Commit Graph
420 Commits
Author SHA1 Message Date
9e93c0a4ef Fix for AllowPrivilegeEscalation : issue-1403 (#1412)
* Fix for AllowPrivilegeEscalation : issue-1403

* fixed syntax error

* Aligned the value for parameter

* Aligned the value for parameter

* Update crds.go

* Aligned the parameter spilo_allow_privilege_escalation

* Parameters sorted in Alphabetical order in manifests yaml

* Parameters sorted in Alphabetical order in manifests yaml

* Update pkg/controller/operator_config.go

* Update docs/reference/operator_parameters.md

Co-authored-by: Neelam Sharma <neelasha@amdocs.com>
Co-authored-by: Felix Kunde <felix-kunde@gmx.de>
2021-03-29 10:37:59 +02:00
machine424 78bfba85d2 create global default privileges in the appropriate prepared databases (#1421) 2021-03-26 14:19:26 +01:00
Felix Kunde c9acd52700 Major version upgrade config (#1386)
* reflect new major version upgrade options everywhere

* emit events during major version upgrade
2021-03-09 15:28:15 +01:00
Felix Kunde ff8143770c Improve rolling upgrades and rolling upgrade continue (#1341)
* add TODOs for moving rooling update label on pods
* steer rolling update via pod annotation
* rename patch method and fix reading flag on sync
* pass only pods to recreatePods function
* do not take address of iterator if you use it later
* add e2e test and pass switchover targets to recreatePods
* add wait_for_pod_failover for e2e test
* add one more e2e test case
* helm chart remove 1.6.0 archive from 1.6.0 archive
* reflect code review feedback
2021-02-26 15:38:58 +01:00
Jan Mussler e837751ae0 Log result. (#1387) 2021-02-26 14:54:26 +01:00
Jan Mussler 636a9a8191 Support major version upgrade via manifest and global upgrades via min version (#1372)
Support major version upgrade trigger via manifest. There is `off` `manual` and `full`. Manual is what you expect, and full will auto upgrade clusters below a certain threshold.
2021-02-25 11:42:43 +01:00
Felix Kunde ca968ca150 Fix empty capabilities (#1380)
* helm chart remove 1.6.0 archive from 1.6.0 archive
* empty pod capabilities should be nil
2021-02-22 17:27:32 +01:00
Felix Kunde 41858a702c making pgTeamMap a pointer (#1349)
* making pgTeamMap a pointer
* init empty map
* add e2e test for additional teams and members
* update test_min_resource_limits
* add more waiting in node_affinity_test
* no need for pointers in map of postgresTeamMebership
* another minor update on node affinity test
* refactor and fix fetching additional members
2021-02-16 10:38:20 +01:00
Michael Seiwald 17da6bc649 Truncate cronjob name at 52 characters (#1208) 2021-02-15 17:00:21 +01:00
Jan Mussler 772f0ca771 Fix volume sync order. (#1340) 2021-02-12 17:36:11 +01:00
zvier 6aeb92f024 code optimization (#1350)
* pre-allocate cap for slice structure
* if clause is no need because of range, and kubelet also use range
  method to get each capability so there is no side-effect

Signed-off-by: Jeff Zvier <zvier20@gmail.com>
2021-02-09 09:35:24 +01:00
Felix Kunde 0cce565b65 fix when adding only one capability (#1339)
* fix when adding only one capability

* fix error messages in unit test
2021-01-29 16:10:27 +01:00
Felix Kunde 12ad8c91fa configurable container capabilities (#1336)
* configurable container capabilities

* revert change on TestTLS

* fix e2e test

* minor fix
2021-01-29 14:54:48 +01:00
Jan Mussler 43168ca622 Also sync volumes on updates. (#1330) 2021-01-25 20:28:37 +01:00
Felix Kunde ac2a00c45e set allowPrivilegeEscalation for deployment templates (#1328)
* set allowPrivilegeEscalation for deployment templates

* securityContext of container, not pod

* aligning

* default service account for pooler
2021-01-25 18:23:29 +01:00
Jan MusslerandFelix Kunde 4a88f00a3f Full AWS gp3 support for iops and througput config. (#1261)
Support new AWS EBS volume type `gp3` with `iops` and `throughput` in the manifest.

Co-authored-by: Felix Kunde <felix-kunde@gmx.de>
2021-01-25 10:07:18 +01:00
Felix Kunde 4ea0b5f432 set AllowPrivilegeEscalation on container securityContext (#1326) 2021-01-22 14:06:19 +01:00
Rafia SabihandRafia Sabih a9b677c957 Use fake client for connection pooler (#1301)
Connection pooler creation, deletion, and synchronization now tested using fake client API.

Co-authored-by: Rafia Sabih <rafia.sabih@zalando.de>
2021-01-19 17:40:20 +01:00
Felix Kunde 258799b420 allow additional members from other teams (#1314) 2021-01-15 15:11:02 +01:00
Rafia SabihandRafia Sabih e398cf8c7e Avoid syncing when possible (#1274)
Avoid extra syncing in case there are no changes in pooler requirements.
Add pooler specific labels to pooler secrets.
Add test case to check for pooler secret creation and deletion.

Co-authored-by: Rafia Sabih <rafia.sabih@zalando.de>
2021-01-14 09:53:09 +01:00
Sergey DudoladovandSergey Dudoladov b7f4cde541 wrap getting Patroni state into retry (#1293)
Retry calls to Patorni API to get cluster state

Co-authored-by: Sergey Dudoladov <sergey.dudoladov@zalando.de>
2021-01-08 15:08:44 +01:00
Sergey DudoladovandSergey Dudoladov 168b679506 add a prefix for the name of a logical backup job (#1287)
* add a prefix for the name of a logical backup job

Co-authored-by: Sergey Dudoladov <sergey.dudoladov@zalando.de>
2021-01-07 10:38:07 +01:00
Felix Kunde 07c4f52ede use pointer type for nodeAffinity (#1263)
* use pointer type for nodeAffinity
2020-12-18 12:07:59 +01:00
Jan MusslerandFelix Kunde a63ad49ef8 Initial commit for new 1.6 release with Postgres 13 support. (#1257)
* Initial commit for new 1.6 release with Postgres 13 support.
* Updating maintainers, Go version, Codeowners.
* Use lazy upgrade image that contains pg13.
* fix typo for ownerReference
* fix clusterrole in helm chart
* reflect GCP logical backup in validation
* improve PostgresTeam docs
* change defaults for enable_pgversion_env_var and storage_resize_mode
* explain manual part of in-place upgrade
* remove gsoc docs

Co-authored-by: Felix Kunde <felix-kunde@gmx.de>
2020-12-17 15:00:29 +01:00
77252e316c Add node affinity support (#1166)
* Adding nodeaffinity support alongside node_readiness_label
* add documentation for node affinity
* add node affinity e2e test
* add unit test for node affinity

Co-authored-by: Steffen Pøhner Henriksen <str3sses@gmail.com>
Co-authored-by: Adrian Astley <adrian.astley@activision.com>
2020-12-16 14:56:28 +01:00
Rafia SabihandRafia Sabih f28706e940 Sync sts at pgversion upgrade (#1256)
When pgversion is updated to a higher major version number, sync statefulSets also.

Co-authored-by: Rafia Sabih <rafia.sabih@zalando.de>
2020-12-16 13:50:24 +01:00
Pavel Tumik fbd04896c2 Add ability to upload logical backup to gcs (#1173)
Support logical backup provider/storage S3 and GCS equivalent
2020-12-16 10:41:08 +01:00
Felix Kunde 929075814a diff SecurityContext of containers (#1255)
* diff SecurityContext of containers
* change log messages to use "does not" vs "doesn't"
2020-12-15 10:06:53 +01:00
Felix Kunde 83fbccac5a new env var for backwards compatability between spilo 12 and 13 (#1254) 2020-12-14 18:43:53 +01:00
Jan Mussler b88d8e34e1 Fix function name in test (#1250)
* Fix function name in test

Error was somehow introduced in last 2 PRs merged.

* Update volumes_test.go
2020-12-12 00:35:27 +01:00
Felix KundeandRafia Sabih 6a97316a69 Support inherited annotations for all major objects (#1236)
* add comments where inherited annotations could be added

* add inheritedAnnotations feature

* return nil if no annotations are set

* minor changes

* first downscaler then inherited annotations

* add unit test for inherited annotations

* add pvc to test + minor changes

* missing comma

* fix nil map assignment

* set annotations in the same order it is done in other places

* replace acidClientSet with acid getters in K8s client

* more fixes on clientSet vs getters

* minor changes

* remove endpoints from annotation test

* refine unit test - but deployment and sts are still empty

* fix checkinng sts and deployment

* make annotations setter one liners

* no need for len check anymore

Co-authored-by: Rafia Sabih <rafia.sabih@zalando.de>
2020-12-11 16:34:01 +01:00
Jan MusslerandFelix Kunde 549f71bb49 Support EBS gp2 to gp3 migration on sync for below 1tb volumes (#1242)
* initial commit for gp3 migration.

* Default volume migration done.

* Added Gomock and one test case with mock.

* Dep update.

* more changes for code gen.

* push fake package.

* Rename var.

* Changes to Makefile and return value.

* Macke mocks phony due to overlap in foldername.

* Learning as one goes. Initialize map.

* Wrong toggle.

* Expect modify call.

* Fix mapping of ids in test.

* Fix volume id.

* volume ids.

* Fixing test setup. Late night...

* create all pvs.

* Fix test case config.

* store volumes and compare.

* More logs.

* Logging of migration action.

* Ensure to log errors.

* Log warning if modify failed, e.g. due to ebs volume state.

* Add more output.

* Skip local e2e tests.

* Reflect k8s volume id in test data. Extract aws volume id from k8s value.

* Finalizing ebs migration.

* More logs. describe fails.

* Fix non existing fields in gp2 discovery.

* Remove nothing to do flag for migration.

* Final commit for migration.

* add new options to all places

Co-authored-by: Felix Kunde <felix-kunde@gmx.de>
2020-12-11 15:52:32 +01:00
Rafia SabihandRafia Sabih 5a6da7275f avoid hard-codeed spilo-role (#1246)
Co-authored-by: Rafia Sabih <rafia.sabih@zalando.de>
2020-12-09 13:00:06 +01:00
Sergey DudoladovandSergey Dudoladov dc9a5b1e61 Introduce PGVERSION (#1172)
* introduce PGVERSION

Co-authored-by: Sergey Dudoladov <sergey.dudoladov@zalando.de>
2020-11-27 18:49:49 +01:00
Sergey DudoladovandSergey Dudoladov 6f5751fe55 raise log level for malformed secrets (#1235)
Co-authored-by: Sergey Dudoladov <sergey.dudoladov@zalando.de>
2020-11-27 18:47:50 +01:00
Boyan Bonevandmicah.coletti@gmail.com 85d1a72cd6 Add scheduler name support - [Update #990] (#1226)
* Add ability to specify alternative schedulers via schedulerName.

Co-authored-by: micah.coletti@gmail.com <micah.coletti@gmail.com>
2020-11-25 10:55:05 +01:00
Jan Mussler c4ae11629b Fix connection pooler deployment selectors (#1213)
Stick with the existing pooler deployment selector labels to make it compatible with existing deployments.

Make the use of additional labels clear and avoid where not needed.

Deployment Selector and Service Selector now do not use extra labels, pod spec does.
2020-11-23 17:18:18 +01:00
49158ecb68 Connection pooler for replica (#1127)
* Enable connection pooler for replica
* Refactor code for connection pooler
  - Move all the relevant code to a separate file
  - Move all the related tests to a separate file
  - Avoid using cluster where not required
  - Simplify the logic in sync and other methods
  - Cleanup of duplicated or unused code
* Fix labels for the replica pods
* Update deleteConnectionPooler to include role
* Adding test cases and other changes
   - Fix unit test and delete secret when required only
   - Make sure we use empty fresh cluster for every test case.
* enhance e2e test
* Disable pooler in complete manifest as this is source for e2e too an creates unnecessary pooler setups.

Co-authored-by: Rafia Sabih <rafia.sabih@zalando.de>
Co-authored-by: Jan Mussler <janm81@gmail.com>
2020-11-13 14:52:21 +01:00
Felix Kunde 3fed565328 check resize mode on update events (#1194)
* check resize mode on update events

* add unit test for PVC resizing

* set resize mode to pvc in charts and manifests

* add test for quantityToGigabyte

* just one debug line for syncing volumes

* extend test and update log msg
2020-11-11 13:22:43 +01:00
Sergey DudoladovandSergey Dudoladov e779eab22f Update e2e pipeline (#1202)
* clean up after test_multi_namespace test

* see the PR description for complete list of changes

Co-authored-by: Sergey Dudoladov <sergey.dudoladov@zalando.de>
2020-11-11 10:21:46 +01:00
Pavel Tumik db0d089e75 Fix cloning from GCS (#1176)
* Fix clone from gcs

* pass google credentials env var if using GS bucket

* remove requirement for timezone as GCS returns timestamp in local time to the region it is in

* Revert "remove requirement for timezone as GCS returns timestamp in local time to the region it is in"

This reverts commit ac4eb350d9.

* update GCS documentation

* remove sentence about logical backups

* reword pod environment configmap section

* fix documentation
2020-11-03 15:05:44 +01:00
Sergey DudoladovandSergey Dudoladov 4f3bb6aa8c Remove operator checks that prevent PG major version upgrade (#1160)
* remove checks that prevent major version upgrade

Co-authored-by: Sergey Dudoladov <sergey.dudoladov@zalando.de>
2020-11-02 16:49:29 +01:00
Jan Mussler c694a72352 Make failure in retry a warning not an error. (#1188) 2020-10-29 13:12:25 +01:00
Felix Kunde d658b9672e PostgresTeam CRD for advanced team management (#1165)
* PostgresTeamCRD for advanced team management

* rework internal structure to be closer to CRD

* superusers instead of admin

* add more util functions and unit tests

* fix initHumanUsers

* check for superusers when creating normal teams

* polishing and fixes

* adding the essential missing pieces

* add documentation and update rbac

* reflect some feedback

* reflect more feedback

* fixing debug logs and raise QueueResyncPeriodTPR

* add two more flags to disable CRD and its superuser support

* fix chart

* update go modules

* move to client 1.19.3 and update codegen
2020-10-28 10:40:10 +01:00
Jan Mussler 3a86dfc8bb End 2 End tests speedup (#1180)
* Improving end 2 end tests, especially speed of execution and error, by implementing proper eventual asserts and timeouts.
* Add documentation for running individual tests
* Fixed String encoding in Patorni state check and error case
* Printing config as multi log line entity, makes it readable and grepable on startup
* Cosmetic changes to logs. Removed quotes from diff. Move all object diffs to text diff. Enabled padding for log level.
* Mount script with tools for easy logaccess and watching objects.
* Set proper update strategy for Postgres operator deployment.
* Move long running test to end. Move pooler test to new functions.
* Remove quote from valid K8s identifiers.
2020-10-28 10:04:33 +01:00
preved911 d9f5d1c9df changed PodEnvironmentSecret location namespace (#1177)
Signed-off-by: Ildar Valiullin <preved.911@gmail.com>
2020-10-22 08:49:30 +02:00
Dmitry Dolgov 1f5d0995a5 Lookup function installation (#1171)
* Lookup function installation

Due to reusing a previous database connection without closing it, lookup
function installation process was skipping the first database in the
list, installing twice into postgres db instead. To prevent that, make
internal initDbConnWithName to overwrite a connection object, and return
the same object only from initDbConn, which is sort of public interface.

Another solution for this would be to modify initDbConnWithName to
return a connection object and then generate one temporary connection
for each db. It sound feasible but after one attempt it seems it
requires a bit more changes around (init, close connections) and
doesn't bring anything significantly better on the table. In case if
some future changes will prove this wrong, do not hesitate to refactor.

Change retry strategy to more insistive one, namely:

* retry on the next sync even if we failed to process one database and
install pooler appliance.

* perform the whole installation unconditionally on update, since the
list of target databases could be changed.

And for the sake of making it even more robust, also log the case when
operator decides to skip installation.

Extend connection pooler e2e test with verification that all dbs have
required schema installed.
2020-10-19 16:18:58 +02:00
Dmitry Dolgov d15f2d3392 Readiness probe (#1169)
Right now there are no readiness probes defined for connection pooler,
which means after a pod restart there is a short time window (between a
container start and connection pooler starting listening to a socket)
when a service can send queries to a new pod, but connection will be
refused. The pooler container is rather lightweight and it start to
listen immediately, so the time window is small, but still.

To fix this add a readiness probe for tcp socket opened by connection
pooler.
2020-10-15 10:16:42 +02:00
Sergey DudoladovandSergey Dudoladov 2a21cc4393 Compare Postgres pod priority on Sync (#1144)
* compare Postgres pod priority on Sync

Co-authored-by: Sergey Dudoladov <sergey.dudoladov@zalando.de>
2020-09-23 17:26:56 +02:00
neelasha-09andFelix Kunde ab95eaa6ef Fixes #1130 (#1139)
* Fixes #1130

Co-authored-by: Felix Kunde <felix-kunde@gmx.de>
2020-09-22 17:16:05 +02:00