Commit Graph

966 Commits

Author SHA1 Message Date
Felix Kunde 5f3f6988bc
add logical-backup build and push to delivery.yaml (#1259)
* add logical-backup build and push to delivery.yaml
* enable manual approval for UI and logical-backup
2020-12-17 12:07:40 +01:00
Felix Kunde 636ba9b846
[UI] reflect new backup paths and cluster status (#1260)
* [UI] reflect new backup paths and cluster status
2020-12-16 15:23:06 +01:00
Pavel Tumik 77252e316c
Add node affinity support (#1166)
* Adding nodeaffinity support alongside node_readiness_label
* add documentation for node affinity
* add node affinity e2e test
* add unit test for node affinity

Co-authored-by: Steffen Pøhner Henriksen <str3sses@gmail.com>
Co-authored-by: Adrian Astley <adrian.astley@activision.com>
2020-12-16 14:56:28 +01:00
Rafia Sabih f28706e940
Sync sts at pgversion upgrade (#1256)
When pgversion is updated to a higher major version number, sync statefulSets also.

Co-authored-by: Rafia Sabih <rafia.sabih@zalando.de>
2020-12-16 13:50:24 +01:00
Pavel Tumik 5076e669cb
Fix timestamp regex (#1178) 2020-12-16 11:17:08 +01:00
Enno Boland 4b90809ade
helm-chart: allow configmaps instead of endpoints for leader elections… (#1037)
* helm-chart: allow configmaps instead of endpoints if leader elections uses the configmaps method

* helm-chart: allow endpoints get even if config maps are used

* helm-chart: allow configmaps instead of endpoints on the operator role too.

Co-authored-by: Enno Boland <mail@eboland.de>
2020-12-16 10:44:25 +01:00
Pavel Tumik fbd04896c2
Add ability to upload logical backup to gcs (#1173)
Support logical backup provider/storage S3 and GCS equivalent
2020-12-16 10:41:08 +01:00
Felix Kunde 929075814a
diff SecurityContext of containers (#1255)
* diff SecurityContext of containers
* change log messages to use "does not" vs "doesn't"
2020-12-15 10:06:53 +01:00
Felix Kunde 83fbccac5a
new env var for backwards compatability between spilo 12 and 13 (#1254) 2020-12-14 18:43:53 +01:00
Felix Kunde 028f23eec7
raise pooler image and fix pgversion config in chart (#1253)
* raise pooler image and fix pgversion config in chart
* enable_ebs_gp3_migration_max_size with quotes
* set ConnectionPoolerMinInstances to 1
2020-12-14 12:37:09 +01:00
Jan Mussler b88d8e34e1
Fix function name in test (#1250)
* Fix function name in test

Error was somehow introduced in last 2 PRs merged.

* Update volumes_test.go
2020-12-12 00:35:27 +01:00
Felix Kunde 6a97316a69
Support inherited annotations for all major objects (#1236)
* add comments where inherited annotations could be added

* add inheritedAnnotations feature

* return nil if no annotations are set

* minor changes

* first downscaler then inherited annotations

* add unit test for inherited annotations

* add pvc to test + minor changes

* missing comma

* fix nil map assignment

* set annotations in the same order it is done in other places

* replace acidClientSet with acid getters in K8s client

* more fixes on clientSet vs getters

* minor changes

* remove endpoints from annotation test

* refine unit test - but deployment and sts are still empty

* fix checkinng sts and deployment

* make annotations setter one liners

* no need for len check anymore

Co-authored-by: Rafia Sabih <rafia.sabih@zalando.de>
2020-12-11 16:34:01 +01:00
Jan Mussler 549f71bb49
Support EBS gp2 to gp3 migration on sync for below 1tb volumes (#1242)
* initial commit for gp3 migration.

* Default volume migration done.

* Added Gomock and one test case with mock.

* Dep update.

* more changes for code gen.

* push fake package.

* Rename var.

* Changes to Makefile and return value.

* Macke mocks phony due to overlap in foldername.

* Learning as one goes. Initialize map.

* Wrong toggle.

* Expect modify call.

* Fix mapping of ids in test.

* Fix volume id.

* volume ids.

* Fixing test setup. Late night...

* create all pvs.

* Fix test case config.

* store volumes and compare.

* More logs.

* Logging of migration action.

* Ensure to log errors.

* Log warning if modify failed, e.g. due to ebs volume state.

* Add more output.

* Skip local e2e tests.

* Reflect k8s volume id in test data. Extract aws volume id from k8s value.

* Finalizing ebs migration.

* More logs. describe fails.

* Fix non existing fields in gp2 discovery.

* Remove nothing to do flag for migration.

* Final commit for migration.

* add new options to all places

Co-authored-by: Felix Kunde <felix-kunde@gmx.de>
2020-12-11 15:52:32 +01:00
Igor Yanchenko b9ef88f842
replace AdditionalProperties with XPreserveUnknownFields FIXES #1206 (#1248) 2020-12-10 16:27:03 +01:00
Thunderbolt 598c05b64b
add to postgresql_type.go omitempty annotation (#1223)
* add to postgresql_type.go omitempty annotation

* add postgresql_type.go additional omitempty

* remove postgresql.go defaults
2020-12-10 14:04:03 +01:00
Rafia Sabih 5a6da7275f
avoid hard-codeed spilo-role (#1246)
Co-authored-by: Rafia Sabih <rafia.sabih@zalando.de>
2020-12-09 13:00:06 +01:00
Sergey Dudoladov dc9a5b1e61
Introduce PGVERSION (#1172)
* introduce PGVERSION

Co-authored-by: Sergey Dudoladov <sergey.dudoladov@zalando.de>
2020-11-27 18:49:49 +01:00
Sergey Dudoladov 6f5751fe55
raise log level for malformed secrets (#1235)
Co-authored-by: Sergey Dudoladov <sergey.dudoladov@zalando.de>
2020-11-27 18:47:50 +01:00
Felix Kunde 2b5382edf3
Add PR template (#1234)
* Add PR template

* move to github folder and rename file

* add headlines
2020-11-27 15:40:35 +01:00
Boyan Bonev 85d1a72cd6
Add scheduler name support - [Update #990] (#1226)
* Add ability to specify alternative schedulers via schedulerName.

Co-authored-by: micah.coletti@gmail.com <micah.coletti@gmail.com>
2020-11-25 10:55:05 +01:00
Felix Kunde cfd83e33c8
preserving fields only when using k8s specs (#1228)
* preserving fields when k8s specs are used with x-kubernetes-preserve-unknown-fields flag
* cleaning up merge errors in postgresql and operatorconfiguration CRD
* add operatorconfiguration CRD and sample manifests in setUpClass of e2e tests
* update generated code and go modules
2020-11-24 16:23:22 +01:00
Jan Mussler c4ae11629b
Fix connection pooler deployment selectors (#1213)
Stick with the existing pooler deployment selector labels to make it compatible with existing deployments.

Make the use of additional labels clear and avoid where not needed.

Deployment Selector and Service Selector now do not use extra labels, pod spec does.
2020-11-23 17:18:18 +01:00
Thunderbolt 580883bc59
add operator ui helm chart value imagePullSecret (#1211) 2020-11-17 10:47:35 +01:00
Jan Mussler 67d1b4b167
compile coverage report and add badges. (#1218)
Add GH badges, and added coveralls support again.
2020-11-16 12:08:28 +01:00
Jan Mussler 65d1a71cc9
Update README.md (#1217)
Removed all the badges, they don't work anyways and are off.
2020-11-16 10:30:18 +01:00
Felix Kunde 3e42e8a896
CRD: preserve unknown fields and add to all category (#1212)
* CRD: preserve unknown fields and add to all category

* allow Pg13

* left over
2020-11-16 10:29:01 +01:00
Jan Mussler a7f453352a
Use Github.com actions to run tests and e2e tests. (#1215)
* Use GH action to run tests and end 2 end tests.
* Remove travis.
2020-11-16 10:15:47 +01:00
Rafia Sabih 49158ecb68
Connection pooler for replica (#1127)
* Enable connection pooler for replica
* Refactor code for connection pooler
  - Move all the relevant code to a separate file
  - Move all the related tests to a separate file
  - Avoid using cluster where not required
  - Simplify the logic in sync and other methods
  - Cleanup of duplicated or unused code
* Fix labels for the replica pods
* Update deleteConnectionPooler to include role
* Adding test cases and other changes
   - Fix unit test and delete secret when required only
   - Make sure we use empty fresh cluster for every test case.
* enhance e2e test
* Disable pooler in complete manifest as this is source for e2e too an creates unnecessary pooler setups.

Co-authored-by: Rafia Sabih <rafia.sabih@zalando.de>
Co-authored-by: Jan Mussler <janm81@gmail.com>
2020-11-13 14:52:21 +01:00
Felix Kunde 3fed565328
check resize mode on update events (#1194)
* check resize mode on update events

* add unit test for PVC resizing

* set resize mode to pvc in charts and manifests

* add test for quantityToGigabyte

* just one debug line for syncing volumes

* extend test and update log msg
2020-11-11 13:22:43 +01:00
Sergey Dudoladov e779eab22f
Update e2e pipeline (#1202)
* clean up after test_multi_namespace test

* see the PR description for complete list of changes

Co-authored-by: Sergey Dudoladov <sergey.dudoladov@zalando.de>
2020-11-11 10:21:46 +01:00
Felix Kunde b379db20ed
fix redundant appending of infrastructure roles (#1192) 2020-11-05 12:04:51 +01:00
Felix Kunde 9a824c38f4
fix identation in operatorconfiguration CRD and jsonPath case (#1195)
* fix identation in operatorconfiguration CRD

* fix jsonPath field case
2020-11-05 11:49:24 +01:00
Jan Mussler 90799d7e7f
More output from test watch script. All namespaces and deployments. (#1193) 2020-11-04 16:40:15 +01:00
Pavel Tumik db0d089e75
Fix cloning from GCS (#1176)
* Fix clone from gcs

* pass google credentials env var if using GS bucket

* remove requirement for timezone as GCS returns timestamp in local time to the region it is in

* Revert "remove requirement for timezone as GCS returns timestamp in local time to the region it is in"

This reverts commit ac4eb350d9.

* update GCS documentation

* remove sentence about logical backups

* reword pod environment configmap section

* fix documentation
2020-11-03 15:05:44 +01:00
Sergey Dudoladov 4f3bb6aa8c
Remove operator checks that prevent PG major version upgrade (#1160)
* remove checks that prevent major version upgrade

Co-authored-by: Sergey Dudoladov <sergey.dudoladov@zalando.de>
2020-11-02 16:49:29 +01:00
Felix Kunde d76419565b
move to apiextensions from v1beta1 to v1 (#746)
* move to apiextensions from v1beta1 to v1

* remove metadata from CRD validation

* some forgotten change
2020-11-02 10:49:29 +01:00
Jan Mussler 7f7beba66b
Improving e2e more (#1185)
* Add curl to operator image.

* Wait for idle operator in delete.
2020-10-29 13:59:22 +01:00
Jan Mussler c694a72352
Make failure in retry a warning not an error. (#1188) 2020-10-29 13:12:25 +01:00
Felix Kunde 9a11e85d57
disable PostgresTeam by default (#1186)
* disable PostgresTeam by default

* fix version in chart
2020-10-28 17:51:37 +01:00
Jakub Warczarek e10e0fec9e
Add support in UI for custom S3 endpoints for backups (#1152)
* Support custom S3 endpoint for backups

* Log info about AWS S3 endpoint during start up
2020-10-28 10:56:50 +01:00
Felix Kunde d658b9672e
PostgresTeam CRD for advanced team management (#1165)
* PostgresTeamCRD for advanced team management

* rework internal structure to be closer to CRD

* superusers instead of admin

* add more util functions and unit tests

* fix initHumanUsers

* check for superusers when creating normal teams

* polishing and fixes

* adding the essential missing pieces

* add documentation and update rbac

* reflect some feedback

* reflect more feedback

* fixing debug logs and raise QueueResyncPeriodTPR

* add two more flags to disable CRD and its superuser support

* fix chart

* update go modules

* move to client 1.19.3 and update codegen
2020-10-28 10:40:10 +01:00
Jan Mussler 3a86dfc8bb
End 2 End tests speedup (#1180)
* Improving end 2 end tests, especially speed of execution and error, by implementing proper eventual asserts and timeouts.
* Add documentation for running individual tests
* Fixed String encoding in Patorni state check and error case
* Printing config as multi log line entity, makes it readable and grepable on startup
* Cosmetic changes to logs. Removed quotes from diff. Move all object diffs to text diff. Enabled padding for log level.
* Mount script with tools for easy logaccess and watching objects.
* Set proper update strategy for Postgres operator deployment.
* Move long running test to end. Move pooler test to new functions.
* Remove quote from valid K8s identifiers.
2020-10-28 10:04:33 +01:00
arminfelder 7730ecfdec
fixed case where, no ready label is defined, but node is unscheduable (#1162)
* fixed case where, no ready label is defined, but node is unscheduable
2020-10-28 09:33:52 +01:00
Felix Kunde e97235aa39
update dependencies oct 2020 (#1184)
* update dependencies oct 2020

* update codegen
2020-10-27 16:59:26 +01:00
preved911 d9f5d1c9df
changed PodEnvironmentSecret location namespace (#1177)
Signed-off-by: Ildar Valiullin <preved.911@gmail.com>
2020-10-22 08:49:30 +02:00
Felix Kunde 22fa0875e2
add maxLength constraint for CRD (#1175)
* add maxLength constraint for CRD
2020-10-22 08:44:04 +02:00
刘新 a8bfe4eb87
Remove repeated initialization of Pod ServiceAccount (#1164)
Co-authored-by: xin.liu <xin.liu@woqutech.com>
2020-10-20 14:18:22 +02:00
Dmitry Dolgov 1f5d0995a5
Lookup function installation (#1171)
* Lookup function installation

Due to reusing a previous database connection without closing it, lookup
function installation process was skipping the first database in the
list, installing twice into postgres db instead. To prevent that, make
internal initDbConnWithName to overwrite a connection object, and return
the same object only from initDbConn, which is sort of public interface.

Another solution for this would be to modify initDbConnWithName to
return a connection object and then generate one temporary connection
for each db. It sound feasible but after one attempt it seems it
requires a bit more changes around (init, close connections) and
doesn't bring anything significantly better on the table. In case if
some future changes will prove this wrong, do not hesitate to refactor.

Change retry strategy to more insistive one, namely:

* retry on the next sync even if we failed to process one database and
install pooler appliance.

* perform the whole installation unconditionally on update, since the
list of target databases could be changed.

And for the sake of making it even more robust, also log the case when
operator decides to skip installation.

Extend connection pooler e2e test with verification that all dbs have
required schema installed.
2020-10-19 16:18:58 +02:00
Dmitry Dolgov d15f2d3392
Readiness probe (#1169)
Right now there are no readiness probes defined for connection pooler,
which means after a pod restart there is a short time window (between a
container start and connection pooler starting listening to a socket)
when a service can send queries to a new pod, but connection will be
refused. The pooler container is rather lightweight and it start to
listen immediately, so the time window is small, but still.

To fix this add a readiness probe for tcp socket opened by connection
pooler.
2020-10-15 10:16:42 +02:00
Alex Stockinger 692c721854
Introduce ENABLE_JSON_LOGGING env variable (#1158) 2020-10-08 15:32:15 +02:00