This commit adds support of a not-yet-released Patroni feature that allows postgres to run as primary in case of a failed leader lock update.
* Add Patroni 'failsafe_mode' local parameter (enable for a single PG cluster)
* Allow configuring Patroni 'failsafe_mode' parameter globally
* add toggle to turn off readiness probes
* include PodManagementPolicy and ReadinessProbe in stateful set comparison
* add URI scheme to generated readiness probe
* create streams only after postgres instances were restarted
* checkAndSetGlobalPostgreSQLConfiguration returns if config has been patched
* restart can be pending even without a config patch
* deprecate ClusterName field of Postgresql type
* remove for teamId from operator API endpints /status /logs /history
* update dns_format_string and yaml template in UI
* Use getSwitchoverCandidate instead of masterCandidate when trying to migrating master pod to a replica
Ref: #1983
* Remove unused masterCandidate (replaced by getSwitchoverCandidate)
Ref: #1983
* allow in place pw rotation of system users
* block postgres user from rotation
* mark pooler pods for replacement
* adding podsGetter where pooler is synced in unit tests
* move rotation code in extra function
* removing inner goroutine in cluster.Switchover and resolve race between processPodEvent and unregisterPodSubscriber
* unlock mutex after handling event, now with non-blocking default case
* reverse membership for additional owner roles
* remove type RoleOriginSpilo
* use e2e images with cron_admin inside
* let operator resolve reversed membership
* make additional owner roles part of the sync user strategy
* add more context in the docs about additional_owner_roles
* return err if teams API fails with StatusCode other than 404
* add unit test for 404 at team members
Co-authored-by: Jociele Padilha <jociele.padilha@zalando.de>
Co-authored-by: Felix Kunde <felix-kunde@gmx.de>
* add the possibility to create a standby cluster that streams from a remote primary
* extending unit tests
* add more docs and e2e test
Co-authored-by: machine424 <ayoubmrini424@gmail.com>
* return only warning if team can't be found
Co-authored-by: Jociele Padilha <jociele.padilha@zalando.de>
Co-authored-by: Felix Kunde <felix-kunde@gmx.de>
* do not create endpoints when use config maps
* delete cluster objects with 'leader' suffix
Co-authored-by: Евграфов Александр Александрович <aevgrafov@cmx.ru>
* feat: add ignored annotations when comparing during sync
Co-authored-by: Felix Kunde <felix-kunde@gmx.de>
Co-authored-by: Moshe Immerman <moshe@flanksource.com>
In Go, when a struct field is not set, it becomes a struct with
default values for all fields. These default values are included
during serialization. This causes issues with schema validation
where optional fields cannot be omitted because default values
are considered invalid.
This patch addresses this issue for `Resources` fields on several
types by using a pointer value.
* Add support for pooler load balancer
Signed-off-by: Sergey Shatunov <me@prok.pw>
* Rename to enable_master_pooler_load_balancer
Signed-off-by: Sergey Shatunov <me@prok.pw>
* target port should be intval
* enhance pooler e2e test
* add new options to crds.go
Co-authored-by: Sergey Shatunov <me@prok.pw>
* Add optional logical backup retention time
* Set defaults for potentially unbound variables, so that the script will work with older operator versions
* Document retention time parameter for logical backups
* Add retention time parameter to resources and charts
Co-authored-by: Felix Kunde <felix-kunde@gmx.de>
* provide event stream API
* check manifest settings for logical decoding before creating streams
* operator updates Postgres config and creates replication user
* name FES like the Postgres cluster
* add delete case and fix updating streams + update unit test
* check if fes CRD exists before syncing
* existing slot must use the same plugin
* make id and payload columns configurable
* sync streams only when they are defined in manifest
* introduce applicationId for separate stream CRDs
* add FES to RBAC in chart
* disable streams in chart
* switch to pgoutput plugin and let operator create publications
* reflect code review and additional refactoring
Co-authored-by: Paŭlo Ebermann <paul.ebermann@zalando.de>
* synchronous_node_count support
* notification about Patroni image version
* default synchronous_node_count to 1
Co-authored-by: Felix Kunde <felix-kunde@gmx.de>
* do not recreate pods if previous Patroni API calls fail
* move retry reads against Patroni API to pod.go
* remove final failover check in node affinity test
* make test_min_resource_limits more robust
* password rotation in K8s secrets
* add db connection to syncSecrets
* add user retention
* add e2e test
* cleanup on username mismatch if rotation was switched off
* add unit test for syncSecrets + new updateSecret func
* include tolerations in statefulset comparison
* provide alternative merge behavior of nodeSelectorTerms for node readiness label
* add config option to change affinity merge behavior
* reworked e2e tests around node affinity
* Make CRD registration configurable and drop RBAC permissions when CRD registration is disabled
* add generated deep copy functions
Co-authored-by: Damian Peckett <d.peckett_admin@mgmt.innovo-cloud.de>
* Add support for manual gs_wal_path in standby
* Remove separate standby version configuration
* Remove setting standby path via cluster/uid/version
Picking up the version doesn't work reliably without making changes to
Spilo. It's clearer to just specify the full S3/GS bucket path.
Co-authored-by: Felix Kunde <felix-kunde@gmx.de>
* init error arrays correctly
* avoid nilPointer when syncing connectionPooler
* getInfrastructureRoles should return error
* fix unit tests and return type for getInfrastructureRoles
* restart master first in some edge cases
* edge case is when desired is lower than effective
* wait after config patch and restart on sync whenever we see pending_restart
* convert options to int to check decrease and add unit test
* minor update to e2e tests
* wait only after restart not every sync
* using spilo 14 e2e images
* improve Patroni config sync
* collect new and updated slots to patch patroni
* refactor httpGet in Patroni and extend unit tests
* GetMemberData should call the patroni endpoint
* add PATCH test
* remove role from installLookupFunction and run it on database sync, too
* fix condition to decide on syncing pooler
* trigger lookup from database sync only if pooler is set
* use empty spec everywhere and do not sync if one lookupfunction was passed
* do not sync pooler after being disabled
This commit adds support for using an Azure storage account as a backup
location.
It uses the existing GCS functionality as a reference for what to do,
and follows the example set by GCS as closely as possible.
The decision to name the cloud provider key "aws_or_gcp" is unfortunate
while adding support for Azure, but I have left it alone to allow for
this changeset to be backwards compatible.
* refactor restarting instances and reduce listPods calls
* only add parameters to set if it differs from effective config
* update e2e test for updating Postgres config
* patch config only once
* reorder e2e tests to follow alphabetical sorting
* e2e: finish waiting for pod failover only if all pods were replaced
* wait for sync in rolling update timeout test
* restart instances via rest api instead of recreating pods
* Ignore differences in bootstrap.dcs when compare SPILO_CONFIGURATION
* isBootstrapOnlyParameter is rewritten, instead of whitelist it uses blacklist
* added e2e test for max_connections decreasing
* documentation updated
* pending_restart flag added to restart api call, wait fot ttl seconds after restart
* refactoring, /restart returns error if pending_restart is set to true and patroni is not pending restart
* restart postgresql instances within pods only if pod's restart is not required
* patroni might need to restart postgresql after pods were recreated if values like max_connections decreased
* instancesRestart is not critical, try to restart pods if not successful
* cleanup
Co-authored-by: Felix Kunde <felix-kunde@gmx.de>
* Create cross namespace secrets
* add test cases
* fixes
* Fixes
- include namespace in secret name only when namespace is provided
- use username.namespace as key to pgUsers only when namespace is
provided
- avoid conflict in the role creation in db by checking namespace
alongwith the username
* Update unit tests
* Fix test case
* Fixes
- update regular expression for usernames
- add test to allow check for valid usernames
- create pg roles with namespace (if any) appended in rolename
* add more test cases for valid usernames
* update docs
* fixes as per review comments
* update e2e
* fixes
* Add toggle to allow namespaced secrets
* update docs
* comment update
* Update e2e/tests/test_e2e.py
* few minor fixes
* fix unit tests
* fix e2e
* fix e2e attempt 2
* fix e2e
Co-authored-by: Rafia Sabih <rafia.sabih@zalando.de>
Co-authored-by: Felix Kunde <felix-kunde@gmx.de>
* Mount additional volumes to 'postgres' container when 'targetContainers' is an empty list
Co-authored-by: Felix Kunde <felix-kunde@gmx.de>
Co-authored-by: Felix Kunde <felix-kunde@gmx.de>
* rename db roles that are removed from manifests
* extend PostgresTeam e2e test
* make suffix configurable and add deprecated field to pgUser struct
* deny LOGIN from deprecated roles
* update feature documentation
* replace statefulset on annotation diff
* remove update annotation function for statefulset
* add unit test for syncing annotations
* add inherited annotation to unit test
* helm chart remove 1.6.0 archive from 1.6.0 archive
* bump operator to v1.6.2
* fix pointer deref
* skip connection pooler sync when empty
* revert pooler change and minor update to version msg
* do not log query on error when creating or altering users
* Fix for AllowPrivilegeEscalation : issue-1403
* fixed syntax error
* Aligned the value for parameter
* Aligned the value for parameter
* Update crds.go
* Aligned the parameter spilo_allow_privilege_escalation
* Parameters sorted in Alphabetical order in manifests yaml
* Parameters sorted in Alphabetical order in manifests yaml
* Update pkg/controller/operator_config.go
* Update docs/reference/operator_parameters.md
Co-authored-by: Neelam Sharma <neelasha@amdocs.com>
Co-authored-by: Felix Kunde <felix-kunde@gmx.de>
* add TODOs for moving rooling update label on pods
* steer rolling update via pod annotation
* rename patch method and fix reading flag on sync
* pass only pods to recreatePods function
* do not take address of iterator if you use it later
* add e2e test and pass switchover targets to recreatePods
* add wait_for_pod_failover for e2e test
* add one more e2e test case
* helm chart remove 1.6.0 archive from 1.6.0 archive
* reflect code review feedback
Support major version upgrade trigger via manifest. There is `off` `manual` and `full`. Manual is what you expect, and full will auto upgrade clusters below a certain threshold.
* making pgTeamMap a pointer
* init empty map
* add e2e test for additional teams and members
* update test_min_resource_limits
* add more waiting in node_affinity_test
* no need for pointers in map of postgresTeamMebership
* another minor update on node affinity test
* refactor and fix fetching additional members
* pre-allocate cap for slice structure
* if clause is no need because of range, and kubelet also use range
method to get each capability so there is no side-effect
Signed-off-by: Jeff Zvier <zvier20@gmail.com>
Avoid extra syncing in case there are no changes in pooler requirements.
Add pooler specific labels to pooler secrets.
Add test case to check for pooler secret creation and deletion.
Co-authored-by: Rafia Sabih <rafia.sabih@zalando.de>
* add default values to operatorconfiguration crd
* leave default for enable_master_load_balancer to true
* add missing bits for new logical backup option
* fix wrong lb tag and update chart package
* bump tp v1.6.0
* update logical-backup image
* Using smaller image for e2e test.
* fix env var name in docs
* add postgresql-client-13 to logical backup image
Co-authored-by: Jan Mussler <janm81@gmail.com>
* Initial commit for new 1.6 release with Postgres 13 support.
* Updating maintainers, Go version, Codeowners.
* Use lazy upgrade image that contains pg13.
* fix typo for ownerReference
* fix clusterrole in helm chart
* reflect GCP logical backup in validation
* improve PostgresTeam docs
* change defaults for enable_pgversion_env_var and storage_resize_mode
* explain manual part of in-place upgrade
* remove gsoc docs
Co-authored-by: Felix Kunde <felix-kunde@gmx.de>
* Adding nodeaffinity support alongside node_readiness_label
* add documentation for node affinity
* add node affinity e2e test
* add unit test for node affinity
Co-authored-by: Steffen Pøhner Henriksen <str3sses@gmail.com>
Co-authored-by: Adrian Astley <adrian.astley@activision.com>
* add comments where inherited annotations could be added
* add inheritedAnnotations feature
* return nil if no annotations are set
* minor changes
* first downscaler then inherited annotations
* add unit test for inherited annotations
* add pvc to test + minor changes
* missing comma
* fix nil map assignment
* set annotations in the same order it is done in other places
* replace acidClientSet with acid getters in K8s client
* more fixes on clientSet vs getters
* minor changes
* remove endpoints from annotation test
* refine unit test - but deployment and sts are still empty
* fix checkinng sts and deployment
* make annotations setter one liners
* no need for len check anymore
Co-authored-by: Rafia Sabih <rafia.sabih@zalando.de>
* initial commit for gp3 migration.
* Default volume migration done.
* Added Gomock and one test case with mock.
* Dep update.
* more changes for code gen.
* push fake package.
* Rename var.
* Changes to Makefile and return value.
* Macke mocks phony due to overlap in foldername.
* Learning as one goes. Initialize map.
* Wrong toggle.
* Expect modify call.
* Fix mapping of ids in test.
* Fix volume id.
* volume ids.
* Fixing test setup. Late night...
* create all pvs.
* Fix test case config.
* store volumes and compare.
* More logs.
* Logging of migration action.
* Ensure to log errors.
* Log warning if modify failed, e.g. due to ebs volume state.
* Add more output.
* Skip local e2e tests.
* Reflect k8s volume id in test data. Extract aws volume id from k8s value.
* Finalizing ebs migration.
* More logs. describe fails.
* Fix non existing fields in gp2 discovery.
* Remove nothing to do flag for migration.
* Final commit for migration.
* add new options to all places
Co-authored-by: Felix Kunde <felix-kunde@gmx.de>
* preserving fields when k8s specs are used with x-kubernetes-preserve-unknown-fields flag
* cleaning up merge errors in postgresql and operatorconfiguration CRD
* add operatorconfiguration CRD and sample manifests in setUpClass of e2e tests
* update generated code and go modules
Stick with the existing pooler deployment selector labels to make it compatible with existing deployments.
Make the use of additional labels clear and avoid where not needed.
Deployment Selector and Service Selector now do not use extra labels, pod spec does.
* Enable connection pooler for replica
* Refactor code for connection pooler
- Move all the relevant code to a separate file
- Move all the related tests to a separate file
- Avoid using cluster where not required
- Simplify the logic in sync and other methods
- Cleanup of duplicated or unused code
* Fix labels for the replica pods
* Update deleteConnectionPooler to include role
* Adding test cases and other changes
- Fix unit test and delete secret when required only
- Make sure we use empty fresh cluster for every test case.
* enhance e2e test
* Disable pooler in complete manifest as this is source for e2e too an creates unnecessary pooler setups.
Co-authored-by: Rafia Sabih <rafia.sabih@zalando.de>
Co-authored-by: Jan Mussler <janm81@gmail.com>
* check resize mode on update events
* add unit test for PVC resizing
* set resize mode to pvc in charts and manifests
* add test for quantityToGigabyte
* just one debug line for syncing volumes
* extend test and update log msg
* clean up after test_multi_namespace test
* see the PR description for complete list of changes
Co-authored-by: Sergey Dudoladov <sergey.dudoladov@zalando.de>
* Fix clone from gcs
* pass google credentials env var if using GS bucket
* remove requirement for timezone as GCS returns timestamp in local time to the region it is in
* Revert "remove requirement for timezone as GCS returns timestamp in local time to the region it is in"
This reverts commit ac4eb350d9.
* update GCS documentation
* remove sentence about logical backups
* reword pod environment configmap section
* fix documentation
* PostgresTeamCRD for advanced team management
* rework internal structure to be closer to CRD
* superusers instead of admin
* add more util functions and unit tests
* fix initHumanUsers
* check for superusers when creating normal teams
* polishing and fixes
* adding the essential missing pieces
* add documentation and update rbac
* reflect some feedback
* reflect more feedback
* fixing debug logs and raise QueueResyncPeriodTPR
* add two more flags to disable CRD and its superuser support
* fix chart
* update go modules
* move to client 1.19.3 and update codegen
* Improving end 2 end tests, especially speed of execution and error, by implementing proper eventual asserts and timeouts.
* Add documentation for running individual tests
* Fixed String encoding in Patorni state check and error case
* Printing config as multi log line entity, makes it readable and grepable on startup
* Cosmetic changes to logs. Removed quotes from diff. Move all object diffs to text diff. Enabled padding for log level.
* Mount script with tools for easy logaccess and watching objects.
* Set proper update strategy for Postgres operator deployment.
* Move long running test to end. Move pooler test to new functions.
* Remove quote from valid K8s identifiers.
* Lookup function installation
Due to reusing a previous database connection without closing it, lookup
function installation process was skipping the first database in the
list, installing twice into postgres db instead. To prevent that, make
internal initDbConnWithName to overwrite a connection object, and return
the same object only from initDbConn, which is sort of public interface.
Another solution for this would be to modify initDbConnWithName to
return a connection object and then generate one temporary connection
for each db. It sound feasible but after one attempt it seems it
requires a bit more changes around (init, close connections) and
doesn't bring anything significantly better on the table. In case if
some future changes will prove this wrong, do not hesitate to refactor.
Change retry strategy to more insistive one, namely:
* retry on the next sync even if we failed to process one database and
install pooler appliance.
* perform the whole installation unconditionally on update, since the
list of target databases could be changed.
And for the sake of making it even more robust, also log the case when
operator decides to skip installation.
Extend connection pooler e2e test with verification that all dbs have
required schema installed.
Right now there are no readiness probes defined for connection pooler,
which means after a pod restart there is a short time window (between a
container start and connection pooler starting listening to a socket)
when a service can send queries to a new pod, but connection will be
refused. The pooler container is rather lightweight and it start to
listen immediately, so the time window is small, but still.
To fix this add a readiness probe for tcp socket opened by connection
pooler.
* post polishing for latest PRs
* update travis and go modules
* make deprecation comments in structs less confusing
* have separate pod priority class es for operator and database pods
* allow using both infrastructure_roles_options
* new default values for user and role definition
* use robot_zmon as parent role
* add operator log to debug
* right name for old secret
* only extract if rolesDefs is empty
* set password1 in old infrastructure role
* fix new infra rile secret
* choose different role key for new secret
* set memberof everywhere
* reenable all tests
* reflect feedback
* remove condition for rolesDefs
Felix Kunde
Polina Bungina
yoshihikoueno
machine424
Dmitry Volodin
Philipp B
Jan Mussler
JBWatenbergScality
Jociele Padilha
mujx
Oleg
neelasha-09
evsasha
preved911
Jakob Gillich
A. Stoewer
david amick
Maksim Zhylinski
Menzorg
Damian Peckett
James McDonald
Rafia Sabih
Wolfgang Ebner
Aaron Peschel
John Rood
Quan Hoang
Sergey Dudoladov
Igor Yanchenko
georgebarbarosie
Michael Seiwald
Tommaso Pozzetti
zvier
Pavel Tumik
Pavel Tumik
Thunderbolt
Boyan Bonev
arminfelder
刘新
Dmitry Dolgov
Alex Stockinger
Rico Berger
hlihhovac