Commit Graph

64 Commits

Author SHA1 Message Date
alfredw33 2b0def5bc8
Support for GCS WAL-E backups (#620)
* Support for WAL_GS_BUCKET and GOOGLE_APPLICATION_CREDENTIALS environtment variables

* Fixed merge issue but also removed all changes to support macos.

* Updated test to new format

* Missed macos specific changes

* Added documentation and addressed comments

* Update docs/administrator.md

* Update docs/administrator.md

* Update e2e/run.sh

Co-authored-by: Felix Kunde <felix-kunde@gmx.de>
2020-06-03 17:33:48 +02:00
Rafia Sabih d52296c323
Propagate annotations to the StatefulSet (#932)
* Initial commit

* Corrections

- set the type of the new  configuration parameter to be array of
  strings
- propagate the annotations to statefulset at sync

* Enable regular expression matching

* Improvements

-handle rollingUpdate flag
-modularize code
-rename config parameter name

* fix merge error

* Pass annotations to connection pooler deployment

* update code-gen

* Add documentation and update manifests

* add e2e test and introduce option in configmap

* fix service annotations test

* Add unit test

* fix e2e tests

* better key lookup of annotations tests

* add debug message for annotation tests

* Fix typos

* minor fix for looping

* Handle update path and renaming

- handle the update path to update sts and connection pooler deployment.
  This way no need to wait for sync
- rename the parameter to downscaler_annotations
- handle other review comments

* another try to fix python loops

* Avoid unneccessary update events

* Update manifests

* some final polishing

* fix cluster_test after polishing

Co-authored-by: Rafia Sabih <rafia.sabih@zalando.de>
Co-authored-by: Felix Kunde <felix-kunde@gmx.de>
2020-05-04 14:46:56 +02:00
Sergey Dudoladov cc635a02e3
Lazy upgrade of the Spilo image (#859)
* initial implementation

* describe forcing the rolling upgrade

* make parameter name more descriptive

* add missing pieces

* address review

* address review

* fix bug in e2e tests

* fix cluster name label in e2e test

* raise test timeout

* load spilo test image

* use available spilo image

* delete replica pod for lazy update test

* fix e2e

* fix e2e with a vengeance

* lets wait for another 30m

* print pod name in error msg

* print pod name in error msg 2

* raise timeout, comment other tests

* subsequent updates of config

* add comma

* fix e2e test

* run unit tests before e2e

* remove conflicting dependency

* Revert "remove conflicting dependency"

This reverts commit 65fc09054b.

* improve cdp build

* dont run unit before e2e tests

* Revert "improve cdp build"

This reverts commit e2a8fa12aa.

Co-authored-by: Sergey Dudoladov <sergey.dudoladov@zalando.de>
Co-authored-by: Felix Kunde <felix-kunde@gmx.de>
2020-04-29 10:07:14 +02:00
Felix Kunde 1d009d9595
bump spilo and pooler version + update docs (#945) 2020-04-28 16:01:13 +02:00
Björn Fischer 168abfe37b
Fully speced global sidecars (#890)
* implement fully speced global sidecars

* fix issue #924
2020-04-27 17:40:22 +02:00
Felix Kunde 64389b8bad
update image and docs for connection pooler (#898) 2020-04-03 16:28:36 +02:00
ReSearchITEng 1249626a60
kubernetes_use_configmap (#887)
* kubernetes_use_configmap

* Update manifests/postgresql-operator-default-configuration.yaml

Co-Authored-By: Felix Kunde <felix-kunde@gmx.de>

* Update manifests/configmap.yaml

Co-Authored-By: Felix Kunde <felix-kunde@gmx.de>

* Update charts/postgres-operator/values.yaml

Co-Authored-By: Felix Kunde <felix-kunde@gmx.de>

* go.fmt

Co-authored-by: Felix Kunde <felix-kunde@gmx.de>
2020-04-02 13:20:45 +02:00
Felix Kunde b43b22dfcc
Call me pooler, not pool (#883)
* rename pooler parts and add example to manifest
* update codegen
* fix manifest and add more details to docs
* reflect renaming also in e2e tests
2020-04-01 10:34:03 +02:00
Felix Kunde ba9cf68650
Change type of pod environment config map to NamespacedName (#870)
* allow PodEnvironmentConfigMap in other namespaces
* update codegen
* update docs and comments
2020-03-25 15:59:31 +01:00
Dmitry Dolgov 9dfa433363
Connection pooler (#799)
Connection pooler support

Add support for a connection pooler. The idea is to make it generic enough to
be able to switch between different implementations (e.g. pgbouncer or
odyssey). Operator needs to create a deployment with pooler and a service for
it to access.

For connection pool to work properly, a database needs to be prepared by
operator, namely a separate user have to be created with an access to an
installed lookup function (to fetch credential for other users).

This setups is supposed to be used only by robot/application users. Usually a
connection pool implementation is more CPU bounded, so it makes sense to create
several pods for connection pool with more emphasize on cpu resources. At the
moment there are no special affinity or tolerations assigned to bring those
pods closer to the database. For availability purposes minimal number of
connection pool pods is 2, ideally they have to be distributed between
different nodes/AZ, but it's not enforced in the operator itself. Available
configuration supposed to be ergonomic and in the normal case require minimum
changes to a manifest to enable connection pool. To have more control over the
configuration and functionality on the pool side one can customize the
corresponding docker image.

Co-authored-by: Felix Kunde <felix-kunde@gmx.de>
2020-03-25 12:57:26 +01:00
Felix Kunde 07c5da35e3
fix minor issues in docs and manifests (#866)
* fix minor issues in docs and manifests
* double retry_timeout_sec
2020-03-18 15:02:13 +01:00
Jonathan Herlin 35b2213e05
Fix typo in values file (#861)
* Fix typo

Co-authored-by: Jonathan Herlin <jonathan.herlin@elits.com>
2020-03-11 11:32:13 +01:00
Felix Kunde b997e3682f
be more permissive with standbys (#842)
* be more permissive with standbys
* reflect feedback and updated docs
2020-02-24 15:14:14 +01:00
Felix Kunde aea9e9bd33
postgres-pod clusterrole (#832)
* define postgres-pod clusterrole and align rbac in chart
* align UI chart rbac with operator and update doc
* operator RBAC needs podsecuritypolicy to grant it to postgres-pod
2020-02-19 12:32:54 +01:00
Jonathan Juares Beber ba60e15d07 Add ServiceAnnotations cluster config (#803)
The [operator parameters][1] already support the
`custom_service_annotations` config.With this parameter is possible to
define custom annotations that will be used on the services created by the
operator. The `custom_service_annotations` as all the other
[operator parameters][1] are defined on the operator level and do not allow
customization on the cluster level. A cluster may require different service
annotations, as for example, set up different cloud load balancers
timeouts, different ingress annotations, and/or enable more customizable
environments.

This commit introduces a new parameter on the cluster level, called
`serviceAnnotations`, responsible for defining custom annotations just for
the services created by the operator to the specifically defined cluster.
It allows a mix of configuration between `custom_service_annotations` and
`serviceAnnotations` where the latest one will have priority. In order to
allow custom service annotations to be used on services without
LoadBalancers (as for example, service mesh services annotations) both
`custom_service_annotations` and `serviceAnnotations` are applied
independently of load-balancing configuration. For retro-compatibility
purposes, `custom_service_annotations` is still under
[Load balancer related options][2]. The two default annotations when using
LoadBalancer services, `external-dns.alpha.kubernetes.io/hostname` and
`service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout` are
still defined by the operator.
`service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout` can
be overridden by `custom_service_annotations` or `serviceAnnotations`,
allowing a more customizable environment.
`external-dns.alpha.kubernetes.io/hostname` can not be overridden once
there is no differentiation between custom service annotations for
replicas and masters.

It updates the documentation and creates the necessary unit and e2e
tests to the above-described feature too.

[1]: https://github.com/zalando/postgres-operator/blob/master/docs/reference/operator_parameters.md
[2]: https://github.com/zalando/postgres-operator/blob/master/docs/reference/operator_parameters.md#load-balancer-related-options
2020-02-10 12:03:25 +01:00
Vito Botta a660d758a5 Add region setting for logical backups to non-AWS storage (#813)
* Add region setting for logical backups to non-AWS storage
2020-02-10 11:48:24 +01:00
Felix Kunde 1f0312a014
make minimum limits boundaries configurable (#808)
* make minimum limits boundaries configurable
* add e2e test
2020-02-03 11:43:18 +01:00
Felix Kunde 97e0d6d388
extend docs and polish manifest examples (#762) 2019-12-12 17:55:41 +01:00
Felix Kunde 107334fe71
Add global option to enable/disable init containers and sidecars (#478)
* Add global option to enable/disable init containers and sidecars
* update dependencies
2019-12-10 15:45:54 +01:00
Felix Kunde a3b34f146f
Add CRD validation (#599)
* add CRD manifests with validation
* update documentation
* patroni slots is not an array but a nested hash map
* make deps call tools
* cover validation in docs and export it in crds.go
* add toggle to disable creation of CRD validation and document it
* use templated service account also for CRD-configured helm deployment
2019-11-28 12:02:05 +01:00
Armin Nesiren 5f87384d7f Passing endpoint, access and secret key to logical-backup container (#628)
* Added possibility to add custom annotations to LoadBalancer service.

* Added parameters for custom endpoint, access and secret key for logical backup.

* Modified dump.sh so it knows how to handle new features. Configurable S3 SSE
2019-11-26 10:40:49 +01:00
Thomas Runyon 535517cd1b Custom annotations 329 (#657)
* Add ability for custom annotations to database pods
2019-11-11 10:45:35 +01:00
Emre Hasegeli 33e1d60703 docs: Fix links on operator parameters reference (#712) 2019-11-08 08:20:43 +01:00
Felix Kunde cd350a4bc1
make run.sh executable from within e2e (#619) 2019-07-24 15:07:32 +02:00
Felix Kunde 0aff809958
use h1 tags to not render titles in sidebar (#626) 2019-07-19 12:50:39 +02:00
Dmitry Dolgov 53fea88391
Update link for pod_antiaffinity_topology_key (#622)
Looks like the anchor was changed in the K8S docs.
2019-07-17 13:04:54 +02:00
Felix Kunde 4fc5822b24
Update docs for v1.2 (#609)
* update docs and move parts from README to index.md
* fix typos, headings and code alignment in docs
2019-07-11 17:19:27 +02:00
Felix Kunde 7c19cf50db
align config map, operator config, helm chart values and templates (#595)
* align config map, operator config, helm chart values and templates
* follow helm chart conventions also in CRD templates
* split up values files and add comments
* avoid yaml confusion in postgres manifests
* bump spilo version and use example for logical_backup_s3_bucket
* add ConfigTarget switch to values
2019-07-08 17:49:25 +02:00
Felix Kunde 36003b8264
enable shmVolume setting in OperatorConfiguration (#605)
* enable shmVolume setting in OperatorConfiguration
2019-07-05 16:48:37 +02:00
Yujun Zhang ff80fc4d0f Fix spelling error (#604)
EBS, not ESB
2019-07-03 14:49:05 +02:00
Markus 93bfed3e75 Add secret mount to operator (#535)
* add secret mount to operator
2019-06-19 12:40:49 +02:00
Felix Kunde 6918394562
Add PDB configuration toggle (#583)
* Don't create an impossible disruption budget for smaller clusters.
* sync PDB also on update
2019-06-18 10:48:21 +02:00
Maxim Ivanov 44acd7e4db Not being able to register CRD is not a fatal error (#444)
Operator proceeds to checking if CRD is present and ready,
and if not, only then it is a fatal error.
2019-06-14 16:08:29 +02:00
Aaron Miller ec5b1d4d58 StatefulSet fsGroup config option to allow non-root spilo (#531)
* StatefulSet fsGroup config option to allow non-root spilo

* Allow Postgres CRD to overide SpiloFSGroup of the Operator.

* Document FSGroup of a Pod cannot be changed after creation.
2019-06-04 16:38:26 +02:00
Felix Kunde 5a0e95ac45
Add CRD configuration to Helm chart values.yaml (#559)
* add templates for CRDs incl. crd-install hooks
* support both config styles in values.yaml
* fix ServiceAccount naming in values.yaml
2019-06-03 14:48:32 +02:00
Erik Inge Bolsø ebda39368e database.go: remove hardcoded .svc.cluster.local dns suffix (#561)
* database.go: substitute hardcoded .svc.cluster.local dns suffix with config parameter

Use the pod's configured dns search path, for clusters where .svc.cluster.local is not correct.
2019-05-31 16:32:00 +02:00
Sergey Dudoladov 3ffc8ac5fa
Sample logical backup image. (#572)
* Sample logical backup image.

Based on the earlier work by Dmitry Dolgov @erthalion
2019-05-31 15:51:58 +02:00
Sergey Dudoladov f3e1e80aaf
Add logical backup (#442)
* Add k8s cron job to spawn logical backups

* Minor doc updates
2019-05-16 15:52:01 +02:00
Aaron Miller 15ec6a920d Config option to allow Spilo container to run non-privileged. (#525)
* Config option to allow Spilo container to run non-privileged.

Runs non-privileged by default.

Fixes #395

* add spilo_privileged to manifests/configmap.yaml

* add spilo_privileged to helm chart's values.yaml
2019-04-03 17:13:39 +02:00
Sergey Dudoladov 0b53dbe5dc
Set statefulset update and management policy explicitly (#515)
* fix logging in retry

* explicitly set the stateful set update strategy to onDelete

* add podManagementPolicy
2019-03-13 11:49:18 +01:00
Sergey Dudoladov f400539b69
Retry moving master pods (#463)
* Retry moving master pods

* bump up master pod wait timeout
2019-02-28 16:19:27 +01:00
Felix Kunde 31e568157b reflect change in github url (#496)
Project was moved from the incubator to the Zalando main org, hence the rename
2019-02-25 11:26:55 +01:00
teuto.net Netzdienste GmbH 26a7fdfa9f Add Pod Anti Affinity (#489)
* Add Pod Anti Affinity
2019-02-21 16:37:03 +01:00
Stephane T d11b23bd71 Add inherited_labels (#459)
* add support for inherited_labels

Signed-off-by: Stephane Tang <hi@stang.sh>

* update docs with inherited_labels

Signed-off-by: Stephane Tang <hi@stang.sh>
2019-02-14 12:29:06 +01:00
Armin Nesiren 6f6a599c90 Added possibility to add custom annotations to LoadBalancer service. (#461)
* Added possibility to add custom annotations to LoadBalancer service.
2019-01-25 11:35:27 +01:00
zerg-junior 4b5d3cd121
Fix golint failures
* Fix golint fails based on the original work from  the user u5surf

* Skip installing Docker as CDP now have one pre-installed (repairs builds on CDP)
2019-01-08 13:04:48 +01:00
zerg-junior 5cfcc453a9
Update CRD configuration docs and fix the CDP build (#414)
* Update CRD configuration docs

* document resource consumption of the operator

* Add talks by Oleksii
2019-01-02 12:01:47 +01:00
Arve Knudsen 8d766e020c Fix reference to enable_database_access in operator_parameters.md (#435) 2019-01-02 10:31:28 +01:00
zerg-junior c0b0b9a832
[WIP] Add 'admin' option to create role (#425)
* Add 'admin' option to create role

* Fix run_locally_script
2018-12-27 10:14:33 +01:00
Dmitry Dolgov d6e6b00770
Add shm_volume option (#427)
Add possibility to mount a tmpfs volume to /dev/shm to avoid issues like
[this](https://github.com/docker-library/postgres/issues/416). To achieve that
two new options were introduced:

* `enableShmVolume` to PostgreSQL manifest, to specify whether or not mount
this volume per database cluster

* `enable_shm_volume` to operator configuration, to specify whether or not mount
per operator.

The first one, `enableShmVolume` takes precedence to allow us to be more flexible.
2018-12-21 16:22:30 +01:00