public_git/postgres-operator
1
0
Fork 0
mirror of https://github.com/zalando/postgres-operator.git synced 2026-10-04 01:20:48 +02:00
Code Issues Projects Releases Wiki Activity

Merge branch 'master' into fes-support

Browse Source
This commit is contained in:
Felix Kunde
2022-02-21 11:29:39 +01:00
parent 66824076e5 3ce0b1e7fa
commit 47aa7b1c64
32 changed files with 1183 additions and 322 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/administrator.md
+172 -24
View File
@@ -3,6 +3,25 @@
Learn how to configure and manage the Postgres Operator in your Kubernetes (K8s)
environment.
## CRD registration and validation
On startup, the operator will try to register the necessary
[CustomResourceDefinitions](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/#customresourcedefinitions)
`Postgresql` and `OperatorConfiguration`. The latter will only get created if
the `POSTGRES_OPERATOR_CONFIGURATION_OBJECT` [environment variable](https://github.com/zalando/postgres-operator/blob/master/manifests/postgres-operator.yaml#L36)
is set in the deployment yaml and is not empty. If the CRDs already exists they
will only be patched. If you do not wish the operator to create or update the
CRDs set `enable_crd_registration` config option to `false`.
CRDs are defined with a `openAPIV3Schema` structural schema against which new
manifests of [`postgresql`](https://github.com/zalando/postgres-operator/blob/master/manifests/postgresql.crd.yaml) or [`OperatorConfiguration`](https://github.com/zalando/postgres-operator/blob/master/manifests/operatorconfiguration.crd.yaml)
resources will be validated. On creation you can bypass the validation with
`kubectl create --validate=false`.
By default, the operator will register the CRDs in the `all` category so
that resources are listed on `kubectl get all` commands. The `crd_categories`
config option allows for customization of categories.
## Upgrading the operator
The Postgres Operator is upgraded by changing the docker image within the
@@ -63,30 +82,6 @@ upgrade procedure, refer to the [corresponding PR in Spilo](https://github.com/z
When `major_version_upgrade_mode` is set to `manual` the operator will run
the upgrade script for you after the manifest is updated and pods are rotated.
## CRD Validation
[CustomResourceDefinitions](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/#customresourcedefinitions)
will be registered with schema validation by default when the operator is
deployed. The `OperatorConfiguration` CRD will only get created if the
`POSTGRES_OPERATOR_CONFIGURATION_OBJECT` [environment variable](https://github.com/zalando/postgres-operator/blob/master/manifests/postgres-operator.yaml#L36)
in the deployment yaml is set and not empty.
When submitting manifests of [`postgresql`](https://github.com/zalando/postgres-operator/blob/master/manifests/postgresql.crd.yaml) or
[`OperatorConfiguration`](https://github.com/zalando/postgres-operator/blob/master/manifests/operatorconfiguration.crd.yaml) custom
resources with kubectl, validation can be bypassed with `--validate=false`. The
operator can also be configured to not register CRDs with validation on `ADD` or
`UPDATE` events. Running instances are not affected when enabling the validation
afterwards unless the manifests is not changed then. Note, that the provided CRD
manifests contain the validation for users to understand what schema is
enforced.
Once the validation is enabled it can only be disabled manually by editing or
patching the CRD manifest:
```bash
kubectl patch crd postgresqls.acid.zalan.do -p '{"spec":{"validation": null}}'
```
## Non-default cluster domain
If your cluster uses a DNS domain other than the default `cluster.local`, this
@@ -293,6 +288,84 @@ that are aggregated into the K8s [default roles](https://kubernetes.io/docs/refe
For Helm deployments setting `rbac.createAggregateClusterRoles: true` adds these clusterroles to the deployment.
## Password rotation in K8s secrets
The operator regularly updates credentials in the K8s secrets if the
`enable_password_rotation` option is set to `true` in the configuration.
It happens only for `LOGIN` roles with an associated secret (manifest roles,
default users from `preparedDatabases`). Furthermore, there are the following
exceptions:
1. Infrastructure role secrets since rotation should happen by the infrastructure.
2. Team API roles that connect via OAuth2 and JWT token (no secrets to these roles anyway).
3. Database owners since ownership on database objects can not be inherited.
4. System users such as `postgres`, `standby` and `pooler` user.
The interval of days can be set with `password_rotation_interval` (default
`90` = 90 days, minimum 1). On each rotation the user name and password values
are replaced in the K8s secret. They belong to a newly created user named after
the original role plus rotation date in YYMMDD format. All priviliges are
inherited meaning that migration scripts should still grant and revoke rights
against the original role. The timestamp of the next rotation is written to the
secret as well. Note, if the rotation interval is decreased it is reflected in
the secrets only if the next rotation date is more days away than the new
length of the interval.
Pods still using the previous secret values which they keep in memory continue
to connect to the database since the password of the corresponding user is not
replaced. However, a retention policy can be configured for users created by
the password rotation feature with `password_rotation_user_retention`. The
operator will ensure that this period is at least twice as long as the
configured rotation interval, hence the default of `180` = 180 days. When
the creation date of a rotated user is older than the retention period it
might not get removed immediately. Only on the next user rotation it is checked
if users can get removed. Therefore, you might want to configure the retention
to be a multiple of the rotation interval.
### Password rotation for single users
From the configuration, password rotation is enabled for all secrets with the
mentioned exceptions. If you wish to first test rotation for a single user (or
just have it enabled only for a few secrets) you can specify it in the cluster
manifest. The rotation and retention intervals can only be configured globally.
```
spec:
usersWithSecretRotation:
- foo_user
- bar_reader_user
```
### Password replacement without extra users
For some use cases where the secret is only used rarely - think of a `flyway`
user running a migration script on pod start - we do not need to create extra
database users but can replace only the password in the K8s secret. This type
of rotation cannot be configured globally but specified in the cluster
manifest:
```
spec:
usersWithInPlaceSecretRotation:
- flyway
- bar_owner_user
```
This would be the recommended option to enable rotation in secrets of database
owners, but only if they are not used as application users for regular read
and write operations.
### Turning off password rotation
When password rotation is turned off again the operator will check if the
`username` value in the secret matches the original username and replace it
with the latter. A new password is assigned and the `nextRotation` field is
cleared. A final lookup for child (rotation) users to be removed is done but
they will only be dropped if the retention policy allows for it. This is to
avoid sudden connection issues in pods which still use credentials of these
users in memory. You have to remove these child users manually or re-enable
password rotation with smaller interval so they get cleaned up.
## Use taints and tolerations for dedicated PostgreSQL nodes
To ensure Postgres pods are running on nodes without any other application pods,
@@ -339,6 +412,81 @@ master pods from being evicted by the K8s runtime. To prevent eviction
completely, specify the toleration by leaving out the `tolerationSeconds` value
(similar to how Kubernetes' own DaemonSets are configured)
## Node readiness labels
The operator can watch on certain node labels to detect e.g. the start of a
Kubernetes cluster upgrade procedure and move master pods off the nodes to be
decommissioned. Key-value pairs for these node readiness labels can be
specified in the configuration (option name is in singular form):
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: postgres-operator
data:
node_readiness_label: "status1:ready,status2:ready"
```
```yaml
apiVersion: "acid.zalan.do/v1"
kind: OperatorConfiguration
metadata:
name: postgresql-configuration
configuration:
kubernetes:
node_readiness_label:
status1: ready
status2: ready
```
The operator will create a `nodeAffinity` on the pods. This makes the
`node_readiness_label` option the global configuration for defining node
affinities for all Postgres clusters. You can have both, cluster-specific and
global affinity, defined and they will get merged on the pods. If
`node_readiness_label_merge` is configured to `"AND"` the node readiness
affinity will end up under the same `matchExpressions` section(s) from the
manifest affinity.
```yaml
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: environment
operator: In
values:
- pci
- key: status1
operator: In
values:
- ready
- key: status2
...
```
If `node_readiness_label_merge` is set to `"OR"` (default) the readiness label
affinty will be appended with its own expressions block:
```yaml
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: environment
...
- matchExpressions:
- key: storage
...
- matchExpressions:
- key: status1
...
- key: status2
...
```
## Enable pod anti affinity
To ensure Postgres pods are running on different topologies, you can use
docs/reference/cluster_manifest.md
+16
View File
@@ -115,6 +115,22 @@ These parameters are grouped directly under the `spec` key in the manifest.
create the K8s secret in that namespace. The part after the first `.` is
considered to be the user name. Optional.
* **usersWithSecretRotation**
list of users to enable credential rotation in K8s secrets. The rotation
interval can only be configured globally. On each rotation a new user will
be added in the database replacing the `username` value in the secret of
the listed user. Although, rotation users inherit all rights from the
original role, keep in mind that ownership is not transferred. See more
details in the [administrator docs](https://github.com/zalando/postgres-operator/blob/master/docs/administrator.md#password-rotation-in-k8s-secrets).
* **usersWithInPlaceSecretRotation**
list of users to enable in-place password rotation in K8s secrets. The
rotation interval can only be configured globally. On each rotation the
password value will be replaced in the secrets which the operator reflects
in the database, too. List only users here that rarely connect to the
database, like a flyway user running a migration on Pod start. See more
details in the [administrator docs](https://github.com/zalando/postgres-operator/blob/master/docs/administrator.md#password-replacement-without-extra-users).
* **databases**
a map of database names to database owners for the databases that should be
created by the operator. The owner users should already exist on the cluster
docs/reference/operator_parameters.md
+37 -7
View File
@@ -75,9 +75,12 @@ Those are top-level keys, containing both leaf keys and groups.
The default is `true`.
* **enable_crd_validation**
toggles if the operator will create or update CRDs with
*deprecated*: toggles if the operator will create or update CRDs with
[OpenAPI v3 schema validation](https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/#validation)
The default is `true`.
The default is `true`. `false` will be ignored, since `apiextensions.io/v1` requires a structural schema definition.
* **crd_categories**
The operator will register CRDs in the `all` category by default so that they will be returned by a `kubectl get all` call. You are free to change categories or leave them empty.
* **enable_lazy_spilo_upgrade**
Instruct operator to update only the statefulsets with new images (Spilo and InitContainers) without immediately doing the rolling update. The assumption is pods will be re-started later with new images, for example due to the node rotation.
@@ -174,6 +177,28 @@ under the `users` key.
Postgres username used for replication between instances. The default is
`standby`.
* **enable_password_rotation**
For all `LOGIN` roles that are not database owners the operator can rotate
credentials in the corresponding K8s secrets by replacing the username and
password. This means, new users will be added on each rotation inheriting
all priviliges from the original roles. The rotation date (in YYMMDD format)
is appended to the names of the new user. The timestamp of the next rotation
is written to the secret. The default is `false`.
* **password_rotation_interval**
If password rotation is enabled (either from config or cluster manifest) the
interval can be configured with this parameter. The measure is in days which
means daily rotation (`1`) is the most frequent interval possible.
Default is `90`.
* **password_rotation_user_retention**
To avoid an ever growing amount of new users due to password rotation the
operator will remove the created users again after a certain amount of days
has passed. The number can be configured with this parameter. However, the
operator will check that the retention policy is at least twice as long as
the rotation interval and update to this minimum in case it is not.
Default is `180`.
## Major version upgrades
Parameters configuring automatic major version upgrades. In a
@@ -344,11 +369,16 @@ configuration they are grouped under the `kubernetes` key.
* **node_readiness_label**
a set of labels that a running and active node should possess to be
considered `ready`. The operator uses values of those labels to detect the
start of the Kubernetes cluster upgrade procedure and move master pods off
the nodes to be decommissioned. When the set is not empty, the operator also
assigns the `Affinity` clause to the Postgres pods to be scheduled only on
`ready` nodes. The default is empty.
considered `ready`. When the set is not empty, the operator assigns the
`nodeAffinity` clause to the Postgres pods to be scheduled only on `ready`
nodes. The default is empty.
* **node_readiness_label_merge**
If a `nodeAffinity` is also specified in the postgres cluster manifest
it will get merged with the `node_readiness_label` affinity on the pods.
The merge strategy can be configured - it can either be "AND" or "OR".
See [user docs](../user.md#use-taints-tolerations-and-node-affinity-for-dedicated-postgresql-nodes)
for more details. Default is "OR".
* **toleration**
a dictionary that should contain `key`, `operator`, `value` and
docs/user.md
+6 -1
View File
@@ -671,7 +671,9 @@ configured [default requests](reference/operator_parameters.md#kubernetes-resour
To ensure Postgres pods are running on nodes without any other application pods,
you can use [taints and tolerations](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/)
and configure the required toleration in the manifest.
and configure the required toleration in the manifest. Tolerations can also be
defined in the [operator config](administrator.md#use-taints-and-tolerations-for-dedicated-postgresql-nodes)
to apply for all Postgres clusters.
```yaml
spec:
@@ -703,6 +705,9 @@ spec:
- pci
```
If you need to define a `nodeAffinity` for all your Postgres clusters use the
`node_readiness_label` [configuration](administrator.md#node-readiness-labels).
## In-place major version upgrade
Starting with Spilo 13, operator supports in-place major version upgrade to a