public_git
/
postgres-operator
mirror of https://github.com/zalando/postgres-operator.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

Update docs for v1.2 (#609) 

* update docs and move parts from README to index.md
* fix typos, headings and code alignment in docs
This commit is contained in:
Felix Kunde 2019-07-11 17:19:27 +02:00 committed by GitHub
parent 1c340c68c6
commit 4fc5822b24
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
17 changed files with 689 additions and 618 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

16
CONTRIBUTING.md
View File

@ -4,15 +4,19 @@ Wanna contribute to the Postgres Operator? Yay - here is how!
## Reporting issues
If you have a question about patroni or have a problem using it, please read the`README` before filing an issue.
Also double check with the current issues on our [Issues Tracker](https://github.com/zalando/postgres-operator/issues).
Before filing an issue, if you have a question about Postgres Operator or have
a problem using it, please read the [concepts](docs/index.md) page or use the
different guides that we provide for [users](docs/user.md),
[developers](docs/developer.md) or [admins](docs/administrator). Also double
check with the current issues on our [Issues Tracker](https://github.com/zalando/postgres-operator/issues).
## Contributing a pull request
1. Submit a comment to the relevant issue or create a new issue describing your proposed change.
1. Do a fork, develop and test your code changes.
1. Include documentation
1. Submit a pull request.
1. Submit a comment to the relevant issue or create a new issue describing your
proposed change.
2. Do a fork, develop and test your code changes.
3. Include documentation
4. Submit a pull request.
You'll get feedback about your pull request as soon as possible.

78
README.md
View File

@ -8,69 +8,39 @@
<img src="docs/diagrams/logo.png" width="200">
## Introduction to the Postgres Operator
The Postgres Operator enables highly-available [PostgreSQL](https://www.postgresql.org/)
clusters on Kubernetes (K8s) powered by [Patroni](https://github.com/zalando/spilo).
It is configured only through manifests to ease integration into automated CI/CD
pipelines with no access to Kubernetes directly.
The Postgres [operator](https://coreos.com/blog/introducing-operators.html)
manages PostgreSQL clusters on Kubernetes:
The Postgres Operator has been developed at Zalando and is being used in
production for over two years.
1. The operator watches additions, updates, and deletions of PostgreSQL cluster
manifests and changes the running clusters accordingly. For example, when a
user submits a new manifest, the operator spawns a new Postgres cluster with
necessary entities such as StatefulSets, Services, and also Postgres roles. See this
[Postgres cluster manifest](manifests/complete-postgres-manifest.yaml)
for settings that a manifest may contain.
## Getting started
2. The operator also watches updates to [its own configuration](manifests/configmap.yaml)
and alters running Postgres clusters if necessary. For instance, if a pod
Docker image is changed, the operator carries out the rolling update. That
is, the operator re-spawns one-by-one pods of each StatefulSet it manages
with the new Docker image.
For a quick first impression follow the instructions of this
[tutorial](docs/quickstart.md).
3. Finally, the operator periodically synchronizes the actual state of each
Postgres cluster with the desired state defined in the cluster's manifest.
## Documentation
4. The operator aims to be hands free and configuration happens only via manifests and its own config.
This enables easy integration in automated deploy pipelines with no access to Kubernetes directly.
There is a browser-friendly version of this documentation at
[postgres-operator.readthedocs.io](https://postgres-operator.readthedocs.io)
# Getting started
* [How it works](docs/index.md)
* [The Postgres experience on K8s](docs/user.md)
* [DBA options - from RBAC to backup](docs/administrator.md)
* [Debug and extend the operator](docs/developer.md)
* [Configuration options](docs/reference/operator_parameters.md)
* [Postgres manifest reference](docs/reference/cluster_manifest.md)
* [Command-line options and environment variables](docs/reference/command_line_and_environment.md)
For a quick first impression follow the instructions of [this](docs/quickstart.md)
tutorial.
## Google Summer of Code
# Google Summer of Code
The Postgres Operator made it to the [Google Summer of Code 2019](https://summerofcode.withgoogle.com/organizations/5429926902104064/)!
Check [our ideas](docs/gsoc-2019/ideas.md#google-summer-of-code-2019)
and start discussions in [the issue tracker](https://github.com/zalando/postgres-operator/issues).
The Postgres Operator made it to the [Google Summer of Code 2019](https://summerofcode.withgoogle.com/)! As a brand new mentoring organization, we are now looking for our first mentees. Check [our ideas](https://github.com/zalando/postgres-operator/blob/master/docs/gsoc-2019/ideas.md#google-summer-of-code-2019) and start discussion in [the issue tracker](https://github.com/zalando/postgres-operator/issues). And don't forget to spread a word about our GSoC participation to attract even more students.
## Table of contents
* [concepts](docs/index.md)
* [user documentation](docs/user.md)
* [administrator documentation](docs/administrator.md)
* [developer documentation](docs/developer.md)
* [operator configuration reference](docs/reference/operator_parameters.md)
* [cluster manifest reference](docs/reference/cluster_manifest.md)
* [command-line options and environment variables](docs/reference/command_line_and_environment.md)
The rest of this document is a tutorial to get you up and running locally with the operator on Minikube.
## Overview of involved entities
Here is a diagram, that summarizes what would be created by the operator, when a
new Postgres cluster CRD is submitted:
![postgresql-operator](docs/diagrams/operator.png "K8S resources, created by operator")
This picture is not complete without an overview of what is inside a single cluster pod, so
let's zoom in:
![pod](docs/diagrams/pod.png "Database pod components")
These two diagrams should help you to understand the basics of what kind of
functionality the operator provides.
There is a browser-friendly version of this documentation at [postgres-operator.readthedocs.io](https://postgres-operator.readthedocs.io)
## Community
## Community
There are two places to get in touch with the community:
1. The [GitHub issue tracker](https://github.com/zalando/postgres-operator/issues)

4
charts/postgres-operator/Chart.yaml
View File

@ -3,7 +3,7 @@ name: postgres-operator
version: 1.2.0
appVersion: 1.2.0
home: https://github.com/zalando/postgres-operator
description: Postgres operator creates and manages PostgreSQL clusters running in Kubernetes
description: Postgres Operator creates and manages PostgreSQL clusters running in Kubernetes
keywords:
- postgres
- operator
@ -16,5 +16,5 @@ maintainers:
- name: kimxogus
email: kgyoo8232@gmail.com
sources:
- https://github.com/zalando-incubator/postgres-operator
- https://github.com/zalando/postgres-operator
engine: gotpl

2
charts/postgres-operator/values-crd.yaml
View File

@ -243,7 +243,7 @@ serviceAccount:
create: true
# The name of the ServiceAccount to use.
# If not set and create is true, a name is generated using the fullname template
# When relying solely on the OperatorConfiguration CRD, set this value to "operator"
# When relying solely on the OperatorConfiguration CRD, this value has to be "operator"
# Otherwise, the operator tries to use the "default" service account which is forbidden
name: operator

2
charts/postgres-operator/values.yaml
View File

@ -224,8 +224,6 @@ serviceAccount:
create: true
# The name of the ServiceAccount to use.
# If not set and create is true, a name is generated using the fullname template
# When relying solely on the OperatorConfiguration CRD, set this value to "operator"
# Otherwise, the operator tries to use the "default" service account which is forbidden
name:
priorityClassName: ""

2
docker/Dockerfile
View File

@ -7,7 +7,7 @@ RUN apk --no-cache add ca-certificates
COPY build/* /
RUN addgroup -g 1000 pgo
RUN adduser -D -u 1000 -G pgo -g 'Postgres operator' pgo
RUN adduser -D -u 1000 -G pgo -g 'Postgres Operator' pgo
USER 1000:1000

4
docker/logical-backup/dump.sh
View File

@ -7,7 +7,7 @@ set -o pipefail
IFS=$'\n\t'
# make script trace visible via `kubectl logs`
set -o xtrace
set -o xtrace
ALL_DB_SIZE_QUERY="select sum(pg_database_size(datname)::numeric) from pg_database;"
PG_BIN=$PG_DIR/$PG_VERSION/bin
@ -35,7 +35,7 @@ function aws_upload {
# mimic bucket setup from Spilo
# to keep logical backups at the same path as WAL
# NB: $LOGICAL_BACKUP_S3_BUCKET_SCOPE_SUFFIX already contains the leading "/" when set by the Postgres operator
# NB: $LOGICAL_BACKUP_S3_BUCKET_SCOPE_SUFFIX already contains the leading "/" when set by the Postgres Operator
PATH_TO_BACKUP=s3://$LOGICAL_BACKUP_S3_BUCKET"/spilo/"$SCOPE$LOGICAL_BACKUP_S3_BUCKET_SCOPE_SUFFIX"/logical_backups/"$(date +%s).sql.gz
if [ -z "$EXPECTED_SIZE" ]; then

236
docs/administrator.md
View File

@ -1,14 +1,19 @@
# How to configure PostgreSQL operator
# Administrator Guide
## Select the namespace to deploy to
Learn how to configure and manage the Postgres Operator in your Kubernetes (K8s)
environment.
## Namespaces
### Select the namespace to deploy to
The operator can run in a namespace other than `default`. For example, to use
the `test` namespace, run the following before deploying the operator's
manifests:
```bash
$ kubectl create namespace test
$ kubectl config set-context $(kubectl config current-context) --namespace=test
kubectl create namespace test
kubectl config set-context $(kubectl config current-context) --namespace=test
```
All subsequent `kubectl` commands will work with the `test` namespace. The
@ -17,18 +22,17 @@ ConfigMap - there. Please note that the namespace for service accounts and
cluster role bindings in [operator RBAC rules](../manifests/operator-service-account-rbac.yaml)
needs to be adjusted to the non-default value.
## Specify the namespace to watch
### Specify the namespace to watch
Watching a namespace for an operator means tracking requests to change
Postgresql clusters in the namespace such as "increase the number of Postgresql
replicas to 5" and reacting to the requests, in this example by actually
scaling up.
Watching a namespace for an operator means tracking requests to change Postgres
clusters in the namespace such as "increase the number of Postgres replicas to
5" and reacting to the requests, in this example by actually scaling up.
By default, the operator watches the namespace it is deployed to. You can
change this by setting the `WATCHED_NAMESPACE` var in the `env` section of the
[operator deployment](../manifests/postgres-operator.yaml) manifest or by
altering the `watched_namespace` field in the operator
[ConfigMap](../manifests/configmap.yaml#L6).
[ConfigMap](../manifests/configmap.yaml#L79).
In the case both are set, the env var takes the precedence. To make the
operator listen to all namespaces, explicitly set the field/env var to "`*`".
@ -36,14 +40,14 @@ Note that for an operator to manage pods in the watched namespace, the
operator's service account (as specified in the operator deployment manifest)
has to have appropriate privileges to access the watched namespace. The
operator may not be able to function in the case it watches all namespaces but
lacks access rights to any of them (except Kubernetes system namespaces like
lacks access rights to any of them (except K8s system namespaces like
`kube-system`). The reason is that for multiple namespaces operations such as
'list pods' execute at the cluster scope and fail at the first violation of
access rights.
The watched namespace also needs to have a (possibly different) service account
in the case database pods need to talk to the Kubernetes API (e.g. when using
Kubernetes-native configuration of Patroni). The operator checks that the
in the case database pods need to talk to the K8s API (e.g. when using
K8s-native configuration of Patroni). The operator checks that the
`pod_service_account_name` exists in the target namespace, and, if not, deploys
there the `pod_service_account_definition` from the operator
[`Config`](../pkg/util/config/config.go) with the default value of:
@ -61,52 +65,65 @@ namespace. The operator performs **no** further syncing of this account.
## Non-default cluster domain
If your cluster uses a different dns domain than `cluster.local`, this needs
to be set in the operator ConfigMap. This is used by the operator to connect
to the clusters after creation.
If your cluster uses a DNS domain other than the default `cluster.local`, this
needs to be set in the operator configuration (`cluster_domain` variable). This
is used by the operator to connect to the clusters after creation.
## Role-based access control for the operator
The `manifests/operator-service-account-rbac.yaml` defines cluster roles and
bindings needed for the operator to function under access control restrictions.
To deploy the operator with this RBAC policy use:
### Service account and cluster roles
The manifest [`operator-service-account-rbac.yaml`](../manifests/operator-service-account-rbac.yaml)
defines the service account, cluster roles and bindings needed for the operator
to function under access control restrictions. To deploy the operator with this
RBAC policy use:
```bash
$ kubectl create -f manifests/configmap.yaml
$ kubectl create -f manifests/operator-service-account-rbac.yaml
$ kubectl create -f manifests/postgres-operator.yaml
$ kubectl create -f manifests/minimal-postgres-manifest.yaml
kubectl create -f manifests/configmap.yaml
kubectl create -f manifests/operator-service-account-rbac.yaml
kubectl create -f manifests/postgres-operator.yaml
kubectl create -f manifests/minimal-postgres-manifest.yaml
```
Note that the service account in `operator-rbac.yaml` is named
`zalando-postgres-operator`. You may have to change the `service_account_name`
in the operator ConfigMap and `serviceAccountName` in the postgres-operator
deployment appropriately.
Note that the service account is named `zalando-postgres-operator`. You may have
to change the `service_account_name` in the operator ConfigMap and
`serviceAccountName` in the `postgres-operator` deployment appropriately. This
is done intentionally to avoid breaking those setups that already work with the
default `operator` account. In the future the operator should ideally be run
under the `zalando-postgres-operator` service account.
This is done intentionally to avoid breaking those setups that already work
with the default `operator` account. In the future the operator should ideally
be run under the `zalando-postgres-operator` service account.
The service account defined in `operator-rbac.yaml` acquires some privileges
not really used by the operator (i.e. we only need `list` and `watch` on
`configmaps` resources), this is also done intentionally to avoid breaking
The service account defined in `operator-service-account-rbac.yaml` acquires
some privileges not used by the operator (i.e. we only need `list` and `watch`
on `configmaps` resources). This is also done intentionally to avoid breaking
things if someone decides to configure the same service account in the
operator's ConfigMap to run postgres clusters.
operator's ConfigMap to run Postgres clusters.
### Use taints and tolerations for dedicated PostgreSQL nodes
### Give K8S users access to create/list `postgresqls`
To ensure Postgres pods are running on nodes without any other application
pods, you can use
[taints and tolerations](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/)
By default `postgresql` custom resources can only be listed and changed by
cluster admins. To allow read and/or write access to other human users apply
the `user-facing-clusterrole` manifest:
```bash
kubectl create -f manifests/user-facing-clusterroles.yaml
```
It creates zalando-postgres-operator:user:view, :edit and :admin clusterroles
that are aggregated into the K8s [default roles](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#default-roles-and-role-bindings).
## Use taints and tolerations for dedicated PostgreSQL nodes
To ensure Postgres pods are running on nodes without any other application pods,
you can use [taints and tolerations](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/)
and configure the required toleration in the operator ConfigMap.
As an example you can set following node taint:
```bash
$ kubectl taint nodes <nodeName> postgres=:NoSchedule
kubectl taint nodes <nodeName> postgres=:NoSchedule
```
And configure the toleration for the PostgreSQL pods by adding following line
And configure the toleration for the Postgres pods by adding following line
to the ConfigMap:
```yaml
@ -119,14 +136,18 @@ data:
...
```
Note that the Kubernetes version 1.13 brings [taint-based eviction](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/#taint-based-evictions) to the beta stage and enables it by default.
Postgres pods by default receive tolerations for `unreachable` and `noExecute` taints with the timeout of `5m`.
Depending on your setup, you may want to adjust these parameters to prevent master pods from being evicted by the Kubernetes runtime.
To prevent eviction completely, specify the toleration by leaving out the `tolerationSeconds` value (similar to how Kubernetes' own DaemonSets are configured)
Note that the K8s version 1.13 brings [taint-based eviction](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/#taint-based-evictions)
to the beta stage and enables it by default. Postgres pods by default receive
tolerations for `unreachable` and `noExecute` taints with the timeout of `5m`.
Depending on your setup, you may want to adjust these parameters to prevent
master pods from being evicted by the K8s runtime. To prevent eviction
completely, specify the toleration by leaving out the `tolerationSeconds` value
(similar to how Kubernetes' own DaemonSets are configured)
### Enable pod anti affinity
## Enable pod anti affinity
To ensure Postgres pods are running on different topologies, you can use [pod anti affinity](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/)
To ensure Postgres pods are running on different topologies, you can use
[pod anti affinity](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/)
and configure the required topology in the operator ConfigMap.
Enable pod anti affinity by adding following line to the operator ConfigMap:
@ -140,9 +161,10 @@ data:
enable_pod_antiaffinity: "true"
```
By default the topology key for the pod anti affinity is set to `kubernetes.io/hostname`,
you can set another topology key e.g. `failure-domain.beta.kubernetes.io/zone` by adding following line
to the operator ConfigMap, see [built-in node labels](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#interlude-built-in-node-labels) for available topology keys:
By default the topology key for the pod anti affinity is set to
`kubernetes.io/hostname`, you can set another topology key e.g.
`failure-domain.beta.kubernetes.io/zone` by adding following line to the
operator ConfigMap, see [built-in node labels](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#interlude-built-in-node-labels) for available topology keys:
```yaml
apiVersion: v1
@ -154,7 +176,7 @@ data:
pod_antiaffinity_topology_key: "failure-domain.beta.kubernetes.io/zone"
```
### Pod Disruption Budget
## Pod Disruption Budget
By default the operator uses a PodDisruptionBudget (PDB) to protect the cluster
from voluntarily disruptions and hence unwanted DB downtime. The `MinAvailable`
@ -168,14 +190,15 @@ The PDB is only relaxed in two scenarios:
The PDB is still in place having `MinAvailable` set to `0`. If enabled it will
be automatically set to `1` on scale up. Disabling PDBs helps avoiding blocking
Kubernetes upgrades in managed K8s environments at the cost of prolonged DB
downtime. See PR #384 for the use case.
downtime. See PR [#384](https://github.com/zalando/postgres-operator/pull/384)
for the use case.
### Add cluster-specific labels
## Add cluster-specific labels
In some cases, you might want to add `labels` that are specific to a given
postgres cluster, in order to identify its child objects.
The typical use case is to add labels that identifies the `Pods` created by the
operator, in order to implement fine-controlled `NetworkPolicies`.
Postgres cluster, in order to identify its child objects. The typical use case
is to add labels that identifies the `Pods` created by the operator, in order
to implement fine-controlled `NetworkPolicies`.
**OperatorConfiguration**
@ -259,7 +282,7 @@ data:
This ConfigMap is then added as a source of environment variables to the
Postgres StatefulSet/pods.
## Limiting the number of instances in clusters with `min_instances` and `max_instances`
## Limiting the number of min and max instances in clusters
As a preventive measure, one can restrict the minimum and the maximum number of
instances permitted by each Postgres cluster managed by the operator. If either
@ -269,9 +292,9 @@ either the min or the max boundary. For instance, of a cluster manifest has 1
instance and the `min_instances` is set to 3, the cluster will be created with 3
instances. By default, both parameters are set to `-1`.
## Load balancers
## Load balancers and allowed IP ranges
For any Postgresql/Spilo cluster, the operator creates two separate Kubernetes
For any Postgres/Spilo cluster, the operator creates two separate K8s
services: one for the master pod and one for replica pods. To expose these
services to an outer network, one can attach load balancers to them by setting
`enableMasterLoadBalancer` and/or `enableReplicaLoadBalancer` to `true` in the
@ -280,72 +303,109 @@ manifest, the operator configmap's settings `enable_master_load_balancer` and
`enable_replica_load_balancer` apply. Note that the operator settings affect
all Postgresql services running in all namespaces watched by the operator.
To limit the range of IP adresses that can reach a load balancer, specify the
To limit the range of IP addresses that can reach a load balancer, specify the
desired ranges in the `allowedSourceRanges` field (applies to both master and
replica load balancers). To prevent exposing load balancers to the entire
Internet, this field is set at cluster creation time to `127.0.0.1/32` unless
overwritten explicitly. If you want to revoke all IP ranges from an existing
cluster, please set the `allowedSourceRanges` field to `127.0.0.1/32` or to an
empty sequence `[]`. Setting the field to `null` or omitting it entirely may
lead to Kubernetes removing this field from the manifest due to its
lead to K8s removing this field from the manifest due to its
[handling of null fields](https://kubernetes.io/docs/concepts/overview/object-management-kubectl/declarative-config/#how-apply-calculates-differences-and-merges-changes).
Then the resultant manifest will not contain the necessary change, and the
operator will respectively do noting with the existing source ranges.
## Running periodic 'autorepair' scans of Kubernetes objects
## Running periodic 'autorepair' scans of K8s objects
The Postgres operator periodically scans all Kubernetes objects belonging to
each cluster and repairs all discrepancies between them and the definitions
generated from the current cluster manifest. There are two types of scans:
The Postgres Operator periodically scans all K8s objects belonging to each
cluster and repairs all discrepancies between them and the definitions generated
from the current cluster manifest. There are two types of scans:
* `sync scan`, running every `resync_period` seconds for every cluster
* `repair scan`, coming every `repair_period` only for those clusters that didn't
report success as a result of the last operation applied to them.
* `repair scan`, coming every `repair_period` only for those clusters that
didn't report success as a result of the last operation applied to them.
## Postgres roles supported by the operator
The operator is capable of maintaining roles of multiple kinds within a
Postgres database cluster:
* **System roles** are roles necessary for the proper work of Postgres itself such as a replication role or the initial superuser role. The operator delegates creating such roles to Patroni and only establishes relevant secrets.
* **System roles** are roles necessary for the proper work of Postgres itself
such as a replication role or the initial superuser role. The operator delegates
creating such roles to Patroni and only establishes relevant secrets.
* **Infrastructure roles** are roles for processes originating from external systems, e.g. monitoring robots. The operator creates such roles in all Postgres clusters it manages assuming that Kubernetes secrets with the relevant credentials exist beforehand.
* **Infrastructure roles** are roles for processes originating from external
systems, e.g. monitoring robots. The operator creates such roles in all Postgres
clusters it manages, assuming that K8s secrets with the relevant
credentials exist beforehand.
* **Per-cluster robot users** are also roles for processes originating from external systems but defined for an individual Postgres cluster in its manifest. A typical example is a role for connections from an application that uses the database.
* **Per-cluster robot users** are also roles for processes originating from
external systems but defined for an individual Postgres cluster in its manifest.
A typical example is a role for connections from an application that uses the
database.
* **Human users** originate from the Teams API that returns a list of the team members given a team id. The operator differentiates between (a) product teams that own a particular Postgres cluster and are granted admin rights to maintain it, and (b) Postgres superuser teams that get the superuser access to all Postgres databases running in a Kubernetes cluster for the purposes of maintaining and troubleshooting.
* **Human users** originate from the Teams API that returns a list of the team
members given a team id. The operator differentiates between (a) product teams
that own a particular Postgres cluster and are granted admin rights to maintain
it, and (b) Postgres superuser teams that get the superuser access to all
Postgres databases running in a K8s cluster for the purposes of maintaining and
troubleshooting.
## Understanding rolling update of Spilo pods
The operator logs reasons for a rolling update with the `info` level and a diff between the old and new StatefulSet specs with the `debug` level. To benefit from numerous escape characters in the latter log entry, view it in CLI with `echo -e`. Note that the resultant message will contain some noise because the `PodTemplate` used by the operator is yet to be updated with the default values used internally in Kubernetes.
The operator logs reasons for a rolling update with the `info` level and a diff
between the old and new StatefulSet specs with the `debug` level. To benefit
from numerous escape characters in the latter log entry, view it in CLI with
`echo -e`. Note that the resultant message will contain some noise because the
`PodTemplate` used by the operator is yet to be updated with the default values
used internally in K8s.
## Logical backups
The operator can manage k8s cron jobs to run logical backups of Postgres clusters. The cron job periodically spawns a batch job that runs a single pod. The backup script within this pod's container can connect to a DB for a logical backup. The operator updates cron jobs during Sync if the job schedule changes; the job name acts as the job identifier. These jobs are to be enabled for each indvidual Postgres cluster by setting `enableLogicalBackup: true` in its manifest. Notes:
The operator can manage k8s cron jobs to run logical backups of Postgres
clusters. The cron job periodically spawns a batch job that runs a single pod.
The backup script within this pod's container can connect to a DB for a logical
backup. The operator updates cron jobs during Sync if the job schedule changes;
the job name acts as the job identifier. These jobs are to be enabled for each
individual Postgres cluster by setting `enableLogicalBackup: true` in its
manifest. Notes:
1. The [example image](../docker/logical-backup/Dockerfile) implements the backup via `pg_dumpall` and upload of compressed and encrypted results to an S3 bucket; the default image ``registry.opensource.zalan.do/acid/logical-backup`` is the same image built with the Zalando-internal CI pipeline. `pg_dumpall` requires a `superuser` access to a DB and runs on the replica when possible.
1. The [example image](../docker/logical-backup/Dockerfile) implements the
backup via `pg_dumpall` and upload of compressed and encrypted results to an S3
bucket; the default image ``registry.opensource.zalan.do/acid/logical-backup``
is the same image built with the Zalando-internal CI pipeline. `pg_dumpall`
requires a `superuser` access to a DB and runs on the replica when possible.
2. Due to the [limitation of Kubernetes cron jobs](https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/#cron-job-limitations) it is highly advisable to set up additional monitoring for this feature; such monitoring is outside of the scope of operator responsibilities.
2. Due to the [limitation of K8s cron jobs](https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/#cron-job-limitations)
it is highly advisable to set up additional monitoring for this feature; such
monitoring is outside of the scope of operator responsibilities.
3. The operator does not remove old backups.
4. You may use your own image by overwriting the relevant field in the operator configuration. Any such image must ensure the logical backup is able to finish [in presence of pod restarts](https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#handling-pod-and-container-failures) and [simultaneous invocations](https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/#cron-job-limitations) of the backup cron job.
4. You may use your own image by overwriting the relevant field in the operator
configuration. Any such image must ensure the logical backup is able to finish
[in presence of pod restarts](https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#handling-pod-and-container-failures)
and [simultaneous invocations](https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/#cron-job-limitations)
of the backup cron job.
5. For that feature to work, your RBAC policy must enable operations on the `cronjobs` resource from the `batch` API group for the operator service account. See [example RBAC](../manifests/operator-service-account-rbac.yaml)
5. For that feature to work, your RBAC policy must enable operations on the
`cronjobs` resource from the `batch` API group for the operator service account.
See [example RBAC](../manifests/operator-service-account-rbac.yaml)
## Access to cloud resources from clusters in non cloud environment
## Access to cloud resources from clusters in non-cloud environment
To access cloud resources like S3 from a cluster in a bare metal setup you can use
`additional_secret_mount` and `additional_secret_mount_path` config parameters.
With this you can provision cloud credentials to the containers in the pods of the StatefulSet.
This works this way that it mounts a volume from the given secret in the pod and this can
then accessed in the container over the configured mount path. Via [Custum Pod Environment Variables](#custom-pod-environment-variables)
you can then point the different cloud sdk's (aws, google etc.) to this mounted secret.
With this credentials the cloud sdk can then access cloud resources to upload logs etc.
To access cloud resources like S3 from a cluster on bare metal you can use
`additional_secret_mount` and `additional_secret_mount_path` configuration
parameters. The cloud credentials will be provisioned in the Postgres containers
by mounting an additional volume from the given secret to database pods. They
can then be accessed over the configured mount path. Via
[Custom Pod Environment Variables](#custom-pod-environment-variables) you can
point different cloud SDK's (AWS, GCP etc.) to this mounted secret, e.g. to
access cloud resources for uploading logs etc.
A secret can be pre provisioned in different ways:
A secret can be pre-provisioned in different ways:
* Generic secret created via `kubectl create secret generic some-cloud-creds --from-file=some-cloud-credentials-file.json`
* Automaticly provisioned via a Controller like [kube-aws-iam-controller](https://github.com/mikkeloscar/kube-aws-iam-controller). This controller would then also rotate the credentials. Please visit the documention for more information.
* Automatically provisioned via a custom K8s controller like
[kube-aws-iam-controller](https://github.com/mikkeloscar/kube-aws-iam-controller)

295
docs/developer.md
View File

@ -1,128 +1,28 @@
## Intro
# Developer Guide
See [minikube installation guide](https://github.com/kubernetes/minikube/releases)
Make sure you use the latest version of Minikube.
After the installation, issue
```bash
$ minikube start
```
Note: if you are running on a Mac, you may also use Docker for Mac Kubernetes
instead of a docker-machine.
Once you have it started successfully, use [the quickstart
guide](https://github.com/kubernetes/minikube#quickstart) in order to test your
that your setup is working.
Note: if you use multiple Kubernetes clusters, you can switch to Minikube with
`kubectl config use-context minikube`
## Deploying the operator
### Kubernetes manifest
A ConfigMap is used to store the configuration of the operator. Alternatively,
a CRD-based configuration can be used, as described [here](reference/operator_parameters).
```bash
$ kubectl --context minikube create -f manifests/configmap.yaml
```
First you need to install the service account definition in your Minikube cluster.
```bash
$ kubectl --context minikube create -f manifests/operator-service-account-rbac.yaml
```
Next deploy the postgres-operator from the docker image Zalando is using:
```bash
$ kubectl --context minikube create -f manifests/postgres-operator.yaml
```
If you prefer to build the image yourself follow up down below.
### Helm chart
Alternatively, the operator can be installed by using the provided [Helm](https://helm.sh/)
chart which saves you the manual steps. Therefore, you would need to install
the helm CLI on your machine. After initializing helm (and its server
component Tiller) in your local cluster you can install the operator chart.
You can define a release name that is prepended to the operator resource's
names.
Use `--name zalando` to match with the default service account name as older
operator versions do not support custom names for service accounts. When relying
solely on the CRD-based configuration edit the `serviceAccount` section in the
[values yaml file](../charts/values.yaml) by setting the name to `"operator"`.
```bash
$ helm init
$ helm install --name zalando ./charts/postgres-operator
```
## Check if CustomResourceDefinition has been registered
```bash
$ kubectl --context minikube get crd
NAME KIND
postgresqls.acid.zalan.do CustomResourceDefinition.v1beta1.apiextensions.k8s.io
```
## Create a new Spilo cluster
```bash
$ kubectl --context minikube create -f manifests/minimal-postgres-manifest.yaml
```
## Watch pods being created
```bash
$ kubectl --context minikube get pods -w --show-labels
```
## Connect to PostgreSQL
We can use the generated secret of the `postgres` robot user to connect to our
`acid-minimal-cluster` master running in Minikube:
```bash
$ export HOST_PORT=$(minikube service acid-minimal-cluster --url | sed 's,.*/,,')
$ export PGHOST=$(echo $HOST_PORT | cut -d: -f 1)
$ export PGPORT=$(echo $HOST_PORT | cut -d: -f 2)
$ export PGPASSWORD=$(kubectl --context minikube get secret postgres.acid-minimal-cluster.credentials -o 'jsonpath={.data.password}' | base64 -d)
$ psql -U postgres
```
# Setup development environment
The following steps guide you through the setup to work on the operator itself.
Read this guide if you want to debug the operator, fix bugs or contribute new
features and tests.
## Setting up Go
Postgres operator is written in Go. Use the [installation
instructions](https://golang.org/doc/install#install) if you don't have Go on
your system. You won't be able to compile the operator with Go older than 1.7.
We recommend installing [the latest one](https://golang.org/dl/).
Postgres Operator is written in Go. Use the [installation instructions](https://golang.org/doc/install#install)
if you don't have Go on your system. You won't be able to compile the operator
with Go older than 1.7. We recommend installing [the latest one](https://golang.org/dl/).
Go projects expect their source code and all the dependencies to be located
under the [GOPATH](https://github.com/golang/go/wiki/GOPATH). Normally, one
would create a directory for the GOPATH (i.e. ~/go) and place the source code
under the ~/go/src subdirectories.
under the ~/go/src sub directories.
Given the schema above, the postgres operator source code located at
Given the schema above, the Postgres Operator source code located at
`github.com/zalando/postgres-operator` should be put at
-`~/go/src/github.com/zalando/postgres-operator`.
```bash
$ export GOPATH=~/go
$ mkdir -p ${GOPATH}/src/github.com/zalando/
$ cd ${GOPATH}/src/github.com/zalando/
$ git clone https://github.com/zalando/postgres-operator.git
export GOPATH=~/go
mkdir -p ${GOPATH}/src/github.com/zalando/
cd ${GOPATH}/src/github.com/zalando/
git clone https://github.com/zalando/postgres-operator.git
```
## Building the operator
@ -130,55 +30,74 @@ Given the schema above, the postgres operator source code located at
You need Glide to fetch all dependencies. Install it with:
```bash
$ make tools
make tools
```
Next, install dependencies with glide by issuing:
```bash
$ make deps
make deps
```
This would take a while to complete. You have to redo `make deps` every time
you dependencies list changes, i.e. after adding a new library dependency.
Build the operator docker image and pushing it to Pier One:
Build the operator with the `make docker` command. You may define the TAG
variable to assign an explicit tag to your docker image and the IMAGE to set
the image name. By default, the tag is computed with
`git describe --tags --always --dirty` and the image is
`registry.opensource.zalan.do/acid/postgres-operator`
```bash
$ make docker push
export TAG=$(git describe --tags --always --dirty)
make docker
```
You may define the TAG variable to assign an explicit tag to your docker image
and the IMAGE to set the image name. By default, the tag is computed with
`git describe --tags --always --dirty` and the image is
`pierone.stups.zalan.do/acid/postgres-operator`
Building the operator binary (for testing the out-of-cluster option):
```bash
$ make
make
```
The binary will be placed into the build directory.
## Deploying self build image
The fastest way to run your docker image locally is to reuse the docker from
minikube. The following steps will get you the docker image built and deployed.
The fastest way to run and test your docker image locally is to reuse the docker
from [minikube](https://github.com/kubernetes/minikube/releases) or use the
`load docker-image` from [kind](https://kind.sigs.k8s.io/). The following steps
will get you the docker image built and deployed.
```bash
$ eval $(minikube docker-env)
$ export TAG=$(git describe --tags --always --dirty)
$ make docker
$ sed -e "s/\(image\:.*\:\).*$/\1$TAG/" manifests/postgres-operator.yaml|kubectl --context minikube create -f -
# minikube
eval $(minikube docker-env)
make docker
# kind
make docker
kind load docker-image <image> --name <kind-cluster-name>
```
# Code generation
Then create a new Postgres Operator deployment. You can reuse the provided
manifest but replace the version and tag. Don't forget to also apply
configuration and RBAC manifests first, e.g.:
The operator employs k8s-provided code generation to obtain deep copy methods
and Kubernetes-like APIs for its custom resource definitons, namely the Postgres
CRD and the operator CRD. The usage of the code generation follows conventions
from the k8s community. Relevant scripts live in the `hack` directory:
```bash
kubectl create -f manifests/configmap.yaml
kubectl create -f manifests/operator-service-account-rbac.yaml
sed -e "s/\(image\:.*\:\).*$/\1$TAG/" manifests/postgres-operator.yaml | kubectl create -f -
# check if the operator is coming up
kubectl get pod -l name=postgres-operator
```
## Code generation
The operator employs K8s-provided code generation to obtain deep copy methods
and K8s-like APIs for its custom resource definitions, namely the
Postgres CRD and the operator CRD. The usage of the code generation follows
conventions from the k8s community. Relevant scripts live in the `hack`
directory:
* `update-codegen.sh` triggers code generation for the APIs defined in `pkg/apis/acid.zalan.do/`,
* `verify-codegen.sh` checks if the generated code is up-to-date (to be used within CI).
@ -199,18 +118,22 @@ operator itself with `--v=8` to log all REST API requests). To attach a debugger
to the operator, use the `-outofcluster` option to run the operator locally on
the developer's laptop (and not in a docker container).
# Debugging the operator
## Debugging the operator
There is a web interface in the operator to observe its internal state. The
operator listens on port 8080. It is possible to expose it to the
localhost:8080 by doing:
`localhost:8080` by doing:
$ kubectl --context minikube port-forward $(kubectl --context minikube get pod -l name=postgres-operator -o jsonpath={.items..metadata.name}) 8080:8080
```bash
kubectl --context minikube port-forward $(kubectl --context minikube get pod -l name=postgres-operator -o jsonpath={.items..metadata.name}) 8080:8080
```
The inner 'query' gets the name of the postgres operator pod, and the outer
The inner query gets the name of the Postgres Operator pod, and the outer one
enables port forwarding. Afterwards, you can access the operator API with:
$ curl --location http://127.0.0.1:8080/$endpoint | jq .
```bash
curl --location http://127.0.0.1:8080/$endpoint | jq .
```
The available endpoints are listed below. Note that the worker ID is an integer
from 0 up to 'workers' - 1 (value configured in the operator configuration and
@ -242,14 +165,13 @@ The operator also supports pprof endpoints listed at the
* /debug/pprof/trace
It's possible to attach a debugger to troubleshoot postgres-operator inside a
docker container. It's possible with gdb and
[delve](https://github.com/derekparker/delve). Since the latter one is a
specialized debugger for golang, we will use it as an example. To use it you
need:
docker container. It's possible with [gdb](https://www.gnu.org/software/gdb/)
and [delve](https://github.com/derekparker/delve). Since the latter one is a
specialized debugger for Go, we will use it as an example. To use it you need:
* Install delve locally
```
```bash
go get -u github.com/derekparker/delve/cmd/dlv
```
@ -261,7 +183,8 @@ RUN go get github.com/derekparker/delve/cmd/dlv
```
* Update the `Makefile` to build the project with debugging symbols. For that
you need to add `gcflags` to a build target for corresponding OS (e.g. linux)
you need to add `gcflags` to a build target for corresponding OS (e.g.
GNU/Linux)
```
-gcflags "-N -l"
@ -276,61 +199,75 @@ CMD ["/root/go/bin/dlv", "--listen=:DLV_PORT", "--headless=true", "--api-version
* Forward the listening port
```
```bash
kubectl port-forward POD_NAME DLV_PORT:DLV_PORT
```
* Attach to it
```
$ dlv connect 127.0.0.1:DLV_PORT
```bash
dlv connect 127.0.0.1:DLV_PORT
```
## Unit tests
To run all unit tests, you can simply do:
```
$ go test ./...
```bash
go test ./...
```
For go 1.9 `vendor` directory would be excluded automatically. For previous
versions you can exclude it manually:
```
$ go test $(glide novendor)
```bash
go test $(glide novendor)
```
In case if you need to debug your unit test, it's possible to use delve:
```
$ dlv test ./pkg/util/retryutil/
```bash
dlv test ./pkg/util/retryutil/
Type 'help' for list of commands.
(dlv) c
PASS
```
To test the multinamespace setup, you can use
To test the multi-namespace setup, you can use
```
```bash
./run_operator_locally.sh --rebuild-operator
```
It will automatically create an `acid-minimal-cluster` in the namespace `test`.
Then you can for example check the Patroni logs:
```
```bash
kubectl logs acid-minimal-cluster-0
```
## End-to-end tests
The operator provides reference e2e (end-to-end) tests to ensure various infra parts work smoothly together.
Each e2e execution tests a Postgres operator image built from the current git branch. The test runner starts a [kind](https://kind.sigs.k8s.io/) (local k8s) cluster and Docker container with tests. The k8s API client from within the container connects to the `kind` cluster using the standard Docker `bridge` network.
The tests utilize examples from `/manifests` (ConfigMap is used for the operator configuration) to avoid maintaining yet another set of configuration files. The kind cluster is deleted if tests complete successfully.
The operator provides reference e2e (end-to-end) tests to ensure various infra
parts work smoothly together. Each e2e execution tests a Postgres Operator image
built from the current git branch. The test runner starts a [kind](https://kind.sigs.k8s.io/)
(local k8s) cluster and Docker container with tests. The k8s API client from
within the container connects to the `kind` cluster using the standard Docker
`bridge` network. The tests utilize examples from `/manifests` (ConfigMap is
used for the operator configuration) to avoid maintaining yet another set of
configuration files. The kind cluster is deleted if tests complete successfully.
End-to-end tests are executed automatically during builds; to invoke them locally use `make e2e-run` from the project's top directory. Run `make e2e-tools e2e-build` to install `kind` and build the tests' image locally before the first run.
End-to-end tests are executed automatically during builds:
End-to-end tests are written in Python and use `flake8` for code quality. Please run flake8 [before submitting a PR](http://flake8.pycqa.org/en/latest/user/using-hooks.html).
```bash
# invoke them from the project's top directory
make e2e-run
# install kind and build test image before first run
make e2e-tools e2e-build
```
End-to-end tests are written in Python and use `flake8` for code quality.
Please run flake8 [before submitting a PR](http://flake8.pycqa.org/en/latest/user/using-hooks.html).
## Introduce additional configuration parameters
@ -340,17 +277,17 @@ be updated. As explained [here](reference/operator_parameters.md), it's possible
to configure the operator either with a ConfigMap or CRD, but currently we aim
to synchronize parameters everywhere.
When choosing a parameter name for a new option in a PG manifest, keep in mind
the naming conventions there. The `snake_case` variables come from the Patroni/Postgres world, while the `camelCase` from the k8s world.
When choosing a parameter name for a new option in a Postgres cluster manifest,
keep in mind the naming conventions there. We use `camelCase` for manifest
parameters (with exceptions for certain Patroni/Postgres options) and
`snake_case` variables in the configuration. Only introduce new manifest
variables if you feel a per-cluster configuration is necessary.
Note: If one option is defined in the operator configuration and in the cluster
[manifest](../manifests/complete-postgres-manifest.yaml), the latter takes
precedence.
So, first define the parameters in:
* the [ConfigMap](../manifests/configmap.yaml) manifest
* the CR's [default configuration](../manifests/postgresql-operator-default-configuration.yaml)
* the Helm chart [values](../charts/postgres-operator/values.yaml)
### Go code
Update the following Go files that obtain the configuration parameter from the
manifest files:
@ -358,12 +295,30 @@ manifest files:
* [operator_config.go](../pkg/controller/operator_config.go)
* [config.go](../pkg/util/config/config.go)
Postgres manifest parameters are defined in the [api package](../pkg/apis/acid.zalan.do/v1/postgresql_type.go).
The operator behavior has to be implemented at least in [k8sres.go](../pkg/cluster/k8sres.go).
Please, reflect your changes in tests, for example in:
* [config_test.go](../pkg/util/config/config_test.go)
* [k8sres_test.go](../pkg/cluster/k8sres_test.go)
* [util_test.go](../pkg/apis/acid.zalan.do/v1/util_test.go)
Finally, document the new configuration option(s) for the operator in its
[reference](reference/operator_parameters.md) document and explain the feature
in the [administrator docs](administrator.md).
### Updating manifest files
For the CRD-based configuration, please update the following files:
* the default [OperatorConfiguration](../manifests/postgresql-operator-default-configuration.yaml)
* the Helm chart's [values-crd file](../charts/postgres-operator/values.yaml)
Reflect the changes in the ConfigMap configuration as well (note that numeric
and boolean parameters have to use double quotes here):
* [ConfigMap](../manifests/configmap.yaml) manifest
* the Helm chart's default [values file](../charts/postgres-operator/values.yaml)
### Updating documentation
Finally, add a section for each new configuration option and/or cluster manifest
parameter in the reference documents:
* [config reference](reference/operator_parameters.md)
* [manifest reference](reference/cluster_manifest.md)
It also helps users to explain new features with examples in the
[administrator docs](administrator.md).

59
docs/index.md
View File

@ -1,49 +1,68 @@
# Introduction
# Concepts
The Postgres [operator](https://coreos.com/blog/introducing-operators.html)
manages PostgreSQL clusters on Kubernetes:
manages PostgreSQL clusters on Kubernetes (K8s):
1. The operator watches additions, updates, and deletions of PostgreSQL cluster
manifests and changes the running clusters accordingly. For example, when a
user submits a new manifest, the operator fetches that manifest and spawns a
new Postgres cluster along with all necessary entities such as Kubernetes
new Postgres cluster along with all necessary entities such as K8s
StatefulSets and Postgres roles. See this
[Postgres cluster manifest](https://github.com/zalando/postgres-operator/blob/master/manifests/complete-postgres-manifest.yaml)
[Postgres cluster manifest](../manifests/complete-postgres-manifest.yaml)
for settings that a manifest may contain.
2. The operator also watches updates to [its own configuration](https://github.com/zalando/postgres-operator/blob/master/manifests/configmap.yaml)
and alters running Postgres clusters if necessary. For instance, if a pod
docker image is changed, the operator carries out the rolling update. That
is, the operator re-spawns one-by-one pods of each StatefulSet it manages
2. The operator also watches updates to [its own configuration](../manifests/configmap.yaml)
and alters running Postgres clusters if necessary. For instance, if the
docker image in a pod is changed, the operator carries out the rolling
update, which means it re-spawns pods of each managed StatefulSet one-by-one
with the new Docker image.
3. Finally, the operator periodically synchronizes the actual state of each
Postgres cluster with the desired state defined in the cluster's manifest.
## Concepts
4. The operator aims to be hands free as configuration works only via manifests.
This enables easy integration in automated deploy pipelines with no access to
K8s directly.
### Scope
## Scope
The scope of the postgres operator is on provisioning, modifying configuration
The scope of the Postgres Operator is on provisioning, modifying configuration
and cleaning up Postgres clusters that use Patroni, basically to make it easy
and convenient to run Patroni based clusters on Kubernetes. The provisioning
and modifying includes Kubernetes resources on one side but also e.g. database
and convenient to run Patroni based clusters on K8s. The provisioning
and modifying includes K8s resources on one side but also e.g. database
and role provisioning once the cluster is up and running. We try to leave as
much work as possible to Kubernetes and to Patroni where it fits, especially
much work as possible to K8s and to Patroni where it fits, especially
the cluster bootstrap and high availability. The operator is however involved
in some overarching orchestration, like rolling updates to improve the user
experience.
Monitoring of clusters is not in scope, for this good tools already exist from
ZMON to Prometheus and more Postgres specific options.
Monitoring or tuning Postgres is not in scope of the operator in the current
state. Other tools like [ZMON](https://opensource.zalando.com/zmon/),
[Prometheus](https://prometheus.io/) or more Postgres specific options can be
used to complement it.
### Status
## Overview of involved entities
Here is a diagram, that summarizes what would be created by the operator, when a
new Postgres cluster CRD is submitted:
![postgresql-operator](diagrams/operator.png "K8S resources, created by operator")
This picture is not complete without an overview of what is inside a single
cluster pod, so let's zoom in:
![pod](diagrams/pod.png "Database pod components")
These two diagrams should help you to understand the basics of what kind of
functionality the operator provides.
## Status
This project is currently in active development. It is however already
[used internally by Zalando](https://jobs.zalando.com/tech/blog/postgresql-in-a-time-of-kubernetes/)
in order to run Postgres clusters on Kubernetes in larger numbers for staging
in order to run Postgres clusters on K8s in larger numbers for staging
environments and a growing number of production clusters. In this environment
the operator is deployed to multiple Kubernetes clusters, where users deploy
the operator is deployed to multiple K8s clusters, where users deploy
manifests via our CI/CD infrastructure or rely on a slim user interface to
create manifests.
@ -59,4 +78,4 @@ Please, report any issues discovered to https://github.com/zalando/postgres-oper
4. "Blue elephant on-demand: Postgres + Kubernetes" talk by Oleksii Kliukin and Jan Mussler, FOSDEM 2018: [video](https://fosdem.org/2018/schedule/event/blue_elephant_on_demand_postgres_kubernetes/) | [slides (pdf)](https://www.postgresql.eu/events/fosdem2018/sessions/session/1735/slides/59/FOSDEM%202018_%20Blue_Elephant_On_Demand.pdf)
3. "Kube-Native Postgres" talk by Josh Berkus, KubeCon 2017: [video](https://www.youtube.com/watch?v=Zn1vd7sQ_bc)
5. "Kube-Native Postgres" talk by Josh Berkus, KubeCon 2017: [video](https://www.youtube.com/watch?v=Zn1vd7sQ_bc)

150
docs/quickstart.md
View File

@ -1,54 +1,82 @@
## Prerequisites:
# Quickstart
In order to run the Postgres Operator locally in minikube you need to install
the following tools:
This guide aims to give you a quick look and feel for using the Postgres
Operator on a local Kubernetes environment.
* [minikube](https://github.com/kubernetes/minikube/releases)
* [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl-binary-via-curl)
## Prerequisites
Note that you can also use built-in Kubernetes support in the Docker Desktop
Since the Postgres Operator is designed for the Kubernetes (K8s) framework,
hence set it up first. For local tests we recommend to use one of the following
solutions:
* [minikube](https://github.com/kubernetes/minikube/releases), which creates a
single-node K8s cluster inside a VM (requires KVM or VirtualBox),
* [kind](https://kind.sigs.k8s.io/), which allows creating multi-nodes K8s
clusters running on Docker (requires Docker)
To interact with the K8s infrastructure install it's CLI runtime [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl-binary-via-curl).
This quickstart assumes that you have started minikube or created a local kind
cluster. Note that you can also use built-in K8s support in the Docker Desktop
for Mac to follow the steps of this tutorial. You would have to replace
`minikube start` and `minikube delete` with your launch actions for the Docker
built-in Kubernetes support.
built-in K8s support.
Clone the repository and change to the directory. Then start minikube.
## Configuration Options
Configuring the Postgres Operator is only possible before deploying a new
Postgres cluster. This can work in two ways: via a ConfigMap or a custom
`OperatorConfiguration` object. More details on configuration can be found
[here](reference/operator_parameters.md).
## Deployment options
The Postgres Operator can be deployed in the following ways:
* Manual deployment
* Helm chart
* Operator Lifecycle Manager (OLM)
### Manual deployment setup
The Postgres Operator can be installed simply by applying yaml manifests. Note,
we provide the `/manifests` directory as an example only; you should consider
adjusting the manifests to your K8s environment (e.g. namespaces).
```bash
# First, clone the repository and change to the directory
git clone https://github.com/zalando/postgres-operator.git
cd postgres-operator
minikube start
```
If you want to configure the Postgres Operator it must happen before deploying a
Postgres cluster. This can happen in two ways: Via a ConfigMap or a
`OperatorConfiguration` object, which adheres a CustomResourceDefinition (CRD).
More details on configuration can be found [here](reference/operator_parameters.md).
## Manual deployment setup
The Postgres Operator can be installed simply by applying yaml manifests.
```bash
# apply the manifests in the following order
kubectl create -f manifests/configmap.yaml # configuration
kubectl create -f manifests/operator-service-account-rbac.yaml # identity and permissions
kubectl create -f manifests/postgres-operator.yaml # deployment
```
## Helm chart
When using kubectl 1.14 or newer the mentioned manifests could be also be
bundled in one [Kustomization](https://github.com/kubernetes-sigs/kustomize)
manifest.
For convenience, we have automated starting the operator with minikube using the
`run_operator_locally` script. It applies the [`acid-minimal-cluster`](../manifests/minimal-postgres-manifest).
manifest.
```bash
./run_operator_locally.sh
```
### Helm chart
Alternatively, the operator can be installed by using the provided [Helm](https://helm.sh/)
chart which saves you the manual steps. Therefore, you would need to install
the helm CLI on your machine. After initializing helm (and its server
component Tiller) in your local cluster you can install the operator chart.
You can define a release name that is prepended to the operator resource's
names.
chart which saves you the manual steps. Therefore, install the helm CLI on your
machine. After initializing helm (and its server component Tiller) in your local
cluster you can install the operator chart. You can define a release name that
is prepended to the operator resource's names.
Use `--name zalando` to match with the default service account name as older
operator versions do not support custom names for service accounts. When relying
solely on the CRD-based configuration edit the `serviceAccount` section in the
[values yaml file](../charts/values.yaml) by setting the name to `"operator"`.
operator versions do not support custom names for service accounts. To use
CRD-based configuration you need to specify the [values-crd yaml file](../charts/values-crd.yaml).
```bash
# 1) initialize helm
@ -57,6 +85,20 @@ helm init
helm install --name zalando ./charts/postgres-operator
```
### Operator Lifecycle Manager (OLM)
The [Operator Lifecycle Manager (OLM)](https://github.com/operator-framework/operator-lifecycle-manager)
has been designed to facilitate management of K8s operators. It has to be
installed in your K8s environment. When OLM is set up simply download and deploy
the Postgres Operator with the following command:
```bash
kubectl create -f https://operatorhub.io/install/postgres-operator.yaml
```
This installs the operator in the `operators` namespace. More information can be
found on [operatorhub.io](https://operatorhub.io/operator/postgres-operator).
## Create a Postgres cluster
Starting the operator may take a few seconds. Check if the operator pod is
@ -96,48 +138,38 @@ kubectl get svc -l application=spilo -L spilo-role
## Connect to the Postgres cluster via psql
You can retrieve the host and port of the Postgres master from minikube.
Retrieve the password from the Kubernetes Secret that is created in your cluster.
You can create a port-forward on a database pod to connect to Postgres. See the
[user guide](user.md#connect-to-postgresql) for instructions. With minikube it's
also easy to retrieve the connections string from the K8s service that is
pointing to the master pod:
```bash
export HOST_PORT=$(minikube service acid-minimal-cluster --url | sed 's,.*/,,')
export PGHOST=$(echo $HOST_PORT | cut -d: -f 1)
export PGPORT=$(echo $HOST_PORT | cut -d: -f 2)
```
Retrieve the password from the K8s Secret that is created in your cluster.
```bash
export PGPASSWORD=$(kubectl get secret postgres.acid-minimal-cluster.credentials -o 'jsonpath={.data.password}' | base64 -d)
psql -U postgres
```
## Delete a Postgres cluster
To delete a Postgres cluster simply delete the postgresql custom resource.
To delete a Postgres cluster simply delete the `postgresql` custom resource.
```bash
kubectl delete postgresql acid-minimal-cluster
# tear down cleanly
minikube delete
```
This should remove the associated StatefulSet, database Pods, Services and
Endpoints. The PersistentVolumes are released and the PodDisruptionBudget is
deleted. Secrets however are not deleted and backups will remain in place.
## Running and testing the operator
The best way to test the operator is to run it in [minikube](https://kubernetes.io/docs/getting-started-guides/minikube/).
Minikube is a tool to run Kubernetes cluster locally.
For convenience, we have automated starting the operator and submitting the
`acid-minimal-cluster`. From inside the cloned repository execute the
`run_operator_locally` shell script.
```bash
./run_operator_locally.sh
```
Note we provide the `/manifests` directory as an example only; you should
consider adjusting the manifests to your particular setting.
### Configuration Options
The operator can be configured with the provided ConfigMap
(`manifests/configmap.yaml`) or the operator's own CRD. See
[developer docs](developer.md) for details.
When deleting a cluster while it is still starting up or got stuck during that
phase it can [happen](https://github.com/zalando/postgres-operator/issues/551)
that the `postgresql` resource is deleted leaving orphaned components behind.
This can cause troubles when creating a new Postgres cluster. For a fresh setup
you can delete your local minikube or kind cluster and start again.

141
docs/reference/cluster_manifest.md
View File

@ -1,10 +1,12 @@
Individual postgres clusters are described by the Kubernetes *cluster manifest*
that has the structure defined by the `postgres CRD` (custom resource
# Cluster manifest reference
Individual Postgres clusters are described by the Kubernetes *cluster manifest*
that has the structure defined by the `postgresql` CRD (custom resource
definition). The following section describes the structure of the manifest and
the purpose of individual keys. You can take a look at the examples of the
[minimal](https://github.com/zalando/postgres-operator/blob/master/manifests/minimal-postgres-manifest.yaml)
[minimal](../manifests/minimal-postgres-manifest.yaml)
and the
[complete](https://github.com/zalando/postgres-operator/blob/master/manifests/complete-postgres-manifest.yaml)
[complete](../manifests/complete-postgres-manifest.yaml)
cluster manifests.
When Kubernetes resources, such as memory, CPU or volumes, are configured,
@ -19,7 +21,7 @@ applied.
## Manifest structure
A postgres manifest is a `YAML` document. On the top level both individual
A Postgres manifest is a `YAML` document. On the top level both individual
parameters and parameter groups can be defined. Parameter names are written
in camelCase.
@ -54,26 +56,31 @@ These parameters are grouped directly under the `spec` key in the manifest.
name of the team the cluster belongs to. Changing it after the cluster
creation is not supported. Required field.
* **numberOfInstances**
total number of instances for a given cluster. The operator parameters
`max_instances` and `min_instances` may also adjust this number. Required
field.
* **dockerImage**
custom docker image that overrides the **docker_image** operator parameter.
It should be a [Spilo](https://github.com/zalando/spilo) image. Optional.
It should be a [Spilo](https://github.com/zalando/spilo) image. Optional.
* **spiloFSGroup**
the Persistent Volumes for the spilo pods in the StatefulSet will be owned
and writable by the group ID specified. This will override the **spilo_fsgroup**
the Persistent Volumes for the Spilo pods in the StatefulSet will be owned and
writable by the group ID specified. This will override the **spilo_fsgroup**
operator parameter. This is required to run Spilo as a non-root process, but
requires a custom spilo image. Note the FSGroup of a Pod cannot be changed
without recreating a new Pod.
requires a custom Spilo image. Note the FSGroup of a Pod cannot be changed
without recreating a new Pod. Optional.
* **enableMasterLoadBalancer**
boolean flag to override the operator defaults (set by the
`enable_master_load_balancer` parameter) to define whether to enable the load
balancer pointing to the postgres primary. Optional.
balancer pointing to the Postgres primary. Optional.
* **enableReplicaLoadBalancer**
boolean flag to override the operator defaults (set by the
`enable_replica_load_balancer` parameter) to define whether to enable the
load balancer pointing to the postgres standby instances. Optional.
load balancer pointing to the Postgres standby instances. Optional.
* **allowedSourceRanges**
when one or more load balancers are enabled for the cluster, this parameter
@ -82,11 +89,6 @@ These parameters are grouped directly under the `spec` key in the manifest.
this parameter. Optional, when empty the load balancer service becomes
inaccessible from outside of the Kubernetes cluster.
* **numberOfInstances**
total number of instances for a given cluster. The operator parameters
`max_instances` and `min_instances` may also adjust this number. Required
field.
* **users**
a map of usernames to user flags for the users that should be created in the
cluster by the operator. User flags are a list, allowed elements are
@ -114,7 +116,7 @@ These parameters are grouped directly under the `spec` key in the manifest.
that should be assigned to the cluster pods. When not specified, the value
is taken from the `pod_priority_class_name` operator parameter, if not set
then the default priority class is taken. The priority class itself must be
defined in advance.
defined in advance. Optional.
* **enableShmVolume**
Start a database pod without limitations on shm memory. By default docker
@ -126,35 +128,36 @@ These parameters are grouped directly under the `spec` key in the manifest.
about mounting a volume will be made based on operator configuration
(`enable_shm_volume`, which is `true` by default). It it's present and value
is `false`, then no volume will be mounted no matter how operator was
configured (so you can override the operator configuration).
configured (so you can override the operator configuration). Optional.
* **enableLogicalBackup**
Determines if the logical backup of this cluster should be taken and uploaded
to S3. Default: false.
to S3. Default: false. Optional.
* **logicalBackupSchedule**
Schedule for the logical backup k8s cron job. Please take [the reference schedule format](https://kubernetes.io/docs/tasks/job/automated-tasks-with-cron-jobs/#schedule)
into account. Default: "30 00 \* \* \*"
Schedule for the logical backup k8s cron job. Please take
[the reference schedule format](https://kubernetes.io/docs/tasks/job/automated-tasks-with-cron-jobs/#schedule)
into account. Optional. Default is: "30 00 \* \* \*"
## Postgres parameters
Those parameters are grouped under the `postgresql` top-level key.
Those parameters are grouped under the `postgresql` top-level key, which is
required in the manifest.
* **version**
the postgres major version of the cluster. Looks at the [Spilo
the Postgres major version of the cluster. Looks at the [Spilo
project](https://github.com/zalando/spilo/releases) for the list of supported
versions. Changing the cluster version once the cluster has been bootstrapped
is not supported. Required field.
* **parameters**
a dictionary of postgres parameter names and values to apply to the resulting
cluster. Optional (Spilo automatically sets reasonable defaults for
parameters like work_mem or max_connections).
a dictionary of Postgres parameter names and values to apply to the resulting
cluster. Optional (Spilo automatically sets reasonable defaults for parameters
like `work_mem` or `max_connections`).
## Patroni parameters
Those parameters are grouped under the `patroni` top-level key. See the [patroni
Those parameters are grouped under the `patroni` top-level key. See the [Patroni
documentation](https://patroni.readthedocs.io/en/latest/SETTINGS.html) for the
explanation of `ttl` and `loop_wait` parameters.
@ -176,19 +179,19 @@ explanation of `ttl` and `loop_wait` parameters.
authentication. Optional.
* **ttl**
patroni `ttl` parameter value, optional. The default is set by the Spilo
Patroni `ttl` parameter value, optional. The default is set by the Spilo
docker image. Optional.
* **loop_wait**
patroni `loop_wait` parameter value, optional. The default is set by the
Patroni `loop_wait` parameter value, optional. The default is set by the
Spilo docker image. Optional.
* **retry_timeout**
patroni `retry_timeout` parameter value, optional. The default is set by the
Patroni `retry_timeout` parameter value, optional. The default is set by the
Spilo docker image. Optional.
* **maximum_lag_on_failover**
patroni `maximum_lag_on_failover` parameter value, optional. The default is
Patroni `maximum_lag_on_failover` parameter value, optional. The default is
set by the Spilo docker image. Optional.
* **slots**
@ -199,38 +202,34 @@ explanation of `ttl` and `loop_wait` parameters.
automatically created by Patroni for cluster members and permanent replication
slots. Optional.
* **standby**
initializes cluster as a standby creating a cascading replication, where standby leader is streaming from specified remote location
## Postgres container resources
Those parameters define [CPU and memory requests and
limits](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/)
for the postgres container. They are grouped under the `resources` top-level
key. There are two subgroups, `requests` and `limits`.
Those parameters define [CPU and memory requests and limits](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/)
for the Postgres container. They are grouped under the `resources` top-level
key with subgroups `requests` and `limits`.
### Requests
CPU and memory requests for the postgres container.
CPU and memory requests for the Postgres container.
* **cpu**
CPU requests for the postgres container. Optional, overrides the
CPU requests for the Postgres container. Optional, overrides the
`default_cpu_requests` operator configuration parameter. Optional.
* **memory**
memory requests for the postgres container. Optional, overrides the
memory requests for the Postgres container. Optional, overrides the
`default_memory_request` operator configuration parameter. Optional.
#### Limits
### Limits
CPU and memory limits for the postgres container.
CPU and memory limits for the Postgres container.
* **cpu**
CPU limits for the postgres container. Optional, overrides the
CPU limits for the Postgres container. Optional, overrides the
`default_cpu_limits` operator configuration parameter. Optional.
* **memory**
memory limits for the postgres container. Optional, overrides the
memory limits for the Postgres container. Optional, overrides the
`default_memory_limits` operator configuration parameter. Optional.
## Parameters defining how to clone the cluster from another one
@ -257,8 +256,13 @@ under the `clone` top-level key and do not affect the already running cluster.
timestamp. When this parameter is set the operator will not consider cloning
from the live cluster, even if it is running, and instead goes to S3. Optional.
* **s3_wal_path**
the url to S3 bucket containing the WAL archive of the cluster to be cloned.
Optional.
* **s3_endpoint**
the url of the S3-compatible service should be set when cloning from non AWS S3. Optional.
the url of the S3-compatible service should be set when cloning from non AWS
S3. Optional.
* **s3_access_key_id**
the access key id, used for authentication on S3 service. Optional.
@ -267,13 +271,25 @@ under the `clone` top-level key and do not affect the already running cluster.
the secret access key, used for authentication on S3 service. Optional.
* **s3_force_path_style**
to enable path-style addressing(i.e., http://s3.amazonaws.com/BUCKET/KEY) when connecting to an S3-compatible service
that lack of support for sub-domain style bucket URLs (i.e., http://BUCKET.s3.amazonaws.com/KEY). Optional.
to enable path-style addressing(i.e., http://s3.amazonaws.com/BUCKET/KEY)
when connecting to an S3-compatible service that lack of support for
sub-domain style bucket URLs (i.e., http://BUCKET.s3.amazonaws.com/KEY).
Optional.
### EBS volume resizing
## Standby cluster
On startup, an existing `standby` top-level key creates a standby Postgres
cluster streaming from a remote location. So far only streaming from a S3 WAL
archive is supported.
* **s3_wal_path**
the url to S3 bucket containing the WAL archive of the remote primary.
Required when the `standby` section is present.
## EBS volume resizing
Those parameters are grouped under the `volume` top-level key and define the
properties of the persistent storage that stores postgres data.
properties of the persistent storage that stores Postgres data.
* **size**
the size of the target EBS volume. Usual Kubernetes size modifiers, i.e. `Gi`
@ -286,13 +302,13 @@ properties of the persistent storage that stores postgres data.
for the details on storage classes. Optional.
* **subPath**
Subpath to use when mounting volume into Spilo container
Subpath to use when mounting volume into Spilo container. Optional.
### Sidecar definitions
## Sidecar definitions
Those parameters are defined under the `sidecars` key. They consist of a list
of dictionaries, each defining one sidecar (an extra container running
along the main postgres container on the same pod). The following keys can be
along the main Postgres container on the same pod). The following keys can be
defined in the sidecar dictionary:
* **name**
@ -306,16 +322,11 @@ defined in the sidecar dictionary:
(https://kubernetes.io/docs/tasks/inject-data-application/environment-variable-expose-pod-information/)
for environment variables. Optional.
* **resources** see below. Optional.
* **resources**
[CPU and memory requests and limits](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container)
for each sidecar container. Optional.
#### Sidecar container resources
Those parameters define [CPU and memory requests and
limits](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/)
for the sidecar container. They are grouped under the `resources` key for each sidecar.
There are two subgroups, `requests` and `limits`.
##### Requests
### Requests
CPU and memory requests for the sidecar container.
@ -327,7 +338,7 @@ CPU and memory requests for the sidecar container.
memory requests for the sidecar container. Optional, overrides the
`default_memory_request` operator configuration parameter. Optional.
##### Limits
### Limits
CPU and memory limits for the sidecar container.

6
docs/reference/command_line_and_environment.md
View File

@ -50,9 +50,9 @@ The following environment variables are accepted by the operator:
`scalyr_api_key` operator parameter.
* **CRD_READY_WAIT_TIMEOUT**
defines the timeout for the complete postgres CRD creation. When not set
defines the timeout for the complete `postgresql` CRD creation. When not set
default is 30s.
* **CRD_READY_WAIT_INTERVAL**
defines the interval between consecutive attempts waiting for the postgres
CRD to be created. The default is 5s.
defines the interval between consecutive attempts waiting for the
`postgresql` CRD to be created. The default is 5s.

117
docs/reference/operator_parameters.md
View File

@ -1,3 +1,5 @@
# Configuration parameters
There are two mutually-exclusive methods to set the Postgres Operator
configuration.
@ -8,12 +10,12 @@ configuration.
maps. String values containing ':' should be enclosed in quotes. The
configuration is flat, parameter group names below are not reflected in the
configuration structure. There is an
[example](https://github.com/zalando/postgres-operator/blob/master/manifests/configmap.yaml)
[example](../manifests/configmap.yaml)
* CRD-based configuration. The configuration is stored in a custom YAML
manifest. The manifest is an instance of the custom resource definition (CRD)
called `OperatorConfiguration`. The operator registers this CRD during the
start and uses it for configuration if the [operator deployment manifest ](https://github.com/zalando/postgres-operator/blob/master/manifests/postgres-operator.yaml#L21)
start and uses it for configuration if the [operator deployment manifest](../manifests/postgres-operator.yaml#L36)
sets the `POSTGRES_OPERATOR_CONFIGURATION_OBJECT` env variable to a non-empty
value. The variable should point to the `postgresql-operator-configuration`
object in the operator's namespace.
@ -22,7 +24,7 @@ configuration.
simply represented in the usual YAML way. There are no default values built-in
in the operator, each parameter that is not supplied in the configuration
receives an empty value. In order to create your own configuration just copy
the [default one](https://github.com/zalando/postgres-operator/blob/master/manifests/postgresql-operator-default-configuration.yaml)
the [default one](../manifests/postgresql-operator-default-configuration.yaml)
and change it.
To test the CRD-based configuration locally, use the following
@ -32,10 +34,10 @@ configuration.
kubectl create -f manifests/postgresql-operator-default-configuration.yaml
kubectl get operatorconfigurations postgresql-operator-default-configuration -o yaml
```
Note that the operator first attempts to register the CRD of the `OperatorConfiguration`
and then waits for an instance to be created. In between these two event the
operator pod may be failing since it cannot fetch the not-yet-existing
`OperatorConfiguration` instance.
Note that the operator first attempts to register the CRD of the
`OperatorConfiguration` and then waits for an instance to be created. In
between these two event the operator pod may be failing since it cannot fetch
the not-yet-existing `OperatorConfiguration` instance.
The CRD-based configuration is more powerful than the one based on ConfigMaps
and should be used unless there is a compatibility requirement to use an already
@ -56,11 +58,11 @@ parameters, those parameters have no effect and are replaced by the
`CRD_READY_WAIT_INTERVAL` and `CRD_READY_WAIT_TIMEOUT` environment variables.
They will be deprecated and removed in the future.
For the configmap operator configuration, the [default parameter values](https://github.com/zalando-incubator/postgres-operator/blob/master/pkg/util/config/config.go#L14)
For the configmap configuration, the [default parameter values](../pkg/util/config/config.go#L14)
mentioned here are likely to be overwritten in your local operator installation
via your local version of the operator configmap. In the case you use the
operator CRD, all the CRD defaults are provided in the
[operator's default configuration manifest](https://github.com/zalando-incubator/postgres-operator/blob/master/manifests/postgresql-operator-default-configuration.yaml)
[operator's default configuration manifest](../manifests/postgresql-operator-default-configuration.yaml)
Variable names are underscore-separated words.
@ -75,7 +77,7 @@ Those are top-level keys, containing both leaf keys and groups.
Kubernetes-native DCS).
* **docker_image**
Spilo docker image for postgres instances. For production, don't rely on the
Spilo docker image for Postgres instances. For production, don't rely on the
default image, as it might be not the most up-to-date one. Instead, build
your own Spilo image from the [github
repository](https://github.com/zalando/spilo).
@ -98,12 +100,12 @@ Those are top-level keys, containing both leaf keys and groups.
create/update/delete/sync clusters concurrently. The default is `4`.
* **max_instances**
operator will cap the number of instances in any managed postgres cluster up
operator will cap the number of instances in any managed Postgres cluster up
to the value of this parameter. When `-1` is specified, no limits are applied.
The default is `-1`.
* **min_instances**
operator will run at least the number of instances for any given postgres
operator will run at least the number of instances for any given Postgres
cluster equal to the value of this parameter. When `-1` is specified, no
limits are applied. The default is `-1`.
@ -120,7 +122,7 @@ Those are top-level keys, containing both leaf keys and groups.
containers with high memory limits due to the lack of memory on Kubernetes
cluster nodes. This affects all containers created by the operator (Postgres,
Scalyr sidecar, and other sidecars); to set resources for the operator's own
container, change the [operator deployment manually](https://github.com/zalando/postgres-operator/blob/master/manifests/postgres-operator.yaml#L13).
container, change the [operator deployment manually](../manifests/postgres-operator.yaml#L20).
The default is `false`.
## Postgres users
@ -129,11 +131,11 @@ Parameters describing Postgres users. In a CRD-configuration, they are grouped
under the `users` key.
* **super_username**
postgres `superuser` name to be created by `initdb`. The default is
Postgres `superuser` name to be created by `initdb`. The default is
`postgres`.
* **replication_username**
postgres username used for replication between instances. The default is
Postgres username used for replication between instances. The default is
`standby`.
## Kubernetes resources
@ -158,17 +160,16 @@ configuration they are grouped under the `kubernetes` key.
This definition must bind pod service account to a role with permission
sufficient for the pods to start and for Patroni to access k8s endpoints;
service account on its own lacks any such rights starting with k8s v1.8. If
not excplicitly defined by the user, a simple definition that binds the
not explicitly defined by the user, a simple definition that binds the
account to the operator's own 'zalando-postgres-operator' cluster role will
be used. The default is empty.
* **pod_terminate_grace_period**
Postgres pods are [terminated
forcefully](https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods)
Postgres pods are [terminated forcefully](https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods)
after this timeout. The default is `5m`.
* **watched_namespace**
The operator watches for postgres objects in the given namespace. If not
The operator watches for Postgres objects in the given namespace. If not
specified, the value is taken from the operator namespace. A special `*`
value makes it watch all namespaces. The default is empty (watch the operator
pod namespace).
@ -195,9 +196,9 @@ configuration they are grouped under the `kubernetes` key.
`{username}.{cluster}.credentials.{tprkind}.{tprgroup}`.
* **cluster_domain**
defines the default dns domain for the kubernetes cluster the operator is
defines the default DNS domain for the kubernetes cluster the operator is
running in. The default is `cluster.local`. Used by the operator to connect
to the postgres clusters after creation.
to the Postgres clusters after creation.
* **oauth_token_secret_name**
a name of the secret containing the `OAuth2` token to pass to the teams API.
@ -217,9 +218,8 @@ configuration they are grouped under the `kubernetes` key.
* **inherited_labels**
list of labels that can be inherited from the cluster manifest, and added to
each child objects (`StatefulSet`, `Pod`, `Service` and `Endpoints`) created
by the opertor.
Typical use case is to dynamically pass labels that are specific to a given
postgres cluster, in order to implement `NetworkPolicy`.
by the operator. Typical use case is to dynamically pass labels that are
specific to a given Postgres cluster, in order to implement `NetworkPolicy`.
The default is empty.
* **cluster_name_label**
@ -243,7 +243,7 @@ configuration they are grouped under the `kubernetes` key.
* **pod_environment_configmap**
a name of the ConfigMap with environment variables to populate on every pod.
Right now this ConfigMap is searched in the namespace of the postgres cluster.
Right now this ConfigMap is searched in the namespace of the Postgres cluster.
All variables from that ConfigMap are injected to the pod's environment, on
conflicts they are overridden by the environment variables generated by the
operator. The default is empty.
@ -254,8 +254,9 @@ configuration they are grouped under the `kubernetes` key.
be defined in advance. Default is empty (use the default priority class).
* **spilo_fsgroup**
the Persistent Volumes for the spilo pods in the StatefulSet will be owned and writable by the group ID specified.
This is required to run Spilo as a non-root process, but requires a custom spilo image. Note the FSGroup of a Pod
the Persistent Volumes for the Spilo pods in the StatefulSet will be owned and
writable by the group ID specified. This is required to run Spilo as a
non-root process, but requires a custom Spilo image. Note the FSGroup of a Pod
cannot be changed without recreating a new Pod.
* **spilo_privileged**
@ -291,19 +292,19 @@ Those parameters are grouped under the `postgres_pod_resources` key in a
CRD-based configuration.
* **default_cpu_request**
CPU request value for the postgres containers, unless overridden by
CPU request value for the Postgres containers, unless overridden by
cluster-specific settings. The default is `100m`.
* **default_memory_request**
memory request value for the postgres containers, unless overridden by
memory request value for the Postgres containers, unless overridden by
cluster-specific settings. The default is `100Mi`.
* **default_cpu_limit**
CPU limits for the postgres containers, unless overridden by cluster-specific
CPU limits for the Postgres containers, unless overridden by cluster-specific
settings. The default is `3`.
* **default_memory_limit**
memory limits for the postgres containers, unless overridden by cluster-specific
memory limits for the Postgres containers, unless overridden by cluster-specific
settings. The default is `1Gi`.
## Operator timeouts
@ -334,11 +335,11 @@ CRD-based configuration.
cluster or recreating pods. The default is `10m`.
* **ready_wait_interval**
the interval between consecutive attempts waiting for the postgres CRD to be
created. The default is `5s`.
the interval between consecutive attempts waiting for the `postgresql` CRD to
be created. The default is `5s`.
* **ready_wait_timeout**
the timeout for the complete postgres CRD creation. The default is `30s`.
the timeout for the complete `postgresql` CRD creation. The default is `30s`.
## Load balancer related options
@ -394,7 +395,7 @@ yet officially supported.
Spilo are S3 and GCS. The default is empty.
* **log_s3_bucket**
S3 bucket to use for shipping postgres daily logs. Works only with S3 on AWS.
S3 bucket to use for shipping Postgres daily logs. Works only with S3 on AWS.
The bucket has to be present and accessible by Postgres pods. The default is
empty.
@ -413,6 +414,28 @@ yet officially supported.
* **additional_secret_mount_path**
Path to mount the above Secret in the filesystem of the container(s). The default is empty.
## Logical backup
These parameters configure a k8s cron job managed by the operator to produce
Postgres logical backups. In the CRD-based configuration those parameters are
grouped under the `logical_backup` key.
* **logical_backup_schedule**
Backup schedule in the cron format. Please take the
[reference schedule format](https://kubernetes.io/docs/tasks/job/automated-tasks-with-cron-jobs/#schedule)
into account. Default: "30 00 \* \* \*"
* **logical_backup_docker_image**
An image for pods of the logical backup job. The [example image](../../docker/logical-backup/Dockerfile)
runs `pg_dumpall` on a replica if possible and uploads compressed results to
an S3 bucket under the key `/spilo/pg_cluster_name/cluster_k8s_uuid/logical_backups`.
The default image is the same image built with the Zalando-internal CI
pipeline. Default: "registry.opensource.zalan.do/acid/logical-backup"
* **logical_backup_s3_bucket**
S3 bucket to store backup results. The bucket has to be present and
accessible by Postgres pods. Default: empty.
## Debugging the operator
Options to aid debugging of the operator itself. Grouped under the `debug` key.
@ -423,7 +446,7 @@ Options to aid debugging of the operator itself. Grouped under the `debug` key.
* **enable_database_access**
boolean parameter that toggles the functionality of the operator that require
access to the postgres database, i.e. creating databases and users. The
access to the Postgres database, i.e. creating databases and users. The
default is `true`.
## Automatic creation of human users in the database
@ -442,7 +465,7 @@ key.
`https://teams.example.com/api/`.
* **team_api_role_configuration**
postgres parameters to apply to each team member role. The default is
Postgres parameters to apply to each team member role. The default is
'*log_statement:all*'. It is possible to supply multiple options, separating
them by commas. Options containing commas within the value are not supported,
with the exception of the `search_path`. For instance:
@ -527,23 +550,3 @@ scalyr sidecar. In the CRD-based configuration they are grouped under the
* **scalyr_memory_limit**
Memory limit value for the Scalyr sidecar. The default is `1Gi`.
## Logical backup
These parameters configure a k8s cron job managed by the operator to produce
Postgres logical backups. In the CRD-based configuration those parameters are
grouped under the `logical_backup` key.
* **logical_backup_schedule**
Backup schedule in the cron format. Please take [the reference schedule format](https://kubernetes.io/docs/tasks/job/automated-tasks-with-cron-jobs/#schedule) into account. Default: "30 00 \* \* \*"
* **logical_backup_docker_image**
An image for pods of the logical backup job. The [example image](../../docker/logical-backup/Dockerfile)
runs `pg_dumpall` on a replica if possible and uploads compressed results to
an S3 bucket under the key `/spilo/pg_cluster_name/cluster_k8s_uuid/logical_backups`.
The default image is the same image built with the Zalando-internal CI
pipeline. Default: "registry.opensource.zalan.do/acid/logical-backup"
* **logical_backup_s3_bucket**
S3 bucket to store backup results. The bucket has to be present and
accessible by Postgres pods. Default: empty.

165
docs/user.md
View File

@ -1,7 +1,11 @@
# User Guide
Learn how to work with the Postgres Operator in a Kubernetes (K8s) environment.
## Create a manifest for a new PostgreSQL cluster
As an example you can take this
[minimal example](https://github.com/zalando/postgres-operator/blob/master/manifests/minimal-postgres-manifest.yaml):
Make sure you have [set up](quickstart.md) the operator. Then you can create a
new Postgres cluster by applying manifest like this [minimal example](../manifests/minimal-postgres-manifest.yaml):
```yaml
apiVersion: "acid.zalan.do/v1"
@ -29,27 +33,19 @@ spec:
version: "10"
```
## Create a new Spilo cluster
Once you cloned the Postgres Operator [repository](https://github.com/zalando/postgres-operator)
you can find this example also in the manifests folder:
```bash
$ kubectl create -f manifests/minimal-postgres-manifest.yaml
kubectl create -f manifests/minimal-postgres-manifest.yaml
```
## Watch pods being created
```bash
$ kubectl get pods -w --show-labels
kubectl get pods -w --show-labels
```
## Give K8S users access to create/list postgresqls
```bash
$ kubectl create -f manifests/user-facing-clusterroles.yaml
```
Creates zalando-postgres-operator:users:view, :edit and :admin clusterroles that are
aggregated into the default roles.
## Connect to PostgreSQL
With a `port-forward` on one of the database pods (e.g. the master) you can
@ -69,33 +65,37 @@ Open another CLI and connect to the database. Use the generated secret of the
in Minikube:
```bash
$ export PGPASSWORD=$(kubectl get secret postgres.acid-minimal-cluster.credentials -o 'jsonpath={.data.password}' | base64 -d)
$ psql -U postgres -p 6432
export PGPASSWORD=$(kubectl get secret postgres.acid-minimal-cluster.credentials -o 'jsonpath={.data.password}' | base64 -d)
psql -U postgres -p 6432
```
# Defining database roles in the operator
## Defining database roles in the operator
Postgres operator allows defining roles to be created in the resulting database
Postgres Operator allows defining roles to be created in the resulting database
cluster. It covers three use-cases:
* `manifest roles`: create application roles specific to the cluster described in the manifest.
* `infrastructure roles`: create application roles that should be automatically created on every
* `manifest roles`: create application roles specific to the cluster described
in the manifest.
* `infrastructure roles`: create application roles that should be automatically
created on every
cluster managed by the operator.
* `teams API roles`: automatically create users for every member of the team owning the database
cluster.
* `teams API roles`: automatically create users for every member of the team
owning the database cluster.
In the next sections, we will cover those use cases in more details.
## Manifest roles
### Manifest roles
Manifest roles are defined directly in the cluster manifest. See
[minimal postgres manifest](https://github.com/zalando/postgres-operator/blob/master/manifests/minimal-postgres-manifest.yaml)
for an example of `zalando` role, defined with `superuser` and `createdb`
flags.
[minimal postgres manifest](../manifests/minimal-postgres-manifest.yaml)
for an example of `zalando` role, defined with `superuser` and `createdb` flags.
Manifest roles are defined as a dictionary, with a role name as a key and a
list of role options as a value. For a role without any options it is best to supply the empty
list `[]`. It is also possible to leave this field empty as in our example manifests, but in certain cases such empty field may removed by Kubernetes [due to the `null` value it gets](https://kubernetes.io/docs/concepts/overview/object-management-kubectl/declarative-config/#how-apply-calculates-differences-and-merges-changes) (`foobar_user:` is equivalent to `foobar_user: null`).
list of role options as a value. For a role without any options it is best to
supply the empty list `[]`. It is also possible to leave this field empty as in
our example manifests. In certain cases such empty field may be missing later
removed by K8s [due to the `null` value it gets](https://kubernetes.io/docs/concepts/overview/object-management-kubectl/declarative-config/#how-apply-calculates-differences-and-merges-changes)
(`foobar_user:` is equivalent to `foobar_user: null`).
The operator accepts the following options: `superuser`, `inherit`, `login`,
`nologin`, `createrole`, `createdb`, `replication`, `bypassrls`.
@ -107,13 +107,13 @@ The operator automatically generates a password for each manifest role and
places it in the secret named
`{username}.{team}-{clustername}.credentials.postgresql.acid.zalan.do` in the
same namespace as the cluster. This way, the application running in the
Kubernetes cluster and working with the database can obtain the password right
from the secret, without ever sharing it outside of the cluster.
K8s cluster and connecting to Postgres can obtain the password right from the
secret, without ever sharing it outside of the cluster.
At the moment it is not possible to define membership of the manifest role in
other roles.
## Infrastructure roles
### Infrastructure roles
An infrastructure role is a role that should be present on every PostgreSQL
cluster managed by the operator. An example of such a role is a monitoring
@ -122,7 +122,7 @@ user. There are two ways to define them:
* With the infrastructure roles secret only
* With both the the secret and the infrastructure role ConfigMap.
### Infrastructure roles secret
#### Infrastructure roles secret
The infrastructure roles secret is specified by the `infrastructure_roles_secret_name`
parameter. The role definition looks like this (values are base64 encoded):
@ -134,22 +134,21 @@ parameter. The role definition looks like this (values are base64 encoded):
```
The block above describes the infrastructure role 'dbuser' with password
'secret' that is a member of the 'operator' role. For the following
definitions one must increase the index, i.e. the next role will be defined as
'user2' and so on. The resulting role will automatically be a login role.
'secret' that is a member of the 'operator' role. For the following definitions
one must increase the index, i.e. the next role will be defined as 'user2' and
so on. The resulting role will automatically be a login role.
Note that with definitions that solely use the infrastructure roles secret
there is no way to specify role options (like superuser or nologin) or role
memberships. This is where the ConfigMap comes into play.
### Secret plus ConfigMap
#### Secret plus ConfigMap
A [ConfigMap](https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/)
allows for defining more details regarding the infrastructure roles.
Therefore, one should use the new style that specifies infrastructure roles
using both the secret and a ConfigMap. The ConfigMap must have the same name as
the secret. The secret should contain an entry with 'rolename:rolepassword' for
each role.
allows for defining more details regarding the infrastructure roles. Therefore,
one should use the new style that specifies infrastructure roles using both the
secret and a ConfigMap. The ConfigMap must have the same name as the secret.
The secret should contain an entry with 'rolename:rolepassword' for each role.
```yaml
dbuser: c2VjcmV0
@ -179,14 +178,14 @@ Since an infrastructure role is created uniformly on all clusters managed by
the operator, it makes no sense to define it without the password. Such
definitions will be ignored with a prior warning.
See [infrastructure roles secret](https://github.com/zalando/postgres-operator/blob/master/manifests/infrastructure-roles.yaml)
and [infrastructure roles configmap](https://github.com/zalando/postgres-operator/blob/master/manifests/infrastructure-roles-configmap.yaml) for the examples.
See [infrastructure roles secret](../manifests/infrastructure-roles.yaml)
and [infrastructure roles configmap](../manifests/infrastructure-roles-configmap.yaml)
for the examples.
## Use taints and tolerations for dedicated PostgreSQL nodes
To ensure Postgres pods are running on nodes without any other application
pods, you can use
[taints and tolerations](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/)
To ensure Postgres pods are running on nodes without any other application pods,
you can use [taints and tolerations](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/)
and configure the required toleration in the manifest.
```yaml
@ -208,7 +207,6 @@ You can spin up a new cluster as a clone of the existing one, using a clone
section in the spec. There are two options here:
* Clone directly from a source cluster using `pg_basebackup`
* Clone from an S3 bucket
### Clone directly
@ -247,10 +245,10 @@ spec:
```
Here `cluster` is a name of a source cluster that is going to be cloned. A new
cluster will be cloned from S3, using the latest backup before the
`timestamp`. In this case, `uid` field is also mandatory - operator will use it
to find a correct key inside an S3 bucket. You can find this field from
metadata of a source cluster:
cluster will be cloned from S3, using the latest backup before the `timestamp`.
In this case, `uid` field is also mandatory - operator will use it to find a
correct key inside an S3 bucket. You can find this field in the metadata of the
source cluster:
```yaml
apiVersion: acid.zalan.do/v1
@ -263,7 +261,8 @@ metadata:
Note that timezone is required for `timestamp`. Otherwise, offset is relative
to UTC, see [RFC 3339 section 5.6) 3339 section 5.6](https://www.ietf.org/rfc/rfc3339.txt).
For non AWS S3 following settings can be set to support cloning from other S3 implementations:
For non AWS S3 following settings can be set to support cloning from other S3
implementations:
```yaml
apiVersion: "acid.zalan.do/v1"
@ -283,7 +282,12 @@ spec:
## Setting up a standby cluster
Standby clusters are like normal cluster but they are streaming from a remote cluster. As the first version of this feature, the only scenario covered by operator is to stream from a wal archive of the master. Following the more popular infrastructure of using Amazon's S3 buckets, it is mentioned as s3_wal_path here. To make a cluster as standby add a section standby in the YAML file as follows.
Standby clusters are like normal cluster but they are streaming from a remote
cluster. As the first version of this feature, the only scenario covered by
operator is to stream from a WAL archive of the master. Following the more
popular infrastructure of using Amazon's S3 buckets, it is mentioned as
`s3_wal_path` here. To start a cluster as standby add the following `standby`
section in the YAML file:
```yaml
spec:
@ -293,15 +297,24 @@ spec:
Things to note:
- An empty string is provided in s3_wal_path of the standby cluster will result in error and no statefulset will be created.
- An empty string in the `s3_wal_path` field of the standby cluster will result
in an error and no statefulset will be created.
- Only one pod can be deployed for stand-by cluster.
- To manually promote the standby_cluster, use patronictl and remove config entry.
- There is no way to transform a non-standby cluster to standby cluster through operator. Hence, if a cluster is created without standby section in YAML and later modified by adding that section, there will be no effect on the cluster. However, it can be done through Patroni by adding the [standby_cluster] (https://github.com/zalando/patroni/blob/bd2c54581abb42a7d3a3da551edf0b8732eefd27/docs/replica_bootstrap.rst#standby-cluster) section using patronictl edit-config. Note that the transformed standby cluster will not be doing any streaming, rather will just be in standby mode and allow read-only transactions only.
- To manually promote the standby_cluster, use `patronictl` and remove config
entry.
- There is no way to transform a non-standby cluster to a standby cluster
through the operator. Adding the standby section to the manifest of a running
Postgres cluster will have no effect. However, it can be done through Patroni
by adding the [standby_cluster] (https://github.com/zalando/patroni/blob/bd2c54581abb42a7d3a3da551edf0b8732eefd27/docs/replica_bootstrap.rst#standby-cluster)
section using `patronictl edit-config`. Note that the transformed standby
cluster will not be doing any streaming. It will be in standby mode and allow
read-only transactions only.
## Sidecar Support
Each cluster can specify arbitrary sidecars to run. These containers could be used for
log aggregation, monitoring, backups or other tasks. A sidecar can be specified like this:
Each cluster can specify arbitrary sidecars to run. These containers could be
used for log aggregation, monitoring, backups or other tasks. A sidecar can be
specified like this:
```yaml
apiVersion: "acid.zalan.do/v1"
@ -326,21 +339,21 @@ spec:
value: "any-k8s-env-things"
```
In addition to any environment variables you specify, the following environment variables
are always passed to sidecars:
In addition to any environment variables you specify, the following environment
variables are always passed to sidecars:
- `POD_NAME` - field reference to `metadata.name`
- `POD_NAMESPACE` - field reference to `metadata.namespace`
- `POSTGRES_USER` - the superuser that can be used to connect to the database
- `POSTGRES_PASSWORD` - the password for the superuser
The PostgreSQL volume is shared with sidecars and is mounted at `/home/postgres/pgdata`.
The PostgreSQL volume is shared with sidecars and is mounted at
`/home/postgres/pgdata`.
## InitContainers Support
Each cluster can specify arbitrary init containers to run. These containers can be
used to run custom actions before any normal and sidecar containers start.
Each cluster can specify arbitrary init containers to run. These containers can
be used to run custom actions before any normal and sidecar containers start.
An init container can be specified like this:
```yaml
@ -361,7 +374,6 @@ spec:
`initContainers` accepts full `v1.Container` definition.
## Increase volume size
PostgreSQL operator supports statefulset volume resize if you're using the
@ -390,19 +402,26 @@ actions:
* call AWS API to change the volume size
* connect to the pod using `kubectl exec` and resize the filesystem with
`resize2fs`.
* connect to pod using `kubectl exec` and resize filesystem with `resize2fs`.
Fist step has a limitation, AWS rate-limits this operation to no more than once
every 6 hours.
Note that if the statefulset is scaled down before resizing the size changes
are only applied to the volumes attached to the running pods. The size of the
volumes that correspond to the previously running pods is not changed.
every 6 hours. Note, that if the statefulset is scaled down before resizing the
new size is only applied to the volumes attached to the running pods. The
size of volumes that correspond to the previously running pods is not changed.
## Logical backups
If you add
You can enable logical backups from the cluster manifest by adding the following
parameter in the spec section:
```
enableLogicalBackup: true
```
to the cluster manifest, the operator will create and sync a k8s cron job to do periodic logical backups of this particular Postgres cluster. Due to the [limitation of Kubernetes cron jobs](https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/#cron-job-limitations) it is highly advisable to set up additional monitoring for this feature; such monitoring is outside of the scope of operator responsibilities. See [configuration reference](reference/cluster_manifest.md) and [administrator documentation](administrator.md) for details on how backups are executed.
The operator will create and sync a K8s cron job to do periodic logical backups
of this particular Postgres cluster. Due to the [limitation of K8s cron jobs](https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/#cron-job-limitations)
it is highly advisable to set up additional monitoring for this feature; such
monitoring is outside the scope of operator responsibilities. See
[configuration reference](reference/cluster_manifest.md) and
[administrator documentation](administrator.md) for details on how backups are
executed.

26
manifests/complete-postgres-manifest.yaml
View File

@ -50,12 +50,12 @@ spec:
- hostssl all all 0.0.0.0/0 md5
- host all all 0.0.0.0/0 md5
# slots:
# permanent_physical_1:
# type: physical
# permanent_logical_1:
# type: logical
# database: foo
# plugin: pgoutput
# - permanent_physical_1:
# type: physical
# - permanent_logical_1:
# type: logical
# database: foo
# plugin: pgoutput
ttl: 30
loop_wait: &loop_wait 10
retry_timeout: 10
@ -63,15 +63,15 @@ spec:
# restore a Postgres DB with point-in-time-recovery
# with a non-empty timestamp, clone from an S3 bucket using the latest backup before the timestamp
# with an empty/absent timestamp, clone from an existing alive cluster using pg_basebackup
# clone:
# uid: "efd12e58-5786-11e8-b5a7-06148230260c"
# cluster: "acid-batman"
# timestamp: "2017-12-19T12:40:33+01:00" # timezone required (offset relative to UTC, see RFC 3339 section 5.6)
# s3_wal_path: "s3://custom/path/to/bucket"
# clone:
# uid: "efd12e58-5786-11e8-b5a7-06148230260c"
# cluster: "acid-batman"
# timestamp: "2017-12-19T12:40:33+01:00" # timezone required (offset relative to UTC, see RFC 3339 section 5.6)
# s3_wal_path: "s3://custom/path/to/bucket"
# run periodic backups with k8s cron jobs
# enableLogicalBackup: true
# logicalBackupSchedule: "30 00 * * *"
# enableLogicalBackup: true
# logicalBackupSchedule: "30 00 * * *"
maintenanceWindows:
- 01:00-06:00 #UTC
- Sat:00:00-04:00

4
run_operator_locally.sh
View File

@ -1,6 +1,6 @@
#!/usr/bin/env bash
#
# Deploy a Postgres operator to a minikube aka local Kubernetes cluster
# Deploy a Postgres Operator to a minikube aka local Kubernetes cluster
# Optionally re-build the operator binary beforehand to test local changes
# Known limitations:
@ -213,7 +213,7 @@ function submit_postgresql_manifest(){
function main(){
if ! [[ $(basename "$PWD") == "postgres-operator" ]]; then
echo "Please execute the script only from the root directory of the Postgres operator repo."
echo "Please execute the script only from the root directory of the Postgres Operator repo."
exit 1
fi