public_git/postgres-operator
1
0
Fork 0
mirror of https://github.com/zalando/postgres-operator.git synced 2026-09-30 18:06:37 +02:00
Code Issues Projects Releases Wiki Activity

Update docs for v1.2 (#609)

Browse Source 
* update docs and move parts from README to index.md
* fix typos, headings and code alignment in docs
This commit is contained in:
Felix Kunde
2019-07-11 17:19:27 +02:00
committed by GitHub
parent 1c340c68c6
commit 4fc5822b24
17 changed files with 689 additions and 618 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/developer.md
+125 -170
View File
@@ -1,128 +1,28 @@
## Intro
# Developer Guide
See [minikube installation guide](https://github.com/kubernetes/minikube/releases)
Make sure you use the latest version of Minikube.
After the installation, issue
```bash
$ minikube start
```
Note: if you are running on a Mac, you may also use Docker for Mac Kubernetes
instead of a docker-machine.
Once you have it started successfully, use [the quickstart
guide](https://github.com/kubernetes/minikube#quickstart) in order to test your
that your setup is working.
Note: if you use multiple Kubernetes clusters, you can switch to Minikube with
`kubectl config use-context minikube`
## Deploying the operator
### Kubernetes manifest
A ConfigMap is used to store the configuration of the operator. Alternatively,
a CRD-based configuration can be used, as described [here](reference/operator_parameters).
```bash
$ kubectl --context minikube create -f manifests/configmap.yaml
```
First you need to install the service account definition in your Minikube cluster.
```bash
$ kubectl --context minikube create -f manifests/operator-service-account-rbac.yaml
```
Next deploy the postgres-operator from the docker image Zalando is using:
```bash
$ kubectl --context minikube create -f manifests/postgres-operator.yaml
```
If you prefer to build the image yourself follow up down below.
### Helm chart
Alternatively, the operator can be installed by using the provided [Helm](https://helm.sh/)
chart which saves you the manual steps. Therefore, you would need to install
the helm CLI on your machine. After initializing helm (and its server
component Tiller) in your local cluster you can install the operator chart.
You can define a release name that is prepended to the operator resource's
names.
Use `--name zalando` to match with the default service account name as older
operator versions do not support custom names for service accounts. When relying
solely on the CRD-based configuration edit the `serviceAccount` section in the
[values yaml file](../charts/values.yaml) by setting the name to `"operator"`.
```bash
$ helm init
$ helm install --name zalando ./charts/postgres-operator
```
## Check if CustomResourceDefinition has been registered
```bash
$ kubectl --context minikube get crd
NAME KIND
postgresqls.acid.zalan.do CustomResourceDefinition.v1beta1.apiextensions.k8s.io
```
## Create a new Spilo cluster
```bash
$ kubectl --context minikube create -f manifests/minimal-postgres-manifest.yaml
```
## Watch pods being created
```bash
$ kubectl --context minikube get pods -w --show-labels
```
## Connect to PostgreSQL
We can use the generated secret of the `postgres` robot user to connect to our
`acid-minimal-cluster` master running in Minikube:
```bash
$ export HOST_PORT=$(minikube service acid-minimal-cluster --url | sed 's,.*/,,')
$ export PGHOST=$(echo $HOST_PORT | cut -d: -f 1)
$ export PGPORT=$(echo $HOST_PORT | cut -d: -f 2)
$ export PGPASSWORD=$(kubectl --context minikube get secret postgres.acid-minimal-cluster.credentials -o 'jsonpath={.data.password}' | base64 -d)
$ psql -U postgres
```
# Setup development environment
The following steps guide you through the setup to work on the operator itself.
Read this guide if you want to debug the operator, fix bugs or contribute new
features and tests.
## Setting up Go
Postgres operator is written in Go. Use the [installation
instructions](https://golang.org/doc/install#install) if you don't have Go on
your system. You won't be able to compile the operator with Go older than 1.7.
We recommend installing [the latest one](https://golang.org/dl/).
Postgres Operator is written in Go. Use the [installation instructions](https://golang.org/doc/install#install)
if you don't have Go on your system. You won't be able to compile the operator
with Go older than 1.7. We recommend installing [the latest one](https://golang.org/dl/).
Go projects expect their source code and all the dependencies to be located
under the [GOPATH](https://github.com/golang/go/wiki/GOPATH). Normally, one
would create a directory for the GOPATH (i.e. ~/go) and place the source code
under the ~/go/src subdirectories.
under the ~/go/src sub directories.
Given the schema above, the postgres operator source code located at
Given the schema above, the Postgres Operator source code located at
`github.com/zalando/postgres-operator` should be put at
-`~/go/src/github.com/zalando/postgres-operator`.
```bash
$ export GOPATH=~/go
$ mkdir -p ${GOPATH}/src/github.com/zalando/
$ cd ${GOPATH}/src/github.com/zalando/
$ git clone https://github.com/zalando/postgres-operator.git
export GOPATH=~/go
mkdir -p ${GOPATH}/src/github.com/zalando/
cd ${GOPATH}/src/github.com/zalando/
git clone https://github.com/zalando/postgres-operator.git
```
## Building the operator
@@ -130,55 +30,74 @@ Given the schema above, the postgres operator source code located at
You need Glide to fetch all dependencies. Install it with:
```bash
$ make tools
make tools
```
Next, install dependencies with glide by issuing:
```bash
$ make deps
make deps
```
This would take a while to complete. You have to redo `make deps` every time
you dependencies list changes, i.e. after adding a new library dependency.
Build the operator docker image and pushing it to Pier One:
Build the operator with the `make docker` command. You may define the TAG
variable to assign an explicit tag to your docker image and the IMAGE to set
the image name. By default, the tag is computed with
`git describe --tags --always --dirty` and the image is
`registry.opensource.zalan.do/acid/postgres-operator`
```bash
$ make docker push
export TAG=$(git describe --tags --always --dirty)
make docker
```
You may define the TAG variable to assign an explicit tag to your docker image
and the IMAGE to set the image name. By default, the tag is computed with
`git describe --tags --always --dirty` and the image is
`pierone.stups.zalan.do/acid/postgres-operator`
Building the operator binary (for testing the out-of-cluster option):
```bash
$ make
make
```
The binary will be placed into the build directory.
## Deploying self build image
The fastest way to run your docker image locally is to reuse the docker from
minikube. The following steps will get you the docker image built and deployed.
The fastest way to run and test your docker image locally is to reuse the docker
from [minikube](https://github.com/kubernetes/minikube/releases) or use the
`load docker-image` from [kind](https://kind.sigs.k8s.io/). The following steps
will get you the docker image built and deployed.
```bash
$ eval $(minikube docker-env)
$ export TAG=$(git describe --tags --always --dirty)
$ make docker
$ sed -e "s/\(image\:.*\:\).*$/\1$TAG/" manifests/postgres-operator.yaml|kubectl --context minikube create -f -
# minikube
eval $(minikube docker-env)
make docker
# kind
make docker
kind load docker-image <image> --name <kind-cluster-name>
```
# Code generation
Then create a new Postgres Operator deployment. You can reuse the provided
manifest but replace the version and tag. Don't forget to also apply
configuration and RBAC manifests first, e.g.:
The operator employs k8s-provided code generation to obtain deep copy methods
and Kubernetes-like APIs for its custom resource definitons, namely the Postgres
CRD and the operator CRD. The usage of the code generation follows conventions
from the k8s community. Relevant scripts live in the `hack` directory:
```bash
kubectl create -f manifests/configmap.yaml
kubectl create -f manifests/operator-service-account-rbac.yaml
sed -e "s/\(image\:.*\:\).*$/\1$TAG/" manifests/postgres-operator.yaml | kubectl create -f -
# check if the operator is coming up
kubectl get pod -l name=postgres-operator
```
## Code generation
The operator employs K8s-provided code generation to obtain deep copy methods
and K8s-like APIs for its custom resource definitions, namely the
Postgres CRD and the operator CRD. The usage of the code generation follows
conventions from the k8s community. Relevant scripts live in the `hack`
directory:
* `update-codegen.sh` triggers code generation for the APIs defined in `pkg/apis/acid.zalan.do/`,
* `verify-codegen.sh` checks if the generated code is up-to-date (to be used within CI).
@@ -199,18 +118,22 @@ operator itself with `--v=8` to log all REST API requests). To attach a debugger
to the operator, use the `-outofcluster` option to run the operator locally on
the developer's laptop (and not in a docker container).
# Debugging the operator
## Debugging the operator
There is a web interface in the operator to observe its internal state. The
operator listens on port 8080. It is possible to expose it to the
localhost:8080 by doing:
`localhost:8080` by doing:
$ kubectl --context minikube port-forward $(kubectl --context minikube get pod -l name=postgres-operator -o jsonpath={.items..metadata.name}) 8080:8080
```bash
kubectl --context minikube port-forward $(kubectl --context minikube get pod -l name=postgres-operator -o jsonpath={.items..metadata.name}) 8080:8080
```
The inner 'query' gets the name of the postgres operator pod, and the outer
The inner query gets the name of the Postgres Operator pod, and the outer one
enables port forwarding. Afterwards, you can access the operator API with:
$ curl --location http://127.0.0.1:8080/$endpoint | jq .
```bash
curl --location http://127.0.0.1:8080/$endpoint | jq .
```
The available endpoints are listed below. Note that the worker ID is an integer
from 0 up to 'workers' - 1 (value configured in the operator configuration and
@@ -242,14 +165,13 @@ The operator also supports pprof endpoints listed at the
* /debug/pprof/trace
It's possible to attach a debugger to troubleshoot postgres-operator inside a
docker container. It's possible with gdb and
[delve](https://github.com/derekparker/delve). Since the latter one is a
specialized debugger for golang, we will use it as an example. To use it you
need:
docker container. It's possible with [gdb](https://www.gnu.org/software/gdb/)
and [delve](https://github.com/derekparker/delve). Since the latter one is a
specialized debugger for Go, we will use it as an example. To use it you need:
* Install delve locally
```
```bash
go get -u github.com/derekparker/delve/cmd/dlv
```
@@ -261,7 +183,8 @@ RUN go get github.com/derekparker/delve/cmd/dlv
```
* Update the `Makefile` to build the project with debugging symbols. For that
you need to add `gcflags` to a build target for corresponding OS (e.g. linux)
you need to add `gcflags` to a build target for corresponding OS (e.g.
GNU/Linux)
```
-gcflags "-N -l"
@@ -276,61 +199,75 @@ CMD ["/root/go/bin/dlv", "--listen=:DLV_PORT", "--headless=true", "--api-version
* Forward the listening port
```
```bash
kubectl port-forward POD_NAME DLV_PORT:DLV_PORT
```
* Attach to it
```
$ dlv connect 127.0.0.1:DLV_PORT
```bash
dlv connect 127.0.0.1:DLV_PORT
```
## Unit tests
To run all unit tests, you can simply do:
```
$ go test ./...
```bash
go test ./...
```
For go 1.9 `vendor` directory would be excluded automatically. For previous
versions you can exclude it manually:
```
$ go test $(glide novendor)
```bash
go test $(glide novendor)
```
In case if you need to debug your unit test, it's possible to use delve:
```
$ dlv test ./pkg/util/retryutil/
```bash
dlv test ./pkg/util/retryutil/
Type 'help' for list of commands.
(dlv) c
PASS
```
To test the multinamespace setup, you can use
To test the multi-namespace setup, you can use
```
```bash
./run_operator_locally.sh --rebuild-operator
```
It will automatically create an `acid-minimal-cluster` in the namespace `test`.
Then you can for example check the Patroni logs:
```
```bash
kubectl logs acid-minimal-cluster-0
```
## End-to-end tests
The operator provides reference e2e (end-to-end) tests to ensure various infra parts work smoothly together.
Each e2e execution tests a Postgres operator image built from the current git branch. The test runner starts a [kind](https://kind.sigs.k8s.io/) (local k8s) cluster and Docker container with tests. The k8s API client from within the container connects to the `kind` cluster using the standard Docker `bridge` network.
The tests utilize examples from `/manifests` (ConfigMap is used for the operator configuration) to avoid maintaining yet another set of configuration files. The kind cluster is deleted if tests complete successfully.
The operator provides reference e2e (end-to-end) tests to ensure various infra
parts work smoothly together. Each e2e execution tests a Postgres Operator image
built from the current git branch. The test runner starts a [kind](https://kind.sigs.k8s.io/)
(local k8s) cluster and Docker container with tests. The k8s API client from
within the container connects to the `kind` cluster using the standard Docker
`bridge` network. The tests utilize examples from `/manifests` (ConfigMap is
used for the operator configuration) to avoid maintaining yet another set of
configuration files. The kind cluster is deleted if tests complete successfully.
End-to-end tests are executed automatically during builds; to invoke them locally use `make e2e-run` from the project's top directory. Run `make e2e-tools e2e-build` to install `kind` and build the tests' image locally before the first run.
End-to-end tests are executed automatically during builds:
End-to-end tests are written in Python and use `flake8` for code quality. Please run flake8 [before submitting a PR](http://flake8.pycqa.org/en/latest/user/using-hooks.html).
```bash
# invoke them from the project's top directory
make e2e-run
# install kind and build test image before first run
make e2e-tools e2e-build
```
End-to-end tests are written in Python and use `flake8` for code quality.
Please run flake8 [before submitting a PR](http://flake8.pycqa.org/en/latest/user/using-hooks.html).
## Introduce additional configuration parameters
@@ -340,17 +277,17 @@ be updated. As explained [here](reference/operator_parameters.md), it's possible
to configure the operator either with a ConfigMap or CRD, but currently we aim
to synchronize parameters everywhere.
When choosing a parameter name for a new option in a PG manifest, keep in mind
the naming conventions there. The `snake_case` variables come from the Patroni/Postgres world, while the `camelCase` from the k8s world.
When choosing a parameter name for a new option in a Postgres cluster manifest,
keep in mind the naming conventions there. We use `camelCase` for manifest
parameters (with exceptions for certain Patroni/Postgres options) and
`snake_case` variables in the configuration. Only introduce new manifest
variables if you feel a per-cluster configuration is necessary.
Note: If one option is defined in the operator configuration and in the cluster
[manifest](../manifests/complete-postgres-manifest.yaml), the latter takes
precedence.
So, first define the parameters in:
* the [ConfigMap](../manifests/configmap.yaml) manifest
* the CR's [default configuration](../manifests/postgresql-operator-default-configuration.yaml)
* the Helm chart [values](../charts/postgres-operator/values.yaml)
### Go code
Update the following Go files that obtain the configuration parameter from the
manifest files:
@@ -358,12 +295,30 @@ manifest files:
* [operator_config.go](../pkg/controller/operator_config.go)
* [config.go](../pkg/util/config/config.go)
Postgres manifest parameters are defined in the [api package](../pkg/apis/acid.zalan.do/v1/postgresql_type.go).
The operator behavior has to be implemented at least in [k8sres.go](../pkg/cluster/k8sres.go).
Please, reflect your changes in tests, for example in:
* [config_test.go](../pkg/util/config/config_test.go)
* [k8sres_test.go](../pkg/cluster/k8sres_test.go)
* [util_test.go](../pkg/apis/acid.zalan.do/v1/util_test.go)
Finally, document the new configuration option(s) for the operator in its
[reference](reference/operator_parameters.md) document and explain the feature
in the [administrator docs](administrator.md).
### Updating manifest files
For the CRD-based configuration, please update the following files:
* the default [OperatorConfiguration](../manifests/postgresql-operator-default-configuration.yaml)
* the Helm chart's [values-crd file](../charts/postgres-operator/values.yaml)
Reflect the changes in the ConfigMap configuration as well (note that numeric
and boolean parameters have to use double quotes here):
* [ConfigMap](../manifests/configmap.yaml) manifest
* the Helm chart's default [values file](../charts/postgres-operator/values.yaml)
### Updating documentation
Finally, add a section for each new configuration option and/or cluster manifest
parameter in the reference documents:
* [config reference](reference/operator_parameters.md)
* [manifest reference](reference/cluster_manifest.md)
It also helps users to explain new features with examples in the
[administrator docs](administrator.md).