postgres-operator/docs/developer.md

## Intro

See [minikube installation guide](https://github.com/kubernetes/minikube/releases)

Make sure you use the latest version of Minikube.

After the installation, issue

```bash
    $ minikube start
```

Note: if you are running on a Mac, you may also use Docker for Mac Kubernetes
instead of a docker-machine.

Once you have it started successfully, use [the quickstart
guide](https://github.com/kubernetes/minikube#quickstart) in order to test your
that your setup is working.

Note: if you use multiple Kubernetes clusters, you can switch to Minikube with
`kubectl config use-context minikube`

## Create ConfigMap

ConfigMap is used to store the configuration of the operator

```bash
    $ kubectl --context minikube  create -f manifests/configmap.yaml
```

## Deploying the operator

### - Kubernetes manifest

First you need to install the service account definition in your Minikube cluster.

```bash
    $ kubectl --context minikube create -f manifests/operator-service-account-rbac.yaml
```

Next deploy the postgres-operator from the docker image Zalando is using:

```bash
    $ kubectl --context minikube create -f manifests/postgres-operator.yaml
```

If you prefer to build the image yourself follow up down below.

### - Helm chart

You can install postgres-operator also with a [Helm](https://helm.sh/) chart.
This requires installing the Helm CLI first and then initializing it in the
cluster.

```bash
    $ helm init
    $ helm install --name my-release ./charts/postgres-operator
```

## Check if CustomResourceDefinition has been registered

```bash
    $ kubectl --context minikube   get crd

	NAME                          KIND
	postgresqls.acid.zalan.do     CustomResourceDefinition.v1beta1.apiextensions.k8s.io
```

## Create a new Spilo cluster

```bash
    $ kubectl --context minikube  create -f manifests/minimal-postgres-manifest.yaml
```

## Watch pods being created

```bash
    $ kubectl --context minikube  get pods -w --show-labels
```

## Connect to PostgreSQL

We can use the generated secret of the `postgres` robot user to connect to our
`acid-minimal-cluster` master running in Minikube:

```bash
    $ export HOST_PORT=$(minikube service acid-minimal-cluster --url | sed 's,.*/,,')
    $ export PGHOST=$(echo $HOST_PORT | cut -d: -f 1)
    $ export PGPORT=$(echo $HOST_PORT | cut -d: -f 2)
    $ export PGPASSWORD=$(kubectl --context minikube get secret postgres.acid-minimal-cluster.credentials -o 'jsonpath={.data.password}' | base64 -d)
    $ psql -U postgres
```

# Setup development environment

The following steps guide you through the setup to work on the operator itself.

## Setting up Go

Postgres operator is written in Go. Use the [installation
instructions](https://golang.org/doc/install#install) if you don't have Go on
your system. You won't be able to compile the operator with Go older than 1.7.
We recommend installing [the latest one](https://golang.org/dl/).

Go projects expect their source code and all the dependencies to be located
under the [GOPATH](https://github.com/golang/go/wiki/GOPATH). Normally, one
would create a directory for the GOPATH (i.e. ~/go) and place the source code
under the ~/go/src subdirectories.

Given the schema above, the postgres operator source code located at
`github.com/zalando/postgres-operator` should be put at
-`~/go/src/github.com/zalando/postgres-operator`.

```bash
    $ export GOPATH=~/go
    $ mkdir -p ${GOPATH}/src/github.com/zalando/
    $ cd ${GOPATH}/src/github.com/zalando/
    $ git clone https://github.com/zalando/postgres-operator.git
```

## Building the operator

You need Glide to fetch all dependencies. Install it with:

```bash
    $ make tools
```

Next, install dependencies with glide by issuing:

```bash
    $ make deps
```

This would take a while to complete. You have to redo `make deps` every time
you dependencies list changes, i.e. after adding a new library dependency.

Build the operator docker image and pushing it to Pier One:

```bash
    $ make docker push
```

You may define the TAG variable to assign an explicit tag to your docker image
and the IMAGE to set the image name. By default, the tag is computed with
`git describe --tags --always --dirty` and the image is
`pierone.stups.zalan.do/acid/postgres-operator`

Building the operator binary (for testing the out-of-cluster option):

```bash
    $ make
```

The binary will be placed into the build directory.

## Deploying self build image

The fastest way to run your docker image locally is to reuse the docker from
minikube. The following steps will get you the docker image built and deployed.

```bash
    $ eval $(minikube docker-env)
    $ export TAG=$(git describe --tags --always --dirty)
    $ make docker
    $ sed -e "s/\(image\:.*\:\).*$/\1$TAG/" manifests/postgres-operator.yaml|kubectl --context minikube create  -f -
```

# Code generation

The operator employs k8s-provided code generation to obtain deep copy methods
and Kubernetes-like APIs for its custom resource definitons, namely the Postgres
CRD and the operator CRD. The usage of the code generation follows conventions
from the k8s community. Relevant scripts live in the `hack` directory:
* `update-codegen.sh` triggers code generation for the APIs defined in `pkg/apis/acid.zalan.do/`,
* `verify-codegen.sh` checks if the generated code is up-to-date (to be used within CI).

The `/pkg/generated/` contains the resultant code. To make these scripts work,
you may need to `export GOPATH=$(go env GOPATH)`

References for code generation are:
* [Relevant pull request](https://github.com/zalando/postgres-operator/pull/369)
See comments there for minor issues that can sometimes broke the generation process.
* [Code generator source code](https://github.com/kubernetes/code-generator)
* [Code Generation for CustomResources](https://blog.openshift.com/kubernetes-deep-dive-code-generation-customresources/) - intro post on the topic
* Code generation in [Prometheus](https://github.com/coreos/prometheus-operator) and [etcd](https://github.com/coreos/etcd-operator) operators

To debug the generated API locally, use the
[kubectl proxy](https://kubernetes.io/docs/tasks/access-kubernetes-api/http-proxy-access-api/)
and `kubectl --v=8` log level to display contents of HTTP requests (run the
operator itself with `--v=8` to log all REST API requests). To attach a debugger
to the operator, use the `-outofcluster` option to run the operator locally on
the developer's laptop (and not in a docker container).

# Debugging the operator

There is a web interface in the operator to observe its internal state. The
operator listens on port 8080. It is possible to expose it to the
localhost:8080 by doing:

    $ kubectl --context minikube port-forward $(kubectl --context minikube get pod -l name=postgres-operator -o jsonpath={.items..metadata.name}) 8080:8080

The inner 'query' gets the name of the postgres operator pod, and the outer
enables port forwarding. Afterwards, you can access the operator API with:

    $ curl http://127.0.0.1:8080/$endpoint| jq .

The available endpoints are listed below. Note that the worker ID is an integer
from 0 up to 'workers' - 1 (value configured in the operator configuration and
defaults to 4)

* /databases - all databases per cluster
* /workers/all/queue - state of the workers queue (cluster events to process)
* /workers/$id/queue - state of the queue for the worker $id
* /workers/$id/logs - log of the operations performed by a given worker
* /clusters/ - list of teams and clusters known to the operator
* /clusters/$team - list of clusters for the given team
* /clusters/$team/$namespace/$clustername - detailed status of the cluster,
  including the specifications for CRD, master and replica services, endpoints
  and statefulsets, as well as any errors and the worker that cluster is
  assigned to.
* /clusters/$team/$namespace/$clustername/logs/ - logs of all operations
  performed to the cluster so far.
* /clusters/$team/$namespace/$clustername/history/ - history of cluster changes
  triggered by the changes of the manifest (shows the somewhat obscure diff and
  what exactly has triggered the change)

The operator also supports pprof endpoints listed at the
[pprof package](https://golang.org/pkg/net/http/pprof/), such as:

* /debug/pprof/
* /debug/pprof/cmdline
* /debug/pprof/profile
* /debug/pprof/symbol
* /debug/pprof/trace

It's possible to attach a debugger to troubleshoot postgres-operator inside a
docker container. It's possible with gdb and
[delve](https://github.com/derekparker/delve). Since the latter one is a
specialized debugger for golang, we will use it as an example. To use it you
need:

* Install delve locally

```
go get -u github.com/derekparker/delve/cmd/dlv
```

* Add following dependencies to the `Dockerfile`

```
RUN apk --no-cache add go git musl-dev
RUN go get github.com/derekparker/delve/cmd/dlv
```

* Update the `Makefile` to build the project with debugging symbols. For that
  you need to add `gcflags` to a build target for corresponding OS (e.g. linux)

```
-gcflags "-N -l"
```

* Run `postgres-operator` under the delve. For that you need to replace
  `ENTRYPOINT` with the following `CMD`:

```
CMD ["/root/go/bin/dlv", "--listen=:DLV_PORT", "--headless=true", "--api-version=2", "exec", "/postgres-operator"]
```

* Forward the listening port

```
kubectl port-forward POD_NAME DLV_PORT:DLV_PORT
```

* Attach to it

```
$ dlv connect 127.0.0.1:DLV_PORT
```

## Unit tests

To run all unit tests, you can simply do:

```
$ go test ./...
```

For go 1.9 `vendor` directory would be excluded automatically. For previous
versions you can exclude it manually:

```
$ go test $(glide novendor)
```

In case if you need to debug your unit test, it's possible to use delve:

```
$ dlv test ./pkg/util/retryutil/
Type 'help' for list of commands.
(dlv) c
PASS
```

To test the multinamespace setup, you can use

```
./run_operator_locally.sh --rebuild-operator
```
It will automatically create an `acid-minimal-cluster` in the namespace `test`.
Then you can for example check the Patroni logs:

```
kubectl logs acid-minimal-cluster-0
```

## End-to-end tests

The operator provides e2e (end-to-end) tests to ensure various infra parts work smoothly together.
Such tests employ [kind](https://kind.sigs.k8s.io/) to start a local k8s cluster.
We intend to execute e2e tests during builds, but you can invoke them locally with `make e2e` from the project's top directory. Each e2e execution tests a Postgres operator image built from the current git branch.

### Smoke tests

The first provided collection of [smoke tests](../e2e/tests/test_smoke.py) covers the most basic operator capabilities. These tests utilize examples from `/manifests` (ConfigMap is used for the operator configuration) to avoid maintaining yet another set of configuration files. The `kind-smoke-test-postgres-operator` cluster is deleted if tests complete successfully.

### Contributing tests suites

1. Consider using a separate `kind` cluster per a logically separate collection of tests. This approach enables simple clean up after test success by deleting the relevant cluster.
2. Make tests time out; for now we use the [timeout-decorator](https://github.com/pnpnpn/timeout-decorator) for that purpose.
3. Please briefly describe the use case for the new test collection here and in the comments.


## Introduce additional configuration parameters

In the case you want to add functionality to the operator that shall be
controlled via the operator configuration there are a few places that need to
be updated. As explained [here](reference/operator_parameters.md), it's possible
to configure the operator either with a ConfigMap or CRD, but currently we aim
to synchronize parameters everywhere.

Note: If one option is defined in the operator configuration and in the cluster
[manifest](../manifests/complete-postgres-manifest.yaml), the latter takes
precedence.

So, first define the parameters in:
* the [ConfigMap](../manifests/configmap.yaml) manifest
* the CR's [default configuration](../manifests/postgresql-operator-default-configuration.yaml)
* the Helm chart [values](../charts/postgres-operator/values.yaml)

Update the following Go files that obtain the configuration parameter from the
manifest files:
* [operator_configuration_type.go](../pkg/apis/acid.zalan.do/v1/operator_configuration_type.go)
* [operator_config.go](../pkg/controller/operator_config.go)
* [config.go](../pkg/util/config/config.go)

The operator behavior has to be implemented at least in [k8sres.go](../pkg/cluster/k8sres.go).
Please, reflect your changes in tests, for example in:
* [config_test.go](../pkg/util/config/config_test.go)
* [k8sres_test.go](../pkg/cluster/k8sres_test.go)
* [util_test.go](../pkg/apis/acid.zalan.do/v1/util_test.go)

Finally, document the new configuration option(s) for the operator in its
[reference](reference/operator_parameters.md) document and explain the feature
in the [administrator docs](administrator.md).