Commit Graph

256 Commits

Author SHA1 Message Date
Oleksii Kliukin c25e849fe4 Fix a failure to create new statefulset at sync.
Also do a fmt run.
2017-11-08 18:24:17 +01:00
Murat Kabilov 86803406db
use sync methods while updating the cluster 2017-11-03 12:00:43 +01:00
Georg Kunz 47dd766fa7 Add node toleration config to PodSpec (#151)
* Add node toleration config to PodSpec

This allows to taint nodes dedicated to Postgres and prevents other pods from running on these nodes.

* Document taint and toleration setup

And remove setting from default operator ConfigMap

* Allow to overwrite tolerations with Postgres manifest
2017-11-02 19:10:44 +01:00
Oleksii Kliukin ce960e892a
Create new databases and change owners of existing ones during sync. (#153)
* Create new databases and change owners of existing ones during sync.
2017-11-02 17:46:33 +01:00
Oleksii Kliukin 7a76be7d3e Minor fixes around PDB (pod-distruption-budget) syncing: (#147)
- Call comparison function in the case of the sync as well as for update
- Include full cluster name in PDB name
- Assign cluster labels to the PDB object
2017-10-23 12:26:59 +02:00
Murat Kabilov c17aabb642 fix pod disruption budget labels (#146) 2017-10-20 15:01:51 +02:00
Murat Kabilov 661b141849 Fix Pod Disruption Budget null pointer exception 2017-10-20 11:43:50 +02:00
Murat Kabilov a1deae198b add missing master matchLabel for the PDB (#144) 2017-10-20 11:26:40 +02:00
Oleksii Kliukin eba23279c8 Kube cluster upgrade 2017-10-19 10:49:42 +02:00
Oleksii Kliukin 1dbf259c76 Retry opening DB connections. (#140)
Make sure DB connection retry also reopens a connection after closing it
2017-10-18 16:28:00 +02:00
Oleksii Kliukin 99870d8eac Fix division by zero when connecting to the DB.
Apparently the retry function's first parameter is the duration of
a single attempt and it cannot be zero.
2017-10-18 10:44:49 +02:00
Murat Kabilov 202f2de988 Retry connecting to pg 2017-10-17 17:03:50 +02:00
Murat Kabilov 6c4cb4e9da Perform manual failover during the scale down 2017-10-16 17:41:23 +02:00
Murat Kabilov 5b29576a8e Remove redundant constants 2017-10-16 15:52:48 +02:00
Murat Kabilov 3b32265258 Set status of the cluster on sync fail/success 2017-10-12 15:10:42 +02:00
Jan Mussler cec695d48e Superuser toggle for team members
Make superuser toggleable for team members. Add and "admin" role to team members if superuser is disabled.
2017-10-12 15:01:54 +02:00
Murat Kabilov 8d5faaa5a5 return idle status when worker has nothing to do 2017-10-11 15:42:20 +02:00
Oleksii Kliukin 793defef72 Fix pod wait timeouts.
Previously, a timer had been reset on every message received through
the pod channel.
2017-10-11 14:58:37 +02:00
Murat Kabilov 83c8d6c419 Extend diagnostic api with worker status info 2017-10-11 12:26:09 +02:00
Murat Kabilov 32aa7270e6 Use round-robin strategy while assigning workers 2017-10-09 16:56:27 +02:00
Murat Kabilov 71a540ff48 Merge branch 'master' into crd 2017-10-09 11:55:18 +02:00
Murat Kabilov 2f3bb1e265 set the proper name for the crd related constants file 2017-10-09 11:01:46 +02:00
Murat Kabilov a35e9c6119 move from tpr to crd 2017-10-06 15:12:08 +02:00
Murat Kabilov 3b8c06416e skip manual failover for 1-pod clusters 2017-10-05 13:30:15 +03:00
Jan Mussler c4af0ac6a6 Update cluster.go 2017-10-05 10:58:23 +02:00
Jan M 4a1170855a Adding '_' to allowed chars. 2017-10-05 10:53:19 +02:00
Murat Kabilov 48ec6b35b9 perform manual failover on pg cluster rolling upgrade 2017-10-04 16:56:47 +03:00
Murat Kabilov 00194d0130 create dbs on cluster create 2017-10-04 16:24:27 +03:00
Murat Kabilov 5cfdabb63e fix regexp for api endpoint urls 2017-09-28 12:00:40 +02:00
Murat Kabilov be8bf22c00 add missing return 2017-09-28 11:23:56 +02:00
Murat Kabilov 93d4bf2b55 Merge branch 'master' into api-improvements 2017-09-26 14:47:13 +02:00
Murat Kabilov 9a66e09b88 cluster history api endpoint 2017-09-26 14:30:45 +02:00
Murat Kabilov 19de2a24b7 go lint 2017-09-26 13:44:30 +02:00
Murat Kabilov ed476ae85d add missing comment for the method 2017-09-26 13:39:13 +02:00
Murat Kabilov f77852a152 store time of the cluster event 2017-09-26 13:17:23 +02:00
Murat Kabilov c44cfff988 add Diff util method 2017-09-26 13:13:15 +02:00
Murat Kabilov c67f06956e fix comments for ringlogger 2017-09-26 13:12:38 +02:00
Murat Kabilov d876f4d88e set secret name template via config map 2017-09-18 14:25:09 +02:00
Oleksii Kliukin 7667847bfe Feature/validate role options (#101)
Be more rigorous about validating user flags.

Only accept CREATE ROLE flags that doesn't have any params (i.e.
not ADMIN or CONNECTION LIMIT). Check that both flag and NOflag
are not used at the same time.
2017-09-15 13:57:48 +02:00
Murat Kabilov 969a06f521 Use DCS_ENABLE_KUBERNETES_API=true environment to enable kubernetes native deployment 2017-09-14 11:39:49 +02:00
Murat Kabilov 8430ee86c9 add comments on roles 2017-09-11 17:44:32 +02:00
Murat Kabilov 83f1dbe80c add missing return in the api server 2017-09-11 17:44:32 +02:00
Murat Kabilov 90b49a24ba make postgresql roles public 2017-09-11 17:44:32 +02:00
Oleksii Kliukin 8b85935a7a Allow cloning clusters from the operator. (#90)
Allow cloning clusters from the operator.

The changes add a new JSON node `clone` with possible values `cluster`
and `timestamp`. `cluster` is mandatory, and setting a non-empty
`timestamp` triggers wal-e point in time recovery. Spilo and Patroni do
the whole heavy-lifting, the operator just defines certain variables and
gathers some data about how to connect to the host to clone or the
target S3 bucket.

As a minor change, set the image pull policy to IfNotPresent instead
of Always to simplify local testing.

Change the default replication username to standby.
2017-09-08 16:47:03 +02:00
Oleksii Kliukin a0a9e8f849 Feature/configure replication role (#97)
Configure superuser and replication usernames
2017-09-07 10:12:34 +02:00
Murat Kabilov 4db5bd13d1 delete cluster key from the clusters list only when delete procedure is finished 2017-09-04 18:48:03 +02:00
Murat Kabilov 39c123e96a fetch cluster resources by name, not by label selectors 2017-09-04 18:03:54 +02:00
Murat Kabilov 8aa11ecee2 Add patroni api client 2017-08-30 16:01:18 +02:00
Murat Kabilov 899c0bef45 Use warningf instead of warnf 2017-08-30 14:35:56 +02:00
Murat Kabilov 53ceede3cb show worker queue size in the cluster status 2017-08-28 12:05:33 +02:00
Murat Kabilov f44c8e1206 Make pod termination grace period configurable 2017-08-18 16:52:19 +02:00
Murat Kabilov 71dfb33b2b make pod termination grace period configurable 2017-08-18 16:38:25 +02:00
Murat Kabilov 83760ebbef discard cluster events from the queue on cluster delete;
delete cluster from the clusters map before deleting cluster itself
2017-08-17 12:24:23 +02:00
Murat Kabilov f2c23021bb generate clusterEvent queue key in a separate function 2017-08-17 12:20:03 +02:00
Murat Kabilov 5967837875 pass the name of the status in the log message on set cluster status failure 2017-08-17 12:18:53 +02:00
Murat Kabilov bf5879a71f normalize log message text 2017-08-17 12:16:19 +02:00
Murat Kabilov dad8e2f49f make cluster event queue consumption non-blocking 2017-08-15 16:03:19 +02:00
Murat Kabilov d2828e5ece remove var shading; fix imports 2017-08-15 15:59:10 +02:00
Murat Kabilov 272d7e1bcf rename service field to services as it contains service per role 2017-08-15 15:55:56 +02:00
Murat Kabilov 38e0ffecf7 make controllerinformer interface private;
use named regexp groups
2017-08-15 14:07:16 +02:00
Murat Kabilov 82d5583809 add diagnostic api http server 2017-08-15 12:20:09 +02:00
Murat Kabilov 51fdfb90f7 log cluster and controller events in the ringlog via logrus hook 2017-08-15 12:16:09 +02:00
Murat Kabilov 82f58b57d8 add cluster and controller methods for getting status 2017-08-15 12:11:06 +02:00
Murat Kabilov 4ee28e3818 add ringlog 2017-08-15 11:59:09 +02:00
Murat Kabilov 58572bb43f move controller config to the spec package 2017-08-15 11:41:46 +02:00
Murat Kabilov 606d000022 fix test 2017-08-15 10:41:04 +02:00
Murat Kabilov 5470f20be4 always pass a cluster name as a logger field 2017-08-15 10:29:18 +02:00
Murat Kabilov e26db66cb5 start all the log messages with lowercase letters 2017-08-15 10:12:36 +02:00
Oleksii Kliukin 87a379f663 Avoid reusing closed DB connection. (#79)
Set DB connection to nil upon closing it.
2017-08-10 18:19:35 +02:00
Oleksii Kliukin f15f93f479 Bugfix/close db connections (#78)
Open and close DB connections on-demand.

Previously, we used to leave the DB connection open while the
cluster was registered with the operator, potentially resutling
in dangled connections if the operator terminates abnormally.

Small refactoring around the role syncing code.
2017-08-10 10:10:00 +02:00
Oleksii Kliukin 8b58782a4a fix pam_role_name parameter name. 2017-08-02 17:55:06 +02:00
Oleksii Kliukin a8b5b77cc4 Fix missing labels in the replica service selector. 2017-08-02 17:46:24 +02:00
Murat Kabilov cf663cb841 Fix golint warnings 2017-08-01 16:08:56 +02:00
Murat Kabilov c02a740e10 Fix setting debug logger level 2017-08-01 11:51:03 +02:00
Murat Kabilov 1211220208 Skip running empty set of queries 2017-08-01 10:09:09 +02:00
Murat Kabilov 6183203f4d fix cluster event queue processing 2017-07-31 10:30:49 +02:00
Murat Kabilov 2fe22ff614 Remove pod dispatcher 2017-07-27 14:16:49 +02:00
Murat Kabilov 3ad4b127c4 Fix graceful shutdown
graceful shutdown of goroutines on operator exit
2017-07-27 12:54:22 +02:00
Murat Kabilov 1f8b37f33d Make use of kubernetes client-go v4
* client-go v4.0.0-beta0
* remove unnecessary methods for tpr object
* rest client: use interface instead of structure pointer
* proper names for constants; some clean up for log messages
* remove teams api client from controller and make it per cluster
2017-07-25 15:25:17 +02:00
Oleksii Kliukin 4455f1b639 Feature/unit tests (#53)
- Avoid relying on Clientset structure to call Kubernetes API functions.
While Clientset is a convinient "catch-all" abstraction for calling
REST API related to different Kubernetes objects, it's impossible to
mock. Replacing it wih the kubernetes.Interface would be quite
straightforward, but would require an exra level of mocked interfaces,
because of the versioning. Instead, a new interface is defined, which
contains only the objects we need of the pre-defined versions.

-  Move KubernetesClient to k8sutil package.
- Add more tests.
2017-07-24 16:56:46 +02:00
Murat Kabilov 4f36e447c3 Skip config params with no values (#62) 2017-07-14 17:22:25 +02:00
Oleksii Kliukin a8ed1e25b4 Avoid re-creating master pod if it is empty during sync. (#58)
Fixes #59
2017-07-12 10:57:20 +02:00
Oleksii Kliukin 00150711e4 Configure load balancer on a per-cluster and operator-wide level (#57)
* Deny all requests to the load balancer by default.
* Operator-wide toggle for the load-balancer.
* Define per-cluster useLoadBalancer option.

If useLoadBalancer is not set - then operator-wide defaults take place. If it
is true - the load balancer is created, otherwise a service type clusterIP is
created.

Internally, we have to completely replace the service if the service type
changes. We cannot patch, since some fields from the old service that will
remain after patch are incompatible with the new one, and handling them
explicitly when updating the service is ugly and error-prone. We cannot
update the service because of the immutable fields, that leaves us the only
option of deleting the old service and creating the new one. Unfortunately,
there is still an issue of unnecessary removal of endpoints associated with
the service, it will be addressed in future commits.

* Revert the unintended effect of go fmt

* Recreate endpoints on service update.

When the service type is changed, the service is deleted and then
the one with the new type is created. Unfortnately, endpoints are
deleted as well. Re-create them afterwards, preserving the original
addresses stored in them.

* Improve error messages and comments. Use generate instead of gen in names.
2017-06-30 13:38:49 +02:00
Murat Kabilov 9a6b0b8c37 Tests for teams API (#46) 2017-06-12 17:29:32 +02:00
Oleksii Kliukin 987990fb0e Move service annotation patch template into the constants. 2017-06-12 10:24:23 +02:00
Oleksii Kliukin 17826ee434 Go fmt run. 2017-06-12 10:24:23 +02:00
Oleksii Kliukin 51d73fb172 Replace service annotations when updating services.
In case the whole annotation changes (like the external DNS) we
don't want to keep the old one hanging around. Unline specs, we
don't expect anyone except the operator to change the annotations.

Use StrategicMergePatchType in order to replace the annotations
map completely.
2017-06-12 10:24:23 +02:00
Oleksii Kliukin e0dacd0ca9 Remove an unused export. 2017-06-08 16:17:01 +02:00
Murat Kabilov a9d746878b fix golint warn 2017-06-08 15:52:01 +02:00
Murat Kabilov 1540a2ba65 fix typos;
remove unnecessary tests;
go fmt -s
2017-06-08 15:52:01 +02:00
Murat Kabilov e104a67260 Fix resync of the clusters 2017-06-08 11:51:48 +02:00
Murat Kabilov bdc2db97ac Tests for Specs and Teams API 2017-06-08 10:58:48 +02:00
Oleksii Kliukin bc0e9ab4bc Add error checks per report from errcheck-ng 2017-06-08 10:41:44 +02:00
Murat Kabilov 292a9bda05 Check for dns annotation of the service 2017-06-07 16:41:39 +02:00
Oleksii Kliukin dc36c4ca12 Implement replicaLoadBalancer boolean flag. (#38)
The flag adds a replica service with the name cluster_name-repl and
a DNS name that defaults to {cluster}-repl.{team}.{hostedzone}.

The implementation converted Service field of the cluster into a map
with one or two elements and deals with the cases when the new flag
is changed on a running cluster
(the update and the sync should create or delete the replica service).
In order to pick up master and replica service and master endpoint
when listing cluster resources.

* Update the spec when updating the cluster.
2017-06-07 13:54:17 +02:00
Oleksii Kliukin 7b0ca31bfb Implements EBS volume resizing #35.
In order to support volumes different from EBS and filesystems other than EXT2/3/4 the respective code parts were implemented as interfaces. Adding the new resize for the volume or the filesystem will require implementing the interface, but no other changes in the cluster code itself.

Volume resizing first changes the EBS and the filesystem, and only afterwards is reflected in the Kubernetes "PersistentVolume" object. This is done deliberately to be able to check if the volume needs resizing by peeking at the Size of the PersistentVolume structure. We recheck, nevertheless, in the EBSVolumeResizer, whether the actual EBS volume size doesn't match the spec, since call to the AWS ModifyVolume is counted against the resize limit of once every 6 hours, even for those calls that shouldn't result in an actual resize (i.e. when the size matches the one for the running volume).

As a collateral, split the constants into multiple files, move the volume code into a separate file and fix minor issues related to the error reporting.
2017-06-06 13:53:27 +02:00
Murat Kabilov 1fb05212a9 Refactor teams API package 2017-05-30 10:14:30 +02:00
Murat Kabilov f7aaf8863d Change maintenance window format 2017-05-30 09:56:10 +02:00
Murat Kabilov 1111964fee fix password check in pguserpassword
remove magic number
2017-05-26 18:19:12 +02:00
Murat Kabilov 009db16c7c Use queues for the pod events (#30) 2017-05-23 15:24:14 +02:00