Commit Graph

143 Commits

Author SHA1 Message Date
Oleksii Kliukin c2826b10e2 Merge branch 'master' into fix/go-vet-fixes 2017-05-17 11:30:07 +02:00
Oleksii Kliukin 4457ce4e47 Replace the statefulset if it cannot be updated. (#18)
Updates to statefulset spec for fields other than 'replicas' and
containers' are forbidden. However, it is possible to delete the old
statefulset without deleting its pods and create the new one, using the
changed specs. The new statefulset shall pick up the orphaned pods.

Change the statefulset's comparison to return the combined effect of
all checks, not just the first non-matching field.
2017-05-17 11:28:21 +02:00
Murat Kabilov 6e5d7abcc5 pass cluster by reference 2017-05-17 11:05:15 +02:00
Murat Kabilov 356be8f0f1 skip clusters with invalid spec 2017-05-16 16:46:37 +02:00
Oleksii Kliukin 03064637f1 Allow disabling access to the DB and the Teams API.
Command-line options --nodatabaseaccess and --noteamsapi disable all
teams api interaction and access to the Postgres database. This is
useful for debugging purposes when the operator runs out of cluster
(with --outofcluster flag).

The same effect can be achieved by setting enable_db_access and/or
enable_teams_api to false.
2017-05-12 17:40:48 +02:00
Murat Kabilov 92d7fbf372 replace github.bus.zalan.do with github.cm/zalando-incubator 2017-05-12 11:50:16 +02:00
Oleksii Kliukin 6983f444ed Periodically sync roles with the running clusters. (#102)
The sync adds or alters database roles based on the roles defined
in the cluster's TPR, Team API and operator's infrastructure roles.
At the moment, roles are not deleted, as it would be dangerous for
the robot roles in case TPR is misconfigured. In addition, ALTER
ROLE does not remove role options, i.e. SUPERUSER or CREATEROLE,
neither it removes role membership: only new options are added and
new role membership is granted. So far, options like NOSUPERUSER
and NOCREATEROLE won't be handed correctly, when mixed with the
non-negative counterparts, also NOLOGIN should be processed correctly.
The code assumes that only MD5 passwords are stored in the DB and
will likely break with the new SCRAM auth in PostgreSQL 10.

On the implementation side, create the new interface to abstract
roles merge and creation, move most of the role-based functionality
from cluster/pg into the new 'users' module, strip create user code
of special cases related to human-based users (moving them to init
instead) and fixed the password md5 generator to avoid processing
already encrypted passwords. In addition, moved the system roles
off the slice containing all other roles in order to avoid extra
efforts to avoid creating them.

Also, fix a leak in DB connections when the new connection is not
considered healthy and discarded without being closed. Initialize
the database during the sync phase before syncing users.
2017-05-12 11:41:35 +02:00
Murat Kabilov 2370659c69 Parallel cluster processing
Run operations concerning multiple clusters in parallel. Each cluster gets its
own worker in order to create, update, sync or delete clusters.  Each worker
acquires the lock on a cluster.  Subsequent operations on the same cluster
have to wait until the current one finishes.  There is a pool of parallel
workers, configurable with the `workers` parameter in the configmap and set by
default to 4. The cluster-related tasks  are assigned to the workers based on
a cluster name: the tasks for the same cluster will be always assigned to the
same worker. There is no blocking between workers, although there is a chance
that a single worker will become a bottleneck if too many clusters are
assigned to it; therefore, for large-scale deployments it might be necessary
to bump up workers from the default value.
2017-05-12 11:41:35 +02:00
Oleksii Kliukin 1c4bce86df Avoid "bulk-comparing" pod resources during sync. (#109)
* Avoid "bulk-comparing" pod resources during sync.

First attempt to fix bogus restarts due to the reported mismatch
of container resources where one of the resources is an empty struct,
while the other has all fields set to nil.

In addition, add an ability to set limits and requests per pod, as well as the operator-level defaults.
2017-05-12 11:41:35 +02:00
Murat Kabilov a7c57874d5 Do not create roles if cluster is masterless
fix pod deletion
2017-05-12 11:41:34 +02:00
Murat Kabilov 08c0e3b6dd Use unified type for the namespaced object names 2017-05-12 11:41:34 +02:00
Oleksii Kliukin 71b93b4cc2 Feature/infrastructure roles (#91)
* Add infrastructure roles configured globally.

Those are the roles defined in the operator itself. The operator's
configuration refers to the secret containing role names, passwords
and membership information. While they are referred to as roles, in
reality those are users.

In addition, improve the regex to filter out invalid users and
make sure user secret names are compatible with DNS name spec.

Add an example manifest for the infrastructure roles.
2017-05-12 11:41:33 +02:00
Murat Kabilov 655f6dcadb make cluster resources private 2017-05-12 11:41:33 +02:00
Murat Kabilov 322676a6b9 Skip deleting Pods and PVCs if failed to delete StatefulSet 2017-05-12 11:41:32 +02:00
Murat Kabilov bb4fec25ae Fix deletion of the failed cluster; more debug messages 2017-05-12 11:41:32 +02:00
Oleksii Kliukin 8e658174e8 Fix the issue with calling a non-existent function. 2017-05-12 11:41:31 +02:00
Murat Kabilov d4bb72989a Warn if etcd key for the new cluster already exist 2017-05-12 11:41:31 +02:00
Murat Kabilov 852c5beae5 Check etcd key availability for the new cluster 2017-05-12 11:41:31 +02:00
Oleksii Kliukin 04ed22f73f Remove unnecessary initializations. 2017-05-12 11:41:31 +02:00
Murat Kabilov ee83e196a9 Fix secrets sync
* log if secret already exists
2017-05-12 11:41:30 +02:00
Oleksii Kliukin 8268b07ad2 Set logger level per package instead of doing this globally 2017-05-12 11:41:30 +02:00
Oleksii Kliukin 3a4c6268be Increase log verbosity, namely for object updates.
- add a new environment variable for triggering debug log level
- show both new, old object and diff during syncs and updates
- use pretty package to pretty-print go structures
-
2017-05-12 11:41:29 +02:00
Murat Kabilov c2d2a67ad5 Get config from environment variables;
ignore pg major version change;
get rid of resources package;
2017-05-12 11:41:29 +02:00
Murat Kabilov 79a6726d4d Increase logging verbosity, restructure code 2017-05-12 11:41:28 +02:00
Murat Kabilov 6f7399b36f Sync clusters states
* move statefulset creation from cluster spec to the separate function
* sync cluster state with desired state;
* move out from arrays for cluster resources;
* recreate pods instead of deleting them in case of statefulset change
* check for master while creating cluster/updating pods
* simplify retryutil
* list pvc while listing resources
* name kubernetes resources with capital letter
* do rolling update in case of env variables change
2017-05-12 11:41:27 +02:00
Oleksii Kliukin 1377724b2e Fix a compliation error. 2017-05-12 11:41:27 +02:00
Oleksii Kliukin 31d7426327 ClusterTeamName -> ClusterName. Add a TODO item. 2017-05-12 11:41:27 +02:00
Oleksii Kliukin 55dbacdfa6 Assign DNS name to the cluster.
DNS name is generated from the team name and cluster name.
Use "zalando.org/dnsname" service annotation that makes 'mate' service assign a CNAME to the load balancer name.
2017-05-12 11:41:27 +02:00
Murat Kabilov 486c8ecb07 use neutral name for set cluster status function 2017-05-12 11:41:26 +02:00
Murat Kabilov 1c6e7ac2e7 loadBalancerSourceRanges update 2017-05-12 11:41:26 +02:00
Murat Kabilov fc127069ab remove unnecessary ControllerNamespace 2017-05-12 11:41:26 +02:00
Murat Kabilov 416dace289 get rid of arrays in the kuberesources;
use shorter form of checking for errors
2017-05-12 11:41:26 +02:00
Oleksii Kliukin abd313f2d9 Fix a missing colon. 2017-05-12 11:41:26 +02:00
Oleksii Kliukin f65fab00dd Fix a typo 2017-05-12 11:41:26 +02:00
Oleksii Kliukin 033c28f03a Delete persistent volumes on deletion of the cluster. 2017-05-12 11:41:26 +02:00
Murat Kabilov caa0eab19b Move statefulset creation from cluster spec to the separate function 2017-05-12 11:41:25 +02:00
Oleksii Kliukin a2e78ac2ec Feature/persistent volumes 2017-05-12 11:41:25 +02:00
Murat Kabilov ae77fa15e8 Pod Rolling update
introduce Pod events channel;
add parsing of the MaintenanceWindows section;
skip deleting Etcd key on cluster delete;
use external etcd host;
watch for tpr/pods in the namespace of the operator pod only;
2017-05-12 11:41:25 +02:00
Murat Kabilov dfde075c66 Use TPR object namespace while creating its objects 2017-05-12 11:37:09 +02:00
Murat Kabilov 6e2d64bd50 Create human users from teams api 2017-05-12 11:37:09 +02:00
Murat Kabilov 58506634c4 Create pg users 2017-05-12 11:37:09 +02:00
Murat Kabilov 7e4d0410c2 Use one secret per user 2017-05-12 11:37:09 +02:00
Murat Kabilov abb1173035 Code refactor 2017-05-12 11:37:09 +02:00