Improve infrastructure role definitions (#208)

Enhance definitions of infrastructure roles by allowing membership in multiple roles, role options and per-role configuration to be specified in the infrastructure role configmap, which must have the same name as the infrastructure role secret. See manifests/infrastructure-roles-configmap.yaml for the examples and updated README for the description of different types of database roles supposed by the operator and their purposes.

Change the logic of merging infrastructure roles with the manifest roles when they have the same name, to return the infrastructure role unchanged instead of merging. Previously, we used to propagate flags from the manifest role to the resulting infrastructure one, as there were no way to define flags for the infrastructure role; however, this is not the case anymore.

Code review and tests by @erthalion

README.md

@@ -316,6 +316,100 @@ By default is set to *"log_statement:all"*. See [PostgreSQL documentation on ALT
 The default value is `admin`. Operator will also disallow superuser and replication roles to be redefined.
 
 
+### Defining database roles in the operator
+
+Postgres operator allows defining roles to be created in the resulting database cluster. It covers three use-cases:
+
+* create application roles specific to the cluster described in the manifest: `manifest roles`.
+* create application roles that should be automatically created on every cluster managed by the operator: `infrastructure roles`.
+* automatically create users for every member of the team owning the database cluster: `teams API roles`.
+
+In the next sections, we will cover those use cases in more details.
+
+#### Manifest roles
+
+Manifest roles are defined directly in the cluster manifest. See [minimal postgres manifest](https://github.com/zalando-incubator/postgres-operator/blob/master/manifests/minimal-postgres-manifest.yaml) for an example of `zalando` role, defined with `superuser` and `createdb` flags.
+
+Manifest roles are defined as a dictionary, with a role name as a key and a list of role options as a value. For a role without any options supply an empty list.
+
+The operator accepts the following options:  `superuser`, `inherit`, `login`, `nologin`, `createrole`, `createdb`, `replication`, `bypassrls`.
+
+By default, manifest roles are login roles (aka users), unless `nologin` is specified explicitly.
+
+The operator automatically generates a password for each manifest role and places it in the secret named
+`{username}.{team}-{clustername}.credentials.postgresql.acid.zalan.do` in the same namespace as the cluster.
+This way, the application running in the Kubernetes cluster and working with the database can obtain the password right from the secret, without ever sharing it outside of the cluster.  
+
+At the moment it is not possible to define membership of the manifest role in other roles.
+
+#### Infrastructure roles
+
+An infrastructure role is a role that should be present on every PostgreSQL cluster managed by the operator. An example of such a role is a monitoring user. There are two ways to define them:
+
+* Exclusively via the infrastructure roles secret (specified by the `infrastructure_roles_secret_name` parameter).
+
+The role definition looks like this (values are base64 encoded):
+
+
+    user1: ZGJ1c2Vy
+    password1: c2VjcmV0
+    inrole1: b3BlcmF0b3I=
+
+A block above describes the infrastructure role 'dbuser' with the password 'secret' that is the member of the 'operator' role.
+For the following definitions one must increase the index, i.e. the next role will be defined as 'user2' and so on. Note that there is no way to specify role options (like superuser or nologin) this way, and the resulting role will automatically be a login role.
+
+*  Via both the infrastructure roles secret and the infrastructure role configmap (with the same name as the infrastructure roles secret).
+
+The infrastructure roles secret should contain an entry with 'rolename: rolepassword' for each role, and the role description should be specified in the configmap. Below is the example:
+
+
+    dbuser: c2VjcmV0
+
+and the configmap definition for that user:
+
+    data:
+      dbuser: |
+        inrole: [operator, admin]  # following roles will be assigned to the new user
+        user_flags:
+          - createdb
+        db_parameters:  # db parameters, applied for this particular user
+          log_statement: all
+
+Note that the definition above allows for more details than the one that relies solely on the infrastructure role secret.
+In particular, one can allow membership in multiple roles via the `inrole` array parameter, define role flags via the `user_flags` list
+and supply per-role options through the `db_parameters` dictionary. All those parameters are optional.
+
+The definitions that solely use the infrastructure roles secret are more limited and considered legacy ones; one should use the new style that specifies infrastructure roles using both the secret and the configmap. You can mix both in the infrastructure role secret, as long as your new-style definition can be clearly distinguished from the old-style one (for instance, do not name new-style roles`userN`).
+
+Since an infrastructure role is created uniformly on all clusters managed by the operator, it makes no sense to define it without the password. Such definitions will be ignored with a prior warning.
+
+See [infrastructure roles secret](https://github.com/zalando-incubator/postgres-operator/blob/master/manifests/infrastructure-roles.yaml)
+and [infrastructure roles configmap](https://github.com/zalando-incubator/postgres-operator/blob/master/manifests/infrastructure-roles-configmap.yaml) for the examples.
+
+#### Teams API roles
+
+Teams API roles cover the task of creating human users on the cluster. The operator calls a special Teams API endpoint (configured via the `teams_api_url` parameter) to get the list of human users for the particular cluster. It provides the team id (configured via the `teamId` parameter on the cluster itself) to the teams API.
+
+There is a demo implementation of the teams API server at [fake teams api project](https://github.com/ikitiki/fake-teams-api).
+The operator expects an OAuth2 authentication for the teams API endpoint. To fetch the OAuth2 token, it reads the secret with the name specified by the `oauth_token_secret_name` operator configuration. That secret should contain two fields:
+`read-only-token-type` equal to `Bearer` and `read-only-token-secret`, containing the actual token. It is the task of some external service to rotate those tokens properly.
+
+Once the operator gets the list of team members from the teams API, it creates them as members of the `pam_role_name` role (configured in the operator configuration).  The operator creates them as LOGIN roles and optionally assigns them superuser (if `enable_team_superuser` is set) and `team_admin_role` role (if it is set).
+
+Note that the operator does not create any password for those roles, as those are supposed to authenticate against the OAuth2 endpoint using the [pam-oauth](https://github.com/CyberDem0n/pam-oauth2) module that is the part of [Spilo](https://github.com/zalando/spilo). The operator passes the URL specified in the `pam_configuration` parameter to Spilo, which configures the `pg_hba.conf` authentication for `pam_role_name` group to pass the token provided by the user (as the password) to that URL, together with the username.
+
+The pre-requisite to this is an OAuth2 service that generates tokens for users and provides an URL for authenticating them. Once this infrastructure is in place, it will, combined with `pam_oauth`, give human users strong auto-expiring passwords.
+
+For small installations, the teams API can be disabled by setting `enable_teams_api` to `false` in the operator configuration; then it is the task of the cluster admin to manage human users manually.
+
+#### Role priorities
+
+When there is a naming conflict between roles coming from different origins (i.e. an infrastructure role defined with the same name as the manifest role), the operator will choose the one with the highest priority origin.
+
+System roles (configured with `super_username` and `replication_username` in the operator) have the highest priority; next are team API roles, infrastructure roles and manifest roles.
+
+There is a mechanism that prevents overriding critical roles: it is not possible to override system roles (the operator will give an error even before applying priority rules);  the same applies to the roles mentioned in the `protected_role_names` list in the operator configuration.
+
 ### Debugging the operator itself
 
 There is a web interface in the operator to observe its internal state. The operator listens on port 8080. It is possible to expose it to the localhost:8080 by doing:

glide.lock

@@ -1,8 +1,8 @@
-hash: bb2d336f3efb57376916e47b585ad55611b5023b79044ced59c762ea35427f19
-updated: 2017-10-10T10:40:56.894070487+02:00
+hash: aa008e00a8cf34fa59a081dd67644319f9d5d077bf4de81aa4a25a2d5b8781a1
+updated: 2018-01-15T15:00:53.231516+01:00
 imports:
 - name: github.com/aws/aws-sdk-go
-  version: 760741802ad40f49ae9fc4a69ef6706d2527d62e
+  version: 0cebc639926eb91b0192dae4b28bc808417e764c
   subpackages:
   - aws
   - aws/awserr
@@ -104,7 +104,7 @@ imports:
 - name: github.com/PuerkitoBio/urlesc
   version: 5bd2802263f21d8788851d5305584c82a5c75d7e
 - name: github.com/Sirupsen/logrus
-  version: f006c2ac4710855cf0f916dd6b77acf6b048dc6e
+  version: d682213848ed68c0a260ca37d6dd5ace8423f5ba
 - name: github.com/spf13/pflag
   version: 9ff6c6923cfffbcd502984b8e0c80539a94968b7
 - name: github.com/ugorji/go

glide.yaml

@@ -43,3 +43,4 @@ import:
   - tools/cache
   - tools/clientcmd
   - tools/remotecommand
+- package: gopkg.in/yaml.v2

manifests/infrastructure-roles-configmap.yaml

@@ -0,0 +1,11 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: postgresql-infrastructure-roles
+data:
+  batman: |
+    inrole: [admin]  # following roles will be assigned to the new user
+    user_flags:
+      - createdb
+    db_parameters:  # db parameters, applyed for this particular user
+      log_statement: all

manifests/infrastructure-roles.yaml

@@ -3,7 +3,8 @@ data:
   # required format (w/o quotes): 'propertyNumber: value'
   # allowed properties: 'user', 'password', 'inrole'
   # numbers >= 1 are mandatory
-  
+  # alternatively, supply the user: password pairs and
+  # provide other options in the configmap.
   # robot_zmon_acid_monitoring
   user1: cm9ib3Rfem1vbl9hY2lkX21vbml0b3Jpbmc=
   # robot_zmon
@@ -12,6 +13,9 @@ data:
   user2: dGVzdHVzZXI=
   # foobar
   password2: Zm9vYmFy
+  # user batman with the password justice
+  # look for other fields in the infrastructure roles configmap
+  batman: anVzdGljZQ==
 kind: Secret
 metadata:
   name: postgresql-infrastructure-roles

pkg/cluster/cluster.go

@@ -674,24 +674,18 @@ func (c *Cluster) initRobotUsers() error {
 		if err != nil {
 			return fmt.Errorf("invalid flags for user %q: %v", username, err)
 		}
-		if _, present := c.pgUsers[username]; !present {
-			c.pgUsers[username] = spec.PgUser{
-				Origin:   spec.RoleOriginManifest,
-				Name:     username,
-				Password: util.RandomPassword(constants.PasswordLength),
-				Flags:    flags,
-			}
+		newRole := spec.PgUser{
+			Origin:   spec.RoleOriginManifest,
+			Name:     username,
+			Password: util.RandomPassword(constants.PasswordLength),
+			Flags:    flags,
+		}
+		if currentRole, present := c.pgUsers[username]; present {
+			c.pgUsers[username] = c.resolveNameConflict(&currentRole, &newRole)
 		} else {
-			// avoid overwriting the password if the user is already there. The flags should be
-			// merged here, but since there is no mechanism to define them for non-robot roles
-			// they are assigned from the robot user.
-			c.logger.Debugf("merging manifest and infrastructure user %q data", username)
-			user := c.pgUsers[username]
-			user.Flags = flags
-			c.pgUsers[username] = user
+			c.pgUsers[username] = newRole
 		}
 	}
-
 	return nil
 }
 
@@ -715,41 +709,60 @@ func (c *Cluster) initHumanUsers() error {
 			}
 		}
 
-		if _, present := c.pgUsers[username]; present {
-			c.logger.Warnf("overwriting existing user %q with the data from the teams API", username)
-		}
-
-		c.pgUsers[username] = spec.PgUser{
+		newRole := spec.PgUser{
 			Origin:     spec.RoleOriginTeamsAPI,
 			Name:       username,
 			Flags:      flags,
 			MemberOf:   memberOf,
 			Parameters: c.OpConfig.TeamAPIRoleConfiguration,
 		}
+
+		if currentRole, present := c.pgUsers[username]; present {
+			c.pgUsers[username] = c.resolveNameConflict(&currentRole, &newRole)
+		} else {
+			c.pgUsers[username] = newRole
+		}
 	}
 
 	return nil
 }
 
 func (c *Cluster) initInfrastructureRoles() error {
-	// add infrastucture roles from the operator's definition
-	for username, data := range c.InfrastructureRoles {
+	// add infrastructure roles from the operator's definition
+	for username, newRole := range c.InfrastructureRoles {
 		if !isValidUsername(username) {
 			return fmt.Errorf("invalid username: '%v'", username)
 		}
 		if c.shouldAvoidProtectedOrSystemRole(username, "infrastructure role") {
 			continue
 		}
-		flags, err := normalizeUserFlags(data.Flags)
+		flags, err := normalizeUserFlags(newRole.Flags)
 		if err != nil {
 			return fmt.Errorf("invalid flags for user '%v': %v", username, err)
 		}
-		data.Flags = flags
-		c.pgUsers[username] = data
+		newRole.Flags = flags
+
+		if currentRole, present := c.pgUsers[username]; present {
+			c.pgUsers[username] = c.resolveNameConflict(&currentRole, &newRole)
+		} else {
+			c.pgUsers[username] = newRole
+		}
 	}
 	return nil
 }
 
+// resolves naming conflicts between existing and new roles by chosing either of them.
+func (c *Cluster) resolveNameConflict(currentRole, newRole *spec.PgUser) (result spec.PgUser) {
+	if newRole.Origin >= currentRole.Origin {
+		result = *newRole
+	} else {
+		result = *currentRole
+	}
+	c.logger.Debugf("resolved a conflict of role %q between %s and %s to %s",
+		newRole.Name, newRole.Origin, currentRole.Origin, result.Origin)
+	return
+}
+
 func (c *Cluster) shouldAvoidProtectedOrSystemRole(username, purpose string) bool {
 	if c.isProtectedUsername(username) {
 		c.logger.Warnf("cannot initialize a new %s with the name of the protected user %q", purpose, username)

pkg/cluster/cluster_test.go

@@ -33,9 +33,8 @@ func TestInitRobotUsers(t *testing.T) {
 	}{
 		{
 			manifestUsers: map[string]spec.UserFlags{"foo": {"superuser", "createdb"}},
-			infraRoles:    map[string]spec.PgUser{"foo": {Origin: spec.RoleOriginManifest, Name: "foo", Password: "bar"}},
-			result: map[string]spec.PgUser{"foo": {Origin: spec.RoleOriginManifest,
-				Name: "foo", Password: "bar", Flags: []string{"CREATEDB", "LOGIN", "SUPERUSER"}}},
+			infraRoles:    map[string]spec.PgUser{"foo": {Origin: spec.RoleOriginInfrastructure, Name: "foo", Password: "bar"}},
+			result: map[string]spec.PgUser{"foo": {Origin: spec.RoleOriginInfrastructure,  Name: "foo", Password: "bar"}},
 			err: nil,
 		},
 		{

pkg/controller/util.go

@@ -13,6 +13,7 @@ import (
 	"github.com/zalando-incubator/postgres-operator/pkg/util/config"
 	"github.com/zalando-incubator/postgres-operator/pkg/util/constants"
 	"github.com/zalando-incubator/postgres-operator/pkg/util/k8sutil"
+	"gopkg.in/yaml.v2"
 )
 
 func (c *Controller) makeClusterConfig() cluster.Config {
@@ -96,6 +97,14 @@ func (c *Controller) createCRD() error {
 	})
 }
 
+func readDecodedRole(s string) (*spec.PgUser, error) {
+	var result spec.PgUser
+	if err := yaml.Unmarshal([]byte(s), &result); err != nil {
+		return nil, fmt.Errorf("could not decode yaml role: %v", err)
+	}
+	return &result, nil
+}
+
 func (c *Controller) getInfrastructureRoles(rolesSecret *spec.NamespacedName) (result map[string]spec.PgUser, err error) {
 	if *rolesSecret == (spec.NamespacedName{}) {
 		// we don't have infrastructure roles defined, bail out
@@ -110,16 +119,16 @@ func (c *Controller) getInfrastructureRoles(rolesSecret *spec.NamespacedName) (r
 		return nil, fmt.Errorf("could not get infrastructure roles secret: %v", err)
 	}
 
-	data := infraRolesSecret.Data
+	secretData := infraRolesSecret.Data
 	result = make(map[string]spec.PgUser)
 Users:
 	// in worst case we would have one line per user
-	for i := 1; i <= len(data); i++ {
+	for i := 1; i <= len(secretData); i++ {
 		properties := []string{"user", "password", "inrole"}
 		t := spec.PgUser{Origin: spec.RoleOriginInfrastructure}
 		for _, p := range properties {
 			key := fmt.Sprintf("%s%d", p, i)
-			if val, present := data[key]; !present {
+			if val, present := secretData[key]; !present {
 				if p == "user" {
 					// exit when the user name with the next sequence id is absent
 					break Users
@@ -137,19 +146,48 @@ Users:
 					c.logger.Warningf("unknown key %q", p)
 				}
 			}
-
-			delete(data, key)
+			delete(secretData, key)
 		}
 
 		if t.Name != "" {
+			if t.Password == "" {
+				c.logger.Warningf("infrastructure role %q has no password defined and is ignored", t.Name)
+				continue
+			}
 			result[t.Name] = t
 		}
 	}
 
-	if len(data) != 0 {
-		c.logger.Warningf("%d unprocessed entries in the infrastructure roles' secret", len(data))
-		c.logger.Info(`infrastructure role entries should be in the {key}{id} format, where {key} can be either of "user", "password", "inrole" and the {id} a monotonically increasing integer starting with 1`)
-		c.logger.Debugf("unprocessed entries: %#v", data)
+	// perhaps we have some map entries with usernames, passwords, let's check if we have those users in the configmap
+	if infraRolesMap, err := c.KubeClient.ConfigMaps(rolesSecret.Namespace).Get(rolesSecret.Name, metav1.GetOptions{}); err == nil {
+		// we have a configmap with username - json description, let's read and decode it
+		for role, s := range infraRolesMap.Data {
+			if roleDescr, err := readDecodedRole(s); err != nil {
+				return nil, fmt.Errorf("could not decode role description: %v", err)
+			} else {
+				// check if we have a a password in a configmap
+				c.logger.Debugf("found role description for role %q: %+v", role, roleDescr)
+				if passwd, ok := secretData[role]; ok {
+					roleDescr.Password = string(passwd)
+					delete(secretData, role)
+				} else {
+					c.logger.Warningf("infrastructure role %q has no password defined and is ignored", role)
+					continue
+				}
+				roleDescr.Name = role
+				roleDescr.Origin = spec.RoleOriginInfrastructure
+				result[role] = *roleDescr
+			}
+		}
+	}
+
+	if len(secretData) > 0 {
+		c.logger.Warningf("%d unprocessed entries in the infrastructure roles secret,"+
+			" checking configmap %v", len(secretData), rolesSecret.Name)
+		c.logger.Info(`infrastructure role entries should be in the {key}{id} format,` +
+			` where {key} can be either of "user", "password", "inrole" and the {id}` +
+			` a monotonically increasing integer starting with 1`)
+		c.logger.Debugf("unprocessed entries: %#v", secretData)
 	}
 
 	return result, nil

pkg/controller/util_test.go

@@ -5,6 +5,7 @@ import (
 	"reflect"
 	"testing"
 
+	b64 "encoding/base64"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	v1core "k8s.io/client-go/kubernetes/typed/core/v1"
 	"k8s.io/client-go/pkg/api/v1"
@@ -21,6 +22,10 @@ type mockSecret struct {
 	v1core.SecretInterface
 }
 
+type mockConfigMap struct {
+	v1core.ConfigMapInterface
+}
+
 func (c *mockSecret) Get(name string, options metav1.GetOptions) (*v1.Secret, error) {
 	if name != testInfrastructureRolesSecretName {
 		return nil, fmt.Errorf("NotFound")
@@ -31,20 +36,43 @@ func (c *mockSecret) Get(name string, options metav1.GetOptions) (*v1.Secret, er
 		"user1":     []byte("testrole"),
 		"password1": []byte("testpassword"),
 		"inrole1":   []byte("testinrole"),
+		"foobar":    []byte(b64.StdEncoding.EncodeToString([]byte("password"))),
 	}
 	return secret, nil
 
 }
 
+func (c *mockConfigMap) Get(name string, options metav1.GetOptions) (*v1.ConfigMap, error) {
+	if name != testInfrastructureRolesSecretName {
+		return nil, fmt.Errorf("NotFound")
+	}
+	configmap := &v1.ConfigMap{}
+	configmap.Name = mockController.opConfig.ClusterNameLabel
+	configmap.Data = map[string]string{
+		"foobar": "{}",
+	}
+	return configmap, nil
+}
+
 type MockSecretGetter struct {
 }
 
+type MockConfigMapsGetter struct {
+}
+
 func (c *MockSecretGetter) Secrets(namespace string) v1core.SecretInterface {
 	return &mockSecret{}
 }
 
+func (c *MockConfigMapsGetter) ConfigMaps(namespace string) v1core.ConfigMapInterface {
+	return &mockConfigMap{}
+}
+
 func newMockKubernetesClient() k8sutil.KubernetesClient {
-	return k8sutil.KubernetesClient{SecretsGetter: &MockSecretGetter{}}
+	return k8sutil.KubernetesClient{
+		SecretsGetter:    &MockSecretGetter{},
+		ConfigMapsGetter: &MockConfigMapsGetter{},
+	}
 }
 
 func newMockController() *Controller {
@@ -135,6 +163,12 @@ func TestGetInfrastructureRoles(t *testing.T) {
 					Password: "testpassword",
 					MemberOf: []string{"testinrole"},
 				},
+				"foobar": {
+					Name:     "foobar",
+					Origin:   spec.RoleOriginInfrastructure,
+					Password: b64.StdEncoding.EncodeToString([]byte("password")),
+					MemberOf: nil,
+				},
 			},
 			nil,
 		},

pkg/spec/types.go

@@ -32,12 +32,14 @@ const (
 	fileWithNamespace = "/var/run/secrets/kubernetes.io/serviceaccount/namespace"
 )
 
+// RoleOrigin contains the code of the origin of a role
 type RoleOrigin int
 
+// The rolesOrigin constant values should be sorted by the role priority.
 const (
-	RoleOriginUnknown = iota
-	RoleOriginInfrastructure
+	RoleOriginUnknown RoleOrigin = iota
 	RoleOriginManifest
+	RoleOriginInfrastructure
 	RoleOriginTeamsAPI
 	RoleOriginSystem
 )
@@ -72,12 +74,12 @@ type PodEvent struct {
 
 // PgUser contains information about a single user.
 type PgUser struct {
-	Origin     RoleOrigin
-	Name       string
-	Password   string
-	Flags      []string
-	MemberOf   []string
-	Parameters map[string]string
+	Origin     RoleOrigin        `yaml:"-"`
+	Name       string            `yaml:"-"`
+	Password   string            `yaml:"-"`
+	Flags      []string          `yaml:"user_flags"`
+	MemberOf   []string          `yaml:"inrole"`
+	Parameters map[string]string `yaml:"db_parameters"`
 }
 
 // PgUserMap maps user names to the definitions.
@@ -203,6 +205,20 @@ func (n *NamespacedName) DecodeWorker(value, operatorNamespace string) error {
 	return nil
 }
 
+func (r RoleOrigin) String() string {
+	switch r {
+	case RoleOriginManifest:
+		return "manifest role"
+	case RoleOriginInfrastructure:
+		return "infrastructure role"
+	case RoleOriginTeamsAPI:
+		return "teams API role"
+	case RoleOriginSystem:
+		return "system role"
+	}
+	return "unknown"
+}
+
 // GetOperatorNamespace assumes serviceaccount secret is mounted by kubernetes
 // Placing this func here instead of pgk/util avoids circular import
 func GetOperatorNamespace() string {