Configure load balancer on a per-cluster and operator-wide level (#57)

* Deny all requests to the load balancer by default.
* Operator-wide toggle for the load-balancer.
* Define per-cluster useLoadBalancer option.

If useLoadBalancer is not set - then operator-wide defaults take place. If it
is true - the load balancer is created, otherwise a service type clusterIP is
created.

Internally, we have to completely replace the service if the service type
changes. We cannot patch, since some fields from the old service that will
remain after patch are incompatible with the new one, and handling them
explicitly when updating the service is ugly and error-prone. We cannot
update the service because of the immutable fields, that leaves us the only
option of deleting the old service and creating the new one. Unfortunately,
there is still an issue of unnecessary removal of endpoints associated with
the service, it will be addressed in future commits.

* Revert the unintended effect of go fmt

* Recreate endpoints on service update.

When the service type is changed, the service is deleted and then
the one with the new type is created. Unfortnately, endpoints are
deleted as well. Re-create them afterwards, preserving the original
addresses stored in them.

* Improve error messages and comments. Use generate instead of gen in names.

pkg/util/config/config.go

@@ -53,6 +53,7 @@ type Config struct {
 	DebugLogging         bool           `name:"debug_logging" default:"true"`
 	EnableDBAccess       bool           `name:"enable_database_access" default:"true"`
 	EnableTeamsAPI       bool           `name:"enable_teams_api" default:"true"`
+	EnableLoadBalancer   bool           `name:"enable_load_balancer" default:"true"`
 	MasterDNSNameFormat  stringTemplate `name:"master_dns_name_format" default:"{cluster}.{team}.{hostedzone}"`
 	ReplicaDNSNameFormat stringTemplate `name:"replica_dns_name_format" default:"{cluster}-repl.{team}.{hostedzone}"`
 	Workers              uint32         `name:"workers" default:"4"`

pkg/util/constants/annotations.go

@@ -2,10 +2,10 @@ package constants
 
 // Names and values in Kubernetes annotation for services, statefulsets and volumes
 const (
-	ZalandoDNSNameAnnotation           = "external-dns.alpha.kubernetes.io/hostname"
-	ElbTimeoutAnnotationName           = "service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout"
-	ElbTimeoutAnnotationValue          = "3600"
-	KubeIAmAnnotation                  = "iam.amazonaws.com/role"
-	VolumeStorateProvisionerAnnotation = "pv.kubernetes.io/provisioned-by"
-	ServiceMetadataAnnotationFormat    = `{"metadata":{"annotations": {"$patch":"replace", %s}}}`
+	ZalandoDNSNameAnnotation               = "external-dns.alpha.kubernetes.io/hostname"
+	ElbTimeoutAnnotationName               = "service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout"
+	ElbTimeoutAnnotationValue              = "3600"
+	KubeIAmAnnotation                      = "iam.amazonaws.com/role"
+	VolumeStorateProvisionerAnnotation     = "pv.kubernetes.io/provisioned-by"
+	ServiceMetadataAnnotationReplaceFormat = `{"metadata":{"annotations": {"$patch":"replace", %s}}}`
 )