public_git
/
postgres-operator
mirror of https://github.com/zalando/postgres-operator.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

[UI] add tab for monthly costs per cluster (#796) 

* add tab for monthly costs per cluster
* sync run_local and update version number
* lowering resources
* some Makefile polishing and updated admin docs on UI
* extend admin docs on UI
* add api-service manifest for operator
* set min limits in UI to default min limits of operator
* reflect new UI helm charts in docs
* make cluster name label configurable
This commit is contained in:
Felix Kunde 2020-02-19 12:58:24 +01:00 committed by GitHub
parent aea9e9bd33
commit d5660f65bb
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
17 changed files with 257 additions and 86 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
charts/postgres-operator-ui/Chart.yaml
View File

@ -1,7 +1,7 @@
apiVersion: v1
name: postgres-operator-ui
version: 0.1.0
appVersion: 1.2.0
appVersion: 1.3.0
home: https://github.com/zalando/postgres-operator
description: Postgres Operator UI provides a graphical interface for a convenient database-as-a-service user experience
keywords:
@ -12,6 +12,8 @@ keywords:
- patroni
- spilo
maintainers:
- name: Zalando
email: opensource@zalando.de
- name: siku4
email: sk@sik-net.de
sources:

5
charts/postgres-operator/templates/service.yaml
View File

@ -8,6 +8,7 @@ metadata:
app.kubernetes.io/instance: {{ .Release.Name }}
name: {{ template "postgres-operator.fullname" . }}
spec:
type: ClusterIP
ports:
- port: 8080
protocol: TCP
@ -15,7 +16,3 @@ spec:
selector:
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/name: {{ template "postgres-operator.name" . }}
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}

72
docs/administrator.md
View File

@ -480,37 +480,71 @@ A secret can be pre-provisioned in different ways:
## Setting up the Postgres Operator UI
With the v1.2 release the Postgres Operator is shipped with a browser-based
Since the v1.2 release the Postgres Operator is shipped with a browser-based
configuration user interface (UI) that simplifies managing Postgres clusters
with the operator. The UI runs with Node.js and comes with it's own Docker
image.
with the operator.
Run NPM to continuously compile `tags/js` code. Basically, it creates an
`app.js` file in: `static/build/app.js`
### Building the UI image
```
(cd ui/app && npm start)
```
To build the Docker image open a shell and change to the `ui` folder. Then run:
The UI runs with Node.js and comes with it's own Docker
image. However, installing Node.js to build the operator UI is not required. It
is handled via Docker containers when running:
```bash
docker build -t registry.opensource.zalan.do/acid/postgres-operator-ui:v1.2.0 .
make docker
```
Apply all manifests for the `ui/manifests` folder to deploy the Postgres
Operator UI on K8s. For local tests you don't need the Ingress resource.
### Configure endpoints and options
The UI talks to the K8s API server as well as the Postgres Operator [REST API](developer.md#debugging-the-operator).
K8s API server URLs are loaded from the machine's kubeconfig environment by
default. Alternatively, a list can also be passed when starting the Python
application with the `--cluster` option.
The Operator API endpoint can be configured via the `OPERATOR_API_URL`
environment variables in the [deployment manifest](../ui/manifests/deployment.yaml#L40).
You can also expose the operator API through a [service](../manifests/api-service.yaml).
Some displayed options can be disabled from UI using simple flags under the
`OPERATOR_UI_CONFIG` field in the deployment.
### Deploy the UI on K8s
Now, apply all manifests from the `ui/manifests` folder to deploy the Postgres
Operator UI on K8s. Replace the image tag in the deployment manifest if you
want to test the image you've built with `make docker`. Make sure the pods for
the operator and the UI are both running.
```bash
kubectl apply -f ui/manifests
sed -e "s/\(image\:.*\:\).*$/\1$TAG/" manifests/deployment.yaml | kubectl apply -f manifests/
kubectl get all -l application=postgres-operator-ui
```
Make sure the pods for the operator and the UI are both running. For local
testing you need to apply proxying and port forwarding so that the UI can talk
to the K8s and Postgres Operator REST API. You can use the provided
`run_local.sh` script for this. Make sure it uses the correct URL to your K8s
API server, e.g. for minikube it would be `https://192.168.99.100:8443`.
### Local testing
For local testing you need to apply K8s proxying and operator pod port
forwarding so that the UI can talk to the K8s and Postgres Operator REST API.
The Ingress resource is not needed. You can use the provided `run_local.sh`
script for this. Make sure that:
* Python dependencies are installed on your machine
* the K8s API server URL is set for kubectl commands, e.g. for minikube it would usually be `https://192.168.99.100:8443`.
* the pod label selectors for port forwarding are correct
When testing with minikube you have to build the image in its docker environment
(running `make docker` doesn't do it for you). From the `ui` directory execute:
```bash
# compile and build operator UI
make docker
# build in image in minikube docker env
eval $(minikube docker-env)
docker build -t registry.opensource.zalan.do/acid/postgres-operator-ui:v1.3.0 .
# apply UI manifests next to a running Postgres Operator
kubectl apply -f manifests/
# install python dependencies to run UI locally
pip3 install -r requirements
./run_local.sh
```

10
docs/operator-ui.md
View File

@ -31,9 +31,13 @@ status page.
![pgui-waiting-for-master](diagrams/pgui-waiting-for-master.png "Waiting for master pod")
Usually, the startup should only take up to 1 minute. If you feel the process
got stuck click on the "Logs" button to inspect the operator logs. From the
"Status" field in the top menu you can also retrieve the logs and queue of each
worker the operator is using. The number of concurrent workers can be
got stuck click on the "Logs" button to inspect the operator logs. If the logs
look fine, but the UI seems to got stuck, check if you are have configured the
same [cluster name label](../ui/manifests/deployment.yaml#L45) like for the
[operator](../manifests/configmap.yaml#L13).
From the "Status" field in the top menu you can also retrieve the logs and queue
of each worker the operator is using. The number of concurrent workers can be
[configured](reference/operator_parameters.md#general).
![pgui-operator-logs](diagrams/pgui-operator-logs.png "Checking operator logs")

57
docs/quickstart.md
View File

@ -52,6 +52,7 @@ cd postgres-operator
kubectl create -f manifests/configmap.yaml # configuration
kubectl create -f manifests/operator-service-account-rbac.yaml # identity and permissions
kubectl create -f manifests/postgres-operator.yaml # deployment
kubectl create -f manifests/api-service.yaml # operator API to be used by UI
```
There is a [Kustomization](https://github.com/kubernetes-sigs/kustomize)
@ -104,7 +105,7 @@ kubectl create -f https://operatorhub.io/install/postgres-operator.yaml
This installs the operator in the `operators` namespace. More information can be
found on [operatorhub.io](https://operatorhub.io/operator/postgres-operator).
## Create a Postgres cluster
## Check if Postgres Operator is running
Starting the operator may take a few seconds. Check if the operator pod is
running before applying a Postgres cluster manifest.
@ -115,7 +116,61 @@ kubectl get pod -l name=postgres-operator
# if you've created the operator using helm chart
kubectl get pod -l app.kubernetes.io/name=postgres-operator
```
If the operator doesn't get into `Running` state, either check the latest K8s
events of the deployment or pod with `kubectl describe` or inspect the operator
logs:
```bash
kubectl logs "$(kubectl get pod -l name=postgres-operator --output='name')"
```
## Deploy the operator UI
In the following paragraphs we describe how to access and manage PostgreSQL
clusters from the command line with kubectl. But it can also be done from the
browser-based [Postgres Operator UI](operator-ui.md). Before deploying the UI
make sure the operator is running and its REST API is reachable through a
[K8s service](../manifests/api-service.yaml). The URL to this API must be
configured in the [deployment manifest](../ui/manifests/deployment.yaml#L43)
of the UI.
To deploy the UI simply apply all its manifests files or use the UI helm chart:
```bash
# manual deployment
kubectl apply -f ui/manifests/
# or helm chart
helm install postgres-operator-ui ./charts/postgres-operator-ui
```
Like with the operator, check if the UI pod gets into `Running` state:
```bash
# if you've created the operator using yaml manifests
kubectl get pod -l name=postgres-operator-ui
# if you've created the operator using helm chart
kubectl get pod -l app.kubernetes.io/name=postgres-operator-ui
```
You can now access the web interface by port forwarding the UI pod (mind the
label selector) and enter `localhost:8081` in your browser:
```bash
kubectl port-forward "$(kubectl get pod -l name=postgres-operator-ui --output='name')" 8081
```
Available option are explained in detail in the [UI docs](operator-ui.md).
## Create a Postgres cluster
If the operator pod is running it listens to new events regarding `postgresql`
resources. Now, it's time to submit your first Postgres cluster manifest.
```bash
# create a Postgres cluster
kubectl create -f manifests/minimal-postgres-manifest.yaml
```

12
manifests/api-service.yaml Normal file
View File

@ -0,0 +1,12 @@
apiVersion: v1
kind: Service
metadata:
name: postgres-operator
spec:
type: ClusterIP
ports:
- port: 8080
protocol: TCP
targetPort: 8080
selector:
name: postgres-operator

1
manifests/kustomization.yaml
View File

@ -4,3 +4,4 @@ resources:
- configmap.yaml
- operator-service-account-rbac.yaml
- postgres-operator.yaml
- api-service.yaml

20
ui/Makefile
View File

@ -1,17 +1,6 @@
.PHONY: clean test appjs docker push mock
BINARY ?= postgres-operator-ui
BUILD_FLAGS ?= -v
CGO_ENABLED ?= 0
ifeq ($(RACE),1)
BUILD_FLAGS += -race -a
CGO_ENABLED=1
endif
LOCAL_BUILD_FLAGS ?= $(BUILD_FLAGS)
LDFLAGS ?= -X=main.version=$(VERSION)
IMAGE ?= registry.opensource.zalan.do/acid/$(BINARY)
IMAGE ?= registry.opensource.zalan.do/acid/postgres-operator-ui
VERSION ?= $(shell git describe --tags --always --dirty)
TAG ?= $(VERSION)
GITHEAD = $(shell git rev-parse --short HEAD)
@ -32,8 +21,11 @@ appjs:
docker run $(TTYFLAGS) -u $$(id -u) -v $$(pwd):/workdir -w /workdir/app node:10.1.0-alpine npm run build
docker: appjs
docker build --build-arg "VERSION=$(VERSION)" -t "$(IMAGE):$(TAG)" .
@echo 'Docker image $(IMAGE):$(TAG) can now be used.'
echo `(env)`
echo "Tag ${TAG}"
echo "Version ${VERSION}"
echo "git describe $(shell git describe --tags --always --dirty)"
docker build --rm -t "$(IMAGE):$(TAG)" -f Dockerfile .
push: docker
docker push "$(IMAGE):$(TAG)"

2
ui/app/package.json
View File

@ -1,6 +1,6 @@
{
"name": "postgres-operator-ui",
"version": "1.0.0",
"version": "1.3.0",
"description": "PostgreSQL Operator UI",
"main": "src/app.js",
"config": {

16
ui/app/src/new.tag.pug
View File

@ -408,7 +408,7 @@ new
ref='cpuLimit'
type='number'
placeholder='{ cpu.state.limit.initialValue }'
min='1'
min='250'
required
value='{ cpu.state.limit.state }'
onchange='{ cpu.state.limit.edit }'
@ -434,7 +434,7 @@ new
onkeyup='{ memory.state.request.edit }'
)
.input-group-addon
.input-units Gi
.input-units Mi
.input-group
.input-group-addon.resource-type Limit
@ -442,14 +442,14 @@ new
ref='memoryLimit'
type='number'
placeholder='{ memory.state.limit.initialValue }'
min='1'
min='250'
required
value='{ memory.state.limit.state }'
onchange='{ memory.state.limit.edit }'
onkeyup='{ memory.state.limit.edit }'
)
.input-group-addon
.input-units Gi
.input-units Mi
.col-lg-3
help-general(config='{ opts.config }')
@ -519,10 +519,10 @@ new
resources:
requests:
cpu: {{ cpu.state.request.state }}m
memory: {{ memory.state.request.state }}Gi
memory: {{ memory.state.request.state }}Mi
limits:
cpu: {{ cpu.state.limit.state }}m
memory: {{ memory.state.limit.state }}Gi{{#if restoring}}
memory: {{ memory.state.limit.state }}Mi{{#if restoring}}
clone:
cluster: "{{ backup.state.name.state }}"
@ -786,8 +786,8 @@ new
return instance
}
this.cpu = DynamicResource({ request: 100, limit: 1000 })
this.memory = DynamicResource({ request: 1, limit: 1 })
this.cpu = DynamicResource({ request: 100, limit: 500 })
this.memory = DynamicResource({ request: 100, limit: 500 })
this.backup = DynamicSet({
type: () => 'empty',

3
ui/app/src/postgresql.tag.pug
View File

@ -76,6 +76,9 @@ postgresql
.alert.alert-danger(if='{ progress.requestStatus !== "OK" }') Create request failed
.alert.alert-success(if='{ progress.requestStatus === "OK" }') Create request successful ({ new Date(progress.createdTimestamp).toLocaleString() })
.alert.alert-info(if='{ !progress.postgresql }') PostgreSQL cluster manifest pending
.alert.alert-success(if='{ progress.postgresql }') PostgreSQL cluster manifest created
.alert.alert-info(if='{ !progress.statefulSet }') StatefulSet pending
.alert.alert-success(if='{ progress.statefulSet }') StatefulSet created

79
ui/app/src/postgresqls.tag.pug
View File

@ -45,12 +45,14 @@ postgresqls
thead
tr
th(style='width: 120px') Team
th(style='width: 130px') Namespace
th Name
th(style='width: 50px') Pods
th(style='width: 140px') CPU
th(style='width: 130px') Memory
th(style='width: 100px') Size
th(style='width: 130px') Namespace
th Name
th(style='width: 120px') Cost/Month
th(stlye='width: 120px')
tbody
tr(
@ -58,19 +60,21 @@ postgresqls
hidden='{ !namespaced_name.toLowerCase().includes(filter.state.toLowerCase()) }'
)
td { team }
td { nodes }
td { cpu } / { cpu_limit }
td { memory } / { memory_limit }
td { volume_size }
td(style='white-space: pre')
| { namespace }
td
a(
href='/#/status/{ cluster_path(this) }'
)
| { name }
td { nodes }
td { cpu } / { cpu_limit }
td { memory } / { memory_limit }
td { volume_size }
td { calcCosts(nodes, cpu, memory, volume_size) }$
td
.btn-group.pull-right(
aria-label='Cluster { qname } actions'
@ -124,12 +128,14 @@ postgresqls
thead
tr
th(style='width: 120px') Team
th(style='width: 130px') Namespace
th Name
th(style='width: 50px') Pods
th(style='width: 140px') CPU
th(style='width: 130px') Memory
th(style='width: 100px') Size
th(style='width: 130px') Namespace
th Name
th(style='width: 120px') Cost/Month
th(stlye='width: 120px')
tbody
tr(
@ -137,20 +143,20 @@ postgresqls
hidden='{ !namespaced_name.toLowerCase().includes(filter.state.toLowerCase()) }'
)
td { team }
td { nodes }
td { cpu } / { cpu_limit }
td { memory } / { memory_limit }
td { volume_size }
td(style='white-space: pre')
| { namespace }
td
a(
href='/#/status/{ cluster_path(this) }'
)
| { name }
td { nodes }
td { cpu } / { cpu_limit }
td { memory } / { memory_limit }
td { volume_size }
td { calcCosts(nodes, cpu, memory, volume_size) }$
td
.btn-group.pull-right(
aria-label='Cluster { qname } actions'
@ -223,6 +229,45 @@ postgresqls
+ '/' + encodeURI(cluster.name)
)
const calcCosts = this.calcCosts = (nodes, cpu, memory, disk) => {
costs = nodes * (toCores(cpu) * opts.config.cost_core + toMemory(memory) * opts.config.cost_memory + toDisk(disk) * opts.config.cost_ebs)
return costs.toFixed(2)
}
const toDisk = this.toDisk = value => {
if(value.endsWith("Gi")) {
value = value.substring(0, value.length-2)
value = Number(value)
return value
}
return value
}
const toMemory = this.toMemory = value => {
if (value.endsWith("Mi")) {
value = value.substring(0, value.length-2)
value = Number(value) / 1000.
return value
}
else if(value.endsWith("Gi")) {
value = value.substring(0, value.length-2)
value = Number(value)
return value
}
return value
}
const toCores = this.toCores = value => {
if (value.endsWith("m")) {
value = value.substring(0, value.length-1)
value = Number(value) / 1000.
return value
}
return value
}
this.on('mount', () =>
jQuery
.get('/postgresqls')

23
ui/manifests/deployment.yaml
View File

@ -4,23 +4,23 @@ metadata:
name: "postgres-operator-ui"
namespace: "default"
labels:
application: "postgres-operator-ui"
name: "postgres-operator-ui"
team: "acid"
spec:
replicas: 1
selector:
matchLabels:
application: "postgres-operator-ui"
name: "postgres-operator-ui"
template:
metadata:
labels:
application: "postgres-operator-ui"
name: "postgres-operator-ui"
team: "acid"
spec:
serviceAccountName: postgres-operator-ui
containers:
- name: "service"
image: registry.opensource.zalan.do/acid/postgres-operator-ui:v1.2.0
image: registry.opensource.zalan.do/acid/postgres-operator-ui:v1.3.0
ports:
- containerPort: 8081
protocol: "TCP"
@ -32,8 +32,8 @@ spec:
timeoutSeconds: 1
resources:
limits:
cpu: "300m"
memory: "3000Mi"
cpu: "200m"
memory: "200Mi"
requests:
cpu: "100m"
memory: "100Mi"
@ -41,7 +41,9 @@ spec:
- name: "APP_URL"
value: "http://localhost:8081"
- name: "OPERATOR_API_URL"
value: "http://localhost:8080"
value: "http://postgres-operator:8080"
- name: "OPERATOR_CLUSTER_NAME_LABEL"
value: "cluster-name"
- name: "TARGET_NAMESPACE"
value: "default"
- name: "TEAMS"
@ -60,9 +62,14 @@ spec:
"replica_load_balancer_visible": true,
"resources_visible": true,
"users_visible": true,
"cost_ebs": 0.119,
"cost_core": 0.0575,
"cost_memory": 0.014375,
"postgresql_versions": [
"12",
"11",
"10",
"9.6"
"9.6",
"9.5"
]
}

12
ui/operator_ui/main.py
View File

@ -76,6 +76,7 @@ ACCESS_TOKEN_URL = getenv('ACCESS_TOKEN_URL')
TOKENINFO_URL = getenv('OAUTH2_TOKEN_INFO_URL')
OPERATOR_API_URL = getenv('OPERATOR_API_URL', 'http://postgres-operator')
OPERATOR_CLUSTER_NAME_LABEL = getenv('OPERATOR_CLUSTER_NAME_LABEL', 'cluster-name')
OPERATOR_UI_CONFIG = getenv('OPERATOR_UI_CONFIG', '{}')
OPERATOR_UI_MAINTENANCE_CHECK = getenv('OPERATOR_UI_MAINTENANCE_CHECK', '{}')
READ_ONLY_MODE = getenv('READ_ONLY_MODE', False) in [True, 'true']
@ -84,6 +85,13 @@ SUPERUSER_TEAM = getenv('SUPERUSER_TEAM', 'acid')
TARGET_NAMESPACE = getenv('TARGET_NAMESPACE')
GOOGLE_ANALYTICS = getenv('GOOGLE_ANALYTICS', False)
# storage pricing, i.e. https://aws.amazon.com/ebs/pricing/
COST_EBS = float(getenv('COST_EBS', 0.119)) # GB per month
# compute costs, i.e. https://www.ec2instances.info/?region=eu-central-1&selected=m5.2xlarge
COST_CORE = 30.5 * 24 * float(getenv('COST_CORE', 0.0575)) # Core per hour m5.2xlarge / 8.
COST_MEMORY = 30.5 * 24 * float(getenv('COST_MEMORY', 0.014375)) # Memory GB m5.2xlarge / 32.
WALE_S3_ENDPOINT = getenv(
'WALE_S3_ENDPOINT',
'https+path://s3-eu-central-1.amazonaws.com:443',
@ -293,6 +301,9 @@ DEFAULT_UI_CONFIG = {
'dns_format_string': '{0}.{1}.{2}',
'pgui_link': '',
'static_network_whitelist': {},
'cost_ebs': COST_EBS,
'cost_core': COST_CORE,
'cost_memory': COST_MEMORY
}
@ -1003,6 +1014,7 @@ def main(port, secret_key, debug, clusters: list):
logger.info(f'App URL: {APP_URL}')
logger.info(f'Authorize URL: {AUTHORIZE_URL}')
logger.info(f'Operator API URL: {OPERATOR_API_URL}')
logger.info(f'Operator cluster name label: {OPERATOR_CLUSTER_NAME_LABEL}')
logger.info(f'Readonly mode: {"enabled" if READ_ONLY_MODE else "disabled"}') # noqa
logger.info(f'Spilo S3 backup bucket: {SPILO_S3_BACKUP_BUCKET}')
logger.info(f'Spilo S3 backup prefix: {SPILO_S3_BACKUP_PREFIX}')

6
ui/operator_ui/spiloutils.py
View File

@ -3,7 +3,7 @@ from datetime import datetime, timezone
from furl import furl
from json import dumps
from logging import getLogger
from os import environ
from os import environ, getenv
from requests import Session
from urllib.parse import urljoin
from uuid import UUID
@ -16,6 +16,8 @@ logger = getLogger(__name__)
session = Session()
OPERATOR_CLUSTER_NAME_LABEL = getenv('OPERATOR_CLUSTER_NAME_LABEL', 'cluster-name')
def request(cluster, path, **kwargs):
if 'timeout' not in kwargs:
@ -137,7 +139,7 @@ def read_pods(cluster, namespace, spilo_cluster):
cluster=cluster,
resource_type='pods',
namespace=namespace,
label_selector={'version': spilo_cluster},
label_selector={OPERATOR_CLUSTER_NAME_LABEL: spilo_cluster},
)

12
ui/requirements.txt
View File

@ -1,14 +1,14 @@
Flask-OAuthlib==0.9.5
Flask==1.0.2
backoff==1.5.0
boto3==1.5.14
boto==2.48.0
Flask==1.1.1
backoff==1.8.1
boto3==1.10.4
boto==2.49.0
click==6.7
furl==1.0.1
furl==1.0.2
gevent==1.2.2
jq==0.1.6
json_delta>=2.0
kubernetes==3.0.0
requests==2.20.1
requests==2.22.0
stups-tokens>=1.1.19
wal_e==1.1.0

7
ui/run_local.sh
View File

@ -19,10 +19,15 @@ default_operator_ui_config='{
"nat_gateways_visible": false,
"resources_visible": true,
"users_visible": true,
"cost_ebs": 0.119,
"cost_core": 0.0575,
"cost_memory": 0.014375,
"postgresql_versions": [
"12",
"11",
"10",
"9.6"
"9.6",
"9.5"
],
"static_network_whitelist": {
"localhost": ["172.0.0.1/32"]