create CDC event stream CRD (#1570)

* provide event stream API
* check manifest settings for logical decoding before creating streams
* operator updates Postgres config and creates replication user
* name FES like the Postgres cluster
* add delete case and fix updating streams + update unit test
* check if fes CRD exists before syncing
* existing slot must use the same plugin
* make id and payload columns configurable
* sync streams only when they are defined in manifest
* introduce applicationId for separate stream CRDs
* add FES to RBAC in chart
* disable streams in chart
* switch to pgoutput plugin and let operator create publications
* reflect code review and additional refactoring

Co-authored-by: Paŭlo Ebermann <paul.ebermann@zalando.de>
This commit is contained in:
Felix Kunde
2022-02-28 10:09:42 +01:00
committed by GitHub
co-authored by Paŭlo Ebermann
parent 8b404fd049
commit d8a159ef1a
42 changed files with 2330 additions and 31 deletions
+20 -1
View File
@@ -3,6 +3,7 @@ kind: postgresql
metadata:
name: acid-test-cluster
# labels:
# application: test-app
# environment: demo
# annotations:
# "acid.zalan.do/controller": "second-operator"
@@ -17,7 +18,7 @@ spec:
- superuser
- createdb
foo_user: []
# flyway: []
# flyway: []
# usersWithSecretRotation:
# - foo_user
# usersWithInPlaceSecretRotation:
@@ -203,3 +204,21 @@ spec:
# operator: In
# values:
# - enabled
# Enables change data capture streams for defined database tables
# streams:
# - applicationId: test-app
# database: foo
# tables:
# data.state_pending_outbox:
# eventType: test-app.status-pending
# data.state_approved_outbox:
# eventType: test-app.status-approved
# data.orders_outbox:
# eventType: test-app.order-completed
# idColumn: o_id
# payloadColumn: o_payload
# # Optional. Filter ignores events before a certain txnId and lsn. Can be used to skip bad events
# filter:
# data.orders_outbox: "[?(@.source.txId > 500 && @.source.lsn > 123456)]"
# batchSize: 1000