From cf51e5c0baaedd182a22218fe9c022d40d84d44e Mon Sep 17 00:00:00 2001 From: Danil Uzlov <36223296+d-uzlov@users.noreply.github.com> Date: Wed, 26 Mar 2025 10:42:22 +0000 Subject: [PATCH] add user documentation for proxy --- docs/proxy-driver.md | 197 +++++++++++++++++++ examples/proxy.yaml | 20 ++ src/driver/controller-proxy/compatibility.md | 3 + 3 files changed, 220 insertions(+) create mode 100644 docs/proxy-driver.md create mode 100644 examples/proxy.yaml diff --git a/docs/proxy-driver.md b/docs/proxy-driver.md new file mode 100644 index 0000000..70a173d --- /dev/null +++ b/docs/proxy-driver.md @@ -0,0 +1,197 @@ + +# Proxy driver + +Proxy driver allow you to run several drivers in one instance of democratic-csi. + +Usually democratic-csi requires you to run a separate instance for each storage class. +This is heavily implied by the CSI protocol. + +However, it is still _possible_ to have several storage classes work via a single CSI driver. +The `proxy` driver makes the best effort to allow dynamic storage classes. + +This file has user documentation for proxy driver. Technical details that are relevant only for development can be found [here](../src/driver/controller-proxy/compatibility.md). + +## Terminology + +"Proxy" is the driver created via `driver: proxy` in the main config. +Other drivers are referred to as "real driver" or "underlying driver". + +"Connection" is equivalent to a democratic-csi deployment without a proxy. +Each connection has a separate `.yaml` config file. +Each real driver is associated with a single connection name. + +## Compatibility + +Drivers that are tested and work without issues: + +- `freenas-nfs` +- `freenas-iscsi` +- `freenas-api-nfs` +- `freenas-api-iscsi` +- `zfs-generic-nfs` +- `zfs-generic-iscsi` +- `zfs-generic-nvmeof` +- `local-hostpath` + +Drivers that are not tested but should work fine: + +- `freenas-smb` +- `freenas-api-smb` +- `zfs-generic-smb` +- `synology-iscsi` +- `nfs-client` +- `smb-client` +- `lustre-client` +- `node-manual` +- `zfs-local-dataset` +- `zfs-local-zvol` + +Drivers that are known to be incompatible with proxy: + +- `objectivefs` +- `zfs-local-ephemeral-inline` + +All `local` drivers need `proxy.nodeTopology.type == node` to work properly. + +## Config layout + +Like all other drivers, proxy driver needs the main config file. See [proxy.yaml example](../examples/proxy.yaml). +Additionally, proxy needs config files for other drivers to be in the container filesystem. + +Initially proxy doesn't know what real drivers to use. +In Kubernetes you configure real drivers via `parameters.connection` field in a Storage Class. +In other Container Orchestrators look for equivalent settings. + +In the example `proxy.configFolder` value is `/mnt/connections/`. +This means that proxy will look for real driver config files in this folder. + +Config file must have the following name: `.yaml`. + +Connection names are arbitrary, you just need to make sure that name of the config file +matches connection name from the storage class. + +Connection configuration can be added and updated dynamically, +as long as you make sure that files mounted into democratic-csi driver container are updated. + +## Limitations + +Proxy driver has a few limitations. + +Since it provides a common interface for several drivers, these drivers need to be similar enough. +For example, you can't mix `local-hostpath` and `freenas-nfs` drivers, because they simply need different modes of deployment. +Generally, you can mix drivers if it's possible to switch between them by just changing config file. + +Another limitation is connection name length. +Connection name SHOULD be short. It is added as a prefix into Volume Handle value. Volume handles have limited maximum length. +If your volume handle is already very long, adding connection name to it may cause volume creation to fail. +Whether or not this is relevant at all for you depends on your real driver config. +For example, snapshot handles are usually very long. Volume handle length is usually fixed, unless you customized it. + +You will probably be fine when using connection name under 20 symbols. +You can probably use longer names, but the shorter the better. + +Another limitation is that connection name is saved in volumes. +Connection name MUST be immutable. +If you create a volume, and then change connection name, you will get errors at different operations. +You may still be able to mount this volume, but volume size expansion or volume deletion will not work anymore. +It would be analogous to deleting the whole democratic-csi deployment when not using proxy driver. + +If you want to change connection name, you need to add a new config file for new connection, +and create a new storage class that will use this new connection. +You can then delete all volumes that are using the old connection, and only then you can delete the old connection config. + +## Simple k8s example + +Imagine that currently you have several deployments of democratic-csi +with different configs that you want to merge into a single deployment. + +First set democratic-csi [config values for proxy](../examples/proxy.yaml) (here is a minimalistic example): + +```yaml +driver: proxy +proxy: + configFolder: /mnt/connections/ +``` + +Then adjust your helm values for democratic-csi deployment: + +- (optional) delete all built-in storage classes +- (required) add extra volumes to the controller deployment + +```yaml +csiDriver: + name: org.democratic-csi.proxy +controller: + extraVolumes: + - name: connections + secret: + secretName: connections + driver: + extraVolumeMounts: + - name: connections + mountPath: /mnt/connections +``` + +Then create a secret for config files that will contain real driver config later: + +```bash +# don't forget to adjust namespace +kubectl create secret generic connections +``` + +Then you can deploy democratic-csi with proxy driver. +Now you should have an empty deployment that works +but can't create any real drivers or connect to any real backend. + +Let's add connections to proxy: +you need to update the `connections` secret. +Let's say that you need 2 storage classes: `nfs` and `iscsi`. +You need to have 2 separate config files for them. + +```bash +# don't forget to adjust namespace +kubectl create secret generic connections \ + --from-file example-nfs.yaml=./path/to/nfs/config.yaml \ + --from-file example-iscsi.yaml=./path/to/iscsi/config.yaml \ + -o yaml --dry-run=client | kl apply -f - +``` + +As you can see, you don't need to restart democratic-csi to add or update connections. +If you change the `connections` secret too quickly, then you may need to wait a few seconds +for files to get remounted info filesystem, but no restart is needed. + +Then create corresponding storage classes: + +```yaml +--- +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: nfs +provisioner: org.democratic-csi.proxy +reclaimPolicy: Delete +allowVolumeExpansion: true +parameters: + connection: example-nfs + fsType: nfs +--- +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: iscsi +provisioner: org.democratic-csi.proxy +reclaimPolicy: Delete +allowVolumeExpansion: true +parameters: + connection: example-iscsi + fsType: ext4 +``` + +Now you should be able to create volumes using these 2 storage classes. + +Notice that storage class name does not match connection name. +Also, local file names don't match connection name. +This is done to make example easier to understand. + +In real deployment you'll probably want to keep local file names, connection names, +and mounted file names synced for easier management. diff --git a/examples/proxy.yaml b/examples/proxy.yaml new file mode 100644 index 0000000..1084b13 --- /dev/null +++ b/examples/proxy.yaml @@ -0,0 +1,20 @@ + +driver: proxy + +proxy: + # path to folder where real driver config files are located + # this value is required for the proxy to work + configFolder: /mnt/connections/ + # connections that are no longer used are eventually deleted + # when timeout is 0, caching will be completely disabled, each request will create a new driver + # when timeout is -1, cache timeout is disabled, drivers are cached forever + # default: 60 + cacheTimeoutMinutes: 60 + # Node topology defines isolated groups of nodes + # local drivers need node topology + # All other drivers do not depend on topology, and will work fine with simpler cluster topology + nodeTopology: + # 'cluster': the whole cluster has unified storage + # 'node': each node has its own storage. Required for 'local-' drivers + # default: cluster + type: cluster diff --git a/src/driver/controller-proxy/compatibility.md b/src/driver/controller-proxy/compatibility.md index 6e29227..51756f2 100644 --- a/src/driver/controller-proxy/compatibility.md +++ b/src/driver/controller-proxy/compatibility.md @@ -11,6 +11,9 @@ which isn't ideal for small deployments, and is incompatible with democratic-csi A great discussion of difficulties per storage class state can be found here: - https://github.com/container-storage-interface/spec/issues/370 +This file contains implementation details relevant for development. +If you are searching for user documentation and deployment example go [here](../../../docs/proxy-driver.md). + ## Terminology and structure "Proxy" is the driver created via `driver: proxy` in the main config.