160 lines
5.2 KiB
Markdown
160 lines
5.2 KiB
Markdown
# Introduction
|
|
|
|
`democratic-csi` implements the `csi` (container storage interface) spec
|
|
providing storage for various container orchestration systems (ie: Kubernetes).
|
|
|
|
The current focus is providing storage via iscsi/nfs from zfs-based storage
|
|
systems, predominantly `FreeNAS / TrueNAS` and `ZoL` on `Ubuntu`.
|
|
|
|
The current drivers implement the depth and breadth of the `csi` spec, so you
|
|
have access to resizing, snapshots, clones, etc functionality.
|
|
|
|
`democratic-csi` is 2 things:
|
|
|
|
- several implementations of `csi` drivers
|
|
- `freenas-nfs` (manages zfs datasets to share over nfs)
|
|
- `freenas-iscsi` (manages zfs zvols to share over iscsi)
|
|
- `freenas-smb` (manages zfs datasets to share over smb)
|
|
- `zfs-generic-nfs` (works with any ZoL installation...ie: Ubuntu)
|
|
- `zfs-generic-iscsi` (works with any ZoL installation...ie: Ubuntu)
|
|
- `zfs-local-ephemeral-inline` (provisions node-local zfs datasets)
|
|
- `nfs-client` (crudely provisions storage using a shared nfs share/directory for all volumes)
|
|
- framework for developing `csi` drivers
|
|
|
|
If you have any interest in providing a `csi` driver, simply open an issue to
|
|
discuss. The project provides an extensive framework to build from making it
|
|
relatively easy to implement new drivers.
|
|
|
|
# Installation
|
|
|
|
Predominantly 3 things are needed:
|
|
|
|
- node prep
|
|
- server prep
|
|
- deploy the driver into the cluster (`helm` chart provided with sample
|
|
`values.yaml`)
|
|
|
|
## Node Prep
|
|
|
|
You should install/configure the requirements for both nfs and iscsi.
|
|
|
|
Follow the instructions here: https://netapp-trident.readthedocs.io/en/stable-v20.04/kubernetes/operations/tasks/worker.html
|
|
|
|
If you are running Kubernetes with rancher/rke please see the following:
|
|
|
|
- https://github.com/rancher/rke/issues/1846
|
|
|
|
### freenas-smb
|
|
|
|
If using with Windows based machines you may need to enable guest access (even
|
|
if you are connecting with credentiasl)
|
|
|
|
```
|
|
Set-ItemProperty HKLM:\SYSTEM\CurrentControlSet\Services\LanmanWorkstation\Parameters AllowInsecureGuestAuth -Value 1
|
|
Restart-Service LanmanWorkstation -Force
|
|
```
|
|
|
|
### zfs-local-ephemeral-inline
|
|
|
|
This `driver` provisions node-local ephemeral storage on a per-pod basis. Each
|
|
node should have an identically named zfs pool created and avaialble to the
|
|
`driver`. Note, this is _NOT_ the same thing as using the docker zfs storage
|
|
driver (although the same pool could be used). No other requirements are
|
|
necessary.
|
|
|
|
- https://github.com/kubernetes/enhancements/blob/master/keps/sig-storage/20190122-csi-inline-volumes.md
|
|
- https://kubernetes-csi.github.io/docs/ephemeral-local-volumes.html
|
|
|
|
## Server Prep
|
|
|
|
Server preparation depends slightly on which `driver` you are using.
|
|
|
|
### FreeNAS (freenas-nfs, freenas-iscsi, freenas-smb)
|
|
|
|
Ensure the following services are configurged and running:
|
|
|
|
- ssh (if you use a password for authentication make sure it is allowed)
|
|
- ensure `zsh`, `bash`, or `sh` is set as the root shell, `csh` gives false errors due to quoting
|
|
- nfs
|
|
- iscsi
|
|
- smb
|
|
|
|
### ZoL (zfs-generic-nfs, zfs-generic-iscsi)
|
|
|
|
Ensure ssh and zfs is installed on the server and that you have installed
|
|
`targetcli`.
|
|
|
|
- `yum install targetcli -y`
|
|
- `apt-get -y install targetcli-fb`
|
|
|
|
## Helm Installation
|
|
|
|
```
|
|
helm repo add democratic-csi https://democratic-csi.github.io/charts/
|
|
helm repo update
|
|
# helm v2
|
|
helm search democratic-csi/
|
|
|
|
# helm v3
|
|
helm search repo democratic-csi/
|
|
|
|
# copy proper values file from https://github.com/democratic-csi/charts/tree/master/stable/democratic-csi/examples
|
|
# edit as appropriate
|
|
# examples are from helm v2, alter as appropriate for v3
|
|
|
|
# add --create-namespace for helm v3
|
|
helm upgrade \
|
|
--install \
|
|
--values freenas-iscsi.yaml \
|
|
--namespace democratic-csi \
|
|
zfs-iscsi democratic-csi/democratic-csi
|
|
|
|
helm upgrade \
|
|
--install \
|
|
--values freenas-nfs.yaml \
|
|
--namespace democratic-csi \
|
|
zfs-nfs democratic-csi/democratic-csi
|
|
```
|
|
|
|
## Multiple Deployments
|
|
|
|
You may install multiple deployments of each/any driver. It requires the following:
|
|
|
|
- Use a new helm release name for each deployment
|
|
- Make sure you have a unique `csiDriver.name` in the values file
|
|
- Use unqiue names for your storage classes (per cluster)
|
|
- Use a unique parent dataset (ie: don't try to use the same parent across deployments or clusters)
|
|
|
|
# Snapshot Support
|
|
|
|
Install beta (v1.17+) CRDs (once per cluster):
|
|
|
|
- https://github.com/kubernetes-csi/external-snapshotter/tree/master/client/config/crd
|
|
|
|
```
|
|
kubectl apply -f snapshot.storage.k8s.io_volumesnapshotclasses.yaml
|
|
kubectl apply -f snapshot.storage.k8s.io_volumesnapshotcontents.yaml
|
|
kubectl apply -f snapshot.storage.k8s.io_volumesnapshots.yaml
|
|
```
|
|
|
|
Install snapshot controller (once per cluster):
|
|
|
|
- https://github.com/kubernetes-csi/external-snapshotter/tree/master/deploy/kubernetes/snapshot-controller
|
|
|
|
```
|
|
# replace namespace references to your liking
|
|
kubectl apply -f rbac-snapshot-controller.yaml
|
|
kubectl apply -f setup-snapshot-controller.yaml
|
|
```
|
|
|
|
Install `democratic-csi` as usual with `volumeSnapshotClasses` defined as appropriate.
|
|
|
|
- https://kubernetes.io/docs/concepts/storage/volume-snapshots/
|
|
- https://github.com/kubernetes-csi/external-snapshotter#usage
|
|
|
|
# Related
|
|
|
|
- https://github.com/nmaupu/freenas-provisioner
|
|
- https://github.com/travisghansen/freenas-iscsi-provisioner
|
|
- https://datamattsson.tumblr.com/post/624751011659202560/welcome-truenas-core-container-storage-provider
|