![Image](https://img.shields.io/docker/pulls/democraticcsi/democratic-csi.svg)
![Image](https://img.shields.io/github/workflow/status/democratic-csi/democratic-csi/CI?style=flat-square)

# Introduction

## What is Democratic-CSI?  

`Democratic-CSI` implements the `CSI` (Container Storage Interface) specifications providing storage for various container orchestration systems (*ie: Kubernetes, Nomad, OpenShift*).

The current *focus* is providing storage via iSCSI or NFS from ZFS-based storage systems, predominantly `TrueNAS / FreeNAS` and `ZoL on Ubuntu`.  
The current *drivers* implement the depth and breadth of the `csi` specifications, so you have access to resizing, snapshots, clones, etc functionality.

## What can Democratic-CSI offer?

**Several implementations of `CSI` drivers**  
:arrow_forward: `freenas-nfs` (manages zfs datasets to share over nfs)  
:arrow_forward: `freenas-iscsi` (manages zfs zvols to share over iscsi)  
:arrow_forward: `freenas-smb` (manages zfs datasets to share over smb)  
:arrow_forward: `freenas-api-nfs` experimental use with SCALE only (manages zfs datasets to share over nfs)  
:arrow_forward: `freenas-api-iscsi` experimental use with SCALE only (manages zfs zvols to share over iscsi)  
:arrow_forward: `freenas-api-smb` experimental use with SCALE only (manages zfs datasets to share over smb)  
:arrow_forward: `zfs-generic-nfs` (works with any ZoL installation...ie: Ubuntu)  
:arrow_forward: `zfs-generic-iscsi` (works with any ZoL installation...ie: Ubuntu)  
:arrow_forward: `zfs-local-ephemeral-inline` (provisions node-local zfs datasets)  
:arrow_forward: `synology-iscsi` experimental (manages volumes to share over iscsi)  
:arrow_forward: `lustre-client` (crudely provisions storage using a shared lustre share/directory for all volumes)  
:arrow_forward: `nfs-client` (crudely provisions storage using a shared nfs share/directory for all volumes)  
:arrow_forward: `smb-client` (crudely provisions storage using a shared smb share/directory for all volumes)  
:arrow_forward: `node-manual` (allows connecting to manually created smb, nfs, lustre, and iscsi volumes, see sample PVs in the `examples` directory)  

**Development**  
:arrow_forward: Framework for developing `CSI` drivers

If you have any interest in providing a `CSI` driver, simply open an issue to
discuss. The project provides an extensive framework to build and making it
relatively easy to implement new drivers.

## Community Guides

[Using TrueNAS to provide persistent storage for Kubernetes](https://jonathangazeley.com/2021/01/05/using-truenas-to-provide-persistent-storage-for-kubernetes/)  
[Migrating from `NFS-client-provisioner` to `democratic-CSI`](https://gist.github.com/admun/4372899f20421a947b7544e5fc9f9117)  
[Migrating between storage classes using `Velero`](https://gist.github.com/deefdragon/d58a4210622ff64088bd62a5d8a4e8cc)

# Installation

Predominantly 3 prerequisites are needed:

- Nodes preperation (ie: Kubernetes cluster nodes)
- Storage server preperation
- Deployment of the driver into the cluster (`helm` chart provided with sample
  `values.yaml`)

## :wrench: Node preperation

Alright, you have chosen your driver. Let's start by configuring the prerequisites for your Node.  
You can choose to use either **NFS** or **iSCSI** or **both**.

### NFS configuration

___

#### RHEL / CentOS

```bash
sudo yum install -y nfs-utils
```

#### Ubuntu / Debian

```bash
sudo apt-get install -y nfs-common
```

### iSCSI configuration

___

**RHEL / CentOS**  
Install the following system packages:

```bash
sudo yum install -y lsscsi iscsi-initiator-utils sg3_utils device-mapper-multipath
```

Enable multipathing:

```bash
sudo mpathconf --enable --with_multipathd y
```

Ensure that `iscsid` and `multipathd` are running:

```bash
sudo systemctl enable iscsid multipathd && sudo systemctl start iscsid multipathd
```

Start and enable iSCSI:

```bash
sudo systemctl enable iscsi && sudo systemctl start iscsi
```

**Ubuntu / Debian**  
Install the following system packages:

```bash
sudo apt-get install -y open-iscsi lsscsi sg3-utils multipath-tools scsitools
```

#### Multipathing

`Multipath` is supported for the `iSCSI`-based drivers. Simply setup multipath to your liking and set multiple portals in the config as appropriate.  
*NOTE:* If you are running Kubernetes with Rancher/RKE please see the following:  
[Support host iscsi simultaneously with kubelet iscsi (pvc)](https://github.com/rancher/rke/issues/1846>)

Add the mutlipath configuration:

```bash
sudo tee /etc/multipath.conf <<-'EOF'
defaults {
    user_friendly_names yes
    find_multipaths yes
}
EOF
```

Enable the `multipath-tools` service and restart to load the configuration:

```bash
sudo systemctl enable multipath-tools && sudo service multipath-tools restart
```

Ensure that `open-iscsi` and `multipath-tools` are enabled and running:

```bash
sudo systemctl status multipath-tools
sudo systemctl enable open-iscsi.service
sudo service open-iscsi start
sudo systemctl status open-iscsi
```

### FreeNAS-SMB

___
If using with Windows based machines you may need to enable guest access (even
if you are connecting with credentials)

```powershell
Set-ItemProperty HKLM:\SYSTEM\CurrentControlSet\Services\LanmanWorkstation\Parameters AllowInsecureGuestAuth -Value 1 ; Restart-Service LanmanWorkstation -Force
```

### ZFS-local-ephemeral-inline

___
This `driver` provisions node-local ephemeral storage on a per-pod basis. Each node should have an identically named ZFS pool created and avaialble to the `driver`.  
*NOTE:* This is *NOT* the same thing as using the docker ZFS storage
driver (although the same pool could be used). No other requirements are
necessary. More regarding to this can be found here: [Pod Inline Volume Support](https://kubernetes-csi.github.io/docs/ephemeral-local-volumes.html)

## :wrench: Storage server preperation

Storage server preperation depends slightly on which `driver` you are using.
The recommended version of FreeNAS / TrueNAS is 12.0-U2 or higher, however the driver should work
with much older versions as well.

### TrueNAS / FreeNAS (freenas-nfs, freenas-iscsi, freenas-smb, freenas-api-nfs, freenas-api-iscsi, freenas-api-smb)

#### API without SSH

___
Configuration templates can be found [HERE](https://github.com/D1StrX/democratic-csi/blob/667354978e497fb4624d52e909609ca278e4bd25/examples/api-with-ssh)  
The various `freenas-api-*` drivers are currently EXPERIMENTAL and can only be used with SCALE 21.08+. Fundamentally these drivers remove the need for `ssh` connections and do all operations entirely with the TrueNAS api. With that in mind, any `ssh/shell/etc` requirements below can be safely ignored. Also note the following known issues:

- [Additional middleware changes to support Democratic CSI use of native API](https://jira.ixsystems.com/browse/NAS-111870)
- [TrueNAS Scale 21.08 - Could not log into all portals](https://github.com/democratic-csi/democratic-csi/issues/112)
- [Pure api based truenas driver (ssh dependency removed)](https://github.com/democratic-csi/democratic-csi/issues/101)

[Continue configuration](#Service-configuration)

#### API with SSH

___
Configuration templates can be found [HERE](https://github.com/D1StrX/democratic-csi/blob/667354978e497fb4624d52e909609ca278e4bd25/examples/api-with-ssh)

[Continue configuration](#Service-configuration)

### Service configuration  

Ensure the following services are *configured*, *running* and starting automatically:  

#### SSH configuration

___

- When creating a custom user (e.g., `CSI`):
  - Ensure `ZSH`, `BASH`, or `SH` is set as `shell`, `CSH` gives false errors due to quoting (also applicable when using `root`)  
  &emsp;![image](https://user-images.githubusercontent.com/40062371/147365044-007b2657-30f9-428b-ae12-7622a572866d.png)
  - Ensure that user has passwordless `sudo` privileges:  
    *NOTE:* This could get reset by FreeNAS if you alter the user via the GUI later
    - On TrueNAS CORE 12.0-u3 or higher, open the Shell:  

      ```bash
      cli
      ```

      After you enter the truenas cli and are at that prompt:

      ```bash
      account user query select=id,username,uid,sudo_nopasswd
      ```

      find the `id` of the user you want to update (note, this is distinct from the `uid`)

      ```bash
      account user update id=<id> sudo=true
      ```

      ```bash
      account user update id=<id> sudo_nopasswd=true
      ```

      (Optional) If you want to enable passwordless authentication via CLI:

      ```bash
      account user update id=<id> password_disabled=true
      ```

      Exit the CLI by pressing `ctrl-d`

    - On other versions add the user to the sudoers file:  

      ```bash
      visudo
      ```

      ```bash
      <username> ALL=(ALL) NOPASSWD:ALL
      ```

      Confirm sudoers file is appropriate:

      ```bash
      cat /usr/local/etc/sudoers
      ```

  - `CSI` has a homefolder, this is used to store its SSH Public Key
  &emsp;![image](https://user-images.githubusercontent.com/40062371/147370105-6030b22e-ceb3-4768-b4a0-8e55fafe7f0f.png)
  - Add the user to `wheel` or create/use a group that will be used for permissions later on

#### **NFS configuration**  

___

- Bind the interface to the NFS service
- It is recommended to use NFS 3

#### **iSCSI configuration**  

___
*NOTE:* (Fixed in 12.0-U2+) when using the FreeNAS API concurrently, the `/etc/ctl.conf` file on the server can become invalid, some sample scripts are provided in the `contrib` directory to clean things up ie:  
Copy the script to the server and directly and run - `./ctld-config-watchdog-db.sh | logger -t ctld-config-watchdog-db.sh &`  
Please read the scripts and set the variables appropriate for your server.

- Ensure you have pre*emptively created portals, initatior groups, auths
  - Make note of the respective IDs (the true ID may not reflect what is
    visible in the UI)
  - IDs can be visible by clicking the the `Edit` link and finding the ID in the
    browser address bar
  - Optionally you may use the following to retrieve appropiate IDs:
    - `curl --header "Accept: application/json" --user root:<password> 'http(s)://<ip>/api/v2.0/iscsi/portal'`
    - `curl --header "Accept: application/json" --user root:<password> 'http(s)://<ip>/api/v2.0/iscsi/initiator'`
    - `curl --header "Accept: application/json" --user root:<password> 'http(s)://<ip>/api/v2.0/iscsi/auth'`

### **SMB configuration**  

___

- Bind the interface to the SMB service

### **YAML Values configuration**

___
Instruct `Democratic-CSI` to use `sudo` by uncommenting the following in your configuration template:

  ```yaml
  zfs:
    cli:
      sudoEnabled: true
  ```

Starting with TrueNAS CORE 12 it is also possible to use an `apiKey` instead of
the user/root password for the HTTP connection.
The `apiKey` can be generated by clicking on the `Settings icon` -> `API Keys` -> `ADD`  
![image](https://user-images.githubusercontent.com/40062371/147371451-ff712de3-cce0-448e-b59f-29269179d2d6.png)

Issues to review:  
[ixsystems NAS-108519](https://jira.ixsystems.com/browse/NAS-108519)  
[ixsystems NAS-108520](https://jira.ixsystems.com/browse/NAS-108520)  
[ixsystems NAS-108521](https://jira.ixsystems.com/browse/NAS-108521)  
[ixsystems NAS-108522](https://jira.ixsystems.com/browse/NAS-108522)  
[ixsystems NAS-107219](https://jira.ixsystems.com/browse/NAS-107219)  

### **ZoL (zfs-generic-nfs, zfs-generic-iscsi)**

___

Ensure ssh and zfs is installed on the nfs/iscsi server and that you have installed
`targetcli`.

 ```bash
 sudo yum install targetcli -y
 ```

 ```bash
 sudo apt-get -y install targetcli-fb
 ```

### **Synology (synology-iscsi)**

___
Ensure iSCSI Manager has been installed and is generally setup/configured.

## **Helm Installation**

___
Copy proper example Values file from the examples:  
[API without SSH](https://github.com/D1StrX/democratic-csi/blob/667354978e497fb4624d52e909609ca278e4bd25/examples/api-without-ssh)  
[API with SSH](https://github.com/D1StrX/democratic-csi/blob/667354978e497fb4624d52e909609ca278e4bd25/examples/api-with-ssh)  

Add the `Democratic-CSI` Helm repository:

```bash
helm search repo democratic-csi/
```

Update your Helm repository to get latest charts:

```bash
helm repo update
```

### **Helm V3**

___

Install `Democratic-CSI` with your configured values. Helm V3 requires that you `--create-namespace`

```bash
helm install zfs-nfs democratic-csi/democratic-csi --values truenas-isci.yaml --create-namespace democratic-csi
```

Update/Upgrade Values:

```bash
helm upgrade <name> democratic-csi/democratic-csi --values <freenas-*>.yaml --namespace <namespace>
```

### **Helm V2**

___
Install `Democratic-CSI` with your configured values.

```bash
helm upgrade \
--install \
--values freenas-nfs.yaml \
--namespace democratic-csi \
zfs-nfs democratic-csi/democratic-csi
```

### **On non standard Kubelet paths**

Some distrobutions, such as `minikube` and `microk8s` use a non-standard kubelet path. In such cases it is  ecessary to provide a new kubelet host path, microk8s example below:

```bash
microk8s helm upgrade \
  --install \
  --values freenas-nfs.yaml \
  --set node.kubeletHostPath="/var/snap/microk8s/common/var/lib/kubelet"  \
  --namespace democratic-csi \
  zfs-nfs democratic-csi/democratic-csi
```

- microk8s - `/var/snap/microk8s/common/var/lib/kubelet`
- pivotal - `/var/vcap/data/kubelet`

### **OpenShift**

`Democratic-CSI` generally works fine with openshift. Some special parameters
need to be set with helm (support added in chart version `0.6.1`):

```bash
# for sure required
--set node.rbac.openshift.privileged=true
--set node.driver.localtimeHostPath=false

# unlikely, but in special circumstances may be required
--set controller.rbac.openshift.privileged=true
```

### **Nomad**

`Democratic-CSI` works with Nomad in a functioning but limted capacity. See the [Nomad docs](docs/nomad.md) for details.

## **Multiple Deployments**

You may install multiple deployments of each/any driver. It requires the following:

- Use a new helm release name for each deployment
- Make sure you have a unique `csiDriver.name` in the values file
- Use unqiue names for your storage classes (per cluster)
- Use a unique parent dataset (ie: don't try to use the same parent across deployments or clusters)

## **Snapshot Support**  

[**External-snapshotter CRD**](https://github.com/kubernetes-csi/external-snapshotter/tree/master/client/config/crd)  
Install beta (v1.17+) CRDs (one per cluster):

```bash
kubectl apply -f snapshot.storage.k8s.io_volumesnapshotclasses.yaml
```

```bash
kubectl apply -f snapshot.storage.k8s.io_volumesnapshotcontents.yaml
```

```bash
kubectl apply -f snapshot.storage.k8s.io_volumesnapshots.yaml
```

[**External-snapshotter Controller**](https://github.com/kubernetes-csi/external-snapshotter/tree/master/deploy/kubernetes/snapshot-controller)  
Install snapshot controller (once per cluster):  
Add `--namespace` references to your need.

```bash
kubectl apply -f rbac-snapshot-controller.yaml
```

```bash
kubectl apply -f setup-snapshot-controller.yaml
```

Install `Democratic-CSI` with `volumeSnapshotClasses` appropriatly defined.

[Volume-snapshots](https://kubernetes.io/docs/concepts/storage/volume-snapshots/)  
[External-snapshotter Usage](https://github.com/kubernetes-csi/external-snapshotter#usage)  
[Democratic-CSI Issue](https://github.com/democratic-csi/democratic-csi/issues/129#issuecomment-961489810)

## Migrating from freenas-provisioner and freenas-iscsi-provisioner

It is possible to migrate all volumes from the non-csi freenas provisioners
to `Democratic-CSI`.

Copy the `contrib/freenas-provisioner-to-democratic-csi.sh` script from the project to your workstation. Read the script in detail, and edit the variables
to your needs to start migrating!

# **Sponsors**

A special shout out to the wonderful sponsors of this project!

[![ixSystems](https://www.ixsystems.com/wp-content/uploads/2021/06/ix_logo_200x47.png "ixSystems")](http://ixsystems.com/)

## **Related**

[Freenas provisioner](https://github.com/nmaupu/freenas-provisioner)  
[Freenas iSCSI provisioner](https://github.com/travisghansen/freenas-iscsi-provisioner)  
[Welcome TrueNAS Core container storageprovider](https://datamattsson.tumblr.com/post/624751011659202560/welcome-truenas-core-container-storage-provider)
[Truenas CSI](https://github.com/dravanet/truenas-csi)  
[Synology CSI](https://github.com/SynologyOpenSource/synology-csi)