public_git
/
democratic-csi
mirror of https://github.com/democratic-csi/democratic-csi.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

markdown fixes

This commit is contained in:
D1StrX 2022-01-12 21:12:06 +01:00
parent a3ab293d60
commit a458636a8f
1 changed files with 121 additions and 98 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

219
README.md
View File

@ -10,7 +10,7 @@
The current *focus* is providing storage via iSCSI or NFS from ZFS-based storage systems, predominantly `TrueNAS / FreeNAS` and `ZoL on Ubuntu`. The current *focus* is providing storage via iSCSI or NFS from ZFS-based storage systems, predominantly `TrueNAS / FreeNAS` and `ZoL on Ubuntu`.
The current *drivers* implement the depth and breadth of the `csi` specifications, so you have access to resizing, snapshots, clones, etc functionality. The current *drivers* implement the depth and breadth of the `csi` specifications, so you have access to resizing, snapshots, clones, etc functionality.
## What can Democratic-CSI offer? ## What can Democratic-CSI offer?
**Several implementations of `CSI` drivers** **Several implementations of `CSI` drivers**
:arrow_forward: `freenas-nfs` (manages zfs datasets to share over nfs) :arrow_forward: `freenas-nfs` (manages zfs datasets to share over nfs)
@ -43,18 +43,19 @@ relatively easy to implement new drivers.
# Installation # Installation
Predominantly 3 prerequisites are needed: Predominantly 3 prerequisites are needed:
- Nodes preperation (ie: Kubernetes cluster nodes) - Nodes preperation (ie: Kubernetes cluster nodes)
- Storage server preperation - Storage server preperation
- Deployment of the driver into the cluster (`helm` chart provided with sample - Deployment of the driver into the cluster (`helm` chart provided with sample
`values.yaml`) `values.yaml`)
## Node preperation ## :wrench: Node preperation
Alright, you have chosen your driver. Let's start by configuring the prerequisites for your Node. Alright, you have chosen your driver. Let's start by configuring the prerequisites for your Node.
You can choose to use either **NFS** or **iSCSI** or **both**. You can choose to use either **NFS** or **iSCSI** or **both**.
### **NFS configuration** ### NFS configuration
___ ___
@ -74,7 +75,7 @@ sudo apt-get install -y nfs-common
___ ___
#### RHEL / CentOS **RHEL / CentOS**
Install the following system packages: Install the following system packages:
```bash ```bash
@ -106,14 +107,15 @@ Install the following system packages:
sudo apt-get install -y open-iscsi lsscsi sg3-utils multipath-tools scsitools sudo apt-get install -y open-iscsi lsscsi sg3-utils multipath-tools scsitools
``` ```
**Multipathing** #### Multipathing
`Multipath` is supported for the `iSCSI`-based drivers. Simply setup multipath to your liking and set multiple portals in the config as appropriate. `Multipath` is supported for the `iSCSI`-based drivers. Simply setup multipath to your liking and set multiple portals in the config as appropriate.
*NOTE:* If you are running Kubernetes with Rancher/RKE please see the following: *NOTE:* If you are running Kubernetes with Rancher/RKE please see the following:
[Support host iscsi simultaneously with kubelet iscsi (pvc)](https://github.com/rancher/rke/issues/1846>) [Support host iscsi simultaneously with kubelet iscsi (pvc)](https://github.com/rancher/rke/issues/1846>)
<br/>
Add the mutlipath configuration: Add the mutlipath configuration:
```
```bash
sudo tee /etc/multipath.conf <<-'EOF' sudo tee /etc/multipath.conf <<-'EOF'
defaults { defaults {
user_friendly_names yes user_friendly_names yes
@ -121,151 +123,166 @@ defaults {
} }
EOF EOF
``` ```
Enable the `multipath-tools` service and restart to load the configuration: Enable the `multipath-tools` service and restart to load the configuration:
```
```bash
sudo systemctl enable multipath-tools && sudo service multipath-tools restart sudo systemctl enable multipath-tools && sudo service multipath-tools restart
``` ```
Ensure that `open-iscsi` and `multipath-tools` are enabled and running: Ensure that `open-iscsi` and `multipath-tools` are enabled and running:
```
```bash
sudo systemctl status multipath-tools sudo systemctl status multipath-tools
sudo systemctl enable open-iscsi.service sudo systemctl enable open-iscsi.service
sudo service open-iscsi start sudo service open-iscsi start
sudo systemctl status open-iscsi sudo systemctl status open-iscsi
``` ```
<br/>
### **FreeNAS-SMB** </span> ### FreeNAS-SMB
___ ___
If using with Windows based machines you may need to enable guest access (even If using with Windows based machines you may need to enable guest access (even
if you are connecting with credentials) if you are connecting with credentials)
``` ```powershell
Set-ItemProperty HKLM:\SYSTEM\CurrentControlSet\Services\LanmanWorkstation\Parameters AllowInsecureGuestAuth -Value 1 ; Restart-Service LanmanWorkstation -Force Set-ItemProperty HKLM:\SYSTEM\CurrentControlSet\Services\LanmanWorkstation\Parameters AllowInsecureGuestAuth -Value 1 ; Restart-Service LanmanWorkstation -Force
``` ```
<br/>
### **ZFS-local-ephemeral-inline** ### ZFS-local-ephemeral-inline
___ ___
This `driver` provisions node-local ephemeral storage on a per-pod basis. Each node should have an identically named ZFS pool created and avaialble to the `driver`. This `driver` provisions node-local ephemeral storage on a per-pod basis. Each node should have an identically named ZFS pool created and avaialble to the `driver`.
*NOTE:* This is *NOT* the same thing as using the docker ZFS storage *NOTE:* This is *NOT* the same thing as using the docker ZFS storage
driver (although the same pool could be used). No other requirements are driver (although the same pool could be used). No other requirements are
necessary. More regarding to this can be found here: [Pod Inline Volume Support](https://kubernetes-csi.github.io/docs/ephemeral-local-volumes.html) necessary. More regarding to this can be found here: [Pod Inline Volume Support](https://kubernetes-csi.github.io/docs/ephemeral-local-volumes.html)
<br/> ## :wrench: Storage server preperation
## **Storage server preperation**
Storage server preperation depends slightly on which `driver` you are using. Storage server preperation depends slightly on which `driver` you are using.
The recommended version of FreeNAS / TrueNAS is 12.0-U2 or higher, however the driver should work The recommended version of FreeNAS / TrueNAS is 12.0-U2 or higher, however the driver should work
with much older versions as well. with much older versions as well.
### **TrueNAS / FreeNAS (freenas-nfs, freenas-iscsi, freenas-smb, freenas-api-nfs, freenas-api-iscsi, freenas-api-smb)** ### TrueNAS / FreeNAS (freenas-nfs, freenas-iscsi, freenas-smb, freenas-api-nfs, freenas-api-iscsi, freenas-api-smb)
<br/>
#### API without SSH
**API without SSH**
___ ___
Configuration templates can be found [HERE](https://github.com/D1StrX/democratic-csi/blob/667354978e497fb4624d52e909609ca278e4bd25/examples/api-with-ssh) Configuration templates can be found [HERE](https://github.com/D1StrX/democratic-csi/blob/667354978e497fb4624d52e909609ca278e4bd25/examples/api-with-ssh)
The various `freenas-api-*` drivers are currently EXPERIMENTAL and can only be used with SCALE 21.08+. Fundamentally these drivers remove the need for `ssh` connections and do all operations entirely with the TrueNAS api. With that in mind, any `ssh/shell/etc` requirements below can be safely ignored. Also note the following known issues: The various `freenas-api-*` drivers are currently EXPERIMENTAL and can only be used with SCALE 21.08+. Fundamentally these drivers remove the need for `ssh` connections and do all operations entirely with the TrueNAS api. With that in mind, any `ssh/shell/etc` requirements below can be safely ignored. Also note the following known issues:
* [Additional middleware changes to support Democratic CSI use of native API](https://jira.ixsystems.com/browse/NAS-111870) - [Additional middleware changes to support Democratic CSI use of native API](https://jira.ixsystems.com/browse/NAS-111870)
* [TrueNAS Scale 21.08 - Could not log into all portals](https://github.com/democratic-csi/democratic-csi/issues/112) - [TrueNAS Scale 21.08 - Could not log into all portals](https://github.com/democratic-csi/democratic-csi/issues/112)
* [Pure api based truenas driver (ssh dependency removed)](https://github.com/democratic-csi/democratic-csi/issues/101) - [Pure api based truenas driver (ssh dependency removed)](https://github.com/democratic-csi/democratic-csi/issues/101)
[Continue configuration](#Service-configuration) [Continue configuration](#Service-configuration)
<br/>
**API with SSH** #### API with SSH
___ ___
Configuration templates can be found [HERE](https://github.com/D1StrX/democratic-csi/blob/667354978e497fb4624d52e909609ca278e4bd25/examples/api-with-ssh) Configuration templates can be found [HERE](https://github.com/D1StrX/democratic-csi/blob/667354978e497fb4624d52e909609ca278e4bd25/examples/api-with-ssh)
[Continue configuration](#Service-configuration) [Continue configuration](#Service-configuration)
<br/>
### **Service configuration** ### Service configuration
Ensure the following services are *configured*, *running* and starting automatically: Ensure the following services are *configured*, *running* and starting automatically:
#### **SSH configuration** #### SSH configuration
___ ___
* When creating a custom user (e.g., `CSI`):
* Ensure `ZSH`, `BASH`, or `SH` is set as `shell`, `CSH` gives false errors due to quoting (also applicable when using `root`) - When creating a custom user (e.g., `CSI`):
- Ensure `ZSH`, `BASH`, or `SH` is set as `shell`, `CSH` gives false errors due to quoting (also applicable when using `root`)
&emsp;![image](https://user-images.githubusercontent.com/40062371/147365044-007b2657-30f9-428b-ae12-7622a572866d.png) &emsp;![image](https://user-images.githubusercontent.com/40062371/147365044-007b2657-30f9-428b-ae12-7622a572866d.png)
* Ensure that user has passwordless `sudo` privileges: - Ensure that user has passwordless `sudo` privileges:
*NOTE:* This could get reset by FreeNAS if you alter the user via the GUI later *NOTE:* This could get reset by FreeNAS if you alter the user via the GUI later
* On TrueNAS CORE 12.0-u3 or higher, open the Shell: - On TrueNAS CORE 12.0-u3 or higher, open the Shell:
```
```bash
cli cli
``` ```
After you enter the truenas cli and are at that prompt: After you enter the truenas cli and are at that prompt:
```
```bash
account user query select=id,username,uid,sudo_nopasswd account user query select=id,username,uid,sudo_nopasswd
``` ```
find the `id` of the user you want to update (note, this is distinct from the `uid`) find the `id` of the user you want to update (note, this is distinct from the `uid`)
``` ```bash
account user update id=<id> sudo=true account user update id=<id> sudo=true
``` ```
```
```bash
account user update id=<id> sudo_nopasswd=true account user update id=<id> sudo_nopasswd=true
``` ```
(Optional) If you want to enable passwordless authentication via CLI: (Optional) If you want to enable passwordless authentication via CLI:
```
```bash
account user update id=<id> password_disabled=true account user update id=<id> password_disabled=true
``` ```
Exit the CLI by pressing `ctrl-d` Exit the CLI by pressing `ctrl-d`
* On other versions add the user to the sudoers file: - On other versions add the user to the sudoers file:
```
```bash
visudo visudo
``` ```
```
```bash
<username> ALL=(ALL) NOPASSWD:ALL <username> ALL=(ALL) NOPASSWD:ALL
``` ```
Confirm sudoers file is appropriate: Confirm sudoers file is appropriate:
```
```bash
cat /usr/local/etc/sudoers cat /usr/local/etc/sudoers
``` ```
* `CSI` has a homefolder, this is used to store its SSH Public Key
&emsp;![image](https://user-images.githubusercontent.com/40062371/147370105-6030b22e-ceb3-4768-b4a0-8e55fafe7f0f.png)
* Add the user to `wheel` or create/use a group that will be used for permissions later on
<br/> - `CSI` has a homefolder, this is used to store its SSH Public Key
&emsp;![image](https://user-images.githubusercontent.com/40062371/147370105-6030b22e-ceb3-4768-b4a0-8e55fafe7f0f.png)
- Add the user to `wheel` or create/use a group that will be used for permissions later on
#### **NFS configuration** #### **NFS configuration**
___
* Bind the interface to the NFS service
* It is recommended to use NFS 3
<br/> ___
- Bind the interface to the NFS service
- It is recommended to use NFS 3
#### **iSCSI configuration** #### **iSCSI configuration**
___ ___
*NOTE:* (Fixed in 12.0-U2+) when using the FreeNAS API concurrently, the `/etc/ctl.conf` file on the server can become invalid, some sample scripts are provided in the `contrib` directory to clean things up ie: *NOTE:* (Fixed in 12.0-U2+) when using the FreeNAS API concurrently, the `/etc/ctl.conf` file on the server can become invalid, some sample scripts are provided in the `contrib` directory to clean things up ie:
Copy the script to the server and directly and run - `./ctld-config-watchdog-db.sh | logger -t ctld-config-watchdog-db.sh &` Copy the script to the server and directly and run - `./ctld-config-watchdog-db.sh | logger -t ctld-config-watchdog-db.sh &`
Please read the scripts and set the variables correctly for your server. Please read the scripts and set the variables appropriate for your server.
* Ensure you have pre*emptively created portals, initatior groups, auths
* Make note of the respective IDs (the true ID may not reflect what is
visible in the UI)
* IDs can be visible by clicking the the `Edit` link and finding the ID in the
browser address bar
* Optionally you may use the following to retrieve appropiate IDs:
* `curl --header "Accept: application/json" --user root:<password> 'http(s)://<ip>/api/v2.0/iscsi/portal'`
* `curl --header "Accept: application/json" --user root:<password> 'http(s)://<ip>/api/v2.0/iscsi/initiator'`
* `curl --header "Accept: application/json" --user root:<password> 'http(s)://<ip>/api/v2.0/iscsi/auth'`
<br/> - Ensure you have pre*emptively created portals, initatior groups, auths
- Make note of the respective IDs (the true ID may not reflect what is
visible in the UI)
- IDs can be visible by clicking the the `Edit` link and finding the ID in the
browser address bar
- Optionally you may use the following to retrieve appropiate IDs:
- `curl --header "Accept: application/json" --user root:<password> 'http(s)://<ip>/api/v2.0/iscsi/portal'`
- `curl --header "Accept: application/json" --user root:<password> 'http(s)://<ip>/api/v2.0/iscsi/initiator'`
- `curl --header "Accept: application/json" --user root:<password> 'http(s)://<ip>/api/v2.0/iscsi/auth'`
### **SMB configuration** ### **SMB configuration**
___
* Bind the interface to the SMB service
<br/> ___
- Bind the interface to the SMB service
### **YAML Values configuration** ### **YAML Values configuration**
___ ___
Instruct `Democratic-CSI` to use `sudo` by uncommenting the following in your configuration template: Instruct `Democratic-CSI` to use `sudo` by uncommenting the following in your configuration template:
``` ```bash
zfs: zfs:
cli: cli:
sudoEnabled: true sudoEnabled: true
@ -283,60 +300,67 @@ Issues to review:
[ixsystems NAS-108522](https://jira.ixsystems.com/browse/NAS-108522) [ixsystems NAS-108522](https://jira.ixsystems.com/browse/NAS-108522)
[ixsystems NAS-107219](https://jira.ixsystems.com/browse/NAS-107219) [ixsystems NAS-107219](https://jira.ixsystems.com/browse/NAS-107219)
<br/>
### **ZoL (zfs-generic-nfs, zfs-generic-iscsi)** ### **ZoL (zfs-generic-nfs, zfs-generic-iscsi)**
___ ___
Ensure ssh and zfs is installed on the nfs/iscsi server and that you have installed Ensure ssh and zfs is installed on the nfs/iscsi server and that you have installed
`targetcli`. `targetcli`.
``` ```bash
sudo yum install targetcli -y sudo yum install targetcli -y
``` ```
```
```bash
sudo apt-get -y install targetcli-fb sudo apt-get -y install targetcli-fb
``` ```
<br/>
### **Synology (synology-iscsi)** ### **Synology (synology-iscsi)**
___ ___
Ensure iSCSI Manager has been installed and is generally setup/configured. Ensure iSCSI Manager has been installed and is generally setup/configured.
<br/>
## **Helm Installation** ## **Helm Installation**
___ ___
Copy proper example Values file from the examples: Copy proper example Values file from the examples:
[API without SSH](https://github.com/D1StrX/democratic-csi/blob/667354978e497fb4624d52e909609ca278e4bd25/examples/api-without-ssh) [API without SSH](https://github.com/D1StrX/democratic-csi/blob/667354978e497fb4624d52e909609ca278e4bd25/examples/api-without-ssh)
[API with SSH](https://github.com/D1StrX/democratic-csi/blob/667354978e497fb4624d52e909609ca278e4bd25/examples/api-with-ssh) [API with SSH](https://github.com/D1StrX/democratic-csi/blob/667354978e497fb4624d52e909609ca278e4bd25/examples/api-with-ssh)
Add the `Democratic-CSI` Helm repository: Add the `Democratic-CSI` Helm repository:
```
```bash
helm search repo democratic-csi/ helm search repo democratic-csi/
``` ```
Update your Helm repository to get latest charts: Update your Helm repository to get latest charts:
```
```bash
helm repo update helm repo update
``` ```
### **Helm V3** ### **Helm V3**
___ ___
Install `Democratic-CSI` with your configured values. Helm V3 requires that you `--create-namespace` Install `Democratic-CSI` with your configured values. Helm V3 requires that you `--create-namespace`
```
```bash
helm install zfs-nfs democratic-csi/democratic-csi --values truenas-isci.yaml --create-namespace democratic-csi helm install zfs-nfs democratic-csi/democratic-csi --values truenas-isci.yaml --create-namespace democratic-csi
``` ```
Update/Upgrade Values: Update/Upgrade Values:
```
```bash
helm upgrade <name> democratic-csi/democratic-csi --values <freenas-*>.yaml --namespace <namespace> helm upgrade <name> democratic-csi/democratic-csi --values <freenas-*>.yaml --namespace <namespace>
``` ```
### **Helm V2** ### **Helm V2**
___ ___
Install `Democratic-CSI` with your configured values. Install `Democratic-CSI` with your configured values.
```
```bash
helm upgrade \ helm upgrade \
--install \ --install \
--values freenas-nfs.yaml \ --values freenas-nfs.yaml \
@ -347,6 +371,7 @@ zfs-nfs democratic-csi/democratic-csi
### **On non standard Kubelet paths** ### **On non standard Kubelet paths**
Some distrobutions, such as `minikube` and `microk8s` use a non-standard kubelet path. In such cases it is ecessary to provide a new kubelet host path, microk8s example below: Some distrobutions, such as `minikube` and `microk8s` use a non-standard kubelet path. In such cases it is ecessary to provide a new kubelet host path, microk8s example below:
```bash ```bash
microk8s helm upgrade \ microk8s helm upgrade \
--install \ --install \
@ -356,15 +381,15 @@ microk8s helm upgrade \
zfs-nfs democratic-csi/democratic-csi zfs-nfs democratic-csi/democratic-csi
``` ```
* microk8s - `/var/snap/microk8s/common/var/lib/kubelet` - microk8s - `/var/snap/microk8s/common/var/lib/kubelet`
* pivotal - `/var/vcap/data/kubelet` - pivotal - `/var/vcap/data/kubelet`
### **OpenShift** ### **OpenShift**
`Democratic-CSI` generally works fine with openshift. Some special parameters `Democratic-CSI` generally works fine with openshift. Some special parameters
need to be set with helm (support added in chart version `0.6.1`): need to be set with helm (support added in chart version `0.6.1`):
``` ```bash
# for sure required # for sure required
--set node.rbac.openshift.privileged=true --set node.rbac.openshift.privileged=true
--set node.driver.localtimeHostPath=false --set node.driver.localtimeHostPath=false
@ -387,11 +412,12 @@ You may install multiple deployments of each/any driver. It requires the followi
- Use a unique parent dataset (ie: don't try to use the same parent across deployments or clusters) - Use a unique parent dataset (ie: don't try to use the same parent across deployments or clusters)
## **Snapshot Support** ## **Snapshot Support**
Install beta (v1.17+) CRDs (one per cluster): Install beta (v1.17+) CRDs (one per cluster):
- https://github.com/kubernetes-csi/external-snapshotter/tree/master/client/config/crd - <https://github.com/kubernetes-csi/external-snapshotter/tree/master/client/config/crd>
``` ```bash
kubectl apply -f snapshot.storage.k8s.io_volumesnapshotclasses.yaml kubectl apply -f snapshot.storage.k8s.io_volumesnapshotclasses.yaml
kubectl apply -f snapshot.storage.k8s.io_volumesnapshotcontents.yaml kubectl apply -f snapshot.storage.k8s.io_volumesnapshotcontents.yaml
kubectl apply -f snapshot.storage.k8s.io_volumesnapshots.yaml kubectl apply -f snapshot.storage.k8s.io_volumesnapshots.yaml
@ -399,9 +425,9 @@ kubectl apply -f snapshot.storage.k8s.io_volumesnapshots.yaml
Install snapshot controller (once per cluster): Install snapshot controller (once per cluster):
- https://github.com/kubernetes-csi/external-snapshotter/tree/master/deploy/kubernetes/snapshot-controller - <https://github.com/kubernetes-csi/external-snapshotter/tree/master/deploy/kubernetes/snapshot-controller>
``` ```bash
# replace namespace references to your liking # replace namespace references to your liking
kubectl apply -f rbac-snapshot-controller.yaml kubectl apply -f rbac-snapshot-controller.yaml
kubectl apply -f setup-snapshot-controller.yaml kubectl apply -f setup-snapshot-controller.yaml
@ -409,11 +435,12 @@ kubectl apply -f setup-snapshot-controller.yaml
Install `Democratic-CSI` as usual with `volumeSnapshotClasses` defined as appropriate. Install `Democratic-CSI` as usual with `volumeSnapshotClasses` defined as appropriate.
- https://kubernetes.io/docs/concepts/storage/volume-snapshots/ - <https://kubernetes.io/docs/concepts/storage/volume-snapshots/>
- https://github.com/kubernetes-csi/external-snapshotter#usage - <https://github.com/kubernetes-csi/external-snapshotter#usage>
- https://github.com/democratic-csi/democratic-csi/issues/129#issuecomment-961489810 - <https://github.com/democratic-csi/democratic-csi/issues/129#issuecomment-961489810>
## **Migrating from freenas-provisioner and freenas-iscsi-provisioner** ## **Migrating from freenas-provisioner and freenas-iscsi-provisioner**
It is possible to migrate all volumes from the non-csi freenas provisioners It is possible to migrate all volumes from the non-csi freenas provisioners
to `Democratic-CSI`. to `Democratic-CSI`.
@ -421,20 +448,16 @@ Copy the `contrib/freenas-provisioner-to-democratic-csi.sh` script from the
project to your workstation, read the script in detail, and edit the variables project to your workstation, read the script in detail, and edit the variables
to your needs to start migrating! to your needs to start migrating!
<br/>
# **Sponsors** # **Sponsors**
A special shout out to the wonderful sponsors of this project! A special shout out to the wonderful sponsors of this project!
[![ixSystems](https://www.ixsystems.com/wp-content/uploads/2021/06/ix_logo_200x47.png "ixSystems")](http://ixsystems.com/) [![ixSystems](https://www.ixsystems.com/wp-content/uploads/2021/06/ix_logo_200x47.png "ixSystems")](http://ixsystems.com/)
<br/>
## **Related** ## **Related**
- https://github.com/nmaupu/freenas-provisioner - <https://github.com/nmaupu/freenas-provisioner>
- https://github.com/travisghansen/freenas-iscsi-provisioner - <https://github.com/travisghansen/freenas-iscsi-provisioner>
- https://datamattsson.tumblr.com/post/624751011659202560/welcome-truenas-core-container-storage-provider - <https://datamattsson.tumblr.com/post/624751011659202560/welcome-truenas-core-container-storage-provider>
- https://github.com/dravanet/truenas-csi - <https://github.com/dravanet/truenas-csi>
- https://github.com/SynologyOpenSource/synology-csi - <https://github.com/SynologyOpenSource/synology-csi>