From a458636a8fd653c1d945b0116922337257755d60 Mon Sep 17 00:00:00 2001 From: D1StrX Date: Wed, 12 Jan 2022 21:12:06 +0100 Subject: [PATCH] markdown fixes --- README.md | 219 ++++++++++++++++++++++++++++++------------------------ 1 file changed, 121 insertions(+), 98 deletions(-) diff --git a/README.md b/README.md index 53f0e9c..c6e2244 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,7 @@ The current *focus* is providing storage via iSCSI or NFS from ZFS-based storage systems, predominantly `TrueNAS / FreeNAS` and `ZoL on Ubuntu`. The current *drivers* implement the depth and breadth of the `csi` specifications, so you have access to resizing, snapshots, clones, etc functionality. -## What can Democratic-CSI offer? +## What can Democratic-CSI offer? **Several implementations of `CSI` drivers** :arrow_forward: `freenas-nfs` (manages zfs datasets to share over nfs) @@ -43,18 +43,19 @@ relatively easy to implement new drivers. # Installation -Predominantly 3 prerequisites are needed: +Predominantly 3 prerequisites are needed: + - Nodes preperation (ie: Kubernetes cluster nodes) - Storage server preperation - Deployment of the driver into the cluster (`helm` chart provided with sample `values.yaml`) -## Node preperation +## :wrench: Node preperation Alright, you have chosen your driver. Let's start by configuring the prerequisites for your Node. You can choose to use either **NFS** or **iSCSI** or **both**. -### **NFS configuration** +### NFS configuration ___ @@ -74,7 +75,7 @@ sudo apt-get install -y nfs-common ___ -#### RHEL / CentOS +**RHEL / CentOS** Install the following system packages: ```bash @@ -106,14 +107,15 @@ Install the following system packages: sudo apt-get install -y open-iscsi lsscsi sg3-utils multipath-tools scsitools ``` -**Multipathing** +#### Multipathing + `Multipath` is supported for the `iSCSI`-based drivers. Simply setup multipath to your liking and set multiple portals in the config as appropriate. *NOTE:* If you are running Kubernetes with Rancher/RKE please see the following: [Support host iscsi simultaneously with kubelet iscsi (pvc)](https://github.com/rancher/rke/issues/1846>) -
Add the mutlipath configuration: -``` + +```bash sudo tee /etc/multipath.conf <<-'EOF' defaults { user_friendly_names yes @@ -121,151 +123,166 @@ defaults { } EOF ``` + Enable the `multipath-tools` service and restart to load the configuration: -``` + +```bash sudo systemctl enable multipath-tools && sudo service multipath-tools restart ``` + Ensure that `open-iscsi` and `multipath-tools` are enabled and running: -``` + +```bash sudo systemctl status multipath-tools sudo systemctl enable open-iscsi.service sudo service open-iscsi start sudo systemctl status open-iscsi ``` -
-### **FreeNAS-SMB** +### FreeNAS-SMB + ___ If using with Windows based machines you may need to enable guest access (even if you are connecting with credentials) -``` +```powershell Set-ItemProperty HKLM:\SYSTEM\CurrentControlSet\Services\LanmanWorkstation\Parameters AllowInsecureGuestAuth -Value 1 ; Restart-Service LanmanWorkstation -Force ``` -
-### **ZFS-local-ephemeral-inline** +### ZFS-local-ephemeral-inline + ___ This `driver` provisions node-local ephemeral storage on a per-pod basis. Each node should have an identically named ZFS pool created and avaialble to the `driver`. *NOTE:* This is *NOT* the same thing as using the docker ZFS storage driver (although the same pool could be used). No other requirements are necessary. More regarding to this can be found here: [Pod Inline Volume Support](https://kubernetes-csi.github.io/docs/ephemeral-local-volumes.html) -
- -## **Storage server preperation** +## :wrench: Storage server preperation Storage server preperation depends slightly on which `driver` you are using. The recommended version of FreeNAS / TrueNAS is 12.0-U2 or higher, however the driver should work with much older versions as well. -### **TrueNAS / FreeNAS (freenas-nfs, freenas-iscsi, freenas-smb, freenas-api-nfs, freenas-api-iscsi, freenas-api-smb)** -
+### TrueNAS / FreeNAS (freenas-nfs, freenas-iscsi, freenas-smb, freenas-api-nfs, freenas-api-iscsi, freenas-api-smb) + +#### API without SSH -**API without SSH** ___ Configuration templates can be found [HERE](https://github.com/D1StrX/democratic-csi/blob/667354978e497fb4624d52e909609ca278e4bd25/examples/api-with-ssh) The various `freenas-api-*` drivers are currently EXPERIMENTAL and can only be used with SCALE 21.08+. Fundamentally these drivers remove the need for `ssh` connections and do all operations entirely with the TrueNAS api. With that in mind, any `ssh/shell/etc` requirements below can be safely ignored. Also note the following known issues: -* [Additional middleware changes to support Democratic CSI use of native API](https://jira.ixsystems.com/browse/NAS-111870) -* [TrueNAS Scale 21.08 - Could not log into all portals](https://github.com/democratic-csi/democratic-csi/issues/112) -* [Pure api based truenas driver (ssh dependency removed)](https://github.com/democratic-csi/democratic-csi/issues/101) +- [Additional middleware changes to support Democratic CSI use of native API](https://jira.ixsystems.com/browse/NAS-111870) +- [TrueNAS Scale 21.08 - Could not log into all portals](https://github.com/democratic-csi/democratic-csi/issues/112) +- [Pure api based truenas driver (ssh dependency removed)](https://github.com/democratic-csi/democratic-csi/issues/101) [Continue configuration](#Service-configuration) -
-**API with SSH** +#### API with SSH + ___ Configuration templates can be found [HERE](https://github.com/D1StrX/democratic-csi/blob/667354978e497fb4624d52e909609ca278e4bd25/examples/api-with-ssh) [Continue configuration](#Service-configuration) -
-### **Service configuration** +### Service configuration + Ensure the following services are *configured*, *running* and starting automatically: -#### **SSH configuration** +#### SSH configuration + ___ -* When creating a custom user (e.g., `CSI`): - * Ensure `ZSH`, `BASH`, or `SH` is set as `shell`, `CSH` gives false errors due to quoting (also applicable when using `root`) + +- When creating a custom user (e.g., `CSI`): + - Ensure `ZSH`, `BASH`, or `SH` is set as `shell`, `CSH` gives false errors due to quoting (also applicable when using `root`)  ![image](https://user-images.githubusercontent.com/40062371/147365044-007b2657-30f9-428b-ae12-7622a572866d.png) - * Ensure that user has passwordless `sudo` privileges: + - Ensure that user has passwordless `sudo` privileges: *NOTE:* This could get reset by FreeNAS if you alter the user via the GUI later - * On TrueNAS CORE 12.0-u3 or higher, open the Shell: - ``` + - On TrueNAS CORE 12.0-u3 or higher, open the Shell: + + ```bash cli ``` + After you enter the truenas cli and are at that prompt: - ``` + + ```bash account user query select=id,username,uid,sudo_nopasswd ``` + find the `id` of the user you want to update (note, this is distinct from the `uid`) - ``` + ```bash account user update id= sudo=true ``` - ``` + + ```bash account user update id= sudo_nopasswd=true ``` + (Optional) If you want to enable passwordless authentication via CLI: - ``` + + ```bash account user update id= password_disabled=true ``` + Exit the CLI by pressing `ctrl-d` - * On other versions add the user to the sudoers file: - ``` + - On other versions add the user to the sudoers file: + + ```bash visudo ``` - ``` + + ```bash ALL=(ALL) NOPASSWD:ALL ``` + Confirm sudoers file is appropriate: - ``` + + ```bash cat /usr/local/etc/sudoers ``` - - * `CSI` has a homefolder, this is used to store its SSH Public Key -  ![image](https://user-images.githubusercontent.com/40062371/147370105-6030b22e-ceb3-4768-b4a0-8e55fafe7f0f.png) - * Add the user to `wheel` or create/use a group that will be used for permissions later on -
+ - `CSI` has a homefolder, this is used to store its SSH Public Key +  ![image](https://user-images.githubusercontent.com/40062371/147370105-6030b22e-ceb3-4768-b4a0-8e55fafe7f0f.png) + - Add the user to `wheel` or create/use a group that will be used for permissions later on #### **NFS configuration** -___ -* Bind the interface to the NFS service -* It is recommended to use NFS 3 -
+___ + +- Bind the interface to the NFS service +- It is recommended to use NFS 3 #### **iSCSI configuration** + ___ *NOTE:* (Fixed in 12.0-U2+) when using the FreeNAS API concurrently, the `/etc/ctl.conf` file on the server can become invalid, some sample scripts are provided in the `contrib` directory to clean things up ie: Copy the script to the server and directly and run - `./ctld-config-watchdog-db.sh | logger -t ctld-config-watchdog-db.sh &` -Please read the scripts and set the variables correctly for your server. -* Ensure you have pre*emptively created portals, initatior groups, auths - * Make note of the respective IDs (the true ID may not reflect what is - visible in the UI) - * IDs can be visible by clicking the the `Edit` link and finding the ID in the - browser address bar - * Optionally you may use the following to retrieve appropiate IDs: - * `curl --header "Accept: application/json" --user root: 'http(s):///api/v2.0/iscsi/portal'` - * `curl --header "Accept: application/json" --user root: 'http(s):///api/v2.0/iscsi/initiator'` - * `curl --header "Accept: application/json" --user root: 'http(s):///api/v2.0/iscsi/auth'` +Please read the scripts and set the variables appropriate for your server. -
+- Ensure you have pre*emptively created portals, initatior groups, auths + - Make note of the respective IDs (the true ID may not reflect what is + visible in the UI) + - IDs can be visible by clicking the the `Edit` link and finding the ID in the + browser address bar + - Optionally you may use the following to retrieve appropiate IDs: + - `curl --header "Accept: application/json" --user root: 'http(s):///api/v2.0/iscsi/portal'` + - `curl --header "Accept: application/json" --user root: 'http(s):///api/v2.0/iscsi/initiator'` + - `curl --header "Accept: application/json" --user root: 'http(s):///api/v2.0/iscsi/auth'` ### **SMB configuration** -___ -* Bind the interface to the SMB service -
+___ + +- Bind the interface to the SMB service ### **YAML Values configuration** + ___ Instruct `Democratic-CSI` to use `sudo` by uncommenting the following in your configuration template: - ``` + ```bash zfs: cli: sudoEnabled: true @@ -283,60 +300,67 @@ Issues to review: [ixsystems NAS-108522](https://jira.ixsystems.com/browse/NAS-108522) [ixsystems NAS-107219](https://jira.ixsystems.com/browse/NAS-107219) -
- ### **ZoL (zfs-generic-nfs, zfs-generic-iscsi)** + ___ Ensure ssh and zfs is installed on the nfs/iscsi server and that you have installed `targetcli`. - ``` + ```bash sudo yum install targetcli -y ``` - ``` + + ```bash sudo apt-get -y install targetcli-fb ``` -
- ### **Synology (synology-iscsi)** + ___ Ensure iSCSI Manager has been installed and is generally setup/configured. -
- ## **Helm Installation** + ___ Copy proper example Values file from the examples: [API without SSH](https://github.com/D1StrX/democratic-csi/blob/667354978e497fb4624d52e909609ca278e4bd25/examples/api-without-ssh) [API with SSH](https://github.com/D1StrX/democratic-csi/blob/667354978e497fb4624d52e909609ca278e4bd25/examples/api-with-ssh) Add the `Democratic-CSI` Helm repository: -``` + +```bash helm search repo democratic-csi/ ``` + Update your Helm repository to get latest charts: -``` + +```bash helm repo update ``` ### **Helm V3** + ___ Install `Democratic-CSI` with your configured values. Helm V3 requires that you `--create-namespace` -``` + +```bash helm install zfs-nfs democratic-csi/democratic-csi --values truenas-isci.yaml --create-namespace democratic-csi ``` + Update/Upgrade Values: -``` + +```bash helm upgrade democratic-csi/democratic-csi --values .yaml --namespace ``` ### **Helm V2** + ___ Install `Democratic-CSI` with your configured values. -``` + +```bash helm upgrade \ --install \ --values freenas-nfs.yaml \ @@ -347,6 +371,7 @@ zfs-nfs democratic-csi/democratic-csi ### **On non standard Kubelet paths** Some distrobutions, such as `minikube` and `microk8s` use a non-standard kubelet path. In such cases it is ecessary to provide a new kubelet host path, microk8s example below: + ```bash microk8s helm upgrade \ --install \ @@ -356,15 +381,15 @@ microk8s helm upgrade \ zfs-nfs democratic-csi/democratic-csi ``` -* microk8s - `/var/snap/microk8s/common/var/lib/kubelet` -* pivotal - `/var/vcap/data/kubelet` +- microk8s - `/var/snap/microk8s/common/var/lib/kubelet` +- pivotal - `/var/vcap/data/kubelet` ### **OpenShift** `Democratic-CSI` generally works fine with openshift. Some special parameters need to be set with helm (support added in chart version `0.6.1`): -``` +```bash # for sure required --set node.rbac.openshift.privileged=true --set node.driver.localtimeHostPath=false @@ -387,11 +412,12 @@ You may install multiple deployments of each/any driver. It requires the followi - Use a unique parent dataset (ie: don't try to use the same parent across deployments or clusters) ## **Snapshot Support** + Install beta (v1.17+) CRDs (one per cluster): -- https://github.com/kubernetes-csi/external-snapshotter/tree/master/client/config/crd +- -``` +```bash kubectl apply -f snapshot.storage.k8s.io_volumesnapshotclasses.yaml kubectl apply -f snapshot.storage.k8s.io_volumesnapshotcontents.yaml kubectl apply -f snapshot.storage.k8s.io_volumesnapshots.yaml @@ -399,9 +425,9 @@ kubectl apply -f snapshot.storage.k8s.io_volumesnapshots.yaml Install snapshot controller (once per cluster): -- https://github.com/kubernetes-csi/external-snapshotter/tree/master/deploy/kubernetes/snapshot-controller +- -``` +```bash # replace namespace references to your liking kubectl apply -f rbac-snapshot-controller.yaml kubectl apply -f setup-snapshot-controller.yaml @@ -409,11 +435,12 @@ kubectl apply -f setup-snapshot-controller.yaml Install `Democratic-CSI` as usual with `volumeSnapshotClasses` defined as appropriate. -- https://kubernetes.io/docs/concepts/storage/volume-snapshots/ -- https://github.com/kubernetes-csi/external-snapshotter#usage -- https://github.com/democratic-csi/democratic-csi/issues/129#issuecomment-961489810 +- +- +- ## **Migrating from freenas-provisioner and freenas-iscsi-provisioner** + It is possible to migrate all volumes from the non-csi freenas provisioners to `Democratic-CSI`. @@ -421,20 +448,16 @@ Copy the `contrib/freenas-provisioner-to-democratic-csi.sh` script from the project to your workstation, read the script in detail, and edit the variables to your needs to start migrating! -
- # **Sponsors** A special shout out to the wonderful sponsors of this project! [![ixSystems](https://www.ixsystems.com/wp-content/uploads/2021/06/ix_logo_200x47.png "ixSystems")](http://ixsystems.com/) -
- ## **Related** -- https://github.com/nmaupu/freenas-provisioner -- https://github.com/travisghansen/freenas-iscsi-provisioner -- https://datamattsson.tumblr.com/post/624751011659202560/welcome-truenas-core-container-storage-provider -- https://github.com/dravanet/truenas-csi -- https://github.com/SynologyOpenSource/synology-csi +- +- +- +- +-