prep for release, documentation

Signed-off-by: Travis Glenn Hansen <travisghansen@yahoo.com>

README.md

@@ -18,15 +18,21 @@ have access to resizing, snapshots, clones, etc functionality.
   - `freenas-nfs` (manages zfs datasets to share over nfs)
   - `freenas-iscsi` (manages zfs zvols to share over iscsi)
   - `freenas-smb` (manages zfs datasets to share over smb)
+  - `freenas-api-nfs` experimental use with SCALE only (manages zfs datasets to share over nfs)
+  - `freenas-api-iscsi` experimental use with SCALE only (manages zfs zvols to share over iscsi)
+  - `freenas-api-smb` experimental use with SCALE only (manages zfs datasets to share over smb)
   - `zfs-generic-nfs` (works with any ZoL installation...ie: Ubuntu)
   - `zfs-generic-iscsi` (works with any ZoL installation...ie: Ubuntu)
   - `zfs-local-ephemeral-inline` (provisions node-local zfs datasets)
+  - `synology-iscsi` experimental (manages volumes to share over iscsi)
+  - `lustre-client` (crudely provisions storage using a shared lustre
+    share/directory for all volumes)
   - `nfs-client` (crudely provisions storage using a shared nfs share/directory
     for all volumes)
   - `smb-client` (crudely provisions storage using a shared smb share/directory
     for all volumes)
-  - `node-manual` (allows connecting to manually created smb, nfs, and iscsi
-    volumes, see sample PVs in the `examples` directory)
+  - `node-manual` (allows connecting to manually created smb, nfs, lustre, and
+    iscsi volumes, see sample PVs in the `examples` directory)
 - framework for developing `csi` drivers
 
 If you have any interest in providing a `csi` driver, simply open an issue to
@@ -139,11 +145,16 @@ necessary.
 
 Server preparation depends slightly on which `driver` you are using.
 
-### FreeNAS (freenas-nfs, freenas-iscsi, freenas-smb)
+### FreeNAS (freenas-nfs, freenas-iscsi, freenas-smb, freenas-api-nfs, freenas-api-iscsi, freenas-api-smb)
 
 The recommended version of FreeNAS is 12.0-U2+, however the driver should work
 with much older versions as well.
 
+The various `freenas-api-*` drivers are currently EXPERIMENTAL and can only be
+used with SCALE 21.08+. Fundamentally these drivers remove the need for `ssh`
+connections and do all operations entirely with the TrueNAS api. With that in
+mind, any ssh/shell/etc requirements below can be safely ignored.
+
 Ensure the following services are configurged and running:
 
 - ssh (if you use a password for authentication make sure it is allowed)
@@ -208,6 +219,10 @@ Ensure ssh and zfs is installed on the nfs/iscsi server and that you have instal
 - `sudo yum install targetcli -y`
 - `sudo apt-get -y install targetcli-fb`
 
+### Synology (synology-iscsi)
+
+Ensure iscsi manager has been installed and is generally setup/configured.
+
 ## Helm Installation
 
 ```
@@ -252,6 +267,9 @@ microk8s helm upgrade \
   zfs-nfs democratic-csi/democratic-csi
 ```
 
+- microk8s - `/var/snap/microk8s/common/var/lib/kubelet`
+- pivotal - `/var/vcap/data/kubelet`
+
 ### openshift
 
 `democratic-csi` generally works fine with openshift. Some special parameters

examples/freenas-api-iscsi.yaml

@@ -0,0 +1,85 @@
+driver: freenas-api-iscsi
+instance_id:
+httpConnection:
+  protocol: http
+  host: server address
+  port: 80
+  # use only 1 of apiKey or username/password
+  # if both are present, apiKey is preferred
+  # apiKey is only available starting in TrueNAS-12
+  #apiKey:
+  username: root
+  password:
+  allowInsecure: true
+  # use apiVersion 2 for TrueNAS-12 and up (will work on 11.x in some scenarios as well)
+  # leave unset for auto-detection
+  #apiVersion: 2
+zfs:
+  # can be used to override defaults if necessary
+  # the example below is useful for TrueNAS 12
+  #cli:
+  #  sudoEnabled: true
+  #
+  #  leave paths unset for auto-detection
+  #  paths:
+  #    zfs: /usr/local/sbin/zfs
+  #    zpool: /usr/local/sbin/zpool
+  #    sudo: /usr/local/bin/sudo
+  #    chroot: /usr/sbin/chroot
+
+  # can be used to set arbitrary values on the dataset/zvol
+  # can use handlebars templates with the parameters from the storage class/CO
+  #datasetProperties:
+  #  "org.freenas:description": "{{ parameters.[csi.storage.k8s.io/pvc/namespace] }}/{{ parameters.[csi.storage.k8s.io/pvc/name] }}"
+  #  "org.freenas:test": "{{ parameters.foo }}"
+  #  "org.freenas:test2": "some value"
+  
+  # total volume name (zvol/<datasetParentName>/<pvc name>) length cannot exceed 63 chars
+  # https://www.ixsystems.com/documentation/freenas/11.2-U5/storage.html#zfs-zvol-config-opts-tab
+  # standard volume naming overhead is 46 chars
+  # datasetParentName should therefore be 17 chars or less
+  datasetParentName: tank/k8s/b/vols
+  # do NOT make datasetParentName and detachedSnapshotsDatasetParentName overlap
+  # they may be siblings, but neither should be nested in the other 
+  detachedSnapshotsDatasetParentName: tanks/k8s/b/snaps
+  # "" (inherit), lz4, gzip-9, etc
+  zvolCompression:
+  # "" (inherit), on, off, verify
+  zvolDedup:
+  zvolEnableReservation: false
+  # 512, 1K, 2K, 4K, 8K, 16K, 64K, 128K default is 16K
+  zvolBlocksize:
+iscsi:
+  targetPortal: "server[:port]"
+  # for multipath
+  targetPortals: [] # [ "server[:port]", "server[:port]", ... ]
+  # leave empty to omit usage of -I with iscsiadm
+  interface:
+
+  # MUST ensure uniqueness
+  # full iqn limit is 223 bytes, plan accordingly
+  # default is "{{ name }}"
+  #nameTemplate: "{{ parameters.[csi.storage.k8s.io/pvc/namespace] }}-{{ parameters.[csi.storage.k8s.io/pvc/name] }}"
+  namePrefix: csi-
+  nameSuffix: "-clustera"
+  # add as many as needed
+  targetGroups:
+    # get the correct ID from the "portal" section in the UI
+    - targetGroupPortalGroup: 1
+      # get the correct ID from the "initiators" section in the UI
+      targetGroupInitiatorGroup: 1
+      # None, CHAP, or CHAP Mutual
+      targetGroupAuthType: None
+      # get the correct ID from the "Authorized Access" section of the UI
+      # only required if using Chap
+      targetGroupAuthGroup:
+
+  extentInsecureTpc: true
+  extentXenCompat: false
+  extentDisablePhysicalBlocksize: true
+  # 512, 1024, 2048, or 4096,
+  extentBlocksize: 512
+  # "" (let FreeNAS decide, currently defaults to SSD), Unknown, SSD, 5400, 7200, 10000, 15000
+  extentRpm: "SSD"
+  # 0-100 (0 == ignore)
+  extentAvailThreshold: 0

examples/freenas-api-nfs.yaml

@@ -0,0 +1,58 @@
+driver: freenas-api-nfs
+instance_id:
+httpConnection:
+  protocol: http
+  host: server address
+  port: 80
+  # use only 1 of apiKey or username/password
+  # if both are present, apiKey is preferred
+  # apiKey is only available starting in TrueNAS-12
+  #apiKey:
+  username: root
+  password:
+  allowInsecure: true
+  # use apiVersion 2 for TrueNAS-12 and up (will work on 11.x in some scenarios as well)
+  # leave unset for auto-detection
+  #apiVersion: 2
+zfs:
+  # can be used to override defaults if necessary
+  # the example below is useful for TrueNAS 12
+  #cli:
+  #  sudoEnabled: true
+  #
+  #  leave paths unset for auto-detection
+  #  paths:
+  #    zfs: /usr/local/sbin/zfs
+  #    zpool: /usr/local/sbin/zpool
+  #    sudo: /usr/local/bin/sudo
+  #    chroot: /usr/sbin/chroot
+  
+  # can be used to set arbitrary values on the dataset/zvol
+  # can use handlebars templates with the parameters from the storage class/CO
+  #datasetProperties:
+  #  "org.freenas:description": "{{ parameters.[csi.storage.k8s.io/pvc/namespace] }}/{{ parameters.[csi.storage.k8s.io/pvc/name] }}"
+  #  "org.freenas:test": "{{ parameters.foo }}"
+  #  "org.freenas:test2": "some value"
+
+  datasetParentName: tank/k8s/a/vols
+  # do NOT make datasetParentName and detachedSnapshotsDatasetParentName overlap
+  # they may be siblings, but neither should be nested in the other
+  detachedSnapshotsDatasetParentName: tank/k8s/a/snaps
+  datasetEnableQuotas: true
+  datasetEnableReservation: false
+  datasetPermissionsMode: "0777"
+  datasetPermissionsUser: 0
+  datasetPermissionsGroup: 0
+  #datasetPermissionsAcls:
+  #- "-m everyone@:full_set:allow"
+  #- "-m u:kube:full_set:allow"
+
+nfs:
+  shareHost: server address
+  shareAlldirs: false
+  shareAllowedHosts: []
+  shareAllowedNetworks: []
+  shareMaprootUser: root
+  shareMaprootGroup: root
+  shareMapallUser: ""
+  shareMapallGroup: ""

examples/freenas-api-smb.yaml

@@ -0,0 +1,77 @@
+driver: freenas-api-smb
+instance_id:
+httpConnection:
+  protocol: http
+  host: server address
+  port: 80
+  # use only 1 of apiKey or username/password
+  # if both are present, apiKey is preferred
+  # apiKey is only available starting in TrueNAS-12
+  #apiKey:
+  username: root
+  password:
+  allowInsecure: true
+  # use apiVersion 2 for TrueNAS-12 and up (will work on 11.x in some scenarios as well)
+  # leave unset for auto-detection
+  #apiVersion: 2
+zfs:
+  # can be used to override defaults if necessary
+  # the example below is useful for TrueNAS 12
+  #cli:
+  #  sudoEnabled: true
+  #
+  #  leave paths unset for auto-detection
+  #  paths:
+  #    zfs: /usr/local/sbin/zfs
+  #    zpool: /usr/local/sbin/zpool
+  #    sudo: /usr/local/bin/sudo
+  #    chroot: /usr/sbin/chroot
+  
+  # can be used to set arbitrary values on the dataset/zvol
+  # can use handlebars templates with the parameters from the storage class/CO
+  #datasetProperties:
+  #  "org.freenas:description": "{{ parameters.[csi.storage.k8s.io/pvc/namespace] }}/{{ parameters.[csi.storage.k8s.io/pvc/name] }}"
+  #  "org.freenas:test": "{{ parameters.foo }}"
+  #  "org.freenas:test2": "some value"
+
+  datasetProperties:
+    aclmode: restricted
+    casesensitivity: mixed
+
+  datasetParentName: tank/k8s/a/vols
+  # do NOT make datasetParentName and detachedSnapshotsDatasetParentName overlap
+  # they may be siblings, but neither should be nested in the other
+  detachedSnapshotsDatasetParentName: tank/k8s/a/snaps
+  datasetEnableQuotas: true
+  datasetEnableReservation: false
+  datasetPermissionsMode: "0777"
+  datasetPermissionsUser: 0
+  datasetPermissionsGroup: 0
+  datasetPermissionsAcls:
+  - "-m everyone@:full_set:allow"
+  #- "-m u:kube:full_set:allow"
+
+smb:
+  shareHost: server address
+  nameTemplate: ""
+  namePrefix: ""
+  nameSuffix: ""
+  
+  # if any of the shareFoo parameters do not work with your version of FreeNAS
+  # simply comment the param (and use the configuration template if necessary)
+
+  shareAuxiliaryConfigurationTemplate: |
+    #guest ok = yes
+    #guest only = yes
+  shareHome: false
+  shareAllowedHosts: []
+  shareDeniedHosts: []
+  #shareDefaultPermissions: true
+  shareGuestOk: true
+  #shareGuestOnly: true
+  #shareShowHiddenFiles: true
+  shareRecycleBin: true
+  shareBrowsable: false
+  shareAccessBasedEnumeration: true
+  shareTimeMachine: false
+  #shareStorageTask:

examples/freenas-smb.yaml

@@ -1,4 +1,4 @@
-driver: freenas-nfs
+driver: freenas-smb
 instance_id:
 httpConnection:
   protocol: http

examples/node-common.yaml

@@ -0,0 +1,23 @@
+# common options for the node service
+
+node:
+  mount:
+    # should fsck be executed before mounting the fs
+    checkFilesystem:
+      xfs:
+        enabled: false
+        customOptions: []
+      ext4:
+        enabled: false
+        customOptions: []
+        customFilesystemOptions: []
+  format:
+    xfs:
+      customOptions: []
+      #- -K
+      # ...
+    ext4:
+      customOptions: []
+      #- -E
+      #- nodiscard
+      # ...

examples/synology-iscsi.yaml

@@ -0,0 +1,89 @@
+driver: synology-iscsi
+httpConnection:
+  protocol: http
+  host: server address
+  port: 5000
+  username: admin
+  password: password
+  allowInsecure: true
+  # should be uniqe across all installs to the same nas
+  session: "democratic-csi"
+  serialize: true
+
+synology:
+  # choose the proper volume for your system
+  volume: /volume1
+
+iscsi:
+  targetPortal: "server[:port]"
+  # for multipath
+  targetPortals: [] # [ "server[:port]", "server[:port]", ... ]
+  # leave empty to omit usage of -I with iscsiadm
+  interface: ""
+  # can be whatever you would like
+  baseiqn: "iqn.2000-01.com.synology:csi."
+
+  # MUST ensure uniqueness
+  # full iqn limit is 223 bytes, plan accordingly
+  namePrefix: ""
+  nameSuffix: ""
+
+  # documented below are several blocks
+  # pick the option appropriate for you based on what your backing fs is and desired features
+  # you do not need to alter dev_attribs under normal circumstances but they may be altered in advanced use-cases
+  lunTemplate:
+    # btrfs thin provisioning
+    type: "BLUN"
+    # tpws = Hardware-assisted zeroing
+    # caw = Hardware-assisted locking
+    # 3pc = Hardware-assisted data transfer
+    # tpu = Space reclamation
+    # can_snapshot = Snapshot
+    #dev_attribs:
+    #- dev_attrib: emulate_tpws
+    #  enable: 1
+    #- dev_attrib: emulate_caw
+    #  enable: 1
+    #- dev_attrib: emulate_3pc
+    #  enable: 1
+    #- dev_attrib: emulate_tpu
+    #  enable: 0
+    #- dev_attrib: can_snapshot
+    #  enable: 1
+
+    # btfs thick provisioning
+    # only zeroing and locking supported
+    #type: "BLUN_THICK"
+    # tpws = Hardware-assisted zeroing
+    # caw = Hardware-assisted locking
+    #dev_attribs:
+    #- dev_attrib: emulate_tpws
+    #  enable: 1
+    #- dev_attrib: emulate_caw
+    #  enable: 1
+
+    # ext4 thinn provisioning UI sends everything with enabled=0
+    #type: "THIN"
+
+    # ext4 thin with advanced legacy features set
+    # can only alter tpu (all others are set as enabled=1)
+    #type: "ADV"
+    #dev_attribs:
+    #- dev_attrib: emulate_tpu
+    #  enable: 1
+
+    # ext4 thick
+    # can only alter caw
+    #type: "FILE"
+    #dev_attribs:
+    #- dev_attrib: emulate_caw
+    #  enable: 1
+
+  lunSnapshotTemplate:
+    is_locked: true
+    # https://kb.synology.com/en-me/DSM/tutorial/What_is_file_system_consistent_snapshot
+    is_app_consistent: true
+
+  targetTemplate:
+    auth_type: 0
+    max_sessions: 0