Add rancheros.md

README.md

@@ -264,6 +264,7 @@ Longhorn will always try to maintain at least given number of healthy replicas f
 ### [Google Kubernetes Engine](./docs/gke.md)
 ### [Upgrade](./docs/upgrade.md)
 ### [Deal with Kubernetes node failure](./docs/node-failure.md)
+### [Use CSI driver on RancherOS + RKE](./docs/rancheros.md)
 
 ## Troubleshooting
 You can click `Generate Support Bundle` link at the bottom of the UI to download a zip file contains Longhorn related configuration and logs.

docs/rancheros.md

@@ -0,0 +1,64 @@
+# Longhorn CSI on RancherOS + RKE
+
+## Requirements
+  1. Kubernetes v1.11 or higher.
+  2. Longhorn v0.4.1 or higher.
+  3. RancherOS Ubuntu console.
+
+
+## Instruction
+### For Kubernetes v1.11 only 
+  The following step is not needed for Kubernetes v1.12+.
+
+  Add extra_binds for kubelet in RKE `cluster.yml`:
+  ```
+   services:
+     kubelet:
+       extra_binds:
+       - "/opt/rke/var/lib/kubelet/plugins:/var/lib/kubelet/plugins" 
+   ```
+   
+### For each node:
+  #### 1. Switch to ubuntu console
+
+  `sudo ros console switch ubuntu`, then type `y`
+
+  #### 2. Install open-iscsi for each node. 
+  ```
+  sudo su
+  apt update
+  apt install -y open-iscsi
+  ```
+  #### 3. Modify configuration for iscsi. 
+    
+  1. Open config file `/etc/iscsi/iscsid.conf`
+  2. Comment `iscsid.startup = /bin/systemctl start iscsid.socket`
+  3. Uncomment `iscsid.startup = /sbin/iscsid`
+    
+
+
+## Background 
+CSI doesn't work with RancherOS + RKE before Longhorn v0.4.1. The reason is:
+
+1. RancherOS sets argument `root-dir=/opt/rke/var/lib/kubelet` for kubelet, , which is different from the default value `/var/lib/kubelet`.
+                                                                             
+2. **For k8s v1.12+**
+
+     Kubelet will detect the `csi.sock` according to argument `<--kubelet-registration-path>` passed in by Kubernetes CSI driver-registrar, and `<drivername>-reg.sock` (for Longhorn, it's `io.rancher.longhorn-reg.sock`) on kubelet path `<root-dir>/plugins`.
+   
+   **For k8s v1.11**
+   
+     Kubelet will find both sockets on kubelet path `/var/lib/kubelet/plugins`.
+   
+3. By default, Longhorn CSI driver create and expose these 2 sock files on host path `/var/lib/kubelet/plugins`.
+
+4. Then kubelet cannot find `<drivername>-reg.sock`, so CSI driver doesn't work.
+
+5. Furthermore, kubelet will instruct CSI plugin to mount Longhorn volume on `<root-dir>/pods/<pod-name>/volumes/kubernetes.io~csi/<volume-name>/mount`.
+
+   But this path inside CSI plugin container won't be binded mount on host path. And the mount operation for Longhorn volume is meaningless.
+   
+   Hence Kubernetes cannot connect to Longhorn using CSI driver.
+
+## Reference
+https://github.com/kubernetes-csi/driver-registrar