diff --git a/README.md b/README.md index 2fd3092..cdf5a34 100644 --- a/README.md +++ b/README.md @@ -264,6 +264,7 @@ Longhorn will always try to maintain at least given number of healthy replicas f ### [Google Kubernetes Engine](./docs/gke.md) ### [Upgrade](./docs/upgrade.md) ### [Deal with Kubernetes node failure](./docs/node-failure.md) +### [Use CSI driver on RancherOS + RKE](./docs/rancheros.md) ## Troubleshooting You can click `Generate Support Bundle` link at the bottom of the UI to download a zip file contains Longhorn related configuration and logs. diff --git a/docs/rancheros.md b/docs/rancheros.md new file mode 100644 index 0000000..3ce64f7 --- /dev/null +++ b/docs/rancheros.md @@ -0,0 +1,64 @@ +# Longhorn CSI on RancherOS + RKE + +## Requirements + 1. Kubernetes v1.11 or higher. + 2. Longhorn v0.4.1 or higher. + 3. RancherOS Ubuntu console. + + +## Instruction +### For Kubernetes v1.11 only + The following step is not needed for Kubernetes v1.12+. + + Add extra_binds for kubelet in RKE `cluster.yml`: + ``` + services: + kubelet: + extra_binds: + - "/opt/rke/var/lib/kubelet/plugins:/var/lib/kubelet/plugins" + ``` + +### For each node: + #### 1. Switch to ubuntu console + + `sudo ros console switch ubuntu`, then type `y` + + #### 2. Install open-iscsi for each node. + ``` + sudo su + apt update + apt install -y open-iscsi + ``` + #### 3. Modify configuration for iscsi. + + 1. Open config file `/etc/iscsi/iscsid.conf` + 2. Comment `iscsid.startup = /bin/systemctl start iscsid.socket` + 3. Uncomment `iscsid.startup = /sbin/iscsid` + + + +## Background +CSI doesn't work with RancherOS + RKE before Longhorn v0.4.1. The reason is: + +1. RancherOS sets argument `root-dir=/opt/rke/var/lib/kubelet` for kubelet, , which is different from the default value `/var/lib/kubelet`. + +2. **For k8s v1.12+** + + Kubelet will detect the `csi.sock` according to argument `<--kubelet-registration-path>` passed in by Kubernetes CSI driver-registrar, and `-reg.sock` (for Longhorn, it's `io.rancher.longhorn-reg.sock`) on kubelet path `/plugins`. + + **For k8s v1.11** + + Kubelet will find both sockets on kubelet path `/var/lib/kubelet/plugins`. + +3. By default, Longhorn CSI driver create and expose these 2 sock files on host path `/var/lib/kubelet/plugins`. + +4. Then kubelet cannot find `-reg.sock`, so CSI driver doesn't work. + +5. Furthermore, kubelet will instruct CSI plugin to mount Longhorn volume on `/pods//volumes/kubernetes.io~csi//mount`. + + But this path inside CSI plugin container won't be binded mount on host path. And the mount operation for Longhorn volume is meaningless. + + Hence Kubernetes cannot connect to Longhorn using CSI driver. + +## Reference +https://github.com/kubernetes-csi/driver-registrar \ No newline at end of file