diff --git a/docs/csi-config.md b/docs/csi-config.md index 8dbce88..377b797 100644 --- a/docs/csi-config.md +++ b/docs/csi-config.md @@ -1,57 +1,11 @@ -# Longhorn CSI on RancherOS/CoreOS + RKE or K3S +# Longhorn CSI on K3S ## Requirements 1. Kubernetes v1.11 or higher. 2. Longhorn v0.4.1 or higher. - 3. For RancherOS only: Ubuntu console. ## Instruction -### For RancherOS/CoreOS + Kubernetes v1.11 only - The following step is not needed for Kubernetes v1.12 and above. - - Add extra_binds for kubelet in RKE `cluster.yml`: - ``` - services: - kubelet: - extra_binds: - - "/opt/rke/var/lib/kubelet/plugins:/var/lib/kubelet/plugins" - ``` - -### For each node: -#### RancherOS: - ##### 1. Switch to ubuntu console - - `sudo ros console switch ubuntu`, then type `y` - - ##### 2. Install open-iscsi for each node. - ``` - sudo apt update - sudo apt install -y open-iscsi - ``` - ##### 3. Modify configuration for iscsi. - - 1. Open config file `/etc/iscsi/iscsid.conf` - 2. Comment `iscsid.startup = /bin/systemctl start iscsid.socket` - 3. Uncomment `iscsid.startup = /sbin/iscsid` - -#### CoreOS: - - ##### 1. If you want to enable iSCSI daemon automatically at boot, you need to enable the systemd service: - - ``` - sudo su - systemctl enable iscsid - reboot - ``` - - ##### 2. Or just start the iSCSI daemon for the current session: - - ``` - sudo su - systemctl start iscsid - ``` - #### K3S: ##### 1. For Longhorn v0.7.0 and above Longhorn v0.7.0 and above support k3s v0.10.0 and above only by default. @@ -73,21 +27,6 @@ This error is due to Longhorn cannot detect where is the root dir setup for Kube User can override the root-dir detection by manually setting argument `kubelet-root-dir` here: https://github.com/rancher/longhorn/blob/master/deploy/longhorn.yaml#L329 -#### How to find `root-dir`? - -**For RancherOS/CoreOS** - -Run `ps aux | grep kubelet` and get argument `--root-dir` on host node. - -e.g. -``` -$ ps aux | grep kubelet -root 3755 4.4 2.9 744404 120020 ? Ssl 00:45 0:02 kubelet --root-dir=/opt/rke/var/lib/kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins -``` -You will find `root-dir` in the cmdline of proc `kubelet`. If it's not set, the default value `/var/lib/kubelet` would be used. In the case of RancherOS/CoreOS, the root-dir would be `/opt/rke/var/lib/kubelet` as shown above. - -If kubelet is using a configuration file, you would need to check the configuration file to locate the `root-dir` parameter. - **For K3S v0.10.0-** Run `ps aux | grep k3s` and get argument `--data-dir` or `-d` on k3s server node. @@ -107,33 +46,8 @@ If K3S is using a configuration file, you would need to check the configuration It is always `/var/lib/kubelet` ## Background -#### CSI doesn't work with RancherOS/CoreOS + RKE before Longhorn v0.4.1 -The reason is: - -1. RKE sets argument `root-dir=/opt/rke/var/lib/kubelet` for kubelet in the case of RancherOS or CoreOS, which is different from the default value `/var/lib/kubelet`. - -2. **For k8s v1.12 and above** - - Kubelet will detect the `csi.sock` according to argument `<--kubelet-registration-path>` passed in by Kubernetes CSI driver-registrar, and `-reg.sock` (for Longhorn, it's `io.rancher.longhorn-reg.sock`) on kubelet path `/plugins`. - - **For k8s v1.11** - - Kubelet will find both sockets on kubelet path `/var/lib/kubelet/plugins`. - -3. By default, Longhorn CSI driver create and expose these 2 sock files on host path `/var/lib/kubelet/plugins`. - -4. Then kubelet cannot find `-reg.sock`, so CSI driver doesn't work. - -5. Furthermore, kubelet will instruct CSI plugin to mount Longhorn volume on `/pods//volumes/kubernetes.io~csi//mount`. - - But this path inside CSI plugin container won't be binded mount on host path. And the mount operation for Longhorn volume is meaningless. - - Hence Kubernetes cannot connect to Longhorn using CSI driver. - #### Longhorn versions before v0.7.0 don't work on K3S v0.10.0 or above K3S now sets its kubelet directory to `/var/lib/kubelet`. See [the K3S release comment](https://github.com/rancher/k3s/releases/tag/v0.10.0) for details. ## Reference https://github.com/kubernetes-csi/driver-registrar - -https://coreos.com/os/docs/latest/iscsi.html