64 lines
2.1 KiB
Markdown
64 lines
2.1 KiB
Markdown
|
# Longhorn CSI on RancherOS + RKE
|
|||
|
|
|
|||
|
|
## Requirements
|
|||
|
|
1. Kubernetes v1.11 or higher.
|
|||
|
|
2. Longhorn v0.4.1 or higher.
|
|||
|
|
3. RancherOS Ubuntu console.
|
|||
|
|
|
|||
|
|
|
|||
|
|
## Instruction
|
|||
|
|
### For Kubernetes v1.11 only
|
|||
|
|
The following step is not needed for Kubernetes v1.12+.
|
|||
|
|
|
|||
|
|
Add extra_binds for kubelet in RKE `cluster.yml`:
|
|||
|
|
```
|
|||
|
|
services:
|
|||
|
|
kubelet:
|
|||
|
|
extra_binds:
|
|||
|
|
- "/opt/rke/var/lib/kubelet/plugins:/var/lib/kubelet/plugins"
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### For each node:
|
|||
|
|
#### 1. Switch to ubuntu console
|
|||
|
|
|
|||
|
|
`sudo ros console switch ubuntu`, then type `y`
|
|||
|
|
|
|||
|
|
#### 2. Install open-iscsi for each node.
|
|||
|
|
```
|
|||
|
|
sudo su
|
|||
|
|
apt update
|
|||
|
|
apt install -y open-iscsi
|
|||
|
|
```
|
|||
|
|
#### 3. Modify configuration for iscsi.
|
|||
|
|
|
|||
|
|
1. Open config file `/etc/iscsi/iscsid.conf`
|
|||
|
|
2. Comment `iscsid.startup = /bin/systemctl start iscsid.socket`
|
|||
|
|
3. Uncomment `iscsid.startup = /sbin/iscsid`
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
## Background
|
|||
|
|
CSI doesn't work with RancherOS + RKE before Longhorn v0.4.1. The reason is:
|
|||
|
|
|
|||
|
|
1. RancherOS sets argument `root-dir=/opt/rke/var/lib/kubelet` for kubelet, , which is different from the default value `/var/lib/kubelet`.
|
|||
|
|
|
|||
|
|
2. **For k8s v1.12+**
|
|||
|
|
|
|||
|
|
Kubelet will detect the `csi.sock` according to argument `<--kubelet-registration-path>` passed in by Kubernetes CSI driver-registrar, and `<drivername>-reg.sock` (for Longhorn, it's `io.rancher.longhorn-reg.sock`) on kubelet path `<root-dir>/plugins`.
|
|||
|
|
|
|||
|
|
**For k8s v1.11**
|
|||
|
|
|
|||
|
|
Kubelet will find both sockets on kubelet path `/var/lib/kubelet/plugins`.
|
|||
|
|
|
|||
|
|
3. By default, Longhorn CSI driver create and expose these 2 sock files on host path `/var/lib/kubelet/plugins`.
|
|||
|
|
|
|||
|
|
4. Then kubelet cannot find `<drivername>-reg.sock`, so CSI driver doesn't work.
|
|||
|
|
|
|||
|
|
5. Furthermore, kubelet will instruct CSI plugin to mount Longhorn volume on `<root-dir>/pods/<pod-name>/volumes/kubernetes.io~csi/<volume-name>/mount`.
|
|||
|
|
|
|||
|
|
But this path inside CSI plugin container won't be binded mount on host path. And the mount operation for Longhorn volume is meaningless.
|
|||
|
|
|
|||
|
|
Hence Kubernetes cannot connect to Longhorn using CSI driver.
|
|||
|
|
|
|||
|
|
## Reference
|
|||
|
|
https://github.com/kubernetes-csi/driver-registrar
|