add CoreOS support and rename rancheros.md to csi-config.md
This commit is contained in:
Shuo Wu
Sheng Yang
parent
9a17007661
commit
15f0b9c26b
@ -264,7 +264,7 @@ Longhorn will always try to maintain at least given number of healthy replicas f
|
|||||||
### [Google Kubernetes Engine](./docs/gke.md)
|
### [Google Kubernetes Engine](./docs/gke.md)
|
||||||
### [Upgrade](./docs/upgrade.md)
|
### [Upgrade](./docs/upgrade.md)
|
||||||
### [Deal with Kubernetes node failure](./docs/node-failure.md)
|
### [Deal with Kubernetes node failure](./docs/node-failure.md)
|
||||||
### [Use CSI driver on RancherOS + RKE](./docs/rancheros.md)
|
### [Use CSI driver on RancherOS/CoreOS + RKE or K3S](./docs/csi-config.md)
|
||||||
### [Restore a backup to an image file](./docs/restore-to-file.md)
|
### [Restore a backup to an image file](./docs/restore-to-file.md)
|
||||||
|
|
||||||
## Troubleshooting
|
## Troubleshooting
|
||||||
|
|||||||
119
docs/csi-config.md
Normal file
119
docs/csi-config.md
Normal file
@ -0,0 +1,119 @@
|
|||||||
|
# Longhorn CSI on RancherOS/CoreOS + RKE or K3S
|
||||||
|
|
||||||
|
## Requirements
|
||||||
|
1. Kubernetes v1.11 or higher.
|
||||||
|
2. Longhorn v0.4.1 or higher.
|
||||||
|
3. For RancherOS only: Ubuntu console.
|
||||||
|
|
||||||
|
|
||||||
|
## Instruction
|
||||||
|
### For Kubernetes v1.11 only
|
||||||
|
The following step is not needed for Kubernetes v1.12+.
|
||||||
|
|
||||||
|
Add extra_binds for kubelet in RKE `cluster.yml`:
|
||||||
|
```
|
||||||
|
services:
|
||||||
|
kubelet:
|
||||||
|
extra_binds:
|
||||||
|
- "/opt/rke/var/lib/kubelet/plugins:/var/lib/kubelet/plugins"
|
||||||
|
```
|
||||||
|
|
||||||
|
### For each node:
|
||||||
|
#### RancherOS:
|
||||||
|
##### 1. Switch to ubuntu console
|
||||||
|
|
||||||
|
`sudo ros console switch ubuntu`, then type `y`
|
||||||
|
|
||||||
|
##### 2. Install open-iscsi for each node.
|
||||||
|
```
|
||||||
|
sudo apt update
|
||||||
|
sudo apt install -y open-iscsi
|
||||||
|
```
|
||||||
|
##### 3. Modify configuration for iscsi.
|
||||||
|
|
||||||
|
1. Open config file `/etc/iscsi/iscsid.conf`
|
||||||
|
2. Comment `iscsid.startup = /bin/systemctl start iscsid.socket`
|
||||||
|
3. Uncomment `iscsid.startup = /sbin/iscsid`
|
||||||
|
|
||||||
|
#### CoreOS:
|
||||||
|
|
||||||
|
##### 1. If you want to enable iSCSI daemon automatically at boot, you need to enable the systemd service:
|
||||||
|
|
||||||
|
```
|
||||||
|
sudo su
|
||||||
|
systemctl enable iscsid
|
||||||
|
reboot
|
||||||
|
```
|
||||||
|
|
||||||
|
##### 2. Or just start the iSCSI daemon for the current session:
|
||||||
|
|
||||||
|
```
|
||||||
|
sudo su
|
||||||
|
systemctl start iscsid
|
||||||
|
```
|
||||||
|
|
||||||
|
#### K3S:
|
||||||
|
No extra configuration is needed.
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
### Common issues
|
||||||
|
#### Failed to get arg root-dir: Cannot get kubelet root dir, no related proc for root-dir detection ...
|
||||||
|
|
||||||
|
This error is due to Longhorn cannot find out where is the root dir setup for Kubelet, so the CSI plugin installation would fail.
|
||||||
|
|
||||||
|
User can override the root-dir detection by manually setting argument `kubelet-root-dir` here:
|
||||||
|
https://github.com/rancher/longhorn/blob/master/deploy/longhorn.yaml#L329
|
||||||
|
|
||||||
|
**How to find `root-dir`?**
|
||||||
|
|
||||||
|
**For RancherOS/CoreOS**
|
||||||
|
|
||||||
|
Run `ps aux | grep kubelet` and get argument `--root-dir` on host node.
|
||||||
|
|
||||||
|
e.g.
|
||||||
|
```
|
||||||
|
$ ps aux | grep kubelet
|
||||||
|
root 3755 4.4 2.9 744404 120020 ? Ssl 00:45 0:02 kubelet --root-dir=/opt/rke/var/lib/kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins
|
||||||
|
```
|
||||||
|
You will find `root-dir` in the cmdline of proc `kubelet`. By default it is `/opt/rke/var/lil/kubelet` if you are using RKE.
|
||||||
|
|
||||||
|
**For K3S**
|
||||||
|
|
||||||
|
Run `ps aux | grep k3s` and get argument `--data-dir` or `-d` on k3s node.
|
||||||
|
|
||||||
|
e.g.
|
||||||
|
```
|
||||||
|
$ ps uax | grep k3s
|
||||||
|
root 4160 0.0 0.0 51420 3948 pts/0 S+ 00:55 0:00 sudo /usr/local/bin/k3s server --data-dir /opt/test/kubelet
|
||||||
|
root 4161 49.0 4.0 259204 164292 pts/0 Sl+ 00:55 0:04 /usr/local/bin/k3s server --data-dir /opt/test/kubelet
|
||||||
|
```
|
||||||
|
You will find `data-dir` in the cmdline of proc `k3s`. By default it is not set and `/var/lib/rancher/k3s` will be used. Then joining `data-dir` with `/agent/kubelet` you will get the `root-dir`. So the default `root-dir` is `/var/lib/rancher/k3s/agent/kubelet`.
|
||||||
|
|
||||||
|
|
||||||
|
## Background
|
||||||
|
CSI doesn't work with RancherOS/CoreOS + RKE before Longhorn v0.4.1. The reason is:
|
||||||
|
|
||||||
|
1. RKE sets argument `root-dir=/opt/rke/var/lib/kubelet` for kubelet in the case of RancherOS or CoreOS, which is different from the default value `/var/lib/kubelet`.
|
||||||
|
|
||||||
|
2. **For k8s v1.12+**
|
||||||
|
|
||||||
|
Kubelet will detect the `csi.sock` according to argument `<--kubelet-registration-path>` passed in by Kubernetes CSI driver-registrar, and `<drivername>-reg.sock` (for Longhorn, it's `io.rancher.longhorn-reg.sock`) on kubelet path `<root-dir>/plugins`.
|
||||||
|
|
||||||
|
**For k8s v1.11**
|
||||||
|
|
||||||
|
Kubelet will find both sockets on kubelet path `/var/lib/kubelet/plugins`.
|
||||||
|
|
||||||
|
3. By default, Longhorn CSI driver create and expose these 2 sock files on host path `/var/lib/kubelet/plugins`.
|
||||||
|
|
||||||
|
4. Then kubelet cannot find `<drivername>-reg.sock`, so CSI driver doesn't work.
|
||||||
|
|
||||||
|
5. Furthermore, kubelet will instruct CSI plugin to mount Longhorn volume on `<root-dir>/pods/<pod-name>/volumes/kubernetes.io~csi/<volume-name>/mount`.
|
||||||
|
|
||||||
|
But this path inside CSI plugin container won't be binded mount on host path. And the mount operation for Longhorn volume is meaningless.
|
||||||
|
|
||||||
|
Hence Kubernetes cannot connect to Longhorn using CSI driver.
|
||||||
|
|
||||||
|
## Reference
|
||||||
|
https://github.com/kubernetes-csi/driver-registrar
|
||||||
|
|
||||||
|
https://coreos.com/os/docs/latest/iscsi.html
|
||||||
@ -1,63 +0,0 @@
|
|||||||
# Longhorn CSI on RancherOS + RKE
|
|
||||||
|
|
||||||
## Requirements
|
|
||||||
1. Kubernetes v1.11 or higher.
|
|
||||||
2. Longhorn v0.4.1 or higher.
|
|
||||||
3. RancherOS Ubuntu console.
|
|
||||||
|
|
||||||
|
|
||||||
## Instruction
|
|
||||||
### For Kubernetes v1.11 only
|
|
||||||
The following step is not needed for Kubernetes v1.12+.
|
|
||||||
|
|
||||||
Add extra_binds for kubelet in RKE `cluster.yml`:
|
|
||||||
```
|
|
||||||
services:
|
|
||||||
kubelet:
|
|
||||||
extra_binds:
|
|
||||||
- "/opt/rke/var/lib/kubelet/plugins:/var/lib/kubelet/plugins"
|
|
||||||
```
|
|
||||||
|
|
||||||
### For each node:
|
|
||||||
#### 1. Switch to ubuntu console
|
|
||||||
|
|
||||||
`sudo ros console switch ubuntu`, then type `y`
|
|
||||||
|
|
||||||
#### 2. Install open-iscsi for each node.
|
|
||||||
```
|
|
||||||
sudo apt update
|
|
||||||
sudo apt install -y open-iscsi
|
|
||||||
```
|
|
||||||
#### 3. Modify configuration for iscsi.
|
|
||||||
|
|
||||||
1. Open config file `/etc/iscsi/iscsid.conf`
|
|
||||||
2. Comment `iscsid.startup = /bin/systemctl start iscsid.socket`
|
|
||||||
3. Uncomment `iscsid.startup = /sbin/iscsid`
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
## Background
|
|
||||||
CSI doesn't work with RancherOS + RKE before Longhorn v0.4.1. The reason is:
|
|
||||||
|
|
||||||
1. RancherOS sets argument `root-dir=/opt/rke/var/lib/kubelet` for kubelet, , which is different from the default value `/var/lib/kubelet`.
|
|
||||||
|
|
||||||
2. **For k8s v1.12+**
|
|
||||||
|
|
||||||
Kubelet will detect the `csi.sock` according to argument `<--kubelet-registration-path>` passed in by Kubernetes CSI driver-registrar, and `<drivername>-reg.sock` (for Longhorn, it's `io.rancher.longhorn-reg.sock`) on kubelet path `<root-dir>/plugins`.
|
|
||||||
|
|
||||||
**For k8s v1.11**
|
|
||||||
|
|
||||||
Kubelet will find both sockets on kubelet path `/var/lib/kubelet/plugins`.
|
|
||||||
|
|
||||||
3. By default, Longhorn CSI driver create and expose these 2 sock files on host path `/var/lib/kubelet/plugins`.
|
|
||||||
|
|
||||||
4. Then kubelet cannot find `<drivername>-reg.sock`, so CSI driver doesn't work.
|
|
||||||
|
|
||||||
5. Furthermore, kubelet will instruct CSI plugin to mount Longhorn volume on `<root-dir>/pods/<pod-name>/volumes/kubernetes.io~csi/<volume-name>/mount`.
|
|
||||||
|
|
||||||
But this path inside CSI plugin container won't be binded mount on host path. And the mount operation for Longhorn volume is meaningless.
|
|
||||||
|
|
||||||
Hence Kubernetes cannot connect to Longhorn using CSI driver.
|
|
||||||
|
|
||||||
## Reference
|
|
||||||
https://github.com/kubernetes-csi/driver-registrar
|
|
||||||
Loading…
Reference in New Issue
Block a user