ivampiresp/longhorn
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Document upgrade path from v0.1, v0.2 to v0.3

Browse Source
This commit is contained in:
James Oliver 2018-08-17 17:19:22 -07:00 committed by Sheng Yang
parent bb1ac25afd
commit 0dc32d7676
2 changed files with 180 additions and 80 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

14
README.md
View File

@ -66,19 +66,23 @@ Both `kube-apiserver` and `kubelet` should have `--feature-gates=MountPropagatio
2. Google GKE: `/home/kubernetes/flexvolume` 2. Google GKE: `/home/kubernetes/flexvolume`
3. For other distro, please find the correct directory by running `ps aux|grep kubelet` on the host and check the `--volume-plugin-dir` parameter. If there is none, it would be the default value `/usr/libexec/kubernetes/kubelet-plugins/volume/exec/` . 3. For other distro, please find the correct directory by running `ps aux|grep kubelet` on the host and check the `--volume-plugin-dir` parameter. If there is none, it would be the default value `/usr/libexec/kubernetes/kubelet-plugins/volume/exec/` .
# Upgrading
For instructions on how to upgrade Longhorn v0.1 or v0.2 to v0.3, [see this document](docs/upgrade.md#upgrade).
# Deployment # Deployment
Create the deployment of Longhorn in your Kubernetes cluster is easy. Create the deployment of Longhorn in your Kubernetes cluster is easy.
If you're using Rancher RKE, or other distro with Kubernetes v1.10+ and Mount Propagation enabled, you can just do: If you're using Rancher RKE, or other distro with Kubernetes v1.10+ and Mount Propagation enabled, you can just do:
``` ```
kubectl create -f https://raw.githubusercontent.com/rancher/longhorn/v0.3-rc/deploy/longhorn.yaml kubectl apply -f https://raw.githubusercontent.com/rancher/longhorn/v0.3-rc/deploy/longhorn.yaml
``` ```
If you're using Flexvolume driver with other Kubernetes Distro, replace the value of $FLEXVOLUME_DIR in the following command with your own Flexvolume Directory as specified above. If you're using Flexvolume driver with other Kubernetes Distro, replace the value of $FLEXVOLUME_DIR in the following command with your own Flexvolume Directory as specified above.
``` ```
FLEXVOLUME_DIR="/home/kubernetes/flexvolume/" FLEXVOLUME_DIR="/home/kubernetes/flexvolume/"
curl -s https://raw.githubusercontent.com/rancher/longhorn/v0.3-rc/deploy/longhorn.yaml|sed "s#^\( *\)value: \"/var/lib/kubelet/volumeplugins\"#\1value: \"${FLEXVOLUME_DIR}\"#g" > longhorn.yaml curl -s https://raw.githubusercontent.com/rancher/longhorn/v0.3-rc/deploy/longhorn.yaml|sed "s#^\( *\)value: \"/var/lib/kubelet/volumeplugins\"#\1value: \"${FLEXVOLUME_DIR}\"#g" > longhorn.yaml
kubectl create -f longhorn.yaml kubectl apply -f longhorn.yaml
``` ```
For Google Kubernetes Engine (GKE) users, see [here](#google-kubernetes-engine) before proceed. For Google Kubernetes Engine (GKE) users, see [here](#google-kubernetes-engine) before proceed.
@ -145,12 +149,12 @@ Longhorn provides persistent volume directly to Kubernetes through one of the Lo
Use following command to create a default Longhorn StorageClass named `longhorn`. Use following command to create a default Longhorn StorageClass named `longhorn`.
``` ```
kubectl create -f https://raw.githubusercontent.com/rancher/longhorn/v0.3-rc/examples/storageclass.yaml kubectl apply -f https://raw.githubusercontent.com/rancher/longhorn/v0.3-rc/examples/storageclass.yaml
``` ```
Now you can create a pod using Longhorn like this: Now you can create a pod using Longhorn like this:
``` ```
kubectl create -f https://raw.githubusercontent.com/rancher/longhorn/v0.3-rc/examples/pvc.yaml kubectl apply -f https://raw.githubusercontent.com/rancher/longhorn/v0.3-rc/examples/pvc.yaml
``` ```
The yaml contains two parts: The yaml contains two parts:
@ -214,7 +218,7 @@ We provides two testing purpose backupstore based on NFS server and Minio S3 ser
Use following command to setup a Minio S3 server for BackupStore after `longhorn-system` was created. Use following command to setup a Minio S3 server for BackupStore after `longhorn-system` was created.
``` ```
kubectl create -f https://raw.githubusercontent.com/rancher/longhorn/v0.3-rc/deploy/backupstores/minio-backupstore.yaml kubectl apply -f https://raw.githubusercontent.com/rancher/longhorn/v0.3-rc/deploy/backupstores/minio-backupstore.yaml
``` ```
Now set `Settings/General/BackupTarget` to Now set `Settings/General/BackupTarget` to

242
docs/upgrade.md
View File

@ -1,100 +1,196 @@
# Upgrade # Upgrade
Here we would cover how to upgrade from Longhorn v0.2 to Longhorn v0.3 release. Here we cover how to upgrade to Longhorn v0.3 from all previous releases.
## Backup your existing data ## Backup Existing Volumes
1. It's recommended to create a latest backup for every volume to the backupstore before upgrade.
2. Make sure no volume is in degraded or faulted state. It's recommended to create a recent backup of every volume to the backupstore
3. Shutdown related Kubernetes pods. Detach all the volumes. Make sure all the volumes are detached before proceeding. before upgrade.
4. Backup CRD yaml to local directory:
Create an on-cluster backupstore if you haven't already. We'll use NFS in this
example.
``` ```
kubectl -n longhorn-system get volumes.longhorn.rancher.io -o yaml > longhorn-v0.2-backup-volumes.yaml kubectl apply -f https://raw.githubusercontent.com/rancher/longhorn/v0.3-rc/deploy/backupstores/nfs-backupstore.yaml
kubectl -n longhorn-system get engines.longhorn.rancher.io -o yaml > longhorn-v0.2-backup-engines.yaml
kubectl -n longhorn-system get replicas.longhorn.rancher.io -o yaml > longhorn-v0.2-backup-replicas.yaml
kubectl -n longhorn-system get settings.longhorn.rancher.io -o yaml > longhorn-v0.2-backup-settings.yaml
``` ```
5. Noted the value of BackupTarget in the setting. The user would need to reset after upgrade.
## Upgrade from v0.2 to v0.3 On Settings page, set Backup Target to
`nfs://longhorn-test-nfs-svc.default:/opt/backupstore` and click `Save`.
Please be aware that the upgrade will incur API downtime. Navigate to each volume detail page and click `Take Snapshot`. Click the new
snapshot and click `Backup`.
### 1. Remove the old manager ## Check For Issues
Make sure no volume is in degraded or faulted state. Wait for degraded
volumes to heal and delete/restore faulted volumes before proceeding.
## Detach Volumes
Shutdown all Kubernetes Pods using Longhorn volumes in order to detach the
volumes. The easiest way to achieve this is by deleting all workloads. If
this is not desirable, some workloads may be suspended. We will cover how
each workload can be modified to shut down its pods.
### CronJob
Edit the cronjob with `kubectl edit cronjob/<name>`.
Set `.spec.suspend` to `true`.
Wait for any currently executing jobs to complete, or terminate them by
deleting relevant pods.
### DaemonSet
Delete the daemonset with `kubectl delete ds/<name>`.
There is no way to suspend this workload.
### Deployment
Edit the deployment with `kubectl edit deploy/<name>`.
Set `.spec.replicas` to `0`.
### Job
Consider allowing the single-run job to complete.
Otherwise, delete the job with `kubectl delete job/<name>`.
### Pod
Delete the pod with `kubectl delete pod/<name>`.
There is no way to suspend a pod not managed by a workload controller.
### ReplicaSet
Edit the replicaset with `kubectl edit replicaset/<name>`.
Set `.spec.replicas` to `0`.
### ReplicationController
Edit the replicationcontroller with `kubectl edit rc/<name>`.
Set `.spec.replicas` to `0`.
### StatefulSet
Edit the statefulset with `kubectl edit statefulset/<name>`.
Set `.spec.replicas` to `0`.
Detach all remaining volumes from Longhorn UI. These volumes were most likely
created and attached outside of Kubernetes via Longhorn UI or REST API.
## Uninstall Old Version
Make note of `BackupTarget` on the `Setting` page. You will need to manually
set `BackupTarget` after upgrading from either v0.1 or v0.2.
Delete Longhorn components.
For Longhorn `v0.1`:
```
kubectl delete -f https://raw.githubusercontent.com/llparse/longhorn/v0.1/deploy/uninstall-for-upgrade.yaml
```
For Longhorn `v0.2`:
``` ```
kubectl delete -f https://raw.githubusercontent.com/rancher/longhorn/v0.2/deploy/uninstall-for-upgrade.yaml kubectl delete -f https://raw.githubusercontent.com/rancher/longhorn/v0.2/deploy/uninstall-for-upgrade.yaml
``` ```
### 2. Install the new manager If both commands returned `Not found` for all components, Longhorn is probably
deployed in a different namespace. Determine which namespace is in use and
We will use `kubectl apply` instead of `kubectl create` to install the new version of the manager. adjust `NAMESPACE` accordingly:
If you're using Rancher RKE, or other distro with Kubernetes v1.10+ and Mount Propagation enabled, you can just do:
``` ```
kubectl apply -f https://raw.githubusercontent.com/rancher/longhorn/v0.3-rc/deploy/longhorn.yaml NAMESPACE=longhorn-custom-ns
``` curl -sSfL https://raw.githubusercontent.com/rancher/longhorn/v0.1/deploy/uninstall-for-upgrade.yaml|sed "s#^\( *\)namespace: longhorn#\1namespace: ${NAMESPACE}#g" > longhorn.yaml
If you're using Flexvolume driver with other Kubernetes Distro, replace the value of $FLEXVOLUME_DIR in the following command with your own Flexvolume Directory as specified above. kubectl delete -f longhorn.yaml
```
FLEXVOLUME_DIR="/home/kubernetes/flexvolume/"
curl -s https://raw.githubusercontent.com/rancher/longhorn/v0.3-rc/deploy/longhorn.yaml|sed "s#^\( *\)value: \"/var/lib/kubelet/volumeplugins\"#\1value: \"${FLEXVOLUME_DIR}\"#g" > longhorn.yaml
kubectl apply -f longhorn.yaml
``` ```
For Google Kubernetes Engine (GKE) users, see [here](./gke.md) before proceed. ## Backup Longhorn System
Longhorn Manager and Longhorn Driver will be deployed as daemonsets in a separate namespace called `longhorn-system`, as you can see in the yaml file. Backup Longhorn CRD yaml to local directory.
When you see those pods has started correctly as follows, you've deployed the Longhorn successfully. ### v0.1
Check your backups to make sure Longhorn was running in namespace `longhorn`.
Deployed with CSI driver:
``` ```
# kubectl -n longhorn-system get pod NAMESPACE=longhorn
NAME READY STATUS RESTARTS AGE kubectl -n ${NAMESPACE} get volumes.longhorn.rancher.io -o yaml > longhorn-v0.1-backup-volumes.yaml
csi-attacher-0 1/1 Running 0 6h kubectl -n ${NAMESPACE} get engines.longhorn.rancher.io -o yaml > longhorn-v0.1-backup-engines.yaml
csi-provisioner-0 1/1 Running 0 6h kubectl -n ${NAMESPACE} get replicas.longhorn.rancher.io -o yaml > longhorn-v0.1-backup-replicas.yaml
engine-image-ei-57b85e25-8v65d 1/1 Running 0 7d kubectl -n ${NAMESPACE} get settings.longhorn.rancher.io -o yaml > longhorn-v0.1-backup-settings.yaml
engine-image-ei-57b85e25-gjjs6 1/1 Running 0 7d
engine-image-ei-57b85e25-t2787 1/1 Running 0 7d
longhorn-csi-plugin-4cpk2 2/2 Running 0 6h
longhorn-csi-plugin-ll6mq 2/2 Running 0 6h
longhorn-csi-plugin-smlsh 2/2 Running 0 6h
longhorn-driver-deployer-7b5bdcccc8-fbncl 1/1 Running 0 6h
longhorn-manager-7x8x8 1/1 Running 0 6h
longhorn-manager-8kqf4 1/1 Running 0 6h
longhorn-manager-kln4h 1/1 Running 0 6h
longhorn-ui-f849dcd85-cgkgg 1/1 Running 0 5d
```
Or with Flexvolume driver
```
# kubectl -n longhorn-system get pod
NAME READY STATUS RESTARTS AGE
engine-image-ei-57b85e25-8v65d 1/1 Running 0 7d
engine-image-ei-57b85e25-gjjs6 1/1 Running 0 7d
engine-image-ei-57b85e25-t2787 1/1 Running 0 7d
longhorn-driver-deployer-5469b87b9c-b9gm7 1/1 Running 0 2h
longhorn-flexvolume-driver-lth5g 1/1 Running 0 2h
longhorn-flexvolume-driver-tpqf7 1/1 Running 0 2h
longhorn-flexvolume-driver-v9mrj 1/1 Running 0 2h
longhorn-manager-7x8x8 1/1 Running 0 9h
longhorn-manager-8kqf4 1/1 Running 0 9h
longhorn-manager-kln4h 1/1 Running 0 9h
longhorn-ui-f849dcd85-cgkgg 1/1 Running 0 5d
``` ```
### 3. Upgrade Engine Images and set BackupTarget ### v0.2
Check your backups to make sure Longhorn was running in namespace
`longhorn-system`.
```
NAMESPACE=longhorn-system
kubectl -n ${NAMESPACE} get volumes.longhorn.rancher.io -o yaml > longhorn-v0.2-backup-volumes.yaml
kubectl -n ${NAMESPACE} get engines.longhorn.rancher.io -o yaml > longhorn-v0.2-backup-engines.yaml
kubectl -n ${NAMESPACE} get replicas.longhorn.rancher.io -o yaml > longhorn-v0.2-backup-replicas.yaml
kubectl -n ${NAMESPACE} get settings.longhorn.rancher.io -o yaml > longhorn-v0.2-backup-settings.yaml
```
1. Wait until the UI is up. ## Delete CRDs in Different Namespace
2. Set the BackupTarget in the setting to the same value as before upgrade.
3. Make all the volumes are all detached. This is only required for Rancher users running Longhorn App `v0.1`. Delete all
4. Select all the volumes using batch selection. Click batch operation button CRDs from your namespace which is probably `longhorn`.
```
NAMESPACE=longhorn
kubectl -n ${NAMESPACE} get volumes.longhorn.rancher.io -o yaml | sed "s/\- longhorn.rancher.io//g" | kubectl apply -f -
kubectl -n ${NAMESPACE} get engines.longhorn.rancher.io -o yaml | sed "s/\- longhorn.rancher.io//g" | kubectl apply -f -
kubectl -n ${NAMESPACE} get replicas.longhorn.rancher.io -o yaml | sed "s/\- longhorn.rancher.io//g" | kubectl apply -f -
kubectl -n ${NAMESPACE} get settings.longhorn.rancher.io -o yaml | sed "s/\- longhorn.rancher.io//g" | kubectl apply -f -
kubectl -n ${NAMESPACE} delete volumes.longhorn.rancher.io --all
kubectl -n ${NAMESPACE} delete engines.longhorn.rancher.io --all
kubectl -n ${NAMESPACE} delete replicas.longhorn.rancher.io --all
kubectl -n ${NAMESPACE} delete settings.longhorn.rancher.io --all
```
## Install Longhorn v0.3
### Rancher 2.x
For Rancher users who are running Longhorn v0.1, delete the Longhorn App from
`Catalog Apps` screen in Rancher UI. *Do not click the upgrade button.* Launch
Longhorn App template version `0.3.0-rc4`.
### Other Kubernetes Distro
For Longhorn v0.2 users who are not using Rancher, follow
[the official Longhorn Deployment instructions](../README.md#deployment).
## Restore Longhorn System
This step is only required for Rancher users running Longhorn App `v0.1`.
```
NAMESPACE=longhorn-system
sed "s#^\( *\)namespace: .*#\1namespace: ${NAMESPACE}#g" longhorn-v0.1-backup-settings.yaml | kubectl apply -f -
sed "s#^\( *\)namespace: .*#\1namespace: ${NAMESPACE}#g" longhorn-v0.1-backup-replicas.yaml | kubectl apply -f -
sed "s#^\( *\)namespace: .*#\1namespace: ${NAMESPACE}#g" longhorn-v0.1-backup-engines.yaml | kubectl apply -f -
sed "s#^\( *\)namespace: .*#\1namespace: ${NAMESPACE}#g" longhorn-v0.1-backup-volumes.yaml | kubectl apply -f -
```
## Access UI and Set BackupTarget
Wait until the longhorn-ui pod is `Running`:
```
kubectl -n longhorn-system get pod -w
```
[Access the UI](../README.md#access-the-ui).
On `Setting > General`, set `Backup Target` to the backup target used in
the previous version. In our example, this is
`nfs://longhorn-test-nfs-svc.default:/opt/backupstore`.
## Upgrade Engine Images
Ensure all volumes are detached. If any are still attached, detach them now
and wait until they are in `Detached` state.
Select all the volumes using batch selection. Click batch operation button
`Upgrade Engine`, choose the only engine image available in the list. It's `Upgrade Engine`, choose the only engine image available in the list. It's
the default engine shipped with the manager for this release. the default engine shipped with the manager for this release.
5. Now attach the volume one by one, to see if the volume works correctly.
## Attach Volumes
Now we will resume all workloads by reversing the changes we made to detach
the volumes. Any volume not part of a K8s workload or pod must be attached
manually.
## Note ## Note
Upgrade is always tricky. Keep backups for the volumes are critical. Upgrade is always tricky. Keeping recent backups for volumes is critical.
If you have any issues, please reported it at
https://github.com/rancher/longhorn/issues , with your backup yaml files as well
as manager logs.
If you have any issues, please report it at
https://github.com/rancher/longhorn/issues and include your backup yaml files
as well as manager logs.