Add LEP for csi snapshot support
Longhorn #304 Signed-off-by: Joshua Moody <joshua.moody@rancher.com>
This commit is contained in:
Joshua Moody
Sheng Yang
parent
93c0d6a8af
commit
9cb7bdc160
263
enhancements/20200904-csi-snapshot-support.md
Normal file
263
enhancements/20200904-csi-snapshot-support.md
Normal file
@ -0,0 +1,263 @@
|
|||||||
|
# CSI Snapshot Support
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
To allow users to create/restore/delete backups programmatically,
|
||||||
|
we want to add support for the csi snapshot mechanism.
|
||||||
|
This way existing tools can be used to create kubernetes `VolumeSnapshot` api resources,
|
||||||
|
which we can react to in the csi plugin.
|
||||||
|
|
||||||
|
### Related Issues
|
||||||
|
|
||||||
|
https://github.com/longhorn/longhorn/issues/304
|
||||||
|
https://github.com/longhorn/longhorn/issues/610
|
||||||
|
https://github.com/longhorn/longhorn/issues/1127
|
||||||
|
https://github.com/kubernetes/community/blob/master/contributors/design-proposals/storage/csi-snapshot.md
|
||||||
|
|
||||||
|
## Motivation
|
||||||
|
|
||||||
|
Provide user with programmatic access for backup operations via the standardized csi interface.
|
||||||
|
https://kubernetes.io/docs/concepts/storage/volume-snapshots/#provisioning-volume-snapshot
|
||||||
|
|
||||||
|
### Goals
|
||||||
|
|
||||||
|
- add csi snapshot support to our csi driver
|
||||||
|
|
||||||
|
### Non-goals
|
||||||
|
|
||||||
|
- VolumeBackup crd refactor
|
||||||
|
- Changes to the longhorn backup code
|
||||||
|
|
||||||
|
## Proposal
|
||||||
|
|
||||||
|
- support csi CreateSnapshot call
|
||||||
|
- support csi DeleteSnapshot call
|
||||||
|
- support snapshot as ContentSource for restoration during CreateVolume calls
|
||||||
|
|
||||||
|
### User Stories
|
||||||
|
|
||||||
|
Currently, it's hard for users to interact with the backup system programmatically,
|
||||||
|
after this enhancement the users will be able to use the standard kubernetes csi mechanisms for
|
||||||
|
backup creation / deletion and restoration of a new volume based on a backup.
|
||||||
|
|
||||||
|
### User Experience In Detail
|
||||||
|
|
||||||
|
#### Backup creation via VolumeSnapshot resource
|
||||||
|
|
||||||
|
The user can request a backup of a volume by creation of a kubernetes `VolumeSnapshot` object.
|
||||||
|
Example below for a volume named `test-vol`
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
apiVersion: snapshot.storage.k8s.io/v1beta1
|
||||||
|
kind: VolumeSnapshot
|
||||||
|
metadata:
|
||||||
|
name: test-snapshot-pvc
|
||||||
|
spec:
|
||||||
|
volumeSnapshotClassName: longhorn
|
||||||
|
source:
|
||||||
|
persistentVolumeClaimName: test-vol
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Restoration via VolumeSnapshot resource
|
||||||
|
|
||||||
|
The user can request the creation of a volume based on a prior created `VolumeSnapshot` object.
|
||||||
|
Example below for a volume named `test-vol-restore`
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
apiVersion: v1
|
||||||
|
kind: PersistentVolumeClaim
|
||||||
|
metadata:
|
||||||
|
name: test-vol-restore
|
||||||
|
spec:
|
||||||
|
storageClassName: longhorn
|
||||||
|
dataSource:
|
||||||
|
name: test-vol-snapshot
|
||||||
|
kind: VolumeSnapshot
|
||||||
|
apiGroup: snapshot.storage.k8s.io
|
||||||
|
accessModes:
|
||||||
|
- ReadWriteOnce
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
storage: 2Gi
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Restoration of an existing Longhorn backup (pre-provisioning)
|
||||||
|
|
||||||
|
The user can request the creation of a volume based on a prior longhorn backup, that was not created via the csi layer.
|
||||||
|
The user needs to create a `VolumeSnapshotContent` object with an associated `VolumeSnapshot` object.
|
||||||
|
The `snapshotHandle` of the `VolumeSnapshotContent` needs to point to an existing longhorn backup.
|
||||||
|
Example below for a volume named `test-restore-existing-backup`
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
apiVersion: snapshot.storage.k8s.io/v1beta1
|
||||||
|
kind: VolumeSnapshotContent
|
||||||
|
metadata:
|
||||||
|
name: test-existing-backup
|
||||||
|
spec:
|
||||||
|
volumeSnapshotClassName: longhorn
|
||||||
|
driver: driver.longhorn.io
|
||||||
|
deletionPolicy: Delete
|
||||||
|
source:
|
||||||
|
# NOTE: change this to point to an existing backup on the backupstore
|
||||||
|
snapshotHandle: bs://test-vol/backup-625159fb469e492e
|
||||||
|
volumeSnapshotRef:
|
||||||
|
name: test-snapshot-existing-backup
|
||||||
|
namespace: default
|
||||||
|
```
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
apiVersion: snapshot.storage.k8s.io/v1beta1
|
||||||
|
kind: VolumeSnapshot
|
||||||
|
metadata:
|
||||||
|
name: test-snapshot-existing-backup
|
||||||
|
spec:
|
||||||
|
volumeSnapshotClassName: longhorn
|
||||||
|
source:
|
||||||
|
volumeSnapshotContentName: test-existing-backup
|
||||||
|
```
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
apiVersion: v1
|
||||||
|
kind: PersistentVolumeClaim
|
||||||
|
metadata:
|
||||||
|
name: test-restore-existing-backup
|
||||||
|
spec:
|
||||||
|
storageClassName: longhorn
|
||||||
|
dataSource:
|
||||||
|
name: test-snapshot-existing-backup
|
||||||
|
kind: VolumeSnapshot
|
||||||
|
apiGroup: snapshot.storage.k8s.io
|
||||||
|
accessModes:
|
||||||
|
- ReadWriteOnce
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
storage: 2Gi
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Backup deletion via VolumeSnapshot resource
|
||||||
|
|
||||||
|
The user can request the deletion of a backup by removing the associated `VolumeSnapshot` object.
|
||||||
|
If the DeletionPolicy is Delete, then the referenced longhorn backup will be deleted along with the `VolumeSnapshotContent` object.
|
||||||
|
If the DeletionPolicy is Retain, then both the referenced longhorn backup and `VolumeSnapshotContent` object remain.
|
||||||
|
The default DeletionPolicy is set to Delete, if the user wants to retain the longhorn backups,
|
||||||
|
the user can create a snapshotClass with DeletionPolicy set to Retain.
|
||||||
|
https://kubernetes.io/docs/concepts/storage/volume-snapshot-classes/#deletionpolicy
|
||||||
|
|
||||||
|
Example below for a snapshot named `test-snapshot-pvc`
|
||||||
|
`kubectl delete volumesnapshots test-snapshot-pvc`
|
||||||
|
|
||||||
|
Deletion is triggered by deleting the VolumeSnapshot object, and the DeletionPolicy will be followed.
|
||||||
|
|
||||||
|
The default deletion policy
|
||||||
|
|
||||||
|
### API changes
|
||||||
|
|
||||||
|
no changes necessary
|
||||||
|
|
||||||
|
## Design
|
||||||
|
|
||||||
|
### Implementation Overview
|
||||||
|
|
||||||
|
#### Implement CSI CreateSnapshot call
|
||||||
|
|
||||||
|
The creation of a `VolumeSnapshot` resource triggers the creation of a longhorn snapshot afterwards,
|
||||||
|
that snapshot will be backed up via the longhorn backup mechanism to a user defined backupstore (s3, nfs).
|
||||||
|
While the backup isn't completed the csi snapshot will be marked not ready to use.
|
||||||
|
|
||||||
|
The csi-snapshot-request name is generated by the csi snapshotter, based on the configured prefix + `VolumeSnapshot.uuid`
|
||||||
|
This information is also used to generate the `VolumeSnapshotContent.name = snapcontent-uuid`
|
||||||
|
|
||||||
|
The csi-snapshotter will always call the longhorn csi plugin with the same csi snapshot request name for a specific Snapshot object.
|
||||||
|
This means we have to use the csi snapshotter generated name, for the longhorn snapshot
|
||||||
|
or introduce a CRD (VolumeBackup) so that we have a persisted way of associating
|
||||||
|
csiSnapshot - longhornSnapshot - longhornBackup
|
||||||
|
|
||||||
|
If we would ignore the requested name, we would end up generating a new snapshot for each successive call,
|
||||||
|
since a backup can take a long time for very big volumes, that would not be the desired behavior.
|
||||||
|
|
||||||
|
To be able to lookup the backup during a following `DeleteSnapshot` or `CreateVolume` call,
|
||||||
|
we encode the backupVolume and backupName as part of the snapshotID
|
||||||
|
which is returned from the csi CreateSnapshot call.
|
||||||
|
|
||||||
|
this will be set as the `VolumeSnapshotContent.snapshotHandle` for the kubernetes created `VolumeSnapshotContent` object.
|
||||||
|
We use the format `type://backupVolume/backupName` where the default type equals `bs` for direct references to longhorn backups.
|
||||||
|
|
||||||
|
This is so that in the future we can refer to a custom kubernetes resource, which we can use for backup metadata persistence.
|
||||||
|
|
||||||
|
#### Implement CSI DeleteSnapshot call
|
||||||
|
|
||||||
|
For backup deletion all we have todo is decode the snapshotID then trigger the backup delete calls via the longhorn api.
|
||||||
|
|
||||||
|
#### Add CSI CreateVolume ContentSource support.
|
||||||
|
|
||||||
|
For the volume creation based on a CSI snapshot we can decode the snapshotID to lookup the backup in the backupstore.
|
||||||
|
This will provide us with the backupURL which we can add to the `fromBackup` field in the longhorn created Volume resource.
|
||||||
|
As part of prior work longhorn already knows how to restore these backups, this was initially used for a StorageClass
|
||||||
|
parameter, by reusing the same mechanism we don't have to maintain multiple code paths for volume restoration.
|
||||||
|
|
||||||
|
### Test plan
|
||||||
|
|
||||||
|
See examples of the necessary yaml manifests in the `User Experience In Detail` section.
|
||||||
|
|
||||||
|
Creation test:
|
||||||
|
- create volume
|
||||||
|
- write data to volume
|
||||||
|
- create a `VolumeSnapshot` object
|
||||||
|
- wait for `VolumeSnapshot` to be ready to use
|
||||||
|
- check for backup existence on the backupstore
|
||||||
|
|
||||||
|
Deletion test:
|
||||||
|
- create volume
|
||||||
|
- write data to volume
|
||||||
|
- create a `VolumeSnapshot` object
|
||||||
|
- wait for `VolumeSnapshot` to be ready to use
|
||||||
|
- check for backup existence on the backupstore
|
||||||
|
- delete `VolumeSnapshot` object
|
||||||
|
- wait for backup removal from the backupstore
|
||||||
|
|
||||||
|
Restore csi snapshot test:
|
||||||
|
- create volume
|
||||||
|
- write data to volume
|
||||||
|
- create a `VolumeSnapshot` object
|
||||||
|
- wait for `VolumeSnapshot` to be ready to use
|
||||||
|
- check for backup existence on the backupstore
|
||||||
|
- create PVC with content source set to the `VolumeSnapshot` object
|
||||||
|
- wait for volume restoration
|
||||||
|
- verify restored volume data == previously written data
|
||||||
|
|
||||||
|
Restore existing longhorn backup test:
|
||||||
|
- create volume
|
||||||
|
- write data to volume
|
||||||
|
- create a longhorn backup
|
||||||
|
- check for backup existence on the backupstore
|
||||||
|
- create a `VolumeSnapshotContent` object pointing to the longhorn backup
|
||||||
|
- create a `VolumeSnapshot` object pointing to the `VolumeSnapshotContent` object
|
||||||
|
- create PVC with content source set to the `VolumeSnapshot` object
|
||||||
|
- wait for volume restoration
|
||||||
|
- verify restored volume data == previously written data
|
||||||
|
|
||||||
|
### Upgrade strategy
|
||||||
|
|
||||||
|
For csi snapshot support the user needs to update their kubernetes installation to at least 1.17
|
||||||
|
|
||||||
|
For environments where the user has pinned their csi images (airgap)
|
||||||
|
the users need to manually provide the following images:
|
||||||
|
|
||||||
|
- longhornio/csi-provisioner:v1.6.0
|
||||||
|
- longhornio/csi-snapshotter:v2.1.1
|
||||||
|
|
||||||
|
We upgraded the csi-provsioner from 1.4 to 1.6, which still supports kubernetes 1.13 as a minimum version
|
||||||
|
|
||||||
|
## Note
|
||||||
|
|
||||||
|
The CRDs and snapshot controller installations are the responsibility of the Kubernetes distribution.
|
||||||
|
See: https://kubernetes.io/docs/concepts/storage/volume-snapshots/#introduction
|
||||||
|
|
||||||
|
We are discussing whether longhorn can provide these as part of the longhorn installation, but there isn't really
|
||||||
|
a good way of making sure that there isn't a snapshot controller already deployed in the cluster.
|
||||||
|
|
||||||
|
Make sure your cluster contains the below crds, rancher rke did not deploy them for me.
|
||||||
|
https://github.com/kubernetes-csi/external-snapshotter/tree/master/client/config/crd
|
||||||
|
|
||||||
|
Make sure your cluster contains the snapshot controller, rancher rke did not deploy it for me.
|
||||||
|
https://github.com/kubernetes-csi/external-snapshotter/tree/master/deploy/kubernetes/snapshot-controller
|
||||||
Loading…
Reference in New Issue
Block a user