79 lines
2.6 KiB
Markdown
79 lines
2.6 KiB
Markdown
# Concurrent Backup Restore Per Node Limit
|
|
|
|
## Summary
|
|
Longhorn has no boundary on the number of concurrent volume backup restoring.
|
|
|
|
Having a new `concurrent-backup-restore-per-node-limit` setting allows the user to limit the concurring backup restoring. Setting this restriction lowers the potential risk of overloading the cluster when volumes restoring from backup concurrently. For ex: during the Longhorn system restore.
|
|
|
|
### Related Issues
|
|
https://github.com/longhorn/longhorn/issues/4558
|
|
|
|
## Motivation
|
|
### Goals
|
|
Introduce a new `concurrent-backup-restore-per-node-limit` setting to define the boundary of the concurrent volume backup restoring.
|
|
|
|
### Non-goals
|
|
`None`
|
|
|
|
## Proposal
|
|
1. Introduce a new `concurrent-backup-restore-per-node-limit` setting.
|
|
1. Track the number of per-node volumes restoring from backup with atomic count (thread-safe) in the engine monitor.
|
|
|
|
### User Stories
|
|
Allow the user to set the concurrent backup restore per node limit to control the risk of cluster overload when Longhorn volume is restoring from backup concurrently.
|
|
|
|
### User Experience In Detail
|
|
1. Longhorn holds the engine backup restore when the number of volume backups restoring on a node reaches the `concurrent-backup-restore-per-node-limit`.
|
|
1. The volume backup restore continues when the number of volume backups restoring on a node is below the limit.
|
|
|
|
## Design
|
|
|
|
### Implementation Overview
|
|
|
|
#### The `concurrent-backup-restore-per-node-limit` Setting
|
|
|
|
This setting controls how many engines on a node can restore the backup concurrently.
|
|
|
|
Longhorn engine monitor backs off when the volume [backup restoring count](#track-the-volume-backup-restoring-per-node) reaches the setting limit.
|
|
|
|
Set the value to **0** to disable backup restore.
|
|
|
|
```
|
|
Category = SettingCategoryGeneral,
|
|
Type = integer
|
|
Default = 5 # same as the default replica rebuilding number
|
|
```
|
|
|
|
#### Track the volume backup restoring per node
|
|
|
|
1. Create a new atomic counter in the engine controller.
|
|
```
|
|
type EngineController struct {
|
|
restoringCounter util.Counter
|
|
}
|
|
```
|
|
1. Pass the restoring counter to each of its engine monitors.
|
|
```
|
|
type EngineMonitor struct {
|
|
restoringCounter util.Counter
|
|
}
|
|
```
|
|
|
|
1. Increase the restoring counter before backup restore.
|
|
> Ignore DR volumes (volume.Status.IsStandby).
|
|
1. Decrease the restoring counter when the backup restore caller method ends
|
|
|
|
### Test plan
|
|
|
|
- Test the setting should block backup restore when creating multiple volumes from the backup at the same time.
|
|
- Test the setting should be per-node limited.
|
|
- Test the setting should not have effect on DR volumes.
|
|
|
|
### Upgrade strategy
|
|
|
|
`None`
|
|
|
|
## Note [optional]
|
|
|
|
`None`
|