OKD / OCP 4.14

Signed-off-by: Arthur <arthur@arthurvardevanyan.com>

.github/ISSUE_TEMPLATE/bug.md

@@ -0,0 +1,48 @@
+---
+name: Bug report
+about: Create a bug report
+title: "[BUG]"
+labels: ["kind/bug", "require/qa-review-coverage", "require/backport"]
+assignees: ''
+
+---
+
+## Describe the bug (🐛 if you encounter this issue)
+
+<!--A clear and concise description of what the bug is.-->
+
+## To Reproduce
+
+<!--Provide the steps to reproduce the behavior.-->
+
+## Expected behavior
+
+<!--A clear and concise description of what you expected to happen.-->
+
+## Support bundle for troubleshooting
+
+<!--Provide a support bundle when the issue happens. You can generate a support bundle using the link at the footer of the Longhorn UI. Check [here](https://longhorn.io/docs/latest/advanced-resources/support-bundle/).-->
+
+## Environment
+
+<!-- Suggest checking the doc of the best practices of using Longhorn. [here](https://longhorn.io/docs/1.5.1/best-practices)-->
+ - Longhorn version:
+ - Installation method (e.g. Rancher Catalog App/Helm/Kubectl):
+ - Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version:
+   - Number of management node in the cluster:
+   - Number of worker node in the cluster:
+ - Node config
+   - OS type and version:
+   - Kernel version:
+   - CPU per node:
+   - Memory per node:
+   - Disk type(e.g. SSD/NVMe/HDD):
+   - Network bandwidth between the nodes:
+ - Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal):
+ - Number of Longhorn volumes in the cluster:
+ - Impacted Longhorn resources:
+   - Volume names:
+
+## Additional context
+
+<!--Add any other context about the problem here.-->

.github/ISSUE_TEMPLATE/bug_report.md

@@ -1,49 +0,0 @@
----
-name: Bug report
-about: Create a bug report
-title: "[BUG]"
-labels: kind/bug
-assignees: ''
-
----
-
-## Describe the bug (🐛 if you encounter this issue)
-
-A clear and concise description of what the bug is.
-
-## To Reproduce
-
-Steps to reproduce the behavior:
-1. Go to '...'
-2. Click on '....'
-3. Perform '....'
-4. See error
-
-## Expected behavior
-
-A clear and concise description of what you expected to happen.
-
-## Log or Support bundle
-
-If applicable, add the Longhorn managers' log or support bundle when the issue happens. 
-You can generate a Support Bundle using the link at the footer of the Longhorn UI.
-
-## Environment
-
- - Longhorn version:
- - Installation method (e.g. Rancher Catalog App/Helm/Kubectl):
- - Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version:
-   - Number of management node in the cluster:
-   - Number of worker node in the cluster:
- - Node config
-   - OS type and version:
-   - CPU per node:
-   - Memory per node:
-   - Disk type(e.g. SSD/NVMe):
-   - Network bandwidth between the nodes:
- - Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal):
- - Number of Longhorn volumes in the cluster:
-
-## Additional context
-
-Add any other context about the problem here.

.github/ISSUE_TEMPLATE/doc.md

@@ -7,15 +7,10 @@ assignees: ''
 
 ---
 
-## What's the task? Please describe
+## What's the document you plan to update? Why? Please describe
 
-A clear and concise description of what the document is.
-
-## Describe the items of the task (DoD, definition of done) you'd like
-> Please use a task list for items on a separate line with a clickable checkbox https://docs.github.com/en/issues/tracking-your-work-with-issues/about-task-lists
-
-- [ ] `item 1`
+<!--A clear and concise description of what the document is.-->
 
 ## Additional context
 
-Add any other context or screenshots about the document request here.
+<!--Add any other context or screenshots about the document request here.-->

.github/ISSUE_TEMPLATE/feature.md

@@ -2,23 +2,23 @@
 name: Feature request
 about: Suggest an idea/feature
 title: "[FEATURE] "
-labels: kind/enhancement
+labels: ["kind/enhancement", "require/lep", "require/doc", "require/auto-e2e-test"]
 assignees: ''
 
 ---
 
 ## Is your feature request related to a problem? Please describe (👍 if you like this request)
 
-A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+<!--A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]-->
 
 ## Describe the solution you'd like
 
-A clear and concise description of what you want to happen
+<!--A clear and concise description of what you want to happen-->
 
 ## Describe alternatives you've considered
 
-A clear and concise description of any alternative solutions or features you've considered.
+<!--A clear and concise description of any alternative solutions or features you've considered.-->
 
 ## Additional context
 
-Add any other context or screenshots about the feature request here.
+<!--Add any other context or screenshots about the feature request here.-->

.github/ISSUE_TEMPLATE/improvement.md

@@ -2,23 +2,23 @@
 name: Improvement request
 about: Suggest an improvement of an existing feature
 title: "[IMPROVEMENT] "
-labels: kind/improvement
+labels: ["kind/improvement", "require/doc", "require/auto-e2e-test", "require/backport"]
 assignees: ''
 
 ---
 
 ## Is your improvement request related to a feature? Please describe (👍 if you like this request)
 
-A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+<!--A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]-->
 
 ## Describe the solution you'd like
 
-A clear and concise description of what you want to happen.
+<!--A clear and concise description of what you want to happen.-->
 
 ## Describe alternatives you've considered
 
-A clear and concise description of any alternative solutions or features you've considered.
+<!--A clear and concise description of any alternative solutions or features you've considered.-->
 
 ## Additional context
 
-Add any other context or screenshots about the feature request here.
+<!--Add any other context or screenshots about the feature request here.-->

.github/ISSUE_TEMPLATE/infra.md

@@ -0,0 +1,24 @@
+---
+name: Infra
+about: Create an test/dev infra task
+title: "[INFRA] "
+labels: kind/infra
+assignees: ''
+
+---
+
+## What's the test to develop? Please describe
+
+<!--A clear and concise description of what test/dev infra you want to develop.-->
+
+## Describe the items of the test development (DoD, definition of done) you'd like
+
+<!--
+Please use a task list for items on a separate line with a clickable checkbox https://docs.github.com/en/issues/tracking-your-work-with-issues/about-task-lists
+
+- [ ] `item 1`
+-->
+
+## Additional context
+
+<!--Add any other context or screenshots about the test infra request here.-->

.github/ISSUE_TEMPLATE/question.md

@@ -7,7 +7,8 @@ assignees: ''
 
 ---
 ## Question
-> Suggest to use https://github.com/longhorn/longhorn/discussions to ask questions.
+
+<!--Suggest to use https://github.com/longhorn/longhorn/discussions to ask questions.-->
 
 ## Environment
 
@@ -15,6 +16,7 @@ assignees: ''
  - Kubernetes version:
  - Node config
    - OS type and version
+   - Kernel version
    - CPU per node:
    - Memory per node:
    - Disk type
@@ -23,4 +25,4 @@ assignees: ''
 
 ## Additional context
 
-Add any other context about the problem here.
+<!--Add any other context about the problem here.-->

.github/ISSUE_TEMPLATE/refactor.md

@@ -1,6 +1,6 @@
 ---
-name: Refactoring request
-about: Suggest a refactoring request of an existing feature or design
+name: Refactor request
+about: Suggest a refactoring request for an existing implementation
 title: "[REFACTOR] "
 labels: kind/refactoring
 assignees: ''
@@ -9,16 +9,16 @@ assignees: ''
 
 ## Is your improvement request related to a feature? Please describe
 
-A clear and concise description of what the problem is.
+<!--A clear and concise description of what the problem is.-->
 
 ## Describe the solution you'd like
 
-A clear and concise description of what you want to happen.
+<!--A clear and concise description of what you want to happen.-->
 
 ## Describe alternatives you've considered
 
-A clear and concise description of any alternative solutions or features you've considered.
+<!--A clear and concise description of any alternative solutions or features you've considered.-->
 
 ## Additional context
 
-Add any other context or screenshots about the refactoring request here.
+<!--Add any other context or screenshots about the refactoring request here.-->

.github/ISSUE_TEMPLATE/release.md

@@ -0,0 +1,35 @@
+---
+name: Release task
+about: Create a release task
+title: "[RELEASE]"
+labels: release/task
+assignees: ''
+
+---
+
+**What's the task? Please describe.**
+Action items for releasing v<x.y.z>
+
+**Describe the sub-tasks.**
+  - Pre-Release
+    - [ ] Regression test plan (manual) - @khushboo-rancher 
+    - [ ] Run e2e regression for pre-GA milestones (`install`, `upgrade`) - @yangchiu 
+    - [ ] Run security testing of container images for pre-GA milestones - @yangchiu 
+    - [ ] Verify longhorn chart PR to ensure all artifacts are ready for GA (`install`, `upgrade`)  @chriscchien 
+    - [ ] Run core testing (install, upgrade) for the GA build from the previous patch and the last patch of the previous feature release (1.4.2). - @yangchiu 
+  - Release
+    - [ ] Release longhorn/chart from the release branch to publish to ArtifactHub
+    - [ ] Release note
+	     - [ ] Deprecation note
+	     - [ ] Upgrade notes including highlighted notes, deprecation, compatible changes, and others impacting the current users
+  - Post-Release
+    - [ ] Create a new release branch of manager/ui/tests/engine/longhorn instance-manager/share-manager/backing-image-manager when creating the RC1
+    - [ ] Update https://github.com/longhorn/longhorn/blob/master/deploy/upgrade_responder_server/chart-values.yaml @PhanLe1010 
+    - [ ] Add another request for the rancher charts for the next patch release (`1.5.1`) @rebeccazzzz  
+  - Rancher charts: verify the chart is able to install & upgrade - @khushboo-rancher 
+    - [ ] rancher/image-mirrors update @weizhe0422 (@PhanLe1010 )
+        - https://github.com/rancher/image-mirror/pull/412
+    - [ ] rancher/charts 2.7 branches for rancher marketplace @weizhe0422 (@PhanLe1010)
+        - `dev-2.7`: https://github.com/rancher/charts/pull/2766
+
+cc @longhorn/qa @longhorn/dev 

.github/ISSUE_TEMPLATE/task.md

@@ -9,13 +9,16 @@ assignees: ''
 
 ## What's the task? Please describe
 
-A clear and concise description of what the task is.
+<!--A clear and concise description of what the task is.-->
 
-## Describe the items of the task (DoD, definition of done) you'd like
-> Please use a task list for items on a separate line with a clickable checkbox https://docs.github.com/en/issues/tracking-your-work-with-issues/about-task-lists
+## Describe the sub-tasks
+
+<!--
+Please use a task list for items on a separate line with a clickable checkbox https://docs.github.com/en/issues/tracking-your-work-with-issues/about-task-lists
 
 - [ ] `item 1`
+-->
 
 ## Additional context
 
-Add any other context or screenshots about the task request here.
+<!--Add any other context or screenshots about the task request here.-->

.github/ISSUE_TEMPLATE/test.md

@@ -9,13 +9,16 @@ assignees: ''
 
 ## What's the test to develop? Please describe
 
-A clear and concise description of what the test you want to develop.
+<!--A clear and concise description of what test you want to develop.-->
 
-## Describe the items of the test development (DoD, definition of done) you'd like
-> Please use a task list for items on a separate line with a clickable checkbox https://docs.github.com/en/issues/tracking-your-work-with-issues/about-task-lists
+## Describe the tasks for the test
+
+<!--
+Please use a task list for items on a separate line with a clickable checkbox https://docs.github.com/en/issues/tracking-your-work-with-issues/about-task-lists
 
 - [ ] `item 1`
+-->
 
 ## Additional context
 
-Add any other context or screenshots about the test request here.
+<!--Add any other context or screenshots about the test request here.-->

.github/ISSUE_TEMPLATE/test_infra.md

@@ -1,21 +0,0 @@
----
-name: Test infra
-about: Create a test-infra task
-title: "[TEST-INFRA] "
-labels: kind/test
-assignees: ''
-
----
-
-## What's the test to develop? Please describe
-
-A clear and concise description of what the test infra you want to develop.
-
-## Describe the items of the test development (DoD, definition of done) you'd like
-> Please use a task list for items on a separate line with a clickable checkbox https://docs.github.com/en/issues/tracking-your-work-with-issues/about-task-lists
-
-- [ ] `item 1`
-
-## Additional context
-
-Add any other context or screenshots about the test infra request here.

.github/workflows/add-to-projects.yml

@@ -1,32 +1,40 @@
 name: Add-To-Projects
 on:
   issues:
-    types: [opened, labeled]
+    types: [ opened, labeled ]
 jobs:
   community:
     runs-on: ubuntu-latest
     steps:
-      - name: Is Longhorn Member
-        uses: tspascoal/get-user-teams-membership@v1.0.4
-        id: is-longhorn-member
-        with:
-          username: ${{ github.event.issue.user.login }}
-          organization: longhorn
-          GITHUB_TOKEN: ${{ secrets.CUSTOM_GITHUB_TOKEN }}
-      - name: Add To Community Project
-        if: fromJSON(steps.is-longhorn-member.outputs.teams)[0] == null
-        uses: actions/add-to-project@v0.3.0
-        with:
-          project-url: https://github.com/orgs/longhorn/projects/5
-          github-token: ${{ secrets.CUSTOM_GITHUB_TOKEN }}
+    - name: Is Longhorn Member
+      uses: tspascoal/get-user-teams-membership@v1.0.4
+      id: is-longhorn-member
+      with:
+        username: ${{ github.event.issue.user.login }}
+        organization: longhorn
+        GITHUB_TOKEN: ${{ secrets.CUSTOM_GITHUB_TOKEN }}
+    - name: Add To Community Project
+      if: fromJSON(steps.is-longhorn-member.outputs.teams)[0] == null
+      uses: actions/add-to-project@v0.3.0
+      with:
+        project-url: https://github.com/orgs/longhorn/projects/5
+        github-token: ${{ secrets.CUSTOM_GITHUB_TOKEN }}
 
   qa:
     runs-on: ubuntu-latest
     steps:
-      - name: Add To QA & Devops Project
-        uses: actions/add-to-project@v0.3.0
-        with:
-          project-url: https://github.com/orgs/longhorn/projects/4
-          github-token: ${{ secrets.CUSTOM_GITHUB_TOKEN }}
-          labeled: kind/test, area/test-infra
-          label-operator: OR
+    - name: Is Longhorn Member
+      uses: tspascoal/get-user-teams-membership@v1.0.4
+      id: is-longhorn-member
+      with:
+        username: ${{ github.event.issue.user.login }}
+        organization: longhorn
+        GITHUB_TOKEN: ${{ secrets.CUSTOM_GITHUB_TOKEN }}
+    - name: Add To QA & DevOps Project
+      if: fromJSON(steps.is-longhorn-member.outputs.teams)[0] != null
+      uses: actions/add-to-project@v0.3.0
+      with:
+        project-url: https://github.com/orgs/longhorn/projects/4
+        github-token: ${{ secrets.CUSTOM_GITHUB_TOKEN }}
+        labeled: kind/test, area/infra
+        label-operator: OR

.github/workflows/close-issue.yml

@@ -1,50 +1,50 @@
 name: Close-Issue
 on:
   issues:
-    types: [unlabeled]
+    types: [ unlabeled ]
 jobs:
   backport:
     runs-on: ubuntu-latest
     if: contains(github.event.label.name, 'backport/')
     steps:
-      - name: Get Backport Version
-        uses: xom9ikk/split@v1
-        id: split
-        with:
-          string: ${{ github.event.label.name }}
-          separator: /
-      - name: Check if Backport Issue Exists
-        uses: actions-cool/issues-helper@v3
-        id: if-backport-issue-exists
-        with:
-          actions: 'find-issues'
-          token: ${{ github.token }}
-          title-includes: |
-            [BACKPORT][v${{ steps.split.outputs._1 }}]${{ github.event.issue.title }}
-      - name: Close Backport Issue
-        if: fromJSON(steps.if-backport-issue-exists.outputs.issues)[0] != null
-        uses: actions-cool/issues-helper@v3
-        with:
-          actions: 'close-issue'
-          token: ${{ github.token }}
-          issue-number: ${{ fromJSON(steps.if-backport-issue-exists.outputs.issues)[0].number }}
+    - name: Get Backport Version
+      uses: xom9ikk/split@v1
+      id: split
+      with:
+        string: ${{ github.event.label.name }}
+        separator: /
+    - name: Check if Backport Issue Exists
+      uses: actions-cool/issues-helper@v3
+      id: if-backport-issue-exists
+      with:
+        actions: 'find-issues'
+        token: ${{ github.token }}
+        title-includes: |
+          [BACKPORT][v${{ steps.split.outputs._1 }}]${{ github.event.issue.title }}
+    - name: Close Backport Issue
+      if: fromJSON(steps.if-backport-issue-exists.outputs.issues)[0] != null
+      uses: actions-cool/issues-helper@v3
+      with:
+        actions: 'close-issue'
+        token: ${{ github.token }}
+        issue-number: ${{ fromJSON(steps.if-backport-issue-exists.outputs.issues)[0].number }}
 
   automation:
     runs-on: ubuntu-latest
     if: contains(github.event.label.name, 'require/automation-e2e')
     steps:
-      - name: Check if Automation Issue Exists
-        uses: actions-cool/issues-helper@v3
-        id: if-automation-issue-exists
-        with:
-          actions: 'find-issues'
-          token: ${{ github.token }}
-          title-includes: |
-            [TEST]${{ github.event.issue.title }}
-      - name: Close Automation Test Issue
-        if: fromJSON(steps.if-automation-issue-exists.outputs.issues)[0] != null
-        uses: actions-cool/issues-helper@v3
-        with:
-          actions: 'close-issue'
-          token: ${{ github.token }}
-          issue-number: ${{ fromJSON(steps.if-automation-issue-exists.outputs.issues)[0].number }}
+    - name: Check if Automation Issue Exists
+      uses: actions-cool/issues-helper@v3
+      id: if-automation-issue-exists
+      with:
+        actions: 'find-issues'
+        token: ${{ github.token }}
+        title-includes: |
+          [TEST]${{ github.event.issue.title }}
+    - name: Close Automation Test Issue
+      if: fromJSON(steps.if-automation-issue-exists.outputs.issues)[0] != null
+      uses: actions-cool/issues-helper@v3
+      with:
+        actions: 'close-issue'
+        token: ${{ github.token }}
+        issue-number: ${{ fromJSON(steps.if-automation-issue-exists.outputs.issues)[0].number }}

.github/workflows/create-issue.yml

@@ -1,97 +1,114 @@
 name: Create-Issue
 on:
   issues:
-    types: [labeled]
+    types: [ labeled ]
 jobs:
   backport:
     runs-on: ubuntu-latest
     if: contains(github.event.label.name, 'backport/')
     steps:
-      - name: Get Backport Version
-        uses: xom9ikk/split@v1
-        id: split
-        with:
-          string: ${{ github.event.label.name }}
-          separator: /
-      - name: Check if Backport Issue Exists
-        uses: actions-cool/issues-helper@v3
-        id: if-backport-issue-exists
-        with:
-          actions: 'find-issues'
-          token: ${{ github.token }}
-          issue-state: 'all'
-          title-includes: |
-            [BACKPORT][v${{ steps.split.outputs._1 }}]${{ github.event.issue.title }}
-      - name: Get Milestone Object
-        if: fromJSON(steps.if-backport-issue-exists.outputs.issues)[0] == null
-        uses: longhorn/bot/milestone-action@master
-        id: milestone
-        with:
-          token: ${{ github.token }}
-          repository: ${{ github.repository }}
-          milestone_name: v${{ steps.split.outputs._1 }}
-      - name: Get Labels
-        if: fromJSON(steps.if-backport-issue-exists.outputs.issues)[0] == null
-        id: labels
-        run: |
-          RAW_LABELS="${{ join(github.event.issue.labels.*.name, ' ') }}"
-          RAW_LABELS="${RAW_LABELS} kind/backport"
-          echo "RAW LABELS: $RAW_LABELS"
-          LABELS=$(echo "$RAW_LABELS" | sed -r 's/\s*backport\S+//g' | sed -r 's/\s*require\/auto-e2e-test//g' | xargs | sed 's/ /, /g')
-          echo "LABELS: $LABELS"
-          echo "::set-output name=labels::$LABELS"
-      - name: Create Backport Issue
-        if: fromJSON(steps.if-backport-issue-exists.outputs.issues)[0] == null
-        uses: dacbd/create-issue-action@v1
-        id: new-issue
-        with:
-          token: ${{ github.token }}
-          title: |
-            [BACKPORT][v${{ steps.split.outputs._1 }}]${{ github.event.issue.title }}
-          body: |
-            backport ${{ github.event.issue.html_url }}
-          labels: ${{ steps.labels.outputs.labels }}
-          milestone: ${{ fromJSON(steps.milestone.outputs.data).number }}
-          assignees: ${{ join(github.event.issue.assignees.*.login, ', ') }}
-      - name: Get Repo Id
-        if: fromJSON(steps.if-backport-issue-exists.outputs.issues)[0] == null
-        uses: octokit/request-action@v2.x
-        id: repo
-        with:
-          route: GET /repos/${{ github.repository }}
-        env:
-          GITHUB_TOKEN: ${{ github.token }}
-      - name: Add Backport Issue To Release
-        if: fromJSON(steps.if-backport-issue-exists.outputs.issues)[0] == null
-        uses: longhorn/bot/add-zenhub-release-action@master
-        with:
-          zenhub_token: ${{ secrets.ZENHUB_TOKEN }}
-          repo_id: ${{ fromJSON(steps.repo.outputs.data).id }}
-          issue_number: ${{ steps.new-issue.outputs.number }}
-          release_name: ${{ steps.split.outputs._1 }}
+    - name: Is Longhorn Member
+      uses: tspascoal/get-user-teams-membership@v1.0.4
+      id: is-longhorn-member
+      with:
+        username: ${{ github.actor }}
+        organization: longhorn
+        GITHUB_TOKEN: ${{ secrets.CUSTOM_GITHUB_TOKEN }}
+    - name: Get Backport Version
+      if: fromJSON(steps.is-longhorn-member.outputs.teams)[0] != null
+      uses: xom9ikk/split@v1
+      id: split
+      with:
+        string: ${{ github.event.label.name }}
+        separator: /
+    - name: Check if Backport Issue Exists
+      if: fromJSON(steps.is-longhorn-member.outputs.teams)[0] != null
+      uses: actions-cool/issues-helper@v3
+      id: if-backport-issue-exists
+      with:
+        actions: 'find-issues'
+        token: ${{ github.token }}
+        issue-state: 'all'
+        title-includes: |
+          [BACKPORT][v${{ steps.split.outputs._1 }}]${{ github.event.issue.title }}
+    - name: Get Milestone Object
+      if: fromJSON(steps.is-longhorn-member.outputs.teams)[0] != null && fromJSON(steps.if-backport-issue-exists.outputs.issues)[0] == null
+      uses: longhorn/bot/milestone-action@master
+      id: milestone
+      with:
+        token: ${{ github.token }}
+        repository: ${{ github.repository }}
+        milestone_name: v${{ steps.split.outputs._1 }}
+    - name: Get Labels
+      if: fromJSON(steps.is-longhorn-member.outputs.teams)[0] != null && fromJSON(steps.if-backport-issue-exists.outputs.issues)[0] == null
+      id: labels
+      run: |
+        RAW_LABELS="${{ join(github.event.issue.labels.*.name, ' ') }}"
+        RAW_LABELS="${RAW_LABELS} kind/backport"
+        echo "RAW LABELS: $RAW_LABELS"
+        LABELS=$(echo "$RAW_LABELS" | sed -r 's/\s*backport\S+//g' | sed -r 's/\s*require\/auto-e2e-test//g' | xargs | sed 's/ /, /g')
+        echo "LABELS: $LABELS"
+        echo "labels=$LABELS" >> $GITHUB_OUTPUT
+    - name: Create Backport Issue
+      if: fromJSON(steps.is-longhorn-member.outputs.teams)[0] != null && fromJSON(steps.if-backport-issue-exists.outputs.issues)[0] == null
+      uses: dacbd/create-issue-action@v1
+      id: new-issue
+      with:
+        token: ${{ github.token }}
+        title: |
+          [BACKPORT][v${{ steps.split.outputs._1 }}]${{ github.event.issue.title }}
+        body: |
+          backport ${{ github.event.issue.html_url }}
+        labels: ${{ steps.labels.outputs.labels }}
+        milestone: ${{ fromJSON(steps.milestone.outputs.data).number }}
+        assignees: ${{ join(github.event.issue.assignees.*.login, ', ') }}
+    - name: Get Repo Id
+      if: fromJSON(steps.is-longhorn-member.outputs.teams)[0] != null && fromJSON(steps.if-backport-issue-exists.outputs.issues)[0] == null
+      uses: octokit/request-action@v2.x
+      id: repo
+      with:
+        route: GET /repos/${{ github.repository }}
+      env:
+        GITHUB_TOKEN: ${{ github.token }}
+    - name: Add Backport Issue To Release
+      if: fromJSON(steps.is-longhorn-member.outputs.teams)[0] != null && fromJSON(steps.if-backport-issue-exists.outputs.issues)[0] == null
+      uses: longhorn/bot/add-zenhub-release-action@master
+      with:
+        zenhub_token: ${{ secrets.ZENHUB_TOKEN }}
+        repo_id: ${{ fromJSON(steps.repo.outputs.data).id }}
+        issue_number: ${{ steps.new-issue.outputs.number }}
+        release_name: ${{ steps.split.outputs._1 }}
 
   automation:
     runs-on: ubuntu-latest
     if: contains(github.event.label.name, 'require/auto-e2e-test')
     steps:
-      - name: Check if Automation Issue Exists
-        uses: actions-cool/issues-helper@v3
-        id: if-automation-issue-exists
-        with:
-          actions: 'find-issues'
-          token: ${{ github.token }}
-          issue-state: 'all'
-          title-includes: |
-            [TEST]${{ github.event.issue.title }}
-      - name: Create Automation Test Issue
-        if: fromJSON(steps.if-automation-issue-exists.outputs.issues)[0] == null
-        uses: dacbd/create-issue-action@v1
-        with:
-          token: ${{ github.token }}
-          title: |
-            [TEST]${{ github.event.issue.title }}
-          body: |
-            adding/updating auto e2e test cases for ${{ github.event.issue.html_url }} if they can be automated
-            
-            cc @longhorn/qa
-          labels: kind/test
+    - name: Is Longhorn Member
+      uses: tspascoal/get-user-teams-membership@v1.0.4
+      id: is-longhorn-member
+      with:
+        username: ${{ github.actor }}
+        organization: longhorn
+        GITHUB_TOKEN: ${{ secrets.CUSTOM_GITHUB_TOKEN }}
+    - name: Check if Automation Issue Exists
+      if: fromJSON(steps.is-longhorn-member.outputs.teams)[0] != null
+      uses: actions-cool/issues-helper@v3
+      id: if-automation-issue-exists
+      with:
+        actions: 'find-issues'
+        token: ${{ github.token }}
+        issue-state: 'all'
+        title-includes: |
+          [TEST]${{ github.event.issue.title }}
+    - name: Create Automation Test Issue
+      if: fromJSON(steps.is-longhorn-member.outputs.teams)[0] != null && fromJSON(steps.if-automation-issue-exists.outputs.issues)[0] == null
+      uses: dacbd/create-issue-action@v1
+      with:
+        token: ${{ github.token }}
+        title: |
+          [TEST]${{ github.event.issue.title }}
+        body: |
+          adding/updating auto e2e test cases for ${{ github.event.issue.html_url }} if they can be automated
+          
+          cc @longhorn/qa
+        labels: kind/test

.github/workflows/stale.yaml

@@ -4,25 +4,25 @@ on:
   workflow_call:
   workflow_dispatch:
   schedule:
-    - cron: '30 1 * * *'
+  - cron: '30 1 * * *'
 
 jobs:
   stale:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/stale@v4
-        with:
-          stale-issue-message: 'This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.'
-          stale-pr-message: 'This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.'
-          close-issue-message: 'This issue was closed because it has been stalled for 5 days with no activity.'
-          close-pr-message: 'This PR was closed because it has been stalled for 10 days with no activity.'
-          days-before-stale: 30
-          days-before-pr-stale: 45
-          days-before-close: 5
-          days-before-pr-close: 10
-          stale-issue-label: 'stale'
-          stale-pr-label: 'stale'
-          exempt-all-assignees: true
-          exempt-issue-labels: 'kind/bug,kind/doc,kind/enhancement,kind/poc,kind/refactoring,kind/test,kind/task,kind/backport,kind/regression,kind/evaluation'
-          exempt-draft-pr: true
-          exempt-all-milestones: true
+    - uses: actions/stale@v4
+      with:
+        stale-issue-message: 'This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.'
+        stale-pr-message: 'This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.'
+        close-issue-message: 'This issue was closed because it has been stalled for 5 days with no activity.'
+        close-pr-message: 'This PR was closed because it has been stalled for 10 days with no activity.'
+        days-before-stale: 30
+        days-before-pr-stale: 45
+        days-before-close: 5
+        days-before-pr-close: 10
+        stale-issue-label: 'stale'
+        stale-pr-label: 'stale'
+        exempt-all-assignees: true
+        exempt-issue-labels: 'kind/bug,kind/doc,kind/enhancement,kind/poc,kind/refactoring,kind/test,kind/task,kind/backport,kind/regression,kind/evaluation'
+        exempt-draft-pr: true
+        exempt-all-milestones: true

CHANGELOG/CHANGELOG-1.4.2.md

@@ -0,0 +1,92 @@
+## Release Note
+### **v1.4.2 released!** 🎆
+
+Longhorn v1.4.2 is the latest stable version of Longhorn 1.4.
+It introduces improvements and bug fixes in the areas of stability, performance, space efficiency, resilience, and so on. Please try it out and provide feedback. Thanks for all the contributions!
+
+> For the definition of stable or latest release, please check [here](https://github.com/longhorn/longhorn#releases).
+
+## Installation
+
+> **Please ensure your Kubernetes cluster is at least v1.21 before installing v1.4.2.**
+
+Longhorn supports 3 installation ways including Rancher App Marketplace, Kubectl, and Helm. Follow the installation instructions [here](https://longhorn.io/docs/1.4.2/deploy/install/).
+
+## Upgrade
+
+> **Please read the [important notes](https://longhorn.io/docs/1.4.2/deploy/important-notes/) first and ensure your Kubernetes cluster is at least v1.21 before upgrading to Longhorn v1.4.2 from v1.3.x/v1.4.x, which are only supported source versions.**
+
+Follow the upgrade instructions [here](https://longhorn.io/docs/1.4.2/deploy/upgrade/).
+
+## Deprecation & Incompatibilities
+
+N/A
+
+## Known Issues after Release
+
+Please follow up on [here](https://github.com/longhorn/longhorn/wiki/Outstanding-Known-Issues-of-Releases) about any outstanding issues found after this release.
+
+
+## Highlights
+  
+  - [IMPROVEMENT] Use PDB to protect Longhorn components from unexpected drains ([3304](https://github.com/longhorn/longhorn/issues/3304)) - @yangchiu @PhanLe1010
+  - [IMPROVEMENT] Introduce timeout mechanism for the sparse file syncing service ([4305](https://github.com/longhorn/longhorn/issues/4305)) - @yangchiu @ChanYiLin
+  - [IMPROVEMENT] Recurring jobs create new snapshots while being not able to clean up old ones ([4898](https://github.com/longhorn/longhorn/issues/4898)) - @mantissahz @chriscchien
+  
+## Improvement
+  
+  - [IMPROVEMENT] Support bundle collects dmesg, syslog and related information of longhorn nodes ([5073](https://github.com/longhorn/longhorn/issues/5073)) - @weizhe0422 @roger-ryao
+  - [IMPROVEMENT] Fix BackingImage uploading/downloading flow to prevent client timeout ([5443](https://github.com/longhorn/longhorn/issues/5443)) - @ChanYiLin @chriscchien
+  - [IMPROVEMENT] Create a new setting so that Longhorn removes PDB for instance-manager-r that doesn't have any running instance inside it ([5549](https://github.com/longhorn/longhorn/issues/5549)) - @PhanLe1010 @khushboo-rancher
+  - [IMPROVEMENT] Deprecate the setting `allow-node-drain-with-last-healthy-replica` and replace it by `node-drain-policy` setting ([5585](https://github.com/longhorn/longhorn/issues/5585)) - @yangchiu @PhanLe1010
+  - [IMPROVEMENT][UI] Recurring jobs create new snapshots while being not able to clean up old one ([5610](https://github.com/longhorn/longhorn/issues/5610)) - @mantissahz @smallteeths @roger-ryao
+  - [IMPROVEMENT] Only activate replica if it doesn't have deletion timestamp during volume engine upgrade ([5632](https://github.com/longhorn/longhorn/issues/5632)) - @PhanLe1010 @roger-ryao
+  - [IMPROVEMENT] Clean up backup target if the backup target setting is unset ([5655](https://github.com/longhorn/longhorn/issues/5655)) - @yangchiu @ChanYiLin
+  
+## Resilience
+  
+  - [BUG] Directly mark replica as failed if the node is deleted ([5542](https://github.com/longhorn/longhorn/issues/5542)) - @weizhe0422 @roger-ryao
+  - [BUG] RWX volume is stuck at detaching when the attached node is down  ([5558](https://github.com/longhorn/longhorn/issues/5558)) - @derekbit @roger-ryao
+  - [BUG] Backup monitor gets stuck in an infinite loop if backup isn't found ([5662](https://github.com/longhorn/longhorn/issues/5662)) - @derekbit @chriscchien
+  - [BUG] Resources such as replicas are somehow not mutated when network is unstable  ([5762](https://github.com/longhorn/longhorn/issues/5762)) - @derekbit @roger-ryao
+  - [BUG] Instance manager may not update instance status for a minute after starting ([5809](https://github.com/longhorn/longhorn/issues/5809)) - @ejweber @chriscchien
+  
+## Bugs
+  
+  - [BUG] Delete a uploading backing image, the corresponding LH temp file is not deleted ([3682](https://github.com/longhorn/longhorn/issues/3682)) - @ChanYiLin @chriscchien
+  - [BUG] Can not create backup in engine image not fully deployed cluster ([5248](https://github.com/longhorn/longhorn/issues/5248)) - @ChanYiLin @roger-ryao
+  - [BUG] Upgrade engine --> spec.restoreVolumeRecurringJob and spec.snapshotDataIntegrity Unsupported value ([5485](https://github.com/longhorn/longhorn/issues/5485)) - @yangchiu @derekbit
+  - [BUG] Bulk backup deletion cause restoring volume to finish with attached state. ([5506](https://github.com/longhorn/longhorn/issues/5506)) - @ChanYiLin @roger-ryao
+  - [BUG] volume expansion starts for no reason, gets stuck on current size > expected size ([5513](https://github.com/longhorn/longhorn/issues/5513)) - @mantissahz @roger-ryao
+  - [BUG] RWX volume attachment failed if tried more enough times ([5537](https://github.com/longhorn/longhorn/issues/5537)) - @yangchiu @derekbit
+  - [BUG] instance-manager-e emits `Wait for process pvc-xxxx to shutdown` constantly ([5575](https://github.com/longhorn/longhorn/issues/5575)) - @derekbit @roger-ryao
+  - [BUG] Support bundle kit should respect node selector & taint toleration ([5614](https://github.com/longhorn/longhorn/issues/5614)) - @yangchiu @c3y1huang
+  - [BUG] Value overlapped in page Instance Manager Image ([5622](https://github.com/longhorn/longhorn/issues/5622)) - @smallteeths @chriscchien
+  - [BUG] Instance manager PDB created with wrong selector thus blocking the draining of the wrongly selected node forever ([5680](https://github.com/longhorn/longhorn/issues/5680)) - @PhanLe1010 @chriscchien
+  - [BUG] During volume live engine upgrade, if the replica pod is killed, the volume is stuck in upgrading forever ([5684](https://github.com/longhorn/longhorn/issues/5684)) - @yangchiu @PhanLe1010
+  - [BUG] Instance manager PDBs cannot be removed if the longhorn-manager pod on its spec node is not available ([5688](https://github.com/longhorn/longhorn/issues/5688)) - @PhanLe1010 @roger-ryao
+  - [BUG] Rebuild rebuilding is possibly issued to a wrong replica ([5709](https://github.com/longhorn/longhorn/issues/5709)) - @ejweber @roger-ryao
+  - [BUG] longhorn upgrade is not upgrading engineimage ([5740](https://github.com/longhorn/longhorn/issues/5740)) - @shuo-wu @chriscchien
+  - [BUG] `test_replica_auto_balance_when_replica_on_unschedulable_node` Error in creating volume with nodeSelector and dataLocality parameters ([5745](https://github.com/longhorn/longhorn/issues/5745)) - @c3y1huang @roger-ryao
+  - [BUG] Unable to backup volume after NFS server IP change ([5856](https://github.com/longhorn/longhorn/issues/5856)) - @derekbit @roger-ryao
+  
+## Misc
+  
+  - [TASK] Check and update the networking doc & example YAMLs ([5651](https://github.com/longhorn/longhorn/issues/5651)) - @yangchiu @shuo-wu
+  
+## Contributors
+
+- @ChanYiLin
+- @PhanLe1010
+- @c3y1huang
+- @chriscchien
+- @derekbit
+- @ejweber
+- @innobead
+- @khushboo-rancher
+- @mantissahz
+- @roger-ryao
+- @shuo-wu
+- @smallteeths
+- @weizhe0422
+- @yangchiu

CHANGELOG/CHANGELOG-1.4.3.md

@@ -0,0 +1,74 @@
+## Release Note
+### **v1.4.3 released!** 🎆
+
+Longhorn v1.4.3 is the latest stable version of Longhorn 1.4.
+It introduces improvements and bug fixes in the areas of stability, resilience, and so on. Please try it out and provide feedback. Thanks for all the contributions!
+
+> For the definition of stable or latest release, please check [here](https://github.com/longhorn/longhorn#releases).
+
+## Installation
+
+> **Please ensure your Kubernetes cluster is at least v1.21 before installing v1.4.3.**
+
+Longhorn supports 3 installation ways including Rancher App Marketplace, Kubectl, and Helm. Follow the installation instructions [here](https://longhorn.io/docs/1.4.3/deploy/install/).
+
+## Upgrade
+
+> **Please read the [important notes](https://longhorn.io/docs/1.4.3/deploy/important-notes/) first and ensure your Kubernetes cluster is at least v1.21 before upgrading to Longhorn v1.4.3 from v1.3.x/v1.4.x, which are only supported source versions.**
+
+Follow the upgrade instructions [here](https://longhorn.io/docs/1.4.3/deploy/upgrade/).
+
+## Deprecation & Incompatibilities
+
+N/A
+
+## Known Issues after Release
+
+Please follow up on [here](https://github.com/longhorn/longhorn/wiki/Outstanding-Known-Issues-of-Releases) about any outstanding issues found after this release.
+
+
+## Improvement
+  
+  - [IMPROVEMENT] Assign the pods to the same node where the strict-local volume is present ([5448](https://github.com/longhorn/longhorn/issues/5448)) - @c3y1huang @chriscchien
+  
+## Resilience
+  
+  - [BUG] filesystem corrupted after delete instance-manager-r for a locality best-effort volume ([5801](https://github.com/longhorn/longhorn/issues/5801)) - @yangchiu @ChanYiLin @mantissahz
+  
+## Bugs
+  
+  - [BUG] 'Upgrade Engine' still shows up in a specific situation when engine already upgraded ([3063](https://github.com/longhorn/longhorn/issues/3063)) - @weizhe0422 @PhanLe1010 @smallteeths
+  - [BUG] DR volume even after activation remains in standby mode if there are one or more failed replicas. ([3069](https://github.com/longhorn/longhorn/issues/3069)) - @yangchiu @mantissahz
+  - [BUG] Prevent Longhorn uninstallation from getting stuck due to backups in error ([5868](https://github.com/longhorn/longhorn/issues/5868)) - @ChanYiLin @mantissahz
+  - [BUG]  Unable to create support bundle if the previous one stayed in ReadyForDownload phase ([5882](https://github.com/longhorn/longhorn/issues/5882)) - @c3y1huang @roger-ryao
+  - [BUG] share-manager for a given pvc keep restarting (other pvc are working fine) ([5954](https://github.com/longhorn/longhorn/issues/5954)) - @yangchiu @derekbit
+  - [BUG] Replica auto-rebalance doesn't respect node selector ([5971](https://github.com/longhorn/longhorn/issues/5971)) - @c3y1huang @roger-ryao
+  - [BUG] Extra snapshot generated when clone from a detached volume ([5986](https://github.com/longhorn/longhorn/issues/5986)) - @weizhe0422 @ejweber
+  - [BUG] User created snapshot deleted after node drain and uncordon ([5992](https://github.com/longhorn/longhorn/issues/5992)) - @yangchiu @mantissahz
+  - [BUG] In some specific situation, system backup auto deleted when creating another one ([6045](https://github.com/longhorn/longhorn/issues/6045)) - @c3y1huang @chriscchien
+  - [BUG] Backing Image deletion stuck if it's deleted during uploading process and bids is ready-for-transfer state ([6086](https://github.com/longhorn/longhorn/issues/6086)) - @WebberHuang1118 @chriscchien
+  - [BUG] Backing image manager fails when SELinux is enabled ([6108](https://github.com/longhorn/longhorn/issues/6108)) - @ejweber @chriscchien
+  - [BUG] test_dr_volume_with_restore_command_error failed ([6130](https://github.com/longhorn/longhorn/issues/6130)) - @mantissahz @roger-ryao
+  - [BUG] Longhorn doesn't remove the system backups crd on uninstallation ([6185](https://github.com/longhorn/longhorn/issues/6185)) - @c3y1huang @khushboo-rancher
+  - [BUG] Test case test_ha_backup_deletion_recovery failed in rhel or rockylinux arm64 environment ([6213](https://github.com/longhorn/longhorn/issues/6213)) - @yangchiu @ChanYiLin @mantissahz
+  - [BUG] Engine continues to attempt to rebuild replica while detaching ([6217](https://github.com/longhorn/longhorn/issues/6217)) - @yangchiu @ejweber
+  - [BUG] Unable to receive support bundle from UI when it's large (400MB+) ([6256](https://github.com/longhorn/longhorn/issues/6256)) - @c3y1huang @chriscchien 
+  - [BUG] Migration test case failed: unable to detach volume migration is not ready yet ([6238](https://github.com/longhorn/longhorn/issues/6238)) - @yangchiu @PhanLe1010 @khushboo-rancher
+  - [BUG] Restored Volumes stuck in attaching state ([6239](https://github.com/longhorn/longhorn/issues/6239)) - @derekbit @roger-ryao
+  
+## Contributors
+
+- @ChanYiLin
+- @PhanLe1010
+- @WebberHuang1118
+- @c3y1huang
+- @chriscchien
+- @derekbit
+- @ejweber
+- @innobead
+- @khushboo-rancher
+- @mantissahz
+- @roger-ryao
+- @smallteeths
+- @weizhe0422
+- @yangchiu

CHANGELOG/CHANGELOG-1.5.0.md

@@ -0,0 +1,301 @@
+## Release Note
+### **v1.5.0 released!** 🎆
+
+Longhorn v1.5.0 is the latest version of Longhorn 1.5.
+It introduces many enhancements, improvements, and bug fixes as described below including performance, stability, maintenance, resilience, and so on. Please try it and feedback. Thanks for all the contributions!
+
+> For the definition of stable or latest release, please check [here](https://github.com/longhorn/longhorn#releases).
+
+  - [v2 Data Engine based on SPDK - Preview](https://github.com/longhorn/longhorn/issues/5751)
+    > **Please note that this is a preview feature, so should not be used in any production environment. A preview feature is disabled by default and would be changed in the following versions until it becomes general availability.**
+    
+    In addition to the existing iSCSI stack (v1) data engine, we are introducing the v2 data engine based on SPDK (Storage Performance Development Kit). This release includes the introduction of volume lifecycle management, degraded volume handling, offline replica rebuilding, block device management, and orphaned replica management. For the performance benchmark and comparison with v1, check the report [here](https://longhorn.io/docs/1.5.0/spdk/performance-benchmark/).
+
+  - [Longhorn Volume Attachment](https://github.com/longhorn/longhorn/issues/3715)
+    Introducing the new Longhorn VolumeAttachment CR, which ensures exclusive attachment and supports automatic volume attachment and detachment for various headless operations such as volume cloning, backing image export, and recurring jobs.
+
+  - [Cluster Autoscaler - GA](https://github.com/longhorn/longhorn/issues/5238)
+    Cluster Autoscaler was initially introduced as an experimental feature in v1.3. After undergoing automatic validation on different public cloud Kubernetes distributions and receiving user feedback, it has now reached general availability.
+
+  - [Instance Manager Engine & Replica Consolidation](https://github.com/longhorn/longhorn/issues/5208)
+    Previously, there were two separate instance manager pods responsible for volume engine and replica process management. However, this setup required high resource usage, especially during live upgrades. In this release, we have merged these pods into a single instance manager, reducing the initial resource requirements.
+
+  - [Volume Backup Compression Methods](https://github.com/longhorn/longhorn/issues/5189)
+    Longhorn supports different compression methods for volume backups, including lz4, gzip, or no compression. This allows users to choose the most suitable method based on their data type and usage requirements.
+
+  - [Automatic Volume Trim Recurring Job](https://github.com/longhorn/longhorn/issues/5186)
+    While volume filesystem trim was introduced in v1.4, users had to perform the operation manually. From this release, users can create a recurring job that automatically runs the trim process, improving space efficiency without requiring human intervention.
+
+  - [RWX Volume Trim](https://github.com/longhorn/longhorn/issues/5143)
+    Longhorn supports filesystem trim for RWX (Read-Write-Many) volumes, expanding the trim functionality beyond RWO (Read-Write-Once) volumes only.
+
+  - [Upgrade Path Enforcement & Downgrade Prevention](https://github.com/longhorn/longhorn/issues/5131)
+    To ensure compatibility after an upgrade, we have implemented upgrade path enforcement. This prevents unintended downgrades and ensures the system and data remain intact.
+
+  - [Backing Image Management via CSI VolumeSnapshot](https://github.com/longhorn/longhorn/issues/5005)
+    Users can now utilize the unified CSI VolumeSnapshot interface to manage Backing Images similar to volume snapshots and backups.
+
+  - [Snapshot Cleanup & Delete Recurring Job](https://github.com/longhorn/longhorn/issues/3836)
+    Introducing two new recurring job types specifically designed for snapshot cleanup and deletion. These jobs allow users to remove unnecessary snapshots for better space efficiency.
+
+  - [CIFS Backup Store](https://github.com/longhorn/longhorn/issues/3599) & [Azure Backup Store](https://github.com/longhorn/longhorn/issues/1309)
+    To enhance users' backup strategies and align with data governance policies, Longhorn now supports additional backup storage protocols, including CIFS and Azure.
+
+  - [Kubernetes Upgrade Node Drain Policy](https://github.com/longhorn/longhorn/issues/3304)  
+    The new Node Drain Policy provides flexible strategies to protect volume data during Kubernetes upgrades or node maintenance operations. This ensures the integrity and availability of your volumes.
+
+## Installation
+
+> **Please ensure your Kubernetes cluster is at least v1.21 before installing Longhorn v1.5.0.**
+
+Longhorn supports 3 installation ways including Rancher App Marketplace, Kubectl, and Helm. Follow the installation instructions [here](https://longhorn.io/docs/1.5.0/deploy/install/).
+
+## Upgrade
+
+> **Please ensure your Kubernetes cluster is at least v1.21 before upgrading to Longhorn v1.5.0 from v1.4.x. Only support upgrading from 1.4.x.**
+
+Follow the upgrade instructions [here](https://longhorn.io/docs/1.5.0/deploy/upgrade/).
+
+## Deprecation & Incompatibilities
+
+Please check the [important notes](https://longhorn.io/docs/1.5.0/deploy/important-notes/) to know more about deprecated, removed, incompatible features and important changes. If you upgrade indirectly from an older version like v1.3.x, please also check the corresponding important note for each upgrade version path.
+
+## Known Issues after Release
+
+Please follow up on [here](https://github.com/longhorn/longhorn/wiki/Outstanding-Known-Issues-of-Releases) about any outstanding issues found after this release. 
+
+## Highlights
+  
+  - [DOC] Provide the user guide for Kubernetes upgrade ([494](https://github.com/longhorn/longhorn/issues/494)) - @PhanLe1010
+  - [FEATURE] Backups to Azure Blob Storage ([1309](https://github.com/longhorn/longhorn/issues/1309)) - @mantissahz @chriscchien
+  - [IMPROVEMENT] Use PDB to protect Longhorn components from unexpected drains ([3304](https://github.com/longhorn/longhorn/issues/3304)) - @yangchiu @PhanLe1010
+  - [FEATURE] CIFS Backup Store Support ([3599](https://github.com/longhorn/longhorn/issues/3599)) - @derekbit @chriscchien
+  - [IMPROVEMENT] Consolidate volume attach/detach implementation ([3715](https://github.com/longhorn/longhorn/issues/3715)) - @yangchiu @PhanLe1010
+  - [IMPROVEMENT] Periodically clean up volume snapshots ([3836](https://github.com/longhorn/longhorn/issues/3836)) - @c3y1huang @chriscchien
+  - [IMPROVEMENT] Introduce timeout mechanism for the sparse file syncing service ([4305](https://github.com/longhorn/longhorn/issues/4305)) - @yangchiu @ChanYiLin
+  - [IMPROVEMENT] Recurring jobs create new snapshots while being not able to clean up old ones ([4898](https://github.com/longhorn/longhorn/issues/4898)) - @mantissahz @chriscchien
+  - [FEATURE] BackingImage Management via VolumeSnapshot ([5005](https://github.com/longhorn/longhorn/issues/5005)) - @ChanYiLin @chriscchien
+  - [FEATURE] Upgrade path enforcement & downgrade prevention ([5131](https://github.com/longhorn/longhorn/issues/5131)) - @yangchiu @mantissahz
+  - [FEATURE] Support RWX volume trim ([5143](https://github.com/longhorn/longhorn/issues/5143)) - @derekbit @chriscchien
+  - [FEATURE] Auto Trim via recurring job ([5186](https://github.com/longhorn/longhorn/issues/5186)) - @c3y1huang @chriscchien
+  - [FEATURE] Introduce faster compression and multiple threads for volume backup & restore  ([5189](https://github.com/longhorn/longhorn/issues/5189)) - @derekbit @roger-ryao
+  - [FEATURE] Consolidate Instance Manager Engine & Replica for resource consumption reduction ([5208](https://github.com/longhorn/longhorn/issues/5208)) - @yangchiu @c3y1huang
+  - [FEATURE] Cluster Autoscaler Support GA ([5238](https://github.com/longhorn/longhorn/issues/5238)) - @yangchiu @c3y1huang
+  - [FEATURE] Update K8s version support and component/pkg/build dependencies for Longhorn 1.5 ([5595](https://github.com/longhorn/longhorn/issues/5595)) - @yangchiu @ejweber
+  - [FEATURE] Support SPDK Data Engine - Preview ([5751](https://github.com/longhorn/longhorn/issues/5751)) - @derekbit @shuo-wu @DamiaSan
+  
+## Enhancements
+  
+  - [FEATURE] Allow users to directly activate a restoring/DR volume as long as there is one ready replica. ([1512](https://github.com/longhorn/longhorn/issues/1512)) - @mantissahz @weizhe0422
+  - [REFACTOR] volume controller refactoring/split up, to simplify the control flow ([2527](https://github.com/longhorn/longhorn/issues/2527)) - @PhanLe1010 @chriscchien
+  - [FEATURE] Import and export SPDK longhorn volumes to longhorn sparse file directory ([4100](https://github.com/longhorn/longhorn/issues/4100)) - @DamiaSan
+  - [FEATURE] Add a global `storage reserved` setting for newly created longhorn nodes' disks ([4773](https://github.com/longhorn/longhorn/issues/4773)) - @mantissahz @chriscchien
+  - [FEATURE] Support backup volumes during system backup ([5011](https://github.com/longhorn/longhorn/issues/5011)) - @c3y1huang @chriscchien
+  - [FEATURE] Support SPDK lvol shallow copy for newly replica creation ([5217](https://github.com/longhorn/longhorn/issues/5217)) - @DamiaSan
+  - [FEATURE] Introduce longhorn-spdk-engine for SPDK volume management ([5282](https://github.com/longhorn/longhorn/issues/5282)) - @shuo-wu
+  - [FEATURE] Support replica-zone-soft-anti-affinity setting per volume ([5358](https://github.com/longhorn/longhorn/issues/5358)) - @ChanYiLin @smallteeths @chriscchien
+  - [FEATURE] Install Opt-In NetworkPolicies  ([5403](https://github.com/longhorn/longhorn/issues/5403)) - @yangchiu @ChanYiLin
+  - [FEATURE] Create Longhorn SPDK Engine component with basic fundamental functions ([5406](https://github.com/longhorn/longhorn/issues/5406)) - @shuo-wu
+  - [FEATURE] Add status APIs for shallow copy and IO pause/resume ([5647](https://github.com/longhorn/longhorn/issues/5647)) - @DamiaSan
+  - [FEATURE] Introduce a new disk type, disk management and replica scheduler for SPDK volumes ([5683](https://github.com/longhorn/longhorn/issues/5683)) - @derekbit @roger-ryao
+  - [FEATURE] Support replica scheduling for SPDK volume  ([5711](https://github.com/longhorn/longhorn/issues/5711)) - @derekbit
+  - [FEATURE] Create SPDK gRPC service for instance manager ([5712](https://github.com/longhorn/longhorn/issues/5712)) - @shuo-wu
+  - [FEATURE] Environment check script for Longhorn with SPDK ([5738](https://github.com/longhorn/longhorn/issues/5738)) - @derekbit @chriscchien
+  - [FEATURE] Deployment manifests for helping install SPDK dependencies, utilities and libraries ([5739](https://github.com/longhorn/longhorn/issues/5739)) - @yangchiu @derekbit
+  - [FEATURE] Implement Disk gRPC Service in Instance Manager for collecting SPDK disk statistics from SPDK gRPC service  ([5744](https://github.com/longhorn/longhorn/issues/5744)) - @derekbit @chriscchien
+  - [FEATURE] Support for SPDK RAID1 by setting the minimum number of base_bdevs to 1 ([5758](https://github.com/longhorn/longhorn/issues/5758)) - @yangchiu @DamiaSan
+  - [FEATURE] Add a global setting for enabling and disabling SPDK feature ([5778](https://github.com/longhorn/longhorn/issues/5778)) - @yangchiu @derekbit
+  - [FEATURE] Identify and manage orphaned lvols and raid bdevs if the associated `Volume` resources are not existing ([5827](https://github.com/longhorn/longhorn/issues/5827)) - @yangchiu @derekbit
+  - [FEATURE] Longhorn UI for SPDK feature ([5846](https://github.com/longhorn/longhorn/issues/5846)) - @smallteeths @chriscchien
+  - [FEATURE] UI modification to work with new AD mechanism (Longhorn UI -> Longhorn API) ([6004](https://github.com/longhorn/longhorn/issues/6004)) - @yangchiu @smallteeths
+  - [FEATURE] Replica offline rebuild over SPDK - data engine ([6067](https://github.com/longhorn/longhorn/issues/6067)) - @shuo-wu
+  - [FEATURE] Support automatic offline replica rebuilding of volumes using SPDK data engine ([6071](https://github.com/longhorn/longhorn/issues/6071)) - @yangchiu @derekbit
+  
+## Improvement
+  
+  - [IMPROVEMENT] Do not count the failure replica reuse failure caused by the disconnection ([1923](https://github.com/longhorn/longhorn/issues/1923)) - @yangchiu @mantissahz
+  - [IMPROVEMENT] Consider changing the over provisioning default/recommendation to 100% percentage (no over provisioning) ([2694](https://github.com/longhorn/longhorn/issues/2694)) - @c3y1huang @chriscchien
+  - [BUG] StorageClass of pv and pvc of a recovered pv should not always be default. ([3506](https://github.com/longhorn/longhorn/issues/3506)) - @ChanYiLin @smallteeths @roger-ryao
+  - [IMPROVEMENT] Auto-attach volume for K8s CSI snapshot ([3726](https://github.com/longhorn/longhorn/issues/3726)) - @weizhe0422 @PhanLe1010
+  - [IMPROVEMENT] Change Longhorn API to create/delete snapshot CRs instead of calling engine CLI ([3995](https://github.com/longhorn/longhorn/issues/3995)) - @yangchiu @PhanLe1010
+  - [IMPROVEMENT] Add support for crypto parameters for RWX volumes ([4829](https://github.com/longhorn/longhorn/issues/4829)) - @mantissahz @roger-ryao
+  - [IMPROVEMENT] Remove the global setting `mkfs-ext4-parameters` ([4914](https://github.com/longhorn/longhorn/issues/4914)) - @ejweber @roger-ryao
+  - [IMPROVEMENT] Move all snapshot related settings at one place. ([4930](https://github.com/longhorn/longhorn/issues/4930)) - @smallteeths @roger-ryao
+  - [IMPROVEMENT] Remove system managed component image settings  ([5028](https://github.com/longhorn/longhorn/issues/5028)) - @mantissahz @chriscchien
+  - [IMPROVEMENT] Set default `engine-replica-timeout` value for engine controller start command ([5031](https://github.com/longhorn/longhorn/issues/5031)) - @derekbit @chriscchien
+  - [IMPROVEMENT] Support bundle collects dmesg, syslog and related information of longhorn nodes ([5073](https://github.com/longhorn/longhorn/issues/5073)) - @weizhe0422 @roger-ryao
+  - [IMPROVEMENT] Collect volume, system, feature info for metrics for better usage awareness ([5235](https://github.com/longhorn/longhorn/issues/5235)) - @c3y1huang @chriscchien @roger-ryao
+  - [IMPROVEMENT] Update uninstallation info to include the 'Deleting Confirmation Flag'  in chart ([5250](https://github.com/longhorn/longhorn/issues/5250)) - @PhanLe1010 @roger-ryao
+  - [IMPROVEMENT] Disable Revision Counter for Strict-Local dataLocality ([5257](https://github.com/longhorn/longhorn/issues/5257)) - @derekbit @roger-ryao
+  - [IMPROVEMENT] Fix Guaranteed Engine Manager CPU recommendation formula in UI ([5338](https://github.com/longhorn/longhorn/issues/5338)) - @c3y1huang @smallteeths @roger-ryao
+  - [IMPROVEMENT] Update PSP validation in the Longhorn upstream chart  ([5339](https://github.com/longhorn/longhorn/issues/5339)) - @yangchiu @PhanLe1010
+  - [IMPROVEMENT] Update ganesha nfs to 4.2.3 ([5356](https://github.com/longhorn/longhorn/issues/5356)) - @derekbit @roger-ryao
+  - [IMPROVEMENT] Set write-cache of longhorn block device to off explicitly ([5382](https://github.com/longhorn/longhorn/issues/5382)) - @derekbit @chriscchien
+  - [IMPROVEMENT] Clean up unused backupstore mountpoint ([5391](https://github.com/longhorn/longhorn/issues/5391)) - @derekbit @chriscchien
+  - [DOC] Update Kubernetes version info to have consistent description from the longhorn documentation in chart ([5399](https://github.com/longhorn/longhorn/issues/5399)) - @ChanYiLin @roger-ryao
+  - [IMPROVEMENT] Fix BackingImage uploading/downloading flow to prevent client timeout ([5443](https://github.com/longhorn/longhorn/issues/5443)) - @ChanYiLin @chriscchien
+  - [IMPROVEMENT] Assign the pods to the same node where the strict-local volume is present ([5448](https://github.com/longhorn/longhorn/issues/5448)) - @c3y1huang @chriscchien
+  - [IMPROVEMENT] Have explicitly message when trying to attach a volume which it's engine and replica were on deleted node  ([5545](https://github.com/longhorn/longhorn/issues/5545)) - @ChanYiLin @chriscchien
+  - [IMPROVEMENT] Create a new setting so that Longhorn removes PDB for instance-manager-r that doesn't have any running instance inside it ([5549](https://github.com/longhorn/longhorn/issues/5549)) - @PhanLe1010 @roger-ryao
+  - [IMPROVEMENT] Merge conversion/admission webhook and recovery backend services into longhorn-manager ([5590](https://github.com/longhorn/longhorn/issues/5590)) - @ChanYiLin @chriscchien
+  - [IMPROVEMENT][UI] Recurring jobs create new snapshots while being not able to clean up old one ([5610](https://github.com/longhorn/longhorn/issues/5610)) - @mantissahz @smallteeths @roger-ryao
+  - [IMPROVEMENT] Only activate replica if it doesn't have deletion timestamp during volume engine upgrade ([5632](https://github.com/longhorn/longhorn/issues/5632)) - @PhanLe1010 @roger-ryao
+  - [IMPROVEMENT] Clean up backup target if the backup target setting is unset ([5655](https://github.com/longhorn/longhorn/issues/5655)) - @yangchiu @ChanYiLin
+  - [IMPROVEMENT] Bump CSI sidecar components' version ([5672](https://github.com/longhorn/longhorn/issues/5672)) - @yangchiu @ejweber
+  - [IMPROVEMENT] Configure log level of Longhorn components ([5888](https://github.com/longhorn/longhorn/issues/5888)) - @ChanYiLin @weizhe0422
+  - [IMPROVEMENT] Remove development toolchain from Longhorn images ([6022](https://github.com/longhorn/longhorn/issues/6022)) - @ChanYiLin @derekbit
+  - [IMPROVEMENT] Reduce replica process's number of allocated ports  ([6079](https://github.com/longhorn/longhorn/issues/6079)) - @ChanYiLin @derekbit
+  - [IMPROVEMENT] UI supports automatic replica rebuilding for SPDK volumes ([6107](https://github.com/longhorn/longhorn/issues/6107)) - @smallteeths @roger-ryao
+  - [IMPROVEMENT] Minor UX changes for Longhorn SPDK ([6126](https://github.com/longhorn/longhorn/issues/6126)) - @derekbit @roger-ryao
+  - [IMPROVEMENT] Instance manager spdk_tgt resilience due to spdk_tgt crash ([6155](https://github.com/longhorn/longhorn/issues/6155)) - @yangchiu @derekbit
+  - [IMPROVEMENT] Determine number of replica/engine port count in longhorn-manager (control plane) instead ([6163](https://github.com/longhorn/longhorn/issues/6163)) - @derekbit @chriscchien
+  - [IMPROVEMENT] SPDK client should functions after encountering decoding error ([6191](https://github.com/longhorn/longhorn/issues/6191)) - @yangchiu @shuo-wu
+  
+## Performance
+  
+  - [REFACTORING] Evaluate the impact of removing the client side compression for backup blocks ([1409](https://github.com/longhorn/longhorn/issues/1409)) - @derekbit
+  
+## Resilience
+  
+  - [BUG] If backing image downloading fails on one node, it doesn't try on other nodes. ([3746](https://github.com/longhorn/longhorn/issues/3746)) - @ChanYiLin
+  - [BUG] Replica rebuilding caused by rke2/kubelet restart ([5340](https://github.com/longhorn/longhorn/issues/5340)) - @derekbit @chriscchien
+  - [BUG] Volume restoration will never complete if attached node is down ([5464](https://github.com/longhorn/longhorn/issues/5464)) - @derekbit @weizhe0422 @chriscchien
+  - [BUG] Node disconnection test failed ([5476](https://github.com/longhorn/longhorn/issues/5476)) - @yangchiu @derekbit
+  - [BUG] Physical node down test failed ([5477](https://github.com/longhorn/longhorn/issues/5477)) - @derekbit @chriscchien
+  - [BUG] Backing image with sync failure ([5481](https://github.com/longhorn/longhorn/issues/5481)) - @ChanYiLin @roger-ryao
+  - [BUG] share-manager pod failed to restart after kubelet restart ([5507](https://github.com/longhorn/longhorn/issues/5507)) - @yangchiu @derekbit
+  - [BUG] Directly mark replica as failed if the node is deleted ([5542](https://github.com/longhorn/longhorn/issues/5542)) - @weizhe0422 @roger-ryao
+  - [BUG] RWX volume is stuck at detaching when the attached node is down  ([5558](https://github.com/longhorn/longhorn/issues/5558)) - @derekbit @roger-ryao
+  - [BUG] Unable to export RAID1 bdev in degraded state  ([5650](https://github.com/longhorn/longhorn/issues/5650)) - @chriscchien @DamiaSan
+  - [BUG] Backup monitor gets stuck in an infinite loop if backup isn't found ([5662](https://github.com/longhorn/longhorn/issues/5662)) - @derekbit @chriscchien
+  - [BUG] Resources such as replicas are somehow not mutated when network is unstable  ([5762](https://github.com/longhorn/longhorn/issues/5762)) - @derekbit @roger-ryao
+  - [BUG] filesystem corrupted after delete instance-manager-r for a locality best-effort volume ([5801](https://github.com/longhorn/longhorn/issues/5801)) - @yangchiu @ChanYiLin @mantissahz
+  
+## Stability
+  
+  - [BUG] nfs backup broken - NFS server: mkdir - file exists ([4626](https://github.com/longhorn/longhorn/issues/4626)) - @yangchiu @derekbit
+  - [BUG] Memory leak in CSI plugin caused by stuck umount processes if the RWX volume is already gone ([5296](https://github.com/longhorn/longhorn/issues/5296)) - @derekbit @roger-ryao
+  
+## Bugs
+  
+  - [BUG] 'Upgrade Engine' still shows up in a specific situation when engine already upgraded ([3063](https://github.com/longhorn/longhorn/issues/3063)) - @weizhe0422 @PhanLe1010 @smallteeths
+  - [BUG] DR volume even after activation remains in standby mode if there are one or more failed replicas. ([3069](https://github.com/longhorn/longhorn/issues/3069)) - @yangchiu @mantissahz
+  - [BUG] volume not able to attach with raw type backing image ([3437](https://github.com/longhorn/longhorn/issues/3437)) - @yangchiu @ChanYiLin
+  - [BUG] Delete a uploading backing image, the corresponding LH temp file is not deleted ([3682](https://github.com/longhorn/longhorn/issues/3682)) - @ChanYiLin @chriscchien
+  - [BUG] Cloned PVC from detached volume will stuck at not ready for workload ([3692](https://github.com/longhorn/longhorn/issues/3692)) - @PhanLe1010 @chriscchien
+  - [BUG] Block device volume failed to unmount when it is detached unexpectedly ([3778](https://github.com/longhorn/longhorn/issues/3778)) - @PhanLe1010 @chriscchien
+  - [BUG] After migration of Longhorn from Rancher old UI to dashboard, the csi-plugin doesn't update ([4519](https://github.com/longhorn/longhorn/issues/4519)) - @mantissahz @roger-ryao
+  - [BUG] Volumes Stuck in Attach/Detach Loop when running on OpenShift/OKD ([4988](https://github.com/longhorn/longhorn/issues/4988)) - @ChanYiLin
+  - [BUG] Longhorn 1.3.2 fails to backup & restore volumes behind Internet proxy  ([5054](https://github.com/longhorn/longhorn/issues/5054)) - @mantissahz @chriscchien
+  - [BUG] Instance manager pod does not respect of node taint? ([5161](https://github.com/longhorn/longhorn/issues/5161)) - @ejweber
+  - [BUG] RWX doesn't work with release 1.4.0 due to end grace update error from recovery backend ([5183](https://github.com/longhorn/longhorn/issues/5183)) - @derekbit @chriscchien
+  - [BUG] Incorrect indentation of charts/questions.yaml ([5196](https://github.com/longhorn/longhorn/issues/5196)) - @mantissahz @roger-ryao
+  - [BUG] Updating option "Allow snapshots removal during trim" for old volumes failed  ([5218](https://github.com/longhorn/longhorn/issues/5218)) - @shuo-wu @roger-ryao
+  - [BUG] Since 1.4.0 RWX volume failing regularly ([5224](https://github.com/longhorn/longhorn/issues/5224)) - @derekbit
+  - [BUG] Can not create backup in engine image not fully deployed cluster ([5248](https://github.com/longhorn/longhorn/issues/5248)) - @ChanYiLin @roger-ryao
+  - [BUG] Incorrect router retry mechanism ([5259](https://github.com/longhorn/longhorn/issues/5259)) - @mantissahz @chriscchien
+  - [BUG] System Backup is stuck at Uploading if there are PVs not provisioned by CSI driver ([5286](https://github.com/longhorn/longhorn/issues/5286)) - @c3y1huang @chriscchien
+  - [BUG] Sync up with backup target during DR volume activation ([5292](https://github.com/longhorn/longhorn/issues/5292)) - @yangchiu @weizhe0422
+  - [BUG] environment_check.sh does not handle different kernel versions in cluster correctly ([5304](https://github.com/longhorn/longhorn/issues/5304)) - @achims311 @roger-ryao
+  - [BUG] instance-manager-r high memory consumption ([5312](https://github.com/longhorn/longhorn/issues/5312)) - @derekbit @roger-ryao
+  - [BUG] Unable to upgrade longhorn from v1.3.2 to master-head ([5368](https://github.com/longhorn/longhorn/issues/5368)) - @yangchiu @derekbit
+  - [BUG] Modify engineManagerCPURequest and replicaManagerCPURequest won't raise resource request in instance-manager-e pod ([5419](https://github.com/longhorn/longhorn/issues/5419)) - @c3y1huang
+  - [BUG] Error message not consistent between create/update recurring job when retain number greater than 50 ([5434](https://github.com/longhorn/longhorn/issues/5434)) - @c3y1huang @chriscchien
+  - [BUG] Do not copy Host header to API requests forwarded to Longhorn Manager ([5438](https://github.com/longhorn/longhorn/issues/5438)) - @yangchiu @smallteeths
+  - [BUG] RWX Volume attachment is getting Failed ([5456](https://github.com/longhorn/longhorn/issues/5456)) - @derekbit
+  - [BUG] test case test_backup_lock_deletion_during_restoration failed ([5458](https://github.com/longhorn/longhorn/issues/5458)) - @yangchiu @derekbit
+  - [BUG] Unable to create support bundle agent pod in air-gap environment ([5467](https://github.com/longhorn/longhorn/issues/5467)) - @yangchiu @c3y1huang
+  - [BUG] Example of data migration doesn't work for hidden/./dot-files) ([5484](https://github.com/longhorn/longhorn/issues/5484)) - @hedefalk @shuo-wu @chriscchien
+  - [BUG] Upgrade engine --> spec.restoreVolumeRecurringJob and spec.snapshotDataIntegrity Unsupported value ([5485](https://github.com/longhorn/longhorn/issues/5485)) - @yangchiu @derekbit
+  - [BUG] test case test_dr_volume_with_backup_block_deletion failed ([5489](https://github.com/longhorn/longhorn/issues/5489)) - @yangchiu @derekbit
+  - [BUG] Bulk backup deletion cause restoring volume to finish with attached state. ([5506](https://github.com/longhorn/longhorn/issues/5506)) - @ChanYiLin @roger-ryao
+  - [BUG] volume expansion starts for no reason, gets stuck on current size > expected size ([5513](https://github.com/longhorn/longhorn/issues/5513)) - @mantissahz @roger-ryao
+  - [BUG] RWX volume attachment failed if tried more enough times ([5537](https://github.com/longhorn/longhorn/issues/5537)) - @yangchiu @derekbit
+  - [BUG] instance-manager-e emits `Wait for process pvc-xxxx to shutdown` constantly ([5575](https://github.com/longhorn/longhorn/issues/5575)) - @derekbit @roger-ryao
+  - [BUG] Support bundle kit should respect node selector & taint toleration ([5614](https://github.com/longhorn/longhorn/issues/5614)) - @yangchiu @c3y1huang
+  - [BUG] Value overlapped in page Instance Manager Image ([5622](https://github.com/longhorn/longhorn/issues/5622)) - @smallteeths @chriscchien
+  - [BUG] Updated Rocky 9 (and others) can't attach due to SELinux ([5627](https://github.com/longhorn/longhorn/issues/5627)) - @yangchiu @ejweber
+  - [BUG] Fix misleading error messages when creating a mount point for a backup store ([5630](https://github.com/longhorn/longhorn/issues/5630)) - @derekbit
+  - [BUG] Instance manager PDB created with wrong selector thus blocking the draining of the wrongly selected node forever ([5680](https://github.com/longhorn/longhorn/issues/5680)) - @PhanLe1010 @chriscchien
+  - [BUG] During volume live engine upgrade, if the replica pod is killed, the volume is stuck in upgrading forever ([5684](https://github.com/longhorn/longhorn/issues/5684)) - @yangchiu @PhanLe1010
+  - [BUG] Instance manager PDBs cannot be removed if the longhorn-manager pod on its spec node is not available ([5688](https://github.com/longhorn/longhorn/issues/5688)) - @PhanLe1010 @roger-ryao
+  - [BUG] Rebuild rebuilding is possibly issued to a wrong replica ([5709](https://github.com/longhorn/longhorn/issues/5709)) - @ejweber @roger-ryao
+  - [BUG] Observing repilca on new IM-r before upgrading of volume ([5729](https://github.com/longhorn/longhorn/issues/5729)) - @c3y1huang
+  - [BUG] longhorn upgrade is not upgrading engineimage ([5740](https://github.com/longhorn/longhorn/issues/5740)) - @shuo-wu @chriscchien
+  - [BUG] `test_replica_auto_balance_when_replica_on_unschedulable_node` Error in creating volume with nodeSelector and dataLocality parameters ([5745](https://github.com/longhorn/longhorn/issues/5745)) - @c3y1huang @roger-ryao
+  - [BUG] Unable to backup volume after NFS server IP change ([5856](https://github.com/longhorn/longhorn/issues/5856)) - @derekbit @roger-ryao
+  - [BUG] Prevent Longhorn uninstallation from getting stuck due to backups in error ([5868](https://github.com/longhorn/longhorn/issues/5868)) - @ChanYiLin @mantissahz
+  - [BUG]  Unable to create support bundle if the previous one stayed in ReadyForDownload phase ([5882](https://github.com/longhorn/longhorn/issues/5882)) - @c3y1huang @roger-ryao
+  - [BUG] share-manager for a given pvc keep restarting (other pvc are working fine) ([5954](https://github.com/longhorn/longhorn/issues/5954)) - @yangchiu @derekbit
+  - [BUG] Replica auto-rebalance doesn't respect node selector ([5971](https://github.com/longhorn/longhorn/issues/5971)) - @c3y1huang @roger-ryao
+  - [BUG] Volume detached automatically after upgrade Longhorn ([5983](https://github.com/longhorn/longhorn/issues/5983)) - @yangchiu @PhanLe1010
+  - [BUG] Extra snapshot generated when clone from a detached volume ([5986](https://github.com/longhorn/longhorn/issues/5986)) - @weizhe0422 @ejweber
+  - [BUG] User created snapshot deleted after node drain and uncordon ([5992](https://github.com/longhorn/longhorn/issues/5992)) - @yangchiu @mantissahz
+  - [BUG] Webhook PDBs are not removed after upgrading to master-head ([6026](https://github.com/longhorn/longhorn/issues/6026)) - @weizhe0422 @PhanLe1010
+  - [BUG] In some specific situation, system backup auto deleted when creating another one ([6045](https://github.com/longhorn/longhorn/issues/6045)) - @c3y1huang @chriscchien
+  - [BUG] Backing Image deletion stuck if it's deleted during uploading process and bids is ready-for-transfer state ([6086](https://github.com/longhorn/longhorn/issues/6086)) - @WebberHuang1118 @chriscchien
+  - [BUG] A backup target backed by a Samba server is not recognized ([6100](https://github.com/longhorn/longhorn/issues/6100)) - @derekbit @weizhe0422
+  - [BUG] Backing image manager fails when SELinux is enabled ([6108](https://github.com/longhorn/longhorn/issues/6108)) - @ejweber @chriscchien
+  - [BUG] Force delete volume make SPDK disk unschedule ([6110](https://github.com/longhorn/longhorn/issues/6110)) - @derekbit
+  - [BUG] share-manager terminated during Longhorn upgrading causes rwx volume not working ([6120](https://github.com/longhorn/longhorn/issues/6120)) - @yangchiu @derekbit
+  - [BUG] SPDK Volume snapshotList API Error ([6123](https://github.com/longhorn/longhorn/issues/6123)) - @derekbit @chriscchien
+  - [BUG] test_recurring_jobs_allow_detached_volume failed ([6124](https://github.com/longhorn/longhorn/issues/6124)) - @ChanYiLin @roger-ryao
+  - [BUG] Cron job triggered replica rebuilding keeps repeating itself after corrupting snapshot data ([6129](https://github.com/longhorn/longhorn/issues/6129)) - @yangchiu @mantissahz
+  - [BUG] test_dr_volume_with_restore_command_error failed ([6130](https://github.com/longhorn/longhorn/issues/6130)) - @mantissahz @roger-ryao
+  - [BUG] RWX volume remains attached after workload deleted if it's upgraded from v1.4.2 ([6139](https://github.com/longhorn/longhorn/issues/6139)) - @PhanLe1010 @chriscchien
+  - [BUG] timestamp or checksum not matched in test_snapshot_hash_detect_corruption test case ([6145](https://github.com/longhorn/longhorn/issues/6145)) - @yangchiu @derekbit
+  - [BUG] When a v2 volume is attached in maintenance mode, removing a replica will lead to volume stuck in attaching-detaching loop  ([6166](https://github.com/longhorn/longhorn/issues/6166)) - @derekbit @chriscchien
+  - [BUG] Misleading offline rebuilding hint if offline rebuilding is not enabled ([6169](https://github.com/longhorn/longhorn/issues/6169)) - @smallteeths @roger-ryao
+  - [BUG] Longhorn doesn't remove the system backups crd on uninstallation ([6185](https://github.com/longhorn/longhorn/issues/6185)) - @c3y1huang @khushboo-rancher
+  - [BUG] Volume attachment related error logs in uninstaller pod ([6197](https://github.com/longhorn/longhorn/issues/6197)) - @yangchiu @PhanLe1010
+  - [BUG] Test case test_ha_backup_deletion_recovery failed in rhel or rockylinux arm64 environment ([6213](https://github.com/longhorn/longhorn/issues/6213)) - @yangchiu @ChanYiLin @mantissahz
+  - [BUG] migration test cases could fail due to unexpected volume controllers and replicas status ([6215](https://github.com/longhorn/longhorn/issues/6215)) - @yangchiu @PhanLe1010
+  - [BUG] Engine continues to attempt to rebuild replica while detaching ([6217](https://github.com/longhorn/longhorn/issues/6217)) - @yangchiu @ejweber
+  
+## Misc
+  
+  - [TASK] Remove deprecated volume spec recurringJobs and storageClass recurringJobs field ([2865](https://github.com/longhorn/longhorn/issues/2865)) - @c3y1huang @chriscchien
+  - [TASK] Remove deprecated fields after CRD API version bump ([3289](https://github.com/longhorn/longhorn/issues/3289)) - @c3y1huang @roger-ryao
+  - [TASK] Replace jobq lib with an alternative way for listing remote backup volumes and info ([4176](https://github.com/longhorn/longhorn/issues/4176)) - @ChanYiLin @chriscchien
+  - [DOC] Update the Longhorn document in Uninstalling Longhorn using kubectl ([4841](https://github.com/longhorn/longhorn/issues/4841)) - @roger-ryao
+  - [TASK] Remove a deprecated feature `disable-replica-rebuild` from longhorn-manager ([4997](https://github.com/longhorn/longhorn/issues/4997)) - @ejweber @chriscchien
+  - [TASK]  Update the distro matrix supports on Longhorn docs for 1.5 ([5177](https://github.com/longhorn/longhorn/issues/5177)) - @yangchiu
+  - [TASK] Clarify if any upcoming K8s API deprecation/removal will impact Longhorn 1.4 ([5180](https://github.com/longhorn/longhorn/issues/5180)) - @PhanLe1010
+  - [TASK] Revert affinity for Longhorn user deployed components ([5191](https://github.com/longhorn/longhorn/issues/5191)) - @weizhe0422 @ejweber
+  - [TASK] Add GitHub action for CI to lib repos for supporting dependency bot ([5239](https://github.com/longhorn/longhorn/issues/5239)) - 
+  - [DOC] Update the readme of longhorn-spdk-engine about using new Longhorn (RAID1) bdev ([5256](https://github.com/longhorn/longhorn/issues/5256)) - @DamiaSan
+  - [TASK][UI] add new recurring job tasks ([5272](https://github.com/longhorn/longhorn/issues/5272)) - @smallteeths @chriscchien
+  - [DOC] Update the node maintenance doc to cover upgrade prerequisites for Rancher ([5278](https://github.com/longhorn/longhorn/issues/5278)) - @PhanLe1010
+  - [TASK] Run build-engine-test-images automatically when having incompatible engine on master ([5400](https://github.com/longhorn/longhorn/issues/5400)) - @yangchiu
+  - [TASK] Update k8s.gcr.io to registry.k8s.io in repos ([5432](https://github.com/longhorn/longhorn/issues/5432)) - @yangchiu
+  - [TASK][UI] add new recurring job task - filesystem trim ([5529](https://github.com/longhorn/longhorn/issues/5529)) - @smallteeths @chriscchien
+  - doc: update prerequisites in chart readme to make it consistent with documentation v1.3.x ([5531](https://github.com/longhorn/longhorn/pull/5531)) - @ChanYiLin
+  - [FEATURE] Remove deprecated `allow-node-drain-with-last-healthy-replica` ([5620](https://github.com/longhorn/longhorn/issues/5620)) - @weizhe0422 @PhanLe1010
+  - [FEATURE] Set recurring jobs to PVCs ([5791](https://github.com/longhorn/longhorn/issues/5791)) - @yangchiu @c3y1huang
+  - [TASK] Automatically update crds.yaml in longhorn repo from longhorn-manager repo ([5854](https://github.com/longhorn/longhorn/issues/5854)) - @yangchiu
+  - [IMPROVEMENT] Remove privilege requirement from lifecycle jobs ([5862](https://github.com/longhorn/longhorn/issues/5862)) - @mantissahz @chriscchien
+  - [TASK][UI] support new aio typed instance managers ([5876](https://github.com/longhorn/longhorn/issues/5876)) - @smallteeths @chriscchien
+  - [TASK] Remove `Guaranteed Engine Manager CPU`, `Guaranteed Replica Manager CPU`, and `Guaranteed Engine CPU` settings. ([5917](https://github.com/longhorn/longhorn/issues/5917)) - @c3y1huang @roger-ryao
+  - [TASK][UI] Support volume backup policy ([6028](https://github.com/longhorn/longhorn/issues/6028)) - @smallteeths @chriscchien
+  - [TASK] Reduce BackupConcurrentLimit and RestoreConcurrentLimit default values  ([6135](https://github.com/longhorn/longhorn/issues/6135)) - @derekbit @chriscchien
+  
+## Contributors
+
+- @ChanYiLin
+- @DamiaSan
+- @PhanLe1010
+- @WebberHuang1118
+- @achims311
+- @c3y1huang
+- @chriscchien
+- @derekbit
+- @ejweber
+- @hedefalk
+- @innobead
+- @khushboo-rancher
+- @mantissahz
+- @roger-ryao
+- @shuo-wu
+- @smallteeths
+- @weizhe0422
+- @yangchiu

CHANGELOG/CHANGELOG-1.5.1.md

@@ -0,0 +1,65 @@
+## Release Note
+### **v1.5.1 released!** 🎆
+
+Longhorn v1.5.1 is the latest version of Longhorn 1.5.
+This release introduces bug fixes as described below about 1.5.0 upgrade issues, stability, troubleshooting and so on. Please try it and feedback. Thanks for all the contributions!
+
+> For the definition of stable or latest release, please check [here](https://github.com/longhorn/longhorn#releases).
+
+## Installation
+
+> **Please ensure your Kubernetes cluster is at least v1.21 before installing v1.5.1.**
+
+Longhorn supports 3 installation ways including Rancher App Marketplace, Kubectl, and Helm. Follow the installation instructions [here](https://longhorn.io/docs/1.5.1/deploy/install/).
+
+## Upgrade
+
+> **Please read the [important notes](https://longhorn.io/docs/1.5.1/deploy/important-notes/) first and ensure your Kubernetes cluster is at least v1.21 before upgrading to Longhorn v1.5.1 from v1.4.x/v1.5.0, which are only supported source versions.**
+
+Follow the upgrade instructions [here](https://longhorn.io/docs/1.5.1/deploy/upgrade/).
+
+## Deprecation & Incompatibilities
+
+N/A
+
+## Known Issues after Release
+
+Please follow up on [here](https://github.com/longhorn/longhorn/wiki/Outstanding-Known-Issues-of-Releases) about any outstanding issues found after this release.
+
+## Improvement
+  
+  - [IMPROVEMENT] Implement/fix the unit tests of Volume Attachment and volume controller ([6005](https://github.com/longhorn/longhorn/issues/6005)) - @PhanLe1010
+  - [QUESTION] Repetitive warnings and errors in a new longhorn setup ([6257](https://github.com/longhorn/longhorn/issues/6257)) - @derekbit @c3y1huang @roger-ryao
+  
+## Resilience
+  
+  - [BUG] 1.5.0 Upgrade: Longhorn conversion webhook server fails ([6259](https://github.com/longhorn/longhorn/issues/6259)) - @derekbit @roger-ryao
+  - [BUG] Race leaves snapshot CRs that cannot be deleted ([6298](https://github.com/longhorn/longhorn/issues/6298)) - @yangchiu @PhanLe1010 @ejweber
+  
+## Bugs
+  
+  - [BUG] Engine continues to attempt to rebuild replica while detaching ([6217](https://github.com/longhorn/longhorn/issues/6217)) - @yangchiu @ejweber
+  - [BUG] Upgrade to 1.5.0 failed: validator.longhorn.io denied the request if having orphan resources ([6246](https://github.com/longhorn/longhorn/issues/6246)) - @derekbit @roger-ryao
+  - [BUG] Unable to receive support bundle from UI when it's large (400MB+) ([6256](https://github.com/longhorn/longhorn/issues/6256)) - @c3y1huang @chriscchien
+  - [BUG] Longhorn Manager Pods CrashLoop after upgrade from 1.4.0 to 1.5.0 while backing up volumes ([6264](https://github.com/longhorn/longhorn/issues/6264)) - @ChanYiLin @roger-ryao
+  - [BUG] Can not delete type=`bi` VolumeSnapshot if related backing image not exist ([6266](https://github.com/longhorn/longhorn/issues/6266)) - @ChanYiLin @chriscchien
+  - [BUG] 1.5.0: AttachVolume.Attach failed for volume, the volume is currently attached to a different node ([6287](https://github.com/longhorn/longhorn/issues/6287)) - @yangchiu @derekbit
+  - [BUG] test case test_setting_priority_class failed in master and v1.5.x ([6319](https://github.com/longhorn/longhorn/issues/6319)) - @derekbit @chriscchien
+  - [BUG] Unused webhook and recovery backend deployment left in helm chart ([6252](https://github.com/longhorn/longhorn/issues/6252)) - @ChanYiLin @chriscchien
+  
+## Misc
+  
+  - [DOC] v1.5.0 additional outgoing firewall ports need to be opened 9501 9502 9503 ([6317](https://github.com/longhorn/longhorn/issues/6317)) - @ChanYiLin @chriscchien
+  
+## Contributors
+
+- @ChanYiLin
+- @PhanLe1010
+- @c3y1huang
+- @chriscchien
+- @derekbit
+- @ejweber
+- @innobead
+- @roger-ryao
+- @yangchiu
+

MAINTAINERS

@@ -3,5 +3,6 @@ The list of current Longhorn maintainers:
 Name, <Email>, @GitHubHandle
 Sheng Yang, <sheng@yasker.org>, @yasker
 Shuo Wu, <shuo.wu@suse.com>, @shuo-wu
-Joshua Moody, <joshua.moody@suse.com>, @joshimoo
 David Ko, <dko@suse.com>, @innobead
+Derek Su, <derek.su@suse.com>, @derekbit
+Phan Le, <phan.le@suse.com>, @PhanLe1010

README.md

@@ -31,17 +31,18 @@ You can read more technical details of Longhorn [here](https://longhorn.io/).
 
 > **NOTE**:
 > - __\<version\>*__ means the release branch is under active support and will have periodic follow-up patch releases.
-> - __Latest__ release means the release version is the most recent release of the release branch.
-> - __Stable__ release means the release version is stable and has been widely adopted by users.
+> - __Latest__ release means the version is the latest release of the newest release branch.
+> - __Stable__ release means the version is stable and has been widely adopted by users.
 
 https://github.com/longhorn/longhorn/releases
 
 | Release   | Version | Type           | Release Note (Changelog)                                       | Important Note                                              |
-|-----------|---------|----------------|----------------------------------------------------------------|-------------------------------------------------------------| 
-| **1.4***  | 1.4.2   | Latest, Stable | [🔗](https://github.com/longhorn/longhorn/releases/tag/v1.4.2) | [🔗](https://longhorn.io/docs/1.4.2/deploy/important-notes) |
-| **1.3***  | 1.3.3   | Latest, Stable | [🔗](https://github.com/longhorn/longhorn/releases/tag/v1.3.3) | [🔗](https://longhorn.io/docs/1.3.3/deploy/important-notes) |
-| 1.2       | 1.2.6   | Latest, Stable | [🔗](https://github.com/longhorn/longhorn/releases/tag/v1.2.6) | [🔗](https://longhorn.io/docs/1.2.6/deploy/important-notes) |
-| 1.1       | 1.1.3   | Latest, Stable | [🔗](https://github.com/longhorn/longhorn/releases/tag/v1.1.3) |                                                             |
+|-----------|---------|----------------|----------------------------------------------------------------|-------------------------------------------------------------|
+| **1.5***  | 1.5.1   | Latest         | [🔗](https://github.com/longhorn/longhorn/releases/tag/v1.5.1) | [🔗](https://longhorn.io/docs/1.5.1/deploy/important-notes) |
+| **1.4***  | 1.4.4   | Stable         | [🔗](https://github.com/longhorn/longhorn/releases/tag/v1.4.4) | [🔗](https://longhorn.io/docs/1.4.4/deploy/important-notes) |
+| 1.3       | 1.3.3   | Stable         | [🔗](https://github.com/longhorn/longhorn/releases/tag/v1.3.3) | [🔗](https://longhorn.io/docs/1.3.3/deploy/important-notes) |
+| 1.2       | 1.2.6   | Stable         | [🔗](https://github.com/longhorn/longhorn/releases/tag/v1.2.6) | [🔗](https://longhorn.io/docs/1.2.6/deploy/important-notes) |
+| 1.1       | 1.1.3   | Stable         | [🔗](https://github.com/longhorn/longhorn/releases/tag/v1.1.3) |                                                             |
 
 # Roadmap
 

chart/Chart.yaml

@@ -1,7 +1,7 @@
 apiVersion: v1
 name: longhorn
-version: 1.4.0-dev
-appVersion: v1.4.0-dev
+version: 1.6.0-dev
+appVersion: v1.6.0-dev
 kubeVersion: ">=1.21.0-0"
 description: Longhorn is a distributed block storage system for Kubernetes.
 keywords:

chart/README.md

@@ -51,7 +51,7 @@ helm repo update
 - With Helm 2, the following command will create the `longhorn-system` namespace and install the Longhorn chart together.
 ```
 helm install longhorn/longhorn --name longhorn --namespace longhorn-system
-``` 
+```
 - With Helm 3, the following commands will create the `longhorn-system` namespace first, then install the Longhorn chart.
 
 ```
@@ -74,5 +74,253 @@ helm uninstall longhorn -n longhorn-system
 kubectl delete namespace longhorn-system
 ```
 
+## Values
+
+The `values.yaml` contains items used to tweak a deployment of this chart.
+
+### Cattle Settings
+
+| Key | Type | Default | Description |
+|-----|------|---------|-------------|
+| global.cattle.systemDefaultRegistry | string | `""` | System default registry |
+| global.cattle.windowsCluster.defaultSetting.systemManagedComponentsNodeSelector | string | `"kubernetes.io/os:linux"` | Node selector for Longhorn system managed components |
+| global.cattle.windowsCluster.defaultSetting.taintToleration | string | `"cattle.io/os=linux:NoSchedule"` | Toleration for Longhorn system managed components |
+| global.cattle.windowsCluster.enabled | bool | `false` | Enable this to allow Longhorn to run on the Rancher deployed Windows cluster |
+| global.cattle.windowsCluster.nodeSelector | object | `{"kubernetes.io/os":"linux"}` | Select Linux nodes to run Longhorn user deployed components |
+| global.cattle.windowsCluster.tolerations | list | `[{"effect":"NoSchedule","key":"cattle.io/os","operator":"Equal","value":"linux"}]` | Tolerate Linux nodes to run Longhorn user deployed components |
+
+### Network Policies
+
+| Key | Type | Default | Description |
+|-----|------|---------|-------------|
+| networkPolicies.enabled | bool | `false` | Enable NetworkPolicies to limit access to the Longhorn pods |
+| networkPolicies.type | string | `"k3s"` | Create the policy based on your distribution to allow access for the ingress. Options: `k3s`, `rke2`, `rke1` |
+
+### Image Settings
+
+| Key | Type | Default | Description |
+|-----|------|---------|-------------|
+| image.csi.attacher.repository | string | `"longhornio/csi-attacher"` | Specify CSI attacher image repository. Leave blank to autodetect |
+| image.csi.attacher.tag | string | `"v4.2.0"` | Specify CSI attacher image tag. Leave blank to autodetect |
+| image.csi.livenessProbe.repository | string | `"longhornio/livenessprobe"` | Specify CSI liveness probe image repository. Leave blank to autodetect  |
+| image.csi.livenessProbe.tag | string | `"v2.9.0"` | Specify CSI liveness probe image tag. Leave blank to autodetect |
+| image.csi.nodeDriverRegistrar.repository | string | `"longhornio/csi-node-driver-registrar"` | Specify CSI node driver registrar image repository. Leave blank to autodetect |
+| image.csi.nodeDriverRegistrar.tag | string | `"v2.7.0"` | Specify CSI node driver registrar image tag. Leave blank to autodetect |
+| image.csi.provisioner.repository | string | `"longhornio/csi-provisioner"` | Specify CSI provisioner image repository. Leave blank to autodetect |
+| image.csi.provisioner.tag | string | `"v3.4.1"` | Specify CSI provisioner image tag. Leave blank to autodetect |
+| image.csi.resizer.repository | string | `"longhornio/csi-resizer"` | Specify CSI driver resizer image repository. Leave blank to autodetect |
+| image.csi.resizer.tag | string | `"v1.7.0"` | Specify CSI driver resizer image tag. Leave blank to autodetect |
+| image.csi.snapshotter.repository | string | `"longhornio/csi-snapshotter"` | Specify CSI driver snapshotter image repository. Leave blank to autodetect |
+| image.csi.snapshotter.tag | string | `"v6.2.1"` | Specify CSI driver snapshotter image tag. Leave blank to autodetect. |
+| image.longhorn.backingImageManager.repository | string | `"longhornio/backing-image-manager"` | Specify Longhorn backing image manager image repository |
+| image.longhorn.backingImageManager.tag | string | `"master-head"` | Specify Longhorn backing image manager image tag  |
+| image.longhorn.engine.repository | string | `"longhornio/longhorn-engine"` | Specify Longhorn engine image repository |
+| image.longhorn.engine.tag | string | `"master-head"` | Specify Longhorn engine image tag |
+| image.longhorn.instanceManager.repository | string | `"longhornio/longhorn-instance-manager"` | Specify Longhorn instance manager image repository |
+| image.longhorn.instanceManager.tag | string | `"master-head"` | Specify Longhorn instance manager image tag |
+| image.longhorn.manager.repository | string | `"longhornio/longhorn-manager"` | Specify Longhorn manager image repository |
+| image.longhorn.manager.tag | string | `"master-head"` | Specify Longhorn manager image tag |
+| image.longhorn.shareManager.repository | string | `"longhornio/longhorn-share-manager"` | Specify Longhorn share manager image repository |
+| image.longhorn.shareManager.tag | string | `"master-head"` | Specify Longhorn share manager image tag |
+| image.longhorn.supportBundleKit.repository | string | `"longhornio/support-bundle-kit"` | Specify Longhorn support bundle manager image repository |
+| image.longhorn.supportBundleKit.tag | string | `"v0.0.27"` | Specify Longhorn support bundle manager image tag |
+| image.longhorn.ui.repository | string | `"longhornio/longhorn-ui"` | Specify Longhorn ui image repository |
+| image.longhorn.ui.tag | string | `"master-head"` | Specify Longhorn ui image tag |
+| image.openshift.oauthProxy.repository | string | `"quay.io/openshift/origin-oauth-proxy"` | For openshift user. Specify oauth proxy image repository |
+| image.openshift.oauthProxy.tag | float | `4.13` | For openshift user. Specify oauth proxy image tag. Note: Use your OCP/OKD 4.X Version, Current Stable is 4.13 |
+| image.pullPolicy | string | `"IfNotPresent"` | Image pull policy which applies to all user deployed Longhorn Components. e.g, Longhorn manager, Longhorn driver, Longhorn UI |
+
+### Service Settings
+
+| Key | Description |
+|-----|-------------|
+| service.manager.nodePort | NodePort port number (to set explicitly, choose port between 30000-32767) |
+| service.manager.type | Define Longhorn manager service type. |
+| service.ui.nodePort | NodePort port number (to set explicitly, choose port between 30000-32767) |
+| service.ui.type | Define Longhorn UI service type. Options: `ClusterIP`, `NodePort`, `LoadBalancer`, `Rancher-Proxy` |
+
+### StorageClass Settings
+
+| Key | Type | Default | Description |
+|-----|------|---------|-------------|
+| persistence.backingImage.dataSourceParameters | string | `nil` | Specify the data source parameters for the backing image used in Longhorn StorageClass. This option accepts a json string of a map. e.g., `'{\"url\":\"https://backing-image-example.s3-region.amazonaws.com/test-backing-image\"}'`. |
+| persistence.backingImage.dataSourceType | string | `nil` | Specify the data source type for the backing image used in Longhorn StorageClass. If the backing image does not exists, Longhorn will use this field to create a backing image. Otherwise, Longhorn will use it to verify the selected backing image. |
+| persistence.backingImage.enable | bool | `false` | Set backing image for Longhorn StorageClass |
+| persistence.backingImage.expectedChecksum | string | `nil` | Specify the expected SHA512 checksum of the selected backing image in Longhorn StorageClass |
+| persistence.backingImage.name | string | `nil` | Specify a backing image that will be used by Longhorn volumes in Longhorn StorageClass. If not exists, the backing image data source type and backing image data source parameters should be specified so that Longhorn will create the backing image before using it |
+| persistence.defaultClass | bool | `true` | Set Longhorn StorageClass as default |
+| persistence.defaultClassReplicaCount | int | `3` | Set replica count for Longhorn StorageClass |
+| persistence.defaultDataLocality | string | `"disabled"` | Set data locality for Longhorn StorageClass. Options: `disabled`, `best-effort` |
+| persistence.defaultFsType | string | `"ext4"` | Set filesystem type for Longhorn StorageClass |
+| persistence.defaultMkfsParams | string | `""` | Set mkfs options for Longhorn StorageClass |
+| persistence.defaultNodeSelector.enable | bool | `false` | Enable Node selector for Longhorn StorageClass |
+| persistence.defaultNodeSelector.selector | string | `""` | This selector enables only certain nodes having these tags to be used for the volume. e.g. `"storage,fast"` |
+| persistence.migratable | bool | `false` | Set volume migratable for Longhorn StorageClass |
+| persistence.reclaimPolicy | string | `"Delete"` | Define reclaim policy. Options: `Retain`, `Delete` |
+| persistence.recurringJobSelector.enable | bool | `false` | Enable recurring job selector for Longhorn StorageClass |
+| persistence.recurringJobSelector.jobList | list | `[]` | Recurring job selector list for Longhorn StorageClass. Please be careful of quotes of input. e.g., `[{"name":"backup", "isGroup":true}]` |
+| persistence.removeSnapshotsDuringFilesystemTrim | string | `"ignored"` | Allow automatically removing snapshots during filesystem trim for Longhorn StorageClass. Options: `ignored`, `enabled`, `disabled` |
+
+### CSI Settings
+
+| Key | Description |
+|-----|-------------|
+| csi.attacherReplicaCount | Specify replica count of CSI Attacher. Leave blank to use default count: 3 |
+| csi.kubeletRootDir | Specify kubelet root-dir. Leave blank to autodetect |
+| csi.provisionerReplicaCount | Specify replica count of CSI Provisioner. Leave blank to use default count: 3 |
+| csi.resizerReplicaCount | Specify replica count of CSI Resizer. Leave blank to use default count: 3 |
+| csi.snapshotterReplicaCount | Specify replica count of CSI Snapshotter. Leave blank to use default count: 3 |
+
+### Longhorn Manager Settings
+
+Longhorn system contains user deployed components (e.g, Longhorn manager, Longhorn driver, Longhorn UI) and system managed components (e.g, instance manager, engine image, CSI driver, etc.).
+These settings only apply to Longhorn manager component.
+
+| Key | Type | Default | Description |
+|-----|------|---------|-------------|
+| longhornManager.log.format | string | `"plain"` | Options: `plain`, `json` |
+| longhornManager.nodeSelector | object | `{}` | Select nodes to run Longhorn manager |
+| longhornManager.priorityClass | string | `nil` | Priority class for longhorn manager |
+| longhornManager.serviceAnnotations | object | `{}` | Annotation used in Longhorn manager service |
+| longhornManager.tolerations | list | `[]` | Tolerate nodes to run Longhorn manager |
+
+### Longhorn Driver Settings
+
+Longhorn system contains user deployed components (e.g, Longhorn manager, Longhorn driver, Longhorn UI) and system managed components (e.g, instance manager, engine image, CSI driver, etc.).
+These settings only apply to Longhorn driver component.
+
+| Key | Type | Default | Description |
+|-----|------|---------|-------------|
+| longhornDriver.nodeSelector | object | `{}` | Select nodes to run Longhorn driver |
+| longhornDriver.priorityClass | string | `nil` | Priority class for longhorn driver |
+| longhornDriver.tolerations | list | `[]` | Tolerate nodes to run Longhorn driver |
+
+### Longhorn UI Settings
+
+Longhorn system contains user deployed components (e.g, Longhorn manager, Longhorn driver, Longhorn UI) and system managed components (e.g, instance manager, engine image, CSI driver, etc.).
+These settings only apply to Longhorn UI component.
+
+| Key | Type | Default | Description |
+|-----|------|---------|-------------|
+| longhornUI.nodeSelector | object | `{}` | Select nodes to run Longhorn UI |
+| longhornUI.priorityClass | string | `nil` | Priority class count for longhorn ui |
+| longhornUI.replicas | int | `2` | Replica count for longhorn ui |
+| longhornUI.tolerations | list | `[]` | Tolerate nodes to run Longhorn UI |
+
+### Ingress Settings
+
+| Key | Type | Default | Description |
+|-----|------|---------|-------------|
+| ingress.annotations | string | `nil` | Ingress annotations done as key:value pairs |
+| ingress.enabled | bool | `false` | Set to true to enable ingress record generation |
+| ingress.host | string | `"sslip.io"` | Layer 7 Load Balancer hostname |
+| ingress.ingressClassName | string | `nil` | Add ingressClassName to the Ingress Can replace the kubernetes.io/ingress.class annotation on v1.18+ |
+| ingress.path | string | `"/"` | If ingress is enabled you can set the default ingress path then you can access the UI by using the following full path {{host}}+{{path}} |
+| ingress.secrets | string | `nil` | If you're providing your own certificates, please use this to add the certificates as secrets |
+| ingress.secureBackends | bool | `false` | Enable this in order to enable that the backend service will be connected at port 443 |
+| ingress.tls | bool | `false` | Set this to true in order to enable TLS on the ingress record |
+| ingress.tlsSecret | string | `"longhorn.local-tls"` | If TLS is set to true, you must declare what secret will store the key/certificate for TLS |
+
+### Private Registry Settings
+
+Longhorn can be installed in an air gapped environment with private registry settings. Please refer to **Air Gap Installation** in our official site [link](https://longhorn.io/docs)
+
+| Key | Description |
+|-----|-------------|
+| privateRegistry.createSecret | Set `true` to create a new private registry secret |
+| privateRegistry.registryPasswd | Password used to authenticate to private registry |
+| privateRegistry.registrySecret | If create a new private registry secret is true, create a Kubernetes secret with this name; else use the existing secret of this name. Use it to pull images from your private registry |
+| privateRegistry.registryUrl | URL of private registry. Leave blank to apply system default registry |
+| privateRegistry.registryUser | User used to authenticate to private registry |
+
+### OS/Kubernetes Distro Settings
+
+#### Opensift Settings
+
+Please also refer to this document [ocp-readme](https://github.com/longhorn/longhorn/blob/master/chart/ocp-readme.md) for more details
+
+| Key | Type | Default | Description |
+|-----|------|---------|-------------|
+| openshift.enabled | bool | `false` | Enable when using openshift |
+| openshift.ui.port | int | `443` | UI port in openshift environment |
+| openshift.ui.proxy | int | `8443` | UI proxy in openshift environment |
+| openshift.ui.route | string | `"longhorn-ui"` | UI route in openshift environment |
+
+### Other Settings
+
+| Key | Default | Description |
+|-----|---------|-------------|
+| annotations | `{}` | Annotations to add to the Longhorn Manager DaemonSet Pods. Optional. |
+| enablePSP | `false` | For Kubernetes < v1.25, if your cluster enables Pod Security Policy admission controller, set this to `true` to ship longhorn-psp which allow privileged Longhorn pods to start |
+
+### System Default Settings
+
+For system default settings, you can first leave blank to use default values which will be applied when installing Longhorn.
+You can then change them through UI after installation.
+For more details like types or options, you can refer to **Settings Reference** in our official site [link](https://longhorn.io/docs)
+
+| Key | Description |
+|-----|-------------|
+| defaultSettings.allowEmptyDiskSelectorVolume | Allow Scheduling Empty Disk Selector Volumes To Any Disk |
+| defaultSettings.allowEmptyNodeSelectorVolume | Allow Scheduling Empty Node Selector Volumes To Any Node |
+| defaultSettings.allowRecurringJobWhileVolumeDetached | If this setting is enabled, Longhorn will automatically attaches the volume and takes snapshot/backup  when it is the time to do recurring snapshot/backup. |
+| defaultSettings.allowVolumeCreationWithDegradedAvailability | This setting allows user to create and attach a volume that doesn't have all the replicas scheduled at the time of creation. |
+| defaultSettings.autoCleanupSystemGeneratedSnapshot | This setting enables Longhorn to automatically cleanup the system generated snapshot after replica rebuild is done. |
+| defaultSettings.autoDeletePodWhenVolumeDetachedUnexpectedly | If enabled, Longhorn will automatically delete the workload pod that is managed by a controller (e.g. deployment, statefulset, daemonset, etc...)  when Longhorn volume is detached unexpectedly (e.g. during Kubernetes upgrade, Docker reboot, or network disconnect). By deleting the pod, its controller restarts the pod and Kubernetes handles volume reattachment and remount. |
+| defaultSettings.autoSalvage | If enabled, volumes will be automatically salvaged when all the replicas become faulty e.g. due to network disconnection. Longhorn will try to figure out which replica(s) are usable, then use them for the volume. By default true. |
+| defaultSettings.backingImageCleanupWaitInterval | This interval in minutes determines how long Longhorn will wait before cleaning up the backing image file when there is no replica in the disk using it. |
+| defaultSettings.backingImageRecoveryWaitInterval | This interval in seconds determines how long Longhorn will wait before re-downloading the backing image file when all disk files of this backing image become failed or unknown. |
+| defaultSettings.backupCompressionMethod | This setting allows users to specify backup compression method. |
+| defaultSettings.backupConcurrentLimit | This setting controls how many worker threads per backup concurrently. |
+| defaultSettings.backupTarget | The endpoint used to access the backupstore. Available: NFS, CIFS, AWS, GCP, AZURE. |
+| defaultSettings.backupTargetCredentialSecret | The name of the Kubernetes secret associated with the backup target. |
+| defaultSettings.backupstorePollInterval | In seconds. The backupstore poll interval determines how often Longhorn checks the backupstore for new backups. Set to 0 to disable the polling. By default 300. |
+| defaultSettings.concurrentAutomaticEngineUpgradePerNodeLimit | This setting controls how Longhorn automatically upgrades volumes' engines to the new default engine image after upgrading Longhorn manager. The value of this setting specifies the maximum number of engines per node that are allowed to upgrade to the default engine image at the same time. If the value is 0, Longhorn will not automatically upgrade volumes' engines to default version. |
+| defaultSettings.concurrentReplicaRebuildPerNodeLimit | This setting controls how many replicas on a node can be rebuilt simultaneously. |
+| defaultSettings.concurrentVolumeBackupRestorePerNodeLimit | This setting controls how many volumes on a node can restore the backup concurrently. Set the value to **0** to disable backup restore. |
+| defaultSettings.createDefaultDiskLabeledNodes | Create default Disk automatically only on Nodes with the label "node.longhorn.io/create-default-disk=true" if no other disks exist. If disabled, the default disk will be created on all new nodes when each node is first added. |
+| defaultSettings.defaultDataLocality | Longhorn volume has data locality if there is a local replica of the volume on the same node as the pod which is using the volume. |
+| defaultSettings.defaultDataPath | Default path to use for storing data on a host. By default "/var/lib/longhorn/" |
+| defaultSettings.defaultLonghornStaticStorageClass | The 'storageClassName' is given to PVs and PVCs that are created for an existing Longhorn volume. The StorageClass name can also be used as a label, so it is possible to use a Longhorn StorageClass to bind a workload to an existing PV without creating a Kubernetes StorageClass object. By default 'longhorn-static'. |
+| defaultSettings.defaultReplicaCount | The default number of replicas when a volume is created from the Longhorn UI. For Kubernetes configuration, update the `numberOfReplicas` in the StorageClass. By default 3. |
+| defaultSettings.deletingConfirmationFlag | This flag is designed to prevent Longhorn from being accidentally uninstalled which will lead to data lost. |
+| defaultSettings.disableRevisionCounter | This setting is only for volumes created by UI. By default, this is false meaning there will be a reivision counter file to track every write to the volume. During salvage recovering Longhorn will pick the replica with largest reivision counter as candidate to recover the whole volume. If revision counter is disabled, Longhorn will not track every write to the volume. During the salvage recovering, Longhorn will use the 'volume-head-xxx.img' file last modification time and file size to pick the replica candidate to recover the whole volume. |
+| defaultSettings.disableSchedulingOnCordonedNode | Disable Longhorn manager to schedule replica on Kubernetes cordoned node. By default true. |
+| defaultSettings.engineReplicaTimeout | In seconds. The setting specifies the timeout between the engine and replica(s), and the value should be between 8 to 30 seconds. The default value is 8 seconds. |
+| defaultSettings.failedBackupTTL | In minutes. This setting determines how long Longhorn will keep the backup resource that was failed. Set to 0 to disable the auto-deletion. |
+| defaultSettings.fastReplicaRebuildEnabled | This feature supports the fast replica rebuilding. It relies on the checksum of snapshot disk files, so setting the snapshot-data-integrity to **enable** or **fast-check** is a prerequisite. |
+| defaultSettings.guaranteedInstanceManagerCPU | This integer value indicates how many percentage of the total allocatable CPU on each node will be reserved for each instance manager Pod. You can leave it with the default value, which is 12%. |
+| defaultSettings.kubernetesClusterAutoscalerEnabled | Enabling this setting will notify Longhorn that the cluster is using Kubernetes Cluster Autoscaler. |
+| defaultSettings.logLevel | The log level Panic, Fatal, Error, Warn, Info, Debug, Trace used in longhorn manager. Default to Info. |
+| defaultSettings.nodeDownPodDeletionPolicy | Defines the Longhorn action when a Volume is stuck with a StatefulSet/Deployment Pod on a node that is down. |
+| defaultSettings.nodeDrainPolicy | Define the policy to use when a node with the last healthy replica of a volume is drained. |
+| defaultSettings.offlineReplicaRebuilding | This setting allows users to enable the offline replica rebuilding for volumes using v2 data engine. |
+| defaultSettings.orphanAutoDeletion | This setting allows Longhorn to delete the orphan resource and its corresponding orphaned data automatically like stale replicas. Orphan resources on down or unknown nodes will not be cleaned up automatically. |
+| defaultSettings.priorityClass | priorityClass for longhorn system componentss |
+| defaultSettings.recurringFailedJobsHistoryLimit | This setting specifies how many failed backup or snapshot job histories should be retained. History will not be retained if the value is 0. |
+| defaultSettings.recurringSuccessfulJobsHistoryLimit | This setting specifies how many successful backup or snapshot job histories should be retained. History will not be retained if the value is 0. |
+| defaultSettings.removeSnapshotsDuringFilesystemTrim | This setting allows Longhorn filesystem trim feature to automatically mark the latest snapshot and its ancestors as removed and stops at the snapshot containing multiple children. |
+| defaultSettings.replicaAutoBalance | Enable this setting automatically rebalances replicas when discovered an available node. |
+| defaultSettings.replicaDiskSoftAntiAffinity | Allow scheduling on disks with existing healthy replicas of the same volume. By default true. |
+| defaultSettings.replicaFileSyncHttpClientTimeout | In seconds. The setting specifies the HTTP client timeout to the file sync server. |
+| defaultSettings.replicaReplenishmentWaitInterval | In seconds. The interval determines how long Longhorn will wait at least in order to reuse the existing data on a failed replica rather than directly creating a new replica for a degraded volume. |
+| defaultSettings.replicaSoftAntiAffinity | Allow scheduling on nodes with existing healthy replicas of the same volume. By default false. |
+| defaultSettings.replicaZoneSoftAntiAffinity | Allow scheduling new Replicas of Volume to the Nodes in the same Zone as existing healthy Replicas. Nodes don't belong to any Zone will be treated as in the same Zone. Notice that Longhorn relies on label `topology.kubernetes.io/zone=<Zone name of the node>` in the Kubernetes node object to identify the zone. By default true. |
+| defaultSettings.restoreConcurrentLimit | This setting controls how many worker threads per restore concurrently. |
+| defaultSettings.restoreVolumeRecurringJobs | Restore recurring jobs from the backup volume on the backup target and create recurring jobs if not exist during a backup restoration. |
+| defaultSettings.snapshotDataIntegrity | This setting allows users to enable or disable snapshot hashing and data integrity checking. |
+| defaultSettings.snapshotDataIntegrityCronjob | Unix-cron string format. The setting specifies when Longhorn checks the data integrity of snapshot disk files. |
+| defaultSettings.snapshotDataIntegrityImmediateCheckAfterSnapshotCreation | Hashing snapshot disk files impacts the performance of the system. The immediate snapshot hashing and checking can be disabled to minimize the impact after creating a snapshot. |
+| defaultSettings.storageMinimalAvailablePercentage | If the minimum available disk capacity exceeds the actual percentage of available disk capacity, the disk becomes unschedulable until more space is freed up. By default 25. |
+| defaultSettings.storageNetwork | Longhorn uses the storage network for in-cluster data traffic. Leave this blank to use the Kubernetes cluster network. |
+| defaultSettings.storageOverProvisioningPercentage | The over-provisioning percentage defines how much storage can be allocated relative to the hard drive's capacity. By default 200. |
+| defaultSettings.storageReservedPercentageForDefaultDisk | The reserved percentage specifies the percentage of disk space that will not be allocated to the default disk on each new Longhorn node. |
+| defaultSettings.supportBundleFailedHistoryLimit | This setting specifies how many failed support bundles can exist in the cluster. Set this value to **0** to have Longhorn automatically purge all failed support bundles. |
+| defaultSettings.systemManagedComponentsNodeSelector | nodeSelector for longhorn system components |
+| defaultSettings.systemManagedPodsImagePullPolicy | This setting defines the Image Pull Policy of Longhorn system managed pod. e.g. instance manager, engine image, CSI driver, etc. The new Image Pull Policy will only apply after the system managed pods restart. |
+| defaultSettings.taintToleration | taintToleration for longhorn system components |
+| defaultSettings.upgradeChecker | Upgrade Checker will check for new Longhorn version periodically. When there is a new version available, a notification will appear in the UI. By default true. |
+| defaultSettings.v2DataEngine | This allows users to activate v2 data engine based on SPDK. Currently, it is in the preview phase and should not be utilized in a production environment. |
+
 ---
 Please see [link](https://github.com/longhorn/longhorn) for more information.

chart/README.md.gotmpl

@@ -0,0 +1,253 @@
+# Longhorn Chart
+
+> **Important**: Please install the Longhorn chart in the `longhorn-system` namespace only.
+
+> **Warning**: Longhorn doesn't support downgrading from a higher version to a lower version.
+
+## Source Code
+
+Longhorn is 100% open source software. Project source code is spread across a number of repos:
+
+1. Longhorn Engine -- Core controller/replica logic https://github.com/longhorn/longhorn-engine
+2. Longhorn Instance Manager -- Controller/replica instance lifecycle management https://github.com/longhorn/longhorn-instance-manager
+3. Longhorn Share Manager -- NFS provisioner that exposes Longhorn volumes as ReadWriteMany volumes. https://github.com/longhorn/longhorn-share-manager
+4. Backing Image Manager -- Backing image file lifecycle management. https://github.com/longhorn/backing-image-manager
+5. Longhorn Manager -- Longhorn orchestration, includes CSI driver for Kubernetes https://github.com/longhorn/longhorn-manager
+6. Longhorn UI -- Dashboard https://github.com/longhorn/longhorn-ui
+
+## Prerequisites
+
+1. A container runtime compatible with Kubernetes (Docker v1.13+, containerd v1.3.7+, etc.)
+2. Kubernetes >= v1.21
+3. Make sure `bash`, `curl`, `findmnt`, `grep`, `awk` and `blkid` has been installed in all nodes of the Kubernetes cluster.
+4. Make sure `open-iscsi` has been installed, and the `iscsid` daemon is running on all nodes of the Kubernetes cluster. For GKE, recommended Ubuntu as guest OS image since it contains `open-iscsi` already.
+
+## Upgrading to Kubernetes v1.25+
+
+Starting in Kubernetes v1.25, [Pod Security Policies](https://kubernetes.io/docs/concepts/security/pod-security-policy/) have been removed from the Kubernetes API.
+
+As a result, **before upgrading to Kubernetes v1.25** (or on a fresh install in a Kubernetes v1.25+ cluster), users are expected to perform an in-place upgrade of this chart with `enablePSP` set to `false` if it has been previously set to `true`.
+
+> **Note:**
+> If you upgrade your cluster to Kubernetes v1.25+ before removing PSPs via a `helm upgrade` (even if you manually clean up resources), **it will leave the Helm release in a broken state within the cluster such that further Helm operations will not work (`helm uninstall`, `helm upgrade`, etc.).**
+>
+> If your charts get stuck in this state, you may have to clean up your Helm release secrets.
+Upon setting `enablePSP` to false, the chart will remove any PSP resources deployed on its behalf from the cluster. This is the default setting for this chart.
+
+As a replacement for PSPs, [Pod Security Admission](https://kubernetes.io/docs/concepts/security/pod-security-admission/) should be used. Please consult the Longhorn docs for more details on how to configure your chart release namespaces to work with the new Pod Security Admission and apply Pod Security Standards.
+
+## Installation
+1. Add Longhorn chart repository.
+```
+helm repo add longhorn https://charts.longhorn.io
+```
+
+2. Update local Longhorn chart information from chart repository.
+```
+helm repo update
+```
+
+3. Install Longhorn chart.
+- With Helm 2, the following command will create the `longhorn-system` namespace and install the Longhorn chart together.
+```
+helm install longhorn/longhorn --name longhorn --namespace longhorn-system
+```
+- With Helm 3, the following commands will create the `longhorn-system` namespace first, then install the Longhorn chart.
+
+```
+kubectl create namespace longhorn-system
+helm install longhorn longhorn/longhorn --namespace longhorn-system
+```
+
+## Uninstallation
+
+With Helm 2 to uninstall Longhorn.
+```
+kubectl -n longhorn-system patch -p '{"value": "true"}' --type=merge lhs deleting-confirmation-flag
+helm delete longhorn --purge
+```
+
+With Helm 3 to uninstall Longhorn.
+```
+kubectl -n longhorn-system patch -p '{"value": "true"}' --type=merge lhs deleting-confirmation-flag
+helm uninstall longhorn -n longhorn-system
+kubectl delete namespace longhorn-system
+```
+
+## Values
+
+The `values.yaml` contains items used to tweak a deployment of this chart.
+
+### Cattle Settings
+
+| Key | Type | Default | Description |
+|-----|------|---------|-------------|
+{{- range .Values }}
+  {{- if hasPrefix "global" .Key }}
+| {{ .Key }} | {{ .Type }} | {{ if .Default }}{{ .Default }}{{ else }}{{ .AutoDefault }}{{ end }} | {{ if .Description }}{{ .Description }}{{ else }}{{ .AutoDescription }}{{ end }} |
+  {{- end }}
+{{- end }}
+
+### Network Policies
+
+| Key | Type | Default | Description |
+|-----|------|---------|-------------|
+{{- range .Values }}
+  {{- if hasPrefix "networkPolicies" .Key }}
+| {{ .Key }} | {{ .Type }} | {{ if .Default }}{{ .Default }}{{ else }}{{ .AutoDefault }}{{ end }} | {{ if .Description }}{{ .Description }}{{ else }}{{ .AutoDescription }}{{ end }} |
+  {{- end }}
+{{- end }}
+
+### Image Settings
+
+| Key | Type | Default | Description |
+|-----|------|---------|-------------|
+{{- range .Values }}
+  {{- if hasPrefix "image" .Key }}
+| {{ .Key }} | {{ .Type }} | {{ if .Default }}{{ .Default }}{{ else }}{{ .AutoDefault }}{{ end }} | {{ if .Description }}{{ .Description }}{{ else }}{{ .AutoDescription }}{{ end }} |
+  {{- end }}
+{{- end }}
+
+### Service Settings
+
+| Key | Description |
+|-----|-------------|
+{{- range .Values }}
+  {{- if (and (hasPrefix "service" .Key) (not (contains "Account" .Key))) }}
+| {{ .Key }} | {{ if .Description }}{{ .Description }}{{ else }}{{ .AutoDescription }}{{ end }} |
+  {{- end }}
+{{- end }}
+
+### StorageClass Settings
+
+| Key | Type | Default | Description |
+|-----|------|---------|-------------|
+{{- range .Values }}
+  {{- if hasPrefix "persistence" .Key }}
+| {{ .Key }} | {{ .Type }} | {{ if .Default }}{{ .Default }}{{ else }}{{ .AutoDefault }}{{ end }} | {{ if .Description }}{{ .Description }}{{ else }}{{ .AutoDescription }}{{ end }} |
+  {{- end }}
+{{- end }}
+
+### CSI Settings
+
+| Key | Description |
+|-----|-------------|
+{{- range .Values }}
+  {{- if hasPrefix "csi" .Key }}
+| {{ .Key }} | {{ if .Description }}{{ .Description }}{{ else }}{{ .AutoDescription }}{{ end }} |
+  {{- end }}
+{{- end }}
+
+### Longhorn Manager Settings
+
+Longhorn system contains user deployed components (e.g, Longhorn manager, Longhorn driver, Longhorn UI) and system managed components (e.g, instance manager, engine image, CSI driver, etc.).
+These settings only apply to Longhorn manager component.
+
+| Key | Type | Default | Description |
+|-----|------|---------|-------------|
+{{- range .Values }}
+  {{- if hasPrefix "longhornManager" .Key }}
+| {{ .Key }} | {{ .Type }} | {{ if .Default }}{{ .Default }}{{ else }}{{ .AutoDefault }}{{ end }} | {{ if .Description }}{{ .Description }}{{ else }}{{ .AutoDescription }}{{ end }} |
+  {{- end }}
+{{- end }}
+
+### Longhorn Driver Settings
+
+Longhorn system contains user deployed components (e.g, Longhorn manager, Longhorn driver, Longhorn UI) and system managed components (e.g, instance manager, engine image, CSI driver, etc.).
+These settings only apply to Longhorn driver component.
+
+| Key | Type | Default | Description |
+|-----|------|---------|-------------|
+{{- range .Values }}
+  {{- if hasPrefix "longhornDriver" .Key }}
+| {{ .Key }} | {{ .Type }} | {{ if .Default }}{{ .Default }}{{ else }}{{ .AutoDefault }}{{ end }} | {{ if .Description }}{{ .Description }}{{ else }}{{ .AutoDescription }}{{ end }} |
+  {{- end }}
+{{- end }}
+
+### Longhorn UI Settings
+
+Longhorn system contains user deployed components (e.g, Longhorn manager, Longhorn driver, Longhorn UI) and system managed components (e.g, instance manager, engine image, CSI driver, etc.).
+These settings only apply to Longhorn UI component.
+
+| Key | Type | Default | Description |
+|-----|------|---------|-------------|
+{{- range .Values }}
+  {{- if hasPrefix "longhornUI" .Key }}
+| {{ .Key }} | {{ .Type }} | {{ if .Default }}{{ .Default }}{{ else }}{{ .AutoDefault }}{{ end }} | {{ if .Description }}{{ .Description }}{{ else }}{{ .AutoDescription }}{{ end }} |
+  {{- end }}
+{{- end }}
+
+### Ingress Settings
+
+| Key | Type | Default | Description |
+|-----|------|---------|-------------|
+{{- range .Values }}
+  {{- if hasPrefix "ingress" .Key }}
+| {{ .Key }} | {{ .Type }} | {{ if .Default }}{{ .Default }}{{ else }}{{ .AutoDefault }}{{ end }} | {{ if .Description }}{{ .Description }}{{ else }}{{ .AutoDescription }}{{ end }} |
+  {{- end }}
+{{- end }}
+
+### Private Registry Settings
+
+Longhorn can be installed in an air gapped environment with private registry settings. Please refer to **Air Gap Installation** in our official site [link](https://longhorn.io/docs)
+
+| Key | Description |
+|-----|-------------|
+{{- range .Values }}
+  {{- if hasPrefix "privateRegistry" .Key }}
+| {{ .Key }} | {{ if .Description }}{{ .Description }}{{ else }}{{ .AutoDescription }}{{ end }} |
+  {{- end }}
+{{- end }}
+
+### OS/Kubernetes Distro Settings
+
+#### Opensift Settings
+
+Please also refer to this document [ocp-readme](https://github.com/longhorn/longhorn/blob/master/chart/ocp-readme.md) for more details
+
+| Key | Type | Default | Description |
+|-----|------|---------|-------------|
+{{- range .Values }}
+  {{- if hasPrefix "openshift" .Key }}
+| {{ .Key }} | {{ .Type }} | {{ if .Default }}{{ .Default }}{{ else }}{{ .AutoDefault }}{{ end }} | {{ if .Description }}{{ .Description }}{{ else }}{{ .AutoDescription }}{{ end }} |
+  {{- end }}
+{{- end }}
+
+### Other Settings
+
+| Key | Default | Description |
+|-----|---------|-------------|
+{{- range .Values }}
+  {{- if not (or (hasPrefix "defaultSettings" .Key)
+  (hasPrefix "networkPolicies" .Key)
+  (hasPrefix "image" .Key)
+  (hasPrefix "service" .Key)
+  (hasPrefix "persistence" .Key)
+  (hasPrefix "csi" .Key)
+  (hasPrefix "longhornManager" .Key)
+  (hasPrefix "longhornDriver" .Key)
+  (hasPrefix "longhornUI" .Key)
+  (hasPrefix "privateRegistry" .Key)
+  (hasPrefix "ingress" .Key)
+  (hasPrefix "openshift" .Key)
+  (hasPrefix "global" .Key)) }}
+| {{ .Key }} | {{ if .Default }}{{ .Default }}{{ else }}{{ .AutoDefault }}{{ end }} | {{ if .Description }}{{ .Description }}{{ else }}{{ .AutoDescription }}{{ end }} |
+  {{- end }}
+{{- end }}
+
+### System Default Settings
+
+For system default settings, you can first leave blank to use default values which will be applied when installing Longhorn.
+You can then change them through UI after installation.
+For more details like types or options, you can refer to **Settings Reference** in our official site [link](https://longhorn.io/docs)
+
+| Key | Description |
+|-----|-------------|
+{{- range .Values }}
+  {{- if hasPrefix "defaultSettings" .Key }}
+| {{ .Key }} | {{ if .Description }}{{ .Description }}{{ else }}{{ .AutoDescription }}{{ end }} |
+  {{- end }}
+{{- end }}
+
+---
+Please see [link](https://github.com/longhorn/longhorn) for more information.

chart/ocp-readme.md

@@ -0,0 +1,177 @@
+# OpenShift / OKD Extra Configuration Steps
+
+- [OpenShift / OKD Extra Configuration Steps](#openshift--okd-extra-configuration-steps)
+  - [Notes](#notes)
+  - [Known Issues](#known-issues)
+  - [Preparing Nodes (Optional)](#preparing-nodes-optional)
+    - [Default /var/lib/longhorn setup](#default-varliblonghorn-setup)
+    - [Separate /var/mnt/longhorn setup](#separate-varmntlonghorn-setup)
+      - [Create Filesystem](#create-filesystem)
+      - [Mounting Disk On Boot](#mounting-disk-on-boot)
+      - [Label and Annotate Nodes](#label-and-annotate-nodes)
+  - [Example values.yaml](#example-valuesyaml)
+  - [Installation](#installation)
+  - [Refs](#refs)
+
+## Notes
+
+Main changes and tasks for OCP are:
+
+- On OCP / OKD, the Operating System is Managed by the Cluster
+- OCP Imposes [Security Context Constraints](https://docs.openshift.com/container-platform/4.11/authentication/managing-security-context-constraints.html)
+  - This requires everything to run with the least privilege possible. For the moment every component has been given access to run as higher privilege.
+  - Something to circle back on is network polices and which components can have their privileges reduced without impacting functionality.
+    - The UI probably can be for example.
+- openshift/oauth-proxy for authentication to the Longhorn Ui
+  - **⚠️** Currently Scoped to Authenticated Users that can delete a longhorn settings object.
+    - **⚠️** Since the UI it self is not protected, network policies will need to be created to prevent namespace <--> namespace communication against the pod or service object directly.
+    - Anyone with access to the UI Deployment can remove the route restriction. (Namespace Scoped Admin)
+- Option to use separate disk in /var/mnt/longhorn & MachineConfig file to mount /var/mnt/longhorn
+- Adding finalizers for mount propagation
+
+## Known Issues
+
+- General Feature/Issue Thread
+  - [[FEATURE] Deploying Longhorn on OKD/Openshift](https://github.com/longhorn/longhorn/issues/1831)
+- 4.10 / 1.23:
+  - 4.10.0-0.okd-2022-03-07-131213 to 4.10.0-0.okd-2022-07-09-073606
+    - Tested, No Known Issues
+- 4.11 / 1.24:
+  - 4.11.0-0.okd-2022-07-27-052000 to 4.11.0-0.okd-2022-11-19-050030
+    - Tested, No Known Issues
+  - 4.11.0-0.okd-2022-12-02-145640, 4.11.0-0.okd-2023-01-14-152430:
+    - Workaround: [[BUG] Volumes Stuck in Attach/Detach Loop](https://github.com/longhorn/longhorn/issues/4988)
+      - [MachineConfig Patch](https://github.com/longhorn/longhorn/issues/4988#issuecomment-1345676772)
+- 4.12 / 1.25:
+  - 4.12.0-0.okd-2022-12-05-210624 to 4.12.0-0.okd-2023-01-20-101927
+    - Tested, No Known Issues
+  - 4.12.0-0.okd-2023-01-21-055900 to 4.12.0-0.okd-2023-02-18-033438:
+    - Workaround: [[BUG] Volumes Stuck in Attach/Detach Loop](https://github.com/longhorn/longhorn/issues/4988)
+      - [MachineConfig Patch](https://github.com/longhorn/longhorn/issues/4988#issuecomment-1345676772)
+  - 4.12.0-0.okd-2023-03-05-022504 - 4.12.0-0.okd-2023-04-16-041331:
+    - Tested, No Known Issues
+- 4.13 / 1.26:
+  - 4.13.0-0.okd-2023-05-03-001308 - 4.13.0-0.okd-2023-08-18-135805:
+    - Tested, No Known Issues
+- 4.14 / 1.27:
+  - 4.14.0-0.okd-2023-08-12-022330 - 4.14.0-0.okd-2023-10-28-073550:
+    - Tested, No Known Issues
+
+## Preparing Nodes (Optional)
+
+Only required if you require additional customizations, such as storage-less nodes, or secondary disks.
+
+### Default /var/lib/longhorn setup
+
+Label each node for storage with:
+
+```bash
+oc get nodes --no-headers | awk '{print $1}'
+
+export NODE="worker-0"
+oc label node "${NODE}" node.longhorn.io/create-default-disk=true
+```
+
+### Separate /var/mnt/longhorn setup
+
+#### Create Filesystem
+
+On the storage nodes create a filesystem with the label longhorn:
+
+```bash
+oc get nodes --no-headers | awk '{print $1}'
+
+export NODE="worker-0"
+oc debug node/${NODE} -t -- chroot /host bash
+
+# Validate Target Drive is Present
+lsblk
+
+export DRIVE="sdb" #vdb
+sudo mkfs.ext4 -L longhorn /dev/${DRIVE}
+```
+
+> ⚠️ Note: If you add New Nodes After the below Machine Config is applied, you will need to also reboot the node.
+
+#### Mounting Disk On Boot
+
+The Secondary Drive needs to be mounted on every boot. Save the Concents and Apply the MachineConfig with `oc apply -f`:
+
+> ⚠️ This will trigger an machine config profile update and reboot all worker nodes on the cluster
+
+```yaml
+apiVersion: machineconfiguration.openshift.io/v1
+kind: MachineConfig
+metadata:
+  labels:
+    machineconfiguration.openshift.io/role: worker
+  name: 71-mount-storage-worker
+spec:
+  config:
+    ignition:
+      version: 3.2.0
+    systemd:
+      units:
+        - name: var-mnt-longhorn.mount
+          enabled: true
+          contents: |
+            [Unit]
+            Before=local-fs.target
+            [Mount]
+            Where=/var/mnt/longhorn
+            What=/dev/disk/by-label/longhorn
+            Options=rw,relatime,discard
+            [Install]
+            WantedBy=local-fs.target
+```
+
+#### Label and Annotate Nodes
+
+Label and annotate storage nodes like this:
+
+```bash
+oc get nodes --no-headers | awk '{print $1}'
+
+export NODE="worker-0"
+oc annotate node ${NODE} --overwrite node.longhorn.io/default-disks-config='[{"path":"/var/mnt/longhorn","allowScheduling":true}]'
+oc label node ${NODE} node.longhorn.io/create-default-disk=config
+```
+
+## Example values.yaml
+
+Minimum Adjustments Required
+
+```yaml
+openshift:
+  oauthProxy:
+    repository: quay.io/openshift/origin-oauth-proxy
+    tag: 4.14  # Use Your OCP/OKD 4.X Version, Current Stable is 4.14
+
+# defaultSettings: # Preparing nodes (Optional)
+  # createDefaultDiskLabeledNodes: true
+
+openshift:
+  enabled: true
+  ui:
+    route: "longhorn-ui"
+    port: 443
+    proxy: 8443
+```
+
+## Installation
+
+```bash
+# helm template ./chart/ --namespace longhorn-system --values ./chart/values.yaml --no-hooks > longhorn.yaml # Local Testing
+helm template longhorn --namespace longhorn-system --values values.yaml --no-hooks  > longhorn.yaml
+oc create namespace longhorn-system -o yaml --dry-run=client | oc apply -f -
+oc apply -f longhorn.yaml -n longhorn-system
+```
+
+## Refs
+
+- <https://docs.openshift.com/container-platform/4.11/storage/persistent_storage/persistent-storage-iscsi.html>
+- <https://docs.okd.io/4.11/storage/persistent_storage/persistent-storage-iscsi.html>
+- okd 4.5: <https://github.com/longhorn/longhorn/issues/1831#issuecomment-702690613>
+- okd 4.6: <https://github.com/longhorn/longhorn/issues/1831#issuecomment-765884631>
+- oauth-proxy: <https://github.com/openshift/oauth-proxy/blob/master/contrib/sidecar.yaml>
+- <https://github.com/longhorn/longhorn/issues/1831>

chart/questions.yaml

@@ -89,7 +89,7 @@ questions:
     label: Longhorn Support Bundle Kit Image Repository
     group: "Longhorn Images Settings"
   - variable: image.longhorn.supportBundleKit.tag
-    default: v0.0.24
+    default: v0.0.27
     description: "Specify Longhorn Support Bundle Manager Image Tag"
     type: string
     label: Longhorn Support Bundle Kit Image Tag
@@ -244,7 +244,7 @@ questions:
     group: "Longhorn CSI Driver Settings"
   - variable: defaultSettings.backupTarget
     label: Backup Target
-    description: "The endpoint used to access the backupstore. NFS and S3 are supported."
+    description: "The endpoint used to access the backupstore. Available: NFS, CIFS, AWS, GCP, AZURE"
     group: "Longhorn Default Settings"
     type: string
     default:
@@ -256,8 +256,7 @@ questions:
     default:
   - variable: defaultSettings.allowRecurringJobWhileVolumeDetached
     label: Allow Recurring Job While Volume Is Detached
-    description: 'If this setting is enabled, Longhorn will automatically attaches the volume and takes snapshot/backup when it is the time to do recurring snapshot/backup.
-Note that the volume is not ready for workload during the period when the volume was automatically attached. Workload will have to wait until the recurring job finishes.'
+    description: 'If this setting is enabled, Longhorn will automatically attaches the volume and takes snapshot/backup when it is the time to do recurring snapshot/backup.'
     group: "Longhorn Default Settings"
     type: boolean
     default: "false"
@@ -275,11 +274,7 @@ Note that the volume is not ready for workload during the period when the volume
     default: "/var/lib/longhorn/"
   - variable: defaultSettings.defaultDataLocality
     label: Default Data Locality
-    description: 'We say a Longhorn volume has data locality if there is a local replica of the volume on the same node as the pod which is using the volume.
-This setting specifies the default data locality when a volume is created from the Longhorn UI. For Kubernetes configuration, update the `dataLocality` in the StorageClass
-The available modes are:
-- **disabled**. This is the default option. There may or may not be a replica on the same node as the attached volume (workload)
-- **best-effort**. This option instructs Longhorn to try to keep a replica on the same node as the attached volume (workload). Longhorn will not stop the volume, even if it cannot keep a replica local to the attached volume (workload) due to environment limitation, e.g. not enough disk space, incompatible disk tags, etc.'
+    description: 'Longhorn volume has data locality if there is a local replica of the volume on the same node as the pod which is using the volume.'
     group: "Longhorn Default Settings"
     type: enum
     options:
@@ -294,17 +289,7 @@ The available modes are:
     default: "false"
   - variable: defaultSettings.replicaAutoBalance
     label: Replica Auto Balance
-    description: 'Enable this setting automatically rebalances replicas when discovered an available node.
-The available global options are:
-- **disabled**. This is the default option. No replica auto-balance will be done.
-- **least-effort**. This option instructs Longhorn to balance replicas for minimal redundancy.
-- **best-effort**. This option instructs Longhorn to balance replicas for even redundancy.
-Longhorn also support individual volume setting. The setting can be specified in volume.spec.replicaAutoBalance, this overrules the global setting.
-The available volume spec options are:
-- **ignored**. This is the default option that instructs Longhorn to inherit from the global setting.
-- **disabled**. This option instructs Longhorn no replica auto-balance should be done.
-- **least-effort**. This option instructs Longhorn to balance replicas for minimal redundancy.
-- **best-effort**. This option instructs Longhorn to balance replicas for even redundancy.'
+    description: 'Enable this setting automatically rebalances replicas when discovered an available node.'
     group: "Longhorn Default Settings"
     type: enum
     options:
@@ -327,6 +312,14 @@ The available volume spec options are:
     min: 0
     max: 100
     default: 25
+  - variable: defaultSettings.storageReservedPercentageForDefaultDisk
+    label: Storage Reserved Percentage For Default Disk
+    description: "The reserved percentage specifies the percentage of disk space that will not be allocated to the default disk on each new Longhorn node."
+    group: "Longhorn Default Settings"
+    type: int
+    min: 0
+    max: 100
+    default: 30
   - variable: defaultSettings.upgradeChecker
     label: Enable Upgrade Checker
     description: 'Upgrade Checker will check for new Longhorn version periodically. When there is a new version available, a notification will appear in the UI. By default true.'
@@ -356,22 +349,14 @@ The available volume spec options are:
     default: 300
   - variable: defaultSettings.failedBackupTTL
     label: Failed Backup Time to Live
-    description: "In minutes. This setting determines how long Longhorn will keep the backup resource that was failed. Set to 0 to disable the auto-deletion.
-Failed backups will be checked and cleaned up during backupstore polling which is controlled by **Backupstore Poll Interval** setting.
-Hence this value determines the minimal wait interval of the cleanup. And the actual cleanup interval is multiple of **Backupstore Poll Interval**.
-Disabling **Backupstore Poll Interval** also means to disable failed backup auto-deletion."
+    description: "In minutes. This setting determines how long Longhorn will keep the backup resource that was failed. Set to 0 to disable the auto-deletion."
     group: "Longhorn Default Settings"
     type: int
     min: 0
     default: 1440
   - variable: defaultSettings.restoreVolumeRecurringJobs
     label: Restore Volume Recurring Jobs
-    description: "Restore recurring jobs from the backup volume on the backup target and create recurring jobs if not exist during a backup restoration.
-Longhorn also supports individual volume setting. The setting can be specified on Backup page when making a backup restoration, this overrules the global setting.
-The available volume setting options are:
-- **ignored**. This is the default option that instructs Longhorn to inherit from the global setting.
-- **enabled**. This option instructs Longhorn to restore recurring jobs/groups from the backup target forcibly.
-- **disabled**. This option instructs Longhorn no restoring recurring jobs/groups should be done."
+    description: "Restore recurring jobs from the backup volume on the backup target and create recurring jobs if not exist during a backup restoration."
     group: "Longhorn Default Settings"
     type: boolean
     default: "false"
@@ -391,9 +376,7 @@ The available volume setting options are:
     default: 1
   - variable: defaultSettings.supportBundleFailedHistoryLimit
     label: SupportBundle Failed History Limit
-    description: "This setting specifies how many failed support bundles can exist in the cluster.
-The retained failed support bundle is for analysis purposes and needs to clean up manually.
-Set this value to **0** to have Longhorn automatically purge all failed support bundles."
+    description: "This setting specifies how many failed support bundles can exist in the cluster. Set this value to **0** to have Longhorn automatically purge all failed support bundles."
     group: "Longhorn Default Settings"
     type: int
     min: 0
@@ -406,9 +389,7 @@ Set this value to **0** to have Longhorn automatically purge all failed support
     default: "true"
   - variable: defaultSettings.autoDeletePodWhenVolumeDetachedUnexpectedly
     label: Automatically Delete Workload Pod when The Volume Is Detached Unexpectedly
-    description: 'If enabled, Longhorn will automatically delete the workload pod that is managed by a controller (e.g. deployment, statefulset, daemonset, etc...) when Longhorn volume is detached unexpectedly (e.g. during Kubernetes upgrade, Docker reboot, or network disconnect). By deleting the pod, its controller restarts the pod and Kubernetes handles volume reattachment and remount.
-If disabled, Longhorn will not delete the workload pod that is managed by a controller. You will have to manually restart the pod to reattach and remount the volume.
-**Note:** This setting does not apply to the workload pods that do not have a controller. Longhorn never deletes them.'
+    description: 'If enabled, Longhorn will automatically delete the workload pod that is managed by a controller (e.g. deployment, statefulset, daemonset, etc...) when Longhorn volume is detached unexpectedly (e.g. during Kubernetes upgrade, Docker reboot, or network disconnect). By deleting the pod, its controller restarts the pod and Kubernetes handles volume reattachment and remount.'
     group: "Longhorn Default Settings"
     type: boolean
     default: "true"
@@ -424,13 +405,27 @@ If disabled, Longhorn will not delete the workload pod that is managed by a cont
     group: "Longhorn Default Settings"
     type: boolean
     default: "true"
+  - variable: defaultSettings.replicaDiskSoftAntiAffinity
+    label: Replica Disk Level Soft Anti-Affinity
+    description: 'Allow scheduling on disks with existing healthy replicas of the same volume. By default true.'
+    group: "Longhorn Default Settings"
+    type: boolean
+    default: "true"
+  - variable: defaultSettings.allowEmptyNodeSelectorVolume
+    label: Allow Empty Node Selector Volume
+    description: "Allow Scheduling Empty Node Selector Volumes To Any Node"
+    group: "Longhorn Default Settings"
+    type: boolean
+    default: "true"
+  - variable: defaultSettings.allowEmptyDiskSelectorVolume
+    label: Allow Empty Disk Selector Volume
+    description: "Allow Scheduling Empty Disk Selector Volumes To Any Disk"
+    group: "Longhorn Default Settings"
+    type: boolean
+    default: "true"
   - variable: defaultSettings.nodeDownPodDeletionPolicy
     label: Pod Deletion Policy When Node is Down
-    description: "Defines the Longhorn action when a Volume is stuck with a StatefulSet/Deployment Pod on a node that is down.
-- **do-nothing** is the default Kubernetes behavior of never force deleting StatefulSet/Deployment terminating pods. Since the pod on the node that is down isn't removed, Longhorn volumes are stuck on nodes that are down.
-- **delete-statefulset-pod** Longhorn will force delete StatefulSet terminating pods on nodes that are down to release Longhorn volumes so that Kubernetes can spin up replacement pods.
-- **delete-deployment-pod** Longhorn will force delete Deployment terminating pods on nodes that are down to release Longhorn volumes so that Kubernetes can spin up replacement pods.
-- **delete-both-statefulset-and-deployment-pod** Longhorn will force delete StatefulSet/Deployment terminating pods on nodes that are down to release Longhorn volumes so that Kubernetes can spin up replacement pods."
+    description: "Defines the Longhorn action when a Volume is stuck with a StatefulSet/Deployment Pod on a node that is down."
     group: "Longhorn Default Settings"
     type: enum
     options:
@@ -441,10 +436,7 @@ If disabled, Longhorn will not delete the workload pod that is managed by a cont
     default: "do-nothing"
   - variable: defaultSettings.nodeDrainPolicy
     label: Node Drain Policy
-    description: "Define the policy to use when a node with the last healthy replica of a volume is drained.
-- **block-if-contains-last-replica** Longhorn will block the drain when the node contains the last healthy replica of a volume.
-- **allow-if-replica-is-stopped** Longhorn will allow the drain when the node contains the last healthy replica of a volume but the replica is stopped. WARNING: possible data loss if the node is removed after draining. Select this option if you want to drain the node and do in-place upgrade/maintenance.
-- **always-allow** Longhorn will allow the drain even though the node contains the last healthy replica of a volume. WARNING: possible data loss if the node is removed after draining. Also possible data corruption if the last replica was running during the draining."
+    description: "Define the policy to use when a node with the last healthy replica of a volume is drained."
     group: "Longhorn Default Settings"
     type: enum
     options:
@@ -454,29 +446,21 @@ If disabled, Longhorn will not delete the workload pod that is managed by a cont
     default: "block-if-contains-last-replica"
   - variable: defaultSettings.replicaReplenishmentWaitInterval
     label: Replica Replenishment Wait Interval
-    description: "In seconds. The interval determines how long Longhorn will wait at least in order to reuse the existing data on a failed replica rather than directly creating a new replica for a degraded volume.
-Warning: This option works only when there is a failed replica in the volume. And this option may block the rebuilding for a while in the case."
+    description: "In seconds. The interval determines how long Longhorn will wait at least in order to reuse the existing data on a failed replica rather than directly creating a new replica for a degraded volume."
     group: "Longhorn Default Settings"
     type: int
     min: 0
     default: 600
   - variable: defaultSettings.concurrentReplicaRebuildPerNodeLimit
     label: Concurrent Replica Rebuild Per Node Limit
-    description: "This setting controls how many replicas on a node can be rebuilt simultaneously.
-Typically, Longhorn can block the replica starting once the current rebuilding count on a node exceeds the limit. But when the value is 0, it means disabling the replica rebuilding.
-WARNING:
-- The old setting \"Disable Replica Rebuild\" is replaced by this setting.
-- Different from relying on replica starting delay to limit the concurrent rebuilding, if the rebuilding is disabled, replica object replenishment will be directly skipped.
-- When the value is 0, the eviction and data locality feature won't work. But this shouldn't have any impact to any current replica rebuild and backup restore."
+    description: "This setting controls how many replicas on a node can be rebuilt simultaneously."
     group: "Longhorn Default Settings"
     type: int
     min: 0
     default: 5
   - variable: defaultSettings.concurrentVolumeBackupRestorePerNodeLimit
     label: Concurrent Volume Backup Restore Per Node Limit
-    description: "This setting controls how many volumes on a node can restore the backup concurrently.
-Longhorn blocks the backup restore once the restoring volume count exceeds the limit.
-Set the value to **0** to disable backup restore."
+    description: "This setting controls how many volumes on a node can restore the backup concurrently. Set the value to **0** to disable backup restore."
     group: "Longhorn Default Settings"
     type: int
     min: 0
@@ -525,45 +509,14 @@ Set the value to **0** to disable backup restore."
     default: 60
   - variable: defaultSettings.backingImageRecoveryWaitInterval
     label: Backing Image Recovery Wait Interval
-    description: "This interval in seconds determines how long Longhorn will wait before re-downloading the backing image file when all disk files of this backing image become failed or unknown.
-    WARNING:
-      - This recovery only works for the backing image of which the creation type is \"download\".
-      - File state \"unknown\" means the related manager pods on the pod is not running or the node itself is down/disconnected."
+    description: "This interval in seconds determines how long Longhorn will wait before re-downloading the backing image file when all disk files of this backing image become failed or unknown."
     group: "Longhorn Default Settings"
     type: int
     min: 0
     default: 300
-  - variable: defaultSettings.guaranteedEngineManagerCPU
-    label: Guaranteed Engine Manager CPU
-    description: "This integer value indicates how many percentage of the total allocatable CPU on each node will be reserved for each engine manager Pod. For example, 10 means 10% of the total CPU on a node will be allocated to each engine manager pod on this node. This will help maintain engine stability during high node workload.
-    In order to prevent unexpected volume engine crash as well as guarantee a relative acceptable IO performance, you can use the following formula to calculate a value for this setting:
-    Guaranteed Engine Manager CPU = The estimated max Longhorn volume engine count on a node * 0.1 / The total allocatable CPUs on the node * 100.
-    The result of above calculation doesn't mean that's the maximum CPU resources the Longhorn workloads require. To fully exploit the Longhorn volume I/O performance, you can allocate/guarantee more CPU resources via this setting.
-    If it's hard to estimate the usage now, you can leave it with the default value, which is 12%. Then you can tune it when there is no running workload using Longhorn volumes.
-    WARNING:
-      - Value 0 means unsetting CPU requests for engine manager pods.
-      - Considering the possible new instance manager pods in the further system upgrade, this integer value is range from 0 to 40. And the sum with setting 'Guaranteed Engine Manager CPU' should not be greater than 40.
-      - One more set of instance manager pods may need to be deployed when the Longhorn system is upgraded. If current available CPUs of the nodes are not enough for the new instance manager pods, you need to detach the volumes using the oldest instance manager pods so that Longhorn can clean up the old pods automatically and release the CPU resources. And the new pods with the latest instance manager image will be launched then.
-      - This global setting will be ignored for a node if the field \"EngineManagerCPURequest\" on the node is set.
-      - After this setting is changed, all engine manager pods using this global setting on all the nodes will be automatically restarted. In other words, DO NOT CHANGE THIS SETTING WITH ATTACHED VOLUMES."
-    group: "Longhorn Default Settings"
-    type: int
-    min: 0
-    max: 40
-    default: 12
-  - variable: defaultSettings.guaranteedReplicaManagerCPU
-    label: Guaranteed Replica Manager CPU
-    description: "This integer value indicates how many percentage of the total allocatable CPU on each node will be reserved for each replica manager Pod. 10 means 10% of the total CPU on a node will be allocated to each replica manager pod on this node. This will help maintain replica stability during high node workload.
-    In order to prevent unexpected volume replica crash as well as guarantee a relative acceptable IO performance, you can use the following formula to calculate a value for this setting:
-    Guaranteed Replica Manager CPU = The estimated max Longhorn volume replica count on a node * 0.1 / The total allocatable CPUs on the node * 100.
-    The result of above calculation doesn't mean that's the maximum CPU resources the Longhorn workloads require. To fully exploit the Longhorn volume I/O performance, you can allocate/guarantee more CPU resources via this setting.
-    If it's hard to estimate the usage now, you can leave it with the default value, which is 12%. Then you can tune it when there is no running workload using Longhorn volumes.
-    WARNING:
-      - Value 0 means unsetting CPU requests for replica manager pods.
-      - Considering the possible new instance manager pods in the further system upgrade, this integer value is range from 0 to 40. And the sum with setting 'Guaranteed Replica Manager CPU' should not be greater than 40.
-      - One more set of instance manager pods may need to be deployed when the Longhorn system is upgraded. If current available CPUs of the nodes are not enough for the new instance manager pods, you need to detach the volumes using the oldest instance manager pods so that Longhorn can clean up the old pods automatically and release the CPU resources. And the new pods with the latest instance manager image will be launched then.
-      - This global setting will be ignored for a node if the field \"ReplicaManagerCPURequest\" on the node is set.
-      - After this setting is changed, all replica manager pods using this global setting on all the nodes will be automatically restarted. In other words, DO NOT CHANGE THIS SETTING WITH ATTACHED VOLUMES."
+  - variable: defaultSettings.guaranteedInstanceManagerCPU
+    label: Guaranteed Instance Manager CPU
+    description: "This integer value indicates how many percentage of the total allocatable CPU on each node will be reserved for each instance manager Pod. You can leave it with the default value, which is 12%."
     group: "Longhorn Default Settings"
     type: int
     min: 0
@@ -571,18 +524,13 @@ Set the value to **0** to disable backup restore."
     default: 12
   - variable: defaultSettings.logLevel
     label: Log Level
-    description: "The log level Panic, Fatal, Error, Warn, Info, Debug, Trace used in longhorn manager. By default Debug."
+    description: "The log level Panic, Fatal, Error, Warn, Info, Debug, Trace used in longhorn manager. Default to Info."
     group: "Longhorn Default Settings"
     type: string
-    default: "Debug"
+    default: "Info"
 - variable: defaultSettings.kubernetesClusterAutoscalerEnabled
   label: Kubernetes Cluster Autoscaler Enabled (Experimental)
-  description: "Enabling this setting will notify Longhorn that the cluster is using Kubernetes Cluster Autoscaler.
-  Longhorn prevents data loss by only allowing the Cluster Autoscaler to scale down a node that met all conditions:
-    - No volume attached to the node.
-    - Is not the last node containing the replica of any volume.
-    - Is not running backing image components pod.
-    - Is not running share manager components pod."
+  description: "Enabling this setting will notify Longhorn that the cluster is using Kubernetes Cluster Autoscaler."
   group: "Longhorn Default Settings"
   type: boolean
   default: false
@@ -594,20 +542,13 @@ Set the value to **0** to disable backup restore."
   default: false
 - variable: defaultSettings.storageNetwork
   label: Storage Network
-  description: "Longhorn uses the storage network for in-cluster data traffic. Leave this blank to use the Kubernetes cluster network.
-	To segregate the storage network, input the pre-existing NetworkAttachmentDefinition in \"<namespace>/<name>\" format.
-	WARNING:
-	  - The cluster must have pre-existing Multus installed, and NetworkAttachmentDefinition IPs are reachable between nodes.
-	  - DO NOT CHANGE THIS SETTING WITH ATTACHED VOLUMES. Longhorn will try to block this setting update when there are attached volumes.
-	  - When applying the setting, Longhorn will restart all manager, instance-manager, and backing-image-manager pods."
+  description: "Longhorn uses the storage network for in-cluster data traffic. Leave this blank to use the Kubernetes cluster network."
   group: "Longhorn Default Settings"
   type: string
   default:
 - variable: defaultSettings.deletingConfirmationFlag
   label: Deleting Confirmation Flag
-  description: "This flag is designed to prevent Longhorn from being accidentally uninstalled which will lead to data lost.
-	Set this flag to **true** to allow Longhorn uninstallation.
-	If this flag **false**, Longhorn uninstallation job will fail. "
+  description: "This flag is designed to prevent Longhorn from being accidentally uninstalled which will lead to data lost."
   group: "Longhorn Default Settings"
   type: boolean
   default: "false"
@@ -619,11 +560,7 @@ Set the value to **0** to disable backup restore."
   default: "8"
 - variable: defaultSettings.snapshotDataIntegrity
   label: Snapshot Data Integrity
-  description: "This setting allows users to enable or disable snapshot hashing and data integrity checking.
-  Available options are
-    - **disabled**: Disable snapshot disk file hashing and data integrity checking.
-    - **enabled**: Enables periodic snapshot disk file hashing and data integrity checking. To detect the filesystem-unaware corruption caused by bit rot or other issues in snapshot disk files, Longhorn system periodically hashes files and finds corrupted ones. Hence, the system performance will be impacted during the periodical checking.
-    - **fast-check**: Enable snapshot disk file hashing and fast data integrity checking. Longhorn system only hashes snapshot disk files if their are not hashed or the modification time are changed. In this mode, filesystem-unaware corruption cannot be detected, but the impact on system performance can be minimized."
+  description: "This setting allows users to enable or disable snapshot hashing and data integrity checking."
   group: "Longhorn Default Settings"
   type: string
   default: "disabled"
@@ -635,17 +572,13 @@ Set the value to **0** to disable backup restore."
   default: "false"
 - variable: defaultSettings.snapshotDataIntegrityCronjob
   label: Snapshot Data Integrity Check CronJob
-  description: "Unix-cron string format. The setting specifies when Longhorn checks the data integrity of snapshot disk files.
-  Warning: Hashing snapshot disk files impacts the performance of the system. It is recommended to run data integrity checks during off-peak times and to reduce the frequency of checks."
+  description: "Unix-cron string format. The setting specifies when Longhorn checks the data integrity of snapshot disk files."
   group: "Longhorn Default Settings"
   type: string
   default: "0 0 */7 * *"
 - variable: defaultSettings.removeSnapshotsDuringFilesystemTrim
   label: Remove Snapshots During Filesystem Trim
-  description: "This setting allows Longhorn filesystem trim feature to automatically mark the latest snapshot and its ancestors as removed and stops at the snapshot containing multiple children.\n\n
-    Since Longhorn filesystem trim feature can be applied to the volume head and the followed continuous removed or system snapshots only.\n\n
-    Notice that trying to trim a removed files from a valid snapshot will do nothing but the filesystem will discard this kind of in-memory trimmable file info.\n\n
-    Later on if you mark the snapshot as removed and want to retry the trim, you may need to unmount and remount the filesystem so that the filesystem can recollect the trimmable file info."
+  description: "This setting allows Longhorn filesystem trim feature to automatically mark the latest snapshot and its ancestors as removed and stops at the snapshot containing multiple children."
   group: "Longhorn Default Settings"
   type: boolean
   default: "false"
@@ -663,11 +596,7 @@ Set the value to **0** to disable backup restore."
   default: "30"
 - variable: defaultSettings.backupCompressionMethod
   label: Backup Compression Method
-  description: "This setting allows users to specify backup compression method.
-  Available options are
-    - **none**: Disable the compression method. Suitable for multimedia data such as encoded images and videos.
-    - **lz4**: Fast compression method. Suitable for flat files.
-    - **gzip**: A bit of higher compression ratio but relatively slow."
+  description: "This setting allows users to specify backup compression method."
   group: "Longhorn Default Settings"
   type: string
   default: "lz4"
@@ -677,24 +606,30 @@ Set the value to **0** to disable backup restore."
   group: "Longhorn Default Settings"
   type: int
   min: 1
-  default: 5
+  default: 2
 - variable: defaultSettings.restoreConcurrentLimit
   label: Restore Concurrent Limit Per Backup
   description: "This setting controls how many worker threads per restore concurrently."
   group: "Longhorn Default Settings"
   type: int
   min: 1
-  default: 5
-- variable: defaultSettings.spdk
-  label: Enable SPDK Data Engine (Preview Feature)
-  description: "This allows users to activate SPDK data engine. Currently, it is in the preview phase and should not be utilized in a production environment.
-	WARNING:
-	  - The cluster must have pre-existing Multus installed, and NetworkAttachmentDefinition IPs are reachable between nodes.
-	  - DO NOT CHANGE THIS SETTING WITH ATTACHED VOLUMES. Longhorn will try to block this setting update when there are attached volumes.
-	  - When applying the setting, Longhorn will restart all instance-manager pods."
-  group: "Longhorn SPDK Data Engine Settings"
+  default: 2
+- variable: defaultSettings.v2DataEngine
+  label: V2 Data Engine
+  description: "This allows users to activate v2 data engine based on SPDK. Currently, it is in the preview phase and should not be utilized in a production environment."
+  group: "Longhorn V2 Data Engine (Preview Feature) Settings"
   type: boolean
   default: false
+- variable: defaultSettings.offlineReplicaRebuilding
+  label: Offline Replica Rebuilding
+  description: "This setting allows users to enable the offline replica rebuilding for volumes using v2 data engine."
+  group: "Longhorn V2 Data Engine (Preview Feature) Settings"
+  required: true
+  type: enum
+  options:
+  - "enabled"
+  - "disabled"
+  default: "enabled"
 - variable: persistence.defaultClass
   default: "true"
   description: "Set as default StorageClass for Longhorn"
@@ -704,7 +639,7 @@ Set the value to **0** to disable backup restore."
   type: boolean
 - variable: persistence.reclaimPolicy
   label: Storage Class Retain Policy
-  description: "Define reclaim policy (Retain or Delete)"
+  description: "Define reclaim policy. Options: `Retain`, `Delete`"
   group: "Longhorn Storage Class Settings"
   required: true
   type: enum
@@ -721,7 +656,7 @@ Set the value to **0** to disable backup restore."
   max: 10
   default: 3
 - variable: persistence.defaultDataLocality
-  description: "Set data locality for Longhorn StorageClass"
+  description: "Set data locality for Longhorn StorageClass. Options: `disabled`, `best-effort`"
   label: Default Storage Class Data Locality
   group: "Longhorn Storage Class Settings"
   type: enum
@@ -743,18 +678,18 @@ Set the value to **0** to disable backup restore."
     group: "Longhorn Storage Class Settings"
     type: string
     default:
-- variable: defaultSettings.defaultNodeSelector.enable
-  description: "Enable recurring Node selector for Longhorn StorageClass"
+- variable: persistence.defaultNodeSelector.enable
+  description: "Enable Node selector for Longhorn StorageClass"
   group: "Longhorn Storage Class Settings"
   label: Enable Storage Class Node Selector
   type: boolean
   default: false
   show_subquestion_if: true
   subquestions:
-  - variable: defaultSettings.defaultNodeSelector.selector
+  - variable: persistence.defaultNodeSelector.selector
     label: Storage Class Node Selector
-    description: 'We use NodeSelector when we want to bind PVC via StorageClass into desired mountpoint on the nodes tagged whith its value'
-    group: "Longhorn Default Settings"
+    description: 'This selector enables only certain nodes having these tags to be used for the volume. e.g. `"storage,fast"`'
+    group: "Longhorn Storage Class Settings"
     type: string
     default:
 - variable: persistence.backingImage.enable
@@ -807,7 +742,7 @@ Set the value to **0** to disable backup restore."
     type: string
     default:
 - variable: persistence.removeSnapshotsDuringFilesystemTrim
-  description: "Allow automatically removing snapshots during filesystem trim for Longhorn StorageClass"
+  description: "Allow automatically removing snapshots during filesystem trim for Longhorn StorageClass. Options: `ignored`, `enabled`, `disabled`"
   label: Default Storage Class Remove Snapshots During Filesystem Trim
   group: "Longhorn Storage Class Settings"
   type: enum
@@ -838,7 +773,7 @@ Set the value to **0** to disable backup restore."
     label: Ingress Path
 - variable: service.ui.type
   default: "Rancher-Proxy"
-  description: "Define Longhorn UI service type"
+  description: "Define Longhorn UI service type. Options: `ClusterIP`, `NodePort`, `LoadBalancer`, `Rancher-Proxy`"
   type: enum
   options:
     - "ClusterIP"
@@ -880,7 +815,7 @@ Set the value to **0** to disable backup restore."
   subquestions:
   - variable: networkPolicies.type
     label: Network Policies for Ingress
-    description: "Create the policy to allow access for the ingress, select the distribution."
+    description: "Create the policy based on your distribution to allow access for the ingress. Options: `k3s`, `rke2`, `rke1`"
     show_if: "networkPolicies.enabled=true&&ingress.enabled=true"
     type: enum
     default: "rke2"

chart/templates/clusterrole.yaml

@@ -37,6 +37,9 @@ rules:
 - apiGroups: ["longhorn.io"]
   resources: ["volumes", "volumes/status", "engines", "engines/status", "replicas", "replicas/status", "settings",
               "engineimages", "engineimages/status", "nodes", "nodes/status", "instancemanagers", "instancemanagers/status",
+  {{- if .Values.openshift.enabled }}
+              "engineimages/finalizers", "nodes/finalizers", "instancemanagers/finalizers",
+  {{- end }}
               "sharemanagers", "sharemanagers/status", "backingimages", "backingimages/status",
               "backingimagemanagers", "backingimagemanagers/status", "backingimagedatasources", "backingimagedatasources/status",
               "backuptargets", "backuptargets/status", "backupvolumes", "backupvolumes/status", "backups", "backups/status",
@@ -59,3 +62,16 @@ rules:
 - apiGroups: ["rbac.authorization.k8s.io"]
   resources: ["roles", "rolebindings", "clusterrolebindings", "clusterroles"]
   verbs: ["*"]
+{{- if .Values.openshift.enabled }}
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: longhorn-ocp-privileged-role
+  labels: {{- include "longhorn.labels" . | nindent 4 }}
+rules:
+- apiGroups: ["security.openshift.io"]
+  resources: ["securitycontextconstraints"]
+  resourceNames: ["anyuid", "privileged"]
+  verbs: ["use"]
+{{- end }}

chart/templates/clusterrolebinding.yaml

@@ -25,3 +25,25 @@ subjects:
 - kind: ServiceAccount
   name: longhorn-support-bundle
   namespace: {{ include "release_namespace" . }}
+{{- if .Values.openshift.enabled }}
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: longhorn-ocp-privileged-bind
+  labels: {{- include "longhorn.labels" . | nindent 4 }}
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: longhorn-ocp-privileged-role
+subjects:
+- kind: ServiceAccount
+  name: longhorn-service-account
+  namespace: {{ include "release_namespace" . }}
+- kind: ServiceAccount
+  name: longhorn-ui-service-account
+  namespace: {{ include "release_namespace" . }}
+- kind: ServiceAccount
+  name: default # supportbundle-agent-support-bundle uses default sa
+  namespace: {{ include "release_namespace" . }}
+{{- end }}

chart/templates/crds.yaml

@@ -1316,8 +1316,8 @@ spec:
                 type: boolean
               backendStoreDriver:
                 enum:
-                - longhorn
-                - spdk
+                - v1
+                - v2
                 type: string
               backupVolume:
                 type: string
@@ -1326,6 +1326,7 @@ spec:
               disableFrontend:
                 type: boolean
               engineImage:
+                description: 'Deprecated: Replaced by field `image`.'
                 type: string
               frontend:
                 enum:
@@ -1334,6 +1335,8 @@ spec:
                 - nvmf
                 - ""
                 type: string
+              image:
+                type: string
               logRequested:
                 type: boolean
               nodeID:
@@ -2418,8 +2421,8 @@ spec:
                 type: boolean
               backendStoreDriver:
                 enum:
-                - longhorn
-                - spdk
+                - v1
+                - v2
                 type: string
               backingImage:
                 type: string
@@ -2432,6 +2435,7 @@ spec:
               diskPath:
                 type: string
               engineImage:
+                description: 'Deprecated: Replaced by field `image`.'
                 type: string
               engineName:
                 type: string
@@ -2441,6 +2445,8 @@ spec:
                 type: string
               healthyAt:
                 type: string
+              image:
+                type: string
               logRequested:
                 type: boolean
               nodeID:
@@ -3314,8 +3320,8 @@ spec:
                 type: string
               backendStoreDriver:
                 enum:
-                - longhorn
-                - spdk
+                - v1
+                - v2
                 type: string
               backingImage:
                 type: string
@@ -3342,6 +3348,7 @@ spec:
               encrypted:
                 type: boolean
               engineImage:
+                description: 'Deprecated: Replaced by field `image`.'
                 type: string
               fromBackup:
                 type: string
@@ -3352,6 +3359,8 @@ spec:
                 - nvmf
                 - ""
                 type: string
+              image:
+                type: string
               lastAttachedBy:
                 type: string
               migratable:
@@ -3366,6 +3375,13 @@ spec:
                 type: array
               numberOfReplicas:
                 type: integer
+              offlineReplicaRebuilding:
+                description: OfflineReplicaRebuilding is used to determine if the offline replica rebuilding feature is enabled or not
+                enum:
+                - ignored
+                - disabled
+                - enabled
+                type: string
               replicaAutoBalance:
                 enum:
                 - ignored
@@ -3373,15 +3389,22 @@ spec:
                 - least-effort
                 - best-effort
                 type: string
+              replicaDiskSoftAntiAffinity:
+                description: Replica disk soft anti affinity of the volume. Set enabled to allow replicas to be scheduled in the same disk.
+                enum:
+                - ignored
+                - enabled
+                - disabled
+                type: string
               replicaSoftAntiAffinity:
-                description: Replica soft anti affinity of the volume. Set enabled to allow replicas to be scheduled on the same node
+                description: Replica soft anti affinity of the volume. Set enabled to allow replicas to be scheduled on the same node.
                 enum:
                 - ignored
                 - enabled
                 - disabled
                 type: string
               replicaZoneSoftAntiAffinity:
-                description: Replica zone soft anti affinity of the volume. Set enabled to allow replicas to be scheduled in the same zone
+                description: Replica zone soft anti affinity of the volume. Set enabled to allow replicas to be scheduled in the same zone.
                 enum:
                 - ignored
                 - enabled
@@ -3503,6 +3526,8 @@ spec:
                 type: string
               lastDegradedAt:
                 type: string
+              offlineReplicaRebuildingRequired:
+                type: boolean
               ownerID:
                 type: string
               pendingNodeID:

chart/templates/default-setting.yaml

@@ -15,6 +15,7 @@ data:
     {{ if not (kindIs "invalid" .Values.defaultSettings.replicaAutoBalance) }}replica-auto-balance: {{ .Values.defaultSettings.replicaAutoBalance }}{{ end }}
     {{ if not (kindIs "invalid" .Values.defaultSettings.storageOverProvisioningPercentage) }}storage-over-provisioning-percentage: {{ .Values.defaultSettings.storageOverProvisioningPercentage }}{{ end }}
     {{ if not (kindIs "invalid" .Values.defaultSettings.storageMinimalAvailablePercentage) }}storage-minimal-available-percentage: {{ .Values.defaultSettings.storageMinimalAvailablePercentage }}{{ end }}
+    {{ if not (kindIs "invalid" .Values.defaultSettings.storageReservedPercentageForDefaultDisk) }}storage-reserved-percentage-for-default-disk: {{ .Values.defaultSettings.storageReservedPercentageForDefaultDisk }}{{ end }}
     {{ if not (kindIs "invalid" .Values.defaultSettings.upgradeChecker) }}upgrade-checker: {{ .Values.defaultSettings.upgradeChecker }}{{ end }}
     {{ if not (kindIs "invalid" .Values.defaultSettings.defaultReplicaCount) }}default-replica-count: {{ .Values.defaultSettings.defaultReplicaCount }}{{ end }}
     {{ if not (kindIs "invalid" .Values.defaultSettings.defaultDataLocality) }}default-data-locality: {{ .Values.defaultSettings.defaultDataLocality }}{{ end }}
@@ -50,6 +51,7 @@ data:
     {{ if not (kindIs "invalid" .Values.defaultSettings.autoDeletePodWhenVolumeDetachedUnexpectedly) }}auto-delete-pod-when-volume-detached-unexpectedly: {{ .Values.defaultSettings.autoDeletePodWhenVolumeDetachedUnexpectedly }}{{ end }}
     {{ if not (kindIs "invalid" .Values.defaultSettings.disableSchedulingOnCordonedNode) }}disable-scheduling-on-cordoned-node: {{ .Values.defaultSettings.disableSchedulingOnCordonedNode }}{{ end }}
     {{ if not (kindIs "invalid" .Values.defaultSettings.replicaZoneSoftAntiAffinity) }}replica-zone-soft-anti-affinity: {{ .Values.defaultSettings.replicaZoneSoftAntiAffinity }}{{ end }}
+    {{ if not (kindIs "invalid" .Values.defaultSettings.replicaDiskSoftAntiAffinity) }}replica-disk-soft-anti-affinity: {{ .Values.defaultSettings.replicaDiskSoftAntiAffinity }}{{ end }}
     {{ if not (kindIs "invalid" .Values.defaultSettings.nodeDownPodDeletionPolicy) }}node-down-pod-deletion-policy: {{ .Values.defaultSettings.nodeDownPodDeletionPolicy }}{{ end }}
     {{ if not (kindIs "invalid" .Values.defaultSettings.nodeDrainPolicy) }}node-drain-policy: {{ .Values.defaultSettings.nodeDrainPolicy }}{{ end }}
     {{ if not (kindIs "invalid" .Values.defaultSettings.replicaReplenishmentWaitInterval) }}replica-replenishment-wait-interval: {{ .Values.defaultSettings.replicaReplenishmentWaitInterval }}{{ end }}
@@ -62,8 +64,7 @@ data:
     {{ if not (kindIs "invalid" .Values.defaultSettings.concurrentAutomaticEngineUpgradePerNodeLimit) }}concurrent-automatic-engine-upgrade-per-node-limit: {{ .Values.defaultSettings.concurrentAutomaticEngineUpgradePerNodeLimit }}{{ end }}
     {{ if not (kindIs "invalid" .Values.defaultSettings.backingImageCleanupWaitInterval) }}backing-image-cleanup-wait-interval: {{ .Values.defaultSettings.backingImageCleanupWaitInterval }}{{ end }}
     {{ if not (kindIs "invalid" .Values.defaultSettings.backingImageRecoveryWaitInterval) }}backing-image-recovery-wait-interval: {{ .Values.defaultSettings.backingImageRecoveryWaitInterval }}{{ end }}
-    {{ if not (kindIs "invalid" .Values.defaultSettings.guaranteedEngineManagerCPU) }}guaranteed-engine-manager-cpu: {{ .Values.defaultSettings.guaranteedEngineManagerCPU }}{{ end }}
-    {{ if not (kindIs "invalid" .Values.defaultSettings.guaranteedReplicaManagerCPU) }}guaranteed-replica-manager-cpu: {{ .Values.defaultSettings.guaranteedReplicaManagerCPU }}{{ end }}
+    {{ if not (kindIs "invalid" .Values.defaultSettings.guaranteedInstanceManagerCPU) }}guaranteed-instance-manager-cpu: {{ .Values.defaultSettings.guaranteedInstanceManagerCPU }}{{ end }}
     {{ if not (kindIs "invalid" .Values.defaultSettings.kubernetesClusterAutoscalerEnabled) }}kubernetes-cluster-autoscaler-enabled: {{ .Values.defaultSettings.kubernetesClusterAutoscalerEnabled }}{{ end }}
     {{ if not (kindIs "invalid" .Values.defaultSettings.orphanAutoDeletion) }}orphan-auto-deletion: {{ .Values.defaultSettings.orphanAutoDeletion }}{{ end }}
     {{ if not (kindIs "invalid" .Values.defaultSettings.storageNetwork) }}storage-network: {{ .Values.defaultSettings.storageNetwork }}{{ end }}
@@ -79,4 +80,7 @@ data:
     {{ if not (kindIs "invalid" .Values.defaultSettings.backupCompressionMethod) }}backup-compression-method: {{ .Values.defaultSettings.backupCompressionMethod }}{{ end }}
     {{ if not (kindIs "invalid" .Values.defaultSettings.backupConcurrentLimit) }}backup-concurrent-limit: {{ .Values.defaultSettings.backupConcurrentLimit }}{{ end }}
     {{ if not (kindIs "invalid" .Values.defaultSettings.restoreConcurrentLimit) }}restore-concurrent-limit: {{ .Values.defaultSettings.restoreConcurrentLimit }}{{ end }}
-    {{ if not (kindIs "invalid" .Values.defaultSettings.spdk) }}spdk: {{ .Values.defaultSettings.spdk }}{{ end }}
+    {{ if not (kindIs "invalid" .Values.defaultSettings.v2DataEngine) }}v2-data-engine: {{ .Values.defaultSettings.v2DataEngine }}{{ end }}
+    {{ if not (kindIs "invalid" .Values.defaultSettings.offlineReplicaRebuilding) }}offline-replica-rebuilding: {{ .Values.defaultSettings.offlineReplicaRebuilding }}{{ end }}
+    {{ if not (kindIs "invalid" .Values.defaultSettings.allowEmptyNodeSelectorVolume) }}allow-empty-node-selector-volume: {{ .Values.defaultSettings.allowEmptyNodeSelectorVolume }}{{ end }}
+    {{ if not (kindIs "invalid" .Values.defaultSettings.allowEmptyDiskSelectorVolume) }}allow-empty-disk-selector-volume: {{ .Values.defaultSettings.allowEmptyDiskSelectorVolume }}{{ end }}

chart/templates/deployment-ui.yaml

@@ -1,3 +1,41 @@
+{{- if .Values.openshift.enabled }}
+{{- if .Values.openshift.ui.route }}
+# https://github.com/openshift/oauth-proxy/blob/master/contrib/sidecar.yaml
+# Create a proxy service account and ensure it will use the route "proxy"
+# Create a secure connection to the proxy via a route
+apiVersion: route.openshift.io/v1
+kind: Route
+metadata:
+  labels: {{- include "longhorn.labels" . | nindent 4 }}
+    app: longhorn-ui
+  name: {{ .Values.openshift.ui.route }}
+  namespace: {{ include "release_namespace" . }}
+spec:
+  to:
+    kind: Service
+    name: longhorn-ui
+  tls:
+    termination: reencrypt
+---
+apiVersion: v1
+kind: Service
+metadata:
+  labels: {{- include "longhorn.labels" . | nindent 4 }}
+    app: longhorn-ui
+  name: longhorn-ui
+  namespace: {{ include "release_namespace" . }}
+  annotations:
+    service.alpha.openshift.io/serving-cert-secret-name: longhorn-ui-tls
+spec:
+  ports:
+  - name: longhorn-ui
+    port: {{ .Values.openshift.ui.port | default 443 }}
+    targetPort: {{ .Values.openshift.ui.proxy | default 8443 }}
+  selector:
+    app: longhorn-ui
+---
+{{- end }}
+{{- end }}
 apiVersion: apps/v1
 kind: Deployment
 metadata:
@@ -15,6 +53,7 @@ spec:
       labels: {{- include "longhorn.labels" . | nindent 8 }}
         app: longhorn-ui
     spec:
+      serviceAccountName: longhorn-ui-service-account
       affinity:
         podAntiAffinity:
           preferredDuringSchedulingIgnoredDuringExecution:
@@ -28,6 +67,28 @@ spec:
                   - longhorn-ui
               topologyKey: kubernetes.io/hostname
       containers:
+      {{- if .Values.openshift.enabled }}
+      {{- if .Values.openshift.ui.route }}
+      - name: oauth-proxy
+        image: {{ template "registry_url" . }}{{ .Values.image.openshift.oauthProxy.repository }}:{{ .Values.image.openshift.oauthProxy.tag }}
+        imagePullPolicy: IfNotPresent
+        ports:
+        - containerPort: {{ .Values.openshift.ui.proxy | default 8443 }}
+          name: public
+        args:
+        - --https-address=:{{ .Values.openshift.ui.proxy | default 8443 }}
+        - --provider=openshift
+        - --openshift-service-account=longhorn-ui-service-account
+        - --upstream=http://localhost:8000
+        - --tls-cert=/etc/tls/private/tls.crt
+        - --tls-key=/etc/tls/private/tls.key
+        - --cookie-secret=SECRET
+        - --openshift-sar={"namespace":"{{ include "release_namespace" . }}","group":"longhorn.io","resource":"setting","verb":"delete"}
+        volumeMounts:
+          - mountPath: /etc/tls/private
+            name: longhorn-ui-tls
+      {{- end }}
+      {{- end }}
       - name: longhorn-ui
         image: {{ template "registry_url" . }}{{ .Values.image.longhorn.ui.repository }}:{{ .Values.image.longhorn.ui.tag }}
         imagePullPolicy: {{ .Values.image.pullPolicy }}
@@ -47,6 +108,13 @@ spec:
           - name: LONGHORN_UI_PORT
             value: "8000"
       volumes:
+      {{- if .Values.openshift.enabled }}
+      {{- if .Values.openshift.ui.route }}
+      - name: longhorn-ui-tls
+        secret:
+          secretName: longhorn-ui-tls
+      {{- end }}
+      {{- end }}
       - emptyDir: {}
         name: nginx-cache
       - emptyDir: {}

chart/templates/preupgrade-job.yaml

@@ -1,3 +1,4 @@
+{{- if .Values.helmPreUpgradeCheckerJob.enabled }}
 apiVersion: batch/v1
 kind: Job
 metadata:
@@ -54,3 +55,4 @@ spec:
 {{ toYaml .Values.longhornManager.nodeSelector | indent 8 }}
         {{- end }}
       {{- end }}
+{{- end }}

chart/templates/serviceaccount.yaml

@@ -11,6 +11,25 @@ metadata:
 ---
 apiVersion: v1
 kind: ServiceAccount
+metadata:
+  name: longhorn-ui-service-account
+  namespace: {{ include "release_namespace" . }}
+  labels: {{- include "longhorn.labels" . | nindent 4 }}
+  {{- with .Values.serviceAccount.annotations }}
+  annotations:
+    {{- toYaml . | nindent 4 }}
+  {{- end }}
+  {{- if .Values.openshift.enabled }}
+  {{- if .Values.openshift.ui.route }}
+  {{- if not .Values.serviceAccount.annotations }}
+  annotations:
+  {{- end }}
+    serviceaccounts.openshift.io/oauth-redirectreference.primary: '{"kind":"OAuthRedirectReference","apiVersion":"v1","reference":{"kind":"Route","name":"longhorn-ui"}}'
+  {{- end }}
+  {{- end }}
+---
+apiVersion: v1
+kind: ServiceAccount
 metadata:
   name: longhorn-support-bundle
   namespace: {{ include "release_namespace" . }}

chart/values.yaml

@@ -3,181 +3,350 @@
 # Declare variables to be passed into your templates.
 global:
   cattle:
+    # -- System default registry
     systemDefaultRegistry: ""
     windowsCluster:
-      # Enable this to allow Longhorn to run on the Rancher deployed Windows cluster
+      # -- Enable this to allow Longhorn to run on the Rancher deployed Windows cluster
       enabled: false
-      # Tolerate Linux node taint
+      # -- Tolerate Linux nodes to run Longhorn user deployed components
       tolerations:
       - key: "cattle.io/os"
         value: "linux"
         effect: "NoSchedule"
         operator: "Equal"
-      # Select Linux nodes
+      # -- Select Linux nodes to run Longhorn user deployed components
       nodeSelector:
         kubernetes.io/os: "linux"
-      # Recognize toleration and node selector for Longhorn run-time created components
       defaultSetting:
+        # -- Toleration for Longhorn system managed components
         taintToleration: cattle.io/os=linux:NoSchedule
+        # -- Node selector for Longhorn system managed components
         systemManagedComponentsNodeSelector: kubernetes.io/os:linux
 
 networkPolicies:
+  # -- Enable NetworkPolicies to limit access to the Longhorn pods
   enabled: false
-  # Available types: k3s, rke2, rke1
+  # -- Create the policy based on your distribution to allow access for the ingress. Options: `k3s`, `rke2`, `rke1`
   type: "k3s"
 
 image:
   longhorn:
     engine:
+      # -- Specify Longhorn engine image repository
       repository: longhornio/longhorn-engine
+      # -- Specify Longhorn engine image tag
       tag: master-head
     manager:
+      # -- Specify Longhorn manager image repository
       repository: longhornio/longhorn-manager
+      # -- Specify Longhorn manager image tag
       tag: master-head
     ui:
+      # -- Specify Longhorn ui image repository
       repository: longhornio/longhorn-ui
+      # -- Specify Longhorn ui image tag
       tag: master-head
     instanceManager:
+      # -- Specify Longhorn instance manager image repository
       repository: longhornio/longhorn-instance-manager
+      # -- Specify Longhorn instance manager image tag
       tag: master-head
     shareManager:
+      # -- Specify Longhorn share manager image repository
       repository: longhornio/longhorn-share-manager
+      # -- Specify Longhorn share manager image tag
       tag: master-head
     backingImageManager:
+      # -- Specify Longhorn backing image manager image repository
       repository: longhornio/backing-image-manager
+      # -- Specify Longhorn backing image manager image tag 
       tag: master-head
     supportBundleKit:
+      # -- Specify Longhorn support bundle manager image repository
       repository: longhornio/support-bundle-kit
-      tag: v0.0.24
+      # -- Specify Longhorn support bundle manager image tag
+      tag: v0.0.27
   csi:
     attacher:
+      # -- Specify CSI attacher image repository. Leave blank to autodetect
       repository: longhornio/csi-attacher
+      # -- Specify CSI attacher image tag. Leave blank to autodetect
       tag: v4.2.0
     provisioner:
+      # -- Specify CSI provisioner image repository. Leave blank to autodetect
       repository: longhornio/csi-provisioner
+      # -- Specify CSI provisioner image tag. Leave blank to autodetect
       tag: v3.4.1
     nodeDriverRegistrar:
+      # -- Specify CSI node driver registrar image repository. Leave blank to autodetect
       repository: longhornio/csi-node-driver-registrar
+      # -- Specify CSI node driver registrar image tag. Leave blank to autodetect
       tag: v2.7.0
     resizer:
+      # -- Specify CSI driver resizer image repository. Leave blank to autodetect
       repository: longhornio/csi-resizer
+      # -- Specify CSI driver resizer image tag. Leave blank to autodetect
       tag: v1.7.0
     snapshotter:
+      # -- Specify CSI driver snapshotter image repository. Leave blank to autodetect
       repository: longhornio/csi-snapshotter
+      # -- Specify CSI driver snapshotter image tag. Leave blank to autodetect.
       tag: v6.2.1
     livenessProbe:
+      # -- Specify CSI liveness probe image repository. Leave blank to autodetect 
       repository: longhornio/livenessprobe
+      # -- Specify CSI liveness probe image tag. Leave blank to autodetect
       tag: v2.9.0
+  openshift:
+    oauthProxy:
+      # -- For openshift user. Specify oauth proxy image repository
+      repository: quay.io/openshift/origin-oauth-proxy
+      # -- For openshift user. Specify oauth proxy image tag. Note: Use your OCP/OKD 4.X Version, Current Stable is 4.14
+      tag: 4.14
+  # -- Image pull policy which applies to all user deployed Longhorn Components. e.g, Longhorn manager, Longhorn driver, Longhorn UI
   pullPolicy: IfNotPresent
 
 service:
   ui:
+    # -- Define Longhorn UI service type. Options: `ClusterIP`, `NodePort`, `LoadBalancer`, `Rancher-Proxy`
     type: ClusterIP
+    # -- NodePort port number (to set explicitly, choose port between 30000-32767)
     nodePort: null
   manager:
+    # -- Define Longhorn manager service type.
     type: ClusterIP
+    # -- NodePort port number (to set explicitly, choose port between 30000-32767)
     nodePort: ""
-    loadBalancerIP: ""
-    loadBalancerSourceRanges: ""
 
 persistence:
+  # -- Set Longhorn StorageClass as default
   defaultClass: true
+  # -- Set filesystem type for Longhorn StorageClass
   defaultFsType: ext4
+  # -- Set mkfs options for Longhorn StorageClass
   defaultMkfsParams: ""
+  # -- Set replica count for Longhorn StorageClass
   defaultClassReplicaCount: 3
-  defaultDataLocality: disabled # best-effort otherwise
+  # -- Set data locality for Longhorn StorageClass. Options: `disabled`, `best-effort`
+  defaultDataLocality: disabled
+  # -- Define reclaim policy. Options: `Retain`, `Delete`
   reclaimPolicy: Delete
+  # -- Set volume migratable for Longhorn StorageClass
   migratable: false
   recurringJobSelector:
+    # -- Enable recurring job selector for Longhorn StorageClass
     enable: false
+    # -- Recurring job selector list for Longhorn StorageClass. Please be careful of quotes of input. e.g., `[{"name":"backup", "isGroup":true}]`
     jobList: []
   backingImage:
+    # -- Set backing image for Longhorn StorageClass
     enable: false
+    # -- Specify a backing image that will be used by Longhorn volumes in Longhorn StorageClass. If not exists, the backing image data source type and backing image data source parameters should be specified so that Longhorn will create the backing image before using it
     name: ~
+    # -- Specify the data source type for the backing image used in Longhorn StorageClass.
+    # If the backing image does not exists, Longhorn will use this field to create a backing image. Otherwise, Longhorn will use it to verify the selected backing image.
     dataSourceType: ~
+    # -- Specify the data source parameters for the backing image used in Longhorn StorageClass. This option accepts a json string of a map. e.g., `'{\"url\":\"https://backing-image-example.s3-region.amazonaws.com/test-backing-image\"}'`.
     dataSourceParameters: ~
+    # -- Specify the expected SHA512 checksum of the selected backing image in Longhorn StorageClass
     expectedChecksum: ~
   defaultNodeSelector:
-    enable: false # disable by default
+    # -- Enable Node selector for Longhorn StorageClass
+    enable: false
+    # -- This selector enables only certain nodes having these tags to be used for the volume. e.g. `"storage,fast"`
     selector: ""
-  removeSnapshotsDuringFilesystemTrim: ignored # "enabled" or "disabled" otherwise
+  # -- Allow automatically removing snapshots during filesystem trim for Longhorn StorageClass. Options: `ignored`, `enabled`, `disabled`
+  removeSnapshotsDuringFilesystemTrim: ignored
+
+helmPreUpgradeCheckerJob:
+  enabled: true
 
 csi:
+  # -- Specify kubelet root-dir. Leave blank to autodetect
   kubeletRootDir: ~
+  # -- Specify replica count of CSI Attacher. Leave blank to use default count: 3
   attacherReplicaCount: ~
+  # -- Specify replica count of CSI Provisioner. Leave blank to use default count: 3
   provisionerReplicaCount: ~
+  # -- Specify replica count of CSI Resizer. Leave blank to use default count: 3
   resizerReplicaCount: ~
+  # -- Specify replica count of CSI Snapshotter. Leave blank to use default count: 3
   snapshotterReplicaCount: ~
 
 defaultSettings:
+  # -- The endpoint used to access the backupstore. Available: NFS, CIFS, AWS, GCP, AZURE.
   backupTarget: ~
+  # -- The name of the Kubernetes secret associated with the backup target.
   backupTargetCredentialSecret: ~
+  # -- If this setting is enabled, Longhorn will automatically attaches the volume and takes snapshot/backup 
+  # when it is the time to do recurring snapshot/backup.
   allowRecurringJobWhileVolumeDetached: ~
+  # -- Create default Disk automatically only on Nodes with the label "node.longhorn.io/create-default-disk=true" if no other disks exist.
+  # If disabled, the default disk will be created on all new nodes when each node is first added.
   createDefaultDiskLabeledNodes: ~
+  # -- Default path to use for storing data on a host. By default "/var/lib/longhorn/"
   defaultDataPath: ~
+  # -- Longhorn volume has data locality if there is a local replica of the volume on the same node as the pod which is using the volume.
   defaultDataLocality: ~
+  # -- Allow scheduling on nodes with existing healthy replicas of the same volume. By default false.
   replicaSoftAntiAffinity: ~
+  # -- Enable this setting automatically rebalances replicas when discovered an available node.
   replicaAutoBalance: ~
+  # -- The over-provisioning percentage defines how much storage can be allocated relative to the hard drive's capacity. By default 200.
   storageOverProvisioningPercentage: ~
+  # -- If the minimum available disk capacity exceeds the actual percentage of available disk capacity,
+  # the disk becomes unschedulable until more space is freed up. By default 25.
   storageMinimalAvailablePercentage: ~
+  # -- The reserved percentage specifies the percentage of disk space that will not be allocated to the default disk on each new Longhorn node.
+  storageReservedPercentageForDefaultDisk: ~
+  # -- Upgrade Checker will check for new Longhorn version periodically.
+  # When there is a new version available, a notification will appear in the UI. By default true.
   upgradeChecker: ~
+  # -- The default number of replicas when a volume is created from the Longhorn UI.
+  # For Kubernetes configuration, update the `numberOfReplicas` in the StorageClass. By default 3.
   defaultReplicaCount: ~
+  # -- The 'storageClassName' is given to PVs and PVCs that are created for an existing Longhorn volume. The StorageClass name can also be used as a label,
+  # so it is possible to use a Longhorn StorageClass to bind a workload to an existing PV without creating a Kubernetes StorageClass object.
+  # By default 'longhorn-static'.
   defaultLonghornStaticStorageClass: ~
+  # -- In seconds. The backupstore poll interval determines how often Longhorn checks the backupstore for new backups.
+  # Set to 0 to disable the polling. By default 300.
   backupstorePollInterval: ~
+  # -- In minutes. This setting determines how long Longhorn will keep the backup resource that was failed. Set to 0 to disable the auto-deletion.
   failedBackupTTL: ~
+  # -- Restore recurring jobs from the backup volume on the backup target and create recurring jobs if not exist during a backup restoration.
   restoreVolumeRecurringJobs: ~
+  # -- This setting specifies how many successful backup or snapshot job histories should be retained. History will not be retained if the value is 0.
   recurringSuccessfulJobsHistoryLimit: ~
+  # -- This setting specifies how many failed backup or snapshot job histories should be retained. History will not be retained if the value is 0.
   recurringFailedJobsHistoryLimit: ~
+  # -- This setting specifies how many failed support bundles can exist in the cluster.
+  # Set this value to **0** to have Longhorn automatically purge all failed support bundles.
   supportBundleFailedHistoryLimit: ~
+  # -- taintToleration for longhorn system components
   taintToleration: ~
+  # -- nodeSelector for longhorn system components
   systemManagedComponentsNodeSelector: ~
+  # -- priorityClass for longhorn system componentss
   priorityClass: ~
+  # -- If enabled, volumes will be automatically salvaged when all the replicas become faulty e.g. due to network disconnection.
+  # Longhorn will try to figure out which replica(s) are usable, then use them for the volume. By default true.
   autoSalvage: ~
+  # -- If enabled, Longhorn will automatically delete the workload pod that is managed by a controller (e.g. deployment, statefulset, daemonset, etc...) 
+  # when Longhorn volume is detached unexpectedly (e.g. during Kubernetes upgrade, Docker reboot, or network disconnect).
+  # By deleting the pod, its controller restarts the pod and Kubernetes handles volume reattachment and remount.
   autoDeletePodWhenVolumeDetachedUnexpectedly: ~
+  # -- Disable Longhorn manager to schedule replica on Kubernetes cordoned node. By default true.
   disableSchedulingOnCordonedNode: ~
+  # -- Allow scheduling new Replicas of Volume to the Nodes in the same Zone as existing healthy Replicas.
+  # Nodes don't belong to any Zone will be treated as in the same Zone.
+  # Notice that Longhorn relies on label `topology.kubernetes.io/zone=<Zone name of the node>` in the Kubernetes node object to identify the zone.
+  # By default true.
   replicaZoneSoftAntiAffinity: ~
+  # -- Allow scheduling on disks with existing healthy replicas of the same volume. By default true.
+  replicaDiskSoftAntiAffinity: ~
+  # -- Defines the Longhorn action when a Volume is stuck with a StatefulSet/Deployment Pod on a node that is down.
   nodeDownPodDeletionPolicy: ~
-  nodeDrainPolicy : ~
+  # -- Define the policy to use when a node with the last healthy replica of a volume is drained.
+  nodeDrainPolicy: ~
+  # -- In seconds. The interval determines how long Longhorn will wait at least in order to reuse the existing data on a failed replica
+  # rather than directly creating a new replica for a degraded volume.
   replicaReplenishmentWaitInterval: ~
+  # -- This setting controls how many replicas on a node can be rebuilt simultaneously.
   concurrentReplicaRebuildPerNodeLimit: ~
+  # -- This setting controls how many volumes on a node can restore the backup concurrently. Set the value to **0** to disable backup restore.
   concurrentVolumeBackupRestorePerNodeLimit: ~
+  # -- This setting is only for volumes created by UI.
+  # By default, this is false meaning there will be a reivision counter file to track every write to the volume.
+  # During salvage recovering Longhorn will pick the replica with largest reivision counter as candidate to recover the whole volume.
+  # If revision counter is disabled, Longhorn will not track every write to the volume.
+  # During the salvage recovering, Longhorn will use the 'volume-head-xxx.img' file last modification time and
+  # file size to pick the replica candidate to recover the whole volume.
   disableRevisionCounter: ~
+  # -- This setting defines the Image Pull Policy of Longhorn system managed pod.
+  # e.g. instance manager, engine image, CSI driver, etc.
+  # The new Image Pull Policy will only apply after the system managed pods restart.
   systemManagedPodsImagePullPolicy: ~
+  # -- This setting allows user to create and attach a volume that doesn't have all the replicas scheduled at the time of creation.
   allowVolumeCreationWithDegradedAvailability: ~
+  # -- This setting enables Longhorn to automatically cleanup the system generated snapshot after replica rebuild is done.
   autoCleanupSystemGeneratedSnapshot: ~
+  # -- This setting controls how Longhorn automatically upgrades volumes' engines to the new default engine image after upgrading Longhorn manager.
+  # The value of this setting specifies the maximum number of engines per node that are allowed to upgrade to the default engine image at the same time.
+  # If the value is 0, Longhorn will not automatically upgrade volumes' engines to default version.
   concurrentAutomaticEngineUpgradePerNodeLimit: ~
+  # -- This interval in minutes determines how long Longhorn will wait before cleaning up the backing image file when there is no replica in the disk using it.
   backingImageCleanupWaitInterval: ~
+  # -- This interval in seconds determines how long Longhorn will wait before re-downloading the backing image file
+  # when all disk files of this backing image become failed or unknown.
   backingImageRecoveryWaitInterval: ~
-  guaranteedEngineManagerCPU: ~
-  guaranteedReplicaManagerCPU: ~
+  # -- This integer value indicates how many percentage of the total allocatable CPU on each node will be reserved for each instance manager Pod.
+  # You can leave it with the default value, which is 12%.
+  guaranteedInstanceManagerCPU: ~
+  # -- Enabling this setting will notify Longhorn that the cluster is using Kubernetes Cluster Autoscaler.
   kubernetesClusterAutoscalerEnabled: ~
+  # -- This setting allows Longhorn to delete the orphan resource and its corresponding orphaned data automatically like stale replicas.
+  # Orphan resources on down or unknown nodes will not be cleaned up automatically.
   orphanAutoDeletion: ~
+  # -- Longhorn uses the storage network for in-cluster data traffic. Leave this blank to use the Kubernetes cluster network.
   storageNetwork: ~
+  # -- This flag is designed to prevent Longhorn from being accidentally uninstalled which will lead to data lost.
   deletingConfirmationFlag: ~
+  # -- In seconds. The setting specifies the timeout between the engine and replica(s), and the value should be between 8 to 30 seconds.
+  # The default value is 8 seconds.
   engineReplicaTimeout: ~
+  # -- This setting allows users to enable or disable snapshot hashing and data integrity checking.
   snapshotDataIntegrity: ~
+  # -- Hashing snapshot disk files impacts the performance of the system.
+  # The immediate snapshot hashing and checking can be disabled to minimize the impact after creating a snapshot.
   snapshotDataIntegrityImmediateCheckAfterSnapshotCreation: ~
+  # -- Unix-cron string format. The setting specifies when Longhorn checks the data integrity of snapshot disk files.
   snapshotDataIntegrityCronjob: ~
+  # -- This setting allows Longhorn filesystem trim feature to automatically mark the latest snapshot and
+  # its ancestors as removed and stops at the snapshot containing multiple children.
   removeSnapshotsDuringFilesystemTrim: ~
+  # -- This feature supports the fast replica rebuilding.
+  # It relies on the checksum of snapshot disk files, so setting the snapshot-data-integrity to **enable** or **fast-check** is a prerequisite.
   fastReplicaRebuildEnabled: ~
+  # -- In seconds. The setting specifies the HTTP client timeout to the file sync server.
   replicaFileSyncHttpClientTimeout: ~
+  # -- The log level Panic, Fatal, Error, Warn, Info, Debug, Trace used in longhorn manager. Default to Info.
   logLevel: ~
+  # -- This setting allows users to specify backup compression method.
   backupCompressionMethod: ~
+  # -- This setting controls how many worker threads per backup concurrently.
   backupConcurrentLimit: ~
+  # -- This setting controls how many worker threads per restore concurrently.
   restoreConcurrentLimit: ~
-  spdk: ~
+  # -- This allows users to activate v2 data engine based on SPDK.
+  # Currently, it is in the preview phase and should not be utilized in a production environment.
+  v2DataEngine: ~
+  # -- This setting allows users to enable the offline replica rebuilding for volumes using v2 data engine.
+  offlineReplicaRebuilding: ~
+  # -- Allow Scheduling Empty Node Selector Volumes To Any Node
+  allowEmptyNodeSelectorVolume: ~
+  # -- Allow Scheduling Empty Disk Selector Volumes To Any Disk
+  allowEmptyDiskSelectorVolume: ~
+
 privateRegistry:
+  # -- Set `true` to create a new private registry secret
   createSecret: ~
+  # -- URL of private registry. Leave blank to apply system default registry
   registryUrl: ~
+  # -- User used to authenticate to private registry
   registryUser: ~
+  # -- Password used to authenticate to private registry
   registryPasswd: ~
+  # -- If create a new private registry secret is true, create a Kubernetes secret with this name; else use the existing secret of this name. Use it to pull images from your private registry
   registrySecret: ~
 
 longhornManager:
   log:
-    ## Allowed values are `plain` or `json`.
+    # -- Options: `plain`, `json`
     format: plain
+  # -- Priority class for longhorn manager
   priorityClass: ~
+  # -- Tolerate nodes to run Longhorn manager
   tolerations: []
   ## If you want to set tolerations for Longhorn Manager DaemonSet, delete the `[]` in the line above
   ## and uncomment this example block
@@ -185,11 +354,13 @@ longhornManager:
   #   operator: "Equal"
   #   value: "value"
   #   effect: "NoSchedule"
+  # -- Select nodes to run Longhorn manager
   nodeSelector: {}
   ## If you want to set node selector for Longhorn Manager DaemonSet, delete the `{}` in the line above
   ## and uncomment this example block
   #  label-key1: "label-value1"
   #  label-key2: "label-value2"
+  # -- Annotation used in Longhorn manager service
   serviceAnnotations: {}
   ## If you want to set annotations for the Longhorn Manager service, delete the `{}` in the line above
   ## and uncomment this example block
@@ -197,7 +368,9 @@ longhornManager:
   #  annotation-key2: "annotation-value2"
 
 longhornDriver:
+  # -- Priority class for longhorn driver
   priorityClass: ~
+  # -- Tolerate nodes to run Longhorn driver
   tolerations: []
   ## If you want to set tolerations for Longhorn Driver Deployer Deployment, delete the `[]` in the line above
   ## and uncomment this example block
@@ -205,6 +378,7 @@ longhornDriver:
   #   operator: "Equal"
   #   value: "value"
   #   effect: "NoSchedule"
+  # -- Select nodes to run Longhorn driver
   nodeSelector: {}
   ## If you want to set node selector for Longhorn Driver Deployer Deployment, delete the `{}` in the line above
   ## and uncomment this example block
@@ -212,8 +386,11 @@ longhornDriver:
   #  label-key2: "label-value2"
 
 longhornUI:
+  # -- Replica count for longhorn ui
   replicas: 2
+  # -- Priority class count for longhorn ui
   priorityClass: ~
+  # -- Tolerate nodes to run Longhorn UI
   tolerations: []
   ## If you want to set tolerations for Longhorn UI Deployment, delete the `[]` in the line above
   ## and uncomment this example block
@@ -221,84 +398,37 @@ longhornUI:
   #   operator: "Equal"
   #   value: "value"
   #   effect: "NoSchedule"
+  # -- Select nodes to run Longhorn UI
   nodeSelector: {}
   ## If you want to set node selector for Longhorn UI Deployment, delete the `{}` in the line above
   ## and uncomment this example block
   #  label-key1: "label-value1"
   #  label-key2: "label-value2"
 
-longhornConversionWebhook:
-  replicas: 2
-  priorityClass: ~
-  tolerations: []
-  ## If you want to set tolerations for Longhorn conversion webhook Deployment, delete the `[]` in the line above
-  ## and uncomment this example block
-  # - key: "key"
-  #   operator: "Equal"
-  #   value: "value"
-  #   effect: "NoSchedule"
-  nodeSelector: {}
-  ## If you want to set node selector for Longhorn conversion webhook Deployment, delete the `{}` in the line above
-  ## and uncomment this example block
-  #  label-key1: "label-value1"
-  #  label-key2: "label-value2"
-
-longhornAdmissionWebhook:
-  replicas: 2
-  priorityClass: ~
-  tolerations: []
-  ## If you want to set tolerations for Longhorn admission webhook Deployment, delete the `[]` in the line above
-  ## and uncomment this example block
-  # - key: "key"
-  #   operator: "Equal"
-  #   value: "value"
-  #   effect: "NoSchedule"
-  nodeSelector: {}
-  ## If you want to set node selector for Longhorn admission webhook Deployment, delete the `{}` in the line above
-  ## and uncomment this example block
-  #  label-key1: "label-value1"
-  #  label-key2: "label-value2"
-
-longhornRecoveryBackend:
-  replicas: 2
-  priorityClass: ~
-  tolerations: []
-  ## If you want to set tolerations for Longhorn recovery backend Deployment, delete the `[]` in the line above
-  ## and uncomment this example block
-  # - key: "key"
-  #   operator: "Equal"
-  #   value: "value"
-  #   effect: "NoSchedule"
-  nodeSelector: {}
-  ## If you want to set node selector for Longhorn recovery backend Deployment, delete the `{}` in the line above
-  ## and uncomment this example block
-  #  label-key1: "label-value1"
-  #  label-key2: "label-value2"
-
 ingress:
-  ## Set to true to enable ingress record generation
+  # -- Set to true to enable ingress record generation
   enabled: false
 
-  ## Add ingressClassName to the Ingress
-  ## Can replace the kubernetes.io/ingress.class annotation on v1.18+
+  # -- Add ingressClassName to the Ingress
+  # Can replace the kubernetes.io/ingress.class annotation on v1.18+
   ingressClassName: ~
 
+  # -- Layer 7 Load Balancer hostname
   host: sslip.io
 
-  ## Set this to true in order to enable TLS on the ingress record
+  # -- Set this to true in order to enable TLS on the ingress record
   tls: false
 
-  ## Enable this in order to enable that the backend service will be connected at port 443
+  # -- Enable this in order to enable that the backend service will be connected at port 443
   secureBackends: false
 
-  ## If TLS is set to true, you must declare what secret will store the key/certificate for TLS
+  # -- If TLS is set to true, you must declare what secret will store the key/certificate for TLS
   tlsSecret: longhorn.local-tls
 
-  ## If ingress is enabled you can set the default ingress path
-  ## then you can access the UI by using the following full path {{host}}+{{path}}
+  # -- If ingress is enabled you can set the default ingress path
+  # then you can access the UI by using the following full path {{host}}+{{path}}
   path: /
 
-  ## Ingress annotations done as key:value pairs
   ## If you're using kube-lego, you will want to add:
   ## kubernetes.io/tls-acme: true
   ##
@@ -306,10 +436,12 @@ ingress:
   ## ref: https://github.com/kubernetes/ingress-nginx/blob/master/docs/annotations.md
   ##
   ## If tls is set to true, annotation ingress.kubernetes.io/secure-backends: "true" will automatically be set
+  # -- Ingress annotations done as key:value pairs
   annotations:
   #  kubernetes.io/ingress.class: nginx
   #  kubernetes.io/tls-acme: true
 
+  # -- If you're providing your own certificates, please use this to add the certificates as secrets
   secrets:
   ## If you're providing your own certificates, please use this to add the certificates as secrets
   ## key and certificate should start with -----BEGIN CERTIFICATE----- or
@@ -324,17 +456,25 @@ ingress:
   #   key:
   #   certificate:
 
-#  For Kubernetes < v1.25, if your cluster enables Pod Security Policy admission controller,
-#  set this to `true` to ship longhorn-psp which allow privileged Longhorn pods to start
+# -- For Kubernetes < v1.25, if your cluster enables Pod Security Policy admission controller,
+# set this to `true` to ship longhorn-psp which allow privileged Longhorn pods to start
 enablePSP: false
 
-## Specify override namespace, specifically this is useful for using longhorn as sub-chart
-## and its release namespace is not the `longhorn-system`
-namespaceOverride: ""
-
-# Annotations to add to the Longhorn Manager DaemonSet Pods. Optional.
+# -- Annotations to add to the Longhorn Manager DaemonSet Pods. Optional.
 annotations: {}
 
 serviceAccount:
-  # Annotations to add to the service account
+  # -- Annotations to add to the service account
   annotations: {}
+
+## openshift settings
+openshift:
+  # -- Enable when using openshift
+  enabled: false
+  ui:
+    # -- UI route in openshift environment
+    route: "longhorn-ui"
+    # -- UI port in openshift environment
+    port: 443
+    # -- UI proxy in openshift environment
+    proxy: 8443

deploy/backupstores/cifs-backupstore.yaml

@@ -0,0 +1,87 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: cifs-secret
+  namespace: longhorn-system
+type: Opaque
+data:
+  CIFS_USERNAME: bG9uZ2hvcm4tY2lmcy11c2VybmFtZQ== # longhorn-cifs-username
+  CIFS_PASSWORD: bG9uZ2hvcm4tY2lmcy1wYXNzd29yZA== # longhorn-cifs-password
+---
+apiVersion: v1
+kind: Secret
+metadata:
+  name: cifs-secret
+  namespace: default
+type: Opaque
+data:
+  CIFS_USERNAME: bG9uZ2hvcm4tY2lmcy11c2VybmFtZQ== # longhorn-cifs-username
+  CIFS_PASSWORD: bG9uZ2hvcm4tY2lmcy1wYXNzd29yZA== # longhorn-cifs-password
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: longhorn-test-cifs
+  namespace: default
+  labels:
+    app: longhorn-test-cifs
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: longhorn-test-cifs
+  template:
+    metadata:
+      labels:
+        app: longhorn-test-cifs
+    spec:
+      volumes:
+      - name: cifs-volume
+        emptyDir: {}
+      containers:
+      - name: longhorn-test-cifs-container
+        image: derekbit/samba:latest
+        ports:
+        - containerPort: 139
+        - containerPort: 445
+        imagePullPolicy: Always
+        env:
+        - name: EXPORT_PATH
+          value: /opt/backupstore
+        - name: CIFS_DISK_IMAGE_SIZE_MB
+          value: "4096"
+        - name: CIFS_USERNAME
+          valueFrom:
+            secretKeyRef:
+              name: cifs-secret
+              key: CIFS_USERNAME
+        - name: CIFS_PASSWORD
+          valueFrom:
+            secretKeyRef:
+              name: cifs-secret
+              key: CIFS_PASSWORD
+        securityContext:
+          privileged: true
+          capabilities:
+            add: ["SYS_ADMIN", "DAC_READ_SEARCH"]
+        volumeMounts:
+        - name: cifs-volume
+          mountPath: "/opt/backupstore"
+        args: ["-u", "$(CIFS_USERNAME);$(CIFS_PASSWORD)", "-s", "backupstore;$(EXPORT_PATH);yes;no;no;all;none"]
+---
+kind: Service
+apiVersion: v1
+metadata:
+  name: longhorn-test-cifs-svc
+  namespace: default
+spec:
+  selector:
+    app: longhorn-test-cifs
+  clusterIP: None
+  ports:
+  - name: netbios-port
+    port: 139
+    targetPort: 139
+  - name: microsoft-port
+    port: 445
+    targetPort: 445

deploy/backupstores/nfs-backupstore.yaml

@@ -6,7 +6,6 @@ metadata:
   labels:
     app: longhorn-test-nfs
 spec:
-  spec:
   selector:
     matchLabels:
       app: longhorn-test-nfs

deploy/longhorn-images.txt

@@ -10,4 +10,4 @@ longhornio/longhorn-instance-manager:master-head
 longhornio/longhorn-manager:master-head
 longhornio/longhorn-share-manager:master-head
 longhornio/longhorn-ui:master-head
-longhornio/support-bundle-kit:v0.0.24
+longhornio/support-bundle-kit:v0.0.27

deploy/longhorn.yaml

@@ -14,7 +14,18 @@ metadata:
   labels:
     app.kubernetes.io/name: longhorn
     app.kubernetes.io/instance: longhorn
-    app.kubernetes.io/version: v1.4.0-dev
+    app.kubernetes.io/version: v1.6.0-dev
+---
+# Source: longhorn/templates/serviceaccount.yaml
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: longhorn-ui-service-account
+  namespace: longhorn-system
+  labels:
+    app.kubernetes.io/name: longhorn
+    app.kubernetes.io/instance: longhorn
+    app.kubernetes.io/version: v1.6.0-dev
 ---
 # Source: longhorn/templates/serviceaccount.yaml
 apiVersion: v1
@@ -25,7 +36,7 @@ metadata:
   labels:
     app.kubernetes.io/name: longhorn
     app.kubernetes.io/instance: longhorn
-    app.kubernetes.io/version: v1.4.0-dev
+    app.kubernetes.io/version: v1.6.0-dev
 ---
 # Source: longhorn/templates/default-setting.yaml
 apiVersion: v1
@@ -36,7 +47,7 @@ metadata:
   labels:
     app.kubernetes.io/name: longhorn
     app.kubernetes.io/instance: longhorn
-    app.kubernetes.io/version: v1.4.0-dev
+    app.kubernetes.io/version: v1.6.0-dev
 data:
   default-setting.yaml: |-
 ---
@@ -49,7 +60,7 @@ metadata:
   labels:
     app.kubernetes.io/name: longhorn
     app.kubernetes.io/instance: longhorn
-    app.kubernetes.io/version: v1.4.0-dev
+    app.kubernetes.io/version: v1.6.0-dev
 data:
   storageclass.yaml: |
     kind: StorageClass
@@ -79,7 +90,7 @@ metadata:
   labels:
     app.kubernetes.io/name: longhorn
     app.kubernetes.io/instance: longhorn
-    app.kubernetes.io/version: v1.4.0-dev
+    app.kubernetes.io/version: v1.6.0-dev
     longhorn-manager: ""
   name: backingimagedatasources.longhorn.io
 spec:
@@ -250,7 +261,7 @@ metadata:
   labels:
     app.kubernetes.io/name: longhorn
     app.kubernetes.io/instance: longhorn
-    app.kubernetes.io/version: v1.4.0-dev
+    app.kubernetes.io/version: v1.6.0-dev
     longhorn-manager: ""
   name: backingimagemanagers.longhorn.io
 spec:
@@ -426,7 +437,7 @@ metadata:
   labels:
     app.kubernetes.io/name: longhorn
     app.kubernetes.io/instance: longhorn
-    app.kubernetes.io/version: v1.4.0-dev
+    app.kubernetes.io/version: v1.6.0-dev
     longhorn-manager: ""
   name: backingimages.longhorn.io
 spec:
@@ -585,7 +596,7 @@ metadata:
   labels:
     app.kubernetes.io/name: longhorn
     app.kubernetes.io/instance: longhorn
-    app.kubernetes.io/version: v1.4.0-dev
+    app.kubernetes.io/version: v1.6.0-dev
     longhorn-manager: ""
   name: backups.longhorn.io
 spec:
@@ -781,7 +792,7 @@ metadata:
   labels:
     app.kubernetes.io/name: longhorn
     app.kubernetes.io/instance: longhorn
-    app.kubernetes.io/version: v1.4.0-dev
+    app.kubernetes.io/version: v1.6.0-dev
     longhorn-manager: ""
   name: backuptargets.longhorn.io
 spec:
@@ -964,7 +975,7 @@ metadata:
   labels:
     app.kubernetes.io/name: longhorn
     app.kubernetes.io/instance: longhorn
-    app.kubernetes.io/version: v1.4.0-dev
+    app.kubernetes.io/version: v1.6.0-dev
     longhorn-manager: ""
   name: backupvolumes.longhorn.io
 spec:
@@ -1131,7 +1142,7 @@ metadata:
   labels:
     app.kubernetes.io/name: longhorn
     app.kubernetes.io/instance: longhorn
-    app.kubernetes.io/version: v1.4.0-dev
+    app.kubernetes.io/version: v1.6.0-dev
     longhorn-manager: ""
   name: engineimages.longhorn.io
 spec:
@@ -1323,7 +1334,7 @@ metadata:
   labels:
     app.kubernetes.io/name: longhorn
     app.kubernetes.io/instance: longhorn
-    app.kubernetes.io/version: v1.4.0-dev
+    app.kubernetes.io/version: v1.6.0-dev
     longhorn-manager: ""
   name: engines.longhorn.io
 spec:
@@ -1419,8 +1430,8 @@ spec:
                 type: boolean
               backendStoreDriver:
                 enum:
-                - longhorn
-                - spdk
+                - v1
+                - v2
                 type: string
               backupVolume:
                 type: string
@@ -1429,6 +1440,7 @@ spec:
               disableFrontend:
                 type: boolean
               engineImage:
+                description: 'Deprecated: Replaced by field `image`.'
                 type: string
               frontend:
                 enum:
@@ -1437,6 +1449,8 @@ spec:
                 - nvmf
                 - ""
                 type: string
+              image:
+                type: string
               logRequested:
                 type: boolean
               nodeID:
@@ -1678,7 +1692,7 @@ metadata:
   labels:
     app.kubernetes.io/name: longhorn
     app.kubernetes.io/instance: longhorn
-    app.kubernetes.io/version: v1.4.0-dev
+    app.kubernetes.io/version: v1.6.0-dev
     longhorn-manager: ""
   name: instancemanagers.longhorn.io
 spec:
@@ -1919,7 +1933,7 @@ metadata:
   labels:
     app.kubernetes.io/name: longhorn
     app.kubernetes.io/instance: longhorn
-    app.kubernetes.io/version: v1.4.0-dev
+    app.kubernetes.io/version: v1.6.0-dev
     longhorn-manager: ""
   name: nodes.longhorn.io
 spec:
@@ -2163,7 +2177,7 @@ metadata:
   labels:
     app.kubernetes.io/name: longhorn
     app.kubernetes.io/instance: longhorn
-    app.kubernetes.io/version: v1.4.0-dev
+    app.kubernetes.io/version: v1.6.0-dev
     longhorn-manager: ""
   name: orphans.longhorn.io
 spec:
@@ -2434,7 +2448,7 @@ metadata:
   labels:
     app.kubernetes.io/name: longhorn
     app.kubernetes.io/instance: longhorn
-    app.kubernetes.io/version: v1.4.0-dev
+    app.kubernetes.io/version: v1.6.0-dev
     longhorn-manager: ""
   name: replicas.longhorn.io
 spec:
@@ -2538,8 +2552,8 @@ spec:
                 type: boolean
               backendStoreDriver:
                 enum:
-                - longhorn
-                - spdk
+                - v1
+                - v2
                 type: string
               backingImage:
                 type: string
@@ -2552,6 +2566,7 @@ spec:
               diskPath:
                 type: string
               engineImage:
+                description: 'Deprecated: Replaced by field `image`.'
                 type: string
               engineName:
                 type: string
@@ -2561,6 +2576,8 @@ spec:
                 type: string
               healthyAt:
                 type: string
+              image:
+                type: string
               logRequested:
                 type: boolean
               nodeID:
@@ -2651,7 +2668,7 @@ metadata:
   labels:
     app.kubernetes.io/name: longhorn
     app.kubernetes.io/instance: longhorn
-    app.kubernetes.io/version: v1.4.0-dev
+    app.kubernetes.io/version: v1.6.0-dev
     longhorn-manager: ""
   name: settings.longhorn.io
 spec:
@@ -2742,7 +2759,7 @@ metadata:
   labels:
     app.kubernetes.io/name: longhorn
     app.kubernetes.io/instance: longhorn
-    app.kubernetes.io/version: v1.4.0-dev
+    app.kubernetes.io/version: v1.6.0-dev
     longhorn-manager: ""
   name: sharemanagers.longhorn.io
 spec:
@@ -2857,7 +2874,7 @@ metadata:
   labels:
     app.kubernetes.io/name: longhorn
     app.kubernetes.io/instance: longhorn
-    app.kubernetes.io/version: v1.4.0-dev
+    app.kubernetes.io/version: v1.6.0-dev
     longhorn-manager: ""
   name: snapshots.longhorn.io
 spec:
@@ -2984,7 +3001,7 @@ metadata:
   labels:
     app.kubernetes.io/name: longhorn
     app.kubernetes.io/instance: longhorn
-    app.kubernetes.io/version: v1.4.0-dev
+    app.kubernetes.io/version: v1.6.0-dev
     longhorn-manager: ""
   name: supportbundles.longhorn.io
 spec:
@@ -3110,7 +3127,7 @@ metadata:
   labels:
     app.kubernetes.io/name: longhorn
     app.kubernetes.io/instance: longhorn
-    app.kubernetes.io/version: v1.4.0-dev
+    app.kubernetes.io/version: v1.6.0-dev
     longhorn-manager: ""
   name: systembackups.longhorn.io
 spec:
@@ -3238,7 +3255,7 @@ metadata:
   labels:
     app.kubernetes.io/name: longhorn
     app.kubernetes.io/instance: longhorn
-    app.kubernetes.io/version: v1.4.0-dev
+    app.kubernetes.io/version: v1.6.0-dev
     longhorn-manager: ""
   name: systemrestores.longhorn.io
 spec:
@@ -3340,7 +3357,7 @@ metadata:
   labels:
     app.kubernetes.io/name: longhorn
     app.kubernetes.io/instance: longhorn
-    app.kubernetes.io/version: v1.4.0-dev
+    app.kubernetes.io/version: v1.6.0-dev
     longhorn-manager: ""
   name: volumes.longhorn.io
 spec:
@@ -3462,8 +3479,8 @@ spec:
                 type: string
               backendStoreDriver:
                 enum:
-                - longhorn
-                - spdk
+                - v1
+                - v2
                 type: string
               backingImage:
                 type: string
@@ -3490,6 +3507,7 @@ spec:
               encrypted:
                 type: boolean
               engineImage:
+                description: 'Deprecated: Replaced by field `image`.'
                 type: string
               fromBackup:
                 type: string
@@ -3500,6 +3518,8 @@ spec:
                 - nvmf
                 - ""
                 type: string
+              image:
+                type: string
               lastAttachedBy:
                 type: string
               migratable:
@@ -3514,6 +3534,13 @@ spec:
                 type: array
               numberOfReplicas:
                 type: integer
+              offlineReplicaRebuilding:
+                description: OfflineReplicaRebuilding is used to determine if the offline replica rebuilding feature is enabled or not
+                enum:
+                - ignored
+                - disabled
+                - enabled
+                type: string
               replicaAutoBalance:
                 enum:
                 - ignored
@@ -3521,15 +3548,22 @@ spec:
                 - least-effort
                 - best-effort
                 type: string
+              replicaDiskSoftAntiAffinity:
+                description: Replica disk soft anti affinity of the volume. Set enabled to allow replicas to be scheduled in the same disk.
+                enum:
+                - ignored
+                - enabled
+                - disabled
+                type: string
               replicaSoftAntiAffinity:
-                description: Replica soft anti affinity of the volume. Set enabled to allow replicas to be scheduled on the same node
+                description: Replica soft anti affinity of the volume. Set enabled to allow replicas to be scheduled on the same node.
                 enum:
                 - ignored
                 - enabled
                 - disabled
                 type: string
               replicaZoneSoftAntiAffinity:
-                description: Replica zone soft anti affinity of the volume. Set enabled to allow replicas to be scheduled in the same zone
+                description: Replica zone soft anti affinity of the volume. Set enabled to allow replicas to be scheduled in the same zone.
                 enum:
                 - ignored
                 - enabled
@@ -3651,6 +3685,8 @@ spec:
                 type: string
               lastDegradedAt:
                 type: string
+              offlineReplicaRebuildingRequired:
+                type: boolean
               ownerID:
                 type: string
               pendingNodeID:
@@ -3693,7 +3729,7 @@ metadata:
   labels:
     app.kubernetes.io/name: longhorn
     app.kubernetes.io/instance: longhorn
-    app.kubernetes.io/version: v1.4.0-dev
+    app.kubernetes.io/version: v1.6.0-dev
     longhorn-manager: ""
   name: volumeattachments.longhorn.io
 spec:
@@ -3822,7 +3858,7 @@ metadata:
   labels:
     app.kubernetes.io/name: longhorn
     app.kubernetes.io/instance: longhorn
-    app.kubernetes.io/version: v1.4.0-dev
+    app.kubernetes.io/version: v1.6.0-dev
 rules:
 - apiGroups:
   - apiextensions.k8s.io
@@ -3888,7 +3924,7 @@ metadata:
   labels:
     app.kubernetes.io/name: longhorn
     app.kubernetes.io/instance: longhorn
-    app.kubernetes.io/version: v1.4.0-dev
+    app.kubernetes.io/version: v1.6.0-dev
 roleRef:
   apiGroup: rbac.authorization.k8s.io
   kind: ClusterRole
@@ -3906,7 +3942,7 @@ metadata:
   labels:
     app.kubernetes.io/name: longhorn
     app.kubernetes.io/instance: longhorn
-    app.kubernetes.io/version: v1.4.0-dev
+    app.kubernetes.io/version: v1.6.0-dev
 roleRef:
   apiGroup: rbac.authorization.k8s.io
   kind: ClusterRole
@@ -3923,7 +3959,7 @@ metadata:
   labels:
     app.kubernetes.io/name: longhorn
     app.kubernetes.io/instance: longhorn
-    app.kubernetes.io/version: v1.4.0-dev
+    app.kubernetes.io/version: v1.6.0-dev
     app: longhorn-manager
   name: longhorn-backend
   namespace: longhorn-system
@@ -3944,7 +3980,7 @@ metadata:
   labels:
     app.kubernetes.io/name: longhorn
     app.kubernetes.io/instance: longhorn
-    app.kubernetes.io/version: v1.4.0-dev
+    app.kubernetes.io/version: v1.6.0-dev
     app: longhorn-ui
   name: longhorn-frontend
   namespace: longhorn-system
@@ -3965,7 +4001,7 @@ metadata:
   labels:
     app.kubernetes.io/name: longhorn
     app.kubernetes.io/instance: longhorn
-    app.kubernetes.io/version: v1.4.0-dev
+    app.kubernetes.io/version: v1.6.0-dev
     app: longhorn-conversion-webhook
   name: longhorn-conversion-webhook
   namespace: longhorn-system
@@ -3986,7 +4022,7 @@ metadata:
   labels:
     app.kubernetes.io/name: longhorn
     app.kubernetes.io/instance: longhorn
-    app.kubernetes.io/version: v1.4.0-dev
+    app.kubernetes.io/version: v1.6.0-dev
     app: longhorn-admission-webhook
   name: longhorn-admission-webhook
   namespace: longhorn-system
@@ -4007,7 +4043,7 @@ metadata:
   labels:
     app.kubernetes.io/name: longhorn
     app.kubernetes.io/instance: longhorn
-    app.kubernetes.io/version: v1.4.0-dev
+    app.kubernetes.io/version: v1.6.0-dev
     app: longhorn-recovery-backend
   name: longhorn-recovery-backend
   namespace: longhorn-system
@@ -4028,7 +4064,7 @@ metadata:
   labels:
     app.kubernetes.io/name: longhorn
     app.kubernetes.io/instance: longhorn
-    app.kubernetes.io/version: v1.4.0-dev
+    app.kubernetes.io/version: v1.6.0-dev
   name: longhorn-engine-manager
   namespace: longhorn-system
 spec:
@@ -4044,7 +4080,7 @@ metadata:
   labels:
     app.kubernetes.io/name: longhorn
     app.kubernetes.io/instance: longhorn
-    app.kubernetes.io/version: v1.4.0-dev
+    app.kubernetes.io/version: v1.6.0-dev
   name: longhorn-replica-manager
   namespace: longhorn-system
 spec:
@@ -4060,7 +4096,7 @@ metadata:
   labels:
     app.kubernetes.io/name: longhorn
     app.kubernetes.io/instance: longhorn
-    app.kubernetes.io/version: v1.4.0-dev
+    app.kubernetes.io/version: v1.6.0-dev
     app: longhorn-manager
   name: longhorn-manager
   namespace: longhorn-system
@@ -4073,7 +4109,7 @@ spec:
       labels:
         app.kubernetes.io/name: longhorn
         app.kubernetes.io/instance: longhorn
-        app.kubernetes.io/version: v1.4.0-dev
+        app.kubernetes.io/version: v1.6.0-dev
         app: longhorn-manager
     spec:
       containers:
@@ -4095,7 +4131,7 @@ spec:
         - --backing-image-manager-image
         - "longhornio/backing-image-manager:master-head"
         - --support-bundle-manager-image
-        - "longhornio/support-bundle-kit:v0.0.24"
+        - "longhornio/support-bundle-kit:v0.0.27"
         - --manager-image
         - "longhornio/longhorn-manager:master-head"
         - --service-account
@@ -4165,7 +4201,7 @@ metadata:
   labels:
     app.kubernetes.io/name: longhorn
     app.kubernetes.io/instance: longhorn
-    app.kubernetes.io/version: v1.4.0-dev
+    app.kubernetes.io/version: v1.6.0-dev
 spec:
   replicas: 1
   selector:
@@ -4176,7 +4212,7 @@ spec:
       labels:
         app.kubernetes.io/name: longhorn
         app.kubernetes.io/instance: longhorn
-        app.kubernetes.io/version: v1.4.0-dev
+        app.kubernetes.io/version: v1.6.0-dev
         app: longhorn-driver-deployer
     spec:
       initContainers:
@@ -4231,7 +4267,7 @@ metadata:
   labels:
     app.kubernetes.io/name: longhorn
     app.kubernetes.io/instance: longhorn
-    app.kubernetes.io/version: v1.4.0-dev
+    app.kubernetes.io/version: v1.6.0-dev
     app: longhorn-ui
   name: longhorn-ui
   namespace: longhorn-system
@@ -4245,9 +4281,10 @@ spec:
       labels:
         app.kubernetes.io/name: longhorn
         app.kubernetes.io/instance: longhorn
-        app.kubernetes.io/version: v1.4.0-dev
+        app.kubernetes.io/version: v1.6.0-dev
         app: longhorn-ui
     spec:
+      serviceAccountName: longhorn-ui-service-account
       affinity:
         podAntiAffinity:
           preferredDuringSchedulingIgnoredDuringExecution:

deploy/prerequisite/longhorn-iscsi-selinux-workaround.yaml

@@ -0,0 +1,35 @@
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: longhorn-iscsi-selinux-workaround
+  labels:
+    app: longhorn-iscsi-selinux-workaround
+  annotations:
+    command: &cmd if ! rpm -q policycoreutils > /dev/null 2>&1; then echo "failed to apply workaround; only applicable in Fedora based distros with SELinux enabled"; exit; elif cd /tmp && echo '(allow iscsid_t self (capability (dac_override)))' > local_longhorn.cil && semodule -vi local_longhorn.cil && rm -f local_longhorn.cil; then echo "applied workaround successfully"; else echo "failed to apply workaround; error code $?"; fi
+spec:
+  selector:
+    matchLabels:
+      app: longhorn-iscsi-selinux-workaround
+  template:
+    metadata:
+      labels:
+        app: longhorn-iscsi-selinux-workaround
+    spec:
+      hostPID: true
+      initContainers:
+      - name: iscsi-selinux-workaround
+        command:
+          - nsenter
+          - --mount=/proc/1/ns/mnt
+          - --
+          - bash
+          - -c
+          - *cmd
+        image: alpine:3.17
+        securityContext:
+          privileged: true
+      containers:
+      - name: sleep
+        image: registry.k8s.io/pause:3.1
+  updateStrategy:
+    type: RollingUpdate

deploy/prerequisite/longhorn-spdk-setup.yaml

@@ -5,7 +5,7 @@ metadata:
   labels:
     app: longhorn-spdk-setup
   annotations:
-    command: &cmd rm -rf ${SPDK_DIR}; git clone -b longhorn https://github.com/longhorn/spdk.git ${SPDK_DIR} && bash ${SPDK_DIR}/scripts/setup.sh ${SPDK_OPTION}; if [ $? -eq 0 ]; then echo "vm.nr_hugepages=$((HUGEMEM/2))" >> /etc/sysctl.conf; echo "SPDK environment is configured successfully"; else echo "Failed to configure SPDK environment error code $?"; fi; rm -rf ${SPDK_DIR}
+    command: &cmd OS=$(grep -E "^ID_LIKE=" /etc/os-release | cut -d '=' -f 2); if [[ -z "${OS}" ]]; then OS=$(grep -E "^ID=" /etc/os-release | cut -d '=' -f 2); fi; if [[ "${OS}" == *"debian"* ]]; then sudo apt-get update -q -y && sudo apt-get install -q -y git; elif [[ "${OS}" == *"suse"* ]]; then sudo zypper --gpg-auto-import-keys -q refresh && sudo zypper --gpg-auto-import-keys -q install -y git; else sudo yum makecache -q -y && sudo yum --setopt=tsflags=noscripts install -q -y git; fi && if [ $? -eq 0 ]; then echo "git install successfully"; else echo "git install failed error code $?"; fi && rm -rf ${SPDK_DIR}; git clone -b longhorn https://github.com/longhorn/spdk.git ${SPDK_DIR} && bash ${SPDK_DIR}/scripts/setup.sh ${SPDK_OPTION}; if [ $? -eq 0 ]; then echo "vm.nr_hugepages=$((HUGEMEM/2))" >> /etc/sysctl.conf; echo "SPDK environment is configured successfully"; else echo "Failed to configure SPDK environment error code $?"; fi; rm -rf ${SPDK_DIR}
 spec:
   selector:
     matchLabels:
@@ -33,7 +33,7 @@ spec:
         - name: SPDK_OPTION
           value: ""
         - name: HUGEMEM
-          value: "2048"
+          value: "1024"
         - name: PCI_ALLOWED
           value: "none"
         - name: DRIVER_OVERRIDE

deploy/upgrade_responder_server/README.md

@@ -1,5 +1,7 @@
-This directory contains the helm values for the Longhorn upgrade responder server. 
+# Upgrade Responder Helm Chart
+
+This directory contains the helm values for the Longhorn upgrade responder server.
 The values are in the file `./chart-values.yaml`.
 When you update the content of `./chart-values.yaml`, automation pipeline will update the Longhorn upgrade responder.
-
-The chart source chart is in `chart.yaml`
+Information about the source chart is in `chart.yaml`.
+See [dev/upgrade-responder](../../dev/upgrade-responder/README.md) for manual deployment steps.

deploy/upgrade_responder_server/chart-values.yaml

@@ -14,27 +14,359 @@ secret:
   # Set this to false if you don't want to manage these secrets with helm
   managed: false
 
+resources:
+  limits:
+    cpu: 400m
+    memory: 512Mi
+  requests:
+    cpu: 200m
+    memory: 256Mi
+
 # This configmap contains information about the latest release
 # of the application that is using this Upgrade Responder
 configMap:
   responseConfig: |-
     {
-      "Versions": [
+      "versions": [
         {
-          "Name": "v1.3.3",
-          "ReleaseDate": "2023-04-19T00:00:00Z",
-          "Tags": [
+          "name": "v1.3.3",
+          "releaseDate": "2023-04-19T00:00:00Z",
+          "tags": [
             "stable"
           ]
         },
         {
-          "Name": "v1.4.2",
-          "ReleaseDate": "2023-05-12T00:00:00Z",
-          "Tags": [
+          "name": "v1.4.3",
+          "releaseDate": "2023-07-14T00:00:00Z",
+          "tags": [
             "latest", 
             "stable"
           ]
+        },
+        {
+          "name": "v1.5.1",
+          "releaseDate": "2023-07-19T00:00:00Z",
+          "tags": [
+            "latest"
+          ]
         }
       ]
     }
+  requestSchema: |-
+    {
+      "appVersionSchema": {
+        "dataType": "string",
+        "maxLen": 200
+      },
+      "extraTagInfoSchema": {
+        "hostKernelRelease": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "hostOsDistro": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "kubernetesNodeProvider": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "kubernetesVersion": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingAllowRecurringJobWhileVolumeDetached": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingAllowVolumeCreationWithDegradedAvailability": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingAutoCleanupSystemGeneratedSnapshot": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingAutoDeletePodWhenVolumeDetachedUnexpectedly": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingAutoSalvage": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingBackupCompressionMethod": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingBackupTarget": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingCrdApiVersion": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingCreateDefaultDiskLabeledNodes": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingDefaultDataLocality": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingDisableRevisionCounter": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingDisableSchedulingOnCordonedNode": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingFastReplicaRebuildEnabled": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingKubernetesClusterAutoscalerEnabled": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingNodeDownPodDeletionPolicy": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingNodeDrainPolicy": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingOfflineReplicaRebuilding": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingOrphanAutoDeletion": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingPriorityClass": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingRegistrySecret": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingRemoveSnapshotsDuringFilesystemTrim": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingReplicaAutoBalance": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingReplicaSoftAntiAffinity": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingReplicaZoneSoftAntiAffinity": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingReplicaDiskSoftAntiAffinity": {
+          "dataType": "string",
+          "maxLen": 200
+        }
+        "longhornSettingRestoreVolumeRecurringJobs": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingSnapshotDataIntegrity": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingSnapshotDataIntegrityCronjob": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingSnapshotDataIntegrityImmediateCheckAfterSnapshotCreation": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingStorageNetwork": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingSystemManagedComponentsNodeSelector": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingSystemManagedPodsImagePullPolicy": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingTaintToleration": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingV2DataEngine": {
+          "dataType": "string",
+          "maxLen": 200
+        }
+      },
+      "extraFieldInfoSchema": {
+        "longhornInstanceManagerAverageCpuUsageMilliCores": {
+          "dataType": "float"
+        },
+        "longhornInstanceManagerAverageMemoryUsageBytes": {
+          "dataType": "float"
+        },
+        "longhornManagerAverageCpuUsageMilliCores": {
+          "dataType": "float"
+        },
+        "longhornManagerAverageMemoryUsageBytes": {
+          "dataType": "float"
+        },
+        "longhornNamespaceUid": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornNodeCount": {
+          "dataType": "float"
+        },
+        "longhornNodeDiskHDDCount": {
+          "dataType": "float"
+        },
+        "longhornNodeDiskNVMeCount": {
+          "dataType": "float"
+        },
+        "longhornNodeDiskSSDCount": {
+          "dataType": "float"
+        },
+        "longhornSettingBackingImageCleanupWaitInterval": {
+          "dataType": "float"
+        },
+        "longhornSettingBackingImageRecoveryWaitInterval": {
+          "dataType": "float"
+        },
+        "longhornSettingBackupConcurrentLimit": {
+          "dataType": "float"
+        },
+        "longhornSettingBackupstorePollInterval": {
+          "dataType": "float"
+        },
+        "longhornSettingConcurrentAutomaticEngineUpgradePerNodeLimit": {
+          "dataType": "float"
+        },
+        "longhornSettingConcurrentReplicaRebuildPerNodeLimit": {
+          "dataType": "float"
+        },
+        "longhornSettingConcurrentVolumeBackupRestorePerNodeLimit": {
+          "dataType": "float"
+        },
+        "longhornSettingDefaultReplicaCount": {
+          "dataType": "float"
+        },
+        "longhornSettingEngineReplicaTimeout": {
+          "dataType": "float"
+        },
+        "longhornSettingFailedBackupTtl": {
+          "dataType": "float"
+        },
+        "longhornSettingGuaranteedInstanceManagerCpu": {
+          "dataType": "float"
+        },
+        "longhornSettingRecurringFailedJobsHistoryLimit": {
+          "dataType": "float"
+        },
+        "longhornSettingRecurringSuccessfulJobsHistoryLimit": {
+          "dataType": "float"
+        },
+        "longhornSettingReplicaFileSyncHttpClientTimeout": {
+          "dataType": "float"
+        },
+        "longhornSettingReplicaReplenishmentWaitInterval": {
+          "dataType": "float"
+        },
+        "longhornSettingRestoreConcurrentLimit": {
+          "dataType": "float"
+        },
+        "longhornSettingStorageMinimalAvailablePercentage": {
+          "dataType": "float"
+        },
+        "longhornSettingStorageOverProvisioningPercentage": {
+          "dataType": "float"
+        },
+        "longhornSettingStorageReservedPercentageForDefaultDisk": {
+          "dataType": "float"
+        },
+        "longhornSettingSupportBundleFailedHistoryLimit": {
+          "dataType": "float"
+        },
+        "longhornVolumeAccessModeRwoCount": {
+          "dataType": "float"
+        },
+        "longhornVolumeAccessModeRwxCount": {
+          "dataType": "float"
+        },
+        "longhornVolumeAccessModeUnknownCount": {
+          "dataType": "float"
+        },
+        "longhornVolumeAverageActualSizeBytes": {
+          "dataType": "float"
+        },
+        "longhornVolumeAverageNumberOfReplicas": {
+          "dataType": "float"
+        },
+        "longhornVolumeAverageSizeBytes": {
+          "dataType": "float"
+        },
+        "longhornVolumeAverageSnapshotCount": {
+          "dataType": "float"
+        },
+        "longhornVolumeDataLocalityBestEffortCount": {
+          "dataType": "float"
+        },
+        "longhornVolumeDataLocalityDisabledCount": {
+          "dataType": "float"
+        },
+        "longhornVolumeDataLocalityStrictLocalCount": {
+          "dataType": "float"
+        },
+        "longhornVolumeFrontendBlockdevCount": {
+          "dataType": "float"
+        },
+        "longhornVolumeFrontendIscsiCount": {
+          "dataType": "float"
+        },
+        "longhornVolumeOfflineReplicaRebuildingDisabledCount": {
+          "dataType": "float"
+        },
+        "longhornVolumeOfflineReplicaRebuildingEnabledCount": {
+          "dataType": "float"
+        },
+        "longhornVolumeReplicaAutoBalanceDisabledCount": {
+          "dataType": "float"
+        },
+        "longhornVolumeReplicaSoftAntiAffinityFalseCount": {
+          "dataType": "float"
+        },
+        "longhornVolumeReplicaZoneSoftAntiAffinityTrueCount": {
+          "dataType": "float"
+        },
+        "longhornVolumeReplicaDiskSoftAntiAffinityTrueCount": {
+          "dataType": "float"
+        },
+        "longhornVolumeRestoreVolumeRecurringJobFalseCount": {
+          "dataType": "float"
+        },
+        "longhornVolumeSnapshotDataIntegrityDisabledCount": {
+          "dataType": "float"
+        },
+        "longhornVolumeSnapshotDataIntegrityFastCheckCount": {
+          "dataType": "float"
+        },
+        "longhornVolumeUnmapMarkSnapChainRemovedFalseCount": {
+          "dataType": "float"
+        }
+      }
+    }
 

deploy/upgrade_responder_server/chart.yaml

@@ -1,5 +1,5 @@
 url: https://github.com/longhorn/upgrade-responder.git
-commit: 3c78890f5415744af1923eac01f98636ac52a113
+commit: 116f807836c29185038cfb005708f0a8d41f4d35
 releaseName: longhorn-upgrade-responder
 namespace: longhorn-upgrade-responder
 

dev/upgrade-responder/install.sh

@@ -1,6 +1,7 @@
 #!/bin/bash
 
 UPGRADE_RESPONDER_REPO="https://github.com/longhorn/upgrade-responder.git"
+UPGRADE_RESPONDER_REPO_BRANCH="master"
 UPGRADE_RESPONDER_VALUE_YAML="upgrade-responder-value.yaml"
 UPGRADE_RESPONDER_IMAGE_REPO="longhornio/upgrade-responder"
 UPGRADE_RESPONDER_IMAGE_TAG="master-head"
@@ -13,7 +14,7 @@ DEPLOYMENT_TIMEOUT_SEC=300
 DEPLOYMENT_WAIT_INTERVAL_SEC=5
 
 temp_dir=$(mktemp -d)
-trap 'rm -r "${temp_dir}"' EXIT
+trap 'rm -rf "${temp_dir}"' EXIT # -f because packed Git files (.pack, .idx) are write protected.
 
 cp -a ./* ${temp_dir}
 cd ${temp_dir}
@@ -59,12 +60,338 @@ secret:
   influxDBUrl: "${INFLUXDB_URL}"
   influxDBUser: "root"
   influxDBPassword: "root"
+configMap:
+  responseConfig: |-
+    {
+      "versions": [{
+        "name": "v1.0.0",
+        "releaseDate": "2020-05-18T12:30:00Z",
+        "tags": ["latest"]
+      }]
+    }
+  requestSchema: |-
+    {
+      "appVersionSchema": {
+        "dataType": "string",
+        "maxLen": 200
+      },
+      "extraTagInfoSchema": {
+        "hostKernelRelease": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "hostOsDistro": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "kubernetesNodeProvider": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "kubernetesVersion": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingAllowRecurringJobWhileVolumeDetached": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingAllowVolumeCreationWithDegradedAvailability": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingAutoCleanupSystemGeneratedSnapshot": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingAutoDeletePodWhenVolumeDetachedUnexpectedly": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingAutoSalvage": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingBackupCompressionMethod": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingBackupTarget": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingCrdApiVersion": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingCreateDefaultDiskLabeledNodes": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingDefaultDataLocality": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingDisableRevisionCounter": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingDisableSchedulingOnCordonedNode": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingFastReplicaRebuildEnabled": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingKubernetesClusterAutoscalerEnabled": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingNodeDownPodDeletionPolicy": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingNodeDrainPolicy": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingOfflineReplicaRebuilding": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingOrphanAutoDeletion": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingPriorityClass": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingRegistrySecret": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingRemoveSnapshotsDuringFilesystemTrim": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingReplicaAutoBalance": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingReplicaSoftAntiAffinity": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingReplicaZoneSoftAntiAffinity": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingReplicaDiskSoftAntiAffinity": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingRestoreVolumeRecurringJobs": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingSnapshotDataIntegrity": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingSnapshotDataIntegrityCronjob": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingSnapshotDataIntegrityImmediateCheckAfterSnapshotCreation": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingStorageNetwork": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingSystemManagedComponentsNodeSelector": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingSystemManagedPodsImagePullPolicy": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingTaintToleration": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornSettingV2DataEngine": {
+          "dataType": "string",
+          "maxLen": 200
+        }
+      },
+      "extraFieldInfoSchema": {
+        "longhornInstanceManagerAverageCpuUsageMilliCores": {
+          "dataType": "float"
+        },
+        "longhornInstanceManagerAverageMemoryUsageBytes": {
+          "dataType": "float"
+        },
+        "longhornManagerAverageCpuUsageMilliCores": {
+          "dataType": "float"
+        },
+        "longhornManagerAverageMemoryUsageBytes": {
+          "dataType": "float"
+        },
+        "longhornNamespaceUid": {
+          "dataType": "string",
+          "maxLen": 200
+        },
+        "longhornNodeCount": {
+          "dataType": "float"
+        },
+        "longhornNodeDiskHDDCount": {
+          "dataType": "float"
+        },
+        "longhornNodeDiskNVMeCount": {
+          "dataType": "float"
+        },
+        "longhornNodeDiskSSDCount": {
+          "dataType": "float"
+        },
+        "longhornSettingBackingImageCleanupWaitInterval": {
+          "dataType": "float"
+        },
+        "longhornSettingBackingImageRecoveryWaitInterval": {
+          "dataType": "float"
+        },
+        "longhornSettingBackupConcurrentLimit": {
+          "dataType": "float"
+        },
+        "longhornSettingBackupstorePollInterval": {
+          "dataType": "float"
+        },
+        "longhornSettingConcurrentAutomaticEngineUpgradePerNodeLimit": {
+          "dataType": "float"
+        },
+        "longhornSettingConcurrentReplicaRebuildPerNodeLimit": {
+          "dataType": "float"
+        },
+        "longhornSettingConcurrentVolumeBackupRestorePerNodeLimit": {
+          "dataType": "float"
+        },
+        "longhornSettingDefaultReplicaCount": {
+          "dataType": "float"
+        },
+        "longhornSettingEngineReplicaTimeout": {
+          "dataType": "float"
+        },
+        "longhornSettingFailedBackupTtl": {
+          "dataType": "float"
+        },
+        "longhornSettingGuaranteedInstanceManagerCpu": {
+          "dataType": "float"
+        },
+        "longhornSettingRecurringFailedJobsHistoryLimit": {
+          "dataType": "float"
+        },
+        "longhornSettingRecurringSuccessfulJobsHistoryLimit": {
+          "dataType": "float"
+        },
+        "longhornSettingReplicaFileSyncHttpClientTimeout": {
+          "dataType": "float"
+        },
+        "longhornSettingReplicaReplenishmentWaitInterval": {
+          "dataType": "float"
+        },
+        "longhornSettingRestoreConcurrentLimit": {
+          "dataType": "float"
+        },
+        "longhornSettingStorageMinimalAvailablePercentage": {
+          "dataType": "float"
+        },
+        "longhornSettingStorageOverProvisioningPercentage": {
+          "dataType": "float"
+        },
+        "longhornSettingStorageReservedPercentageForDefaultDisk": {
+          "dataType": "float"
+        },
+        "longhornSettingSupportBundleFailedHistoryLimit": {
+          "dataType": "float"
+        },
+        "longhornVolumeAccessModeRwoCount": {
+          "dataType": "float"
+        },
+        "longhornVolumeAccessModeRwxCount": {
+          "dataType": "float"
+        },
+        "longhornVolumeAccessModeUnknownCount": {
+          "dataType": "float"
+        },
+        "longhornVolumeAverageActualSizeBytes": {
+          "dataType": "float"
+        },
+        "longhornVolumeAverageNumberOfReplicas": {
+          "dataType": "float"
+        },
+        "longhornVolumeAverageSizeBytes": {
+          "dataType": "float"
+        },
+        "longhornVolumeAverageSnapshotCount": {
+          "dataType": "float"
+        },
+        "longhornVolumeDataLocalityBestEffortCount": {
+          "dataType": "float"
+        },
+        "longhornVolumeDataLocalityDisabledCount": {
+          "dataType": "float"
+        },
+        "longhornVolumeDataLocalityStrictLocalCount": {
+          "dataType": "float"
+        },
+        "longhornVolumeFrontendBlockdevCount": {
+          "dataType": "float"
+        },
+        "longhornVolumeFrontendIscsiCount": {
+          "dataType": "float"
+        },
+        "longhornVolumeOfflineReplicaRebuildingDisabledCount": {
+          "dataType": "float"
+        },
+        "longhornVolumeOfflineReplicaRebuildingEnabledCount": {
+          "dataType": "float"
+        },
+        "longhornVolumeReplicaAutoBalanceDisabledCount": {
+          "dataType": "float"
+        },
+        "longhornVolumeReplicaSoftAntiAffinityFalseCount": {
+          "dataType": "float"
+        },
+        "longhornVolumeReplicaZoneSoftAntiAffinityTrueCount": {
+          "dataType": "float"
+        },
+        "longhornVolumeReplicaDiskSoftAntiAffinityTrueCount": {
+          "dataType": "float"
+        },
+        "longhornVolumeRestoreVolumeRecurringJobFalseCount": {
+          "dataType": "float"
+        },
+        "longhornVolumeSnapshotDataIntegrityDisabledCount": {
+          "dataType": "float"
+        },
+        "longhornVolumeSnapshotDataIntegrityFastCheckCount": {
+          "dataType": "float"
+        },
+        "longhornVolumeUnmapMarkSnapChainRemovedFalseCount": {
+          "dataType": "float"
+        }
+      }
+    }
 image:
   repository: ${UPGRADE_RESPONDER_IMAGE_REPO}
   tag: ${UPGRADE_RESPONDER_IMAGE_TAG}
 EOF
 
-    git clone ${UPGRADE_RESPONDER_REPO}
+    git clone -b ${UPGRADE_RESPONDER_REPO_BRANCH} ${UPGRADE_RESPONDER_REPO}
     helm upgrade --install ${APP_NAME}-upgrade-responder upgrade-responder/chart -f ${UPGRADE_RESPONDER_VALUE_YAML}
     wait_for_deployment "${APP_NAME}-upgrade-responder"
 }

enhancements/20220428-storage-network-through-grpc-proxy.md

@@ -51,7 +51,7 @@ https://github.com/longhorn/longhorn/issues/3546
 
 - Introduce a new gRPC server in Instance Manager.
 
-- Keep re-usable connections between Manager and Instance Managers.
+- Keep reusable connections between Manager and Instance Managers.
 
 - Allow Manager to fall back to engine binary call when communicating with old Instance Manager.
 
@@ -101,7 +101,7 @@ So I can decide when to upgrade the Engine Image.
 1. When updating the setting I see engine/replica instance manager pod and backing image manager pods is restarted.
 1. I attach the volumes.
 1. I describe Engine, Replica, and BackingImageManager, and see the `storageIP` in CR status is in the range of the `NetworkAttachmentDefinition` subnet/CIDR. I also see the `storageIP` is different from the `ip` in CR status.
-1. I describe the Engine and see the `replicaAddressMap` in CR spec and status is using the storage IP. 
+1. I describe the Engine and see the `replicaAddressMap` in CR spec and status is using the storage IP.
 1. I see pod logs indicate the network directions.
 
 #### Story 2 - upgrade

enhancements/20220727-dedicated-recovery-backend-for-rwx-volume-nfs-server.md

@@ -68,7 +68,7 @@ While the node where the share-manager pod is running is down, the share-manager
                               │                          │
                           HTTP API         ┌─────────────┴──────────────┐
                               │            │                            │
-                              │            │ endpint 1                  │ endpoint N
+                              │            │ endpoint 1                 │ endpoint N
      ┌──────────────────────┐ │  ┌─────────▼────────┐          ┌────────▼─────────┐
      │  share-manager pod   │ │  │ recovery-backend │          │ recovery-backend │
      │                      │ │  │      pod         │          │      pod         │

enhancements/20221213-reimplement-longhorn-engine-with-SPDK.md

@@ -0,0 +1,181 @@
+# Reimplement Longhorn Engine with SPDK
+
+
+## Summary
+
+The Storage Performance Development Kit [SPDK](https://spdk.io) provides a set of tools and C libraries for writing high performance, scalable, user-mode storage applications. It achieves high performance through the use of a number of key techniques:
+
+* Moving all of the necessary drivers into userspace, which avoids syscalls and enables zero-copy access from the application.
+* Polling hardware for completions instead of relying on interrupts, which lowers both total latency and latency variance.
+* Avoiding all locks in the I/O path, instead relying on message passing.
+
+SPDK has several features that allow it to perform tasks similar to what the `longhorn-engine` currently needs:
+
+* [Block Device](https://spdk.io/doc/bdev.html) layer, often simply called bdev, intends to be equivalent to the operating system block storage layer that often sits immediately above the device drivers in a traditional kernel storage stack. SPDK provides also virtual bdev modules which creates block devices on existing bdev, for example Logical Volumes or RAID1.
+* [Logical volumes](https://spdk.io/doc/logical_volumes.html) library is a flexible storage space management system. It allows creating and managing virtual block devices with variable size on top of other bdevs. The SPDK Logical Volume library is built on top [Blobstore](https://spdk.io/doc/blob.html) which is a persistent, power-fail safe block allocator designed to be used as the local storage system backing a higher level storage service, typically in lieu of a traditional filesystem. Logical volumes have a couple of features like Thinly Provisioning and Snapshots similar to what actual Longhorn-Engine provides.
+* [NVMe over Fabrics](https://spdk.io/doc/nvmf.html) is a feature to presents block devices over a fabrics such as Ethernet, supporting RDMA and TCP transports. The standard Linux kernel initiators for NVMe-oF interoperate with these SPDK NVMe-oF targets, so with this feature we can serve bdev over the network or to other processes
+
+
+## Motivation
+
+These are the reasons that have driven us:
+
+* Use SPDK to improve performance of Longhorn
+* Use SPDK functionality to improve reliability and robustness
+* Use SPDK to take advantage of the new features that are continuously added to the framework
+
+### Goals
+
+* Implement all actual `longhorn-engine` functionalities
+* Continue to support multiple `longhorn-engine` versions concurrently
+* Maintain as much as possible the same user experience between Longhorn with and without SPDK
+* Lay the groundwork for extending Longhorn to sharding and aggegration of storage devices
+
+
+## Proposal
+
+SPDK implements a JSON-RPC 2.0 server to allow external management tools to dynamically configure SPDK components ([documentation](https://spdk.io/doc/jsonrpc.html)).
+
+What we aim is to create an external orchestrator that, with JSON-RPC calls towards multiple instances of `spdk_tgt` app running in different machines, could manage the durability and reliability of data. Actually, not all needed functionalities to do that are already available in SPDK, so some new JSON-RPC commands will be developed over SPDK. This orchestrator is implemented in longhorn manager pods and will use a new process, called `longhorn-spdk-engine` in continuity with actual `longhorn-engine`, to talk with `spdk_tgt`.
+
+* The main purpose of `longhorn-spdk-engine` is to create and export via NVMe-oF logical volumes from multiple replica nodes (one of them likely local), attach to these volumes on a controller node, use resulting bdevs to create a RAID1 bdev and exporting it via NVMe-oF locally. At this point NVMe Kernel module can be used to connect to this NVMe-oF subsystem and so to create a block device `/dev/nvmeXnY` to be used by the Longhorn CSI driver. In this way we will have multiple replica of the same data written on this block device.
+* Below a diagram that shows the control plane of the proposal ![SPDK New Architecture](./image/spdk-control-plane.png)
+* In release 23.01, support for ublk will be added in SPDK: with this functionality we can directly create a block device without using the NVMe layer on Linux kernel versions >6.0. This will be a quite big enhancement over using NVMe-oF locally.
+
+The `longhorn-spdk-engine` will be responsible to make all others control operations, like for example creating snapshots over all replicas of the same volume. Other functionalities orchestrated through the engine will be the remote rebuild, a complete rebuild of the entire snapshot stack of a volume needed to add or repair a replica, the backup and restore, export/import of a SPDK logical volumes to/from sparse files stored on an external storage system via S3.
+
+The `longhorn-spdk-engine` will be developed in Go so maybe we can reuse some code from `longhorn-engine`, for example gRPC handling to receive control commands and error handling during snapshot/backup/restore operations.
+
+
+What about the data plane, below a comparison between actual architecture and new design:
+* longhorn-engine ![](./image/engine-data-plane.png)
+* spdk_tgt       
+  ![](./image/spdk-data-plane.png)
+
+## Design
+
+
+### Implementation Overview
+
+Actually there is a `longhorn-engine` controller and some `longhorn-engine` replica for every volume to manage. All these instances are started and controlled by the `instance-manager`, so on every node belonging to the cluster we have one instance of `instance-manager` and multiple instances of `longhorn-engine`. Every volume is stored in a sequence of sparse files representing the live data and the snapshots. With SPDK we have a different situation, because `spdk_tgt` can take the control of an entire disk, so in every node we will have a single instance of SPDK that will handle all the volumes created by Longhorn.
+
+To orchestrate SPDK instances running on different nodes in a way to make up a set of replicas, we will introduce, as discussed before, the `longhorn-spdk-engine`; to make the volume management lighter we will have an instance of the engine per volume. `longhorn-spdk-engine` will implement actual gRPC interface used by `longhorn-engine` to talk with `instance-manager`, so this last one will became the portal to communicate with `longhorn-manager` by different data plane.
+
+`spdk_tgt` by default starts with a single thread, but it can be configured to use multiple threads: we can have a thread per core available on the CPU. This will increase the performance but comes with the cost of an high CPU utilization. Working in polling mode instead than in interrupt mode, CPU core utilization by a single thread is always rising 100% even with no workload to handle. This could be a problem, so we can configure `spdk_tgt` with dynamic scheduler: in this way, if no workload is present, only one core will be used and only one thread will continue polling. Other thread will be put in a idle state and will become active again only when needed. Moreover, dynamic scheduler has a way to reduce the CPU frequency. (See future work section.)
+
+
+### Snapshots
+
+When `longhorn-spdk-engine` receive a snapshot request from `instance-manager`, before to proceed all I/O operations over volume's block device `/dev/nvmeXnY` must be stopped to ensure that snapshots over all the replicas contains the same data.
+Actually there is no way to suspend the I/O operations over a block device, so we will have to implement this feature into SPDK. But in RAID bdev there are already some private functions to suspend I/O (they will be used for example in base bdev removing), maybe we can use and improve them. These functions actually enqueue all the I/O operations received during the suspend time.
+
+Once received a snapshot request, `longhorn-spdk-engine` will call the JSON-RPC to make a snapshot over the local replica of the volume involved. The snapshot RPC command will ensure to freeze all I/O over the logical volume to be snapshotted, so all pending I/O will be executed before the snapshot.
+
+SPDK logical volume have a couple of features that we will use:
+* clone, used to create new logical volume based on a snapshot. It can be used to revert a volume to a snapshot too, cloning a new volume, deleting the old one and then renaming the new one as the old one
+* decouple, feature that can be used to delete a snapshot, first decoupling the child volume from this snapshot and then deleting the snapshot.
+
+
+### Replica rebuild
+
+RAID replica rebuild is actually under development, so we don'know exactly hot it will be implemented, but we can suppose that we will not use it because presumably it will work only at bdev layer.
+When a new replica has to be added or a replica has to be rebuilt, we have to recreate the entire snapshot stack of each volume that are hosted on that node. Actually SPDK doesn't have nothing to do that, but after discussing with core maintainers we arranged a procedure. Let's make an example.
+
+Supposing we have to rebuild a volume with two layer of snapshots, snapshotA is the oldest and snapshotB the younger, basically we have to (in _italic_ what we miss):
+
+* create a new volume on the node to be rebuilt
+* export this volume via NVMe-oF
+* attach to this volume in the node where we have the source data
+* _copy snapshotA over the attached volume_
+* perform a snapshot over the exported volume
+* repeat the copy and snapshot operations for snapshotB
+
+What we have to implement is a JSON-RPC to copy a logical volume over an arbitrary bdev (that in our case will represent a remote volume exported via NVMe-oF and locally attached) _while the top layer is also being modified_ (see next section).
+
+So, in this way we can rebuild the snapshot stack of a volume. But what about the live data? Actual `longhorn-engine` make the replica rebuild in an "hot" way, i.e., during the rebuilding phase it is writing over the live data of the new volume. So, how can we reproduce this with SPDK? First of all we have to wait the conclusion of RAID1 bdev's review process to see what kind of replica rebuild will be implemented. But, supposing that the rebuild feature will not be useful for us, we will need to create a couple of additional JSON-RPC over SPDK to implement the following procedure (in _italic_ what we miss):
+
+* create a new volume over the node to be rebuilt
+* export this volume via NVMe-oF
+* attach to this volume in the node where we have the RAID1
+* _add the bdev of the attached volume to the RAID1 bdev excluded from the read balancing_
+* wait for the snapshot stack rebuilding to finish
+* _change the upper volume of the snapshot stack from the current to this one with the live data_
+* _enable the bdev of the attached volume for RAID1 read balancing_
+
+What we have at the end of the rebuilding's phase is a snapshot stack with an empty volume at the top, while in the RAID1 we have a volume with the live data but without any snapshot. So we have to couple these 2 stacks exchanging the upper volume and to do that we need a new JSON-RPC. We will need to implement the JSON-RPC to enable/disable a bdev from the RAID1 read balancing too.
+
+
+### Backup and Restore
+
+Backup will be implemented exporting a volume to a sparse file and then save this file over an external storage via S3. SPDK already has a `spdk_dd` application that can copy a bdev to a file and this app has an option to preserve bdev sparseness. But using spdk_dd has some problems: actually the sparse option works only with bdev that represent a local logical volume, not an exported one via NVMe-oF. So to backup a volume we cannot work on a remote node where to export this volume, we need to work on the node where we have the data source. But in this way, to perform a backup, we would need to stop the `spdk_tgt` app, run the `spdk_dd` and then restart the `spdk_tgt`. This operation is needed because it could not be safe to run multiple spdk applications over the same disk (even if spdk_dd would read from a read only volume) and moreover `spdk_dd` could not see the volume to export if this has been created after the last restart of `spdk_tgt` app. This because blobstore metadata, and so newly created logical volume, are saved on disk only on application exit.
+
+Stopping `spdk_tgt` is not acceptable because it would suspend operation over all other volumes hosted in this node so, to solve these problems, we have 2 possible solutions:
+
+* create a JSON-RPC command to export logical volume to a sparse file, so that we can make the operation directly over the `spdk_tgt` app
+* create a custom NVMe-oF command to implement the seek_data and seek_hole functionalities of bdev used by `spdk_dd` to skip holes
+
+With the second solution we could export the volume via NVMe-oF to a dedicated node where to perform the backup with `spdk_dd`application.
+
+The restore operation can be done in a couple of way:
+* read the backup sparse file and write its content into the longhorn block device. In this way data will be fully replicated
+* clone from backup over each replica, importing the backup sparse file into a new thinly provisioned logical volume. We can perform this operation over the local node, owner of the new volume, if for the backup process we choose to develop a JSON-RPC to export/import logical volume to/from sparse files. Otherwise we can do it or over a dedicated node with `spdk_dd` application, that handle sparse file with SEEK_HOLE and SEEK_DATA functionalities of `lseek`.
+
+If we leverage the same backup & restore mechanism of `longhorn-engine`, we can restore a backup done by the actual engine to a SPDK volume.
+
+
+### Remote Control
+
+The JSON-RPC API by default is only available over the `/var/tmp/spdk.sock` Unix domain socket, but SPDK offer the sample python script [rpc_http_proxy](https://spdk.io/doc/jsonrpc_proxy.html) that provides http server which listens for JSON objects from users. Otherwise we could use the `socat` application to forward requests received from an IP socket towards a Unix socket. Both `socat` and `rpc_http_proxy` can perform user authentication with password.
+
+
+### Upgrade Strategy
+
+What kind of upgrade/migration will we support?
+For out-of-cluster migration we can use the Restore procedure to create SPDK logical volumes starting from existing Longhorn files. Instead for in-cluster migration we can retain read support for the old format, writing new data over SPDK.
+
+Whatabout `spdk_tgt` updates, we can perform a rolling update strategy updating nodes one by one. Stopping `spdk_tgt` over a node will cause:
+* stop of all the volumes controlled in the node. To avoid service interruption the node must be evacuated before the update. The cheat is to delay the update until the node has to be rebooted for a kernel update.
+* stop of all the replicas hosted in the node. This is not a problem because during the update the I/O will be redirected towards other replica of the volume. To make a clean update of a node, before to stop `spdk_tgt`, we have to notify all the nodes that have a bdev imported via NMVMe-oF from this node to detach controllers involved.
+
+Moreover this is a good time to introduce backup versioning, which allows us to change/improve the backup format [REF: GH3175](https://github.com/longhorn/longhorn/issues/3175)
+
+
+### Future Work
+
+* For Edge use cases, energy efficiency is important. We may need further enhancements and an interrupt-driven mode during low load periods for the scheduler. [Here](https://www.snia.org/educational-library/spdk-schedulers-%E2%80%93-saving-cpu-cores-polled-mode-storage-application-2021) an introduction to SPDK Schedulers that describes briefly the interrupt mode.
+
+
+### Roadmap
+
+For Longhorn 1.5, we need to have the below capabilities:
+* replica (RAID1)
+* snapshot (create, delete/purge, revert)
+* replica rebuilding
+* volume clone
+
+For 1.6, we need the rest of the feature parity functions:
+* volume backup & restore
+* DR volume restore (incremental restore from another volume backup)
+* volume encryption
+* create volume from the backing image
+* create backing image from volume  
+* volume expansion
+* volume trim
+* volume metrics (bandwidth, latency, IOPS)
+* volume data integrity (snapshot checksum)
+
+SPDK uses a quarterly release cycle, next release will be 23.01 (January 2023). Assuming actual RAID1 implementation will be available in 23.01 release, actually the JSON-RPC we need to implement over SPDK are:
+* suspend I/O operation
+* copy a snapshot over an arbitrary bdev
+* add bdev to raid1 in read balancing disabled mode
+* enable/disable bdev in raid1 read balancing
+* export/import file to/from bdev or implement seek_data/hole in NVMe-oF
+
+The first development is necessary for snapshot, the last one for backup/restore and the other three developments are necessary for replica rebuilding.
+The snapshot copy has already been discussed with SPDK core maintainers, so an upstream development can be made.
+
+
+### Limitations
+Actual RAID1 implementation is not still complete, so actually we have some limitations:
+* read balancing has been developed but is still under review, so it is available only in SPDK Gerrit
+* replica rebuild is still under development, so it isn't available. As a consequence of this, actually RAID1 miss the functionality to add a new base bdev to an existing RAID1 bdev.

enhancements/20230420-engine-identity-validation.md

@@ -110,32 +110,32 @@ in this LEP are backwards compatible. All gRPC metadata validation is by demand
 upgraded) client only injects some metadata (e.g. `volume-name` but not `instance-name`), the server only validates the
 metadata provided.
 
-Add an `instance-name` flag to the `longhorn controller <volume-name>` command (e.g. `longhorn controller <volume-name>
---instance-name <instance-name>`). This command already accepts the volume name, so an additional `volume-name` flag
-would be redundant. The longhorn-engine controller server remembers its volume and instance name.
+Add a global `volume-name` flag and a global `engine-instance-name` flag to the engine CLI (e.g. `longhorn -volume-name
+<volume-name> -engine-instance-name <engine-instance-name> <command> <args>`). Virtually all CLI commands create a
+controller client and these flags allow appropriate gRPC metadata to be injected into every client request. Requests
+that reach the wrong longhorn-engine controller server are rejected.
 
-Add a `volume-name` flag and an `instance-name` flag to the `longhorn replica <directory>` command (e.g. `longhorn
-replica <directory> -volume-name <volume-name> -instance-name <instance-name>`). The longhorn-engine replica server
-remembers its volume and instance name.
+Use the global `engine-instance-name` flag and the pre-existing `volume-name` positional argument to allow the
+longhorn-engine controller server to remember its volume and instance name (e.g. `longhorn -engine-instance-name
+<instance-name> controller <volume-name>`). Ignore the global `volume-name` flag, as it is redundant.
 
-Add a `volume-name` flag and an `instance-name` flag to the `longhorn replica <directory>` command (e.g. `longhorn
-replica <directory> -volume-name <volume-name> -instance-name <instance-name>`). The longhorn-engine sync-agent server
-remembers its volume and instance name.
+Use the global `volume-name` flag or the pre-existing local `volume-name` flag and a new `replica-instance-name` flag to
+allow the longhorn-engine replica server to remember its volume and instance name (e.g. `longhorn -volume-name
+<volume-name> replica <directory> -replica-instance-name <replica-instance-name>`).
 
-Add a `volume-name` and `engine-instance-name` flag to every CLI command that launches an asynchronous task (e.g.
-`longhorn ls-replica -volume-name <volume-name> -engine-instance-name <engine-instance-name>`). All such
-commands create a controller client and these flags allow appropriate gRPC metadata to be injected into every client
-request. Requests that reach the wrong longhorn-engine controller server are rejected.
+Use the global `volume-name` flag and a new `replica-instance-name` flag to allow the longhorn-engine sync-agent server
+to remember its volume and instance name (e.g. `longhorn -volume-name <volume-name> sync-agent -replica-instance-name
+<replica-instance-name>`).
 
 Add an additional `replica-instance-name` flag to CLI commands that launch asynchronous tasks that communicate directly
-with the longhorn-engine replica server (e.g. `longhorn add-replica <address> -size <size> -current-size <current-size>
--volume-name <volume-name> -engine-instance-name <engine-instance-name> -replica-instance-name
-<replica-instance-name>`). All such commands create a replica client and these flags allow appropriate gRPC metadata to
-be injected into every client request. Requests that reach the wrong longhorn-engine replica server are rejected.
+with the longhorn-engine replica server (e.g. `longhorn -volume-name <volume-name> add-replica <address> -size <size>
+ -current-size <current-size> -replica-instance-name <replica-instance-name>`). All such commands create a replica
+client and these flags allow appropriate gRPC metadata to be injected into every client request. Requests that reach the
+wrong longhorn-engine replica server are rejected.
 
-Return 5 NOT FOUND with an appropriate message when metadata validation fails. In this case, the NOT_FOUND refers to the
-longhorn-engine component the caller was actually attempting to communicate with. (The particular return code is
-definitely open to discussion.)
+Return 9 FAILED_PRECONDITION with an appropriate message when metadata validation fails. This code is chosen in
+accordance with the [RPC API](https://grpc.github.io/grpc/core/md_doc_statuscodes.html), which instructs developers to
+use FAILED_PRECONDITION if the client should not retry until the system system has been explicitly fixed.
 
 #### Longhorn-Instance-Manager
 
@@ -259,7 +259,7 @@ func ReplicaCmd() cli.Command {
 				Usage: "Name of the volume (for validation purposes)",
 			},
 			cli.StringFlag{
-				Name:  "instance-name",
+				Name:  "replica-instance-name",
 				Value: "",
 				Usage: "Name of the instance (for validation purposes)",
 			},
@@ -318,8 +318,9 @@ validation.
 
 #### Longhorn-Manager Integration
 
-Ensure the engine and replica controllers launch engine and replica processes with `--volume-name` and `--instance-name`
-flags so that these processes can validate identifying gRPC metadata coming from requests.
+Ensure the engine and replica controllers launch engine and replica processes with `-volume-name` and
+`-engine-instance-name` or `-replica-instance-name` flags so that these processes can validate identifying gRPC metadata
+coming from requests.
 
 Ensure the engine controller supplies correct information to the ProxyEngineService client functions so that identity
 validation can occur in the lower layers.
@@ -328,6 +329,7 @@ validation can occur in the lower layers.
 
 This issue/LEP was inspired by [longhorn/longhorn#5709](https://github.com/longhorn/longhorn/issues/5709). In the
 situation described in this issue:
+
 1. An engine controller with out-of-date information (including a replica address the associated volume does not own)
    [issues a ReplicaAdd
    command](https://github.com/longhorn/longhorn-manager/blob/a7dd20cdbdb1a3cea4eb7490f14d94d2b0ef273a/controller/engine_controller.go#L1819)
@@ -339,8 +341,9 @@ situation described in this issue:
    is used to expand the replica before a followup failure to actually add the replica to the controller's backend.
 
 After this improvement, the above scenario will be impossible:
-1. Both the engine and replica controllers will launch engine and replica processes with the `--volume-name` and
-   `--instance-name` flags.
+
+1. Both the engine and replica controllers will launch engine and replica processes with the `-volume-name` and
+   `-engine-instance-name` or `replica-instance-name` flags.
 2. When the engine controller issues a ReplicaAdd command, it will do so using the expanded embedded
    `ProxyEngineRequest` message (with `volume_name` and `instance_name` fields) and an additional
    `replica_instance_name` field.
@@ -353,9 +356,10 @@ After this improvement, the above scenario will be impossible:
 
 ### Test plan
 
-TODO: Integration test plan. 
+#### TODO: Integration Test Plan
 
 In my test environment, I have experimented with:
+
 - Running new versions of all components, making gRPC calls to the longhorn-engine controller and replica processes with
   wrong gRPC metadata, and verifying that these calls fail.
 - Running new versions of all components, making gRPC calls to instance-manager with an incorrect volume-name or
@@ -367,10 +371,30 @@ This is really a better fit for a negative testing scenario (do something that w
 communication, then verify that communication fails), but we have already eliminated the only known recreate for
 [longhorn/longhorn#5709](https://github.com/longhorn/longhorn/issues/5709).
 
-TODO: Engine integration test plan.
+#### Engine Integration Test Plan
 
-I'm unfamiliar with our engine integration tests, but it should be fairly easy to create negative tests that verify the
-expected behavior.
+Rework test fixtures so that:
+
+- All controller and replica processes are created with the information needed for identity validation.
+- It is convenient to create controller and replica clients with the information needed for identity validation.
+- gRPC metadata is automatically injected into controller and replica client requests when clients have the necessary
+  information.
+
+Do not modify the behavior of existing tests. Since these tests were using clients with identity validation information,
+no identity validation is performed.
+
+- Modify functions/fixtures that create engine/replica processes to allow the new flags to be passed, but do not pass
+  them by default.
+- Modify engine/replica clients used by tests to allow for metadata injection, but do not enable it by default.
+
+Create new tests that:
+
+- Ensure validation fails when a directly created client attempts to communicate with a controller or replica server
+  using the wrong identity validation information.
+- Ensure validation fails when an indirectly created client (by the engine) tries to communicate with a replica server
+  using the wrong identity validation information.
+- Ensure validation fails when an indirectly created client (by a CLI command) tries to communicate with a controller or
+  replica server using the wrong identity validation information.
 
 ### Upgrade strategy
 
@@ -382,7 +406,8 @@ supported version (as governed by the `CLIAPIVersion`). Even if other components
 metadata to non-upgraded processes, it will be ignored.
 
 We will only populate extra ProxyEngineService fields when longhorn-manager is running with an update ProxyEngineService
-client. 
+client.
+
 - RPCs from an old client to a new ProxyEngineService server will succeed, but without the extra fields,
   instance-manager will have no useful gRPC metadata to inject into its longhorn-engine requests.
 - RPCs from a new client to an old ProxyEngineService will succeed, but instance-manager will ignore the new fields and

enhancements/20230523-support-spdk-volumes.md

@@ -0,0 +1,146 @@
+# Support Volumes using V2 Data Engine
+
+## Summary
+
+Longhorn's storage stack, based on iSCSI and a customized protocol, has limitations such as increased I/O latencies and reduced IOPS due to the longer data path. This makes it less suitable for latency-critical applications. To overcome these challenges, Longhorn introduces the Storage Performance Development Kit (SPDK) to enhance overall performance. With SPDK integration, Longhorn optimizes system efficiency, addresses latency concerns, and provides a high-performance storage solution capable of meeting diverse workload demands.
+
+### Related Issues
+
+- [[FEATURE] Add a global setting for enabling and disabling SPDK feature](https://github.com/longhorn/longhorn/issues/5778)
+- [[FEATURE] Support replica scheduling for SPDK volume](https://github.com/longhorn/longhorn/issues/5711)
+- [[FEATURE] Implement Disk gRPC Service in Instance Manager for collecting SPDK disk statistics from SPDK gRPC service](https://github.com/longhorn/longhorn/issues/5744)
+- [[FEATURE] Identify and manage orphaned lvols and raid bdevs if the associated Volume resources are not existing](https://github.com/longhorn/longhorn/issues/5827)
+
+## Motivation
+
+### Goals
+
+- Introduce backend store drivers
+    - `v1`: legacy data path
+    - `v2`: a newly introduced data path based on SPDK
+- Introduce disk types and management
+- Support volume creation, attachment, detachment and deletion
+- Support orphaned replica collection
+
+### Non-goals [optional]
+
+- Support runtime replica rebuilding
+- Support changing number of replicas of a volume
+- Support volume expansion
+- Support volume backup
+
+## Proposal
+
+### User Stories
+
+Longhorn's storage stack is built upon iSCSI and a customized protocol. However, the longer data path associated with this architecture introduces certain limitations, resulting in increased I/O latencies and reduced IOPS. Consequently, Longhorn may not be the ideal choice for latency-critical applications, as the performance constraints could impede their deployment on the platform.
+
+By incorporating SPDK, Longhorn leverages its capabilities to significantly improve performance levels. The integration of SPDK enables Longhorn to optimize system efficiency, mitigate latency concerns, and deliver a high-performance storage solution that can better meet the demands of diverse workloads.
+
+### User Experience In Detail
+
+- Environment Setup
+    - Configure Kernel Modules (uio and uio_pci_generic) and Huge Pages for SPDK
+        
+        ```bash
+        kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/master/deploy/prerequisite/longhorn-spdk-setup.yaml
+        ```
+        
+    - Install NVMe Userspace Tool and Load `nvme-tcp` Kernel Module
+        
+        nvme-cli on each node and make sure that the version of nvme-cli is equal to or greater than version `1.12` .
+        
+        ```bash
+        kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/master/deploy/prerequisite/longhorn-nvme-cli-installation.yaml
+        ```
+        
+    - Restart `kubelet`
+        
+        Modifying the Huge Page configuration of a node requires either a restart of kubelet or a complete reboot of the node. This step is crucial to ensure that the changes take effect and are properly applied.
+        
+- Install Longhorn system
+- Enable SPDK Support
+    
+    Enable the SPDK feature by changing the `v2-data-engine` setting to `true` after installation. Following this, the instance-manager pods shall be automatically restarted.
+    
+- Add Disks for volumes using v2 data engine
+    - Legacy disks are classified as `filesystem`-type disks
+    - Add one or multiple `block`-type disks into `node.Spec.Disks`
+        
+        ```bash
+        block-disk-example1:
+          allowScheduling: true
+          evictionRequested: false
+          path: /path/to/block/device
+          storageReserved: 0
+          tags: []
+          diskType: block
+        ```
+        
+- Create a storage class utilizing the enhanced performance capabilities offered by SPDK
+    
+    ```bash
+    kind: StorageClass
+    apiVersion: storage.k8s.io/v1
+    metadata:
+      name: longhorn-v2-data-engine
+    provisioner: driver.longhorn.io
+    allowVolumeExpansion: true
+    reclaimPolicy: Delete
+    volumeBindingMode: Immediate
+    parameters:
+      numberOfReplicas: "2"
+      staleReplicaTimeout: "2880"
+      fromBackup: ""
+      fsType: "ext4"
+      backendStoreDriver: "v2"
+    ```
+    
+- Create workloads that use Longhorn volumes provisioning based on the storage class.
+
+### API changes
+
+
+## Design
+
+### Implementation Overview
+
+- Global settings
+    - `v2-data-engine`: This setting allows users to enable v2 data engine support. Default: false.
+    - `v2-data-engine-hugepage-limit`: This setting allows users to specify the 2 MiB hugepage size for v2 data engine. Default: 2048.
+- CRD
+    - Introduce `diskType` in `node.Spec.Disks`
+        - `filesystem`: disks for legacy volumes. These disks, which are actually directories, store and organize data in a hierarchical manner.
+        - `block`: block disks for volumes using v2 data engine
+        
+        The replica scheduler assigns replicas of legacy volumes to `filesystem`-type disks while replicas of volumes using v2 data engine are scheduled to `block`-type disks.
+        
+    - Introduce `backendStoreDriver` in `volume.Spec`, `engine.Spec` and `replica.Spec`.
+        - `backendStoreDriver` is utilized to differentiate between volume types and their associated data paths.
+- Introduce `Instance`, `Disk` and `SPDK` gRPC services
+    
+     ![gRPC Services](image/spdk_services.png)
+    
+    - `Instance` gRPC service: It is tasked with managing various operations related to instance management, including creation, deletion, retrieval, listing, and watching. An instance, either an engine or a replica of a legacy volume, represents a process. On the other hand, for replicas of volumes using v2 data engine, an instance represents a logical volume. In the case of an engine for an volume using v2 data engine, an instance is associated with a raid bdev, a frontend NVMe target/initiator pair and a bind mount device.
+
+    - `Disk` gRPC service: It is responsible for managing various disk operations, including creation, deletion, and retrieval. Additionally, it provides functionalities to list or delete replica instances associated with the disks. In the case of a legacy volume, a replica instance is represented as a replica directory on the disk. On the other hand, for an volume using v2 data engine, a replica instance is a replica chained by logical volumes.
+
+    - `SPDK` gRPC service: It manages replicas chained by logical volumes and engines constructed using SPDK raid1 bdevs. In addition, the service is responsible for the communication with `spdk_tgt`.
+
+- Proxy gRPC service APIs
+    - Update gRPC service APIs for support different disk type, filesystem and block, and data engines, v1 and v2.
+
+- Disk orchestration
+    
+    Within the Longhorn system, an aio bdev and an lvstore are created on top of a block-type disk. Replicas in terms of logical volumes (lvols) are then created on the lvstore.
+    
+    ![Disks for Volumes Using V2 Data Engine](image/spdk_disks.png)
+    
+- Orphaned replicas collection
+
+    The features have been integrated into the existing framework for collecting and cleaning up orphaned replicas.
+
+## Test Plan
+
+
+## Note [optional]

enhancements/20230526-volume-backup-policy-for-longhorn-system-backup.md

@@ -4,7 +4,7 @@ The current implementation of the Longhorn system backup lacks integration with
 
 ## Summary
 
-This document proposed to include the volume backup feature in the Longhorn system backup by introducing volume backup policies. 
+This document proposed to include the volume backup feature in the Longhorn system backup by introducing volume backup policies.
 
 By implementing the volume backup policies, users will gain the ability to define how volume data should be backed up during the Longhorn system backup.
 
@@ -30,7 +30,7 @@ Overall, the proposed volume backup policies aim to improve the Longhorn system
 
 1. When volume backup policy is specified:
    - `if-not-present`: Longhorn will create a backup for volumes that do not have an existing backup.
-   - `alway`: Longhorn will create a backup for all volumes, regardless of their existing backups.
+   - `always`: Longhorn will create a backup for all volumes, regardless of their existing backups.
    - `disabled`: Longhorn will not create any backups for volumes.
 1. If a volume backup policy is not specified, the policy will be automatically set to `if-not-present`. This ensures that volumes without any existing backups will be backed up during the Longhorn system backup.
 
@@ -50,7 +50,7 @@ In cases where I don't explicitly specify the volume backup policy during the sy
 
 To set the volume backup policy, users can set the volume backup policy when creating the system backup through the UI. Alternatively, users can specify it in the manifest when creating the SystemBackup custom resource using the kubectl command.
 
-In scenarios where no specific volume backup policy is provided, Longhorn will automatically set the policy as `if-not-present`. 
+In scenarios where no specific volume backup policy is provided, Longhorn will automatically set the policy as `if-not-present`.
 
 ### API changes
 
@@ -75,7 +75,7 @@ Add a new `volumeBackupPolicy` field to the HTTP request and response payload.
 
 #### Mutate empty volume backup policy
 
-When the volume backup policy is not provided in the SystemBackup custom resource, automatically set the policy to `if-not-present`. 
+When the volume backup policy is not provided in the SystemBackup custom resource, automatically set the policy to `if-not-present`.
 
 ### Test plan
 

enhancements/20230616-automatic-offline-replica-rebuild.md

@@ -0,0 +1,86 @@
+# Automatic Offline Replica Rebuilding
+
+## Summary
+
+Currently, Longhorn does not have the capability to support online replica rebuilding for volumes utilizing the V2 Data Engine. However, an automatic offline replica rebuilding mechanism has been implemented as a solution to address this limitation.
+
+### Related Issues
+
+https://github.com/longhorn/longhorn/issues/6071
+
+## Motivation
+
+### Goals
+
+1. Support volumes using v2 data engine
+
+### Non-goals
+
+2. Support volumes using v1 data engine
+
+## Proposal
+
+## User Stories
+
+
+In the event of abnormal power outages or network partitions, replicas of a volume may be lost, resulting in volume degradation. Unfortunately, volumes utilizing the v2 data engine do not currently have the capability for online replica rebuilding. As a solution to address this limitation, Longhorn has implemented an automatic offline replica rebuilding mechanism.
+
+When a degraded volume is detached, this mechanism places the volume in maintenance mode and initiates the rebuilding process. After the rebuilding is successfully completed, the volume is detached according to the user's specified expectations.
+
+### User Experience In Details
+
+- If a volume using the v2 data engine is degraded, the online replica rebuilding process is currently unsupported.
+
+- If offline replica rebuilding feature is enabled when one of the conditions is met
+    - Global setting `offline-replica-rebuild` is `enabled` and `Volume.Spec.OfflineReplicaRebuilding` is  `ignored`
+    - `Volume.Spec.OfflineReplicaRebuilding` is `enabled`
+    The volume's `Spec.OfflineReplicaRebuildingRequired` is set to `true` if a volume is degraded.
+
+- When a degraded volume is detached, this mechanism places the volume in maintenance mode and initiates the rebuilding process. After the rebuilding is successfully completed, the volume is detached according to the user's specified expectations.
+
+- If a user attaches the volume without enabling maintenance mode while the replica rebuilding process is in progress, the ongoing replica rebuilding operation will be terminated. 
+
+## Design
+
+### Implementation Overview
+
+**Settings**
+
+- Add global setting `offline-replica-rebuilding`. Default value is `enabled`. The available options are:
+  - `enabled`
+  - `disable`
+
+**CRD**
+
+- Add `Volume.Spec.OfflineReplicaRebuilding`. The available options are:
+  - ignored`: The volume's offline replica rebuilding behavior follows the settings defined by the global setting `offline-replica-rebuilding`.
+  - `enabled`: Offline replica rebuilding of the volume is always enabled.
+  - `disabled`: Offline replica rebuilding of the volume is always disabled.
+
+- Add `Volume.Status.OfflineReplicaRebuildingRequired`
+
+**Controller**
+
+- Add `volume-rebuilding-controller` for creating and deleting `volume-rebuilding-controller` attachment ticket.
+
+**Logics**
+
+1. A volume-controller sets 'Volume.Status.OfflineReplicaRequired' to 'true' when it realizes a v2 data engine is degraded.
+
+2. If a volume's `Volume.Status.OfflineReplicaRebuildingRequired` is `true`, volume-rebuilding-controller creates a `volume-rebuilding-controller` attachment ticket with frontend disabled and lower priority than tickets with workloads.
+
+3. When the volume is detached, volume-attachment-controller attaches the volume with a `volume-rebuilding-controller` attachment ticket in maintenance mode.
+
+4. volume-controller triggers replica rebuilding.
+
+5. After finishing the replica rebuilding, the volume-controller sets `Volume.Status.OfflineReplicaRebuildingRequired` to `false` if a number of healthy replicas is expected.
+
+6. volume-rebuilding-controller deletes the 'volume-rebuilding-controller' attachment ticket.
+
+7. volume-attachment-controller is aware of the deletion of the `volume-rebuilding-controller` attachment ticket, which causes volume detachment.
+
+### Test Plan
+
+### Integration Tests
+
+1. Degraded Volume lifecycle (creation, attachment, detachment and deletion) and automatic replica rebuilding

enhancements/20230619-spdk-engine.md

@@ -0,0 +1,100 @@
+# SPDK Engine
+
+## Summary
+Longhorn will take advantage of SPDK to launch the second version engine with higher performance.
+
+### Related Issues
+https://github.com/longhorn/longhorn/issues/5406
+https://github.com/longhorn/longhorn/issues/5282
+https://github.com/longhorn/longhorn/issues/5751
+
+## Motivation
+### Goals
+1. Have a set of APIs that talks with spdk_tgt to operate SPDK components.
+2. Launch a control panel that manage and operate SPDK engines and replica.
+
+## Proposal
+1. The SPDK engine architecture is different from the legacy engine: 
+   1. Unlike the legacy engine, the data flow will be taken over by SPDK. The new engine or replica won't directly touch the data handling. The new engine or replica is actually one or a set of SPDK components handled by spdk_tgt.
+   2. Since the main task is to manage SPDK components and abstract them as Longhorn engines or replicas, we can use a single service rather than separate processes to launch and manage engine or replicas.
+   3. As SPDK handles the disks by itself, the disk management logic should be moved to SPDK engine service as well. 
+2. The abstraction of SPDK engine and replica:
+   1. A data disk will be abstracted as an aio bdev + a lvstore. 
+   2. Each snapshot or volume head file is a logical volume (lvol) inside a lvstore.
+   3. A remote replica is finally exposed as a NVMe-oF subsystem, in which the corresponding SPDK lvol stand behind. While a local replica is just a lvol.
+   4. An engine backend is actually a SPDK RAID1 bdev, which may consist of multiple attached replica NVMe-oF subsystems and local lvol.
+   5. An engine frontend is typically a NVMe-oF initiator plus a NVMe-oF subsystem of the RAID bdev.
+3. Do spdk_tgt initializations during instance manager startup.
+
+
+### User Stories
+#### Launch SPDK volumes
+Before the enhancement, users need to launch a RAID1 bdev then expose it as a NVMe-oF initiator as the Longhorn SPDK engine manually by following [the doc](https://github.com/longhorn/longhorn-spdk-engine/wiki/How-to-setup-a-RAID1-block-device-with-SPDK). Besides, rebuilding replicas would be pretty complicated.
+
+After the enhancement, users can directly launch and control Longhorn SPDK engine via the gRPC SPDK engine service. And the rebuilding can be triggered and handled automatically.
+
+### API Changes
+- The new gRPC SPDK engine service:
+  - Replica:
+    | API | Caller | Input | Output | Comments |
+    | --- | --- | --- | --- | --- |
+    | Create | Instance manager proxy | name, lvsName, lvsUUID string, specSize uint64, exposeRequired bool | err error | Create a new replica or start an existing one |
+    | Delete | Instance manager proxy | name string, cleanupRequired bool | err error | Remove or stop an existing replica |
+    | List   | Instance manager proxy |  | replicas map\[string\]Replica, err error | Get all abstracted replica info from the cache of the SPDK engine service |
+    | Get    | Instance manager proxy |  | replica Replica, err error | Get the abstracted replica info from the cache of the SPDK engine service |
+    | Watch  | Instance manager proxy |  | ReplicaStream, err error | Establish a streaming for the replica update notification |
+    | SnapshotCreate | Instance manager proxy | name, snapshotName string | err error | |
+    | SnapshotDelete | Instance manager proxy | name, snapshotName string | err error | |
+    | Rebuilding APIs | The engine inside one gRPC SPDK engine service | | | This set of APIs is responsible for starting and finishing the rebuilding for source replica or destination replica. And it help start data transmission from src to dst |
+  - Engine:
+    | API | Caller | Input | Output | Comments |
+    | --- | --- | --- | --- | --- |
+    | Create | Instance manager proxy | name, lvsName, lvsUUID string, specSize uint64, exposeRequired bool | err error | Start a new engine and connect it with corresponding replicas |
+    | Delete | Instance manager proxy | name string, cleanupRequired bool | err error | Stop an existing engine |
+    | List   | Instance manager proxy |  | engines map\[string\]Engine, err error | Get the abstracted engine info from the cache of the SPDK engine service |
+    | Get    | Instance manager proxy |  | engine Engine, err error | Get the abstracted engine info from the cache of the SPDK engine service |
+    | Watch  | Instance manager proxy |  | EngineStream, err error | Establish a streaming for the engine update notification |
+    | SnapshotCreate | Instance manager proxy | name, snapshotName string | err error | |
+    | SnapshotDelete | Instance manager proxy | name, snapshotName string | err error | |
+    | ReplicaAdd    | Instance manager proxy | engineName, replicaName, replicaAddress string | err error | Find a healthy RW replica as source replica then rebuild the destination replica. To rebuild a replica, the engine will call rebuilding start and finish APIs for both replicas and launch data transmission |
+    | ReplicaDelete | Instance manager proxy | engineName, replicaName, replicaAddress string | err error | Remove a replica from the engine |
+  - Disk:
+    | API | Caller | Input | Output | Comments |
+    | --- | --- | --- | --- | --- |
+    | Create | Instance manager proxy | diskName, diskUUID, diskPath string, blockSize int64 | disk Disk, err error | Use the specified block device as blob store |
+    | Delete | Instance manager proxy | diskName, diskUUID string | err error | Remove a store from spdk_tgt |
+    | Get    | Instance manager proxy | diskName string | disk Disk, err error | Detect the store status and get the abstracted disk info from spdk_tgt |
+
+## Design
+### Implementation Overview
+#### [Go SPDK Helper](https://github.com/longhorn/go-spdk-helper):
+- The SPDK Target is exposed as a [JSON-RPC service](https://spdk.io/doc/jsonrpc.html). 
+- Instead of using the existing sample python script [rpc_http_proxy](https://spdk.io/doc/jsonrpc_proxy.html), we will have a helper repo similar to [longhorn/go-iscsi-helper](https://github.com/longhorn/go-iscsi-helper) to talk with spdk_tgt over Unix domain socket `/var/tmp/spdk.sock`..
+  - The SPDK target config and launching. Then live upgrade, and shutdown if necessary/possible.
+  - The JSON RPC client that directly talks with spdk_tgt.
+  - The exposed Golang SPDK component operating APIs. e.g., lvstore, lvol, RAID creation, deletion, and list.
+  - The NVMe initiator handling Golang APIs (for the engine frontend).
+
+#### [SPDK Engine](https://github.com/longhorn/go-spdk-helper):
+- Launch a gRPC server as the control panel.
+- Have a goroutine that periodically check and update engine/replica caches.
+- Implement the engine/replica/disk APIs listed above.
+- Notify upper layers about the engine/replica update via streaming.
+
+#### Instance Manager:
+- Start spdk_tgt on demand.
+- Update the proxy service so that it forwards SPDK engine/replica requests to the gRPC service.
+
+### Test Plan
+#### Integration tests
+1. Starting and stopping related tests: If Longhorn can start or stop one engine + multiple replicas correctly. 
+2. Basic IO tests: If Data can be r/w correctly. And if data still exists after restart.
+3. Basic snapshot tests: If snapshots can be created and keeps identical among all replicas. If a snapshot can be deleted from all replicas. If snapshot revert work.
+
+#### Manual tests
+1. SPDK volume creation/deletion/attachment/detachment tests.
+2. Basic IO tests: If Data can be r/w correctly when volume is degraded or healthy. And if data still exists after restart.
+3. Basic offline rebuilding tests.
+
+### Upgrade strategy
+This is an experimental engine. We do not need to consider the upgrade or compatibility issues now.

enhancements/20230718-disk-anti-affinity.md

@@ -0,0 +1,155 @@
+# Disk Anti-Affinity
+
+## Summary
+
+Longhorn supports multiple disks per node, but there is currently no way to ensure that two replicas for the same
+volume that schedule to the same node end up on different disks. In fact, the replica scheduler currently doesn't make
+any attempt achieve this goal, even when it is possible to do so.
+
+With the addition of a Disk Anti-Affinity feature, the Longhorn replica scheduler will attempt to schedule two replicas
+for the same volume to different disks when possible. Optionally, the scheduler will refuse to schedule a replica to a
+disk that has another replica for the same volume.
+
+Although the comparison is not perfect, this enhancement can be thought of as enabling RAID 1 for Longhorn (mirroring
+across multiple disks on the same node).
+
+See the [Motivation section](#motivation) for potential benefits.
+
+### Related Issues
+
+- https://github.com/longhorn/longhorn/issues/3823
+- https://github.com/longhorn/longhorn/issues/5149
+
+### Existing Related Features
+
+#### Replica Node Level Soft Anti-Affinity
+
+Disabled by default. When disabled, prevents the scheduling of a replica to a node with an existing healthy replica of
+the same volume.
+
+Can also be set at the volume level to override the global default.
+
+#### Replica Zone Level Soft Anti-Affinity
+
+Enabled by default. When disabled, prevents the scheduling of a replica to a zone with an existing healthy replica of
+the same volume.
+
+Can also be set at the volume level to override the global default.
+
+## Motivation
+
+- Large, multi-node clusters will likely not benefit from this enhancement.
+- Single-node clusters and small, multi-node clusters (on which the number of replicas per volume exceeds the number
+  of available nodes) will experience:
+  - Increased data durability. If a single disk fails, a healthy replica will still exist on an disk that
+    has not failed.
+  - Increased data availability. If a single disk on a node becomes unavailable, but the node itself remains
+    healthy, at least one replica remains healthy. On a single-node cluster, this can directly prevent a volume crash.
+    On a small, multi-node cluster, this can prevent a future volume crash due to the loss of a different node.
+
+### Goals
+
+- In all situations, the Longhorn replica scheduler will make a best effort to ensure two replicas for the same volume
+  do not schedule to the same disk.
+- Optionally, the scheduler will refuse to schedule a replica to a disk that has another replica of the same volume.
+
+## Proposal
+
+### User Stories
+
+#### Story 1
+
+My cluster consists of a single node with multiple attached SSDs. When I create any new volume, I want replicas to
+distribute across these disks so that I can recover from n - 1 disk failures. If there are not as many available disks
+as desired replicas, I want Longhorn to do the best it can.
+
+#### Story 2
+
+My cluster consists of a single node with multiple attached SSDs. When I create any new volume, I want replicas to
+distribute across these disks so that I can recover from n - 1 disk failure. If there are not as many available disks
+as desired replicas, I want scheduling to fail obviously. It is important that I know my volumes aren't being protected
+so I can take action.
+
+#### Story 3
+
+My cluster consists of a single node with multiple attached SSDs. When I create a specific, high-priority volume, I want
+replicas to distribute across these disks so that I can recover from n - 1 disk failure. If there are not as many
+available disks as desired replicas, I want scheduling to fail obviously. It is important that I know high-priority
+volume isn't being protected so I can take action.
+
+### User Experience In Detail
+
+### API changes
+
+Introduce a new Replica Disk Level Soft Anti-Affinity setting with the following definition. By default, set it to
+`true`. While it is generally desirable to schedule replicas to different disks, it would break with existing behavior
+to refuse to schedule replicas when different disks are not available.
+
+```golang
+SettingDefinitionReplicaDiskSoftAntiAffinity = SettingDefinition{
+    DisplayName: "Replica Disk Level Soft Anti-Affinity",
+    Description: "Allow scheduling on disks with existing healthy replicas of the same volume",
+    Category:    SettingCategoryScheduling,
+    Type:        SettingTypeBool,
+    Required:    true,
+    ReadOnly:    false,
+    Default:     "true",
+}
+```
+
+Introduce a new `spec.replicaDiskSoftAntiAffinity` volume field. By default, set it to `ignored`. Similar to the
+existing `spec.replicaSoftAntiAffinity` and `spec.replicaSoftZoneAntiAffinityFields`, override the global setting if
+this field is set to `enabled` or `disabled`.
+
+```yaml
+replicaDiskSoftAntiAffinity:
+  description: Replica disk soft anti affinity of the volume. Set enabled
+    to allow replicas to be scheduled in the same disk.
+  enum:
+  - ignored
+  - enabled
+  - disabled
+  type: string
+```
+
+## Design
+
+### Implementation Overview
+
+The current replica scheduler does the following:
+
+1. Determines which nodes a replica can be scheduled to based on node condition and the `ReplicaSoftAntiAffinity` and
+   `ReplicaZoneSoftAntiAffinity` settings.
+1. Creates a list of all schedulable disks on these nodes.
+1. Chooses the disk with the most available space for scheduling.
+
+Add a step so that the replica scheduler:
+
+1. Determines which nodes a replica can be scheduled to based on node condition and the `ReplicaSoftAntiAffinity` and
+   `ReplicaZoneSoftAntiAffinity` settings.
+1. Creates a list of all schedulable disks on these nodes.
+1. Filters the list to include only disks with the least number of existing matching replicas and optionally only disks
+   with no existing matching replicas.
+1. Chooses the disk from the filtered list with the most available space for scheduling.
+
+### Test plan
+
+Minimally implement two new test cases:
+
+1. In a cluster that includes nodes with multiple available disks, create a volume with
+   `spec.replicaSoftAntiAffinity = true`, `spec.replicaDiskSoftAntiAffinity = true`, and `numberOfReplicas` equal to the
+   total number of disks in the cluster. Confirm that each replica schedules to a different disk. It may be necessary
+   to tweak additional factors. For example, ensure that one disk has enough free space that the old scheduling
+   behavior would assign two replicas to it instead of distributing the replicas evenly among the disks.
+1. In a cluster that includes nodes with multiple available disks, create a volume with
+   `spec.replicaSoftAntiAffinity = true`, `spec.replicaDiskSoftAntiAffinity = false`, and `numberOfReplicas` equal to
+   one more than the total number of disks in the cluster. Confirm that a replica fails to schedule. Previously,
+   multiple replicas would have scheduled to the same disk and no error would have occurred.
+
+### Upgrade strategy
+
+The Replica Disk Level Soft Anti-Affinity setting defaults to `true` to maintain backwards compatibility. It if is set
+to `false``, new replicas that require scheduling will follow the new behavior.
+
+The `spec.replicaDiskSoftAntiAffinity` volume field defaults to `ignored` to maintain backwards compatibility. If it is
+set to `enabled` on a volume, new replicas for that volume that require scheduling will follow the new behavior.

enhancements/20230807-backingimage-backup-support.md

@@ -0,0 +1,267 @@
+# BackingImage Backup Support
+
+## Summary
+This feature enables Longhorn to backup the BackingImage to backup store and restore it.
+
+### Related Issues
+
+- [FEATURE] Restore BackingImage for BackupVolume in a new cluster [#4165](https://github.com/longhorn/longhorn/issues/4165)
+
+## Motivation
+
+### Goals
+
+- When a Volume with a BackingImage being backed up, the BackingImage will also be backed up.
+- User can manually back up the BackingImage.
+- When restoring a Volume with a BackingImage, the BackingImage will also be restored.
+- User can manually restore the BackingImage.
+- All BackingImages are backed up in blocks.
+- If the block contains the same data, BackingImages will reuse the same block in backup store instead of uploading another identical one.
+
+
+## Proposal
+
+### User Stories
+
+With this feature, there is no need for user to manually handle BackingImage across cluster when backing up and restoring the Volumes with BackingImages.
+
+### User Experience In Detail
+
+Before this feature:
+The BackingImage will not be backed up automatically when backing up a Volume with the BackingImage. So the user needs to prepare the BackingImage again in another cluster before restoring the Volume back.
+
+After this feature:
+A BackingImage will be backed up automatically when a Volume with the BackingImage is being backed up. User can also manually back up a BackingImage independently.
+Then, when the Volume with the BackingImage is being restored from backup store, Longhorn will restore the BackingImage at the same time automatically. User can also manually restore the BackingImage independently.
+
+This improve the user experience and reduce the operation overhead.
+
+
+## Design
+
+### Implementation Overview
+
+#### Backup BackingImage - BackupStore
+
+- Backup `BackingImage` is not the same as backup `Volume` which consists of a series of `Snapshots`. Instead, a `BackingImage` already has all the blocks we need to backup. Therefore, we don't need to find the delta between two `BackingImages` like what we do for`Snapshots` which delta might exist in other `Snapshots` between the current `Snapshot` and the last backup `Snapshot`.
+- All the `BackingImages` share the same block pools in backup store, so we can reuse the blocks to increase the backup speed and save the space. This can happen when user create v1 `BackingImage`, use the image to add more data and then export another v2 `BackingImage`.
+- For restoration, we still restore fully on one of the ready disk.
+- Different from `Volume` backup, `BackingImage` does not have any size limit. It can be less than 2MB or not a multiple of 2MB. Thus, the last block might not be 2MB.
+
+- When backing up `BackingImage`
+    1. `preload()`: the BackingImage to get the all the sectors that have data inside.
+    2. `createBackupBackingMapping()`: to get all the blocks we need to backup
+        - Block: offset + size (2MB for each block, last block might less than 2MB)
+    3. `backupMappings()`: write the block to the backup store
+        - if the block is already in the backup store, skip it.
+    4. `saveBackupBacking()`: save the metadata of the `BackupBackingImage` including the block mapping to the backup store. Mapping needs to include block size.
+
+- When restoring `BackingImage`
+    - `loadBackupBacking()`: load the metadata of the `BackupBackingImage` from the backup store
+    - `populateBlocksForFullRestore() + restoreBlocks()`: based on the mapping, write the block data to the correct offset.
+
+- We backup the blocks in async way to increase the backup speed.
+- For qcow2 `BackingImage`, the format is not the same as raw file, we can't detect the hole and the data sector. So we back up all the blocks.
+
+
+#### Backup BackingImage - Controller
+
+1. Add a new CRD `backupbackingimage.longhorn.io`
+    ```go
+    type BackupBackingImageSpec struct {
+        Labels           map[string]string `json:"labels"`
+        BackingImageName string            `json:"backingImageName"`
+        SyncRequestedAt  metav1.Time       `json:"syncRequestedAt"`
+    }
+
+    type BackupBackingImageStatus struct {
+        OwnerID           string                  `json:"ownerID"`
+        Checksum          string                  `json:"checksum"`
+        URL               string                  `json:"url"`
+        Size              string                  `json:"size"`
+        Labels            map[string]string       `json:"labels"`
+        State             BackupBackingImageState `json:"state"`
+        Progress          int                     `json:"progress"`
+        Error             string                  `json:"error,omitempty"`
+        Messages          map[string]string       `json:"messages"`
+        ManagerAddress    string                  `json:"managerAddress"`
+        BackupCreatedAt   string                  `json:"backupCreatedAt"`
+        LastSyncedAt      metav1.Time             `json:"lastSyncedAt"`
+        CompressionMethod BackupCompressionMethod `json:"compressionMethod"`
+    }
+    ```
+    ```go
+    type BackupBackingImageState string
+
+    const (
+        BackupBackingImageStateNew        = BackupBackingImageState("")
+        BackupBackingImageStatePending    = BackupBackingImageState("Pending")
+        BackupBackingImageStateInProgress = BackupBackingImageState("InProgress")
+        BackupBackingImageStateCompleted  = BackupBackingImageState("Completed")
+        BackupBackingImageStateError      = BackupBackingImageState("Error")
+        BackupBackingImageStateUnknown    = BackupBackingImageState("Unknown")
+    )
+    ```
+    - Field `Spec.ManagerAddress` indicates the address of the backing-image-manager running BackingImage backup.
+    - Field `Status.Checksum` records the checksum of the BackingImage. Users may create a new BackingImage with the same name but different content after deleting an old one or there is another BackingImage with the same name in another cluster. To avoid the confliction, we use checksum to check if they are the same.
+    - If cluster already has the `BackingImage` with the same name as in the backup store, we still create the `BackupBackingImage` CR. User can use the checksum to check if they are the same. Therefore we don't use `UUID` across cluster since user might already prepare the same BackingImage with the same name and content in another cluster.
+
+2. Add a new controller `BackupBackingImageController`.
+    - Workflow
+        - Check and update the ownership.
+        - Do cleanup if the deletion timestamp is set.
+            - Cleanup the backup `BackingImage` on backup store
+            - Stop the monitoring
+        - If `Status.LastSyncedAt.IsZero() && Spec.BackingImageName != ""` means **it is created by the User/API layer**, we need to do the backup
+            - Start the monitor
+            - Pick one `BackingImageManager`
+            - Request `BackingImageManager` to backup the `BackingImage` by calling `CreateBackup()` grpc
+        - Else it means the `BackupBackingImage` CR is created by `BackupTargetController` and the backup `BackingImage` already exists in the remote backup target before the CR creation.
+            - Use `backupTargetClient` to get the info of the backup `BackingImage`
+            - Sync the status
+
+3. In `BackingImageManager - manager(backing_image.go)`
+    - Implement `CreateBackup()` grpc
+        - Backup `BackingImage` to backup store in blocks
+
+4. In controller `BackupTargetController`
+    - Workflow
+        - Implement `syncBackupBackingImage()` function
+            - Create the `BackupBackingImage` CRs whose name are in the backup store but not in the cluster
+            - Delete the `BackupBackingImage` CRs whose name are in the cluster but not in the backup store
+            - Request `BackupBackingImageController` to reconcile those `BackupBackingImage` CRs
+
+5. Add a backup API for `BackingImage`
+    - Add new action `backup` to `BackingImage` (`"/v1/backingimages/{name}"`)
+        - create `BackupBackingImage` CR to init the backup process
+        - if `BackupBackingImage` already exists, it means there is already a `BackupBackingImage` in backup store, user can check the checksum to verify if they are the same.
+    - API Watch: establish a streaming connection to report BackupBackingImage info.
+
+6. Trigger
+    - Back up through `BackingImage` operation manually
+    - Back up `BackingImage` when user back up the volume
+        - in `SnapshotBackup()` API
+            - we get the `BackingImage` of the `Volume`
+            - back up `BackingImage` if the `BackupBackingImage` does not exist
+
+
+#### Restoring BackingImage - Controller
+
+2. Add new data source type `restore` for `BackingImageDataSource`
+    ```go
+    type BackingImageDataSourceType string
+
+    const (
+        BackingImageDataSourceTypeDownload         = BackingImageDataSourceType("download")
+        BackingImageDataSourceTypeUpload           = BackingImageDataSourceType("upload")
+        BackingImageDataSourceTypeExportFromVolume = BackingImageDataSourceType("export-from-volume")
+        BackingImageDataSourceTypeRestore          = BackingImageDataSourceType("restore")
+
+        DataSourceTypeRestoreParameterBackupURL    = "backup-url"
+    )
+
+    // BackingImageDataSourceSpec defines the desired state of the Longhorn backing image data source
+    type BackingImageDataSourceSpec struct {
+        NodeID          string                     `json:"nodeID"`
+        UUID            string                     `json:"uuid"`
+        DiskUUID        string                     `json:"diskUUID"`
+        DiskPath        string                     `json:"diskPath"`
+        Checksum        string                     `json:"checksum"`
+        SourceType      BackingImageDataSourceType `json:"sourceType"`
+        Parameters      map[string]string          `json:"parameters"`
+        FileTransferred bool                       `json:"fileTransferred"`
+    }
+    ```
+3. Create BackingImage APIs
+    - No need to change
+        - Create BackingImage CR with `type=restore` and `restore-url=${URL}`
+        - If BackingImage already exists in the cluster, user can use checksum to verify if they are the same.
+4. In `BackingImageController`
+    - No need to change, it will create the `BackingImageDataSource` CR
+5. In `BackingImageDataSourceController`
+    - No need to change, it will create the `BackingImageDataSourcePod` to do the restore.
+6. In `BackingImageManager - data_source`
+    - When init the service, if the type is `restore`, then restore from `backup-url` by requesting sync service in the same pod.
+        ```go
+        requestURL := fmt.Sprintf("http://%s/v1/files", client.Remote)
+        req, err := http.NewRequest("POST", requestURL, nil)
+        q := req.URL.Query()
+        q.Add("action", "restoreFromBackupURL")
+        q.Add("url", backupURL)
+        q.Add("file-path", filePath)
+        q.Add("uuid", uuid)
+        q.Add("disk-uuid", diskUUID)
+        q.Add("expected-checksum", expectedChecksum)
+        ````
+    - In `sync/service` implement `restoreFromBackupURL()` to restore the `BackingImage` from backup store to the local disk.
+7. In `BackingImageDataSourceController`
+    - No need to change, it will take over control when `BackingImageDataSource` status is `ReadyForTransfer`.
+    - If it failed to restore the `BackingImage`, the status of the `BackingImage` will be failed and  `BackingImageDataSourcePod` will be cleaned up and retry with backoff limit like `type=download`. The process is the same as other `BackingImage` creation process.
+8. Trigger
+    - Restore through `BackingImage` operation manually
+    - Restore when user restore the `Volume` with `BackingImage`
+        - Restoring a Volume is actually requesting `Create` a Volume with `fromBackup` in the spec
+        - In `Create()` API we check if the `Volume` has `fromBackup` parameters and has `BackingImage`
+        - Check if `BackingImage` exists
+        - Check and restore `BackupBackingImage` if `BackingImage` does not exist
+        - Restore `BackupBackingImage` by creating `BackingImage` with type `restore` and `backupURL`
+        - Then Create the `Volume` CR so the admission webhook won't failed because of missing `BackingImage` ([ref](https://github.com/longhorn/longhorn-manager/blob/master/webhook/resources/volume/validator.go#L86))
+    - Restore when user create `Volume` through `CSI`
+        - In `CreateVolume()` we check if the `Volume` has `fromBackup` parameters and has `BackingImage`
+        - In `checkAndPrepareBackingImage()`, we restore `BackupBackingImage` by creating `BackingImage` with type `restore` and `backupURL`
+
+#### API and UI changes In Summary
+
+1. `longhorn-ui`:
+    - Add a new page of `BackupBackingImage` like `Backup`
+        - The columns on `BackupBackingImage` list page should be: `Name`, `Size`, `State`, `Created At`, `Operation`.
+        - `Name` can be clicked and will show `Checksum` of the `BackupBackingImage`
+        - `State`: `BackupBackingImageState` of the `BackupBackingImage` CR
+        - `Operation` includes
+            - `restore`
+            - `delete`
+    - Add a new operation `backup` for every `BackingImage` in the `BackingImage` page
+
+2. `API`:
+    - Add new action `backup` to `BackingImage` (`"/v1/backingimages/{name}"`)
+        - create `BackupBackingImage` CR to init the backup process
+    - `BackupBackingImage`
+        - `GET "/v1/backupbackingimages"`: get all `BackupBackingImage`
+        - API Watch: establish a streaming connection to report `BackupBackingImage` info change.
+
+### Test plan
+
+Integration tests
+
+1. `BackupBackingImage` Basic Operation
+    - Setup
+        - Create a `BackingImage`
+        - Setup the backup target
+    - Back up `BackingImage`
+        - `BackupBackingImage` CR should be complete
+    - Delete the `BackingImage` in the cluster
+    - Restore the `BackupBackingImage`
+    - Checksum should be the same
+
+2. Back up `BackingImage` when backing up and restoring Volume
+    - Setup
+        - Create a `BackingImage`
+        - Setup the backup target
+        - Create a Volume with the `BackingImage`
+    - Backup the `Volume`
+    - `BackupBackingImage` CR should be created and complete
+    - Delete the `BackingImage`
+    - Restore the Volume with same `BackingImage`
+    - `BackingImage` should be restored and the `Volume` should also be restored successfully
+    - `Volume` checksum is the same
+
+Manual tests
+
+1. `BackupBackingImage` reuse blocks
+    - Setup
+        - Create a `BackingImage` A
+        - Setup the backup target
+    - Create a `Volume` with `BackingImage` A, write some data and export to another `BackingImage` B
+    - Back up `BackingImage` A
+    - Back up `BackingImage` B
+    - Check it reuses the blocks when backing up `BackingImage` B (by trace log)

enhancements/20230815-engine-upgrade-enforcement.md

@@ -0,0 +1,73 @@
+# Engine Upgrade Enforcement
+
+## Summary
+
+The current Longhorn upgrade process lacks enforcement of the engine version, potentially leading to compatibility issues. To address this concern, we propose the implementation of an Engine Upgrade Enforcement feature.
+
+### Related Issues
+
+https://github.com/longhorn/longhorn/issues/5842
+## Motivation
+
+Longhorn needs to be able to safely deprecated and remove the certain fields, such as [[TASK] Remove deprecated instances field and instance type from instance manager CR #5844](https://github.com/longhorn/longhorn/issues/5844)
+
+### Goals
+
+The primary goal of this proposal is to enhance Longhorn's upgrade mechanism by introducing logic that prevents upgrading to Longhorn versions while there are incompatible engine images in use.
+### Non-goals [optional]
+
+`None`
+
+## Proposal
+
+This proposal focuses on preventing users from upgrading to unsupported or incompatible engine versions. This enhancement will build upon the existing pre-upgrade checks to include validation of engine version compatibility.
+
+### User Stories
+
+#### Story 1: Preventing Incompatible Upgrades
+
+Previously, users had the freedom to continue using an older engine version after a Longhorn upgrade. With the proposed enhancement, the Longhorn upgrade process will be blocked if it includes an incompatible engine version. This will enforce users to manually upgrade the engine to a compatible version before proceeding with the Longhorn upgrade.
+
+### User Experience In Detail
+
+User will perform upgrade a usual. Longhorn will examine the compatibility of the current engine version. If the current engine version is incompatible with the target engine version for the upgrade, Longhorn will halt the upgrade process and prompt the user to address the engine version mismatch before proceeding.
+
+### API changes
+
+`None`
+
+## Design
+
+### Implementation Overview
+
+The implementation approach for this feature will be similar to the [Upgrade Path Enforcement feature](https://github.com/longhorn/longhorn/blob/master/enhancements/20230315-upgrade-path-enforcement.md).
+
+Key implementation steps include:
+
+1. Enhance the function [CheckUpgradePathSupported(...)](https://github.com/longhorn/longhorn-manager/blob/v1.5.1/upgrade/util/util.go#L168) to include the new checks.
+```
+func CheckUpgradePathSupported(namespace string, lhClient lhclientset.Interface) error {
+	if err := CheckLHUpgradePathSupported(namespace, lhClient); err != nil {
+		return err
+	}
+
+	return CheckEngineUpgradePathSupported(namespace, lhClient, emeta.GetVersion())
+}
+```
+
+1. Retrieve the current engine images being used and record the versions.
+1. Prevent upgrades if the targeting engine version is detact to be downgrading.
+1. Prevent upgrades if the engine image version is lower than [the minimum required version for the new engine image controller API](https://github.com/longhorn/longhorn-engine/blob/v1.5.1/pkg/meta/version.go#L10).
+
+### Test plan
+
+- Create unit test for the new logic.
+- Run manual test to verify the handling of incompatible engine image versions (e.g., Longhorn v1.4.x -> v1.5.x -> v1.6.x.)
+
+### Upgrade strategy
+
+`None`
+
+## Note [optional]
+
+`None`

examples/block/crypto/deployment_with_pvc.yaml

@@ -0,0 +1,44 @@
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: longhorn-block-pvc
+  namespace: default
+spec:
+  accessModes:
+    - ReadWriteOnce
+  volumeMode: Block
+  storageClassName: longhorn-crypto-global
+  resources:
+    requests:
+      storage: 2Gi
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: volume-test
+  labels:
+    app: volume-test
+  namespace: default
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: volume-test
+  strategy:
+    type: Recreate
+  template:
+    metadata:
+      labels:
+        app: volume-test
+    spec:
+      restartPolicy: Always
+      containers:
+      - image: nginx:stable-alpine
+        name: volume-test
+        volumeDevices:
+          - devicePath: /dev/longhorn/testblk
+            name: block-vol
+      volumes:
+      - name: block-vol
+        persistentVolumeClaim:
+          claimName: longhorn-block-pvc

examples/block/crypto/secret-crypto-global.yaml

@@ -0,0 +1,8 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: longhorn-crypto
+  namespace: longhorn-system
+stringData:
+  CRYPTO_KEY_VALUE: "Simple passphrase"
+  CRYPTO_KEY_PROVIDER: "secret" # this is optional we currently only support direct keys via secrets

examples/block/crypto/storageclass-crypto-global.yaml

@@ -0,0 +1,32 @@
+kind: StorageClass
+apiVersion: storage.k8s.io/v1
+metadata:
+  name: longhorn-crypto-global
+provisioner: driver.longhorn.io
+allowVolumeExpansion: true
+parameters:
+  numberOfReplicas: "3"
+  staleReplicaTimeout: "2880" # 48 hours in minutes
+  fromBackup: ""
+  encrypted: "true"
+  # we currently don't need secrets for volume creation
+  # but it allows for failing the CreateVolume call early
+  # if the required secret has not been setup yet.
+  csi.storage.k8s.io/provisioner-secret-name: "longhorn-crypto"
+  csi.storage.k8s.io/provisioner-secret-namespace: "longhorn-system"
+  csi.storage.k8s.io/node-publish-secret-name: "longhorn-crypto"
+  csi.storage.k8s.io/node-publish-secret-namespace: "longhorn-system"
+  csi.storage.k8s.io/node-stage-secret-name: "longhorn-crypto"
+  csi.storage.k8s.io/node-stage-secret-namespace: "longhorn-system"
+  # These two are for online expansion of encrypto volumes.
+  # But you need to enable the CSINodeExpandSecret feature gate for
+  # the kube-apiserver and kubelet. For more details, see:
+  # https://kubernetes.io/blog/2022/09/21/kubernetes-1-25-use-secrets-while-expanding-csi-volumes-on-node-alpha/
+  # csi.storage.k8s.io/node-expand-secret-name: "longhorn-crypto"
+  # csi.storage.k8s.io/node-expand-secret-namespace: "longhorn-system"
+  # we only need crypto keys for node operations, I left these as examples
+  # in case we implement external key vaults in the future
+  # csi.storage.k8s.io/controller-publish-secret-name: "longhorn-crypto"
+  # csi.storage.k8s.io/controller-publish-secret-namespace: "longhorn-system"
+  # csi.storage.k8s.io/controller-expand-secret-name: "longhorn-crypto"
+  # csi.storage.k8s.io/controller-expand-secret-namespace: "longhorn-system"

examples/storageclass.yaml

@@ -7,7 +7,7 @@ allowVolumeExpansion: true
 reclaimPolicy: Delete
 volumeBindingMode: Immediate
 parameters:
-  numberOfReplicas: "2"
+  numberOfReplicas: "3"
   staleReplicaTimeout: "2880"
   fromBackup: ""
   fsType: "ext4"
@@ -21,3 +21,4 @@ parameters:
 #  nodeSelector: "storage,fast"
 #  recurringJobSelector: '[{"name":"snap-group", "isGroup":true},
 #                          {"name":"backup", "isGroup":false}]'
+#  nfsOptions: "soft,timeo=150,retrans=3"

examples/v2/pod_with_pvc.yaml

@@ -6,7 +6,7 @@ metadata:
 spec:
   accessModes:
     - ReadWriteOnce
-  storageClassName: longhorn-spdk
+  storageClassName: longhorn-v2-data-engine
   resources:
     requests:
       storage: 2Gi

examples/v2/storageclass.yaml

@@ -1,17 +1,17 @@
 kind: StorageClass
 apiVersion: storage.k8s.io/v1
 metadata:
-  name: longhorn-spdk
+  name: longhorn-v2-data-engine
 provisioner: driver.longhorn.io
 allowVolumeExpansion: true
 reclaimPolicy: Delete
 volumeBindingMode: Immediate
 parameters:
   # backup, backingImage and snapshot related parameters are not supported in SPDK preview stage
-  numberOfReplicas: "2"
+  numberOfReplicas: "3"
   staleReplicaTimeout: "2880"
   fsType: "ext4"
-  backendStoreDriver: "spdk"
+  backendStoreDriver: "v2"
 #  mkfsParams: "-I 256 -b 4096 -O ^metadata_csum,^64bit"
 #  nodeSelector: "storage,fast"
 #  recurringJobSelector: '[{"name":"snap-group", "isGroup":true},
@@ -23,5 +23,6 @@ parameters:
 #  backingImageChecksum: "SHA512 checksum of the backing image"
 #  unmapMarkSnapChainRemoved: "ignored"
 #  diskSelector: "ssd,fast"
+#  nfsOptions: "soft,timeo=150,retrans=3"
 
 

scripts/cleanup.sh

@@ -1,64 +0,0 @@
-#!/bin/bash
-
-NAMESPACE=${NAMESPACE:-longhorn-system}
-
-remove_and_wait() {
-  local crd=$1
-  out=`kubectl -n ${NAMESPACE} delete $crd --all 2>&1`
-  if [ $? -ne 0 ]; then
-    echo $out
-    return
-  fi
-  while true; do
-    out=`kubectl -n ${NAMESPACE} get $crd -o yaml | grep 'items: \[\]'`
-    if [ $? -eq 0 ]; then
-      break
-    fi
-    sleep 1
-  done
-  echo all $crd instances deleted
-}
-
-remove_crd_instances() {
-  remove_and_wait volumes.longhorn.rancher.io
-  # TODO: remove engines and replicas once we fix https://github.com/rancher/longhorn/issues/273
-  remove_and_wait engines.longhorn.rancher.io
-  remove_and_wait replicas.longhorn.rancher.io
-  remove_and_wait engineimages.longhorn.rancher.io
-  remove_and_wait settings.longhorn.rancher.io
-  # do this one last; manager crashes
-  remove_and_wait nodes.longhorn.rancher.io
-}
-
-# Delete driver related workloads in specific order
-remove_driver() {
-  kubectl -n ${NAMESPACE} delete deployment.apps/longhorn-driver-deployer
-  kubectl -n ${NAMESPACE} delete daemonset.apps/longhorn-csi-plugin
-  kubectl -n ${NAMESPACE} delete statefulset.apps/csi-attacher
-  kubectl -n ${NAMESPACE} delete service/csi-attacher
-  kubectl -n ${NAMESPACE} delete statefulset.apps/csi-provisioner
-  kubectl -n ${NAMESPACE} delete service/csi-provisioner
-  kubectl -n ${NAMESPACE} delete daemonset.apps/longhorn-flexvolume-driver
-}
-
-# Delete all workloads in the namespace
-remove_workloads() {
-  kubectl -n ${NAMESPACE} get daemonset.apps -o yaml | kubectl delete -f -
-  kubectl -n ${NAMESPACE} get deployment.apps -o yaml | kubectl delete -f -
-  kubectl -n ${NAMESPACE} get replicaset.apps -o yaml | kubectl delete -f -
-  kubectl -n ${NAMESPACE} get statefulset.apps -o yaml | kubectl delete -f -
-  kubectl -n ${NAMESPACE} get pods -o yaml | kubectl delete -f -
-  kubectl -n ${NAMESPACE} get service -o yaml | kubectl delete -f -
-}
-
-# Delete CRD definitions with longhorn.rancher.io in the name
-remove_crds() {
-  for crd in $(kubectl get crd -o jsonpath={.items[*].metadata.name} | tr ' ' '\n' | grep longhorn.rancher.io); do
-    kubectl delete crd/$crd
-  done
-}
-
-remove_crd_instances
-remove_driver
-remove_workloads
-remove_crds

scripts/environment_check.sh

@@ -109,16 +109,16 @@ set_packages_and_check_cmd() {
 detect_node_kernel_release() {
   local pod="$1"
 
-  KERNEL_RELEASE=$(kubectl exec -i $pod -- nsenter --mount=/proc/1/ns/mnt -- bash -c 'uname -r')
+  KERNEL_RELEASE=$(kubectl exec $pod -- nsenter --mount=/proc/1/ns/mnt -- bash -c 'uname -r')
   echo "$KERNEL_RELEASE"
 }
 
 detect_node_os() {
   local pod="$1"
 
-  OS=$(kubectl exec -i $pod -- nsenter --mount=/proc/1/ns/mnt -- bash -c 'grep -E "^ID_LIKE=" /etc/os-release | cut -d= -f2')
+  OS=$(kubectl exec $pod -- nsenter --mount=/proc/1/ns/mnt -- bash -c 'grep -E "^ID_LIKE=" /etc/os-release | cut -d= -f2')
   if [[ -z "${OS}" ]]; then
-    OS=$(kubectl exec -i $pod -- nsenter --mount=/proc/1/ns/mnt -- bash -c 'grep -E "^ID=" /etc/os-release | cut -d= -f2')
+    OS=$(kubectl exec $pod -- nsenter --mount=/proc/1/ns/mnt -- bash -c 'grep -E "^ID=" /etc/os-release | cut -d= -f2')
   fi
   echo "$OS"
 }
@@ -273,12 +273,34 @@ check_nodes() {
   fi
 }
 
+verlte() {
+    printf '%s\n' "$1" "$2" | sort -C -V
+}
+
+verlt() {
+    ! verlte "$2" "$1"
+}
+
+check_kernel_release() {
+  local pod=$1
+
+  recommended_kernel_release="5.8"
+
+  local kernel=$(detect_node_kernel_release ${pod})
+
+  if verlt "$kernel" "$recommended_kernel_release"  ; then
+    local node=$(kubectl get ${pod} --no-headers -o=custom-columns=:.spec.nodeName)
+    warn "Node $node has outdated kernel release: $kernel. Recommending kernel release >= $recommended_kernel_release"
+    return 1
+  fi
+}
+
 check_iscsid() {
   local pod=$1
 
-  kubectl exec -t ${pod} -- nsenter --mount=/proc/1/ns/mnt -- bash -c "systemctl status --no-pager iscsid.service" > /dev/null 2>&1
+  kubectl exec ${pod} -- nsenter --mount=/proc/1/ns/mnt -- bash -c "systemctl status --no-pager iscsid.service" > /dev/null 2>&1
   if [ $? -ne 0 ]; then
-    kubectl exec -t ${pod} -- nsenter --mount=/proc/1/ns/mnt -- bash -c "systemctl status --no-pager iscsid.socket" > /dev/null 2>&1
+    kubectl exec ${pod} -- nsenter --mount=/proc/1/ns/mnt -- bash -c "systemctl status --no-pager iscsid.socket" > /dev/null 2>&1
       if [ $? -ne 0 ]; then
       node=$(kubectl get ${pod} --no-headers -o=custom-columns=:.spec.nodeName)
       error "Neither iscsid.service nor iscsid.socket is not running on ${node}"
@@ -290,7 +312,7 @@ check_iscsid() {
 check_multipathd() {
   local pod=$1
 
-  kubectl exec -t $pod -- nsenter --mount=/proc/1/ns/mnt -- bash -c "systemctl status --no-pager multipathd.service" > /dev/null 2>&1
+  kubectl exec $pod -- nsenter --mount=/proc/1/ns/mnt -- bash -c "systemctl status --no-pager multipathd.service" > /dev/null 2>&1
   if [ $? = 0 ]; then
     node=$(kubectl get ${pod} --no-headers -o=custom-columns=:.spec.nodeName)
     warn "multipathd is running on ${node}"
@@ -320,7 +342,7 @@ check_packages() {
 check_package() {
   local package=$1
 
-  kubectl exec -i $pod -- nsenter --mount=/proc/1/ns/mnt -- timeout 30 bash -c "$CHECK_CMD $package" > /dev/null 2>&1
+  kubectl exec $pod -- nsenter --mount=/proc/1/ns/mnt -- timeout 30 bash -c "$CHECK_CMD $package" > /dev/null 2>&1
   if [ $? -ne 0 ]; then
     node=$(kubectl get ${pod} --no-headers -o=custom-columns=:.spec.nodeName)
     error "$package is not found in $node."
@@ -341,7 +363,7 @@ check_nfs_client() {
   fi
 
   for option in "${options[@]}"; do
-    kubectl exec -t ${pod} -- nsenter --mount=/proc/1/ns/mnt -- bash -c "[ -f /boot/config-${kernel} ]" > /dev/null 2>&1
+    kubectl exec ${pod} -- nsenter --mount=/proc/1/ns/mnt -- bash -c "[ -f /boot/config-${kernel} ]" > /dev/null 2>&1
     if [ $? -ne 0 ]; then
       warn "Failed to check $option on node ${node}, because /boot/config-${kernel} does not exist on node ${node}"
       continue
@@ -368,18 +390,18 @@ check_kernel_module() {
     return 1
   fi
 
-  kubectl exec -t ${pod} -- nsenter --mount=/proc/1/ns/mnt -- bash -c "[ -e /boot/config-${kernel} ]" > /dev/null 2>&1
+  kubectl exec ${pod} -- nsenter --mount=/proc/1/ns/mnt -- bash -c "[ -e /boot/config-${kernel} ]" > /dev/null 2>&1
   if [ $? -ne 0 ]; then
     warn "Failed to check kernel config option ${option}, because /boot/config-${kernel} does not exist on node ${node}"
     return 1
   fi
 
-  value=$(kubectl exec -t ${pod} -- nsenter --mount=/proc/1/ns/mnt -- bash -c "grep "^$option=" /boot/config-${kernel} | cut -d= -f2")
+  value=$(kubectl exec ${pod} -- nsenter --mount=/proc/1/ns/mnt -- bash -c "grep "^$option=" /boot/config-${kernel} | cut -d= -f2")
   if [ -z "${value}" ]; then
     error "Failed to find kernel config $option on node ${node}"
     return 1
   elif [ "${value}" = "m" ]; then
-    kubectl exec -t ${pod} -- nsenter --mount=/proc/1/ns/mnt -- bash -c "lsmod | grep ${module}" > /dev/null 2>&1
+    kubectl exec ${pod} -- nsenter --mount=/proc/1/ns/mnt -- bash -c "lsmod | grep ${module}" > /dev/null 2>&1
     if [ $? -ne 0 ]; then
       node=$(kubectl get ${pod} --no-headers -o=custom-columns=:.spec.nodeName)
       error "kernel module ${module} is not enabled on ${node}"
@@ -397,7 +419,7 @@ check_hugepage() {
   local pod=$1
   local expected_nr_hugepages=$2
 
-  nr_hugepages=$(kubectl exec -i ${pod} -- nsenter --mount=/proc/1/ns/mnt -- bash -c 'cat /proc/sys/vm/nr_hugepages')
+  nr_hugepages=$(kubectl exec ${pod} -- nsenter --mount=/proc/1/ns/mnt -- bash -c 'cat /proc/sys/vm/nr_hugepages')
   if [ $? -ne 0 ]; then
     error "Failed to check hugepage size on node ${node}"
     return 1
@@ -412,7 +434,7 @@ check_hugepage() {
 function check_nvme_cli() {
   local pod=$1
 
-  value=$(kubectl exec -i $pod -- nsenter --mount=/proc/1/ns/mnt -- bash -c 'nvme version' 2>/dev/null)
+  value=$(kubectl exec $pod -- nsenter --mount=/proc/1/ns/mnt -- bash -c 'nvme version' 2>/dev/null)
   if [ $? -ne 0 ]; then
     node=$(kubectl get ${pod} --no-headers -o=custom-columns=:.spec.nodeName)
     error "Failed to check nvme-cli version on node ${node}"
@@ -432,14 +454,14 @@ function check_sse42_support() {
 
   node=$(kubectl get ${pod} --no-headers -o=custom-columns=:.spec.nodeName)
 
-  machine=$(kubectl exec -i $pod -- nsenter --mount=/proc/1/ns/mnt -- bash -c 'uname -m' 2>/dev/null)
+  machine=$(kubectl exec $pod -- nsenter --mount=/proc/1/ns/mnt -- bash -c 'uname -m' 2>/dev/null)
   if [ $? -ne 0 ]; then
     error "Failed to check machine on node ${node}"
     return 1
   fi
 
   if [ "$machine" = "x86_64" ]; then
-    sse42_support=$(kubectl exec -i $pod -- nsenter --mount=/proc/1/ns/mnt -- bash -c 'grep -o sse4_2 /proc/cpuinfo | wc -l' 2>/dev/null)
+    sse42_support=$(kubectl exec $pod -- nsenter --mount=/proc/1/ns/mnt -- bash -c 'grep -o sse4_2 /proc/cpuinfo | wc -l' 2>/dev/null)
     if [ $? -ne 0 ]; then
       error "Failed to check SSE4.2 instruction set on node ${node}"
       return 1
@@ -462,14 +484,14 @@ Usage: $0 [OPTIONS]
 
 Options:
     -s, --enable-spdk           Enable checking SPDK prerequisites
-    -p, --expected-nr-hugepages Expected number of hugepages for SPDK. Default: 1024
+    -p, --expected-nr-hugepages Expected number of 2 MiB hugepages for SPDK. Default: 512
     -h, --help                  Show this help message and exit
 EOF
     exit 0
 }
 
 enable_spdk=false
-expected_nr_hugepages=1024
+expected_nr_hugepages=512
 while [[ $# -gt 0 ]]; do
     opt="$1"
     case $opt in
@@ -493,7 +515,7 @@ done
 ######################################################
 # Main logics
 ######################################################
-DEPENDENCIES=("kubectl" "jq" "mktemp")
+DEPENDENCIES=("kubectl" "jq" "mktemp" "sort" "printf")
 check_local_dependencies "${DEPENDENCIES[@]}"
 
 # Check the each host has a unique hostname (for RWX volume)
@@ -507,6 +529,7 @@ create_ds
 wait_ds_ready
 
 check_mount_propagation
+check_nodes "kernel release" check_kernel_release
 check_nodes "iscsid" check_iscsid
 check_nodes "multipathd" check_multipathd
 check_nodes "packages" check_packages

scripts/helm-docs.sh

@@ -0,0 +1,15 @@
+#!/bin/bash
+## Reference: https://github.com/norwoodj/helm-docs
+
+set -o errexit
+set -o xtrace
+
+PRJ_DIR=$(readlink -f "$(dirname "${BASH_SOURCE[0]}")/.." 2>/dev/null || realpath "$(dirname "${BASH_SOURCE[0]}")/.." 2>/dev/null)
+CHART_DIR="$PRJ_DIR/chart"
+echo "$CHART_DIR"
+
+echo "Running Helm-Docs"
+sudo docker run \
+    -v "$CHART_DIR:/helm-docs" \
+    -u $(id -u) \
+    jnorwood/helm-docs:v1.9.1

scripts/save-images.sh

@@ -15,6 +15,11 @@ while [[ $# -gt 0 ]]; do
 		shift # past argument
 		shift # past value
 		;;
+		-p|--platform)
+		platform="$2"
+		shift # past argument
+		shift # past value
+		;;
 		-h|--help)
 		help="true"
 		shift
@@ -28,8 +33,9 @@ while [[ $# -gt 0 ]]; do
 done
 
 usage () {
-	echo "USAGE: $0 [--image-list longhorn-images.txt] [--images longhorn-images.tar.gz]"
+	echo "USAGE: $0 [--image-list longhorn-images.txt] [--images longhorn-images.tar.gz] [--platform linux/amd64]"
 	echo "  [-l|--images-list path] text file with list of images. 1 per line."
+	echo "  [-p|--platform linux/arch] if using images-list path, pulls the image with the specified platform"
 	echo "  [-i|--images path] tar.gz generated by docker save. If this flag is empty, the script does not export images to a tar.gz file"
 	echo "  [-h|--help] Usage message"
 }
@@ -42,9 +48,13 @@ fi
 set -e -x
 
 for i in $(cat ${list}); do
-	docker pull ${i}
+    if [ -n "$platform" ]; then
+        docker pull ${i} --platform $platform
+    else
+        docker pull ${i}
+    fi
 done
 
 if [[ $images ]]; then
 	docker save $(cat ${list} | tr '\n' ' ') | gzip -c > ${images}
-fi
+fi

uninstall/uninstall.yaml

@@ -66,7 +66,7 @@ rules:
   - apiGroups: ["longhorn.io"]
     resources: ["volumes", "engines", "replicas", "settings", "engineimages", "nodes", "instancemanagers", "sharemanagers",
                 "backingimages", "backingimagemanagers", "backingimagedatasources", "backuptargets", "backupvolumes", "backups",
-                "recurringjobs", "orphans", "snapshots", "supportbundles", "systembackups", "systemrestores"]
+                "recurringjobs", "orphans", "snapshots", "supportbundles", "systembackups", "systemrestores", "volumeattachments"]
     verbs: ["*"]
   - apiGroups: ["coordination.k8s.io"]
     resources: ["leases"]