OCP / OKD Documentation and Helm Support (#5004)
* Add OCP support to helm chart. Signed-off-by: Bin Guo <bin.guo@casa-systems.com> Signed-off-by: Arthur <arthur@arthurvardevanyan.com> Co-authored-by: Bin Guo <bin.guo@casa-systems.com> Co-authored-by: David Ko <dko@suse.com> Co-authored-by: binguo-casa <70552879+binguo-casa@users.noreply.github.com>
This commit is contained in:
Arthur Vardevanyan
GitHub
parent
39187d64d5
commit
67b4b38a12
177
chart/ocp-readme.md
Normal file
177
chart/ocp-readme.md
Normal file
@ -0,0 +1,177 @@
|
||||
# OpenShift / OKD Extra Configuration Steps
|
||||
|
||||
- [OpenShift / OKD Extra Configuration Steps](#openshift--okd-extra-configuration-steps)
|
||||
- [Notes](#notes)
|
||||
- [Known Issues](#known-issues)
|
||||
- [Preparing Nodes (Optional)](#preparing-nodes-optional)
|
||||
- [Default /var/lib/longhorn setup](#default-varliblonghorn-setup)
|
||||
- [Separate /var/mnt/longhorn setup](#separate-varmntlonghorn-setup)
|
||||
- [Create Filesystem](#create-filesystem)
|
||||
- [Mounting Disk On Boot](#mounting-disk-on-boot)
|
||||
- [Label and Annotate Nodes](#label-and-annotate-nodes)
|
||||
- [Example values.yaml](#example-valuesyaml)
|
||||
- [Installation](#installation)
|
||||
- [Refs](#refs)
|
||||
|
||||
## Notes
|
||||
|
||||
Main changes and tasks for OCP are:
|
||||
|
||||
- On OCP / OKD, the Operating System is Managed by the Cluster
|
||||
- OCP Imposes [Security Context Constraints](https://docs.openshift.com/container-platform/4.11/authentication/managing-security-context-constraints.html)
|
||||
- This requires everything to run with the least privilege possible. For the moment every component has been given access to run as higher privilege.
|
||||
- Something to circle back on is network polices and which components can have their privileges reduced without impacting functionality.
|
||||
- The UI probably can be for example.
|
||||
- openshift/oauth-proxy for authentication to the Longhorn Ui
|
||||
- **⚠️** Currently Scoped to Authenticated Users that can delete a longhorn settings object.
|
||||
- **⚠️** Since the UI it self is not protected, network policies will need to be created to prevent namespace <--> namespace communication against the pod or service object directly.
|
||||
- Anyone with access to the UI Deployment can remove the route restriction. (Namespace Scoped Admin)
|
||||
- Option to use separate disk in /var/mnt/longhorn & MachineConfig file to mount /var/mnt/longhorn
|
||||
- Adding finalizers for mount propagation
|
||||
|
||||
## Known Issues
|
||||
|
||||
- General Feature/Issue Thread
|
||||
- [[FEATURE] Deploying Longhorn on OKD/Openshift](https://github.com/longhorn/longhorn/issues/1831)
|
||||
- 4.10 / 1.23:
|
||||
- 4.10.0-0.okd-2022-03-07-131213 to 4.10.0-0.okd-2022-07-09-073606
|
||||
- Tested, No Known Issues
|
||||
- 4.11 / 1.24:
|
||||
- 4.11.0-0.okd-2022-07-27-052000 to 4.11.0-0.okd-2022-11-19-050030
|
||||
- Tested, No Known Issues
|
||||
- 4.11.0-0.okd-2022-12-02-145640, 4.11.0-0.okd-2023-01-14-152430:
|
||||
- Workaround: [[BUG] Volumes Stuck in Attach/Detach Loop](https://github.com/longhorn/longhorn/issues/4988)
|
||||
- [MachineConfig Patch](https://github.com/longhorn/longhorn/issues/4988#issuecomment-1345676772)
|
||||
- 4.12 / 1.25:
|
||||
- 4.12.0-0.okd-2022-12-05-210624 to 4.12.0-0.okd-2023-01-20-101927
|
||||
- Tested, No Known Issues
|
||||
- 4.12.0-0.okd-2023-01-21-055900 to 4.12.0-0.okd-2023-02-18-033438:
|
||||
- Workaround: [[BUG] Volumes Stuck in Attach/Detach Loop](https://github.com/longhorn/longhorn/issues/4988)
|
||||
- [MachineConfig Patch](https://github.com/longhorn/longhorn/issues/4988#issuecomment-1345676772)
|
||||
- 4.12.0-0.okd-2023-03-05-022504 - 4.12.0-0.okd-2023-04-16-041331:
|
||||
- Tested, No Known Issues
|
||||
- 4.13 / 1.26:
|
||||
- 4.13.0-0.okd-2023-05-03-001308 - 4.13.0-0.okd-2023-08-18-135805:
|
||||
- Tested, No Known Issues
|
||||
- 4.14 / 1.27:
|
||||
- 4.14.0-0.okd-2023-08-12-022330 - 4.14.0-0.okd-2023-08-12-022330:
|
||||
- Tested, No Known Issues
|
||||
|
||||
## Preparing Nodes (Optional)
|
||||
|
||||
Only required if you require additional customizations, such as storage-less nodes, or secondary disks.
|
||||
|
||||
### Default /var/lib/longhorn setup
|
||||
|
||||
Label each node for storage with:
|
||||
|
||||
```bash
|
||||
oc get nodes --no-headers | awk '{print $1}'
|
||||
|
||||
export NODE="worker-0"
|
||||
oc label node "${NODE}" node.longhorn.io/create-default-disk=true
|
||||
```
|
||||
|
||||
### Separate /var/mnt/longhorn setup
|
||||
|
||||
#### Create Filesystem
|
||||
|
||||
On the storage nodes create a filesystem with the label longhorn:
|
||||
|
||||
```bash
|
||||
oc get nodes --no-headers | awk '{print $1}'
|
||||
|
||||
export NODE="worker-0"
|
||||
oc debug node/${NODE} -t -- chroot /host bash
|
||||
|
||||
# Validate Target Drive is Present
|
||||
lsblk
|
||||
|
||||
export DRIVE="sdb" #vdb
|
||||
sudo mkfs.ext4 -L longhorn /dev/${DRIVE}
|
||||
```
|
||||
|
||||
> ⚠️ Note: If you add New Nodes After the below Machine Config is applied, you will need to also reboot the node.
|
||||
|
||||
#### Mounting Disk On Boot
|
||||
|
||||
The Secondary Drive needs to be mounted on every boot. Save the Concents and Apply the MachineConfig with `oc apply -f`:
|
||||
|
||||
> ⚠️ This will trigger an machine config profile update and reboot all worker nodes on the cluster
|
||||
|
||||
```yaml
|
||||
apiVersion: machineconfiguration.openshift.io/v1
|
||||
kind: MachineConfig
|
||||
metadata:
|
||||
labels:
|
||||
machineconfiguration.openshift.io/role: worker
|
||||
name: 71-mount-storage-worker
|
||||
spec:
|
||||
config:
|
||||
ignition:
|
||||
version: 3.2.0
|
||||
systemd:
|
||||
units:
|
||||
- name: var-mnt-longhorn.mount
|
||||
enabled: true
|
||||
contents: |
|
||||
[Unit]
|
||||
Before=local-fs.target
|
||||
[Mount]
|
||||
Where=/var/mnt/longhorn
|
||||
What=/dev/disk/by-label/longhorn
|
||||
Options=rw,relatime,discard
|
||||
[Install]
|
||||
WantedBy=local-fs.target
|
||||
```
|
||||
|
||||
#### Label and Annotate Nodes
|
||||
|
||||
Label and annotate storage nodes like this:
|
||||
|
||||
```bash
|
||||
oc get nodes --no-headers | awk '{print $1}'
|
||||
|
||||
export NODE="worker-0"
|
||||
oc annotate node ${NODE} --overwrite node.longhorn.io/default-disks-config='[{"path":"/var/mnt/longhorn","allowScheduling":true}]'
|
||||
oc label node ${NODE} node.longhorn.io/create-default-disk=config
|
||||
```
|
||||
|
||||
## Example values.yaml
|
||||
|
||||
Minimum Adjustments Required
|
||||
|
||||
```yaml
|
||||
openshift:
|
||||
oauthProxy:
|
||||
repository: quay.io/openshift/origin-oauth-proxy
|
||||
tag: 4.13 # Use Your OCP/OKD 4.X Version, Current Stable is 4.13
|
||||
|
||||
# defaultSettings: # Preparing nodes (Optional)
|
||||
# createDefaultDiskLabeledNodes: true
|
||||
|
||||
openshift:
|
||||
enabled: true
|
||||
ui:
|
||||
route: "longhorn-ui"
|
||||
port: 443
|
||||
proxy: 8443
|
||||
```
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
# helm template ./chart/ --namespace longhorn-system --values ./chart/values.yaml --no-hooks > longhorn.yaml # Local Testing
|
||||
helm template longhorn --namespace longhorn-system --values values.yaml --no-hooks > longhorn.yaml
|
||||
oc create namespace longhorn-system -o yaml --dry-run=client | oc apply -f -
|
||||
oc apply -f longhorn.yaml -n longhorn-system
|
||||
```
|
||||
|
||||
## Refs
|
||||
|
||||
- <https://docs.openshift.com/container-platform/4.11/storage/persistent_storage/persistent-storage-iscsi.html>
|
||||
- <https://docs.okd.io/4.11/storage/persistent_storage/persistent-storage-iscsi.html>
|
||||
- okd 4.5: <https://github.com/longhorn/longhorn/issues/1831#issuecomment-702690613>
|
||||
- okd 4.6: <https://github.com/longhorn/longhorn/issues/1831#issuecomment-765884631>
|
||||
- oauth-proxy: <https://github.com/openshift/oauth-proxy/blob/master/contrib/sidecar.yaml>
|
||||
- <https://github.com/longhorn/longhorn/issues/1831>
|
||||
@ -37,6 +37,9 @@ rules:
|
||||
- apiGroups: ["longhorn.io"]
|
||||
resources: ["volumes", "volumes/status", "engines", "engines/status", "replicas", "replicas/status", "settings",
|
||||
"engineimages", "engineimages/status", "nodes", "nodes/status", "instancemanagers", "instancemanagers/status",
|
||||
{{- if .Values.openshift.enabled }}
|
||||
"engineimages/finalizers", "nodes/finalizers", "instancemanagers/finalizers",
|
||||
{{- end }}
|
||||
"sharemanagers", "sharemanagers/status", "backingimages", "backingimages/status",
|
||||
"backingimagemanagers", "backingimagemanagers/status", "backingimagedatasources", "backingimagedatasources/status",
|
||||
"backuptargets", "backuptargets/status", "backupvolumes", "backupvolumes/status", "backups", "backups/status",
|
||||
@ -59,3 +62,16 @@ rules:
|
||||
- apiGroups: ["rbac.authorization.k8s.io"]
|
||||
resources: ["roles", "rolebindings", "clusterrolebindings", "clusterroles"]
|
||||
verbs: ["*"]
|
||||
{{- if .Values.openshift.enabled }}
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRole
|
||||
metadata:
|
||||
name: longhorn-ocp-privileged-role
|
||||
labels: {{- include "longhorn.labels" . | nindent 4 }}
|
||||
rules:
|
||||
- apiGroups: ["security.openshift.io"]
|
||||
resources: ["securitycontextconstraints"]
|
||||
resourceNames: ["anyuid", "privileged"]
|
||||
verbs: ["use"]
|
||||
{{- end }}
|
||||
|
||||
@ -25,3 +25,25 @@ subjects:
|
||||
- kind: ServiceAccount
|
||||
name: longhorn-support-bundle
|
||||
namespace: {{ include "release_namespace" . }}
|
||||
{{- if .Values.openshift.enabled }}
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRoleBinding
|
||||
metadata:
|
||||
name: longhorn-ocp-privileged-bind
|
||||
labels: {{- include "longhorn.labels" . | nindent 4 }}
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: ClusterRole
|
||||
name: longhorn-ocp-privileged-role
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: longhorn-service-account
|
||||
namespace: {{ include "release_namespace" . }}
|
||||
- kind: ServiceAccount
|
||||
name: longhorn-ui-service-account
|
||||
namespace: {{ include "release_namespace" . }}
|
||||
- kind: ServiceAccount
|
||||
name: default # supportbundle-agent-support-bundle uses default sa
|
||||
namespace: {{ include "release_namespace" . }}
|
||||
{{- end }}
|
||||
|
||||
@ -1,3 +1,41 @@
|
||||
{{- if .Values.openshift.enabled }}
|
||||
{{- if .Values.openshift.ui.route }}
|
||||
# https://github.com/openshift/oauth-proxy/blob/master/contrib/sidecar.yaml
|
||||
# Create a proxy service account and ensure it will use the route "proxy"
|
||||
# Create a secure connection to the proxy via a route
|
||||
apiVersion: route.openshift.io/v1
|
||||
kind: Route
|
||||
metadata:
|
||||
labels: {{- include "longhorn.labels" . | nindent 4 }}
|
||||
app: longhorn-ui
|
||||
name: {{ .Values.openshift.ui.route }}
|
||||
namespace: {{ include "release_namespace" . }}
|
||||
spec:
|
||||
to:
|
||||
kind: Service
|
||||
name: longhorn-ui
|
||||
tls:
|
||||
termination: reencrypt
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
labels: {{- include "longhorn.labels" . | nindent 4 }}
|
||||
app: longhorn-ui
|
||||
name: longhorn-ui
|
||||
namespace: {{ include "release_namespace" . }}
|
||||
annotations:
|
||||
service.alpha.openshift.io/serving-cert-secret-name: longhorn-ui-tls
|
||||
spec:
|
||||
ports:
|
||||
- name: longhorn-ui
|
||||
port: {{ .Values.openshift.ui.port | default 443 }}
|
||||
targetPort: {{ .Values.openshift.ui.proxy | default 8443 }}
|
||||
selector:
|
||||
app: longhorn-ui
|
||||
---
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
@ -15,6 +53,7 @@ spec:
|
||||
labels: {{- include "longhorn.labels" . | nindent 8 }}
|
||||
app: longhorn-ui
|
||||
spec:
|
||||
serviceAccountName: longhorn-ui-service-account
|
||||
affinity:
|
||||
podAntiAffinity:
|
||||
preferredDuringSchedulingIgnoredDuringExecution:
|
||||
@ -28,6 +67,28 @@ spec:
|
||||
- longhorn-ui
|
||||
topologyKey: kubernetes.io/hostname
|
||||
containers:
|
||||
{{- if .Values.openshift.enabled }}
|
||||
{{- if .Values.openshift.ui.route }}
|
||||
- name: oauth-proxy
|
||||
image: {{ template "registry_url" . }}{{ .Values.image.openshift.oauthProxy.repository }}:{{ .Values.image.openshift.oauthProxy.tag }}
|
||||
imagePullPolicy: IfNotPresent
|
||||
ports:
|
||||
- containerPort: {{ .Values.openshift.ui.proxy | default 8443 }}
|
||||
name: public
|
||||
args:
|
||||
- --https-address=:{{ .Values.openshift.ui.proxy | default 8443 }}
|
||||
- --provider=openshift
|
||||
- --openshift-service-account=longhorn-ui-service-account
|
||||
- --upstream=http://localhost:8000
|
||||
- --tls-cert=/etc/tls/private/tls.crt
|
||||
- --tls-key=/etc/tls/private/tls.key
|
||||
- --cookie-secret=SECRET
|
||||
- --openshift-sar={"namespace":"{{ include "release_namespace" . }}","group":"longhorn.io","resource":"setting","verb":"delete"}
|
||||
volumeMounts:
|
||||
- mountPath: /etc/tls/private
|
||||
name: longhorn-ui-tls
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
- name: longhorn-ui
|
||||
image: {{ template "registry_url" . }}{{ .Values.image.longhorn.ui.repository }}:{{ .Values.image.longhorn.ui.tag }}
|
||||
imagePullPolicy: {{ .Values.image.pullPolicy }}
|
||||
@ -47,6 +108,13 @@ spec:
|
||||
- name: LONGHORN_UI_PORT
|
||||
value: "8000"
|
||||
volumes:
|
||||
{{- if .Values.openshift.enabled }}
|
||||
{{- if .Values.openshift.ui.route }}
|
||||
- name: longhorn-ui-tls
|
||||
secret:
|
||||
secretName: longhorn-ui-tls
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
- emptyDir: {}
|
||||
name: nginx-cache
|
||||
- emptyDir: {}
|
||||
|
||||
@ -11,6 +11,25 @@ metadata:
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: longhorn-ui-service-account
|
||||
namespace: {{ include "release_namespace" . }}
|
||||
labels: {{- include "longhorn.labels" . | nindent 4 }}
|
||||
{{- with .Values.serviceAccount.annotations }}
|
||||
annotations:
|
||||
{{- toYaml . | nindent 4 }}
|
||||
{{- end }}
|
||||
{{- if .Values.openshift.enabled }}
|
||||
{{- if .Values.openshift.ui.route }}
|
||||
{{- if not .Values.serviceAccount.annotations }}
|
||||
annotations:
|
||||
{{- end }}
|
||||
serviceaccounts.openshift.io/oauth-redirectreference.primary: '{"kind":"OAuthRedirectReference","apiVersion":"v1","reference":{"kind":"Route","name":"longhorn-ui"}}'
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: longhorn-support-bundle
|
||||
namespace: {{ include "release_namespace" . }}
|
||||
|
||||
@ -68,6 +68,10 @@ image:
|
||||
livenessProbe:
|
||||
repository: longhornio/livenessprobe
|
||||
tag: v2.9.0
|
||||
openshift:
|
||||
oauthProxy:
|
||||
repository: quay.io/openshift/origin-oauth-proxy
|
||||
tag: 4.13 # Use Your OCP/OKD 4.X Version, Current Stable is 4.13
|
||||
pullPolicy: IfNotPresent
|
||||
|
||||
service:
|
||||
@ -293,3 +297,11 @@ annotations: {}
|
||||
serviceAccount:
|
||||
# Annotations to add to the service account
|
||||
annotations: {}
|
||||
|
||||
## openshift settings
|
||||
openshift:
|
||||
enabled: false # true
|
||||
ui:
|
||||
route: "longhorn-ui"
|
||||
port: 443
|
||||
proxy: 8443
|
||||
|
||||
@ -19,6 +19,17 @@ metadata:
|
||||
# Source: longhorn/templates/serviceaccount.yaml
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: longhorn-ui-service-account
|
||||
namespace: longhorn-system
|
||||
labels:
|
||||
app.kubernetes.io/name: longhorn
|
||||
app.kubernetes.io/instance: longhorn
|
||||
app.kubernetes.io/version: v1.6.0-dev
|
||||
---
|
||||
# Source: longhorn/templates/serviceaccount.yaml
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: longhorn-support-bundle
|
||||
namespace: longhorn-system
|
||||
@ -4257,6 +4268,7 @@ spec:
|
||||
app.kubernetes.io/version: v1.6.0-dev
|
||||
app: longhorn-ui
|
||||
spec:
|
||||
serviceAccountName: longhorn-ui-service-account
|
||||
affinity:
|
||||
podAntiAffinity:
|
||||
preferredDuringSchedulingIgnoredDuringExecution:
|
||||
|
||||
Loading…
Reference in New Issue
Block a user