diff --git a/chart/ocp-readme.md b/chart/ocp-readme.md new file mode 100644 index 0000000..16586de --- /dev/null +++ b/chart/ocp-readme.md @@ -0,0 +1,177 @@ +# OpenShift / OKD Extra Configuration Steps + +- [OpenShift / OKD Extra Configuration Steps](#openshift--okd-extra-configuration-steps) + - [Notes](#notes) + - [Known Issues](#known-issues) + - [Preparing Nodes (Optional)](#preparing-nodes-optional) + - [Default /var/lib/longhorn setup](#default-varliblonghorn-setup) + - [Separate /var/mnt/longhorn setup](#separate-varmntlonghorn-setup) + - [Create Filesystem](#create-filesystem) + - [Mounting Disk On Boot](#mounting-disk-on-boot) + - [Label and Annotate Nodes](#label-and-annotate-nodes) + - [Example values.yaml](#example-valuesyaml) + - [Installation](#installation) + - [Refs](#refs) + +## Notes + +Main changes and tasks for OCP are: + +- On OCP / OKD, the Operating System is Managed by the Cluster +- OCP Imposes [Security Context Constraints](https://docs.openshift.com/container-platform/4.11/authentication/managing-security-context-constraints.html) + - This requires everything to run with the least privilege possible. For the moment every component has been given access to run as higher privilege. + - Something to circle back on is network polices and which components can have their privileges reduced without impacting functionality. + - The UI probably can be for example. +- openshift/oauth-proxy for authentication to the Longhorn Ui + - **⚠️** Currently Scoped to Authenticated Users that can delete a longhorn settings object. + - **⚠️** Since the UI it self is not protected, network policies will need to be created to prevent namespace <--> namespace communication against the pod or service object directly. + - Anyone with access to the UI Deployment can remove the route restriction. (Namespace Scoped Admin) +- Option to use separate disk in /var/mnt/longhorn & MachineConfig file to mount /var/mnt/longhorn +- Adding finalizers for mount propagation + +## Known Issues + +- General Feature/Issue Thread + - [[FEATURE] Deploying Longhorn on OKD/Openshift](https://github.com/longhorn/longhorn/issues/1831) +- 4.10 / 1.23: + - 4.10.0-0.okd-2022-03-07-131213 to 4.10.0-0.okd-2022-07-09-073606 + - Tested, No Known Issues +- 4.11 / 1.24: + - 4.11.0-0.okd-2022-07-27-052000 to 4.11.0-0.okd-2022-11-19-050030 + - Tested, No Known Issues + - 4.11.0-0.okd-2022-12-02-145640, 4.11.0-0.okd-2023-01-14-152430: + - Workaround: [[BUG] Volumes Stuck in Attach/Detach Loop](https://github.com/longhorn/longhorn/issues/4988) + - [MachineConfig Patch](https://github.com/longhorn/longhorn/issues/4988#issuecomment-1345676772) +- 4.12 / 1.25: + - 4.12.0-0.okd-2022-12-05-210624 to 4.12.0-0.okd-2023-01-20-101927 + - Tested, No Known Issues + - 4.12.0-0.okd-2023-01-21-055900 to 4.12.0-0.okd-2023-02-18-033438: + - Workaround: [[BUG] Volumes Stuck in Attach/Detach Loop](https://github.com/longhorn/longhorn/issues/4988) + - [MachineConfig Patch](https://github.com/longhorn/longhorn/issues/4988#issuecomment-1345676772) + - 4.12.0-0.okd-2023-03-05-022504 - 4.12.0-0.okd-2023-04-16-041331: + - Tested, No Known Issues +- 4.13 / 1.26: + - 4.13.0-0.okd-2023-05-03-001308 - 4.13.0-0.okd-2023-08-18-135805: + - Tested, No Known Issues +- 4.14 / 1.27: + - 4.14.0-0.okd-2023-08-12-022330 - 4.14.0-0.okd-2023-08-12-022330: + - Tested, No Known Issues + +## Preparing Nodes (Optional) + +Only required if you require additional customizations, such as storage-less nodes, or secondary disks. + +### Default /var/lib/longhorn setup + +Label each node for storage with: + +```bash +oc get nodes --no-headers | awk '{print $1}' + +export NODE="worker-0" +oc label node "${NODE}" node.longhorn.io/create-default-disk=true +``` + +### Separate /var/mnt/longhorn setup + +#### Create Filesystem + +On the storage nodes create a filesystem with the label longhorn: + +```bash +oc get nodes --no-headers | awk '{print $1}' + +export NODE="worker-0" +oc debug node/${NODE} -t -- chroot /host bash + +# Validate Target Drive is Present +lsblk + +export DRIVE="sdb" #vdb +sudo mkfs.ext4 -L longhorn /dev/${DRIVE} +``` + +> ⚠️ Note: If you add New Nodes After the below Machine Config is applied, you will need to also reboot the node. + +#### Mounting Disk On Boot + +The Secondary Drive needs to be mounted on every boot. Save the Concents and Apply the MachineConfig with `oc apply -f`: + +> ⚠️ This will trigger an machine config profile update and reboot all worker nodes on the cluster + +```yaml +apiVersion: machineconfiguration.openshift.io/v1 +kind: MachineConfig +metadata: + labels: + machineconfiguration.openshift.io/role: worker + name: 71-mount-storage-worker +spec: + config: + ignition: + version: 3.2.0 + systemd: + units: + - name: var-mnt-longhorn.mount + enabled: true + contents: | + [Unit] + Before=local-fs.target + [Mount] + Where=/var/mnt/longhorn + What=/dev/disk/by-label/longhorn + Options=rw,relatime,discard + [Install] + WantedBy=local-fs.target +``` + +#### Label and Annotate Nodes + +Label and annotate storage nodes like this: + +```bash +oc get nodes --no-headers | awk '{print $1}' + +export NODE="worker-0" +oc annotate node ${NODE} --overwrite node.longhorn.io/default-disks-config='[{"path":"/var/mnt/longhorn","allowScheduling":true}]' +oc label node ${NODE} node.longhorn.io/create-default-disk=config +``` + +## Example values.yaml + +Minimum Adjustments Required + +```yaml +openshift: + oauthProxy: + repository: quay.io/openshift/origin-oauth-proxy + tag: 4.13 # Use Your OCP/OKD 4.X Version, Current Stable is 4.13 + +# defaultSettings: # Preparing nodes (Optional) + # createDefaultDiskLabeledNodes: true + +openshift: + enabled: true + ui: + route: "longhorn-ui" + port: 443 + proxy: 8443 +``` + +## Installation + +```bash +# helm template ./chart/ --namespace longhorn-system --values ./chart/values.yaml --no-hooks > longhorn.yaml # Local Testing +helm template longhorn --namespace longhorn-system --values values.yaml --no-hooks > longhorn.yaml +oc create namespace longhorn-system -o yaml --dry-run=client | oc apply -f - +oc apply -f longhorn.yaml -n longhorn-system +``` + +## Refs + +- +- +- okd 4.5: +- okd 4.6: +- oauth-proxy: +- diff --git a/chart/templates/clusterrole.yaml b/chart/templates/clusterrole.yaml index e652a34..afc76a2 100644 --- a/chart/templates/clusterrole.yaml +++ b/chart/templates/clusterrole.yaml @@ -37,6 +37,9 @@ rules: - apiGroups: ["longhorn.io"] resources: ["volumes", "volumes/status", "engines", "engines/status", "replicas", "replicas/status", "settings", "engineimages", "engineimages/status", "nodes", "nodes/status", "instancemanagers", "instancemanagers/status", + {{- if .Values.openshift.enabled }} + "engineimages/finalizers", "nodes/finalizers", "instancemanagers/finalizers", + {{- end }} "sharemanagers", "sharemanagers/status", "backingimages", "backingimages/status", "backingimagemanagers", "backingimagemanagers/status", "backingimagedatasources", "backingimagedatasources/status", "backuptargets", "backuptargets/status", "backupvolumes", "backupvolumes/status", "backups", "backups/status", @@ -59,3 +62,16 @@ rules: - apiGroups: ["rbac.authorization.k8s.io"] resources: ["roles", "rolebindings", "clusterrolebindings", "clusterroles"] verbs: ["*"] +{{- if .Values.openshift.enabled }} +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: longhorn-ocp-privileged-role + labels: {{- include "longhorn.labels" . | nindent 4 }} +rules: +- apiGroups: ["security.openshift.io"] + resources: ["securitycontextconstraints"] + resourceNames: ["anyuid", "privileged"] + verbs: ["use"] +{{- end }} diff --git a/chart/templates/clusterrolebinding.yaml b/chart/templates/clusterrolebinding.yaml index 8ab944b..2e34f01 100644 --- a/chart/templates/clusterrolebinding.yaml +++ b/chart/templates/clusterrolebinding.yaml @@ -25,3 +25,25 @@ subjects: - kind: ServiceAccount name: longhorn-support-bundle namespace: {{ include "release_namespace" . }} +{{- if .Values.openshift.enabled }} +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: longhorn-ocp-privileged-bind + labels: {{- include "longhorn.labels" . | nindent 4 }} +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: longhorn-ocp-privileged-role +subjects: +- kind: ServiceAccount + name: longhorn-service-account + namespace: {{ include "release_namespace" . }} +- kind: ServiceAccount + name: longhorn-ui-service-account + namespace: {{ include "release_namespace" . }} +- kind: ServiceAccount + name: default # supportbundle-agent-support-bundle uses default sa + namespace: {{ include "release_namespace" . }} +{{- end }} diff --git a/chart/templates/deployment-ui.yaml b/chart/templates/deployment-ui.yaml index 6bad5cd..0ee86c7 100644 --- a/chart/templates/deployment-ui.yaml +++ b/chart/templates/deployment-ui.yaml @@ -1,3 +1,41 @@ +{{- if .Values.openshift.enabled }} +{{- if .Values.openshift.ui.route }} +# https://github.com/openshift/oauth-proxy/blob/master/contrib/sidecar.yaml +# Create a proxy service account and ensure it will use the route "proxy" +# Create a secure connection to the proxy via a route +apiVersion: route.openshift.io/v1 +kind: Route +metadata: + labels: {{- include "longhorn.labels" . | nindent 4 }} + app: longhorn-ui + name: {{ .Values.openshift.ui.route }} + namespace: {{ include "release_namespace" . }} +spec: + to: + kind: Service + name: longhorn-ui + tls: + termination: reencrypt +--- +apiVersion: v1 +kind: Service +metadata: + labels: {{- include "longhorn.labels" . | nindent 4 }} + app: longhorn-ui + name: longhorn-ui + namespace: {{ include "release_namespace" . }} + annotations: + service.alpha.openshift.io/serving-cert-secret-name: longhorn-ui-tls +spec: + ports: + - name: longhorn-ui + port: {{ .Values.openshift.ui.port | default 443 }} + targetPort: {{ .Values.openshift.ui.proxy | default 8443 }} + selector: + app: longhorn-ui +--- +{{- end }} +{{- end }} apiVersion: apps/v1 kind: Deployment metadata: @@ -15,6 +53,7 @@ spec: labels: {{- include "longhorn.labels" . | nindent 8 }} app: longhorn-ui spec: + serviceAccountName: longhorn-ui-service-account affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: @@ -28,6 +67,28 @@ spec: - longhorn-ui topologyKey: kubernetes.io/hostname containers: + {{- if .Values.openshift.enabled }} + {{- if .Values.openshift.ui.route }} + - name: oauth-proxy + image: {{ template "registry_url" . }}{{ .Values.image.openshift.oauthProxy.repository }}:{{ .Values.image.openshift.oauthProxy.tag }} + imagePullPolicy: IfNotPresent + ports: + - containerPort: {{ .Values.openshift.ui.proxy | default 8443 }} + name: public + args: + - --https-address=:{{ .Values.openshift.ui.proxy | default 8443 }} + - --provider=openshift + - --openshift-service-account=longhorn-ui-service-account + - --upstream=http://localhost:8000 + - --tls-cert=/etc/tls/private/tls.crt + - --tls-key=/etc/tls/private/tls.key + - --cookie-secret=SECRET + - --openshift-sar={"namespace":"{{ include "release_namespace" . }}","group":"longhorn.io","resource":"setting","verb":"delete"} + volumeMounts: + - mountPath: /etc/tls/private + name: longhorn-ui-tls + {{- end }} + {{- end }} - name: longhorn-ui image: {{ template "registry_url" . }}{{ .Values.image.longhorn.ui.repository }}:{{ .Values.image.longhorn.ui.tag }} imagePullPolicy: {{ .Values.image.pullPolicy }} @@ -47,6 +108,13 @@ spec: - name: LONGHORN_UI_PORT value: "8000" volumes: + {{- if .Values.openshift.enabled }} + {{- if .Values.openshift.ui.route }} + - name: longhorn-ui-tls + secret: + secretName: longhorn-ui-tls + {{- end }} + {{- end }} - emptyDir: {} name: nginx-cache - emptyDir: {} diff --git a/chart/templates/serviceaccount.yaml b/chart/templates/serviceaccount.yaml index a563d68..b0d6dd5 100644 --- a/chart/templates/serviceaccount.yaml +++ b/chart/templates/serviceaccount.yaml @@ -11,6 +11,25 @@ metadata: --- apiVersion: v1 kind: ServiceAccount +metadata: + name: longhorn-ui-service-account + namespace: {{ include "release_namespace" . }} + labels: {{- include "longhorn.labels" . | nindent 4 }} + {{- with .Values.serviceAccount.annotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} + {{- if .Values.openshift.enabled }} + {{- if .Values.openshift.ui.route }} + {{- if not .Values.serviceAccount.annotations }} + annotations: + {{- end }} + serviceaccounts.openshift.io/oauth-redirectreference.primary: '{"kind":"OAuthRedirectReference","apiVersion":"v1","reference":{"kind":"Route","name":"longhorn-ui"}}' + {{- end }} + {{- end }} +--- +apiVersion: v1 +kind: ServiceAccount metadata: name: longhorn-support-bundle namespace: {{ include "release_namespace" . }} diff --git a/chart/values.yaml b/chart/values.yaml index 081fb28..e6474f4 100644 --- a/chart/values.yaml +++ b/chart/values.yaml @@ -68,6 +68,10 @@ image: livenessProbe: repository: longhornio/livenessprobe tag: v2.9.0 + openshift: + oauthProxy: + repository: quay.io/openshift/origin-oauth-proxy + tag: 4.13 # Use Your OCP/OKD 4.X Version, Current Stable is 4.13 pullPolicy: IfNotPresent service: @@ -293,3 +297,11 @@ annotations: {} serviceAccount: # Annotations to add to the service account annotations: {} + +## openshift settings +openshift: + enabled: false # true + ui: + route: "longhorn-ui" + port: 443 + proxy: 8443 diff --git a/deploy/longhorn.yaml b/deploy/longhorn.yaml index 1f851bb..eb6c1ca 100644 --- a/deploy/longhorn.yaml +++ b/deploy/longhorn.yaml @@ -19,6 +19,17 @@ metadata: # Source: longhorn/templates/serviceaccount.yaml apiVersion: v1 kind: ServiceAccount +metadata: + name: longhorn-ui-service-account + namespace: longhorn-system + labels: + app.kubernetes.io/name: longhorn + app.kubernetes.io/instance: longhorn + app.kubernetes.io/version: v1.6.0-dev +--- +# Source: longhorn/templates/serviceaccount.yaml +apiVersion: v1 +kind: ServiceAccount metadata: name: longhorn-support-bundle namespace: longhorn-system @@ -4257,6 +4268,7 @@ spec: app.kubernetes.io/version: v1.6.0-dev app: longhorn-ui spec: + serviceAccountName: longhorn-ui-service-account affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: