178 lines
6.4 KiB
Markdown
178 lines
6.4 KiB
Markdown
# OpenShift / OKD Extra Configuration Steps
|
|
|
|
- [OpenShift / OKD Extra Configuration Steps](#openshift--okd-extra-configuration-steps)
|
|
- [Notes](#notes)
|
|
- [Known Issues](#known-issues)
|
|
- [Preparing Nodes (Optional)](#preparing-nodes-optional)
|
|
- [Default /var/lib/longhorn setup](#default-varliblonghorn-setup)
|
|
- [Separate /var/mnt/longhorn setup](#separate-varmntlonghorn-setup)
|
|
- [Create Filesystem](#create-filesystem)
|
|
- [Mounting Disk On Boot](#mounting-disk-on-boot)
|
|
- [Label and Annotate Nodes](#label-and-annotate-nodes)
|
|
- [Example values.yaml](#example-valuesyaml)
|
|
- [Installation](#installation)
|
|
- [Refs](#refs)
|
|
|
|
## Notes
|
|
|
|
Main changes and tasks for OCP are:
|
|
|
|
- On OCP / OKD, the Operating System is Managed by the Cluster
|
|
- OCP Imposes [Security Context Constraints](https://docs.openshift.com/container-platform/4.11/authentication/managing-security-context-constraints.html)
|
|
- This requires everything to run with the least privilege possible. For the moment every component has been given access to run as higher privilege.
|
|
- Something to circle back on is network polices and which components can have their privileges reduced without impacting functionality.
|
|
- The UI probably can be for example.
|
|
- openshift/oauth-proxy for authentication to the Longhorn Ui
|
|
- **⚠️** Currently Scoped to Authenticated Users that can delete a longhorn settings object.
|
|
- **⚠️** Since the UI it self is not protected, network policies will need to be created to prevent namespace <--> namespace communication against the pod or service object directly.
|
|
- Anyone with access to the UI Deployment can remove the route restriction. (Namespace Scoped Admin)
|
|
- Option to use separate disk in /var/mnt/longhorn & MachineConfig file to mount /var/mnt/longhorn
|
|
- Adding finalizers for mount propagation
|
|
|
|
## Known Issues
|
|
|
|
- General Feature/Issue Thread
|
|
- [[FEATURE] Deploying Longhorn on OKD/Openshift](https://github.com/longhorn/longhorn/issues/1831)
|
|
- 4.10 / 1.23:
|
|
- 4.10.0-0.okd-2022-03-07-131213 to 4.10.0-0.okd-2022-07-09-073606
|
|
- Tested, No Known Issues
|
|
- 4.11 / 1.24:
|
|
- 4.11.0-0.okd-2022-07-27-052000 to 4.11.0-0.okd-2022-11-19-050030
|
|
- Tested, No Known Issues
|
|
- 4.11.0-0.okd-2022-12-02-145640, 4.11.0-0.okd-2023-01-14-152430:
|
|
- Workaround: [[BUG] Volumes Stuck in Attach/Detach Loop](https://github.com/longhorn/longhorn/issues/4988)
|
|
- [MachineConfig Patch](https://github.com/longhorn/longhorn/issues/4988#issuecomment-1345676772)
|
|
- 4.12 / 1.25:
|
|
- 4.12.0-0.okd-2022-12-05-210624 to 4.12.0-0.okd-2023-01-20-101927
|
|
- Tested, No Known Issues
|
|
- 4.12.0-0.okd-2023-01-21-055900 to 4.12.0-0.okd-2023-02-18-033438:
|
|
- Workaround: [[BUG] Volumes Stuck in Attach/Detach Loop](https://github.com/longhorn/longhorn/issues/4988)
|
|
- [MachineConfig Patch](https://github.com/longhorn/longhorn/issues/4988#issuecomment-1345676772)
|
|
- 4.12.0-0.okd-2023-03-05-022504 - 4.12.0-0.okd-2023-04-16-041331:
|
|
- Tested, No Known Issues
|
|
- 4.13 / 1.26:
|
|
- 4.13.0-0.okd-2023-05-03-001308 - 4.13.0-0.okd-2023-08-18-135805:
|
|
- Tested, No Known Issues
|
|
- 4.14 / 1.27:
|
|
- 4.14.0-0.okd-2023-08-12-022330 - 4.14.0-0.okd-2023-10-28-073550:
|
|
- Tested, No Known Issues
|
|
|
|
## Preparing Nodes (Optional)
|
|
|
|
Only required if you require additional customizations, such as storage-less nodes, or secondary disks.
|
|
|
|
### Default /var/lib/longhorn setup
|
|
|
|
Label each node for storage with:
|
|
|
|
```bash
|
|
oc get nodes --no-headers | awk '{print $1}'
|
|
|
|
export NODE="worker-0"
|
|
oc label node "${NODE}" node.longhorn.io/create-default-disk=true
|
|
```
|
|
|
|
### Separate /var/mnt/longhorn setup
|
|
|
|
#### Create Filesystem
|
|
|
|
On the storage nodes create a filesystem with the label longhorn:
|
|
|
|
```bash
|
|
oc get nodes --no-headers | awk '{print $1}'
|
|
|
|
export NODE="worker-0"
|
|
oc debug node/${NODE} -t -- chroot /host bash
|
|
|
|
# Validate Target Drive is Present
|
|
lsblk
|
|
|
|
export DRIVE="sdb" #vdb
|
|
sudo mkfs.ext4 -L longhorn /dev/${DRIVE}
|
|
```
|
|
|
|
> ⚠️ Note: If you add New Nodes After the below Machine Config is applied, you will need to also reboot the node.
|
|
|
|
#### Mounting Disk On Boot
|
|
|
|
The Secondary Drive needs to be mounted on every boot. Save the Concents and Apply the MachineConfig with `oc apply -f`:
|
|
|
|
> ⚠️ This will trigger an machine config profile update and reboot all worker nodes on the cluster
|
|
|
|
```yaml
|
|
apiVersion: machineconfiguration.openshift.io/v1
|
|
kind: MachineConfig
|
|
metadata:
|
|
labels:
|
|
machineconfiguration.openshift.io/role: worker
|
|
name: 71-mount-storage-worker
|
|
spec:
|
|
config:
|
|
ignition:
|
|
version: 3.2.0
|
|
systemd:
|
|
units:
|
|
- name: var-mnt-longhorn.mount
|
|
enabled: true
|
|
contents: |
|
|
[Unit]
|
|
Before=local-fs.target
|
|
[Mount]
|
|
Where=/var/mnt/longhorn
|
|
What=/dev/disk/by-label/longhorn
|
|
Options=rw,relatime,discard
|
|
[Install]
|
|
WantedBy=local-fs.target
|
|
```
|
|
|
|
#### Label and Annotate Nodes
|
|
|
|
Label and annotate storage nodes like this:
|
|
|
|
```bash
|
|
oc get nodes --no-headers | awk '{print $1}'
|
|
|
|
export NODE="worker-0"
|
|
oc annotate node ${NODE} --overwrite node.longhorn.io/default-disks-config='[{"path":"/var/mnt/longhorn","allowScheduling":true}]'
|
|
oc label node ${NODE} node.longhorn.io/create-default-disk=config
|
|
```
|
|
|
|
## Example values.yaml
|
|
|
|
Minimum Adjustments Required
|
|
|
|
```yaml
|
|
openshift:
|
|
oauthProxy:
|
|
repository: quay.io/openshift/origin-oauth-proxy
|
|
tag: 4.14 # Use Your OCP/OKD 4.X Version, Current Stable is 4.14
|
|
|
|
# defaultSettings: # Preparing nodes (Optional)
|
|
# createDefaultDiskLabeledNodes: true
|
|
|
|
openshift:
|
|
enabled: true
|
|
ui:
|
|
route: "longhorn-ui"
|
|
port: 443
|
|
proxy: 8443
|
|
```
|
|
|
|
## Installation
|
|
|
|
```bash
|
|
# helm template ./chart/ --namespace longhorn-system --values ./chart/values.yaml --no-hooks > longhorn.yaml # Local Testing
|
|
helm template longhorn --namespace longhorn-system --values values.yaml --no-hooks > longhorn.yaml
|
|
oc create namespace longhorn-system -o yaml --dry-run=client | oc apply -f -
|
|
oc apply -f longhorn.yaml -n longhorn-system
|
|
```
|
|
|
|
## Refs
|
|
|
|
- <https://docs.openshift.com/container-platform/4.11/storage/persistent_storage/persistent-storage-iscsi.html>
|
|
- <https://docs.okd.io/4.11/storage/persistent_storage/persistent-storage-iscsi.html>
|
|
- okd 4.5: <https://github.com/longhorn/longhorn/issues/1831#issuecomment-702690613>
|
|
- okd 4.6: <https://github.com/longhorn/longhorn/issues/1831#issuecomment-765884631>
|
|
- oauth-proxy: <https://github.com/openshift/oauth-proxy/blob/master/contrib/sidecar.yaml>
|
|
- <https://github.com/longhorn/longhorn/issues/1831>
|