ivampiresp/longhorn
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update deploy example.yaml

Browse Source 
Major updates:
1. Longhorn will now run in the `longhorn-system` namespace by default.
2. Improvement on Longhorn Driver includes dependency check when started and
automatic static-linked jq installation
3. Use ganesha as the NFS server for testing to remove the dependency of
`nfs-kernel-server` on the host.
This commit is contained in:
Sheng Yang 2018-01-15 17:16:18 -08:00
parent 2c6328d7cc
commit 1dd3618256
3 changed files with 121 additions and 71 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

72
README.md
View File

@ -4,7 +4,7 @@ Longhorn is a distributed block storage system built using containers and micros
Longhorn is lightweight, reliable, and easy-to-use. It is particularly suitable as persistent storage for containers. It supports snapshots, backups, and even allows you to schedule recurring snapshots and backups!
You can read more details of Longhorn and its design here: http://rancher.com/microservices-block-storage/.
You can read more details of Longhorn and its design [here](http://rancher.com/microservices-block-storage/).
Longhorn is an experimental software. We appreciate your comments as we continue to work on it!
@ -21,22 +21,20 @@ Longhorn is 100% open source software. Project source code is spread across a nu
1. Docker v1.13+
2. Kubernetes v1.8+
3. Make sure `jq`, `curl`, `findmnt`, `grep`, `awk` and `blkid` has been installed in all nodes of the Kubernetes cluster.
3. Make sure `curl`, `findmnt`, `grep`, `awk` and `blkid` has been installed in all nodes of the Kubernetes cluster.
4. Make sure `open-iscsi` has been installed in all nodes of the Kubernetes cluster. For GKE, recommended Ubuntu as guest OS image since it contains `open-iscsi` already.
## Deployment
Create the deployment of Longhorn in your Kubernetes cluster is easy. For example, for GKE, you will only need to run `kubectl create -f deploy/example.yaml`.
Create the deployment of Longhorn in your Kubernetes cluster is easy. For most Kubernetes setup (except GKE), you will only need to run `kubectl create -f deploy/example.yaml`.
The configuration yaml will be slight different for each environment, for example:
For Google Kubernetes Engine (GKE) users, see [here](#google-kubernetes-engine) before proceed.
1. GKE requires user to manually claim himself as cluster admin to enable RBAC, using `kubectl create clusterrolebinding cluster-admin-binding --clusterrole=cluster-admin --user=<name@example.com>` (in which `name@example.com` is the user's account name in GCE, and it's case sensitive). See [here](https://cloud.google.com/kubernetes-engine/docs/how-to/role-based-access-control) for details.
2. The default Flexvolume plugin directory is different with GKE 1.8+, which is at `/home/kubernetes/flexvolume`. You can find it by running `ps aux|grep kubelet` on the host and check the `--flex-volume-plugin-dir` parameter. If there is none, the default `/usr/libexec/kubernetes/kubelet-plugins/volume/exec/` will be used.
Longhorn Manager and Longhorn Driver will be deployed as daemonsets, as you can see in the yaml file.
Longhorn Manager and Longhorn Driver will be deployed as daemonsets in a separate namespace called `longhorn-system`, as you can see in the yaml file.
When you see those pods has started correctly as follows, you've deployed the Longhorn successfully.
```
# kubectl -n longhorn-system get pod
NAME READY STATUS RESTARTS AGE
longhorn-driver-7b8l7 1/1 Running 0 3h
longhorn-driver-tqrlw 1/1 Running 0 3h
@ -48,11 +46,10 @@ longhorn-ui-76674c87b9-89swr 1/1 Running 0 3h
```
## Access the UI
Use `kubectl get svc` to get the external service IP for UI:
Use `kubectl -n longhorn-system get svc` to get the external service IP for UI:
```
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.20.240.1 <none> 443/TCP 9d
longhorn-backend ClusterIP 10.20.248.250 <none> 9500/TCP 58m
longhorn-frontend LoadBalancer 10.20.245.110 100.200.200.123 80:30697/TCP 58m
```
@ -169,32 +166,59 @@ spec:
Longhorn supports backing up to a NFS server. In order to use this feature, you need to have a NFS server running and accessible in the Kubernetes cluster. Here we provides a simple way help to setup a testing NFS server.
### Requirements
1. Make sure `nfs-kernel-server` has been installed in all nodes of kubernetes.
### Deployment
Longhorn's backup feature requires an NFS server or an S3 endpoint. You can setup a simple NFS server on the same host and use that to store backups.
The deployment for the simple nfs server is also very easy.
```
kubectl create -f deploy/example-backupstore.yaml
```
It will create a simple NFS server in the `default` namespace, which can be addressed as `longhorn-test-nfs-svc.default` for other pods in the cluster.
This NFS server won't save any data after you delete the Deployment. It's for development and testing only.
WARNING: This NFS server won't save any data after you delete it. It's for development and testing only.
After this script completes, using the following URL as the Backup Target in the Longhorn setting:
```
nfs://longhorn-nfs-svc:/opt/backupstore
nfs://longhorn-test-nfs-svc.default:/opt/backupstore
```
Open Longhorn UI, go to Setting, fill the Backup Target field with the URL above, click Save. Now you should able to use the backup feature of Longhorn.
## Google Kubernetes Engine
The configuration yaml will be slight different for Google Kubernetes Engine (GKE):
1. GKE requires user to manually claim himself as cluster admin to enable RBAC. User need to execute following command before create the Longhorn system using yaml files.
```
kubectl create clusterrolebinding cluster-admin-binding --clusterrole=cluster-admin --user=<name@example.com>
```
In which `name@example.com` is the user's account name in GCE, and it's case sensitive.
See [here](https://cloud.google.com/kubernetes-engine/docs/how-to/role-based-access-control) for details.
2. The default Flexvolume plugin directory is different with GKE 1.8+, which is at `/home/kubernetes/flexvolume`. User need to use
```
- name: flexvolume-longhorn-mount
hostPath:
path: /home/kubernetes/flexvolume/
```
instead of
```
- name: flexvolume-longhorn-mount
hostPath:
path: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/
```
in the last part of the Longhorn system deployment yaml file.
See [Troubleshooting](#troubleshooting) for details.
## Troubleshooting
### Volume can be attached/detached from UI, but Kubernetes Pod/Deployment etc cannot use it
Check if volume plugin directory has been set correctly.
By default, Kubernetes use `/usr/libexec/kubernetes/kubelet-plugins/volume/exec/` as the directory for volume plugin drivers, as stated in the [official document](https://github.com/kubernetes/community/blob/master/contributors/devel/flexvolume.md#prerequisites).
But some vendors may choose to change the directory due to various reasons. For example, GKE uses `/home/kubernetes/flexvolume` instead.
User can find the correct directory by running `ps aux|grep kubelet` on the host and check the `--flex-volume-plugin-dir` parameter. If there is none, the default `/usr/libexec/kubernetes/kubelet-plugins/volume/exec/` will be used.
## License
Copyright (c) 2014-2017 [Rancher Labs, Inc.](http://rancher.com)
Copyright (c) 2014-2018 [Rancher Labs, Inc.](http://rancher.com)
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.

52
deploy/example-backupstore.yaml
View File

@ -1,39 +1,35 @@
apiVersion: extensions/v1beta1
kind: Deployment
apiVersion: v1
kind: Pod
metadata:
name: longhorn-test-backupstore
name: longhorn-test-nfs
labels:
app: longhorn-nfs
app: longhorn-test-nfs
spec:
replicas: 1
template:
metadata:
labels:
app: longhorn-nfs
spec:
containers:
- name: longhorn-test-backupstore-pod
image: docker.io/erezhorev/dockerized_nfs_server
securityContext:
privileged: true
ports:
# dummy port to keep k8s happy
- containerPort: 1111
name: longhorn-nfs
args: ["/opt/backupstore"]
containers:
- name: longhorn-test-nfs-container
image: janeczku/nfs-ganesha:latest
imagePullPolicy: Always
env:
- name: EXPORT_ID
value: "14"
- name: EXPORT_PATH
value: /opt/backupstore
- name: PSEUDO_PATH
value: /opt/backupstore
command: ["bash", "-c", "mkdir -p /opt/backupstore && /opt/start_nfs.sh"]
securityContext:
capabilities:
add: ["SYS_ADMIN", "DAC_READ_SEARCH"]
---
kind: Service
apiVersion: v1
metadata:
labels:
app: longhorn-nfs
name: longhorn-nfs-svc
name: longhorn-test-nfs-svc
spec:
selector:
app: longhorn-nfs
app: longhorn-test-nfs
clusterIP: None
ports:
# dummy port to keep k8s happy
- name: longhorn-nfs
port: 1111
targetPort: longhorn-nfs
- name: notnecessary
port: 1234
targetPort: 1234

68
deploy/example.yaml
View File

@ -1,15 +1,13 @@
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
apiVersion: v1
kind: Namespace
metadata:
name: longhorn-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: longhorn-bind
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: longhorn-role
subjects:
- kind: ServiceAccount
name: longhorn-service-account
namespace: default
namespace: longhorn-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
@ -25,6 +23,9 @@ rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["*"]
- apiGroups: ["batch"]
resources: ["jobs"]
verbs: ["*"]
- apiGroups: ["longhorn.rancher.io"]
resources: ["nodes"]
verbs: ["*"]
@ -41,10 +42,18 @@ rules:
resources: ["controllers"]
verbs: ["*"]
---
apiVersion: v1
kind: ServiceAccount
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: longhorn-bind
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: longhorn-role
subjects:
- kind: ServiceAccount
name: longhorn-service-account
namespace: longhorn-system
---
apiVersion: extensions/v1beta1
kind: DaemonSet
@ -52,6 +61,7 @@ metadata:
labels:
app: longhorn-manager
name: longhorn-manager
namespace: longhorn-system
spec:
template:
metadata:
@ -67,8 +77,7 @@ spec:
mountPath: /data/
containers:
- name: longhorn-manager
image: rancher/longhorn-manager:6c51e02
imagePullPolicy: Always
image: rancher/longhorn-manager:4d21cac
securityContext:
privileged: true
command: ["launch-manager", "-d",
@ -124,6 +133,7 @@ metadata:
labels:
app: longhorn-manager
name: longhorn-backend
namespace: longhorn-system
spec:
selector:
app: longhorn-manager
@ -139,6 +149,7 @@ metadata:
labels:
app: longhorn-ui
name: longhorn-ui
namespace: longhorn-system
spec:
replicas: 1
template:
@ -148,8 +159,7 @@ spec:
spec:
containers:
- name: longhorn-ui
image: rancher/longhorn-ui:b161e3a
imagePullPolicy: IfNotPresent
image: rancher/longhorn-ui:99622cb
ports:
- containerPort: 8000
name: longhorn-ui
@ -163,6 +173,7 @@ metadata:
labels:
app: longhorn-ui
name: longhorn-frontend
namespace: longhorn-system
spec:
selector:
app: longhorn-ui
@ -177,6 +188,7 @@ apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: longhorn-driver
namespace: longhorn-system
spec:
template:
metadata:
@ -184,8 +196,17 @@ spec:
labels:
app: longhorn-driver
spec:
initContainers:
- name: init-container
image: rancher/longhorn-driver:4d21cac
securityContext:
privileged: true
command: ["/checkdependency.sh"]
volumeMounts:
- name: host-proc-mount
mountPath: /host/proc/
containers:
- image: rancher/longhorn-driver:5260c7b
- image: rancher/longhorn-driver:4d21cac
imagePullPolicy: Always
name: longhorn-driver-container
command: ["/entrypoint.sh"]
@ -194,6 +215,8 @@ spec:
volumeMounts:
- mountPath: /flexmnt
name: flexvolume-longhorn-mount
- mountPath: /binmnt
name: usr-local-bin-mount
env:
- name: LONGHORN_BACKEND_SVC
value: "longhorn-backend"
@ -204,5 +227,12 @@ spec:
volumes:
- name: flexvolume-longhorn-mount
hostPath:
path: /home/kubernetes/flexvolume
#path: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/
path: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/
#FOR GKE
#path: /home/kubernetes/flexvolume/
- name: usr-local-bin-mount
hostPath:
path: /usr/local/bin/
- name: host-proc-mount
hostPath:
path: /proc/