In this Blog I will explain how to install a distributed filesystem on a kubernetes cluster. To run stateful docker images (e.g. a Database like PostgreSQL) you have two choices.
- run the service on a dedicated node – this avoids the lost of data if kubernetes re-schedules your server to another node
- use a distributed storage solution like ceph or glusterfs storage
Gluster is a scalable network filesystem. This allows you to create a large, distributed storage solution on common hard ware. You can connect a gluster storage to Kubernetes to abstract the volume from your services.
Install
You can install Glusterfs on any node this includes the kubernetes worker nodes. The following guide explains how to intal Glusterfs on Debian 10(buster). You will find more information about installation here.
Repeat the following installation on each node you wish to join your gluster network storage. Run the commands as root:
$ su
# Add the GPG key to apt:
$ wget -O - https://download.gluster.org/pub/gluster/glusterfs/7/rsa.pub | apt-key add -
# Add the source (s/amd64/arm64/ as necessary):
$ echo deb [arch=amd64] https://download.gluster.org/pub/gluster/glusterfs/7/LATEST/Debian/buster/amd64/apt buster main > /etc/apt/sources.list.d/gluster.list
# Install...
$ apt update
$ apt install -y glusterfs-server
Start and enable the GlusterFS Service on all the servers.
$ sudo systemctl start glusterd
$ sudo systemctl enable glusterd
To test the gluster status run connect into one of your cluster-nodes:
$ sudo service glusterd status
Setup Gluster Network
Now you can check form one of your gluster nodes if you can reach each other node
$ gluster peer probe [gluster-node-ip]
where gluster-node-ip is the IP Adress or the DNS name of one of your gluster nodes. Now you can check the peer status on each node:
$ gluster peer status
Uuid: vvvv-qqq-zzz-yyyyy-xxxxx
State: Peer in Cluster (Connected)
Other names:
[YOUR-GLUSTER-NODE-NAME]
Setup a Volume
Next you can set up a GlusterFS volume. For that create a data volume on all servers:
$ mkdir -p /data/glusterfs/brick1/gv0
From any single worker node run:
$ gluster volume create gv0 replica 2 [GLUSTER-NODE1]:/data/glusterfs/brick1/gv0 [GLUSTER-NODE2]:/data/glusterfs/brick1/gv0
volume create: gv0: success: please start the volume to access data
replace [GLUSTER-NODE1] with the gluster node dns name or ip address.
Note: the directory must not be on the root partition. At least you should provide 3 gluster nodes.
Now you can start your new volume ‘gv0’:
$ sudo gluster volume start gv0
volume start: gv0: success
With the following command you can check the status of the new volume:
$ sudo gluster volume info
Find more about the setup [here}(https://docs.gluster.org/en/latest/Quick-Start-Guide/Quickstart/).
Kubernetes
Now as your gluster filesystem is up and running it’s time to tell your kubernetes from the new storage.
glusterfs-client
Fist you need to install the glusterfs-client package on your master node. The client is used by the kubernetes scheduler to create the gluster volumes.
$ sudo apt install gluster-client
Persistence Volume Example
The following is an example how to create a volume claim for the GlusterFS within a pod. First you need to create a persistence volume (pv)
gluster-pv.yaml:
apiVersion: v1
kind: PersistentVolume
metadata:
# The name of the PV, which is referenced in pod definitions or displayed in various oc volume commands.
name: gluster-pv
spec:
capacity:
# The amount of storage allocated to this volume.
storage: 1Gi
accessModes:
# labels to match a PV and a PVC. They currently do not define any form of access control.
- ReadWriteMany
# The glusterfs plug-in defines the volume type being used
glusterfs:
endpoints: gluster-cluster
# Gluster volume name, preceded by /
path: /gv0
readOnly: false
# volume reclaim policy indicates that the volume will be preserved after the pods accessing it terminate.
# Accepted values include Retain, Delete, and Recycle.
persistentVolumeReclaimPolicy: Retain
The gluster volume must be created before.
Persistence Volume Claim Example
The persistent volume claim (PVC) specifies the desired access mode and storage capacity. Currently, based on only these two attributes, a PVC is bound to the PV created before. Once a PV is bound to a PVC, that PV is essentially tied to the PVC’s project and cannot be bound to by another PVC. There is a one-to-one mapping of PVs and PVCs. However, multiple pods in the same project can use the same PVC.
gluster-pvc.yaml:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: gluster-claim
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Gi
The claim name is referenced by the pod under its volumes section.
Deployment Example
Within a deployment you can than mount a volume based on this claim. See the following example:
apiVersion: apps/v1
kind: Deployment
.....
spec:
....
template:
....
spec:
containers:
.....
image: postgres:9.6.1
# run as root because of glusterfs
securityContext:
runAsUser: 0
allowPrivilegeEscalation: false
volumeMounts:
- mountPath: /var/lib/postgresql/data
name: dbdata
readOnly: false
restartPolicy: Always
volumes:
- name: dbdata
persistentVolumeClaim:
claimName: gluster-claim
....
Find also a more detailed example here.
Gluster does not support so called “structured data”, meaning live, SQL databases. Of course, using Gluster to backup and restore the database would be fine
I got this from https://docs.gluster.org/en/latest/Install-Guide/Overview/
GlusterFS is fine for putting static content like web page files and backups but if you run a very busy and/or large database using GlusterFS for storage it will ultimately be sadness and tears
So what you recommend? is Ceph fine for databases?
For kubernetes my recommendation is Longhorn. In compare to ceph, is more easy to install and has an active community. See: https://github.com/imixs/imixs-cloud/blob/master/doc/LONGHORN.md
Ceph is not so easy to install and to manage and you need definitely more time and human resources.
Nice Blog – Big Thanks to you. That one really helps me
btw: there is a type in glusterfs-client install cmd. I think it has to be “apt install glusterfs-client” 😉