Kubernetes – PersistentVolume: MountVolume.SetUp failed

During testing Ceph & Kubernetes in combination with the ceph-csi plugin in run into a problem with some of my deployments. For some reason the deployment of a POD failed with the following event log:

Events:                                                                                                                                                                                                Type     Reason            Age                From                 Message      
---------------------------------------------------------------------------------------------
Warning  FailedScheduling  29s                default-scheduler        0/4 nodes are available: 4 persistentvolumeclaim "index" not found. 
Warning  FailedScheduling  25s (x3 over 29s)  default-scheduler        0/4 nodes are available: 4 pod has unbound immediate PersistentVolumeClaims. 
Normal   Scheduled         11s                default-scheduler        Successfully assigned office-demo-internal/documents-7c6c86466b-sqbmt to worker-3
Warning  FailedMount       3s (x5 over 11s)   kubelet, worker-3  MountVolume.SetUp failed for volume "demo-internal-index" : rpc error: code = Internal desc = mount failed: exit status 32
Mounting command: mount  
Mounting arguments: -t ext4 -o bind,_netdev /var/lib/kubelet/plugins/kubernetes.io/csi/pv/demo-internal-index/globalmount/demo-internal /var/lib/kubelet/pods/af2f33e0-06da-4429-9f75-908981cb85c3/volumes/kubernetes.io~csi/demo-internal-index/mount
Output: mount: /var/lib/kubelet/pods/af2f33e0-06da-4429-9f75-9034535485c3/volumes/kubernetes.io~csi/demo-internal-index/mount: special device /var/lib/kubelet/plugins/kubernetes.io/csi/pv/demo-internal-index/globalmount/demo-internal does not exist. 

The csi-plugin logs messages like:

 csi-rbdplugin Mounting command: mount                                                                                                                                                                 
 csi-rbdplugin Mounting arguments: -t ext4 -o bind,_netdev /var/lib/kubelet/plugins/kubernetes.io/csi/pv/demo-internal-index/globalmount/demo-internal-imixs /var/lib/kubelet/pods/af2f33e0-34535-4429- 
 9f75-908981cb85c3/volumes/kubernetes.io~csi/demo-internal-index/mount                                                                                                                                 
 csi-rbdplugin Output: mount: /var/lib/kubelet/pods/af2f33e0-06da-4429-35445-908981cb85c3/volumes/kubernetes.io~csi/demo-internal-index/mount: special device /var/lib/kubelet/plugins/kubernetes.io/cs 
 i/pv/demo-internal-index/globalmount/demo-internal does not exist.                                                                                                                              
 csi-rbdplugin E0613 15:56:55.814449   32379 utils.go:136] ID: 33 Req-ID: demo-internal-imixs GRPC error: rpc error: code = Internal desc = mount failed: exit status 32                               
 csi-rbdplugin Mounting command: mount                                                                                                                                                                 
 csi-rbdplugin Mounting arguments: -t ext4 -o bind,_netdev /var/lib/kubelet/plugins/kubernetes.io/csi/pv/demo-internal-index/globalmount/demo-internal /var/lib/kubelet/pods/af2f33e0-06da-5552- 
 9f75-908981cb85c3/volumes/kubernetes.io~csi/demo-internal-index/mount                                                                                                                                 
 csi-rbdplugin Output: mount: /var/lib/kubelet/pods/af2f33e0-06da-4429-9f75-908981cb85c3/volumes/kubernetes.io~csi/demo-internal-index/mount: special device /var/lib/kubelet/plugins/kubernetes.io/cs 
 i/pv/demo-internal-index/globalmount/demo-internal does not exist. 

After investigating many hours, I figured out that on the corresponding worker node there was something wrong with the corresponding PV directory

/var/lib/kubelet/plugins/kubernetes.io/csi/pv/demo-internal-index/

After deleting this directory on the worker node, everything works again. See also the discussion here.

Leave a Reply

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.