During testing Ceph & Kubernetes in combination with the ceph-csi plugin in run into a problem with some of my deployments. For some reason the deployment of a POD failed with the following event log:
Events: Type Reason Age From Message
---------------------------------------------------------------------------------------------
Warning FailedScheduling 29s default-scheduler 0/4 nodes are available: 4 persistentvolumeclaim "index" not found.
Warning FailedScheduling 25s (x3 over 29s) default-scheduler 0/4 nodes are available: 4 pod has unbound immediate PersistentVolumeClaims.
Normal Scheduled 11s default-scheduler Successfully assigned office-demo-internal/documents-7c6c86466b-sqbmt to worker-3
Warning FailedMount 3s (x5 over 11s) kubelet, worker-3 MountVolume.SetUp failed for volume "demo-internal-index" : rpc error: code = Internal desc = mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t ext4 -o bind,_netdev /var/lib/kubelet/plugins/kubernetes.io/csi/pv/demo-internal-index/globalmount/demo-internal /var/lib/kubelet/pods/af2f33e0-06da-4429-9f75-908981cb85c3/volumes/kubernetes.io~csi/demo-internal-index/mount
Output: mount: /var/lib/kubelet/pods/af2f33e0-06da-4429-9f75-9034535485c3/volumes/kubernetes.io~csi/demo-internal-index/mount: special device /var/lib/kubelet/plugins/kubernetes.io/csi/pv/demo-internal-index/globalmount/demo-internal does not exist. The csi-plugin logs messages like:
csi-rbdplugin Mounting command: mount
csi-rbdplugin Mounting arguments: -t ext4 -o bind,_netdev /var/lib/kubelet/plugins/kubernetes.io/csi/pv/demo-internal-index/globalmount/demo-internal-imixs /var/lib/kubelet/pods/af2f33e0-34535-4429-
9f75-908981cb85c3/volumes/kubernetes.io~csi/demo-internal-index/mount
csi-rbdplugin Output: mount: /var/lib/kubelet/pods/af2f33e0-06da-4429-35445-908981cb85c3/volumes/kubernetes.io~csi/demo-internal-index/mount: special device /var/lib/kubelet/plugins/kubernetes.io/cs
i/pv/demo-internal-index/globalmount/demo-internal does not exist.
csi-rbdplugin E0613 15:56:55.814449 32379 utils.go:136] ID: 33 Req-ID: demo-internal-imixs GRPC error: rpc error: code = Internal desc = mount failed: exit status 32
csi-rbdplugin Mounting command: mount
csi-rbdplugin Mounting arguments: -t ext4 -o bind,_netdev /var/lib/kubelet/plugins/kubernetes.io/csi/pv/demo-internal-index/globalmount/demo-internal /var/lib/kubelet/pods/af2f33e0-06da-5552-
9f75-908981cb85c3/volumes/kubernetes.io~csi/demo-internal-index/mount
csi-rbdplugin Output: mount: /var/lib/kubelet/pods/af2f33e0-06da-4429-9f75-908981cb85c3/volumes/kubernetes.io~csi/demo-internal-index/mount: special device /var/lib/kubelet/plugins/kubernetes.io/cs
i/pv/demo-internal-index/globalmount/demo-internal does not exist. After investigating many hours, I figured out that on the corresponding worker node there was something wrong with the corresponding PV directory
/var/lib/kubelet/plugins/kubernetes.io/csi/pv/demo-internal-index/After deleting this directory on the worker node, everything works again. See also the discussion here.