Lost ssh access to a node
Have you lost ssh access to one of your Kubernetes nodes? Why do you even need ssh access to nodes in the first place? Well, maybe something is stuck, or you need to see a config with your own eyes… I don’t know and I don’t care, they are your servers, not mine…
I’m assuming you have admin level into kubernetes API.
Talk is cheap, show me the code®:
$ NODE_NAME=master-0
$ kubectl create -n kube-system -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
name: root-shell-$NODE_NAME
namespace: kube-system
spec:
nodeName: $NODE_NAME
containers:
- command:
- /bin/cat
image: alpine:3
name: root-shell
tty: true
stdin: true
volumeMounts:
- mountPath: /host
name: hostroot
hostNetwork: true
hostPID: true
hostIPC: true
tolerations:
- effect: NoSchedule
operator: Exists
- effect: NoExecute
operator: Exists
volumes:
- hostPath:
path: /
name: hostroot
EOF
This pod will create a privileged POD into the node master-0 (change it to your node name) running /bin/cat forever. Now you simply exec into it and change the host’s root to pod’s root:$ kubectl -n kube-system exec -it root-shell-$NODE_NAME chroot /host /bin/bash
[root@master-0 /]# id
uid=0(root) gid=0(root) groups=0(root)
Profit!
PS: Here is a DaemonSet for the lazy
$ kubectl create -n kube-system -f - <<EOF
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: root-shell
namespace: kube-system
spec:
revisionHistoryLimit: 0
selector:
matchLabels:
app: root-shell
template:
metadata:
labels:
app: root-shell
spec:
terminationGracePeriodSeconds: 0
containers:
- command:
- /bin/cat
image: alpine:3
name: root-shell
tty: true
stdin: true
volumeMounts:
- mountPath: /host
name: hostroot
hostNetwork: true
tolerations:
- effect: NoSchedule
operator: Exists
- effect: NoExecute
operator: Exists
volumes:
- hostPath:
path: /
name: hostroot
EOF