Setting a node IP is important when you have multiple NICs for a Kubernetes host. This is my topology.
I was attempted to do some failure scenarios of Kubernetes virtual nodes by removing underlying datastores, turning them off, et el, and when they came back they were not added to the Kubernetes cluster. What did the logs say?
Oct 09 11:57:40 k8s-master01a kubelet[2878]: [map[address:192.168.120.201 type:ExternalIP] map[type:ExternalIP address:172.16.253.201] map[address:192.168.120.201 type:InternalIP] map[address:172.16.253.201 type:InternalIP]
Oct 09 11:57:40 k8s-master01a kubelet[2878]: doesn't match $setElementOrder list:
Oct 09 11:57:40 k8s-master01a kubelet[2878]: [map[type:ExternalIP] map[type:InternalIP] map[type:ExternalIP] map[type:InternalIP] map[type:Hostname]]
Oct 09 11:57:41 k8s-master01a kubelet[2878]: E1009 11:57:41.035254 2878 kubelet_node_status.go:391] Error updating node status, will retry: failed to patch status "{\"status\":{\"$setElementOrder/addresses\":[{\"type\":\
Oct 09 11:57:41 k8s-master01a kubelet[2878]: [map[address:192.168.120.201 type:ExternalIP] map[type:ExternalIP address:172.16.253.201] map[address:192.168.120.201 type:InternalIP] map[address:172.16.253.201 type:InternalIP]
Oct 09 11:57:41 k8s-master01a kubelet[2878]: doesn't match $setElementOrder list:
Oct 09 11:57:41 k8s-master01a kubelet[2878]: [map[type:ExternalIP] map[type:InternalIP] map[type:ExternalIP] map[type:InternalIP] map[type:Hostname]]
Oct 09 11:57:41 k8s-master01a kubelet[2878]: E1009 11:57:41.035312 2878 kubelet_node_status.go:379] Unable to update node status: update node status exceeds retry count
I can see here that a management IP is being used as the
In this case I have a Kubernetes node that has three interfaces. Ens160 has Kubernetes API IP 172.16.253.201, ens192 has an interface on OVS, and ens 224 has the node management IP of 192.168.120.201. Based on the logs the private management IP was trying to be used as the Kubernetes API IP (it should be external IP) and the Kubernetes API was trying to use node management (Internal IP) for the Network Container Plugin. Not ideal.
This results in the following:
[email protected]:~/vcp$ kubectl get node NAME STATUS ROLES AGE VERSION k8s-master01a NotReady master 15h v1.11.3 k8s-node01a NotReady none 4h v1.11.3 k8s-node02a NotReady none 4h v1.11.3
Editing the daemon for Kubelet can be done by editing the kubeadm.conf file. Edit with sudo vi /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
Add –node-ip
# Note: This dropin only works with kubeadm and kubelet v1.11+ [Service] Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf" Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.="yaml"" # This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env # This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use # the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file. EnvironmentFile=-/etc/default/kubelet ExecStart= ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS --cloud-provider=vsphere --cloud-config=/etc/vsphereconf/vsphere.conf --node-ip 172.16.253.201
After you edit the file and append the --node-ip value. I added each node --node-ip 172.16.253.201 and changed the IP respectively. You will need to reload the daemon and restart the kubelet service. Run sudo systemctl daemon-reload && sudo systemctl restart kubelet. Check the node status now:
[email protected]:~/vcp$ kubectl get node NAME STATUS ROLES AGE VERSION k8s-master01a Ready master 15h v1.11.3 k8s-node01a Ready. none 4h v1.11.3 k8s-node02a Ready node 4h v1.11.3
This will have an explicit IP address defined for public communications and won’t use the other interfaces to attempt to communicate.
Oh, the things you learn outside of a lab!