Symptoms
I was doing update on Kubernetes cluster I had not touch in a while when I noticed weird behaviour:
- the output of
get command was incomplete
- the
-o wide option had no effect on the command output
Example
The get nodes command is incomplete
1
2
3
4
5
|
➜ .kube git:(master) ✗ kubectl --kubeconfig dropship-dev-uat get nodes
NAME AGE
ip-172-20-33-51.ca-central-1.compute.internal 613d
ip-172-20-55-121.ca-central-1.compute.internal 628d
ip-172-20-85-9.ca-central-1.compute.internal 558d
|
The -o wide has no effect
1
2
3
4
5
6
|
➜ .kube git:(master) ✗ kubectl --kubeconfig dropship-dev-uat get nodes -o wide
NAME AGE
ip-172-20-33-51.ca-central-1.compute.internal 613d
ip-172-20-55-121.ca-central-1.compute.internal 628d
ip-172-20-85-9.ca-central-1.compute.internal 558d
|
Same behaviour can be seen for pods, svc etc
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
|
➜ .kube git:(master) ✗ kubectl --kubeconfig dropship-dev-uat get pods | head -3
NAME AGE
admin-deployment-c47f669fb-cnf2d 95d
admin-deployment-c47f669fb-qtldc 95d
➜ .kube git:(master) ✗ kubectl --kubeconfig dropship-dev-uat get pods -o wide | head -3
NAME AGE
admin-deployment-c47f669fb-cnf2d 95d
admin-deployment-c47f669fb-qtldc 95d
➜ .kube git:(master) ✗ kubectl --kubeconfig dropship-dev-uat get svc | head -3
NAME AGE
admin 503d
bestbuy 488d
➜ .kube git:(master) ✗ kubectl --kubeconfig dropship-dev-uat get svc -o wide | head -3
NAME AGE
admin 503d
bestbuy 488d
➜ .kube git:(master) ✗ kubectl --kubeconfig dropship-dev-uat get deploy -o wide | head -3
NAME AGE
admin-deployment 503d
bestbuy-deployment 488d
|
1
2
3
4
5
6
7
8
9
10
11
12
13
|
➜ .kube git:(master) ✗ kubectl --kubeconfig dropship-dev-uat get pods -o yaml | head -10
apiVersion: v1
items:
- apiVersion: v1
kind: Pod
metadata:
annotations:
deploymentDate: 2018-11-08T12:56:27+0000
kubernetes.io/limit-ranger: 'LimitRanger plugin set: cpu request for container
admin'
creationTimestamp: "2019-10-12T01:30:33Z"
....
....
|
Root cause and solution
The cluster was built in 2018 using kops and has not been upgraded since. While we were working on other projects, the local kubectl was upgraded few times and and currently the client is newer than server.
1
2
3
4
|
➜ .kube git:(master) ✗ kubectl --kubeconfig dropship-dev-uat version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-14T04:24:34Z", GoVersion:"go1.12.13", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"d2835416544f298c919e2ead3be3d0864b52323b", GitTreeState:"clean", BuildDate:"2018-02-07T11:55:20Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
|
The newer version of client does not seem to work with older server version - it would look like regression testing was not 100% complete.
I started to install older versions of the clients, going backwards. The version 1.11.7 from January 2019 works OK
1
2
3
4
5
6
7
8
|
VER=v1.11.7
curl -LO https://storage.googleapis.com/kubernetes-release/release/${VER}/bin/darwin/amd64/kubectl
mv kubectl kubectl1117
chmod +x kubectl1117
➜ .kube git:(master) ✗ ./kubectl1117 --kubeconfig dropship-dev-uat version
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.7", GitCommit:"65ecaf0671341311ce6aea0edab46ee69f65d59e", GitTreeState:"clean", BuildDate:"2019-01-24T19:32:00Z", GoVersion:"go1.10.7", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"d2835416544f298c919e2ead3be3d0864b52323b", GitTreeState:"clean", BuildDate:"2018-02-07T11:55:20Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
|
➜ .kube git:(master) ✗ ./kubectl1117 --kubeconfig dropship-dev-uat get nodes
NAME STATUS ROLES AGE VERSION
ip-172-20-33-51.ca-central-1.compute.internal Ready node 1y v1.9.3
ip-172-20-55-121.ca-central-1.compute.internal Ready master 1y v1.9.3
ip-172-20-85-9.ca-central-1.compute.internal Ready node 1y v1.9.3
# IP's masked
➜ .kube git:(master) ✗ ./kubectl1117 --kubeconfig dropship-dev-uat get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
ip-172-20-33-51.ca-central-1.compute.internal Ready node 1y v1.9.3 172.20.33.51 35.xx.xx.171 Debian GNU/Linux 8 (jessie) 4.4.115-k8s docker://17.3.2
ip-172-20-55-121.ca-central-1.compute.internal Ready master 1y v1.9.3 172.20.55.121 35.xx.xx.175 Debian GNU/Linux 8 (jessie) 4.4.115-k8s docker://17.3.2
ip-172-20-85-9.ca-central-1.compute.internal Ready node 1y v1.9.3 172.20.85.9 52.xx.xx.218 Debian GNU/Linux 8 (jessie) 4.4.115-k8s docker://17.3.2
➜ .kube git:(master) ✗ ./kubectl1117 --kubeconfig dropship-dev-uat get pods | head -3
NAME READY STATUS RESTARTS AGE
admin-deployment-c47f669fb-cnf2d 1/1 Running 0 95d
admin-deployment-c47f669fb-qtldc 1/1 Running 0 95d
➜ .kube git:(master) ✗ ./kubectl1117 --kubeconfig dropship-dev-uat get pods -o wide | head -3
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
admin-deployment-c47f669fb-cnf2d 1/1 Running 0 95d 100.96.5.136 ip-172-20-85-9.ca-central-1.compute.internal <none>
admin-deployment-c47f669fb-qtldc 1/1 Running 0 95d 100.96.4.211 ip-172-20-33-51.ca-central-1.compute.internal <none>
|
The alternative solution (better one) would be upgrading the cluster to latest version of Kubernetes. However, this is much more time consuming and it would require to bring all clusters to same version - DEV, UAT and PROD and would require thorough testing. This quick workaround allowed us to move ahead.
Writting it down in case somebody else has same issue.
Author
Miro Adamy
LastMod
2020-01-15
License
(c) 2006-2019 Miro Adamy