Kops

INSTALLATION

Manual

download kops

curl -LO https://github.com/kubernetes/kops/releases/download/$(curl -s https://api.github.com/repos/kubernetes/kops/releases/latest | grep tag_name | cut -d '"' -f 4)/kops-linux-amd64
chmox +d kops-linux-amd64
sudo mv kops-linux-amd64 /usr/local/bin/kops

check version

kops version

create ROUTE53 hosted zone

aws route53 create-hosted-zone --name kops.mile-kitic.com --caller-reference 1

create s3 bucket

aws s3 mb s3://clusters-eu-central-1.kops.mile-kitic.com

export kops vars

export KOPS_STATE_STORE=s3://clusters-eu-central-1.kops.mile-kitic.com
export KOPS_CLUSTER_NAME=clusters-eu-central-1.kops.mile-kitic.com

create cluster

kops create cluster --name=clusters-eu-central-1.kops.mile-kitic.com \
  --zones=eu-central-1a \
  --node-count=2 \
  --master-count=1 \
  --master-size=t2.medium \
  --node-size=t2.medium \
  --networking flannel-vxlan \
  --topology private \
  --bastion \
  --image="ami-076431be05aaf8080" \
  --yes \
  --dry-run \
  --output yaml

Setup via Terraform

https://github.com/edesibe/kops

GENERAL KOPS

export AWS_REGION=eu-west-1
export AWS_SDK_LOAD_CONFIG=1
export AWS_PROFILE=prod
export KOPS_STATE_STORE=s3://kops-state
export KOPS_CLUSTER_NAME=prod.eu-west-1.cluster.corp.net

Export config

kops export kubecfg --kubeconfig ~/.kube/prod.config

▶ kops get secrets
TYPE			NAME						ID
Keypair			apiserver-aggregator		6600710412948936133508940282
Keypair			apiserver-aggregator-ca		6600710375870867289260536484
Keypair			apiserver-proxy-client		6600710414140481145213141368
Keypair			aws-iam-authenticator		6607795293393135215241074886
Keypair			ca							6600710377875095085181090847
Keypair			calico-client				6600710416288819959483742202
Keypair			etcd						6600710416871732328283990617
Keypair			etcd-client					6600710415724700764591966752
Keypair			kops						6600710413264784829873989096
Keypair			kube-controller-manager		6600710415352598513879815498
Keypair			kube-proxy					6600710416671716436823890644
Keypair			kube-scheduler				6600710412364908770893704482
Keypair			kubecfg						6600710416268079078836592763
Keypair			kubelet						6600710415504220626607130432
Keypair			kubelet-api					6600710415985092000487118843
Keypair			master						6600710413930713016653989982
SSHPublicKey	admin						35:26:bc:7e:10:af:f7:0a:b8:c6:86:0b:12:20:12:38
SSHPublicKey	admin1						6a:2b:c0:8d:9e:20:f8:b2:3c:41:c4:e0:80:7a:88:ca
Secret			admin
Secret			kube
Secret			kube-proxy
Secret			kubelet
Secret			system:controller_manager
Secret			system:dns
Secret			system:logging
Secret			system:monitoring
Secret			system:scheduler

Fingerprints

Fingerprint in kops secrets

2048 MD5:6a:2b:c0:8d:9e:20:f8:b2:3c:41:c4:e0:80:7a:88:ca milekitic.main (RSA)

Fingerprint on aws ui

writing RSA key
(stdin)= f9:70:ce:9e:1d:25:aa:b1:2c:f5:e3:95:01:c8:9c:00

Fingerprint on aws ui

writing RSA key
(stdin)= f9:70:ce:9e:1d:25:aa:b1:2c:f5:e3:95:01:c8:9c:00

TESTING

aws route53 create-hosted-zone --name kops-test.mile-kitic.com --caller-reference 1
aws s3 mb s3://clusters1.kops.mile-kitic.com
export KOPS_STATE_STORE=s3://clusters1.kops.mile-kitic.com

Create k8s cluster with kops

kops create cluster --name=clusters1.kops.mile-kitic.com \
  --zones=eu-central-1a \
  --node-count=2 \
  --master-count=3 \
  --master-size=t2.medium \
  --node-size=t2.medium \
  --networking flannel-vxlan \
  --topology private \
  --bastion \
  --image="ami-031c08681db8c400e" \
  --yes

kops has set your kubectl context to clusters1.kops.mile-kitic.com

Cluster is starting.  It should be ready in a few minutes.

Suggestions:
 * validate cluster: kops validate cluster
 * list nodes: kubectl get nodes --show-labels
 * ssh to the bastion: ssh -A -i ~/.ssh/id_rsa [email protected]
 * the admin user is specific to Debian. If not using Debian please use the appropriate user based on your OS.
 * read about installing addons at: https://github.com/kubernetes/kops/blob/master/docs/addons.md.

Wait for 15m.

Creation of some deployments and services

k create deployment nginx --image=nginx

[mjv@devko] ~
▶ k expose deployment nginx --name=nginx --port=80 --target-port=80 --protocol=TCP --type=LoadBalancer

[mjv@devko] ~
▶ k scale deployment nginx --replicas=4

[mjv@devko] ~
▶ k get all
NAME                         READY   STATUS    RESTARTS   AGE
pod/nginx-554b9c67f9-qd6t2   1/1     Running   0          36s
pod/nginx-554b9c67f9-spfh6   1/1     Running   0          3m35s
pod/nginx-554b9c67f9-ss4hq   1/1     Running   0          36s
pod/nginx-554b9c67f9-vwcvp   1/1     Running   0          36s

NAME                 TYPE           CLUSTER-IP       EXTERNAL-IP                                                                  PORT(S)        AGE
service/kubernetes   ClusterIP      100.64.0.1       <none>                                                                       443/TCP        20m
service/nginx        LoadBalancer   100.66.133.166   a83024b37b63a4bdc8026c5127961121-1734708003.eu-central-1.elb.amazonaws.com   80:31176/TCP   2m25s

NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx   4/4     4            4           3m35s

NAME                               DESIRED   CURRENT   READY   AGE
replicaset.apps/nginx-554b9c67f9   4         4         4       3m35s

Testing deployed app

[mjv@devko] ~
▶ curl a83024b37b63a4bdc8026c5127961121-1734708003.eu-central-1.elb.amazonaws.com
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

UPDATING KEYS

[mjv@devko] ~
▶ kops delete secret --name clusters1.kops.mile-kitic.com sshpublickey admin
[mjv@devko] ~
▶ kops create secret --name clusters1.kops.mile-kitic.com sshpublickey admin -i ~/.ssh/milekitic.main.pub
[mjv@devko] ~
▶ kops update cluster --yes
Using cluster from kubectl context: clusters1.kops.mile-kitic.com

I0206 11:30:03.431729    8103 executor.go:103] Tasks: 0 done / 128 total; 53 can run
I0206 11:30:04.444497    8103 executor.go:103] Tasks: 53 done / 128 total; 32 can run
I0206 11:30:05.191895    8103 executor.go:103] Tasks: 85 done / 128 total; 30 can run
I0206 11:30:06.608875    8103 executor.go:103] Tasks: 115 done / 128 total; 8 can run
I0206 11:30:06.753015    8103 dnsname.go:111] AliasTarget for "api.clusters1.kops.mile-kitic.com." is "api-clusters1-kops-devtec-eb2lhn-1760079497.eu-central-1.elb.amazonaws.com."
I0206 11:30:06.921588    8103 dnsname.go:111] AliasTarget for "bastion.clusters1.kops.mile-kitic.com." is "bastion-clusters1-kops-de-3r2cup-169256782.eu-central-1.elb.amazonaws.com."
I0206 11:30:07.111495    8103 executor.go:103] Tasks: 123 done / 128 total; 5 can run
I0206 11:30:07.362129    8103 executor.go:103] Tasks: 128 done / 128 total; 0 can run
I0206 11:30:07.362170    8103 dns.go:155] Pre-creating DNS records
I0206 11:30:07.749726    8103 update_cluster.go:305] Exporting kubecfg for cluster
kops has set your kubectl context to clusters1.kops.mile-kitic.com

Cluster changes have been applied to the cloud.


Changes may require instances to restart: kops rolling-update cluster

▶ kops rolling-update cluster --yes
Using cluster from kubectl context: clusters1.kops.mile-kitic.com

NAME					STATUS				NEEDUPDATE	READY	MIN	MAX	NODES
bastions				NeedsUpdate			1			0		1	1	0
master-eu-central-1a-1	NeedsUpdate			1			0		1	1	1
master-eu-central-1a-2	NeedsUpdate			1			0		1	1	1
master-eu-central-1a-3	NeedsUpdate			1			0		1	1	1
nodes					NeedsUpdate			2			0		2	2	2
I0206 11:30:19.944970    8146 instancegroups.go:303] Stopping instance "i-03b05933ae8c20b4f", in group "bastions.clusters1.kops.mile-kitic.com" (this may take a while).
I0206 11:30:20.135977    8146 instancegroups.go:196] waiting for 15s after terminating instance
I0206 11:30:35.136155    8146 instancegroups.go:200] Deleted a bastion instance, i-03b05933ae8c20b4f, and continuing with rolling-update.
W0206 11:30:35.738116    8146 aws_cloud.go:671] ignoring instance  as it is terminating: i-03b05933ae8c20b4f in autoscaling group: bastions.clusters1.kops.mile-kitic.com
I0206 11:30:35.947316    8146 instancegroups.go:163] Draining the node: "ip-172-20-34-191.eu-central-1.compute.internal".
WARNING: ignoring DaemonSet-managed Pods: kube-system/kube-flannel-ds-8gh7q
evicting pod "dns-controller-5f75dc8d88-s9jsn"
I0206 11:30:41.401851    8146 instancegroups.go:352] Waiting for 5s for pods to stabilize after draining.
I0206 11:30:46.402081    8146 instancegroups.go:183] deleting node "ip-172-20-34-191.eu-central-1.compute.internal" from kubernetes
I0206 11:30:46.454799    8146 instancegroups.go:301] Stopping instance "i-0d7080a71d22b4bff", node "ip-172-20-34-191.eu-central-1.compute.internal", in group "master-eu-central-1a-1.masters.clusters1.kops.mile-kitic.com" (this may take a while).
I0206 11:30:46.852537    8146 instancegroups.go:196] waiting for 15s after terminating instance
I0206 11:31:01.852785    8146 instancegroups.go:207] Validating the cluster.
W0206 11:31:02.476636    8146 aws_cloud.go:671] ignoring instance  as it is terminating: i-03b05933ae8c20b4f in autoscaling group: bastions.clusters1.kops.mile-kitic.com
W0206 11:31:02.476663    8146 aws_cloud.go:671] ignoring instance  as it is terminating: i-0d7080a71d22b4bff in autoscaling group: master-eu-central-1a-1.masters.clusters1.kops.mile-kitic.com
I0206 11:31:02.627617    8146 instancegroups.go:275] Cluster did not pass validation, will try again in "30s" until duration "15m0s" expires: machine "i-077578902eb59add0" has not yet joined cluster.
I0206 11:31:33.475336    8146 instancegroups.go:275] Cluster did not pass validation, will try again in "30s" until duration "15m0s" expires: machine "i-077578902eb59add0" has not yet joined cluster.
I0206 11:32:03.398420    8146 instancegroups.go:275] Cluster did not pass validation, will try again in "30s" until duration "15m0s" expires: machine "i-077578902eb59add0" has not yet joined cluster.
I0206 11:32:33.519982    8146 instancegroups.go:275] Cluster did not pass validation, will try again in "30s" until duration "15m0s" expires: machine "i-077578902eb59add0" has not yet joined cluster.
I0206 11:33:03.417486    8146 instancegroups.go:275] Cluster did not pass validation, will try again in "30s" until duration "15m0s" expires: machine "i-077578902eb59add0" has not yet joined cluster.
I0206 11:33:33.452308    8146 instancegroups.go:275] Cluster did not pass validation, will try again in "30s" until duration "15m0s" expires: machine "i-077578902eb59add0" has not yet joined cluster.
I0206 11:34:03.512656    8146 instancegroups.go:275] Cluster did not pass validation, will try again in "30s" until duration "15m0s" expires: machine "i-077578902eb59add0" has not yet joined cluster.
I0206 11:34:38.918875    8146 instancegroups.go:275] Cluster did not pass validation, will try again in "30s" until duration "15m0s" expires: machine "i-077578902eb59add0" has not yet joined cluster.
I0206 11:35:03.658700    8146 instancegroups.go:275] Cluster did not pass validation, will try again in "30s" until duration "15m0s" expires: machine "i-077578902eb59add0" has not yet joined cluster.
I0206 11:35:33.444978    8146 instancegroups.go:275] Cluster did not pass validation, will try again in "30s" until duration "15m0s" expires: kube-system pod "kube-scheduler-ip-172-20-57-26.eu-central-1.compute.internal" is pending.
I0206 11:36:03.471287    8146 instancegroups.go:275] Cluster did not pass validation, will try again in "30s" until duration "15m0s" expires: kube-system pod "kube-controller-manager-ip-172-20-57-26.eu-central-1.compute.internal" is pending.
I0206 11:36:33.476485    8146 instancegroups.go:278] Cluster validated.
I0206 11:36:34.044427    8146 instancegroups.go:163] Draining the node: "ip-172-20-63-161.eu-central-1.compute.internal".
WARNING: ignoring DaemonSet-managed Pods: kube-system/kube-flannel-ds-sl9b2
I0206 11:36:34.209124    8146 instancegroups.go:352] Waiting for 5s for pods to stabilize after draining.
I0206 11:36:39.209291    8146 instancegroups.go:183] deleting node "ip-172-20-63-161.eu-central-1.compute.internal" from kubernetes
I0206 11:36:39.258372    8146 instancegroups.go:301] Stopping instance "i-0ab7a87455f4c3c11", node "ip-172-20-63-161.eu-central-1.compute.internal", in group "master-eu-central-1a-2.masters.clusters1.kops.mile-kitic.com" (this may take a while).
I0206 11:36:39.515269    8146 instancegroups.go:196] waiting for 15s after terminating instance
I0206 11:36:54.515463    8146 instancegroups.go:207] Validating the cluster.
W0206 11:36:55.098074    8146 aws_cloud.go:671] ignoring instance  as it is terminating: i-0ab7a87455f4c3c11 in autoscaling group: master-eu-central-1a-2.masters.clusters1.kops.mile-kitic.com
I0206 11:36:55.246047    8146 instancegroups.go:275] Cluster did not pass validation, will try again in "30s" until duration "15m0s" expires: machine "i-0aae77dd6dde4a823" has not yet joined cluster.
W0206 11:37:25.888510    8146 aws_cloud.go:671] ignoring instance  as it is terminating: i-0ab7a87455f4c3c11 in autoscaling group: master-eu-central-1a-2.masters.clusters1.kops.mile-kitic.com
I0206 11:37:26.045229    8146 instancegroups.go:275] Cluster did not pass validation, will try again in "30s" until duration "15m0s" expires: machine "i-0aae77dd6dde4a823" has not yet joined cluster.
I0206 11:37:56.052929    8146 instancegroups.go:275] Cluster did not pass validation, will try again in "30s" until duration "15m0s" expires: machine "i-0aae77dd6dde4a823" has not yet joined cluster.
I0206 11:38:26.025906    8146 instancegroups.go:275] Cluster did not pass validation, will try again in "30s" until duration "15m0s" expires: machine "i-0aae77dd6dde4a823" has not yet joined cluster.
I0206 11:38:56.011925    8146 instancegroups.go:275] Cluster did not pass validation, will try again in "30s" until duration "15m0s" expires: machine "i-0aae77dd6dde4a823" has not yet joined cluster.
I0206 11:39:26.472982    8146 instancegroups.go:275] Cluster did not pass validation, will try again in "30s" until duration "15m0s" expires: machine "i-0aae77dd6dde4a823" has not yet joined cluster.
I0206 11:39:55.965713    8146 instancegroups.go:275] Cluster did not pass validation, will try again in "30s" until duration "15m0s" expires: machine "i-0aae77dd6dde4a823" has not yet joined cluster.
I0206 11:40:25.984054    8146 instancegroups.go:275] Cluster did not pass validation, will try again in "30s" until duration "15m0s" expires: machine "i-0aae77dd6dde4a823" has not yet joined cluster.
I0206 11:40:56.057183    8146 instancegroups.go:275] Cluster did not pass validation, will try again in "30s" until duration "15m0s" expires: machine "i-0aae77dd6dde4a823" has not yet joined cluster.
I0206 11:41:25.968674    8146 instancegroups.go:275] Cluster did not pass validation, will try again in "30s" until duration "15m0s" expires: kube-system pod "etcd-manager-events-ip-172-20-39-249.eu-central-1.compute.internal" is pending.
I0206 11:41:56.057730    8146 instancegroups.go:278] Cluster validated.
I0206 11:41:56.671697    8146 instancegroups.go:163] Draining the node: "ip-172-20-63-105.eu-central-1.compute.internal".
WARNING: ignoring DaemonSet-managed Pods: kube-system/kube-flannel-ds-4sbpv
evicting pod "dns-controller-5f75dc8d88-2wh9g"
I0206 11:42:04.127647    8146 instancegroups.go:352] Waiting for 5s for pods to stabilize after draining.
I0206 11:42:09.127843    8146 instancegroups.go:183] deleting node "ip-172-20-63-105.eu-central-1.compute.internal" from kubernetes
I0206 11:42:09.183416    8146 instancegroups.go:301] Stopping instance "i-0a17a9f53d9405344", node "ip-172-20-63-105.eu-central-1.compute.internal", in group "master-eu-central-1a-3.masters.clusters1.kops.mile-kitic.com" (this may take a while).
I0206 11:42:09.599246    8146 instancegroups.go:196] waiting for 15s after terminating instance
I0206 11:42:24.599415    8146 instancegroups.go:207] Validating the cluster.
W0206 11:42:25.482382    8146 aws_cloud.go:671] ignoring instance  as it is terminating: i-0a17a9f53d9405344 in autoscaling group: master-eu-central-1a-3.masters.clusters1.kops.mile-kitic.com
I0206 11:42:25.681706    8146 instancegroups.go:275] Cluster did not pass validation, will try again in "30s" until duration "15m0s" expires: machine "i-0e9d3aca6621a71b6" has not yet joined cluster.
I0206 11:42:56.834121    8146 instancegroups.go:275] Cluster did not pass validation, will try again in "30s" until duration "15m0s" expires: machine "i-0e9d3aca6621a71b6" has not yet joined cluster.
I0206 11:43:26.464015    8146 instancegroups.go:275] Cluster did not pass validation, will try again in "30s" until duration "15m0s" expires: machine "i-0e9d3aca6621a71b6" has not yet joined cluster.
I0206 11:43:56.713517    8146 instancegroups.go:275] Cluster did not pass validation, will try again in "30s" until duration "15m0s" expires: machine "i-0e9d3aca6621a71b6" has not yet joined cluster.
I0206 11:44:26.402007    8146 instancegroups.go:275] Cluster did not pass validation, will try again in "30s" until duration "15m0s" expires: machine "i-0e9d3aca6621a71b6" has not yet joined cluster.
I0206 11:44:56.478691    8146 instancegroups.go:275] Cluster did not pass validation, will try again in "30s" until duration "15m0s" expires: machine "i-0e9d3aca6621a71b6" has not yet joined cluster, component "etcd-1" is unhealthy.
I0206 11:45:26.448815    8146 instancegroups.go:275] Cluster did not pass validation, will try again in "30s" until duration "15m0s" expires: machine "i-0e9d3aca6621a71b6" has not yet joined cluster.
I0206 11:45:56.478514    8146 instancegroups.go:275] Cluster did not pass validation, will try again in "30s" until duration "15m0s" expires: master "ip-172-20-37-197.eu-central-1.compute.internal" is not ready, kube-system pod "kube-flannel-ds-mk6fv" is pending.
I0206 11:46:26.556229    8146 instancegroups.go:278] Cluster validated.
I0206 11:46:27.134611    8146 instancegroups.go:163] Draining the node: "ip-172-20-48-188.eu-central-1.compute.internal".
WARNING: ignoring DaemonSet-managed Pods: kube-system/kube-flannel-ds-6g4hv
evicting pod "kube-dns-autoscaler-577b4774b5-5xw8z"
evicting pod "nginx-554b9c67f9-ss4hq"
evicting pod "nginx-554b9c67f9-qd6t2"
evicting pod "kube-dns-5fdb85bb5b-5mmpz"
I0206 11:47:04.621707    8146 instancegroups.go:352] Waiting for 5s for pods to stabilize after draining.
I0206 11:47:09.621867    8146 instancegroups.go:183] deleting node "ip-172-20-48-188.eu-central-1.compute.internal" from kubernetes
I0206 11:47:09.667532    8146 instancegroups.go:301] Stopping instance "i-0000187c6eac049f0", node "ip-172-20-48-188.eu-central-1.compute.internal", in group "nodes.clusters1.kops.mile-kitic.com" (this may take a while).
I0206 11:47:10.049967    8146 instancegroups.go:196] waiting for 15s after terminating instance
I0206 11:47:25.050143    8146 instancegroups.go:207] Validating the cluster.
W0206 11:47:25.701822    8146 aws_cloud.go:671] ignoring instance  as it is terminating: i-0000187c6eac049f0 in autoscaling group: nodes.clusters1.kops.mile-kitic.com
I0206 11:47:25.881349    8146 instancegroups.go:275] Cluster did not pass validation, will try again in "30s" until duration "15m0s" expires: machine "i-081e44b03ff708f01" has not yet joined cluster.
I0206 11:47:56.643679    8146 instancegroups.go:275] Cluster did not pass validation, will try again in "30s" until duration "15m0s" expires: machine "i-081e44b03ff708f01" has not yet joined cluster.
I0206 11:48:27.125269    8146 instancegroups.go:275] Cluster did not pass validation, will try again in "30s" until duration "15m0s" expires: machine "i-081e44b03ff708f01" has not yet joined cluster.
I0206 11:48:56.668502    8146 instancegroups.go:275] Cluster did not pass validation, will try again in "30s" until duration "15m0s" expires: machine "i-081e44b03ff708f01" has not yet joined cluster.
I0206 11:49:26.682963    8146 instancegroups.go:275] Cluster did not pass validation, will try again in "30s" until duration "15m0s" expires: node "ip-172-20-35-38.eu-central-1.compute.internal" is not ready, kube-system pod "kube-flannel-ds-mbkwx" is pending, kube-system pod "kube-proxy-ip-172-20-35-38.eu-central-1.compute.internal" is pending.
I0206 11:49:56.684333    8146 instancegroups.go:278] Cluster validated.
I0206 11:49:56.684380    8146 instancegroups.go:163] Draining the node: "ip-172-20-51-237.eu-central-1.compute.internal".
WARNING: ignoring DaemonSet-managed Pods: kube-system/kube-flannel-ds-qssf8
evicting pod "kube-dns-autoscaler-577b4774b5-hb94p"
evicting pod "nginx-554b9c67f9-jxrh6"
evicting pod "nginx-554b9c67f9-vwcvp"
evicting pod "kube-dns-5fdb85bb5b-9bbh6"
evicting pod "nginx-554b9c67f9-kxcp2"
evicting pod "nginx-554b9c67f9-spfh6"
evicting pod "kube-dns-5fdb85bb5b-tnxkx"
error when evicting pod "kube-dns-5fdb85bb5b-tnxkx" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
evicting pod "kube-dns-5fdb85bb5b-tnxkx"
error when evicting pod "kube-dns-5fdb85bb5b-tnxkx" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
evicting pod "kube-dns-5fdb85bb5b-tnxkx"
error when evicting pod "kube-dns-5fdb85bb5b-tnxkx" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
evicting pod "kube-dns-5fdb85bb5b-tnxkx"
error when evicting pod "kube-dns-5fdb85bb5b-tnxkx" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
evicting pod "kube-dns-5fdb85bb5b-tnxkx"
error when evicting pod "kube-dns-5fdb85bb5b-tnxkx" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
evicting pod "kube-dns-5fdb85bb5b-tnxkx"
error when evicting pod "kube-dns-5fdb85bb5b-tnxkx" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
evicting pod "kube-dns-5fdb85bb5b-tnxkx"
error when evicting pod "kube-dns-5fdb85bb5b-tnxkx" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
evicting pod "kube-dns-5fdb85bb5b-tnxkx"
error when evicting pod "kube-dns-5fdb85bb5b-tnxkx" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
evicting pod "kube-dns-5fdb85bb5b-tnxkx"
I0206 11:51:17.653388    8146 instancegroups.go:352] Waiting for 5s for pods to stabilize after draining.
I0206 11:51:22.653583    8146 instancegroups.go:183] deleting node "ip-172-20-51-237.eu-central-1.compute.internal" from kubernetes
I0206 11:51:22.712921    8146 instancegroups.go:301] Stopping instance "i-04f1cf9cfdea8641c", node "ip-172-20-51-237.eu-central-1.compute.internal", in group "nodes.clusters1.kops.mile-kitic.com" (this may take a while).
I0206 11:51:23.115261    8146 instancegroups.go:196] waiting for 15s after terminating instance
I0206 11:51:38.115455    8146 instancegroups.go:207] Validating the cluster.
W0206 11:51:38.917894    8146 aws_cloud.go:671] ignoring instance  as it is terminating: i-04f1cf9cfdea8641c in autoscaling group: nodes.clusters1.kops.mile-kitic.com
I0206 11:51:39.089199    8146 instancegroups.go:275] Cluster did not pass validation, will try again in "30s" until duration "15m0s" expires: InstanceGroup "nodes" did not have enough nodes 1 vs 2.
W0206 11:52:09.695008    8146 aws_cloud.go:671] ignoring instance  as it is terminating: i-04f1cf9cfdea8641c in autoscaling group: nodes.clusters1.kops.mile-kitic.com
I0206 11:52:09.865767    8146 instancegroups.go:275] Cluster did not pass validation, will try again in "30s" until duration "15m0s" expires: machine "i-0cf444ab3994395d4" has not yet joined cluster.
W0206 11:52:39.701371    8146 aws_cloud.go:671] ignoring instance  as it is terminating: i-04f1cf9cfdea8641c in autoscaling group: nodes.clusters1.kops.mile-kitic.com
I0206 11:52:39.866654    8146 instancegroups.go:275] Cluster did not pass validation, will try again in "30s" until duration "15m0s" expires: machine "i-0cf444ab3994395d4" has not yet joined cluster.
I0206 11:53:09.868757    8146 instancegroups.go:275] Cluster did not pass validation, will try again in "30s" until duration "15m0s" expires: machine "i-0cf444ab3994395d4" has not yet joined cluster.
I0206 11:53:39.966089    8146 instancegroups.go:275] Cluster did not pass validation, will try again in "30s" until duration "15m0s" expires: node "ip-172-20-36-135.eu-central-1.compute.internal" is not ready, kube-system pod "kube-flannel-ds-hstqv" is pending, kube-system pod "kube-proxy-ip-172-20-36-135.eu-central-1.compute.internal" is pending.
I0206 11:54:09.903339    8146 instancegroups.go:278] Cluster validated.
I0206 11:54:09.903410    8146 rollingupdate.go:184] Rolling update completed for cluster "clusters1.kops.mile-kitic.com"!
Testing deployment url after update
▶ curl a83024b37b63a4bdc8026c5127961121-1734708003.eu-central-1.elb.amazonaws.com
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

CHAGING IMAGE FROM COREOS TO UBUNTU

▶ kops get ig nodes -o yaml
Using cluster from kubectl context: clusters1.kops.mile-kitic.com

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2020-02-06T10:04:55Z"
  labels:
    kops.k8s.io/cluster: clusters1.kops.mile-kitic.com
  name: nodes
spec:
  image: ami-031c08681db8c400e
  machineType: t2.medium
  maxSize: 2
  minSize: 2
  nodeLabels:
    kops.k8s.io/instancegroup: nodes
  role: Node
  subnets:
  - eu-central-1a
[mjv@devko] ~
▶ kops edit ig nodes
Using cluster from kubectl context: clusters1.kops.mile-kitic.com

[mjv@devko] ~
▶ kops get ig nodes -o yaml
Using cluster from kubectl context: clusters1.kops.mile-kitic.com

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2020-02-06T10:04:55Z"
  generation: 1
  labels:
    kops.k8s.io/cluster: clusters1.kops.mile-kitic.com
  name: nodes
spec:
  image: ami-0b418580298265d5c
  machineType: t2.medium
  maxSize: 2
  minSize: 2
  nodeLabels:
    kops.k8s.io/instancegroup: nodes
  role: Node
  subnets:
  - eu-central-1a

TRIGGERING IMAGE UPDATE

[mjv@devko] ~
▶ kops update cluster clusters1.kops.mile-kitic.com --yes
I0206 11:59:52.935094    9285 executor.go:103] Tasks: 0 done / 128 total; 53 can run
I0206 11:59:53.852663    9285 executor.go:103] Tasks: 53 done / 128 total; 32 can run
I0206 11:59:54.466277    9285 executor.go:103] Tasks: 85 done / 128 total; 30 can run
I0206 11:59:55.800083    9285 executor.go:103] Tasks: 115 done / 128 total; 8 can run
I0206 11:59:55.944026    9285 dnsname.go:111] AliasTarget for "api.clusters1.kops.mile-kitic.com." is "api-clusters1-kops-devtec-eb2lhn-1760079497.eu-central-1.elb.amazonaws.com."
I0206 11:59:56.083104    9285 dnsname.go:111] AliasTarget for "bastion.clusters1.kops.mile-kitic.com." is "bastion-clusters1-kops-de-3r2cup-169256782.eu-central-1.elb.amazonaws.com."
I0206 11:59:56.278397    9285 executor.go:103] Tasks: 123 done / 128 total; 5 can run
I0206 11:59:56.513361    9285 executor.go:103] Tasks: 128 done / 128 total; 0 can run
I0206 11:59:56.513433    9285 dns.go:155] Pre-creating DNS records
I0206 11:59:56.913388    9285 update_cluster.go:305] Exporting kubecfg for cluster
kops has set your kubectl context to clusters1.kops.mile-kitic.com

Cluster changes have been applied to the cloud.


Changes may require instances to restart: kops rolling-update cluster

[mjv@devko] ~
▶ kops rolling-update cluster
Using cluster from kubectl context: clusters1.kops.mile-kitic.com

NAME			STATUS		NEEDUPDATE	READY	MIN	MAX	NODES
bastions		Ready		0		1	1	1	0
master-eu-central-1a-1	Ready		0		1	1	1	1
master-eu-central-1a-2	Ready		0		1	1	1	1
master-eu-central-1a-3	Ready		0		1	1	1	1
nodes			NeedsUpdate	2		0	2	2	2

Must specify --yes to rolling-update.

[mjv@devko] ~
▶ kops rolling-update cluster --yes
Using cluster from kubectl context: clusters1.kops.mile-kitic.com

NAME			STATUS		NEEDUPDATE	READY	MIN	MAX	NODES
bastions		Ready		0		1	1	1	0
master-eu-central-1a-1	Ready		0		1	1	1	1
master-eu-central-1a-2	Ready		0		1	1	1	1
master-eu-central-1a-3	Ready		0		1	1	1	1
nodes			NeedsUpdate	2		0	2	2	2
I0206 12:00:21.191250    9327 instancegroups.go:163] Draining the node: "ip-172-20-35-38.eu-central-1.compute.internal".
WARNING: ignoring DaemonSet-managed Pods: kube-system/kube-flannel-ds-mbkwx
evicting pod "nginx-554b9c67f9-tm7v2"
evicting pod "kube-dns-5fdb85bb5b-jgdwn"
evicting pod "kube-dns-5fdb85bb5b-9xr7d"
evicting pod "nginx-554b9c67f9-5fpg9"
evicting pod "nginx-554b9c67f9-p7hxf"
evicting pod "kube-dns-autoscaler-577b4774b5-6bz5h"
evicting pod "nginx-554b9c67f9-5dwsn"
error when evicting pod "kube-dns-5fdb85bb5b-9xr7d" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
evicting pod "kube-dns-5fdb85bb5b-9xr7d"
error when evicting pod "kube-dns-5fdb85bb5b-9xr7d" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
evicting pod "kube-dns-5fdb85bb5b-9xr7d"
error when evicting pod "kube-dns-5fdb85bb5b-9xr7d" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
evicting pod "kube-dns-5fdb85bb5b-9xr7d"
error when evicting pod "kube-dns-5fdb85bb5b-9xr7d" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
evicting pod "kube-dns-5fdb85bb5b-9xr7d"
error when evicting pod "kube-dns-5fdb85bb5b-9xr7d" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
evicting pod "kube-dns-5fdb85bb5b-9xr7d"
error when evicting pod "kube-dns-5fdb85bb5b-9xr7d" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
evicting pod "kube-dns-5fdb85bb5b-9xr7d"
I0206 12:01:24.525570    9327 instancegroups.go:352] Waiting for 5s for pods to stabilize after draining.
I0206 12:01:29.525843    9327 instancegroups.go:183] deleting node "ip-172-20-35-38.eu-central-1.compute.internal" from kubernetes
I0206 12:01:29.582184    9327 instancegroups.go:301] Stopping instance "i-081e44b03ff708f01", node "ip-172-20-35-38.eu-central-1.compute.internal", in group "nodes.clusters1.kops.mile-kitic.com" (this may take a while).
I0206 12:01:29.967236    9327 instancegroups.go:196] waiting for 15s after terminating instance
I0206 12:01:44.967390    9327 instancegroups.go:207] Validating the cluster.
W0206 12:01:45.770007    9327 aws_cloud.go:671] ignoring instance  as it is terminating: i-081e44b03ff708f01 in autoscaling group: nodes.clusters1.kops.mile-kitic.com
I0206 12:01:46.009333    9327 instancegroups.go:275] Cluster did not pass validation, will try again in "30s" until duration "15m0s" expires: machine "i-083775669fdfe0ab6" has not yet joined cluster.
I0206 12:02:16.895982    9327 instancegroups.go:275] Cluster did not pass validation, will try again in "30s" until duration "15m0s" expires: machine "i-083775669fdfe0ab6" has not yet joined cluster.
I0206 12:02:46.817621    9327 instancegroups.go:275] Cluster did not pass validation, will try again in "30s" until duration "15m0s" expires: machine "i-083775669fdfe0ab6" has not yet joined cluster.
I0206 12:03:16.837203    9327 instancegroups.go:275] Cluster did not pass validation, will try again in "30s" until duration "15m0s" expires: machine "i-083775669fdfe0ab6" has not yet joined cluster.
I0206 12:03:46.921865    9327 instancegroups.go:275] Cluster did not pass validation, will try again in "30s" until duration "15m0s" expires: machine "i-083775669fdfe0ab6" has not yet joined cluster.
I0206 12:04:16.887691    9327 instancegroups.go:275] Cluster did not pass validation, will try again in "30s" until duration "15m0s" expires: node "ip-172-20-60-131.eu-central-1.compute.internal" is not ready, kube-system pod "kube-flannel-ds-68tpq" is pending, kube-system pod "kube-proxy-ip-172-20-60-131.eu-central-1.compute.internal" is pending.
I0206 12:04:46.841627    9327 instancegroups.go:278] Cluster validated.
I0206 12:04:46.841691    9327 instancegroups.go:163] Draining the node: "ip-172-20-36-135.eu-central-1.compute.internal".
WARNING: ignoring DaemonSet-managed Pods: kube-system/kube-flannel-ds-hstqv
evicting pod "nginx-554b9c67f9-nzs86"
evicting pod "kube-dns-autoscaler-577b4774b5-4p9tv"
evicting pod "kube-dns-5fdb85bb5b-ppj85"
evicting pod "nginx-554b9c67f9-g5kmm"
evicting pod "nginx-554b9c67f9-vxnzg"
evicting pod "kube-dns-5fdb85bb5b-v6trb"
evicting pod "nginx-554b9c67f9-s44cn"
error when evicting pod "kube-dns-5fdb85bb5b-v6trb" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
evicting pod "kube-dns-5fdb85bb5b-v6trb"
error when evicting pod "kube-dns-5fdb85bb5b-v6trb" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
evicting pod "kube-dns-5fdb85bb5b-v6trb"
error when evicting pod "kube-dns-5fdb85bb5b-v6trb" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
evicting pod "kube-dns-5fdb85bb5b-v6trb"
error when evicting pod "kube-dns-5fdb85bb5b-v6trb" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
evicting pod "kube-dns-5fdb85bb5b-v6trb"
error when evicting pod "kube-dns-5fdb85bb5b-v6trb" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
evicting pod "kube-dns-5fdb85bb5b-v6trb"
I0206 12:05:45.100556    9327 instancegroups.go:352] Waiting for 5s for pods to stabilize after draining.
I0206 12:05:50.100788    9327 instancegroups.go:183] deleting node "ip-172-20-36-135.eu-central-1.compute.internal" from kubernetes
I0206 12:05:50.145809    9327 instancegroups.go:301] Stopping instance "i-0cf444ab3994395d4", node "ip-172-20-36-135.eu-central-1.compute.internal", in group "nodes.clusters1.kops.mile-kitic.com" (this may take a while).
I0206 12:05:50.575507    9327 instancegroups.go:196] waiting for 15s after terminating instance
I0206 12:06:05.575723    9327 instancegroups.go:207] Validating the cluster.
W0206 12:06:06.352480    9327 aws_cloud.go:671] ignoring instance  as it is terminating: i-0cf444ab3994395d4 in autoscaling group: nodes.clusters1.kops.mile-kitic.com
I0206 12:06:06.597811    9327 instancegroups.go:275] Cluster did not pass validation, will try again in "30s" until duration "15m0s" expires: InstanceGroup "nodes" did not have enough nodes 1 vs 2.
I0206 12:06:37.523124    9327 instancegroups.go:275] Cluster did not pass validation, will try again in "30s" until duration "15m0s" expires: machine "i-0cc202ef7f82e12f0" has not yet joined cluster.
I0206 12:07:07.454126    9327 instancegroups.go:275] Cluster did not pass validation, will try again in "30s" until duration "15m0s" expires: machine "i-0cc202ef7f82e12f0" has not yet joined cluster.
I0206 12:07:37.423231    9327 instancegroups.go:275] Cluster did not pass validation, will try again in "30s" until duration "15m0s" expires: machine "i-0cc202ef7f82e12f0" has not yet joined cluster.
I0206 12:08:07.408808    9327 instancegroups.go:275] Cluster did not pass validation, will try again in "30s" until duration "15m0s" expires: machine "i-0cc202ef7f82e12f0" has not yet joined cluster.
I0206 12:08:37.406871    9327 instancegroups.go:278] Cluster validated.
I0206 12:08:37.406905    9327 rollingupdate.go:184] Rolling update completed for cluster "clusters1.kops.mile-kitic.com"!

[mjv@devko] ~
▶ k get nodes -o wide
NAME                                             STATUS   ROLES    AGE    VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                                        KERNEL-VERSION    CONTAINER-RUNTIME
ip-172-20-37-197.eu-central-1.compute.internal   Ready    master   24m    v1.15.7   172.20.37.197   <none>        Container Linux by CoreOS 2303.3.0 (Rhyolite)   4.19.86-coreos    docker://18.6.3
ip-172-20-39-249.eu-central-1.compute.internal   Ready    master   29m    v1.15.7   172.20.39.249   <none>        Container Linux by CoreOS 2303.3.0 (Rhyolite)   4.19.86-coreos    docker://18.6.3
ip-172-20-51-213.eu-central-1.compute.internal   Ready    node     115s   v1.15.7   172.20.51.213   <none>        Ubuntu 18.04.3 LTS                              4.15.0-1057-aws   docker://18.6.3
ip-172-20-57-26.eu-central-1.compute.internal    Ready    master   34m    v1.15.7   172.20.57.26    <none>        Container Linux by CoreOS 2303.3.0 (Rhyolite)   4.19.86-coreos    docker://18.6.3
ip-172-20-60-131.eu-central-1.compute.internal   Ready    node     6m1s   v1.15.7   172.20.60.131   <none>        Ubuntu 18.04.3 LTS                              4.15.0-1057-aws   docker://18.6.3

RESOURCES

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html https://medium.com/bench-engineering/deploying-kubernetes-clusters-with-kops-and-terraform-832b89250e8e https://github.com/kubernetes/kops/blob/master/docs/cli/kops_get_clusters.md https://github.com/kubernetes/kops/blob/master/docs/state.md https://medium.com/bench-engineering/deploying-kubernetes-clusters-with-kops-and-terraform-832b89250e8e https://github.com/kubernetes/kops/blob/master/docs/terraform.md https://github.com/kubernetes/kops/blob/master/docs/security.md https://github.com/kubernetes/kops/blob/master/docs/bastion.md https://github.com/kubernetes/kops/blob/master/docs/cluster_spec.md
https://github.com/kubernetes/kops/blob/master/docs/security.md https://stackoverflow.com/questions/55046152/changing-the-key-to-the-kops-cluster-doesnt-let-me-ssh-to-master-nodes-from-bas
https://medium.com/@ikod/setting-up-kubernetes-cluster-in-aws-with-private-topology-7552374c7d7a https://github.com/kubernetes/kops/blob/master/docs/examples/kops-tests-private-net-bastion-host.md
https://github.com/kubernetes/kops/issues/2142 https://medium.com/bench-engineering/deploying-kubernetes-clusters-with-kops-and-terraform-832b89250e8e
https://kops.sigs.k8s.io/cli/kops_create_cluster/
https://github.com/kubernetes/kops