MEHColeman

A Kubernetes Walkthrough

2022-08-28T00:00:00+01:00

Here are all the notes I need to get me from zero to installation, to deployment, ingress, encryption and monitoring of a service!

Create a Cluster

You can use various k8s implementations: Minikube or Rancher (with k3s) are good starting points, but check out minikube, k8s, k3s, k0s, microk8s, etc. They each have their useful features (like minikube tunnel, minikube ip or traefik), but are generally compatible with each other.

To administer your kubernetes service, you must have an administration tool - kubectl.

Check your k8s is up and running:

$ minikube version
minikube version: v1.18.0
commit: ec61815d60f66a6e4f6353030a40b12362557caa-dirty

$ minikube start
* minikube v1.18.0 on Ubuntu 18.04 (amd64)
* Using the none driver based on existing profile

X The requested memory allocation of 2200MiB does not leave room for system overhead (total system memory: 2460MiB). You may face stability issues.
* Suggestion: Start minikube with less memory allocated: 'minikube start --memory=2200mb'

* Starting control plane node minikube in cluster minikube
* Restarting existing none bare metal machine for "minikube" ...
* OS release is Ubuntu 18.04.5 LTS
* Preparing Kubernetes v1.20.2 on Docker 19.03.13 ...
  - kubelet.resolv-conf=/run/systemd/resolve/resolv.conf
* Configuring local host environment ...
* Verifying Kubernetes components...
  - Using image gcr.io/k8s-minikube/storage-provisioner:v4
* Enabled addons: storage-provisioner, default-storageclass
* Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

Check that kubectl is installed:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.4", GitCommit:"e87da0bd6e03ec3fea7933c4b5263d151aafd07c", GitTreeState:"clean", BuildDate:"2021-02-18T16:12:00Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"faecb196815e248d3ecfb03c680a4507229c2a56", GitTreeState:"clean", BuildDate:"2021-01-13T13:20:00Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}

Get info about cluster:

$ kubectl cluster-info
Kubernetes control plane is running at https://10.0.0.6:8443
KubeDNS is running at https://10.0.0.6:8443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

View node info:

$ kubectl get nodes
NAME       STATUS   ROLES                  AGE   VERSION
minikube   Ready    control-plane,master   77s   v1.20.2

Deploy an App

Create a deployment with a publicly available docker image:

$ kubectl create deployment  kubernetes-bootcamp --image=gcr.io/google-samples/kubernetes-bootcamp:v1
deployment.apps/kubernetes-bootcamp created

Show deployments:

$ kubectl get deployments
NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
kubernetes-bootcamp   1/1     1            1           68s

Create a proxy that will forward requests into the kubernetes private network:

$ kubectl proxy
Starting to serve on 127.0.0.1:8001

You can then make simple requests to this proxy, like this:

$ curl http://localhost:8001/version
{
  "major": "1",
  "minor": "20",
  "gitVersion": "v1.20.2",
  "gitCommit": "faecb196815e248d3ecfb03c680a4507229c2a56",
  "gitTreeState": "clean",
  "buildDate": "2021-01-13T13:20:00Z",
  "goVersion": "go1.15.5",
  "compiler": "gc",
  "platform": "linux/amd64"

In order to pass more sophisticated messages to one of our managed applications, we need its id:

$ kubectl get pods
NAME                                   READY   STATUS    RESTARTS   AGE
kubernetes-bootcamp-57978f5f5d-c4nng   1/1     Running   0          10m

$ kubectl get pods -o go-template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}'
kubernetes-bootcamp-57978f5f5d-c4nng

$ export POD_NAME=$(kubectl get pods -o go-template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}')

Then we can send messages to this pod:

$ curl http://localhost:8001/api/v1/namespaces/default/pods/$POD_NAME/proxy
Hello Kubernetes bootcamp! | Running on: kubernetes-bootcamp-fb5c67579-c8m9c | v=1

We can also message the proxy api directly:

$ curl http://localhost:8001/api/v1/namespaces/default/pods/$POD_NAME

This requests shows you the full pod configuration:

{
  "kind": "Pod",
  "apiVersion": "v1",
  "metadata": {
    "name": "kubernetes-bootcamp-57978f5f5d-c4nng",
    "generateName": "kubernetes-bootcamp-57978f5f5d-",
    "namespace": "default",
    "uid": "8fe45ce3-0a35-4149-a6be-2117ee7cbf7f",
    "resourceVersion": "997",
    "creationTimestamp": "2022-08-24T12:26:29Z",
    "labels": {
      "app": "kubernetes-bootcamp",
      "pod-template-hash": "57978f5f5d"
    },
    "ownerReferences": [
      {
        "apiVersion": "apps/v1",
        "kind": "ReplicaSet",
        "name": "kubernetes-bootcamp-57978f5f5d",
        "uid": "b6042aad-5fec-4a22-bca5-e6c562aaf6ab",
        "controller": true,
        "blockOwnerDeletion": true
      }
    ],
    "managedFields": [
      {
        "manager": "kube-controller-manager",
        "operation": "Update",
        "apiVersion": "v1",
        "time": "2022-08-24T12:26:29Z",
        "fieldsType": "FieldsV1",
        "fieldsV1": {"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:app":{},"f:pod-template-hash":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"b6042aad-5fec-4a22-bca5-e6c562aaf6ab\"}":{".":{},"f:apiVersion":{},"f:blockOwnerDeletion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:spec":{"f:containers":{"k:{\"name\":\"kubernetes-bootcamp\"}":{".":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{}}}
      },
      {
        "manager": "kubelet",
        "operation": "Update",
        "apiVersion": "v1",
        "time": "2022-08-24T12:26:32Z",
        "fieldsType": "FieldsV1",
        "fieldsV1": {"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"172.18.0.6\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}
      }
    ]
  },
  "spec": {
    "volumes": [
      {
        "name": "default-token-9dbv6",
        "secret": {
          "secretName": "default-token-9dbv6",
          "defaultMode": 420
        }
      }
    ],
    "containers": [
      {
        "name": "kubernetes-bootcamp",
        "image": "gcr.io/google-samples/kubernetes-bootcamp:v1",
        "resources": {

        },
        "volumeMounts": [
          {
            "name": "default-token-9dbv6",
            "readOnly": true,
            "mountPath": "/var/run/secrets/kubernetes.io/serviceaccount"
          }
        ],
        "terminationMessagePath": "/dev/termination-log",
        "terminationMessagePolicy": "File",
        "imagePullPolicy": "IfNotPresent"
      }
    ],
    "restartPolicy": "Always",
    "terminationGracePeriodSeconds": 30,
    "dnsPolicy": "ClusterFirst",
    "serviceAccountName": "default",
    "serviceAccount": "default",
    "nodeName": "minikube",
    "securityContext": {

    },
    "schedulerName": "default-scheduler",
    "tolerations": [
      {
        "key": "node.kubernetes.io/not-ready",
        "operator": "Exists",
        "effect": "NoExecute",
        "tolerationSeconds": 300
      },
      {
        "key": "node.kubernetes.io/unreachable",
        "operator": "Exists",
        "effect": "NoExecute",
        "tolerationSeconds": 300
      }
    ],
    "priority": 0,
    "enableServiceLinks": true,
    "preemptionPolicy": "PreemptLowerPriority"
  },
  "status": {
    "phase": "Running",
    "conditions": [
      {
        "type": "Initialized",
        "status": "True",
        "lastProbeTime": null,
        "lastTransitionTime": "2022-08-24T12:26:29Z"
      },
      {
        "type": "Ready",
        "status": "True",
        "lastProbeTime": null,
        "lastTransitionTime": "2022-08-24T12:26:32Z"
      },
      {
        "type": "ContainersReady",
        "status": "True",
        "lastProbeTime": null,
        "lastTransitionTime": "2022-08-24T12:26:32Z"
      },
      {
        "type": "PodScheduled",
        "status": "True",
        "lastProbeTime": null,
        "lastTransitionTime": "2022-08-24T12:26:29Z"
      }
    ],
    "hostIP": "10.0.0.8",
    "podIP": "172.18.0.6",
    "podIPs": [
      {
        "ip": "172.18.0.6"
      }
    ],
    "startTime": "2022-08-24T12:26:29Z",
    "containerStatuses": [
      {
        "name": "kubernetes-bootcamp",
        "state": {
          "running": {
            "startedAt": "2022-08-24T12:26:31Z"
          }
        },
        "lastState": {

        },
        "ready": true,
        "restartCount": 0,
        "image": "jocatalin/kubernetes-bootcamp:v1",
        "imageID": "docker-pullable://jocatalin/kubernetes-bootcamp@sha256:0d6b8ee63bb57c5f5b6156f446b3bc3b3c143d233037f3a2f00e279c8fcc64af",
        "containerID": "docker://f47b1cd0383064d111cc112310e3fb3680a0ec7874817cfbac7473b0a6ecb186",
        "started": true
      }
    ],
    "qosClass": "BestEffort"
  }
}

This is the reference that kubernetes checks to ensure that what is actually running is consistent with the configuration that we want to be running. Any differences are automatically rectified by the kubernetes controller.

You will usually communicate with a pod without creating this proxy, though. Instead, you will create a Service.

App Status

There are a couple of go-to ways to get information about pods, nodes, services, deployments, etc. get and describe work with most objects.

$ kubectl get nodes

$ kubectl get nodes
NAME                   STATUS   ROLES                  AGE   VERSION
lima-rancher-desktop   Ready    control-plane,master   11d   v1.24.4+k3s1

Outputting in wide format lets you see more information about objects.

$ kubectl get nodes -o wide
NAME                   STATUS   ROLES                  AGE   VERSION        INTERNAL-IP    EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION   CONTAINER-RUNTIME
lima-rancher-desktop   Ready    control-plane,master   11d   v1.24.4+k3s1   192.168.1.62   <none>        Alpine Linux v3.16   5.15.57-0-virt   containerd://1.6.6

describe gives more detail, and shows recent events that might be useful to know about:

$ kubectl describe nodes
Name:               lima-rancher-desktop
Roles:              control-plane,master
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/instance-type=k3s
                    beta.kubernetes.io/os=linux
                    egress.k3s.io/cluster=true
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=lima-rancher-desktop
                    kubernetes.io/os=linux
                    node-role.kubernetes.io/control-plane=true
                    node-role.kubernetes.io/master=true
                    node.kubernetes.io/instance-type=k3s
Annotations:        alpha.kubernetes.io/provided-node-ip: 192.168.1.62
                    flannel.alpha.coreos.com/backend-data: {"VNI":1,"VtepMAC":"9e:dd:b2:30:cf:33"}
                    flannel.alpha.coreos.com/backend-type: vxlan
                    flannel.alpha.coreos.com/kube-subnet-manager: true
                    flannel.alpha.coreos.com/public-ip: 192.168.1.62
                    k3s.io/hostname: lima-rancher-desktop
                    k3s.io/internal-ip: 192.168.1.62
                    k3s.io/node-args:
                      ["server","--https-listen-port","6443","--flannel-iface","rd0","--disable","traefik","--container-runtime-endpoint","/run/k3s/containerd/c...
                    k3s.io/node-config-hash: G4I3ATTKUNLCZ2PTZF77CHL2LW2RZWWPWVR3A32X4RKONERQ6UCA====
                    k3s.io/node-env: {"K3S_DATA_DIR":"/var/lib/rancher/k3s/data/577968fa3d58539cc4265245941b7be688833e6bf5ad7869fa2afe02f15f1cd2"}
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Tue, 23 Aug 2022 18:07:39 +0100
Taints:             <none>
Unschedulable:      false
Lease:
  HolderIdentity:  lima-rancher-desktop
  AcquireTime:     <unset>
  RenewTime:       Sun, 04 Sep 2022 15:43:19 +0100
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Sun, 04 Sep 2022 15:42:39 +0100   Fri, 02 Sep 2022 23:40:25 +0100   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Sun, 04 Sep 2022 15:42:39 +0100   Fri, 02 Sep 2022 23:40:25 +0100   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Sun, 04 Sep 2022 15:42:39 +0100   Fri, 02 Sep 2022 23:40:25 +0100   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Sun, 04 Sep 2022 15:42:39 +0100   Sat, 03 Sep 2022 20:04:03 +0100   KubeletReady                 kubelet is posting ready status
Addresses:
  InternalIP:  192.168.1.62
  Hostname:    lima-rancher-desktop
Capacity:
  cpu:                2
  ephemeral-storage:  102625208Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             6091188Ki
  pods:               110
Allocatable:
  cpu:                2
  ephemeral-storage:  99833802265
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             6091188Ki
  pods:               110
System Info:
  Machine ID:                 a54277578e5fd09a29516dc8b2a2a049
  System UUID:                a54277578e5fd09a29516dc8b2a2a049
  Boot ID:                    8ff701c3-731e-4676-8ffe-8322150e7807
  Kernel Version:             5.15.57-0-virt
  OS Image:                   Alpine Linux v3.16
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  containerd://1.6.6
  Kubelet Version:            v1.24.4+k3s1
  Kube-Proxy Version:         v1.24.4+k3s1
PodCIDR:                      10.42.0.0/24
PodCIDRs:                     10.42.0.0/24
ProviderID:                   k3s://lima-rancher-desktop
Non-terminated Pods:          (8 in total)
  Namespace                   Name                                         CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                                         ------------  ----------  ---------------  -------------  ---
  kube-system                 svclb-lb-blog-98ecb8d3-ggnkv                 0 (0%)        0 (0%)      0 (0%)           0 (0%)         6d3h
  default                     blog-f496555c7-m72qc                         0 (0%)        0 (0%)      0 (0%)           0 (0%)         23h
  default                     blog-f496555c7-f7w7l                         0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d5h
  default                     blog-f496555c7-q7wjt                         0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d5h
  default                     ingress-nginx-controller-6bf7bc7f94-m78b8    100m (5%)     0 (0%)      90Mi (1%)        0 (0%)         19h
  kube-system                 metrics-server-668d979685-6fnj7              100m (5%)     0 (0%)      70Mi (1%)        0 (0%)         11d
  kube-system                 coredns-b96499967-5gtw5                      100m (5%)     0 (0%)      70Mi (1%)        170Mi (2%)     11d
  kube-system                 local-path-provisioner-7b7dc8d6f5-kls56      0 (0%)        0 (0%)      0 (0%)           0 (0%)         11d
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests    Limits
  --------           --------    ------
  cpu                300m (15%)  0 (0%)
  memory             230Mi (3%)  170Mi (2%)
  ephemeral-storage  0 (0%)      0 (0%)
  hugepages-1Gi      0 (0%)      0 (0%)
  hugepages-2Mi      0 (0%)      0 (0%)
Events:              <none>

You can use kubectl to examine logs, or inspect the pod environment:

$ kubectl logs $POD_NAME
Kubernetes Bootcamp App Started At: 2022-08-24T12:55:49.787Z | Running On:  kubernetes-bootcamp-fb5c67579-c8m9c

Running On: kubernetes-bootcamp-fb5c67579-c8m9c | Total Requests: 1 | App Uptime: 493.64 seconds | Log Time: 2022-08-24T13:04:03.427Z
Running On: kubernetes-bootcamp-fb5c67579-c8m9c | Total Requests: 2 | App Uptime: 539.727 seconds | Log Time: 2022-08-24T13:04:49.514Z


$ kubectl exec $POD_NAME -- env
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=kubernetes-bootcamp-fb5c67579-c8m9c
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1
KUBERNETES_SERVICE_HOST=10.96.0.1
KUBERNETES_SERVICE_PORT=443
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_PORT=tcp://10.96.0.1:443
KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443
NPM_CONFIG_LOGLEVEL=info
NODE_VERSION=6.3.1
HOME=/root

…or even open a shell:

$ kubectl exec -ti $POD_NAME -- bash
root@kubernetes-bootcamp-fb5c67579-c8m9c:/#

You can also use top commands to get resource stats:

$ kubectl top node
NAME                   CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
lima-rancher-desktop   791m         39%    1496Mi          25%
$ kubectl top pod
NAME                        CPU(cores)   MEMORY(bytes)
blog-f496555c7-7fn7z        0m           2Mi
blog-f496555c7-fcc8z        0m           7Mi
blog-f496555c7-whnml        0m           2Mi
cm-acme-http-solver-hkpw6   0m           3Mi

Monitoring

Lens

Lens is a nice UI tool for viewing and managing k8s clusters.

$ brew install --cask lens

Prometheus

Prometheus and Grafana are a useful combination for monitoring your cluster. A seperarate note to come on these.. In the meantime, see: Control plane metrics with Prometheus - Amazon EKS

file:///Volumes/protected_data_1/Media/YouTube%20Notes/Professional/DevOps/Kubernetes/Jeff_Geerling/20210210-Kubernetes_101-_Episode_10-_Monitoring_with_Lens_Prometheus_and_Grafana-zW-E8THfvPY.mp4

Creating a Service

A service defines a collection of pods and a policy and IP address by which to access them. A default kubernetes service is instantiated when the cluster is started:

$ kubectl get services
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   25s

To access this externally, we need the expose command.

$ kubectl expose deployment/kubernetes-bootcamp --type="NodePort" --port 8080
service/kubernetes-bootcamp exposed

$ kubectl get services
NAME                  TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
kubernetes            ClusterIP   10.96.0.1        <none>        443/TCP          3m49s
kubernetes-bootcamp   NodePort    10.100.188.108   <none>        8080:31601/TCP   69s

For creating an external web server, you’d want to use a LoadBalancer type of service. This is usually provided by cloud services, and is not available on minikube. However, k3s comes with traefik (although I actually prefer to make use of ingress-nginx) We can then interrogate the service to view its configuration:

$ kubectl describe services/kubernetes-bootcamp
Name:                     kubernetes-bootcamp
Namespace:                default
Labels:                   app=kubernetes-bootcamp
Annotations:              <none>
Selector:                 app=kubernetes-bootcamp
Type:                     NodePort
IP Families:              <none>
IP:                       10.100.188.108
IPs:                      10.100.188.108
Port:                     <unset>  8080/TCP
TargetPort:               8080/TCP
NodePort:                 <unset>  31601/TCP
Endpoints:                172.18.0.2:8080
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

To parse the specific node port ID, we can use:

$ kubectl get services/kubernetes-bootcamp -o go-template='{{(index .spec.ports 0).nodePort}}'
31601

$ export NODE_PORT=$(kubectl get services/kubernetes-bootcamp -o go-template='{{(index .spec.ports 0).nodePort}}')
$ echo NODE_PORT=$NODE_PORT
NODE_PORT=31601

And to access the application via the service:

$ curl $(minikube ip):$NODE_PORT
Hello Kubernetes bootcamp! | Running on: kubernetes-bootcamp-fb5c67579-sfbjd | v=1

Labels

A label will automatically be generated. You can see this with describe deployment

$ kubectl describe deployment
Name:                   kubernetes-bootcamp
Namespace:              default
CreationTimestamp:      Wed, 24 Aug 2022 16:39:19 +0000
Labels:                 app=kubernetes-bootcamp
Annotations:            deployment.kubernetes.io/revision: 1
Selector:               app=kubernetes-bootcamp
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  app=kubernetes-bootcamp
  Containers:
   kubernetes-bootcamp:
    Image:        gcr.io/google-samples/kubernetes-bootcamp:v1
    Port:         8080/TCP
    Host Port:    0/TCP
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   kubernetes-bootcamp-fb5c67579 (1/1 replicas created)
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  10m   deployment-controller  Scaled up replica set kubernetes-bootcamp-fb5c67579 to 1

Here you can see that the deployment label is app=kubernetes-bootcamp You can use this to view the pods and services of the deployment:

$ kubectl get pods -l app=kubernetes-bootcamp
NAME                                  READY   STATUS    RESTARTS   AGE
kubernetes-bootcamp-fb5c67579-sfbjd   1/1     Running   0          20m

$ kubectl get services -l app=kubernetes-bootcamp
NAME                  TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
kubernetes-bootcamp   NodePort   10.100.188.108   <none>        8080:31601/TCP   18m

We can also add new labels to a pod with:

$ kubectl label pods $POD_NAME version=v1
pod/kubernetes-bootcamp-fb5c67579-sfbjd labeled

Deleting a service

You can use a label to delete a service:

$ kubectl delete service -l app=kubernetes-bootcamp
service "kubernetes-bootcamp" deleted

The pod and node will continue as before, simply no longer exposed. To shut down the application, you need to also delete the deployment.

Scaling Kubernetes

I’ve split my notes about k8s scaling solutions in two:

[[Kubernetes Application Scaling]] deals with scaling workloads by creating more pods.
[[Kubernetes Load Balancing]] considers exposing those pods in services and load balancing work between them.

Deployments

Rolling Updates

Performing a rolling update is simple: Update the configuration so that a new version of the application image is specified:

$ kubectl set image deployments/kubernetes-bootcamp kubernetes-bootcamp=jocatalin/kubernetes-bootcamp:v2
deployment.apps/kubernetes-bootcamp image updated

Once the deployment application is changed, kubernetes will automatically migrate the deployment to match the specification. By default, only 1 image becomes unavailable at any 1 time.

$ kubectl rollout status deployments/kubernetes-bootcamp
deployment "kubernetes-bootcamp" successfully rolled out

If a deployment fails or has a problem, you can rollback to the last known good state:

$ kubectl rollout undo deployments/kubernetes-bootcamp
deployment.apps/kubernetes-bootcamp rolled back

Rollback

kubectl rollout history deployment blog
...

kubectl rollout undo deployment blog

For more sophisticated deployment, check out Argo CD (blog post coming soon).

References

Interactive Tutorial - Creating a Cluster | Kubernetes

Switching Contexts with Kubectx

2022-06-12T00:00:00+01:00

Kubectx is a tiny cli shortcut to aid in k8s management and switching between contexts and namespaces.

Normally you would control this sort of thing with a more complicated command. To set the default namespace for the current context, you would use:

kubectl config set-context --current --namespace=<namespace>

To return to the default:

kubectl config set-context --current --namespace=""

The easier-to-remember version is:

brew install kubectx

kubens               # list namespaces
kubens <namespace>   # set default namespace

kubectx              # list contexts
kubectx <context}    # set context

Gamifying My Productivity

2022-06-10T00:00:00+01:00

I’m trying an experiment to improve my productivity, plus also creating a cryptocurrency token to play around with. Here’s how the two go together.

This video popped up in my youtube recommendations last week. It’s over 5 minutes long, so it was pretty hard to make it all the way through, but it starts with a good reminder that our world has so many small things vying for our attention, it is sometimes hard to focus. It’s so easy to get stuck into the comfortable reward cycle of scrolling through reddit, opening another wikipedia tab, checking the socials, just one more level on that mobile game…. I’ve even found myself closing the reddit app only to immediately reflexively open it back up again as my brain immediately starts searching for the next thing to occupy itself.

The problem is that pleasure experienced without prior requirement for pursuit is terrible for us.

Dr Andrew Huberman

All these little ways of injecting dopamine into your brain without any effort feel great, but they’re terrible if you want to keep being productive. Getting your dopamine through effort… requires effort. Why not just skip that bit. Getting rewards without effort is a difficult habit to break, and ultimately, self-destructive. And it’s getting easier and easier to do that more and more.

A great productivity technique is to ensure that your rewards are tied to achievements. More importantly, it’s a kinda good way to lead your life. Rewarding yourself after achieving something through effort is a virtuous reward cycle in itself. Making that a habit is hard, and there are so many tempting, easy ways to break it.

So, how about creating a framework, within which you establish for yourself a set of rules that ensure that you recognise both the cost and the value of both your efforts and your rewards.

To have a play with these ideas, I’ve come up with a scheme which I’m trying out… I give myself points for doing productive or virtuous stuff like working on my book, going to the gym, working on my side-project, or even getting to bed early! ¹

I can then spend these things on rewards: things that I like to do, but aren’t a productive use of my time. Instead of feeling guilty about scrolling through reddit, or watching another episode of It’s Always Sunny for the n^th time, I can feel happy that I’m doing a fun thing that I earned.

Sometimes that’s quite a shift of perspective.

The point is to think about how I spend my time in a fun and engaging way.

This system is easy to tweak. If I think I’ve been going to McDonald’s too often, because it’s just around the corner and it’s so convenient, I can make it cost more points. If I’d like to spend more time at the gym, and can reward myself more for going. I’ve also found myself changing my activities in interesting ways. For example, I might think to myself “Hmm, I’ll still watch this YouTube video, but if I do sit-ups at the same time, I can earn points too!” or, “I’ve finished for the evening, but if I do just one more Pomodoro on my side project, I’ll have extra for a KFC tomorrow, and its the Big Bucket Deal on Tuesdays, ooooh….”

The point is not to turn everything into an activity with a price and start micromanaging my life, but to create a framework with which I can reflect upon my lifestyle and think about how I spend my time in a fun and engaging way. So far, I’m getting a lot out of it.

Using the Blockchain to become more productive

Now, to implement my scheme in the real world, I’ve created a Solana fungible token called Pocket Money (POCK). I have created two accounts - a bank account which I use to pay myself reward points, and an earnings account where I keep my points, and spend them on goodies. The tokens zip back and forth between the two accounts as I keep track of my time. Solana has very cheap and practically instant transactions, so you can do it as much as you like.

Solana is a super-fast and easy to use cryptocurrency. It’s had some problems in it’s journey, but it it’s more than adequate for my use case. It’s also very easy to create your own token. Or, send me a message, and I’ll send you some pocket money. Or, you could buy some Pocket Money super easily here. Do you need a trust-less, distributed, open, blockchain network by which to track this stuff? No. But, t’s fun and nerdy, and I’m loving it. Of course, you could get a bunch of physical coins) and keep track that way.

Think of the Children

As the name of the Token suggests, you could also use this token with your kids. Mow the lawn: get some Pocket Money; play Minecraft, spend your Pocket Money. “Oh, you want to skip school and go to the toy store? Sorry you don’t have enough Pocket Money™ for that.” You know, that sort of thing.

Conclusion

It’s not a perfect system, and it would be awful to obsessively live your life this way, but it’s been an interesting way to reflect upon the worth of my time, how I spend it, and how I appreciate the good things in my life, and the efforts I’ve made to receive them. Also, creating a cryptocurrency is fun!

It’s not for everyone, but if you think it might work for you, I encourage you give it a go. Should you use it as an employer to micro-incentivise your staff in some sort of Black Mirror distopian future hellscape? NOOOO STAAAAAHP.

Cover photo by Mark Cruz on Unsplash

I don’t get tired when I need to. I can easily stay up till 4am if I’m engrossed in something, but I will bitterly regret it the next day… ↩

Creating a Solana Crypto Token

2022-06-06T00:00:00+01:00

Solana is a fast, proof-of-history token framework with a Rust language ‘contract’ system.

I created a couple of Solana tokens. Here’s how…

Steps:

Download Solana platform and the SPL CLI:

sh -c "$(curl -sSfL https://release.solana.com/v1.10.23/install)" (See the Solana Installations Docs for details.
Install Rust
Install the Solana token CLI (cargo install spl-token-cli)
Mint a new token
Create an icon
Register the token on github. (Solana token registry)

I’ve made a very lightweight docker image here with the appropriate tools and some helpful scripts for playing around with tokens. You can go through the token creation process below using this container.

Creating a fungible Solana token

To do a test run first, switch to a test net:

$ solana config get
Config File: /root/.config/solana/cli/config.yml
RPC URL: https://api.mainnet-beta.solana.com
WebSocket URL: wss://api.mainnet-beta.solana.com/ (computed)
Keypair Path: /root/.config/solana/id.json
Commitment: confirmed
$ solana config set --url https://api.devnet.solana.com
Config File: /root/.config/solana/cli/config.yml
RPC URL: https://api.devnet.solana.com
WebSocket URL: wss://api.devnet.solana.com/ (computed)
Keypair Path: /root/.config/solana/id.json
Commitment: confirmed

Later, when you want to create real tokens on the mainnet, you can set the RPC URL back to it’s original value:

$ solana config set --url https://api.mainnet-beta.solana.com

Also, create an encryption key pair:

$ solana-keygen new  -o /root/.config/solana/id.json

This will create a new key pair. Make a note of the BIP39 passphrase used, and the seed phrase. The private key is stored in plaintext, and is thus very insecure. Don’t leave that plaintext file hanging around, or don’t store anything of value in this wallet.

Once you have the passphrase and seedphrase stored, you can regenerate your key pair at any time with:

$ solana-keygen recover --force prompt://
<enter the seed phrase>
<enter the passphrase twice>

You could also store your keys in a hardware wallet like ledger.

$ solana-keygen pubkey usb://ledger

However, USB passthrough with docker on a Mac is not straightforward, so I’m not doing that. If you want to, it’s possible, but apparently you need to ditch docker desktop (which uses Hyperkit, which doesn’t support USB passthrough) and use a KVM like VirtualBox.

Anyway, the next step is to ensure that the public key displayed matches your original public key.

$ solana address
2ASn1atx6oHF4vkyF9vEBBMreJPcaXEiaiFieah9vyZK

Then, send yourself some SOL. You’ll need it to pay for the proceeding transactions. On the testnet, you can airdrop yourself some:

$ solana airdrop 1
Requesting airdrop of 1 SOL

Signature: 4qGBTAcuo8g728CXE5mmn3447vfhBSzs9EmzKh5p25ZK1P9feJ1j3P8ekmfieXYSbT6A8XusderoPU7ceVfA8oCd

1 SOL
$ solana balance
1 SOL

So, you received 1 SOL, and the Signature returned refers to the blackchain transaction ID where your airdrop occured. You can search for this with the Solana blockchain explorer. That link points to the mainnet blockchain, but there is a button on the page to choose testnet or devnet.

1 SOL is far more than you need to pay for the transactions and account rents involved here.

On the mainnet, you’ll need to buy some on an exchange. I used Coinbase.

You can quickly check your account details:

$ solana account 2ASn1atx6oHF4vkyF9vEBBMreJPcaXEiaiFieah9vyZK

Public Key: 2ASn1atx6oHF4vkyF9vEBBMreJPcaXEiaiFieah9vyZK
Balance: 1 SOL
Owner: 11111111111111111111111111111111
Executable: false
Rent Epoch: 321

Now to create a new Solana token! I wanted to create a fungible token with 2 decimal places:

$ spl-token create-token --decimals 2        # 9 is default
Creating token DmSx9d931vZt4HLZ6c4Cyt3ka3EAC2mmeD6JEfgCeQoQ

Signature: 4KLmXg4TPbsp9N1xGgXLFErPDjcipGLsbgNmpgkr6NdM2jjy5TKRJjqZeQJB2NrHyzRjrb9BEHRKKLwXrLhrWHJQ

This creates the token and reports the token unique identifier. You’ll need that later. Again, the signature is the ID of the transaction on the blockchain which created your new token

Next, create an account for your new token:

$ spl-token create-account DmSx9d931vZt4HLZ6c4Cyt3ka3EAC2mmeD6JEfgCeQoQ
Creating account EnrMVfUNHMotdNAxjzjPakvPXYRG7xggsE78z6z5U6ps

Signature: L5H13wni3i1sfRWu2qYJSiR46re8PWU8taoCSBBFLWK5fXEB29Gf4YGfK4C3oXRMa2RLvnyamBrQa6Z4iB4paXP

Mint a bunch of tokens, and add them to your new account:

# spl-token mint <token ID> <amount> [optional account address]
$ spl-token mint DmSx9d931vZt4HLZ6c4Cyt3ka3EAC2mmeD6JEfgCeQoQ 1000000000000
Minting 1000000000000 tokens
  Token: DmSx9d931vZt4HLZ6c4Cyt3ka3EAC2mmeD6JEfgCeQoQ
  Recipient: EnrMVfUNHMotdNAxjzjPakvPXYRG7xggsE78z6z5U6ps

Signature: afM8He5RBtXVLSJhiW3m52YEj7iJc2qvBToxrQBp999p1QdAH2yA7gEySpQTRmK2NTZ7gZABh8QC6hUsmTu8P6r

Verify the token supply:

$ spl-token supply DmSx9d931vZt4HLZ6c4Cyt3ka3EAC2mmeD6JEfgCeQoQ
18446744073.709551615

Note that the amount minted is i) measured in increments of whatever the decimal value the token has been created with, and ii) a maximum of a u64 bit integer. So in this case, I had created a token with 9 decimal places. That meant that it reached the u64 representation limit, and produced just as many tokens as it could.

Now, check your balances and account details:

$ spl-token balance DmSx9d931vZt4HLZ6c4Cyt3ka3EAC2mmeD6JEfgCeQoQ
18446744073.709551615

$ solana balance
0.99647912 SOL
$ solana account DmSx9d931vZt4HLZ6c4Cyt3ka3EAC2mmeD6JEfgCeQoQ

Public Key: DmSx9d931vZt4HLZ6c4Cyt3ka3EAC2mmeD6JEfgCeQoQ
Balance: 0.0014616 SOL
Owner: TokenkegQfeZyiNwAJbNbGKPFXCWuBvf9Ss623VQ5DA
Executable: false
Rent Epoch: 321
Length: 82 (0x52) bytes
0000:   01 00 00 00  68 fa 77 ea  11 06 fd c3  10 ec 82 59   ....h.w........Y
0010:   4a 2d e0 a3  50 c0 9b 5d  36 3b e1 d4  87 4b 20 32   J-..P..]6;...K 2
0020:   95 43 6c 1e  ff ff ff ff  ff ff ff ff  09 01 00 00   .Cl.............
0030:   00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00   ................
0040:   00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00   ................
0050:   00 00                                                ..

Now you can transfer your SOL and tokens to other accounts

$ solana transfer --allow-unfunded-recipient 2ASn1atx6oHF4vkyF9vEBBMreJPcaXEiaiFieah9vyZK 0.15

OMG! I now have 0.15 SOL in my account, and am now a god among men!

# spl-token transfer <options> <token ID> <amount> <recipient address>
$ spl-token transfer --allow-unfunded-recipient --fund-recipient DmSx9d931vZt4HLZ6c4Cyt3ka3EAC2mmeD6JEfgCeQoQ 1000 2ASn1atx6oHF4vkyF9vEBBMreJPcaXEiaiFieah9vyZK

If the address you want to send tokens to isn’t already funded, you can fund it as part of the transaction.

There you go! Now you have SOL and tokens in any account you want. Hooray!

If you use Ledger, your ledger live software doesn’t currently show Solana tokens. You’ll need to use, say, Solflare. Then authenticate your account by connecting your Ledger device. Solflare can then display the correct token balances!

Finishing Touches

You can register your token, and provide an icon, by providing some basic details to the (Solana token registry). Simply add edit the token list file submit a pull request. The PR should be automatically closed and merged by a bot within about an hour.

References

How to Create a Solana Token in 5 Steps » Moralis » The Ultimate Web3 Development Platform

Create and List a Solana Token in a UI with Zero Development in 5 Minutes

Proxmox Installation Checklist

2022-05-12T00:00:00+01:00

Here is a short list of things you might want to run through as you are installing a proxmox server.

Make sure you’ve downloaded the latest version of Proxmox.
Have a copy of, or a direct link to, any of the VM ISOs you know you are about to install.
Add all the hard drives you anticipate using before the install, it’s a bit easier than adding them later.
Install
Enable updates: Check Package Repositories - Proxmox VE for details
- go to /etc/apt/sources.list
- ensure security updates are always enabled
- you can disable enterprise update if you don’t have a subscription
- unstable updates are actually pretty good, but don’t use for production systems. From the above link, the current lines are:
```
# PVE pve-no-subscription repository provided by proxmox.com,
# NOT recommended for production use
deb http://download.proxmox.com/debian/pve bullseye pve-no-subscription
```
If you don’t have a proxmox subscription, also comment out the line in /etc/apt/sources.list.d/pve-enterprise.list
You can then apt-get update and apt dist-upgrade to ensure you’ve got the latest updates. If you added the unstable source, you’ll probably have updates, even if you just installed the latest version of Proxmox.
If available, ensure IOMMU (or VT-d for intel) is enabled in your BIOS. Especially if you want GPU passthrough.

Update the /etc/default/grub file:

#GRUB_CMDLINE_LINUX_DEFAULT="quiet"
#GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on"

After editing this file, run update-grub

then add VFIO modules by editing /etc/modules. Add
```
vfio
vfio_iommu_type1
vfei_pci
vfio_virqfd
```

On the GUI, under Datacenter/Storage remove the local-lvm drive. Then, in the shell:

lvremove /dev/pve/data
lvresice -l +100%FREE /dev/pve/root
resize2fs /dev/mapper/pve-root

On the GUI, Datacenter/Storage -> local. Edit this and ensure that all content types are selected.
Disable the subscription nag: edit the /usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js file and find the text “no valid subscription”. Prepend void{( // to Ext.Msg.show
Install glances for killing VMs that don’t have the Qemu agent installed.
Run dpkg-reconfigure locales to fix nag messages when using the command line.

Post VM Installation Checklist

Ensure you can ssh into your VM. (This means both configuring SSH to be available, and installing ssh keys).
Run your standard post-installation scripts/ ansible playbooks.
Check best practices for your installed OS.

Adding Qemu-Guest-Agent to a *nix VM

To get extra, useful information on Promox, you should ensure that your VM OS has the qemu-guest-agent tools installed and running.

Step 1: Log in using SSH

You must be logged in via SSH as sudo or root user. Please read this article for instructions if you don’t know how to connect.

Step 2: Install qemu guest agent

$ apt update && apt -y install qemu-guest-agent

Step 3: Enable and Start Qemu Agent

$ systemctl enable qemu-guest-agent
$ systemctl start qemu-guest-agent

Step 4: Verify

Verify that the Qemu quest agent is running

$ systemctl status qemu-guest-agent

Probably also add open-vm-tools on Proxmox and open-vm-tools-desktop on your VM.

Creating Kubernetes YAML

2022-04-29T00:00:00+01:00

Using --dry-run is a quick way to create well formed k8s yaml:

$ kubectl create ingress blog --class=nginx --rule="blog.mehcoleman.com/*=blog:80" --dry-run=client -o yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  creationTimestamp: null
  name: blog
spec:
  ingressClassName: nginx
  rules:
  - host: blog.mehcoleman.com
    http:
      paths:
      - backend:
          service:
            name: blog
            port:
              number: 80
        path: /
        pathType: Prefix

See How To Create Kubernetes YAML Manifests Quickly for many other examples, and my post on K8s configuration management methods for an overview of how they might be used.

Kubernetes Application Scaling

2022-04-02T00:00:00+01:00

Here are some thoughts on Kubernetes application scaling, both manual and autoscaling.

You can set scaling for a service, to ensure that an appropriate number of pods are available to fulfil requests. An integrated load balancer is part of the Service.

A vanilla deployment starts with 1 pod:

$ kubectl get deployments
NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
kubernetes-bootcamp   1/1     1            1           47s

The scaling replication set can be viewed with:

$ kubectl get rs
NAME                            DESIRED   CURRENT   READY   AGE
kubernetes-bootcamp-fb5c67579   1         1         1       47s

This can be changed with:

$ kubectl scale deployments/kubernetes-bootcamp --replicas=4
deployment.apps/kubernetes-bootcamp scaled

Now, extra pods with replica configurations will be created:

$ kubectl get deployments
NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
kubernetes-bootcamp   4/4     4            4           21m

$ kubectl get pods -o wide
NAME                                  READY   STATUS    RESTARTS   AGE   IP           NODE       NOMINATED NODE   READINESS GATES
kubernetes-bootcamp-fb5c67579-p4pxk   1/1     Running   0          63s   172.18.0.7   minikube   <none>           <none>
kubernetes-bootcamp-fb5c67579-q42js   1/1     Running   0          63s   172.18.0.9   minikube   <none>           <none>
kubernetes-bootcamp-fb5c67579-qzxjl   1/1     Running   0          63s   172.18.0.8   minikube   <none>           <none>
kubernetes-bootcamp-fb5c67579-rjmz4   1/1     Running   0          21m   172.18.0.2   minikube   <none>           <none>

$ kubectl get rs
NAME                            DESIRED   CURRENT   READY   AGE
kubernetes-bootcamp-fb5c67579   4         4         4       21m

Likewise, we can scale down with:

$ kubectl scale deployments/kubernetes-bootcamp --replicas=2
deployment.apps/kubernetes-bootcamp scaled

Scaling events are logged in the deployments event log. See:

$ kubectl describe deployments/kubernetes-bootcamp
Name:                   kubernetes-bootcamp
Namespace:              default
CreationTimestamp:      Wed, 24 Aug 2022 17:40:26 +0000
Labels:                 app=kubernetes-bootcamp
Annotations:            deployment.kubernetes.io/revision: 1
Selector:               app=kubernetes-bootcamp
Replicas:               2 desired | 2 updated | 2 total | 2 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  app=kubernetes-bootcamp
  Containers:
   kubernetes-bootcamp:
    Image:        gcr.io/google-samples/kubernetes-bootcamp:v1
    Port:         8080/TCP
    Host Port:    0/TCP
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Progressing    True    NewReplicaSetAvailable
  Available      True    MinimumReplicasAvailable
OldReplicaSets:  <none>
NewReplicaSet:   kubernetes-bootcamp-fb5c67579 (2/2 replicas created)
Events:
  Type    Reason             Age    From                   Message
  ----    ------             ----   ----                   -------
  Normal  ScalingReplicaSet  29m    deployment-controller  Scaled up replica set kubernetes-bootcamp-fb5c67579 to 1
  Normal  ScalingReplicaSet  9m23s  deployment-controller  Scaled up replica set kubernetes-bootcamp-fb5c67579 to 4
  Normal  ScalingReplicaSet  48s    deployment-controller  Scaled down replica set kubernetes-bootcamp-fb5c67579 to 2

We can see that the load-balancing is working by making multiple requests to the application. Below, we view the service information, extract the node port, and make a request to the service, which is handled by one of the 4 pods.

$ kubectl describe services/kubernetes-bootcamp
Name:                     kubernetes-bootcamp
Namespace:                default
Labels:                   app=kubernetes-bootcamp
Annotations:              <none>
Selector:                 app=kubernetes-bootcamp
Type:                     NodePort
IP Families:              <none>
IP:                       10.108.183.223
IPs:                      10.108.183.223
Port:                     <unset>  8080/TCP
TargetPort:               8080/TCP
NodePort:                 <unset>  32555/TCP
Endpoints:                172.18.0.2:8080,172.18.0.7:8080,172.18.0.8:8080 + 1 more...
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>
$ export NODE_PORT=$(kubectl get services/kubernetes-bootcamp -o go-template='{{(index .spec.ports 0).nodePort}}')
$ echo NODE_PORT=$NODE_PORT
NODE_PORT=32555
$ curl $(minikube ip):$NODE_PORT
Hello Kubernetes bootcamp! | Running on: kubernetes-bootcamp-fb5c67579-qzxjl | v=1
$ curl $(minikube ip):$NODE_PORT
Hello Kubernetes bootcamp! | Running on: kubernetes-bootcamp-fb5c67579-qzxjl | v=1
$ curl $(minikube ip):$NODE_PORT
Hello Kubernetes bootcamp! | Running on: kubernetes-bootcamp-fb5c67579-rjmz4 | v=1
$ curl $(minikube ip):$NODE_PORT
Hello Kubernetes bootcamp! | Running on: kubernetes-bootcamp-fb5c67579-p4pxk | v=1
$ curl $(minikube ip):$NODE_PORT
Hello Kubernetes bootcamp! | Running on: kubernetes-bootcamp-fb5c67579-qzxjl | v=1
$ curl $(minikube ip):$NODE_PORT
Hello Kubernetes bootcamp! | Running on: kubernetes-bootcamp-fb5c67579-qzxjl | v=1
$ curl $(minikube ip):$NODE_PORT
Hello Kubernetes bootcamp! | Running on: kubernetes-bootcamp-fb5c67579-qzxjl | v=1

Autoscaling

You can also get kubernetes to autoscale, and more advanced still, use a service mesh for more sophisticated control.

To autoscale, you need to be able to monitor your nodes. Check whether metrics are installed:

$ kubectl top nodes
NAME                   CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
lima-rancher-desktop   387m         19%    1375Mi          23%

Yes! Rancher comes with metrics included (See Rancher Docs: Metrics Server).

If you see:

$ kubectl top nodes
error: Metrics API not availble.

This means that metrics is not installed on your system.

You can use Helm to install Bitnami’s metrics-server chart. This might require extra configuration, depending on your deployment platform.

From Helm Charts to deploy Metrics Server in Kubernetes:

helm repo add bitnami https://charts.bitnami.com/bitnami
helm install my-release bitnami/metrics-server

Once the metrics server is running, you can create a horizontal pod autoscaler:

$ kubectl autoscale [-n namespace] deployment blog --min=1 --max=5 --cpu-percent=75
$ kubectl get hpa [-n namespace] blog

Remote Pair Programming

2022-03-21T00:00:00+00:00

Pair programming is a fantastic way of sharing ideas, training, being productive and writing high quality code. Remote pair programming can be just as effective. I love pairing on projects. If you have an interesting open source project you’re working on, and want to pair on it, hit me up.

Different types of pair programming

There are a couple of common approaches to pair programming that I see a lot. Approaches and the specific details can be heavily dependent on the people involved, their preferred way of thinking and learning, and their level of expertise.

They can be split into categories, either by looking at the interpersonal management of the work, or by the technical differences in editing source code.

Firstly, we can look at different methods of managing the workload:

Unstructured. Simply, two people collaborating on solving a problem, without any particular set of rules about how this should be done.
One person is driving the direction of the development “i.e. “We need to write a test that checks for this”, the other is at the keyboard, implementing those directions, i.e. writing the test. This can be broken down further according to how prescriptive the leader is. Do they get right into the details, or are they taking a 10,000 ft approach to direction?
One person is leading the task and at the keyboard. They are describing what they are doing. The other person could be learning or mentoring. They could also be taking notes and, from a slightly detached viewpoint, spotting mistakes or thinking strategically.
One person writes the next test. The partner then takes control and writes the code that gets the test to pass. They then write the next test, and pass back to the first person to write the code and repeat the loop.

These approaches are orthogonal to whether you are working remotely or in an office together.

We can also look at the technical details two people writing one thing.

Two people sit in front of the same computer together. The keyboard swaps between them. As they swap, the same files are open in the same editor. So, they must both use a shared editor and set of keybindings.
Two people sit beside each other on their own computers. When they swap they can use their own preferred editor, key bindings, etc. This is generally more ergonomically pleasant. The pair will need to synchronise the source files they have been working on when they swap roles.

These technical approaches can also be largely orthogonal to whether you are pairing remotely. There are some useful tools to screen share, edit files simultaneously, or sync code seamlessly.

Many general purpose chat & conference tools have screen-sharing facilities that are perfectly sufficient. As long as you can communicate effectively and the screen lag is not terrible, you’ll be fine. For example, Slack or Zoom, along with a lot of git commits and git pulls works fine.

This technique works fine for me, but some people like a much finer grain of swapping who is in control than 1 red-green-refactor, or 1 ‘commits-worth’ of work. If you like to swap who’s controlling the keyboard in the middle of a sentence, you’ll love mob. This allows a group of collaborators to work together on a shared codebase quickly and with minimal friction. Each collaborator calls mob start to start sharing code on a WIP branch. One person works on the code. To hand over, run mob next. Once you’re happy you’ve got a ‘commits-worth’ of code, mob commit squashes the WIP commits into a single commit and commits onto the current branch. Then you can do a regular git push to ensure your remote repo is current. Eventually, you’ll want mob done to end your collaboration session.

Pop is another good looking tool that allows screen sharing, collaborative drawing and mouse & keyboard control. If you like working on a closely shared environment (where you have the same editor and keyboard shortcuts, etc), this works well. If you like the more individual approach, you can also use this tool as a good screen sharing chat tool with some nice extra features.

Tuple is a great screen sharing tool for macOS users (a linux version is in beta). In addition to the ‘standard features, it has nice additions for helping to diagnose screen share performance to ensure your screen share quality stays sharp, and integration with OS features (like link sharing and enabling ‘Do Not Disturb’) that remove friction from a pairing session.

Pair programming and remote pairing are such a fantastic way to work. The great thing is that there is no One True Way to do it. Find one that works for you and your pairing partner.

Proxmox: GPU Passthrough

2022-03-04T00:00:00+00:00

This guide: Proxmox PCI-Passthrough – gareth.com has almost everything you need.

The process is quite straightforward. Ensure that the following modules are added to your /etc/modules file on your Proxmox node:

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

Blacklist the graphics drivers form your Proxmox node, so that the graphics cards will be available for use:

echo "blacklist nvidia" >> /etc/modprobe.d/blacklist.conf
echo "blacklist radeon" >> /etc/modprobe.d/blacklist.conf
echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf

Find the ID of the graphics card you wish to pass-through.

lspci | grep VGA

If you have multiple graphics cards installed, Check you have the right one. They might share the same vendor ID if they are the same model. Once you have the ID, (it will be something like 03:00, get the vendor IDs:

lspci -n -s 03:00

Add the resultant IDs to /etc/modprobe.d/vfio.conf

options vfio-pci ids=10de:1380,10de:0fbc disable_vga=1

Load the new configs

update-initramfs -u

And restart the server.

Finally, add the PCI device to the hardware section of your VM.

Notes

You need to be aware of the IOMMU groups that each device is attached to. See Intro to IOMMU groups for KVM virtual machines - YouTube.

Also, one GPU will be attached to the Proxmox host device, and its ROM will be written in such a way that it will not work correctly for virtualisation, even if it appears to be an available option. You will need to patch the ROM (See GPU BIOS Info) if you only have one GPU installed, or you have several others but also want to use this one.

Add USB Devices too

You might be doing GPU passthrough so that you can drive an OS directly through your PC connected to a monitor. In this case, you probably want to passthrough some USB devices (like keyboard, mouse and perhaps bluetooth) too. This is even easier than GPU passthrough.

The command lsusb will help identify usb ports and devices. You may want to passthrough bluetooth, keyboard and mouse. They will need to be plugged in to the machine hosting Proxmox as you use the GUI to add them.

Virtualised GPUs

You can go one step beyond passthrough GPUs, and virtualise GPUs to split them across multiple VMs. This is more involved. See Proxmox vGPU Gaming Tutorial - Share Your GPU With Multiple VMs! - YouTube

The AMD Reset Bug

Many AMD graphics cards have a bug where they cannot be initialised more that once per power cycle. Since the booting of the Proxmox server counts as the first initialisation, they fail when trying to passthrough to a VM. There are some fixes available, for some cards.

See Working around the AMD GPU Reset bug on Proxmox using vendor-reset – Nicholas Sherlock

Proxmox: Windows 11 Installation

2022-02-18T00:00:00+00:00

Here’s how I installed Windows 11 onto Proxmox:

Install Proxmox > 7.0
Upload the Windows 11 installation ISO from Microsoft and the latest virtio-win iso from GitHub - virtio-win/virtio-win-pkg-scripts: Scripts for packaging virtio-win drivers
Create a new VM. Pay attention to the following configuration settings:

When running through the installation wizard, check the following:

In the System: Make sure the SCSI controller is VirtIO SCSI; Turn on the Qemu Agent; Ensure UEFI BIOS is chosen; Use the same drive location for the primary drive, the EFI disk and the TPM storage; Add the virtual TPM, ensure it’s version 2.0; Ensure machine type is Q35
Hard Disk: Bus Device: SCSI, I think, VirtIO Block will also work, but slower
CPU: CPU type should be Host (to passthrough everything)
Network: Model VirtIO

Once all this is configured, create your lovely new VM, but don’t start it yet:

Next, add the VirtIO disk so that the additional drivers will be available when installing Windows 11: Go to the VM, Click Hardware, and add a new CD/DVD ROM. Attach the virtio-win ISO..
Now you can boot the virtual machine. It should boot into the installation wizard.
Continue, with a custom install.
Eventually, you will given an opportunity to install more drivers. Browse to the virtio ISO
Pick the amd64 folder, w11 folder, and add the SCSI controller.
Next, add the network driver too: Pick the NetKVM folder, w11, amd64
Go through the rest of the installation process, then when complete, run the driver installation exe on the virtIO iso to install any remaining drivers required: Open windows explorer, Open the root directory, and install the extensions.

If you have problems it may be because the virtIO developer certificates are not trusted by Windows 11. You will need to manually trust them. Try:
- Go into windows settings -> drivers, and manually adding a certificate. There will be at least two different certificates across the range of folders containing drivers.
- Once you’ve added them both, go to the Drivers panel in system settings and Add Driver.
- Then choose the browse... options and check the box to automatically find drivers recursively. All the drivers should then be added, and things should start working correctly.
If that didn’t help, check Error: 8004E00F when trying to access COM+ Applications in - Microsoft Community

Also, to make admin easier, you might need to enable SSHing into windows. I used this so I could update the Nvidia BIOS on my graphics cards.

You might also want to enable Proxmox GPU Passthrough or Proxmox GPU Virtualisation.

MEHColeman

A Kubernetes Walkthrough

Create a Cluster

Deploy an App

App Status

Monitoring

Lens

Prometheus

Creating a Service

Labels

Deleting a service

Scaling Kubernetes

Deployments

Rolling Updates

Rollback

References

Switching Contexts with Kubectx

Gamifying My Productivity

Using the Blockchain to become more productive

Think of the Children

Conclusion

Creating a Solana Crypto Token

Creating a fungible Solana token

Finishing Touches

References

Proxmox Installation Checklist

Post VM Installation Checklist

Adding Qemu-Guest-Agent to a *nix VM

Step 1: Log in using SSH

Step 2: Install qemu guest agent

Step 3: Enable and Start Qemu Agent

Step 4: Verify

Creating Kubernetes YAML

Kubernetes Application Scaling

Autoscaling

Remote Pair Programming

Different types of pair programming

Proxmox: GPU Passthrough

Notes

Add USB Devices too

Virtualised GPUs

The AMD Reset Bug

See Also

Proxmox: Windows 11 Installation