Category Archives: CKS

Supply Chain Security – Static Analysis


title: “Supply Chain Security – Static Analysis”
date: 2020-12-13T20:37:33
slug: supply-chain-security-static-analysis


Check your yaml file with Kubesec:

docker run -i kubesec/kubesec:512c5e0 scan /dev/stdin < pod.yaml

In OPA Conftest: Container are not allowed to run as root
Create a policy File:

$ cat policy/deployment.rego
# from https://www.conftest.dev
package main

deny[msg] {
 input.kind = "Deployment"
 not input.spec.template.spec.securityContext.runAsNonRoot = true
 msg = "Containers must not run as root"
}

deny[msg] {
 input.kind = "Deployment"
 not input.spec.selector.matchLabels.app
 msg = "Containers must provide app label for pod selectors"
}
docker run --rm -v $(pwd):/project instrumenta/conftest test deploy.yaml

Reduce Image Footprint with Mulit-Stage


title: “Reduce Image Footprint with Mulit-Stage”
date: 2020-12-13T20:14:19
slug: reduce-image-footprint-with-mulit-stage


Use multi-stage builds

With multi-stage builds, you use multiple FROM statements in your Dockerfile. Each FROM instruction can use a different base, and each of them begins a new stage of the build. You can selectively copy artifacts from one stage to another, leaving behind everything you don’t want in the final image. To show how this works, let’s adapt the Dockerfile from the previous section to use multi-stage builds.

Dockerfile:

# build container stage 1
FROM ubuntu
ARG DEBIAN\_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y golang-go
COPY app.go .
RUN CGO\_ENABLED=0 go build app.go

# app container stage 2
FROM alpine
COPY --from=0 /app .
CMD ["./app"]

Use specific Versions:

# build container stage 1
FROM ubuntu
ARG DEBIAN\_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y golang-go
COPY app.go .
RUN CGO\_ENABLED=0 go build app.go

# app container stage 2
FROM alpine:3.11.6
COPY --from=0 /app .
CMD ["./app"]

Dont run as root

# build container stage 1
FROM ubuntu:20.04
ARG DEBIAN\_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y golang-go=2:1.13~1ubuntu2
COPY app.go .
RUN pwd
RUN CGO\_ENABLED=0 go build app.go

# app container stage 2
FROM alpine:3.12.0
RUN addgroup -S appgroup && adduser -S appuser -G appgroup -h /home/appuser
COPY --from=0 /app /home/appuser/
USER appuser
CMD ["/home/appuser/app"]

Make File Systems Read only (chmod a-w /etc)

# build container stage 1
FROM ubuntu:20.04
ARG DEBIAN\_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y golang-go=2:1.13~1ubuntu2
COPY app.go .
RUN pwd
RUN CGO\_ENABLED=0 go build app.go

# app container stage 2
FROM alpine:3.12.0
RUN chmod a-r /etc
RUN addgroup -S appgroup && adduser -S appuser -G appgroup -h /home/appuser
COPY --from=0 /app /home/appuser/
USER appuser
CMD ["/home/appuser/app"]

Remove Shell Access

# build container stage 1
FROM ubuntu:20.04
ARG DEBIAN\_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y golang-go=2:1.13~1ubuntu2
COPY app.go .
RUN pwd
RUN CGO\_ENABLED=0 go build app.go

# app container stage 2
FROM alpine:3.12.0
RUN addgroup -S appgroup && adduser -S appuser -G appgroup -h /home/appuser
RUN rm -rf /bin/\*
COPY --from=0 /app /home/appuser/
USER appuser
CMD ["/home/appuser/app"]

Open Policy Agent (OPA)


title: “Open Policy Agent (OPA)”
date: 2020-12-13T18:08:39
slug: open-policy-agent-opa


Install OPA: kubectl create -f https://raw.githubusercontent.com/killer-sh/cks-course-environment/master/course-content/opa/gatekeeper.yaml
Create DenyAll Policy for Pods: https://github.com/killer-sh/cks-course-environment/tree/master/course-content/opa/deny-all

https://play.openpolicyagent.org

https://github.com/BouweCeunen/gatekeeper-policies

https://github.com/open-policy-agent/gatekeeper-library/tree/master/library/general
https://github.com/open-policy-agent/gatekeeper/tree/master/demo/basic

Example required memorylimit:

cat requiredresources-template.yaml
apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
 name: k8srequiredresources
spec:
 crd:
 spec:
 names:
 kind: K8sRequiredResources
 listKind: K8sRequiredResourcesList
 plural: k8srequiredresources
 singular: k8srequiredresources
 validation:
 # Schema for the `parameters` field
 openAPIV3Schema:
 properties:
 requests\_cpu:
 type: string
 requests\_memory:
 type: string
 limits\_cpu:
 type: string
 limits\_memory:
 type: string
 targets:
 - target: admission.k8s.gatekeeper.sh
 rego: |

 package k8srequiredresources

 violation[{"msg": msg}] {
 container := input.review.object.spec.containers[\_]
 #not container.resources.limits
 not container.resources.limits.memory
 msg := sprintf("container <%v> has no memory limits", [container.name])
 }
cat resources-policy.yml
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredResources
metadata:
 name: resources-policy
spec:
 match:
 kinds:
 - apiGroups: ["batch", "extensions", "apps", ""]
 kinds: ["Deployment", "Pod", "CronJob", "Job", "StatefulSet", "DaemonSet"]

OS Level Security Domains


title: “OS Level Security Domains”
date: 2020-12-13T16:05:21
slug: os-level-security-domains


Enable PodSecurityPolicy in /etc/kubernetes/manifests/kube-apiserver.yaml (add PodSecurityPolicy)

spec:
 containers:
 - command:
 - kube-apiserver
 - --advertise-address=10.156.0.6
 - --allow-privileged=true
 - --authorization-mode=Node,RBAC
 - --client-ca-file=/etc/kubernetes/pki/ca.crt
 - --enable-admission-plugins=NodeRestriction,PodSecurityPolicy

Create a PodSecurityPolicy:

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
 name: example
spec:
 allowPrivilegeEscalation: false
 privileged: false
 seLinux:
 rule: RunAsAny
 supplementalGroups:
 rule: RunAsAny
 runAsUser:
 rule: RunAsAny
 fsGroup:
 rule: RunAsAny
 volumes:
 - '\*'

Create a Role and assign it to the default SA:

kubectl create role psp-access --verb=use --resource=podsecuritypolicies
kubectl create rolebinding psp-access --role=psp-access --serviceaccount=default:default

Assign top all SA in Namepsace team-red
kubectl create rolebinding psp-mount --clusterrole=psp-mount --group=system:serviceaccounts -n team-red

Container Runtime Sandboxes


title: “Container Runtime Sandboxes”
date: 2020-12-12T13:16:53
slug: container-runtime-sandboxes


Run a container with an own Kernel/Runtime
Check on which container daemon is the Node Running:

# k get node -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
cks-master Ready master 4d1h v1.19.3 10.156.0.2 Ubuntu 18.04.5 LTS 5.4.0-1030-gcp docker://19.3.6
cks-worker Ready 4d1h v1.19.3 10.156.0.3 Ubuntu 18.04.5 LTS 5.4.0-1030-gcp docker://19.3.6

Install gVisor/runsc

curl -fsSL https://gvisor.dev/archive.key | sudo apt-key add -
sudo add-apt-repository "deb https://storage.googleapis.com/gvisor/releases release main"
sudo apt-get update && sudo apt-get install -y runsc

cat <<EOF > /etc/default/kubelet
KUBELET\_EXTRA\_ARGS="--container-runtime remote --container-runtime-endpoint unix:///run/containerd/containerd.sock"
EOF
systemctl daemon-reload
systemctl restart kubelet

cat <<EOF > /etc/containerd/config.toml
disabled\_plugins = ["restart"]
[plugins.linux]
 shim\_debug = true
[plugins.cri.containerd.runtimes.runsc]
 runtime\_type = "io.containerd.runsc.v1"
EOF

# crictl should use containerd as default
cat <<EOF > /etc/crictl.yaml
runtime-endpoint: unix:///run/containerd/containerd.sock
EOF
systemctl restart containerd

Check on which container daemon is the Node Running:

# k get node -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
cks-master Ready master 4d1h v1.19.3 10.156.0.2 Ubuntu 18.04.5 LTS 5.4.0-1030-gcp docker://19.3.6
cks-worker Ready 4d1h v1.19.3 10.156.0.3 Ubuntu 18.04.5 LTS 5.4.0-1030-gcp containerd://1.3.3

Create a RuntimeClass:

cat <<EOF | kubectl apply -f -
apiVersion: node.k8s.io/v1beta1
kind: RuntimeClass
metadata:
 name: gvisor
handler: runsc
EOF

Manage Kubernetes Secrets


title: “Manage Kubernetes Secrets”
date: 2020-12-11T22:36:47
slug: manage-kubernetes-secrets


Encrypting your data in ETCD

Create a new encryption config file:

head -c 32 /dev/urandom | base64
vi /etc/kubernetes/etcd/ec.yaml

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
 - resources:
 - secrets
 providers:
 - aescbc:
 keys:
 - name: key1
 secret: <----- Password must be 16, 32 or 64 Bit long
 - identity: {}

Add a kube-api Parameter:

vi /etc/kubernetes/manifests/kube-apiserver.yaml
--encryption-provider-config=/etc/kubernetes/etcd/ec.yaml

Add under Volumemount:
 - mountPath: /etc/kubernetes/etcd
 name: etcd
 readOnly: true

Add under Volumes:
 - hostPath:
 path: /etc/kubernetes/etcd
 type: DirectoryOrCreate
 name: etcd

Read a secret from etcd

ETCDCTL\_API=3 etcdctl --cert /etc/kubernetes/pki/apiserver-etcd-client.crt --key /etc/kubernetes/pki/apiserver-etcd-client.key --cacert /etc/kubernetes/pki/etcd/ca.crt get /registry/secrets/default/secure-ingress

Rewrite a secred (for encoding)

k get secret secure-ingress -o yaml|k replace -f -

Rewrite all Secrets, after this, remove the “- identity: {}” Provider:

kubectl get secrets --all-namespaces -o json | kubectl replace -f -

Cluster Hardening – Upgrade Kubernetes


title: “Cluster Hardening – Upgrade Kubernetes”
date: 2020-12-10T21:54:21
slug: cluster-hardening-upgrade-kubernetes


kubectl drain
Do the upgrade
kubectl uncordon

Upgrade the Cluster to one minor Version up

Masternode:

k drain xxxxx --ignore-daemonsets
apt-cache show kubeadm | grep 1.19 (next Version)
apt install kubeadm=xxxxx kubelet=xxxx kubectl=xxxxx
kubeadm upgrade plan
kubeadm upgrade apply 1.19.3 (suggested command)
k uncordon xxxx

Workernode:

k drain xxxxx --ignore-daemonsets
apt-cache show kubeadm | grep 1.19
apt install kubeadm=xxxxx kubelet=xxxx kubectl=xxxxx
systemctl restart kubelet
k uncordon xxxx

Cluster Hardening – Restrict API Access


title: “Cluster Hardening – Restrict API Access”
date: 2020-12-10T21:11:57
slug: cluster-hardening-restrict-api-access


Disable anonymous Access (- –anonymous-auth= false)
vi /etc/kubernetes/manifests/kube-apiserver.yaml

apiVersion: v1
kind: Pod
metadata:
 annotations:
 kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 10.156.0.2:6443
 creationTimestamp: null
 labels:
 component: kube-apiserver
 tier: control-plane
 name: kube-apiserver
 namespace: kube-system
spec:
 containers:
 - command:
 - kube-apiserver
 - --anonymous-auth=false
 - --advertise-address=10.156.0.2
 - --allow-privileged=true
 - --authorization-mode=Node,RBAC

Disable anonymous auth: – –anonymous-auth=false
Disable Insecure Port in /etc/kubernetes/manifests/kube-apiserver.yaml by setting the port to 0
Disable the Node Port by comment out: # – –kubernetes-service-node-port=31000 # delete or set to 0