Skip to content

Latest commit

 

History

History
1022 lines (957 loc) · 37.8 KB

Ubuntu_Documentation_No_KVM_snap.adoc

File metadata and controls

1022 lines (957 loc) · 37.8 KB

Installing Kubernetes on LinuxOne - Canonical Distribution

Table of Contents

Note: This setup documents running a two node (1 Master & 1 Worker) Kubernetes Cluster on LinuxOne cloud using LinuxOne supported linux distribution Canonical Ubuntu 18.04.

Also please do note that, all the lines starting with # in the code blocks are comments and are not intended to be ran, although they are harmless when ran ;).

Versions of all the components used

The cluster is based on docker as container run-time and the components are installed as systemd services with flannel as the overlay networking model. The versions of each components used are as follows

Component/ Infrastructure Version

LinuxOne Instance flavour

LinuxONE-Medium

LinuxOne Instance OS

Ubuntu 18.04

Kubernetes

1.9.8

Docker

17.12.1-ce

Etcd

3.2.9

Flannel

0.10.0

Setting up the Nodes before installing the components

This setup requires about 2 LinuxOne cloud instances. One for master and one for worker. Make sure you install Ubuntu on both the instances. SSH into the nodes and follow the below steps in the nodes specified across each step

  1. Set the hostnames for each of the nodes using the utility hostnamectl as follows. Preferrably set the hostnames as k8s-master & k8s-worker respectively.

    # On Master Node
sudo hostnamectl set-hostname k8s-master
# On Worker Node
sudo hostnamectl set-hostname k8s-worker

  2. Note down the IP addresses of both the nodes. In the context of this document, IP addresses of the nodes are as follows

    Node IP

    Master

    148.100.99.99

    Worker

    148.100.99.100

  3. Install vim, net-tools and snap on both the nodes. Vim is needed to edit files, although preinstalled nano can be used. Curl is needed to test services running on a pod during the testing. Net-tools is needed to test and troubleshoot working of flannel. Snap is the package manager we will be using to install Kubernetes components.

    sudo apt-get -y install vim curl net-tools snapd

  4. Install docker on Worker node(s).

    sudo apt install docker.io
sudo systemctl enable docker.service
sudo systemctl start docker.service

  5. Enable IP forward settings on all the worker node(s) so that, the containers are able to ping the internet using the following commands

    sysctl -w net.ipv4.ip_forward=1
sysctl -w net.ipv4.conf.all.forwarding=1
sysctl -w net.bridge.bridge-nf-call-iptables=1

  6. Create the below folders in order to save certificates & configuration files later

    • On Master Node

      sudo mkdir -p /root/srv/kubernetes
sudo mkdir /var/lib/flanneld/

    • On Worker Node(s)

      sudo mkdir -p /root/srv/kubernetes
sudo mkdir /var/lib/flanneld/

  7. Kubernetes is doesn’t work as it is expected with swap memory on. Disable swap space on the worker node(s), by running

    swapoff -a
sudo sed -i 's/\/swapfile/#\/swapfile/g' /etc/fstab

  8. Copy the contents of SSH public key of the Master node in the authorized_keys file on all the worker node(s). By doing this, we can scp files into the worker node(s) from the master node without having to authenticate.

    # On the master node, check if you already have the file 'id_rsa.pub' in '.ssh/'. If not generate the key using the following steps
ssh-keygen
# Give default settings by pressing enter
cat .ssh/id_rsa.pub
#Copy the contents from the output of the above command
# On the worker node(s) paste the copied contents at the end in the file '.ssh/authorized_keys'
echo "<key_copied>" >> .ssh/authorized_keys

  9. It is recommended to login as root on all the nodes, as we will be dealing with files and tools which require root permissions. You can login as root from another account using sudo -i. If not, use sudo before a command wherever required. Anyways, just to be safe I will include sudo where required.

Installing Kubernetes binaries on the Nodes

In this setup, we will be using snap package manager to install all the components. Run the following commands on the respective nodes

Master Node

sudo snap install kube-apiserver --channel=1.9/stable --classic
sudo snap install kube-controller-manager --channel=1.9/stable --classic
sudo snap install kube-scheduler --channel=1.9/stable --classic
sudo snap install kubectl --channel=1.9/stable --classic

Worker Node

sudo snap install kubelet --channel=1.9/stable --classic
sudo snap install kube-proxy --channel=1.9/stable --classic

Creating Certificates & Authentication to enable secure communication across all the Kubernetes components

Run all the following steps and thereby generate the files in the Master node, and then copy the specific mentioned certs and config files to the worker nodes.

Generating Certificates

CA - Certificate Authority

cd /root/srv/kubernetes
openssl genrsa -out ca-key.pem 2048
openssl req -x509 -new -nodes -key ca-key.pem -days 10000 -out ca.pem -subj "/CN=kube-ca"

Master Node OpenSSL config

cat > openssl.cnf <<EOF
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name

[req_distinguished_name]

[v3_req]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names

[alt_names]
DNS.1 = kubernetes
DNS.2 = kubernetes.default
DNS.3 = kubernetes.default.svc
DNS.4 = kubernetes.default.svc.cluster.local
IP.1 = 127.0.0.1
IP.2 = 148.100.99.99 # Master IP
IP.3 = 100.65.0.1 # Service IP
EOF

Kube-apiserver certificates

openssl genrsa -out apiserver-key.pem 2048
openssl req -new -key apiserver-key.pem -out apiserver.csr -subj "/CN=kube-apiserver" -config openssl.cnf
openssl x509 -req -in apiserver.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial \
  -out apiserver.pem -days 7200 -extensions v3_req -extfile openssl.cnf

Admin certificates

openssl genrsa -out admin-key.pem 2048
openssl req -new -key admin-key.pem -out admin.csr -subj "/CN=admin"
openssl x509 -req -in admin.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out admin.pem -days 7200

Kube-controller-manager certificates

openssl genrsa -out kube-controller-manager-key.pem 2048
openssl req -new -key kube-controller-manager-key.pem -out kube-controller-manager.csr -subj "/CN=kube-controller-manager"
openssl x509 -req -in kube-controller-manager.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out kube-controller-manager.pem -days 7200

Kube-scheduler certificates

openssl genrsa -out kube-scheduler-key.pem 2048
openssl req -new -key kube-scheduler-key.pem -out kube-scheduler.csr -subj "/CN=kube-scheduler"
openssl x509 -req -in kube-scheduler.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out kube-scheduler.pem -days 7200

Worker OpenSSL config

cat > worker-openssl.cnf << EOF
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[v3_req]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names
[alt_names]
IP.1 = 148.100.99.100
#IP.2 If more workers are to be configured
EOF

Kubelet certificates

openssl genrsa -out kubelet-key.pem 2048
openssl req -new -key kubelet-key.pem -out kubelet.csr -subj "/CN=kubelet" -config worker-openssl.cnf
openssl x509 -req -in kubelet.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out kubelet.pem -days 7200 -extensions v3_req -extfile worker-openssl.cnf

Kube-proxy certificates

openssl genrsa -out kube-proxy-key.pem 2048
openssl req -new -key kube-proxy-key.pem -out kube-proxy.csr -subj "/CN=kube-proxy"
openssl x509 -req -in kube-proxy.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out kube-proxy.pem -days 7200

Worker node certificates

Note: k8sworker here (and also in the next few steps) refers to the hostname of the worker & 148.100.99.100 refers to the worker node IP. Repeat the procedure after changing the hostname and IP to configure more worker nodes.

openssl genrsa -out k8sworker-worker-key.pem 2048
WORKER_IP=148.100.99.100 openssl req -new -key k8sworker-worker-key.pem -out k8sworker-worker.csr -subj "/CN=system:node:k8sworker" -config worker-openssl.cnf
WORKER_IP=148.100.99.100 openssl x509 -req -in k8sworker-worker.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out k8sworker-worker.pem -days 7200 -extensions v3_req -extfile worker-openssl.cnf

Etcd OpenSSL config

cat > etcd-openssl.cnf <<EOF
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth,serverAuth
subjectAltName = @alt_names
[alt_names]
IP.1 = 148.100.99.99
EOF

Etcd certificates

openssl genrsa -out etcd.key 2048
openssl req -new -key etcd.key -out etcd.csr -subj "/CN=etcd" -extensions v3_req -config etcd-openssl.cnf -sha256
openssl x509 -req -sha256 -CA ca.pem -CAkey ca-key.pem -CAcreateserial \
  -in etcd.csr -out etcd.crt -extensions v3_req -extfile etcd-openssl.cnf -days 7200

Copy the required certificates to the Worker node

# Run the below step on the Master node
scp ca.pem etcd.crt etcd.key kubelet.pem kubelet-key.pem [email protected]:/root/srv/kubernetes/

Generating Kubeconfig files

Admin Kubeconfig

TOKEN=$(dd if=/dev/urandom bs=128 count=1 2>/dev/null | base64 | tr -d "=+/" | dd bs=32 count=1 2>/dev/null)
kubectl config set-cluster linux1.k8s --certificate-authority=/root/srv/kubernetes/ca.pem --embed-certs=true --server=https://148.100.99.99:6443
kubectl config set-credentials admin --client-certificate=/root/srv/kubernetes/admin.pem --client-key=/root/srv/kubernetes/admin-key.pem --embed-certs=true --token=$TOKEN
kubectl config set-context linux1.k8s --cluster=linux1.k8s --user=admin
kubectl config use-context linux1.k8s
cat ~/.kube/config #Create config file

Kube-controller-manager Kubeconfig

TOKEN=$(dd if=/dev/urandom bs=128 count=1 2>/dev/null | base64 | tr -d "=+/" | dd bs=32 count=1 2>/dev/null)
kubectl config set-cluster linux1.k8s --certificate-authority=/root/srv/kubernetes/ca.pem --embed-certs=true --server=https://148.100.99.99:6443 --kubeconfig=/root/srv/kubernetes/kube-controller-manager.kubeconfig
kubectl config set-credentials kube-controller-manager --client-certificate=/root/srv/kubernetes/kube-controller-manager.pem --client-key=/root/srv/kubernetes/kube-controller-manager-key.pem --embed-certs=true --token=$TOKEN --kubeconfig=/root/srv/kubernetes/kube-controller-manager.kubeconfig
kubectl config set-context linux1.k8s --cluster=linux1.k8s --user=kube-controller-manager --kubeconfig=/root/srv/kubernetes/kube-controller-manager.kubeconfig
kubectl config use-context linux1.k8s --kubeconfig=/root/srv/kubernetes/kube-controller-manager.kubeconfig

Kube-scheduler Kubeconfig

TOKEN=$(dd if=/dev/urandom bs=128 count=1 2>/dev/null | base64 | tr -d "=+/" | dd bs=32 count=1 2>/dev/null)
kubectl config set-cluster linux1.k8s --certificate-authority=/root/srv/kubernetes/ca.pem --embed-certs=true --server=https://148.100.99.99:6443 --kubeconfig=/root/srv/kubernetes/kube-scheduler.kubeconfig
kubectl config set-credentials kube-scheduler --client-certificate=/root/srv/kubernetes/kube-scheduler.pem --client-key=/root/srv/kubernetes/kube-scheduler-key.pem --embed-certs=true --token=$TOKEN --kubeconfig=/root/srv/kubernetes/kube-scheduler.kubeconfig
kubectl config set-context linux1.k8s --cluster=linux1.k8s --user=kube-scheduler --kubeconfig=/root/srv/kubernetes/kube-scheduler.kubeconfig
kubectl config use-context linux1.k8s --kubeconfig=/root/srv/kubernetes/kube-scheduler.kubeconfig

Kubelet Kubeconfig (for each Worker Node)

TOKEN=$(dd if=/dev/urandom bs=128 count=1 2>/dev/null | base64 | tr -d "=+/" | dd bs=32 count=1 2>/dev/null)
kubectl config set-cluster linux1.k8s --certificate-authority=/root/srv/kubernetes/ca.pem --embed-certs=true --server=https://148.100.99.99:6443 --kubeconfig=/root/srv/kubernetes/k8sworker-worker.kubeconfig
kubectl config set-credentials k8sworker --client-certificate=/root/srv/kubernetes/k8sworker-worker.pem --client-key=/root/srv/kubernetes/k8sworker-worker-key.pem --embed-certs=true --token=$TOKEN --kubeconfig=/root/srv/kubernetes/k8sworker-worker.kubeconfig
kubectl config set-context linux1.k8s --cluster=linux1.k8s --user=k8sworker --kubeconfig=/root/srv/kubernetes/k8sworker-worker.kubeconfig
kubectl config use-context linux1.k8s --kubeconfig=/root/srv/kubernetes/k8sworker-worker.kubeconfig

Copy the required config files to the worker node(s)

Similar to how we copied the certificates, we need to copy the configurations to the worker node(s)

# Run the below command on the master node
scp k8sworker-worker.kubeconfig [email protected]:/root/srv/kubernetes/kubelet.kubeconfig

Setting up Etcd

Installing Etcd

snap install --channel=3.2 etcd --classic

Configuring Etcd

Create a file in /var/snap/etcd/common/etcd.conf.yml with the following contents

# This is the configuration file for the etcd server.

# Human-readable name for this member.
name: 'master'

# Path to the data directory.
data-dir:

# Path to the dedicated wal directory.
wal-dir:

# Number of committed transactions to trigger a snapshot to disk.
snapshot-count: 10000

# Time (in milliseconds) of a heartbeat interval.
heartbeat-interval: 100

# Time (in milliseconds) for an election to timeout.
election-timeout: 1000

# Raise alarms when backend size exceeds the given quota. 0 means use the
# default quota.
quota-backend-bytes: 0

# List of comma separated URLs to listen on for peer traffic.
listen-peer-urls: https://148.100.99.99:2380

# List of comma separated URLs to listen on for client traffic.
listen-client-urls: https://148.100.99.99:2379

# Maximum number of snapshot files to retain (0 is unlimited).
max-snapshots: 5

# Maximum number of wal files to retain (0 is unlimited).
max-wals: 5

# Comma-separated white list of origins for CORS (cross-origin resource sharing).
cors:

# List of this member's peer URLs to advertise to the rest of the cluster.
# The URLs needed to be a comma-separated list.
initial-advertise-peer-urls: https://148.100.99.99:2380

# List of this member's client URLs to advertise to the public.
# The URLs needed to be a comma-separated list.
advertise-client-urls: https://148.100.99.99:2379

# Discovery URL used to bootstrap the cluster.
discovery:

# Valid values include 'exit', 'proxy'
discovery-fallback: 'proxy'

# HTTP proxy to use for traffic to discovery service.
discovery-proxy:

# DNS domain used to bootstrap initial cluster.
discovery-srv:

# Initial cluster configuration for bootstrapping.
initial-cluster: 'master=https://148.100.99.99:2380'

# Initial cluster token for the etcd cluster during bootstrap.
initial-cluster-token: 'etcd-cluster'

# Initial cluster state ('new' or 'existing').
initial-cluster-state: 'new'

# Reject reconfiguration requests that would cause quorum loss.
strict-reconfig-check: false

# Valid values include 'on', 'readonly', 'off'
proxy: 'off'

# Time (in milliseconds) an endpoint will be held in a failed state.
proxy-failure-wait: 5000

# Time (in milliseconds) of the endpoints refresh interval.
proxy-refresh-interval: 30000

# Time (in milliseconds) for a dial to timeout.
proxy-dial-timeout: 1000

# Time (in milliseconds) for a write to timeout.
proxy-write-timeout: 5000

# Time (in milliseconds) for a read to timeout.
proxy-read-timeout: 0

client-transport-security:
  # DEPRECATED: Path to the client server TLS CA file.
  ca-file:

  # Path to the client server TLS cert file.
  cert-file: '/root/srv/kubernetes/etcd.crt'

  # Path to the client server TLS key file.
  key-file: '/root/srv/kubernetes/etcd.key'

  # Enable client cert authentication.
  client-cert-auth: true

  # Path to the client server TLS trusted CA key file.
  trusted-ca-file: '/root/srv/kubernetes/ca.pem'

  # Client TLS using generated certificates
  auto-tls: false

peer-transport-security:
  # DEPRECATED: Path to the peer server TLS CA file.
  ca-file:

  # Path to the peer server TLS cert file.
  cert-file: '/root/srv/kubernetes/etcd.crt'

  # Path to the peer server TLS key file.
  key-file: '/root/srv/kubernetes/etcd.key'

  # Enable peer client cert authentication.
  client-cert-auth: true

  # Path to the peer server TLS trusted CA key file.
  trusted-ca-file: '/root/srv/kubernetes/ca.pem'

  # Peer TLS using generated certificates.
  auto-tls: false

# Enable debug-level logging for etcd.
debug: true

# Specify a particular log level for each etcd package (eg: 'etcdmain=CRITICAL,etcdserver=DEBUG'.
log-package-levels: 'DEBUG'

# Force to create a new one member cluster.
force-new-cluster: false

Now run the etcd systemd service

systemctl enable snap.etcd.etcd.service
systemctl start snap.etcd.etcd.service
systemctl status snap.etcd.etcd.service --no-pager

Test etcd cluster

etcdctl --endpoints https://148.100.99.99:2379 --cert-file /root/srv/kubernetes/etcd.crt --key-file /root/srv/kubernetes/etcd.key --ca-file /root/srv/kubernetes/ca.pem cluster-health

This should return cluster is healthy if etcd is running correctly.

Setting up Flannel

Flannel should be installed on all the nodes

Installing Flannel

cd ~/
wget https://github.com/coreos/flannel/releases/download/v0.10.0/flanneld-s390x
chmod +x flanneld-s390x
sudo cp flanneld-s390x /usr/local/bin/flanneld

Adding an entry to etcd

This should be run only once and only on the Master node

etcdctl --endpoints https://148.100.99.99:2379 --cert-file /root/srv/kubernetes/etcd.crt --key-file /root/srv/kubernetes/etcd.key --ca-file /root/srv/kubernetes/ca.pem set /coreos.com/network/config '{ "Network": "100.64.0.0/16", "SubnetLen": 24, "Backend": {"Type": "vxlan"} }'

Running Flanneld as a systemd service

Check the interface on which the nodes are connected using ip a. Here it is enc1. Replace it with the correct interface.

sudo cat > /etc/systemd/system/flanneld.service << EOF
[Unit]
Description=Network fabric for containers
Documentation=https://github.com/coreos/flannel
After=network.target
After=network-online.target
Wants=network-online.target
After=etcd.service
Before=docker.service

[Service]
Type=notify
Restart=always
RestartSec=5
ExecStart= /usr/local/bin/flanneld \\
  -etcd-endpoints=https://148.100.99.99:2379 \\
  -iface=enc1000 \\
  -ip-masq=true \\
  -subnet-file=/var/lib/flanneld/subnet.env \\
  -etcd-cafile=/root/srv/kubernetes/ca.pem \\
  -etcd-certfile=/root/srv/kubernetes/etcd.crt \\
  -etcd-keyfile=/root/srv/kubernetes/etcd.key

[Install]
WantedBy=multi-user.target
EOF
sudo systemctl enable flanneld
sudo systemctl start flanneld
sudo systemctl status flanneld --no-pager

The 'iface' here should be the interface through which the nodes communicate and flannel will be configured for this interface.

Changing Docker Settings

add the following lines to the /lib/systemd/system/docker.service EnvironmentFile=/var/lib/flanneld/subnet.env and change the line ExecStart=/usr/bin/dockerd -H fd:// to ExecStart=/usr/bin/dockerd -H fd:// --bip=${FLANNEL_SUBNET} --mtu=${FLANNEL_MTU} --iptables=false --ip-masq=false --ip-forward=true. The file should now some what look like

[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target docker.socket firewalld.service
Wants=network-online.target
Requires=docker.socket

[Service]
Type=notify
# FlannelD subnet setup
EnvironmentFile=/var/lib/flanneld/subnet.env
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd -H fd:// --bip=${FLANNEL_SUBNET} --mtu=${FLANNEL_MTU} --iptables=false --ip-masq=false --ip-forward=true
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=1048576
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
# restart the docker process if it exits prematurely
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s

[Install]
WantedBy=multi-user.target

Then run the following commands

sudo systemctl daemon-reload
sudo systemctl stop docker
sudo systemctl start docker

Testing flanneld

Once flanneld is started and docker daemon is restarted, running route -n on Master node and Worker node(s) the bridge established can be seen with the interface name as 'flannelx'. Also the IP of the nodes on the flannel networks can be seen by running ip a on all the nodes.

Setting up Kubernetes Components

Master Components

Kube-api-server configuration

Set the following parameters for Kube-apiserver using the snap set command

snap set kube-apiserver bind-address=0.0.0.0 \
advertise-address=148.100.99.99 \
admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,ResourceQuota \
allow-privileged=true \
anonymous-auth=false \
apiserver-count=1 \
authorization-mode=Node,RBAC,AlwaysAllow \
authorization-rbac-super-user=admin \
etcd-cafile=/root/srv/kubernetes/ca.pem \
etcd-certfile=/root/srv/kubernetes/etcd.crt \
etcd-keyfile=/root/srv/kubernetes/etcd.key \
etcd-servers=https://148.100.99.99:2379 \
enable-swagger-ui=true event-ttl=1h \
kubelet-certificate-authority=/root/srv/kubernetes/ca.pem \
kubelet-client-certificate=/root/srv/kubernetes/kubelet.pem \
kubelet-client-key=/root/srv/kubernetes/kubelet-key.pem \
kubelet-https=true \
client-ca-file=/root/srv/kubernetes/ca.pem \
runtime-config=api/all=true,batch/v2alpha1=true,rbac.authorization.k8s.io/v1alpha1=true \
secure-port=6443 \
service-cluster-ip-range=100.65.0.0/24 \
storage-backend=etcd3 \
tls-cert-file=/root/srv/kubernetes/apiserver.pem \
tls-private-key-file=/root/srv/kubernetes/apiserver-key.pem \
tls-ca-file=/root/srv/kubernetes/ca.pem \
logtostderr=true \
v=6

Run the service

systemctl enable snap.kube-apiserver.daemon
systemctl start snap.kube-apiserver.daemon
systemctl status snap.kube-apiserver.daemon --no-pager

Kube-scheduler configuration

Set the following parameters for Kube-scheduler using the snap set command

snap set kube-scheduler leader-elect=true \
kubeconfig=/root/srv/kubernetes/kube-scheduler.kubeconfig \
master=https://148.100.99.99:6443 \
v=2

Run the service

systemctl enable snap.kube-scheduler.daemon
systemctl start snap.kube-scheduler.daemon
systemctl status snap.kube-scheduler.daemon --no-pager

Kube-controller-manager configuration

Set the following parameters for Kube-controller-manager using the snap set command

snap set kube-controller-manager v=2 \
allocate-node-cidrs=true \
attach-detach-reconcile-sync-period=1m0s \
cluster-cidr=100.64.0.0/16 \
cluster-name=k8s.virtual.local \
leader-elect=true \
root-ca-file=/root/srv/kubernetes/ca.pem \
service-account-private-key-file=/root/srv/kubernetes/apiserver-key.pem \
use-service-account-credentials=true \
kubeconfig=/root/srv/kubernetes/kube-controller-manager.kubeconfig \
cluster-signing-cert-file=/root/srv/kubernetes/ca.pem \
cluster-signing-key-file=/root/srv/kubernetes/ca-key.pem \
service-cluster-ip-range=100.65.0.0/24 \
configure-cloud-routes=false \
master=https://148.100.99.99:6443 \
allow-privileged=true

Run the service

systemctl enable snap.kube-controller-manager.daemon
systemctl start snap.kube-controller-manager.daemon
systemctl status snap.kube-controller-manager.daemon --no-pager

Worker Components

Kubelet configuration

Set the following parameters for Kubelet using the snap set command

snap set kubelet kubeconfig=/root/srv/kubernetes/kubelet.kubeconfig \
address=148.100.99.100 \
allow-privileged=true \
tls-cert-file=/root/srv/kubernetes/kubelet.pem \
tls-private-key-file=/root/srv/kubernetes/kubelet-key.pem \
cert-dir=/root/srv/kubernetes \
container-runtime=docker \
serialize-image-pulls=false \
register-node=true \
cluster-dns=100.65.0.10 \
cluster-domain=cluster.local \
runtime-cgroups=/systemd/system.slice \
kubelet-cgroups=/systemd/system.slice \
docker=unix:///var/run/docker.sock \
node-labels=kubernetes.io/role=master,node-role.kubernetes.io/master= \
v=2 \
hostname-override=148.100.99.100 \
port=10250 \
logtostderr=true \
pod-manifest-path=/root/srv/kubernetes/manifests

Run the service

systemctl enable snap.kubelet.daemon
systemctl start snap.kubelet.daemon
systemctl status snap.kubelet.daemon --no-pager

Kube-proxy configuration

Set the following parameters for Kube-proxy using the snap set command

snap set kube-proxy cluster-cidr=100.64.0.0/16 \
masquerade-all=true \
kubeconfig=/root/srv/kubernetes/kubelet.kubeconfig \
proxy-mode=iptables  \
v=2

Run the service

systemctl enable snap.kube-proxy.daemon
systemctl start snap.kube-proxy.daemon
systemctl status snap.kube-proxy.daemon --no-pager

Testing the cluster

Now that we have deployed the cluster let’s test it.

Test if Kuberenetes Api Server is Running

Running kubectl version should return the version of both kubectl and kube-api-server

Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.9", GitCommit:"57729ea3d9a1b75f3fc7bbbadc597ba707d47c8a", GitTreeState:"clean", BuildDate:"2018-06-29T01:14:35Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/s390x"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.9", GitCommit:"57729ea3d9a1b75f3fc7bbbadc597ba707d47c8a", GitTreeState:"clean", BuildDate:"2018-06-29T01:07:01Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/s390x"}

Test if all the components are healthy

Running kubectl get componentstatus should return the status of all the components

NAME                 STATUS    MESSAGE             ERROR
scheduler            Healthy   ok
controller-manager   Healthy   ok
etcd-0               Healthy   {"health":"true"}

Test if the node is registered

Running kubectl get nodes should return the nodes sucessfully registered with the server and status of each node.

NAME             STATUS    ROLES     AGE       VERSION
148.100.98.235   Ready     master    5h        v1.9.8

Deploy ngninx

Let’s run an Ngnix app on the cluster.

kubectl run nginx --image=nginx --port=80 --replicas=3
kubectl get pods -o wide
kubectl expose deployment nginx --type NodePort
NODE_PORT=$(kubectl get svc nginx --output=jsonpath='{range .spec.ports[0]}{.nodePort}')
curl http://148.100.99.100:${NODE_PORT} #The IP is of Worker node

Setting up Kube-dns

I will use the following yaml derived from the official kubernetes repository, made some changes (which are highlighted in red). Make sure the clusterIP here is same as what we provide as parameters to the Kubernetes components. The yaml file can be directly download from here.

# Copyright 2016 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

apiVersion: v1
kind: Service
metadata:
  name: kube-dns
  namespace: kube-system
  labels:
    k8s-app: kube-dns
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
    kubernetes.io/name: "KubeDNS"
spec:
  selector:
    k8s-app: kube-dns
  clusterIP: 100.65.0.10
  ports:
  - name: dns
    port: 53
    protocol: UDP
  - name: dns-tcp
    port: 53
    protocol: TCP
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kube-dns
  namespace: kube-system
  labels:
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: kube-dns
  namespace: kube-system
  labels:
    addonmanager.kubernetes.io/mode: EnsureExists
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: kube-dns
  namespace: kube-system
  labels:
    k8s-app: kube-dns
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  # replicas: not specified here:
  # 1. In order to make Addon Manager do not reconcile this replicas parameter.
  # 2. Default is 1.
  # 3. Will be tuned in real time if DNS horizontal auto-scaling is turned on.
  strategy:
    rollingUpdate:
      maxSurge: 10%
      maxUnavailable: 0
  selector:
    matchLabels:
      k8s-app: kube-dns
  template:
    metadata:
      labels:
        k8s-app: kube-dns
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ''
    spec:
      tolerations:
      - key: "CriticalAddonsOnly"
        operator: "Exists"
      volumes:
      - name: kube-dns-config
        configMap:
          name: kube-dns
          optional: true
      - hostPath:
          path: /root/srv/kubernetes/
        name: ssl-certs-kubernetes
      containers:
      - name: kubedns
        image: gcr.io/google_containers/k8s-dns-kube-dns-s390x:1.14.7
        resources:
          # TODO: Set memory limits when we've profiled the container for large
          # clusters, then set request = limit to keep this container in
          # guaranteed class. Currently, this container falls into the
          # "burstable" category so the kubelet doesn't backoff from restarting it.
          limits:
            memory: 170Mi
          requests:
            cpu: 100m
            memory: 70Mi
        livenessProbe:
          httpGet:
            path: /healthcheck/kubedns
            port: 10054
            scheme: HTTP
          initialDelaySeconds: 60
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 5
        readinessProbe:
          httpGet:
            path: /readiness
            port: 8081
            scheme: HTTP
          # we poll on pod startup for the Kubernetes master service and
          # only setup the /readiness HTTP server once that's available.
          initialDelaySeconds: 3
          timeoutSeconds: 5
        args:
        - --domain=cluster.local.
        - --dns-port=10053
        - --kube-master-url=https://148.100.99.99:6443
        - --config-dir=/kube-dns-config
        - --kubecfg-file=/root/srv/kubernetes/kubelet.kubeconfig
        - --v=2
        env:
        - name: PROMETHEUS_PORT
          value: "10055"
        ports:
        - containerPort: 10053
          name: dns-local
          protocol: UDP
        - containerPort: 10053
          name: dns-tcp-local
          protocol: TCP
        - containerPort: 10055
          name: metrics
          protocol: TCP
        volumeMounts:
        - name: kube-dns-config
          mountPath: /kube-dns-config
        - name: ssl-certs-kubernetes
          mountPath: /root/srv/kubernetes
          readOnly: true
      - name: dnsmasq
        image: gcr.io/google_containers/k8s-dns-dnsmasq-nanny-s390x:1.14.7
        livenessProbe:
          httpGet:
            path: /healthcheck/dnsmasq
            port: 10054
            scheme: HTTP
          initialDelaySeconds: 60
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 5
        args:
        - -v=2
        - -logtostderr
        - -configDir=/etc/k8s/dns/dnsmasq-nanny
        - -restartDnsmasq=true
        - --
        - -k
        - --cache-size=1000
        - --no-negcache
        - --log-facility=-
        - --server=/cluster.local/127.0.0.1#10053
        - --server=/in-addr.arpa/127.0.0.1#10053
        - --server=/ip6.arpa/127.0.0.1#10053
        ports:
        - containerPort: 53
          name: dns
          protocol: UDP
        - containerPort: 53
          name: dns-tcp
          protocol: TCP
        # see: https://github.com/kubernetes/kubernetes/issues/29055 for details
        resources:
          requests:
            cpu: 150m
            memory: 20Mi
        volumeMounts:
        - name: kube-dns-config
          mountPath: /etc/k8s/dns/dnsmasq-nanny
      - name: sidecar
        image: gcr.io/google_containers/k8s-dns-sidecar-s390x:1.14.7
        livenessProbe:
          httpGet:
            path: /metrics
            port: 10054
            scheme: HTTP
          initialDelaySeconds: 60
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 5
        args:
        - --v=2
        - --logtostderr
        - --probe=kubedns,127.0.0.1:10053,kubernetes.default.svc.cluster.local,5,SRV
        - --probe=dnsmasq,127.0.0.1:53,kubernetes.default.svc.cluster.local,5,SRV
        ports:
        - containerPort: 10054
          name: metrics
          protocol: TCP
        resources:
          requests:
            memory: 20Mi
            cpu: 10m
      dnsPolicy: Default  # Don't use cluster DNS.
      serviceAccountName: kube-dns

To deploy the kube-dns pod, save the yaml as say kube-dns.yaml and run the command kubectl apply -f kube-dns.yaml. In a few minutes, the pods will start running. Run the following commands to verify if kubedns is running as intended.

kubectl create -f https://k8s.io/examples/admin/dns/busybox.yaml
#Wait for the pod to start
kubectl exec -ti busybox -- nslookup kubernetes.default.svc.cluster.local
# This should result in returning the Kubernetes cluster IP

If in case the 'nslookup' is throwing up an error saying can’t reach server, try stopping the systemd-resolve service and try.

Troubleshooting