Kubernetes

来自Gea-Suan Lin's Wiki
跳到导航 跳到搜索

Kubernetes是一套由Google所发展出来的布署系统。

环境

这边是以Ubuntu 18.04为基础,在AWS上使用一台c5.2xlarge与五台r5.large测试(单master版本),或是再加上三台t3.small并且使用ELB(多master版本)。

安装

先安装Docker,然后安装Kubernetes的套件:

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -; echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list; sudo apt update; sudo apt install -y kubelet kubeadm kubectl

单master版本设定

这边使用Calico当作网络层:

sudo kubeadm init --pod-network-cidr=192.168.0.0/16

把上面执行结果输出的命令拿到别台用sudo跑,像是这样的指令:

sudo kubeadm join a.b.c.d:6443 --token xxxxxx.xxxxxxxxxxxxxxxx --discovery-token-ca-cert-hash sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

接着回到当初跑kubeadm init的机器上,把设定档放到自己目录下:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

接下来启用Calico设定:

kubectl apply -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml
kubectl apply -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml

设定好之后不会马上通,可以用kubectl get nodes --watch可以看到机器会因为retry从NotReady变成Ready

多master版本设定

大多数的设定与单master版本的设定相同,请参考前面对单master版本的说明。

在多master版本中需要三台主机确保HA,以及一组load balancer提供让程式可以操作Kubernetes(预设为port 6443)。

由于我们的测试是在AWS上面,所以这边使用了ELB(Classic版本,Internal,仅使用TCP Proxy)来当作load balancer(即下面设定中的internal-test-gslin-k8s-apiserver-XXXXXXXXXX.us-east-1.elb.amazonaws.com:6443)。

我们需要透过设定档设定对应的controlPaneEndpointpodSubnet(因为--config--pod-network-cidr不能同时使用):

apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
kubernetesVersion: stable
apiServer:
  certSANs:
  - "internal-test-gslin-k8s-apiserver-XXXXXXXXXX.us-east-1.elb.amazonaws.com"
controlPlaneEndpoint: "internal-test-gslin-k8s-apiserver-XXXXXXXXXX.us-east-1.elb.amazonaws.com:6443"
networking:
  podSubnet: 192.168.0.0/16

然后再透过设定档初始化cluster的第一台主机:

sudo kubeadm init --config=kubeadm-config.yaml

依照单master设定复制完设定档:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

再来是设定CNI,在官方文件上是建议使用Weave:

kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

接下来是复制档案到其他机器上,这边是让tmuxscreen的使用者比较方便,可以直接剪下贴上:

cd /etc/kubernetes/
sudo tar -zc -f - pki/{ca.*,sa.*,front-proxy-ca.*,etcd/ca.*} admin.conf | uuencode a.tar.gz

然后到目录下产生a.tar.gz再解开:

cd /etc/kubernetes/
sudo uudecode
sudo tar zxvf a.tar.gz

加入时除了本来的指令外,需要加上--experimental-control-plane(只有这三台主机要加,其他的机器不用):

sudo kubeadm join internal-test-gslin-k8s-apiserver-XXXXXXXXXX.us-east-1.elb.amazonaws.com:6443 --token xxxxxx.xxxxxxxxxxxxxxxx --discovery-token-ca-cert-hash sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx --experimental-control-plane

事后增加主机

token预设是24小时有效,在过期后需要重新建立:

sudo kubeadm token create

可以透过这个指令在建立token的同时,产生完整的加入指令:

echo sudo kubeadm join $(kubeadm config view | grep ^controlPlaneEndpoint | awk '{print $2}') --token $(sudo kubeadm token create) --discovery-token-ca-cert-hash sha256:$(openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //')

使用所有主机

一开始的主机(master)不会被分配到需求(因为安全设计),如果你希望让master也可以加入,可以透过以下的指令让master分配到需求:

kubectl taint nodes --all node-role.kubernetes.io/master-

标签

可以针对主机进行标签,供之后的nodeSelector使用:

kubectl label nodes ip-172-31-1-1 instancetype=c5
kubectl label nodes ip-172-31-1-2 ip-172-31-1-3 ip-172-31-1-4 ip-172-31-1-5 ip-172-31-1-6 instancetype=r5

范例

简单的版本,跑两个Pod并且跑起sshd,并且把Host的Port 80拿来用(会是SSH):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sshd
spec:
  replicas: 2
  selector:
    matchLabels:
      app: sshd
  template:
    metadata:
      labels:
        app: sshd
    spec:
      containers:
      - name: sshd
        image: ubuntu:18.04
        command: ["/bin/sh", "-c"]
        args:
          - export DEBIAN_FRONTEND=noninteractive;
            sed -i 's/archive.ubuntu.com/us.archive.ubuntu.com/' /etc/apt/sources.list;
            apt update;
            apt install -y iproute2 iputils-ping locales mtr-tiny net-tools openssh-server telnet tzdata wget;
            mkdir /run/sshd; /usr/sbin/sshd;
            sleep 3153600000
        ports:
          - containerPort: 22
            hostPort: 80

复杂一点的版本:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: example-r5
spec:
  replicas: 5
  serviceName: example-r5
  selector:
    matchLabels:
      app: example-r5
  template:
    metadata:
      labels:
        app: example-r5
    spec:
      containers:
      - name: example-r5
        image: ubuntu:18.04
        command: ["/bin/sh", "-c"]
        args:
          - export DEBIAN_FRONTEND=noninteractive;
            sed -i 's/archive.ubuntu.com/us.archive.ubuntu.com/' /etc/apt/sources.list;
            apt update;
            apt install -y iproute2 iputils-ping locales mtr-tiny net-tools tzdata wget;
            sleep 3153600000
        resources:
          requests:
            memory: "15Gi"
      nodeSelector:
        instancetype: r5
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: example-c5
spec:
  replicas: 1
  serviceName: example-c5
  selector:
    matchLabels:
      app: example-c5
  template:
    metadata:
      labels:
        app: example-c5
    spec:
      containers:
      - name: example-c5
        image: ubuntu:18.04
        command: ["/bin/sh", "-c"]
        args:
          - export DEBIAN_FRONTEND=noninteractive;
            sed -i 's/archive.ubuntu.com/us.archive.ubuntu.com/' /etc/apt/sources.list;
            apt update;
            apt install -y iproute2 iputils-ping locales mtr-tiny net-tools tzdata wget;
            sleep 3153600000
        resources:
          requests:
            cpu: "7000m"
      nodeSelector:
        instancetype: c5

相关连结

外部链接