Kubernetes

来自Gea-Suan Lin's Wiki
跳到导航 跳到搜索
The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.

Kubernetes是一套由Google所发展出来的布署系统。

环境

这边是以Ubuntu 18.04为基础,在AWS上使用一台c5.2xlarge与五台r5.large测试(单master版本),或是再加上三台t3.small并且使用ELB(多master版本)。

安装

先安装Docker,然后安装Kubernetes的套件:

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -; echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list; sudo apt update; sudo apt install -y kubelet kubeadm kubectl

单master版本设定

这边使用Calico当作网络层:

sudo kubeadm init --pod-network-cidr=192.168.0.0/16

把上面执行结果输出的命令拿到别台用sudo跑,像是这样的指令:

sudo kubeadm join a.b.c.d:6443 --token xxxxxx.xxxxxxxxxxxxxxxx --discovery-token-ca-cert-hash sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

接着回到当初跑kubeadm init的机器上,把设定档放到自己目录下:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

接下来启用Calico设定:

kubectl apply -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml
kubectl apply -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml

设定好之后不会马上通,可以用kubectl get nodes --watch可以看到机器会因为retry从NotReady变成Ready

多master版本设定

大多数的设定与单master版本的设定相同,请参考前面对单master版本的说明。

在多master版本中需要三台主机确保HA,以及一组load balancer提供让程式可以操作Kubernetes(预设为port 6443)。

由于我们的测试是在AWS上面,所以这边使用了ELB(Classic版本,Internal,仅使用TCP Proxy)来当作load balancer(即下面设定中的internal-test-gslin-k8s-apiserver-XXXXXXXXXX.us-east-1.elb.amazonaws.com:6443)。

我们需要透过设定档设定对应的controlPaneEndpointpodSubnet(因为--config--pod-network-cidr不能同时使用):

apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
kubernetesVersion: stable
apiServer:
  certSANs:
  - "internal-test-gslin-k8s-apiserver-XXXXXXXXXX.us-east-1.elb.amazonaws.com"
controlPlaneEndpoint: "internal-test-gslin-k8s-apiserver-XXXXXXXXXX.us-east-1.elb.amazonaws.com:6443"
networking:
  podSubnet: 192.168.0.0/16

然后再透过设定档初始化cluster的第一台主机:

sudo kubeadm init --config=kubeadm-config.yaml

依照单master设定复制完设定档:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

再来是设定CNI,在官方文件上是建议使用Weave:

kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

接下来是复制档案到其他机器上,这边是让tmuxscreen的使用者比较方便,可以直接剪下贴上:

cd /etc/kubernetes/
sudo tar -zc -f - pki/{ca.*,sa.*,front-proxy-ca.*,etcd/ca.*} admin.conf | uuencode a.tar.gz

然后到目录下产生a.tar.gz再解开:

cd /etc/kubernetes/
sudo uudecode
sudo tar zxvf a.tar.gz

加入时除了本来的指令外,需要加上--experimental-control-plane(只有这三台主机要加,其他的机器不用):

sudo kubeadm join internal-test-gslin-k8s-apiserver-XXXXXXXXXX.us-east-1.elb.amazonaws.com:6443 --token xxxxxx.xxxxxxxxxxxxxxxx --discovery-token-ca-cert-hash sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx --experimental-control-plane

事后增加主机

token预设是24小时有效,在过期后需要重新建立:

sudo kubeadm token create

可以透过这个指令在建立token的同时,产生完整的加入指令:

echo sudo kubeadm join $(kubeadm config view | grep ^controlPlaneEndpoint | awk '{print $2}') --token $(sudo kubeadm token create) --discovery-token-ca-cert-hash sha256:$(openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //')

使用所有主机

一开始的主机(master)不会被分配到需求(因为安全设计),如果你希望让master也可以加入,可以透过以下的指令让master分配到需求:

kubectl taint nodes --all node-role.kubernetes.io/master-

标签

可以针对主机进行标签,供之后的nodeSelector使用:

kubectl label nodes ip-172-31-1-1 instancetype=c5
kubectl label nodes ip-172-31-1-2 ip-172-31-1-3 ip-172-31-1-4 ip-172-31-1-5 ip-172-31-1-6 instancetype=r5

范例

简单的版本,跑两个Pod并且跑起sshd,并且把Host的Port 80拿来用(会是SSH):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sshd
spec:
  replicas: 2
  selector:
    matchLabels:
      app: sshd
  template:
    metadata:
      labels:
        app: sshd
    spec:
      containers:
      - name: sshd
        image: ubuntu:18.04
        command: ["/bin/sh", "-c"]
        args:
          - export DEBIAN_FRONTEND=noninteractive;
            sed -i 's/archive.ubuntu.com/us.archive.ubuntu.com/' /etc/apt/sources.list;
            apt update;
            apt install -y iproute2 iputils-ping locales mtr-tiny net-tools openssh-server telnet tzdata wget;
            mkdir /run/sshd; /usr/sbin/sshd;
            sleep 3153600000
        ports:
          - containerPort: 22
            hostPort: 80

复杂一点的版本:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: example-r5
spec:
  replicas: 5
  serviceName: example-r5
  selector:
    matchLabels:
      app: example-r5
  template:
    metadata:
      labels:
        app: example-r5
    spec:
      containers:
      - name: example-r5
        image: ubuntu:18.04
        command: ["/bin/sh", "-c"]
        args:
          - export DEBIAN_FRONTEND=noninteractive;
            sed -i 's/archive.ubuntu.com/us.archive.ubuntu.com/' /etc/apt/sources.list;
            apt update;
            apt install -y iproute2 iputils-ping locales mtr-tiny net-tools tzdata wget;
            sleep 3153600000
        resources:
          requests:
            memory: "15Gi"
      nodeSelector:
        instancetype: r5
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: example-c5
spec:
  replicas: 1
  serviceName: example-c5
  selector:
    matchLabels:
      app: example-c5
  template:
    metadata:
      labels:
        app: example-c5
    spec:
      containers:
      - name: example-c5
        image: ubuntu:18.04
        command: ["/bin/sh", "-c"]
        args:
          - export DEBIAN_FRONTEND=noninteractive;
            sed -i 's/archive.ubuntu.com/us.archive.ubuntu.com/' /etc/apt/sources.list;
            apt update;
            apt install -y iproute2 iputils-ping locales mtr-tiny net-tools tzdata wget;
            sleep 3153600000
        resources:
          requests:
            cpu: "7000m"
      nodeSelector:
        instancetype: c5

相关连结

外部链接