Kubernetes

来自Gea-Suan Lin's Wiki
跳到导航 跳到搜索
The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.

Kubernetes是一套由Google所發展出來的佈署系統。

環境

這邊是以Ubuntu 18.04為基礎,在AWS上使用一台c5.2xlarge與五台r5.large測試(單master版本),或是再加上三台t3.small並且使用ELB(多master版本)。

安裝

先安裝Docker,然後安裝Kubernetes的套件:

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -; echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list; sudo apt update; sudo apt install -y kubelet kubeadm kubectl

單master版本設定

這邊使用Calico當作網路層:

sudo kubeadm init --pod-network-cidr=192.168.0.0/16

把上面執行結果輸出的命令拿到別台用sudo跑,像是這樣的指令:

sudo kubeadm join a.b.c.d:6443 --token xxxxxx.xxxxxxxxxxxxxxxx --discovery-token-ca-cert-hash sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

接著回到當初跑kubeadm init的機器上,把設定檔放到自己目錄下:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

接下來啟用Calico設定:

kubectl apply -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml
kubectl apply -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml

設定好之後不會馬上通,可以用kubectl get nodes --watch可以看到機器會因為retry從NotReady變成Ready

多master版本設定

大多數的設定與單master版本的設定相同,請參考前面對單master版本的說明。

在多master版本中需要三台主機確保HA,以及一組load balancer提供讓程式可以操作Kubernetes(預設為port 6443)。

由於我們的測試是在AWS上面,所以這邊使用了ELB(Classic版本,Internal,僅使用TCP Proxy)來當作load balancer(即下面設定中的internal-test-gslin-k8s-apiserver-XXXXXXXXXX.us-east-1.elb.amazonaws.com:6443)。

我們需要透過設定檔設定對應的controlPaneEndpointpodSubnet(因為--config--pod-network-cidr不能同時使用):

apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
kubernetesVersion: stable
apiServer:
  certSANs:
  - "internal-test-gslin-k8s-apiserver-XXXXXXXXXX.us-east-1.elb.amazonaws.com"
controlPlaneEndpoint: "internal-test-gslin-k8s-apiserver-XXXXXXXXXX.us-east-1.elb.amazonaws.com:6443"
networking:
  podSubnet: 192.168.0.0/16

然後再透過設定檔初始化cluster的第一台主機:

sudo kubeadm init --config=kubeadm-config.yaml

依照單master設定複製完設定檔:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

再來是設定CNI,在官方文件上是建議使用Weave:

kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

接下來是複製檔案到其他機器上,這邊是讓tmuxscreen的使用者比較方便,可以直接剪下貼上:

cd /etc/kubernetes/
sudo tar -zc -f - pki/{ca.*,sa.*,front-proxy-ca.*,etcd/ca.*} admin.conf | uuencode a.tar.gz

然後到目錄下產生a.tar.gz再解開:

cd /etc/kubernetes/
sudo uudecode
sudo tar zxvf a.tar.gz

加入時除了本來的指令外,需要加上--experimental-control-plane(只有這三台主機要加,其他的機器不用):

sudo kubeadm join internal-test-gslin-k8s-apiserver-XXXXXXXXXX.us-east-1.elb.amazonaws.com:6443 --token xxxxxx.xxxxxxxxxxxxxxxx --discovery-token-ca-cert-hash sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx --experimental-control-plane

事後增加主機

token預設是24小時有效,在過期後需要重新建立:

sudo kubeadm token create

可以透過這個指令在建立token的同時,產生完整的加入指令:

echo sudo kubeadm join $(kubeadm config view | grep ^controlPlaneEndpoint | awk '{print $2}') --token $(sudo kubeadm token create) --discovery-token-ca-cert-hash sha256:$(openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //')

使用所有主機

一開始的主機(master)不會被分配到需求(因為安全設計),如果你希望讓master也可以加入,可以透過以下的指令讓master分配到需求:

kubectl taint nodes --all node-role.kubernetes.io/master-

標籤

可以針對主機進行標籤,供之後的nodeSelector使用:

kubectl label nodes ip-172-31-1-1 instancetype=c5
kubectl label nodes ip-172-31-1-2 ip-172-31-1-3 ip-172-31-1-4 ip-172-31-1-5 ip-172-31-1-6 instancetype=r5

範例

簡單的版本,跑兩個Pod並且跑起sshd,並且把Host的Port 80拿來用(會是SSH):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sshd
spec:
  replicas: 2
  selector:
    matchLabels:
      app: sshd
  template:
    metadata:
      labels:
        app: sshd
    spec:
      containers:
      - name: sshd
        image: ubuntu:18.04
        command: ["/bin/sh", "-c"]
        args:
          - export DEBIAN_FRONTEND=noninteractive;
            sed -i 's/archive.ubuntu.com/us.archive.ubuntu.com/' /etc/apt/sources.list;
            apt update;
            apt install -y iproute2 iputils-ping locales mtr-tiny net-tools openssh-server telnet tzdata wget;
            mkdir /run/sshd; /usr/sbin/sshd;
            sleep 3153600000
        ports:
          - containerPort: 22
            hostPort: 80

複雜一點的版本:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: example-r5
spec:
  replicas: 5
  serviceName: example-r5
  selector:
    matchLabels:
      app: example-r5
  template:
    metadata:
      labels:
        app: example-r5
    spec:
      containers:
      - name: example-r5
        image: ubuntu:18.04
        command: ["/bin/sh", "-c"]
        args:
          - export DEBIAN_FRONTEND=noninteractive;
            sed -i 's/archive.ubuntu.com/us.archive.ubuntu.com/' /etc/apt/sources.list;
            apt update;
            apt install -y iproute2 iputils-ping locales mtr-tiny net-tools tzdata wget;
            sleep 3153600000
        resources:
          requests:
            memory: "15Gi"
      nodeSelector:
        instancetype: r5
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: example-c5
spec:
  replicas: 1
  serviceName: example-c5
  selector:
    matchLabels:
      app: example-c5
  template:
    metadata:
      labels:
        app: example-c5
    spec:
      containers:
      - name: example-c5
        image: ubuntu:18.04
        command: ["/bin/sh", "-c"]
        args:
          - export DEBIAN_FRONTEND=noninteractive;
            sed -i 's/archive.ubuntu.com/us.archive.ubuntu.com/' /etc/apt/sources.list;
            apt update;
            apt install -y iproute2 iputils-ping locales mtr-tiny net-tools tzdata wget;
            sleep 3153600000
        resources:
          requests:
            cpu: "7000m"
      nodeSelector:
        instancetype: c5

相關連結

外部連結