大道至简

学必求其心得，业必贵于专精。

0%

K8S集群创建（二进制）

发表于 更新于 分类于 
1
2
3
4
mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.backup
wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
yum clean all
yum makecache

4.3 系统初始化

4.3.0 先他妈yum update一下，别问为什么，问就是坑

4.3.1 关闭防火墙（所有节点）

• 关闭防火墙
systemctl stop firewalld
• 禁止防火墙开机自启
systemctl disable firewalld

4.3.2 关闭selinux（所有节点）

1
2
3
4
5
6
• 永久关闭
# 永久
sed -i 's/enforcing/disabled/' /etc/selinux/config
• 临时关闭（不建议使用这条命令）
# 临时
setenforce 0

4.3.3 关闭swap分区（所有节点）

1
2
3
4
5
6
reboot
	• 永久关闭swap分区
	# 永久
	sed -ri 's/.*swap.*/#&/' /etc/fstab
	• 临时关闭swap分区（不建议使用这条命令）
	swapoff -a

4.3.4 配置主机名（所有节点）

1
2
3
4
5
6
7
8
• 设置主机名
hostnamectl set-hostname <hostname>
• 设置master的主机名
hostnamectl set-hostname k8s-master
• 设置node1的主机名
hostnamectl set-hostname k8s-node1
• 设置node2的主机名
hostnamectl set-hostname k8s-node2

4.3.5 配置hosts文件（所有节点）

1
2
3
4
5
6
	• 在每个节点添加hosts
cat >> /etc/hosts << EOF
172.20.3.32 k8s-master
172.20.3.34 k8s-node1
172.20.3.46 k8s-node2
EOF

4.3.6 将桥接的IPv4流量传递到iptables的链（所有节点）

加载br_netfilter模块
1
modprobe br_netfilter
查看是否加载
1
lsmod | grep br_netfilter
生效
1
2
3
4
5
6
7
sysctl --system
	• 在每个节点添加h如下的命令
	cat > /etc/sysctl.d/k8s.conf << EOF
	net.bridge.bridge-nf-call-ip6tables = 1
	net.bridge.bridge-nf-call-iptables = 1
	net.ipv4.ip_forward = 1
	EOF

4.3.7 配置时间同步（所有节点）

1
2
3
ntpdate time.windows.com
• 在每个节点添加h如下的命令
yum install ntpdate -y

4.3.8 开启ipvs（所有节点）

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
• 在每个节点安装ipset和ipvsadm
yum -y install ipset ipvsadm
• 在所有节点执行如下脚本
cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
EOF
• 授权、运行、检查是否加载
chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4
• 检查是否加载
lsmod | grep -e ipvs -e nf_conntrack_ipv4

4.4 所有节点安装Docker/kubeadm/kubelet/kubectl

4.4.1 概述

• k8s默认CRI（容器运行时）为Docker，因此需要先安装Docker。

4.4.2 安装Docker

1
2
3
4
5
6
7
8
9
10
11
yum -y install docker-ce-18.06.3.ce-3.el7
systemctl enable docker && systemctl start docker
docker version
sudo tee /etc/docker/daemon.json <<-'EOF'
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "registry-mirrors": ["https://b9pmyelo.mirror.aliyuncs.com"]
}
EOF
sudo systemctl daemon-reload
sudo systemctl restart docker
• 安装Docker

1
wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo

• 设置Docker镜像加速器，同时指定docker的cgroupdriver为systemd，这个systemd很重要，docker和k8s的cgroup driver必须一致，否则无法安装

1
sudo mkdir -p /etc/docker

4.4.3 添加阿里云的YUM软件源 (如果k8s的源无法外网访问，请移步到：当服务器无法连接外网时如何处理 中的下载k8s源 步骤)

><< EOF
1
2
3
4
5
6
7
8
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

4.4.4 安装kubeadm、kubelet和kubectl

1
2
3
4
5
6
7
修改KUBELET_EXTRA_ARGS="--cgroup-driver=systemd"
	• 由于版本更新频繁，这里指定版本号部署
	yum install -y kubelet-1.18.0 kubeadm-1.18.0 kubectl-1.18.0
	• 为了实现Docker使用的cgroup driver和kubelet使用的cgroup drver一致，建议修改"/etc/sysconfig/kubelet"文件的内容：
	vim /etc/sysconfig/kubelet
	• 设置为开机自启动即可，由于没有生成配置文件，集群初始化后自动启动：
	systemctl enable kubelet

4.5 部署k8s的Master节点

• 部署k8s的Master节点（172.20.80.254）：
# 由于默认拉取镜像地址k8s.gcr.io国内无法访问，这里需要指定阿里云镜像仓库地址

1
2
3
4
5
6
kubeadm init \
  --apiserver-advertise-address=172.20.3.32 \
  --image-repository registry.aliyuncs.com/google_containers \
  --kubernetes-version v1.18.0 \
  --service-cidr=10.96.0.0/12 \
  --pod-network-cidr=10.244.0.0/16

• 请将图中 kubeadm join.... 这两行话拷贝下来，后面node加入集群时要用

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
[init] Using Kubernetes version: v1.18.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 172.20.3.32]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-master localhost] and IPs [172.20.3.32 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-master localhost] and IPs [172.20.3.32 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
W0102 17:30:13.051267    9870 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[control-plane] Creating static Pod manifest for "kube-scheduler"
W0102 17:30:13.052340    9870 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 19.002034 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.18" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node k8s-master as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node k8s-master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: ntxv5o.hxhljd63gskip2y9
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:
1
2
kubeadm join 172.20.3.32:6443 --token ntxv5o.hxhljd63gskip2y9 \
    --discovery-token-ca-cert-hash sha256:e8f3f469eff4931066679909ce7916c196e25b78a86a379f9303ecc5951a2751

4.6 番外篇之删除节点（非必须勿用）

注意：以下操作都是在master下操作。

1
2
3
4
5
6
• 先将节点设置为维护模式(k8s-node1是节点名称)
kubectl drain k8s-node1 --delete-local-data --force --ignore-daemonsets node/k8s-node1
• 删除节点
kubectl delete node k8s-node1
• 确认是否已删除
kubectl get nodes

4.7 部署CNI网络插件

• 根据提示，在Master节点使用kubectl工具查看节点状态：

1
kubectl get nodes

• 在Master节点部署CNI网络插件(可能会失败，如果失败，请下载到本地，然后安装，如果wget也不行，请将网址拷贝到你自己的电脑上下载，下载完毕后将yml文件再传到服务器上，可以使用scp命令（见下图中的scp命令使用）
wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

• 查看部署CNI网络插件进度：

1
kubectl get pods -A

• 再次在Master节点使用kubectl工具查看节点状态：

1
kubectl get nodes

• 查看集群健康状态：

1
2
kubectl get cs
kubectl cluster-info

• node节点使用kubectl：

1
2
3
mkdir -p $HOME/.kube
 scp  root@172.20.3.32:/etc/kubernetes/admin.conf  /root/.kube/config
 sudo chown $(id -u):$(id -g) $HOME/.kube/config

4.8配置nfs挂载环境

1).安装NFS服务

1
2
3
#所有机器安装 
yum install -y nfs-utils
yum install -y rpcbind

2).设置共享文件夹

1
2
mkdir -p /nfs/data/
chmod 755 /nfs/data

3).配置文件修改（master）

1
2
3
4
5
vim /etc/exports加入以下内容：
/nfs/ *(async,insecure,no_root_squash,no_subtree_check,rw)
/nfs/data/ *(async,insecure,no_root_squash,no_subtree_check,rw)
使配置生效：
exportfs -r

4).启动服务

1
2
3
4
systemctl start rpcbind 
systemctl start nfs
systemctl enable rpcbind                # 设置开机启动
systemctl enable nfs-server.service     # 设置开机启动

动态持久化
5).把docker pull registry.cn-hangzhou.aliyuncs.com/open-ali/nfs-client-provisioner:latest镜像加载到各个节点上
6).kubectl apply -f cluster-admin.rbac.yaml -n kube-system创建权限（以下为cluster-admin.rbac.yaml内容）

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
cat>cluster-admin.rbac.yaml<<-EOF
---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    k8s-app: dcm-rbac
  name: k8s-admin
  namespace: kube-system

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: k8s-admin-crb
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: k8s-admin
  namespace: kube-system
  EOF

7).kubectl apply -f nfs-client-provisioner.yaml -n kube-system创建nfs-client-provisioner（以下为nfs-client-provisioner.yaml内容）

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
cat>nfs-client-provisioner.yaml<<-EOF
kind: Deployment
apiVersion: apps/v1
metadata:
  name: nfs-client-provisioner
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nfs-client-provisioner
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: nfs-client-provisioner
    spec:
      serviceAccountName: k8s-admin
      containers:
        - name: nfs-client-provisioner
          image: registry.cn-hangzhou.aliyuncs.com/open-ali/nfs-client-provisioner:latest
          imagePullPolicy: Never
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME
              value: fuseim.pri/ifs
            - name: NFS_SERVER
              value: 172.20.3.32
            - name: NFS_PATH
              value: /nfs/data
      volumes:
        - name: nfs-client-root
          nfs:
            server: 172.20.3.32
            path: /nfs/data
EOF

8).kubectl apply -f nfs-client-class.yaml 创建storageclass（以下为nfs-client-class.yaml内容）

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
cat>nfs-client-class.yaml<<-EOF
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: managed-nfs-storage
provisioner: fuseim.pri/ifs
EOF
9).kubectl create -f nfs-client-pvc.yaml创建pvc使用对应nfs-client-provisioner验证nfs是否配置成功
cat>nfs-client-pvc.yaml<<-EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nfs-client-pvc
  namespace: kube-system
spec:
  accessModes:
  - ReadWriteMany
    storageClassName: managed-nfs-storage
    resources:
    requests:
      storage: 100Mi
    EOF

9）kubectl get pvc nfs-client-pvc -n kube-system 查看pvc 是否自动关联上pv
#然后，我们进入到NFS的export目录，可以看到对应该volumename的目录已经创建出来了。其中volume的名字是namespace，PVCname以及uuid的组合：
#注意，出现pvc在pending的原因可能为nfs-client-provisionerpod出现了问题，删除重建的时候会出现镜像问题

10）kubectl delete pvc nfs-client-pvc -n kube-system 验证结束后删除pvc