集群：创建日志
开始测试虚拟机连通性—————-
测试[192.168.133.71]联通成功—————-
测试[192.168.133.71, 192.168.133.160]联通成功—————-
测试[192.168.133.71, 192.168.133.160, 192.168.133.174]联通成功—————-
2022-06-01 10:24:23 ^[[36mDEBUG^[[0m generate custom cluster files in /root/kubetelecom/data/clusters/3301001654050243721921
2022-06-01 10:24:23 ^[[36mDEBUG^[[0m cluster 3301001654050243721921: files successfully created.
ansible-playbook -i /root/kubetelecom/data/clusters/3301001654050243721921/hosts -e @clusters/3301001654050243721921/config.yml ../ansible/playbooks/basic.yml
2022-06-01 10:24:23 ^[[32mINFO^[[0m 集群:3301001654050243721921 部署配置节点基础环境开始……
PLAY [cluster_node] ************************************************************
TASK [Gathering Facts] *********************************************************
ok: [192.168.133.160]
ok: [192.168.133.174]
ok: [192.168.133.71]
TASK [prepare : 删除centos/redhat默认安装] *******************************************
changed: [192.168.133.160] => (item=firewalld)
changed: [192.168.133.160] => (item=python-firewall)
changed: [192.168.133.160] => (item=firewalld-filesystem)
changed: [192.168.133.174] => (item=firewalld)
changed: [192.168.133.71] => (item=firewalld)
changed: [192.168.133.174] => (item=python-firewall)
changed: [192.168.133.71] => (item=python-firewall)
changed: [192.168.133.174] => (item=firewalld-filesystem)
changed: [192.168.133.71] => (item=firewalld-filesystem)
TASK [prepare : 准备离线安装包目录] *****************************************************
TASK [prepare : 分发 basic_centos7 离线包] ******************************************
changed: [192.168.133.71]
changed: [192.168.133.174]
TASK [prepare : 安装 basic_centos7 离线包] ******************************************
changed: [192.168.133.160]
TASK [prepare : 临时关闭 selinux] **************************************************
TASK [prepare : 永久关闭 selinux] **************************************************
TASK [prepare : 禁止rsyslog获取journald日志1] ****************************************
TASK [prepare : 禁止rsyslog获取journald日志2] ****************************************
TASK [prepare : 重启rsyslog服务] ***************************************************
TASK [prepare : 禁用系统 swap] *****************************************************
TASK [prepare : 删除fstab swap 相关配置] *********************************************
TASK [prepare : 转换内核版本为浮点数] ****************************************************
TASK [prepare : 设置 nf_conntrack_ipv4 模块名] **************************************
TASK [prepare : 加载内核模块] ********************************************************
changed: [192.168.133.174] => (item=br_netfilter)
changed: [192.168.133.71] => (item=br_netfilter)
changed: [192.168.133.160] => (item=br_netfilter)
changed: [192.168.133.160] => (item=ip_vs)
changed: [192.168.133.160] => (item=ip_vs_rr)
changed: [192.168.133.174] => (item=ip_vs)
changed: [192.168.133.71] => (item=ip_vs)
changed: [192.168.133.160] => (item=ip_vs_wrr)
changed: [192.168.133.160] => (item=ip_vs_sh)
changed: [192.168.133.174] => (item=ip_vs_rr)
changed: [192.168.133.71] => (item=ip_vs_rr)
changed: [192.168.133.160] => (item=nf_conntrack_ipv4)
changed: [192.168.133.174] => (item=ip_vs_wrr)
changed: [192.168.133.71] => (item=ip_vs_wrr)
changed: [192.168.133.174] => (item=ip_vs_sh)
changed: [192.168.133.71] => (item=ip_vs_sh)
changed: [192.168.133.174] => (item=nf_conntrack_ipv4)
changed: [192.168.133.71] => (item=nf_conntrack_ipv4)
TASK [prepare : 启用systemd自动加载模块服务] *********************************************
ok: [192.168.133.160]
ok: [192.168.133.174]
ok: [192.168.133.71]
TASK [prepare : 增加内核模块开机加载配置] **************************************************
TASK [prepare : 设置系统参数] ********************************************************
TASK [prepare : 生效系统参数] ********************************************************
TASK [prepare : 创建 systemd 配置目录] ***********************************************
TASK [prepare : 设置系统 ulimits] **************************************************
TASK [prepare : 把SCTP列入内核模块黑名单] ************************************************
TASK [prepare some dirs] *******************************************************
changed: [192.168.133.160] => (item=/opt/kubetelecom/bin)
changed: [192.168.133.71] => (item=/opt/kubetelecom/bin)
changed: [192.168.133.174] => (item=/opt/kubetelecom/bin)
changed: [192.168.133.160] => (item=/root/.kube)
changed: [192.168.133.174] => (item=/root/.kube)
changed: [192.168.133.71] => (item=/root/.kube)
TASK [获取是否已经安装docker] **********************************************************
TASK [准备docker相关目录] ************************************************************
ok: [192.168.133.160] => (item=/opt/kubetelecom/bin)
ok: [192.168.133.71] => (item=/opt/kubetelecom/bin)
ok: [192.168.133.174] => (item=/opt/kubetelecom/bin)
changed: [192.168.133.160] => (item=/etc/docker)
changed: [192.168.133.71] => (item=/etc/docker)
changed: [192.168.133.174] => (item=/etc/docker)
ok: [192.168.133.160] => (item=/etc/bash_completion.d)
ok: [192.168.133.71] => (item=/etc/bash_completion.d)
ok: [192.168.133.174] => (item=/etc/bash_completion.d)
TASK [下载 docker 二进制文件] *********************************************************
TASK [解压安装 docker] *************************************************************
TASK [docker命令自动补全] ************************************************************
TASK [docker国内镜像加速] ************************************************************
TASK [docker : flush-iptables] *************************************************
TASK [创建docker的systemd unit文件] *************************************************
TASK [开机启用docker 服务] ***********************************************************
TASK [开启docker 服务] *************************************************************
TASK [轮询等待docker服务运行] **********************************************************
changed: [192.168.133.174]
changed: [192.168.133.71]
changed: [192.168.133.160]
TASK [配置 docker 命令软链接] *********************************************************
ok: [192.168.133.160]
ok: [192.168.133.71]
ok: [192.168.133.174]
TASK [docker : 下载采集组件依赖镜像] *****************************************************
changed: [192.168.133.160]
changed: [192.168.133.174]
changed: [192.168.133.71]
TASK [docker : 加载采集组件依赖镜像] *****************************************************
changed: [192.168.133.160]
changed: [192.168.133.174]
TASK [docker : 向 ‘/etc/hosts’ 中添加 HARBOR DOMAIN] *******************************
TASK [docker : 向 cloud-init hosts template (redhat) 添加 HARBOR DOMAIN] **********
TASK [docker : 配置主机登录 HARBOR 镜像仓库] *********************************************
changed: [192.168.133.160]
changed: [192.168.133.71]
changed: [192.168.133.174]
192.168.133.160 : ok=40 changed=33 unreachable=0 failed=0 skipped=78 rescued=0 ignored=0
PLAY RECAP *********************************************************************
192.168.133.174 : ok=40 changed=33 unreachable=0 failed=0 skipped=78 rescued=0 ignored=0
192.168.133.71 : ok=40 changed=33 unreachable=0 failed=0 skipped=78 rescued=0 ignored=0
完成集群注册
完成集群节点注册指令获取
完成集群关联信息及集群节点注册指令入库
192.168.133.71 ansible_ssh_user=root ansible_ssh_pass=”f(ttcBxi”
2022-06-01 10:32:37 ^[[32mINFO^[[0m start to reg a work node:192.168.133.71 into cluster:3301001654050243721921
PLAY [192.168.133.71] **********************************************************
TASK [Gathering Facts] *********************************************************
ok: [192.168.133.71]
[WARNING]: Consider using ‘become’, ‘become_method’, and ‘become_user’ rather
TASK [执行主机注册命令] ****************************************************************
changed: [192.168.133.71]
than running sudo
192.168.133.71 : ok=2 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
PLAY RECAP *********************************************************************
192.168.133.160 ansible_ssh_user=root ansible_ssh_pass=”f(ttcBxi”
2022-06-01 10:36:00 ^[[32mINFO^[[0m start to reg a work node:192.168.133.160 into cluster:3301001654050243721921
PLAY [192.168.133.160] *********************************************************
TASK [Gathering Facts] *********************************************************
ok: [192.168.133.160]
TASK [执行主机注册命令] ****************************************************************
[WARNING]: Consider using ‘become’, ‘become_method’, and ‘become_user’ rather
than running sudo
changed: [192.168.133.160]
192.168.133.160 : ok=2 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
PLAY RECAP *********************************************************************
192.168.133.174 ansible_ssh_user=root ansible_ssh_pass=”f(ttcBxi”
2022-06-01 10:37:47 ^[[32mINFO^[[0m start to reg a work node:192.168.133.174 into cluster:3301001654050243721921
PLAY [192.168.133.174] *********************************************************
TASK [Gathering Facts] *********************************************************
ok: [192.168.133.174]
Pre-pulling kubernetes images
Pre-pulling kubernetes images
Pre-pulling kubernetes images
TASK [执行主机注册命令] ****************************************************************
[WARNING]: Consider using ‘become’, ‘become_method’, and ‘become_user’ rather
changed: [192.168.133.174]
than running sudo
PLAY RECAP *********************************************************************
192.168.133.174 : ok=2 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
集群部署指令下达完毕，完成集群节点注册, 集群组建中。。。
Pre-pulling kubernetes images
Pre-pulling kubernetes images
Pre-pulling kubernetes images
Pre-pulling kubernetes images
Pre-pulling kubernetes images
Pre-pulling kubernetes images
Pre-pulling kubernetes images
Pre-pulling kubernetes images
Pre-pulling kubernetes images
Pre-pulling kubernetes images
Pre-pulling kubernetes images
Pre-pulling kubernetes images
Pre-pulling kubernetes images
Pre-pulling kubernetes images
Pre-pulling kubernetes images
Pre-pulling kubernetes images
[etcd] Building up etcd plane..
[etcd] Building up etcd plane..
Waiting for [etcd-fix-perm] container to exit on host [192.168.133.71]
[remove/etcd-fix-perm] Successfully removed container on host [192.168.133.71]
[remove/etcd-fix-perm] Successfully removed container on host [192.168.133.71]
[remove/etcd-fix-perm] Successfully removed container on host [192.168.133.71]
[remove/etcd-fix-perm] Successfully removed container on host [192.168.133.71]
[etcd] Successfully started [etcd] container on host [192.168.133.71]
[etcd] Successfully started [etcd] container on host [192.168.133.71]
[etcd] Successfully started [rke-log-linker] container on host [192.168.133.71]
[controlplane] Building up Controller Plane..
[healthcheck] Start Healthcheck on service [kube-apiserver] on host [192.168.133.71]
[healthcheck] Start Healthcheck on service [kube-apiserver] on host [192.168.133.71]
[healthcheck] Start Healthcheck on service [kube-apiserver] on host [192.168.133.71]
[healthcheck] Start Healthcheck on service [kube-apiserver] on host [192.168.133.71]
[healthcheck] service [kube-apiserver] on host [192.168.133.71] is healthy
[healthcheck] Start Healthcheck on service [kube-controller-manager] on host [192.168.133.71]
[healthcheck] service [kube-controller-manager] on host [192.168.133.71] is healthy
[healthcheck] service [kube-controller-manager] on host [192.168.133.71] is healthy
[healthcheck] Start Healthcheck on service [kube-scheduler] on host [192.168.133.71]
[healthcheck] service [kube-scheduler] on host [192.168.133.71] is healthy
[sidekick] Sidekick container already created on host [192.168.133.71]
[healthcheck] Start Healthcheck on service [kubelet] on host [192.168.133.71]
[healthcheck] Start Healthcheck on service [kubelet] on host [192.168.133.71]
[healthcheck] service [kubelet] on host [192.168.133.71] is healthy
[worker] Successfully started [rke-log-linker] container on host [192.168.133.71]
[healthcheck] Start Healthcheck on service [kube-proxy] on host [192.168.133.71]
[healthcheck] service [kube-proxy] on host [192.168.133.71] is healthy
[worker] Successfully started Worker Plane..
[addons] Executing deploy job rke-network-plugin
[addons] Executing deploy job rke-network-plugin
[addons] Executing deploy job rke-network-plugin
[addons] Executing deploy job rke-network-plugin
[addons] Executing deploy job rke-coredns-addon
[addons] Executing deploy job rke-coredns-addon
[addons] Executing deploy job rke-coredns-addon
[addons] Executing deploy job rke-coredns-addon
[addons] Executing deploy job rke-coredns-addon
[addons] Executing deploy job rke-coredns-addon
[addons] Executing deploy job rke-metrics-addon
[addons] Executing deploy job rke-metrics-addon
[addons] Executing deploy job rke-metrics-addon
[addons] Executing deploy job rke-metrics-addon
[addons] Executing deploy job rke-metrics-addon
[addons] Executing deploy job rke-ingress-controller
[addons] Executing deploy job rke-ingress-controller
[addons] Executing deploy job rke-ingress-controller
[addons] Executing deploy job rke-ingress-controller
Finished building Kubernetes cluster successfully
Finished building Kubernetes cluster successfully
Waiting for API to be available
Waiting for API to be available
Waiting for API to be available
Waiting for API to be available
Waiting for API to be available
Waiting for API to be available
集群已经创建成功，准备进行后续操作
开启集群监控组件prometheus——————-
正在生成集群token文件————————
集群监控prometheus开启成功，正在进行下一步操作
生成集群级别token———————-
集群token文件生成成功！
集群token文件生成成功，正在进行下一步操作……
在master节点安装、配置kubectl及创建集群级别token，并使用模板文件将promethues的svc端口以NodePort方式暴露
2022-06-01 10:53:14 ^[[32mINFO^[[0m 开始对的 3301001654050243721921 集群的master节点 192.168.133.71 进行相关配置
PLAY [192.168.133.71] **********************************************************
TASK [Gathering Facts] *********************************************************
ok: [192.168.133.71]
TASK [kube-master : 准备kubectl客户端所需目录] ******************************************
ok: [192.168.133.71] => (item=/opt/kubetelecom/bin)
ok: [192.168.133.71] => (item=~/.kube)
changed: [192.168.133.71] => (item=/opt/kubetelecom/template)
TASK [kube-master : 下载 kubectl 二进制文件] ******************************************
TASK [kube-master : 解压安装 kubectl] **********************************************
TASK [kube-master : 添加 kubectl 命令自动补全] *****************************************
TASK [kube-master : 安装kubeconfig配置文件 ~/.kube/config] ***************************
TASK [kube-master : 创建集群级别token] ***********************************************
TASK [kube-master : 安装prometheus模板文件] ******************************************
TASK [kube-master : 修改promethues的service,将端口以NodePort方式暴露] *********************
changed: [192.168.133.251]
PLAY RECAP *********************************************************************
192.168.133.251 : ok=9 changed=8 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
执行成功！
获取集群级别token
eyJhbGciOiJSUzI1NiIsImtpZCI6IlRhYkh5X1JmYW9HUG5xbENPLVRfNmRhNmtJcnRTZXBpdWVyQTZmVVJZbzQifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJrOHMtYWRtaW4tdG9rZW4tcDJ2dm4iLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiazhzLWFkbWluIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiODhjZTcwN2ItY2RkZi00MmIyLTk1ZjgtZDUyNTE4NDNhNjBjIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmUtc3lzdGVtOms4cy1hZG1pbiJ9.3JAIv4mzn4rgyb0NtUA–j2C9wtYGuS4JHYzPjCIDE7Px4A3Fte9CWMnK5yYcQsK8FWK3q4rEmH67P-6lczAQzqksaX-S2S93-BkpIqCrzgmgGp3QD3xp1KWJ4BPJJ7FgdmWOboIiT8u2tbtMgnbGRUB_YUerpxLqP51UVRaT_SzsIlvV2wj5AihUsCYt4qr7sECgaqDqsguMVQmWIP_I5nsDlnF67coDbi00Gu6mwGwHQFFA3Lioq54ae1BYGUYB9FnPuz6ccIfdeg8SnXbz4o4Lu3O3JZduIfy6wiNsXj3FHOWLxgYMFP9aypxmr-CHtDBsfKJGhHlX8DNiBqRlg
集群级别token生成成功!
集群初始化成功——————–部署完成！