作者声明：本篇文章系本人依照真实部署过程原创，未经许可，谢绝转载。
1.概述
今天我们要搭建一个kubernetes环境，主要使用kubeadm工具来搭建。
2.kubeadm简介
kubeadm是Kubernetes官方提供的用于快速安装Kubernetes集群的工具。
点击查看Kubernetes发布的版本
3.操作系统准备
3.1. 硬件环境
我们使用VM虚拟机来安装一个master节点两个node节点。
虚拟机安装CentOS Linux7.6 配置如下：
内存：1G （测试1G没问题）
CPU：2个 （必须大于等于2）
磁盘：20G
3.2. 系统环境
1 | # cat /etc/redhat-release |
4.安装步骤
4.1. 网络同步时间确保集群内时钟同步
1 | # systemctl start chronyd.service |
4.2. 主机名称解析
1 | # echo '127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4' > /etc/hosts |
4.3. 关闭iptables
1 | # systemctl status iptables |
4.4. 关闭防火墙：
1 | # systemctl status firewalld |
4.5. 关闭selinux
1 | # getenforce |
编辑 /etc/selinux/config 文件
设置 SELINUX=disabled
4.6. 关闭 swap(也可以不关闭，需要在后续改配置文件)
1 | # free -m |
编辑 /etc/fstab 文件 下注释掉所有swap
4.7. 检查net.bridge设置
1 | # sysctl -a | grep bridge |
检查
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
是否为1
如果不是1
修改方式：修改 /etc/sysctl.conf 文件
或者新建 /etc/sysctl.d/k8s.conf 增加
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
让文件生效
sysctl -p /etc/sysctl.d/k8s.conf
4.8. yum仓库准备
1 | CentOS-Base的yum仓库 |
4.8. 安装Docker
1 | 安装docker（默认安装最新版本，也可以安装指定版本） |
4.9. 安装kubeadm和kubelet和kubectl
1 | 安装kubelet kubeadm kubectl（默认安装最新版本，目前版本是1.13.4） |
4.10. 没有关闭swap需要修改参数
swap启动的时候 设置防止报错
修改 /etc/sysconfig/kubelet 文件
KUBELET_EXTRA_ARGS=”–fail-swap-on=false”
4.11. 安装网络插件（flannel或calico）
这里我选择安装flannel
flannel 默认的设置是:10.244.0.0/16
flannel安装方法1
2执行以下语句
# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
以上是安装好了一个虚拟机，我们需要把虚拟机复制2份，总共3个虚拟机。
设置3个虚拟机的主机名和ip分别为(可根据实际自行设定)
| 主机名 | ip |
|---|---|
| master | 192.168.110.140 |
| node01 | 192.168.110.141 |
| node02 | 192.168.110.142 |
5.镜像准备
查看master 节点上需要的镜像文件1
2
3
4
5
6
7
8# kubeadm config images list
k8s.gcr.io/kube-apiserver:v1.13.4
k8s.gcr.io/kube-controller-manager:v1.13.4
k8s.gcr.io/kube-scheduler:v1.13.4
k8s.gcr.io/kube-proxy:v1.13.4
k8s.gcr.io/pause:3.1
k8s.gcr.io/etcd:3.2.24
k8s.gcr.io/coredns:1.2.6
为了解决国内普遍访问不到k8s.gcr.io的问题，我们从mirrorgooglecontainers下载image，然后打个tag来绕过网络限制：
1 | docker pull mirrorgooglecontainers/kube-apiserver:v1.13.4 |
6. kubernetes master初始化
1 | 查看kubeadm版本 |
初始化信息1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68[init] Using Kubernetes version: v1.13.4
[preflight] Running pre-flight checks
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 18.09.2. Latest validated version: 18.06
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [master.k8s.com localhost] and IPs [192.168.110.140 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [master.k8s.com localhost] and IPs [192.168.110.140 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [master.k8s.com kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.110.140]
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 24.503251 seconds
[uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.13" in namespace kube-system with the configuration for the kubelets in the cluster
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "master.k8s.com" as an annotation
[mark-control-plane] Marking the node master.k8s.com as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node master.k8s.com as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: 0me0lv.xn1vhr6ub80qexyu
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstraptoken] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstraptoken] creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes master has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of machines by running the following on each node
as root:
kubeadm join 192.168.110.140:6443 --token 0me0lv.xn1vhr6ub80qexyu --discovery-token-ca-cert-hash sha256:700e4cb4ce01c0be5a40e8278ebf531847090b53d53b115ca3031bfb25532b0b
根据提示在master节点下执行以下命令1
2
3# mkdir -p $HOME/.kube
# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
# sudo chown $(id -u):$(id -g) $HOME/.kube/config
7. kubernetes node01加入集群
node01执行以下命令加入集群1
#kubeadm join 192.168.110.140:6443 --token 0me0lv.xn1vhr6ub80qexyu --discovery-token-ca-cert-hash sha256:700e4cb4ce01c0be5a40e8278ebf531847090b53d53b115ca3031bfb25532b0b --ignore-preflight-errors=Swap
执行命令后提示信息1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23[preflight] Running pre-flight checks
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 18.09.2. Latest validated version: 18.06
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
[discovery] Trying to connect to API Server "192.168.110.140:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://192.168.110.140:6443"
[discovery] Requesting info from "https://192.168.110.140:6443" again to validate TLS against the pinned public key
[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "192.168.110.140:6443"
[discovery] Successfully established connection with API Server "192.168.110.140:6443"
[join] Reading configuration from the cluster...
[join] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.13" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[tlsbootstrap] Waiting for the kubelet to perform the TLS Bootstrap...
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "node01.k8s.com" as an annotation
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the master to see this node join the cluster.
8. kubernetes node02加入集群
node02执行以下命令加入集群1
#kubeadm join 192.168.110.140:6443 --token 0me0lv.xn1vhr6ub80qexyu --discovery-token-ca-cert-hash sha256:700e4cb4ce01c0be5a40e8278ebf531847090b53d53b115ca3031bfb25532b0b --ignore-preflight-errors=Swap
以上是配置好了kubernetes 一个master节点和两个node节点
9. 验证集群状态
分别在3个节点上执行命令启动kubelet1
# system start kubelet
在master节点上执行命令查看集群节点状态1
2
3
4
5
6# kubectl get nodes
得到以下信息
NAME STATUS ROLES AGE VERSION
master.k8s.com Ready master 18m v1.13.4
node01.k8s.com Ready <none> 11m v1.13.4
node02.k8s.com Ready <none> 9m39s v1.13.4
在master节点上执行命令查看系统pod运行状态1
2
3
4
5
6
7
8
9
10
11
12
13
14
15# kubectl get pods -n kube-system
得到以下信息
NAME READY STATUS RESTARTS AGE
coredns-86c58d9df4-467w8 1/1 Running 0 23m
coredns-86c58d9df4-qrjx2 1/1 Running 0 23m
etcd-master.k8s.com 1/1 Running 0 22m
kube-apiserver-master.k8s.com 1/1 Running 0 22m
kube-controller-manager-master.k8s.com 1/1 Running 0 22m
kube-flannel-ds-amd64-4hx5q 1/1 Running 0 5m1s
kube-flannel-ds-amd64-nhbqs 1/1 Running 0 5m1s
kube-flannel-ds-amd64-pf5dg 1/1 Running 0 5m1s
kube-proxy-g85lj 1/1 Running 0 14m
kube-proxy-kb9l7 1/1 Running 0 23m
kube-proxy-nczpr 1/1 Running 0 16m
kube-scheduler-master.k8s.com 1/1 Running 0 22m
9. 错误排查
在master节点上执行命令得到如下状态1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master.k8s.com NotReady master 15m v1.13.4
node01.k8s.com NotReady <none> 8m6s v1.13.4
node02.k8s.com NotReady <none> 6m31s v1.13.4
# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-86c58d9df4-467w8 0/1 Pending 0 17m
coredns-86c58d9df4-qrjx2 0/1 Pending 0 17m
etcd-master.k8s.com 1/1 Running 0 16m
kube-apiserver-master.k8s.com 1/1 Running 0 16m
kube-controller-manager-master.k8s.com 1/1 Running 0 16m
kube-proxy-g85lj 1/1 Running 0 8m19s
kube-proxy-kb9l7 1/1 Running 0 17m
kube-proxy-nczpr 1/1 Running 0 9m54s
kube-scheduler-master.k8s.com 1/1 Running 0 16m
以上说明网络没有配置好，需要我们安装网络，此时参考 4.11. 安装网络插件（flannel或calico）
执行命令安装flannel1
# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
执行命令后再次验证节点和pod状态
以上完整安装kubernetes 13.3.4 整个过程