Kubernetes Master

Kubernetes Master节点部署三个服务:kube-apiserverkube-controller-managerkube-scheduler和一个命令工具kubectl.

Master节点来负责整个集群的管理和控制,其中

kube-apiserver: 服务提供了HTTP Rest接口的关键服务进程,是Kuberneters里所有资源的增删改查等操作的唯一入口,也是集群控制的入口进程.

kube-controller-manager: 服务 是kubernetes里面所有资源对象的自动化控制中心,可以理解为资源对象的”大总管”

kube-scheduler: 服务负责资源调度(pod调度)的进程,相当于公交公司的”调度室”。

安装Kube-apiserver

相关参数介绍

• —logtostderr:启用日志

• —-v:日志等级

• —log-dir:日志目录

• —etcd-servers:etcd集群地址

• —bind-address:监听地址

• —secure-port:https安全端口

• —advertise-address:集群通告地址

• —allow-privileged:启用授权

• —service-cluster-ip-range:Service虚拟IP地址段

• —enable-admission-plugins:准入控制模块

• —authorization-mode:认证授权,启用RBAC授权和节点自管理

• —enable-bootstrap-token-auth:启用TLS bootstrap机制

• —token-auth-file:bootstrap token文件

• —service-node-port-range:Service nodeport类型默认分配端口范围

• —kubelet-client-xxx:apiserver访问kubelet客户端证书

• —tls-xxx-file:apiserver https证书

• 1.20版本后必须加的参数:—service-account-issuer,—service-account-signing-key-file

• —etcd-xxxfile:连接Etcd集群证书

• —audit-log-xxx:审计日志

• 启动聚合层相关配置:—requestheader-client-ca-file,—proxy-client-cert-file,—proxy-client-key-file,—requestheader-allowed-names,—requestheader-extra-headers-prefix,—requestheader-group-headers,—requestheader-username-headers,—enable-aggregator-routing

更多参数介绍:https://kubernetes.io/zh/docs/reference/command-line-tools-reference/kube-apiserver/

创建证书

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
cat > /opt/certs/apiserver-csr.json <<EOF
{
"CN": "kubernetes",
"hosts": [
"127.0.0.1",
"192.168.0.1",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local",
"10.1.1.50",
"10.1.1.100",
"10.1.1.110",
"10.1.1.120",
"10.1.1.130"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "beijing",
"L": "beijing",
"O": "apiserver",
"OU": "kubernetes"
}
]
}
EOF

## 生成证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes apiserver-csr.json |cfssljson -bare apiserver
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
cat > /opt/certs/sa-csr.json <<EOF
{
"CN": "ServiceAccount",
"hosts": [
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "beijing",
"L": "beijing",
"O": "ServiceAccount",
"OU": "kubernetes"
}
]
}
EOF

## 生成证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes sa-csr.json |cfssljson -bare sa

安装kube-apiserver

下载地址: https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.24.md

注:打开链接你会发现里面有很多包,下载一个server包就够了,包含了Master和Worker Node二进制文件。

1
2
3
4
5
6
7
8
mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs} 
tar zxvf kubernetes-server-linux-amd64.tar.gz
cd kubernetes/server/bin
cp kube-apiserver /opt/kubernetes/bin
cp kubectl /usr/bin/
scp 10.1.1.11:/opt/certs/apiserver*.pem /opt/kubernetes/ssl
scp 10.1.1.11:/opt/certs/ca*.pem /opt/kubernetes/ssl
scp 10.1.1.11:/opt/certs/sa*.pem /opt/kubernetes/ssl

配置apiserver文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
cat > /opt/kubernetes/cfg/kube-apiserver.conf <<EOF
KUBE_APISERVER_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--etcd-servers=https://10.1.1.100:2379,https://10.1.1.130:2379,https://10.1.1.120:2379 \\
--bind-address=10.1.1.100 \\
--secure-port=6443 \\
--advertise-address=10.1.1.100 \\
--allow-privileged=true \\
--service-cluster-ip-range=192.168.0.0/16 \\
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \\
--authorization-mode=RBAC,Node \\
--enable-bootstrap-token-auth=true \\
--token-auth-file=/opt/kubernetes/cfg/token.csv \\
--service-node-port-range=30000-32767 \\
--kubelet-client-certificate=/opt/kubernetes/ssl/apiserver.pem \\
--kubelet-client-key=/opt/kubernetes/ssl/apiserver-key.pem \\
--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname \\
--client-ca-file=/opt/kubernetes/ssl/ca.pem \\
--service-account-key-file=/opt/kubernetes/ssl/sa.pem \\
--service-account-issuer=https://kubernetes.default.svc.cluster.local \\
--service-account-signing-key-file=/opt/kubernetes/ssl/sa-key.pem \\
--etcd-cafile=/opt/etcd/ssl/ca.pem \\
--etcd-certfile=/opt/etcd/ssl/etcd.pem \\
--etcd-keyfile=/opt/etcd/ssl/etcd-key.pem \\
--tls-cert-file=/opt/kubernetes/ssl/apiserver.pem \\
--tls-private-key-file=/opt/kubernetes/ssl/apiserver-key.pem \\
--proxy-client-cert-file=/opt/kubernetes/ssl/apiserver.pem \\
--proxy-client-key-file=/opt/kubernetes/ssl/apiserver-key.pem \\
--requestheader-allowed-names=aggregator \\
--requestheader-client-ca-file=/opt/kubernetes/ssl/ca.pem \\
--requestheader-extra-headers-prefix=X-Remote-Extra- \\
--requestheader-group-headers=X-Remote-Group \\
--requestheader-username-headers=X-Remote-User \\
--enable-aggregator-routing=true \\
--audit-log-maxage=30 \\
--audit-log-maxbackup=3 \\
--audit-log-maxsize=100 \\
--audit-log-path=/opt/kubernetes/logs/k8s-audit.log"
EOF

配置kube-apiserver启动文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
cat > /usr/lib/systemd/system/kube-apiserver.service <<EOF
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-apiserver.conf
ExecStart=/opt/kubernetes/bin/kube-apiserver \$KUBE_APISERVER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

创建配置token文件

1
2
3
 cat > /opt/kubernetes/cfg/token.csv <<EOF
bc43e407e311d78b60da186fdd347fc8,kubelet-bootstrap,10001,"system:node-bootstrapper"
EOF

格式:token,用户名,UID,用户组

token也可自行生成替换:

1
head -c 16 /dev/urandom | od -An -t x | tr -d ' '

启动apiserver

1
2
3
4
5
6
systemctl daemon-reload
systemctl enable --now kube-apiserver

## 检查配置是否启动
# netstat -lnpt|grep 6443
tcp 0 0 10.1.1.100:6443 0.0.0.0:* LISTEN 6905/kube-apiserver

常见错误

1
2
# 此处为etcd正常关闭报错,故可忽略。
[transport] transport: loopyWriter.run returning. connection error: desc = "transport is closing"

授权apiserver访问kubelet

应用场景:例如kubectl logs

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
cat > /opt/kubernetes/cfg/apiserver-to-kubelet-rbac.yaml <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: system:kube-apiserver-to-kubelet
rules:
- apiGroups:
- ""
resources:
- nodes/proxy
- nodes/stats
- nodes/log
- nodes/spec
- nodes/metrics
- pods/log
verbs:
- "*"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: system:kube-apiserver
namespace: ""
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:kube-apiserver-to-kubelet
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: kubernetes
EOF

kubectl apply -f /opt/kubernetes/cfg/apiserver-to-kubelet-rbac.yaml

安装Kube-controller-manager

创建证书

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
cat > /opt/certs/kube-controller-manager-csr.json << EOF
{
"CN": "system:kube-controller-manager",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing",
"O": "system:masters",
"OU": "kubernetes"
}
]
}
EOF
# 生成证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager

生成kubeconfig

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
scp 10.1.1.11:/opt/certs/kube-controller-manager*.pem /opt/kubernetes/ssl/
KUBE_CONFIG="/opt/kubernetes/cfg/kube-controller-manager.kubeconfig"
KUBE_APISERVER="https://10.1.1.100:6443"

# 设置集群参数
kubectl config set-cluster kubernetes \
--certificate-authority=/opt/kubernetes/ssl/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=${KUBE_CONFIG}
# 设置客户端认证参数
kubectl config set-credentials kube-controller-manager \
--client-certificate=/opt/kubernetes/ssl/kube-controller-manager.pem \
--client-key=/opt/kubernetes/ssl/kube-controller-manager-key.pem \
--embed-certs=true \
--kubeconfig=${KUBE_CONFIG}
# 设置上下文参数
kubectl config set-context default \
--cluster=kubernetes \
--user=kube-controller-manager \
--kubeconfig=${KUBE_CONFIG}
# 设置默认上下文
kubectl config use-context default --kubeconfig=${KUBE_CONFIG}

kube-controller-manager配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
cd /server/tools/kubernetes/server/bin
cp kube-controller-manager /opt/kubernetes/bin

cat > /opt/kubernetes/cfg/kube-controller-manager.conf << EOF
KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--leader-elect=true \\
--kubeconfig=/opt/kubernetes/cfg/kube-controller-manager.kubeconfig \\
--cluster-cidr=172.7.0.0/16 \\
--service-cluster-ip-range=192.168.0.0/16 \\
--cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \\
--cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \\
--root-ca-file=/opt/kubernetes/ssl/ca.pem \\
--service-account-private-key-file=/opt/kubernetes/ssl/sa-key.pem \\
--cluster-signing-duration=87600h0m0s"
EOF

• —kubeconfig:连接apiserver配置文件

• —leader-elect:当该组件启动多个时,自动选举(HA)

• —cluster-signing-cert-file/—cluster-signing-key-file:自动为kubelet颁发证书的CA,与apiserver保持一致

配置kube-controller-manager启动文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
cat > /usr/lib/systemd/system/kube-controller-manager.service << EOF
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-controller-manager.conf
ExecStart=/opt/kubernetes/bin/kube-controller-manager \$KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

启动kube-controller-manager

1
2
3
4
5
6
7
systemctl daemon-reload
systemctl enable --now kube-controller-manager

# 检查
netstat -lnpt|grep kube
tcp 0 0 10.1.1.100:6443 0.0.0.0:* LISTEN 6905/kube-apiserver
tcp6 0 0 :::10257 :::* LISTEN 7253/kube-controlle

安装kube-scheduler

生成kube-scheduler证书

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
cat > /opt/certs/kube-scheduler-csr.json << EOF
{
"CN": "system:kube-scheduler",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing",
"O": "system:masters",
"OU": "kubernetes"
}
]
}
EOF

# 生成证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-scheduler-csr.json | cfssljson -bare kube-scheduler

生成kubeconfig

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
scp 10.1.1.11:/opt/certs/kube-scheduler*.pem /opt/kubernetes/ssl/

KUBE_CONFIG="/opt/kubernetes/cfg/kube-scheduler.kubeconfig"
KUBE_APISERVER="https://10.1.1.100:6443"

# 设置集群参数
kubectl config set-cluster kubernetes \
--certificate-authority=/opt/kubernetes/ssl/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=${KUBE_CONFIG}
# 设置客户端认证参数
kubectl config set-credentials kube-scheduler \
--client-certificate=/opt/kubernetes/ssl/kube-scheduler.pem \
--client-key=/opt/kubernetes/ssl/kube-scheduler-key.pem \
--embed-certs=true \
--kubeconfig=${KUBE_CONFIG}
# 设置上下文参数
kubectl config set-context default \
--cluster=kubernetes \
--user=kube-scheduler \
--kubeconfig=${KUBE_CONFIG}
# 设置默认上下文
kubectl config use-context default --kubeconfig=${KUBE_CONFIG}

创建kube-scheduler配置

1
2
3
4
5
6
7
8
9
10
11
cd /server/tools/kubernetes/server/bin
cp kube-scheduler /opt/kubernetes/bin

cat > /opt/kubernetes/cfg/kube-scheduler.conf << EOF
KUBE_SCHEDULER_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--leader-elect \\
--kubeconfig=/opt/kubernetes/cfg/kube-scheduler.kubeconfig \\
--bind-address=127.0.0.1"
EOF

• —kubeconfig:连接apiserver配置文件

• —leader-elect:当该组件启动多个时,自动选举(HA)

kube-scheduler启动文件

1
2
3
4
5
6
7
8
9
10
11
12
13
cat > /usr/lib/systemd/system/kube-scheduler.service << EOF
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-scheduler.conf
ExecStart=/opt/kubernetes/bin/kube-scheduler \$KUBE_SCHEDULER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

启动kube-scheduler

1
2
3
4
5
6
7
8
9
10
11
systemctl daemon-reload
systemctl enable --now kube-scheduler

# 检查
netstat -lnpt|grep kube
tcp 0 0 10.1.1.100:6443 0.0.0.0:* LISTEN 6905/kube-apiserver
tcp 0 0 127.0.0.1:10259 0.0.0.0:* LISTEN 7378/kube-scheduler
tcp6 0 0 :::10257 :::* LISTEN 7253/kube-controlle

[root@k8s-master1 ~]# tailf /opt/kubernetes/logs/kube-scheduler.INFO
I0516 22:16:14.820411 7378 leaderelection.go:258] successfully acquired lease kube-system/kube-scheduler

查看集群状态

生成kubectl连接集群的证书

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
cat > admin-csr.json <<EOF
{
"CN": "admin",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing",
"O": "system:masters",
"OU": "System"
}
]
}
EOF

# 创建证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin

生成kubeconfig文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
scp 10.1.1.11:/opt/certs/admin*.pem /opt/kubernetes/ssl/ 

mkdir /root/.kube

KUBE_CONFIG="/root/.kube/config"
KUBE_APISERVER="https://10.1.1.100:6443"

# 设置集群参数
kubectl config set-cluster kubernetes \
--certificate-authority=/opt/kubernetes/ssl/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=${KUBE_CONFIG}
# 设置客户端认证参数
kubectl config set-credentials cluster-admin \
--client-certificate=/opt/kubernetes/ssl/admin.pem \
--client-key=/opt/kubernetes/ssl/admin-key.pem \
--embed-certs=true \
--kubeconfig=${KUBE_CONFIG}
# 设置上下文参数
kubectl config set-context default \
--cluster=kubernetes \
--user=cluster-admin \
--kubeconfig=${KUBE_CONFIG}
# 设置默认上下文
kubectl config use-context default --kubeconfig=${KUBE_CONFIG}

查询

1
2
3
4
5
6
7
8
[root@k8s-master1 ~]# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
etcd-0 Healthy {"health":"true"}
etcd-1 Healthy {"health":"true"}
controller-manager Healthy ok
etcd-2 Healthy {"health":"true"}

如上输出说明Master节点组件运行正常。

授权kubelet-bootstrap用户允许请求证书

1
2
3
kubectl create clusterrolebinding kubelet-bootstrap \
--clusterrole=system:node-bootstrapper \
--user=kubelet-bootstrap