Kubernetes-v1.24版安装部署之Mater节点安装部署

Kubernetes Master

Kubernetes Master节点部署三个服务：kube-apiserver、kube-controller-manager、kube-scheduler和一个命令工具kubectl.

Master节点来负责整个集群的管理和控制，其中

kube-apiserver: 服务提供了HTTP Rest接口的关键服务进程，是Kuberneters里所有资源的增删改查等操作的唯一入口，也是集群控制的入口进程.

kube-controller-manager: 服务 是kubernetes里面所有资源对象的自动化控制中心，可以理解为资源对象的”大总管”

kube-scheduler: 服务负责资源调度(pod调度)的进程，相当于公交公司的”调度室”。

安装Kube-apiserver

相关参数介绍

• –logtostderr：启用日志

• —v：日志等级

• –log-dir：日志目录

• –etcd-servers：etcd集群地址

• –bind-address：监听地址

• –secure-port：https安全端口

• –advertise-address：集群通告地址

• –allow-privileged：启用授权

• –service-cluster-ip-range：Service虚拟IP地址段

• –enable-admission-plugins：准入控制模块

• –authorization-mode：认证授权，启用RBAC授权和节点自管理

• –enable-bootstrap-token-auth：启用TLS bootstrap机制

• –token-auth-file：bootstrap token文件

• –service-node-port-range：Service nodeport类型默认分配端口范围

• –kubelet-client-xxx：apiserver访问kubelet客户端证书

• –tls-xxx-file：apiserver https证书

• 1.20版本后必须加的参数：–service-account-issuer，–service-account-signing-key-file

• –etcd-xxxfile：连接Etcd集群证书

• –audit-log-xxx：审计日志

• 启动聚合层相关配置：–requestheader-client-ca-file，–proxy-client-cert-file，–proxy-client-key-file，–requestheader-allowed-names，–requestheader-extra-headers-prefix，–requestheader-group-headers，–requestheader-username-headers，–enable-aggregator-routing

更多参数介绍：https://kubernetes.io/zh/docs/reference/command-line-tools-reference/kube-apiserver/

创建证书

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
cat > /opt/certs/apiserver-csr.json <<EOF
{
    "CN": "kubernetes",
    "hosts": [
        "127.0.0.1",
        "192.168.0.1",
        "kubernetes.default",
        "kubernetes.default.svc",
        "kubernetes.default.svc.cluster",
        "kubernetes.default.svc.cluster.local",
        "10.1.1.50",
        "10.1.1.100",
        "10.1.1.110",
        "10.1.1.120",
        "10.1.1.130"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "beijing",
            "L": "beijing",
            "O": "apiserver",
            "OU": "kubernetes"
        }
    ]
}
EOF

## 生成证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes apiserver-csr.json |cfssljson -bare apiserver
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
cat > /opt/certs/sa-csr.json <<EOF
{
    "CN": "ServiceAccount",
    "hosts": [
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "beijing",
            "L": "beijing",
            "O": "ServiceAccount",
            "OU": "kubernetes"
        }
    ]
}
EOF

## 生成证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes sa-csr.json |cfssljson -bare sa

安装kube-apiserver

下载地址： https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.24.md

注：打开链接你会发现里面有很多包，下载一个server包就够了，包含了Master和Worker Node二进制文件。

1
2
3
4
5
6
7
8
mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs} 
tar zxvf kubernetes-server-linux-amd64.tar.gz
cd kubernetes/server/bin
cp kube-apiserver /opt/kubernetes/bin
cp kubectl /usr/bin/
scp 10.1.1.11:/opt/certs/apiserver*.pem /opt/kubernetes/ssl
scp 10.1.1.11:/opt/certs/ca*.pem /opt/kubernetes/ssl
scp 10.1.1.11:/opt/certs/sa*.pem /opt/kubernetes/ssl

配置apiserver文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
cat > /opt/kubernetes/cfg/kube-apiserver.conf <<EOF
KUBE_APISERVER_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--etcd-servers=https://10.1.1.100:2379,https://10.1.1.130:2379,https://10.1.1.120:2379 \\
--bind-address=10.1.1.100 \\
--secure-port=6443 \\
--advertise-address=10.1.1.100 \\
--allow-privileged=true \\
--service-cluster-ip-range=192.168.0.0/16 \\
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \\
--authorization-mode=RBAC,Node \\
--enable-bootstrap-token-auth=true \\
--token-auth-file=/opt/kubernetes/cfg/token.csv \\
--service-node-port-range=30000-32767 \\
--kubelet-client-certificate=/opt/kubernetes/ssl/apiserver.pem \\
--kubelet-client-key=/opt/kubernetes/ssl/apiserver-key.pem \\
--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname \\
--client-ca-file=/opt/kubernetes/ssl/ca.pem \\
--service-account-key-file=/opt/kubernetes/ssl/sa.pem \\
--service-account-issuer=https://kubernetes.default.svc.cluster.local \\
--service-account-signing-key-file=/opt/kubernetes/ssl/sa-key.pem \\
--etcd-cafile=/opt/etcd/ssl/ca.pem \\
--etcd-certfile=/opt/etcd/ssl/etcd.pem \\
--etcd-keyfile=/opt/etcd/ssl/etcd-key.pem \\
--tls-cert-file=/opt/kubernetes/ssl/apiserver.pem  \\
--tls-private-key-file=/opt/kubernetes/ssl/apiserver-key.pem \\
--proxy-client-cert-file=/opt/kubernetes/ssl/apiserver.pem \\
--proxy-client-key-file=/opt/kubernetes/ssl/apiserver-key.pem \\
--requestheader-allowed-names=aggregator \\
--requestheader-client-ca-file=/opt/kubernetes/ssl/ca.pem \\
--requestheader-extra-headers-prefix=X-Remote-Extra- \\
--requestheader-group-headers=X-Remote-Group \\
--requestheader-username-headers=X-Remote-User \\
--enable-aggregator-routing=true \\
--audit-log-maxage=30 \\
--audit-log-maxbackup=3 \\
--audit-log-maxsize=100 \\
--audit-log-path=/opt/kubernetes/logs/k8s-audit.log"
EOF

配置kube-apiserver启动文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
cat > /usr/lib/systemd/system/kube-apiserver.service <<EOF
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-apiserver.conf
ExecStart=/opt/kubernetes/bin/kube-apiserver \$KUBE_APISERVER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

创建配置token文件

1
2
3
 cat > /opt/kubernetes/cfg/token.csv <<EOF
bc43e407e311d78b60da186fdd347fc8,kubelet-bootstrap,10001,"system:node-bootstrapper"
EOF

格式：token，用户名，UID，用户组

token也可自行生成替换：

1
head -c 16 /dev/urandom | od -An -t x | tr -d ' '

启动apiserver

1
2
3
4
5
6
systemctl daemon-reload
systemctl enable --now kube-apiserver

## 检查配置是否启动
# netstat -lnpt|grep 6443
tcp        0      0 10.1.1.100:6443         0.0.0.0:*               LISTEN      6905/kube-apiserver

常见错误

1
2
# 此处为etcd正常关闭报错，故可忽略。
[transport] transport: loopyWriter.run returning. connection error: desc = "transport is closing"

授权apiserver访问kubelet

应用场景：例如kubectl logs

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
cat > /opt/kubernetes/cfg/apiserver-to-kubelet-rbac.yaml <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:kube-apiserver-to-kubelet
rules:
  - apiGroups:
      - "*"
    resources:
      - nodes
      - nodes/proxy
      - nodes/stats
      - nodes/log
      - nodes/spec
      - nodes/metrics
      - pods/log
    verbs:
      - "*"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:kube-apiserver
  namespace: ""
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:kube-apiserver-to-kubelet
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: kubernetes
EOF

kubectl apply -f /opt/kubernetes/cfg/apiserver-to-kubelet-rbac.yaml

安装Kube-controller-manager

创建证书

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
cat > /opt/certs/kube-controller-manager-csr.json << EOF
{
  "CN": "system:kube-controller-manager",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "BeiJing", 
      "ST": "BeiJing",
      "O": "system:masters",
      "OU": "kubernetes"
    }
  ]
}
EOF
# 生成证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager

生成kubeconfig

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
scp 10.1.1.11:/opt/certs/kube-controller-manager*.pem /opt/kubernetes/ssl/
KUBE_CONFIG="/opt/kubernetes/cfg/kube-controller-manager.kubeconfig"
KUBE_APISERVER="https://10.1.1.100:6443"

# 设置集群参数
kubectl config set-cluster kubernetes \
  --certificate-authority=/opt/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=${KUBE_CONFIG}
# 设置客户端认证参数 
kubectl config set-credentials kube-controller-manager \
  --client-certificate=/opt/kubernetes/ssl/kube-controller-manager.pem \
  --client-key=/opt/kubernetes/ssl/kube-controller-manager-key.pem \
  --embed-certs=true \
  --kubeconfig=${KUBE_CONFIG}
# 设置上下文参数  
kubectl config set-context default \
  --cluster=kubernetes \
  --user=kube-controller-manager \
  --kubeconfig=${KUBE_CONFIG}
# 设置默认上下文 
kubectl config use-context default --kubeconfig=${KUBE_CONFIG}

kube-controller-manager配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
cd /server/tools/kubernetes/server/bin
cp kube-controller-manager /opt/kubernetes/bin

cat > /opt/kubernetes/cfg/kube-controller-manager.conf << EOF        
KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--leader-elect=true \\
--kubeconfig=/opt/kubernetes/cfg/kube-controller-manager.kubeconfig \\
--authentication-kubeconfig=/opt/kubernetes/cfg/kube-controller-manager.kubeconfig \\
--authorization-kubeconfig=/opt/kubernetes/cfg/kube-controller-manager.kubeconfig \\
--bind-address=127.0.0.1 \\
--cluster-cidr=172.7.0.0/16 \\
--service-cluster-ip-range=192.168.0.0/16 \\
--cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \\
--cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem  \\
--root-ca-file=/opt/kubernetes/ssl/ca.pem \\
--service-account-private-key-file=/opt/kubernetes/ssl/sa-key.pem \\
--cluster-signing-duration=87600h0m0s"
EOF

• –kubeconfig：连接apiserver配置文件

• –leader-elect：当该组件启动多个时，自动选举（HA）

• –cluster-signing-cert-file/–cluster-signing-key-file：自动为kubelet颁发证书的CA，与apiserver保持一致

配置kube-controller-manager启动文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
cat > /usr/lib/systemd/system/kube-controller-manager.service << EOF
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-controller-manager.conf
ExecStart=/opt/kubernetes/bin/kube-controller-manager \$KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

启动kube-controller-manager

1
2
3
4
5
6
7
systemctl daemon-reload
systemctl enable --now kube-controller-manager

# 检查
netstat -lnpt|grep kube
tcp        0      0 10.1.1.100:6443         0.0.0.0:*               LISTEN      6905/kube-apiserver
tcp6       0      0 :::10257                :::*                    LISTEN      7253/kube-controlle

安装kube-scheduler

生成kube-scheduler证书

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
cat > /opt/certs/kube-scheduler-csr.json << EOF
{
  "CN": "system:kube-scheduler",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "BeiJing",
      "ST": "BeiJing",
      "O": "system:masters",
      "OU": "kubernetes"
    }
  ]
}
EOF

# 生成证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-scheduler-csr.json | cfssljson -bare kube-scheduler

生成kubeconfig

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
scp 10.1.1.11:/opt/certs/kube-scheduler*.pem /opt/kubernetes/ssl/

KUBE_CONFIG="/opt/kubernetes/cfg/kube-scheduler.kubeconfig"
KUBE_APISERVER="https://10.1.1.100:6443"

# 设置集群参数
kubectl config set-cluster kubernetes \
  --certificate-authority=/opt/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=${KUBE_CONFIG}
# 设置客户端认证参数 
kubectl config set-credentials kube-scheduler \
  --client-certificate=/opt/kubernetes/ssl/kube-scheduler.pem \
  --client-key=/opt/kubernetes/ssl/kube-scheduler-key.pem \
  --embed-certs=true \
  --kubeconfig=${KUBE_CONFIG}
# 设置上下文参数  
kubectl config set-context default \
  --cluster=kubernetes \
  --user=kube-scheduler \
  --kubeconfig=${KUBE_CONFIG}
# 设置默认上下文 
kubectl config use-context default --kubeconfig=${KUBE_CONFIG}

创建kube-scheduler配置

1
2
3
4
5
6
7
8
9
10
11
12
13
cd /server/tools/kubernetes/server/bin
cp kube-scheduler /opt/kubernetes/bin

cat > /opt/kubernetes/cfg/kube-scheduler.conf << EOF
KUBE_SCHEDULER_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--leader-elect \\
--kubeconfig=/opt/kubernetes/cfg/kube-scheduler.kubeconfig \\
--authentication-kubeconfig=/opt/kubernetes/cfg/kube-scheduler.kubeconfig \\
--authorization-kubeconfig=/opt/kubernetes/cfg/kube-scheduler.kubeconfig \\
--bind-address=127.0.0.1"
EOF

• –kubeconfig：连接apiserver配置文件

• –leader-elect：当该组件启动多个时，自动选举（HA）

kube-scheduler启动文件

1
2
3
4
5
6
7
8
9
10
11
12
13
cat > /usr/lib/systemd/system/kube-scheduler.service << EOF
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-scheduler.conf
ExecStart=/opt/kubernetes/bin/kube-scheduler \$KUBE_SCHEDULER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

启动kube-scheduler

1
2
3
4
5
6
7
8
9
10
11
systemctl daemon-reload
systemctl enable --now kube-scheduler

# 检查
netstat -lnpt|grep kube
tcp        0      0 10.1.1.100:6443         0.0.0.0:*               LISTEN      6905/kube-apiserver 
tcp        0      0 127.0.0.1:10259         0.0.0.0:*               LISTEN      7378/kube-scheduler 
tcp6       0      0 :::10257                :::*                    LISTEN      7253/kube-controlle 

[root@k8s-master1 ~]# tailf /opt/kubernetes/logs/kube-scheduler.INFO 
I0516 22:16:14.820411    7378 leaderelection.go:258] successfully acquired lease kube-system/kube-scheduler

查看集群状态

生成kubectl连接集群的证书

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
cat > admin-csr.json <<EOF
{
  "CN": "admin",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "BeiJing",
      "ST": "BeiJing",
      "O": "system:masters",
      "OU": "System"
    }
  ]
}
EOF

# 创建证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin

生成kubeconfig文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
scp 10.1.1.11:/opt/certs/admin*.pem /opt/kubernetes/ssl/ 

mkdir /root/.kube

KUBE_CONFIG="/root/.kube/config"
KUBE_APISERVER="https://10.1.1.100:6443"

# 设置集群参数
kubectl config set-cluster kubernetes \
  --certificate-authority=/opt/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=${KUBE_CONFIG}
# 设置客户端认证参数 
kubectl config set-credentials cluster-admin \
  --client-certificate=/opt/kubernetes/ssl/admin.pem \
  --client-key=/opt/kubernetes/ssl/admin-key.pem \
  --embed-certs=true \
  --kubeconfig=${KUBE_CONFIG}
# 设置上下文参数 
kubectl config set-context default \
  --cluster=kubernetes \
  --user=cluster-admin \
  --kubeconfig=${KUBE_CONFIG}
# 设置默认上下文
kubectl config use-context default --kubeconfig=${KUBE_CONFIG}

查询

1
2
3
4
5
6
7
8
[root@k8s-master1 ~]# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE             ERROR
scheduler            Healthy   ok                  
etcd-0               Healthy   {"health":"true"}   
etcd-1               Healthy   {"health":"true"}   
controller-manager   Healthy   ok                  
etcd-2               Healthy   {"health":"true"}

如上输出说明Master节点组件运行正常。

授权kubelet-bootstrap用户允许请求证书

1
2
3
kubectl create clusterrolebinding kubelet-bootstrap \
--clusterrole=system:node-bootstrapper \
--user=kubelet-bootstrap