Kubernete-- 利用kubeadm 搭建一个kubernate集群

目标

• 利用 kubeadm 搭建一个四节点的k8s测试集群
• 利用harbor搭建一个单节点的私有镜像仓库
• k8s集群与私有镜像仓库整合
• 部署dashboard
• 部署Heapster 监控与统计

前期准备

准备以下4个节点，一个为k8s的master节点，2个为node节点，最后一个作为私有仓库镜像，系统为centos7.2：

hostname	ip	说明
k8s.master	192.168.2.130	k8s master节点
k8s.node1	192.168.2.131	k8s node节点
k8s.node2	192.168.2.132	k8s node节点
k8s.harbor	192.168.2.139	k8s 仓库镜像节点

K8S 安装

1. 所有节点的操作

更新yum

sudo yum update -y

执行以下命令关闭防火墙

systemctl stop firewalld && systemctl disable firewalldsetenforce 0

将SELINUX的值改成disabled

vim /etc/selinux/configSELINUX=disabled

为了使上面的改动生效，需要重启CentOS；

安装docker 官方文档：

#1.配置仓库sudo yum install -y yum-utils device-mapper-persistent-data lvm2sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo #2.可以选择是否开启edge和test仓库sudo yum-config-manager --enable docker-ce-edgesudo yum-config-manager --enable docker-ce-testsudo yum-config-manager --disable docker-ce-edgesudo yum-config-manager --disable docker-ce-test #3.安装docker-cesudo yum install docker-ce     #由于repo中默认只开启stable仓库，故这里安装的是最新稳定版17.09 #4.可以查看所有仓库中所有docker版本，并选择特定版本安装yum list docker-ce --showduplicates | sort -r docker-ce.x86_64            17.09.0.ce-1.el7.centos            docker-ce-stabledocker-ce.x86_64            17.09.0.ce-1.el7.centos            @docker-ce-stabledocker-ce.x86_64            17.06.2.ce-1.el7.centos            docker-ce-stabledocker-ce.x86_64            17.06.1.ce-1.el7.centos            docker-ce-stabledocker-ce.x86_64            17.06.0.ce-1.el7.centos            docker-ce-stabledocker-ce.x86_64            17.03.2.ce-1.el7.centos            docker-ce-stabledocker-ce.x86_64            17.03.1.ce-1.el7.centos            docker-ce-stabledocker-ce.x86_64            17.03.0.ce-1.el7.centos            docker-ce-stable sudo yum install 
    
       例如：sudo yum install docker-ce-17.09.0.ce #5.启动并加入开机启动sudo systemctl start dockersudo systemctl enable docker #6.关闭docker-daemonsudo systemctl stop dockersudo systemctl disable docker
    

设置系统参数

cat <
    
       /etc/sysctl.d/k8s.confnet.bridge.bridge-nf-call-ip6tables = 1net.bridge.bridge-nf-call-iptables = 1EOFsysctl --system
    

关闭swap

swapoff -a

临时关闭，重启后失效

修改 /etc/fstab 文件，注释掉 SWAP 的自动挂载，使用free -m确认swap已经关闭。 > swappiness参数调整，修改/etc/sysctl.d/k8s.conf添加下面一行：

vm.swappiness=0 执行sysctl -p /etc/sysctl.d/k8s.conf使修改生效。

如果服务器可以翻墙，可以直接通过yum命令安装，内容如下： vim /etc/yum.repos.d/kubernetes.repo

[kubernetes]name=Kubernetesbaseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64enabled=1gpgcheck=1repo_gpgcheck=1gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg        https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg

安装kubernetes相关的应用

yum updateyum install -y kubelet kubeadm kubectl

如果不能翻墙，只能先下载下来，然后安装，需要安装的rpm包url地址可以在这个网页中找到：

curl https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64/repodata/primary.xml

1.9.2百度网盘下载地址

https://pan.baidu.com/s/1i6Qbo4d

我们这里只需要安装四个rpm包，kubectl,kubeadm, kubelet以及kubernetes-cni，可以直接搜索上面的网页然后找到合适版本的rpm包。我们这里安装最新版本1.9.3,对应的地址如下：

kubeadm-1.9.2-0.x86_64.rpmkubelet-1.9.3-0.x86_64.rpmkubernetes-cni-0.6.0-0.x86_64.rpmkubectl-1.9.2-0.x86_64.rpmsocat-1.7.3.2-2.el7.x86_64.rpmyum install -y *.rpmsystemctl enable kubelet && systemctl start kubelet

journalctl -xeu kubelet查看错误信息

修改kubernetes的网络配置: 修改文件/etc/systemd/system/kubelet.service.d/10-kubeadm.conf，找到下面这一行：

Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=systemd"

把systemd改成cgroupfs；

然后执行以下命令重启kubelet：

systemctl restart kubeletsystemctl status kubelet

接下来开始基于Kubeadm 创建k8s集群，不过在开始之前，我们先准备下需要用到的镜像，因为kubeadm创建的k8s集群中的kub-api, kube-scheduler, kube-proxy, kube-controller-manager,etcd等服务都是直接拉取镜像跑在k8s集群中，为了避免安装过程中下载镜像浪费太多时间，这里先把镜像下载好。 我们直接用1.9.2，如果服务器可以翻墙，直接拉取镜像：

#必须镜像gcr.io/google_containers/kube-apiserver-amd64:v1.9.2gcr.io/google_containers/kube-proxy-amd64:v1.9.2gcr.io/google_containers/kube-controller-manager-amd64:v1.9.2gcr.io/google_containers/kube-scheduler-amd64:v1.9.2gcr.io/google_containers/etcd-amd64:3.1.11gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.7gcr.io/google_containers/k8s-dns-kube-dns-amd641.14.7gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.7gcr.io/google_containers/pause-amd64:3.0#calicoquay.io/calico/node:v2.6.7quay.io/calico/kube-controllers:v1.0.3quay.io/coreos/etcd:v3.1.10#flannelquay.io/coreos/flannel:v0.9.1-amd64#Dashboardk8s.gcr.io/kubernetes-dashboard-amd64

导入镜像

### 导入镜像 ###[root@k8s-master images]# docker load -i cni.tar                  .                  .                  .[root@k8s-master images]# docker imagesREPOSITORY                                                       TAG                 IMAGE ID            CREATED             SIZEquay.io/calico/node                                              v2.6.7              7c694b9cac81        8 days ago          281.6 MBgcr.io/google_containers/kube-controller-manager-amd64           v1.9.2              769d889083b6        3 weeks ago         137.8 MBgcr.io/google_containers/kube-proxy-amd64                        v1.9.2              e6754bb0a529        3 weeks ago         109.1 MBgcr.io/google_containers/kube-apiserver-amd64                    v1.9.2              7109112be2c7        3 weeks ago         210.4 MBgcr.io/google_containers/kube-scheduler-amd64                    v1.9.2              2bf081517538        3 weeks ago         62.71 MBquay.io/calico/kube-controllers                                  v1.0.3              34aebe64326d        3 weeks ago         52.25 MBk8s.gcr.io/kubernetes-dashboard-amd64                            v1.8.2              c87ea0497294        3 weeks ago         102.3 MBquay.io/calico/cni                                               v1.11.2             6f0a76fc7dd2        7 weeks ago         70.78 MBgcr.io/google_containers/etcd-amd64                              3.1.11              59d36f27cceb        9 weeks ago         193.9 MBquay.io/coreos/flannel                                           v0.9.1-amd64        2b736d06ca4c        12 weeks ago        51.31 MBgcr.io/google_containers/k8s-dns-sidecar-amd64                   1.14.7              db76ee297b85        3 months ago        42.03 MBgcr.io/google_containers/k8s-dns-kube-dns-amd64                  1.14.7              5d049a8c4eec        3 months ago        50.27 MBgcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64             1.14.7              5feec37454f4        3 months ago        40.95 MBquay.io/coreos/etcd                                              v3.1.10             47bb9dd99916        6 months ago        34.56 MBgcr.io/google_containers/pause-amd64                             3.0                 99e59f495ffa        21 months ago       746.9 kB

在各个节点上运行ip_vs 因为可能遇到

[root@k8s-master /etc/modprobe.d]$ docker logs 71a413c7b015I0307 03:32:57.394549       1 feature_gate.go:184] feature gates: map[]time="2018-03-07T03:32:57Z" level=warning msg="Running modprobe ip_vs failed with message: `modprobe: ERROR: could not insert 'ip_vs': Exec format error\ninsmod /lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/net/netfilter/ipvs/ip_vs.ko.xz`, error: exit status 1"time="2018-03-07T03:32:57Z" level=error msg="Could not get ipvs family information from the kernel. It is possible that ipvs is not enabled in your kernel. Native loadbalancing will not work until this is fixed."

在kube-proxy 运行之前先运行ip_vs

[root@k8s-master ~]$ modprobe -r ip_vs[root@k8s-master ~]$ lsmod|grep ip_vs[root@k8s-master ~]$ modprobe ip_vs[root@k8s-master ~]$ lsmod|grep ip_vsip_vs                 141092  0nf_conntrack          133387  9 ip_vs,nf_nat,nf_nat_ipv4,nf_nat_ipv6,xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_netlink,nf_conntrack_ipv4,nf_conntrack_ipv6libcrc32c              12644  3 ip_vs,nf_nat,nf_conntrack[root@k8s-master ~]$ modprobe -r ip_vs[root@k8s-master ~]$ insmod /lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/net/netfilter/ipvs/ip_vs.ko.xz[root@k8s-master ~]$ lsmod|grep ip_vsip_vs                 141092  0nf_conntrack          133387  9 ip_vs,nf_nat,nf_nat_ipv4,nf_nat_ipv6,xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_netlink,nf_conntrack_ipv4,nf_conntrack_ipv6libcrc32c              12644  3 ip_vs,nf_nat,nf_conntrack[root@k8s-master ~]$

2. master节点安装

使用kubeadm初始化master节点

kubeadm init --kubernetes-version=v1.9.2 --pod-network-cidr=172.16.0.0/16 --apiserver-advertise-address=192.168.2.130

注意命令执行后的最后一句kubeadm join --token d8b040.0aa73933666c2865 192.168.2.130:6443 --discovery-token-ca-cert-hash sha256:c24a3a8404036a9e96e30dcada132eceb35e5db3e036c9e4ea8809cb9c623531，在node加入的时候我们就是用这一句来将node加入到集群的

kubeadm会自动检查当前环境是否有上次命令执行的"残留"。如果有，必须清理后再行执行init。我们可以通过"kubeadm reset"来清理环境，以备重来。

创建kube目录，添加kubectl配置

mkdir -p $HOME/.kubesudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/configsudo chown $(id -u):$(id -g) $HOME/.kube/config

检查k8s组件状态

kubectl get cs

添加Calico(网络组件)

注意修改CALICO_IPV4POOL_CIDR参数

wget https://docs.projectcalico.org/v2.6/getting-started/kubernetes/installation/hosted/kubeadm/1.6/calico.yamlkubectl apply -f calico.yaml

2. node节点安装

kubeadm join --token d8b040.0aa73933666c2865 192.168.2.130:6443 --discovery-token-ca-cert-hash sha256:c24a3a8404036a9e96e30dcada132eceb35e5db3e036c9e4ea8809cb9c623531

在master上查看

kubectl get nodeskubectl get pods --all-namespaceskubectl describe node ******

节点重置/移除节点

### 驱离k8s-node-1节点上的pod ###[root@k8s-master ~]# kubectl drain k8s.node1 --delete-local-data --force --ignore-daemonsets### 删除节点 ###[root@k8s-master ~]# kubectl delete node k8s.node1### 重置节点 ###[root@k8s-node-1 ~]# kubeadm reset

token过期后重新生成token

#重新生成新的token[root@k8s-master ~]# kubeadm token create[root@k8s-master ~]# kubeadm token listTOKEN                     TTL       EXPIRES                     USAGES                   DESCRIPTION   EXTRA GROUPSaa78f6.8b4cafc8ed26c34f   23h       2018-03-26T16:36:29+08:00   authentication,signing   
    
             system:bootstrappers:kubeadm:default-node-token#获取ca证书sha256编码hash值[root@k8s-master ~]# openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'0fd95a9bc67a7bf0ef42da968a0d55d92e52898ec37c971bd77ee501d845b538#节点加入集群kubeadm join --token aa78f6.8b4cafc8ed26c34f 192.168.2.130:6443 --discovery-token-ca-cert-hash sha256:0fd95a9bc67a7bf0ef42da968a0d55d92e52898ec37c971bd77ee501d845b538 --skip-preflight-checks
    

--skip-preflight-checks，可以防止每次初始化都去检查配置文件，否则可能在多次init后报错[etcd在使用、kubelet在使

3. 部署Dashboard 在master节点上执行：

wget https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yamlkubectl apply -f kubernetes-dashboard.yaml

默认配置部署的dashboard账号kubernetes-dashboard绑定的角色为kubernetes-dashboard-minimal，我们需要为其配置一个权限更高的用户 Dashboard的服务方式已经被修改为nodeport

vi kubernetes-dashboard-rbac-admin.yml

apiVersion: v1kind: ServiceAccountmetadata:  labels:    k8s-app: kubernetes-dashboard  name: kubernetes-dashboard-admin  namespace: kube-system---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRoleBindingmetadata:  name: kubernetes-dashboard-admin  labels:    k8s-app: kubernetes-dashboardroleRef:  apiGroup: rbac.authorization.k8s.io  kind: ClusterRole  name: cluster-adminsubjects:- kind: ServiceAccount  name: kubernetes-dashboard-admin  namespace: kube-system

[root@k8s-master yml]# kubectl apply -f kubernetes-dashboard-rbac-admin.yml

我们使用自己创建的kubernetes-dashboard-admin用户登录dashboard，并且使用token方式登录

### 找到kubernetes-dashboard-admin的secret记录[root@k8s-master yml]# kubectl get secret --all-namespaces | grep kubernetes-dashboard-adminkube-system   kubernetes-dashboard-admin-token-2w8mg           kubernetes.io/service-account-token   3         22h## 查看secret的token值[root@master images]# kubectl get secret --all-namespaces | grep kubernetes-dashboard-adminkube-system   kubernetes-dashboard-admin-token-2w8mg           kubernetes.io/service-account-token   3         18h[root@master images]# kubectl describe secret kubernetes-dashboard-admin-token-2w8mg -n kube-systemName:         kubernetes-dashboard-admin-token-2w8mgNamespace:    kube-systemLabels:       
    
     Annotations:  kubernetes.io/service-account.name=kubernetes-dashboard-admin              kubernetes.io/service-account.uid=b89b863f-1d24-11e8-9bec-080027fdd465Type:  kubernetes.io/service-account-tokenData====ca.crt:     1025 bytesnamespace:  11 bytestoken:      eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZC1hZG1pbi10b2tlbi0ydzhtZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZC1hZG1pbiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImI4OWI4NjNmLTFkMjQtMTFlOC05YmVjLTA4MDAyN2ZkZDQ2NSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlLXN5c3RlbTprdWJlcm5ldGVzLWRhc2hib2FyZC1hZG1pbiJ9.VNxhEYW_QXUMJC-M4H4Mbe_ziDz8g-SbmYtDzCGAN-n6Cpbu0xsle9ZN-uAzMaZIE35aPGhs3AncnnuhBNzdXp3PDYtZCuepaskLQXc73hcXYoRTmNjHM2y23oQQliatBl_ICsujiWI7cbmkvZKU0YVmicn3TYgHKVODFxexvxN5kS9Pirv-vT6g03SnvFCktAteV2nO6pQele0Qyt6UGPZX1lnJDrmAGr3_jzo9YGSDdcZePchiJnBOVh4P0ufRqPylNij5-Upxi3uCU10sOuySeaUuPMerHffW0rafrRPeD_6cyB0cAqawj3CTwFl1IbR4-UOsTZlId1LkCoaCCg
    

查看dashboard的端口，并使用上面的token登录

kubectl get svc --all-namespaces

kubectl describe svc kubernetes-dashboard-admin

4. 部署Heapster 监控与统计

wget https://github.com/kubernetes/heapster/archive/v1.3.0.tar.gztar -zxvf v1.3.0.tar.gzcd heapster-1.3.0/deploy/kube-config/influxdbkubectl create -f ./*

5. k8s 添加私有镜像 利用kubectl创建docker-registry的secret

kubectl create secret docker-registry myregistrykey --docker-server=DOCKER_REGISTRY_SERVER --docker-username=DOCKER_USER --docker-password=DOCKER_PASSWORD --docker-email=DOCKER_EMAILkubectl get secret --all-namespaces   #查看创建的secret