Kubernetes 完全教程

Kubernetes 的 API Spec 以及安全机制

王渊命 @jolestar

Agenda

Kubernetes 的 API Spec
- YAML
- API Spec 概述
- Label 和 Selector 机制
- Pod 详解
- Service/ReplicaSet/Deployment/DaemonSet/StatefulSet/Job/CronJob/ConfigMap
- Kubernetes API Object 的连带删除机制
Kubernetes 的安全机制
- 认证（Authentication）
- 授权（Authorization）
- Admission Control

YAML 简介

YAML is a human-readable data serialization language. It is commonly used for configuration files, but could be used in many applications where data is being stored or transmitted.

空格缩进表示层级关系，不支持 TAB
: 用来分割 key/value
- 表示数组元素，每行一个，也可以用方括号（[] ）和逗号（,）来区分。
字符串可以不用引号，也支持 " 和 '
可以用 --- 区隔文档，把多个文档合并到同一个文件中
支持多行字符输入，通过 | 保留换行符，或者 > 折叠换行
支持锚点标记（&）和参考标记（*）避免重复
# 表示注释
是 json 格式的超集

YAML Example

# Invoice example
receipt:     Oz-Ware Purchase Invoice
customer:
    given:   Dorothy   
items:
    - part_no:   A4786
      descrip:   Water Bucket (Filled)
      price:     1.47
      quantity:  4
    - part_no:   E1628
      descrip:   High Heeled "Ruby" Slippers
      price:     133.7
      quantity:  1
bill-to:  &id001
    street: | 
            123 Tornado Alley
            Suite 16
    city:   East Centerville
ship-to:  *id001   
specialDelivery:  >
    Follow the Yellow Brick
    Road to the Emerald City.

Kubernets API Spec 概述

apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    app: web
    env: prod
  annotations:
    mydomain.com/custom-extend-key: value
spec:
  containers:
  - name: nginx
    image: nginx:1.7.9
    ports:
    - containerPort: 80

Kubernetes API Spec

apiVersion: Object Schema 版本 (v1,batch/v1,storage.k8s.io/v1)
kind: Object Schema 类型（List,Pod,Node,Service)
metadata:
- namespace
- name
- uid
- resourceVersion
- labels
- annotations

Kubernetes API Spec

spec
status

Label 和 Selector

equality/inequality-based

environment = production
tier != frontend

set-based

environment in (production, qa)
tier notin (frontend, backend)
partition
!partition

kubectl get pods -l environment=production,tier=frontend

Pod

Pod 的实现原理（参看预备课）
Pod Spec
Pod 生命周期

Container

command/args
image/imagePullPolicy: Always, Never, IfNotPresent
livenessProbe/readinessProbe: exec,httpGet,tcp
resources
volumeMounts

Pod Spec

apiVersion: core/v1
activeDeadlineSeconds/terminationGracePeriodSeconds
dnsPolicy: ClusterFirst,ClusterFirstWithHostNet,Default
hostNetwork/hostPID/hostIPC
initContainers/containers
imagePullSecrets
serviceAccountName
lifecycle: Hooks: postStart/preStop
restartPolicy: Always, OnFailure, Never
nodeName/nodeSelector
tolerations
affinity
volumes

Pod 生命周期

PodStatus
- phase Pending/Running/Succeeded/Failed/Unknown
- conditions PodScheduled,Initialized,Ready
- containerStatuses

Pod 生命周期-创建

https://dzone.com/articles/kubernetes-lifecycle-of-a-pod

PodTemplateSpec

metadata
spec（PodSpec）

PodDisruptionBudget

apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: zk-pdb
spec:
  selector:
    matchLabels:
      app: zk
  maxUnavailable: 1

Service

apiVersion: v1
spec
- clusterIP: None/IP
- externalIPs
- ports
- selector
- externalName
- type: ExternalName, ClusterIP, NodePort, LoadBalancer

Service

apiVersion: v1
kind: Service
metadata:
  labels:
    k8s-app: kube-dns
  name: kube-dns
  namespace: kube-system
spec:
  clusterIP: 10.96.0.10
  ports:
  - name: dns
    port: 53
    protocol: UDP
    targetPort: 53
  selector:
    k8s-app: kube-dns
  type: ClusterIP

selector 只支持 equality-based

Endpoint

addresses
notReadyAddresses
ports

apiVersion: v1
kind: Endpoints
metadata:
  name: helloworld
  namespace: default
subsets:
- addresses:
  - ip: 192.168.100.244
    nodeName: i-0elqbqzu
  - ip: 192.168.101.247
    nodeName: i-7d7zqw6x
  ports:
  - port: 80
    protocol: TCP

ReplicaSet (ReplicationController)

apiVersion: apps/v1beta2 (1.8 以前是 extensions/v1beta1)
spec
- minReadySeconds
- replicas
- selector
- template: PodTemplateSpec

Deployment

apiVersion: apps/v1beta2 （1.8以前是 apps/v1beta1）
spec
- minReadySeconds
- paused
- replicas
- selector
- strategy: Recreate/RollingUpdate
  - rollingUpdate.maxSurge
  - rollingUpdate.maxUnavailable
- revisionHistoryLimit
- template: PodTemplateSpec

Deployment

apiVersion: apps/v1beta2
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80

Deployment Rolling updates

kubectl set image deployment <deployment> <container>=<image>
kubectl rollout status deployment <deployment>
kubectl rollout pause deployment <deployment>
kubectl rollout resume deployment <deployment>
kubectl rollout undo deployment <deployment>

https://kubernetes.io/docs/tutorials/kubernetes-basics/update-intro/

DaemonSet

apiVersion: apps/v1beta2 (1.8以前是 extensions/v1beta1)
spec
- minReadySeconds
- revisionHistoryLimit
- selector
- template
- updateStrategy: RollingUpdate,OnDelete

StatefulSet

apiVersion: apps/v1beta2 (1.8以前是 apps/v1beta1)
spec
- podManagementPolicy: OrderedReady,Parallel
- serviceName
- replicas
- revisionHistoryLimit
- selector
- template
- updateStrategy: OnDelete,RollingUpdate
  - rollingUpdate.partition
- volumeClaimTemplates: PersistentVolumeClaim

StatefulSet

Pod Identity
- ordinal: [0,N)
- pod name: $(statefulset name)-$(ordinal)
- pod dns: $(pod name).$(service name).$(namespace).svc.cluster.local
Storage
- PersistentVolume Provisioner
- volumeClaimTemplates 的定义优先级高于 PodTemplate 中的定义
- PVC 的生命周期独立于 StatefulSet

Job

apiVersion: batch/v1
spec
- activeDeadlineSeconds
- backoffLimit: default 6
- completions
- parallelism
- selector
- manualSelector
- template: PodTemplateSpec
  - RestartPolicy: Never,OnFailure

Job

apiVersion: batch/v1
kind: Job
metadata:
  name: pi
spec:
  template:
    metadata:
      name: pi
    spec:
      containers:
      - name: pi
        image: perl
        command: ["perl",  "-Mbignum=bpi", "-wle", "print bpi(2000)"]
      restartPolicy: Never
      backoffLimit: 4

CronJob

apiVersion: batch/v1beta1
spec
- concurrencyPolicy: Allow, Forbid, Replace
- startingDeadlineSeconds
- successfulJobsHistoryLimit: default 3
- failedJobsHistoryLimit: default 1
- jobTemplate: JobTemplateSpec
- schedule: Cron format
- suspend

CronJob

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: hello
spec:
  schedule: "*/1 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: hello
            image: busybox
            args:
            - /bin/sh
            - -c
            - date; echo Hello from the Kubernetes cluster
          restartPolicy: OnFailure

ConfigMap

apiVersion: v1

data: object

kind: ConfigMap
apiVersion: v1
metadata:
  name: example-config
  namespace: default
data:
  examplekey: hello
  example.property: |-
    property.1=value-1
    property.2=value-2
    property.3=value-3

使用：env,volume

Kubernetes API Object 的连带删除机制

Label/Selector
metadata.ownerReferences
kubectl delete --cascade=false

Kubernetes 安全机制

认证（Authentication）：Client Certificates, Password, and Plain Tokens, Bootstrap Tokens, JWT Tokens
授权（Authorization）：ABAC, RBAC, Webhook
Admission Control

Kubernetes 安全机制

Kubernetes 的认证（Authentication）

user (normal users, service accounts)
group

Kubernetes 的认证（Authentication）

X509 Client Certs

 openssl req -new -key jbeda.pem -out jbeda-csr.pem -subj "/CN=jbeda/O=app1/O=app2"

https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster/

Static Token File

apiserver --token-auth-file=SOMEFILE

 token,user,uid,"group1,group2,group3"

Authorization: Bearer token-xxxx

Static Password File

apiserver --basic-auth-file=SOMEFILE

 password,user,uid,"group1,group2,group3"

Authorization: Basic BASE64ENCODED(USER:PASSWORD)

Kubernetes 的认证（Authentication）

OpenID Connect Tokens
- OAuth2 + id_token(JSON Web Token)
Webhook Token Authentication
- apiserver
  - --authentication-token-webhook-config-file
  - --authentication-token-webhook-cache-ttl
Authenticating Proxy
- --requestheader-username-headers
- --requestheader-group-headers

ServiceAcccount

一种将身份和资源绑定的机制（AWS Role, GCE ServiceAccount）
通过以下机制实现
- ServiceAccount admission
  - 自动检查或者设置 pod 的 serviceAccountName
  - 增加 token volume 并且挂载（/var/run/secrets/kubernetes.io/serviceaccount）
- Token controller
- Service account controller
- client sdk 自动读取固定路径下的 token
支持关联 imagePullSecret

Kubernetes 的授权（Authorization）

授权的主要要素

User
Group
Verb: get, list, create, update, patch, watch, proxy, redirect, delete, deletecollection
Resource
Namespace
API
API Group

Kubernetes Authorization Modules

Node

ABAC

 {"apiVersion": "abac.authorization.kubernetes.io/v1beta1", "kind": "Policy", 
 "spec": {"user": "alice", "namespace": "*", "resource": "*", "apiGroup": "*"}}

RBAC
Webhook

Kubernetes RBAC Authorization

Role/ClusterRole

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: default
  name: pod-reader
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "watch", "list"]

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: secret-reader
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "watch", "list"]

Kubernetes RBAC Authorization

RoleBinding/ClusterRoleBinding

kind: RoleBinding/ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: read-pods
  namespace: default
subjects:
- kind: User/Group/ServiceAccount
  name: jane
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role/ClusterRole
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

Admission Control

插件设置: apiserver --admission-control
主要插件：
- AlwaysPullImages
- DefaultStorageClass
- GenericAdmissionWebhook
- ImagePolicyWebhook
- Initializers: https://github.com/kelseyhightower/kubernetes-initializer-tutorial
- LimitRanger
- PodNodeSelector
- PodPreset
- ResourceQuota: https://kubernetes.io/docs/concepts/policy/resource-quotas/
- ServiceAccount
- PodSecurityPolicy: https://kubernetes.io/docs/concepts/policy/pod-security-policy/

演示

DaemonSet https://github.com/QingCloudAppcenter/kubernetes/blob/master/k8s/addons/kube-proxy

Service 以及 Deployment 滚动升级

wget https://raw.githubusercontent.com/jolestar/kubernetes-complete-course/master/example/helloworld.yaml
kubectl apply -f helloworld.yaml
kubectl scale --replicas=6 deployment/helloworld
curl -H "accept: application/yaml" helloworld/status
kubectl get replicaset
kubectl set image deployment/helloworld web=jolestar/go-probe:v0.2
kubectl rollout status deployment/helloworld
kubectl rollout pause deployment/helloworld
kubectl get replicaset
kubectl rollout resume deployment/helloworld
kubectl rollout undo deployment/helloworld
curl -H "accept: application/yaml" helloworld/status

将 pod 从 Deployment 摘除
RBAC 以及 ServiceAccount 的使用 https://github.com/QingCloudAppcenter/kubernetes/tree/master/k8s/addons/dashboard

演示

通过 StatefulSet 部署 zookeeper 集群 https://kubernetes.io/docs/tutorials/stateful-application/zookeeper/

 wget https://raw.githubusercontent.com/jolestar/kubernetes-complete-course/master/example/zookeeper_mini.yaml
 kubectl apply -f zookeeper_mini.yaml
 kubectl exec zk-0 -- cat /opt/zookeeper/conf/zoo.cfg
 kubectl exec zk-0 -- /opt/zookeeper/bin/zkCli.sh create /test "test"
 kubectl exec zk-0 -- cat /opt/zookeeper/bin/start-zookeeper|less
 kubectl exec zk-0 -- /opt/zookeeper/bin/zkCli.sh create ls /
 kubectl exec zk-1 -- /opt/zookeeper/bin/zkCli.sh create ls /
 kubectl delete -f zookeeper_mini.yaml
 kubectl get pvc
 kubectl get pv
 kubectl apply -f zookeeper_mini.yaml
 kubectl exec zk-0 -- /opt/zookeeper/bin/zkCli.sh create ls /

总结

对 Kubernetes 的 Spec 以及安全机制有一个整体的理解

Kubernetes 的描述文件能做什么，如何将当前系统组件通过 Kubernetes 中的对象描述出来
Kubernetes 提供了哪些安全机制，如何和当前内部权限系统整合
Kubernetes 提供的扩展插件能做什么，如何通过插件在集群全局配置策略，约束和规范应用的部署行为

关于 API Spec 中的版本以及支持的属性，参看 Api Reference 1.7 1.8 下节课：通过应用案例解析 kubernetes 中的 pod 放置策略，autoscale 等。

作业

通过 StatefulSet 部署 kafka，参看 https://github.com/kow3ns/kubernetes-kafka, 并理解如何利用 Kubernetes 的能力部署分布式系统。
通过证书生成工具，生成一个普通用户证书并让 Kubernetes 批准，然后使用该证书操作集群，而不是默认的 admin 账号。
思考并尝试通过 Kubernetes 搭建一个日志收集分析系统。

关于我

个人博客: http://jolestar.com 课程 Github：https://github.com/jolestar/kubernetes-complete-course

Files

chapter-4-kubernetes-api-and-security.md

Latest commit

History

chapter-4-kubernetes-api-and-security.md

File metadata and controls

Kubernetes 完全教程

Kubernetes 的 API Spec 以及安全机制

王渊命 @jolestar

Agenda

YAML 简介

YAML Example

Kubernets API Spec 概述

Kubernetes API Spec

Kubernetes API Spec

Label 和 Selector

Pod

Container

Pod Spec

Pod 生命周期

Pod 生命周期-创建

PodTemplateSpec

PodDisruptionBudget

Service

Service

Endpoint

ReplicaSet (ReplicationController)

Deployment

Deployment

Deployment Rolling updates

DaemonSet

StatefulSet

StatefulSet

Job

Job

CronJob

CronJob

ConfigMap

Kubernetes API Object 的连带删除机制

Kubernetes 安全机制

Kubernetes 安全机制

Kubernetes 的认证（Authentication）

Kubernetes 的认证（Authentication）

Kubernetes 的认证（Authentication）

ServiceAcccount

Kubernetes 的授权（Authorization）

Kubernetes Authorization Modules

Kubernetes RBAC Authorization

Kubernetes RBAC Authorization

Admission Control

演示

演示

总结

作业

关于我