An Intro to Kops

I have been talking a bit about Kubernetes lately so I figured I would cover how to deploy and maintain a Kubernetes cluster. I use kops, but multiple options exist. Read on to learn more!

Options to deploy k8s

You have a few options to deploy k8s today, namely:

• Roll your own: kops, kubeadm, etc
• Use a configuration management tool: CloudFormation, Terraform, etc
• Use a private cloud managed solution: AKS, EKS, GKE, VKE, etc
• Use a third-party managed solution: Platform9, Ranger, StratoScale, etc

Which option should you choose? Like most things, it depends. Already using configuration management tools and comfortable with them? Could be a good option. k8s not your area of expertise or where you want to differentiate? Perhaps a managed solution is best.

My team has a ton of operational expertise so we opted to roll our own. We tested kops first and really liked it. Kubeadm is the recommended k8s way to deploy k8s, but in terms of Github likes, kops has a lot more traction.

Kops features

No matter what solution you choose you need to ensure it fits your requirements. Here are the features of kops:

• Automates the provisioning of Kubernetes clusters in AWS and GCE
• Deploys Highly Available (HA) Kubernetes Masters
• Built on a state-sync model for dry-runs and automatic idempotency
• Ability to generate Terraform
• Supports custom Kubernetes add-ons
• Command line autocompletion
• YAML Manifest Based API Configuration
• Templating and dry-run modes for creating Manifests
• Choose from eight different CNI Networking providers out-of-the-box
• Supports upgrading from kube-up
• Capability to add containers, as hooks, and files to nodes via a cluster manifest

The items in bold were either very important to us or something we thought would be nice to have.

Kops deployment process

The process for deploying k8s via kops was not great when I was getting started. Since then it has become much better. The basic steps on AWS are:

1. Create a role with the appropriate permissions
2. Configure DNS if applicable
3. Configure a S3 bucket
4. Define how you want your cluster to look
5. Deploy with kops

Given these steps are pretty well-defined now, I figured I would dig more into step 4:

kops create cluster \
    --cloud aws \
    --encrypt-etcd-storage \
    --authorization rbac \
    --kubernetes-version ${KUBERNETES_VERSION} \
    --state ${KOPS_STATE_STORE} \
    --ssh-public-key ${KOPS_CLUSTER_NAME}_id_rsa.pub \
    --node-count ${NODE_COUNT} \
    --zones ${ZONES} \
    --node-size ${NODE_SIZE} \
    --master-size ${MASTER_SIZE} \
    --master-zones ${ZONES} \
    --networking calico \
    --topology private \
    --bastion \
    --yes

Some things to note:

• K8s is often NOT deployed securely by default (this is getting better with time) — some of the options above are default now, but were not until recently
• Zones is important if you want multi-AZ HA of master/nodes — be sure to read the HA guide as there are some gotchas
• For security reasons, k8s should always be private and access should always be done via a bastion host
• Unfortunately, the above is just scratching the service — there are A LOT of additional considerations

Kops commands

I already covered the primary kops command kops create cluster, but there are others to be aware of as well:

• kops get clusters which lists all the available clusters in the registry
• kops edit cluster which can be used to make changes to a cluster
• kops delete cluster which can be used to delete a cluster
• kops update cluster which can be used to update a cluster
• kops rolling-update cluster which is used to roll out changes that require a restart

The edit command allows you to do things like:

• Change the k8s version
• Add or remove nodes
• Change node types

After editing you need to update and potentially to a rolling-update.

Note: In addition to the cluster there is also a notion of “instance groups” which is a grouping of nodes that maps to an autoscaling group.

Post deployment activities

Once k8s is deployed, there are some other activities you will want to perform. These include:

• Installing tools like Helm — while the tiller will be removed in v3, it is still required today
• Installing monitoring tools like cadvisor, ELK, Prometheus, Grafana, etc
• Any patches to the default system

The last bullet is an extension to a comment I made earlier — you need to prioritize security. Some things to consider:

• Patching admission control in kube API server

    kubeAPIServer:
      admissionControl:
      - Initializers
      - NamespaceLifecycle
      - LimitRanger
      - ServiceAccount
      - PersistentVolumeLabel
      - DefaultStorageClass
      - DefaultTolerationSeconds
      - NodeRestriction
      - ResourceQuota
      - AlwaysPullImages
      - DenyEscalatingExec

• Avoid default credential injection into pods

  $ kubectl patch serviceaccount default -p "automountServiceAccountToken: false"

• Install and configure kube2iam

As a bonus, you may want to allow volume expansion:

$ kubectl patch storageclasses.storage.k8s.io gp2 -p "allowVolumeExpansion: true"

Summary

As you can see, deploying k8s is pretty straight forward when using kops, but the approach is not for everyone. Unfortunately, security and availability require some forethought as the default configuration is not meant for production environments. While availability is pretty straightforward to understand and configure, security is not. I will cover k8s security in more depth in a future post.

© 2018, Steve Flanders. All rights reserved.