http://queue.acm.org/detail.cfm?id=2898444
article-borg-omega-and-kubernetes-lessons#no-direct-accessIn contrast to Omega, which exposes the store directly to trusted control-plane components, state in Kubernetes is accessed exclusively through a domain-specific REST API that applies higher-level versioning, validation, semantics, and policy, in support of a more diverse array of clients. article-borg-omega-and-kubernetes-lessons#no-direct-access
article-borg-omega-and-kubernetes-lessons#utilization-higher-than-normsThe resource isolation provided by containers has enabled Google to drive utilization significantly higher than industry norms article-borg-omega-and-kubernetes-lessons#utilization-higher-than-norms
article-borg-omega-and-kubernetes-lessons#isolation-that-does-not-workcontainers cannot prevent interference in resources that the operating-system kernel doesn't manage, such as level 3 processor caches and memory bandwidth, and containers need to be supported by an additional security layer (such as virtual machines) to protect against the kinds of malicious actors found in the cloud. article-borg-omega-and-kubernetes-lessons#isolation-that-does-not-work
article-borg-omega-and-kubernetes-lessons#only-containersthe container has become the sole runnable entity supported by the Google infrastructure article-borg-omega-and-kubernetes-lessons#only-containers
article-borg-omega-and-kubernetes-lessons#annotations-communicate-application-structureannotations... "can be used to communicate application structure" article-borg-omega-and-kubernetes-lessons#annotations-communicate-application-structure
article-borg-omega-and-kubernetes-lessons#annotation-set-by-various-things"annotations can be set by the container itself or other actors in the management system" article-borg-omega-and-kubernetes-lessons#annotation-set-by-various-things
article-borg-omega-and-kubernetes-lessons#always-in-a-podalways runs an application container inside a top-level pod, even if the pod contains a single container article-borg-omega-and-kubernetes-lessons#always-in-a-pod
article-borg-omega-and-kubernetes-lessons#pod-makes-it-easy-to-add-support-serviceit's easy to add a new small support service, because it operates in the private execution environment provided by its own container article-borg-omega-and-kubernetes-lessons#pod-makes-it-easy-to-add-support-service
article-borg-omega-and-kubernetes-lessons#three-basic-fieldsevery Kubernetes object has three basic fields in its description: Object Metadata, Specification (or Spec), and Status. article-borg-omega-and-kubernetes-lessons#three-basic-fields
article-borg-omega-and-kubernetes-lessons#field-details
The Object Metadata is the same for all objects in the system; it contains information such as the object's name, UID (unique identifier), an object version number (for optimistic concurrency control), and labels (key-value pairs, see below). The contents of Spec and Status vary by object type, but their concept does not: Spec is used to describe the desired state of the object, whereas Status provides read-only information about the current state of the object.
article-borg-omega-and-kubernetes-lessons#field-details
article-borg-omega-and-kubernetes-lessons#dynamic-apisTo further this consistency, Kubernetes is being extended to enable users to add their own APIs dynamically, alongside the core Kubernetes functionality. article-borg-omega-and-kubernetes-lessons#dynamic-apis
article-borg-omega-and-kubernetes-lessons#three-forms-of-replicated-pods
Kubernetes has three different forms of replicated pods:
* ReplicationController: run-forever replicated containers (e.g., web servers).
* DaemonSet: ensure a single instance on each node in the cluster (e.g., logging agents).
* Job: a run-to-completion controller that knows how to run a (possibly parallelized) batch job from start to finish.
article-borg-omega-and-kubernetes-lessons#three-forms-of-replicated-pods
article-borg-omega-and-kubernetes-lessons#all-action-based-on-observationall action is based on observation rather than a state diagram article-borg-omega-and-kubernetes-lessons#all-action-based-on-observation
article-borg-omega-and-kubernetes-lessons#control-through-choreographyThe design of Kubernetes as a combination of microservices and small control loops is an example of control through choreography—achieving a desired emergent behavior by combining the effects of separate, autonomous entities that collaborate. This is a conscious design choice in contrast to a centralized orchestration system, which may be easier to construct at first but tends to become brittle and rigid over time, especially in the presence of unanticipated errors or state changes. article-borg-omega-and-kubernetes-lessons#control-through-choreography
article-borg-omega-and-kubernetes-lessons#ip-address-per-podwe decided that Kubernetes would allocate an IP address per pod, thus aligning network identity (IP address) with application identity article-borg-omega-and-kubernetes-lessons#ip-address-per-pod
article-borg-omega-and-kubernetes-lessons#networking-underlaysAll of the popular cloud platforms provide networking underlays that enable IP-per-pod article-borg-omega-and-kubernetes-lessons#networking-underlays
article-borg-omega-and-kubernetes-lessons#label-selectorsLabel selectors are the grouping mechanism in Kubernetes, and define the scope of all management operations that can span multiple entities. article-borg-omega-and-kubernetes-lessons#label-selectors
article-borg-omega-and-kubernetes-lessons#remove-from-load-balancer-by-changing-label
But the flexibility of labels has compensating advantages—for example, the separation of controllers and pods means that it is possible to "orphan" and "adopt" containers. Consider a load-balanced service that uses a label selector to identify the set of pods to send traffic to. If one of these pods starts misbehaving, that pod can be quarantined from serving requests by removing one or more of the labels that cause it to be targeted by the Kubernetes service load balancer. The pod is no longer serving traffic, but it will remain up and can be debugged in situ.
article-borg-omega-and-kubernetes-lessons#remove-from-load-balancer-by-changing-label
article-borg-omega-and-kubernetes-lessons#decoupled-client-componentsthe client components are decoupled from one another and can evolve or be replaced independently (which is especially important in the open-source environment) article-borg-omega-and-kubernetes-lessons#decoupled-client-components
article-borg-omega-and-kubernetes-lessons#api-server-centralization-benefitsthe centralization makes it easy to enforce common semantics, invariants, and policies article-borg-omega-and-kubernetes-lessons#api-server-centralization-benefits
article-borg-omega-and-kubernetes-lessons#configuration-is-code
The result is the kind of inscrutable "configuration is code" that people were trying to avoid by eliminating hard-coded parameters in the application's source code. It doesn't reduce operational complexity or make the configurations easier to debug or change; it just moves the computations from a real programming language to a domain-specific one, which typically has weaker development tools (e.g., debuggers, unit test frameworks, etc).
article-borg-omega-and-kubernetes-lessons#configuration-is-code
article-borg-omega-and-kubernetes-lessons#programmatic-configuration
We believe the most effective approach is to accept this need, embrace the inevitability of programmatic configuration, and maintain a clean separation between computation and data. The language to represent the data should be a simple, data-only format such as JSON or YAML, and programmatic modification of this data should be done in a real programming language, where there are well-understood semantics, as well as good tooling. Interestingly, this same separation of computation and data can be seen in the disparate field of front-end development with frameworks such as Angular that maintain a crisp separation between the worlds of markup (data) and JavaScript (computation).
article-borg-omega-and-kubernetes-lessons#programmatic-configuration
article-borg-omega-and-kubernetes-lessons#open-challenges
article-borg-omega-and-kubernetes-lessons#open-challenges