/ open source

Kubernetes updating many deployments at once

Subscribie is a website builder targeting the niche of users who want to build a subscription based website. It allows users to quickly spin up a website and sell items on subscription. It's GPLv3 licenced. There's a hosted Subscription website version.

Requirements: Ability to scale to 10s, 100s of sites

  • Be able to spin up 10s, 100s (thousands) of sites
  • Allow each of these sites to be customised, based on user input (danger!)

To do this, we take advantage of many of the useful core concepts available in Kubernetes

  • Labels; if you label objects well, you can address it and manipulate it with ease
  • Rolling updates
  • Keeping every site online whilst upgrading the sites to a new version of the software

Basic architecure, a pipeline

It's basically a queue of sites which are waiting to be deployed (each site only takes a new secconds), and then once deployed they can be managed as individual deployments , or upgrades all at the same time though a staged rollout.

The deployment of a new site looks something like this:

  1. Users submit new Subscription website's they want to run
  2. New site data get submitted into a CouchDB instance
  3. A kubernetes polls CouchDB for new sites to be deployed
  4. A new site is born

update-kubernetes-deployments-en-masse
https://swimlanes.io/u/dwAEHeAMS

  • Each site is a kubernetes deployment
  • When a user signs up to create a new site, a Kubernetes deployment manifest is created and submitted to the cluster
  • Each deployment is labeled with subscribie=site, and also the individual site name

This allows us to, when we want to, either individually or en-masse upgrade all subscribie sites once a new versino of Subscribie (via a docker image) becomes available. Updating a fleet of sites simply becomes:

Updating all sites at once to a new version:

for deployment in $(kubectl get deployments -o jsonpath='{.items[*].metadata.name}' -l subscribie=site)
do
  echo "Updating $deployment"
  kubectl --record deployment.apps/$deployment set image deployment.v1.apps/$deployment subscribie=subscribie/subscribie:v0.11
done

The above would upgrade all sites to the new image, whist keeping every site online.

Something went wrong? We can also rollback, which may be as simple as:

Rolling back the update

for deployment in $(kubectl get deployments -o jsonpath='{.items[*].metadata.name}' -l subscribie=site)
do
  echo "Rolling back $deployment"
  kubectl rollout undo deployment $deployment 
done

The above would roll back the update accross all sites. To view the rollout history of a spesific deployment:

kubectl rollout history deployment $deployment

References / Further reading: