Kubernetes updating many deployments at once
Subscribie is a website builder targeting the niche of users who want to build a subscription based website. It allows users to quickly spin up a website and sell items on subscription. It's GPLv3 licenced. There's a hosted Subscription website version.
Requirements: Ability to scale to 10s, 100s of sites
- Be able to spin up 10s, 100s (thousands) of sites
- Allow each of these sites to be customised, based on user input (danger!)
To do this, we take advantage of many of the useful core concepts available in Kubernetes
- Labels; if you label objects well, you can address it and manipulate it with ease
- Rolling updates
- Keeping every site online whilst upgrading the sites to a new version of the software
Basic architecure, a pipeline
It's basically a queue of sites which are waiting to be deployed (each site only takes a new secconds), and then once deployed they can be managed as individual deployments , or upgrades all at the same time though a staged rollout.
The deployment of a new site looks something like this:
- Users submit new Subscription website's they want to run
- New site data get submitted into a CouchDB instance
- A kubernetes polls CouchDB for new sites to be deployed
- A new site is born
https://swimlanes.io/u/dwAEHeAMS
- Each site is a kubernetes deployment
- When a user signs up to create a new site, a Kubernetes deployment manifest is created and submitted to the cluster
- Each deployment is labeled with
subscribie=site, and also the individual site name
This allows us to, when we want to, either individually or en-masse upgrade all subscribie sites once a new versino of Subscribie (via a docker image) becomes available. Updating a fleet of sites simply becomes:
Updating all sites at once to a new version:
for deployment in $(kubectl get deployments -o jsonpath='{.items[*].metadata.name}' -l subscribie=site)
do
echo "Updating $deployment"
kubectl --record deployment.apps/$deployment set image deployment.v1.apps/$deployment subscribie=subscribie/subscribie:v0.11
done
The above would upgrade all sites to the new image, whist keeping every site online.
Something went wrong? We can also rollback, which may be as simple as:
Rolling back the update
for deployment in $(kubectl get deployments -o jsonpath='{.items[*].metadata.name}' -l subscribie=site)
do
echo "Rolling back $deployment"
kubectl rollout undo deployment $deployment
done
The above would roll back the update accross all sites. To view the rollout history of a spesific deployment:
kubectl rollout history deployment $deployment
References / Further reading:
- https://shapeshed.com/rolling-deployments-with-kubernetes/#recording-history
- Cert-Manager for automated tls certificates for each site (shout out to James and contributers for making the complicated possible)
- Rook / Ceph for persistant storage with readwrite many volumes https://rook.io/docs/rook/v1.0/ceph-quickstart.html