Data Persistence In Containerised Environment With AutoPoilot

At the moment you can't talk about containerisation without thinking about a hole host of dependant topics:

• Auto discovery - Container Name Services & what do you mean port 80 is already in use?
• Managing persistent storage in a containerised environment - Where do I put my database & oh damn where was my database?
• Scaling Horizontally - without being tied to one provider, possible?
• Replication, Snapshots and recovery.
• Auto recovery & Container Life-cycles (see Autopilot Pattern)

That's a lot! Yet it's just containers right? It's time we talked about containers in production seriously and go balls-deep into the messy world of production containers. This is a deep dive into the history, identifying interesting projects and looking at some real-world deployments.

Auto Recovery, Discovery & Scale

Why do we need this?

If we're going to scale a swam of instances, and allow any of them to break at any time, then we need some way for them to come-back-up, reconfigure and make themselves known to an environment. It's easy to forget data persistence at this point, because it's hard, therefore it tends to get ignored, we're only human (see talk)

Note: Throughout this article we'll make reference to a fictitious LAMP-stack type application because it's simple and everyone gets it.

When you read deeply into containers by comparing the various implementations, you quickly realise there's healthy debate about where & how to put persistent data when using containers.

ZFS has won the argument in many respects for enterprise persistent data storage in the same way that Solaris Zones seem to have a much more evolved implementation of containerisation compared to Linux cgroups. Note, the concept of containerisation (which is older than you may think- at-least as early as 1982 ).

If you get excited / stampeded by Brian Cantrill's talks on these topics enough your quickly convinced of this and go away with an appreciation of the history behind chroot, jails, and eventually what we now refer to as containers.

"The work that was prior to jails, was chroot.
chroot was kinda the first way we had of containing a program, and what chroot does is, it changes your root filesystem to be a specified directory in the filesystem so that now your new 'root' of this process, is this specified
directory."

...One funny detail about chroot is, we don't know why exactly why it was done! In fact, the most complete explanation as to why it was done is in [this paper] in the footnote.

tbc..
GHOST_URL/docker-vs-solaris-containers-tldr-its-a-question-of-kernel-not-application-2/