Re: [atomic-devel] Moving towards containerized Kube/layered packages

[Daniel J Walsh <dwalsh redhat com>]

I like the idea.


On 08/22/2016 10:13 AM, Colin Walters wrote:
> Hi, I'd like to propose a fairly fundamental rework of Atomic Host.  TL;DR:
>
> - Move towards "system containers" (or layered packages) for flannel/etcd
> - Move towards containers (system, or Docker) for kubernetes-master
> - Move towards layered packages for kubernetes-node and storage (ceph/gluster)
>
> In progress PR:
>
> https://github.com/CentOS/sig-atomic-buildscripts/pull/144
>
> There are advantages to this and disadvantages; I think we'll have some
> short term transition pain, but past that short term the advantages
> outweigh the disadvantages a lot.
>
> == Advantage: Version flexibility ==
>
> etcd for should really have its own identity in a clustered environment, and
> not necessarily roll fowards/backwards with the underlying host version.  I've
> had users report things like hitting an e.g. Kubernetes or Docker issue and rolling back their
> host, which rolled back etcd as well, but the store isn't forwards-compatible,
> which then breaks.  There's also a transition to etcd2 coming, which again
> one should really manage distinct from host upgrades.
>
> Another major example is that while we chose to include Kubernetes in
> the host, it's a fast moving project, and many people want to use a newer
> version, or a different distribution like OpenShift.  The version flexibility
> also applies to other components like Ceph/Gluster and flannel.
>
> == Advantage: Size and fit to purpose ==
>
> We included things like the Ceph and GlusterFS drivers in the base
> host, but that was before we had layered packages, and there's
> also continuing progress on containerized drivers.  If one is using
> an existing IaaS environment like OpenStack or AWS, many users
> want to reuse Cinder/AWS, rather than maintaining their own storage.
>
> Similarly, while flannel is a good general purpose tool, there are
> lots of alternatives, and some users already have existing SDN solutions.
>
> == Disadvantage: More assembly required ==
>
> This is a superficial disadvantage I think - in practice, since we didn't
> pick a single official installation/upgrade system (like OpenShift has
> openshift-ansible), if you want to run a Kubernetes cluster, you need
> to do a lot of assembly anyways.  Adding a bit more to that I suspect
> isn't going to be too bad for most users.
>
> Down the line I'd like to revisit the installation/upgrade story - there's
> work happening upstream in
> https://github.com/kubernetes/contrib/tree/master/ansible
> and I think there's also interest and some work in
> having parts of openshift-ansible be available for baseline Kubernetes
> and accessible on Galaxy etc.
>
> == Disadvantage: Dependency on new tooling ==
>
> Both `rpm-ostree pkg-add` and `atomic install --system` are pretty new.
> They both could use better documentation, more real world testing, and
> in particular haven't gained management tool awareness yet (for example, they need
> better Ansible support). 
>
> == Summary ==
>
> If people agree, I'd like to merge the PR pretty soon and do a new CentOS AH Alpha,
> and we can collaborate on updating docs/tools around this.  For Fedora...it's
> probably simplest to leave 24 alone and just do 25 for now.
>
> What I'd like to focus on is having AH be more of a good "building block"
> rather than positioning it as a complete solution.  We ensure that the base
> Docker/kernel/SELinux/systemd block works together, system management tools
> work, and look at working more in the upstream Kubernetes (and OpenShift)
> communities, particularly around Ansible.
>