Re: [atomic-devel] Kubernetes manual setup

[Colin Walters <walters verbum org>]

On Wed, Feb 21, 2018, at 12:34 PM, Chris Negus wrote:

> In my mind, this means that someone trying out vanilla Kubernetes will 
> start with some OS outside of the Fedora/RHEL/CentOS ecosystem. My 
> question is, is it okay to let this content die? Or should we encourage 
> some way to still manually use Kubernetes on Fedora (Atomic or not)?

This is a pretty deep question with a lot of implications and history involved.
Let me start by stating something that's fairly obvious: Kubernetes is pretty
popular!  There are multiple "distros" of Kubernetes now, including now *two*
products owned by Red Hat.

My personal opinion (now) is that we are primarily producing a base operating system; we need to
support these different Kubernetes distributions; as well as non-Kubernetes
cases.

Including Kubernetes directly in AH was clearly a historical mistake, one that
we are just finally digging ourselves out from.  Part of the issue here is that
in Atomic we are introducing *two* entirely new ways to deliver software;
via system containers as well as package layering now.  And Kubernetes could
be done via both.

In practice, I think system containers for Kubernetes make a lot of sense because
then you get a natural decoupling of the base OS from your container service; they
live at different cadences.  But it's been a challenge because it's quite different
from all of the other ways to run Kube.

Kubernetes *also* is an excellent use case for the Fedora "Modularity" effort - having
just one version of Kubernetes included in the "Everything" repository doesn't make
sense when upstream supports multiple.

The way I would describe this then is that we are providing technology that is intended
to support this use case, and we also need to be participating in issues that cross
the OS/Kubernetes boundary (for example: SELinux policy, host update management).
So I think our current efforts at having system containers for upstream Kubernetes maintained by
"us" are indeed the right approach - we need to ensure the basics work.  Ideally
of course we have more people gaining traction/buy-in in the upstream Kubernetes community.
In practice I think a whole lot of that is mostly "non-Atomic" CentOS/RHEL using RPMs
or just plain dropping binaries in /opt - hopefully we can get some of the upstream
community using the technologies we've developed here.  But it's tricky because
we have other Kubernetes distributions like OpenShift which are also a primary use case.

Anyways another important thing here is: we need to *clearly* distinguish the "dev/test"
scenario from the "production" use case.  For the "dev/test" case there's minikube/minishift
as well as `oc cluster up` which I use a lot (and a major bonus of using Linux as
a desktop including Fedora Atomic Workstation is that you can `oc cluster up` directly
on metal and avoid virt overhead).