Re: [atomic-devel] Kubernetes in containers

[Colin Walters <walters verbum org>]

> What do you guys think about switching to Kubernetes-in-a-container
> by default?  We should likely keep the Kubernetes in the Atomic Host
> available for some time, but how long?
> 
> I know this would affect some of the Ansible and documentation work
> out there.

Looking around a bit more, one thing too is that there's also
work on containerizing OpenShift (which includes Kubernetes at the core).
I suspect a lot of users would like to make use of some of the OpenShift
additions, such as source-to-image on an Atomic Host cluster.

Moving to a consistent out-of-host Kubernetes would allow us to
more easily support that.

This afternoon I fired up a new Fedora Atomic 22 host and
played around with:
https://docs.openshift.org/latest/getting_started/administrators.html

It quickly became very clear to me that the containerized OpenShift
is not the primary focus of the documentation so far.

I have no idea why the docs suggest -v /tmp/kubernetes:/tmp/kubernetes.
The predictable-filename-in-tmp security issue aside, it seems
we really want -v /var/lib/openshift:/var/lib/openshift, as that's where
critical things like the auto-generated keys live.

(It might help if we changed it to use /etc/origin on the host, i.e.
 keep the same file paths as the non-containerized version, so that
 it's easier to migrate).

Pairing with that, we could also patch the Dockerfile to make
use of the /usr/bin/atomic LABEL INSTALL.

There's a lot of effort here:

 - Figuring out how to deduplicate some of the packaging work
   with existing Kubernetes
 - More cleanly separate the "develop OpenShift" mode the
   Vagrantfile and Dockerfile focus on to something separate
 - How to manage upgrades around persistent config files for the containers
 - Updating all of the documentation
 - More I'm forgetting

But the end result seems like it would make sense; other thoughts
with respect to containerized Kubernetes and OpenShift?