Re: [atomic-devel] Kubernetes in containers

[Clayton Coleman <ccoleman redhat com>]

The install docs are wrong, see
https://github.com/openshift/openshift-docs/pull/959

Other comments inline

On Wed, Sep 16, 2015 at 6:21 PM, Colin Walters <walters verbum org> wrote:
>> What do you guys think about switching to Kubernetes-in-a-container
>> by default?  We should likely keep the Kubernetes in the Atomic Host
>> available for some time, but how long?
>>
>> I know this would affect some of the Ansible and documentation work
>> out there.
>
> Looking around a bit more, one thing too is that there's also
> work on containerizing OpenShift (which includes Kubernetes at the core).
> I suspect a lot of users would like to make use of some of the OpenShift
> additions, such as source-to-image on an Atomic Host cluster.
>
> Moving to a consistent out-of-host Kubernetes would allow us to
> more easily support that.
>
> This afternoon I fired up a new Fedora Atomic 22 host and
> played around with:
> https://docs.openshift.org/latest/getting_started/administrators.html
>
> It quickly became very clear to me that the containerized OpenShift
> is not the primary focus of the documentation so far.
>
> I have no idea why the docs suggest -v /tmp/kubernetes:/tmp/kubernetes.
> The predictable-filename-in-tmp security issue aside, it seems
> we really want -v /var/lib/openshift:/var/lib/openshift, as that's where
> critical things like the auto-generated keys live.
>
> (It might help if we changed it to use /etc/origin on the host, i.e.
>  keep the same file paths as the non-containerized version, so that
>  it's easier to migrate).


We don't want to encourage local host paths in the future.  For now it's fine


>
> Pairing with that, we could also patch the Dockerfile to make
> use of the /usr/bin/atomic LABEL INSTALL.


I believe Aaron opened a PR for that, but it was for the wrong Dockerfile.


>
> There's a lot of effort here:
>
>  - Figuring out how to deduplicate some of the packaging work
>    with existing Kubernetes
>  - More cleanly separate the "develop OpenShift" mode the
>    Vagrantfile and Dockerfile focus on to something separate

We have "official" images that we've been publishing for a long time -
those Dockerfiles are not intended to be for development.  The one in
the root of the repo is a remnant and should be removed.  There should
be an official "run vagrant" image.  I was hoping the fedora CDK would
be done by now, but it is not.


>  - How to manage upgrades around persistent config files for the containers


The bulk will move into etcd and be defaulted, the remainder
(connection info, admin secrets) is probably going to have to come
from the host.  There will probably have to be a "slurp in config"
option on master startup forever.



>  - Updating all of the documentation
>  - More I'm forgetting
>
> But the end result seems like it would make sense; other thoughts
> with respect to containerized Kubernetes and OpenShift?

It's going to be a while before Kube supports pure additive function.
There are a fair number of hurdles to climb there - admission
controllers, landing the authenticators and authorizers, and then
config injection.  I'd hazard it's at least 3-6 months, and a lot of
the refactors in Kube recently are going to slow that down (because
we're changing things we probably didn't *need* to change).  For the
vast majority of deployments, containerized Kube with OS on top is
interesting, but not truly necessary (3 masters + 3 etcds in a fashion
similar to https://github.com/openshift/origin/blob/master/examples/ha/openshift-ha.yaml).
Containerized Kubelet as a standalone should probably be the first
step - but until config changes land, setting several hundred node
flags on every docker container is not my idea of fun.