Re: [atomic-devel] looking for feedback on running kubernetes in system containers

[Spyros Trigazis <strigazi gmail com>]

Hi,

So far, I have only tried etcd, works well but the only piece missing is
a way to pass TLS credentials which is quite important for certain deployments like ours. My next goal is flannel. Flannel will require
TLS creds as well. To do it, I rebuilt the image to bindmount them. 

To be honest, I didn't try kube components because the version isn't
newer than the one in fedora-atomic and since we don't use ansible
we need some modifications. If kube was newer I would be more
motivated :).

Spyros

On 27 April 2017 at 18:59, Jason Brooks <jbrooks redhat com> wrote:

I've been working on running kubernetes, flannel and etcd in system
containers, and setting up a cluster using the ansible scripts at
kubernetes/contrib.

I wrote a blog post about it here:
https://jebpages.com/2017/04/11/testing-system-containerized-kube-and-friends/

These are my system containers:
https://github.com/jasonbrooks/atomic-system-containers/tree/kube-containers

and my ansible branch:
https://github.com/jasonbrooks/contrib/tree/system-containers/ansible

I've changed the etcd and flannel containers to bind mount config dirs
in /etc, so that the ansible can config them using the same operations
it'd use for non-system containers. I'm using tmpfiles.d to put a link
to the etcdctl from the container into /usr/local/bin/etcd because
ansible expects and needs etcdctl to be on the host to set up the
flannel network, and linking to the etcdctl from the container again
lets us reuse the same ansible operations as for non system container
case.

The kube containers are based on the ones I'm maintaining in the
fedora and centos container registries, and they also get configs from
bind mounted /etc/kubernetes. Like with the etcd container, I'm
creating a link from the kube-apiserver container's kubectl to
/usr/local/bin/kubectl on the host, because the kube-addons service
expects kubectl to be on the host.

I've been using f25-based containers, but this should work with centos
containers, too.

Anyway, if you're interested in this topic, I'd appreciate it if you
gave my post / github forks a look and let me know what you think /
what I'm doing terribly wrong / etc. :)

Thanks, Jason