Re: [atomic-devel] looking for feedback on running kubernetes in system containers

[Jason Brooks <jbrooks redhat com>]

On Fri, Apr 28, 2017 at 1:05 AM, Spyros Trigazis <strigazi gmail com> wrote:
> Hi,
>
> So far, I have only tried etcd, works well but the only piece missing is
> a way to pass TLS credentials which is quite important for certain
> deployments like ours. My next goal is flannel. Flannel will require
> TLS creds as well. To do it, I rebuilt the image to bindmount them.

The ansible scripts handle this, and they put the certs in
/etc/etcd/certs -- I'm bind mounting /etc/etcd to accommodate this.
Where do you put your certs?

It's a similar situation w/ flannel, w/ certs in /etc/flanneld/certs.

>
> To be honest, I didn't try kube components because the version isn't
> newer than the one in fedora-atomic and since we don't use ansible
> we need some modifications. If kube was newer I would be more
> motivated :).

Good idea. I just built rawhide versions of these containers that you
can check out by swapping the tag fc25 for rawhide. They have kube
1.6.1. I haven't tested them yet, though.

Jason

>
> Spyros
>
> On 27 April 2017 at 18:59, Jason Brooks <jbrooks redhat com> wrote:
>>
>> I've been working on running kubernetes, flannel and etcd in system
>> containers, and setting up a cluster using the ansible scripts at
>> kubernetes/contrib.
>>
>> I wrote a blog post about it here:
>>
>> https://jebpages.com/2017/04/11/testing-system-containerized-kube-and-friends/
>>
>> These are my system containers:
>>
>> https://github.com/jasonbrooks/atomic-system-containers/tree/kube-containers
>>
>> and my ansible branch:
>> https://github.com/jasonbrooks/contrib/tree/system-containers/ansible
>>
>> I've changed the etcd and flannel containers to bind mount config dirs
>> in /etc, so that the ansible can config them using the same operations
>> it'd use for non-system containers. I'm using tmpfiles.d to put a link
>> to the etcdctl from the container into /usr/local/bin/etcd because
>> ansible expects and needs etcdctl to be on the host to set up the
>> flannel network, and linking to the etcdctl from the container again
>> lets us reuse the same ansible operations as for non system container
>> case.
>>
>> The kube containers are based on the ones I'm maintaining in the
>> fedora and centos container registries, and they also get configs from
>> bind mounted /etc/kubernetes. Like with the etcd container, I'm
>> creating a link from the kube-apiserver container's kubectl to
>> /usr/local/bin/kubectl on the host, because the kube-addons service
>> expects kubectl to be on the host.
>>
>> I've been using f25-based containers, but this should work with centos
>> containers, too.
>>
>> Anyway, if you're interested in this topic, I'd appreciate it if you
>> gave my post / github forks a look and let me know what you think /
>> what I'm doing terribly wrong / etc. :)
>>
>> Thanks, Jason
>>
>