Re: [atomic-devel] Kubeadm vs. SELinux

[Devan Goodwin <dgoodwin redhat com>]

Yeah I've been working with the sig on this since the effort started,
and at various points we got setenforce 0 out of there, but it keeps
creeping back in when something else goes wrong.

The discovery pod is not long for this world and should be gone by 1.6
when this goes beta, we made that one unconfined as it was a rush job
and temporary so it could read secret data.

The etcd pod kubeadm sets up uses /var/lib/etcd, I ended up making
this one unconfined as well so we could avoid setenforce 0 at launch,
I have TODO's to figure out what to do here and could use advice. I'm
not sure if I should try to configure via the pod definition, or try
to find ways to set correct context on /var/lib/etcd (which gets
deleted on kubeadm reset) Jason it looks like your PR might address
this so I will test that out today.

At this point we were in the clear, but then the weave guys (who wrote
most of the kubeadm docs) found that users installing the weave
networking addon after cluster setup were hitting selinux issues
preventing kube-dns from coming up, and I believe this was when
setenforce 0 came back into play most recently.

However pmorie's work on selinux coming in 1.5 I think has changed
things quite a bit. I did some testing a couple weeks ago and it looks
like this work fixes the weave issue, as well as the one the discovery
pod was hitting with secrets. In short I think we can get setenforce 0
out of the instructions in 1.5. See notes here:
https://github.com/kubernetes/kubeadm/issues/29

Hope that helps, if there are other changes you guys would like let me
know. I will look through the linked blog post on problems with Atomic
and see what we can do there ASAP.

On Tue, Nov 22, 2016 at 8:37 PM, Jason Brooks <jbrooks redhat com> wrote:
> On Tue, Nov 22, 2016 at 4:26 PM, Josh Berkus <jberkus redhat com> wrote:
>> On 11/22/2016 03:27 PM, Clayton Coleman wrote:
>>> Copying Devan as well since he's been working with kubeadm for a while.
>>>
>>>> On Nov 22, 2016, at 5:25 PM, Jason Brooks <jbrooks redhat com> wrote:
>>>>
>>>>> On Tue, Nov 22, 2016 at 2:38 PM, Daniel J Walsh <dwalsh redhat com> wrote:
>>>>>
>>>>>
>>>>>> On 11/22/2016 05:15 PM, Josh Berkus wrote:
>>>>>> Currently, it is not possible to run Kubeadm with SELinux enabled.
>>>>>>
>>>>>> This is bad; it means that Kubernetes' official installation
>>>>>> instructions include `setenforce 0`.  But it's hard to argue the point
>>>>>> when a kubeadm install -- soon to be the main install option for
>>>>>> Kubernetes, and the only one which currently works on Atomic -- simply
>>>>>> doesn't work with SELinux enabled.
>>>>>>
>>>>>> The current blocker is that kubeadm init will hang forever at this stage:
>>>>>>
>>>>>> <master/apiclient> created API client, waiting for the control plane to
>>>>>> become ready
>>>>>>
>>>>>>
>>>>>> The errors shown in the journal are here:
>>>>>>
>>>>>> https://gist.github.com/jberkus/4e926c76fbf772ffee4eb774cb0a4c60
>>>>>>
>>>>>> That's on Fedora 25 Atomic.  I've had the exact same experience on
>>>>>> CentOS 7 and RHEL 7, although the error messages are not identical.
>>>>>>
>>>>>> Seems like this is on us to fix, if we want people to keep SELinux
>>>>>> enforcing. I don;t know if we need to push patches to Kubeadm, or to
>>>>>> SELinux, or both.
>>>>>>
>>>>>
>>>>> What AVC's are you seeing?  Where is the bugzilla for this?
>>>>>
>>>>> ausearch -m avc -ts recent
>>>>
>>>> https://paste.fedoraproject.org/488671/79856867/
>>>>
>>>> This is from a kubeadm that's packaged up in a copr:
>>>> https://copr.fedorainfracloud.org/coprs/jasonbrooks/kube-release/
>>>>
>>>> The kubernetes project provides rpms for centos and ubuntu, and there
>>>> are a few things about the way they pkg it that conflict w/ atomic.
>>>> Some more info at
>>>> https://jebpages.com/2016/11/01/installing-kubernetes-on-centos-atomic-host-with-kubeadm/.
>>>>
>>
>> In addition to this, please note that setenforce 0 is not required on
>> the workers nodes, just on the master.  The kubelet nodes work fine with
>> just relabeling the /var/lib/kubelet directory.
>>
>> It would be really nice if we could somehow do that relabeling as part
>> of the installation package, but I don't see how; it would need to be a
>> patch/fork on kubeadm instead.
>
> The problem containers are etcd and kube-discovery, they're set to
> type unconfined_t to work around selinux, but I believe the correct
> type is spc_t. Changing to spc_t allows the install to continue w/o
> disabling selinux.
>
> I sent a PR to change this: https://github.com/kubernetes/kubernetes/pull/37327
>
>>
>>
>> --
>> --
>> Josh Berkus
>> Project Atomic
>> Red Hat OSAS