Re: [atomic-devel] Kubeadm vs. SELinux

[Daniel J Walsh <dwalsh redhat com>]

On 11/22/2016 07:26 PM, Josh Berkus wrote:
> On 11/22/2016 03:27 PM, Clayton Coleman wrote:
>> Copying Devan as well since he's been working with kubeadm for a while.
>>
>>> On Nov 22, 2016, at 5:25 PM, Jason Brooks <jbrooks redhat com> wrote:
>>>
>>>> On Tue, Nov 22, 2016 at 2:38 PM, Daniel J Walsh <dwalsh redhat com> wrote:
>>>>
>>>>
>>>>> On 11/22/2016 05:15 PM, Josh Berkus wrote:
>>>>> Currently, it is not possible to run Kubeadm with SELinux enabled.
>>>>>
>>>>> This is bad; it means that Kubernetes' official installation
>>>>> instructions include `setenforce 0`.  But it's hard to argue the point
>>>>> when a kubeadm install -- soon to be the main install option for
>>>>> Kubernetes, and the only one which currently works on Atomic -- simply
>>>>> doesn't work with SELinux enabled.
>>>>>
>>>>> The current blocker is that kubeadm init will hang forever at this stage:
>>>>>
>>>>> <master/apiclient> created API client, waiting for the control plane to
>>>>> become ready
>>>>>
>>>>>
>>>>> The errors shown in the journal are here:
>>>>>
>>>>> https://gist.github.com/jberkus/4e926c76fbf772ffee4eb774cb0a4c60
>>>>>
>>>>> That's on Fedora 25 Atomic.  I've had the exact same experience on
>>>>> CentOS 7 and RHEL 7, although the error messages are not identical.
>>>>>
>>>>> Seems like this is on us to fix, if we want people to keep SELinux
>>>>> enforcing. I don;t know if we need to push patches to Kubeadm, or to
>>>>> SELinux, or both.
>>>>>
>>>> What AVC's are you seeing?  Where is the bugzilla for this?
>>>>
>>>> ausearch -m avc -ts recent
>>> https://paste.fedoraproject.org/488671/79856867/
>>>
>>> This is from a kubeadm that's packaged up in a copr:
>>> https://copr.fedorainfracloud.org/coprs/jasonbrooks/kube-release/
>>>
>>> The kubernetes project provides rpms for centos and ubuntu, and there
>>> are a few things about the way they pkg it that conflict w/ atomic.
>>> Some more info at
>>> https://jebpages.com/2016/11/01/installing-kubernetes-on-centos-atomic-host-with-kubeadm/.
>>>
> In addition to this, please note that setenforce 0 is not required on
> the workers nodes, just on the master.  The kubelet nodes work fine with
> just relabeling the /var/lib/kubelet directory.
>
> It would be really nice if we could somehow do that relabeling as part
> of the installation package, but I don't see how; it would need to be a
> patch/fork on kubeadm instead.
>
>
Relabeling of /var/lib/kublet should not be necessary either.  What
label is being applied?  What AVC's show up when it is
not applied?

Just setting random labels has the potential for opening security
holes.   We need to get this directory labeled in such
a way that not all containers can read/write it.