Re: [atomic-devel] Kubeadm vs. SELinux

[Daniel J Walsh <dwalsh redhat com>]

On 11/22/2016 07:37 PM, Jason Brooks wrote:
> On Tue, Nov 22, 2016 at 4:26 PM, Josh Berkus <jberkus redhat com> wrote:
>> On 11/22/2016 03:27 PM, Clayton Coleman wrote:
>>> Copying Devan as well since he's been working with kubeadm for a while.
>>>
>>>> On Nov 22, 2016, at 5:25 PM, Jason Brooks <jbrooks redhat com> wrote:
>>>>
>>>>> On Tue, Nov 22, 2016 at 2:38 PM, Daniel J Walsh <dwalsh redhat com> wrote:
>>>>>
>>>>>
>>>>>> On 11/22/2016 05:15 PM, Josh Berkus wrote:
>>>>>> Currently, it is not possible to run Kubeadm with SELinux enabled.
>>>>>>
>>>>>> This is bad; it means that Kubernetes' official installation
>>>>>> instructions include `setenforce 0`.  But it's hard to argue the point
>>>>>> when a kubeadm install -- soon to be the main install option for
>>>>>> Kubernetes, and the only one which currently works on Atomic -- simply
>>>>>> doesn't work with SELinux enabled.
>>>>>>
>>>>>> The current blocker is that kubeadm init will hang forever at this stage:
>>>>>>
>>>>>> <master/apiclient> created API client, waiting for the control plane to
>>>>>> become ready
>>>>>>
>>>>>>
>>>>>> The errors shown in the journal are here:
>>>>>>
>>>>>> https://gist.github.com/jberkus/4e926c76fbf772ffee4eb774cb0a4c60
>>>>>>
>>>>>> That's on Fedora 25 Atomic.  I've had the exact same experience on
>>>>>> CentOS 7 and RHEL 7, although the error messages are not identical.
>>>>>>
>>>>>> Seems like this is on us to fix, if we want people to keep SELinux
>>>>>> enforcing. I don;t know if we need to push patches to Kubeadm, or to
>>>>>> SELinux, or both.
>>>>>>
>>>>> What AVC's are you seeing?  Where is the bugzilla for this?
>>>>>
>>>>> ausearch -m avc -ts recent
>>>> https://paste.fedoraproject.org/488671/79856867/
>>>>
>>>> This is from a kubeadm that's packaged up in a copr:
>>>> https://copr.fedorainfracloud.org/coprs/jasonbrooks/kube-release/
>>>>
>>>> The kubernetes project provides rpms for centos and ubuntu, and there
>>>> are a few things about the way they pkg it that conflict w/ atomic.
>>>> Some more info at
>>>> https://jebpages.com/2016/11/01/installing-kubernetes-on-centos-atomic-host-with-kubeadm/.
>>>>
>> In addition to this, please note that setenforce 0 is not required on
>> the workers nodes, just on the master.  The kubelet nodes work fine with
>> just relabeling the /var/lib/kubelet directory.
>>
>> It would be really nice if we could somehow do that relabeling as part
>> of the installation package, but I don't see how; it would need to be a
>> patch/fork on kubeadm instead.
> The problem containers are etcd and kube-discovery, they're set to
> type unconfined_t to work around selinux, but I believe the correct
> type is spc_t. Changing to spc_t allows the install to continue w/o
> disabling selinux.
>
> I sent a PR to change this: https://github.com/kubernetes/kubernetes/pull/37327
Correct although it would be nice to get new types for these containers
perhaps
we could build policy for each one that is not unconfined but still
allow other processes
to communicate with them.  etcd_t, probbaly needs very limited access on
a host system
and I have no idea what kube-discovery does.
>
>>
>> --
>> --
>> Josh Berkus
>> Project Atomic
>> Red Hat OSAS