Re: [atomic-devel] Kubeadm vs. SELinux

[Devan Goodwin <dgoodwin redhat com>]

On Wed, Nov 23, 2016 at 9:44 AM, Daniel J Walsh <dwalsh redhat com> wrote:
>
>
> On 11/22/2016 07:37 PM, Jason Brooks wrote:
>> On Tue, Nov 22, 2016 at 4:26 PM, Josh Berkus <jberkus redhat com> wrote:
>>> On 11/22/2016 03:27 PM, Clayton Coleman wrote:
>>>> Copying Devan as well since he's been working with kubeadm for a while.
>>>>
>>>>> On Nov 22, 2016, at 5:25 PM, Jason Brooks <jbrooks redhat com> wrote:
>>>>>
>>>>>> On Tue, Nov 22, 2016 at 2:38 PM, Daniel J Walsh <dwalsh redhat com> wrote:
>>>>>>
>>>>>>
>>>>>>> On 11/22/2016 05:15 PM, Josh Berkus wrote:
>>>>>>> Currently, it is not possible to run Kubeadm with SELinux enabled.
>>>>>>>
>>>>>>> This is bad; it means that Kubernetes' official installation
>>>>>>> instructions include `setenforce 0`.  But it's hard to argue the point
>>>>>>> when a kubeadm install -- soon to be the main install option for
>>>>>>> Kubernetes, and the only one which currently works on Atomic -- simply
>>>>>>> doesn't work with SELinux enabled.
>>>>>>>
>>>>>>> The current blocker is that kubeadm init will hang forever at this stage:
>>>>>>>
>>>>>>> <master/apiclient> created API client, waiting for the control plane to
>>>>>>> become ready
>>>>>>>
>>>>>>>
>>>>>>> The errors shown in the journal are here:
>>>>>>>
>>>>>>> https://gist.github.com/jberkus/4e926c76fbf772ffee4eb774cb0a4c60
>>>>>>>
>>>>>>> That's on Fedora 25 Atomic.  I've had the exact same experience on
>>>>>>> CentOS 7 and RHEL 7, although the error messages are not identical.
>>>>>>>
>>>>>>> Seems like this is on us to fix, if we want people to keep SELinux
>>>>>>> enforcing. I don;t know if we need to push patches to Kubeadm, or to
>>>>>>> SELinux, or both.
>>>>>>>
>>>>>> What AVC's are you seeing?  Where is the bugzilla for this?
>>>>>>
>>>>>> ausearch -m avc -ts recent
>>>>> https://paste.fedoraproject.org/488671/79856867/
>>>>>
>>>>> This is from a kubeadm that's packaged up in a copr:
>>>>> https://copr.fedorainfracloud.org/coprs/jasonbrooks/kube-release/
>>>>>
>>>>> The kubernetes project provides rpms for centos and ubuntu, and there
>>>>> are a few things about the way they pkg it that conflict w/ atomic.
>>>>> Some more info at
>>>>> https://jebpages.com/2016/11/01/installing-kubernetes-on-centos-atomic-host-with-kubeadm/.
>>>>>
>>> In addition to this, please note that setenforce 0 is not required on
>>> the workers nodes, just on the master.  The kubelet nodes work fine with
>>> just relabeling the /var/lib/kubelet directory.
>>>
>>> It would be really nice if we could somehow do that relabeling as part
>>> of the installation package, but I don't see how; it would need to be a
>>> patch/fork on kubeadm instead.
>> The problem containers are etcd and kube-discovery, they're set to
>> type unconfined_t to work around selinux, but I believe the correct
>> type is spc_t. Changing to spc_t allows the install to continue w/o
>> disabling selinux.
>>
>> I sent a PR to change this: https://github.com/kubernetes/kubernetes/pull/37327
> Correct although it would be nice to get new types for these containers
> perhaps
> we could build policy for each one that is not unconfined but still
> allow other processes
> to communicate with them.  etcd_t, probbaly needs very limited access on
> a host system
> and I have no idea what kube-discovery does.

Kube-disocvery is the key piece that lets you run a very short kubeadm
join command, it's just a simple pod that offers up signed data
containing a list of api servers, and the CA cert to talk to them.
It's a temporary solution for kubeadm alpha and is disappearing
entirely, replaced by config maps in core k8s.

We unconfined it in the rush for alpha to avoid shipping instructions
that involved setenforce 0. It couldn't read secrets by default,
though this is fixed in 1.5 with pmorie's work I believe.

etcd container just needs access to /var/lib/etcd, suggestions on most
correct way to handle this would be very welcome.