Re: [atomic-devel] Kubeadm vs. SELinux

[Daniel J Walsh <dwalsh redhat com>]

On 11/23/2016 10:46 AM, Daniel J Walsh wrote:
>
> On 11/23/2016 10:33 AM, Devan Goodwin wrote:
>> On Wed, Nov 23, 2016 at 9:44 AM, Daniel J Walsh <dwalsh redhat com> wrote:
>>> On 11/22/2016 07:37 PM, Jason Brooks wrote:
>>>> On Tue, Nov 22, 2016 at 4:26 PM, Josh Berkus <jberkus redhat com> wrote:
>>>>> On 11/22/2016 03:27 PM, Clayton Coleman wrote:
>>>>>> Copying Devan as well since he's been working with kubeadm for a while.
>>>>>>
>>>>>>> On Nov 22, 2016, at 5:25 PM, Jason Brooks <jbrooks redhat com> wrote:
>>>>>>>
>>>>>>>> On Tue, Nov 22, 2016 at 2:38 PM, Daniel J Walsh <dwalsh redhat com> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>> On 11/22/2016 05:15 PM, Josh Berkus wrote:
>>>>>>>>> Currently, it is not possible to run Kubeadm with SELinux enabled.
>>>>>>>>>
>>>>>>>>> This is bad; it means that Kubernetes' official installation
>>>>>>>>> instructions include `setenforce 0`.  But it's hard to argue the point
>>>>>>>>> when a kubeadm install -- soon to be the main install option for
>>>>>>>>> Kubernetes, and the only one which currently works on Atomic -- simply
>>>>>>>>> doesn't work with SELinux enabled.
>>>>>>>>>
>>>>>>>>> The current blocker is that kubeadm init will hang forever at this stage:
>>>>>>>>>
>>>>>>>>> <master/apiclient> created API client, waiting for the control plane to
>>>>>>>>> become ready
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> The errors shown in the journal are here:
>>>>>>>>>
>>>>>>>>> https://gist.github.com/jberkus/4e926c76fbf772ffee4eb774cb0a4c60
>>>>>>>>>
>>>>>>>>> That's on Fedora 25 Atomic.  I've had the exact same experience on
>>>>>>>>> CentOS 7 and RHEL 7, although the error messages are not identical.
>>>>>>>>>
>>>>>>>>> Seems like this is on us to fix, if we want people to keep SELinux
>>>>>>>>> enforcing. I don;t know if we need to push patches to Kubeadm, or to
>>>>>>>>> SELinux, or both.
>>>>>>>>>
>>>>>>>> What AVC's are you seeing?  Where is the bugzilla for this?
>>>>>>>>
>>>>>>>> ausearch -m avc -ts recent
>>>>>>> https://paste.fedoraproject.org/488671/79856867/
>>>>>>>
>>>>>>> This is from a kubeadm that's packaged up in a copr:
>>>>>>> https://copr.fedorainfracloud.org/coprs/jasonbrooks/kube-release/
>>>>>>>
>>>>>>> The kubernetes project provides rpms for centos and ubuntu, and there
>>>>>>> are a few things about the way they pkg it that conflict w/ atomic.
>>>>>>> Some more info at
>>>>>>> https://jebpages.com/2016/11/01/installing-kubernetes-on-centos-atomic-host-with-kubeadm/.
>>>>>>>
>>>>> In addition to this, please note that setenforce 0 is not required on
>>>>> the workers nodes, just on the master.  The kubelet nodes work fine with
>>>>> just relabeling the /var/lib/kubelet directory.
>>>>>
>>>>> It would be really nice if we could somehow do that relabeling as part
>>>>> of the installation package, but I don't see how; it would need to be a
>>>>> patch/fork on kubeadm instead.
>>>> The problem containers are etcd and kube-discovery, they're set to
>>>> type unconfined_t to work around selinux, but I believe the correct
>>>> type is spc_t. Changing to spc_t allows the install to continue w/o
>>>> disabling selinux.
>>>>
>>>> I sent a PR to change this: https://github.com/kubernetes/kubernetes/pull/37327
>>> Correct although it would be nice to get new types for these containers
>>> perhaps
>>> we could build policy for each one that is not unconfined but still
>>> allow other processes
>>> to communicate with them.  etcd_t, probbaly needs very limited access on
>>> a host system
>>> and I have no idea what kube-discovery does.
>> Kube-disocvery is the key piece that lets you run a very short kubeadm
>> join command, it's just a simple pod that offers up signed data
>> containing a list of api servers, and the CA cert to talk to them.
>> It's a temporary solution for kubeadm alpha and is disappearing
>> entirely, replaced by config maps in core k8s.
>>
>> We unconfined it in the rush for alpha to avoid shipping instructions
>> that involved setenforce 0. It couldn't read secrets by default,
>> though this is fixed in 1.5 with pmorie's work I believe.
>>
>> etcd container just needs access to /var/lib/etcd, suggestions on most
>> correct way to handle this would be very welcome.
> etcd could probably run with a standard container, but have
> /var/lib/etcd mounted in with the equivalent of
>
> docker run -v /var/lib/etcd:/var/lib/etcd:Z
>
> As long as everyone else talks to it via the network SELinux confinement
> should work fine.
>
>
Sounds like we should just ignore kube-discovery and run it as spc_t for
now until it disappears.