Re: [atomic-devel] Kubeadm vs. SELinux

[Daniel J Walsh <dwalsh redhat com>]

If anyone has an SELinux issue, please PING me immediately.  We should
not have a

Red Hat Document that says you need to disable SELinux enforcing,
especially if this

is a security related feature.  For example running containers or the
management system

for running containers.


On 11/23/2016 07:12 AM, Devan Goodwin wrote:
> Yeah I've been working with the sig on this since the effort started,
> and at various points we got setenforce 0 out of there, but it keeps
> creeping back in when something else goes wrong.
>
> The discovery pod is not long for this world and should be gone by 1.6
> when this goes beta, we made that one unconfined as it was a rush job
> and temporary so it could read secret data.
>
> The etcd pod kubeadm sets up uses /var/lib/etcd, I ended up making
> this one unconfined as well so we could avoid setenforce 0 at launch,
> I have TODO's to figure out what to do here and could use advice. I'm
> not sure if I should try to configure via the pod definition, or try
> to find ways to set correct context on /var/lib/etcd (which gets
> deleted on kubeadm reset) Jason it looks like your PR might address
> this so I will test that out today.
>
> At this point we were in the clear, but then the weave guys (who wrote
> most of the kubeadm docs) found that users installing the weave
> networking addon after cluster setup were hitting selinux issues
> preventing kube-dns from coming up, and I believe this was when
> setenforce 0 came back into play most recently.
>
> However pmorie's work on selinux coming in 1.5 I think has changed
> things quite a bit. I did some testing a couple weeks ago and it looks
> like this work fixes the weave issue, as well as the one the discovery
> pod was hitting with secrets. In short I think we can get setenforce 0
> out of the instructions in 1.5. See notes here:
> https://github.com/kubernetes/kubeadm/issues/29
>
> Hope that helps, if there are other changes you guys would like let me
> know. I will look through the linked blog post on problems with Atomic
> and see what we can do there ASAP.
>
> On Tue, Nov 22, 2016 at 8:37 PM, Jason Brooks <jbrooks redhat com> wrote:
>> On Tue, Nov 22, 2016 at 4:26 PM, Josh Berkus <jberkus redhat com> wrote:
>>> On 11/22/2016 03:27 PM, Clayton Coleman wrote:
>>>> Copying Devan as well since he's been working with kubeadm for a while.
>>>>
>>>>> On Nov 22, 2016, at 5:25 PM, Jason Brooks <jbrooks redhat com> wrote:
>>>>>
>>>>>> On Tue, Nov 22, 2016 at 2:38 PM, Daniel J Walsh <dwalsh redhat com> wrote:
>>>>>>
>>>>>>
>>>>>>> On 11/22/2016 05:15 PM, Josh Berkus wrote:
>>>>>>> Currently, it is not possible to run Kubeadm with SELinux enabled.
>>>>>>>
>>>>>>> This is bad; it means that Kubernetes' official installation
>>>>>>> instructions include `setenforce 0`.  But it's hard to argue the point
>>>>>>> when a kubeadm install -- soon to be the main install option for
>>>>>>> Kubernetes, and the only one which currently works on Atomic -- simply
>>>>>>> doesn't work with SELinux enabled.
>>>>>>>
>>>>>>> The current blocker is that kubeadm init will hang forever at this stage:
>>>>>>>
>>>>>>> <master/apiclient> created API client, waiting for the control plane to
>>>>>>> become ready
>>>>>>>
>>>>>>>
>>>>>>> The errors shown in the journal are here:
>>>>>>>
>>>>>>> https://gist.github.com/jberkus/4e926c76fbf772ffee4eb774cb0a4c60
>>>>>>>
>>>>>>> That's on Fedora 25 Atomic.  I've had the exact same experience on
>>>>>>> CentOS 7 and RHEL 7, although the error messages are not identical.
>>>>>>>
>>>>>>> Seems like this is on us to fix, if we want people to keep SELinux
>>>>>>> enforcing. I don;t know if we need to push patches to Kubeadm, or to
>>>>>>> SELinux, or both.
>>>>>>>
>>>>>> What AVC's are you seeing?  Where is the bugzilla for this?
>>>>>>
>>>>>> ausearch -m avc -ts recent
>>>>> https://paste.fedoraproject.org/488671/79856867/
>>>>>
>>>>> This is from a kubeadm that's packaged up in a copr:
>>>>> https://copr.fedorainfracloud.org/coprs/jasonbrooks/kube-release/
>>>>>
>>>>> The kubernetes project provides rpms for centos and ubuntu, and there
>>>>> are a few things about the way they pkg it that conflict w/ atomic.
>>>>> Some more info at
>>>>> https://jebpages.com/2016/11/01/installing-kubernetes-on-centos-atomic-host-with-kubeadm/.
>>>>>
>>> In addition to this, please note that setenforce 0 is not required on
>>> the workers nodes, just on the master.  The kubelet nodes work fine with
>>> just relabeling the /var/lib/kubelet directory.
>>>
>>> It would be really nice if we could somehow do that relabeling as part
>>> of the installation package, but I don't see how; it would need to be a
>>> patch/fork on kubeadm instead.
>> The problem containers are etcd and kube-discovery, they're set to
>> type unconfined_t to work around selinux, but I believe the correct
>> type is spc_t. Changing to spc_t allows the install to continue w/o
>> disabling selinux.
>>
>> I sent a PR to change this: https://github.com/kubernetes/kubernetes/pull/37327
>>
>>>
>>> --
>>> --
>>> Josh Berkus
>>> Project Atomic
>>> Red Hat OSAS