Re: [atomic] Kubernetes and /etc/hosts

[Jonathan Rogers <jrogers emphasys-software com>]

While Seth's workaround allows Kubernetes containers to read /etc/hosts,
I've discovered a similar problem with /dev/shm. A container started
with "docker run" can access /dev/shm normally, but one started by
Kubernetes cannot. After carefully comparing the output from "docker
inspect" for the two situations, I discovered that Kubernetes is using
Docker's IPC mechanism to share /dev/shm between two containers in a
pod. It seems that while Docker would normally set SELinux labels to
allow both containers to read and write, Kubernetes overrides SELinux
labels, disallowing access for all but the source container to /dev/shm.

My container needs to use /dev/shm for IPC within the container only.
I'm only specifying one container per pod so the container which is the
source for /dev/shm must be automatically generated by Kubernetes (it
has a name starting with "k8s_POD"). I don't currently need any of the
fancy cross-container IPC stuff but I guess for now, I'll just disable
SELinux enforcement. Is this a bug in Kubernetes only? Is there a way to
work around this, such as disabling any attempt at cross-container IPC?
I found this bug which seems to describe exactly the problem I'm having:

https://github.com/kubernetes/kubernetes/issues/20899

On 09/08/2016 11:25 AM, Seth Jennings wrote:
> Hey Jonathan,
> 
> Yes this is an issue.
> 
> kube mounts /etc/hosts into the container from something like:
> 
> /var/lib/kubelet/pods/9679b088-75d6-11e6-9f2b-fa163eebc035/etc-host
> 
> on the host.  That file is currently labeled with docker_var_lib_t:
> 
> # ls -alZ etc-hosts
> -rw-r--r--. root root system_u:object_r:docker_var_lib_t:s0 etc-hosts
> 
> This should be resolved by this yet-to-be-released fix to selinux-policy:
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1369159
> 
> Should be fixed in selinux-policy > 3.13.1-95.el7.
> 
> In the meantime, you can override the selinux policy on your system manually:
> 
> # semanage fcontext -l | grep kubelet
> /var/lib/kubelet(/.*)?                             all files
> system_u:object_r:docker_var_lib_t:s0
> 
> # semanage fcontext -a -t svirt_sandbox_file_t "/var/lib/kubelet(/.*)?"
> 
> # restorecon -Rv /var/lib/kubelet
> 
> Hope this helps!
> 
> Seth
> 
> On Sun, Sep 4, 2016 at 8:15 PM, Jonathan Rogers
> <jrogers emphasys-software com> wrote:
>> I'm using CentOS Atomic Host 7. I can run Docker containers directly and
>> they function as expected. I set up a small Kubernetes cluster using the
>> Atomic and Kubernetes documentation. I can also run containers using
>> Kubernetes, but /etc/hosts is unreadable in the containers because of
>> SELinux configuration. I found this be true in containers based on both
>> the "busybox" Docker image as well as my CentOS 6-based image.
>>
>> I see that Kubernetes sets up a mount just for /etc/hosts, overriding
>> Docker's default behavior. Why is this necessary? It seems that
>> Kubernetes fails to apply the necessary label(s) to the hosts file it
>> provides. If I use the chcon command on the host to add
>> "svirt_sandbox_file_t" to the Kubernetes-managed hosts file, the
>> container can read it via /etc/hosts. Of course, disabling SELinux
>> enforcement also avoids the problem.
>>
>> Since this doesn't seem to be a common problem, I can't tell if it's
>> Kubernetes, CentOS or Atomic at fault. I found a long discussion about
>> SELinux vs Kubernetes which seems related. However, the Docker volume in
>> question was generated automatically by Kubernetes without any explicit
>> configuration.
>>
>> https://github.com/projectatomic/adb-atomic-developer-bundle/issues/117
>>
>> --
>> Jonathan Rogers
>> Socialserve.com by Emphasys Software
>> jrogers emphasys-software com
>>
>>
> 


-- 
Jonathan Rogers
Socialserve.com by Emphasys Software
jrogers emphasys-software com