[atomic-devel] Fwd: Trouble with cgroup when starting latest Kubelet

[Jason Brooks <jbrooks redhat com>]

neglected to reply-all

---------- Forwarded message ----------
From: Jason Brooks <jbrooks redhat com>
Date: Thu, Jun 14, 2018 at 3:06 PM
Subject: Re: [atomic-devel] Trouble with cgroup when starting latest Kubelet
To: Rares Vernica <rvernica gmail com>


On Thu, Jun 14, 2018 at 1:37 PM, Rares Vernica <rvernica gmail com> wrote:
> Hi,
>
> I installed the CentOS-Atomic-Host-7-Installer.iso 2018-06-11 on bare metal
> and the latest Kubelet from
> registry.centos.org/centos/kubernetes-sig-kubelet:latest I'm having trouble
> starting Kubelet. I get this:
>
> # systemctl -l status kubelet
> ● kubelet.service - kubernetes-kubelet
>    Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor
> preset: disabled)
>    Active: active (running) since Thu 2018-06-14 13:28:40 PDT; 3min 8s ago
>   Process: 6440 ExecStop=/usr/bin/runc --systemd-cgroup kill kubelet
> (code=exited, status=0/SUCCESS)
>  Main PID: 6452 (runc)
>     Tasks: 8
>    Memory: 8.6M
>    CGroup: /system.slice/kubelet.service
>            └─6452 /usr/bin/runc --systemd-cgroup run kubelet
>
> runc[6452]: E0614 20:31:13.173341    6463 summary.go:102] Failed to get
> system container stats for "/systemd/system.slice": failed to get cgroup
> stats for "/systemd/system.slice": failed to get container info for
> "/systemd/system.slice": unknown container "/systemd/system.slice"
> runc[6452]: E0614 20:31:13.173379    6463 summary.go:102] Failed to get
> system container stats for "/system.slice/docker.service": failed to get
> cgroup stats for "/system.slice/docker.service": failed to get container
> info for "/system.slice/docker.service": unknown container
> "/system.slice/docker.service"
> runc[6452]: E0614 20:31:41.060382    6463 kubelet_network.go:225] Failed to
> ensure that nat chain KUBE-MARK-DROP exists: error creating chain
> "KUBE-MARK-DROP": executable file not found in $PATH:
> runc[6452]: W0614 20:31:43.093989    6463 container_manager_linux.go:574]
> [ContainerManager] Failed to ensure state of "/systemd/system.slice": failed
> to move PID 6463 (in "/system.slice/runc-kubelet.scope") to
> "/systemd/system.slice": mkdir /sys/fs/cgroup/cpuset/systemd: read-only file
> system
>
> In /etc/kubernetes/kubelet I use:
>
> KUBELET_ARGS="--cgroup-driver=systemd
> --runtime-cgroups=/systemd/system.slice
> --kubelet-cgroups=/systemd/system.slice --fail-swap-on=false"
>
> Any ideas of what is going wrong here?

Just tried to reproduce this on centos atomic host 7.1805 and the
kubernetes-kubelet system container and the kubelet is running for me,
although I do see some error messages. The system container may need
some tweaks.

See if you have a problem using package layering to install this
version of the kubelet:

1. uninstall the system container

atomic uninstall kublet

2. configure the yum repo:

cat <<EOF > /etc/yum.repos.d/virt7-kubernetes-110-candidate.repo
[virt7-kubernetes-110-candidate]
name=virt7-kubernetes-110-candidate
baseurl=http://cbs.centos.org/repos/virt7-kubernetes-110-candidate/x86_64/os
enabled=1
gpgcheck=0
EOF

3. install the package

rpm-ostree install kubernetes-node -r




>
> Thanks!
> Rares