Re: [atomic-devel] systemd.unit files and atomic upgrade.

[Eric Paris <eparis redhat com>]

On Fri, 2015-05-22 at 10:06 -0400, Robert Rati wrote:
> I've reproduced this issue pretty easily.  We have symlinks in 
> /etc/systemd/system that point to common unit files on an NFS share. 
> The unit files in the NFS share are usable and functioning on 7.1.0. 
> Then I do:
> 
> ostree remote add --set=gpg-verify=false atomic7.1.1 <url>
> rpm-ostree rebase atomic7.1.1:rhel-atomic-host/7/x86_64/standard
> systemctl reboot
> 
> After the system boots, the kubelets and kube-proxy are not running. 
> Attempting to start the kubelet manually produces a failure because 
> our 
> configuration causes issues with the kube shipped in 7.1.1 (unit file 
> 
> ${VAR} issue).  Doing:
> 
> systemctl daemon-reload; systemctl restart kubelet
> 
> gets me the kubelet from our custom unit files 
> (v0.17.1-629-gd9d12fd3f7036c).
> 
> Then I upgrade to 7.1.2:
> ostree remote add --set=gpg-verify=false atomic7.1.2 <url>
> rpm-ostree rebase atomic7.1.1:rhel-atomic-host/7/x86_64/standard
> systemctl reboot
> 
> After the system boots, the kubelet is not running.  Manually 
> starting 
> the kubelet results in the kubelet version from atomic 7.1.2 running 
> (v0.15.0-123-g0ea87e48640729)
> 
> Throughout all upgrades the files in /etc/systemd/system are 
> untouched. 
>   The symlinks are still pointing to the unit files in the shared 
> storage and the links are still good, the NFS share is mounted, etc.
> 
> systemctl daemon-reload; systemctl restart kubelet
> 
> The above gets me the kubelet from our custom unit files 
> (v0.17.1-629-gd9d12fd3f7036c).
> 
> In each upgrade case, "atomic host upgrade" never did anything but 
> report an error or that no upgrades available.  I tried running it 
> after 
> ostree and rpm-ostree steps, and before and after the reboot.
> 
> It should also be noted that in each upgrade case daemons that were 
> enabled didn't end up starting on the reboot.  From 7.1.0->7.1.1 
> docker, 
> kubelet, and kube-proxy did not start after the reboot.  From 
> 7.1.1->7.1.2 docker and the kubelet did not start after the reboot.

Did this every work after a reboot? I think the key is that you don't
actually have unit files in etc, you have them in nfs!

/etc is "supposed" to be local

I don't see a good way to fix your setup.  Don't use NFS for /etc
Add a daemon-reload after the mounting of remote filesystems, but
that's hard, as you'll need to add After= stuff to your unit files you
have on NFS...   Nothing about this seems feasible really.

Would we 
really expect to put links to NFS shares in /etc/init.d/rc3.d/ and them
to work correctly?


> 
> Rob
> 
> On 05/21/2015 04:07 PM, Tim St Clair wrote:
> > Hey Folks -
> > 
> >     We recently upgraded our cluster 7.1.0->7.1.1->7.1.2 and we 
> > uncovered that our systemd.unit files did not hold across upgrades.
> > 
> > Is/was this a known issue?
> > 
>