Re: [atomic-devel] systemd.unit files and atomic upgrade.

[Robert Rati <rrati redhat com>]

I've reproduced this issue pretty easily.  We have symlinks in 
/etc/systemd/system that point to common unit files on an NFS share. 
The unit files in the NFS share are usable and functioning on 7.1.0. 
Then I do:

ostree remote add --set=gpg-verify=false atomic7.1.1 <url>
rpm-ostree rebase atomic7.1.1:rhel-atomic-host/7/x86_64/standard
systemctl reboot

After the system boots, the kubelets and kube-proxy are not running. 
Attempting to start the kubelet manually produces a failure because our 
configuration causes issues with the kube shipped in 7.1.1 (unit file 
${VAR} issue).  Doing:

systemctl daemon-reload; systemctl restart kubelet

gets me the kubelet from our custom unit files 
(v0.17.1-629-gd9d12fd3f7036c).

Then I upgrade to 7.1.2:
ostree remote add --set=gpg-verify=false atomic7.1.2 <url>
rpm-ostree rebase atomic7.1.1:rhel-atomic-host/7/x86_64/standard
systemctl reboot

After the system boots, the kubelet is not running.  Manually starting 
the kubelet results in the kubelet version from atomic 7.1.2 running 
(v0.15.0-123-g0ea87e48640729)

Throughout all upgrades the files in /etc/systemd/system are untouched. 
 The symlinks are still pointing to the unit files in the shared 
storage and the links are still good, the NFS share is mounted, etc.

systemctl daemon-reload; systemctl restart kubelet

The above gets me the kubelet from our custom unit files 
(v0.17.1-629-gd9d12fd3f7036c).

In each upgrade case, "atomic host upgrade" never did anything but 
report an error or that no upgrades available.  I tried running it after 
ostree and rpm-ostree steps, and before and after the reboot.

It should also be noted that in each upgrade case daemons that were 
enabled didn't end up starting on the reboot.  From 7.1.0->7.1.1 docker, 
kubelet, and kube-proxy did not start after the reboot.  From 
7.1.1->7.1.2 docker and the kubelet did not start after the reboot.

Rob

On 05/21/2015 04:07 PM, Tim St Clair wrote:
> Hey Folks -
>
>     We recently upgraded our cluster 7.1.0->7.1.1->7.1.2 and we uncovered that our systemd.unit files did not hold across upgrades.
>
> Is/was this a known issue?