Re: [atomic-devel] Atomic as Base OS for Standalone Appliance

["Shane O'Donnell" <sodonnell cree com>]

Inline...

On 5/22/18, 12:30 PM, "atomic-devel-bounces projectatomic io on behalf of Colin Walters" <atomic-devel-bounces projectatomic io on behalf of walters verbum org> wrote:

    Hi Shane,

    On Mon, May 21, 2018, at 3:09 PM, Shane O'Donnell wrote:
    > Hey All –
    >
    > We’re building an IoT edge device based on CentOS Atomic Host

    Cool; without knowing more it feels like this falls in between "server"
    and "device"?   If it's more "device" like then the newly formed Fedora
    IoT group is intended to be the place, although as of yet as far as I
    can tell there isn't even a mailing list?

I'm tracking that project now, too, but this is shipping on a quad-core x86_64 w/ 4GB RAM & 256GB SSD.  Definitely more "server" more than "device", but anemic when compared to real servers.

    > As an example, we’re trying to follow the “transactional” nature of
    > Atomic, but we’re finding there’s no real “commit”.  When we need to
    > change something in the OS itself,

    A big topic here is whether you derive from the CAH ostree commits (hostimages),
    or whether you do a custom compose/build.  The latter is fairly straightforward
    and gives you full control.

We're doing a custom compose, but our configs still track pretty closely with CAH.

    > we end up stuck with a “hotfix”
    > flagged deployment that we can’t really “commit” to clear the flag,
    > which leaves us unable to make follow-on transactional changes.

    It depends on what you want to do - we could probably fairly easily
    support a model where there's one or more persistent "layer"s.  I just
    filed this: https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fprojectatomic%2Frpm-ostree%2Fissues%2F1370&data=01%7C01%7Csodonnell%40cree.com%7C8a47a4ebfe5741ab723808d5c0014905%7Ca326555ee95e4f1b87f747eba52c598e%7C0&sdata=78svYf5jTFPuft1%2BjZnp1%2FzZGQ0OebmtMWcDgq5RsjQ%3D&reserved=0

    But again it depends on what content you want there.

We've run into issues in a couple of different scenarios, but as a representative use case:

Most of our containers have bind-mounted volumes.  We use Ansible to create those directories in advance, and a number of them get certs created/added to the directories.  We were originally doing this early in our Ansible processing, then after manipulating kernel args with "ex kargs", found a number of our changes to /etc were lost after a reboot.  We re-ordered things to get around this and introduced a new reboot to effectively 'commit' our changes, but we were hoping to find some way to allow manipulation of the current deployment with a with a "commit" step at the end.

Allowing a "persist overlay" sounds like it would allow us to force the "merge" on demand -- which would certainly address this situation.

    > We are also in the process of migrating our Ansible-based platform
    > customization and software-load (i.e., Docker containers managed via
    > Docker Compose) which seems somewhat unnatural in the Atomic world.

    Hmm.  Today we do support Ansible although most of our efforts
    as you noticed are oriented towards Kubernetes.

I've found a few sparse docs on the support for Ansible as well as watched a YouTube video on it, but I may have overlooked something.  I was also imagining a scenario where our Ansible tasks/playbooks could be modal, where we run a task (e.g., ostree admin unlock), continue running tasks that would manipulate the current deployment (or overlay), then run a task (e.g., ostree admin lock) to commit the current deployment (or merge the current overlay).  Entering with this wrong-headed mindset may have stymied us a bit as well.

    > The docs seem pretty clearly focused on using Atomic as the OS for a
    > cluster, but our standalone deployments seem to leave us with a lot of
    > questions.  I think the following list is a reasonable summary of the
    > big questions we haven’t really figured out yet:
    >
    >
    >   *   Is there a “commit” that would allow us to commit the current
    > “hotfix” to the current deployment?

    Not today, though the above issue covers some of it.   On IRC you mentioned
    kernel arguments; we could add `--inplace` or something.  That'd likely
    be fairly easy to do.  Kernel argument management is a gap for sure.

`--inplace` would work.

On a somewhat related note, we found that building a new GRUB config was broken and was creating a grub.cfg that tried to load "linux16" and "initrd16" vs. "linuxefi"/"initrdefi", but I think this may be a broader CentOS problem and not specific to CAH.

    >   *   All of our appliances start with a common base image.  What’s the
    > recommended approach for changing kernel arguments so they appear in the
    > common base image?

    ostree today does not store kernel arguments in the commit.  See also
    https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fostreedev%2Fostree%2Fissues%2F479&data=01%7C01%7Csodonnell%40cree.com%7C8a47a4ebfe5741ab723808d5c0014905%7Ca326555ee95e4f1b87f747eba52c598e%7C0&sdata=Rw8v24emgGZtiyxeFXbkpz24Q8mf4h7fZlxRnv1OKHs%3D&reserved=0

    The high level thing to understand is that the design mostly is that
    rpm-ostree can be used in the same places/same way as one might use
    "classic" package managers like yum/apt.  Since today one doesn't
    store kernel arguments in rpm/deb packages, ostree also didn't do it
    by default.  Instead you use kickstart or config management or whatever.
    But that doesn't mean we couldn't!   Like I said in the issue, it'd
    probably make sense to do, but we'd have to work through some overlap
    in terms of how it works with Anaconda, etc.

Yes, we're loading our systems with Anaconda/Kickstart, but "ex kargs" seemed a much cleaner approach to remove a default parameter than other options.  Live and learn...

    See also one of our oldest issues:
    https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fprojectatomic%2Frpm-ostree%2Fissues%2F471&data=01%7C01%7Csodonnell%40cree.com%7C8a47a4ebfe5741ab723808d5c0014905%7Ca326555ee95e4f1b87f747eba52c598e%7C0&sdata=OtomwPgvIwBVyEp0g75syMa8T8NN4EsvVLWPKoB3riA%3D&reserved=0

I don't think server-side Ansible would work in our case.  For example, we're creating certs and handling secrets/tokens that are generated at container "initialization" time and then shared with other containers.  I don't think we can do that (sanely) on the server-side.

That said, I can see many other use cases where that would work...

    > Note that we’re both invested and committed to using Atomic, and we’d
    > like to offer up a resource to help flesh out the “standalone” side of
    > the documentation so we both have some agreed upon contract between
    > users and core devs, as well as to better enable new users with this use
    > case.

    That's always nice!

I'd love to save somebody else the learning curve!  __

Shane O.


This e-mail message, including any attachments and previous email messages sent with it, contains CONFIDENTIAL and PROPRIETARY information of Cree, Inc. or its subsidiaries and may be legally PRIVILEGED. You may not use, disclose, reproduce or distribute such information without Cree's authorization. If you have received this message in error, please notify the sender immediately and permanently delete the original message, its attachments and any copies thereof.