Re: [atomic-devel] tools and systemtap containers are available in Fedora

[William Cohen <wcohen redhat com>]

On 10/05/2017 10:33 AM, Jeremy Eder wrote:
> Forgot to add Will Cohen (discussed stap errors with him briefly).  Also my replies won't make it to the dev list since I am not subscribed (just fyi I guess).
> 
> On Thu, Oct 5, 2017 at 9:10 AM, Jeremy Eder <jeder redhat com <mailto:jeder redhat com>> wrote:
> 
>     First of all, that readme is awesome.
> 
>     spot checking the tools container...seems to all "just work" when I run it with atomic run ...
>     blktrace works
>     ethtool works (-K -i -c -S specifically)
>     netstat works
>     pstack works
>     perf top,record,report works
>     iotop works
>     slabtop works
>     lstopo works
>     htop works (wish this was in rhel)
>     nstat works
>     ss works (-tmpie)
>     ifpps works (wish this was in rhel)
>     numastat works (-mczs)
>     pmap works
>     all the sysstat tools work
>     strace works
>     tcpdump works
>     sar works but you have to prepend the /host directory (so, sar -f /host/var/log/sa/sa05)
>     my god tmux is in here?? yes!
> 
> 
>     ​systemtap (aww, no readme?)
> 
>     doesnt work:
>     ​[root 8b7437fed211 /]# cd /usr/share/systemtap/examples/process/                                                                                                                             
>     [root 8b7437fed211 process]# stap cycle_thief.stp
>     ERROR: Couldn't insert module '/tmp/stapslabb9/stap_0811c9eea1bbb81f2fbc5f7bf9df4506_8509.ko': Operation not permitted
>     WARNING: /usr/bin/staprun exited with status: 1
>     Pass 5: run failed.  [man error::pass5]
>     [root 8b7437fed211 process]# 
> 
> 
> 
>     [root dhcp23-91 ~]# atomic run --spc candidate-registry.fedoraproject.org/f26/systemtap <http://candidate-registry.fedoraproject.org/f26/systemtap>
>     docker run --cap-add SYS_MODULE -v /sys/kernel/debug:/sys/kernel/debug -v /usr/src/kernels:/usr/src/kernels -v /usr/lib/modules/:/usr/lib/modules/ -v /usr/lib/debug:/usr/lib/debug -t -i --name systemtap-spc candidate-registry.fedoraproject.org/f26/systemtap <http://candidate-registry.fedoraproject.org/f26/systemtap>
> 
>     This container uses privileged security switches:
> 
>     INFO: --cap-add 
>           Adding capabilities to your container could allow processes from the container to break out onto your host system.
> 
>     For more information on these switches and their security implications, consult the manpage for 'docker run'.
> 
>     [root 10accce504c2 /]# cd /usr/share/systemtap/examples/process/
>     [root 10accce504c2 process]# stap cycle_thief.stp 
>     ERROR: Couldn't insert module '/tmp/stapNEjJDX/stap_4f013e7562b546a0316af840de9f0713_8509.ko': Operation not permitted
>     WARNING: /usr/bin/staprun exited with status: 1
>     Pass 5: run failed.  [man error::pass5]
> 
> 
> 
>     On Thu, Oct 5, 2017 at 3:09 AM, Tomas Tomecek <ttomecek redhat com <mailto:ttomecek redhat com>> wrote:
> 
>         Not sure if the question is for me -- I literally have no idea how to do that.
> 
> 
>         Let me know how I can help,
> 
>         Tomas
> 
> 
>         On Thu, Oct 5, 2017 at 5:04 AM, Dusty Mabe <dusty dustymabe com <mailto:dusty dustymabe com>> wrote:
> 
> 
> 
>             On 09/18/2017 10:48 AM, Tomas Tomecek wrote:
>             > Hello,
>             >
>             > we managed to move tools container from Fedora Dockerfiles github repo to Fedora infra [1]. As a side effects, we put systemtap in a dedicated container.
>             >
>             > We would very much appreciate your feedback here: so if you have some time to take a look at these containers and try them out, it would mean a lot to us.
>             >
>             > Repos:
>             > https://src.fedoraproject.org/container/systemtap <https://src.fedoraproject.org/container/systemtap>
>             > https://src.fedoraproject.org/container/tools <https://src.fedoraproject.org/container/tools>
>             >
>             > The way to access the images:
>             > docker pull candidate-registry.fedoraproject.org/f26/tools <http://candidate-registry.fedoraproject.org/f26/tools> <http://candidate-registry.fedoraproject.org/f26/tools <http://candidate-registry.fedoraproject.org/f26/tools>>
> 
>             just tested out the tools container. can we get this into the official registry?
> 
>             > docker pull candidate-registry.fedoraproject.org/f26/systemtap <http://candidate-registry.fedoraproject.org/f26/systemtap> <http://candidate-registry.fedoraproject.org/f26/systemtap <http://candidate-registry.fedoraproject.org/f26/systemtap>>
>             >
>             > Both images have help files, so please read them prior using the containers:
>             > https://src.fedoraproject.org/container/tools/blob/master/f/root/README.md <https://src.fedoraproject.org/container/tools/blob/master/f/root/README.md>
>             > https://github.com/container-images/systemtap/blob/master/help/help.md <https://github.com/container-images/systemtap/blob/master/help/help.md>
>             >
>             > (or `atomic help $the_container_image`)
>             >
>             > [1] https://pagure.io/atomic-wg/issue/214 <https://pagure.io/atomic-wg/issue/214>
> 
> 
> 
> 
> 
>     -- 
> 
>     -- Jeremy Eder
> 
> 
> 
> 
> -- 
> 
> -- Jeremy Eder

Hi,

I have done some probing around on the environment that Jeremy setup.

The problem seems to be limited to the actual loading of the the generated module in the container.
I was able to do:

# stap -p4 -m cycle_thief /usr/share/systemtap/examples/process/cycle_thief.stp

Then copy the cycle_thief.ko from inside the container to the host machine.  The following command to run things on the host works fine:

# staprun ./cycle_thief.ko

Conversely was albe to load and unload various kernel modules on the host with modprobe and rmprobe, but unable to same operations within the kernel.

What is the list of syscalls allowed?

Maybe run container-check.stp on the host looking at the container that we are trying to run systemtap inside.  How do we find out the process that spawned off that container?  Installed "pstree", started a process in the client that could find in pstree output.  Then:

# ./container_check.stp -v -x 2816
Pass 1: parsed user script and 471 library scripts using 139876virt/46200res/7696shr/38748data kb, in 140usr/30sys/175real ms.
Pass 2: analyzed script: 582 probes, 21 functions, 104 embeds, 110 globals using 308456virt/216372res/9060shr/207328data kb, in 29990usr/390sys/30531real ms.
Pass 3: translated to C into "/tmp/stapVQGA4T/stap_942629388b1b117eb698f8777091b161_1001584_src.c" using 308456virt/216372res/9060shr/207328data kb, in 2880usr/20sys/2926real ms.
Pass 4: compiled C into "stap_942629388b1b117eb698f8777091b161_1001584.ko" in 76330usr/1700sys/78140real ms.
Pass 5: starting run.
starting container_check.stp. monitoring 2816
^C

capabilities used by executables
      executable:      prob capability



capabilities used by syscalls
      executable,              syscall (       capability ) :            count


forbidden syscalls
      executable,              syscall:            count


failed syscalls
      executable,              syscall =            errno:            count
            bash,                 stat =           ENOENT:                1
            bash,                wait4 =           ECHILD:                1
         staprun,          init_module =            EPERM:                1
         staprun,               access =           ENOENT:                1
         staprun,                 stat =           ENOENT:                1
Pass 5: run completed in 10usr/9170sys/22614real ms.


So it looks like init_module syscall is not being allowed.

-Will