gdb is not so great at debugging within modules.

At the suggestion of LKML, I have spent today working on making blk-cgroup, also known as blkiocg, into a module. it has not gone entirely smoothly, as there is an already existing module (cfq-iosched) that depends on it, so the process has involved a good amount of figuring out how module dependencies (in terms of symbol-loading and the language of the kbuild system) work.

Now that I got it finally to build, boot, and load the modules (modprobe even figured out the dependency for me, it was great), I discovered that the kernel panics when trying to unload the subsystem (with what looks like a SIGILL in some memory management code). So, GDBing it up, I have discovered that it is difficult to deal with code that's loaded at runtime that isn't a standard dynamic library.

(gdb)
0x000000006005146a 3504 ss->destroy(ss, dummytop);
(gdb)
0x00000000628202ad in ?? ()
(gdb) break block/blk-cgroup.c:207
No source file named block/blk-cgroup.c.
Make breakpoint pending on future shared library load? (y or [n]) n
(gdb) disassemble 0x628202ad
No function contains specified address.
(gdb) disassemble 0x628202ad 0x62820363
Dump of assembler code from 0x628202ad to 0x62820363:
0x00000000628202ad: push %rbp
...

And you thought 213's bomblab would never be useful!

I take two refcounts before I take two refcounts, and then I take two more

The best solution to the previously mentioned deadlock problem was determined to be having parse_cgroupfs_options take module reference counts (before sget takes the lock), and have rebind_subsystems drop them later. This division of work makes sure that the subsystems will stick around during sget so we can safely drop our own lock in the meantime.

So I was polishing up the deadlock-free version of the patch series today, and found a good bug. After a few routine changes and polishings, I decided to go through a bit more thorough testing than I'd done since implementing this change:

livecd dev # mount -t cgroup none -o test2,net_cls,devices,cpuacct cgroup/
livecd dev # ls cgroup/
cgroup.procs   cpuacct.usage_percpu  devices.list       release_agent
cpuacct.stat   devices.allow         net_cls.classid    tasks
cpuacct.usage  devices.deny          notify_on_release  test2.hax
livecd dev # lsmod
Module                  Size  Used by
cgroup_test2            2880  1
cls_cgroup              5080  1
livecd dev #

Simple enough. Now, cgroups has this functionality where you can't destroy a hierarchy that has children cgroups. (Children cgroups are represented as subdirectories in the filesystem tree, and are made just as you would expect.) It does, however, let you unmount it:

livecd dev # mkdir cgroup/foo
livecd dev # umount cgroup/
livecd dev #

In which case the hierarchy sticks around, invisible. You can make it reappear by mounting with the same list of subsystems; any attempt to mount a subsystem on that hierarchy with a mismatched set of subsystems will fail.

livecd dev # lsmod
Module                  Size  Used by
cgroup_test2            2880  1
cls_cgroup              5080  1
livecd dev #

Great - the modules' reference counts indicate that the hierarchy is still there. Let's bring it back again:

livecd dev # mount -t cgroup none -o cpuacct,test2,net_cls,devices cgroup/
livecd dev # lsmod
Module                  Size  Used by
cgroup_test2            2880  2
cls_cgroup              5080  2
livecd dev #

Oops! The mounting code didn't realize that we weren't changing anything with respect to the subsystems, and we leaked a reference! Fortunately, with the new changes that I made, there was a simple fix - in cgroup_get_sb, added lines 1518-1519:

1426         if (root == opts.new_root) {
1427 +--- 85 lines: We used the new root structure, so this is a new hierarchy ---
1512         } else {
1513                 /*
1514                  * We re-used an existing hierarchy - the new root (if
1515                  * any) is not needed
1516                  */
1517                 cgroup_drop_root(opts.new_root);
1518                 /* no subsys rebinding, so refcounts don't change */
1519                 drop_parsed_module_refcounts(opts.subsys_bits);
1520         }

For extra credit, tell how this bug is also a security vulnerability!

so what's this net_cls thing actually useful for?

Anand came to #cslounge today with an interesting question: is there a logical equivalent of nice for network bandwidth management that i can easily apply to a process? I don't like being unable to browse the web or irc when I scp large amounts of stuff.

I was under the impression that there didn't actually exist a subsystem for controlling that sort of thing, only for "classifying" it - the net_cls subsystem has one control file, which is "classid", and it lets you specify a network class for each cgroup, and it doesn't seem to do much because it doesn't have anything hooking into it because it's a module. Then I found this, which explains the actual secret: each classid can be associated with sockets held by tasks in those cgroups, and then from -userspace- have bandwidth throttling administered! Very slick, and keeps the kernel-side labour to a minimum, so much so that it can even be modularized.

Of course, Anand's particular situation had an easier solution, which is the -l option to scp which lets you specify a bandwidth limit explicitly.

a module UNloading adventure...

I promised myself this morning that I would spend a "little" bit of time thinking about module unloading. An afternoon and half an evening later, I found myself with a newly written 175-line patch...

livecd dev # insmod /mnt/host/cgroup_test1.ko
livecd dev # insmod /mnt/host/cgroup_test2.ko
livecd dev # modprobe cls_cgroup
livecd dev # lsmod
Module Size Used by
cls_cgroup 5064 0
cgroup_test2 2800 0
cgroup_test1 2800 0
livecd dev # mount -t cgroup none -o net_cls,test2 cgroup/
livecd dev # lsmod
Module Size Used by
cls_cgroup 5064 1
cgroup_test2 2800 1
cgroup_test1 2800 0
livecd dev # rmmod cgroup_test1
livecd dev # rmmod cgroup_test2
ERROR: Module cgroup_test2 is in use
livecd dev # umount cgroup
livecd dev # lsmod
Module Size Used by
cls_cgroup 5064 0
cgroup_test2 2800 0
livecd dev # rmmod cgroup_test2
livecd dev #

It still has some FIXMEs, meaning I need to make sure there are no races where there might be races, but I am surprised at how easy that was.

try_module_get vs delete_module

let me introduce you to my friend, whose name is try_module_get.

478 static inline int try_module_get(struct module *module)
479 {
480         int ret = 1;
481
482         if (module) {
483                 unsigned int cpu = get_cpu();
484                 if (likely(module_is_live(module))) {
485                         local_inc(__module_ref_addr(module, cpu));
486                         trace_module_get(module, _THIS_IP_,
487                                 local_read(__module_ref_addr(module, cpu)));
488                 }
489                 else
490                         ret = 0;
491                 put_cpu();
492         }
493         return ret;
494 }

he lives in include/linux/module.h, and he will get a reference count on the module for you, unless its state flag is set to MODULE_STATE_GOING. the get_cpu and put_cpu are SMP macros that disable/enable preemption so you can have a valid smp_processor_id.

great! now let's take a look over at a potential competitor, a system call in kernel/module.c by the name of delete_module. part of his code looks like this:

853         /* Stop the machine so refcounts can't move and disable module. */
854         ret = try_stop_module(mod, flags, &forced);
855         if (ret != 0)
856                 goto out;
857
858         /* Never wait if forced. */
859         if (!forced && module_refcount(mod) != 0)
860                 wait_for_zero_refcount(mod);

he can assist you in two different styles. the most common one is the "remove module immediately", which is what happens with rmmod usually. in this case, the O_NONBLOCK flag is specified. try_stop_module wants to set MODULE_STATE_GOING, and will behave differently depending on this flag.

if O_NONBLOCK is specified, try_stop_module will apply a very big hammer whose name is stop_machine. in this case, it will safely ensure that the reference count is zero (failing otherwise), and then set the MODULE_STATE_GOING flag. this is wonderful: because of the stop_machine hammer, there will be no problems racing with our first friend, try_module_get.

there is another way to invoke rmmod, which is with the --wait flag. if this is specified, try_stop_module will set MODULE_STATE_GOING without worrying about the refcount, and then delete_module will wait for the reference count to drop to zero. the keen-eyed systems hacker will at this point worry, what if we get the following scheduling pattern?

0) module state = MODULE_STATE_LIVE; refcount = 0
1) try_module_get checks module is alive, and succeeds (line 484)
2) delete_module sets MODULE_STATE_GOING flag (line 854)
3) delete_module waits until the refcount is zero, and finishes (line 860)
4) try_module_get increments the refcount (line 485).

not to worry, keen-eyed systems hacker! you will note that our clever friend try_module_get disables preemption on its CPU as it runs. this guarantees that he will not be descheduled during that bit of his code, and therefore, through the wonderful phenomenon of "very small critical section", the problematic execution order won't happen.

"This is mainly for kernel developers and desperate users."

I ran into the CONFIG_MODULES_FORCE_UNLOAD option today, and out of curiosity tried setting it and seeing what would happen.

livecd dev # rmmod -f cls_cgroup
Disabling lock debugging due to kernel taint
livecd dev # ls cgroup/
cgroup.procs net_cls.classid notify_on_release release_agent tasks
livecd dev # cat cgroup/net_cls.classid
cat: cgroup/net_cls.classid: Invalid argument
livecd dev # echo 1 > cgroup/net_cls.classid
bash: echo: write error: Invalid argument
livecd dev # umount cgroup

Modules linked in: [last unloaded: cls_cgroup]
Kernel panic - not syncing: Kernel mode fault at addr 0x0, ip 0x6017b8cc

I was impressed that it didn't die immediately when I tried looking at the file.

livecd / # modprobe cls_cgroup
livecd / # rmmod cls_cgroup
ERROR: Module cls_cgroup is in use
livecd / # rmmod -f cls_cgroup
Disabling lock debugging due to kernel taint
livecd / # cd /dev
livecd dev # mkdir cgroup
livecd dev # mount -t cgroup none -o cls_cgroup cgroup/

Modules linked in: [last unloaded: cls_cgroup]
Kernel panic - not syncing: Kernel mode fault at addr 0x0, ip 0x600fe9aa

Note that this is a case that I'd eventually like to get working with module unloading (without the -f option, of course).

what did the segmentation violation handler say to the general protection handler when the kernel panicked?

livecd / # modprobe cls_cgroup
[New LWP 21722]
linux-nat.c:1152: internal-error: linux_nat_resume: Assertion `lp != NULL' failed.
A problem internal to GDB has been detected,
further debugging may prove unreliable.
Quit this debugging session? (y or n)

Currently merging gdb-7.0 and hoping that won't have the same bug. [edit: new version seems to work fine.]

wrestling dragons

so, next goal after doing a dummy subsystem is to start on the existing builtin subsystems. at least one should end up being modularized, and for others that I examine, analysis should be given on why they can't or would be difficult to be made into modules. so a quick glance over the existing subsystems turned into surprise progress when I discovered that net_cls (which additionally goes by cls_cgroup, net_cls_cgroup, and in all likelihood additional permutations thereof) not only is restricted to one file (devices and ns for example have miscellaneous function calls to them with ifdefs, in contrast), but also already has module_init() et.al. declarations at the bottom!

After a bit of hacking around with cgroup_load_subsys() (my new function) and the subsys_id, and also changing the Kconfig option to tristate, the build system happily modulificatarized net_cls for me. Booting up UML, I realized it was probably time to figure out how to make modprobe do the right thing - as it turns out, UML's documentation is very helpful in this regard. A quick hostfs mount and depmod later, and the module loads and runs just fine. Victoly!

In other news, binding GDB to UML yields some frightening results:

0x00007f614c38b420 in nanosleep () from /lib64/libc.so.6
(gdb) break cgroup_load_subsys
Breakpoint 1 at 0x60051819: file kernel/cgroup.c, line 3619.
(gdb) cont
Continuing.

Program received signal SIGSEGV, Segmentation fault.
memcpy () at arch/um/sys-x86_64/../../x86/lib/memcpy_64.S:68
68 movq %r11, 0*8(%rdi)
Current language: auto; currently asm
(gdb) cont
Continuing.

Program received signal SIGSEGV, Segmentation fault.
0x00007f614c36911b in memset () from /lib64/libc.so.6
(gdb)
Continuing.

Program received signal SIGSEGV, Segmentation fault.
0x00007f614c36873e in memset () from /lib64/libc.so.6
(gdb)
Continuing.

Breakpoint 1, cgroup_load_subsys (ss=0x62828fe0) at kernel/cgroup.c:3619
3619 if (ss->fork || ss->exit)
Current language: auto; currently c
(gdb)

a module loading adventure of the "great success" variety

okay! so, yesterday, and the day before that, and a little bit today, but mostly yesterday (in fact, for just about all of yesterday), I put together two patches in my stgit tree which I called cgroups-revamp-subsys-array.patch and cgroups-subsys-module-interface.patch, satisfying (in a very rudimentary way) #2 and #3 from the roadmap. (#1 turned out to be something I needed to keep but work around anyway... details not important now.) I made sure they compiled and went to bed.

Today, after looking at ns_cgroup.c and devices_cgroup.c and realizing that they couldn't really be modularized easily, I threw together a skeleton subsystem modeled after the other ones that adds a file "hax" that wraps a global variable whose value determines whether you can attach tasks to the cgroup or not. All right, now let's follow this guide that elly pointed me at to get it to build as a module...

WARNING: "cgroup_load_subsys" [/home/bblum/Documents/School/F09/412/hax/cgroup_test1.ko] undefined!
WARNING: "cgroup_add_file" [/home/bblum/Documents/School/F09/412/hax/cgroup_test1.ko] undefined!

Well, they're just warnings, so try loading the module anyway, right? (Note: I use insmod instead of modprobe because the latter wants infrastructure and dependencies, and the former can just take any random file from the filesystem.)

livecd / # mount -t hostfs none -o /home/bblum/412 /mnt/host/
livecd / # cd /mnt/host/hax/
livecd hax # insmod cgroup_test1.ko
cgroup_test1: Unknown symbol cgroup_load_subsys
cgroup_test1: Unknown symbol cgroup_add_file
insmod: error inserting 'cgroup_test1.ko': -1 Unknown symbol in module

It was worth a shot, though. Turns out I need to EXPORT_SYMBOL(...) everything I'll need for the module in kernel/cgroup.c. For now, I just do the functions my subsystem uses; later, I'll need to worry about functions that -any- subsystem might use. Next:

livecd hax # insmod cgroup_test1.ko
cgroup_test1: version magic '2.6.31-rc9-mm1-gf013913 mod_unload ' should be '2.6.31-rc9-mm1-ge40e265 mod_unload '
insmod: error inserting 'cgroup_test1.ko': -1 Invalid module format

It took too long to realize that the kernel I'd most recently booted somehow had something different enough to change the vermagic string from the most recent time I'd built it, which is what I'd built the module against. Okay. Rebooting UML, and going to get it right this time.

livecd hax # insmod cgroup_test1.ko
Kernel panic - not syncing: Kernel mode signal 4
Modules linked in: cgroup_test1(+)
Segmentation fault

I had deliberately left out the ".module = THIS_MODULE" line in test1_subsys when first building it, to see what would happen when cgroup_load_subsys tried to pin the module... and promptly forgotten about it. Putting the line in, finally, and:

livecd dev # lsmod
Module Size Used by
livecd dev # mount -t cgroup none -o test1 cgroup/
mount: special device none does not exist
livecd dev # insmod /mnt/host/hax/cgroup_test1.ko
livecd dev # lsmod
Module Size Used by
cgroup_test1 2512 1 [permanent]
livecd dev # mount -t cgroup none -o test1 cgroup/
livecd dev # ls cgroup/
cgroup.procs notify_on_release release_agent tasks test1.hax
livecd dev # mkdir cgroup/foo
livecd dev # echo $$ > cgroup/foo/tasks
bash: echo: write error: Operation not permitted
livecd dev # echo 42 > cgroup/foo/test1.hax
livecd dev # echo $$ > cgroup/foo/tasks
livecd dev #

:)

roadmap

all right, so it seems like a good idea to map out the ideas and targets I've got in my head, for several reasons. Here's what I've determined I should be doing.

1) Fork/exit callbacks need to go. It's this functionality that cgroups has offered since (presumably) it first hit mainline in which a subsystem can set itself up to get a function called whenever a task forks or exits. Apparently, no subsystem has ever used it, and the presence of it here is going to interact funnily with module-loadable subsystems, so - at the suggestion and approval of Paul - I'm going to strip all callback code out of cgroups. This will be done as a pre-patch to the main patch series I plan on generating.

2) Changing how subsys[] is used.
a) At the bottom of the array will be the entries for builtin subsystems, which will be there at link-time, up until CGROUP_BUILTIN_SUBSYS_COUNT. CGROUP_SUBSYS_COUNT, which used to be that, is now defined as the size of the subsys_bits field in cgroupfs_root (i.e., 32 or 64), and is still the max size of the array. At link time, all entries between the builtin count and the total count will be NULL, and that's where module subsystems will put themselves. (This is done.) Also, the array will need to be surrounded in a rwlock, since when a subsystem registers itself it will need to take a subsys_id. (This is not done.)
b) All code throughout cgroups needs to be able to handle when a subsystem is gone. Each loop that iterates down the array will need to have a check for null pointers (this is done) and take the read-lock (this is not done). There may also be other things that certain loops need to do, situationally - this is as yet unclear.

3) cgroup_init_subsys() needs to be revised to be suitable as a module initcall. It needs to be able to handle failures correctly (the current version will kpanic on initialization fail, since it's assumed to call at boot time only). Of course, because some subsystems will be left as builtins, we'll still need a version suitable for calling at boottime - probably just a wrapper around the adapted module initcall. Also, we'll need to be concurrency-safe now - obviously around the subsys array, and possibly in the other various things that the function does. Among other things, when the module is loaded OR when the module is mounted on a cgroup hierarchy (see note at end of post) we'll need to pin it with try_module_get() to make sure it doesn't go away.

Once we hit this point, it can be said that cgroups has support for modular subsystems. Next, we do the whole "confirming" thing:

4) adapt one or more subsystems to become modules, or perhaps write a new skeleton one for testing, or both. in order to be a module (suppose your module is "foo" as CONFIG_FOO), you need to do the following things:
a) instead of having code interspersed in other code with stuff like #ifdef CONFIG_FOO, it has to be all in the same file (since each .o file is either going to be a builtin or a module). in the kconfig, you need to specify that it's buildable as a module, and in the makefile, you need to make sure that the config file corresponds to the right source file.
b) you need to register a bunch of stuff with the module_suchandsuch() macros - like name, version, author, and most importantly module_init() and module_exit(), which define what functions are called at module load and unload time. (the infrastructure behind this and these macros is a lot of hax.)

I am uncertain whether I'll end up supporting module unloading for cgroups - it seems like it would be useful, given that we have a limit on the number of subsystems loaded at a time. I think this would involve making sure that subsystems can't be unloaded while attached to any mounted hierarchy, but can when not. This likely will necessitate use of cgroup_lock. If we do this, we'll end up pinning the module when we mount a hierarchy - there will be a race here if somebody's trying to unload the module at the same time, so when mounting, pinning all subsystems will have to be done before committing to the mount.

The alternative approach - and the one that I'll go with to begin with, for sure - is to just say nope, never unload, and the module is pinned forever as soon as it's loaded.

understanding is powered by magic

I spent a good hour or two this afternoon poring over the kernel's module build infrastructure (not daring to look at any .c files, of course), looking at various other module code trying to take them as examples. Something clicked in my head then while looking at include/linux/init.h, which hadn't done before, presumably because I hadn't looked at module examples, and suddenly I understood how things wanted to be compiled as modules and have initcalls/exitcalls registered.

It seems (read: I've been advised) that each subsystem will want to have a pointer to its struct module, so it can keep it "pinned" while the subsystem is loaded. This raises an interesting question: where the hell does the module struct come from? include/linux/module.h has a macro called THIS_MODULE which references extern struct module __this_module. Looking at a few examples, some of them have foo.mod.c files, which all look to have the same struct declaration (with perhaps a few differences, namely in the "depends=" string at the end). However, modules that I found living in kernel/ don't tend to have that, though everything else (use of the initcall macros, etc) was the same. Where does this mystery struct come from? A grep through the standard directories revealed nothing; grepping the whole source tree discovered the file scripts/mod/modpost.c which... generates a header file for module code with the relevant struct information. As in, "buf_printf(b, "struct module __this_module\n");" with surrounding context. Ugh.

I need to learn to start trusting the macros (THIS_MODULE, in this case) that look like complete hax instead of trying to figure out what the hax are.

As a note to myself, the CONFIG_CGROUP option settings live in init/Kconfig, and to enable modularization on an option you need to change 'bool' to 'tristate'.