cgroup: rmdir() does not complete

Previous message: [thread] [date] [author]
Next message: [thread] [date] [author]
From: Mark Hills
Date: Thursday, August 26, 2010 - 8:51 am

I am experiencing hung tasks when trying to rmdir() on a cgroup. One task 
spins, others queue up behind it with the following:

  INFO: task soaked-cgroup:27257 blocked for more than 120 seconds.
  "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  soaked-cgrou D ffff8800058157c0     0 27257  29411 0x00000000
  ffff88004ffffdd8 0000000000000086 ffff88004ffffda8 ffff88004ffffeb8
  0000000000000010 ffff880119813780 ffff88004ffffd48 ffff88004fffffd8
  ffff88004fffffd8 000000000000f9b0 00000000000157c0 ffff880137693268
  Call Trace:
  [<ffffffff81115edb>] ? mntput_no_expire+0x24/0xe7
  [<ffffffff81427acd>] __mutex_lock_common+0x14d/0x1b4
  [<ffffffff81108a7c>] ? path_put+0x1d/0x22
  [<ffffffff81427b48>] __mutex_lock_slowpath+0x14/0x16
  [<ffffffff81427c4f>] mutex_lock+0x31/0x4b
  [<ffffffff8110bdf8>] do_rmdir+0x74/0x102
  [<ffffffff8110bebd>] sys_rmdir+0x11/0x13
  [<ffffffff81009b02>] system_call_fastpath+0x16/0x1b

Kernel is from Fedora, 2.6.33.6. In all cases the cgroup contains no 
tasks.

Commit ec64f5 ("fix frequent -EBUSY at rmdir") adds a busy wait loop to 
the rmdir. It looks like what I am seeing here and indicates that some 
cgroup subsystem is busy, indefinitely.

I have not worked out how to reproduce it quickly. My only way is to 
complete a 'dd' command in the cgroup, but then the problem is so rare it 
is slow progress.

Documentation/cgroup.memory.txt describes how force_empty can be required 
in some cases. Does this mean that with the patch above, these cases will 
now spin on rmdir(), instead of returning -EBUSY? How can produce a 
reliable test case requiring memory.force_empty to be used, to test this?

Or is it likely to be some other cause, and how best to find it?

Thanks

-- 
Mark
--
Previous message: [thread] [date] [author]
Next message: [thread] [date] [author]

Messages in current thread:
cgroup: rmdir() does not complete, Mark Hills, (Thu Aug 26, 8:51 am)
Re: cgroup: rmdir() does not complete, Daisuke Nishimura, (Thu Aug 26, 5:56 pm)
Re: cgroup: rmdir() does not complete, Balbir Singh, (Thu Aug 26, 6:20 pm)
Re: cgroup: rmdir() does not complete, KAMEZAWA Hiroyuki, (Thu Aug 26, 6:25 pm)
Re: cgroup: rmdir() does not complete, KAMEZAWA Hiroyuki, (Thu Aug 26, 7:35 pm)
Re: cgroup: rmdir() does not complete, Daisuke Nishimura, (Thu Aug 26, 8:39 pm)
Re: cgroup: rmdir() does not complete, KAMEZAWA Hiroyuki, (Thu Aug 26, 10:42 pm)
Re: cgroup: rmdir() does not complete, KAMEZAWA Hiroyuki, (Thu Aug 26, 11:29 pm)
Re: cgroup: rmdir() does not complete, Balbir Singh, (Mon Aug 30, 12:32 am)
Re: cgroup: rmdir() does not complete, Mark Hills, (Mon Aug 30, 2:13 am)
Re: cgroup: rmdir() does not complete, Mark Hills, (Mon Aug 30, 2:25 am)
Re: cgroup: rmdir() does not complete, Mark Hills, (Wed Sep 1, 4:10 am)
Re: cgroup: rmdir() does not complete, KAMEZAWA Hiroyuki, (Wed Sep 1, 4:42 pm)
Re: cgroup: rmdir() does not complete, Mark Hills, (Thu Sep 2, 2:45 am)
Re: cgroup: rmdir() does not complete, Mark Hills, (Thu Sep 9, 3:01 am)
Re: cgroup: rmdir() does not complete, Balbir Singh, (Thu Sep 9, 3:09 am)
Re: cgroup: rmdir() does not complete, Mark Hills, (Thu Sep 9, 4:36 am)
Re: cgroup: rmdir() does not complete, Peter Zijlstra, (Thu Sep 9, 4:50 am)
Re: cgroup: rmdir() does not complete, Mark Hills, (Thu Sep 9, 4:04 pm)
Re: cgroup: rmdir() does not complete, KAMEZAWA Hiroyuki, (Thu Sep 9, 4:43 pm)
Re: cgroup: rmdir() does not complete, KAMEZAWA Hiroyuki, (Thu Sep 9, 7:16 pm)
Re: cgroup: rmdir() does not complete, Daisuke Nishimura, (Thu Sep 9, 9:05 pm)
Re: cgroup: rmdir() does not complete, KAMEZAWA Hiroyuki, (Thu Sep 9, 9:11 pm)
Re: cgroup: rmdir() does not complete, Mark Hills, (Fri Sep 10, 12:28 am)
Re: cgroup: rmdir() does not complete, KAMEZAWA Hiroyuki, (Fri Sep 10, 12:33 am)
Re: cgroup: rmdir() does not complete, Mark Hills, (Fri Sep 10, 12:51 am)