login
Search
Continuing the earlier discussion about low latency and Ingo Molnar [interview]'s voluntary kernel preemption patch [story], the conversation moved onto the affect a filesystem can have on latency. Specifically, 2.6 maintainer Andrew Morton [interview] noted that ReiserFS was known to have some latency issues in both the 2.4 and 2.6 Linux kernels, "resierfs: yes, it's a problem. I 'fixed' it multiple times in 2.4, but the fixes ended up breaking the fs in subtle ways and I eventually gave up." However, he did go on to note, "actually, the 2.4 low-latency patch does still have some reiserfs fixes, so it's probably better than reiserfs in 2.6."
When asked if ext3 was a better choice for low latency work, Andrew Morton replied, "ext3 is certainly better than [reiserfs], but still has a couple of potential problem spots. ext2 is probably the best at this time." Data is continuing to be collected and reviewed by a number of kernel developers, so the more noticeable latency issues in the 2.6 kernel will likely be addressed soon.
From: Lee Revell [email blocked] Subject: Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch Date: Mon, 12 Jul 2004 18:56:55 -0400 On Sun, 2004-07-11 at 01:25, Andrew Morton wrote: > What we need to do is to encourage audio testers to use ALSA drivers, to > enable CONFIG_SND_DEBUG in the kernel build and to set > /proc/asound/*/*/xrun_debug and to send us the traces which result from > underruns. > OK, here goes. The following traces result from running JACK overnight like so, on an otherwise idle system. Hardware is a VIA EPIA 6000, with a 600Mhz C3 processor. Kernel is 2.6.7 + volunatary_preempt patch. voluntary_preempt and kernel_preemption are both on. jackd -v --realtime -d alsa --outchannels 2 --rate 48000 --shorts --playback --period 32 --nperiods 2 These settings require less than 666 microseconds scheduler latency. The average performance is quite good - 5-20 *microseconds*! load = 0.3754 max usecs: 5.000, spare = 661.000 load = 1.1637 max usecs: 13.000, spare = 653.000 load = 3.0593 max usecs: 33.000, spare = 633.000 load = 1.9050 max usecs: 5.000, spare = 661.000 load = 1.4780 max usecs: 7.000, spare = 659.000 load = 1.2645 max usecs: 7.000, spare = 659.000 load = 1.0076 max usecs: 5.000, spare = 661.000 load = 1.1044 max usecs: 8.000, spare = 658.000 load = 0.9276 max usecs: 5.000, spare = 661.000 load = 1.5148 max usecs: 14.000, spare = 652.000 load = 1.1328 max usecs: 5.000, spare = 661.000 load = 1.0168 max usecs: 6.000, spare = 660.000 However there are periodic XRUNS. I noticed that many of these seem APM-related, so here are the relevant settings: # Power management options (ACPI, APM) # APM (Advanced Power Management) BIOS Support CONFIG_APM=y # CONFIG_APM_IGNORE_USER_SUSPEND is not set # CONFIG_APM_DO_ENABLE is not set CONFIG_APM_CPU_IDLE=y # CONFIG_APM_DISPLAY_BLANK is not set CONFIG_APM_RTC_IS_GMT=y # CONFIG_APM_ALLOW_INTS is not set # CONFIG_APM_REAL_MODE_POWER_OFF is not set I did not intend to activate these settings when I compiled the kernel, but as this is certainly not an idle system, it seems like there may be a bug lurking. The log excerpts below are annotated. Sorry if it is too long. This section of the log is from running JACK overnight: [log] Hopefully this is not too much information. You did ask for details... Lee Revell Mindpipe Audio
From: Andrew Morton [email blocked] Subject: Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch Date: Mon, 12 Jul 2004 16:31:41 -0700 Lee Revell [email blocked] wrote: > > On Sun, 2004-07-11 at 01:25, Andrew Morton wrote: > > What we need to do is to encourage audio testers to use ALSA drivers, to > > enable CONFIG_SND_DEBUG in the kernel build and to set > > /proc/asound/*/*/xrun_debug and to send us the traces which result from > > underruns. > > > > OK, here goes. The following traces result from running JACK overnight > like so, on an otherwise idle system. Hardware is a VIA EPIA 6000, with > a 600Mhz C3 processor. Kernel is 2.6.7 + volunatary_preempt patch. > voluntary_preempt and kernel_preemption are both on. > > jackd -v --realtime -d alsa --outchannels 2 --rate 48000 --shorts > --playback --period 32 --nperiods 2 > > These settings require less than 666 microseconds scheduler latency. > The average performance is quite good - 5-20 *microseconds*! OK, thanks. The problem areas there are the timer-based route cache flushing and reiserfs. We can probably fix the route caceh thing by rescheduling the timer after having handled 1000 routes or whatever, although I do wonder if this is a thing we really need to bother about - what else was that machine up to? resierfs: yes, it's a problem. I "fixed" it multiple times in 2.4, but the fixes ended up breaking the fs in subtle ways and I eventually gave up.
From: Paul Davis [email blocked] Subject: Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch Date: Mon, 12 Jul 2004 19:58:57 -0400 >resierfs: yes, it's a problem. I "fixed" it multiple times in 2.4, but the >fixes ended up breaking the fs in subtle ways and I eventually gave up. andrew, this is really helpful. should we conclude that until some announcement from reiser that they have addressed this, the reiserfs should be avoided on low latency systems? --p
From: Andrew Morton [email blocked] Subject: Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch Date: Mon, 12 Jul 2004 17:06:49 -0700 Paul Davis [email blocked] wrote: > > >resierfs: yes, it's a problem. I "fixed" it multiple times in 2.4, but the > >fixes ended up breaking the fs in subtle ways and I eventually gave up. > > andrew, this is really helpful. should we conclude that until some > announcement from reiser that they have addressed this, the reiserfs > should be avoided on low latency systems? > It seems that way, yes. I do not know how common the holdoffs are in real life. It would be interesting if there was a user report that switching from reiserfs to ext2/ext3 actually made a difference - this would tell us that it is indeed a real-world problem. Note that this info because available because someone set /proc/asound/*/*/xrun_debug. We need more people doing that.
From: Lee Revell [email blocked] Subject: Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch Date: Mon, 12 Jul 2004 21:01:16 -0400 On Mon, 2004-07-12 at 20:06, Andrew Morton wrote: > > It seems that way, yes. I do not know how common the holdoffs are in real > life. It would be interesting if there was a user report that switching > from reiserfs to ext2/ext3 actually made a difference - this would tell us > that it is indeed a real-world problem. > This was not a synthetic benchmark, I would consider this a 'real-world' problem now. Repeating the test with ext3 would just tell you whether it has the same problem. If it is neccesary to get the reiserfs issue addressed, I will repeat the test with an ext3 system in the next few days, I would like to hear from reiser on this before doing much more. > Note that this info because available because someone set > /proc/asound/*/*/xrun_debug. We need more people doing that. > - This goes back to the need for ALSA documentation. Someone needs to write some. This will probably require paying that person. Hopefully SuSe is working on this, though I suspect I would have heard something.
From: Lee Revell [email blocked] Subject: Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch Date: Mon, 12 Jul 2004 20:17:04 -0400 On Mon, 2004-07-12 at 19:31, Andrew Morton wrote: > > OK, thanks. The problem areas there are the timer-based route cache > flushing and reiserfs. > > We can probably fix the route caceh thing by rescheduling the timer after > having handled 1000 routes or whatever, although I do wonder if this is a > thing we really need to bother about - what else was that machine up to? > Gnutella client. Forgot about that. I agree, it is not reasonable to expect low latency with this kind of network traffic happening. I am impressed it worked as well as it did. > resierfs: yes, it's a problem. I "fixed" it multiple times in 2.4, but the > fixes ended up breaking the fs in subtle ways and I eventually gave up. > Interesting. There is an overwhelming consensus amongst Linux audio folks that you should use reiserfs for low latency work. Should I try ext3? Lee
From: Andrew Morton [email blocked] Subject: Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch Date: Mon, 12 Jul 2004 17:46:39 -0700 Lee Revell [email blocked] wrote: > > > resierfs: yes, it's a problem. I "fixed" it multiple times in 2.4, but the > > fixes ended up breaking the fs in subtle ways and I eventually gave up. > > > > Interesting. There is an overwhelming consensus amongst Linux audio > folks that you should use reiserfs for low latency work. It seems to be misplaced. A simple make-a-zillion-teeny-files test here exhibits a 14 millisecond holdoff. > Should I try ext3? ext3 is certainly better than that, but still has a couple of potential problem spots. ext2 is probably the best at this time.
From: Jan Depner [email blocked] Subject: Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch Date: Mon, 12 Jul 2004 19:58:08 -0500 Is this problem strictly with reiserfs in 2.6 or is it also a problem with 2.4. I actually experienced fewer xruns with reiserfs vs ext3 on 2.4 (preempt/ll) but I have no hard numbers to back that up. Jan
From: Andrew Morton [email blocked] Subject: Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch Date: Mon, 12 Jul 2004 18:00:29 -0700 (please don't top-post) Jan Depner [email blocked] wrote: > > Is this problem strictly with reiserfs in 2.6 or is it also a problem > with 2.4. I actually experienced fewer xruns with reiserfs vs ext3 on > 2.4 (preempt/ll) but I have no hard numbers to back that up. Actually, the 2.4 low-latency patch does still have some reiserfs fixes, so it's probably better than reiserfs in 2.6.
Interesting
but what of XFS or (my fav fs now) JFS?
Yeah, I'd like to know, too.
As XFS is specially planned for streaming content... also, what advantages do you see in JFS? (I mean in the context of desktop use).
Ok, OT.
Ok, I admit this is going way offtopic. So please disconsider the question.
JFS doesn't seem to bring anything to the table
JFS is not as fast as ReiserFS when dealing with lots of small files, not as fast as XFS when dealing with large files and/or huge constant data flow to/from disk, not as robust as Ext3 in case of crashes.
So i couldn't find any usage for it.
not bad, but doesn't come up clean after hard crash
I tested JFS not thouroughly, but casually, for multimedia work.
What I liked about it was it's predictable response, no big flushes, very polite, not hitting the CPU, especially in 2.6 kernel.
But after hard crashes, it reliably did not reply journal correctly, and had to be remount -o ro and fscked.
This is on scsi disks, without write caching. Of course, this must be fixable, in the future I look forward to evaluating it again.
So although the max latency may not be lowest (I didn't test, and don't seek to guess, but for the sake of argument and on the balance of probability, let's assume it isn't), even so, it's average seemed quite good. Same case for throughput, not the fastest, but nice and steady.
Simeon
it's fine
XFS is actually best for video work and, in general, for situations where you use very large files (> 1 GB). For a video production machine, a simple reformat of the working disk array to XFS will decrease the frame drops dramatically.
As much as i tried, for audio work (smaller files, smaller bitrate than with video) i couldn't find any difference between Ext3, ReiserFS and XFS. I get about the same amount of xruns on each of them.
Ext3 seems the most robust in case of crashes, so i guess that's it: XFS for the spool array, Ext3 for the system drives.
I would hope you didn't see a
I would hope you didn't see any difference on "small" files because... they should all be in cache! Which makes me wonder why people working on small files care about filesystems? If you have to go to disk, in general, you are facing a millisecond-type latency anyway. I saw, get it all into cache, and if you're worried about performance, spring for the extra RAM.
Not quite true
Eventually those files end up committed to disk. Some filesystems perform poorly on large numbers of small files. Typically, it's due to poor handling of metadata.
For instance, ext2 had pretty crappy performance when there were many files in one directory. I seem to recall filename lookup times grew quadratically or worse. (That's where all the "htree" patches came in.) Also, each small file has an inode associated with it. Depending on how all those are laid out on disk, you could get into some thrashing scenarios if you rifle through many small files.
For large numbers of short-lived small files, you're correct. For large numbers of persistent small files, you should switch filesystems to an appropriate filesystem. Reiser seems to focus on optimizing this case.
I once used a program here at work which essentially abused the filesystem as a database (each "key" was its own file, with a huge tree of thousands of such keys) and boy was it painful waiting for 'ls' and 'du' to operate on that tree. The right filesystem would've made that much nicer, I think.
Not just a few small files, b
Not just a few small files, but tons and tons of small files. Think: a Squid disk cache.
Really?
"Which makes me wonder why people working on small files care about filesystems?"
Well, maybe not in desktop world, but... Consider busy email server, storing messages in Maildir format. Add to it increased quotas expected by today's users... See it now?
Offtopic - Reiser4
What is the state of reiser4?
I remember it was said to be practically done, just two issues to be
addressed, etc..?
Re:Offtopic - Reiser4
There are beta release patches available.
Performs _very_ good when you look at the benchmarks on their webside.
Reiser4 performance
I read an interesting article in the iX (german it magazine), and the performance seems to be very good, but the CPU load seems to be very high too. So it can be, that when using for everyday use the latencies and FS performance can be poor.
We'll see...
First, if you handle more dat
First, if you handle more data then you may use more cpu too IMHO. In other words, they should measure it in total cpu usage instead of the average cpu usage.
Second, they aren't done optimizing it yet, making it stable is their priority.
Never trust benchmarks, or any stats for that matter. Consider them a strong hint if they're done well, not much more. You should test for yourself what is fastest for the things you do (Yes, that's a benchmark too ;-).
"overwhelming consensus" is an overstatement
There are people who seem to prefer ReiserFS, whatever the reason, but "overwhelming consensus" is clearly an exageration.
For normal audio work i couldn't find any practical difference between Ext3 and ReiserFS. I get about the same xruns (lost parts of the sound stream) on both of them, in the same conditions. Since Ext3 seems more robust in case of system crashes (which are not uncommon when playing with customized kernels etc.) i pretty much settled to using Ext3 for most partitions.
For video work XFS is clearly better, as it seems to be better in most cases when working with large files (multi-gigabyte). In situations where frames were dropped constantly on Ext3 or ReiserFS, a simple reformat to XFS caused the drops to disappear entirely without any other change being required.
In conclusion, the hard-drive in my multimedia PC which is dedicated for keeping the multimedia files (the "multimedia spool" hard-drive) has one partition, formated XFS. It does not hurt audio (i get about the same amount of xruns in the same conditions), but it helps video a lot.
All other partitions are Ext3.
We need reimplement new ext2a & ext3a & mke2afs
why do not we reimplement ext2a and ext3a and mke2afs to support more than 4 GiB files and to forget 64bit-XFS for video?
open4free ©
& e2afsck
& e2afsck ;)
It's not the support for very
It's not the support for very large files that matters in this case, it's the fact that XFS was designed from bottom-up for handling large files and/or fast read-writes from-to those files and/or concurrent read-write streams in multi-CPU environments.
In other words, a video engineer's dream. :-) But it's pretty good for databases too, or any other situation when you have to deal with files larger than 100 megs or so.
Uh, check your data.
According to this page, with the current default ext2/ext3 block size (4K), the maximum file size on ext2/ext3 is 2TB. Even with the old default block size (1K), the max filesize is 16GB.
Really the limitations on 32-bit platforms exist in the programs themselves that are not large-file aware.
Added at 4:46PM:
Something tells me the big hiccup for large files under ext2/ext3 is the allocation and writeback policy. Since XFS is extent based (as opposed to hierarchical lists of blocks), I imagine it is highly effective at allocating large tracts of sequential space for high-rate writers. I seem to recall ext2/3 try to do something similar, but at a coarse grain. Those lumps in available bandwidth would turn into dropped frames. XFS can just keep extending the preallocation window as more data arrives.
reiserfs has data=ordered since 2.6.6
ReiserFS has data=ordered since 2.6.6 and should be as good
as ext3 regarding system crashes. (No more garbled files)