As posted to lkml and linux-scsi on 2007-03-15 without reply, see http://marc.info/?l=linux-kernel&m=117395128412313&w=2 for original post: It is not so nice when one can write backup tapes but the tapes cannot be read. I don't know if memory management or the st driver is the culprit, but this is a not so nice situation. I can't even say if the tapes are written correctly as I can't read them (one does not reboot production machines back to 2.4.x just to try to read a backup tape - I don't have 2.6.x older than 2.6.20 on these machines). -- Andreas Steinmetz SPAMmers use firstname.lastname@example.org -
Repeatable oops in our most recently released kernel, nobody bothers to BUG_ON(!PageSlab(page)); that's seriously screwed up. Do you have CONFIG_DEBUG_SLAB enabled? If not, please enable it and retest. -
This is scary. Looking at disassembly of the OOPS: Disassembly of section .text: 00000000 <.text>: 0: 5f pop %edi 1: c3 ret 2: 57 push %edi 3: 89 c1 mov %eax,%ecx 5: 89 d7 mov %edx,%edi 7: 8d 92 00 00 00 40 lea 0x40000000(%edx),%edx d: 56 push %esi e: c1 ea 0c shr $0xc,%edx 11: 53 push %ebx 12: c1 e2 05 shl $0x5,%edx 15: 03 15 40 5d 5a c0 add 0xc05a5d40,%edx At this point, edx has the result of virt_to_page(). 1b: 8b 02 mov (%edx),%eax 1d: f6 c4 40 test $0x40,%ah 20: 74 03 je 0x25 If it's a compound page, look up the real page from ->private. 22: 8b 52 0c mov 0xc(%edx),%edx Now, reload page flags. 25: 8b 02 mov (%edx),%eax And test... 27: a8 80 test $0x80,%al 29: 75 04 jne 0x2f 2b: 0f 0b ud2a 2d: eb fe jmp 0x2d 2f: 39 4a 18 cmp %ecx,0x18(%edx) [snip, snip] EIP is at kmem_cache_free+0x29/0x5a eax: c1800000 ebx: f0ae12c0 ecx: c18f73c0 edx: c1800000 esi: c1919de0 edi: 00000000 ebp: 00001000 esp: f1fe7e14 ds: 007b es: 007b ss: 0068 But somehow eax and edx have the same value 0xc1800000 here. Hmm? Pekka -
Aah, but if you look at contents of the stack: Stack: f0ae12c0 c1919de0 ffffffea c0137f97 00000000 f0ae12c0 c1919e20 c0168d45 f0ae12c0 00001000 c0168fb9 c02a77e3 00001000 00000000 00000000 00000000 00000000 c17bb6e0 00001000 00000000 f1b38be8 00000003 f54ac050 c1b9d6e8 Call Trace: [<c0137f97>] mempool_free+0x48/0x4c [<c0168d45>] bio_free+0x21/0x2c [<c0168fb9>] bio_put+0x22/0x23 You can see that mempool_free is passing a NULL pointer to kmem_cache_free() which doesn't handle it properly. The NULL pointer comes from bio_free() where ->bi_io_vec is NULL because nr_iovecs passed to bio_alloc_bioset() was zero. The question is, why is nr_pages zero in scsi_req_map_sg()? -
Could you try this patch http://marc.info/?l=linux-scsi&m=116464965414878&w=2 I thought st was modified to not send offsets in the last elements but it looks like it wasn't. -
Actually, there are two patches in the email referred to. If the analysis that we're passing NULL to mempool_free is correct, it should be the second one that fixes the problem (the one that checks bio->bi_io_vec before freeing it). Which would mean we have a nr_vecs==0 bio generated by the tar somehow. James -
I think we might only need the first patch if the problem is similar to what the lsi guys were seeing. I thought the problem is that we are not estimating how large the transfer is correctly because we do not take into account offsets at the end. This results in nr_vecs being zero when it should be a valid value. I thought Kai's patch: http://bugzilla.kernel.org/show_bug.cgi?id=7919 http://git.kernel.org/?p=linux/kernel/git/jejb/scsi-misc-2.6.git;a=commitdiff;h=9abe16... fixed the problem on st's side, but I guess not so you are probably right. Here is a patch that dumps the sgl we are getting from st so we can see for sure what we are getting and can decide if we need the first patch, second patch or both.
Oh, I noticed that the subject for the mail references 22.214.171.124 and the patch for st in the bugzilla did not make into 2.6.20 and is not in .3. Could we try the st patch in the bugzilla first? -
On Tue, 20 Mar 2007 00:25:02 +0100 If you're referring to the below patch then it's already in mainline, and has been for a month. Have you tested 2.6.21-rc4? If not, please do so. Perhaps we should merge this into 2.6.20.x? commit 9abe16c670bd3d4ab5519257514f9f291383d104 Author: Kai Makisara <Kai.Makisara@kolumbus.fi> Date: Sat Feb 3 13:21:29 2007 +0200 [SCSI] st: fix Tape dies if wrong block size used, bug 7919 On Thu, 1 Feb 2007, Andrew Morton wrote: > On Thu, 1 Feb 2007 15:34:29 -0800 > email@example.com wrote: > > > http://bugzilla.kernel.org/show_bug.cgi?id=7919 > > > > Summary: Tape dies if wrong block size used > > Kernel Version: 2.6.20-rc5 > > Status: NEW > > Severity: normal > > Owner: firstname.lastname@example.org > > Submitter: email@example.com > > > > > > Most recent kernel where this bug did *NOT* occur: 126.96.36.199 > > > > Other Kernels Tested and Results: > > > > OK 188.8.131.52 > > OK 184.108.40.206 > > OK 220.127.116.11 > > BAD 18.104.22.168 > > BAD 2.6.18-1.2869.fc6 > > BAD 22.214.171.124 + > > BAD 2.6.20-rc5 > > > > NOTE: 2.6.18-1.2869.fc6 is a Fedora modified kernel, all others are from kernel.org > > ... > > Steps to reproduce: > > Get a Adaptec AHA-2940U/UW/D / AIC-7881U card and a tape drive, > > install a recent kernel > > set the tape block size - mt setblk 4096 > > read from or write to tape using wrong block size - tar -b 7 -cvf /dev/tape foo > > Write does not trigger this bug because the driver refuses in fixed block mode writes that are not a multiple of the block size. Read does trigger it in my system. The bug is not associated with any specific HBA. st tries to do direct i/o in fixed block mode with reads that are not a multiple of tape block ...
James, could this also be the cause of a tar based backup going crazy and thinking all data is new under any 2.6.21-rc* kernel I've tested so far with amanda, which in my case uses tar? I've tried the fedora patched tar-1.15-1, and one I hand built right after 1.15-1 came out over a year ago, and they both do it, but only when booted to a 2.6.21-rc* kernel. This obviously will be a show-stopper, either for amanda (and by inference, any app that uses tar), or for the migration of an amanda -- Cheers, Gene "There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order." -Ed Howdershelt (Author) fractal radiation jamming the backbone -
Er, I don't think so .. that sounds like mtime miscompare, which is either a problem with the filesystem or a problem with the way mtime is stored in the tar archive. James -
Well, since the times reported by ls -l --full-time are sane [root@coyote pix]# ls -l --full-time total 924784 -rw-r--r-- 1 root root 985324 2002-06-09 18:14:54.000000000 -0400 0203.jpg [... the rest of a 100k listing, booted to 126.96.36.199-rdsl-0.31] And: [root@coyote pix]# ls -l --full-time total 924784 -rw-r--r-- 1 root root 985324 2002-06-09 18:14:54.000000000 -0400 0203.jpg booted to 2.6.21-rc4 allthough the fractional second is a string of .000000000, even when booted to a tar-unfriendly kernel, then it would tend to point at tar, but two differently built versions of tar have been confirmed as miss-behaving in the presence of a kernel in the 2.6.21 series so far, all of them. I'm going to reboot twice more tonight, once to verify that the output of an ls -l --full-time is as I said above, I'll save this and do that again and clip it in after a reboot to 2.6.21-rc4, and once to 188.8.131.52-rc1 to see if by chance one of those patches is the guilty party. I'll leave the latter running tonight for the amanda run & see what falls out. -- Cheers, Gene "There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order." -Ed Howdershelt (Author) Try to relax and enjoy the crisis. -- Ashleigh Brilliant -