"Incidentally i was thinking about using KVM for automated testing. Important pieces of hardware should get an in-KVM simulator/emulator, that way developers who do not own that hardware can do functionality testing too," Ingo Molnar suggested during a thread discussing a SCSI driver bug fix. Linus Torvalds was originally unimpressed by the idea:
"Using emulators to test device drivers is almost certain to be pointless. The problem with device drivers tends to be timing issues, odd hardware interactions, and lots of strange (and sometimes undocumented) behaviour and dependencies (eg things like 'you have to wait 50us after setting the reset bit until the hardware has actually reset'). These are all things that you'd generally not catch in emulation - because the emulation by necessity is only going to be a very weak picture of the real thing."
Alan Cox countered, "for some things. I do it a bit because you can use it to fake failures that are tricky to do in the real world. It won't tell you the driver works but its surprisingly good for testing for races (forcing IRQ delivery at specific points), buggy hardware you don't posses, and things like media failures and timeouts your real hardware refuses to do." Linus acquiesced conditionally, "I do agree that you likely find bugs, even if quite often it's exactly because the behaviour is something that will never happen on real hardware," then acknowledged previous debugging efforts by Alan, "but failure testing is very useful - I forget who it was who debugged some driver by taking a CD and just scratching it mercilessly to induce read errors ;)" Ingo added, "something like that wont enable 100% coverage (or even reasonable coverage for most hardware), so it's no replacement for actual hard testing, but it could push out the domain of minimally tested code quite a bit and increase the quality of the kernel."
From: Ingo Molnar
Subject: Re: [patch] scsi: fix crash in gdth_timeout()
Date: Oct 15, 12:27 pm 2007
* Boaz Harrosh <bharrosh@panasas.com> wrote:
> > Honestly, the driver is probably rarely run by people that lack the
> > hardware, I bet...
>
> It was all "flight by instruments only". I called for HW testers and
> none came forward. All these changes, apart from "successful downgrade
> to messy" where also needed in order to push important changes to
> scsi.
>
> But a little bird said that QEMU might simulate this HW. SO I guess it
> is QEMU time for me.
heh. Incidentally i was thinking about using KVM for automated testing.
Important pieces of hardware should get an in-KVM simulator/emulator,
that way developers who do not own that hardware can do functionality
testing too. So basically the highest-quality drivers would have an
"inverse driver" in KVM, which simulates the hardware. (that model is
evidently useful to the hardware maker even for new hardware: it can
then also be used to test the Linux compatibility and Linux performance
of future planned releases of the hardware, etc.)
Ingo
-
From: Linus Torvalds
Subject: Re: [patch] scsi: fix crash in gdth_timeout()
Date: Oct 15, 12:38 pm 2007
On Mon, 15 Oct 2007, Ingo Molnar wrote:
>
> heh. Incidentally i was thinking about using KVM for automated testing.
Using emulators to test device drivers is almost certain to be pointless.
The problem with device drivers tends to be timing issues, odd hardware
interactions, and lots of strange (and sometimes undocumented) behaviour
and dependencies (eg things like "you have to wait 50us after setting the
reset bit until the hardware has actually reset").
These are all things that you'd generally not catch in emulation - because
the emulation by necessity is only going to be a very weak picture of the
real thing.
So I suspect you can find the easy stuff, but only by writing insanely
complex device model descriptions in the emulator environment itself, and
only for those device models that have actually been written.
Can it be donein theory? Sure. Practically? Not so sure. Would it help? I
suspect the effort to do the device model would be many times bigger than
*any* conceivable effort to just fix the driver bugs as they get found
through other means.
So we could perhaps have *really* good emulated hardware for a few models
of hw out there, but likely they'd be fewer and less varied platforms than
most kernel developers end up having hidden under their desk anyway..
Linus
-
From: Alan Cox
Subject: Re: [patch] scsi: fix crash in gdth_timeout()
Date: Oct 15, 1:08 pm 2007
On Mon, 15 Oct 2007 12:38:06 -0700 (PDT)
Linus Torvalds <torvalds@linux-foundation.org> wrote:
>
>
> On Mon, 15 Oct 2007, Ingo Molnar wrote:
> >
> > heh. Incidentally i was thinking about using KVM for automated testing.
>
> Using emulators to test device drivers is almost certain to be pointless.
For some things. I do it a bit because you can use it to fake
failures that are tricky to do in the real world. It won't tell you the
driver works but its suprisingly good for testing for races (forcing IRQ
delivery at specific points), buggy hardware you don't posess, and things
like media failures and timeouts your real hardware refuses to do.
Alan
-
From: Linus Torvalds
Subject: Re: [patch] scsi: fix crash in gdth_timeout()
Date: Oct 15, 1:15 pm 2007
On Mon, 15 Oct 2007, Alan Cox wrote:
>
> For some things. I do it a bit because you can use it to fake
> failures that are tricky to do in the real world. It won't tell you the
> driver works but its suprisingly good for testing for races (forcing IRQ
> delivery at specific points), buggy hardware you don't posess, and things
> like media failures and timeouts your real hardware refuses to do.
Heh. I do agree that you likely find bugs, even if quite often it's
exactly because the behaviour is something that will never happen on real
hardware.
But failure testing is very useful - I forget who it was who debugged some
driver by taking a CD and just scrathing it mercilessly to induce read
errors ;)
Having a really *bad* HW emulator can certainly work that way too, even if
it also would probably end up hitting just a few of the potential error
paths..
Linus
-
From: Ingo Molnar
Subject: Re: [patch] scsi: fix crash in gdth_timeout()
Date: Oct 15, 3:23 pm 2007
* Linus Torvalds <torvalds@linux-foundation.org> wrote:
> On Mon, 15 Oct 2007, Ingo Molnar wrote:
> >
> > heh. Incidentally i was thinking about using KVM for automated
> > testing.
>
> Using emulators to test device drivers is almost certain to be
> pointless.
something like that wont enable 100% coverage (or even reasonable
coverage for most hardware), so it's no replacement for actual hard
testing, but it could push out the domain of minimally tested code quite
a bit and increase the quality of the kernel. Races are always tough and
so are bugs on the side of the hardware, but it's the silly boot-time
crash showstoppers and "device does not work anymore" mistakes that
causes us to lose most of the testers and early adopters.
I'm not really worried about the 1% of bugs that are tough to find and
fix (because they are actually fun to find and fix), i'm worried about
the 99% easy and boring bugs - because they annoy users just as much as
the exciting bugs do. If we fix them faster then there's more time (and
more tester stamina) left for the harder to find bugs.
so i think adding redundancy in form of a simplified hw emulator can
certainly not hurt and fundamentally increases robustness - and it will
definitely reduce the chance for a whole host of stupid bugs that are
not in the hardware but are in the ~4 million lines of Linux driver
codebase. Limits and scalability would also be testable: "if i put 32 of
these networking cards into a Linux box, will the Linux driver blow up".
not that i think this is realistic for any significant portion of the
hardware currently - unless some hw maker starts doing it. But KVM will
have a good portion of the core PC platform emulated (APIC, etc.) - and
that's a nice step forward already.
Ingo
-
I'm wondering, how much
I'm wondering, how much effort is actually needed to write a reasonable emulated implementation for some hardware, compared to developing the driver? Could it be done more easily in dynamic languages?
Depends of course on the
Depends of course on the hardware complexity, but it's not too difficult. Language choice may be limited by the emulator.
Basically the emulator talks to the hardware by doing IO or register accesses (reverse of the driver in a way). The hardware can raise or lower IRQs and access memory.
Display, network, serial/parallel, keyboard/mouse etc. hardware may use the standard means to exchange data with the outside world.