From: Soren Sandmann <sandmann@redhat.com>

Subject: [PATCH] x86: add the debugfs interface for the sysprof tool
The sysprof tool is a very easy to use GUI tool to find out where

userspace is spending CPU time. See

http://www.daimi.au.dk/~sandmann/sysprof/

for more information and screenshots on this tool.
Sysprof needs a 200 line kernel module to do it's work, this

module puts some simple profiling data into debugfs.
Signed-off-by: Soren Sandmann <sandmann@redhat.com>

Signed-off-by: Arjan van de Ven <arjan@linux.intel.com>

---

 arch/x86/Kconfig.debug    |   10 ++

 arch/x86/kernel/Makefile  |    2 +

 arch/x86/kernel/sysprof.c |  200 +++++++++++++++++++++++++++++++++++++++++++++

 arch/x86/kernel/sysprof.h |   34 ++++++++

 4 files changed, 246 insertions(+), 0 deletions(-)

 create mode 100644 arch/x86/kernel/sysprof.c

 create mode 100644 arch/x86/kernel/sysprof.h
diff --git a/arch/x86/Kconfig.debug b/arch/x86/Kconfig.debug

index 12c98ea..8eb06c0 100644

--- a/arch/x86/Kconfig.debug

+++ b/arch/x86/Kconfig.debug

@@ -206,6 +206,16 @@ config MMIOTRACE_TEST
 	  Say N, unless you absolutely know what you are doing.
+config SYSPROF

+	tristate "Enable sysprof userland performance sampler"

+	depends on PROFILING

+	help

+	  This option enables the sysprof debugfs file that is used by the

+	  sysprof tool. sysprof is a tool to show where in userspace CPU

+	  time is spent.

+

+	  When in doubt, say N

+

 #

 # IO delay types:

 #

diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile

index 4a4260c..1e8fb66 100644

--- a/arch/x86/kernel/Makefile

+++ b/arch/x86/kernel/Makefile

@@ -80,6 +80,8 @@ obj-$(CONFIG_DEBUG_NX_TEST)	+= test_nx.o

 obj-$(CONFIG_VMI)		+= vmi_32.o vmiclock_32.o

 obj-$(CONFIG_PARAVIRT)		+= paravirt.o paravirt_patch_$(BITS).o
+obj-$(CONFIG_SYSPROF)		+= sysprof.o

+

 ifdef CONFIG_INPUT_PCSPKR

 obj-y				+= pcspeaker.o

 endif

diff --git a/arch/x86/kernel/sysprof.c b/arch/x86/kernel/sysprof.c

new file mode 100644

index 0000000.....
[ message continues ]

Seems a poor idea to me.  Sure, oprofile is "hard to set up", but not if

your distributor already did it for you.
Sidebar: the code uses the utterly crappy register_timer_hook() which 
a) is woefully misnamed and
b) is racy and 
c) will disrupt oprofile if it is being used.  And vice versa.
This code adds a new kernel->userspace interface which is not even

documented in code comments.  It appears to use a pollable debugfs file in

/proc somewhere, carrying an unspecified payload.
What happens when multiple processes are consuming data from the same
checkpatch used to warn that linux/uaccess.h is preferred over asm/uaccess.h

but this is another of those checkpatch features which seems to have
We read `head' before taking the "lock".  Another CPU could modify it after
Why not use spin_trylock()?  Then you get lockdep support too.
And why not use spin_lock()?  At least a comment should be added explaining
And the changelog should explain and justify why we cannot profile root's

code.  I cannot begin to imagine why it was done and I cannot fathom why it

passed uncommented in documentation, code, changelog and "review" comments.

It greatly reduces the usefulness of an already dubious feature.
If this limitation _was_ documented then I'd be in a position to ask what is

special about root, as opposed to some non-root user who has <unspecified>

capabilities.  And why we penalise a root who has dropped <unspecified>

capabilities.  etcetera.
This is ambiguous in a containerised environment.
Ingo, please be alert for anything which exposes raw pids to userspace.
I don't know what a correct and suitable interface might be - perhaps Pavel
The (absent) interface documentation should explain what happens when a
`head' can just merrily advance over `tail' and there is no notification to
Please put a blank line between end-of-variables and start-of-code.
This seems to be a wrong return value?  Shouldn't it just return zero if

...
[ message continues ]

On Sat, Feb 23, 2008 at 10:11 AM, Andrew Morton
I wonder if sysprof should hook to the same interrupt as oprofile then?
On Sat, Feb 23, 2008 at 10:11 AM, Andrew Morton
Agreed. This seems like a perfect fit with relayfs.

--

oprofile uses register_timer_hook() for its oh-crap-nothing-else-works

fallback iirc.  It's a useful fallback: I used it a few centuries ago on

some el-cheapo VIA CPU-based thing we had at Digeo.
It's unclear to me whether all this stuff works with NO_HZ=y, btw.  Didn't

we just lose the regular timer interrupts which these clients depend upon?
--

On Sat, Feb 23, 2008 at 10:11 AM, Andrew Morton
Have you tried sysprof? It's really nice to setup and use compared to

oprofile when profiling user-space.
On Sat, Feb 23, 2008 at 10:11 AM, Andrew Morton
Hmm, it does include <linux/proc_fs.h> but seems to set up a regular

debugfs directory (usually mounted in /sys/debug) and not proc. Arjan?

--

Wanna see how I use oprofile?
box:/home/akpm> akpm-oprofile gcc t.c

Daemon not running

Daemon not running

Using 2.6+ OProfile kernel interface.

Reading module info.

Using log file /var/lib/oprofile/oprofiled.log

Daemon started.

Profiler running.

t.c:6:12: error: token ""x"" is not valid in preprocessor expressions
real    0m0.262s

user    0m0.004s

sys     0m0.004s

Stopping profiling.

Stopping profiling.

Killing daemon.

CPU: CPU with timer interrupt, speed 0 MHz (estimated)

Profiling through timer interrupt

samples  %        symbol name

625      99.2063  mwait_idle

1         0.1587  __handle_mm_fault

1         0.1587  clear_page

1         0.1587  do_page_fault

1         0.1587  flush_tlb_page

1         0.1587  pfn_valid

box:/home/akpm> 
One thirteen-character command!  Why?  Because I actually got off my butt

and wrote a script to hide low-level details.  I wrote the thing five years

ago and don't remember anything about what's in it.
I didn't need to write a new kernel module to enable that

thirteen-character shell script, and I don't believe one needs to write a

new kernel module to put a nice easy-to-use GUI around oprofile either.
This is one of those i-cant-believe-im-having-this-discussion discussions.

--

Hi Andrew,
Sysprof tracks the full stack frame so it can provide meaningful call

tree (who called what) which is invaluable for spotting hot _paths_. I

don't see how oprofile can do that as it tracks instruction pointers only.
			Pekka

--

You could try passing the --callgraph option to opcontrol.
--

Nicholas Miell <nmiell@comcast.net>
--

Hmm, perhaps I am missing something but I don't think that does what

sysprof does. At least I can't find where in the oprofile kernel code

does it save the full stack trace for user-space. John?
                        Pekka

--

Hi,
Ok, so as pointed out by Nicholas/Andrew, oprofile does indeed do

exactly what sysprof does (see

arch/x86/oprofile/backtrace.c::backtrace_address, for example). So,

Soeren, any other reason we can't use the oprofile kernel module for

sysprof?

--

as i pointed it out earlier in the thread, the oprofile implementation

seems buggy because when an event comes from NMI context

__copy_from_user_inatomic() can fault and re-enable NMIs - causing

possible stack recursion/corruption. Does not look like an easy fix.
	Ingo

--

yes, it's very nice and very usable - and that's all that matters

really. It's almost a _duty_ of the mainstream kernel to include that

trivial 200 lines of sysprof code, given how poor instrumentation

support is on Linux.
As a comparison, here's a session of a newbie developer, meeting

oprofile for the first time in his life (using a fresh package,

oprofile-0.9.3-6.fc8):
  ---------------------->

  [ Newbie: WTF, no GUI tool? ]

  [ Narrator: we lose 90% of the developers at this point. ]

  [ Newbie is adventurous and has heard about opcontrol and tries it. ]
  # opcontrol
  [ Newbie sees tons of output. User scratches head. After looking

    around, finds the following option listed:

    "-s/--start       start data collection". That must be it! ]
  # opcontrol -s

  No vmlinux file specified. You must specify the correct vmlinux file, e.g.

  opcontrol --vmlinux=/path/to/vmlinux

  If you do not have a vmlinux file, use

  opcontrol --no-vmlinux

  Enter opcontrol --help for full options
  [ Newbie: WTF? Doesnt oprofile think that what I want to do is to

    profile ... the currently running kernel, wherever a kernel is

    and whatever a vmlinux might be?? ]
  [ Narrator: At this point oprofile has confused about 99% of all

    user-space developers who have no freaking idea about what a vmlinux

    is. ]
  [ Newbie user figures that --no-vmlinux might be the right option: ]
  # opcontrol -s --no-vmlinux

  Option "--setup" not valid with "-s".
  [ Newbie: WTF? what not valid? Why should i care? Damnit, i only want

    to profile stuff!!! ]
  [ The newbie user eventually finds out that opcontrol help text is

    buggy and that -s does not mean --start, but --setup. ]
  [ Narrator: we now have lost 99.99% of the first-time users. ]
  [ Newbie, armed with this nontrivial piece of information: ]
  # opcontrol --start --no-vmlinux
  Using default event: CPU_CLK_UNHALTED:100000:0:1:1

  Using 2.6+ OProfile kernel interface.

  Using l...
[ message continues ]

Hi Ingo,
In all fairness, there's a GUI for oprofile too [1] but I don't think

it supports call trees as sysprof does which is IMHO the killer

feature of the latter.
  1. http://labs.o-hand.com/oprofileui/

--

two things.
Firstly, this isnt an oprofile replacement, this is a pretty separate

concept. Sysprof is more of a tracer than a profiler. (and we are

currently working on merging it into ftrace)
Secondly, real developers who tune user-space code disagree with your

characterisation of oprofile being easy to use. _Please_ just ask any

developer around you who hasnt used oprofile before to try sysprof

versus oprofile - without looking at any docs. Give him minimal context

and two starting points: "opcontrol" and "sysprof" - nothing else. Give

him a very simple test.c code:
  void test_fn(void)

  {

	for (;;)

		gettimeofday(0, 0);

  }
  int main(void)

  {

	test_fn();

  }
and ask him to try oprofile and then to try sysprof as a comparison - to

and figure in which app and which function the overhead is.
then measure the time it takes to solve this task via the two tools and 
that was for x86.git#testing, it's not even in x86.git#mm.
It's 200 lines of pretty well isolated code for something that is

already much more usable to me than 10 years of oprofile. Really, i'd

much rather take 200 lines of poor kernel code written by a userspace

developer for stuff that _already works better_, than to have ~2000

lines of oprofile code and an unusable (to me) user-space tool written

by kernel developers.
	Ingo

--

I don't understand the distinction and I don't see what sysprof (as defined

by its kernel->userspace interface) can do which oprofile cannot.
This is yet another thing which should have been in the damned changlog but
I think you should drop it and we should see a replacement patch which has

all the bugs, inefficiencies and deficiencies addressed and which has a
afacit all of these criticisms surround oprofile's userspace tools only.

--

Obviously I hold no sway here, so there's little point in my continuing

to try and fight this madness, but I have to say my piece. Don't worry,

I'll leave it after this - I know Ingo always gets his way.
Now work out whether this is a result of the kernel code or not. Whilst

there's probably no point me stating this: I object most strenuously to

the idea of merging duplicate code that does part of what oprofile does.
Soren wrote sysprof when he tried an earlier version of oprofile and

found it slightly non-obvious. Instead of doing any of these things:
- complaining to the oprofile list

- checking a version of oprofile written in the last two years

- fixing oprofile user space to be more obvious
he just went on right ahead and re-implemented it so he could write a

GUI on top of it. That *exact same GUI*  could have been on top of

oprofile. Usability problems with opcontrol or opreport could have been

reported (or even - gosh - fixed). Soren was too lazy.
I honestly can't believe that you're telling me you'd rather have more

kernel code than make some tweaks (which, by the way, you have NEVER

complained about to me or to the oprofile mailing list until now)
Wrong. It's shipped with a (rather poor) GUI pretty much since

inception. More recently, there's been at least two GUIs. For example:
Firstly, the distributions should have set this up automatically. That

they don't is a distributor bug. The sheer madness of Linux not leaving

a vmlinux file in a stable known location is hardly something oprofile

can be blamed for.
Secondly, not ONE PERSON INCLUDING YOU has EVER suggested to us that we
It's astonishing that you would know about this, complaining about this,

but not file a bug report. Same goes for the rest.
And once again, please explain how the buggy shell script has anything

to do with kernel code.
What, exactly, happened to scratching an itch? If everyone should be

hanging his head in shame, Ingo, why haven't you used your considerable

influence to ...
[ message continues ]

I think it's fair to say that oth oprofile and sysprof can use some

improvements.  There are a couple of questions that immediately come

to mind, including the most obvious one, *if* as you John clams, the

oprofile kernel had all of the functionality for the GUI, why wasn't

it used --- could it *perhaps* because the kernel interface for

oprofile wasn't documented well?  Heck, even if sysprof is 200 lines

of code versus 2000 lines of kernel code, most people don't write

extra code unless it's because the 2000 lines of pre-existing code
Wrong Answer.  People who write userspace helpers *have* to do the

work of the distro's.  It's a bad, bad, bad, Bad, BAD idea to leave it

up to the distributions.  It means that some distributions won't get

it right; other distributions will do it in different ways, making it

harder for users to switch between distro's and making it harder for

people to write distribution-neutral HOWTO's.
There are plenty of things that can be done, including using search

paths to try to find vmlinuz; or maybe even proposing a new standard

such as say for example /lib/modules/`uname -r`/vmlinux being a

synlink to the location of vmlinux.  We already have

/lib/modules/`uname -r`/build and /lib/modules/`uname -r`/source, for

example.
The abdication of responsibility and the lack of trying to solve the

usability issues is one of the things that really worries me about

*all* of Linux's RAS tools.  We can and should do better!  And it's

really embarassing that the RAS maintainers seem (I assume you are one

of the oprofile maintainers), seem to be blaming this on the victims,

the people who are complaining about using *your* tool.  Yes, it's a

shame that Ingo didn't try to fix your tool; open source, and scratch

your own itch and all of that.  To be sure.  But at the *same* *time*

don't you have enough pride to take a look at a tools which so

obviously has massive lacks in the usability department, and tried to

fix it years ago?  There's more than enough blame to go a...
[ message continues ]

Hi Ted. I would have loved to have fixed it myself, or had one of the

other oprofile contributors do so. Unfortunately I have no control over

what the distributions do. For example:
At the time when I was trying to fix this, I wasn't aware of any way to

propose a new standard and get distributions to follow it - is there

some way now? Informally I discussed this problem several times with

many people without any resolution. As regards searching informal paths,

this is extremely risky - get the wrong vmlinux and we end up with
I'm not abdicating responsibility: I no longer maintain oprofile,

haven't for a long time, and the other contributors don't have any real

time to spend on it either.  I'm well aware there are improvements to be

made: when I (and others) had time, this stuff was improving massively

with each release. When Soren et al had a need for a simple tool + GUI,

they should have helped out with this: it's simply good engineering to

not duplicate efforts. I didn't even get a /single/ email.
regards,

john

--

The way that /lib/modules/`uname -r`/build was standardize was via

mail to LKML, years ago.  It was declared so, "make install" for base

kernel Makefiles did that, and the distro's picked it up pretty

quickly thereafter.
In terms of picking the right vmlinux, how about a kernel patch which

stashes the MD5 checksum of vmlinux in a convenient location the

compressed kernel which can be pulled out via querying

/sys/kernel/vmlinux_cksum?  If the problem is making sure you have the

right vmlinux, there are some fairly simple ways of assuring this ---
Let me make it clear that the problems go far beyond oprofile.  I have

similar issues of disquietude about the easy of use of SystemTap,

kdump, and all of the other RAS system tools.  It may be the problem

is that the companies who fund the development of the RAS tools are

stopping before they can be made turn-key and easy to use by kernel

developers, as opposed to assuming that the distro's will do all of

the hard work productizing them and actually making them *usuable*.
The problem is that not enough mainline kernel developers use these

tools, mostly because they aren't easy enough for them to use.  I

remember complaining about kdump, and I got the same answer, "Oh, it's

the distro's job to make it easy to use."  Which is fine, except that

means very few people actually use it (how many kernel developers use

RHEL and SLES as their day-to-day development OS, as opposed to Fedora

or Debian, et. al.?), and since there aren't lots of kernel developers

using it, once the people who are funded to support the RAS tools get

reassigned to other projects, what's left is in a terrible shape to be

used by mainline kernel developers, and then the tools effectively

become unused and then unmaintained.....
						- Ted

--

i agree with most of your mail but i beg to differ with what you wrote

about my role :-/ The specific bug/issue i discovered with oprofile i

discovered on the very day i wrote about it to lkml.
In any case i'm not the one to fix oprofile - i cannot fix or report all

itches that i have in ~1 billion lines of userspace - i would have to

spend my whole life complaining ;-)
I'm the one to make sure that patches for useful userspace tools that

get submitted to me eventually go upstream, one way or another. Sysprof

has been around for years, had to rely on a (trivial) external module

and AFAIK the feature even predates oprofile's stack-trace support. The

API is butt-ugly and it's being fixed. So if then it should have been

the oprofile folks porting over sysprof to their new API ... claiming

otherwise would rewrite history i think.
a quick glance at oprofile's stack-trace/callgraph support shows it

being rather buggy: it uses __copy_from_user_inatomic() from NMI

context, which is bad because that can fault and re-enable NMIs, causing

stack recursion/corruption. Fixing it is nontrivial. I'm not sure how

much this feature has been tested - but with a sucky userspace kernel

features _do not get tested_ - it's as simple as that! I'd be happier

with sysprof's pure and simple "we only care about time events" initial

approach and evolve this field via actual interest from users.
	Ingo

--

just to make it clear: that "one way or another" very much includes the

possibility that sysprof is modified to make use of the oprofile APIs -

i.e. no kernel patch needed at all.
	Ingo

--

This is not accurate. Sysprof started by me adding a hierarchical call

view to speedprof, a SIGPROF profiler which was basically a hack in

memprof.
Oprofile did not work on my system at the time (Red Hat 9, I believe),

and the website said that I had to apply a patch to the kernel and

recompile, so I didn't try it.
The hierarchical call view in speedprof worked out so well that I

wrote a simple kernel module to produce system-wide stacktraces, and

fed them into speedprof. Since speedprof was just a hack in memprof, I

wrote a new GUI, and sysprof was born.
It was only later I tried oprofile and found it not only much more

difficult to use, but also much less useful when I did get it to work.
Soren

--

This surprises me. Can you please elaborate on why oprofile is "much

less useful" than sysprof?
Anton - who has used oprofile to analyse and tune databases, JVMs,

	compilers and operating systems. Maybe I've been missing out on

	the killer app for all this time!!!

--

it's OK if you use it full time and if you are amongst the 0.001% of the

developer population we call "the best kernel hackers on the planet" ;-) 
It sucks badly if you use it occasionally and have to re-learn its

non-trivial usage and its quirks every time. As it happens, most

userspace developers are in this second category.
(and i'm not worried about the first category at all - if needed they

can write their own OS from scratch ;-)
	Ingo

--

I dream of being a card carrying member of the club but apparently
Or their own profiling infrastructure!
OK I'm done being a pain, time to be constructive. It's becoming clear

we have some work to do in order to make oprofile easier to use. I've

been involved with the project from the early days, and it's hard to be

objective when it comes to usability issues.
Off the top of my head, there are a number of reasons I think it makes

sense to use the existing oprofile kernel code instead of adding the

sysprof kernel module:
- Wide architecture support. A quick look shows 9 architectures with

  oprofile kernel module code.
- biarch support. You can mix 32 or 64bit userspace and the same oprofile

  userspace works with 32 or 64bit kernels. I'm not sure where sysprof

  sits wrt this.
- Java and Mono support. Oprofile has work underway to communicate with

  a running JVM and produce symbolic debugging information:
http://primates.ximian.com/~massi/blog/archive/2007/Nov-19.html
- Robust sample to binary mapping. It appears that sysprof has a race

  where processes can exit before the user process has a chance to do

  the PID to binary mapping.
- Support for the full set of performance monitor events. A 100 or 250Hz

  timer is useful for simple profiling, but eventually you will want

  either faster sampling or different event sampling (eg cache misses).
Oprofile now has a mode to output XML and it shouldn't take much effort

for sysprof to parse this instead of the binary debugfs file.
Anton

--

Nobody has in any way demonstrated that this is due to the design or

implementation of oprofile's kernel component.

--

Could we please be very careful to not conflate oprofile's kernel support

with oprofile userspace?  For the purposes of this discussion the

distinction is most important.
I _assume_ you're referring to oprofile userspace which I agree is

user-hostile.  It is a rich area for others to improve upon.  In so doing

they might even have a need to change oprofile's kernel code.  But simply

giving up and going off and implementing some parallel kernel code is an

utter last resort and should only be done after it has been shown that the

current kernel code is unfixable.

--

i found out about this particular issue just today, when i wrote this

mail, so consider this as my bugreport.
And i have to say, most of the usability deficits in oprofile are very

obvious, so consider this as a general "the oprofile commands suck in

almost every detail" bugreport. Tools should fundamentally be

one-stop-shops. I personally wouldnt mind the lack of a GUI at all if

the command line tool was _obvious to use_. If the principle was: get

the current histogram to the user. A tool should work _hard_ to get

something (_anything_) useful out by default. Transparently start up any

background state and procesing that is needed - and hide it as much as

possible. Try to figure out where the vmlinux is, if there's any. Do

_not_ put the user through any extra chores if not absolutely necessary.

Display information that tells the user what happened. Etc., etc. -

these are all basic principles.
imagine if the "ls" command, instead of listing files, showed 60 lines

of options, and when i picked "-l" it would, instead of listing files

already, ask me: "do you really mean the current filesystem?". And if i

said "list whatever filesystem i wanted" then it would also say "because

dm-crypt is not configured into the kernel I cannot display encrypted

information, use me with --no-crypt".
and as i sit here, i tried "opreport" once again, to just see what it

does by default. It just hung there, displaying nothing. For minutes.

Then i got suspicious and straced it. It loops in stat()s in one huge

directory that has amassed more than 20 thousand sample files in the

last few hours. This is such a basic usecase, tell me that this never

happened to you and that nobody ever came to the idea to display "sorry,

this might take a few minutes, 10% of the files are processed so far"

sort of feel-good messages to the user?
but generally i cannot fix and report and fight over all the crap that

people do - that would mean i'd have to complain about Linux all day,

non-st...
[ message continues ]

What is the added value over oprofile?
--

On Wed, 20 Feb 2008 19:16:15 +0100
it actually works and is usable by humans ;)
what oprofile doesn't do is the nice userland hierarchy of where cpu time is spend.
--

If you want to reach me at my work email, use arjan@linux.intel.com

For development, discussion and tips for power savings,

visit http://www.lesswatts.org

--

# sysprof
(sysprof:12480): Gtk-WARNING **: cannot open display:
so why not fix oprofile callgraph output and build a GUI on top of

oprofile for those of us who really want RSI from mouse movement :-)
--

On Wed, 20 Feb 2008 19:53:42 +0100
feel free to reinvent a whole GUI just to avoid a 200 line kernel module.

sysprof is here. it works. the gui is REALLY nice.

I think it's the wrong tradeoff though... oprofile exists for how long?
--

If you want to reach me at my work email, use arjan@linux.intel.com

For development, discussion and tips for power savings,

visit http://www.lesswatts.org

--

I guess we have to agree to disagree here. Its plain useless from my
Dunno, years, and has served me well.
The thing I worry about is the wild-growth of duplicate functionality

and interfaces. You might say, 'its in /debug' so no API crap, but if

enough user-space depends on it people _will_ complain if it breaks.
Hopefully someone will consolidate stuff - soon. I can agree with the

fact that the oprofile user-interface is quite horrible, and perhaps the

kernel code isn't pretty (never looked at it), so if people want to

replace it, feel free, but offer a full replacement so we can deprecate

and remove the old stuff, and not carry everything around.
Currently we have: readprofile, oprofile, perfmon and now sysprof.
Also, sysprof is a misnomer, you cannot be a system wide profiler and

have code like:
+       if (!is_user) {

+               /* kernel */

+               trace->pid = current->pid;

+               trace->truncated = 0;

+               trace->n_addresses = 1;

+

+               /* 0x1 is taken by sysprof to mean "in kernel" */

+               trace->addresses[0] = 0x1;

+	}
The kernel is an integral part of the system, it can often help to know

where in the kernel time is spent - even if you're not directly

interested in 'fixing' the kernel.
--

On Wed, 20 Feb 2008 21:58:42 +0100
that's fine. Different tools for different people. sysprof isn't aimed

at kernel developers.
"If all you have is an allen wrench, everything looks like Ikea"
it's ok to have different tools for really different things
--

If you want to reach me at my work email, use arjan@linux.intel.com

For development, discussion and tips for power savings,

visit http://www.lesswatts.org

--

That was speaking from userland days.
But you forgot the more important points about API and wild-growth of

duplicate interfaces.
--

On Wed, 20 Feb 2008 22:44:29 +0100
yes there's multiple interfaces. There are multiple interfaces *today*.

Oprofile/perfmon2 are very focused on CPU events and have complex interfaces,

sysprof has a much more simple interface (and yes, very specific to syspref)

that just focuses on samples.
Sadly, I think there's use for both, and forcing both into the same straightjacket is a mistake imo.
--

If you want to reach me at my work email, use arjan@linux.intel.com

For development, discussion and tips for power savings,

visit http://www.lesswatts.org

--

Usable for me is a cli interface. Also, I absolutely love opannotate.
For a fact, most of my professional userspace development days, I had to

profile on remote machines with no X.
--

I definitely agree there.
It's interesting to note that sysprof requires you to run the GUI as

root in order to work. Maybe Ingo and Arjan are confident there are no

bugs in all the libraries that sysprof links to:
# ldd `which sysprof` | wc -l

39
I'm not.
Actually before someone converted it to debugfs, it was even worse, the

sysprof kernel module exported all profiling information to the world:
-r--r--r-- 1 root root 2060 2008-02-25 23:00 /proc/sysprof-trace
Anton

--

thanks, looks good to me - applied.
	Ingo

--

Woah slow down guys. Did I miss the review?
Yes it's a 200 line patch, but it's a 200 line x86 patch. Surely we

should apply some of the same rigour we did when we merged the oprofile

patch? Is it biarch safe? Will it run on powerpc, arm etc?
I'm still struggling to understand why we need this functionality at

all. Lets argue the ABI and not cloud it with a discussion about

userspace eye candy. What does this give you that is an improvement over

the oprofile kernel-user ABI?
Anton

--

note that it was applied to x86.git#testing. It's as if Andrew applied

something to -mm. This is not a guarantee of upstream merging (at all).
	Ingo

--