>>>
>>>
>>>>> Current Situation:
>>>>> The dynamic linker mmap()s executable and data sections of our
>>>>> executable but it doesn't call madvise().
>>>>> By default page faults trigger 131072byte reads. To make matters worse,
>>>>> the compile-time linker + gcc lay out code in a manner that does not
>>>>> correspond to how the resulting executable will be executed(ie the
>>>>> layout is basically random). This means that during startup 15-40mb
>>>>> binaries are read in basically random fashion. Even if one orders the
>>>>> binary optimally, throughput is still suboptimal due to the puny readahead.
>>>>>
>>>>> IO Hints:
>>>>> Fortunately when one specifies madvise(WILLNEED) pagefaults trigger 2mb
>>>>> reads and a binary that tends to take 110 page faults(ie program stops
>>>>> execution and waits for disk) can be reduced down to 6. This has the
>>>>> potential to double application startup of large apps without any clear
>>>>> downsides.
>>>>>
>>>>> Suse ships their glibc with a dynamic linker patch to fadvise()
>>>>> dynamic libraries(not sure why they switched from doing madvise
>>>>> before).
>>>>>
>>>>>
>>> This is interesting. I wonder how SuSE implements the policy.
>>> Do you have the patch or some strace output that demonstrates the
>>> fadvise() call?
>>>
>>>
>> glibc-2.3.90-ld.so-madvise.diff in
>>
http://www.rpmseek.com/rpm/glibc-2.4-31.12.3.src.html?hl=com&cba=0:G:0:3732595:0:15:0:
>>
> 550 Can't open
> /pub/linux/distributions/suse/pub/suse/update/10.1/rpm/src/glibc-2.4-31.12.3.src.rpm:
> No such file or directory
>
> OK I give up.
>
>
>> As I recall they just fadvise the filedescriptor before accessing it.
>>
> Obviously this is a bit risky for small memory systems..
>
>
>>>>> I filed a glibc bug about this at
>>>>>
http://sourceware.org/bugzilla/show_bug.cgi?id=11431 . Uli commented
>>>>> with his concern about wasting memory resources. What is the impact of
>>>>> madvise(WILLNEED) or the fadvise equivalent on systems under memory
>>>>> pressure? Does the kernel simply start ignoring these hints?
>>>>>
>>>>>
>>>> It will throttle based on memory pressure. In idle situations it will
>>>> eat your file cache, however, to satisfy the request.
>>>>
>>>> Now, the file cache should be much bigger than the amount of unneeded
>>>> pages you prefault with the hint over the whole library, so I guess the
>>>> benefit of prefaulting the right pages outweighs the downside of evicting
>>>> some cache for unused library pages.
>>>>
>>>> Still, it's a workaround for deficits in the demand-paging/readahead
>>>> heuristics and thus a bit ugly, I feel. Maybe Wu can help.
>>>>
>>>>
>>> Program page faults are inherently random, so the straightforward
>>> solution would be to increase the mmap read-around size (for desktops
>>> with reasonable large memory), rather than to improve program layout
>>> or readahead heuristics :)
>>>
>>>
>> Program page faults may exhibit random behavior once they've started.
>>
> Right.
>
>
>> During startup page-in pattern of over-engineered OO applications is
>> very predictable. Programs are laid out based on compilation units,
>> which have no relation to how they are executed. Another problem is that
>> any large old application will have lots of code that is either rarely
>> executed or completely dead. Random sprinkling of live code among mostly
>> unneeded code is a problem.
>>
> Agreed.
>
>
>> I'm able to reduce startup pagefaults by 2.5x and mem usage by a few MB
>> with proper binary layout. Even if one lays out a program wrongly, the
>> worst-case pagein pattern will be pretty similar to what it is by default.
>>
> That's great. When will we enjoy your research fruits? :)
>
