From: Jared Hulbert <jaredeh@...>
Subject: [PATCH 00/10] AXFS: Advanced XIP filesystem
Date: Aug 21, 1:44 am 2008

I'd like to get a first round of review on my AXFS filesystem.  This is a simple
read only compressed filesystem like Squashfs and cramfs.  AXFS is special
because it also allows for execute-in-place of your applications.  It is a major
improvement over the cramfs XIP patches that have been floating around for ages.
The biggest improvement is in the way AXFS allows for each page to be XIP or
not.  First, a user collects information about which pages are accessed on a
compressed image for each mmap()ed region from /proc/axfs/volume0.  That
'profile' is used as an input to the image builder.  The resulting image has
only the relevant pages uncompressed and XIP.  The result is smaller memory
sizes and faster launches.
                            

See http://axfs.sourceforge.net for more info.


fs/Kconfig                |   21 +
fs/Makefile               |    1 
fs/axfs/Makefile          |    7 
fs/axfs/axfs_bdev.c       |  158 ++++++++
fs/axfs/axfs_inode.c      |  490 ++++++++++++++++++++++++++
fs/axfs/axfs_mtd.c        |  233 ++++++++++++
fs/axfs/axfs_profiling.c  |  594 +++++++++++++++++++++++++++++++
fs/axfs/axfs_super.c      |  866 ++++++++++++++++++++++++++++++++++++++++++++++
fs/axfs/axfs_uml.c        |   47 ++
fs/axfs/axfs_uncompress.c |   97 +++++
include/linux/axfs.h      |  358 +++++++++++++++++++
11 files changed, 2872 insertions(+)


--

From: Jamie Lokier <jamie@...>
Subject: Re: [PATCH 00/10] AXFS: Advanced XIP filesystem
Date: Aug 21, 7:07 am 2008

Jared Hulbert wrote:
> The biggest improvement is in the way AXFS allows for each page to be XIP or
> not.  First, a user collects information about which pages are accessed on a
> compressed image for each mmap()ed region from /proc/axfs/volume0.  That
> 'profile' is used as an input to the image builder.  The resulting image has
> only the relevant pages uncompressed and XIP.  The result is smaller memory
> sizes and faster launches.

Sounds great, really nice idea.

How does it fare with no MMU?  Can the profiler and image builder lay
out the XIP pages in such a way that no-MMU mmaps can map those regions?

No complaint if not, it would be a nice bonus though.

-- Jamie
--


From: Jared Hulbert <jaredeh@...>
Subject: Re: [PATCH 00/10] AXFS: Advanced XIP filesystem
Date: Aug 21, 10:11 am 2008

> How does it fare with no MMU?  Can the profiler and image builder lay
> out the XIP pages in such a way that no-MMU mmaps can map those regions?
>
> No complaint if not, it would be a nice bonus though.

Sorry.  I don't believe it will work on no-MMU as is.  That said you
_could_ tweak the mkfs tool to lay mmap()'ed regions down contiguously
but then if you mmap() an unprofiled region, well that would be bad.
I suppose you could make axfs_mmap smart enough to handle that.  I
guess the cleanest way would be to just make files lay down
contiguously, you lose some of the space saving but it would work.

I'm not plannin to get to this anytime soon.  But I'd be willing merge
patches.  Can anybody convince me offline that working on no-MMU this
makes financial sense for my employer?  This is getting to be a common
question.  How many noMMU users are out there and why are you so
interested?
--


From: Greg Ungerer <gerg@...>
Subject: Re: [PATCH 00/10] AXFS: Advanced XIP filesystem
Date: Aug 21, 8:12 pm 2008

Hi Jared,

Jared Hulbert wrote:
>> How does it fare with no MMU?  Can the profiler and image builder lay
>> out the XIP pages in such a way that no-MMU mmaps can map those regions?
>>
>> No complaint if not, it would be a nice bonus though.
> 
> Sorry.  I don't believe it will work on no-MMU as is.  That said you
> _could_ tweak the mkfs tool to lay mmap()'ed regions down contiguously
> but then if you mmap() an unprofiled region, well that would be bad.
> I suppose you could make axfs_mmap smart enough to handle that.  I
> guess the cleanest way would be to just make files lay down
> contiguously, you lose some of the space saving but it would work.

That would be enough I think. If you could manually select
which files are contiguous-and-uncompressed that would be
useful for some too here.


> I'm not plannin to get to this anytime soon.  But I'd be willing merge
> patches.  Can anybody convince me offline that working on no-MMU this
> makes financial sense for my employer?  This is getting to be a common
> question.  How many noMMU users are out there and why are you so
> interested?

One of those unknown factors, how many are there?
Who knows, pretty much impossible to tell.

One thing for sure is that many people who do non-MMU setups
are interested in XIP to get the space savings. These are very
often small devices with very constrained RAM and flash. (For
whatever it is worth single NOR flash only boards are common in
these smaller form factors :-)

Regards
Greg


------------------------------------------------------------------------
Greg Ungerer  --  Chief Software Dude       EMAIL:     gerg@snapgear.com
Secure Computing Corporation                PHONE:       +61 7 3435 2888
825 Stanley St,                             FAX:         +61 7 3891 3630
Woolloongabba, QLD, 4102, Australia         WEB: http://www.SnapGear.com
--

From: Carsten Otte <cotte@...>
Subject: Re: [PATCH 00/10] AXFS: Advanced XIP filesystem
Date: Aug 21, 6:25 am 2008

Jared Hulbert wrote:
> I'd like to get a first round of review on my AXFS filesystem.
I like the general approach of it. It's much more flexible than the 
ext2 extension I've done, and the possibility to select XIP vs. 
compression per page is really really neat. I can imagine that people 
will prefer this over the ext2 implementation on s390. It is unclear 
to me how the "secondary block device" thing is supposed to work. 
Could you elaborate a bit on that?

--


From: Jared Hulbert <jaredeh@...>
Subject: Re: [PATCH 00/10] AXFS: Advanced XIP filesystem
Date: Aug 21, 10:30 am 2008

> I like the general approach of it. It's much more flexible than the ext2
> extension I've done, and the possibility to select XIP vs. compression per
> page is really really neat. I can imagine that people will prefer this over
> the ext2 implementation on s390. It is unclear to me how the "secondary
> block device" thing is supposed to work. Could you elaborate a bit on that?

First off we don't yet support direct_access(), but I am planning on that soon.

Sure.  For a system that has say a NOR Flash and a NAND or a embedded
MMC, one can split a filesystem image such that only the XIP parts of
the image are on the NOR while the compressed bits are on the NAND /
eMMC.  The NOR part is accessed as directly addressable memory, while
the NAND would use mtd->read() and the eMMC would use block device
access API's.  In this case I would call this NAND or eMMC the
"secondary device" because the primary device is the NOR.

Assuming my NOR was at /dev//mtd2 and my NAND at /dev/mtd5.  I would
call the following to mount such a system:

mount -t axfs -o second_dev=/dev/mtd5 /dev/mtd2 /mnt/axfs
--


From: Nick Piggin <nickpiggin@...>
Subject: Re: [PATCH 00/10] AXFS: Advanced XIP filesystem
Date: Aug 21, 6:43 am 2008

On Thursday 21 August 2008 20:25, Carsten Otte wrote:
> Jared Hulbert wrote:
> > I'd like to get a first round of review on my AXFS filesystem.
>
> I like the general approach of it. It's much more flexible than the
> ext2 extension I've done, and the possibility to select XIP vs.
> compression per page is really really neat. I can imagine that people
> will prefer this over the ext2 implementation on s390. It is unclear
> to me how the "secondary block device" thing is supposed to work.
> Could you elaborate a bit on that?

Agreed. I haven't had a good look through it yet, but at a glance it
looks pretty neat. The VM side of things looks pretty reasonable
(I fear XIP faulting might have another race or two, but that's a
core mm issue rather than filesystem specific).
--

From: Frans Meulenbroeks <fransmeulenbroeks@...>
Subject: Re: [PATCH 00/10] AXFS: Advanced XIP filesystem
Date: Aug 21, 2:32 am 2008

Jared, nice work!

I've also read your paper from the linux symposium
(http://ols.fedoraproject.org/OLS/Reprints-2008/hulbert-reprint.pdf)

A few questions:
- how does this benchmark compared to cramfs and squashfs in a NAND-only system
  (or is it just not a good plan to use this with NAND-only (of course
I won't get XIP with NAND, I understand that)
- would axfs be suitable as a filesystem on a ram disk?

Background for the last question is that if you do not have the memory
to retain all pages uncompressed (as you would with ramfs), this could
be a nice intermediate format.
Furthermore compared to ramfs, a filesystem on a ramdisk does not need
the initialisation during startup (decompressing the cpio file,
creating the files, copying the data), so when it comes to boot times
a filesystem on a ramdisk (e.g. axfs) could be a better choice.

Appreciate your feedback.

Frans.
--


From: Jared Hulbert <jaredeh@...>
Subject: Re: [PATCH 00/10] AXFS: Advanced XIP filesystem
Date: Aug 21, 3:32 pm 2008

> Jared, nice work!

Thanks.

> A few questions:

I meant to address these before.  Sorry.

> - how does this benchmark compared to cramfs and squashfs in a NAND-only system
>  (or is it just not a good plan to use this with NAND-only (of course
> I won't get XIP with NAND, I understand that)

I don't know, I'm interested to find out.  I just benchmarked that.
Actually it should work very well as a NAND-only fs.  Also you do get
something like XIP with NAND.  If you boot an XIP AXFS image on NAND
or a blkdev it will copy that XIP region into RAM and "XIP" it from
there.  I think this will make it very good for LiveCD's.  Though we
just (minutes ago) realized our testing of that feature was flawed, so
no guarantees.

> - would axfs be suitable as a filesystem on a ram disk?

It could be.  I plan on implementing support for brd.  That might work nicely.
--


From: Nick Piggin <nickpiggin@...>
Subject: Re: [PATCH 00/10] AXFS: Advanced XIP filesystem
Date: Aug 21, 10:04 pm 2008

On Friday 22 August 2008 05:32, Jared Hulbert wrote:
> > Jared, nice work!

> > - would axfs be suitable as a filesystem on a ram disk?
>
> It could be.  I plan on implementing support for brd.  That might work
> nicely.

I was going to take a look at this too. With any luck, it should be
little effort required as it looks like you have the block device
support in place?

This filesystem actually should in theory work fairly well with brd,
because then we wouldn't have to bring the data over into pagecache
for frequently used pages but we can retain the compressed storage
for the infrequently used stuff.

I say in theory because I don't know of any serious users (except
kernel testing) of brd :)
--

From: Dave Chinner <david@...>
Subject: Re: [PATCH 00/10] AXFS: Advanced XIP filesystem
Date: Aug 21, 4:39 am 2008

On Wed, Aug 20, 2008 at 10:44:36PM -0700, Jared Hulbert wrote:
> I'd like to get a first round of review on my AXFS filesystem.  This is a simple
> read only compressed filesystem like Squashfs and cramfs.  AXFS is special
> because it also allows for execute-in-place of your applications.  It is a major
> improvement over the cramfs XIP patches that have been floating around for ages.
> The biggest improvement is in the way AXFS allows for each page to be XIP or
> not.  First, a user collects information about which pages are accessed on a
> compressed image for each mmap()ed region from /proc/axfs/volume0.  That
> 'profile' is used as an input to the image builder.  The resulting image has
> only the relevant pages uncompressed and XIP.  The result is smaller memory
> sizes and faster launches.

FWIW, I'm not sure it's a good idea to name this new filesystem AXFS.
People are almost certainly going to confuse it with XFS despite
the filesystems being aimed at diammetrically opposed ends of the
storage spectrum. ;)

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com
--


From: Jared Hulbert <jaredeh@...>
Subject: Re: [PATCH 00/10] AXFS: Advanced XIP filesystem
Date: Aug 21, 10:19 am 2008

> FWIW, I'm not sure it's a good idea to name this new filesystem AXFS.
> People are almost certainly going to confuse it with XFS despite
> the filesystems being aimed at diammetrically opposed ends of the
> storage spectrum. ;)

In principle I think you are right.  AXFS and XFS are similar names
and it could lead to confusion.  I think XFS should change its name to
prevent confusion.  I think by 2 years AXFS will be used in orders of
magnitude more machines anyway.  ;)

About opposite end of the spectrum...  Carsten just said AXFS might be
nice for s390, so I'm not sure how true that is.

I'm kind of attached to the name now.
--