Re: Handling large files with GIT

Previous message: [thread] [date] [author]
Next message: [thread] [date] [author]
From: Linus Torvalds
Date: Sunday, February 12, 2006 - 10:05 pm

On Sun, 12 Feb 2006, Linus Torvalds wrote:

Final note: this means, for example, that git is relatively bad at 
tracking a "hashed" nested file directory (like the one git itself uses), 
because new files will end up randomly appearing in every directory, and 
no directory is ever "stable".

In contrast, if the directory structure is - for example - something where 
you index files by date, and subdirectories with older dates are thus much 
more naturally likely to be quiescent, the "this tree is the same" 
optimizations work very well.

Basically, a lot of the git speed optimizations depend on "on average, 
things stay the same". We may have 18,000+ files in the kernel, but most 
patches will change maybe five of them. There's a lot of fairly static 
content and the changes have a certain level of "locality". It's normally 
a hundred-line patch to one file, not a hundred files that had one-liners. 
And when 20 files are changed, most of them tend to be in the same 
subdirectory, etc etc.

Taking advantage of those kinds of things is what makes git good at 
handling software projects. But it wouldn't necessarily be how you lay out 
a mail directory, for example. An automated file store might want to 
spread out the changes on purpose.

		Linus
-
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Previous message: [thread] [date] [author]
Next message: [thread] [date] [author]

Messages in current thread:
Handling large files with GIT, Martin Langhoff, (Wed Feb 8, 2:14 am)
Re: Handling large files with GIT, Johannes Schindelin, (Wed Feb 8, 4:54 am)
Re: Handling large files with GIT, Linus Torvalds, (Wed Feb 8, 9:34 am)
Re: Handling large files with GIT, Linus Torvalds, (Wed Feb 8, 10:01 am)
Re: Handling large files with GIT, Junio C Hamano, (Wed Feb 8, 1:11 pm)
Re: Handling large files with GIT, Florian Weimer, (Wed Feb 8, 2:20 pm)
Re: Handling large files with GIT, Martin Langhoff, (Wed Feb 8, 3:35 pm)
Re: Handling large files with GIT, Greg KH, (Wed Feb 8, 9:54 pm)
Re: Handling large files with GIT, Martin Langhoff, (Wed Feb 8, 10:38 pm)
Re: Handling large files with GIT, Ben Clifford, (Sun Feb 12, 6:26 pm)
Re: Handling large files with GIT, Linus Torvalds, (Sun Feb 12, 8:42 pm)
Re: Handling large files with GIT, Martin Langhoff, (Sun Feb 12, 9:40 pm)
Re: Handling large files with GIT, Linus Torvalds, (Sun Feb 12, 9:57 pm)
Re: Handling large files with GIT, Linus Torvalds, (Sun Feb 12, 10:05 pm)
Re: Handling large files with GIT, Jeff Garzik, (Sun Feb 12, 10:55 pm)
Re: Handling large files with GIT, Keith Packard, (Sun Feb 12, 11:07 pm)
Re: Handling large files with GIT, Linus Torvalds, (Mon Feb 13, 9:19 am)
Re: Handling large files with GIT, Ian Molton, (Mon Feb 13, 4:17 pm)
Re: Handling large files with GIT, Martin Langhoff, (Mon Feb 13, 4:19 pm)
Re: Handling large files with GIT, Martin Langhoff, (Mon Feb 13, 5:07 pm)
Re: Handling large files with GIT, Johannes Schindelin, (Tue Feb 14, 11:56 am)
Re: Handling large files with GIT, Linus Torvalds, (Tue Feb 14, 12:52 pm)
Re: Handling large files with GIT, Sam Vilain, (Tue Feb 14, 2:21 pm)
Re: Handling large files with GIT, Linus Torvalds, (Tue Feb 14, 3:01 pm)
Re: Handling large files with GIT, Junio C Hamano, (Tue Feb 14, 3:30 pm)
Re: Handling large files with GIT, Sam Vilain, (Tue Feb 14, 5:40 pm)
Re: Handling large files with GIT, Junio C Hamano, (Tue Feb 14, 6:39 pm)
Re: Handling large files with GIT, Linus Torvalds, (Tue Feb 14, 7:05 pm)
Re: Handling large files with GIT, Martin Langhoff, (Tue Feb 14, 7:07 pm)
Re: Handling large files with GIT, Linus Torvalds, (Tue Feb 14, 7:18 pm)
Re: Handling large files with GIT, Linus Torvalds, (Tue Feb 14, 7:33 pm)
Re: Handling large files with GIT, Linus Torvalds, (Tue Feb 14, 8:58 pm)
Re: Handling large files with GIT, Sam Vilain, (Tue Feb 14, 9:03 pm)
Re: Handling large files with GIT, Junio C Hamano, (Wed Feb 15, 2:54 am)
Re: Handling large files with GIT, Linus Torvalds, (Wed Feb 15, 8:44 am)
Re: Handling large files with GIT, Linus Torvalds, (Wed Feb 15, 10:16 am)
Re: Handling large files with GIT, Linus Torvalds, (Wed Feb 15, 8:25 pm)
Re: Handling large files with GIT, Junio C Hamano, (Wed Feb 15, 8:29 pm)
Re: Handling large files with GIT, Fredrik Kuivinen, (Thu Feb 16, 1:32 pm)