On Mon, 21 Jan 2008, Kevin Ballard wrote:
quoted text
> 
> > It's what shows up when you sha1sum, but it's also as simple as what shows
> > up when you do an "ls -l" and look at a file size.
> 
> It does? Why on earth should it do that? Filename doesn't contribute to the
> listed filesize on OS X.

Umm. What's this inability to see that data is data is data?

Why do you think Unicode has anything in particular to do with filenames?

Those same unicode strings are often part of the file data itself, and 
then that encoding damn well is visible in "ls -l".

Doing

	echo ä > file
	ls -l file

sure shows that "underlying octet" thing that you wanted to avoid so much. 
My point was that those underlying octets are always there, and they do 
matter. The fact that the differences may not be visible when you compare 
the normalized forms doesn't make it any less true.

You can choose to put blinders on and try to claim that normalization is 
invisible, but it's only invisible TO THOSE THINGS THAT DON'T WANT TO SEE 
IT.

But that doesn't change the fact that a lot of things *do* see it. There 
are very few things that are "Unicode specific", and a *lot* of tools that 
are just "general data tools".

And git tries to be a general data tool, not a Unicode-specific one.

quoted text
> I'm not sure what you mean. The byte sequence is different from Latin1 to
> UTF-8 even if you use NFC, so I don't think, in this case, it makes any
> difference whether you use NFC or NFD.
>
> Yes, the codepoints are the same in Latin1 and UTF-8 if you use NFC, but 
> that's hardly relevant. Please correct me if I'm wrong, but I believe 
> Latin1->UTF-8->Latin1 conversion will always produce the same Latin1 
> text whether you use NFC or NFD.

The problem is that the UTF-8 form is different, so if you save things in 
UTF-8 (which we hopefully agree is a sane thing to do), then you should 
try to use a representation that people agree on.

And NFC is the more common normalization form by far, so by normalizing to 
something else, you actually de-normalize as far as those other people are 
concerned.

So if you have to normalize, at least use the normal form!

quoted text
> The only reason it's particularly inconvenient is because it's different from
> what most other systems picked. And if you want to blame someone for that,
> blame Unicode for having so many different normalization forms.

I blame them for encouraging normalization at all.

It's stupid.

You don't need it.

The people who care about "are these strings equivalent" shouldn't do a 
"memcmp()" on them in the first place. And if you don't do a memcmp() on 
things, then you don't need to normalize. 

So you have two cases:
 (a) the cases that care about *identity*. They don't want normalization
 (b) the cases that care about *equivalence*. And they shouldn't do 
      octet-by-octet comparison.

See? Either you want to see equivalence, or you don't. And in neither case 
is normalization the right thing to do (except as *possibly* an internal 
part of the comparison, but there are actually better ways to check for 
equivalence than the brute-force "normalize both and compare results 
bitwise").

			Linus
-
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html