On Jan 21, 2008, at 1:12 PM, Linus Torvalds wrote:

quoted text
> On Mon, 21 Jan 2008, Kevin Ballard wrote:
>> On Jan 21, 2008, at 9:14 AM, Peter Karlsson wrote:
>>>
>>> I happen to prefer the text-as-string-of-characters (or code points,
>>> since you use the other meaning of characters in your posts), =20
>>> since I
>>> come from the text world, having worked a lot on Unicode text
>>> processing.
>>>
>>> You apparently prefer the text-as-sequence-of-octets, which I tend =20=

quoted text
>>> to
>>> dislike because I would have thought computer engineers would have
>>> evolved beyond this when we left the 1900s.
>>
>> I agree. Every single problem that I can recall Linus bringing up =20
>> as a
>> consequence of HFS+ treating filenames as strings [..]
>
> You say "I agree", BUT YOU DON'T EVEN SEEM TO UNDERSTAND WHAT IS =20
> GOING ON.

I could say the same thing about you.

quoted text
> The fact is, text-as-string-of-codepoints (let's make the "codepoints"
> obvious, so that there is no ambiguity, but I'd also like to make it =20=

quoted text
> clear
> that a codepoint *is* how a Unicode character is defined, and a =20
> Unicode
> "string" is actually *defined* to be a sequence of codepoints, and =20
> totally
> independent of normalization!) is fine.
>
> That was never the issue at all. Unicode codepoints are wonderful.
>
> Now, git _also_ heavily depends on the actual encoding of those
> codepoints, since we create hashes etc, so in fact, as far ass git is
> concerned, names have to be in some particular encoding to be =20
> hashed, and
> UTF-8 is the only sane encoding for Unicode. People can blather about
> UCS-2 and UTF-16 and UTF-32 all they want, but the fact is, UTF-8 is
> simply technically superior in so many ways that I don't even =20
> understand
> why anybody ever uses anything else.
>
> So I would not disagree with using UTF-8 at all.
>
> But that is *entirely* a separate issue from "normalization".
>
> Kevin, you seem to think that normalization is somehow forced on you =20=

quoted text
> by
> the "text-as-codepoints" decision, and that is SIMPLY NOT TRUE.
> Normalization is a totally separate decision, and it's a STUPID one,
> because it breaks so many of the _nice_ properties of using UTF-8.

I'm not saying it's forced on you, I'm saying when you treat filenames =20=

as text, it DOESN'T MATTER if the string gets normalized. As long as =20
the string remains equivalent, YOU DON'T CARE about the underlying =20
byte stream.

quoted text
> And THAT is where we differ. It has nothing to do with "octets". It =20=

quoted text
> has
> nothing to do with not liking Unicode. It has nothing to do with
> "strings".
>
> In short:
>
> - normalization is by no means required or even a good feature. It's
>   something you do when you want to know if two strings are =20
> equivalent,
>   but that doesn't actually mean that you should keep the strings
>   normalized all the time!

Alright, fine. I'm not saying HFS+ is right in storing the normalized =20=

version, but I do believe the authors of HFS+ must have had a reason =20
to do that, and I also believe that it shouldn't make any difference =20
to me since it remains equivalent.

quoted text
> - normalization has *nothing* to do with "treating text as octets".
>   That's entirely an encoding issue.

Sure it does. Normalizing a string produces an equivalent string, and =20=

so unless I look at the octets the two strings are, for all intents =20
and purposes, the same.

quoted text
> - of *course* git has to treat things as a binary stream at some =20
> point,
>   since you need that to even compute a SHA1 in the first place, but =20=

quoted text
> that
>   has *nothing* to do with normalization or the lack of it.

You're right, but it doesn't have to treat it as a binary stream at =20
the level I care about. I mean, no matter what you do at some level =20
the string is evaluated as a binary stream. For our purposes, just =20
redefine the hashing algorithm to hash all equivalent strings the =20
same, and you can implement that by using SHA1 on a particular =20
encoding of the string.

quoted text
> Got it? Forced normalization is stupid, because it changes the data =20=

quoted text
> and
> removes information, and unless you know that change is safe, it's the
> wrong thing to do.

Decomposing and recomposing shouldn't lose any information we care =20
about - when treating filenames as text, a<COMBINING DIARESIS> and <A =20=

WITH DIARESIS> are equivalent, and thus no distinction is made between =20=

them. I'm not sure what other information you might be considering =20
lost in this case.

quoted text
> One reason _not_ to do normalization is that if you don't, you can =20
> still
> interact with no ambiguity with other non-Unicode locales. You can =20
> do the
> 1:1 Latin1<->Unicode translation, and you *never* get into trouble. In
> cotnrast, if you normalize, it's no longer a 1:1 translation any =20
> more, and
> you can get into a situation where the translation from Latin1 to =20
> Unicode
> and back results in a *different* filename than the one you started =20=

quoted text
> with!

I don't believe you. See below.

quoted text
> See? That's a *serious*problem*. A system that forces normalization BY
> DEFINITION cannot work with people who use a Latin1 filesystem, =20
> because it
> will corrupt the filenames!
>
> But you are apparently too damn stupid to understand that "data
> corruption" =3D=3D "bad", and too damn stupid to see that "Unicode" =
does =20
quoted text
> not
> mean "Forced normalization".

When have I ever said that Unicode meant Forced normalization?

quoted text
> But I'll try one more time. Let's say that I work on a project where =20=

quoted text
> there
> are some people who use Latin1, and some people who use UTF-8, and =20
> we use
> special characters. It should all work, as long as we use only the =20
> common
> subset, and we teach git to convert to UTF-8 as a common base. Right?
>
> In your *idiotic* world, where you have to normalize and corrupting
> filenames is ok, that doesn't work! It works wonderfully well if you =20=

quoted text
> do
> the obvious 1:1 translation and you do *not* normalize, but the =20
> moment you
> start normalizing, you actually corrupt the filenames!

Wrong.

quoted text
> And yes, the character sequence 'a=A8' is exactly one such sequence. =20=

quoted text
> It's
> perfectly representable in both Latin1 and in UTF-8: in latin1 it is a
> two-character '\x61\xa8', and when doing a Latin1->UTF-8 conversion, =20=

quoted text
> it
> becomes '\x61\xc2\xa8', and you can convert back and forth between =20
> those
> two forms an infinite amount of times, and you never corrupt it.
>
> But the moment you add normalization to the mix, you start screwing =20=

quoted text
> up.
> Suddenly, the sequence '\x61\xa8' in Latin1 becomes (assuming NFD)
> '\xc3\xa4' in UTF-8, and when converted back to Latin1, it is now =20
> '\xe4',
> ie that filename hass been corrupted!

Wrong. '\x61\x18' in Latin1, when converted to UTF-8 (NFD) is still =20
'\x61\xc2\xa8'. You're mixing up DIARESIS (U+00A8) and COMBINING =20
DIARESIS (U+0308).

I suspect this is why you've been yelling so much - you have a =20
fundamental misunderstanding about what normalization is actually doing.

quoted text
> See? Normalization in the face of working together with others is a =20=

quoted text
> total
> and utter mistake, and yes, it really *does* corrupt data. It makes it
> fundamentally impossible to reliably work together with other =20
> encodings -
> even when you do converstion between the two!
>
> [ And that's the really sad part. Non-normalized Unicode can pretty =20=

quoted text
> much
>  be used as a "generic encoding" for just about all locales - if you =20=

quoted text
> know
>  the locale you convert from and to, you can generally use UTF-8 as an
>  internal format, knowing that you can always get the same result =20
> back in
>  the original encoding. Normalization literally breaks that wonderful
>  generic capability of Unicode.
>
>  And the fact that Unicode is such a "generic replacement" for any =20
> locale
>  is exactly what makes it so wonderful, and allows you to fairly
>  seamlessly convert piece-meal from some particular locale to Unicode:
>  even if you have some programs that still work in the original =20
> locale,
>  you know that you can convert back to it without loss of information.
>
>  Except if you normalize. In that case, you *do* lose information, and
>  suddenly one of the best things about Unicode simply disappears.

See above as to why you're not losing the information you so fervently =20=

believe you are.

quoted text
>  As a result, people who force-normalize are idiots. But they seem to
>  also be stupid enough that they don't understand that they are =20
> idiots.
>  Sad.

People who insult others run the risk of looking like a fool when =20
shown to be wrong.

quoted text
>  It's a bit like whitespace. Whitespace "doesn't matter" in text (=3D=3D=
 =20
quoted text
> is
>  equivalent), but an email client that force-normalizes whitespace in
>  text is a really *broken* email client, because it turns out that
>  sometimes even the "equivalent" forms simply do matter. Patches are
>  text, but whitespace is meaningful there.
>
>  Same exact deal: it's good to have the *ability* to normalize
>  whitespace (in email, we call this "text=3Dflowed" or similar), and =
in
quoted text
>  some ceses you might even want to make it the default action, but
>  *forcing* normalization is total idiocy and actually makes the system
>  less useful! ]

Sure, it all depends on what level you need to evaluate text. If we're =20=

talking about english paragraphs, then whitespace can be messed with. =20=

When we're talking about unicode strings, then specific encoding can =20
be messed with. When talking about byte sequence, nothing can be =20
messed with.

In our case, when working on an HFS+ filesystem all you have to care =20
about is the unicode string level. The specific encoding can be messed =20=

with, and the client shouldn't care. Problems only arise when =20
attempting to interoperate with filesystems that work at the byte =20
sequence level.

The only information you lose when doing canonical normalization is =20
what the original byte sequence was. Sure, this is a problem when =20
working on a filesystem that cares about byte sequence, but it's not a =20=

problem when working on a filesystem that cares about the unicode =20
string.

-Kevin Ballard

--=20
Kevin Ballard
http://kevin.sb.org
kevin@sb.org
http://www.tildesoft.com