Fix "leading surrogate" in UTF-8 (actually CESU-8) #607
Conversation
Amusingly, the code which is described as:
This is a leading surrogate; some idiot has...
has a typo: 0xDC00 should be 0xD800.
The comment mentions a "leading surrogate", which is a synonym for a
high-surrogate code unit:
A 16-bit code unit in the range D800_16 to DBFF_16, used in UTF-16
as the leading code unit of a surrogate pair. Also known as a
leading surrogate.
https://unicode.org/glossary/#high_surrogate_code_unit
What libarchive is doing here is adjusting for an invalid conversion of
UTF16 to UTF8; this adjustment is now known as the Compatibility
Encoding Scheme for UTF-16: 8-Bit (CESU-8), published as the Unicode
Technical Report #26 [1].
[1] https://www.unicode.org/reports/tr26/
Essentially, if libarchive detects a surrogate pair (not allowed in
UTF-8 [2]), it tries to construct the desired unicode value (as per
CESU-8).
[2] That unicode value should be encoded with 4 octets, whereas the
surrogate pair requires 6 octets.
Caveat:
- I used the term CESU-8 because later libarchive calls it that -- in
fact, they moved this functionality into cesu8_to_unicode() in:
2011-04-20 Add a check for surrogate pairs in UTF-8...
libarchive c319a15a798653faae34a1f861ab74fbcf053e11
However, "cesu" does not appear in the tarsnap git repo.
|
Amusingly, the code which is described as: has a typo: 0xDC00 should be 0xD800. The comment mentions a "leading surrogate", which is a synonym for a high-surrogate code unit:
What libarchive is doing here is adjusting for an invalid conversion of UTF16 to UTF8; this adjustment is now known as the Compatibility Encoding Scheme for UTF-16: 8-Bit (CESU-8), published as the Unicode Technical Report #26 [1]. [1] https://www.unicode.org/reports/tr26/ Essentially, if libarchive detects a surrogate pair (not allowed in UTF-8 [2]), it tries to construct the desired unicode value (as per CESU-8). [2] That unicode value should be encoded with 4 octets, whereas the surrogate pair requires 6 octets. Caveat:
|
tl;drThere's a typo in libarchive-2.7 which stopped the code from doing what it was supposed to. Modern libarchive uses a different method of doing the same thing (so the typo wasn't explicitly "fixed", but it's not broken any more). ExperimentsTo investigate this, I created Both tar files contain a single 0-byte file called In the "good" version, the tar file begins: whereas the bad one encodes the smile emoji as a surrogate pair, and begins: The diff on those two lines is: (The full diff has more: the filename is repeated 3 times, and the checksum and path length changes. But those aren't important.) tar programsI tested libarchive-2.7, libarchvie 3.6.0 (the default in freebsd 12.4), and gnu tar 1.35, with
(that's the bytes in octal) When I tried applying this fix to libarchive-2.7, it worked fine: (that's the modified libarchive-2.7) |
No description provided.