mirror of
https://github.com/KingDuckZ/kamokan.git
synced 2024-12-05 00:55:40 +00:00
300 lines
22 KiB
Text
300 lines
22 KiB
Text
UTF-8 decoder capability and stress test
|
||
----------------------------------------
|
||
|
||
Markus Kuhn <http://www.cl.cam.ac.uk/~mgk25/> - 2015-08-28 - CC BY 4.0
|
||
|
||
This test file can help you examine, how your UTF-8 decoder handles
|
||
various types of correct, malformed, or otherwise interesting UTF-8
|
||
sequences. This file is not meant to be a conformance test. It does
|
||
not prescribe any particular outcome. Therefore, there is no way to
|
||
"pass" or "fail" this test file, even though the text does suggest a
|
||
preferable decoder behaviour at some places. Its aim is, instead, to
|
||
help you think about, and test, the behaviour of your UTF-8 decoder on a
|
||
systematic collection of unusual inputs. Experience so far suggests
|
||
that most first-time authors of UTF-8 decoders find at least one
|
||
serious problem in their decoder using this file.
|
||
|
||
The test lines below cover boundary conditions, malformed UTF-8
|
||
sequences, as well as correctly encoded UTF-8 sequences of Unicode code
|
||
points that should never occur in a correct UTF-8 file.
|
||
|
||
According to ISO 10646-1:2000, sections D.7 and 2.3c, a device
|
||
receiving UTF-8 shall interpret a "malformed sequence in the same way
|
||
that it interprets a character that is outside the adopted subset" and
|
||
"characters that are not within the adopted subset shall be indicated
|
||
to the user" by a receiving device. One commonly used approach in
|
||
UTF-8 decoders is to replace any malformed UTF-8 sequence by a
|
||
replacement character (U+FFFD), which looks a bit like an inverted
|
||
question mark, or a similar symbol. It might be a good idea to
|
||
visually distinguish a malformed UTF-8 sequence from a correctly
|
||
encoded Unicode character that is just not available in the current
|
||
font but otherwise fully legal, even though ISO 10646-1 doesn't
|
||
mandate this. In any case, just ignoring malformed sequences or
|
||
unavailable characters does not conform to ISO 10646, will make
|
||
debugging more difficult, and can lead to user confusion.
|
||
|
||
Please check, whether a malformed UTF-8 sequence is (1) represented at
|
||
all, (2) represented by exactly one single replacement character (or
|
||
equivalent signal), and (3) the following quotation mark after an
|
||
illegal UTF-8 sequence is correctly displayed, i.e. proper
|
||
resynchronization takes place immediately after any malformed
|
||
sequence. This file says "THE END" in the last line, so if you don't
|
||
see that, your decoder crashed somehow before, which should always be
|
||
cause for concern.
|
||
|
||
All lines in this file are exactly 79 characters long (plus the line
|
||
feed). In addition, all lines end with "|", except for the two test
|
||
lines 2.1.1 and 2.2.1, which contain non-printable ASCII controls
|
||
U+0000 and U+007F. If you display this file with a fixed-width font,
|
||
these "|" characters should all line up in column 79 (right margin).
|
||
This allows you to test quickly, whether your UTF-8 decoder finds the
|
||
correct number of characters in every line, that is whether each
|
||
malformed sequences is replaced by a single replacement character.
|
||
|
||
Note that, as an alternative to the notion of malformed sequence used
|
||
here, it is also a perfectly acceptable (and in some situations even
|
||
preferable) solution to represent each individual byte of a malformed
|
||
sequence with a replacement character. If you follow this strategy in
|
||
your decoder, then please ignore the "|" column.
|
||
|
||
|
||
Here come the tests: |
|
||
|
|
||
1 Some correct UTF-8 text |
|
||
|
|
||
You should see the Greek word 'kosme': "κόσμε" |
|
||
|
|
||
2 Boundary condition test cases |
|
||
|
|
||
2.1 First possible sequence of a certain length |
|
||
|
|
||
2.1.1 1 byte (U-00000000): " |