761 lines
32 KiB
HTML
761 lines
32 KiB
HTML
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 3.2//EN">
|
|
<html>
|
|
<head>
|
|
<meta name="generator" content=
|
|
"HTML Tidy for Linux/x86 (vers 12 April 2005), see www.w3.org">
|
|
<meta name="description" content="A simple, portable and lightweigt C++ library for easy handling of UTF-8 encoded strings">
|
|
<meta name="keywords" content="UTF-8 C++ portable utf8 unicode generic templates">
|
|
<title>UTF8-CPP: UTF-8 with C++ in a Portable Way</title>
|
|
</head>
|
|
<body>
|
|
<p><a href="https://sourceforge.net/projects/utfcpp">The Sourceforge project page</a></p>
|
|
<h2>Table of Contents</h2>
|
|
<ul>
|
|
<li><a href="#introduction">Introduction</a></li>
|
|
<li><a href="#examples">Examples of Use</a></li>
|
|
<li><a href="#reference">Reference</a></li>
|
|
<li><a href="#points">Points of Interest</a></li>
|
|
<li><a href="#conclusion">Conclusion</a></li>
|
|
<li><a href="#references">References</a></li>
|
|
</ul>
|
|
<h2 id="introduction">Introduction</h2>
|
|
<p>Many C++ developers miss an easy and portable way of handling
|
|
Unicode encoded strings. C++ Standard is currently Unicode
|
|
agnostic, and while some work is being done to introduce Unicode to
|
|
the next incarnation called C++0x, for the moment nothing of the
|
|
sort is available. In the meantime, developers use 3rd party
|
|
libraries like ICU, OS specific capabilities, or simply roll out
|
|
their own solutions.</p>
|
|
<p>In order to easily handle UTF-8 encoded Unicode strings, I have
|
|
come up with a set of template functions. For anybody used to work
|
|
with STL algorithms, they should be easy and natural to use. The
|
|
code is freely available for any purpose - check out the license at
|
|
the beginning of the utf8.h file. Be aware, though, that while I
|
|
did some testing, this library has not been used in production yet.
|
|
If you run into bugs or performance issues, please let me know and
|
|
I'll do my best to address them.</p>
|
|
<p>The purpose of this article is not to offer an introduction to
|
|
Unicode in general, and UTF-8 in particular. If you are not
|
|
familiar with Unicode, be sure to check out <a href=
|
|
"http://www.unicode.org/">Unicode Home Page</a> or some other
|
|
source of information for Unicode. Also, it is not my aim to
|
|
advocate the use of UTF-8 encoded strings in C++ programs; if you
|
|
want to handle UTF-8 encoded strings from C++, I am sure you have
|
|
good reasons for it.</p>
|
|
<h2 id="examples">Examples of use</h2>
|
|
<p>To illustrate the use of this utf8 library, we shall open a file
|
|
containing UTF-8 encoded text, check whether it starts with a byte order mark,
|
|
read each line into a <code>std::string</code>, check it for validity, convert the text to UTF-16,
|
|
and back to UTF-8:</p>
|
|
<pre>
|
|
#include <fstream>
|
|
#include <iostream>
|
|
#include <string>
|
|
#include <vector>
|
|
using namespace std;
|
|
|
|
int main()
|
|
{
|
|
if (argc != 2) {
|
|
cout << "\nUsage: docsample filename\n";
|
|
return 0;
|
|
}
|
|
const char* test_file_path = argv[1];
|
|
// Open the test file (must be UTF-8 encoded)
|
|
ifstream fs8(test_file_path);
|
|
if (!fs8.is_open()) {
|
|
cout << "Could not open " << test_file_path << endl;
|
|
return 0;
|
|
}
|
|
|
|
// Read the first line of the file
|
|
unsigned line_count = 1;
|
|
string line;
|
|
if (!getline(fs8, line))
|
|
return 0;
|
|
|
|
// Look for utf-8 byte-order mark at the beginning
|
|
if (line.size() > 2) {
|
|
if (utf8::is_bom(line.c_str()))
|
|
cout << "There is a byte order mark at the beginning of the file\n";
|
|
}
|
|
|
|
// Play with all the lines in the file
|
|
do {
|
|
// check for invalid utf-8 (for a simple yes/no check, there is also utf8::is_valid function)
|
|
string::iterator end_it = utf8::find_invalid(line.begin(), line.end());
|
|
if (end_it != line.end()) {
|
|
cout << "Invalid UTF-8 encoding detected at line " << line_count << "\n";
|
|
cout << "This part is fine: " << string(line.begin(), end_it) << "\n";
|
|
}
|
|
// Get the line length (at least for the valid part)
|
|
int length = utf8::distance(line.begin(), end_it);
|
|
cout << "Length of line " << line_count << " is " << length << "\n";
|
|
|
|
// Convert it to utf-16
|
|
vector<unsigned short> utf16line;
|
|
utf8::utf8to16(line.begin(), end_it, back_inserter(utf16line));
|
|
// And back to utf-8;
|
|
string utf8line;
|
|
utf8::utf16to8(utf16line.begin(), utf16line.end(), back_inserter(utf8line));
|
|
// Confirm that the conversion went OK:
|
|
if (utf8line != string(line.begin(), end_it))
|
|
cout << "Error in UTF-16 conversion at line: " << line_count << "\n";
|
|
|
|
getline(fs8, line);
|
|
line_count++;
|
|
} while (!fs8.eof());
|
|
|
|
return 0;
|
|
}
|
|
</pre>
|
|
<p>In the previous code sample, we have seen the use of the following functions
|
|
from <code>utf8</code> namespace: first we used <code>is_bom</code>
|
|
function to detect UTF-8 byte order mark at the beginning of the
|
|
file; then for each line we performed a detection of invalid UTF-8 sequences with <code>find_invalid</code>;
|
|
the number of characters (more precisely - the number of Unicode code points) in each line was determined
|
|
with a use of <code>utf8::distance</code>; finally, we have converted each line to UTF-16 encoding with
|
|
<code>utf8to16</code> and back to UTF-8 with <code>utf16to8</code>.
|
|
</p>
|
|
<h2 id ="reference">Reference</h2>
|
|
|
|
<h3>Functions From utf8 Namespace</h3>
|
|
<h4>utf8::append</h4>
|
|
<p>Encodes a 32 bit code point as a UTF-8 sequence of octets and
|
|
appends the sequence to a UTF-8 string.</p>
|
|
<code>template <typename octet_iterator> octet_iterator
|
|
append(uint32_t cp, octet_iterator result);</code>
|
|
<p><code>cp</code>: A 32 bit integer representing a code point to
|
|
append to the sequence.<br>
|
|
<code>result</code>: An output iterator to the place in the
|
|
sequence where to append the code point.<br>
|
|
<u>Return value</u>: An iterator pointing to the place after the
|
|
newly appended sequence.</p>
|
|
<p>Example of use:</p>
|
|
<pre>
|
|
unsigned char u[5] = {0,0,0,0,0};
|
|
|
|
unsigned char* end = append(0x0448, u);
|
|
|
|
assert (u[0] == 0xd1 && u[1] == 0x88 && u[2] == 0 && u[3] == 0 && u[4] == 0);
|
|
</pre>
|
|
<p>Note that <code>append</code> does not allocate any memory - it
|
|
is the burden of the caller to make sure there is enough memory
|
|
allocated for the operation. To make things more interesting,
|
|
<code>append</code> can add anywhere between 1 and 4 octets to the
|
|
sequence. In practice, you would most often want to use
|
|
<code>std::back_inserter</code> to ensure that the necessary memory
|
|
is allocated.</p>
|
|
<p>In case of an invalid code point, a
|
|
<code>utf8::invalid_code_point</code> exception is thrown.</p>
|
|
<h4>utf8::next</h4>
|
|
<p>Given the iterator to the beginning of the UTF-8 sequence, it
|
|
returns the code point and moves the iterator to the next
|
|
position.</p>
|
|
<code>template <typename octet_iterator> uint32_t
|
|
next(octet_iterator& it, octet_iterator end);</code>
|
|
<p><code>it</code>: a reference to an iterator pointing to the
|
|
beginning of an UTF-8 encoded code point. After the function
|
|
returns, it is incremented to point to the beginning of the next
|
|
code point.<br>
|
|
<code>end</code>: end of the UTF-8 sequence to be processed. If
|
|
<code>it</code> gets equal to <code>end</code> during the
|
|
extraction of a code point, an <code>utf8::not_enough_room</code>
|
|
exception is thrown.<br>
|
|
<u>Return value</u>: the 32 bit representation of the processed
|
|
UTF-8 code point.</p>
|
|
<p>Example of use:</p>
|
|
<pre>
|
|
char* twochars = "\xe6\x97\xa5\xd1\x88";
|
|
char* w = twochars;
|
|
|
|
int cp = next(w, twochars + 6);
|
|
|
|
assert (cp == 0x65e5);
|
|
assert (w == twochars + 3);
|
|
</pre>
|
|
<p>This function is typically used to iterate through a UTF-8
|
|
encoded string.</p>
|
|
<p>In case of an invalid UTF-8 seqence, a
|
|
<code>utf8::invalid_utf8</code> exception is thrown.</p>
|
|
<h4>utf8::previous</h4>
|
|
<p>Given a reference to an iterator pointing to an octet in a UTF-8
|
|
seqence, it decreases the iterator until it hits the beginning of
|
|
the previous UTF-8 encoded code point and returns the 32 bits
|
|
representation of the code point.</p>
|
|
<code>template <typename octet_iterator> uint32_t
|
|
previous(octet_iterator& it, octet_iterator pass_start);</code>
|
|
<p><code>it</code>: a reference pointing to an octet within a UTF-8
|
|
encoded string. After the function returns, it is decremented to
|
|
point to the beginning of the previous code point.<br>
|
|
<code>pass_start</code>: an iterator to the point in the sequence
|
|
where the search for the beginning of a code point is aborted if no
|
|
result was reached. It is a safety measure to prevent passing the
|
|
beginning of the string in the search for a UTF-8 lead octet.<br>
|
|
<u>Return value</u>: the 32 bit representation of the previous code
|
|
point.</p>
|
|
<p>Example of use:</p>
|
|
<pre>
|
|
char* twochars = "\xe6\x97\xa5\xd1\x88";
|
|
unsigned char* w = twochars + 3;
|
|
|
|
int cp = previous (w, twochars - 1);
|
|
|
|
assert (cp == 0x65e5);
|
|
assert (w == twochars);
|
|
</pre>
|
|
<p>The primary purpose of this function is to iterate backwards
|
|
through a UTF-8 encoded string. Therefore, <code>it</code> will
|
|
typically point to the beginning of a code point, and
|
|
<code>pass_start</code> will point to the octet just before the
|
|
beginning of the string to ensure we don't go backwards too far.
|
|
<code>it</code> is decreased until it points to a lead UTF-8 octet,
|
|
and then the UTF-8 sequence beginning with that octet is decoded to
|
|
a 32 bit representation and returned.</p>
|
|
<p>In case <code>pass_end</code> is reached before a UTF-8 lead
|
|
octet is hit, or if an invalid UTF-8 sequence is started by the
|
|
lead octet, an <code>invalid_utf8</code> exception is thrown</p>
|
|
<h4>utf8::advance</h4>
|
|
<p>Advances an iterator by the specified number of code points
|
|
within an UTF-8 sequence.</p>
|
|
<code>template <typename octet_iterator, typename
|
|
distance_type> void advance (octet_iterator& it,
|
|
distance_type n, octet_iterator end);</code>
|
|
<p><code>it</code>: a reference to an iterator pointing to the
|
|
beginning of an UTF-8 encoded code point. After the function
|
|
returns, it is incremented to point to the nth following code
|
|
point.<br>
|
|
<code>n</code>: a positive integer that shows how many code points
|
|
we want to advance.<br>
|
|
<code>end</code>: end of the UTF-8 sequence to be processed. If
|
|
<code>it</code> gets equal to <code>end</code> during the
|
|
extraction of a code point, an <code>utf8::not_enough_room</code>
|
|
exception is thrown.<br></p>
|
|
<p>Example of use:</p>
|
|
<pre>
|
|
char* twochars = "\xe6\x97\xa5\xd1\x88";
|
|
unsigned char* w = twochars;
|
|
|
|
advance (w, 2, twochars + 6);
|
|
|
|
assert (w == twochars + 5);
|
|
</pre>
|
|
<p>This function works only "forward". In case of a negative
|
|
<code>n</code>, there is no effect.</p>
|
|
<p>In case of an invalid code point, a
|
|
<code>utf8::invalid_code_point</code> exception is thrown.</p>
|
|
<h4>utf8::distance</h4>
|
|
<p>Given the iterators to two UTF-8 encoded code points in a
|
|
seqence, returns the number of code points between them.</p>
|
|
<code>template <typename octet_iterator> typename
|
|
std::iterator_traits<octet_iterator>::difference_type
|
|
distance (octet_iterator first, octet_iterator last);</code>
|
|
<p><code>first</code>: an iterator to a beginning of a UTF-8
|
|
encoded code point.<br>
|
|
<code>last</code>: an iterator to a "post-end" of the last UTF-8
|
|
encoded code point in the sequence we are trying to determine the
|
|
length. It can be the beginning of a new code point, or not.<br>
|
|
<u>Return value</u> the distance between the iterators, in code
|
|
points.</p>
|
|
<p>Example of use:</p>
|
|
<pre>
|
|
char* twochars = "\xe6\x97\xa5\xd1\x88";
|
|
|
|
size_t dist = utf8::distance(twochars, twochars + 5);
|
|
|
|
assert (dist == 2);
|
|
</pre>
|
|
<p>This function is used to find the length (in code points) of a
|
|
UTF-8 encoded string. The reason it is called <em>distance</em>,
|
|
rather than, say, <em>length</em> is mainly because developers are
|
|
used that <em>length</em> is an O(1) function. Computing the length
|
|
of an UTF-8 string is a linear operation, and it looked better to
|
|
model it after <code>std::distance</code> algorithm.</p>
|
|
<p>In case of an invalid UTF-8 seqence, a
|
|
<code>utf8::invalid_utf8</code> exception is thrown. If
|
|
<code>last</code> does not point to the past-of-end of a UTF-8
|
|
seqence, a <code>utf8::not_enough_room</code> exception is
|
|
thrown.</p>
|
|
<h4>utf8::utf16to8</h4>
|
|
<p>Converts a UTF-16 encoded string to UTF-8.</p>
|
|
<code>template <typename u16bit_iterator, typename
|
|
octet_iterator> octet_iterator utf16to8 (u16bit_iterator start,
|
|
u16bit_iterator end, octet_iterator result);</code>
|
|
<p><code>start</code>: an iterator pointing to the beginning of the
|
|
UTF-16 encoded string to convert.<br>
|
|
<code>end</code>: an iterator pointing to pass-the-end of the
|
|
UTF-16 encoded string to convert.<br>
|
|
<code>result</code>: an output iterator to the place in the UTF-8
|
|
string where to append the result of conversion.<br>
|
|
<u>Return value</u>: An iterator pointing to the place after the appended UTF-8 string.</p>
|
|
<p>Example of use:</p>
|
|
<pre>
|
|
unsigned short utf16string[] = {0x41, 0x0448, 0x65e5, 0xd834, 0xdd1e};
|
|
vector<unsigned char> utf8result;
|
|
|
|
utf16to8(utf16string, utf16string + 5, back_inserter(utf8result));
|
|
|
|
assert (utf8result.size() == 10);
|
|
</pre>
|
|
<p>In case of invalid UTF-16 sequence, a
|
|
<code>utf8::invalid_utf16</code> exception is thrown.</p>
|
|
<h4>utf8::utf8to16</h4>
|
|
<p>Converts an UTF-8 encoded string to UTF-16</p>
|
|
<code>template <typename u16bit_iterator, typename
|
|
octet_iterator> u16bit_iterator utf8to16 (octet_iterator start,
|
|
octet_iterator end, u16bit_iterator result);</code>
|
|
<p><code>start</code>: an iterator pointing to the beginning of the
|
|
UTF-8 encoded string to convert. < br /> <code>end</code>: an
|
|
iterator pointing to pass-the-end of the UTF-8 encoded string to
|
|
convert.<br>
|
|
<code>result</code>: an output iterator to the place in the UTF-16
|
|
string where to append the result of conversion.<br>
|
|
<u>Return value</u>: An iterator pointing to the place after the appended UTF-16 string.</p>
|
|
<p>Example of use:</p>
|
|
<pre>
|
|
char utf8_with_surrogates[] = "\xe6\x97\xa5\xd1\x88\xf0\x9d\x84\x9e";
|
|
vector <unsigned short> utf16result;
|
|
|
|
utf8to16(utf8_with_surrogates, utf8_with_surrogates + 9, back_inserter(utf16result));
|
|
|
|
assert (utf16result.size() == 4);
|
|
assert (utf16result[2] == 0xd834);
|
|
assert (utf16result[3] == 0xdd1e);
|
|
</pre>
|
|
<p>In case of an invalid UTF-8 seqence, a
|
|
<code>utf8::invalid_utf8</code> exception is thrown. If
|
|
<code>end</code> does not point to the past-of-end of a UTF-8
|
|
seqence, a <code>utf8::not_enough_room</code> exception is
|
|
thrown.</p>
|
|
<h4>utf8::utf32to8</h4>
|
|
<p>Converts a UTF-32 encoded string to UTF-8.</p>
|
|
<code>template <typename octet_iterator, typename
|
|
u32bit_iterator> octet_iterator utf32to8 (u32bit_iterator start,
|
|
u32bit_iterator end, octet_iterator result);</code>
|
|
<p><code>start</code>: an iterator pointing to the beginning of the
|
|
UTF-32 encoded string to convert.<br>
|
|
<code>end</code>: an iterator pointing to pass-the-end of the
|
|
UTF-32 encoded string to convert.<br>
|
|
<code>result</code>: an output iterator to the place in the UTF-8
|
|
string where to append the result of conversion.<br>
|
|
<u>Return value</u>: An iterator pointing to the place after the appended UTF-8 string.</p>
|
|
<p>Example of use:</p>
|
|
<pre>
|
|
int utf32string[] = {0x448, 0x65E5, 0x10346, 0};
|
|
vector<unsigned char> utf8result;
|
|
|
|
utf32to8(utf32string, utf32string + 3, back_inserter(utf8result));
|
|
|
|
assert (utf8result.size() == 9);
|
|
</pre>
|
|
<p>In case of invalid UTF-32 string, a
|
|
<code>utf8::invalid_code_point</code> exception is thrown.</p>
|
|
<h4>utf8::utf8to32</h4>
|
|
<p>Converts a UTF-8 encoded string to UTF-32.</p>
|
|
<code>template <typename octet_iterator, typename
|
|
u32bit_iterator> u32bit_iterator utf8to32 (octet_iterator start,
|
|
octet_iterator end, u32bit_iterator result);</code>
|
|
<p><code>start</code>: an iterator pointing to the beginning of the
|
|
UTF-8 encoded string to convert.<br>
|
|
<code>end</code>: an iterator pointing to pass-the-end of the UTF-8
|
|
encoded string to convert.<br>
|
|
<code>result</code>: an output iterator to the place in the UTF-32
|
|
string where to append the result of conversion.<br>
|
|
<u>Return value</u>: An iterator pointing to the place after the appended UTF-32 string.</p>
|
|
<p>Example of use:</p>
|
|
<pre>
|
|
char* twochars = "\xe6\x97\xa5\xd1\x88";
|
|
vector<int> utf32result;
|
|
|
|
utf8to32(twochars, twochars + 5, back_inserter(utf32result));
|
|
|
|
assert (utf32result.size() == 2);
|
|
</pre>
|
|
<p>In case of an invalid UTF-8 seqence, a
|
|
<code>utf8::invalid_utf8</code> exception is thrown. If
|
|
<code>end</code> does not point to the past-of-end of a UTF-8
|
|
seqence, a <code>utf8::not_enough_room</code> exception is
|
|
thrown.</p>
|
|
<h4>utf8::find_invalid</h4>
|
|
<p>Detects an invalid sequence within a UTF-8 string.</p>
|
|
<code>template <typename octet_iterator> octet_iterator
|
|
find_invalid(octet_iterator start, octet_iterator end);</code>
|
|
<p><code>start</code>: an iterator pointing to the beginning of the
|
|
UTF-8 string to test for validity.<br>
|
|
<code>end</code>: an iterator pointing to pass-the-end of the UTF-8
|
|
string to test for validity.<br>
|
|
<u>Return value</u>: an iterator pointing to the first invalid
|
|
octet in the UTF-8 string. In case none were found, equals
|
|
<code>end</code>.</p>
|
|
<p>Example of use:</p>
|
|
<pre>
|
|
char utf_invalid[] = "\xe6\x97\xa5\xd1\x88\xfa";
|
|
|
|
char* invalid = find_invalid(utf_invalid, utf_invalid + 6);
|
|
|
|
assert (invalid == utf_invalid + 5);
|
|
</pre>
|
|
<p>This function is typically used to make sure a UTF-8 string is
|
|
valid before processing it with other functions. It is especially
|
|
important to call it if before doing any of the <em>unchecked</em>
|
|
operations on it.</p>
|
|
<h4>utf8::is_valid</h4>
|
|
<p>Checks whether a sequence of octets is a valid UTF-8 string.</p>
|
|
<code>template <typename octet_iterator> bool
|
|
is_valid(octet_iterator start, octet_iterator end);</code>
|
|
<p><code>start</code>: an iterator pointing to the beginning of the
|
|
UTF-8 string to test for validity.<br>
|
|
<code>end</code>: an iterator pointing to pass-the-end of the UTF-8
|
|
string to test for validity.<br>
|
|
<u>Return value</u>: <code>true</code> if the sequence is a valid
|
|
UTF-8 string; <code>false</code> if not.</p>
|
|
Example of use:
|
|
<pre>
|
|
char utf_invalid[] = "\xe6\x97\xa5\xd1\x88\xfa";
|
|
|
|
bool bvalid = is_valid(utf_invalid, utf_invalid + 6);
|
|
|
|
assert (bvalid == false);
|
|
</pre>
|
|
<p><code>is_valid</code> is a shorthand for
|
|
<code>find_invalid(start, end) == end;</code>. You may want to use
|
|
it to make sure that a byte seqence is a valid UTF-8 string without
|
|
the need to know where it fails if it is not valid.</p>
|
|
<h4>utf8::replace_invalid</h4>
|
|
<p>Replaces all invalid UTF-8 sequences within a string with a replacement marker.</p>
|
|
<p><code>template <typename octet_iterator, typename output_iterator>
|
|
output_iterator replace_invalid(octet_iterator start, octet_iterator end, output_iterator out, uint32_t replacement);
|
|
</code></p>
|
|
<p><code>template <typename octet_iterator, typename output_iterator>
|
|
output_iterator replace_invalid(octet_iterator start, octet_iterator end, output_iterator out);
|
|
</code></p>
|
|
<p>
|
|
<code>start</code>: an iterator pointing to the beginning of the
|
|
UTF-8 string to look for invalid UTF-8 sequences.<br>
|
|
<code>end</code>: an iterator pointing to pass-the-end of the UTF-8
|
|
string to look for invalid UTF-8 sequences.<br>
|
|
<code>out</code>: An output iterator to the range where the result of replacement is stored.<br>
|
|
<code>replacement</code>: A Unicode code point for the replacement marker. The version without this
|
|
parameter assumes the value <code>0xfffd</code><br>
|
|
<u>Return value</u>: An iterator pointing to the place after the UTF-8 string with
|
|
replaced invalid sequences.</p>
|
|
<p>Example of use:</p>
|
|
<pre>
|
|
char invalid_sequence[] = "a\x80\xe0\xa0\xc0\xaf\xed\xa0\x80z";
|
|
vector<char> replace_invalid_result;
|
|
|
|
replace_invalid (invalid_sequence, invalid_sequence + sizeof(invalid_sequence), back_inserter(replace_invalid_result), '?');
|
|
|
|
bvalid = is_valid(replace_invalid_result.begin(), replace_invalid_result.end());
|
|
assert (bvalid);
|
|
char* fixed_invalid_sequence = "a????z";
|
|
assert (std::equal(replace_invalid_result.begin(), replace_invalid_result.end(), fixed_invalid_sequence));
|
|
</pre>
|
|
<p>
|
|
<code>replace_invalid</code> does not perform in-place replacement of invalid sequences. Rather, it produces a copy
|
|
of the original string with the invalid sequences replaced with a replacement marker. Therefore, <code>out</code> must
|
|
not be in the <code>[start, end]</code> range.
|
|
</p>
|
|
<p>If <code>end</code> does not point to the past-of-end of a UTF-8 sequence, a <code>utf8::not_enough_room</code>
|
|
exception is thrown.</p>
|
|
<h4>utf8::is_bom</h4>
|
|
<p>Checks whether a sequence of three octets is a UTF-8 byte order
|
|
mark (BOM)</p>
|
|
<code>template <typename octet_iterator> bool is_bom
|
|
(octet_iterator it);</code>
|
|
<p><code>it</code>: beginning of the 3-octet sequence to check<br>
|
|
<u>Return value</u>: <code>true</code> if the sequence is UTF-8
|
|
byte order mark; <code>false</code> if not.</p>
|
|
<p>Example of use:</p>
|
|
<pre>
|
|
unsigned char byte_order_mark[] = {0xef, 0xbb, 0xbf};
|
|
|
|
bool bbom = is_bom(byte_order_mark);
|
|
|
|
assert (bbom == true);
|
|
</pre>
|
|
<p>The typical use of this function is to check the first three
|
|
bytes of a file. If they form the UTF-8 BOM, we want to skip them
|
|
before processing the actual UTF-8 encoded text.</p>
|
|
<h3>Functions From utf8::unchecked Namespace</h3>
|
|
<h4>utf8::unchecked::append</h4>
|
|
<p>Encodes a 32 bit code point as a UTF-8 sequence of octets and
|
|
appends the sequence to a UTF-8 string.</p>
|
|
<code>template <typename octet_iterator> octet_iterator
|
|
append(uint32_t cp, octet_iterator result);</code>
|
|
<p><code>cp</code>: A 32 bit integer representing a code point to
|
|
append to the sequence.<br>
|
|
<code>result</code>: An output iterator to the place in the
|
|
sequence where to append the code point.<br>
|
|
<u>Return value</u>: An iterator pointing to the place after the
|
|
newly appended sequence.</p>
|
|
<p>Example of use:</p>
|
|
<pre>
|
|
unsigned char u[5] = {0,0,0,0,0};
|
|
|
|
unsigned char* end = unchecked::append(0x0448, u);
|
|
|
|
assert (u[0] == 0xd1 && u[1] == 0x88 && u[2] == 0 && u[3] == 0 && u[4] == 0);
|
|
</pre>
|
|
<p>This is a quicker but less safe version of
|
|
<code>utf8::append</code>. It does not check for validity of the
|
|
supplied code point, and may produce an invalid UTF-8 sequence.</p>
|
|
<h4>utf8::unchecked::next</h4>
|
|
<p>Given the iterator to the beginning of a UTF-8 sequence, it
|
|
returns the code point and moves the iterator to the next
|
|
position.</p>
|
|
<code>template <typename octet_iterator> uint32_t
|
|
next(octet_iterator& it);</code>
|
|
<p><code>it</code>: a reference to an iterator pointing to the
|
|
beginning of an UTF-8 encoded code point. After the function
|
|
returns, it is incremented to point to the beginning of the next
|
|
code point.<br>
|
|
<u>Return value</u>: the 32 bit representation of the processed
|
|
UTF-8 code point.</p>
|
|
<p>Example of use:</p>
|
|
<pre>
|
|
char* twochars = "\xe6\x97\xa5\xd1\x88";
|
|
char* w = twochars;
|
|
|
|
int cp = unchecked::next(w);
|
|
|
|
assert (cp == 0x65e5);
|
|
assert (w == twochars + 3);
|
|
</pre>
|
|
<p>This is a quicker but less safe version of
|
|
<code>utf8::next</code>. It does not check for validity of the
|
|
supplied UTF-8 sequence.</p>
|
|
<h4>utf8::unchecked::previous</h4>
|
|
<p>Given a reference to an iterator pointing to an octet in a UTF-8
|
|
seqence, it decreases the iterator until it hits the beginning of
|
|
the previous UTF-8 encoded code point and returns the 32 bits
|
|
representation of the code point.</p>
|
|
<code>template <typename octet_iterator> uint32_t
|
|
previous(octet_iterator& it);</code>
|
|
<p><code>it</code>: a reference pointing to an octet within a UTF-8
|
|
encoded string. After the function returns, it is decremented to
|
|
point to the beginning of the previous code point.<br>
|
|
<u>Return value</u>: the 32 bit representation of the previous code
|
|
point.</p>
|
|
<p>Example of use:</p>
|
|
<pre>
|
|
char* twochars = "\xe6\x97\xa5\xd1\x88";
|
|
char* w = twochars + 3;
|
|
|
|
int cp = unchecked::previous (w);
|
|
|
|
assert (cp == 0x65e5);
|
|
assert (w == twochars);
|
|
</pre>
|
|
<p>This is a quicker but less safe version of
|
|
<code>utf8::previous</code>. It does not check for validity of the
|
|
supplied UTF-8 sequence and offers no boundary checking.</p>
|
|
<h4>utf8::unchecked::advance</h4>
|
|
<p>Advances an iterator by the specified number of code points
|
|
within an UTF-8 sequence.</p>
|
|
<code>template <typename octet_iterator, typename
|
|
distance_type> void advance (octet_iterator& it,
|
|
distance_type n);</code>
|
|
<p><code>it</code>: a reference to an iterator pointing to the
|
|
beginning of an UTF-8 encoded code point. After the function
|
|
returns, it is incremented to point to the nth following code
|
|
point.<br>
|
|
<code>n</code>: a positive integer that shows how many code points
|
|
we want to advance.<br></p>
|
|
<p>Example of use:</p>
|
|
<pre>
|
|
char* twochars = "\xe6\x97\xa5\xd1\x88";
|
|
char* w = twochars;
|
|
|
|
unchecked::advance (w, 2);
|
|
|
|
assert (w == twochars + 5);
|
|
</pre>
|
|
<p>This function works only "forward". In case of a negative
|
|
<code>n</code>, there is no effect.</p>
|
|
<p>This is a quicker but less safe version of
|
|
<code>utf8::advance</code>. It does not check for validity of the
|
|
supplied UTF-8 sequence and offers no boundary checking.</p>
|
|
<h4>utf8::unchecked::distance</h4>
|
|
<p>Given the iterators to two UTF-8 encoded code points in a
|
|
seqence, returns the number of code points between them.</p>
|
|
<code>template <typename octet_iterator> typename
|
|
std::iterator_traits<octet_iterator>::difference_type
|
|
distance (octet_iterator first, octet_iterator last);</code>
|
|
<p><code>first</code>: an iterator to a beginning of a UTF-8
|
|
encoded code point.<br>
|
|
<code>last</code>: an iterator to a "post-end" of the last UTF-8
|
|
encoded code point in the sequence we are trying to determine the
|
|
length. It can be the beginning of a new code point, or not.<br>
|
|
<u>Return value</u> the distance between the iterators, in code
|
|
points.</p>
|
|
<p>Example of use:</p>
|
|
<pre>
|
|
char* twochars = "\xe6\x97\xa5\xd1\x88";
|
|
|
|
size_t dist = utf8::unchecked::distance(twochars, twochars + 5);
|
|
|
|
assert (dist == 2);
|
|
</pre>
|
|
<p>This is a quicker but less safe version of
|
|
<code>utf8::distance</code>. It does not check for validity of the
|
|
supplied UTF-8 sequence.</p>
|
|
<h4>utf8::unchecked::utf16to8</h4>
|
|
<p>Converts a UTF-16 encoded string to UTF-8.</p>
|
|
<code>template <typename u16bit_iterator, typename
|
|
octet_iterator> octet_iterator utf16to8 (u16bit_iterator start,
|
|
u16bit_iterator end, octet_iterator result);</code>
|
|
<p><code>start</code>: an iterator pointing to the beginning of the
|
|
UTF-16 encoded string to convert.<br>
|
|
<code>end</code>: an iterator pointing to pass-the-end of the
|
|
UTF-16 encoded string to convert.<br>
|
|
<code>result</code>: an output iterator to the place in the UTF-8
|
|
string where to append the result of conversion.
|
|
<u>Return value</u>: An iterator pointing to the place after the appended UTF-8 string.</p>
|
|
<p>Example of use:</p>
|
|
<pre>
|
|
unsigned short utf16string[] = {0x41, 0x0448, 0x65e5, 0xd834, 0xdd1e};
|
|
vector<unsigned char> utf8result;
|
|
|
|
unchecked::utf16to8(utf16string, utf16string + 5, back_inserter(utf8result));
|
|
|
|
assert (utf8result.size() == 10);
|
|
</pre>
|
|
<p>This is a quicker but less safe version of
|
|
<code>utf8::utf16to8</code>. It does not check for validity of the
|
|
supplied UTF-16 sequence.</p>
|
|
<h4>utf8::unchecked::utf8to16</h4>
|
|
<p>Converts an UTF-8 encoded string to UTF-16</p>
|
|
<code>template <typename u16bit_iterator, typename
|
|
octet_iterator> u16bit_iterator utf8to16 (octet_iterator start,
|
|
octet_iterator end, u16bit_iterator result);</code>
|
|
<p><code>start</code>: an iterator pointing to the beginning of the
|
|
UTF-8 encoded string to convert. < br /> <code>end</code>: an
|
|
iterator pointing to pass-the-end of the UTF-8 encoded string to
|
|
convert.<br>
|
|
<code>result</code>: an output iterator to the place in the UTF-16
|
|
string where to append the result of conversion.<br>
|
|
<u>Return value</u>: An iterator pointing to the place after the appended UTF-16 string.
|
|
</p>
|
|
|
|
<p>Example of use:</p>
|
|
<pre>
|
|
char utf8_with_surrogates[] = "\xe6\x97\xa5\xd1\x88\xf0\x9d\x84\x9e";
|
|
vector <unsigned short> utf16result;
|
|
|
|
unchecked::utf8to16(utf8_with_surrogates, utf8_with_surrogates + 9, back_inserter(utf16result));
|
|
|
|
assert (utf16result.size() == 4);
|
|
assert (utf16result[2] == 0xd834);
|
|
assert (utf16result[3] == 0xdd1e);
|
|
</pre>
|
|
<p>This is a quicker but less safe version of
|
|
<code>utf8::utf8to16</code>. It does not check for validity of the
|
|
supplied UTF-8 sequence.</p>
|
|
<h4>utf8::unchecked::utf32to8</h4>
|
|
<p>Converts a UTF-32 encoded string to UTF-8.</p>
|
|
<code>template <typename octet_iterator, typename
|
|
u32bit_iterator> octet_iterator utf32to8 (u32bit_iterator start,
|
|
u32bit_iterator end, octet_iterator result);</code>
|
|
<p><code>start</code>: an iterator pointing to the beginning of the
|
|
UTF-32 encoded string to convert.<br>
|
|
<code>end</code>: an iterator pointing to pass-the-end of the
|
|
UTF-32 encoded string to convert.<br>
|
|
<code>result</code>: an output iterator to the place in the UTF-8
|
|
string where to append the result of conversion.<br>
|
|
<u>Return value</u>: An iterator pointing to the place after the appended UTF-8 string.
|
|
</p>
|
|
<p>Example of use:</p>
|
|
<pre>
|
|
int utf32string[] = {0x448, 0x65E5, 0x10346, 0};
|
|
vector<unsigned char> utf8result;
|
|
|
|
utf32to8(utf32string, utf32string + 3, back_inserter(utf8result));
|
|
|
|
assert (utf8result.size() == 9);
|
|
</pre>
|
|
<p>This is a quicker but less safe version of
|
|
<code>utf8::utf32to8</code>. It does not check for validity of the
|
|
supplied UTF-32 sequence.</p>
|
|
<h4>utf8::unchecked::utf8to32</h4>
|
|
<p>Converts a UTF-8 encoded string to UTF-32.</p>
|
|
<code>template <typename octet_iterator, typename
|
|
u32bit_iterator> u32bit_iterator utf8to32 (octet_iterator start,
|
|
octet_iterator end, u32bit_iterator result);</code>
|
|
<p><code>start</code>: an iterator pointing to the beginning of the
|
|
UTF-8 encoded string to convert.<br>
|
|
<code>end</code>: an iterator pointing to pass-the-end of the UTF-8
|
|
encoded string to convert.<br>
|
|
<code>result</code>: an output iterator to the place in the UTF-32
|
|
string where to append the result of conversion.<br>
|
|
<u>Return value</u>: An iterator pointing to the place after the appended UTF-32 string.
|
|
</p>
|
|
<p>Example of use:</p>
|
|
<pre>
|
|
char* twochars = "\xe6\x97\xa5\xd1\x88";
|
|
vector<int> utf32result;
|
|
|
|
unchecked::utf8to32(twochars, twochars + 5, back_inserter(utf32result));
|
|
|
|
assert (utf32result.size() == 2);
|
|
</pre>
|
|
<p>This is a quicker but less safe version of
|
|
<code>utf8::utf8to32</code>. It does not check for validity of the
|
|
supplied UTF-8 sequence.</p>
|
|
<h2 id="points">Points of interest</h2>
|
|
<h4>Design goals and decisions</h4>
|
|
<p>The library was designed to be:</p>
|
|
<ol>
|
|
<li>Generic: for better or worse, there are many C++ string classes
|
|
out there, and the library should work with as many of them as
|
|
possible.</li>
|
|
<li>Portable: the library should be portable both accross different
|
|
platforms and compilers. The only non-portable code is a small
|
|
section that declares unsigned integers of different sizes: three
|
|
typedefs. They can be changed by the users of the library if they
|
|
don't match their platform. The default setting should work for
|
|
Windows (both 32 and 64 bit), and most 32 bit and 64 bit Unix
|
|
derivatives.</li>
|
|
<li>Lightweight: follow the "pay only for what you use"
|
|
guidline.</li>
|
|
<li>Unintrusive: avoid forcing any particular design or even
|
|
programming style on the user. This is a library, not a
|
|
framework.</li>
|
|
</ol>
|
|
<h4>Alternatives</h4>
|
|
<p>In case you want to look into other means of working with UTF-8
|
|
strings from C++, here is the list of solutions I am aware of:</p>
|
|
<ol>
|
|
<li><a href="http://icu.sourceforge.net/">ICU Library</a>. It is
|
|
very powerful, complete, feature-rich, mature, and widely used.
|
|
Also big, intrusive, non-generic, and doesn't play well with the
|
|
Standard Library. I definitelly recommend looking at ICU even if
|
|
you don't plan to use it.</li>
|
|
<li><a href=
|
|
"http://www.gtkmm.org/gtkmm2/docs/tutorial/html/ch03s04.html">Glib::ustring</a>.
|
|
A class specifically made to work with UTF-8 strings, and also feel
|
|
like <code>std::string</code>. If you prefer to have yet another
|
|
string class in your code, it may be worth a look. Be aware of the
|
|
licensing issues, though.</li>
|
|
<li>Platform dependent solutions: Windows and POSIX have functions
|
|
to convert strings from one encoding to another. That is only a
|
|
subset of what my library offers, but if that is all you need it
|
|
may be good enough, especially given the fact that these functions
|
|
are mature and tested in production.</li>
|
|
</ol>
|
|
<h2 id="conclusion">Conclusion</h2>
|
|
<p>Until Unicode becomes officially recognized by the C++ Standard
|
|
Library, we need to use other means to work with UTF-8 strings.
|
|
Template functions I describe in this article may be a good step in
|
|
this direction.</p>
|
|
<h2 id="references">References</h2>
|
|
<ol>
|
|
<li><a href="http://www.unicode.org/">The Unicode
|
|
Consortium</a>.</li>
|
|
<li><a href="http://icu.sourceforge.net/">ICU Library</a>.</li>
|
|
<li><a href="http://en.wikipedia.org/wiki/UTF-8">UTF-8 at
|
|
Wikipedia</a></li>
|
|
<li><a href="http://www.cl.cam.ac.uk/~mgk25/unicode.html">UTF-8 and Unicode FAQ for Unix/Linux</a></li>
|
|
</ol>
|
|
</body>
|
|
</html>
|