693 lines
28 KiB
HTML
693 lines
28 KiB
HTML
|
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 3.2//EN">
|
||
|
|
<html>
|
||
|
|
<head>
|
||
|
|
<meta name="generator" content=
|
||
|
|
"HTML Tidy for Linux/x86 (vers 12 April 2005), see www.w3.org">
|
||
|
|
<title></title>
|
||
|
|
</head>
|
||
|
|
<body>
|
||
|
|
<h2>Introduction</h2>
|
||
|
|
<p>Many C++ developers miss an easy and portable way of handling
|
||
|
|
Unicode encoded strings. C++ Standard is currently Unicode
|
||
|
|
agnostic, and while some work is being done to introduce Unicode to
|
||
|
|
the next incarnation called C++0x, for the moment nothing of the
|
||
|
|
sort is available. In the meantime, developers use 3rd party
|
||
|
|
libraries like ICU, OS specific capabilities, or simply roll out
|
||
|
|
their own solutions.</p>
|
||
|
|
<p>In order to easily handle UTF-8 encoded Unicode strings, I have
|
||
|
|
come up with a set of template functions. For anybody used to work
|
||
|
|
with STL algorithms, they should be easy and natural to use. The
|
||
|
|
code is freely available for any purpose - check out the license at
|
||
|
|
the beginning of the utf8.h file. Be aware, though, that while I
|
||
|
|
did some testing, this library has not been used in production yet.
|
||
|
|
If you run into bugs or performance issues, please let me know and
|
||
|
|
I'll do my best to address them.</p>
|
||
|
|
<p>The purpose of this article is not to offer an introduction to
|
||
|
|
Unicode in general, and UTF-8 in particular. If you are not
|
||
|
|
familiar with Unicode, be sure to check out <a href=
|
||
|
|
"http://www.unicode.org/">Unicode Home Page</a> or some other
|
||
|
|
source of information for Unicode. Also, it is not my aim to
|
||
|
|
advocate the use of UTF-8 encoded strings in C++ programs; if you
|
||
|
|
want to handle UTF-8 encoded strings from C++, I am sure you have
|
||
|
|
good reasons for it.</p>
|
||
|
|
<h2>Examples of use</h2>
|
||
|
|
<p>To illustrate the use of this utf8 library, we shall open a file
|
||
|
|
containing a line of UTF-8 encoded text, read the line into
|
||
|
|
<code>std::string</code>, convert the text to UTF-16, and write it
|
||
|
|
to another file:</p>
|
||
|
|
<pre>
|
||
|
|
#include <fstream>
|
||
|
|
#include <iostream>
|
||
|
|
#include <string>
|
||
|
|
#include <vector>
|
||
|
|
using namespace std;
|
||
|
|
|
||
|
|
int main()
|
||
|
|
{
|
||
|
|
// Open the file with a utf-8 encoded line of text in it
|
||
|
|
ifstream fs8("utf8.txt");
|
||
|
|
if (!fs8.is_open()) {
|
||
|
|
cout << "Could not open utf8.txt" << endl;
|
||
|
|
return 0;
|
||
|
|
}
|
||
|
|
|
||
|
|
// is there a utf8 marker? if yes, skip it.
|
||
|
|
fs8.seekg(0, ios::end);
|
||
|
|
ifstream::pos_type file_length = fs8.tellg();
|
||
|
|
fs8.seekg(0, ios::beg);
|
||
|
|
if (file_length > 3) {
|
||
|
|
char bom[3];
|
||
|
|
fs8.read(bom, 3);
|
||
|
|
if (!utf8::is_bom(bom))
|
||
|
|
fs8.seekg(0, ios::beg);
|
||
|
|
}
|
||
|
|
|
||
|
|
// Read the line from the file
|
||
|
|
string text8;
|
||
|
|
getline(fs8, text8);
|
||
|
|
|
||
|
|
// Make sure it is valid utf-8
|
||
|
|
if (!utf8::is_valid(text8.begin(), text8.end())) {
|
||
|
|
cout << "Invalid utf-8 text";
|
||
|
|
return 0;
|
||
|
|
}
|
||
|
|
|
||
|
|
// Convert the text to utf-16
|
||
|
|
vector<unsigned short> text16;
|
||
|
|
text16.push_back(0xfeff); // bom
|
||
|
|
utf8::utf8to16(text8.begin(), text8.end(), back_inserter(text16));
|
||
|
|
|
||
|
|
// Create the file for writing the utf-16 string
|
||
|
|
ofstream fs16("utf16.txt", ios_base::out | ios_base::binary);
|
||
|
|
if (!fs16.is_open()) {
|
||
|
|
cout << "Could not open utf16.txt" << endl;
|
||
|
|
return 0;
|
||
|
|
}
|
||
|
|
|
||
|
|
// Write the utf16 text to the file
|
||
|
|
fs16.write(reinterpret_cast<const char*>(&text16[0]), text16.size() * sizeof (unsigned short));
|
||
|
|
}
|
||
|
|
</pre>
|
||
|
|
<p>In the previous code sample, we have seen the use of 3 functions
|
||
|
|
from <code>utf8</code> namespace: first we used <code>is_bom</code>
|
||
|
|
function to detect UTF-8 byte order mark at the beginning of the
|
||
|
|
file, then <code>is_valid</code> to make sure that the text we
|
||
|
|
loaded is valid UTF-8, and finally <code>utf8to16</code> to convert
|
||
|
|
the text to UTF-16 encoding. Note that the use of
|
||
|
|
<code>is_valid</code> was optional in this case;
|
||
|
|
<code>utf8to16</code> throws an exception in case of invalid UTF-8
|
||
|
|
text.</p>
|
||
|
|
<h2>Reference</h2>
|
||
|
|
<h3>Functions From utf8 Namespace</h3>
|
||
|
|
<h4>utf8::append</h4>
|
||
|
|
<p>Encodes a 32 bit code point as a UTF-8 sequence of octets and
|
||
|
|
appends the sequence to a UTF-8 string.</p>
|
||
|
|
<code>template <typename octet_iterator> octet_iterator
|
||
|
|
append(uint32_t cp, octet_iterator result);</code>
|
||
|
|
<p><code>cp</code>: A 32 bit integer representing a code point to
|
||
|
|
append to the sequence.<br>
|
||
|
|
<code>result</code>: An output iterator to the place in the
|
||
|
|
sequence where to append the code point.<br>
|
||
|
|
<u>Return value</u>: An iterator pointing to the place after the
|
||
|
|
newly appended sequence.</p>
|
||
|
|
<p>Example of use:</p>
|
||
|
|
<pre>
|
||
|
|
unsigned char u[5] = {0,0,0,0,0};
|
||
|
|
|
||
|
|
unsigned char* end = append(0x0448, u);
|
||
|
|
|
||
|
|
assert (u[0] == 0xd1 && u[1] == 0x88 && u[2] == 0 && u[3] == 0 && u[4] == 0);
|
||
|
|
</pre>
|
||
|
|
<p>Note that <code>append</code> does not allocate any memory - it
|
||
|
|
is the burden of the caller to make sure there is enough memory
|
||
|
|
allocated for the operation. To make things more interesting,
|
||
|
|
<code>append</code> can add anywhere between 1 and 4 octets to the
|
||
|
|
sequence. In practice, you would most often want to use
|
||
|
|
<code>std::back_inserter</code> to ensure that the necessary memory
|
||
|
|
is allocated.</p>
|
||
|
|
<p>In case of an invalid code point, a
|
||
|
|
<code>utf8::invalid_code_point</code> exception is thrown.</p>
|
||
|
|
<h4>utf8::next</h4>
|
||
|
|
<p>Given the iterator to the beginning of the UTF-8 sequence, it
|
||
|
|
returns the code point and moves the iterator to the next
|
||
|
|
position.</p>
|
||
|
|
<code>template <typename octet_iterator> uint32_t
|
||
|
|
next(octet_iterator& it, octet_iterator end);</code>
|
||
|
|
<p><code>it</code>: a reference to an iterator pointing to the
|
||
|
|
beginning of an UTF-8 encoded code point. After the function
|
||
|
|
returns, it is incremented to point to the beginning of the next
|
||
|
|
code point.<br>
|
||
|
|
<code>end</code>: end of the UTF-8 sequence to be processed. If
|
||
|
|
<code>it</code> gets equal to <code>end</code> during the
|
||
|
|
extraction of a code point, an <code>utf8::not_enough_room</code>
|
||
|
|
exception is thrown.<br>
|
||
|
|
<u>Return value</u>: the 32 bit representation of the processed
|
||
|
|
UTF-8 code point.</p>
|
||
|
|
<p>Example of use:</p>
|
||
|
|
<pre>
|
||
|
|
unsigned char twochars[] = {0xE6, 0x97, 0xA5, 0xd1, 0x88, 0x0};
|
||
|
|
unsigned char* w = twochars;
|
||
|
|
|
||
|
|
int cp = next(w, twochars + 6);
|
||
|
|
|
||
|
|
assert (cp == 0x65e5);
|
||
|
|
assert (w == twochars + 3);
|
||
|
|
</pre>
|
||
|
|
<p>This function is typically used to iterate through a UTF-8
|
||
|
|
encoded string.</p>
|
||
|
|
<p>In case of an invalid UTF-8 seqence, a
|
||
|
|
<code>utf8::invalid_utf8</code> exception is thrown.</p>
|
||
|
|
<h4>utf8::previous</h4>
|
||
|
|
<p>Given a reference to an iterator pointing to an octet in a UTF-8
|
||
|
|
seqence, it decreases the iterator until it hits the beginning of
|
||
|
|
the previous UTF-8 encoded code point and returns the 32 bits
|
||
|
|
representation of the code point.</p>
|
||
|
|
<code>template <typename octet_iterator> uint32_t
|
||
|
|
previous(octet_iterator& it, octet_iterator pass_start);</code>
|
||
|
|
<p><code>it</code>: a reference pointing to an octet within a UTF-8
|
||
|
|
encoded string. After the function returns, it is decremented to
|
||
|
|
point to the beginning of the previous code point.<br>
|
||
|
|
<code>pass_start</code>: an iterator to the point in the sequence
|
||
|
|
where the search for the beginning of a code point is aborted if no
|
||
|
|
result was reached. It is a safety measure to prevent passing the
|
||
|
|
beginning of the string in the search for a UTF-8 lead octet.<br>
|
||
|
|
<u>Return value</u>: the 32 bit representation of the previous code
|
||
|
|
point.</p>
|
||
|
|
<p>Example of use:</p>
|
||
|
|
<pre>
|
||
|
|
unsigned char twochars[] = {0xE6, 0x97, 0xA5, 0xd1, 0x88, 0x0};
|
||
|
|
unsigned char* w = twochars + 3;
|
||
|
|
|
||
|
|
int cp = previous (w, twochars - 1);
|
||
|
|
|
||
|
|
assert (cp == 0x65e5);
|
||
|
|
assert (w == twochars);
|
||
|
|
</pre>
|
||
|
|
<p>The primary purpose of this function is to iterate backwards
|
||
|
|
through a UTF-8 encoded string. Therefore, <code>it</code> will
|
||
|
|
typically point to the beginning of a code point, and
|
||
|
|
<code>pass_start</code> will point to the octet just before the
|
||
|
|
beginning of the string to ensure we don't go backwards too far.
|
||
|
|
<code>it</code> is decreased until it points to a lead UTF-8 octet,
|
||
|
|
and then the UTF-8 sequence beginning with that octet is decoded to
|
||
|
|
a 32 bit representation and returned.</p>
|
||
|
|
<p>In case <code>pass_end</code> is reached before a UTF-8 lead
|
||
|
|
octet is hit, or if an invalid UTF-8 sequence is started by the
|
||
|
|
lead octet, an <code>invalid_utf8</code> exception is thrown</p>
|
||
|
|
<h4>utf8::advance</h4>
|
||
|
|
<p>Advances an iterator by the specified number of code points
|
||
|
|
within an UTF-8 sequence.</p>
|
||
|
|
<code>template <typename octet_iterator, typename
|
||
|
|
distance_type> void advance (octet_iterator& it,
|
||
|
|
distance_type n, octet_iterator end);</code>
|
||
|
|
<p><code>it</code>: a reference to an iterator pointing to the
|
||
|
|
beginning of an UTF-8 encoded code point. After the function
|
||
|
|
returns, it is incremented to point to the nth following code
|
||
|
|
point.<br>
|
||
|
|
<code>n</code>: a positive integer that shows how many code points
|
||
|
|
we want to advance.<br>
|
||
|
|
<code>end</code>: end of the UTF-8 sequence to be processed. If
|
||
|
|
<code>it</code> gets equal to <code>end</code> during the
|
||
|
|
extraction of a code point, an <code>utf8::not_enough_room</code>
|
||
|
|
exception is thrown.<br></p>
|
||
|
|
<p>Example of use:</p>
|
||
|
|
<pre>
|
||
|
|
unsigned char twochars[] = {0xE6, 0x97, 0xA5, 0xd1, 0x88, 0x0};
|
||
|
|
unsigned char* w = twochars;
|
||
|
|
|
||
|
|
advance (w, 2, twochars + 6);
|
||
|
|
|
||
|
|
assert (w == twochars + 5);
|
||
|
|
</pre>
|
||
|
|
<p>This function works only "forward". In case of a negative
|
||
|
|
<code>n</code>, there is no effect.</p>
|
||
|
|
<p>In case of an invalid code point, a
|
||
|
|
<code>utf8::invalid_code_point</code> exception is thrown.</p>
|
||
|
|
<h4>utf8::distance</h4>
|
||
|
|
<p>Given the iterators to two UTF-8 encoded code points in a
|
||
|
|
seqence, returns the number of code points between them.</p>
|
||
|
|
<code>template <typename octet_iterator> typename
|
||
|
|
std::iterator_traits<octet_iterator>::difference_type
|
||
|
|
distance (octet_iterator first, octet_iterator last);</code>
|
||
|
|
<p><code>first</code>: an iterator to a beginning of a UTF-8
|
||
|
|
encoded code point.<br>
|
||
|
|
<code>last</code>: an iterator to a "post-end" of the last UTF-8
|
||
|
|
encoded code point in the sequence we are trying to determine the
|
||
|
|
length. It can be the beginning of a new code point, or not.<br>
|
||
|
|
<u>Return value</u> the distance between the iterators, in code
|
||
|
|
points.</p>
|
||
|
|
<p>Example of use:</p>
|
||
|
|
<pre>
|
||
|
|
unsigned char twochars[] = {0xE6, 0x97, 0xA5, 0xd1, 0x88, 0x0};
|
||
|
|
|
||
|
|
size_t dist = utf8::distance(twochars, twochars + 5);
|
||
|
|
|
||
|
|
assert (dist == 2);
|
||
|
|
</pre>
|
||
|
|
<p>This function is used to find the length (in code points) of a
|
||
|
|
UTF-8 encoded string. The reason it is called <em>distance</em>,
|
||
|
|
rather than, say, <em>length</em> is mainly because developers are
|
||
|
|
used that <em>length</em> is an O(1) function. Computing the length
|
||
|
|
of an UTF-8 string is a linear operation, and it looked better to
|
||
|
|
model it after <code>std::distance</code> algorithm.</p>
|
||
|
|
<p>In case of an invalid UTF-8 seqence, a
|
||
|
|
<code>utf8::invalid_utf8</code> exception is thrown. If
|
||
|
|
<code>last</code> does not point to the past-of-end of a UTF-8
|
||
|
|
seqence, a <code>utf8::not_enough_room</code> exception is
|
||
|
|
thrown.</p>
|
||
|
|
<h4>utf8::utf16to8</h4>
|
||
|
|
<p>Converts a UTF-16 encoded string to UTF-8.</p>
|
||
|
|
<code>template <typename u16bit_iterator, typename
|
||
|
|
octet_iterator> void utf16to8 (u16bit_iterator start,
|
||
|
|
u16bit_iterator end, octet_iterator result);</code>
|
||
|
|
<p><code>start</code>: an iterator pointing to the beginning of the
|
||
|
|
UTF-16 encoded string to convert.<br>
|
||
|
|
<code>end</code>: an iterator pointing to pass-the-end of the
|
||
|
|
UTF-16 encoded string to convert.<br>
|
||
|
|
<code>result</code>: an output iterator to the place in the UTF-8
|
||
|
|
string where to append the result of conversion.</p>
|
||
|
|
<p>Example of use:</p>
|
||
|
|
<pre>
|
||
|
|
unsigned short utf16string[] = {0x41, 0x0448, 0x65e5, 0xd834, 0xdd1e};
|
||
|
|
vector<unsigned char> utf8result;
|
||
|
|
|
||
|
|
utf16to8(utf16string, utf16string + 5, back_inserter(utf8result));
|
||
|
|
|
||
|
|
assert (utf8result.size() == 10);
|
||
|
|
</pre>
|
||
|
|
<p>In case of invalid UTF-16 sequence, a
|
||
|
|
<code>utf8::invalid_utf16</code> exception is thrown.</p>
|
||
|
|
<h4>utf8::utf8to16</h4>
|
||
|
|
<p>Converts an UTF-8 encoded string to UTF-16</p>
|
||
|
|
<code>template <typename u16bit_iterator, typename
|
||
|
|
octet_iterator> void utf8to16 (octet_iterator start,
|
||
|
|
octet_iterator end, u16bit_iterator result);</code>
|
||
|
|
<p><code>start</code>: an iterator pointing to the beginning of the
|
||
|
|
UTF-8 encoded string to convert. < br /> <code>end</code>: an
|
||
|
|
iterator pointing to pass-the-end of the UTF-8 encoded string to
|
||
|
|
convert.<br>
|
||
|
|
<code>result</code>: an output iterator to the place in the UTF-16
|
||
|
|
string where to append the result of conversion.</p>
|
||
|
|
<p>Example of use:</p>
|
||
|
|
<pre>
|
||
|
|
unsigned char utf8_with_surrogates[] = {0xE6, 0x97, 0xA5, 0xd1, 0x88,
|
||
|
|
0xf0, 0x9d, 0x84, 0x9e};
|
||
|
|
vector <unsigned short> utf16result;
|
||
|
|
|
||
|
|
utf8to16(utf8_with_surrogates, utf8_with_surrogates + 9, back_inserter(utf16result));
|
||
|
|
|
||
|
|
assert (utf16result.size() == 4);
|
||
|
|
assert (utf16result[2] == 0xd834);
|
||
|
|
assert (utf16result[3] == 0xdd1e);
|
||
|
|
</pre>
|
||
|
|
<p>In case of an invalid UTF-8 seqence, a
|
||
|
|
<code>utf8::invalid_utf8</code> exception is thrown. If
|
||
|
|
<code>last</code> does not point to the past-of-end of a UTF-8
|
||
|
|
seqence, a <code>utf8::not_enough_room</code> exception is
|
||
|
|
thrown.</p>
|
||
|
|
<h4>utf8::utf32to8</h4>
|
||
|
|
<p>Converts a UTF-32 encoded string to UTF-8.</p>
|
||
|
|
<code>template <typename octet_iterator, typename
|
||
|
|
u32bit_iterator> void utf32to8 (u32bit_iterator start,
|
||
|
|
u32bit_iterator end, octet_iterator result);</code>
|
||
|
|
<p><code>start</code>: an iterator pointing to the beginning of the
|
||
|
|
UTF-32 encoded string to convert.<br>
|
||
|
|
<code>end</code>: an iterator pointing to pass-the-end of the
|
||
|
|
UTF-32 encoded string to convert.<br>
|
||
|
|
<code>result</code>: an output iterator to the place in the UTF-8
|
||
|
|
string where to append the result of conversion.</p>
|
||
|
|
<p>Example of use:</p>
|
||
|
|
<pre>
|
||
|
|
int utf32string[] = {0x448, 0x65E5, 0x10346, 0};
|
||
|
|
vector<unsigned char> utf8result;
|
||
|
|
|
||
|
|
utf32to8(utf32string, utf32string + 3, back_inserter(utf8result));
|
||
|
|
|
||
|
|
assert (utf8result.size() == 9);
|
||
|
|
</pre>
|
||
|
|
<p>In case of invalid UTF-32 string, a
|
||
|
|
<code>utf8::invalid_code_point</code> exception is thrown.</p>
|
||
|
|
<h4>utf8::utf8to32</h4>
|
||
|
|
<p>Converts a UTF-8 encoded string to UTF-32.</p>
|
||
|
|
<code>template <typename octet_iterator, typename
|
||
|
|
u32bit_iterator> void utf8to32 (octet_iterator start,
|
||
|
|
octet_iterator end, u32bit_iterator result);</code>
|
||
|
|
<p><code>start</code>: an iterator pointing to the beginning of the
|
||
|
|
UTF-8 encoded string to convert.<br>
|
||
|
|
<code>end</code>: an iterator pointing to pass-the-end of the UTF-8
|
||
|
|
encoded string to convert.<br>
|
||
|
|
<code>result</code>: an output iterator to the place in the UTF-32
|
||
|
|
string where to append the result of conversion.</p>
|
||
|
|
<p>Example of use:</p>
|
||
|
|
<pre>
|
||
|
|
unsigned char twochars[] = {0xE6, 0x97, 0xA5, 0xd1, 0x88, 0x0};
|
||
|
|
vector<int> utf32result;
|
||
|
|
|
||
|
|
utf8to32(twochars, twochars + 5, back_inserter(utf32result));
|
||
|
|
|
||
|
|
assert (utf32result.size() == 2);
|
||
|
|
</pre>
|
||
|
|
<p>In case of an invalid UTF-8 seqence, a
|
||
|
|
<code>utf8::invalid_utf8</code> exception is thrown. If
|
||
|
|
<code>last</code> does not point to the past-of-end of a UTF-8
|
||
|
|
seqence, a <code>utf8::not_enough_room</code> exception is
|
||
|
|
thrown.</p>
|
||
|
|
<h4>utf8::find_invalid</h4>
|
||
|
|
<p>Detects an invalid sequence within a UTF-8 string.</p>
|
||
|
|
<code>template <typename octet_iterator> octet_iterator
|
||
|
|
find_invalid(octet_iterator start, octet_iterator end);</code>
|
||
|
|
<p><code>start</code>: an iterator pointing to the beginning of the
|
||
|
|
UTF-8 string to test for validity.<br>
|
||
|
|
<code>end</code>: an iterator pointing to pass-the-end of the UTF-8
|
||
|
|
string to test for validity.<br>
|
||
|
|
<u>Return value</u>: an iterator pointing to the first invalid
|
||
|
|
octet in the UTF-8 string. In case none were found, equals
|
||
|
|
<code>end</code>.</p>
|
||
|
|
<p>Example of use:</p>
|
||
|
|
<pre>
|
||
|
|
unsigned char utf_invalid[] = {0xE6, 0x97, 0xA5, 0xd1, 0x88, 0xfa};
|
||
|
|
|
||
|
|
unsigned char* invalid = find_invalid(utf_invalid, utf_invalid + 6);
|
||
|
|
|
||
|
|
assert (invalid == utf_invalid + 5);
|
||
|
|
</pre>
|
||
|
|
<p>This function is typically used to make sure a UTF-8 string is
|
||
|
|
valid before processing it with other functions. It is especially
|
||
|
|
important to call it if before doing any of the <em>unchecked</em>
|
||
|
|
operations on it.</p>
|
||
|
|
<h4>utf8::is_valid</h4>
|
||
|
|
<p>Checks whether a sequence of octets is a valid UTF-8 string.</p>
|
||
|
|
<code>template <typename octet_iterator> bool
|
||
|
|
is_valid(octet_iterator start, octet_iterator end);</code>
|
||
|
|
<p><code>start</code>: an iterator pointing to the beginning of the
|
||
|
|
UTF-8 string to test for validity.<br>
|
||
|
|
<code>end</code>: an iterator pointing to pass-the-end of the UTF-8
|
||
|
|
string to test for validity.<br>
|
||
|
|
<u>Return value</u>: <code>true</code> if the sequence is a valid
|
||
|
|
UTF-8 string; <code>false</code> if not.</p>
|
||
|
|
Example of use:
|
||
|
|
<pre>
|
||
|
|
unsigned char utf_invalid[] = {0xE6, 0x97, 0xA5, 0xd1, 0x88, 0xfa};
|
||
|
|
|
||
|
|
bool bvalid = is_valid(utf_invalid, utf_invalid + 6);
|
||
|
|
|
||
|
|
assert (bvalid == false);
|
||
|
|
</pre>
|
||
|
|
<p><code>is_valid</code> is a shorthand for
|
||
|
|
<code>find_invalid(start, end) == end;</code>. You may want to use
|
||
|
|
it to make sure that a byte seqence is a valid UTF-8 string without
|
||
|
|
the need to know where it fails if it is not valid.</p>
|
||
|
|
<h4>utf8::is_bom</h4>
|
||
|
|
<p>Checks whether a sequence of three octets is a UTF-8 byte order
|
||
|
|
mark (BOM)</p>
|
||
|
|
<code>template <typename octet_iterator> bool is_bom
|
||
|
|
(octet_iterator it);</code>
|
||
|
|
<p><code>it</code> Beginning of the 3-octet sequence to check<br>
|
||
|
|
<u>Return value</u>: <code>true</code> if the sequence is UTF-8
|
||
|
|
byte order mark; <code>false</code> if not.</p>
|
||
|
|
<p>Example of use:</p>
|
||
|
|
<pre>
|
||
|
|
unsigned char byte_order_mark[] = {0xef, 0xbb, 0xbf};
|
||
|
|
|
||
|
|
bool bbom = is_bom(byte_order_mark);
|
||
|
|
|
||
|
|
assert (bbom == true);
|
||
|
|
</pre>
|
||
|
|
<p>The typical use of this function is to check the first three
|
||
|
|
bytes of a file. If they form the UTF-8 BOM, we want to skip them
|
||
|
|
before processing the actual UTF-8 encoded text.</p>
|
||
|
|
<h3>Functions From utf8::unchecked Namespace</h3>
|
||
|
|
<h4>utf8::unchecked::append</h4>
|
||
|
|
<p>Encodes a 32 bit code point as a UTF-8 sequence of octets and
|
||
|
|
appends the sequence to a UTF-8 string.</p>
|
||
|
|
<code>template <typename octet_iterator> octet_iterator
|
||
|
|
append(uint32_t cp, octet_iterator result);</code>
|
||
|
|
<p><code>cp</code>: A 32 bit integer representing a code point to
|
||
|
|
append to the sequence.<br>
|
||
|
|
<code>result</code>: An output iterator to the place in the
|
||
|
|
sequence where to append the code point.<br>
|
||
|
|
<u>Return value</u>: An iterator pointing to the place after the
|
||
|
|
newly appended sequence.</p>
|
||
|
|
<p>Example of use:</p>
|
||
|
|
<pre>
|
||
|
|
unsigned char u[5] = {0,0,0,0,0};
|
||
|
|
|
||
|
|
unsigned char* end = unchecked::append(0x0448, u);
|
||
|
|
|
||
|
|
assert (u[0] == 0xd1 && u[1] == 0x88 && u[2] == 0 && u[3] == 0 && u[4] == 0);
|
||
|
|
</pre>
|
||
|
|
<p>This is a quicker but less safe version of
|
||
|
|
<code>utf8::append</code>. It does not check for validity of the
|
||
|
|
supplied code point, and may produce an invalid UTF-8 sequence.</p>
|
||
|
|
<h4>utf8::unchecked::next</h4>
|
||
|
|
<p>Given the iterator to the beginning of a UTF-8 sequence, it
|
||
|
|
returns the code point and moves the iterator to the next
|
||
|
|
position.</p>
|
||
|
|
<code>template <typename octet_iterator> uint32_t
|
||
|
|
next(octet_iterator& it);</code>
|
||
|
|
<p><code>it</code>: a reference to an iterator pointing to the
|
||
|
|
beginning of an UTF-8 encoded code point. After the function
|
||
|
|
returns, it is incremented to point to the beginning of the next
|
||
|
|
code point.<br>
|
||
|
|
<u>Return value</u>: the 32 bit representation of the processed
|
||
|
|
UTF-8 code point.</p>
|
||
|
|
<p>Example of use:</p>
|
||
|
|
<pre>
|
||
|
|
unsigned char twochars[] = {0xE6, 0x97, 0xA5, 0xd1, 0x88, 0x0};
|
||
|
|
unsigned char* w = twochars;
|
||
|
|
|
||
|
|
int cp = unchecked::next(w);
|
||
|
|
|
||
|
|
assert (cp == 0x65e5);
|
||
|
|
assert (w == twochars + 3);
|
||
|
|
</pre>
|
||
|
|
<p>This is a quicker but less safe version of
|
||
|
|
<code>utf8::next</code>. It does not check for validity of the
|
||
|
|
supplied UTF-8 sequence.</p>
|
||
|
|
<h4>utf8::unchecked::previous</h4>
|
||
|
|
<p>Given a reference to an iterator pointing to an octet in a UTF-8
|
||
|
|
seqence, it decreases the iterator until it hits the beginning of
|
||
|
|
the previous UTF-8 encoded code point and returns the 32 bits
|
||
|
|
representation of the code point.</p>
|
||
|
|
<code>template <typename octet_iterator> uint32_t
|
||
|
|
previous(octet_iterator& it);</code>
|
||
|
|
<p><code>it</code>: a reference pointing to an octet within a UTF-8
|
||
|
|
encoded string. After the function returns, it is decremented to
|
||
|
|
point to the beginning of the previous code point.<br>
|
||
|
|
<u>Return value</u>: the 32 bit representation of the previous code
|
||
|
|
point.</p>
|
||
|
|
<p>Example of use:</p>
|
||
|
|
<pre>
|
||
|
|
unsigned char twochars[] = {0xE6, 0x97, 0xA5, 0xd1, 0x88, 0x0};
|
||
|
|
unsigned char* w = twochars + 3;
|
||
|
|
|
||
|
|
int cp = unchecked::previous (w);
|
||
|
|
|
||
|
|
assert (cp == 0x65e5);
|
||
|
|
assert (w == twochars);
|
||
|
|
</pre>
|
||
|
|
<p>This is a quicker but less safe version of
|
||
|
|
<code>utf8::previous</code>. It does not check for validity of the
|
||
|
|
supplied UTF-8 sequence and offers no boundary checking.</p>
|
||
|
|
<h4>utf8::unchecked::advance</h4>
|
||
|
|
<p>Advances an iterator by the specified number of code points
|
||
|
|
within an UTF-8 sequence.</p>
|
||
|
|
<code>template <typename octet_iterator, typename
|
||
|
|
distance_type> void advance (octet_iterator& it,
|
||
|
|
distance_type n);</code>
|
||
|
|
<p><code>it</code>: a reference to an iterator pointing to the
|
||
|
|
beginning of an UTF-8 encoded code point. After the function
|
||
|
|
returns, it is incremented to point to the nth following code
|
||
|
|
point.<br>
|
||
|
|
<code>n</code>: a positive integer that shows how many code points
|
||
|
|
we want to advance.<br></p>
|
||
|
|
<p>Example of use:</p>
|
||
|
|
<pre>
|
||
|
|
unsigned char twochars[] = {0xE6, 0x97, 0xA5, 0xd1, 0x88, 0x0};
|
||
|
|
unsigned char* w = twochars;
|
||
|
|
|
||
|
|
unchecked::advance (w, 2);
|
||
|
|
|
||
|
|
assert (w == twochars + 5);
|
||
|
|
</pre>
|
||
|
|
<p>This function works only "forward". In case of a negative
|
||
|
|
<code>n</code>, there is no effect.</p>
|
||
|
|
<p>This is a quicker but less safe version of
|
||
|
|
<code>utf8::advance</code>. It does not check for validity of the
|
||
|
|
supplied UTF-8 sequence and offers no boundary checking.</p>
|
||
|
|
<h4>utf8::unchecked::distance</h4>
|
||
|
|
<p>Given the iterators to two UTF-8 encoded code points in a
|
||
|
|
seqence, returns the number of code points between them.</p>
|
||
|
|
<code>template <typename octet_iterator> typename
|
||
|
|
std::iterator_traits<octet_iterator>::difference_type
|
||
|
|
distance (octet_iterator first, octet_iterator last);</code>
|
||
|
|
<p><code>first</code>: an iterator to a beginning of a UTF-8
|
||
|
|
encoded code point.<br>
|
||
|
|
<code>last</code>: an iterator to a "post-end" of the last UTF-8
|
||
|
|
encoded code point in the sequence we are trying to determine the
|
||
|
|
length. It can be the beginning of a new code point, or not.<br>
|
||
|
|
<u>Return value</u> the distance between the iterators, in code
|
||
|
|
points.</p>
|
||
|
|
<p>Example of use:</p>
|
||
|
|
<pre>
|
||
|
|
unsigned char twochars[] = {0xE6, 0x97, 0xA5, 0xd1, 0x88, 0x0};
|
||
|
|
|
||
|
|
size_t dist = utf8::unchecked::distance(twochars, twochars + 5);
|
||
|
|
|
||
|
|
assert (dist == 2);
|
||
|
|
</pre>
|
||
|
|
<p>This is a quicker but less safe version of
|
||
|
|
<code>utf8::distance</code>. It does not check for validity of the
|
||
|
|
supplied UTF-8 sequence.</p>
|
||
|
|
<h4>utf8::unchecked::utf16to8</h4>
|
||
|
|
<p>Converts a UTF-16 encoded string to UTF-8.</p>
|
||
|
|
<code>template <typename u16bit_iterator, typename
|
||
|
|
octet_iterator> void utf16to8 (u16bit_iterator start,
|
||
|
|
u16bit_iterator end, octet_iterator result);</code>
|
||
|
|
<p><code>start</code>: an iterator pointing to the beginning of the
|
||
|
|
UTF-16 encoded string to convert.<br>
|
||
|
|
<code>end</code>: an iterator pointing to pass-the-end of the
|
||
|
|
UTF-16 encoded string to convert.<br>
|
||
|
|
<code>result</code>: an output iterator to the place in the UTF-8
|
||
|
|
string where to append the result of conversion.</p>
|
||
|
|
<p>Example of use:</p>
|
||
|
|
<pre>
|
||
|
|
unsigned short utf16string[] = {0x41, 0x0448, 0x65e5, 0xd834, 0xdd1e};
|
||
|
|
vector<unsigned char> utf8result;
|
||
|
|
|
||
|
|
unchecked::utf16to8(utf16string, utf16string + 5, back_inserter(utf8result));
|
||
|
|
|
||
|
|
assert (utf8result.size() == 10);
|
||
|
|
</pre>
|
||
|
|
<p>This is a quicker but less safe version of
|
||
|
|
<code>utf8::utf16to8</code>. It does not check for validity of the
|
||
|
|
supplied UTF-16 sequence.</p>
|
||
|
|
<h4>utf8::unchecked::utf8to16</h4>
|
||
|
|
<p>Converts an UTF-8 encoded string to UTF-16</p>
|
||
|
|
<code>template <typename u16bit_iterator, typename
|
||
|
|
octet_iterator> void utf8to16 (octet_iterator start,
|
||
|
|
octet_iterator end, u16bit_iterator result);</code>
|
||
|
|
<p><code>start</code>: an iterator pointing to the beginning of the
|
||
|
|
UTF-8 encoded string to convert. < br /> <code>end</code>: an
|
||
|
|
iterator pointing to pass-the-end of the UTF-8 encoded string to
|
||
|
|
convert.<br>
|
||
|
|
<code>result</code>: an output iterator to the place in the UTF-16
|
||
|
|
string where to append the result of conversion.</p>
|
||
|
|
<p>Example of use:</p>
|
||
|
|
<pre>
|
||
|
|
unsigned char utf8_with_surrogates[] = {0xE6, 0x97, 0xA5, 0xd1, 0x88,
|
||
|
|
0xf0, 0x9d, 0x84, 0x9e};
|
||
|
|
vector <unsigned short> utf16result;
|
||
|
|
|
||
|
|
unchecked::utf8to16(utf8_with_surrogates, utf8_with_surrogates + 9, back_inserter(utf16result));
|
||
|
|
|
||
|
|
assert (utf16result.size() == 4);
|
||
|
|
assert (utf16result[2] == 0xd834);
|
||
|
|
assert (utf16result[3] == 0xdd1e);
|
||
|
|
</pre>
|
||
|
|
<p>This is a quicker but less safe version of
|
||
|
|
<code>utf8::utf8to16</code>. It does not check for validity of the
|
||
|
|
supplied UTF-8 sequence.</p>
|
||
|
|
<h4>utf8::unchecked::utf32to8</h4>
|
||
|
|
<p>Converts a UTF-32 encoded string to UTF-8.</p>
|
||
|
|
<code>template <typename octet_iterator, typename
|
||
|
|
u32bit_iterator> void utf32to8 (u32bit_iterator start,
|
||
|
|
u32bit_iterator end, octet_iterator result);</code>
|
||
|
|
<p><code>start</code>: an iterator pointing to the beginning of the
|
||
|
|
UTF-32 encoded string to convert.<br>
|
||
|
|
<code>end</code>: an iterator pointing to pass-the-end of the
|
||
|
|
UTF-32 encoded string to convert.<br>
|
||
|
|
<code>result</code>: an output iterator to the place in the UTF-8
|
||
|
|
string where to append the result of conversion.</p>
|
||
|
|
<p>Example of use:</p>
|
||
|
|
<pre>
|
||
|
|
int utf32string[] = {0x448, 0x65E5, 0x10346, 0};
|
||
|
|
vector<unsigned char> utf8result;
|
||
|
|
|
||
|
|
utf32to8(utf32string, utf32string + 3, back_inserter(utf8result));
|
||
|
|
|
||
|
|
assert (utf8result.size() == 9);
|
||
|
|
</pre>
|
||
|
|
<p>This is a quicker but less safe version of
|
||
|
|
<code>utf8::utf32to8</code>. It does not check for validity of the
|
||
|
|
supplied UTF-32 sequence.</p>
|
||
|
|
<h4>utf8::unchecked::utf8to32</h4>
|
||
|
|
<p>Converts a UTF-8 encoded string to UTF-32.</p>
|
||
|
|
<code>template <typename octet_iterator, typename
|
||
|
|
u32bit_iterator> void utf8to32 (octet_iterator start,
|
||
|
|
octet_iterator end, u32bit_iterator result);</code>
|
||
|
|
<p><code>start</code>: an iterator pointing to the beginning of the
|
||
|
|
UTF-8 encoded string to convert.<br>
|
||
|
|
<code>end</code>: an iterator pointing to pass-the-end of the UTF-8
|
||
|
|
encoded string to convert.<br>
|
||
|
|
<code>result</code>: an output iterator to the place in the UTF-32
|
||
|
|
string where to append the result of conversion.</p>
|
||
|
|
<p>Example of use:</p>
|
||
|
|
<pre>
|
||
|
|
unsigned char twochars[] = {0xE6, 0x97, 0xA5, 0xd1, 0x88, 0x0};
|
||
|
|
vector<int> utf32result;
|
||
|
|
|
||
|
|
unchecked::utf8to32(twochars, twochars + 5, back_inserter(utf32result));
|
||
|
|
|
||
|
|
assert (utf32result.size() == 2);
|
||
|
|
</pre>
|
||
|
|
<p>This is a quicker but less safe version of
|
||
|
|
<code>utf8::utf8to32</code>. It does not check for validity of the
|
||
|
|
supplied UTF-8 sequence.</p>
|
||
|
|
<h2>Points of interest</h2>
|
||
|
|
<h4>Design goals and decisions</h4>
|
||
|
|
<p>The library was designed to be:</p>
|
||
|
|
<ol>
|
||
|
|
<li>Generic: for better or worse, there are many C++ string classes
|
||
|
|
out there, and the library should work with as many of them as
|
||
|
|
possible.</li>
|
||
|
|
<li>Portable: the library should be portable both accross different
|
||
|
|
platforms and compilers. The only non-portable code is a small
|
||
|
|
section that declares unsigned integers of different sizes: three
|
||
|
|
typedefs. They can be changed by the users of the library if they
|
||
|
|
don't match their platform. The default setting should work for
|
||
|
|
Windows (both 32 and 64 bit), and most 32 bit and 64 bit Unix
|
||
|
|
derivatives.</li>
|
||
|
|
<li>Lightweight: follow the "pay only for what you use"
|
||
|
|
guidline.</li>
|
||
|
|
<li>Unintrusive: avoid forcing any particular design or even
|
||
|
|
programming style on the user. This is a library, not a
|
||
|
|
framework.</li>
|
||
|
|
</ol>
|
||
|
|
<h4>Alternatives</h4>
|
||
|
|
<p>In case you want to look into other means of working with UTF-8
|
||
|
|
strings from C++, here is the list of solutions I am aware of:</p>
|
||
|
|
<ol>
|
||
|
|
<li><a href="http://icu.sourceforge.net/">ICU Library</a>. It is
|
||
|
|
very powerful, complete, feature-rich, mature, and widely used.
|
||
|
|
Also big, intrusive, non-generic, and doesn't play well with the
|
||
|
|
Standard Library. I definitelly recommend looking at ICU even if
|
||
|
|
you don't plan to use it.</li>
|
||
|
|
<li><a href=
|
||
|
|
"http://www.gtkmm.org/gtkmm2/docs/tutorial/html/ch03s04.html">Glib::ustring</a>.
|
||
|
|
A class specifically made to work with UTF-8 strings, and also feel
|
||
|
|
like <code>std::string</code>. If you prefer to have yet another
|
||
|
|
string class in your code, it may be worth a look. Be aware of the
|
||
|
|
licensing issues, though.</li>
|
||
|
|
<li>Platform dependent solutions: Windows and POSIX have functions
|
||
|
|
to convert strings from one encoding to another. That is only a
|
||
|
|
subset of what my library offers, but if that is all you need it
|
||
|
|
may be good enough, especially given the fact that these functions
|
||
|
|
are mature and tested in production.</li>
|
||
|
|
</ol>
|
||
|
|
<h2>Conclusion</h2>
|
||
|
|
<p>Until Unicode becomes officially recognized by the C++ Standard
|
||
|
|
Library, we need to use other means to work with UTF-8 strings.
|
||
|
|
Template functions I describe in this article may be a good step in
|
||
|
|
this direction.</p>
|
||
|
|
<h2>References</h2>
|
||
|
|
<ol>
|
||
|
|
<li><a href="http://www.unicode.org/">The Unicode
|
||
|
|
Consortium</a>.</li>
|
||
|
|
<li><a href="http://icu.sourceforge.net/">ICU Library</a>.</li>
|
||
|
|
<li><a href="http://en.wikipedia.org/wiki/UTF-8">UTF-8 at
|
||
|
|
Wikipedia</a></li>
|
||
|
|
</ol>
|
||
|
|
</body>
|
||
|
|
</html>
|