View source for CESU-8

{{Short description|Encoding scheme for Unicode}}
The '''Compatibility Encoding Scheme for UTF-16: 8-Bit''' ('''CESU-8''') is a variant of [[UTF-8]] that is described in [[Unicode]] Technical Report #26.<ref>{{Cite web |url=https://www.unicode.org/reports/tr26/ |title=Unicode Technical Report #26 - Compatibility Encoding Scheme for UTF-16: 8-Bit (CESU-8) |last=McGowan |first=Rick |publisher=Unicode Consortium}}</ref>  A Unicode code point from the Basic Multilingual Plane (BMP), i.e. a code point in the range U+0000 to U+FFFF, is encoded in the same way as in UTF-8.  A Unicode supplementary character, i.e. a code point in the range U+10000 to U+10FFFF, is first represented as a surrogate pair, like in [[UTF-16]], and then each surrogate code point is encoded in UTF-8.  Therefore, CESU-8 needs six bytes (3 bytes per surrogate) for each Unicode supplementary character while UTF-8 needs only four. Though not specified in the technical report, ''unpaired'' surrogates are also encoded as 3 bytes each, and CESU-8 is exactly the same as applying an older UCS-2 to UTF-8 converter to UTF-16 data.

The encoding of Unicode non-BMP characters works out to <code>11101101 1010yyyy 10xxxxxx 11101101 1011xxxx 10xxxxxx</code> (yyyy represents the top five bits of the character minus one). The byte values 0xF0—0xF4 will not appear in CESU-8, as they start the 4-byte encodings used by UTF-8.

CESU-8 is not an official part of the Unicode Standard, because Unicode Technical Reports are informative documents only.<ref>{{Cite web |url=https://www.unicode.org/reports/about-reports.html#Types |title=About Unicode Technical Reports - Types of Unicode Technical Reports: UAX, UTS, UTR |publisher=Unicode Consortium}}</ref>  It should be used exclusively for internal processing and never for external data exchange.

Supporting CESU-8 in [[HTML]] documents is prohibited by the W3C<ref>{{Cite web |url=https://www.w3.org/TR/html51/syntax.html#character-encodings |title=8.2.2.3. Character encodings |website=HTML 5.1 Standard |publisher=W3C}}</ref><ref>{{Cite web |url=https://www.w3.org/TR/html5/syntax.html#character-encodings |title=8.2.2.3. Character encodings |website=HTML 5 Standard |publisher=W3C}}</ref> and [[Organization:WHATWG|WHATWG]]<ref>{{Cite web |url=https://html.spec.whatwg.org/multipage/parsing.html#character-encodings |title=12.2.3.3 Character encodings |website=HTML Living Standard |publisher=WHATWG}}</ref> HTML standards, as it would present a [[Cross-site scripting|cross-site scripting]] vulnerability.<ref>{{Cite web |url=https://developer.mozilla.org/en-US/docs/Web/HTML/Element/meta |title=&lt;meta> - HTML |website=MDN Web Docs |publisher=Mozilla}}</ref>

Java's [[UTF-8#Modified UTF-8|Modified UTF-8]] is CESU-8 with a special overlong encoding of the NUL character (U+0000) as the two-byte sequence <code>C0 80</code>.<ref>{{cite web |title=Java SE documentation for Interface java.io.DataInput, subsection on Modified UTF-8 |url=https://docs.oracle.com/javase/8/docs/api/java/io/DataInput.html#modified-utf-8 |year=2015 |publisher=[[Company:Oracle Corporation|Oracle Corporation]] |access-date=2021-04-30}}</ref>

The [[Software:Oracle database|Oracle database]] uses CESU-8 for its "UTF8" character set. Standard UTF-8 can be obtained using the character set "AL32UTF8" (since Oracle version 9.0).

==Examples==

{| class="wikitable"
|-----
! Code point 
| U+0045 || colspan="2" | U+0205 || colspan="12" | U+10400
|-----
! Character 
| E || colspan="2" | ȅ || colspan="12" | 𐐀
|-----
! UTF-8
| 45
| style="border-right: 1px #dddddd solid" | C8
| style="border-left: 1px #dddddd solid" | 85
| colspan="3" style="border-right: 1px #dddddd solid" | F0
| colspan="3" style="border-left: 1px #dddddd solid; border-right: 1px #dddddd solid" | 90
| colspan="3" style="border-left: 1px #dddddd solid; border-right: 1px #dddddd solid" | 90
| colspan="3" style="border-left: 1px #dddddd solid" | 80
|-----
! UTF-16
| 0045 || colspan="2" | 0205 || colspan="6" | D801
| colspan="6" | DC00
|-----
! CESU-8
| 45
| style="border-right: 1px #dddddd solid" | C8
| style="border-left: 1px #dddddd solid" | 85
| colspan="2" style="border-right: 1px #dddddd solid" | ED
| colspan="2" style="border-left: 1px #dddddd solid; border-right: 1px #dddddd solid" | A0
| colspan="2" style="border-left: 1px #dddddd solid" | 81
| colspan="2" style="border-right: 1px #dddddd solid" | ED
| colspan="2" style="border-left: 1px #dddddd solid; border-right: 1px #dddddd solid" | B0
| colspan="2" style="border-left: 1px #dddddd solid" | 80
|}

==References==
{{Reflist}}

==External links==
* [https://www.unicode.org/reports/tr26/ Unicode Technical Report #26]
* [https://docs.oracle.com/javase/8/docs/technotes/guides/jni/spec/types.html#modified_utf_8_strings Modified UTF-8 definition]
* [http://demo.icu-project.org/icu-bin/convexp?conv=CESU-8 Graphical View of CESU-8 in ICU's Converter Explorer]

{{Unicode navigation}}
{{character encoding}}

{{DEFAULTSORT:Cesu-8}}
[[Category:Unicode Transformation Formats]]
[[Category:Character encoding]]

{{Sourceattribution|CESU-8}}