Han unification
Encyclopedia
Han unification is an effort by the authors of Unicode
and the Universal Character Set
to map multiple character sets of the so-called CJK
languages into a single set of unified characters. Han characters are a common feature of written Chinese
(hanzi), Japanese
(kanji
), Korean
(hanja
), and—at least historically—other East and Southeast Asian languages. (See Vietnamese
Hán Tự and Chữ Nôm.)
Modern Chinese, Japanese and Korean typeface
s typically use regional or historical variants of a given Han character
. In the formulation of Unicode, an attempt was made to unify these variants by considering them different glyphs representing the same "grapheme", or orthographic
unit, hence, "Han unification", with the resulting character repertoire sometimes contracted to Unihan.
Unihan can also refer to the Unihan Database maintained by the Unicode Consortium, which provides information about all of the unified Han characters encoded in the Unicode standard, including mappings to various national and industry standards, indices into standard dictionaries, encoded variants, pronunciations in various languages, and an English definition. The database is available to the public as text files and via an interactive Web site. The latter also includes representative glyphs and definitions for compound words drawn from the free Japanese EDICT
and Chinese CEDICT
dictionary projects (which are provided for convenience and are not a formal part of the Unicode standard).
(IRG), made up of experts from the Chinese-speaking countries, North and South Korea, Japan, Vietnam, and other countries, is responsible for the process.
One possible rationale is the desire to limit the size of the full Unicode character set, where CJK characters as represented by discrete ideograms may approach or exceed 100,000 (while those required for ordinary literacy in any language are probably under 3,000). Version 1 of Unicode was designed to fit into 16 bits and only 20,940 characters (32%) out of the possible 65,536 were reserved for these CJK Unified Ideographs
. Later Unicode has been extended to 21 bits allowing many more CJK characters (75,960 are assigned, with room for more).
The secret life of Unicode article located on IBM DeveloperWorks attempts to illustrate part of the motivation for Han unification:
In fact, the three ideographs for "one" are encoded separately in Unicode, as they are not considered national variants. The first and second are used on financial instruments to prevent tampering (they may be considered variants), while the third is the common form in all three countries.
However, Han unification has also caused considerable controversy, particularly among the Japanese public, who, with the nation's literati, have a history of protesting the culling of historically and culturally significant variants. (See Kanji#Orthographic reform and lists of kanji. Today, the list of characters officially recognized for use in proper names continues to expand at a modest pace.)
However, this refers to the fact that some graphemes are composed of several characters. So, for example, the character "a" (U+0061) combined with a circle above (U+030A) (i.e. å
) might be understood by a user as a single grapheme while being composed of multiple Unicode abstract characters. In addition, Unicode also assigns some code points to a small number (other than for compatibility reasons) of formatting characters, whitespace characters, and other abstract characters that are not graphemes, but instead used to control the breaks between lines, words, graphemes and grapheme clusters. With the unified Han ideographs, the Unicode standard makes a departure from prior practices in assigning abstract characters not as graphemes, but according to the underlying meaning of the grapheme: what linguists sometimes call sememe
s. This departure therefore is not simply explained by the oft quoted distinction between an abstract character and a glyph, but is more rooted in the difference between an abstract character assigned as a grapheme and an abstract character assigned as a sememe. In contrast, consider Unicode's unification of punctuation and diacritic
s, where graphemes with widely different meanings (for example, an apostrophe and a single quotation mark) are unified because the graphemes are the same. For Unihan the characters are not unified by their appearance, but by their definition or meaning.
For a grapheme to be represented by various glyphs means that the grapheme has glyph variations that are usually determined by selecting one font or another or using glyph substitution features where multiple glyphs are included in a single font. Such glyph variations are considered by Unicode a feature of rich text protocols and not properly handled by the plain text goals of Unicode. However, when the change from one glyph to another constitutes a change from one grapheme to another—where a glyph cannot possibly still, for example, mean the same grapheme understood as the small letter "a"—Unicode separates those into separate code points. For Unihan the same thing is done whenever the abstract meaning changes, however rather than speaking of the abstract meaning of a grapheme (the letter "a"), the unification of Han ideographs assigns a new code point for each different meaning—even if that meaning is expressed by distinct graphemes in different languages. While a grapheme such as "ö" might mean something different in English (as used in the word "coördinated") than it does in German, it is still the same grapheme and can be easily unified so that English and German can share a common abstract Latin writing system (along with Latin itself).
To deal with the use of different graphemes for the same Unihan sememe, Unicode has relied on several mechanisms to deal with the issue: especially as it relates to rendering text. One has been to treat it as simply a font issue so that different fonts might be used to render Chinese, Japanese or Korean. Also font formats such as OpenType allow for the mapping of alternate glyphs according to language so that a text rendering system can look to the user's environmental settings to determine which glyph to use. The problem with these approaches is that they fail to meet the goals of Unicode to support multilingual text within the same document.
So rather than treat the issue as a rich text problem of glyph alternates, Unicode added the concept of variation selectors, first introduced in version 3.2 and supplemented in version 4.0. While variation selectors are treated as combining characters, they have no associated diacritic or mark. Instead, by combining with a base character, they signal the two character sequence selects a variation (typically in terms of grapheme, but also in terms of underlying meaning as in the case of a location name or other proper noun) of the base character. This then is not a selection of an alternate glyph, but the selection of a grapheme variation or a variation of the base abstract character. Such a two-character sequence however can be easily mapped to a separate single glyph in modern fonts. Since Unicode has assigned 256 separate variation selectors, it is capable of assigning 256 variations for any Han ideograph. Such variations can be specific to one language or another and enable the encoding of plain text that includes such grapheme variations.
Small differences in graphical representation are also problematic when they affect legibility or the wrong cultural tradition. Besides making some Unicode fonts unusable for texts involving multiple "Unihan languages", names or other orthographically sensitive terminology might be displayed incorrectly. (Proper names tend to be especially orthographically conservative—compare this to changing the spelling of one's name to suit a language reform in the U.S. or U.K.) While this may be considered primarily a graphical representation or rendering problem to be overcome by more artful fonts, the widespread use of Unicode would make it difficult to preserve such distinctions. The problem of one character representing semantically different concepts is also present in the Latin part of Unicode. The Unicode character for an apostrophe is the same as the character for a right single quote (’). On the other hand, it is sometimes pointed out that the capital Latin letter "A" is not unified with the Greek letter "Α" (Alpha). This is, of course, desirable for reasons of compatibility, and deals with a much smaller alphabetic character set.
While the unification aspect of Unicode is controversial in some quarters for the reasons given above, Unicode itself does now encode a vast number of seldom-used characters of a more-or-less antiquarian nature.
Some of the controversy stems from the fact that the very decision of performing Han unification was made by the initial Unicode Consortium, which at the time was a consortium of North American companies and organizations (most of them in California), but included no East Asia government representatives. The initial design goal was to create a 16-bit standard, and Han unification was therefore a critical step for avoiding tens of thousands of character duplications. This 16-bit requirement was later abandoned, making the size of the character set less an issue today.
The controversy later extended to the internationally representative ISO: the initial CJK-JRG group favored a proposal (DIS 10646) for a non-unified character set, "which was thrown out in favor of unification with the Unicode Consortium's unified character set by the votes of American and European ISO members" (even though the Japanese position was unclear). Endorsing the Unicode Han unification was a necessary step for the heated ISO 10646/Unicode merger.
Much of the controversy surrounding Han unification is based on the distinction between glyph
s, as defined in Unicode, and the related but distinct idea of graphemes. Unicode assigns abstract characters (graphemes), as opposed to glyphs, which are a particular visual representations of a character in a specific typeface
. One character may be represented by many distinct glyphs, for example a "g" or an "a", both of which may have one loop (a, g) or two (a, g). Yet for a reader of Latin script based languages the two variations of the "a" character are both recognized as the same grapheme. Graphemes present in national character code standards have been added to Unicode, as required by Unicode's Source Separation rule, even where they can be composed of characters already available. The national character code standards existing in CJK languages are considerably more involved, given the technological limitations under which they evolved, and so the official CJK participants in Han unification may well have been amenable to reform.
Unlike European versions, CJK Unicode fonts, due to Han unification, have large but irregular patterns of overlap, requiring language-specific fonts. Unfortunately, language-specific fonts also make it difficult to access to a variant which, as with the "grass" example, happens to appear more typically in another language style. (That is to say, it would be difficult to access "grass" with the four-stroke radical more typical of Traditional Chinese in a Japanese environment, which fonts would typically depict the three-stroke radical.) Unihan proponents tend to favor markup languages for defining language strings, but this would not ensure the use of a specific variant in the case given, only the language-specific font more likely to depict a character as that variant. (At this point, merely stylistic differences do enter in, as a selection of Japanese and Chinese fonts are not likely to be visually compatible.)
Chinese users seem to have fewer objections to Han unification, largely because Unicode did not attempt to unify Simplified Chinese characters (an invention of the People's Republic of China
, and in use among Chinese speakers in the PRC, Singapore
, and Malaysia), with Traditional Chinese characters, as used in Hong Kong, Taiwan (Big5
), and, with some differences, more familiar to Korean and Japanese users. Unicode is seen as neutral with regards to this politically charged issue, and has encoded Simplified and Traditional Chinese glyphs separately (e.g. the ideograph for "discard" is 丟 U+4E1F for Traditional Chinese Big5 #A5E1 and 丢 U+4E22 for Simplified Chinese GB #2210). It is also noted that Traditional and Simplified characters should be encoded separately according to Unicode Han Unification rules, because they are distinguished in pre-existing PRC character sets. Furthermore, as with other variants, Traditional to Simplified characters is not a one-to-one relationship.
However, none of these alternative standards has been as widely adopted as Unicode
, which is now the base character set for many new standards and protocols, and is built into the architecture of operating systems (Microsoft Windows
, Apple
Mac OS X
, and many Unix-like
systems), programming languages (Perl
, Python, C#, Java
, Common LISP
, APL
), and libraries (IBM International Components for Unicode
(ICU) along with the Pango
, Graphite
, Scribe
, Uniscribe
, and ATSUI
rendering engines), font formats (TrueType
and OpenType
) and so on.
(3 varieties: unmarked "Chinese", simplified characters
, and traditional characters
), Japanese
, or Korean
. The browser
should select, for each character, a glyph
(from a font
) suitable to the specified language. (Besides actual character variation—look for differences in stroke order, number, or direction—the typefaces may also reflect different typographical styles, as with serif and non-serif alphabets.) This only works for fallback glyph selection if you have CJK fonts installed on your system and the font selected to display this article does not include glyphs for these characters. Note also that Unicode includes non-graphical language tag characters in the range U+E0000 – U+E007F for plain text language tagging.
Unicode includes support of CJKV radicals, strokes, punctuation, marks and symbols in the following blocks:
Additional compatibility (discouraged use) characters appear in these blocks:
These compatibility characters (excluding the twelve unified ideographs in the CJK Compatibility Ideographs block) are included for compatibility with legacy text handling systems and other legacy character sets. They include forms of characters for vertical text layout and rich text characters that Unicode recommends handling through other means.
An Unihan.zip file is provided on unicode.org. It contains all the data the Unihan team have collected.
A project libUnihan (0.5.3) provides a normalized SQLite Unihan database and corresponding C library. All tables in this database are in fifth normal form.
libUnihan is released as LGPL, while its database, UnihanDb, is released as MIT License
.
Unicode
Unicode is a computing industry standard for the consistent encoding, representation and handling of text expressed in most of the world's writing systems...
and the Universal Character Set
Universal Character Set
The Universal Character Set , defined by the International Standard ISO/IEC 10646, Information technology — Universal multiple-octet coded character set , is a standard set of characters upon which many character encodings are based...
to map multiple character sets of the so-called CJK
CJK
CJK is a collective term for Chinese, Japanese, and Korean, which is used in the field of software and communications internationalization.The term CJKV means CJK plus Vietnamese, which constitute the main East Asian languages.- Characteristics :...
languages into a single set of unified characters. Han characters are a common feature of written Chinese
Chinese language
The Chinese language is a language or language family consisting of varieties which are mutually intelligible to varying degrees. Originally the indigenous languages spoken by the Han Chinese in China, it forms one of the branches of Sino-Tibetan family of languages...
(hanzi), Japanese
Japanese language
is a language spoken by over 130 million people in Japan and in Japanese emigrant communities. It is a member of the Japonic language family, which has a number of proposed relationships with other languages, none of which has gained wide acceptance among historical linguists .Japanese is an...
(kanji
Kanji
Kanji are the adopted logographic Chinese characters hanzi that are used in the modern Japanese writing system along with hiragana , katakana , Indo Arabic numerals, and the occasional use of the Latin alphabet...
), Korean
Korean language
Korean is the official language of the country Korea, in both South and North. It is also one of the two official languages in the Yanbian Korean Autonomous Prefecture in People's Republic of China. There are about 78 million Korean speakers worldwide. In the 15th century, a national writing...
(hanja
Hanja
Hanja is the Korean name for the Chinese characters hanzi. More specifically, it refers to those Chinese characters borrowed from Chinese and incorporated into the Korean language with Korean pronunciation...
), and—at least historically—other East and Southeast Asian languages. (See Vietnamese
Vietnamese language
Vietnamese is the national and official language of Vietnam. It is the mother tongue of 86% of Vietnam's population, and of about three million overseas Vietnamese. It is also spoken as a second language by many ethnic minorities of Vietnam...
Hán Tự and Chữ Nôm.)
Modern Chinese, Japanese and Korean typeface
Typeface
In typography, a typeface is the artistic representation or interpretation of characters; it is the way the type looks. Each type is designed and there are thousands of different typefaces in existence, with new ones being developed constantly....
s typically use regional or historical variants of a given Han character
Variant Chinese character
Variant Chinese characters are Chinese characters that are homophones and synonyms. Almost all variants are allographs in most circumstances, such as casual handwriting...
. In the formulation of Unicode, an attempt was made to unify these variants by considering them different glyphs representing the same "grapheme", or orthographic
Orthography
The orthography of a language specifies a standardized way of using a specific writing system to write the language. Where more than one writing system is used for a language, for example Kurdish, Uyghur, Serbian or Inuktitut, there can be more than one orthography...
unit, hence, "Han unification", with the resulting character repertoire sometimes contracted to Unihan.
Unihan can also refer to the Unihan Database maintained by the Unicode Consortium, which provides information about all of the unified Han characters encoded in the Unicode standard, including mappings to various national and industry standards, indices into standard dictionaries, encoded variants, pronunciations in various languages, and an English definition. The database is available to the public as text files and via an interactive Web site. The latter also includes representative glyphs and definitions for compound words drawn from the free Japanese EDICT
EDICT
The JMdict/EDICT project was started by Jim Breen in 1991 with the aim to provide a machine-readable Japanese to English dictionary. Since that time it has been updated and expanded by many contributors. The dictionaries resulting from the project are simply text files; other programs are needed to...
and Chinese CEDICT
CEDICT
The CEDICT project was started by Paul Denisowski in 1997 and is presently maintained by MDBG, under the name CC-CEDICT, with the aim to provide a complete Chinese to English dictionary with pronunciation in pinyin for the Chinese characters.- Content :...
dictionary projects (which are provided for convenience and are not a formal part of the Unicode standard).
Rationale and controversy
Rules for Han unification are given in the East Asian Scripts chapter of the various versions of the Unicode Standard (Chapter 12 in Unicode 6.0). The Ideographic Rapporteur GroupIdeographic Rapporteur Group
The Ideographic Rapporteur Group advises the Unicode Consortium and the ISO/IEC JTC1/SC2/WG2 on Han character additions to the repertoire of the Unicode and ISO/IEC 10646-1 character set standards, and on Han unification...
(IRG), made up of experts from the Chinese-speaking countries, North and South Korea, Japan, Vietnam, and other countries, is responsible for the process.
One possible rationale is the desire to limit the size of the full Unicode character set, where CJK characters as represented by discrete ideograms may approach or exceed 100,000 (while those required for ordinary literacy in any language are probably under 3,000). Version 1 of Unicode was designed to fit into 16 bits and only 20,940 characters (32%) out of the possible 65,536 were reserved for these CJK Unified Ideographs
CJK Unified Ideographs
The Chinese, Japanese and Korean scripts share a common background. In the process called Han unification the common characters were identified, and named "CJK Unified Ideographs"...
. Later Unicode has been extended to 21 bits allowing many more CJK characters (75,960 are assigned, with room for more).
The secret life of Unicode article located on IBM DeveloperWorks attempts to illustrate part of the motivation for Han unification:
In fact, the three ideographs for "one" are encoded separately in Unicode, as they are not considered national variants. The first and second are used on financial instruments to prevent tampering (they may be considered variants), while the third is the common form in all three countries.
However, Han unification has also caused considerable controversy, particularly among the Japanese public, who, with the nation's literati, have a history of protesting the culling of historically and culturally significant variants. (See Kanji#Orthographic reform and lists of kanji. Today, the list of characters officially recognized for use in proper names continues to expand at a modest pace.)
Graphemes versus glyphs
A grapheme is the smallest abstract unit of meaning in a writing system. Any grapheme has many possible glyph expressions, but all are recognized as the same grapheme by those with reading and writing knowledge of a particular writing system. While Unicode typically assigns characters to code points to express the graphemes within a system of writing, the Unicode standard (section 3.4 D7) does caution:However, this refers to the fact that some graphemes are composed of several characters. So, for example, the character "a" (U+0061) combined with a circle above (U+030A) (i.e. å
Å
Å represents various sounds in several languages. Å is part of the alphabets used for the Alemannic and the Bavarian-Austrian dialects of German...
) might be understood by a user as a single grapheme while being composed of multiple Unicode abstract characters. In addition, Unicode also assigns some code points to a small number (other than for compatibility reasons) of formatting characters, whitespace characters, and other abstract characters that are not graphemes, but instead used to control the breaks between lines, words, graphemes and grapheme clusters. With the unified Han ideographs, the Unicode standard makes a departure from prior practices in assigning abstract characters not as graphemes, but according to the underlying meaning of the grapheme: what linguists sometimes call sememe
Sememe
A sememe is a semantic language unit of meaning, correlative to a morpheme.A sememe is a proposed unit of transmitted or intended meaning; it is atomic or indivisible. A sememe can be the meaning expressed by a morpheme, such as the English pluralizing morpheme -s, which carries the sememic...
s. This departure therefore is not simply explained by the oft quoted distinction between an abstract character and a glyph, but is more rooted in the difference between an abstract character assigned as a grapheme and an abstract character assigned as a sememe. In contrast, consider Unicode's unification of punctuation and diacritic
Diacritic
A diacritic is a glyph added to a letter, or basic glyph. The term derives from the Greek διακριτικός . Diacritic is both an adjective and a noun, whereas diacritical is only an adjective. Some diacritical marks, such as the acute and grave are often called accents...
s, where graphemes with widely different meanings (for example, an apostrophe and a single quotation mark) are unified because the graphemes are the same. For Unihan the characters are not unified by their appearance, but by their definition or meaning.
For a grapheme to be represented by various glyphs means that the grapheme has glyph variations that are usually determined by selecting one font or another or using glyph substitution features where multiple glyphs are included in a single font. Such glyph variations are considered by Unicode a feature of rich text protocols and not properly handled by the plain text goals of Unicode. However, when the change from one glyph to another constitutes a change from one grapheme to another—where a glyph cannot possibly still, for example, mean the same grapheme understood as the small letter "a"—Unicode separates those into separate code points. For Unihan the same thing is done whenever the abstract meaning changes, however rather than speaking of the abstract meaning of a grapheme (the letter "a"), the unification of Han ideographs assigns a new code point for each different meaning—even if that meaning is expressed by distinct graphemes in different languages. While a grapheme such as "ö" might mean something different in English (as used in the word "coördinated") than it does in German, it is still the same grapheme and can be easily unified so that English and German can share a common abstract Latin writing system (along with Latin itself).
To deal with the use of different graphemes for the same Unihan sememe, Unicode has relied on several mechanisms to deal with the issue: especially as it relates to rendering text. One has been to treat it as simply a font issue so that different fonts might be used to render Chinese, Japanese or Korean. Also font formats such as OpenType allow for the mapping of alternate glyphs according to language so that a text rendering system can look to the user's environmental settings to determine which glyph to use. The problem with these approaches is that they fail to meet the goals of Unicode to support multilingual text within the same document.
So rather than treat the issue as a rich text problem of glyph alternates, Unicode added the concept of variation selectors, first introduced in version 3.2 and supplemented in version 4.0. While variation selectors are treated as combining characters, they have no associated diacritic or mark. Instead, by combining with a base character, they signal the two character sequence selects a variation (typically in terms of grapheme, but also in terms of underlying meaning as in the case of a location name or other proper noun) of the base character. This then is not a selection of an alternate glyph, but the selection of a grapheme variation or a variation of the base abstract character. Such a two-character sequence however can be easily mapped to a separate single glyph in modern fonts. Since Unicode has assigned 256 separate variation selectors, it is capable of assigning 256 variations for any Han ideograph. Such variations can be specific to one language or another and enable the encoding of plain text that includes such grapheme variations.
Unihan "abstract characters"
Since the Unihan standard encodes "abstract characters", not "glyphs", the graphical artifacts produced by Unicode have been considered temporary technical hurdles, and at most, cosmetic. However, again, particularly in Japan, due in part to the way in which Chinese characters were incorporated into Japanese writing systems historically, the inability to specify a particular variant is considered a significant obstacle to the use of Unicode in scholarly work. For example, the unification of "grass" (explained above), means that a historical text cannot be encoded so as to preserve its peculiar orthography. Instead, for example, the scholar would be required to locate the desired glyph in a specific typeface in order to convey the text as written, defeating the purpose of a unified character set. Unicode has responded to these needs by assigning variation selectors so that authors can select grapheme variations of particular ideographs (or even other characters).Small differences in graphical representation are also problematic when they affect legibility or the wrong cultural tradition. Besides making some Unicode fonts unusable for texts involving multiple "Unihan languages", names or other orthographically sensitive terminology might be displayed incorrectly. (Proper names tend to be especially orthographically conservative—compare this to changing the spelling of one's name to suit a language reform in the U.S. or U.K.) While this may be considered primarily a graphical representation or rendering problem to be overcome by more artful fonts, the widespread use of Unicode would make it difficult to preserve such distinctions. The problem of one character representing semantically different concepts is also present in the Latin part of Unicode. The Unicode character for an apostrophe is the same as the character for a right single quote (’). On the other hand, it is sometimes pointed out that the capital Latin letter "A" is not unified with the Greek letter "Α" (Alpha). This is, of course, desirable for reasons of compatibility, and deals with a much smaller alphabetic character set.
While the unification aspect of Unicode is controversial in some quarters for the reasons given above, Unicode itself does now encode a vast number of seldom-used characters of a more-or-less antiquarian nature.
Some of the controversy stems from the fact that the very decision of performing Han unification was made by the initial Unicode Consortium, which at the time was a consortium of North American companies and organizations (most of them in California), but included no East Asia government representatives. The initial design goal was to create a 16-bit standard, and Han unification was therefore a critical step for avoiding tens of thousands of character duplications. This 16-bit requirement was later abandoned, making the size of the character set less an issue today.
The controversy later extended to the internationally representative ISO: the initial CJK-JRG group favored a proposal (DIS 10646) for a non-unified character set, "which was thrown out in favor of unification with the Unicode Consortium's unified character set by the votes of American and European ISO members" (even though the Japanese position was unclear). Endorsing the Unicode Han unification was a necessary step for the heated ISO 10646/Unicode merger.
Much of the controversy surrounding Han unification is based on the distinction between glyph
Glyph
A glyph is an element of writing: an individual mark on a written medium that contributes to the meaning of what is written. A glyph is made up of one or more graphemes....
s, as defined in Unicode, and the related but distinct idea of graphemes. Unicode assigns abstract characters (graphemes), as opposed to glyphs, which are a particular visual representations of a character in a specific typeface
Typeface
In typography, a typeface is the artistic representation or interpretation of characters; it is the way the type looks. Each type is designed and there are thousands of different typefaces in existence, with new ones being developed constantly....
. One character may be represented by many distinct glyphs, for example a "g" or an "a", both of which may have one loop (a, g) or two (a, g). Yet for a reader of Latin script based languages the two variations of the "a" character are both recognized as the same grapheme. Graphemes present in national character code standards have been added to Unicode, as required by Unicode's Source Separation rule, even where they can be composed of characters already available. The national character code standards existing in CJK languages are considerably more involved, given the technological limitations under which they evolved, and so the official CJK participants in Han unification may well have been amenable to reform.
Unlike European versions, CJK Unicode fonts, due to Han unification, have large but irregular patterns of overlap, requiring language-specific fonts. Unfortunately, language-specific fonts also make it difficult to access to a variant which, as with the "grass" example, happens to appear more typically in another language style. (That is to say, it would be difficult to access "grass" with the four-stroke radical more typical of Traditional Chinese in a Japanese environment, which fonts would typically depict the three-stroke radical.) Unihan proponents tend to favor markup languages for defining language strings, but this would not ensure the use of a specific variant in the case given, only the language-specific font more likely to depict a character as that variant. (At this point, merely stylistic differences do enter in, as a selection of Japanese and Chinese fonts are not likely to be visually compatible.)
Chinese users seem to have fewer objections to Han unification, largely because Unicode did not attempt to unify Simplified Chinese characters (an invention of the People's Republic of China
People's Republic of China
China , officially the People's Republic of China , is the most populous country in the world, with over 1.3 billion citizens. Located in East Asia, the country covers approximately 9.6 million square kilometres...
, and in use among Chinese speakers in the PRC, Singapore
Singapore
Singapore , officially the Republic of Singapore, is a Southeast Asian city-state off the southern tip of the Malay Peninsula, north of the equator. An island country made up of 63 islands, it is separated from Malaysia by the Straits of Johor to its north and from Indonesia's Riau Islands by the...
, and Malaysia), with Traditional Chinese characters, as used in Hong Kong, Taiwan (Big5
Big5
Big-5 or Big5 is a character encoding method used in Taiwan, Hong Kong, and Macau for Traditional Chinese characters.Mainland China, which uses Simplified Chinese Characters, uses the GB instead.- Organization :...
), and, with some differences, more familiar to Korean and Japanese users. Unicode is seen as neutral with regards to this politically charged issue, and has encoded Simplified and Traditional Chinese glyphs separately (e.g. the ideograph for "discard" is 丟 U+4E1F for Traditional Chinese Big5 #A5E1 and 丢 U+4E22 for Simplified Chinese GB #2210). It is also noted that Traditional and Simplified characters should be encoded separately according to Unicode Han Unification rules, because they are distinguished in pre-existing PRC character sets. Furthermore, as with other variants, Traditional to Simplified characters is not a one-to-one relationship.
Alternatives
Specialist character sets developed to address, or regarded by some as not suffering from, these perceived deficiencies include:- ISO/IEC 2022ISO/IEC 2022ISO/IEC 2022 Information technology—Character code structure and extension techniques, is an ISO standard specifying...
(based on sequence codes to switch between Chinese, Japanese, Korean character sets - hence without unification) - CNS character setCNS character setThe CNS 11643 character set , also officially known as the "Chinese Standard Interchange Code" , is officially the standard character set of the Republic of China....
- CCCII character set
- TRONTRON (encoding)TRON Code is a multi-byte character encoding used in the TRON Project. It is similar to Unicode but does not use Unicode's Han unification process: each character from each CJK character set is encoded separately, including archaic and historical equivalents of modern characters...
- UTF-2000First proposed in 1998. However, , adoption of this proposed counter-standard is nearly non-existent. There has been little definitive standardization process or documents on UTF-2000 except for some conference presentations in 2000 and 2001.
- MojikyoMojikyois a set of computer software and fonts for enhanced logogram word-processing. , it collected 126,560/142,228 characters . Among them, 101,936/128,573 characters belong to the extended CJKV family...
- Big5 extensions
- GCCS and its successor HKSCSHKSCSThe Hong Kong Supplementary Character Set is a set of Chinese characters -- 4,702 in total in the initial release—used in Cantonese, as well as when writing the names of some places in Hong Kong . It evolved from the preceding Government Chinese Character Set or GCCS...
However, none of these alternative standards has been as widely adopted as Unicode
Unicode
Unicode is a computing industry standard for the consistent encoding, representation and handling of text expressed in most of the world's writing systems...
, which is now the base character set for many new standards and protocols, and is built into the architecture of operating systems (Microsoft Windows
Microsoft Windows
Microsoft Windows is a series of operating systems produced by Microsoft.Microsoft introduced an operating environment named Windows on November 20, 1985 as an add-on to MS-DOS in response to the growing interest in graphical user interfaces . Microsoft Windows came to dominate the world's personal...
, Apple
Apple Computer
Apple Inc. is an American multinational corporation that designs and markets consumer electronics, computer software, and personal computers. The company's best-known hardware products include the Macintosh line of computers, the iPod, the iPhone and the iPad...
Mac OS X
Mac OS X
Mac OS X is a series of Unix-based operating systems and graphical user interfaces developed, marketed, and sold by Apple Inc. Since 2002, has been included with all new Macintosh computer systems...
, and many Unix-like
Unix-like
A Unix-like operating system is one that behaves in a manner similar to a Unix system, while not necessarily conforming to or being certified to any version of the Single UNIX Specification....
systems), programming languages (Perl
Perl
Perl is a high-level, general-purpose, interpreted, dynamic programming language. Perl was originally developed by Larry Wall in 1987 as a general-purpose Unix scripting language to make report processing easier. Since then, it has undergone many changes and revisions and become widely popular...
, Python, C#, Java
Java (programming language)
Java is a programming language originally developed by James Gosling at Sun Microsystems and released in 1995 as a core component of Sun Microsystems' Java platform. The language derives much of its syntax from C and C++ but has a simpler object model and fewer low-level facilities...
, Common LISP
Common Lisp
Common Lisp, commonly abbreviated CL, is a dialect of the Lisp programming language, published in ANSI standard document ANSI INCITS 226-1994 , . From the ANSI Common Lisp standard the Common Lisp HyperSpec has been derived for use with web browsers...
, APL
APL programming language
APL is an interactive array-oriented language and integrated development environment, which is available from a number of commercial and noncommercial vendors and for most computer platforms. It is based on a mathematical notation developed by Kenneth E...
), and libraries (IBM International Components for Unicode
International Components for Unicode
International Components for Unicode is an open source project of mature C/C++ and Java libraries for Unicode support, software internationalization and software globalization. ICU is widely portable to many operating systems and environments. It gives applications the same results on all...
(ICU) along with the Pango
Pango
Pango is an LGPL licensed open source computing library used by software developers for laying out and rendering text in high quality, emphasising support for multilingual text...
, Graphite
Graphite (SIL)
Graphite is a programmable Unicode-compliant smart-font technology and rendering system developed by SIL International. It is free software, distributed under the terms of the GNU Lesser General Public License and the Common Public License....
, Scribe
Qt (toolkit)
Qt is a cross-platform application framework that is widely used for developing application software with a graphical user interface , and also used for developing non-GUI programs such as command-line tools and consoles for servers...
, Uniscribe
Uniscribe
Uniscribe is the Microsoft Windows set of services for rendering Unicode-encoded text, especially complex text layout. They are implemented in the DLL USP10.DLL. USP10.dll became available to the public with Windows 2000 and Internet Explorer 5.0...
, and ATSUI
Apple Type Services for Unicode Imaging
The Apple Type Services for Unicode Imaging is the set of services for rendering Unicode-encoded text starting with Mac OS 8.5 and in Mac OS X.It replaced the WorldScript engine for legacy encodings....
rendering engines), font formats (TrueType
TrueType
TrueType is an outline font standard originally developed by Apple Computer in the late 1980s as a competitor to Adobe's Type 1 fonts used in PostScript...
and OpenType
OpenType
OpenType is a format for scalable computer fonts. It was built on its predecessor TrueType, retaining TrueType's basic structure and adding many intricate data structures for prescribing typographic behavior...
) and so on.
Examples of language dependent characters
In each row of the following table, the same character is repeated in all five columns. However, each column is marked (via the HTML lang attribute) as being in a different language: ChineseChinese language
The Chinese language is a language or language family consisting of varieties which are mutually intelligible to varying degrees. Originally the indigenous languages spoken by the Han Chinese in China, it forms one of the branches of Sino-Tibetan family of languages...
(3 varieties: unmarked "Chinese", simplified characters
Simplified Chinese character
Simplified Chinese characters are standardized Chinese characters prescribed in the Xiandai Hanyu Tongyong Zibiao for use in Mainland China. Along with traditional Chinese characters, it is one of many standard character sets of the contemporary Chinese written language...
, and traditional characters
Traditional Chinese character
Traditional Chinese characters refers to Chinese characters in any character set which does not contain newly created characters or character substitutions performed after 1946. It most commonly refers to characters in the standardized character sets of Taiwan, of Hong Kong, or in the Kangxi...
), Japanese
Japanese language
is a language spoken by over 130 million people in Japan and in Japanese emigrant communities. It is a member of the Japonic language family, which has a number of proposed relationships with other languages, none of which has gained wide acceptance among historical linguists .Japanese is an...
, or Korean
Korean language
Korean is the official language of the country Korea, in both South and North. It is also one of the two official languages in the Yanbian Korean Autonomous Prefecture in People's Republic of China. There are about 78 million Korean speakers worldwide. In the 15th century, a national writing...
. The browser
Web browser
A web browser is a software application for retrieving, presenting, and traversing information resources on the World Wide Web. An information resource is identified by a Uniform Resource Identifier and may be a web page, image, video, or other piece of content...
should select, for each character, a glyph
Glyph
A glyph is an element of writing: an individual mark on a written medium that contributes to the meaning of what is written. A glyph is made up of one or more graphemes....
(from a font
Typeface
In typography, a typeface is the artistic representation or interpretation of characters; it is the way the type looks. Each type is designed and there are thousands of different typefaces in existence, with new ones being developed constantly....
) suitable to the specified language. (Besides actual character variation—look for differences in stroke order, number, or direction—the typefaces may also reflect different typographical styles, as with serif and non-serif alphabets.) This only works for fallback glyph selection if you have CJK fonts installed on your system and the font selected to display this article does not include glyphs for these characters. Note also that Unicode includes non-graphical language tag characters in the range U+E0000 – U+E007F for plain text language tagging.
| Code | Chinese (Generic) |
Chinese Simplified |
Chinese Traditional |
Japanese | Korean |
|---|---|---|---|---|---|
| 与 | 与 | 与 | 与 | 与 | |
| U+4ECA | 今 | 今 | 今 | 今 | 今 |
| U+4EE4 | 令 | 令 | 令 | 令 | 令 |
| U+514D | 免 | 免 | 免 | 免 | 免 |
| U+5165 | 入 | 入 | 入 | 入 | 入 |
| U+5168 | 全 | 全 | 全 | 全 | 全 |
| U+5177 | 具 | 具 | 具 | 具 | 具 |
| U+5203 | 刃 | 刃 | 刃 | 刃 | 刃 |
| U+5316 | 化 | 化 | 化 | 化 | 化 |
| U+5340 | 區 | 區 | 區 | 區 | 區 |
| U+5916 | 外 | 外 | 外 | 外 | 外 |
| U+60C5 | 情 | 情 | 情 | 情 | 情 |
| U+624D | 才 | 才 | 才 | 才 | 才 |
| U+6B21 | 次 | 次 | 次 | 次 | 次 |
| U+6D77 | 海 | 海 | 海 | 海 | 海 |
| U+6F22 | 漢 | 漢 | 漢 | 漢 | 漢 |
| U+753B | 画 | 画 | 画 | 画 | 画 |
| U+76F4 | 直 | 直 | 直 | 直 | 直 |
| U+771F | 真 | 真 | 真 | 真 | 真 |
| U+7A7A | 空 | 空 | 空 | 空 | 空 |
| U+7D00 | 紀 | 紀 | 紀 | 紀 | 紀 |
| U+8349 | 草 | 草 | 草 | 草 | 草 |
| U+89D2 | 角 | 角 | 角 | 角 | 角 |
| U+8ACB | 請 | 請 | 請 | 請 | 請 |
| U+9053 | 道 | 道 | 道 | 道 | 道 |
| U+9913 | 餓 | 餓 | 餓 | 餓 | 餓 |
| U+9AA8 | 骨 | 骨 | 骨 | 骨 | 骨 |
Examples of some non-unified Han ideographs
For some glyphs, Unicode has encoded variant characters, making it unnecessary to switch between fonts or language tags. In the following table, the separate rows in each group contains the Unicode equivalent character using different code points. Note that for characters such as 入 (U+5165), the only way to display the two variants is to change font (or language tag) as described in the previous table. However, for 內 (U+5167), there is an alternate character 内 (U+5185) as illustrated below. For some characters, like 兌/兑 (U+514C/U+5151), either method can be used to display the different glyphs.| Code | Chinese (Generic) |
Chinese Simplified |
Chinese Traditional |
Japanese | Korean |
|---|---|---|---|---|---|
| U+9AD8 | 高 | 高 | 高 | 高 | 高 |
| U+9AD9 | 髙 | 髙 | 髙 | 髙 | 髙 |
| U+7D05 | 紅 | 紅 | 紅 | 紅 | 紅 |
| U+7EA2 | 红 | 红 | 红 | 红 | 红 |
| U+4E1F | 丟 | 丟 | 丟 | 丟 | 丟 |
| U+4E22 | 丢 | 丢 | 丢 | 丢 | 丢 |
| U+4E57 | 乗 | 乗 | 乗 | 乗 | 乗 |
| U+4E58 | 乘 | 乘 | 乘 | 乘 | 乘 |
| U+4FA3 | 侣 | 侣 | 侣 | 侣 | 侣 |
| U+4FB6 | 侶 | 侶 | 侶 | 侶 | 侶 |
| U+514C | 兌 | 兌 | 兌 | 兌 | 兌 |
| U+5151 | 兑 | 兑 | 兑 | 兑 | 兑 |
| U+5167 | 內 | 內 | 內 | 內 | 內 |
| U+5185 | 内 | 内 | 内 | 内 | 内 |
| U+7522 | 產 | 產 | 產 | 產 | 產 |
| U+7523 | 産 | 産 | 産 | 産 | 産 |
| U+7A05 | 稅 | 稅 | 稅 | 稅 | 稅 |
| U+7A0E | 税 | 税 | 税 | 税 | 税 |
| U+4E80 | 亀 | 亀 | 亀 | 亀 | 亀 |
| U+9F9C | 龜 | 龜 | 龜 | 龜 | 龜 |
| U+9F9F | 龟 | 龟 | 龟 | 龟 | 龟 |
| U+5225 | 別 | 別 | 別 | 別 | 別 |
| U+522B | 别 | 别 | 别 | 别 | 别 |
| U+4E21 | 両 | 両 | 両 | 両 | 両 |
| U+4E24 | 两 | 两 | 两 | 两 | 两 |
| U+5169 | 兩 | 兩 | 兩 | 兩 | 兩 |
Unicode ranges
Ideographic characters assigned by Unicode appear in the following blocks:- CJK Unified Ideographs (4E00–9FFF)
- CJK Unified Ideographs Extension A (3400–4DBF)
- CJK Unified Ideographs Extension B (20000–2A6DF)
- CJK Unified Ideographs Extension C (2A700–2B73F)
- CJK Unified Ideographs Extension D (2B840–2B81F)
- CJK Compatibility Ideographs (F900–FAFF) (the twelve characters at FA0E, FA0F, FA11, FA13, FA14, FA1F, FA21, FA23, FA24, FA27, FA28 and FA29 are actually "unified ideographs" not "compatibility ideographs")
Unicode includes support of CJKV radicals, strokes, punctuation, marks and symbols in the following blocks:
- CJK Radicals Supplement (2E80–2EFF)
- CJK Symbols and Punctuation (3000–303F) (chart)
- CJK Strokes (31C0–31EF)
- Ideographic Description Characters (2FF0–2FFF)
Additional compatibility (discouraged use) characters appear in these blocks:
- Kangxi Radicals (2F00–2FDF)
- Enclosed CJK Letters and Months (3200–32FF) (chart)
- CJK Compatibility (3300–33FF) (chart)
- CJK Compatibility Ideographs (F900–FAFF) (chart)
- CJK Compatibility Ideographs (2F800–2FA1F)
- CJK Compatibility Forms (FE30–FE4F) (chart)
These compatibility characters (excluding the twelve unified ideographs in the CJK Compatibility Ideographs block) are included for compatibility with legacy text handling systems and other legacy character sets. They include forms of characters for vertical text layout and rich text characters that Unicode recommends handling through other means.
International Ideographs Core
International Ideographs Core (IICore) is an subset of 9810 ideographs derived from the CJK Unified Ideographs tables, designed to be implemented in devices with limited memory, input/output capability, and/or applications where the use of complete ISO 10646 ideographs repertoire is not feasible. There are 9810 characters in current standard.Unihan database files
The Unihan project has always made an effort to make available their build database.An Unihan.zip file is provided on unicode.org. It contains all the data the Unihan team have collected.
A project libUnihan (0.5.3) provides a normalized SQLite Unihan database and corresponding C library. All tables in this database are in fifth normal form.
libUnihan is released as LGPL, while its database, UnihanDb, is released as MIT License
MIT License
The MIT License is a free software license originating at the Massachusetts Institute of Technology . It is a permissive license, meaning that it permits reuse within proprietary software provided all copies of the licensed software include a copy of the MIT License terms...
.
See also
- Chinese character encodingChinese character encodingIn computing, Chinese character encodings can be used to represent text written in the CJK languages — Chinese, Japanese, Korean — and obsolete Vietnamese, all of which use Chinese characters...
- GB 18030GB 18030GB18030 is a Chinese government standard describing the required language and character support necessary for software in China. In addition to the "GB18030 code page" this standard contains requirements about which scripts must be supported, font support, etc....
- SinicizationSinicizationSinicization, Sinicisation or Sinification, is the linguistic assimilation or cultural assimilation of terms and concepts of the language and culture of China...
- Z-variantZ-variantIn Unicode, two glyphs are said to be Z-variants if they share the same etymology but have slightly different appearances and different Unicode codepoints. For example, the Unicode characters U+8AAA 說 and U+8AAC 説 are Z-variants...
- List of CJK fonts
External links
- Unihan Database
- Unicode standard
- IRG Page
- IRG working documents – many big size pdfs, some of them with details of CJK extensions
- Han Unification in Unicode by Otfried Cheong
- Why Unicode Won't Work on the Internet: Linguistic, Political, and Technical Limitations
- Why Unicode Will Work On The Internet
- Per-character summary of differences in characters
- The secret life of Unicode
- GB18030 Support Package for Windows 2000/XP, including Chinese, Tibetan, Yi, Mongolian and Thai font by Microsoft
- Proposal to encode additional grass radicals in the UCS – A humorous proposal to encode all possible variants of the grass radical, made as an April Fool's Day joke
- Unicode Technical Note 26: On the Encoding of Latin, Greek, Cyrillic, and Han
- "Unicode Revisited" – the strong point of view of some people working on the competing TRON proposal
- "Unicode in Japan, guide to a technical and psychological struggle" – A more balanced take on the arguments for and against Unicode for Japanese.