Name: 0README.TXT
Background information - mapping tables for the Mac(TM) OS
Version 6: Nov. 15, 1995 - update info for Hebrew and Thai
(Version 5: Apr. 15, 1995)
Peter Edberg, Apple Computer, Inc. <edberg1@applelink.apple.com>
Copyright (C) 1995 by Apple Computer, Inc., all rights reserved.
0. Preliminaries
----------------
For maximum interchangeability, this file and the accompanying MacOS
mapping tables use only ASCII characters and are intended to be
displayed in a monospaced font. Every line terminates with carriage
return.
Apple, the Apple logo, Mac, and Macintosh are trademarks of Apple
Computer, Inc., registered in the United States and other countries.
QuickDraw and TrueType are trademarks of Apple Computer, Inc. Unicode is
a trademark of Unicode Inc. PostScript is a trademark of Adobe Systems
Inc., which may be registered in certain jurisdictions. IBM is a
registered trademark of International Business Machines Corporation. ITC
Zapf Dingbats is a registered trademark of the International Typeface
Corporation. For the sake of brevity, throughout this document and the
accompanying tables, "Unicode" can be used to refer to the Unicode
standard.
Apple Computer, Inc. ("Apple") makes no warranty or representation,
either express or implied, with respect to this document and the
accompanying tables, their quality, accuracy, or fitness for a
particular purpose. In no event will Apple be liable for direct,
indirect, special, incidental, or consequential damages resulting from
any defect or inaccuracy in this document or the accompanying tables.
1. Introduction
---------------
In order to understand the accompanying MacOS mapping tables, you will
need to understand something about the MacOS Unicode Converter. This
converter has been designed to handle the complex issues that can arise
when converting between Unicode and other character sets, including:
* Round-trip fidelity (and the use of corporate characters);
* Supporting various mapping tolerance levels (strict, loose), in order
to provide both round-trip fidelity and a way to handle characters
that have multiple or ambiguous semantics in some character sets;
* Handling character set variants and extensions, which may require
selective inclusion or exclusion of certain mappings or sets of
mappings;
* Mapping single characters in one set to multiple characters in another
or vice versa (in general, a match may map a sequence of 1 to n
characters in one set to a sequence of 0 to m characters in another);
* Handling mappings that may depend on attributes such as resolved
character direction, vertical or horizontal display direction, etc.
The above issues are described in more detail in sections 2-6. Section 7
provides some general information on MacOS character sets and a list of
MacOS character encodings.
This document and all of the accompanying mapping tables and character
lists are preliminary and subject to change. Updated documents and
tables will be available from the Unicode Inc. ftp site (unicode.org),
the Apple ftp site (ftp.info.apple.com), the Apple Computer World-Wide
Web pages (http://www.info.apple.com), and possibly on diskette from
APDA (Apple's mail-order distribution service for developers).
2. Round-trip fidelity and corporate characters
-----------------------------------------------
For the various national and international standards that were sources
for Unicode, Unicode provides round-trip fidelity: Text in one of those
encodings can be mapped to Unicode and back again with no loss of
information. Characters which were distinct in the source standard are
distinct in Unicode.
However, Unicode does not attempt to provide round-trip fidelity for
most vendor standards. Nevertheless, Apple and other platform vendors
may need to provide such round-trip fidelity for their current encodings
(this can be important in file systems, for example). In order to do
this, Apple maps some characters in the current MacOS encodings to
character codes at the upper end of the Unicode private use area (i.e.
the corporate use zone). In general, these are characters that are
rarely used in text that is interchanged with other systems, or
characters for which mistranslation in interchange would have a minimal
impact on most documents. Apple's usage of character codes in the
corporate use zone is documented in the accompanying file
"MacOS_CorpChars".
There is another round-trip fidelity issue that is important for the
MacOS Unicode converter. Among other things, this converter will be used
to convert between two non-Unicode encodings by using Unicode as an
intermediate form. For example, characters in the MacOS standard Roman
encoding could be converted to ISO/IEC 8859-1 by converting them first
to Unicode, and then converting the Unicode text to ISO/IEC 8859-1.
However, not all MacOS standard Roman characters can be represented in a
distinct way in ISO/IEC 8859-1. In such cases it is useful to know the
subset of MacOS Roman characters that can be converted to 8859-1 and
back (via Unicode) with no loss of information.
3. Mapping tolerance: Strict and loose
--------------------------------------
In many character sets, a single character may have multiple semantics,
either by explicit definition, ambiguous definition, or established
usage. For example, the JIS character 0x2142 (Shift-JIS 0x8161) is
specified in the JIS X0208 standard to have two meanings: "double
vertical line" and "parallel". Each of these meanings corresponds to a
different Unicode character: 0x2016 "DOUBLE VERTICAL LINE" and 0x2225
"PARALLEL TO". When mapping from Unicode to JIS, it is normally
desirable to map both of these Unicode characters to the single JIS
character 0x2142. However, when mapping this JIS character to Unicode,
we can choose only one of the possible Unicode characters.
For some character set X, the converse of the X-to-Unicode mappings are
called "strict" mappings from Unicode to X. In general, strict mappings
permit roundtrip conversion from Unicode to X and back for a subset of
Unicode characters. Strict mappings are useful when round-trip fidelity
is desired for an X-to-Unicode-to-Y mapping.
For some characters in X, there may be additional mappings from Unicode
that fall within the range of explicit or established usage for those
characters; these are called "loose" mappings. It is important to note
that the range of allowed loose mappings is determined by the character
set X.
Furthermore, in some cases it is helpful to map a Unicode character to a
sequence of one or more target characters that may not have the same
meaning or use, but which may provide an approximate graphic
representation of the corresponding Unicode character. These are called
"fallback" mappings.
Some examples of strict and loose mappings:
a) In the JIS example above, JIS 0x2142 is usually mapped to Unicode
0x2016 "DOUBLE VERTICAL LINE". Thus the reverse mapping is a strict
mapping from Unicode to JIS, while mapping Unicode 0x2225 "PARALLEL TO"
to JIS 0x2142 is a loose mapping.
b) When mapping ASCII to Unicode, 0x0A "line feed" and 0x0D "carriage
return" are usually mapped to the Unicode code points 0x000A and 0x000D.
When mapping Unicode to ASCII, loose mappings could include mapping
0x2028 "LINE SEPARATOR" to 0x0A and mapping 0x2029 "PARAGRAPH SEPARATOR"
to 0x0D.
c) Other loose mappings from Unicode to ASCII might include mapping
Unicodes 0x2010 "HYPHEN" and 0x2212 "MINUS SIGN" to ASCII 0x2D "hyphen-
minus".
d) In the conventional mapping from ISO/IEC 8859-1 to Unicode, the
8859-1 character 0xE0 "small letter a with grave accent" is mapped to
Unicode 0x00E0 "LATIN SMALL LETTER A WITH GRAVE", so the reverse mapping
is a strict mapping from Unicode to 8859-1. However, the two-character
Unicode sequence 0x0061+0x0300 ("LATIN SMALL LETTER A" + "COMBINING
GRAVE ACCENT") can also be mapped to 8859-1 0xE0 as a loose mapping.
e) Since Shift-JIS distinguishes halfwidth and fullwidth characters,
loose mappings for Shift-JIS must also keep these distinct. For example,
Shift-JIS 0x814D (JIS 0x212E) "grave accent [fullwidth]" is often mapped
to Unicode 0xFF40, "FULLWIDTH GRAVE ACCENT", and the reverse is a strict
mapping. In this case the Unicode sequence 0x3000+0x0300 "IDEOGRAPHIC
SPACE" + "COMBINING GRAVE ACCENT" can also be mapped to Shift-JIS 0x814D
as a loose mapping. However, the Unicode sequence 0x0020+0x0300 "SPACE"
+ "COMBINING GRAVE ACCENT" should not be mapped to Shift-JIS 0x814D as a
loose mapping, although this sequence could be mapped to Shift-JIS 0x60
"grave accent [halfwidth]" as a loose mapping.
Although the MacOS Unicode converter (and its tables) supports strict,
loose, and fallback mappings, the MacOS character mapping tables
accompanying this document provide only the strict mappings.
4. Character set variants and extensions
----------------------------------------
An example illustrates this issue:
The MacOS standard Japanese character set is based on Shift-JIS with
some additional characters. The additions include:
(a) For one-byte characters, five additions and one modification.
(b) Separately-encoded vertical forms for some punctuation and kana
characters from JIS rows 1, 4, and 5. These vertical forms are in
JIS rows 85, 88, and 89.
(c) Apple extension characters, in JIS rows 9-15.
However, in older versions of the Japanese system, some of the fonts
were based on a different encoding which did not include the Apple
extension characters and which encoded vertical forms in JIS rows 11,
14, and 15. Furthermore, PostScript fonts use a different set of
extensions in rows 9-15.
With the MacOS Unicode converter, several variants can be specified for
the MacOS Japanese character set:
* The standard set, with extensions and vertical forms;
* A reduced version of the standard set, without the separately encoded
vertical forms;
* An alternate set that corresponds to the old font variant;
* An alternate set that corresponds to the PostScript variant;
* A basic "least common denominator" set that works with all the old and
new fonts.
The MacOS Japanese character set mappings provided in the accompanying
tables cover only the standard character set, but they are grouped into
three sections: the basic set, the Apple extensions, and the vertical
forms.
5. Mappings that are not one-to-one
-----------------------------------
In some cases, a character in a non-Unicode character set may map to a
sequence of characters in Unicode. To handle the reverse mapping, the
MacOS Unicode converter can break a Unicode stream into appropriate text
elements (which may consist of more than one Unicode character) and can
look up multi-character Unicode sequences.
For example, the Apple extensions in the MacOS standard Japanese
character set include a character for the circled CJK ideograph for
"big". Although Unicode encodes other circled ideographs as single
characters, it does not encode this one. However, this character can be
represented in Unicode as the Unicode sequence 0x5927+0x20DD, the CJK
ideograph for "big" followed by COMBINING ENCLOSING CIRCLE.
In addition, a single Unicode character (or a multi-character Unicode
sequence) may map to a sequence of multiple characters in another
encoding. For example, the Unicode character 0x00BD "VULGAR FRACTION ONE
HALF" cannot be mapped into the MacOS standard Roman character set as a
single character, but it can be mapped to the sequence 0x31+0xDA+0x32,
"digit one" + "fraction slash" + "digit two" (normally this would be a
loose mapping).
Finally, some Unicode characters may be silently consumed when mapping
to some other encodings. For example, when mapping from Unicode to the
MacOS Arabic character set, resolved direction is used to disambiguate
some mappings (this is discussed in the next section). Direction
override characters (Unicodes 0x202C-0x202E) may be used to control the
resolved direction to achieve proper results. Having fulfilled this
role, the direction override characters can then be discarded. They are
included among the Unicode characters that can be represented in the
MacOS Arabic set (they are represented by the direction inherent in
certain characters), but there is no specific output character that
corresponds to them.
The accompanying mapping tables for the MacOS Japanese character set and
the MacOS Arabic character set include one-to-many mappings.
6. Mappings that depend on attributes
-------------------------------------
Mappings from Unicode to other character sets may depend on attributes
such as resolved character direction, the state of symmetric swapping,
and whether the text should use vertical form codes if available (i.e.
whether the text is intended for vertical display on a system that
cannot automatically substitute vertical forms).
a) Resolved character direction
The MacOS Arabic character set was developed in 1986-1987. At that time
the bidirectional line layout algorithm used in the MacOS was fairly
simple; it used only a few direction classes (instead of the 13 or so
now used in the Unicode bidirectional algorithm). In order to permit
users to handle some tricky layout problems, certain punctuation and
symbol characters have duplicate code points, one with a left-right
direction attribute and the other with a right-left direction attribute.
For example, ampersand is encoded at 0x26 with a left-right attribute,
and at 0xA6 with a right-left attribute. However, there is only one
ampersand character in Unicode. We need to have a way to map both of the
MacOS Arabic ampersand characters to Unicode and back again without loss
of information. Mapping one of the MacOS Arabic ampersand characters to
a code in the Unicode corporate use zone is undesirable, since both of
the ampersand characters are likely to be used in text that is
interchanged with other systems.
The problem is solved with the use of direction override characters and
direction-dependent mappings. When mapping from the MacOS Arabic
character set to Unicode, such problem characters are surrounded with an
appropriate direction override:
MacOS Arabic 0x26 ampersand (left)
-> Unicode 0x202D (LRO) + 0x0026 (AMPERSAND) + 0x202C (PDF)
MacOS Arabic 0xA6 ampersand (right)
-> Unicode 0x202E (RLO) + 0x0026 (AMPERSAND) + 0x202C (PDF)
The mappings from Unicode to MacOS Arabic can be disambiguated by the
use of resolved direction:
Unicode 0x0026 -> MacOS Arabic 0x26 (if L) or 0xA6 (if R)
Direction overrides are also used for some other purposes in mapping
MacOS Arabic characters to Unicode. For example, the single MacOS Arabic
ellipsis character has direction class right-left, while the Unicode
HORIZONTAL ELLIPSIS character has direction class neutral. When mapping
the MacOS ellipsis to Unicode, it is surrounded with a direction
override to help preserve proper text layout. However, resolved
direction is not needed or used when mapping the Unicode HORIZONTAL
ELLIPSIS back to MacOS Arabic.
b) Symmetric swapping
In loose mappings from Unicode to the MacOS Arabic character set, the
state of symmetric swapping (which may be changed by the Unicode
characters 0x206A, 0x206B) affects the mapping of paired characters such
as punctuation and brackets. This does not affect the strict mappings
given in the accompanying tables.
c) Horizontal or vertical display
As noted above, the MacOS standard Japanese character set (for
historical reasons) includes separately-encoded vertical forms for some
punctuation and kana. When Unicode characters in the CJK punctuation and
kana ranges are mapped to MacOS Japanese characters and (1) those
characters are intended for vertical display, (2) they will be displayed
in an environment that does not provide automatic vertical form
substitution, and (3) loose mappings are being used, a vertical display
attribute can be used to map certain Unicode characters to the
corresponding vertical form codes in the MacOS Japanese character set.
Note that this capability is only used for loose mappings, and does not
affect the strict mappings given in the accompanying tables. Also note
that this does not affect mapping of the Unicode vertical presentation
forms (which always map to the MacOS Japanese vertical form codes if
those codes are available in the specified variant). Finally, note that
the QuickDraw(TM) GX display environment does provide automatic vertical
forms substitution with appropriate fonts.
7. MacOS character sets
---------------------------
The MacOS can support multiple character sets. In the current MacOS
architecture these character sets are distinguished primarily by script
code: font family IDs are grouped into ranges, and each range is
associated with a script code.
In some cases, there are several variant encodings that share a single
script code. Usually these are minor variants. To distinguish these
variants, additional information is required, such as font name or
system localization code.
The encodings described here (and in the accompanying tables) are the
encodings used in MacOS versions 7.1 and later. In some cases, certain
earlier system versions have used variants of these encodings.
In all MacOS encodings, character codes 0x00-0x7F are identical to ASCII
(except for MacOS Japanese, which changes reverse solidus to yen sign).
Fonts used as "system" fonts (for menus, dialogs, etc.) have four glyphs
at code points 0x11-0x14 for transient use by the Menu Manager. These
glyphs are not intended as characters for use in normal text, and the
associated code points are not generally interpreted as associated with
these glyphs. (However, a "system font variant" mapping table could
provide mappings for these).
Note that in general, character sets cannot be determined from font
layouts (they are not the same thing!). This is most noticeable with
Arabic, Hebrew, and Devanagari.
The following is a list of current MacOS character sets. The
accompanying tables provide mappings from many of these encodings to
Unicode.
a) MacOS encodings for script code 0, smRoman.
* Standard Roman - this is the default for script code 0 (when the
special cases listed below do not apply). It covers several western
European languages, and includes math operators and various symbols.
* Symbol - this is the encoding for the font named "Symbol". It includes
Greek letters, math operators, and miscellaneous symbols. The layout
of the Symbol character set is identical to the layout of the Adobe
Symbol encoding vector, with the addition of the Apple logo at 0xF0.
The Symbol character set encodes some glyph fragments (of arrows,
brackets, etc.) as well as both serif and sans-serif forms for
copyright, registered, and trade mark sign; round-trip mapping of
these characters requires the use of corporate characters.
* Dingbats - this is the encoding for the font named "Zapf Dingbats".
The layout of the Dingbats character set is identical to or a superset
of the layout of the Adobe Zapf Dingbats encoding vector.
* Turkish - this is the encoding if the script code is 0 and the system
region code (system localization) is 24, verTurkey. It has 7 code
point differences from standard Roman.
* Croatian - this is the encoding if the script code is 0 and the system
region code is 68, verCroatia (or 25, verYugoCroatian, only used in
older systems). It has 20 code point differences from standard Roman,
but only 10 differences in repertoire.
* Icelandic - this is the encoding if the script code is 0 and the
system region code is 21, verIceland. It has 6 code point differences
from standard Roman.
* Romanian - this is the encoding if the script code is 0 and the system
region code is 39, verRomania . It has 6 code point differences from
standard Roman.
* Standard Greek (monotonic) - this is the encoding if the script code
is 0 and the system region code is 20, verGreece. Although a script
code is defined for Greek, the Greek localized system does not use it
(the font family IDs are in the smRoman range). This encoding is based
on the ISO/IEC 8859-7 repertoire with additional Roman characters for
French and German, as well as additional symbols.
Greek system 4.1 used a different encoding that matched 8859-7 code
points for Greek letters. Greek system 6.0.7 also used a variant of
the standard encoding, but it was quickly replaced by Greek system
6.0.7.1 which used the standard encoding.
NOTE- The Greek Language Kit, when released, will use the Greek script
code (its Greek fonts will have family IDs in the smGreek range); see
notes under script code 6 below.
See also the Central European Roman encoding under script code 29
below.
b) MacOS encodings for script code 1, smJapanese.
* Standard Japanese - this is the default for script code 1. As
described above, it is based on a Shift-JIS implementation of JIS
X0208-1990 ("fullwidth") and JIS X0201-1976 ("halfwidth"), with 5
additional one-byte characters and one modified character, a set of
Apple extension characters which include many industry standard
extensions, and separate codes for vertical forms of some punctuation
and kana.
There are two variants of standard Japanese associated with specific
fonts: (1) For MaruGothic and HonMincho TrueType fonts in system
software release J-7.1 and Japanese Language Kit 1.0, and (2) for the
PostScript fonts Gothic BBB and Ryumin, which are used with the screen
fonts ChuGothic and SaiMincho. Although they are supported by the
MacOS Unicode converter, these variants are not documented here or
in the accompanying tables. The MacOS Unicode converter also
supports some artificial variants which are just subsets of the
standard Japanese encoding.
c) MacOS encodings for script code 2, smTradChinese.
* Standard Traditional Chinese - this is an extension of Big-5.
d) MacOS encodings for script code 3, smKorean.
* Standard Korean - this is a "shifted" implementation of KSC 5601-1987
(0xA0 is added to the row and to the column), with some additional
characters.
e) MacOS encodings for script code 4, smArabic.
* Standard Arabic - This is based on the ISO/IEC 8859-6 repertoire, with
additional Arabic letters for Persian and Urdu and with additional
Roman letters for European languages. It has the interesting feature
mentioned above that certain ASCII punctuation and symbol characters
are encoded twice, once for each direction. Digit character codes
0x30-0x39 have left-to-right directionality, and may be displayed with
either European digit forms or Arabic digit forms depending on
context. Digit codes 0xB0-0xB9 have right-left directionality and are
always displayed with Arabic digit forms; these are used for special
layout situations such as part numbers.
f) MacOS encodings for script code 5, smHebrew.
* Standard Hebrew - This is based on the ISO/IEC 8859-8 Hebrew letter
repertoire, but adds Hebrew points, some Hebrew ligatures, some
additional Roman letters for European languages, and some non-ASCII
punctuation. As with standard Arabic, certain ASCII punctuation and
symbol characters are encoded twice, once for each direction. This
is also true for the European digits.
There is one minor variant of standard Hebrew associated with
certain fonts, in which LEFT SINGLE QUOTATION MARK at 0xD4 is
replaced by FIGURE SPACE.
g) MacOS encodings for script code 6, smGreek.
This script code will refer to the encoding used with the Greek
Language Kit, when released. It will either be the standard Greek
encoding described above, or a variant that supports polytonic Greek.
h) MacOS encodings for script code 7, smCyrillic.
* Standard Cyrillic - this is the default for script code 7 (when the
special cases listed below do not apply). It is based on the ISO/IEC
8859-5 Cyrillic character repertoire.
* Ukrainian - this is the encoding if the script code is 7 and the
system region code (system localization) is 62, verUkraine. It has 2
code point differences from standard Cyrillic (it adds a case pair
for GHE WITH UPTURN).
* Bulgarian -
An additional Cyrillic variant has been defined to cover the Cyrillic
characters needed for the languages of the central Asian republics
(plus Russian): Uzbek, Kazakh, Kirghiz, Azerbaijani, Turkmen, Tajik).
i) MacOS encodings for script code 9, smDevanagari.
* Standard Devanagari - This is an extension of IS 13194:1991 (ISCII-91)
but is not yet fully defined. The Devanagari encoding used in system
software versions 6.x was different, and was based on ISCII-88.
j) MacOS encodings for script code 21, smThai.
* Standard Thai - This is based on TIS 620-2533, except that three of
the TIS 620-2533 characters are replaced with other characters. Some
undefined code points in TIS 620-2533 are used for additional
punctuation characters.
k) MacOS encodings for script code 25, smSimpChinese.
* Standard simplified Chinese - this is a "shifted" implementation of
GB 2312-1980 (0xA0 is added to the row and to the column), with some
additional characters.
l) MacOS encodings for script code 29, smEastEurRoman.
* Standard Central European - This is similar to standard Roman, but
with a different (and larger) set of European characters and with
fewer symbols. It covers several Slavic languages (Czech, Polish,
Slovak, Slovenian), Hungarian, and the languages of the Baltic
republics (Estonian, Latvian, Lithuanian).
FILE LIST:
The file names here have been changed from the original files
previously published on the Unicode.org FTP server. The mapping
is as follows:
Original Name "8.3" Name
------------------- ------------
MacOS-ReadMe.txt 0README.TXT
MacOS_Cyrillic.txt CYRILLIC.TXT
MacOS_Japanese.txt JAPAN.TXT
MacOS_Turkish.txt TURKISH.TXT
MacOS_Arabic.txt ARABIC.TXT
MacOS_Dingbats.txt DINGBAT.TXT
MacOS_Roman.txt ROMAN.TXT
MacOS_Ukrainian.txt UKRAINE.TXT
MacOS_CentralEuro.txt CNTEURO.TXT
MacOS_Greek.txt GREEK.TXT
MacOS_Romanian.txt ROMANIA.TXT
MacOS_CorpChars.txt CORPCHR.TXT
MacOS_Hebrew.txt HEBREW.TXT
MacOS_Symbol.txt SYMBOL.TXT
MacOS_Croatian.txt CROATIAN.TXT
MacOS_Icelandic.txt ICELAND.TXT
MacOS_Thai.txt THAI.TXT
