-*-mode: text; coding: utf-8;-*- Copyright (C) 2002-2021 Free Software Foundation, Inc. See the end of the file for license conditions. Importing a new Unicode Standard version into Emacs ------------------------------------------------------------- Emacs uses the following files from the Unicode Character Database (a.k.a. "UCD): . UnicodeData.txt . Blocks.txt . BidiBrackets.txt . BidiMirroring.txt . IVD_Sequences.txt . NormalizationTest.txt . SpecialCasing.txt . emoji-data.txt . BidiCharacterTest.txt First, the first 8 files need to be copied into admin/unidata/, and the file https://www.unicode.org/copyright.html should be copied over copyright.html in admin/unidata (that file and emoji-data.txt might need trailing whitespace removed before they can be committed to the Emacs repository). Then Emacs should be rebuilt for them to take effect. Rebuilding Emacs updates several derived files elsewhere in the Emacs source tree, mainly in lisp/international/. When Emacs is rebuilt for the first time after importing the new files, pay attention to any warning or error messages. In particular, admin/unidata/unidata-gen.el will complain if UnicodeData.txt defines new bidirectional attributes of characters, because unidata-gen.el, bidi.c and dispextern.h need to be updated in that case; failure to do so will cause aborts in redisplay. unidata-gen.el will also complain if the format of the Unicode Copyright notice in copyright.html changed in significant ways; in that case, update the regular expression in unidata-gen-file used to extract the copyright string. Next, review the changes in UnicodeData.txt vs the previous version used by Emacs. Any changes, be it introduction of new scripts or addition of codepoints to existing scripts, might need corresponding changes in the data used for filling the category-table, case-table, and char-width-table. The additional scripts should cause automatic updates in charscript.el, but it is a good idea to look at the results and see if any changes in admin/unidata/blocks.awk are required. The setting of char-width-table around line 1200 of characters.el should be checked against the latest version of the Unicode file EastAsianWidth.txt, and any discrepancies fixed: double-width characters are those marked with W or F in that file. Zero-width characters are not taken from EastAsianWidth.txt, they are those whose Unicode General Category property is one of Mn, Me, or Cf, and also Hangul jungseong and jongseong characters (a.k.a. "Jamo medial vowels" and "Jamo final consonants"). Any new scripts added by UnicodeData.txt will also need updates to script-representative-chars defined in fontset.el, and also the list of OTF script tags in otf-script-alist, whose source is on this page: https://docs.microsoft.com/en-us/typography/opentype/spec/scripttags Other databases in fontset.el might also need to be updated as needed. One notable place to check is the function setup-default-fontset, where new scripts will generally need some addition, most probably to the list of "simple" scripts (search for "Simple"). The function 'ucs-names', defined in lisp/international/mule-cmds.el, might need to be updated because it knows about used and unused ranges of Unicode codepoints, which a new release of the Unicode Standard could change. Next, test normalization functions against NormalizationTests.txt, in the test/ directory run: make lisp/international/ucs-normalize-tests See commentary in test/lisp/international/ucs-normalize-tests.el regarding failing lines. The file BidiCharacterTest.txt should be copied to the test suite, and if its format has changed, the file biditest.el there should be modified to follow suit. If there's trailing whitespace in BidiCharacterTest.txt, it should be removed before committing the new version. Visit "emoji-data.txt" with the rebuilt Emacs, and check that an appropriate font is being used for the emoji (by default Emacs uses "Noto Color Emoji"). Running the following command in that buffer will give you an idea of which codepoints are not supported by whichever font Emacs is using. (defun check-emoji-coverage (font-name-regexp) "Display a buffer containing emoji codepoints for which FONT-NAME is not used. This must be run from a buffer in the format of emoji-data.txt. FONT-NAME-REGEXP is checked using `string-match'." (interactive "MFont Name: ") (save-excursion (goto-char (point-min)) (let (res char name ifont) (while (re-search-forward "; Emoji [^(]+(\\(.\\)[).\uFE0F]" nil t) (setq char (aref (match-string 1) 0)) (setq ifont (car (internal-char-font nil char))) (when ifont (setq name (font-xlfd-name ifont))) (if (or (not ifont) (not (string-match font-name-regexp name))) (setq res (concat (string char) res)))) (when res (with-output-to-temp-buffer "*Check-Emoji-Coverage*" (princ (format "Font not matching '%s' was used for the following characters:\n%s" font-name-regexp (reverse res)))))))) Finally, etc/NEWS should be updated to announce the support for the new Unicode version. Problems, fixmes and other unicode-related issues ------------------------------------------------------------- Notes by fx to record various things of variable importance. Handa needs to check them -- don't take too seriously, especially with regard to completeness. * SINGLE_BYTE_CHAR_P returns true for Latin-1 characters, which has undesirable effects. E.g.: (multibyte-string-p (let ((s "x")) (aset s 0 ?£) s)) => nil (multibyte-string-p (concat [?£])) => nil (text-char-description ?£) => "M-#" These examples are all fixed by the change of 2002-10-14, but there still exist questionable SINGLE_BYTE_CHAR_P in the code (keymap.c and print.c). * Rationalize character syntax and its relationship to the Unicode database. (Applies mainly to symbol and punctuation syntax.) * Fontset handling and customization needs work. We want to relate fonts to scripts, probably based on the Unicode blocks. The presence of small-repertoire 10646-encoded fonts in XFree 4 is a pain, not currently worked round. With the change on 2002-07-26, multiple fonts can be specified in a fontset for a specific range of characters. Each range can also be specified by script. Before using ISO10646 fonts, Emacs checks their repertories to avoid such fonts that don't have a glyph for a specific character. fx has worked on fontset customization, but was stymied by basic problems with the way the default face is dealt with (and something else, I think). This needs revisiting. * Work is also needed on charset and coding system priorities. * The relevant bits of latin1-disp.el need porting (and probably re-naming/updating). See also cyril-util.el. * Quail files need more work now the encoding is largely irrelevant. * What to do with the old coding categories stuff? * The preferred-coding-system property of charsets should probably be junked unless it can be made more useful now. * find-multibyte-characters needs looking at. * Implement Korean cp949/UHC, BIG5-HKSCS and any other important missing charsets. * Lazy-load tables for unify-charset somehow? Actually, Emacs clears out all charset maps and unify-map just before dumping, and they are loaded again on demand by the dumped emacs. But, those maps (char tables) generated while temacs is running can't be removed from the dumped emacs. * iso-2022 charsets get unified on i/o. With the change on 2003-01-06, decoding routines put the 'charset' property onto decoded text, and iso-2022 encoder pay attention to it. Thus, for instance, reading and writing by iso-2022-7bit preserve the original designation sequences. The property name 'preferred-charset' may be better? We may have to utilize this property to decide a font. * Revisit locale processing: look at treating the language and charset parts separately. (Language should affect things like spelling and calendar, but that's not a Unicode issue.) * Handle Unicode combining characters usefully, e.g. diacritics, and handle more scripts specifically (à la Devanagari). There are issues with canonicalization. * We need tabular input methods, e.g. for maths symbols. (Not specific to Unicode.) * Need multibyte text in menus, e.g. for the above. (Not specific to Unicode -- see Emacs etc/TODO, but now mostly works with gtk.) * Populate char-width-table correctly for Unicode characters and worry about what happens when double-width charsets covering non-CJK characters are unified. * There are type errors lurking, e.g. in Fcheck_coding_systems_region. Define ENABLE_CHECKING to find them. * Old auto-save files, and similar files, such as Gnus drafts, containing non-ASCII characters probably won't be re-read correctly. Source file encoding -------------------- Most Emacs source files are encoded in UTF-8 (or in ASCII, which is a subset), but there are a few exceptions, listed below. Perhaps someday many of these files will be converted to UTF-8, for convenience when using tools like 'grep -r', but this might need nontrivial changes to the build process. * chinese-big5 These are verbatim copies of files taken from external sources. They haven't been converted to UTF-8. leim/CXTERM-DIC/4Corner.tit leim/CXTERM-DIC/ARRAY30.tit leim/CXTERM-DIC/ECDICT.tit leim/CXTERM-DIC/ETZY.tit leim/CXTERM-DIC/PY-b5.tit leim/CXTERM-DIC/Punct-b5.tit leim/CXTERM-DIC/QJ-b5.tit leim/CXTERM-DIC/ZOZY.tit leim/MISC-DIC/CTLau-b5.html leim/MISC-DIC/cangjie-table.b5 * chinese-iso-8bit These are verbatim copies of files taken from external sources. They haven't been converted to UTF-8. leim/CXTERM-DIC/CCDOSPY.tit leim/CXTERM-DIC/Punct.tit leim/CXTERM-DIC/QJ.tit leim/CXTERM-DIC/SW.tit leim/CXTERM-DIC/TONEPY.tit leim/MISC-DIC/CTLau.html leim/MISC-DIC/pinyin.map leim/MISC-DIC/ziranma.cin * cp850 This file contains non-ASCII characters in unibyte strings. When editing a keyboard layout it's more convenient to see 'é' than '\202', and the MS-DOS compiler requires the single byte if a backslash escape is not being used. src/msdos.c * iso-2022-cn-ext This file is externally generated from leim/MISC-DIC/cangjie-table.b5 by Big5->CNS converter. It hasn't been converted to UTF-8. leim/MISC-DIC/cangjie-table.cns * japanese-iso-8bit SKK-JISYO.L is a verbatim copy of a file taken from an external source. It hasn't been converted to UTF-8. leim/SKK-DIC/SKK-JISYO.L * japanese-shift-jis This is a verbatim copy of a file taken from an external source. It hasn't been converted to UTF-8. admin/charsets/mapfiles/cns2ucsdkw.txt * iso-2022-jp This contains just one CJK charset, but Emacs currently has no easy way to specify set-charset-priority on a per-file basis, so converting this file to UTF-8 might change the file's appearance when viewed by an Emacs that is operating in some other language environment. etc/tutorials/TUTORIAL.ja * utf-8-emacs These files contain characters that cannot be encoded in UTF-8. lisp/language/ethio-util.el lisp/language/ethiopic.el lisp/language/ind-util.el lisp/language/tibet-util.el lisp/language/tibetan.el lisp/leim/quail/ethiopic.el lisp/leim/quail/tibetan.el lisp/international/titdic-cnv.el * binary files These files contain binary data, and are not text files. Some of the entries in this list are patterns, and stand for any files with the listed extension. *.gz *.icns *.ico *.pbm *.pdf *.png *.sig etc/e/eterm-color etc/package-keyring.gpg msdos/emacs.pif nextstep/GNUstep/Emacs.base/Resources/emacs.tiff nt/icons/hand.cur This file is part of GNU Emacs. GNU Emacs is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. GNU Emacs is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with GNU Emacs. If not, see .