* Re: gutenberg-coding.el -- coding system for Project Gutenberg files
[not found] ` <E1ESdmi-0002fk-Pp@fencepost.gnu.org>
@ 2005-10-24 22:33 ` Kevin Ryde
2005-10-25 2:11 ` Kenichi Handa
0 siblings, 1 reply; 10+ messages in thread
From: Kevin Ryde @ 2005-10-24 22:33 UTC (permalink / raw)
Cc: rms
[-- Attachment #1: Type: text/plain, Size: 557 bytes --]
This is my go at Project Gutenberg ebook/etext coding system detection
adapted to the emacs cvs.
The charset names in the texts are slightly free-form and need an
unhappy amount of massaging. "list.log" below is what I grepped out
of all the current files (about 23000 of them).
Some charset names are obvious typos (I reported them), but it doesn't
hurt to allow them.
2005-10-24 Kevin Ryde <user42@zip.com.au>
* international/mule.el (project-gutenberg-auto-coding-function): New
function.
(auto-coding-functions): Add it.
[-- Attachment #2: mule.el.gutenberg.diff --]
[-- Type: text/plain, Size: 5769 bytes --]
Index: mule.el
===================================================================
RCS file: /cvsroot/emacs/emacs/lisp/international/mule.el,v
retrieving revision 1.227
diff -u -c -r1.227 mule.el
cvs server: conflicting specifications of output style
*** mule.el 23 Oct 2005 18:24:00 -0000 1.227
--- mule.el 24 Oct 2005 22:06:19 -0000
***************
*** 1588,1594 ****
(symbol :tag "Coding system"))))
;; See the bottom of this file for built-in auto coding functions.
! (defcustom auto-coding-functions '(sgml-xml-auto-coding-function
sgml-html-meta-auto-coding-function)
"A list of functions which attempt to determine a coding system.
--- 1588,1595 ----
(symbol :tag "Coding system"))))
;; See the bottom of this file for built-in auto coding functions.
! (defcustom auto-coding-functions '(project-gutenberg-auto-coding-function
! sgml-xml-auto-coding-function
sgml-html-meta-auto-coding-function)
"A list of functions which attempt to determine a coding system.
***************
*** 2204,2209 ****
--- 2205,2307 ----
;;; Built-in auto-coding-functions:
+
+ (defun project-gutenberg-auto-coding-function (size)
+ "Determine character encoding of a Project Gutenberg EBook/Etext.
+ This function is designed for use in `auto-coding-functions'.
+
+ A Project Gutenberg text has \"Project Gutenberg\" in the first line, and a
+ subsequent \"Character set encoding:\" line. The latter gives the coding
+ system.
+
+ Some early non-ASCII texts don't have a \"Character set encoding:\", for
+ those you have to use other Emacs mechanisms (eg. \\[universal-coding-system-argument]).
+
+ See http://www.gutenberg.org for more about Project Gutenberg."
+
+ (and (looking-at ".*Project Gutenberg")
+
+ ;; The regexp here is "^Cha[rt]acter set encoding: *\\(.*\\)", except
+ ;; tweaked to avoid trailing spaces and \r in the match-string.
+ ;;
+ ;; Project Gutenberg files are CRLF line endings (usually) so \r is
+ ;; normal; and trailing spaces have been seen in a few files.
+ ;;
+ ;; "Chatacter" is a typo seen in about 220 files as of 2005 (though
+ ;; only 38 are non-ASCII).
+ ;;
+ (re-search-forward
+ "^Cha[rt]acter set encoding:[ \t\r]*\\(\\([ \t\r]*[^ \t\r\n]+\\)*\\)"
+ ;; only search first 200 lines
+ (save-excursion (forward-line 200) (point))
+ t)
+
+ ;; The character set names are slightly free form. They're perfectly
+ ;; understandable to a human, but need some massaging to get
+ ;; something `locale-charset-to-coding-system' can handle. The stuff
+ ;; below was tested on the full set of files in 2005.
+ ;;
+ ;; Some readme.txt files have "MP3" or the like given as the
+ ;; character set, which is bogus, it refers to the existance of .mp3
+ ;; files, the .txt is plain ascii. We let such cases get the warning
+ ;; message.
+
+ (let* ((orig-charset (match-string 1))
+ (charset (downcase orig-charset)))
+
+ ;; "ascii" -> "us-ascii"
+ ;; "iso-646-us (us-ascii)" -> "us-ascii"
+ (if (member charset '("ascii" "iso-646-us (us-ascii)"))
+ (setq charset "us-ascii"))
+
+ ;; "ascii, with a few iso-8859-1 characters" etc -> "iso-8859-1"
+ ;; "acii, with some iso-8859-1 characters" -> "iso-8859-1"
+ ;; the "acii" is a typo in dvptn10.txt, easy enough to allow it
+ (setq charset (replace-regexp-in-string
+ "^as?cii[ (,]*with.* \\(iso-8859-[0-9]+\\).*"
+ "\\1" charset t))
+
+ ;; "cp-1250" -> "windows-1250"
+ ;; "cp1251" -> "windows-1251"
+ ;; "codepage 1250" -> "windows-1250"
+ ;; "windows codepage 1252" -> "windows-1252"
+ ;; "windows code page 1252" -> "windows-1252"
+ (setq charset (replace-regexp-in-string
+ "^\\(cp\\|codepage\\|windows \\(code ?page\\)?\\)[ -]*"
+ "windows-" charset t t))
+
+ ;; "unicode" alone -> "utf-8", found in 10752-8.txt
+ (setq charset (replace-regexp-in-string "^unicode\r?$" "utf-8"
+ charset t))
+
+ ;; "unicode utf-8" -> "utf-8"
+ (setq charset (replace-regexp-in-string "^unicode utf" "utf"
+ charset t t))
+
+ ;; "unicode (utf-8)" -> "utf-8"
+ (setq charset (replace-regexp-in-string "^unicode (\\(.*\\))$" "\\1"
+ charset t))
+
+ ;; "iso-8858-1" -> "iso-8859-1", typo in 10439-8.txt
+ (setq charset (replace-regexp-in-string "8858" "8859" charset t t))
+
+ ;; "ido-8859-1" -> "iso-8859-1", typo in 10549-8.txt
+ (setq charset (replace-regexp-in-string "^ido-" "iso-" charset t t))
+
+ ;; "iso 8859-1 (latin-1)" -> "latin-1"
+ (setq charset (replace-regexp-in-string
+ "^iso 8859-\\([0-9]+\\) (\\(latin-\\1\\))$"
+ "\\2" charset t))
+
+ ;; "iso=8859-1" -> "iso-8859-1"
+ ;; "big 5" -> "big-5"
+ (setq charset (replace-regexp-in-string "[= ]" "-" charset t t))
+
+ (or (locale-charset-to-coding-system charset)
+ (progn
+ (message "Warning: unknown coding system \"%s\""
+ orig-charset)
+ nil)))))
(defun sgml-xml-auto-coding-function (size)
"Determine whether the buffer is XML, and if so, its encoding.
[-- Attachment #3: list.log --]
[-- Type: text/plain, Size: 5124 bytes --]
Character set encoding: ASCII eg. kimrk12.txt
Character set encoding: ISO8859_1 eg. c1001107.txt
Character set encoding: ACII, with some ISO-8859-1 characters
eg. dvptn10.txt
Character set encoding: ASCII eg. 10001.txt
Character set encoding: ASCII
eg. oh11v10.txt
Character set encoding: ASCII, with 2 ISO-8859-1 characters
eg. prpsl10.txt
Character set encoding: ASCII, with a couple of ISO-8859-1 characters
eg. jrcl610.txt
Character set encoding: ASCII, with a few ISO-8859-1 characters
eg. cnnet10.txt
Character set encoding: ASCII (with a few ISO-8859-1 characters)
eg. ltlbh10.txt
Character set encoding: ASCII, with one ISO-8859-1 character
eg. srhrl10.txt
Character set encoding: ASCII, with some ISO-8859-1 characters
eg. bough11.txt
Character set encoding: ASCII, with two ISO-8859-1 characters
eg. prphi10.txt
Character set encoding: Big 5 eg. dxizi10.txt
Character set encoding: BIG-5 eg. 8dxzj10.txt
Character set encoding: Big5 eg. wesik10.txt
Character set encoding: Codepage 1250 eg. sklep10.txt
Character set encoding: CP-1250 eg. 13083-8.txt
Character set encoding: CP-1251 eg. 14741-8.txt
Character set encoding: CP-1252 eg. 12732-8.txt
Character set encoding: CP1251 eg. 11292-8.txt
Character set encoding: cp1251 eg. kknta10.txt
Character set encoding: CP1252 eg. 8ledo10.txt
Character set encoding: EUC-KR eg. kedct10.txt
Character set encoding: IDO-8859-1 eg. 10549-8.txt
Character set encoding: ISO-646-US (US-ASCII) eg. 107.txt
Character set encoding: ISO-8858-1 eg. 10439-8.txt
Character set encoding: ISO 8859-1 eg. 8bld410.txt
Character set encoding: ISO-8859-1 eg. 10002-8.txt
Character set encoding: iso-8859-1 eg. 10429-8.txt
Character set encoding: ISO=8859-1 eg. 7fool10.txt
Character set encoding: ISO 8859-1 (Latin-1) eg. 8adio10.txt
Character set encoding: iso-8859-15 eg. 8dlrm10.txt
Character set encoding: ISO-8859-2 eg. rnpz810.txt
Character set encoding: ISO Latin-1 eg. 10056-8.txt
Character set encoding: ISO-LATIN-1 eg. 8nggd10.txt
Character set encoding: ISO-Latin-1 eg. hstrd10.txt
Character set encoding: iso-Latin-1 eg. 8wpwl10.txt
Character set encoding: iso-latin-1 eg. 8engl10.txt
Character set encoding: ISO8859-1 eg. 7bjrn10.txt
Character set encoding: ISO8859_1 eg. a1001107.txt
Character set encoding: KOI8-R eg. ktria10.txt
Character set encoding: Latin 1 eg. divrw10.txt
Character set encoding: Latin-1 eg. 8dawn10.txt
Character set encoding: Latin-4 eg. kalev10.txt
Character set encoding: Latin1 eg. 10347-8.txt
Character set encoding: MP3 eg. 10348-m-readme.txt
Character set encoding: MPEG eg. atomi10m-readme.txt
Character set encoding: MPEG Layer 3 (MP3) eg. 1donq3-readme.txt
Character set encoding: Unicode eg. 10752-8.txt
Character set encoding: Unicode (UTF-8) eg. orama10u.txt
Character set encoding: Unicode UTF-8 eg. 11753-0.txt
Character set encoding: US-ASCII eg. 10078.txt
Character set encoding: US-ASCII, MIDI, Lilypond, MP3 and TeX
eg. 10535.txt
Character set encoding: UTF-16 eg. 13083-utf16.txt
Character set encoding: UTF-7 eg. 8cart10.txt
Character set encoding: UTF-8 eg. 10140-0.txt
Character set encoding: utf-8 eg. astrl10.txt
Character set encoding: UTF8 eg. 8gslt10.txt
Character set encoding: Windows-1250 eg. 15201-8.txt
Character set encoding: Windows 1251 eg. olavg10.txt
Character set encoding: Windows-1252 eg. 8clcn10.txt
Character set encoding: Windows Code Page 1252 eg. 8tjna10.txt
Character set encoding: Windows Codepage 1252 eg. 8vepi10.txt
Character set encoding: Windows1253 eg. orama10.txt
Chatacter set encoding: ISO-8859-1 eg. 10021-8.txt
Chatacter set encoding: iso-8859-1 eg. 10026-8.txt
Chatacter set encoding: MP3 eg. 10137-m-readme.txt
Chatacter set encoding: Sibelius 3 SIB format and MP3 audio
eg. 10344-readme.txt
Chatacter set encoding: US-ASCII eg. 10021.txt
[-- Attachment #4: Type: text/plain, Size: 142 bytes --]
_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-devel
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: gutenberg-coding.el -- coding system for Project Gutenberg files
2005-10-24 22:33 ` gutenberg-coding.el -- coding system for Project Gutenberg files Kevin Ryde
@ 2005-10-25 2:11 ` Kenichi Handa
2005-10-25 3:53 ` Stefan Monnier
2005-10-25 20:27 ` Richard M. Stallman
0 siblings, 2 replies; 10+ messages in thread
From: Kenichi Handa @ 2005-10-25 2:11 UTC (permalink / raw)
Cc: rms, emacs-devel
In article <87oe5eeea2.fsf@zip.com.au>, Kevin Ryde <user42@zip.com.au> writes:
> [1 <text/plain (7bit)>]
> This is my go at Project Gutenberg ebook/etext coding system detection
> adapted to the emacs cvs.
> The charset names in the texts are slightly free-form and need an
> unhappy amount of massaging. "list.log" below is what I grepped out
> of all the current files (about 23000 of them).
> Some charset names are obvious typos (I reported them), but it doesn't
> hurt to allow them.
I think that the code is good to be included in Emacs. But,
as it's not a bug fix, and I think not many people benefit
from that (how many people read Gutenberg text file?). So,
I'd like to ask Richard to decide whether we include it now
or postpone it.
---
Kenichi Handa
handa@m17n.org
> 2005-10-24 Kevin Ryde <user42@zip.com.au>
> * international/mule.el (project-gutenberg-auto-coding-function): New
> function.
> (auto-coding-functions): Add it.
> [2 mule.el.gutenberg.diff <text/plain (7bit)>]
> Index: mule.el
> ===================================================================
> RCS file: /cvsroot/emacs/emacs/lisp/international/mule.el,v
> retrieving revision 1.227
> diff -u -c -r1.227 mule.el
> cvs server: conflicting specifications of output style
> *** mule.el 23 Oct 2005 18:24:00 -0000 1.227
> --- mule.el 24 Oct 2005 22:06:19 -0000
> ***************
> *** 1588,1594 ****
> (symbol :tag "Coding system"))))
> ;; See the bottom of this file for built-in auto coding functions.
> ! (defcustom auto-coding-functions '(sgml-xml-auto-coding-function
> sgml-html-meta-auto-coding-function)
> "A list of functions which attempt to determine a coding system.
> --- 1588,1595 ----
> (symbol :tag "Coding system"))))
> ;; See the bottom of this file for built-in auto coding functions.
> ! (defcustom auto-coding-functions '(project-gutenberg-auto-coding-function
> ! sgml-xml-auto-coding-function
> sgml-html-meta-auto-coding-function)
> "A list of functions which attempt to determine a coding system.
> ***************
> *** 2204,2209 ****
> --- 2205,2307 ----
> ;;; Built-in auto-coding-functions:
> +
> + (defun project-gutenberg-auto-coding-function (size)
> + "Determine character encoding of a Project Gutenberg EBook/Etext.
> + This function is designed for use in `auto-coding-functions'.
> +
> + A Project Gutenberg text has \"Project Gutenberg\" in the first line, and a
> + subsequent \"Character set encoding:\" line. The latter gives the coding
> + system.
> +
> + Some early non-ASCII texts don't have a \"Character set encoding:\", for
> + those you have to use other Emacs mechanisms (eg. \\[universal-coding-system-argument]).
> +
> + See http://www.gutenberg.org for more about Project Gutenberg."
> +
> + (and (looking-at ".*Project Gutenberg")
> +
> + ;; The regexp here is "^Cha[rt]acter set encoding: *\\(.*\\)", except
> + ;; tweaked to avoid trailing spaces and \r in the match-string.
> + ;;
> + ;; Project Gutenberg files are CRLF line endings (usually) so \r is
> + ;; normal; and trailing spaces have been seen in a few files.
> + ;;
> + ;; "Chatacter" is a typo seen in about 220 files as of 2005 (though
> + ;; only 38 are non-ASCII).
> + ;;
> + (re-search-forward
> + "^Cha[rt]acter set encoding:[ \t\r]*\\(\\([ \t\r]*[^ \t\r\n]+\\)*\\)"
> + ;; only search first 200 lines
> + (save-excursion (forward-line 200) (point))
> + t)
> +
> + ;; The character set names are slightly free form. They're perfectly
> + ;; understandable to a human, but need some massaging to get
> + ;; something `locale-charset-to-coding-system' can handle. The stuff
> + ;; below was tested on the full set of files in 2005.
> + ;;
> + ;; Some readme.txt files have "MP3" or the like given as the
> + ;; character set, which is bogus, it refers to the existance of .mp3
> + ;; files, the .txt is plain ascii. We let such cases get the warning
> + ;; message.
> +
> + (let* ((orig-charset (match-string 1))
> + (charset (downcase orig-charset)))
> +
> + ;; "ascii" -> "us-ascii"
> + ;; "iso-646-us (us-ascii)" -> "us-ascii"
> + (if (member charset '("ascii" "iso-646-us (us-ascii)"))
> + (setq charset "us-ascii"))
> +
> + ;; "ascii, with a few iso-8859-1 characters" etc -> "iso-8859-1"
> + ;; "acii, with some iso-8859-1 characters" -> "iso-8859-1"
> + ;; the "acii" is a typo in dvptn10.txt, easy enough to allow it
> + (setq charset (replace-regexp-in-string
> + "^as?cii[ (,]*with.* \\(iso-8859-[0-9]+\\).*"
> + "\\1" charset t))
> +
> + ;; "cp-1250" -> "windows-1250"
> + ;; "cp1251" -> "windows-1251"
> + ;; "codepage 1250" -> "windows-1250"
> + ;; "windows codepage 1252" -> "windows-1252"
> + ;; "windows code page 1252" -> "windows-1252"
> + (setq charset (replace-regexp-in-string
> + "^\\(cp\\|codepage\\|windows \\(code ?page\\)?\\)[ -]*"
> + "windows-" charset t t))
> +
> + ;; "unicode" alone -> "utf-8", found in 10752-8.txt
> + (setq charset (replace-regexp-in-string "^unicode\r?$" "utf-8"
> + charset t))
> +
> + ;; "unicode utf-8" -> "utf-8"
> + (setq charset (replace-regexp-in-string "^unicode utf" "utf"
> + charset t t))
> +
> + ;; "unicode (utf-8)" -> "utf-8"
> + (setq charset (replace-regexp-in-string "^unicode (\\(.*\\))$" "\\1"
> + charset t))
> +
> + ;; "iso-8858-1" -> "iso-8859-1", typo in 10439-8.txt
> + (setq charset (replace-regexp-in-string "8858" "8859" charset t t))
> +
> + ;; "ido-8859-1" -> "iso-8859-1", typo in 10549-8.txt
> + (setq charset (replace-regexp-in-string "^ido-" "iso-" charset t t))
> +
> + ;; "iso 8859-1 (latin-1)" -> "latin-1"
> + (setq charset (replace-regexp-in-string
> + "^iso 8859-\\([0-9]+\\) (\\(latin-\\1\\))$"
> + "\\2" charset t))
> +
> + ;; "iso=8859-1" -> "iso-8859-1"
> + ;; "big 5" -> "big-5"
> + (setq charset (replace-regexp-in-string "[= ]" "-" charset t t))
> +
> + (or (locale-charset-to-coding-system charset)
> + (progn
> + (message "Warning: unknown coding system \"%s\""
> + orig-charset)
> + nil)))))
> (defun sgml-xml-auto-coding-function (size)
> "Determine whether the buffer is XML, and if so, its encoding.
> [3 list.log <text/plain (7bit)>]
> Character set encoding: ASCII eg. kimrk12.txt
> Character set encoding: ISO8859_1 eg. c1001107.txt
> Character set encoding: ACII, with some ISO-8859-1 characters
> eg. dvptn10.txt
> Character set encoding: ASCII eg. 10001.txt
> Character set encoding: ASCII
> eg. oh11v10.txt
> Character set encoding: ASCII, with 2 ISO-8859-1 characters
> eg. prpsl10.txt
> Character set encoding: ASCII, with a couple of ISO-8859-1 characters
> eg. jrcl610.txt
> Character set encoding: ASCII, with a few ISO-8859-1 characters
> eg. cnnet10.txt
> Character set encoding: ASCII (with a few ISO-8859-1 characters)
> eg. ltlbh10.txt
> Character set encoding: ASCII, with one ISO-8859-1 character
> eg. srhrl10.txt
> Character set encoding: ASCII, with some ISO-8859-1 characters
> eg. bough11.txt
> Character set encoding: ASCII, with two ISO-8859-1 characters
> eg. prphi10.txt
> Character set encoding: Big 5 eg. dxizi10.txt
> Character set encoding: BIG-5 eg. 8dxzj10.txt
> Character set encoding: Big5 eg. wesik10.txt
> Character set encoding: Codepage 1250 eg. sklep10.txt
> Character set encoding: CP-1250 eg. 13083-8.txt
> Character set encoding: CP-1251 eg. 14741-8.txt
> Character set encoding: CP-1252 eg. 12732-8.txt
> Character set encoding: CP1251 eg. 11292-8.txt
> Character set encoding: cp1251 eg. kknta10.txt
> Character set encoding: CP1252 eg. 8ledo10.txt
> Character set encoding: EUC-KR eg. kedct10.txt
> Character set encoding: IDO-8859-1 eg. 10549-8.txt
> Character set encoding: ISO-646-US (US-ASCII) eg. 107.txt
> Character set encoding: ISO-8858-1 eg. 10439-8.txt
> Character set encoding: ISO 8859-1 eg. 8bld410.txt
> Character set encoding: ISO-8859-1 eg. 10002-8.txt
> Character set encoding: iso-8859-1 eg. 10429-8.txt
> Character set encoding: ISO=8859-1 eg. 7fool10.txt
> Character set encoding: ISO 8859-1 (Latin-1) eg. 8adio10.txt
> Character set encoding: iso-8859-15 eg. 8dlrm10.txt
> Character set encoding: ISO-8859-2 eg. rnpz810.txt
> Character set encoding: ISO Latin-1 eg. 10056-8.txt
> Character set encoding: ISO-LATIN-1 eg. 8nggd10.txt
> Character set encoding: ISO-Latin-1 eg. hstrd10.txt
> Character set encoding: iso-Latin-1 eg. 8wpwl10.txt
> Character set encoding: iso-latin-1 eg. 8engl10.txt
> Character set encoding: ISO8859-1 eg. 7bjrn10.txt
> Character set encoding: ISO8859_1 eg. a1001107.txt
> Character set encoding: KOI8-R eg. ktria10.txt
> Character set encoding: Latin 1 eg. divrw10.txt
> Character set encoding: Latin-1 eg. 8dawn10.txt
> Character set encoding: Latin-4 eg. kalev10.txt
> Character set encoding: Latin1 eg. 10347-8.txt
> Character set encoding: MP3 eg. 10348-m-readme.txt
> Character set encoding: MPEG eg. atomi10m-readme.txt
> Character set encoding: MPEG Layer 3 (MP3) eg. 1donq3-readme.txt
> Character set encoding: Unicode eg. 10752-8.txt
> Character set encoding: Unicode (UTF-8) eg. orama10u.txt
> Character set encoding: Unicode UTF-8 eg. 11753-0.txt
> Character set encoding: US-ASCII eg. 10078.txt
> Character set encoding: US-ASCII, MIDI, Lilypond, MP3 and TeX
> eg. 10535.txt
> Character set encoding: UTF-16 eg. 13083-utf16.txt
> Character set encoding: UTF-7 eg. 8cart10.txt
> Character set encoding: UTF-8 eg. 10140-0.txt
> Character set encoding: utf-8 eg. astrl10.txt
> Character set encoding: UTF8 eg. 8gslt10.txt
> Character set encoding: Windows-1250 eg. 15201-8.txt
> Character set encoding: Windows 1251 eg. olavg10.txt
> Character set encoding: Windows-1252 eg. 8clcn10.txt
> Character set encoding: Windows Code Page 1252 eg. 8tjna10.txt
> Character set encoding: Windows Codepage 1252 eg. 8vepi10.txt
> Character set encoding: Windows1253 eg. orama10.txt
> Chatacter set encoding: ISO-8859-1 eg. 10021-8.txt
> Chatacter set encoding: iso-8859-1 eg. 10026-8.txt
> Chatacter set encoding: MP3 eg. 10137-m-readme.txt
> Chatacter set encoding: Sibelius 3 SIB format and MP3 audio
> eg. 10344-readme.txt
> Chatacter set encoding: US-ASCII eg. 10021.txt
> [4 <text/plain; us-ascii (7bit)>]
> _______________________________________________
> Emacs-devel mailing list
> Emacs-devel@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-devel
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: gutenberg-coding.el -- coding system for Project Gutenberg files
2005-10-25 2:11 ` Kenichi Handa
@ 2005-10-25 3:53 ` Stefan Monnier
2005-10-26 1:43 ` Kevin Ryde
2005-10-25 20:27 ` Richard M. Stallman
1 sibling, 1 reply; 10+ messages in thread
From: Stefan Monnier @ 2005-10-25 3:53 UTC (permalink / raw)
Cc: Kevin Ryde, rms, emacs-devel
>> This is my go at Project Gutenberg ebook/etext coding system detection
>> adapted to the emacs cvs.
Since those files are not used very commonly I think it's difficult to
justify the risk of checking for the presence of a Gutenberg "coding cookie"
on each and every file.
Don't those files have a (set of) standardized extensions?
Stefan
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: gutenberg-coding.el -- coding system for Project Gutenberg files
2005-10-25 2:11 ` Kenichi Handa
2005-10-25 3:53 ` Stefan Monnier
@ 2005-10-25 20:27 ` Richard M. Stallman
1 sibling, 0 replies; 10+ messages in thread
From: Richard M. Stallman @ 2005-10-25 20:27 UTC (permalink / raw)
Cc: user42, emacs-devel
I think that the code is good to be included in Emacs. But,
as it's not a bug fix, and I think not many people benefit
from that (how many people read Gutenberg text file?). So,
I'd like to ask Richard to decide whether we include it now
or postpone it.
It fixes something--whether to call it a bug or a gap is not clear to
me. And it seems harmless. So there is no reason to wait for
the release.
However, the issue of possible false matches that Stefan raised is a
something we should look at first.
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: gutenberg-coding.el -- coding system for Project Gutenberg files
2005-10-25 3:53 ` Stefan Monnier
@ 2005-10-26 1:43 ` Kevin Ryde
2005-10-27 1:29 ` Richard M. Stallman
0 siblings, 1 reply; 10+ messages in thread
From: Kevin Ryde @ 2005-10-26 1:43 UTC (permalink / raw)
Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
> Since those files are not used very commonly I think it's difficult to
> justify the risk of checking for the presence of a Gutenberg "coding cookie"
> on each and every file.
Hopefully the risk is small. Or put it this way, is there likely to
be a file in the matched form, but which is in some encoding other
than the one stated.
I did want to do something tighter for the first-line match, or locate
the end of the header info part, but the format varies too much.
> Don't those files have a (set of) standardized extensions?
The normal files (the ones I'm interested in) are .txt. (There's
others like tex or html, txt is the majority.)
(The downloads are available as .zip containing .txt, and I presume
most people will use that, so something that works from archive-mode
as well as a plain find-file is highly desirable.)
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: gutenberg-coding.el -- coding system for Project Gutenberg files
2005-10-26 1:43 ` Kevin Ryde
@ 2005-10-27 1:29 ` Richard M. Stallman
2005-10-27 2:04 ` Kevin Ryde
2005-10-27 23:57 ` Kevin Ryde
0 siblings, 2 replies; 10+ messages in thread
From: Richard M. Stallman @ 2005-10-27 1:29 UTC (permalink / raw)
Cc: emacs-devel
> Don't those files have a (set of) standardized extensions?
The normal files (the ones I'm interested in) are .txt. (There's
others like tex or html, txt is the majority.)
We could do this only for files called .txt, I suppose.
That would not eliminate false matches, but would limit them.
I did want to do something tighter for the first-line match, or locate
the end of the header info part, but the format varies too much.
How about if you send Project Gutenberg a suggestion asking them if
they would make the format more standard, to help programs DTRT?
If you offer one or two specific suggestions, that would be useful.
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: gutenberg-coding.el -- coding system for Project Gutenberg files
2005-10-27 1:29 ` Richard M. Stallman
@ 2005-10-27 2:04 ` Kevin Ryde
2005-10-28 3:47 ` Richard M. Stallman
2005-10-27 23:57 ` Kevin Ryde
1 sibling, 1 reply; 10+ messages in thread
From: Kevin Ryde @ 2005-10-27 2:04 UTC (permalink / raw)
"Richard M. Stallman" <rms@gnu.org> writes:
>
> How about if you send Project Gutenberg a suggestion asking them if
> they would make the format more standard, to help programs DTRT?
> If you offer one or two specific suggestions, that would be useful.
Yep. Though with about 16000 texts, and a fairly conservative update
policy (from what I read in the faqs) I don't suppose that can happen
immediately.
Which is to say, I think there's merit in emacs working with what's
presently published/mirrored/etc, if it can be done in an acceptable
way.
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: gutenberg-coding.el -- coding system for Project Gutenberg files
2005-10-27 1:29 ` Richard M. Stallman
2005-10-27 2:04 ` Kevin Ryde
@ 2005-10-27 23:57 ` Kevin Ryde
1 sibling, 0 replies; 10+ messages in thread
From: Kevin Ryde @ 2005-10-27 23:57 UTC (permalink / raw)
[-- Attachment #1: Type: text/plain, Size: 939 bytes --]
"Richard M. Stallman" <rms@gnu.org> writes:
>
> We could do this only for files called .txt, I suppose.
> That would not eliminate false matches, but would limit them.
I realized that the only files needing to be matched are the non-ascii
ones with a charset spec. Doh. So I think the test can be for a file
starting with one of
"Project Gutenberg "
"Project Gutenberg's "
"The Project Gutenberg "
"**This is a COPYRIGHTED Project Gutenberg "
and possibly with bytes
0xEF 0xBB 0xBF
before those, which is in some (but not all) utf-8 files. This is
tighter than just "Project Gutenberg" anywhere in the first line. New
diff below.
I put just "..." to match the three marker bytes. I'd like to put
those exactly, but it will be matched against a unibyte buffer (if I'm
not mistaken), and I'm unsure how to give a unibyte string literal, or
a multibyte which will match correctly. (A couple of things I tried
didn't work.)
[-- Attachment #2: mule.el.gutenberg-2.diff --]
[-- Type: text/plain, Size: 6207 bytes --]
Index: mule.el
===================================================================
RCS file: /cvsroot/emacs/emacs/lisp/international/mule.el,v
retrieving revision 1.227
diff -u -c -r1.227 mule.el
cvs server: conflicting specifications of output style
*** mule.el 23 Oct 2005 18:24:00 -0000 1.227
--- mule.el 27 Oct 2005 23:56:29 -0000
***************
*** 1588,1594 ****
(symbol :tag "Coding system"))))
;; See the bottom of this file for built-in auto coding functions.
! (defcustom auto-coding-functions '(sgml-xml-auto-coding-function
sgml-html-meta-auto-coding-function)
"A list of functions which attempt to determine a coding system.
--- 1588,1595 ----
(symbol :tag "Coding system"))))
;; See the bottom of this file for built-in auto coding functions.
! (defcustom auto-coding-functions '(project-gutenberg-auto-coding-function
! sgml-xml-auto-coding-function
sgml-html-meta-auto-coding-function)
"A list of functions which attempt to determine a coding system.
***************
*** 2204,2209 ****
--- 2205,2315 ----
;;; Built-in auto-coding-functions:
+
+ (defun project-gutenberg-auto-coding-function (size)
+ "Determine character encoding of a Project Gutenberg EBook/Etext.
+ This function is designed for use in `auto-coding-functions'.
+
+ A Project Gutenberg text has \"Project Gutenberg\" in the first line, and a
+ subsequent \"Character set encoding:\" line. The latter gives the coding
+ system.
+
+ Some early non-ASCII texts don't have a \"Character set encoding:\", for
+ those you have to use other Emacs mechanisms (eg. \\[universal-coding-system-argument]).
+
+ See http://www.gutenberg.org for more about Project Gutenberg."
+
+ ;; This regexp identifies a gutenberg file, it's kept fairly tight to
+ ;; avoid false matches.
+ ;;
+ ;; Many early gutenberg files have different first lines, but the
+ ;; alternatives here are enough for the non-ascii files existing in 2005.
+ ;;
+ ;; Some (but not all) utf-8 files begin with a marker sequence EF BB BF.
+
+ (and (looking-at "\\(...\\)?\\(Project Gutenberg\\('s\\)?\\|The Project Gutenberg\\|\\**This is a COPYRIGHTED Project Gutenberg\\) ")
+
+ ;; The regexp here is "^Cha[rt]acter set encoding: *\\(.*\\)", except
+ ;; tweaked to avoid trailing spaces and \r in the match-string.
+ ;;
+ ;; Project Gutenberg files are CRLF line endings (usually) so \r is
+ ;; normal; and trailing spaces have been seen in a few files.
+ ;;
+ ;; "Chatacter" is a typo seen in about 220 files as of 2005 (though
+ ;; only 38 are non-ASCII).
+ ;;
+ (re-search-forward
+ "^Cha[rt]acter set encoding:[ \t\r]*\\(\\([ \t\r]*[^ \t\r\n]+\\)*\\)"
+ ;; only search first 200 lines
+ (save-excursion (forward-line 200) (point))
+ t)
+
+ ;; The character set names are slightly free form. They're perfectly
+ ;; understandable to a human, but need some massaging to get
+ ;; something `locale-charset-to-coding-system' can handle. The stuff
+ ;; below was tested on the full set of files in 2005.
+ ;;
+ ;; Some readme.txt files have "MP3" or the like given as the
+ ;; character set, which is bogus, it refers to the existance of .mp3
+ ;; files, the .txt is plain ascii. We let such cases get the warning
+ ;; message.
+
+ (let* ((orig-charset (match-string 1))
+ (charset (downcase orig-charset)))
+
+ ;; "ascii" -> "us-ascii"
+ ;; "iso-646-us (us-ascii)" -> "us-ascii"
+ (if (member charset '("ascii" "iso-646-us (us-ascii)"))
+ (setq charset "us-ascii"))
+
+ ;; "ascii, with a few iso-8859-1 characters" etc -> "iso-8859-1"
+ ;; "acii, with some iso-8859-1 characters" -> "iso-8859-1"
+ ;; the "acii" is a typo in dvptn10.txt, easy enough to allow it
+ (setq charset (replace-regexp-in-string
+ "^as?cii[ (,]*with.* \\(iso-8859-[0-9]+\\).*"
+ "\\1" charset t))
+
+ ;; "cp-1250" -> "windows-1250"
+ ;; "cp1251" -> "windows-1251"
+ ;; "codepage 1250" -> "windows-1250"
+ ;; "windows codepage 1252" -> "windows-1252"
+ ;; "windows code page 1252" -> "windows-1252"
+ (setq charset (replace-regexp-in-string
+ "^\\(cp\\|codepage\\|windows \\(code ?page\\)?\\)[ -]*"
+ "windows-" charset t t))
+
+ ;; "unicode" alone -> "utf-8", found in 10752-8.txt
+ (setq charset (replace-regexp-in-string "^unicode\r?$" "utf-8"
+ charset t))
+
+ ;; "unicode utf-8" -> "utf-8"
+ (setq charset (replace-regexp-in-string "^unicode utf" "utf"
+ charset t t))
+
+ ;; "unicode (utf-8)" -> "utf-8"
+ (setq charset (replace-regexp-in-string "^unicode (\\(.*\\))$" "\\1"
+ charset t))
+
+ ;; "iso-8858-1" -> "iso-8859-1", typo in 10439-8.txt
+ (setq charset (replace-regexp-in-string "8858" "8859" charset t t))
+
+ ;; "ido-8859-1" -> "iso-8859-1", typo in 10549-8.txt
+ (setq charset (replace-regexp-in-string "^ido-" "iso-" charset t t))
+
+ ;; "iso 8859-1 (latin-1)" -> "latin-1"
+ (setq charset (replace-regexp-in-string
+ "^iso 8859-\\([0-9]+\\) (\\(latin-\\1\\))$"
+ "\\2" charset t))
+
+ ;; "iso=8859-1" -> "iso-8859-1"
+ ;; "big 5" -> "big-5"
+ (setq charset (replace-regexp-in-string "[= ]" "-" charset t t))
+
+ (or (locale-charset-to-coding-system charset)
+ (progn
+ (message "Warning: unknown coding system \"%s\""
+ orig-charset)
+ nil)))))
(defun sgml-xml-auto-coding-function (size)
"Determine whether the buffer is XML, and if so, its encoding.
[-- Attachment #3: Type: text/plain, Size: 142 bytes --]
_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-devel
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: gutenberg-coding.el -- coding system for Project Gutenberg files
2005-10-27 2:04 ` Kevin Ryde
@ 2005-10-28 3:47 ` Richard M. Stallman
2005-11-10 21:06 ` Kevin Ryde
0 siblings, 1 reply; 10+ messages in thread
From: Richard M. Stallman @ 2005-10-28 3:47 UTC (permalink / raw)
Cc: emacs-devel
Yep. Though with about 16000 texts, and a fairly conservative update
policy (from what I read in the faqs) I don't suppose that can happen
immediately.
That could well be true. On the other hand, they would not have to do
the update by hand, file by file. If they decide to change a format
so as to standardize, they can run a script to do it.
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: gutenberg-coding.el -- coding system for Project Gutenberg files
2005-10-28 3:47 ` Richard M. Stallman
@ 2005-11-10 21:06 ` Kevin Ryde
0 siblings, 0 replies; 10+ messages in thread
From: Kevin Ryde @ 2005-11-10 21:06 UTC (permalink / raw)
I send a mail to PG suggesting to help automated parsing, I didn't get
a reply. I think it's worth working with what's available now, and
see if something better is possible later.
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2005-11-10 21:06 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <87u0ff9387.fsf@zip.com.au>
[not found] ` <E1ES3v6-0006dW-M6@fencepost.gnu.org>
[not found] ` <87wtk9uqcq.fsf@zip.com.au>
[not found] ` <E1ESdmi-0002fk-Pp@fencepost.gnu.org>
2005-10-24 22:33 ` gutenberg-coding.el -- coding system for Project Gutenberg files Kevin Ryde
2005-10-25 2:11 ` Kenichi Handa
2005-10-25 3:53 ` Stefan Monnier
2005-10-26 1:43 ` Kevin Ryde
2005-10-27 1:29 ` Richard M. Stallman
2005-10-27 2:04 ` Kevin Ryde
2005-10-28 3:47 ` Richard M. Stallman
2005-11-10 21:06 ` Kevin Ryde
2005-10-27 23:57 ` Kevin Ryde
2005-10-25 20:27 ` Richard M. Stallman
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/emacs.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).