auto-coding-function for LaTeX

auto-coding-function for LaTeX

[Arne Jørgensen]

[-- Attachment #1: Type: text/plain, Size: 1133 bytes --]

Hello,

I have written a function (latex-auto-coding-function) that determines
a LaTeX files encoding from i.e. a

  \usepackage[latin1]{inputenc}

line (if it is used).

It uses an alist for mapping the LaTeX input encoding names to Emacs'
coding system names. The entries in the alist that I have commented
out are those standard input encoding names in LaTeX that there are no
emacs codings for (or those that I could not identify the coding
system of).

If you find it useful it can be added to Emacs (the papers I signed
say "emacs gnus" - I don't whether that is for emacs AND gnus or just
for gnus).

I'm not a trained elisp programmer so feel free to comment on or
change the code.

If included it should probably be added to mule.el. I'm don't know
whether the latex-auto-coding-function should be added to
auto-coding-functions by default (sooner or later there will be a lot
of functions in there). I know it should be in my
auto-coding-functions though :-)

Kind regards,
-- 
Arne Jørgensen
Gammel Kongevej 7, 1. th., DK-1610 Copenhagen V, Denmark
mobile: +45 21 65 01 13
email: arne@arnested.dk, <http://arnested.dk/>


[-- Attachment #2: auto-coding-function for LaTeX --]
[-- Type: application/emacs-lisp, Size: 1772 bytes --]

[-- Attachment #3: Type: text/plain, Size: 142 bytes --]

_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-devel

Bug in widget-complete (was: auto-coding-function for LaTeX)

[Arne Jørgensen]

Arne Jørgensen <arne@arnested.dk> writes:

> I have written a function (latex-auto-coding-function) that determines
> a LaTeX files encoding from i.e. a

Testing the above i discovered that widget-complete seems to have a
problem when trying to do completion in an empty field.

I tried with fields of type 'coding-system' and 'files' and for both
it looks like it is trying to do the completion from text outside the
field.

The problem is in a Emacs fresh from the trunk and with
--no-site-file and --no-init-file.

Unfortunately I don't have the time at the moment to investigate the
problem further at the moment.

Kind regards,
-- 
Arne Jørgensen
Gammel Kongevej 7, 1. th., DK-1610 Copenhagen V, Denmark
mobile: +45 21 65 01 13
email: arne@arnested.dk, <http://arnested.dk/>

Re: auto-coding-function for LaTeX

[Juri Linkov]

Arne Jørgensen <arne@arnested.dk> writes:
> If included it should probably be added to mule.el. I'm don't know
> whether the latex-auto-coding-function should be added to
> auto-coding-functions by default (sooner or later there will be a lot
> of functions in there). I know it should be in my
> auto-coding-functions though :-)

I think there should be a new variable for "mode to auto-coding-function"
mapping to be able to define

    (latex-mode . latex-auto-coding-function).

And the fact that currently `auto-coding-functions' contains by default
`(sgml-xml-auto-coding-function sgml-html-meta-auto-coding-function)'
doesn't seem right.  They are called for every buffer, not just HTML
and XML.

Thanks to the new variable `magic-mode-alist' now it is possible
to set SGML and HTML major modes depending on buffer contents.
So these functions could be added to "mode to auto-coding-function"
mapping to make them mode-dependent:

    (xml-mode . sgml-xml-auto-coding-function)
    (html-mode . sgml-html-meta-auto-coding-function)

instead of adding them to the default of `auto-coding-functions'.

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: auto-coding-function for LaTeX

[Arne Jørgensen]

Juri Linkov <juri@jurta.org> writes:

> Arne Jørgensen <arne@arnested.dk> writes:
>> If included it should probably be added to mule.el. I'm don't know
>> whether the latex-auto-coding-function should be added to
>> auto-coding-functions by default (sooner or later there will be a lot
>> of functions in there). I know it should be in my
>> auto-coding-functions though :-)
>
> I think there should be a new variable for "mode to auto-coding-function"
> mapping to be able to define
>
>     (latex-mode . latex-auto-coding-function).
>
> And the fact that currently `auto-coding-functions' contains by default
> `(sgml-xml-auto-coding-function sgml-html-meta-auto-coding-function)'
> doesn't seem right.  They are called for every buffer, not just HTML
> and XML.

Yes. That was why I thouhgt it might not be a good idea to just add
it to `auto-coding-functions'. It looks like a general problem to
have too many functions inspecting _every_ file you open.

> Thanks to the new variable `magic-mode-alist' now it is possible
> to set SGML and HTML major modes depending on buffer contents.
> So these functions could be added to "mode to auto-coding-function"
> mapping to make them mode-dependent:
>
>     (xml-mode . sgml-xml-auto-coding-function)
>     (html-mode . sgml-html-meta-auto-coding-function)
>
> instead of adding them to the default of `auto-coding-functions'.

[ remember I'm no expert so there is a good chance the following is
  wrong ]

As I understand it it is too late to change the coding system in a
proper way after the file is loaded into a buffer. I looked a bit at
various functions and `revert-buffer-with-coding-system' is close to
being useful, but I still asks me if I'm sure I want to revert the
buffer.

[ well ... this _is_ how I understand it ... but there _should_ be a
  way to change the coding system afterwards ]

Kind regards,
-- 
Arne Jørgensen
Gammel Kongevej 7, 1. th., DK-1610 Copenhagen V, Denmark
mobile: +45 21 65 01 13
email: arne@arnested.dk, <http://arnested.dk/>

Re: auto-coding-function for LaTeX

[Arne Jørgensen]

[-- Attachment #1: Type: text/plain, Size: 408 bytes --]

Arne Jørgensen <arne@arnested.dk> writes:

> Yes. That was why I thouhgt it might not be a good idea to just add
> it to `auto-coding-functions'. It looks like a general problem to
> have too many functions inspecting _every_ file you open.

I look a bit more in to it. And it actually seems more reasonable to
put the function (slightly changed) into `file-coding-system-alist'.

It could look like this:


[-- Attachment #2: latex-find-file-coding-system.el --]
[-- Type: application/emacs-lisp, Size: 1875 bytes --]

[-- Attachment #3: Type: text/plain, Size: 305 bytes --]


The function is being called with an argument (arg) but I don't know
what I'm supposed to do with the argument so I ignore it at the
moment.

Kind regards,
-- 
Arne Jørgensen
Gammel Kongevej 7, 1. th., DK-1610 Copenhagen V, Denmark
mobile: +45 21 65 01 13
email: arne@arnested.dk, <http://arnested.dk/>

[-- Attachment #4: Type: text/plain, Size: 142 bytes --]

_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-devel

Re: auto-coding-function for LaTeX

[Reiner Steib]

On Fri, Nov 12 2004, Arne Jørgensen wrote:

> It could look like this: [...]

I have added some other coding systems available in Emacs and I tried
to improve the code and doc-strings.  WDYT?

--8<---------------cut here---------------start------------->8---
(defcustom latex-auto-coding-alist
  '(("ansinew" . windows-1252) ; MS Windows ANSI encoding, extension of Latin-1
    ("applemac" . mac-roman)
    ("ascii" . us-ascii)
    ("cp1250" . cp1250) ; MS Windows encoding, codepage 1250
    ("cp1252" . windows-1252) ; synonym of ansinew
    ("cp437de" . cp437) ; IBM code page 437 (German version): 225 is \ss
    ("cp437" . cp437) ; IBM code page 437: 225 is \beta
    ("cp850" . cp850) ; IBM code page 850
    ("cp852" . cp852) ; IBM code page 852
    ("cp865" . cp865) ; IBM code page 865
    ;; The DECMultinational charaterset used by the OpenVMS system
    ;; ("decmulti" . undecided)
    ("latin1" . iso-8859-1)
    ("latin2" . iso-8859-2)
    ("latin3" . iso-8859-3)
    ("latin4" . iso-8859-4)
    ("latin5" . iso-8859-5)
    ("latin9" . iso-8859-15)
    ;; ("next" . undecided) ; The Next encoding
    ("utf8" . utf-8))
  "Mapping from encoding names used by \"inputenc.sty\" Emacs coding systems.
Used by the function `latex-auto-coding-function'."
  :group 'mule
  :type '(alist :key-type (string :tag "LaTeX input encoding")
		:value-type (coding-system :tag "Coding system")))

(defun latex-find-file-coding-system (arg)
"Determine the encoding of a LaTeX buffer if it uses \"inputenc.sty\".

This function is intended to be added to `auto-coding-functions'.

The mapping from LaTeX's \"inputenc.sty\" encoding names to Emacs
coding system names is determined from `latex-auto-coding-alist'."
  (when (re-search-forward "\\usepackage\\[\\(.*\\)\\]{inputenc}" 1000 t)
    (let* ((match (match-string 1))
	   (inputenc-match (cdr (assoc match latex-auto-coding-alist)))
	   (sym (intern match)))
      (when inputenc-match (setq sym inputenc-match))
      (when (coding-system-p sym)
	sym))))

(add-to-list 'file-coding-system-alist
	     '("\\.tex\\|\\.ltx\\'" . latex-find-file-coding-system))
--8<---------------cut here---------------end--------------->8---

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: auto-coding-function for LaTeX

[Arne Jørgensen]

Reiner Steib <reinersteib+gmane@imap.cc> writes:

> On Fri, Nov 12 2004, Arne Jørgensen wrote:
>
>> It could look like this: [...]
>
> I have added some other coding systems available in Emacs

None of the coding systems you added are present in my Emacs. So why
add them? To make it easier for a user to customized when/if they are
added? Or are they present in your Emacs?

> and I tried to improve the code and doc-strings. WDYT?

Beautiful! 

Since the function no longer is supposed to be in `auto-coding-alist'
I think the name of the alist should be changed though. Maybe
`latex-inputenc-coding-alist' or something.

--8<---------------cut here---------------start------------->8---
(defcustom latex-inputenc-coding-alist
  '(("ansinew" . windows-1252) ; MS Windows ANSI encoding, extension of Latin-1
    ("applemac" . mac-roman)
    ("ascii" . us-ascii)
    ("cp1250" . cp1250) ; MS Windows encoding, codepage 1250
    ("cp1252" . windows-1252) ; synonym of ansinew
    ("cp437de" . cp437) ; IBM code page 437 (German version): 225 is \ss
    ("cp437" . cp437) ; IBM code page 437: 225 is \beta
    ("cp850" . cp850) ; IBM code page 850
    ("cp852" . cp852) ; IBM code page 852
    ("cp865" . cp865) ; IBM code page 865
    ;; The DECMultinational charaterset used by the OpenVMS system
    ;; ("decmulti" . undecided)
    ("latin1" . iso-8859-1)
    ("latin2" . iso-8859-2)
    ("latin3" . iso-8859-3)
    ("latin4" . iso-8859-4)
    ("latin5" . iso-8859-5)
    ("latin9" . iso-8859-15)
    ;; ("next" . undecided) ; The Next encoding
    ("utf8" . utf-8))
  "Mapping from encoding names used by \"inputenc.sty\" to Emacs coding systems.
Used by the function `latex-find-file-coding-system'."
  :group 'mule
  :type '(alist :key-type (string :tag "LaTeX input encoding")
		:value-type (coding-system :tag "Coding system")))

(defun latex-find-file-coding-system (arg)
"Determine the encoding of a LaTeX buffer if it uses \"inputenc.sty\".

This function is intended to be added to `file-coding-system-alist'.

The mapping from LaTeX's \"inputenc.sty\" encoding names to Emacs
coding system names is determined from `latex-auto-coding-alist'."
  (when (re-search-forward "\\usepackage\\[\\(.*\\)\\]{inputenc}" 1000 t)
    (let* ((match (match-string 1))
	   (inputenc-match (cdr (assoc match latex-inputenc-coding-alist)))
	   (sym (intern match)))
      (when inputenc-match (setq sym inputenc-match))
      (when (coding-system-p sym)
	sym))))

(add-to-list 'file-coding-system-alist
	     '("\\.tex\\|\\.ltx\\'" . latex-find-file-coding-system))
--8<---------------cut here---------------end--------------->8---

By the way. The re-search-forward searches until position 1000.
Should that number be customizable? The functions in
`auto-coding-alist' get an argument of 249 for number of positions to
search, but I think that is not enough for LaTeX files.

Kind regards,
-- 
Arne Jørgensen
Gammel Kongevej 7, 1. th., DK-1610 Copenhagen V, Denmark
mobile: +45 21 65 01 13
email: arne@arnested.dk, <http://arnested.dk/>

Re: auto-coding-function for LaTeX

[Reiner Steib]

On Fri, Nov 12 2004, Arne Jørgensen wrote:

> None of the coding systems you added are present in my Emacs. So why
> add them? To make it easier for a user to customized when/if they are
> added? Or are they present in your Emacs?

All mentioned cp* codings are available, but not by default, see `M-x
codepage-setup RET TAB TAB'.  I don't know if there is some mechanism
in Emacs call `codepage-setup' automatically.

I'm not sure about the cp437 vs. cp437de issue (β = \beta vs. ß = \ss
= S SHARP).  Maybe this is good enough, see [1].  It seems that both,
`code-pages.el' and `codepage.el', use S SHARP.

> The re-search-forward searches until position 1000.  Should that
> number be customizable? The functions in `auto-coding-alist' get an
> argument of 249 for number of positions to search, but I think that
> is not enough for LaTeX files.

Yes, a limit of 249 surely is not enough for typical LaTeX files.
Maybe the search could stop either at a certain number or when
"\begin{document}" was found.  It should also be checked that the
"inputenc cookie" isn't a LaTeX comment.

If your code cannot be integrated in Emacs, I would propose to add it
to AUCTeX.

Bye, Reiner.

[1] See e.g. this article by Markus Kuhn in a German newsgroup:

,----[ Translation of the most relevant part: ]
| Microsoft Code Page 437 officially contains LATIN SMALL LETTER SHARP
| S on position 0xE1 (though the designers of IBM CGA card ROM's
| apparently thought more of a beta at that time (~1979)).  Many
| conversation programs ([...]) map both s sharp (U+00DF) and beta
| (U+03B2) to 0xE1 in CP437.  Hopefully GNU iconv starting from glibc
| 2.2 will get this right.
`----

,----[ Original article: ]
| From: mgk25@cl.cam.ac.uk (Markus Kuhn)
| Subject: Re: recode latin1:cp437 und 'ß'
| Newsgroups: de.comp.os.unix.apps, de.comp.standards
| Date: 2000/04/22
| Message-ID: <8ds839$anv$1@pegasus.csx.cam.ac.uk>
| 
| >> Georg Schwarz <schwarz@physik.tu-berlin.de> wrote:
| >> 
| >> > Wenn ich Text auf einem Epson Stylus 800+ direkt drucke, schiebt der
| >> > Druckertreiber ihn zunächst durch recode latin1:cp437 (GNU recode
| >> > 3.4.1). Allerdings wird ein 'ß' nicht korrekt ausgegeben. Ich denke daß
| >> > code page 437 gar kein ß enthält.
| 
| Microsoft Code Page 437 enthält offiziell LATIN SMALL LETTER SHARP S auf
| position 0xE1 (auch wenn die Designer des IBM CGA-Karten ROM's
| offensichtlich damals (~1979) eher wohl an ein beta gedacht haben).
| Sehr gute Konvertierprogramme (und dazu darf man GNU recode 3.5 wegen
| der vielen Fehler in den zudem nur injektiven Tabellen leider
| noch nicht zaehlen!) mappen sowohl das scharfe s (U+00DF) als auch as
| kleine beta (U+03B2) auf 0xE1 in CP437. GNU iconv wird das hoffentlich
| ab glibc 2.2 entlich richtig hinkriegen.
| [...]
| Quellen:
| 
|   ftp://ftp.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/PC/CP437.TXT
|   ftp://ftp.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/PC/CP850.TXT
`----
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: auto-coding-function for LaTeX

[Reiner Steib]

On Fri, Nov 12 2004, Reiner Steib wrote:

> All mentioned cp* codings are available, but not by default, see `M-x
> codepage-setup RET TAB TAB'.

Oops...

,----[ C-h n --> NEWS ]
| ** Many new coding systems are available by loading the `code-pages'
| library.  These include complete versions of most of those in
| codepage.el, based on Unicode mappings.  `codepage-setup' is now
| obsolete and is used only in the MS-DOS port of Emacs.  windows-1252
| and windows-1251 are preloaded since the former is so common and the
| latter is used by GNU locales.
`----

I would suggest to keep the cp* entries in the list.  Or check if the
coding system is available similar to the method used in
`mm-charset-synonym-alist', see lisp/gnus/mm-util.el.  Instead of
`mm-coding-system-p' [1] you may use `coding-system-p'.

Bye, Reiner.

[1] `mm-coding-system-p' is used in Gnus for compatibility with XEmacs.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: auto-coding-function for LaTeX

[Juri Linkov]

Arne Jørgensen <arne@arnested.dk> writes:
> I look a bit more in to it. And it actually seems more reasonable to
> put the function (slightly changed) into `file-coding-system-alist'.
> [...]
> (add-to-list 'file-coding-system-alist '("\\.tex\\|\\.ltx\\'" . latex-find-file-coding-system))

I think this is not the most right thing to do since there may
be LaTeX file extensions other than .tex and .ltx.  The standard value
of `auto-mode-alist' has also .sty, .cl[so], .bbl as file extensions
corresponding to latex-mode.  And users can add own mappings to
`auto-mode-alist' or use `-*- mode:latex -*-' headers.
In that case `file-coding-system-alist' wouldn't work.

> The function is being called with an argument (arg) but I don't know
> what I'm supposed to do with the argument so I ignore it at the
> moment.

`arg' is a list (operation, file-name, ...).

For an example of using it, you could look at the file
textmodes/po.el which is completely devoted to setting correct
encoding for PO files according to the "Charset=..." header.

Perhaps this file should be generalized to support more file types
like you want to do for LaTeX files with `inputenc' cookie.

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: auto-coding-function for LaTeX

[David Kastrup]

Juri Linkov <juri@jurta.org> writes:

> Arne Jørgensen <arne@arnested.dk> writes:
>> I look a bit more in to it. And it actually seems more reasonable to
>> put the function (slightly changed) into `file-coding-system-alist'.
>> [...]
>> (add-to-list 'file-coding-system-alist '("\\.tex\\|\\.ltx\\'" . latex-find-file-coding-system))
>
> I think this is not the most right thing to do since there may
> be LaTeX file extensions other than .tex and .ltx.  The standard value
> of `auto-mode-alist' has also .sty, .cl[so], .bbl as file extensions
> corresponding to latex-mode.  And users can add own mappings to
> `auto-mode-alist' or use `-*- mode:latex -*-' headers.
> In that case `file-coding-system-alist' wouldn't work.

In all of the mentioned cases, the files are not supposed to carry an
inputenc declaration.  So this is not a particularly compelling
reason.  However, we also have, say, .dtx and .drv files.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: auto-coding-function for LaTeX

[Richard Stallman]

    I think there should be a new variable for "mode to auto-coding-function"
    mapping to be able to define

	(latex-mode . latex-auto-coding-function).

Could we please stop looking for ideas for more features to add?
We're supposed to be aiming for a release.

Several bug reports came in the past few days; if you try debugging
one or two of them, that would be the kind of contribution that we
need now.

Re: auto-coding-function for LaTeX

[Arne Jørgensen]

[-- Attachment #1: Type: text/plain, Size: 1253 bytes --]

Arne Jørgensen <arne@arnested.dk> Nov 9, 2004:

> I have written a function (latex-auto-coding-function) that determines
> a LaTeX files encoding from i.e. a
>
>   \usepackage[latin1]{inputenc}
>
> line (if it is used).

I know I've been silent about this for a while, but I've been busy and
had only limited time to weed out some bugs ...

I received some useful comments on the code. Here is a highlight of
what has changed:

 - the function is added to `file-coding-system-alist' for the
   extensions .tex .ltx .dtx and .drv (where you will normally find
   inpuenc stuff)

 - the encoding is search for in the LaTeX commands
   \inputencoding{...} and \usepackage[...]{inputenc} (in that order)

 - if the above command is in a comment (%) it is ignored

 - if the encoding is not found we load code-pages and tries if that
   make things better 

Since you are all working hard on getting Emacs ready for release I
know it will probably not be added to cvs so I put it up on my website
<http://arnested.dk/filer/latexenc.el> and am just writing here to say
thanks for the comments.

Kind regards,
-- 
Arne Jørgensen
Gammel Kongevej 7, 1. th., DK-1610 Copenhagen V, Denmark
mobile: +45 21 65 01 13
email: arne@arnested.dk, <http://arnested.dk/>


[-- Attachment #2: Determine the encoding of a LaTeX buffer --]
[-- Type: application/emacs-lisp, Size: 3692 bytes --]

[-- Attachment #3: Type: text/plain, Size: 142 bytes --]

_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-devel