bug#47455: 27.1; bibtex mode - citation key generation - non-ascii characters

bug#47455: 27.1; bibtex mode - citation key generation - non-ascii characters

[Brian Elmegaard via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[-- Attachment #1: Type: text/plain, Size: 4529 bytes --]



Using C-c C-c in a bibtex cleans the entry and generates a citation key.
If the author name includes non-ascii characters these are included in
the key, even though BibTeX does not accept this.

For example:
@Article{äöü21,
  author =          {æøå äöü},
  title =               {foo},
  journal =          {bar},
  year =               2021}



In GNU Emacs 27.1 (build 1, x86_64-w64-mingw32)
of 2020-08-21 built on CIRROCUMULUS
Repository revision: 86d8d76aa36037184db0b2897c434cdaab1a9ae8
Repository branch: HEAD
Windowing system distributor 'Microsoft Corp.', version 10.0.17134
System Description: Microsoft Windows 10 Enterprise (v10.0.1803.17134.2087)

Recent messages:
Checking 71 files in c:/Program Files (x86)/Emacs/emacs-27.1-x86_64/share/emacs/27.1/lisp/erc...
Checking 34 files in c:/Program Files (x86)/Emacs/emacs-27.1-x86_64/share/emacs/27.1/lisp/emulation...
Checking 180 files in c:/Program Files (x86)/Emacs/emacs-27.1-x86_64/share/emacs/27.1/lisp/emacs-lisp...
Checking 24 files in c:/Program Files (x86)/Emacs/emacs-27.1-x86_64/share/emacs/27.1/lisp/cedet...
Checking 59 files in c:/Program Files (x86)/Emacs/emacs-27.1-x86_64/share/emacs/27.1/lisp/calendar...
Checking 87 files in c:/Program Files (x86)/Emacs/emacs-27.1-x86_64/share/emacs/27.1/lisp/calc...
Checking 113 files in c:/Program Files (x86)/Emacs/emacs-27.1-x86_64/share/emacs/27.1/lisp/obsolete...
Checking 1 files in c:/Users/brel/AppData/Roaming/.emacs.d/lisp/dna-mode...
Checking for load-path shadows...done
Making completion list...

Configured using:
'configure --without-dbus --host=x86_64-w64-mingw32
--without-compress-install 'CFLAGS=-O2 -static''

Configured features:
XPM JPEG TIFF GIF PNG RSVG SOUND NOTIFY W32NOTIFY ACL GNUTLS LIBXML2
HARFBUZZ ZLIB TOOLKIT_SCROLL_BARS MODULES THREADS JSON PDUMPER LCMS2 GMP

Important settings:
  value of $LANG: DAN
  locale-coding-system: cp1252

Major mode: Lisp Interaction

Minor modes in effect:
  save-place-mode: t
  delete-selection-mode: t
  show-paren-mode: t
  recentf-mode: t
  global-auto-revert-mode: t
  cua-mode: t
  TeX-PDF-mode: t
  TeX-source-correlate-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(help-mode pp shadow sort mail-extr emacsbug message rmc puny dired
dired-loaddefs format-spec rfc822 mml mml-sec epa derived epg epg-config
gnus-util rmail rmail-loaddefs text-property-search time-date mm-decode
mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader
sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils
saveplace delsel paren recentf tree-widget wid-edit autorevert
filenotify cua-base cus-start cus-load server tex-mik tex crm advice
texmathp finder-inf info tex-site package easymenu browse-url
url-handlers url-parse auth-source cl-seq eieio eieio-core cl-macs
eieio-loaddefs password-cache json subr-x map url-vars seq byte-opt gv
bytecomp byte-compile cconv cl-loaddefs cl-lib tooltip eldoc electric
uniquify ediff-hook vc-hooks lisp-float-type mwheel dos-w32 ls-lisp
disp-table term/w32-win w32-win w32-vars term/common-win tool-bar dnd
fontset image regexp-opt fringe tabulated-list replace newcomment
text-mode elisp-mode lisp-mode prog-mode register page tab-bar menu-bar
rfn-eshadow isearch timer select scroll-bar mouse jit-lock font-lock
syntax facemenu font-core term/tty-colors frame minibuffer cl-generic
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite charscript charprop
case-table epa-hook jka-cmpr-hook help simple abbrev obarray
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote threads w32notify w32
lcms2 multi-tty make-network-process emacs)

Memory information:
((conses 16 100510 8583)
(symbols 48 10963 1)
(strings 32 33640 2227)
(string-bytes 1 1011136)
(vectors 16 16807)
(vector-slots 8 206002 13918)
(floats 8 44 213)
(intervals 56 279 0)
(buffers 1000 15))

[-- Attachment #2: Type: text/html, Size: 14021 bytes --]

bug#47455: 27.1; bibtex mode - citation key generation - non-ascii characters

[Lars Ingebrigtsen]

Brian Elmegaard <be@mek.dtu.dk> writes:

> Using C-c C-c in a bibtex cleans the entry and generates a citation key.
>
> If the author name includes non-ascii characters these are included in
>
> the key, even though BibTeX does not accept this.

Is this the case for all versions of BibTeX?

> For example:
>
> @Article{äöü21,
>
>   author =          {æøå äöü},
>
>   title =               {foo},
>
>   journal =          {bar},
>
>   year =               2021}

I guess Emacs could an asciification of some sort here, but I'm not sure
there's any that's universally accepted?  I've added Roland to the
CCs -- perhaps he has some comments.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#47455: 27.1; bibtex mode - citation key generation - non-ascii characters

[Roland Winkler]

On Tue May 18 2021 Lars Ingebrigtsen wrote:
> Brian Elmegaard <be@mek.dtu.dk> writes:
> 
> > Using C-c C-c in a bibtex cleans the entry and generates a citation key.
> > If the author name includes non-ascii characters these are included in
> > the key, even though BibTeX does not accept this.
> 
> Is this the case for all versions of BibTeX?

I believe the problem lies here already in BibTeX itself, that is,
BibTeX [like conventional (La)TeX] does not like non-ascii
characters anywhere, not in the key nor anywhere else.

Of course, there is biblatex and also new versions of (La)TeX that
can handle non-ascii characters.  But that's a separate story.

> > For example:
> >
> > @Article{äöü21,
> >   author =          {æøå äöü},
> 
> I guess Emacs could an asciification of some sort here, but I'm
> not sure there's any that's universally accepted?

The default of the user variable bibtex-autokey-transcriptions
handles "LaTeX non-ascii" characters like \"a.  You can customize
these rules to your liking.

I vaguely remember an old thread that started from the very question
raised here and expanding on how asciification can be encapsulated
in some generic piece of elisp code.  But I cannot find it anymore
and I do not know either whether this would be possible at all.  I
believe, everyone agrees on asciification of German umlaute like

  ä -> ae

But beyond that, I do not know how to do this satisfactorily.

bug#47455: 27.1; bibtex mode - citation key generation - non-ascii characters

[Lars Ingebrigtsen]

"Roland Winkler" <winkler@gnu.org> writes:

> I vaguely remember an old thread that started from the very question
> raised here and expanding on how asciification can be encapsulated
> in some generic piece of elisp code.  But I cannot find it anymore
> and I do not know either whether this would be possible at all.  I
> believe, everyone agrees on asciification of German umlaute like
>
>   ä -> ae
>
> But beyond that, I do not know how to do this satisfactorily.

And this gets even more difficult to deal with for non-Latin scripts.

So I'm not sure anything here can be done programmatically...  the
command could output a warning?  "Probably invalid key"?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#47455: 27.1; bibtex mode - citation key generation - non-ascii characters

[Roland Winkler]

On Tue May 25 2021 Lars Ingebrigtsen wrote:
> And this gets even more difficult to deal with for non-Latin scripts.
> 
> So I'm not sure anything here can be done programmatically...  the
> command could output a warning?  "Probably invalid key"?

The warning is a good idea.  Actually, the warning should be issued
if there are non-ascii characters anywhere in a BibTeX key because
(oldfashioned) BibTeX will choke on those no matter where they
appear.  So I'll add a new element for the user variable
bibtex-entry-format for this.  Then users can enable these warnings
if they use oldfashioned BibTeX.  (Those who use modern variants of
BibTeX need not enable these warnings.)

I believe that the real problem here lies in the fact that many
publishers of scientific journals let you download citation records
for their journal articles.  When they offer not only BibTeX-formatted
records but other formats, too, the BibTeX records are often
malformed, decorated with non-ascii characters that BibTeX (and
LaTeX) cannot handle and other things.  I have been fooled a number
of times by "invisible" non-ascii characters.  So I will enable the
new option for myself!

bug#47455: 27.1; bibtex mode - citation key generation - non-ascii characters

[Brian Elmegaard via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Hi

Thanks for looking into this.
I understand your reasoning about this being an issue with the tools used.
In auctex I can also enter \newcommand{\ü}{u} without being warned that it will not work with latex.

The warning seems to be a good idea to me as well.

Brian

-----Original Message-----
From: Roland Winkler <winkler@gnu.org> 
Sent: 26. maj 2021 20:57
To: Lars Ingebrigtsen <larsi@gnus.org>
Cc: 47455@debbugs.gnu.org; Brian Elmegaard <be@mek.dtu.dk>
Subject: Re: bug#47455: 27.1; bibtex mode - citation key generation - non-ascii characters

On Tue May 25 2021 Lars Ingebrigtsen wrote:
> And this gets even more difficult to deal with for non-Latin scripts.
> 
> So I'm not sure anything here can be done programmatically...  the 
> command could output a warning?  "Probably invalid key"?

The warning is a good idea.  Actually, the warning should be issued if there are non-ascii characters anywhere in a BibTeX key because
(oldfashioned) BibTeX will choke on those no matter where they appear.  So I'll add a new element for the user variable bibtex-entry-format for this.  Then users can enable these warnings if they use oldfashioned BibTeX.  (Those who use modern variants of BibTeX need not enable these warnings.)

I believe that the real problem here lies in the fact that many publishers of scientific journals let you download citation records for their journal articles.  When they offer not only BibTeX-formatted records but other formats, too, the BibTeX records are often malformed, decorated with non-ascii characters that BibTeX (and
LaTeX) cannot handle and other things.  I have been fooled a number of times by "invisible" non-ascii characters.  So I will enable the new option for myself!

bug#47455: 27.1; bibtex mode - citation key generation - non-ascii characters

[Roland Winkler]

On Fri May 28 2021 Brian Elmegaard wrote:
> I understand your reasoning about this being an issue with the
> tools used.  In auctex I can also enter \newcommand{\ü}{u} without
> being warned that it will not work with latex.
> 
> The warning seems to be a good idea to me as well.

I realized:

One can also instruct font-lock to use for non-ascii characters
something like font-lock-warning-face (based on a user option for
enabling this behavior).

And Auctex could do the same.

However this fails with something like the unicode character
'ZERO WIDTH SPACE' (which has fooled me occassionally in the very
context we are discussing here).  Is it possible to instruct emacs
to make such "hidden characters" more easily visible, say by using
some display property?

The fontification could also be encapsulated in a minor mode that
one could use for \(Bib\|La\)?TeX files.

Maybe such a minor mode exists already and I am right now
reinventing the wheel?

bug#47455: 27.1; bibtex mode - citation key generation - non-ascii characters

[Lars Ingebrigtsen]

"Roland Winkler" <winkler@gnu.org> writes:

> One can also instruct font-lock to use for non-ascii characters
> something like font-lock-warning-face (based on a user option for
> enabling this behavior).

Sounds like a good idea.

> However this fails with something like the unicode character
> 'ZERO WIDTH SPACE' (which has fooled me occassionally in the very
> context we are discussing here).  Is it possible to instruct emacs
> to make such "hidden characters" more easily visible, say by using
> some display property?

I vaguely remember a discussion about this not too long ago, but I can't
find it now.

All the other space "special" space characters, like NON-BREAKING SPACE
and the like, are fontified specially by default in Emacs 28.  But the
point of ZERO WIDTH SPACE is that it takes no room, which makes it
difficult to fontify.  :-)

But bibtex could apply a special display property here, or fontify the
surrounding characters in a special way.

> The fontification could also be encapsulated in a minor mode that
> one could use for \(Bib\|La\)?TeX files.
>
> Maybe such a minor mode exists already and I am right now
> reinventing the wheel?

I can't recall any such minor mode.  Anybody else?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#47455: 27.1; bibtex mode - citation key generation - non-ascii characters

[Arash Esbati]

Brian Elmegaard via "Bug reports for GNU Emacs, the Swiss army knife of
text editors" <bug-gnu-emacs@gnu.org> writes:

> Thanks for looking into this.
> I understand your reasoning about this being an issue with the tools
> used.  In auctex I can also enter \newcommand{\ü}{u} without being
> warned that it will not work with latex.

It's not about LaTeX as the macro package, it's about the underlying
engine.  This example

    \documentclass[10pt]{article}
    \newcommand\ü{foo}
    \begin{document}
    \ü bar
    \end{document}

works when your run lualatex on it, but chokes with pdflatex.  The
reason is described in ltnews30.pdf:

    Improving Unicode handling in pdfTEX

    [...]  What is not possible when using an 8-bit engine such as
    pdfTEX is to use characters other than ascii letters as part of a
    command name. This is due to the fact that all other characters in
    such engines are not single character tokens, but in fact consist of
    a sequence of bytes and this is not supported in command names.

So yes, it about the tools used :-)

Best, Arash

bug#47455: 27.1; bibtex mode - citation key generation - non-ascii characters

[Roland Winkler]

On Tue Jun 1 2021 Arash Esbati wrote:
> It's not about LaTeX as the macro package, it's about the underlying
> engine.

This is known and doesn't solve the problem.  If you are required to
use LaTeX and not, say, lualatex, the reason for this being whatever
it is, then you should use only ascii characters.

bug#47455: 27.1; bibtex mode - citation key generation - non-ascii characters

[Roland Winkler]

On Tue, Jun 01 2021, Roland Winkler wrote:
> On Tue Jun 1 2021 Arash Esbati wrote:
>> It's not about LaTeX as the macro package, it's about the underlying
>> engine.
>
> This is known and doesn't solve the problem.  If you are required to
> use LaTeX and not, say, lualatex, the reason for this being whatever
> it is, then you should use only ascii characters.

Closing.