bug#51621: 29.0.50; bibtex.el biblatex "2.1.3 Non-standard Types" support

bug#51621: 29.0.50; bibtex.el biblatex "2.1.3 Non-standard Types" support

[Leonard Lausen]

biblatex types such as @audio, @music, @review from section 2.1.3 of the
biblatex documentation are not considered by bibtex.el's biblatex
dialect implementation. The standard styles treat these types (with
exceptions) like @misc. An example of an exception is @review, which is
a more specific variant of the @article type.

I think the types are called non-standard, because the standard
bibliography styles provide no bibliography drivers for these types, but
these types are known to the default data model and will be happily
accepted by biber.

The impact is that emacs will not recognize entries of those types as
valid entries, and prevent referencing them via org-cite or applying
operations such as bibtex-clean-entry.

To reproduce run `emacs -Q`. Then switch to `bibtex-mode` and set
`(bibtex-set-dialect 'biblatex t)`. Finally paste (taken from
https://tex.stackexchange.com/a/435059/106845)

    @audio{Smith2018,
      author={Stacey Vanek Smith and Stanley Plotkin},
      title={The Economics of Vaccines},
      howpublished={NPR},
      organization={Planet Money}
      month=6,
      year=2010,
      series={Planet Money},
      url={https://www.npr.org/templates/transcript/transcript.php?storyId=616926505}
    }

and navigate the cursor into the @audio entry. Then execute `M-x
bibtex-clean-entry` and observe "Not inside a BibTex entry" error
message.

Would it make sense to add support for these types to bibtex.el?



In GNU Emacs 29.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.29, cairo version 1.16.0)
 of 2021-11-05 built on leonard-xps13
Repository revision: 37632dae180e9d5196ef01343fefa9234c5f05b9
Repository branch: feature/pgtk
Repository: https://github.com/leezu/emacs
Windowing system distributor 'System Description: Gentoo/Linux

Configured using:
 'configure --with-cairo --with-x-toolkit=no --with-pgtk --with-modules
 --with-native-compilation --prefix=/home/leonard/local'

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
JSON LCMS2 LIBOTF LIBSYSTEMD LIBXML2 MODULES NATIVE_COMP NOTIFY INOTIFY
PDUMPER PGTK PNG RSVG SECCOMP SOUND THREADS TIFF TOOLKIT_SCROLL_BARS XIM
GTK3 ZLIB

Important settings:
  value of $LC_COLLATE: C
  value of $LC_CTYPE: zh_CN.UTF-8
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: BibTeX

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny dired dired-loaddefs
rfc822 mml mml-sec epa derived epg rfc6068 epg-config gnus-util rmail
rmail-loaddefs auth-source cl-seq eieio eieio-core cl-macs
eieio-loaddefs password-cache json map text-property-search seq gv
byte-opt bytecomp byte-compile cconv mm-decode mm-bodies mm-encode
mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail rfc2047
rfc2045 ietf-drums mm-util mail-prsvr mail-utils bibtex iso8601
time-date subr-x help-mode cl-loaddefs cl-lib china-util iso-transl
tooltip eldoc paren electric uniquify ediff-hook vc-hooks
lisp-float-type elisp-mode mwheel term/pgtk-win pgtk-win term/common-win
tool-bar dnd fontset image regexp-opt fringe tabulated-list replace
newcomment text-mode lisp-mode prog-mode register page tab-bar menu-bar
rfn-eshadow isearch easymenu timer select scroll-bar mouse jit-lock
font-lock syntax font-core term/tty-colors frame minibuffer cl-generic
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite emoji-zwj charscript
charprop case-table epa-hook jka-cmpr-hook help simple abbrev obarray
cl-preloaded nadvice button loaddefs faces cus-face macroexp files
window text-properties overlay sha1 md5 base64 format env code-pages
mule custom widget hashtable-print-readable backquote threads dbusbind
inotify dynamic-setting system-font-setting font-render-setting cairo
move-toolbar gtk x-toolkit pgtk lcms2 multi-tty make-network-process
native-compile emacs)

Memory information:
((conses 16 86191 6454)
 (symbols 48 7175 0)
 (strings 32 24675 1427)
 (string-bytes 1 773743)
 (vectors 16 15122)
 (vector-slots 8 363616 15516)
 (floats 8 24 48)
 (intervals 56 325 0)
 (buffers 992 10))

bug#51621: 29.0.50; bibtex.el biblatex "2.1.3 Non-standard Types" support

[Roland Winkler]

On Fri, Nov 05 2021, Leonard Lausen wrote:
> Would it make sense to add support for these types to bibtex.el?

I myself do not use biblatex.  But it seems to me that the sheer number
of entry types supported by biblatex gets overwhelming and inefficient
for users that may use only a subset of all biblatex entry types.  Could
it make sense to break down the list of biblatex entry types into two
customizable subsets such that commands like bibtex-entry and the menu
"Entry-Types" list only the "important" entry types, whereas functions
like bibtex-validate accept the complete set of entry types supported by
biblatex?

Roland

bug#51621: 29.0.50; bibtex.el biblatex "2.1.3 Non-standard Types" support

[Leo Stein]

[-- Attachment #1: Type: text/plain, Size: 1479 bytes --]

Hi,

I wanted to raise a bug report but found this earlier one from 3 years ago
that was almost the same as my own. I was confused about why BibTeX-mode
was happily fontifying a doi=... field within a @software entry, and
attaching a button to it, but I could not follow the URL — because
BibTeX-mode thinks @software is not an allowed type within the "BibTeX"
"dialect". So, bibtex-parse-entry returns nil. This is because the regex
bibtex-entry-head is built from all "valid" entry types of the "dialect".

However, there is no reason @software, or any other type of entry, should
be an "invalid" entry in a bibtex file. It's the purview of whatever bibtex
style (defined in a .bst file) to determine what to do with different
entries. Some bst's default to passing things along to the @misc entry
type. But, I count 7 different bst's in the 2024 TeXLive tree that define a
@software type, and I'm sure there are many more custom types (e.g. the TeX
User's Group's tugboat.bst defines an entry type for @ctan , i.e. to cite
an entry on the Comprehensive TeX Archive Network).

I'm starting to think that the "dialect" design within bibtex.el was
confused about bibtex vs. biblatex (this is pretty confusing, as we can see
here: https://tex.stackexchange.com/q/25701/34063). However, I'm not sure
what is the correct solution. At the very least, bibtex.el should be more
permissive about what entry types get parsed by bibtex-parse-entry.

Best
Leo

[-- Attachment #2: Type: text/html, Size: 1767 bytes --]

bug#51621: 29.0.50; bibtex.el biblatex "2.1.3 Non-standard Types" support

[Roland Winkler]

> I'm starting to think that the "dialect" design within bibtex.el was
> confused about bibtex vs. biblatex (this is pretty confusing, as we
> can see here: https://tex.stackexchange.com/q/25701/34063). However,
> I'm not sure what is the correct solution. At the very least,
> bibtex.el should be more permissive about what entry types get parsed
> by bibtex-parse-entry.

The range of "acceptable" entry types needs to be compatible with the
BibTeX style files that one wants to use.  Certainly, these style files
can be modified to handle any entry types you like.  But I am not sure
it makes sense to extend the defaults of bibtex.el beyond the defaults
defined by BibTeX and / or biblatex.  If you want to use the full range
of entry types defined by biblatex, you may be served better by making
biblatex your default dialect of bibtex-mode.  (I find it useful if
bibtex-mode keeps track of the entry types known to a dialect.)

I am currently working on a patch for bibtex-mode that will make it
easier for users to customize the entry types known by a dialect,
including the possibility to define aliases for entry types.  This patch
should be installed on master in a few weeks.  (I want to test it
first.)

PS: My reading of the above thread on stackexchange is that it will not
make everyone happy if the distinction between old BibTeX and new
biblatex gets blurred by bibtex-mode.

bug#51621: 29.0.50; bibtex.el biblatex "2.1.3 Non-standard Types" support

[Leo Stein]

[-- Attachment #1: Type: text/plain, Size: 2676 bytes --]

Dear Roland,

First of all, thanks for maintaining this very handy mode!

On Mon, Dec 2, 2024 at 8:05 AM Roland Winkler <winkler@gnu.org> wrote:

> > I'm starting to think that the "dialect" design within bibtex.el was
> > confused about bibtex vs. biblatex (this is pretty confusing, as we
> > can see here: https://tex.stackexchange.com/q/25701/34063). However,
> > I'm not sure what is the correct solution. At the very least,
> > bibtex.el should be more permissive about what entry types get parsed
> > by bibtex-parse-entry.
>
> The range of "acceptable" entry types needs to be compatible with the
> BibTeX style files that one wants to use.  Certainly, these style files
> can be modified to handle any entry types you like.  But I am not sure
> it makes sense to extend the defaults of bibtex.el beyond the defaults
> defined by BibTeX and / or biblatex.


I really wish this was more permissive. Looking at a .bib file, we have no
way of knowing the biblio style that it's going to be set with. We also
have no way of knowing whether the user is going to parse it with bibtex or
biber.


>   If you want to use the full range
> of entry types defined by biblatex, you may be served better by making
> biblatex your default dialect of bibtex-mode.  (I find it useful if
> bibtex-mode keeps track of the entry types known to a dialect.)
>

I am still missing something... as far as I can tell, the "dialect" is just
controlling which entries are valid. Is that right? But this is not within
the purview of whether we use bibtex, or biber+biblatex. It depends on the
biblio style that the user wants to use for setting their bibliography.


>
> I am currently working on a patch for bibtex-mode that will make it
> easier for users to customize the entry types known by a dialect,
> including the possibility to define aliases for entry types.  This patch
> should be installed on master in a few weeks.  (I want to test it
> first.)
>

I'm happy to hear that there will be future improvements. I sincerely
request that parsing of entries be made more permissive — not restricted to
a list of entry types, or relying on the user to make some customizations
[I think most users are not going to discover that it's possible
to customize this].


>
> PS: My reading of the above thread on stackexchange is that it will not
> make everyone happy if the distinction between old BibTeX and new
> biblatex gets blurred by bibtex-mode.
>

Again I don't understand why bibtex-mode is making a distinction. The
syntax of the .bib is identical whether the user wants to use bibtex or
biber+biblatex.

Best,
Leo

[-- Attachment #2: Type: text/html, Size: 3745 bytes --]

bug#51621: 29.0.50; bibtex.el biblatex "2.1.3 Non-standard Types" support

[Roland Winkler]

On Mon, Dec 02 2024, Leo Stein wrote:
> I really wish this was more permissive. Looking at a .bib file, we
> have no way of knowing the biblio style that it's going to be set
> with. We also have no way of knowing whether the user is going to
> parse it with bibtex or biber.

The user needs to know whether she wants to use a bib file with BibTeX
or biblatex and use entry types these programs can handle.  Bibtex-mode
cannot be blamed for this.

> I am still missing something... as far as I can tell, the "dialect" is
> just controlling which entries are valid. Is that right? But this is
> not within the purview of whether we use bibtex, or biber+biblatex. It
> depends on the biblio style that the user wants to use for setting
> their bibliography.

Beyond the defaults documented for BibTeX and biblatex, you are free to
write your own bst style files, and you are free to customize
bibtex-mode to your liking.  Everything we discuss here refers only to
user options of bibtex-mode.

The defaults of bibtex-mode match the defaults specified in the
documentation of BibTeX and biblatex.  It would be confusing for users
of bibtex-mode to deviate from that.

> I'm happy to hear that there will be future improvements. 

The goal is to facilitate the customization of bibtex-mode.  I see no
reason to change the defaults of user variables.

> I sincerely request that parsing of entries be made more permissive —
> not restricted to a list of entry types, or relying on the user to
> make some customizations [I think most users are not going to discover
> that it's possible to customize this].

It is a basic aspect of Emacs that users can customize its behavior.
But Emacs cannot (yet) read the mind of its users and foresee the
customizations they want.

I have heard rumors that reading the users' mind will become a user
option in Emacs 42.  (But I do not know whether this option will be
enabled by default.)

bug#51621: 29.0.50; bibtex.el biblatex "2.1.3 Non-standard Types" support

[Leo Stein]

[-- Attachment #1: Type: text/plain, Size: 3131 bytes --]

On Tue, Dec 3, 2024 at 2:37 PM Roland Winkler <winkler@gnu.org> wrote:

> On Mon, Dec 02 2024, Leo Stein wrote:
> > I really wish this was more permissive. Looking at a .bib file, we
> > have no way of knowing the biblio style that it's going to be set
> > with. We also have no way of knowing whether the user is going to
> > parse it with bibtex or biber.
>
> The user needs to know whether she wants to use a bib file with BibTeX
> or biblatex and use entry types these programs can handle.  Bibtex-mode
> cannot be blamed for this.
>

We are both re-hashing the same points. I will say again that both bibtex
and biber+biblatex can handle any types of entries. I think the more
flexible approach is to permissive about entry types, and allow
bibtex-parse-entry to parse an entry of any type, not just a fixed list of
default types from (i) btxdoc.pdf, a piece of documentation from 1988; and
(ii) biblatex.pdf, the documentation for a latex package, not the parser,
biber, which indeed allows any entry type.


>
> > I am still missing something... as far as I can tell, the "dialect" is
> > just controlling which entries are valid. Is that right? But this is
> > not within the purview of whether we use bibtex, or biber+biblatex. It
> > depends on the biblio style that the user wants to use for setting
> > their bibliography.
>
> Beyond the defaults documented for BibTeX and biblatex, you are free to
> write your own bst style files, and you are free to customize
> bibtex-mode to your liking.  Everything we discuss here refers only to
> user options of bibtex-mode.
>
> The defaults of bibtex-mode match the defaults specified in the
> documentation of BibTeX and biblatex.  It would be confusing for users
> of bibtex-mode to deviate from that.
>

You don't understand why you say it would be confusing. Just like emacs can
not read users' minds, I don't see how you can be sure about what would be
confusing. There are common entry types, e.g. @software, which are in very
wide use with standard bibtex.


>
> > I'm happy to hear that there will be future improvements.
>
> The goal is to facilitate the customization of bibtex-mode.  I see no
> reason to change the defaults of user variables.
>
> > I sincerely request that parsing of entries be made more permissive —
> > not restricted to a list of entry types, or relying on the user to
> > make some customizations [I think most users are not going to discover
> > that it's possible to customize this].
>
> It is a basic aspect of Emacs that users can customize its behavior.
> But Emacs cannot (yet) read the mind of its users and foresee the
> customizations they want.
>
> I have heard rumors that reading the users' mind will become a user
> option in Emacs 42.  (But I do not know whether this option will be
> enabled by default.)
>

You don't have to try to read my mind — I am trying to make my mind known
to the devs by emailing this list. I continue to strongly request
that bibtex-parse-entry should be more permissive about parsing types than
a predefined list from 1988.

[-- Attachment #2: Type: text/html, Size: 3916 bytes --]