bug#68477: 29.1; bibtex-braced-string-syntax-table should handle non-ASCII parentheses

bug#68477: 29.1; bibtex-braced-string-syntax-table should handle non-ASCII parentheses

[Joost Kremers]

Consider the following BibTeX entry:

```
@article{Title,
  title = {{Title}},
  author = {Author},
  year = 1970,
  abstract = {（},
  journal = {Journal},
}
```

Note that the `abstract` field has a single opening FULLWIDTH OPENING
PARENTHESIS (code point #xff08), not the (ASCII) LEFT PARENTHESIS. bibtex-mode
cannot parse this entry correctly. (Do, e.g., `C-c C-c` on the entry, bound to
`bibtex-clean-entry` and you'll get a user error "Syntactically incorrect BibTeX
entry starts here".)

If the field contains an unmatched LEFT PARENTHESIS (code point #x28),
bibtex-mode has to trouble parsing the entry.

This is ultimately due to the fact that `bibtex-braced-string-syntax-table`
contains an entry for the ASCII parentheses, setting them to 'punctuation', but
there is no similar entry for non-ASCII parentheses. 




In GNU Emacs 29.1 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.38, cairo
version 1.17.8)
Windowing system distributor 'The X.Org Foundation', version 11.0.12302003
System Description: Arch Linux

Configured using:
 'configure --sysconfdir=/etc --prefix=/usr --libexecdir=/usr/lib
 --with-tree-sitter --localstatedir=/var --with-cairo --disable-build-details
 --with-harfbuzz --with-libsystemd --with-modules --with-x-toolkit=gtk3
 'CFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions
 -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security
 -fstack-clash-protection -fcf-protection -g
 -ffile-prefix-map=/build/emacs/src=/usr/src/debug/emacs -flto=auto'
 'LDFLAGS=-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -flto=auto''
Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG JSON
LCMS2 LIBOTF LIBSYSTEMD LIBXML2 M17N_FLT MODULES NOTIFY INOTIFY PDUMPER PNG RSVG
SECCOMP SOUND SQLITE3 THREADS TIFF TOOLKIT_SCROLL_BARS TREE_SITTER WEBP X11 XDBE
XIM XINPUT2 XPM GTK3 ZLIB
Important settings:
  value of $LANG: en_GB.UTF-8
  locale-coding-system: utf-8-unix

Major mode: ELisp/l

Minor modes in effect:
  TeX-PDF-mode: t
  TeX-source-correlate-mode: t
  display-fill-column-indicator-mode: t
  flycheck-indicator-mode: t
  global-flycheck-eglot-mode: t
  flycheck-posframe-mode: t
  flycheck-mode: t
  eldoc-box-hover-mode: t
  magit-auto-revert-mode: t
  mu4e-modeline-mode: t
  company-box-mode: t
  csv-field-index-mode: t
  minions-mode: t
  doom-modeline-mode: t
  which-key-mode: t
  global-atomic-chrome-edit-mode: t
  marginalia-mode: t
  all-the-icons-completion-mode: t
  company-prescient-mode: t
  prescient-persist-mode: t
  vertico-multiform-mode: t
  vertico-mode: t
  paredit-mode: t
  global-diff-hl-mode: t
  diff-hl-mode: t
  global-git-commit-mode: t
  aggressive-indent-mode: t
  flyspell-mode: t
  outline-minor-mode: t
  shell-dirtrack-mode: t
  hl-line-mode: t
  goggles-mode: t
  company-quickhelp-mode: t
  company-quickhelp-local-mode: t
  global-company-mode: t
  company-mode: t
  server-mode: t
  electric-pair-mode: t
  recentf-mode: t
  delete-selection-mode: t
  override-global-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  buffer-read-only: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  auto-save-visited-mode: t

Load-path shadows:
~/.emacs.d/lisp/custom hides /usr/share/emacs/29.1/lisp/custom
/home/joost/.emacs.d/elpa/transient-20240113.1627/transient hides /usr/share/emacs/29.1/lisp/transient
/home/joost/.emacs.d/elpa/use-package-20230426.2324/use-package hides /usr/share/emacs/29.1/lisp/use-package/use-package
/home/joost/.emacs.d/elpa/use-package-20230426.2324/use-package-lint hides /usr/share/emacs/29.1/lisp/use-package/use-package-lint
/home/joost/.emacs.d/elpa/use-package-20230426.2324/use-package-diminish hides /usr/share/emacs/29.1/lisp/use-package/use-package-diminish
/home/joost/.emacs.d/elpa/bind-key-20230203.2004/bind-key hides /usr/share/emacs/29.1/lisp/use-package/bind-key
/home/joost/.emacs.d/elpa/use-package-20230426.2324/use-package-delight hides /usr/share/emacs/29.1/lisp/use-package/use-package-delight
/home/joost/.emacs.d/elpa/use-package-20230426.2324/use-package-ensure hides /usr/share/emacs/29.1/lisp/use-package/use-package-ensure
/home/joost/.emacs.d/elpa/use-package-20230426.2324/use-package-jump hides /usr/share/emacs/29.1/lisp/use-package/use-package-jump
/home/joost/.emacs.d/elpa/use-package-20230426.2324/use-package-core hides /usr/share/emacs/29.1/lisp/use-package/use-package-core
/home/joost/.emacs.d/elpa/use-package-20230426.2324/use-package-bind-key hides /usr/share/emacs/29.1/lisp/use-package/use-package-bind-key
/home/joost/.emacs.d/elpa/eglot-20221020.1010/eglot hides /usr/share/emacs/29.1/lisp/progmodes/eglot
/home/joost/.emacs.d/elpa/seq-2.24/seq hides /usr/share/emacs/29.1/lisp/emacs-lisp/seq

Features:
(shadow emacsbug vundo latex-mode-expansions auctex-settings latex latex-flymake
tex-ispell tex-style tex texmathp descr-text expand-region yaml-mode-expansions
text-mode-expansions the-org-mode-expansions python-el-fgallina-expansions
er-basic-expansions expand-region-core expand-region-custom edebug shortdoc
org-ebib ebib-settings ebib ebib-reading-list ebib-notes ebib-filters
ebib-keywords ebib-utils ebib-db parsebib tramp-sh view cl-print help-fns
radix-tree display-fill-column-indicator flycheck-indicator flycheck-ledger
flycheck-eglot flycheck-posframe flycheck eldoc-box magit-bookmark
magit-submodule magit-blame magit-stash magit-reflog magit-bisect magit-push
magit-pull magit-fetch magit-clone magit-remote magit-commit magit-sequence
magit-notes magit-worktree magit-tag magit-merge magit-branch magit-reset
magit-files magit-refs magit-status magit magit-repos magit-apply magit-wip
magit-log which-func magit-diff smerge-mode diff magit-core magit-autorevert
magit-margin magit-transient magit-process magit-mode ace-window avy
vertico-buffer consult vc-git jump-char tabify org-autolist org-indent oc-basic
ol-eww eww mm-url ol-rmail ol-mhe ol-irc ol-info ol-gnus nnselect ol-docview
doc-view jka-compr image-mode exif ol-bibtex ol-bbdb ol-w3m ol-doi org-link-doi
org-capture qp smiley gnus-cite mail-extr textsec uni-scripts idna-mapping
ucs-normalize uni-confusable textsec-check visual-fill-column mu4e-settings
gnus-dired mu4e mu4e-org mu4e-notification mu4e-main mu4e-view gnus-art mm-uu
mml2015 mm-view mml-smime smime dig gnus-sum gnus-group gnus-undo gnus-start
gnus-dbus gnus-cloud nnimap nnmail mail-source utf7 nnoo gnus-spec gnus-int
gnus-range gnus-win gnus nnheader range mu4e-headers mu4e-compose mu4e-draft
mu4e-actions smtpmail mu4e-search mu4e-lists mu4e-bookmarks mu4e-mark
mu4e-message flow-fill mu4e-contacts mu4e-update mu4e-folders mu4e-context
mu4e-query-items mu4e-server mu4e-modeline mu4e-vars mu4e-helpers mu4e-config
mu4e-window mu4e-obsolete misearch multi-isearch cus-edit cus-start cus-load
mm-archive mule-util gnutls network-stream url-cache display-line-numbers
jk-input-methods quail company-box company-box-doc frame-local company-box-icons
company-amelia amelia hi-lock go-translate gts-engine-youdao gts-engine-stardict
gts-engine-deepl gts-engine-google-rpc gts-engine-google gts-engine-bing
gts-implements gts-faces gts-core wgrep-ag wgrep csv2ledger csv-mode sort
parse-csv vterm bookmark term disp-table ehelp vterm-module term/xterm xterm
nushell-ts-mode ielm minions doom-modeline doom-modeline-segments
doom-modeline-env doom-modeline-core shrink-path f nerd-icons nerd-icons-faces
nerd-icons-data nerd-icons-data-mdicon nerd-icons-data-flicon
nerd-icons-data-codicon nerd-icons-data-devicon nerd-icons-data-sucicon
nerd-icons-data-wicon nerd-icons-data-faicon nerd-icons-data-powerline
nerd-icons-data-octicon nerd-icons-data-pomicon nerd-icons-data-ipsicon
which-key atomic-chrome iimage image+ image-file image-converter marginalia
all-the-icons-completion company-prescient prescient char-fold orderless
vertico-multiform vertico dockerfile-mode sh-script smie executable
impatient-mode htmlize jupyter python-pytest projectile lisp-mnt grep ibuf-ext
ibuffer ibuffer-loaddefs paredit eglot array jsonrpc ert debug backtrace
flymake-proc flymake thingatpt warnings imenu ag vc-svn compile find-dired s
diff-hl log-view vc-dir ewoc vc vc-dispatcher diff-mode git-commit magit-git
magit-base magit-section cursor-sensor crm with-editor transient log-edit
message yank-media rfc822 mml mml-sec epa derived epg rfc6068 epg-config
gnus-util mm-decode mm-bodies mm-encode mailabbrev gmm-utils mailheader
pcvs-util add-log autorevert filenotify aggressive-indent sendmail mail-utils
nswbuff finder-inf yaml-mode yaml guess-language flyspell ispell reftex
reftex-loaddefs reftex-vars tab-jump-out yasnippet-snippets yasnippet
company-org-block org-settings org-clock ob-jupyter jupyter-tramp tramp-cache
time-stamp jupyter-server jupyter-server-kernel jupyter-rest-api url-http
url-auth mail-parse rfc2231 rfc2047 rfc2045 mm-util ietf-drums mail-prsvr url-gw
nsm jupyter-org-extensions jupyter-org-client jupyter-repl jupyter-widget-client
websocket bindat simple-httpd pp jupyter-client jupyter-kernel jupyter-monads
thunk jupyter-messages hmac-def jupyter-mime shr text-property-search pixel-fill
kinsoku puny svg jupyter-kernelspec jupyter-env jupyter-base eieio-base ob-shell
ob-scheme ob-clojure ob-python python compat treesit org-element org-persist xdg
org-id org-refile avl-tree dom org ob ob-tangle ob-ref ob-lob ob-table ob-exp
org-macro org-src ob-comint org-pcomplete org-list org-footnote org-faces
org-entities noutline outline icons ob-emacs-lisp ob-core ob-eval org-cycle
org-table ol org-fold org-fold-core org-keys oc org-loaddefs find-func cal-menu
calendar cal-loaddefs org-version org-compat org-macs denote dired
dired-loaddefs tramp tramp-loaddefs trampver tramp-integration tramp-compat rx
shell pcomplete comint ansi-osc ansi-color parse-time format-spec mixed-pitch
face-remap biblio biblio-download biblio-dissemin biblio-ieee biblio-hal
biblio-dblp biblio-crossref biblio-arxiv timezone biblio-doi biblio-core
let-alist url-queue url-file ido hl-line bibtex iso8601 time-date adaptive-wrap
goggles pulse posframe hydra lv company-quickhelp pos-tip all-the-icons
all-the-icons-faces data-material data-weathericons data-octicons data-fileicons
data-faicons data-alltheicons company-keywords company-etags etags fileloop xref
project ring company-gtags company-dabbrev-code company-dabbrev company-ipa
company-files company-clang company-cmake company-semantic company-template
company-css company-capf company solarized-light-theme solarized-theme solarized
solarized-faces color whitespace jk-functions advice dash eshell esh-cmd
generator esh-ext esh-opt esh-proc esh-io esh-arg esh-module esh-groups esh-util
files-x notifications dbus xml cl-extra help-mode edmacro kmacro server
elec-pair recentf tree-widget wid-edit delsel use-package use-package-ensure
use-package-delight use-package-diminish use-package-bind-key bind-key
use-package-core all-the-icons-completion-autoloads all-the-icons-autoloads
apheleia-autoloads easy-mmode async-autoloads avy-autoloads
clojure-mode-autoloads auctex-autoloads tex-site company-box-autoloads
company-prescient-autoloads company-quickhelp-autoloads denote-autoloads
devdocs-browser-autoloads diff-hl-autoloads docker-autoloads
doom-modeline-autoloads eldoc-box-autoloads embark-consult-autoloads
consult-autoloads embark-autoloads expand-region-autoloads
flycheck-eglot-autoloads go-translate-autoloads goggles-autoloads
groovy-mode-autoloads hydra-autoloads impatient-mode-autoloads company-autoloads
js2-mode-autoloads json-process-client-autoloads jsonian-autoloads
jupyter-autoloads ledger-mode-autoloads lv-autoloads magit-autoloads
git-commit-autoloads magit-section-autoloads marginalia-autoloads
markdown-mode-autoloads minions-autoloads multiple-cursors-autoloads
nerd-icons-autoloads numpydoc-autoloads nushell-ts-mode-autoloads
orderless-autoloads org-modern-autoloads pdf-tools-autoloads popup-autoloads
pos-tip-autoloads posframe-autoloads prescient-autoloads python-pytest-autoloads
projectile-autoloads f-autoloads simple-httpd-autoloads sly-overlay-autoloads
sly-autoloads solarized-theme-autoloads tab-jump-out-autoloads tablist-autoloads
tide-autoloads flycheck-autoloads dash-autoloads transient-autoloads
use-package-autoloads vertico-autoloads vterm-autoloads vundo-autoloads
web-mode-autoloads websocket-autoloads which-key-autoloads with-editor-autoloads
info compat-autoloads seq-autoloads yaml-autoloads yaml-mode-autoloads
yasnippet-snippets-autoloads yasnippet-autoloads zmq-autoloads package
browse-url url url-proxy url-privacy url-expand url-methods url-history
url-cookie generate-lisp-file url-domsuf url-util mailcap url-handlers url-parse
auth-source cl-seq eieio eieio-core cl-macs password-cache json subr-x map
byte-opt gv bytecomp byte-compile url-vars cl-loaddefs cl-lib pcase rmc
iso-transl tooltip cconv eldoc paren electric uniquify ediff-hook vc-hooks
lisp-float-type elisp-mode mwheel term/x-win x-win term/common-win x-dnd
tool-bar dnd fontset image regexp-opt fringe tabulated-list replace newcomment
text-mode lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow isearch
easymenu timer select scroll-bar mouse jit-lock font-lock syntax font-core
term/tty-colors frame minibuffer nadvice seq simple cl-generic indonesian
philippine cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet
lao korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese composite emoji-zwj charscript charprop
case-table epa-hook jka-cmpr-hook help abbrev obarray oclosure cl-preloaded
button loaddefs theme-loaddefs faces cus-face macroexp files window
text-properties overlay sha1 md5 base64 format env code-pages mule custom widget
keymap hashtable-print-readable backquote threads dbusbind inotify lcms2
dynamic-setting system-font-setting font-render-setting cairo move-toolbar gtk
x-toolkit xinput2 x multi-tty make-network-process emacs)

Memory information:
((conses 16 2653847 352543)
 (symbols 48 76261 35)
 (strings 32 534865 15540)
 (string-bytes 1 15506595)
 (vectors 16 180209)
 (vector-slots 8 3582941 292982)
 (floats 8 2017 1459)
 (intervals 56 255629 6319)
 (buffers 984 50))

-- 
Joost Kremers
Life has its moments

bug#68477: 29.1; bibtex-braced-string-syntax-table should handle non-ASCII parentheses

[Roland Winkler]

On Mon, Jan 15 2024, Joost Kremers wrote:
> Consider the following BibTeX entry:
>
> ```
> @article{Title,
>   title = {{Title}},
>   author = {Author},
>   year = 1970,
>   abstract = {（},
>   journal = {Journal},
> }
> ```
>
> Note that the `abstract` field has a single opening FULLWIDTH OPENING
> PARENTHESIS (code point #xff08), not the (ASCII) LEFT
> PARENTHESIS. bibtex-mode cannot parse this entry correctly. (Do, e.g.,
> `C-c C-c` on the entry, bound to `bibtex-clean-entry` and you'll get a
> user error "Syntactically incorrect BibTeX entry starts here".)
>
> If the field contains an unmatched LEFT PARENTHESIS (code point #x28),
> bibtex-mode has no trouble parsing the entry.
>
> This is ultimately due to the fact that
> `bibtex-braced-string-syntax-table` contains an entry for the ASCII
> parentheses, setting them to 'punctuation', but there is no similar
> entry for non-ASCII parentheses.

I only discovered this bug report now.  I believe utf-8 defines lots of
pairs of parentheses to which this bug report applies.  Is there a
generic way to modify the syntax of "all" pairs of parentheses (except
the curly brackets "{" and "}") to punctuation so that parsing will
ignore such unpaired parentheses?

(Though, I do not know how common a problem are such unpaired non-ascii
parentheses in BibTeX files.)

bug#68477: 29.1; bibtex-braced-string-syntax-table should handle non-ASCII parentheses

[Eli Zaretskii]

> Cc: 68477@debbugs.gnu.org
> From: Roland Winkler <winkler@gnu.org>
> Date: Thu, 02 Jan 2025 13:09:41 -0600
> 
> > Note that the `abstract` field has a single opening FULLWIDTH OPENING
> > PARENTHESIS (code point #xff08), not the (ASCII) LEFT
> > PARENTHESIS. bibtex-mode cannot parse this entry correctly. (Do, e.g.,
> > `C-c C-c` on the entry, bound to `bibtex-clean-entry` and you'll get a
> > user error "Syntactically incorrect BibTeX entry starts here".)
> >
> > If the field contains an unmatched LEFT PARENTHESIS (code point #x28),
> > bibtex-mode has no trouble parsing the entry.
> >
> > This is ultimately due to the fact that
> > `bibtex-braced-string-syntax-table` contains an entry for the ASCII
> > parentheses, setting them to 'punctuation', but there is no similar
> > entry for non-ASCII parentheses.
> 
> I only discovered this bug report now.  I believe utf-8 defines lots of
> pairs of parentheses to which this bug report applies.  Is there a
> generic way to modify the syntax of "all" pairs of parentheses (except
> the curly brackets "{" and "}") to punctuation so that parsing will
> ignore such unpaired parentheses?

Emacs should already set up the syntax of those characters, see
char-syntax.  Is that what you were looking for?

bug#68477: 29.1; bibtex-braced-string-syntax-table should handle non-ASCII parentheses

[Roland Winkler]

On Thu, Jan 02 2025, Eli Zaretskii wrote:
>> Cc: 68477@debbugs.gnu.org
>> From: Roland Winkler <winkler@gnu.org>
>> Date: Thu, 02 Jan 2025 13:09:41 -0600
>>
>> I only discovered this bug report now.  I believe utf-8 defines lots of
>> pairs of parentheses to which this bug report applies.  Is there a
>> generic way to modify the syntax of "all" pairs of parentheses (except
>> the curly brackets "{" and "}") to punctuation so that parsing will
>> ignore such unpaired parentheses?
>
> Emacs should already set up the syntax of those characters, see
> char-syntax.  Is that what you were looking for?

Here the goal is the other way round: Emacs should treat these extra
parentheses not as parentheses, but as punctuation, so that imbalanced
parentheses do not break the parsing of BibTeX fields that only care
about "{" and "}", and nothing else.

bug#68477: 29.1; bibtex-braced-string-syntax-table should handle non-ASCII parentheses

[Eli Zaretskii]

> From: Roland Winkler <winkler@gnu.org>
> Cc: joostkremers@fastmail.fm,  68477@debbugs.gnu.org
> Date: Thu, 02 Jan 2025 14:58:45 -0600
> 
> On Thu, Jan 02 2025, Eli Zaretskii wrote:
> >> Cc: 68477@debbugs.gnu.org
> >> From: Roland Winkler <winkler@gnu.org>
> >> Date: Thu, 02 Jan 2025 13:09:41 -0600
> >>
> >> I only discovered this bug report now.  I believe utf-8 defines lots of
> >> pairs of parentheses to which this bug report applies.  Is there a
> >> generic way to modify the syntax of "all" pairs of parentheses (except
> >> the curly brackets "{" and "}") to punctuation so that parsing will
> >> ignore such unpaired parentheses?
> >
> > Emacs should already set up the syntax of those characters, see
> > char-syntax.  Is that what you were looking for?
> 
> Here the goal is the other way round: Emacs should treat these extra
> parentheses not as parentheses, but as punctuation, so that imbalanced
> parentheses do not break the parsing of BibTeX fields that only care
> about "{" and "}", and nothing else.

Please elaborate on "treat these extra parentheses as punctuation".

Also, does map-char-table provide you with the means to do what is
needed for bibtex?  Syntax table is a char-table, so if you want to do
something with all the characters of a certain syntax, you can do that
using map-char-table, I think.

bug#68477: 29.1; bibtex-braced-string-syntax-table should handle non-ASCII parentheses

[Roland Winkler]

On Fri, Jan 03 2025, Eli Zaretskii wrote:
> Please elaborate on "treat these extra parentheses as punctuation".

bibtex-braced-string-syntax-table gives curly braces the syntax of
parenthesis characters using

    (modify-syntax-entry ?\{ "(}" st)
    (modify-syntax-entry ?\} "){" st)

but square and round parentheses are treated as punctuation using

    (modify-syntax-entry ?\[ "." st)
    (modify-syntax-entry ?\] "." st)
    (modify-syntax-entry ?\( "." st)
    (modify-syntax-entry ?\) "." st)

The goal is to do the latter with all non-ascii parentheses.

> Also, does map-char-table provide you with the means to do what is
> needed for bibtex?  Syntax table is a char-table, so if you want to do
> something with all the characters of a certain syntax, you can do that
> using map-char-table, I think.

Thanks, yes, map-char-table should be exactly what is needed for
bibtex-braced-string-syntax-table: for any non-ascii character where
char-syntax says it belongs to the syntax class of parenthesis
characters, its syntax needs to be changed to punctuation.

(When I was looking for a solution I had checked the section on syntax
tables in the elisp manual.  But map-char-table is obviously in the
section on character tables.)