bug#36401: 26.0.90; set-auto-mode uses case-insensitive match for magic-mode-alist

bug#36401: 26.0.90; set-auto-mode uses case-insensitive match for magic-mode-alist

[Benjamin Riefenstahl]

My use case: I use "magic-mode-alist" to detect a binary file type for
which I have written a mode.  The magic at the beginning is "mTFf", so
it is simple ASCII, but unusually capitalized.  When I add this to
"magic-mode-alist", files and buffers that start with "mtff" are also
put in my binary file mode.

It turns out that "set-auto-mode" in lisp/files.el is the function that
uses magic-mode-alist, and it just calls "looking-at" to test the type.
Because "case-fold-search" is "t" by default, this catches files
starting with "mtff".

Is this intentional?  If so, the documentation for "magic-mode-alist"
should call this out, so users know they might need to work around it.
If not, I could propose a patch to fix it.

TIA, benny

-- 

In GNU Emacs 26.0.90 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.14.5)
 of 2017-10-18 built on riefenstahl-linux
Repository revision: b970a4a52a1c04afce4916ca2bf337e2b36d26c4
Windowing system distributor 'The X.Org Foundation', version 11.0.11604000
System Description:	Debian GNU/Linux 8.11 (jessie)

Recent messages:
Mark set
Saving file /home/riefenstahl/.emacs.cfg/local.riefenstahl-linux.el...
Wrote /home/riefenstahl/.emacs.cfg/local.riefenstahl-linux.el
(No changes need to be saved)
Auto-saving...done
Saving file /home/riefenstahl/.emacs.cfg/local.riefenstahl-linux.el...
Wrote /home/riefenstahl/.emacs.cfg/local.riefenstahl-linux.el
(No changes need to be saved) [6 times]
Making completion list...
Type "q" in help window to restore its previous buffer.

Configured features:
XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GSETTINGS NOTIFY
ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB
TOOLKIT_SCROLL_BARS GTK3 X11 LCMS2

Important settings:
  value of $LANG: C.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Emacs-Lisp

Minor modes in effect:
  global-magit-file-mode: t
  magit-file-mode: t
  global-git-commit-mode: t
  async-bytecomp-package-mode: t
  desktop-save-mode: t
  display-time-mode: t
  diff-auto-refine-mode: t
  delete-selection-mode: t
  shell-dirtrack-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
~/site-lisp/css-mode hides /usr/local/share/emacs/26.0.90/lisp/textmodes/css-mode

Features:
(shadow sort mail-extr emacsbug sendmail find-dired thingatpt eieio-opt
speedbar sb-image ezimage dframe find-func ruler-mode hl-line hexl pp
cl-print misearch multi-isearch help-fns arc-mode archive-mode
benny-mtff-mode magit-obsolete magit-blame magit-stash magit-bisect
magit-remote magit-commit magit-sequence magit-notes magit-worktree
magit-tag magit-merge magit-branch magit-reset magit-collab ghub-graphql
ghub url-http tls gnutls url-gw nsm url-auth url url-proxy url-privacy
url-expand url-methods url-history url-cookie url-domsuf url-util
mailcap let-alist json magit-files magit-refs magit-status magit
magit-repos magit-apply magit-wip magit-log magit-diff smerge-mode
magit-core magit-autorevert autorevert filenotify magit-process
magit-margin magit-mode git-commit magit-git magit-section magit-utils
crm subr-x magit-popup log-edit message-x message rmc puny rfc822 mml
mml-sec epa epg gnus-util rmail rmail-loaddefs mm-decode mm-bodies
mm-encode mail-parse rfc2231 rfc2047 rfc2045 ietf-drums mailabbrev
mail-utils gmm-utils mailheader pcvs-util add-log with-editor cl-extra
help-mode async-bytecomp async dash elec-pair dired-aux make-mode
time-date map sh-script smie executable desktop frameset highline
ange-ftp generic-x cl autoinsert ps-print ps-print-loaddefs ps-def lpr
dired dired-loaddefs benny-x-clipboard disp-table mm-util mail-prsvr
time server protbuf solar cal-dst cal-islam cal-hebrew holidays
hol-loaddefs vc-git diff-mode easy-mmode diary-lib diary-loaddefs
cal-menu calendar cal-loaddefs which-func imenu jka-compr grep compile
delsel cus-start cus-load generic benny-calendar-cfg derived cc-mode
cc-fonts cc-guess cc-menus cc-cmds cc-styles cc-align cc-engine cc-vars
cc-defs filecache benny-file-cache tramp-sh docker-tramp tramp-cache
tramp tramp-compat tramp-loaddefs trampver ucs-normalize shell pcomplete
comint ansi-color ring parse-time format-spec advice benny-unicode
benny-tools browse-url autoload radix-tree lisp-mnt finder-inf gh-common
marshal eieio-compat rx info package easymenu epg-config url-handlers
url-parse auth-source cl-seq eieio eieio-core cl-macs eieio-loaddefs
password-cache url-vars seq byte-opt gv bytecomp byte-compile cconv
cl-loaddefs cl-lib mule-util tooltip eldoc electric uniquify ediff-hook
vc-hooks lisp-float-type mwheel term/x-win x-win term/common-win x-dnd
tool-bar dnd fontset image regexp-opt fringe tabulated-list replace
newcomment text-mode elisp-mode lisp-mode prog-mode register page
menu-bar rfn-eshadow isearch timer select scroll-bar mouse jit-lock
font-lock syntax facemenu font-core term/tty-colors frame cl-generic
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite charscript charprop
case-table epa-hook jka-cmpr-hook help simple abbrev obarray minibuffer
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote dbusbind inotify lcms2
dynamic-setting system-font-setting font-render-setting move-toolbar gtk
x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 418674 115456)
 (symbols 48 37479 2)
 (miscs 40 2593 3720)
 (strings 32 93366 4129)
 (string-bytes 1 2851917)
 (vectors 16 66469)
 (vector-slots 8 1822112 223144)
 (floats 8 700 564)
 (intervals 56 6177 3982)
 (buffers 992 36)
 (heap 1024 70067 2662))

bug#36401: 26.0.90; set-auto-mode uses case-insensitive match for magic-mode-alist

[Eli Zaretskii]

> From: Benjamin Riefenstahl <Riefenstahl@mecom.de>
> Date: Thu, 27 Jun 2019 11:30:47 +0200
> 
> It turns out that "set-auto-mode" in lisp/files.el is the function that
> uses magic-mode-alist, and it just calls "looking-at" to test the type.
> Because "case-fold-search" is "t" by default, this catches files
> starting with "mtff".
> 
> Is this intentional?  If so, the documentation for "magic-mode-alist"
> should call this out, so users know they might need to work around it.
> If not, I could propose a patch to fix it.

I think this is an omission.  Please do propose a patch to fix it.

Thanks.

bug#36401: 26.0.90; set-auto-mode uses case-insensitive match for magic-mode-alist

[Benjamin Riefenstahl]

[-- Attachment #1: Type: text/plain, Size: 236 bytes --]

Eli Zaretskii writes:
> I think this is an omission.  Please do propose a patch to fix it.

See attached.  This applies against master, but the conflict against
emacs-26 is also just minor.  Please advise if you want this
differently.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Make-REs-in-magic-fallback-mode-alist-case-sensitive.patch --]
[-- Type: text/x-diff, Size: 5036 bytes --]

From ea6ea194e8858789dce220e28ff33dcb5e313d30 Mon Sep 17 00:00:00 2001
From: Benjamin Riefenstahl <b.riefenstahl@turtle-trading.net>
Date: Sun, 14 Jul 2019 17:09:39 +0200
Subject: [PATCH] Make REs in magic-(fallback-)mode-alist case-sensitive.

These variables are used for well-defined file formats where relaxed
case matching is not wanted usually.

* lisp/files.el (magic-mode-alist, magic-fallback-mode-alist): Update
the doc string.
(set-auto-mode): Make looking-at for elements of magic-mode-alist and
magic-fallback-mode-alist use case-fold-search == nil.
* lisp/files.el (files-test-magic-mode-alist-re-baseline)
(files-test-magic-mode-alist-re-no-match)
(files-test-magic-mode-alist-re-case-diff): Add.
---
 lisp/files.el            | 18 ++++++++++--------
 test/lisp/files-tests.el | 27 +++++++++++++++++++++++++++
 2 files changed, 37 insertions(+), 8 deletions(-)

diff --git a/lisp/files.el b/lisp/files.el
index b2249bfcb4..d97f11c32e 100644
--- a/lisp/files.el
+++ b/lisp/files.el
@@ -2945,9 +2945,9 @@ magic-mode-alist
   "Alist of buffer beginnings vs. corresponding major mode functions.
 Each element looks like (REGEXP . FUNCTION) or (MATCH-FUNCTION . FUNCTION).
 After visiting a file, if REGEXP matches the text at the beginning of the
-buffer, or calling MATCH-FUNCTION returns non-nil, `normal-mode' will
-call FUNCTION rather than allowing `auto-mode-alist' to decide the buffer's
-major mode.
+buffer (respecting case), or calling MATCH-FUNCTION returns non-nil,
+`normal-mode' will call FUNCTION rather than allowing `auto-mode-alist' to
+decide the buffer's major mode.
 
 If FUNCTION is nil, then it is not called.  (That is a way of saying
 \"allow `auto-mode-alist' to decide for these files.\")")
@@ -2979,9 +2979,9 @@ magic-fallback-mode-alist
   "Like `magic-mode-alist' but has lower priority than `auto-mode-alist'.
 Each element looks like (REGEXP . FUNCTION) or (MATCH-FUNCTION . FUNCTION).
 After visiting a file, if REGEXP matches the text at the beginning of the
-buffer, or calling MATCH-FUNCTION returns non-nil, `normal-mode' will
-call FUNCTION, provided that `magic-mode-alist' and `auto-mode-alist'
-have not specified a mode for this file.
+buffer (respecting case), or calling MATCH-FUNCTION returns non-nil,
+`normal-mode' will call FUNCTION, provided that `magic-mode-alist' and
+`auto-mode-alist' have not specified a mode for this file.
 
 If FUNCTION is nil, then it is not called.")
 (put 'magic-fallback-mode-alist 'risky-local-variable t)
@@ -3098,7 +3098,8 @@ set-auto-mode
                              ((functionp re)
                               (funcall re))
                              ((stringp re)
-                              (looking-at re))
+                              (let ((case-fold-search nil))
+                                (looking-at re)))
                              (t
                               (error
                                "Problem in magic-mode-alist with element %s"
@@ -3159,7 +3160,8 @@ set-auto-mode
                                            ((functionp re)
                                             (funcall re))
                                            ((stringp re)
-                                            (looking-at re))
+                                            (let ((case-fold-search nil))
+                                              (looking-at re)))
                                            (t
                                             (error
                                              "Problem with magic-fallback-mode-alist element: %s"
diff --git a/test/lisp/files-tests.el b/test/lisp/files-tests.el
index aa5dbe7acf..df2c3f47ae 100644
--- a/test/lisp/files-tests.el
+++ b/test/lisp/files-tests.el
@@ -1282,5 +1282,32 @@ files-tests-file-attributes-equal
   (should (equal (file-size-human-readable 10000 'si " " "bit") "10 kbit"))
   (should (equal (file-size-human-readable 10000 'iec " " "bit") "9.8 Kibit")))
 
+(ert-deftest files-test-magic-mode-alist-re-baseline ()
+  "Test magic-mode-alist with RE, expected behaviour for match."
+  (let ((magic-mode-alist '(("my-tag" . text-mode))))
+    (with-temp-buffer
+      (insert "my-tag")
+      (normal-mode)
+      (should (eq major-mode 'text-mode)))))
+
+(ert-deftest files-test-magic-mode-alist-re-no-match ()
+  "Test magic-mode-alist with RE, expected behaviour for no match."
+  (let ((magic-mode-alist '(("my-tag" . text-mode))))
+    (with-temp-buffer
+      (insert "not-my-tag")
+      (normal-mode)
+      (should (not (eq major-mode 'text-mode))))))
+
+(ert-deftest files-test-magic-mode-alist-re-case-diff ()
+  "Test that regexps in magic-mode-alist are case-sensitive.
+See <https://debbugs.gnu.org/36401>."
+  (let ((case-fold-search t)
+        (magic-mode-alist '(("my-tag" . text-mode))))
+    (with-temp-buffer
+      (goto-char (point-min))
+      (insert "My-Tag")
+      (normal-mode)
+      (should (not (eq major-mode 'text-mode))))))
+
 (provide 'files-tests)
 ;;; files-tests.el ends here
-- 
2.11.0

bug#36401: 26.0.90; set-auto-mode uses case-insensitive match for magic-mode-alist

[Eli Zaretskii]

> From: Benjamin Riefenstahl <b.riefenstahl@turtle-trading.net>
> Cc: Benjamin Riefenstahl <Riefenstahl@mecom.de>,  36401@debbugs.gnu.org
> Date: Sun, 14 Jul 2019 17:23:57 +0200
> 
> Eli Zaretskii writes:
> > I think this is an omission.  Please do propose a patch to fix it.
> 
> See attached.  This applies against master, but the conflict against
> emacs-26 is also just minor.  Please advise if you want this
> differently.

Thanks, pushed to the master branch.