bug#5845: load-library vs. list-load-path-shadows

bug#5845: load-library vs. list-load-path-shadows

[Juanma Barranquero]

[Forward of http://lists.gnu.org/archive/html/emacs-devel/2010-04/msg00182.html]


(This is on Windows, so case insensitive search, etc.)

I installed color-theme in my site-lisp, and then

  M-x load-library <RET> authors <RET>

fails because it finds site-lisp/color-theme/AUTHORS instead of
lisp/emacs-lisp/authors.el[c], which is or isn't a bug (I think it's
not, given that `load-library' clearly says that "LIBRARY is searched
[...] both with and without `load-suffixes'").

But at least `list-load-path-shadows' should use the same heuristics,
so it could point out the problem...

   Juanma

bug#5845: load-library vs. list-load-path-shadows

[Leo]

On 2010-04-06 14:36 +0100, Juanma Barranquero wrote:
> [Forward of http://lists.gnu.org/archive/html/emacs-devel/2010-04/msg00182.html]
>
>
> (This is on Windows, so case insensitive search, etc.)
>
> I installed color-theme in my site-lisp, and then
>
>   M-x load-library <RET> authors <RET>
>
> fails because it finds site-lisp/color-theme/AUTHORS instead of
> lisp/emacs-lisp/authors.el[c], which is or isn't a bug (I think it's
> not, given that `load-library' clearly says that "LIBRARY is searched
> [...] both with and without `load-suffixes'").
>
> But at least `list-load-path-shadows' should use the same heuristics,
> so it could point out the problem...
>
>    Juanma

I wonder if it would be better not to include files with no extensions.
Quite a few packages include things like README/ChangeLog.

Leo

bug#5845: load-library vs. list-load-path-shadows

[Juanma Barranquero]

> I wonder if it would be better not to include files with no extensions.

`load' is used to load ~/.emacs

    Juanma

bug#5845: load-library vs. list-load-path-shadows

[Noam Postavsky]

[-- Attachment #1: Type: text/plain, Size: 1054 bytes --]

Should we just use case-insensitive compare for `windows-nt' systems?
The idea of using "the same heuristics" as `load' sounds nice, but as
far as I can tell, `load' just queries the file system directly. Doing
the same in `load-path-shadow-find' is far too slow (raises execution
time from 0.38s to 5.64s, and that's without any packages adding to
the load-path).

--- i/lisp/emacs-lisp/shadow.el
+++ w/lisp/emacs-lisp/shadow.el
@@ -123,7 +123,9 @@ load-path-shadows-find
         ;; XXX.elc (or vice-versa) when they are in the same directory.
         (setq files-seen-this-dir (cons file files-seen-this-dir))

-        (if (setq orig-dir (assoc file files))
+        (if (setq orig-dir (assoc file files
+                                      (if (memq system-type
'(windows-nt ms-dos))
+                      (lambda (f1 f2) (eq (compare-strings f1 nil nil
f2 nil nil t) t)))))
         ;; This file was seen before, we have a shadowing.
         ;; Report it unless the files are identical.
         (let ((base1 (concat (cdr orig-dir) "/" file))

[-- Attachment #2: naive-shadow-list.el --]
[-- Type: application/octet-stream, Size: 6651 bytes --]

;;; -*- lexical-binding: t -*-

(require 'shadow)

(when nil
  (naive-load-path-shadows-find)
  '("c:/Users/npostavs/src/emacs/bug-5845-load-path-case-shadow/AUTHORS"
   "c:/emacs-24.5/share/emacs/24.5/lisp/emacs-lisp/AUTHORS.elc")

  (load-path-shadows-find)
  nil
  (load-path-shadows-find-v2)
  '("c:/Users/npostavs/src/emacs/emacs-26/admin/AUTHORS"
   "c:/Users/npostavs/src/emacs/bug-5845-load-path-case-shadow/AUTHORS")
  
  (benchmark 1 '(load-path-shadows-find-v2))
  "Elapsed time: 0.381508s (0.028689s in 1 GCs)"

  (benchmark 1 '(naive-load-path-shadows-find))
  "Elapsed time: 5.637040s (0.825316s in 29 GCs)"

  (benchmark 1 '(load-path-shadows-find))
  "Elapsed time: 0.122000s (0.016000s in 2 GCs)")

(defun load-path-shadows-find-v2 (&optional path)
  "Return a list of Emacs Lisp files that create shadows.
This function does the work for `list-load-path-shadows'.

We traverse PATH looking for shadows, and return a \(possibly empty)
even-length list of files.  A file in this list at position 2i shadows
the file in position 2i+1.  Emacs Lisp file suffixes \(.el and .elc)
are stripped from the file names in the list.

See the documentation for `list-load-path-shadows' for further information."
  (let (true-names			; List of dirs considered.
	shadows				; List of shadowings, to be returned.
	files				; File names ever seen, with dirs.
	dir				; The dir being currently scanned.
	curr-files			; This dir's Emacs Lisp files.
	orig-dir			; Where the file was first seen.
	files-seen-this-dir		; Files seen so far in this dir.
	)				; The current file.
    (dolist (pp (or path load-path))
      (setq dir (directory-file-name (file-truename (or pp "."))))
      (if (member dir true-names)
	  ;; We have already considered this PATH redundant directory.
	  ;; Show the redundancy if we are interactive, unless the PATH
	  ;; dir is nil or "." (these redundant directories are just a
	  ;; result of the current working directory, and are therefore
	  ;; not always redundant).
	  (or noninteractive
	      (and pp
		   (not (string= pp "."))
		   (message "Ignoring redundant directory %s" pp)))

	(setq true-names (append true-names (list dir)))
	(setq dir (directory-file-name (or pp ".")))
	(setq curr-files (if (file-accessible-directory-p dir)
			     (directory-files dir nil ".\\.elc?\\(\\.gz\\)?$" t)))
	(and curr-files
	     (not noninteractive)
	     (message "Checking %d files in %s..." (length curr-files) dir))

	(setq files-seen-this-dir nil)

	(dolist (file curr-files)

	  (if (string-match "\\.gz$" file)
	      (setq file (substring file 0 -3)))
	  (setq file (substring
		      file 0 (if (string= (substring file -1) "c") -4 -3)))

	  ;; FILE now contains the current file name, with no suffix.
	  (unless (or (member file files-seen-this-dir)
		      ;; Ignore these files.
		      (member file (list "subdirs" "leim-list"
					 (file-name-sans-extension
					  dir-locals-file))))
	    ;; File has not been seen yet in this directory.
	    ;; This test prevents us declaring that XXX.el shadows
	    ;; XXX.elc (or vice-versa) when they are in the same directory.
	    (setq files-seen-this-dir (cons file files-seen-this-dir))

	    (if (setq orig-dir (assoc file files
				      (if (memq system-type '(windows-nt ms-dos))
					  (lambda (f1 f2) (eq (compare-strings f1 nil nil f2 nil nil t) t)))))
						   
		;; This file was seen before, we have a shadowing.
		;; Report it unless the files are identical.
		(let ((base1 (concat (cdr orig-dir) "/" file))
		      (base2 (concat dir "/" file)))
		  (if (not (and load-path-shadows-compare-text
				(load-path-shadows-same-file-or-nonexistent
				 (concat base1 ".el") (concat base2 ".el"))
				;; This is a bit strict, but safe.
				(load-path-shadows-same-file-or-nonexistent
				 (concat base1 ".elc") (concat base2 ".elc"))))
		      (setq shadows
			    (append shadows (list base1 base2)))))

	      ;; Not seen before, add it to the list of seen files.
	      (setq files (cons (cons file dir) files)))))))
    ;; Return the list of shadowings.
    shadows))

(defun naive-load-path-shadows-find (&optional path)
  "Return a list of Emacs Lisp files that create shadows.
This function does the work for `list-load-path-shadows'.

We traverse PATH looking for shadows, and return a \(possibly empty\)
even-length list of files.  A file in this list at position 2i shadows
the file in position 2i+1.  Emacs Lisp file suffixes \(.el and .elc\)
are stripped from the file names in the list.

See the documentation for `list-load-path-shadows' for further information."
  (let ((true-names nil)
        (path (or path load-path))
        (shadows nil))
    (while (cdr path)
      (let* ((pp (pop path))
             (dir (directory-file-name (file-truename (or pp "."))))
             curr-files)
        (if (member dir true-names)
            ;; We have already considered this PATH redundant directory.
            ;; Show the redundancy if we are interactive, unless the PATH
            ;; dir is nil or "." (these redundant directories are just a
            ;; result of the current working directory, and are therefore
            ;; not always redundant).
            (or noninteractive
                (and pp
                     (not (string= pp "."))
                     (message "Ignoring redundant directory %s" pp)))

          (setq true-names (append true-names (list dir)))
          (setq dir (directory-file-name (or pp ".")))
          (setq curr-files (if (file-accessible-directory-p dir)
                               (directory-files dir nil ".\\.elc?\\(\\.gz\\)?$" t)))
          (and curr-files
               (not noninteractive)
               (message "Checking %d files in %s..." (length curr-files) dir))

          (dolist (file curr-files)

            (if (string-match "\\.gz$" file)
                (setq file (substring file 0 -3)))
            (setq file (substring
                        file 0 (if (string= (substring file -1) "c") -4 -3)))

            ;; FILE now contains the current file name, with no suffix.
            (unless
                ;; Ignore these files.
                (member file (list "subdirs" "leim-list"
                                   (file-name-sans-extension
                                    dir-locals-file)))
              (let ((shadowed (locate-library file nil path)))
                (when shadowed
                  (setq shadows (append shadows (list (concat pp "/" file) shadowed))))))))))
    shadows))

bug#5845: load-library vs. list-load-path-shadows

[Ken Brown]

On 1/16/2018 10:54 AM, Noam Postavsky wrote:
> Should we just use case-insensitive compare for `windows-nt' systems?
> The idea of using "the same heuristics" as `load' sounds nice, but as
> far as I can tell, `load' just queries the file system directly. Doing
> the same in `load-path-shadow-find' is far too slow (raises execution
> time from 0.38s to 5.64s, and that's without any packages adding to
> the load-path).

Would it help to use file-name-case-insensitive-p?

Ken

bug#5845: load-library vs. list-load-path-shadows

[Noam Postavsky]

[-- Attachment #1: Type: text/plain, Size: 603 bytes --]

On Tue, Jan 16, 2018 at 12:25 PM, Ken Brown <kbrown@cornell.edu> wrote:
> On 1/16/2018 10:54 AM, Noam Postavsky wrote:
>>
>> Should we just use case-insensitive compare for `windows-nt' systems?
>> The idea of using "the same heuristics" as `load' sounds nice, but as
>> far as I can tell, `load' just queries the file system directly. Doing
>> the same in `load-path-shadow-find' is far too slow (raises execution
>> time from 0.38s to 5.64s, and that's without any packages adding to
>> the load-path).
>
>
> Would it help to use file-name-case-insensitive-p?

Ah, good point. How about the attached?

[-- Attachment #2: v2-0001-Handle-case-insensitive-filenames-for-load-path-s.patch --]
[-- Type: application/octet-stream, Size: 2060 bytes --]

From acbdb32b0a1973ad3f9c361748d9aa3c461e2449 Mon Sep 17 00:00:00 2001
From: Noam Postavsky <npostavs@gmail.com>
Date: Tue, 16 Jan 2018 16:26:56 -0500
Subject: [PATCH v2] Handle case-insensitive filenames for load-path shadows
 (Bug#5845)

* lisp/emacs-lisp/shadow.el (load-path-shadows-find): Check for
shadowing with case-insensitive matching for files of case-insensitive
directories (as determined by `file-name-case-insentive-p').
---
 lisp/emacs-lisp/shadow.el | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/lisp/emacs-lisp/shadow.el b/lisp/emacs-lisp/shadow.el
index 88a494f..6402695 100644
--- a/lisp/emacs-lisp/shadow.el
+++ b/lisp/emacs-lisp/shadow.el
@@ -78,6 +78,7 @@ load-path-shadows-find
 	shadows				; List of shadowings, to be returned.
 	files				; File names ever seen, with dirs.
 	dir				; The dir being currently scanned.
+	dir-case-sensitive              ; `file-name-case-insentive-p' for dir.
 	curr-files			; This dir's Emacs Lisp files.
 	orig-dir			; Where the file was first seen.
 	files-seen-this-dir		; Files seen so far in this dir.
@@ -104,6 +105,9 @@ load-path-shadows-find
 	     (message "Checking %d files in %s..." (length curr-files) dir))
 
 	(setq files-seen-this-dir nil)
+        ;; We assume that case sensitivity of a directory applies to
+        ;; its files.
+        (setq dir-case-sensitive (file-name-case-sensitive-p dir))
 
 	(dolist (file curr-files)
 
@@ -123,7 +127,9 @@ load-path-shadows-find
 	    ;; XXX.elc (or vice-versa) when they are in the same directory.
 	    (setq files-seen-this-dir (cons file files-seen-this-dir))
 
-	    (if (setq orig-dir (assoc file files))
+            (if (setq orig-dir (assoc file files
+                                      (unless dir-case-sensitive
+                                        (lambda (f1 f2) (eq (compare-strings f1 nil nil f2 nil nil t) t)))))
 		;; This file was seen before, we have a shadowing.
 		;; Report it unless the files are identical.
 		(let ((base1 (concat (cdr orig-dir) "/" file))
-- 
2.6.2.windows.1

bug#5845: load-library vs. list-load-path-shadows

[Eli Zaretskii]

> From: Noam Postavsky <npostavs@users.sourceforge.net>
> Date: Tue, 16 Jan 2018 16:32:11 -0500
> Cc: 5845@debbugs.gnu.org
> 
> > Would it help to use file-name-case-insensitive-p?
> 
> Ah, good point. How about the attached?

LGTM, thanks.  Could we have a test for this issue?

bug#5845: load-library vs. list-load-path-shadows

[Noam Postavsky]

[-- Attachment #1: Type: text/plain, Size: 195 bytes --]

On Wed, Jan 17, 2018 at 10:37 AM, Eli Zaretskii <eliz@gnu.org> wrote:

> LGTM, thanks.  Could we have a test for this issue?

Yes, good idea, caught some silly typos. Should this go to emacs-26?

[-- Attachment #2: v3-0001-Handle-case-insensitive-filenames-for-load-path-s.patch --]
[-- Type: application/octet-stream, Size: 5901 bytes --]

From ac53788b72ba5b9b4a86dcfc12157563650ea6da Mon Sep 17 00:00:00 2001
From: Noam Postavsky <npostavs@gmail.com>
Date: Tue, 16 Jan 2018 16:26:56 -0500
Subject: [PATCH v3] Handle case-insensitive filenames for load-path shadows
 (Bug#5845)

* lisp/emacs-lisp/shadow.el (load-path-shadows-find): Check for
shadowing with case-insensitive matching for files of case-insensitive
directories (as determined by `file-name-case-insensitive-p').
* test/lisp/emacs-lisp/shadow-tests.el: New test.
* test/lisp/emacs-lisp/shadow-resources/p1/foo.el:
* test/lisp/emacs-lisp/shadow-resources/p2/FOO.el: New test files.
---
 lisp/emacs-lisp/shadow.el                       | 10 ++++-
 test/lisp/emacs-lisp/shadow-resources/p1/foo.el |  1 +
 test/lisp/emacs-lisp/shadow-resources/p2/FOO.el |  1 +
 test/lisp/emacs-lisp/shadow-tests.el            | 49 +++++++++++++++++++++++++
 4 files changed, 59 insertions(+), 2 deletions(-)
 create mode 100644 test/lisp/emacs-lisp/shadow-resources/p1/foo.el
 create mode 100644 test/lisp/emacs-lisp/shadow-resources/p2/FOO.el
 create mode 100644 test/lisp/emacs-lisp/shadow-tests.el

diff --git a/lisp/emacs-lisp/shadow.el b/lisp/emacs-lisp/shadow.el
index 88a494f..1788f0d 100644
--- a/lisp/emacs-lisp/shadow.el
+++ b/lisp/emacs-lisp/shadow.el
@@ -78,6 +78,7 @@ load-path-shadows-find
 	shadows				; List of shadowings, to be returned.
 	files				; File names ever seen, with dirs.
 	dir				; The dir being currently scanned.
+        dir-case-insensitive            ; `file-name-case-insentive-p' for dir.
 	curr-files			; This dir's Emacs Lisp files.
 	orig-dir			; Where the file was first seen.
 	files-seen-this-dir		; Files seen so far in this dir.
@@ -104,6 +105,9 @@ load-path-shadows-find
 	     (message "Checking %d files in %s..." (length curr-files) dir))
 
 	(setq files-seen-this-dir nil)
+        ;; We assume that case sensitivity of a directory applies to
+        ;; its files.
+        (setq dir-case-insensitive (file-name-case-insensitive-p dir))
 
 	(dolist (file curr-files)
 
@@ -123,10 +127,12 @@ load-path-shadows-find
 	    ;; XXX.elc (or vice-versa) when they are in the same directory.
 	    (setq files-seen-this-dir (cons file files-seen-this-dir))
 
-	    (if (setq orig-dir (assoc file files))
+            (if (setq orig-dir (assoc file files
+                                      (when dir-case-insensitive
+                                        (lambda (f1 f2) (eq (compare-strings f1 nil nil f2 nil nil t) t)))))
 		;; This file was seen before, we have a shadowing.
 		;; Report it unless the files are identical.
-		(let ((base1 (concat (cdr orig-dir) "/" file))
+                (let ((base1 (concat (cdr orig-dir) "/" (car orig-dir)))
 		      (base2 (concat dir "/" file)))
 		  (if (not (and load-path-shadows-compare-text
 				(load-path-shadows-same-file-or-nonexistent
diff --git a/test/lisp/emacs-lisp/shadow-resources/p1/foo.el b/test/lisp/emacs-lisp/shadow-resources/p1/foo.el
new file mode 100644
index 0000000..465038b
--- /dev/null
+++ b/test/lisp/emacs-lisp/shadow-resources/p1/foo.el
@@ -0,0 +1 @@
+;;; This file intentionally left blank.
diff --git a/test/lisp/emacs-lisp/shadow-resources/p2/FOO.el b/test/lisp/emacs-lisp/shadow-resources/p2/FOO.el
new file mode 100644
index 0000000..465038b
--- /dev/null
+++ b/test/lisp/emacs-lisp/shadow-resources/p2/FOO.el
@@ -0,0 +1 @@
+;;; This file intentionally left blank.
diff --git a/test/lisp/emacs-lisp/shadow-tests.el b/test/lisp/emacs-lisp/shadow-tests.el
new file mode 100644
index 0000000..9d4969f
--- /dev/null
+++ b/test/lisp/emacs-lisp/shadow-tests.el
@@ -0,0 +1,49 @@
+;;; shadow-tests.el --- Test suite for shadow.  -*- lexical-binding: t -*-
+
+;; Copyright (C) 2018 Free Software Foundation, Inc.
+
+;; This file is part of GNU Emacs.
+
+;; GNU Emacs is free software: you can redistribute it and/or modify
+;; it under the terms of the GNU General Public License as published by
+;; the Free Software Foundation, either version 3 of the License, or
+;; (at your option) any later version.
+
+;; GNU Emacs is distributed in the hope that it will be useful,
+;; but WITHOUT ANY WARRANTY; without even the implied warranty of
+;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;; GNU General Public License for more details.
+
+;; You should have received a copy of the GNU General Public License
+;; along with GNU Emacs.  If not, see <https://www.gnu.org/licenses/>.
+
+;;; Code:
+
+(require 'ert)
+(require 'shadow)
+(eval-when-compile (require 'cl-lib))
+
+(defconst shadow-tests-data-directory
+  (expand-file-name "lisp/emacs-lisp/shadow-resources"
+                    (or (getenv "EMACS_TEST_DIRECTORY")
+                        (expand-file-name "../../.."
+                                          (or load-file-name
+                                              buffer-file-name))))
+  "Directory for shadow test files.")
+
+(ert-deftest shadow-case-insensitive ()
+  "Test shadowing for case insensitive filenames."
+  ;; Override `file-name-case-insentive-p' so we test the same thing
+  ;; regardless of what file system we're running on.
+  (cl-letf (((symbol-function 'file-name-case-insensitive-p) (lambda (_f) t)))
+    (should (equal (list (expand-file-name "p1/foo" shadow-tests-data-directory)
+                         (expand-file-name "p2/FOO" shadow-tests-data-directory))
+                   (load-path-shadows-find
+                    (list (expand-file-name "p1/" shadow-tests-data-directory)
+                          (expand-file-name "p2/" shadow-tests-data-directory))))))
+  (cl-letf (((symbol-function 'file-name-case-insensitive-p) (lambda (_f) nil)))
+    (should-not (load-path-shadows-find
+                 (list (expand-file-name "p1/" shadow-tests-data-directory)
+                       (expand-file-name "p2/" shadow-tests-data-directory))))))
+
+;;; shadow-tests.el ends here.
-- 
2.6.2.windows.1

bug#5845: load-library vs. list-load-path-shadows

[Eli Zaretskii]

> From: Noam Postavsky <npostavs@users.sourceforge.net>
> Date: Thu, 18 Jan 2018 12:05:48 -0500
> Cc: Ken Brown <kbrown@cornell.edu>, 5845@debbugs.gnu.org
> 
> > LGTM, thanks.  Could we have a test for this issue?
> 
> Yes, good idea, caught some silly typos. Should this go to emacs-26?

Not sure.  The issue doesn't sound urgent/important, what with its
being unsolved for such a long time.  OTOH, the change is simple and
localized.  WDYT?

bug#5845: load-library vs. list-load-path-shadows

[Noam Postavsky]

On Thu, Jan 18, 2018 at 1:51 PM, Eli Zaretskii <eliz@gnu.org> wrote:

>> Should this go to emacs-26?
>
> Not sure.  The issue doesn't sound urgent/important, what with its
> being unsolved for such a long time.  OTOH, the change is simple and
> localized.  WDYT?

Agreed on both points. The simpleness of the fix outweighs the urgency
for me I think, so I'm leaning to emacs-26.

bug#5845: load-library vs. list-load-path-shadows

[Eli Zaretskii]

> From: Noam Postavsky <npostavs@users.sourceforge.net>
> Date: Thu, 18 Jan 2018 14:39:57 -0500
> Cc: Ken Brown <kbrown@cornell.edu>, 5845@debbugs.gnu.org
> 
> On Thu, Jan 18, 2018 at 1:51 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> >> Should this go to emacs-26?
> >
> > Not sure.  The issue doesn't sound urgent/important, what with its
> > being unsolved for such a long time.  OTOH, the change is simple and
> > localized.  WDYT?
> 
> Agreed on both points. The simpleness of the fix outweighs the urgency
> for me I think, so I'm leaning to emacs-26.

OK.

bug#5845: load-library vs. list-load-path-shadows

[Noam Postavsky]

tag 5845 fixed
close 5845 26.1
quit

On Thu, Jan 18, 2018 at 3:46 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Noam Postavsky <npostavs@users.sourceforge.net>
>> Date: Thu, 18 Jan 2018 14:39:57 -0500
>> Cc: Ken Brown <kbrown@cornell.edu>, 5845@debbugs.gnu.org
>>
>> On Thu, Jan 18, 2018 at 1:51 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>>
>> >> Should this go to emacs-26?
>> >
>> > Not sure.  The issue doesn't sound urgent/important, what with its
>> > being unsolved for such a long time.  OTOH, the change is simple and
>> > localized.  WDYT?
>>
>> Agreed on both points. The simpleness of the fix outweighs the urgency
>> for me I think, so I'm leaning to emacs-26.
>
> OK.

Pushed.

[1: 76040d1]: 2018-01-18 16:26:52 -0500
  Handle case-insensitive filenames for load-path shadows (Bug#5845)
  https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=76040d1eae4464b468481231c15e7fb86f4b11d8