bug#30755: 25.3; Encoding of load-file-name wrong when path to working dir does not contain accented letter

bug#30755: 25.3; Encoding of load-file-name wrong when path to working dir does not contain accented letter

[Márton Marczell]

[-- Attachment #1: Type: text/plain, Size: 4018 bytes --]

My Windows username is Márton.
In one of the files loaded by my .emacs (neotree.el), I added this line:
   (print load-file-name)

1. I start Emacs from the command line with no arguments.
2. I view the *Messages* buffer.

If the directory from which I launched Emacs has an accent in its path,
e.g. "D:\á\", this is printed:
"c:/Users/Márton/AppData/Roaming/.emacs.d/elpa/neotree-0.5.2/neotree.el"
But if it doesn't, e.g. "D:\a\", this is printed:
"c:/Users/Márton/AppData/Roaming/.emacs.d/elpa/neotree-0.5.2/neotree.el"

(This prevents the neotree package from loading images in the latter case.)
Please see also bug#25038, which seems related.




In GNU Emacs 25.3.1 (x86_64-w64-mingw32)
 of 2017-09-26 built on LAPHROAIG
Windowing system distributor 'Microsoft Corp.', version 10.0.16299
Configured using:
 'configure --without-dbus --without-compress-install 'CFLAGS=-O2
 -static -g3' PKG_CONFIG_PATH=/mingw64/lib/pkgconfig'

Configured features:
XPM JPEG TIFF GIF PNG RSVG SOUND NOTIFY ACL GNUTLS LIBXML2 ZLIB
TOOLKIT_SCROLL_BARS

Important settings:
  value of $LANG: HUN
  locale-coding-system: cp1250

Major mode: Messages

Minor modes in effect:
  diff-auto-refine-mode: t
  shell-dirtrack-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  line-number-mode: t
  transient-mark-mode: t

Recent messages:

"TATTARAA TATTARAA TATTARAA"

"c:/Users/Márton/AppData/Roaming/.emacs.d/elpa/neotree-0.5.2/neotree.el"

"c:/Users/Márton/AppData/Roaming/.emacs.d/elpa/neotree-0.5.2/"

Loading c:/Users/Márton/AppData/Roaming/.emacs.d/custom.el (source)...done
For information about GNU Emacs and the GNU system, type C-h C-a.
Making completion list...

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message idna rfc822 mml mml-sec epg
mm-decode mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils
mailheader sendmail rfc2047 rfc2045 ietf-drums mail-utils vc-git
diff-mode easy-mmode cursor-sensor js2-mode etags xref project js
sgml-mode cc-mode cc-fonts cc-guess cc-menus cc-cmds cc-styles cc-align
cc-engine cc-vars cc-defs imenu dired neotree derived finder-inf package
epg-config sql view thingatpt edmacro kmacro python tramp-sh tramp
tramp-compat auth-source cl-seq eieio eieio-core cl-macs gnus-util
mm-util help-fns mail-prsvr password-cache tramp-loaddefs trampver
ucs-normalize shell pcomplete format-spec advice json map seq byte-opt
gv bytecomp byte-compile cl-extra help-mode easymenu cconv comint ring
cl-loaddefs pcase cl-lib ansi-color time-date mule-util tooltip eldoc
electric uniquify ediff-hook vc-hooks lisp-float-type mwheel dos-w32
ls-lisp disp-table w32-win w32-vars term/common-win tool-bar dnd fontset
image regexp-opt fringe tabulated-list newcomment elisp-mode lisp-mode
prog-mode register page menu-bar rfn-eshadow timer select scroll-bar
mouse jit-lock font-lock syntax facemenu font-core frame cl-generic cham
georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese charscript case-table epa-hook
jka-cmpr-hook help simple abbrev minibuffer cl-preloaded nadvice
loaddefs button faces cus-face macroexp files text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote w32notify w32 multi-tty
make-network-process emacs)

Memory information:
((conses 16 370288 19749)
 (symbols 56 34419 0)
 (miscs 48 60 234)
 (strings 32 62682 11501)
 (string-bytes 1 1969857)
 (vectors 16 54523)
 (vector-slots 8 895948 6275)
 (floats 8 467 185)
 (intervals 56 909 73)
 (buffers 976 20))

[-- Attachment #2: Type: text/html, Size: 11258 bytes --]

bug#30755: 25.3; Encoding of load-file-name wrong when path to working dir does not contain accented letter

[Eli Zaretskii]

> From: Márton Marczell <dalokmarcinak@gmail.com>
> Date: Fri, 9 Mar 2018 12:22:05 +0100
> 
> My Windows username is Márton.
> 
> In one of the files loaded by my .emacs (neotree.el), I added this line:
> 
>    (print load-file-name)
> 
> 1. I start Emacs from the command line with no arguments.
> 2. I view the *Messages* buffer.
> 
> If the directory from which I launched Emacs has an accent in its path,
> e.g. "D:\á\", this is printed:
> "c:/Users/Márton/AppData/Roaming/.emacs.d/elpa/neotree-0.5.2/neotree.el"
> But if it doesn't, e.g. "D:\a\", this is printed:
> "c:/Users/Márton/AppData/Roaming/.emacs.d/elpa/neotree-0.5.2/neotree.el"
> 
> (This prevents the neotree package from loading images in the latter case.)

How is load-file-name of the package's .el file related to its
inability to load images?

Also, can you please provide a recipe that takes neotree.el out of the
equation?  AFAIU, it should be enough to have a .el file that is
loaded in .emacs, and have the contents of that file be just

  (print load-file-name)

Is that correct?

> Please see also bug#25038, which seems related.

But that bug was solved, so something else must be at work here.

Thanks.

bug#30755: 25.3; Encoding of load-file-name wrong when path to working dir does not contain accented letter

[Eli Zaretskii]

> From: Márton Marczell <dalokmarcinak@gmail.com>
> Date: Fri, 9 Mar 2018 22:41:12 +0100
> 
> > How is load-file-name of the package's .el file related to its inability to load images?
> 
> It loads images like this:
> 
> (defconst neo-dir
>   (expand-file-name (if load-file-name
>                         (file-name-directory load-file-name)
>                       default-directory)))
> ;…
> (defun neo-buffer--get-icon (name)
>   "Get image by NAME."
>   (let ((icon-path (neo-path--join neo-dir "icons"))
>         image)
>     (setq image (create-image
>                  (neo-path--join icon-path (concat name ".xpm"))
>                  'xpm nil :ascent 'center :mask '(heuristic t)))
>     image))
> 
> So the (wrongly encoded) load-file-name path is used to build the path to the image assets. Which makes image loading fail with the error
> Cannot find image file ‘c:/Users/M rton/AppData/Roaming/.emacs.d/elpa/neotree-0.5.2/icons/leaf.xpm’
> 
> > Also, can you please provide a recipe that takes neotree.el out of the
> > equation?  AFAIU, it should be enough to have a .el file that is
> > loaded in .emacs, and have the contents of that file be just
> >   (print load-file-name)
> > Is that correct?
> 
> Yes that is correct.

OK, thanks.

I tried reproducing this problem, but failed.  I guess there are still
important details of the situation that are missing from the
description.  One detail that might be important is how exactly do you
load neotree.el from your init file.  In my reproduction attempt, I
just created a .emacs file with only one line:

  (load "~/foo/foo.el")

where foo.el had this single line:

   (print load-file-name)

This produced the correct file name in the *Messages* buffer when
Emacs was started from a directory that had no non-ASCII characters,
while the home directory did include non-ASCII characters, which I
believe is your case.

Please provide a full description of how you load your file, and
preferably also try reproducing the problem with a simple setup such
as what I used above (modified to match the exact way you load
neotree.el in the real-life use case), and see if you succeed
reproducing it.

Also, if you repeat your original use case, but before starting Emacs
set HOME=c:\Users\Márton\AppData\Roaming in the environment of the
shell from which you invoke Emacs, does the problem reproduce itself?
(Btw, is the shell from which you start Emacs the stock Windows shell
cmd.exe or something else?)

And finally, please keep the bug address on the CC list of your
responses, so that this discussion gets recorded by the bug tracker.

Thanks.

bug#30755: 25.3; Encoding of load-file-name wrong when path toworking dir does not contain accented letter

[Márton Marczell]

[-- Attachment #1: Type: text/plain, Size: 369 bytes --]

During trying to determine the exact conditions of the error, I started anew with a fresh .emacs.d folder and reinstalled my packages, and the bug disappeared. I guess an earlier version of Emacs (in which that related bug was not yet fixed) wrote something in .emacs.d that set the wrong encoding. 

I still have the old .emacs.d if we want to investigate further.

[-- Attachment #2: Type: text/html, Size: 1707 bytes --]

bug#30755: 25.3; Encoding of load-file-name wrong when path toworking dir does not contain accented letter

[Eli Zaretskii]

> Cc: "30755@debbugs.gnu.org" <30755@debbugs.gnu.org>
> From: Márton Marczell <dalokmarcinak@gmail.com>
> Date: Sat, 17 Mar 2018 15:19:12 +0100
> 
> During trying to determine the exact conditions of the error, I started anew with a fresh .emacs.d folder and
> reinstalled my packages, and the bug disappeared. I guess an earlier version of Emacs (in which that related
> bug was not yet fixed) wrote something in .emacs.d that set the wrong encoding. 

OK, thanks.  I'm therefore closing this bug report.

> I still have the old .emacs.d if we want to investigate further.

I don't see the need.  If the problem ever returns, please reopen this
bug or file a new one.

bug#30755: 25.3; Encoding of load-file-name wrong when path toworking dir does not contain accented letter

[Márton Marczell]

[-- Attachment #1: Type: text/plain, Size: 454 bytes --]

This bug has reappeared and I have investigated its conditions further. The
conditions appear to be the following:
 - Emacs is started with a working directory with only ASCII characters in
the path (no accented letters)
 - There is at least one "multi-file package" in ~/.emacs.d/elpa/
This latter requirement is satisfied even with an empty package (only a
"foo" folder with a foo-pkg.el containing only the necessary define-package
and nothing else.)

[-- Attachment #2: Type: text/html, Size: 572 bytes --]

bug#30755: 25.3; Encoding of load-file-name wrong when path toworking dir does not contain accented letter

[Eli Zaretskii]

> From: Márton Marczell <dalokmarcinak@gmail.com>
> Date: Wed, 9 May 2018 11:01:26 +0200
> Cc: 30755-done@debbugs.gnu.org
> 
> This bug has reappeared and I have investigated its conditions further. The conditions appear to be the
> following:
>  - Emacs is started with a working directory with only ASCII characters in the path (no accented letters)
>  - There is at least one "multi-file package" in ~/.emacs.d/elpa/
> This latter requirement is satisfied even with an empty package (only a "foo" folder with a foo-pkg.el containing
> only the necessary define-package and nothing else.)

Can you provide all the necessary files and a procedure to reproduce
this problem?  Given that the problem is so elusive, I think having a
recipe to reproduce exactly what you do and with the same data will
allow us to investigate much more efficiently.

Thanks.

bug#30755: 25.3; Encoding of load-file-name wrong when pathtoworking dir does not contain accented letter

[Márton Marczell]

[-- Attachment #1: Type: text/plain, Size: 701 bytes --]

> Can you provide all the necessary files and a procedure to reproduce
> this problem?  Given that the problem is so elusive, I think having a
> recipe to reproduce exactly what you do and with the same data will
> allow us to investigate much more efficiently.

I’ve uploaded the repro files at
http://users.itk.ppke.hu/~marma/downloads/emacsdebug.zip

The procedure is:
1. Create a Windows user with an accented letter in its name (mine is “Márton”)
2. Unzip the above files to %APPDATA% (C:\Users\Márton\AppData\Roaming)
3. Start Emacs so that the working directory has no accented letter in the path
4. Notice how in the Messages buffer, the path name is wrongly encoded.



[-- Attachment #2: Type: text/html, Size: 4751 bytes --]

bug#30755: 25.3; Encoding of load-file-name wrong when pathtoworking dir does not contain accented letter

[Eli Zaretskii]

> Cc: "30755@debbugs.gnu.org" <30755@debbugs.gnu.org>
> From: Márton Marczell <dalokmarcinak@gmail.com>
> Date: Fri, 11 May 2018 00:23:18 +0200
> 
> I’ve uploaded the repro files at
> 
> http://users.itk.ppke.hu/~marma/downloads/emacsdebug.zip
> 
> The procedure is:
> 
> 1 Create a Windows user with an accented letter in its name (mine is “Márton”)
> 2 Unzip the above files to %APPDATA% (C:\Users\Márton\AppData\Roaming)
> 3 Start Emacs so that the working directory has no accented letter in the path
> 4 Notice how in the Messages buffer, the path name is wrongly encoded.

Thanks.  Creating a new user is not really feasible for me, but I
think I succeeded recreating this by setting HOME manually to point to
a directory with a non-ASCII name.

I see the problem, and I'm testing a provisional solution.  Can you
build your own Emacs from sources?  If so, I'd like to ask you to test
the solution I came up with.

bug#30755: 25.3; Encoding of load-file-name wrong whenpathtoworking dir does not contain accented letter

[Márton Marczell]

[-- Attachment #1: Type: text/plain, Size: 247 bytes --]

I’ve never built Emacs, but if the instructions at https://gist.github.com/nauhygon/f3b44f51b34e89bc54f8 are not wildly incorrect or out-of-date then I think I can do it. Do I get the code with git? What’s the repo/branch URL I should use?


[-- Attachment #2: Type: text/html, Size: 1632 bytes --]

bug#30755: 25.3; Encoding of load-file-name wrong whenpathtoworking dir does not contain accented letter

[Eli Zaretskii]

On May 16, 2018 10:42:00 AM GMT+03:00, "Márton Marczell" <dalokmarcinak@gmail.com> wrote:
> I’ve never built Emacs, but if the instructions at
> https://gist.github.com/nauhygon/f3b44f51b34e89bc54f8 are not wildly
> incorrect or out-of-date then I think I can do it. Do I get the code
> with git? What’s the repo/branch URL I should use?

Please use the master branch of the Emacs Git repository, and I'd prefer
if you used the instructions in the file nt/INSTALL.W64 that
you find in the repository.

bug#30755: 25.3; Encoding of load-file-name wrongwhenpathtoworking dir does not contain accented letter

[Márton Marczell]

[-- Attachment #1: Type: text/plain, Size: 333 bytes --]

I cloned git.savannah.gnu.org/r/emacs.git and built emacs. I set the HOME envvar to ‘D:\Marci\Programozás\emacsdebug’ which contained the repro case from above. I still get wrong output when the load path is printed:

Loading d:/Marci/Programoz  s/emacsdebug/foo.el (source)...

"d:/Marci/Programozás/emacsdebug/foo.el"

[-- Attachment #2: Type: text/html, Size: 1863 bytes --]

bug#30755: 25.3; Encoding of load-file-name wrongwhenpathtoworking dir does not contain accented letter

[Eli Zaretskii]

> Cc: "30755@debbugs.gnu.org" <30755@debbugs.gnu.org>
> From: Márton Marczell <dalokmarcinak@gmail.com>
> Date: Thu, 17 May 2018 16:54:56 +0200
> 
> I cloned git.savannah.gnu.org/r/emacs.git and built emacs. I set the HOME envvar to
> ‘D:\Marci\Programozás\emacsdebug’ which contained the repro case from above. I still get wrong output
> when the load path is printed:

I didn't yet commit my changes, so the fact you still see the problem
is expected.

Please apply the patch below, rebuild Emacs (by typing "make" at the
shell prompt in the top-level directory of the Emacs tree), and see if
the problem goes away.

Thanks.

--- src/fileio.c~0	2018-02-12 12:40:44.000000000 +0200
+++ src/fileio.c	2018-05-15 18:13:28.240161500 +0300
@@ -865,33 +865,71 @@ the root directory.  */)
       }
   }
   multibyte = STRING_MULTIBYTE (name);
-  if (multibyte != STRING_MULTIBYTE (default_directory))
+  bool defdir_multibyte = STRING_MULTIBYTE (default_directory);
+  if (multibyte != defdir_multibyte)
     {
+      /* We want to make both NAME and DEFAULT_DIRECTORY have the same
+	 multibyteness.  Strategy:
+	 . If either NAME or DEFAULT_DIRECTORY is pure-ASCII, they
+	   can be converted to the multibyteness of the other one
+	   while keeping the same byte sequence.
+	 . If both are non-ASCII, the only safe conversion is to
+	   convert the multibyte one to be unibyte, because the
+	   reverse conversion potentially adds bytes while raw bytes
+	   are converted to their multibyte forms, which we will be
+	   unable to account for, since the information about the
+	   original multibyteness is lost.  If those additional bytes
+	   later leak to system APIs because they are not encoded or
+	   because they are converted to unibyte strings by keeping
+	   the data, file APIs will fail.  */
       if (multibyte)
 	{
-	  unsigned char *p = SDATA (name);
+	  bool name_ascii_p = SCHARS (name) == SBYTES (name);
+	  unsigned char *p = SDATA (default_directory);
 
-	  while (*p && ASCII_CHAR_P (*p))
-	    p++;
-	  if (*p == '\0')
+	  if (!name_ascii_p)
+	    while (*p && ASCII_CHAR_P (*p))
+	      p++;
+	  if (name_ascii_p || *p != '\0')
 	    {
-	      /* NAME is a pure ASCII string, and DEFAULT_DIRECTORY is
-		 unibyte.  Do not convert DEFAULT_DIRECTORY to
-		 multibyte; instead, convert NAME to a unibyte string,
-		 so that the result of this function is also a unibyte
-		 string.  This is needed during bootstrapping and
-		 dumping, when Emacs cannot decode file names, because
-		 the locale environment is not set up.  */
+	      /* DEFAULT_DIRECTORY is unibyte and possibly non-ASCII.
+		 Make a unibyte string out of NAME, and arrange for
+		 the result of this function to be a unibyte string.
+		 This is needed during bootstrapping and dumping, when
+		 Emacs cannot decode file names, because the locale
+		 environment is not set up.  */
 	      name = make_unibyte_string (SSDATA (name), SBYTES (name));
 	      multibyte = 0;
 	    }
 	  else
-	    default_directory = string_to_multibyte (default_directory);
+	    {
+	      /* NAME is non-ASCII and multibyte, and
+		 DEFAULT_DIRECTORY is unibyte and pure-ASCII: make a
+		 multibyte string out of DEFAULT_DIRECTORY's data.  */
+	      default_directory =
+		make_multibyte_string (SSDATA (default_directory),
+				       SCHARS (default_directory),
+				       SCHARS (default_directory));
+	    }
 	}
       else
 	{
-	  name = string_to_multibyte (name);
-	  multibyte = 1;
+	  unsigned char *p = SDATA (name);
+
+	  while (*p && ASCII_CHAR_P (*p))
+	    p++;
+	  if (*p == '\0')
+	    {
+	      /* DEFAULT_DIRECTORY is multibyte and NAME is unibyte
+		 and pure-ASCII.  Make a multibyte string out of
+		 NAME's data.  */
+	      name = make_multibyte_string (SSDATA (name),
+					    SCHARS (name), SCHARS (name));
+	      multibyte = 1;
+	    }
+	  else
+	    default_directory = make_unibyte_string (SSDATA (default_directory),
+						     SBYTES (default_directory));
 	}
     }
 

--- lisp/startup.el~0	2018-03-14 06:40:04.000000000 +0200
+++ lisp/startup.el	2018-05-16 18:48:50.597482900 +0300
@@ -560,9 +560,17 @@
 	    (if default-directory
 		(setq default-directory
                       (if (eq system-type 'windows-nt)
-                          ;; Convert backslashes to forward slashes.
-                          (expand-file-name
-                           (decode-coding-string default-directory coding t))
+                          ;; We pass the decoded default-directory as
+                          ;; the 2nd arg to make sure expand-file-name
+                          ;; sees a multibyte string as the default
+                          ;; directory; this avoids the side effect of
+                          ;; returning a unibyte string from
+                          ;; expand-file-name because it still sees
+                          ;; the undecoded value of default-directory.
+                          (let ((defdir (decode-coding-string default-directory
+                                                              coding t)))
+                            ;; Convert backslashes to forward slashes.
+                            (expand-file-name defdir defdir))
                         (decode-coding-string default-directory coding t))))))
 
 	;; Decode all the important variables and directory lists, now

bug#30755: 25.3; Encoding of load-file-name wrongwhenpathtoworking dir does not contain accented letter

[Márton Marczell]

[-- Attachment #1: Type: text/plain, Size: 36 bytes --]

The patch fixed the problem for me.

[-- Attachment #2: Type: text/html, Size: 61 bytes --]

bug#30755: 25.3; Encoding of load-file-name wrongwhenpathtoworking dir does not contain accented letter

[Eli Zaretskii]

> From: Márton Marczell <dalokmarcinak@gmail.com>
> Date: Fri, 18 May 2018 15:06:20 +0200
> Cc: 30755@debbugs.gnu.org
> 
> The patch fixed the problem for me.

Thanks, I therefore pushed the patch to the master branch.

It is too late to make such changes on the release branch for Emacs
26.1, but if there going to be Emacs 26.2, I will backport the change
to it after Emacs 26.1 is released.

Thank you for all your help in tracking down this elusive problem.

I'm closing the bug.