bug#16667: Accented home dir stops ELPA package

bug#16667: Accented home dir stops ELPA package

[Stefan Monnier]

Package: Emacs
Version: 24.3.50


A student of mine signals the following (with Emacs-24.3 under Windows):

- start Emacs
- M-x list-packages
- install "lmc"
- restart Emacs
- open file foo.elmc
- it's not in LMC-Asm mode as it should

Apparently the problem is linked to the fact that his home directory has
an accented letter.  The ELPA installation produced the following messages:

   For information about GNU Emacs and the GNU system, type C-h C-a.
   Contacting host: elpa.gnu.org:80
   Saving file c:/Users/Propriétaire.AlienwareCedric/AppData/Roaming/.emacs.d/elpa/archives/gnu/archive-contents...
   Wrote c:/Users/Propriétaire.AlienwareCedric/AppData/Roaming/.emacs.d/elpa/archives/gnu/archive-contents
   Contacting host: elpa.gnu.org:80
   Saving file c:/Users/Propriétaire.AlienwareCedric/AppData/Roaming/.emacs.d/elpa/lmc-readme.txt...
   Wrote c:/Users/Propriétaire.AlienwareCedric/AppData/Roaming/.emacs.d/elpa/lmc-readme.txt
   Type C-x 1 to delete the help window, C-M-v to scroll help.
   Install package `lmc'? y
   Contacting host: elpa.gnu.org:80
   Wrote c:/Users/Propriétaire.AlienwareCedric/AppData/Roaming/.emacs.d/elpa/lmc-1.3/lmc.el
   Wrote c:/Users/Propriétaire.AlienwareCedric/AppData/Roaming/.emacs.d/elpa/lmc-1.3/lmc-pkg.el
   Wrote c:/Users/Propriétaire.AlienwareCedric/AppData/Roaming/.emacs.d/elpa/lmc-1.3/lmc-autoloads.el
   Making version-control local to lmc-autoloads.el while let-bound!
   autoload-find-file: Opening input file: no such file or directory, c:/Users/Propriétaire.AlienwareCedric/AppData/Roaming/.emacs.d/elpa/lmc-1.3/lmc-pkg.el

we see the last line has an improperly (en/de)coded utf-8 sequence.

Can someone reproduce this problem?  Is it already fixed in trunk?


        Stefan




In GNU Emacs 24.3.50.1 (x86_64-unknown-linux-gnu, GTK+ Version 2.24.22)
 of 2014-02-02 on pastel
Repository revision: monnier@iro.umontreal.ca-20140130214249-81biuubr7j6wwc30
Windowing system distributor `The X.Org Foundation', version 11.0.11405000
System Description:	Debian GNU/Linux testing (jessie)

Configured using:
 `configure -C --enable-checking --enable-check-lisp-object-type
 'CFLAGS=-Wall -g3 -O0 -Wno-pointer-sign''

Important settings:
  value of $LANG: fr_CH.UTF-8
  locale-coding-system: utf-8-unix

Major mode: InactiveMinibuffer

Minor modes in effect:
  diff-auto-refine-mode: t
  shell-dirtrack-mode: t
  electric-pair-mode: t
  url-handler-mode: t
  global-reveal-mode: t
  reveal-mode: t
  auto-insert-mode: t
  savehist-mode: t
  minibuffer-electric-default-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
SPC o r SPC f b o u n d . <left> SPC ( b u g # <down-mouse-2> 
<mouse-2> C-a C-x C-s <help-echo> <switch-frame> <switch-frame> 
<switch-frame> <switch-frame> <switch-frame> <switch-frame> 
<switch-frame> <switch-frame> <switch-frame> <switch-frame> 
<switch-frame> <switch-frame> <switch-frame> <switch-frame> 
<switch-frame> <switch-frame> <switch-frame> <switch-frame> 
<switch-frame> <switch-frame> <switch-frame> <switch-frame> 
<switch-frame> <switch-frame> <switch-frame> <switch-frame> 
<switch-frame> <switch-frame> <switch-frame> <switch-frame> 
<switch-frame> <switch-frame> <switch-frame> <switch-frame> 
<switch-frame> <switch-frame> <switch-frame> <switch-frame> 
<switch-frame> <switch-frame> <switch-frame> <help-echo> 
<switch-frame> <switch-frame> <help-echo> <switch-frame> 
<switch-frame> <switch-frame> <switch-frame> <switch-frame> 
<switch-frame> <help-echo> <switch-frame> <switch-frame> 
<switch-frame> <switch-frame> <help-echo> <help-echo> 
<help-echo> <help-echo> <switch-frame> <switch-frame> 
<switch-frame> <switch-frame> <help-echo> <help-echo> 
<switch-frame> <switch-frame> <help-echo> <help-echo> 
<help-echo> <switch-frame> <switch-frame> <help-echo> 
<switch-frame> <switch-frame> <help-echo> M-r C-x C-s 
M-x c h - l - c l <tab> <return> C-x C-s <switch-frame> 
<switch-frame> <help-echo> <switch-frame> <switch-frame> 
<switch-frame> <switch-frame> <switch-frame> <switch-frame> 
<switch-frame> <switch-frame> <switch-frame> <switch-frame> 
<switch-frame> <switch-frame> <switch-frame> <switch-frame> 
<switch-frame> <switch-frame> <switch-frame> <switch-frame> 
<switch-frame> <switch-frame> <switch-frame> <switch-frame> 
<switch-frame> <switch-frame> <switch-frame> <switch-frame> 
<switch-frame> <switch-frame> <switch-frame> <switch-frame> 
<switch-frame> <switch-frame> <switch-frame> <switch-frame> 
<switch-frame> <switch-frame> <switch-frame> <switch-frame> 
<switch-frame> <switch-frame> <switch-frame> <switch-frame> 
<switch-frame> <switch-frame> <help-echo> <switch-frame> 
<switch-frame> <switch-frame> <switch-frame> <switch-frame> 
<switch-frame> <switch-frame> <switch-frame> <switch-frame> 
<switch-frame> <switch-frame> <switch-frame> C-x v 
d <M-backspace> <return> <down-mouse-5> <mouse-5> <double-down-mouse-5> 
<double-mouse-5> <down-mouse-1> <mouse-movement> <mouse-1> 
m <down> <down> <down> <down> <down> <down> m <switch-frame> 
<switch-frame> <switch-frame> <switch-frame> <help-echo> 
<switch-frame> <switch-frame> <switch-frame> <switch-frame> 
<switch-frame> <switch-frame> <switch-frame> <switch-frame> 
<switch-frame> <switch-frame> <switch-frame> <switch-frame> 
<switch-frame> <switch-frame> <switch-frame> <switch-frame> 
<switch-frame> <switch-frame> <switch-frame> <switch-frame> 
<switch-frame> <switch-frame> <switch-frame> <switch-frame> 
<switch-frame> <switch-frame> <switch-frame> <switch-frame> 
<switch-frame> <switch-frame> <switch-frame> <switch-frame> 
<switch-frame> <switch-frame> <switch-frame> <switch-frame> 
<switch-frame> <switch-frame> <switch-frame> <switch-frame> 
<switch-frame> <switch-frame> <switch-frame> <switch-frame> 
<switch-frame> <switch-frame> <switch-frame> <switch-frame> 
<switch-frame> <switch-frame> <switch-frame> <switch-frame> 
<switch-frame> <help-echo> <switch-frame> v <switch-frame> 
<switch-frame> <switch-frame> <switch-frame> <switch-frame> 
<switch-frame> <switch-frame> <switch-frame> <switch-frame> 
<switch-frame> <switch-frame> <switch-frame> C-c C-c 
<switch-frame> <switch-frame> <switch-frame> <switch-frame> 
<switch-frame> <switch-frame> <help-echo> <down-mouse-1> 
<mouse-1> C-e <down-mouse-1> <mouse-1> C-e <down-mouse-1> 
<mouse-1> C-e <help-echo> <switch-frame> M-x r e - 
e m - b u <tab> <return>

Recent messages:
Mark set
Press C-c C-c when you are done editing.
Enter a change comment.  Type C-c C-c when done
Checking in /home/monnier/src/emacs/trunk/lisp/ChangeLog /home/monnier/src/emacs/trunk/lisp/emacs-lisp/lisp.el...done
Error: (error "Running bzr status --no-classify lisp/ChangeLog...FAILED (status 3)")
Warnings in `bzr' output: bzr: ERROR: Could not acquire lock "/home/monnier/src/emacs/trunk/.bzr/checkout/dirstate": [Errno 11] Resource temporarily unavailable

Error: (error "Running bzr status --no-classify lisp/emacs-lisp/lisp.el...FAILED (status 3)")
Warnings in `bzr' output: bzr: ERROR: Could not acquire lock "/home/monnier/src/emacs/trunk/.bzr/checkout/dirstate": [Errno 11] Resource temporarily unavailable


Load-path shadows:
/home/monnier/src/emacs/elpa/packages/company/.dir-locals hides /home/monnier/src/emacs/elpa/packages/js2-mode/.dir-locals
/home/monnier/src/emacs/elpa/packages/ada-mode/ada-mode hides /home/monnier/src/emacs/work/lisp/progmodes/ada-mode
/home/monnier/src/emacs/elpa/packages/company/.dir-locals hides /home/monnier/src/emacs/work/lisp/gnus/.dir-locals

Features:
(sort mail-extr emacsbug log-edit message sendmail rfc822 mml mml-sec
mm-decode mm-bodies mm-encode mail-parse rfc2231 rfc2047 rfc2045
ietf-drums mail-utils mailheader pcvs-util vc-sccs vc-svn vc-cvs vc-rcs
vc-dir smerge-mode bug-reference add-log whitespace diff-mode vc
vc-dispatcher xscheme unsafep trace testcover shadow scheme re-builder
profiler pcase inf-lisp ielm pp gmm-utils ert ewoc debug elp edebug
cl-indent eieio-opt speedbar sb-image ezimage dframe completion
autorevert filenotify bbdb-com mailabbrev bbdb timezone sgml-mode
reftex-dcr reftex reftex-vars tex-mode compile shell latexenc
org-protocol org-mouse org-mobile org-agenda org-indent org-feed
org-crypt org-capture org-attach vc-git org-id cus-edit cus-start
cus-load wid-edit org-element org-rmail org-mhe org-irc org-info
org-gnus org-docview doc-view jka-compr image-mode dired org-bibtex
bibtex org-bbdb org-w3m org org-macro org-footnote org-pcomplete
pcomplete org-list org-faces org-entities org-version ob-emacs-lisp ob
ob-tangle org-src ob-ref ob-lob ob-table ob-keys ob-exp ob-comint comint
ansi-color ring ob-core ob-eval org-compat org-macs org-loaddefs
format-spec find-func misearch multi-isearch executable copyright view
cal-china lunar solar cal-dst cal-bahai cal-islam cal-hebrew holidays
hol-loaddefs warnings cal-french vc-bzr filecache vlf diary-lib
diary-loaddefs mule-util cal-move cal-menu calendar cal-loaddefs server
noutline outline easy-mmode flyspell ispell eldoc checkdoc thingatpt
help-mode elec-pair url-handlers url-parse auth-source eieio eieio-core
gnus-util mm-util mail-prsvr password-cache url-vars reveal autoinsert
proof-site proof-autoloads cl-macs gv cl pg-vars time-date savehist
minibuf-eldef disp-table edmacro kmacro cl-loaddefs cl-lib info easymenu
byte-opt bytecomp byte-compile cconv advice help-fns package
bbdb-autoloads agda2 vm-autoloads tooltip electric uniquify ediff-hook
vc-hooks lisp-float-type mwheel x-win x-dnd tool-bar dnd fontset image
regexp-opt fringe tabulated-list newcomment lisp-mode prog-mode register
page menu-bar rfn-eshadow timer select scroll-bar mouse jit-lock
font-lock syntax font-core frame cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
case-table epa-hook jka-cmpr-hook help simple abbrev minibuffer nadvice
loaddefs button faces cus-face macroexp files text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote make-network-process dbusbind
gfilenotify dynamic-setting system-font-setting font-render-setting
move-toolbar gtk x-toolkit x multi-tty emacs)

bug#16667: Accented home dir stops ELPA package

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Thu, 06 Feb 2014 08:27:51 -0500
> Cc: cedric.sauriol@umontreal.ca
> 
> A student of mine signals the following (with Emacs-24.3 under Windows):
> 
> - start Emacs
> - M-x list-packages
> - install "lmc"
> - restart Emacs
> - open file foo.elmc
> - it's not in LMC-Asm mode as it should
> 
> Apparently the problem is linked to the fact that his home directory has
> an accented letter.

Was that letter representable by the current Windows system codepage?
If so, I don't think this is the problem.  If it wasn't representable,
such a directory cannot be used on Windows with Emacs 24.3.

> The ELPA installation produced the following messages:
> 
>    For information about GNU Emacs and the GNU system, type C-h C-a.
>    Contacting host: elpa.gnu.org:80
>    Saving file c:/Users/Propriétaire.AlienwareCedric/AppData/Roaming/.emacs.d/elpa/archives/gnu/archive-contents...
>    Wrote c:/Users/Propriétaire.AlienwareCedric/AppData/Roaming/.emacs.d/elpa/archives/gnu/archive-contents
>    Contacting host: elpa.gnu.org:80
>    Saving file c:/Users/Propriétaire.AlienwareCedric/AppData/Roaming/.emacs.d/elpa/lmc-readme.txt...
>    Wrote c:/Users/Propriétaire.AlienwareCedric/AppData/Roaming/.emacs.d/elpa/lmc-readme.txt
>    Type C-x 1 to delete the help window, C-M-v to scroll help.
>    Install package `lmc'? y
>    Contacting host: elpa.gnu.org:80
>    Wrote c:/Users/Propriétaire.AlienwareCedric/AppData/Roaming/.emacs.d/elpa/lmc-1.3/lmc.el
>    Wrote c:/Users/Propriétaire.AlienwareCedric/AppData/Roaming/.emacs.d/elpa/lmc-1.3/lmc-pkg.el
>    Wrote c:/Users/Propriétaire.AlienwareCedric/AppData/Roaming/.emacs.d/elpa/lmc-1.3/lmc-autoloads.el
>    Making version-control local to lmc-autoloads.el while let-bound!
>    autoload-find-file: Opening input file: no such file or directory, c:/Users/Propriétaire.AlienwareCedric/AppData/Roaming/.emacs.d/elpa/lmc-1.3/lmc-pkg.el
> 
> we see the last line has an improperly (en/de)coded utf-8 sequence.

Hmm... can you tell which code called autoload-find-file?

> Is it already fixed in trunk?

The trunk uses an entirely different machinery for non-ASCII file
names on Windows, so it's hard to tell.  But we did fix quite a few
problems related to building and running Emacs from non-ASCII
directories, so maybe this was one of them.

bug#16667: Accented home dir stops ELPA package

[Stefan Monnier]

> Was that letter representable by the current Windows system codepage?

I assume so, since Emacs works at least to some extend and was able to
access some of the files in that user's directory, as evidenced by the
message-log I sent in my previous message.

> If so, I don't think this is the problem.  If it wasn't representable,
> such a directory cannot be used on Windows with Emacs 24.3.

He tested the same procedure after creating another user with home dir
"/Users/LMC" and the problem disappeared, so the accent looks like
a good explanation.

>> we see the last line has an improperly (en/de)coded utf-8 sequence.
> Hmm... can you tell which code called autoload-find-file?

It's called from autoload-generate-file-autoloads itself called from
package-generate-autoloads via update-directory-autoloads.

A possible explanation is if the `directory-files' call in
update-directory-autoloads (which requests full names, so the output
file names include the "c:/Users/Propriétaire.AlienwareCedric/"
prefix) returns mis-encoded names.  E.g. prepends default-directory but
marks the result as unibyte, or something.

>> Is it already fixed in trunk?
> The trunk uses an entirely different machinery for non-ASCII file
> names on Windows, so it's hard to tell.

Indeed, which is why I wonder if it might be fixed already.

> But we did fix quite a few problems related to building and running
> Emacs from non-ASCII directories, so maybe this was one of them.

That's right.  Could someone try to reproduce the problem in 24.3 and
then see if it's fixed in trunk?


        Stefan

bug#16667: Accented home dir stops ELPA package

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: 16667@debbugs.gnu.org,  cedric.sauriol@umontreal.ca
> Date: Fri, 07 Feb 2014 08:50:22 -0500
> 
> > Hmm... can you tell which code called autoload-find-file?
> 
> It's called from autoload-generate-file-autoloads itself called from
> package-generate-autoloads via update-directory-autoloads.
> 
> A possible explanation is if the `directory-files' call in
> update-directory-autoloads (which requests full names, so the output
> file names include the "c:/Users/Propriétaire.AlienwareCedric/"
> prefix) returns mis-encoded names.  E.g. prepends default-directory but
> marks the result as unibyte, or something.

Yes, but how would a unibyte string escape from C in the first place?
Does any of the functions involved encode file names by hand?

> > But we did fix quite a few problems related to building and running
> > Emacs from non-ASCII directories, so maybe this was one of them.
> 
> That's right.  Could someone try to reproduce the problem in 24.3 and
> then see if it's fixed in trunk?

It's not easy to do, because this is the user's home directory, so you
need to have a user with a non-ASCII character in her name.  And the
C:/Users directory is not generally writable, or at least it isn't
supposed to be.

Why don't you ask that guy to install a development snapshot?  If the
problem persists, we can help him then.

bug#16667: Accented home dir stops ELPA package

[Stefan Monnier]

> Yes, but how would a unibyte string escape from C in the first place?

I suggested that directory-files might take the SDATA of the
default-directory (i.e. its internal utf-8 representation), prepend it
to the file name and mark the result as a unibyte string before
returning it.

I have no evidence saying it's what happens, but based on the message
log something along these lines might be going on and could explain the
behavior.

> Does any of the functions involved encode file names by hand?

You mean the functions in autoload.el or package.el?  I don't think so.

> Why don't you ask that guy to install a development snapshot?  If the
> problem persists, we can help him then.

He's a student of mine, so use of Emacs is mostly imposed and I since he
found a workaround (use that alternate user with all-ASCII name) I think
he's happy enough to move on.


        Stefan

bug#16667: #16667 Accented home dir stops ELPA package

[Noam Postavsky]

close 16667 24.5
quit

> It's not easy to do, because this is the user's home directory, so
> you need to have a user with a non-ASCII character in her name.  And
> the C:/Users directory is not generally writable, or at least it
> isn't supposed to be.

It's not that hard, just make a directory with non-ASCII name, and set
HOME= to it.

I confirmed it happens in 24.3, and works in 24.5 (24.4 crashed on
startup with non-ASCII HOME).

Contacting host: elpa.gnu.org:80
Making version-control local to lmc-autoloads.el while let-bound!
Generating autoloads for lmc.el...done
Saving file c:/Users/npostavs/tmp/Propriétaire.AlienwareCedric/.emacs.d/elpa/lmc-1.4/lmc-autoloads.el...
Wrote c:/Users/npostavs/tmp/Propriétaire.AlienwareCedric/.emacs.d/elpa/lmc-1.4/lmc-autoloads.el
(No changes need to be saved)
(No files need saving)
Checking c:/Users/npostavs/tmp/Propriétaire.AlienwareCedric/.emacs.d/elpa/lmc-1.4...
[2 times]
Compiling c:/Users/npostavs/tmp/Propriétaire.AlienwareCedric/.emacs.d/elpa/lmc-1.4/lmc-pkg.el...done
Wrote c:/Users/npostavs/tmp/Propriétaire.AlienwareCedric/.emacs.d/elpa/lmc-1.4/lmc-pkg.elc
Checking c:/Users/npostavs/tmp/Propriétaire.AlienwareCedric/.emacs.d/elpa/lmc-1.4...
Compiling c:/Users/npostavs/tmp/Propriétaire.AlienwareCedric/.emacs.d/elpa/lmc-1.4/lmc.el...done
Wrote c:/Users/npostavs/tmp/Propriétaire.AlienwareCedric/.emacs.d/elpa/lmc-1.4/lmc.elc
Checking c:/Users/npostavs/tmp/Propriétaire.AlienwareCedric/.emacs.d/elpa/lmc-1.4...
Done (Total of 2 files compiled, 1 skipped)