bug#33524: 27.0.50; Suspicious code in flymake-proc around temporary directories

bug#33524: 27.0.50; Suspicious code in flymake-proc around temporary directories

[Philipp Stephani]

I've noticed that the temporary directory code in flymake-proc has
issues with remote filenames.  For example:

$ emacs -Q -batch -l flymake-proc --eval='(print (flymake-proc-create-temp-with-folder-structure "/:/dir" nil))'

"/tmp/:/dir"

Clearly that's not what was intended.  Rather, this should create the
directory structure on the remote machine.

If we use that filename:

mkdir -p /tmp/:/dir

then stuff will generally work, but trying to remove it will cause an
endless loop and try to remove /dir and /:

$ emacs -Q -batch -l flymake-proc --eval='(flymake-proc--delete-temp-directory "/tmp/:/dir")'
Error [flymake-proc *scratch*]: Failed to delete dir /dir, error ignored
Error [flymake-proc *scratch*]: Failed to delete dir /, error ignored
Error [flymake-proc *scratch*]: Failed to delete dir /, error ignored
[...infinite loop...]
^C
lisp.h:1485: Emacs fatal error: assertion failed: 0 <= nchars
Aborted


The assertion failure seems to be a different issue, but this bug
focuses on the problematic behavior of flymake-proc.  The code for these
functions looks really suspicious and seems to make lots of incorrect
assumptions (about whether temporary-file-directory ends in a slash,
that all files are local, etc.).  Especially the infinite loop in
flymake-proc--delete-temp-directory causes trouble because the only way
out of it is sending a signal to the Emacs process.



In GNU Emacs 27.0.50 (build 46, x86_64-pc-linux-gnu, GTK+ Version 3.22.24)
 of 2018-11-27
Repository revision: e02d375cb6670e2306b9c67d7f6fd2dd1d1b2711
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.11906000
System Description: Debian GNU/Linux buster/sid

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.
Eager macro-expansion failure: (wrong-number-of-arguments (2 . 2) 4)

Configured using:
 'configure --without-threads --enable-gcc-warnings=warn-only
 --enable-gtk-deprecation-warnings --without-pop --with-mailutils
 --enable-checking --enable-check-lisp-object-type --with-modules
 'CFLAGS=-O0 -ggdb3''

Configured features:
XPM JPEG TIFF GIF PNG SOUND DBUS GSETTINGS GLIB NOTIFY INOTIFY GNUTLS
FREETYPE XFT ZLIB TOOLKIT_SCROLL_BARS GTK3 X11 XDBE XIM MODULES JSON GMP

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc dired dired-loaddefs
format-spec rfc822 mml easymenu mml-sec epa derived epg epg-config
gnus-util rmail rmail-loaddefs time-date mm-decode mm-bodies mm-encode
mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail rfc2047
rfc2045 ietf-drums mm-util mail-prsvr mail-utils phst pcase ffap
thingatpt url-parse auth-source cl-seq eieio eieio-core cl-macs
eieio-loaddefs password-cache json map url-vars subr-x rx gnutls puny
seq byte-opt gv bytecomp byte-compile cconv dbus xml cl-loaddefs cl-lib
elec-pair mule-util tooltip eldoc electric uniquify ediff-hook vc-hooks
lisp-float-type mwheel term/x-win x-win term/common-win x-dnd tool-bar
dnd fontset image regexp-opt fringe tabulated-list replace newcomment
text-mode elisp-mode lisp-mode prog-mode register page menu-bar
rfn-eshadow isearch timer select scroll-bar mouse jit-lock font-lock
syntax facemenu font-core term/tty-colors frame cl-generic cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese composite charscript charprop
case-table epa-hook jka-cmpr-hook help simple abbrev obarray minibuffer
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote dbusbind inotify
dynamic-setting system-font-setting font-render-setting move-toolbar gtk
x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 109009 5313)
 (symbols 48 21762 1)
 (strings 32 32871 2004)
 (string-bytes 1 894311)
 (vectors 16 16454)
 (vector-slots 8 533506 11914)
 (floats 8 52 65)
 (intervals 56 230 0)
 (buffers 992 12))

-- 
Google Germany GmbH
Erika-Mann-Straße 33
80636 München

Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

If you received this communication by mistake, please don’t forward it to
anyone else (it may contain confidential or privileged information), please
erase all copies of it, including all attachments, and please let the sender
know it went to the wrong person.  Thanks.

bug#33524: 27.0.50; Suspicious code in flymake-proc around temporary directories

[Michael Albinus]

Philipp Stephani <p.stephani2@gmail.com> writes:

Hi Phillipp,

> I've noticed that the temporary directory code in flymake-proc has
> issues with remote filenames.  For example:
>
> $ emacs -Q -batch -l flymake-proc --eval='(print (flymake-proc-create-temp-with-folder-structure "/:/dir" nil))'
>
> "/tmp/:/dir"
>
> Clearly that's not what was intended.  Rather, this should create the
> directory structure on the remote machine.

I don't understand. "/:/dir" is not a remote file name, it is a quoted
file name. See (info "(emacs) Quoted File Names") for a description.

Could you pls explain what you want to achieve? This would help to
understand your problem.

Best regards, Michael.

bug#33524: 27.0.50; Suspicious code in flymake-proc around temporary directories

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 3300 bytes --]

Am Mi., 28. Nov. 2018 um 09:58 Uhr schrieb Michael Albinus <
michael.albinus@gmx.de>:

> Philipp Stephani <p.stephani2@gmail.com> writes:
>
> Hi Phillipp,
>
> > I've noticed that the temporary directory code in flymake-proc has
> > issues with remote filenames.  For example:
> >
> > $ emacs -Q -batch -l flymake-proc --eval='(print
> (flymake-proc-create-temp-with-folder-structure "/:/dir" nil))'
> >
> > "/tmp/:/dir"
> >
> > Clearly that's not what was intended.  Rather, this should create the
> > directory structure on the remote machine.
>
> I don't understand. "/:/dir" is not a remote file name, it is a quoted
> file name. See (info "(emacs) Quoted File Names") for a description.
>
> Could you pls explain what you want to achieve? This would help to
> understand your problem.
>
>
Sorry for being imprecise. Yes, I mean quoted filenames. (It's possible
that the issue also arises for true remote filenames, but I haven't
checked.)
You can easily reproduce user-facing problems. Assuming you have some
binary installed that would check Java files using the legacy backend:

$ touch /tmp/{a,b}.java
$ emacs -Q -eval '(progn (add-hook (quote prog-mode-hook) (quote
flymake-mode)) (ediff "/:/tmp/a.java" "/:/tmp/b.java"))'

And Emacs immediately hangs because it runs into the endless loop.
Stacktrace in this case is

Debugger entered--Lisp error: (quit)
  display-warning((flymake flymake-proc) "Failed to delete dir /, error
ignored" :error "*Flymake log*")
  flymake--log-1(1 flymake-proc "Failed to delete dir %s, error ignored"
"/")
  flymake-proc--safe-delete-directory("/")
  flymake-proc--delete-temp-directory("/tmp/:/tmp/")
  flymake-proc-simple-java-cleanup()
  #f(compiled-function () #<bytecode 0x11a4ea9>)()
  flymake-proc-legacy-flymake(#f(compiled-function (&rest args) #<bytecode
0x11a4e51>))
  flymake--run-backend(flymake-proc-legacy-flymake)
  #f(compiled-function (backend) #<bytecode
0x117b195>)(flymake-proc-legacy-flymake)
  run-hook-wrapped(#f(compiled-function (backend) #<bytecode 0x117b195>)
flymake-proc-legacy-flymake)
  flymake-start((post-command) nil)
  #f(compiled-function () #<bytecode 0x1180771>)()
  set-window-buffer(nil #<buffer a.java>)
  switch-to-buffer(#<buffer a.java>)
  ediff-setup-windows-multiframe-compare(#<buffer a.java> #<buffer b.java>
nil #<buffer *Ediff Control Panel*>)
  ediff-setup-windows-multiframe(#<buffer a.java> #<buffer b.java> nil
#<buffer *Ediff Control Panel*>)
  ediff-setup-windows-default(#<buffer a.java> #<buffer b.java> nil
#<buffer *Ediff Control Panel*>)
  ediff-setup-windows(#<buffer a.java> #<buffer b.java> nil #<buffer *Ediff
Control Panel*>)
  ediff-setup(#<buffer a.java> "/:/tmp/a.java" #<buffer b.java>
"/:/tmp/b.java" nil nil nil ((ediff-job-name . ediff-files)) nil)
  ediff-files-internal("/:/tmp/a.java" "/:/tmp/b.java" nil nil ediff-files)
  ediff("/:/tmp/a.java" "/:/tmp/b.java")
  (progn (add-hook (quote prog-mode-hook) (quote flymake-mode)) (ediff
"/:/tmp/a.java" "/:/tmp/b.java"))
  eval((progn (add-hook (quote prog-mode-hook) (quote flymake-mode)) (ediff
"/:/tmp/a.java" "/:/tmp/b.java")))
  command-line-1(("-f" "toggle-debug-on-quit" "-eval" "(progn (add-hook
(quote prog-mode-hook) (quote flymake-mode)) (ediff \"/:/tmp/a.java\"
\"/:/tmp/b.java\"))"))
  command-line()
  normal-top-level()

[-- Attachment #2: Type: text/html, Size: 4665 bytes --]

bug#33524: 27.0.50; Suspicious code in flymake-proc around temporary directories

[Michael Albinus]

[-- Attachment #1: Type: text/plain, Size: 2832 bytes --]

Philipp Stephani <p.stephani2@gmail.com> writes:

Hi Philipp,

> Sorry for being imprecise. Yes, I mean quoted filenames. (It's
> possible that the issue also arises for true remote filenames, but I
> haven't checked.)

"True remote filename" is still a wrong phrase. Quoted file names and
remote file names play different games.

> You can easily reproduce user-facing problems. Assuming you have some
> binary installed that would check Java files using the legacy backend:
>
> $ touch /tmp/{a,b}.java
> $ emacs -Q -eval '(progn (add-hook (quote prog-mode-hook) (quote
> flymake-mode)) (ediff "/:/tmp/a.java" "/:/tmp/b.java"))'
>
> And Emacs immediately hangs because it runs into the endless loop.
> Stacktrace in this case is
>
> Debugger entered--Lisp error: (quit)
>   display-warning((flymake flymake-proc) "Failed to delete dir /,
> error ignored" :error "*Flymake log*")
>   flymake--log-1(1 flymake-proc "Failed to delete dir %s, error
> ignored" "/")
>   flymake-proc--safe-delete-directory("/")
>   flymake-proc--delete-temp-directory("/tmp/:/tmp/")
>   flymake-proc-simple-java-cleanup()
>   #f(compiled-function () #<bytecode 0x11a4ea9>)()
>   flymake-proc-legacy-flymake(#f(compiled-function (&rest args)
> #<bytecode 0x11a4e51>))
>   flymake--run-backend(flymake-proc-legacy-flymake)
>   #f(compiled-function (backend) #<bytecode 0x117b195>)
> (flymake-proc-legacy-flymake)
>   run-hook-wrapped(#f(compiled-function (backend) #<bytecode
> 0x117b195>) flymake-proc-legacy-flymake)
>   flymake-start((post-command) nil)
>   #f(compiled-function () #<bytecode 0x1180771>)()
>   set-window-buffer(nil #<buffer a.java>)
>   switch-to-buffer(#<buffer a.java>)
>   ediff-setup-windows-multiframe-compare(#<buffer a.java> #<buffer
> b.java> nil #<buffer *Ediff Control Panel*>)
>   ediff-setup-windows-multiframe(#<buffer a.java> #<buffer b.java> nil
> #<buffer *Ediff Control Panel*>)
>   ediff-setup-windows-default(#<buffer a.java> #<buffer b.java> nil
> #<buffer *Ediff Control Panel*>)
>   ediff-setup-windows(#<buffer a.java> #<buffer b.java> nil #<buffer
> *Ediff Control Panel*>)
>   ediff-setup(#<buffer a.java> "/:/tmp/a.java" #<buffer b.java>
> "/:/tmp/b.java" nil nil nil ((ediff-job-name . ediff-files)) nil)
>   ediff-files-internal("/:/tmp/a.java" "/:/tmp/b.java" nil nil
> ediff-files)
>   ediff("/:/tmp/a.java" "/:/tmp/b.java")
>   (progn (add-hook (quote prog-mode-hook) (quote flymake-mode)) (ediff
> "/:/tmp/a.java" "/:/tmp/b.java"))
>   eval((progn (add-hook (quote prog-mode-hook) (quote flymake-mode))
> (ediff "/:/tmp/a.java" "/:/tmp/b.java")))
>   command-line-1(("-f" "toggle-debug-on-quit" "-eval" "(progn
> (add-hook (quote prog-mode-hook) (quote flymake-mode)) (ediff
> \"/:/tmp/a.java\" \"/:/tmp/b.java\"))"))
>   command-line()
>   normal-top-level()

The following patch fixes this problem:


[-- Attachment #2: Type: text/plain, Size: 633 bytes --]

diff --git a/lisp/progmodes/flymake-proc.el b/lisp/progmodes/flymake-proc.el
index 8600be9b97..e969c5d992 100644
--- a/lisp/progmodes/flymake-proc.el
+++ b/lisp/progmodes/flymake-proc.el
@@ -874,7 +874,7 @@ flymake-proc-create-temp-with-folder-structure
   (unless (stringp file-name)
     (error "Invalid file-name"))
 
-  (let* ((dir       (file-name-directory file-name))
+  (let* ((dir       (file-name-directory (file-name-unquote file-name)))
          ;; Not sure what this slash-pos is all about, but I guess it's just
          ;; trying to remove the leading / of absolute file names.
 	 (slash-pos (string-match "/" dir))

[-- Attachment #3: Type: text/plain, Size: 279 bytes --]


However, I don't know whether this is sufficient, because I don't know
why you use quoted file names in your example. Is this something which
shall be preserved in the temporary directory, created by flymake?

João, do you have further remarks?

Best regards, Michael.

bug#33524: 27.0.50; Suspicious code in flymake-proc around temporary directories

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 4329 bytes --]

No, it looks great, for all I know... Which is very little :-)

Flymake-proc is the "legacy" backend that I packed up in a file. It
probably has many such bugs.

Thanks very much Michael and Philipp,
João



On Fri, Nov 30, 2018 at 12:59 PM Michael Albinus <michael.albinus@gmx.de>
wrote:

> Philipp Stephani <p.stephani2@gmail.com> writes:
>
> Hi Philipp,
>
> > Sorry for being imprecise. Yes, I mean quoted filenames. (It's
> > possible that the issue also arises for true remote filenames, but I
> > haven't checked.)
>
> "True remote filename" is still a wrong phrase. Quoted file names and
> remote file names play different games.
>
> > You can easily reproduce user-facing problems. Assuming you have some
> > binary installed that would check Java files using the legacy backend:
> >
> > $ touch /tmp/{a,b}.java
> > $ emacs -Q -eval '(progn (add-hook (quote prog-mode-hook) (quote
> > flymake-mode)) (ediff "/:/tmp/a.java" "/:/tmp/b.java"))'
> >
> > And Emacs immediately hangs because it runs into the endless loop.
> > Stacktrace in this case is
> >
> > Debugger entered--Lisp error: (quit)
> >   display-warning((flymake flymake-proc) "Failed to delete dir /,
> > error ignored" :error "*Flymake log*")
> >   flymake--log-1(1 flymake-proc "Failed to delete dir %s, error
> > ignored" "/")
> >   flymake-proc--safe-delete-directory("/")
> >   flymake-proc--delete-temp-directory("/tmp/:/tmp/")
> >   flymake-proc-simple-java-cleanup()
> >   #f(compiled-function () #<bytecode 0x11a4ea9>)()
> >   flymake-proc-legacy-flymake(#f(compiled-function (&rest args)
> > #<bytecode 0x11a4e51>))
> >   flymake--run-backend(flymake-proc-legacy-flymake)
> >   #f(compiled-function (backend) #<bytecode 0x117b195>)
> > (flymake-proc-legacy-flymake)
> >   run-hook-wrapped(#f(compiled-function (backend) #<bytecode
> > 0x117b195>) flymake-proc-legacy-flymake)
> >   flymake-start((post-command) nil)
> >   #f(compiled-function () #<bytecode 0x1180771>)()
> >   set-window-buffer(nil #<buffer a.java>)
> >   switch-to-buffer(#<buffer a.java>)
> >   ediff-setup-windows-multiframe-compare(#<buffer a.java> #<buffer
> > b.java> nil #<buffer *Ediff Control Panel*>)
> >   ediff-setup-windows-multiframe(#<buffer a.java> #<buffer b.java> nil
> > #<buffer *Ediff Control Panel*>)
> >   ediff-setup-windows-default(#<buffer a.java> #<buffer b.java> nil
> > #<buffer *Ediff Control Panel*>)
> >   ediff-setup-windows(#<buffer a.java> #<buffer b.java> nil #<buffer
> > *Ediff Control Panel*>)
> >   ediff-setup(#<buffer a.java> "/:/tmp/a.java" #<buffer b.java>
> > "/:/tmp/b.java" nil nil nil ((ediff-job-name . ediff-files)) nil)
> >   ediff-files-internal("/:/tmp/a.java" "/:/tmp/b.java" nil nil
> > ediff-files)
> >   ediff("/:/tmp/a.java" "/:/tmp/b.java")
> >   (progn (add-hook (quote prog-mode-hook) (quote flymake-mode)) (ediff
> > "/:/tmp/a.java" "/:/tmp/b.java"))
> >   eval((progn (add-hook (quote prog-mode-hook) (quote flymake-mode))
> > (ediff "/:/tmp/a.java" "/:/tmp/b.java")))
> >   command-line-1(("-f" "toggle-debug-on-quit" "-eval" "(progn
> > (add-hook (quote prog-mode-hook) (quote flymake-mode)) (ediff
> > \"/:/tmp/a.java\" \"/:/tmp/b.java\"))"))
> >   command-line()
> >   normal-top-level()
>
> The following patch fixes this problem:
>
> diff --git a/lisp/progmodes/flymake-proc.el
> b/lisp/progmodes/flymake-proc.el
> index 8600be9b97..e969c5d992 100644
> --- a/lisp/progmodes/flymake-proc.el
> +++ b/lisp/progmodes/flymake-proc.el
> @@ -874,7 +874,7 @@ flymake-proc-create-temp-with-folder-structure
>    (unless (stringp file-name)
>      (error "Invalid file-name"))
>
> -  (let* ((dir       (file-name-directory file-name))
> +  (let* ((dir       (file-name-directory (file-name-unquote file-name)))
>           ;; Not sure what this slash-pos is all about, but I guess it's
> just
>           ;; trying to remove the leading / of absolute file names.
>          (slash-pos (string-match "/" dir))
>
> However, I don't know whether this is sufficient, because I don't know
> why you use quoted file names in your example. Is this something which
> shall be preserved in the temporary directory, created by flymake?
>
> João, do you have further remarks?
>
> Best regards, Michael.
>


-- 
João Távora

[-- Attachment #2: Type: text/html, Size: 5684 bytes --]

bug#33524: 27.0.50; Suspicious code in flymake-proc around temporary directories

[Michael Albinus]

João Távora <joaotavora@gmail.com> writes:

Hi João,

> No, it looks great, for all I know... Which is very little :-)
>
> Flymake-proc is the "legacy" backend that I packed up in a file. It
> probably has many such bugs.

Thanks for the reply. If Philipp confirms the solution, I'll push it to
master.

> Thanks very much Michael and Philipp,
> João

Best regards, Michael.

bug#33524: 27.0.50; Suspicious code in flymake-proc around temporary directories

[Michael Albinus]

Michael Albinus <michael.albinus@gmx.de> writes:

Hi,

>> No, it looks great, for all I know... Which is very little :-)
>>
>> Flymake-proc is the "legacy" backend that I packed up in a file. It
>> probably has many such bugs.
>
> Thanks for the reply. If Philipp confirms the solution, I'll push it to
> master.

I've pushed the patch to the master branch, closing the bug.

>> Thanks very much Michael and Philipp,
>> João

Best regards, Michael.

bug#33524: 27.0.50; Suspicious code in flymake-proc around temporary directories

[Philipp Stephani]

Am Mo., 17. Dez. 2018 um 11:58 Uhr schrieb Michael Albinus
<michael.albinus@gmx.de>:
>
> Michael Albinus <michael.albinus@gmx.de> writes:
>
> Hi,
>
> >> No, it looks great, for all I know... Which is very little :-)
> >>
> >> Flymake-proc is the "legacy" backend that I packed up in a file. It
> >> probably has many such bugs.
> >
> > Thanks for the reply. If Philipp confirms the solution, I'll push it to
> > master.
>
> I've pushed the patch to the master branch, closing the bug.

Confirmed that the issue no longer happens on master, thanks.

1. Would it make sense to backport this patch onto the release branch?
The bug is annoying, and the fix seems trivial.
2. There's more suspicious code in the legacy backend that assumes
that magic or quoted filenames don't exist, is there a long-term plan
to tackle this? ("Deleting the legacy backend" might be a viable
long-term plan.)

bug#33524: 27.0.50; Suspicious code in flymake-proc around temporary directories

[Michael Albinus]

Philipp Stephani <p.stephani2@gmail.com> writes:

Hi Philipp,

> 1. Would it make sense to backport this patch onto the release branch?
> The bug is annoying, and the fix seems trivial.

That's for Eli to decide. Yes, the patch is trivial, so it seems
harmless to backport.

> 2. There's more suspicious code in the legacy backend that assumes
> that magic or quoted filenames don't exist, is there a long-term plan
> to tackle this? ("Deleting the legacy backend" might be a viable
> long-term plan.)

I'm not in the flymake business. I will help if questions arise, but I'm
not in the position to review flymake completely.

Best regards, Michael.

bug#33524: 27.0.50; Suspicious code in flymake-proc around temporary directories

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Date: Mon, 17 Dec 2018 22:08:46 +0100
> Cc: 33524-done@debbugs.gnu.org,
> 	João Távora <joaotavora@gmail.com>
> 
> Philipp Stephani <p.stephani2@gmail.com> writes:
> 
> Hi Philipp,
> 
> > 1. Would it make sense to backport this patch onto the release branch?
> > The bug is annoying, and the fix seems trivial.
> 
> That's for Eli to decide. Yes, the patch is trivial, so it seems
> harmless to backport.

Fine with me, thanks.

bug#33524: 27.0.50; Suspicious code in flymake-proc around temporary directories

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

>> > 1. Would it make sense to backport this patch onto the release branch?
>> > The bug is annoying, and the fix seems trivial.
>> 
>> That's for Eli to decide. Yes, the patch is trivial, so it seems
>> harmless to backport.
>
> Fine with me, thanks.

Cherry-picked from the master branch.

Best regards, Michael.