bug#38718: Emacs `compile' command does not handle remote (over TRAMP) file name correctly

bug#38718: Emacs `compile' command does not handle remote (over TRAMP) file name correctly

[Duncan Greatwood]

[-- Attachment #1: Type: text/plain, Size: 6458 bytes --]

Emacs version aside, this appears almost identical to the other bug
of same name: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=38648

Using tramp, I had a remote cpp file open in a window.

I pressed "C-c c", which invokes a function in my .emacs called
compile-hereish, which (in this case) invokes the compile command with
the remote directory containing the remote cpp.

Compilation runs as expected, throwing certain cpp syntax errors. Now,
if I click on the red error text in the *compilation* window, I get a
prompt like:
Find this error in (default <remote filename>):
/ssh:<username>@<hostname>:/System/Volumes/Data/<remote path to file>

If I press enter, to confirm, I get a [No match] error

I believe this may be a newly revealed issue under macos Catalina. Of
possible note (especially the last), these are the mounts on local
(macos) machine:

$ sudo mount
  /dev/disk1s5 on / (apfs, local, read-only, journaled)
  devfs on /dev (devfs, local, nobrowse)
  /dev/disk1s1 on /System/Volumes/Data (apfs, local, journaled, nobrowse)
  /dev/disk1s4 on /private/var/vm (apfs, local, journaled, nobrowse) map
  auto_home on /System/Volumes/Data/home (autofs, automounted, nobrowse)

Might the fix in 27.1 referenced in report #38648 need to be back-ported
to 26.x?

As per autogenerated report below, I am using emacs 26.2 installed via
brew on macos.

The remote machine is running Ubuntu and:
$ ssh -V
OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017


In GNU Emacs 26.2 (build 1, x86_64-apple-darwin18.2.0, NS appkit-1671.20
Version 10.14.3 (Build 18D109))
 of 2019-04-12 built on builder10-14.porkrind.org
Windowing system distributor 'Apple', version 10.3.1894
Recent messages:
Tramp: Encoding local file
‘/var/folders/nh/nhxs1bmd3h79lvh_z2lpfl700000gn/T/tramp.KiHs9f.cpp’ using
‘(lambda (beg end) (let ((coding-system-for-write (quote binary))
(coding-system-for-read (quote binary))) (apply (quote
tramp-call-process-region) (quote (tramp-file-name ssh <username> nil
<hostname> nil
/home/<username>/Dropbox/progs/FUSEScatterFS/sbshared/src/SB_RemCli.cpp
nil)) beg end (car (split-string env GZIP= gzip)) t t nil (cdr
(split-string env GZIP= gzip)))) (base64-encode-region (point-min)
(point-max)))’...done
Tramp: Decoding remote file
‘/ssh:<username>@<hostname>:/home/<username>/Dropbox/progs/FUSEScatterFS/sbshared/src/SB_RemCli.cpp’
using ‘(base64 -d -i | env GZIP= gzip -d >%s)’...done
Wrote
/ssh:<username>@<hostname>:/home/<username>/Dropbox/progs/FUSEScatterFS/sbshared/src/SB_RemCli.cpp
Tramp: Opening connection for <username>@<hostname> using ssh...
Tramp: Sending command ‘exec ssh -q -l <username>  -o ControlMaster=auto -o
ControlPath='tramp.%C' -o ControlPersist=no -e none <hostname>’
Tramp: Waiting for prompts from remote shell...done
Tramp: Found remote shell prompt on ‘<hostname>’
Tramp: Opening connection for <username>@<hostname> using ssh...done
Compilation finished
Wrote /Users/<username>/.emacs.d/dotemacs-persistent-scratch.txt

Configured using:
 'configure --with-ns '--enable-locallisppath=/Library/Application
 Support/Emacs/${version}/site-lisp:/Library/Application
 Support/Emacs/site-lisp' --with-modules'

Configured features:
NOTIFY ACL GNUTLS LIBXML2 ZLIB TOOLKIT_SCROLL_BARS NS MODULES THREADS

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Compilation

Minor modes in effect:
  diff-auto-refine-mode: t
  shell-dirtrack-mode: t
  global-undo-tree-mode: t
  undo-tree-mode: t
  global-auto-complete-mode: t
  show-paren-mode: t
  delete-selection-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
/Users/<username>/.emacs.d/lisp/linum hides
/Applications/Emacs.app/Contents/Resources/lisp/linum

Features:
(shadow sort mail-extr emacsbug message rmc puny seq rfc822 mml mml-sec
epa derived epg epg-config gnus-util rmail rmail-loaddefs mm-decode
mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader
sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils
compile thingatpt linum cus-edit cus-start cus-load wid-edit cc-mode
cc-fonts cc-guess cc-menus cc-cmds cc-styles cc-align cc-engine cc-vars
cc-defs vc-git diff-mode easymenu make-mode dired dired-loaddefs files-x
misearch multi-isearch elec-pair tramp-cmds tramp-cache tramp-sh tramp
tramp-compat tramp-loaddefs trampver shell pcomplete comint ansi-color
ring parse-time format-spec auth-source cl-seq eieio byte-opt bytecomp
byte-compile cconv eieio-core eieio-loaddefs password-cache refill
easy-mmode server persistent-todo persistent-scratch
solarized-dark-theme solarized color dash advice undo-tree diff
auto-complete-config auto-complete popup string-inflection paren unbound
cl-macs cl gv delsel edmacro kmacro cl-loaddefs cl-lib time-date tooltip
eldoc electric uniquify ediff-hook vc-hooks lisp-float-type mwheel
term/ns-win ns-win ucs-normalize mule-util term/common-win tool-bar dnd
fontset image regexp-opt fringe tabulated-list replace newcomment
text-mode elisp-mode lisp-mode prog-mode register page menu-bar
rfn-eshadow isearch timer select scroll-bar mouse jit-lock font-lock
syntax facemenu font-core term/tty-colors frame cl-generic cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese composite charscript charprop
case-table epa-hook jka-cmpr-hook help simple abbrev obarray minibuffer
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote threads kqueue cocoa ns
multi-tty make-network-process emacs)

Memory information: ((conses 16 579216 42454) (symbols 48 31163 2)
(miscs 40 320 1931) (strings 32 240818 5539) (string-bytes 1 2586465)
(vectors 16 55106) (vector-slots 8 1767127 124118) (floats 8 330 479)
(intervals 56 9362 551) (buffers 992 35))

[-- Attachment #2: Type: text/html, Size: 7073 bytes --]

bug#38718: Emacs `compile' command does not handle remote (over TRAMP) file name correctly

[Michael Albinus]

Duncan Greatwood <dgbulk@gmail.com> writes:

Hi Duncan,

> Emacs version aside, this appears almost identical to the other bug
> of same name: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=38648
>
> Find this error in (default <remote filename>):
> /ssh:<username>@<hostname>:/System/Volumes/Data/<remote path to file>

Could you pls confirm, that <remote path to file> starts with "/home"?
And that on your local machine, "/home" is a symlink to
"/System/Volumes/Data/home"? If yes, your error is exactly the same as
bug#38648.

> $ sudo mount
>   auto_home on /System/Volumes/Data/home (autofs, automounted, nobrowse)

This looks like my suspicion. Hmm, if this is new in macOS Catalina, all
Emacsen are broken with the same.

> Might the fix in 27.1 referenced in report #38648 need to be
> back-ported to 26.x?

There are no plans for an Emacs 26.4. Emacs 27 will start its pretest
soon, and I don't know whether this problem will trigger a hotfix release.

What we could do is to publish the patch of bug#38648 on top of Emacs
26.3. Since we haven't done such an announcement yet, I don't know where
to do it.

I believe there are other fixes in Emacs 27 for macOS Catalina, which
are not included in Emacs 26.3. Maybe a special cumulative patch of
Emacs 26.3, good for macOS Catalina, could be done?

Eli? John?

Best regards, Michael.

bug#38718: Emacs `compile' command does not handle remote (over TRAMP) file name correctly

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Date: Mon, 23 Dec 2019 10:04:09 +0100
> Cc: 38718@debbugs.gnu.org
> 
> > Might the fix in 27.1 referenced in report #38648 need to be
> > back-ported to 26.x?
> 
> There are no plans for an Emacs 26.4. Emacs 27 will start its pretest
> soon, and I don't know whether this problem will trigger a hotfix release.
> 
> What we could do is to publish the patch of bug#38648 on top of Emacs
> 26.3. Since we haven't done such an announcement yet, I don't know where
> to do it.
> 
> I believe there are other fixes in Emacs 27 for macOS Catalina, which
> are not included in Emacs 26.3. Maybe a special cumulative patch of
> Emacs 26.3, good for macOS Catalina, could be done?
> 
> Eli? John?

I don't want to lose time by having another Emacs 26.x release.  I
think even Emacs 26.3 was a mistake.

If these issues cause sever problems on Catalina, we could suggest
people to apply a few patches to their own builds, and hope that
binary distributions pick up those patches.

bug#38718: Emacs `compile' command does not handle remote (over TRAMP) file name correctly

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

> If these issues cause sever problems on Catalina, we could suggest
> people to apply a few patches to their own builds, and hope that
> binary distributions pick up those patches.

Agreed. I could prepare a patch on top of Emacs 26.3.

But I don't know how to "suggest people to apply a few patches to their
own builds". Is there a canonical way?

Best regards, Michael.

bug#38718: Emacs `compile' command does not handle remote (over TRAMP) file name correctly

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: dgbulk@gmail.com,  38718@debbugs.gnu.org
> Date: Mon, 23 Dec 2019 15:14:34 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > If these issues cause sever problems on Catalina, we could suggest
> > people to apply a few patches to their own builds, and hope that
> > binary distributions pick up those patches.
> 
> Agreed. I could prepare a patch on top of Emacs 26.3.
> 
> But I don't know how to "suggest people to apply a few patches to their
> own builds". Is there a canonical way?

I suggest to have the patches posted in the relevant bug reports, and
have the problem description in etc/PROBLEMS which mentions the bug
numbers and the availability of patches for Emacs versions before 27.

Thanks.

bug#38718: Emacs `compile' command does not handle remote (over TRAMP) file name correctly

[Dmitry Gutov]

On 23.12.2019 16:34, Eli Zaretskii wrote:

> I suggest to have the patches posted in the relevant bug reports, and
> have the problem description in etc/PROBLEMS which mentions the bug
> numbers and the availability of patches for Emacs versions before 27.

Isn't etc/PROBLEMS a part of a release? Which we're not going to make.

bug#38718: Emacs `compile' command does not handle remote (over TRAMP) file name correctly

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

> I suggest to have the patches posted in the relevant bug reports, and
> have the problem description in etc/PROBLEMS which mentions the bug
> numbers and the availability of patches for Emacs versions before 27.

This would be etc/PROBLEMS of Emacs 27.0.50. Why shall it describe not
solved in Emacs 26.3, but solved in Emacs 27.0.50?

One idea I have is to provide a patch-26 package via GNU ELPA. But this
has disadvantages: It could patch only Lisp code. And it must be
guaranteed NOT to run for Emacs 27.

> Thanks.

Best regards, Michael.

bug#38718: Emacs `compile' command does not handle remote (over TRAMP) file name correctly

[Eli Zaretskii]

> Cc: 38718@debbugs.gnu.org, dgbulk@gmail.com
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 23 Dec 2019 17:39:55 +0200
> 
> On 23.12.2019 16:34, Eli Zaretskii wrote:
> 
> > I suggest to have the patches posted in the relevant bug reports, and
> > have the problem description in etc/PROBLEMS which mentions the bug
> > numbers and the availability of patches for Emacs versions before 27.
> 
> Isn't etc/PROBLEMS a part of a release? Which we're not going to make.

We are going to release Emacs 27.

bug#38718: Emacs `compile' command does not handle remote (over TRAMP) file name correctly

[Duncan Greatwood]

[-- Attachment #1: Type: text/plain, Size: 787 bytes --]

> Could you pls confirm, that <remote path to file> starts with "/home"?
[DG] Yes, that's correct

> And that on your local machine, "/home" is a symlink to
> "/System/Volumes/Data/home"?
[DG] Yes, that's also correct

> If yes, your error is exactly the same as
> bug#38648.
[DG] OK, makes sense

Regarding the discussion between Michael and Eli on the most efficient way
to proceed, if you have a patch that can be applied to 26.2 I could
certainly give it a try. Or I can try next version of emacs once released.

If you want to try to prompt the relevant binary distributors to patch, I
believe I am pretty typical for current macos emacs users in that emacs is
installed via "brew cask install emacs" which I understand obtains its
binary from https://emacsformacosx.com/.

Thanks.

[-- Attachment #2: Type: text/html, Size: 1088 bytes --]

bug#38718: Emacs `compile' command does not handle remote (over TRAMP) file name correctly

[Michael Albinus]

Duncan Greatwood <dgbulk@gmail.com> writes:

Hi Duncan,

>> If yes, your error is exactly the same as
>> bug#38648.
> [DG] OK, makes sense
>
> Regarding the discussion between Michael and Eli on the most efficient
> way to proceed, if you have a patch that can be applied to 26.2 I
> could certainly give it a try. Or I can try next version of emacs once
> released.

Well, you might try the patch applied to Emacs 27.0.50 for bug#38648 on
top of Emacs 26. Likely, it will apply.
<http://git.savannah.gnu.org/cgit/emacs.git/patch/lisp/progmodes/compile.el?id=0a10795d0bf4fe21997f907b7d6b1fe13517912b>

> If you want to try to prompt the relevant binary distributors to
> patch, I believe I am pretty typical for current macos emacs users in
> that emacs is installed via "brew cask install emacs" which I
> understand obtains its binary from https://emacsformacosx.com/.

I don't use macOS myself, and I don't know who is building this Emacs
version.

> Thanks.

Best regards, Michael.

bug#38718: Emacs `compile' command does not handle remote (over TRAMP) file name correctly

[Duncan Greatwood]

[-- Attachment #1: Type: text/plain, Size: 3564 bytes --]

Michael,

I can confirm that the patch of Emacs 27.0.50 for bug#38648 applied to
Emacs 26.2
<
http://git.savannah.gnu.org/cgit/emacs.git/patch/lisp/progmodes/compile.el?id=0a10795d0bf4fe21997f907b7d6b1fe13517912b
>
works.

This is macOS Catalina (10.15.2). Emacs 26.2 is the version currently
installed on my Macbook by brew/cask.

I rebuilt from patched version of the sources at emacsformacosx.com, which
I believe reproduces the Emacs version used by and installed by brew. I
hadn't done that before, and leave some notes here on what I did. May not
be the best way, but it did work for me.

So:
To build and patch the brew version of Emacs on macOS (including Catalina):

brew update
brew upgrade
Install https://www.java.com/en/download/mac_download.jsp (if not
already installed)
Install xcode command line tools, if not already present:
  xcode-select --install
Downloaded emacs sources from https://emacsformacosx.com/builds
  Expand the emacs sources (double click in Finder)
Download build tools from https://github.com/caldwell/build-emacs
  Expand the build tools (double click in Finder)
  cd build-emacs-master
  Move the expanded emacs sources folder into the build-emacs-master folder
Patch the code you want to change in emacs sources
  For instance, to patch lisp/progmodes/compile.el
    cd emacs-source/lisp/progmodes
    Create your patch file ("patch.diff") here
    Copy patch contents from Internet patch file to patch.diff
    In the patch.diff file, edit the --- and +++ lines to simplify thus:
      --- compile.el
      +++ compile.el
    patch -p0 <patch.diff
    (This will rename old compile.el to compile.el.orig, create new
compile.el)
    cd ../../..
    (Back to build-emacs-master folder)
tar the emacs sources back up again:
  tar -cvjSf emacs-<XX.Y>.tar.bz2 emacs-<XX.Y>
  (substitute in real emacs sources version number)
Per the build tools' github page:
  To fix the build issue (Catalina fix), do as follows:
      If not installed already:
        brew install libxml2
      Then:
        export LDFLAGS="-L/usr/local/opt/libxml2/lib"
        export CPPFLAGS="-I/usr/local/opt/libxml2/include"
        export PKG_CONFIG_PATH="/usr/local/opt/libxml2/lib/pkgconfig"
  ./build-emacs-from-tar -v -j 4 emacs-<XX.Y>.tar.bz2 release
  (Substitute in the correct emacs sources version number)
  (Note: The build will take a long time)
  ./combine-and-package -v Emacs-<XX.Y>-<AA.BB>-x86_64.tar.bz2
  (Substitute in emacs sources version for XX.Y; substitute in macos
  version - 10.15 for Catalina - for AA.BB)
On success, Emacs-<XX.Y>-universal.dmg is created
Double click Emacs-<XX.Y>-universal.dmg in Finder to install



On Mon, Dec 23, 2019 at 9:34 AM Duncan Greatwood <dgbulk@gmail.com> wrote:

> > Could you pls confirm, that <remote path to file> starts with "/home"?
> [DG] Yes, that's correct
>
> > And that on your local machine, "/home" is a symlink to
> > "/System/Volumes/Data/home"?
> [DG] Yes, that's also correct
>
> > If yes, your error is exactly the same as
> > bug#38648.
> [DG] OK, makes sense
>
> Regarding the discussion between Michael and Eli on the most efficient way
> to proceed, if you have a patch that can be applied to 26.2 I could
> certainly give it a try. Or I can try next version of emacs once released.
>
> If you want to try to prompt the relevant binary distributors to patch, I
> believe I am pretty typical for current macos emacs users in that emacs is
> installed via "brew cask install emacs" which I understand obtains its
> binary from https://emacsformacosx.com/.
>
> Thanks.
>
>

[-- Attachment #2: Type: text/html, Size: 4998 bytes --]

bug#38718: Emacs `compile' command does not handle remote (over TRAMP) file name correctly

[Michael Albinus]

Duncan Greatwood <dgbulk@gmail.com> writes:

> Michael,

Hi Duncan,

> I can confirm that the patch of Emacs 27.0.50 for bug#38648 applied to
> Emacs 26.2
> <http://git.savannah.gnu.org/cgit/emacs.git/patch/lisp/progmodes/compile.el?id=0a10795d0bf4fe21997f907b7d6b1fe13517912b>
>
> works.
>
> This is macOS Catalina (10.15.2). Emacs 26.2 is the version currently
> installed on my Macbook by brew/cask.

Thanks a lot for your feedback!

> I rebuilt from patched version of the sources at emacsformacosx.com,
> which I believe reproduces the Emacs version used by and installed by
> brew. I hadn't done that before, and leave some notes here on what I
> did. May not be the best way, but it did work for me.

I still don't know where to put your notes for the public. Since we have
it here in debbugs, people with a similar problem will find your notes,
and could apply. Hopefully.

Or at least we could point them to your message.

There's nothing else we can do, so I'm closing the bug.

Best regards, Michael.