bug#60505: 29.0.60; Fido Mode and Tramp Completion

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[Julien Roy]

When I C-x C-f and start typing a TRAMP session to an SSH host, for instance
`/ssh:`, then hit <tab>, I see a list of known hosts proposed for
completion. This is the expected behavior. If I then enable fido-mode
and do the same, instead of having known hosts proposed when hitting
<tab>, I only get a list of TRAMP methods for completion, eg. `ssh:`,
`sshx:` and `sshfs:`. I would expect to also have known hosts on
fido-mode.

To reproduce, open emacs -Q; then:
C-x C-f /ssh: <tab>
(This produces the expected behavior of proposing known hosts)
M-x fido-mode
C-x C-f /ssh: <tab>
(Now, only TRAMP methods are proposed for completion)


In GNU Emacs 29.0.60 (build 1, x86_64-pc-linux-gnu, GTK+ Version
 3.24.35, cairo version 1.17.6) of 2022-12-27 built on localhost
Repository revision: ba1ddea9dabf51c9c6e463d667bcce0b48294453
Repository branch: emacs-29
System Description: Gentoo Linux

Configured using:
 'configure --prefix=/usr --build=x86_64-pc-linux-gnu
 --host=x86_64-pc-linux-gnu --mandir=/usr/share/man
 --infodir=/usr/share/info --datadir=/usr/share --sysconfdir=/etc
 --localstatedir=/var/lib --datarootdir=/usr/share
 --disable-silent-rules --docdir=/usr/share/doc/emacs-29.0.9999-r1
 --htmldir=/usr/share/doc/emacs-29.0.9999-r1/html --libdir=/usr/lib64
 --program-suffix=-emacs-29-vcs --includedir=/usr/include/emacs-29-vcs
 --infodir=/usr/share/info/emacs-29-vcs --localstatedir=/var
 --enable-locallisppath=/etc/emacs:/usr/share/emacs/site-lisp
 --without-compress-install --without-hesiod --without-pop
 --with-file-notification=inotify --with-pdumper --enable-acl
 --with-dbus --with-modules --without-gameuser --with-libgmp --with-gpm
 --with-native-compilation=aot --without-json --without-kerberos
 --without-kerberos5 --with-lcms2 --with-xml2 --without-mailutils
 --without-selinux --without-sqlite3 --with-gnutls --without-libsystemd
 --with-threads --with-tree-sitter --without-wide-int --with-sound=alsa
 --with-zlib --with-pgtk --without-x --without-ns
 --with-toolkit-scroll-bars --without-gconf --with-gsettings
 --with-harfbuzz --without-libotf --without-m17n-flt --without-xwidgets
 --with-gif --with-jpeg --with-png --with-rsvg --with-tiff --with-webp
 --without-imagemagick --with-dumping=pdumper 'CFLAGS=-march=znver2 -O2
 -pipe' 'LDFLAGS=-Wl,-O1 -Wl,--as-needed''

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
LCMS2 LIBXML2 MODULES NATIVE_COMP NOTIFY INOTIFY PDUMPER PGTK PNG RSVG
SECCOMP SOUND THREADS TIFF TOOLKIT_SCROLL_BARS TREE_SITTER WEBP XIM GTK3
ZLIB

Important settings:
  value of $LC_MONETARY: en_US.UTF-8
  value of $LC_NUMERIC: en_US.UTF-8
  value of $LC_TIME: en_US.UTF-8
  value of $LANG: en_US.utf8
  locale-coding-system: utf-8-unix

Major mode: Org

Minor modes in effect:
  flyspell-mode: t
  org-indent-mode: t
  server-mode: t
  fido-vertical-mode: t
  icomplete-vertical-mode: t
  icomplete-mode: t
  fido-mode: t
  pixel-scroll-mode: t
  electric-pair-mode: t
  delete-selection-mode: t
  global-hl-line-mode: t
  shell-dirtrack-mode: t
  override-global-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  global-prettify-symbols-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  line-number-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
/usr/share/emacs/site-lisp/cmake-mode hides /usr/share/emacs/site-lisp/cmake/cmake-mode
/usr/share/emacs/site-lisp/desktop-entry-mode hides /usr/share/emacs/site-lisp/desktop-file-utils/desktop-entry-mode

Features:
(shadow emacsbug flyspell ispell org-indent org-element org-persist
org-id org-refile avl-tree generator oc-basic ol-eww eww xdg url-queue
ol-rmail ol-mhe ol-irc ol-info ol-gnus nnselect ol-docview doc-view
filenotify jka-compr image-mode exif ol-bibtex bibtex ol-bbdb ol-w3m
ol-doi org-link-doi org ob ob-tangle ob-ref ob-lob ob-table ob-exp
org-macro org-src ob-comint org-pcomplete org-list org-footnote
org-faces org-entities noutline outline ob-emacs-lisp ob-core ob-eval
org-cycle org-table ol org-fold org-fold-core org-keys oc org-loaddefs
find-func cal-menu calendar cal-loaddefs org-version org-compat org-macs
ebuild-mode skeleton sh-script smie treesit executable bookmark
gnus-notifications gnus-fun notifications mule-util sort gnus-cite
smiley qp mm-archive mail-extr textsec uni-scripts idna-mapping
ucs-normalize uni-confusable textsec-check gnus-async gnus-bcklg gnus-ml
disp-table gnus-topic nndraft nnmh nnrss mm-url nnfolder utf-7 epa-file
network-stream nsm gnus-agent gnus-srvr gnus-score score-mode nnvirtual
gnus-msg gnus-art mm-uu mml2015 mm-view mml-smime smime gnutls dig
gnus-cache gnus-sum shr pixel-fill kinsoku url-file svg dom comp
comp-cstr server display-line-numbers cl-extra checkdoc lisp-mnt
help-mode flymake-proc flymake project compile warnings thingatpt init
init-mentor init-flyspell init-flymake help-at-pt init-tramp init-modus
modus-vivendi-theme modus-themes pcase init-fido icomplete init-gnus
smtpmail gnus-demon nntp gnus-group gnus-undo gnus-start gnus-dbus dbus
xml gnus-cloud nnimap nnmail mail-source utf7 gnus-spec gnus-win nnoo
gnus-int gnus-range message sendmail yank-media puny rfc822 mml mml-sec
epa derived epg rfc6068 epg-config mm-decode mm-bodies mm-encode
mail-parse rfc2231 rfc2047 rfc2045 ietf-drums mailabbrev gmm-utils
mailheader gnus nnheader gnus-util text-property-search mail-utils range
mm-util mail-prsvr init-org init-ui pixel-scroll cua-base dired
dired-loaddefs elec-pair delsel hl-line init-packages lxd-tramp tramp
tramp-loaddefs trampver tramp-integration cus-edit pp cus-load icons
wid-edit files-x tramp-compat rx shell pcomplete comint ansi-osc
ansi-color ring parse-time iso8601 time-date ls-lisp format-spec
use-package use-package-ensure use-package-delight use-package-diminish
use-package-bind-key bind-key easy-mmode use-package-core site-gentoo
mentor-autoloads xml-rpc-autoloads async-autoloads
org-contacts-autoloads yaml-mode-autoloads url-scgi-autoloads package
browse-url url url-proxy url-privacy url-expand url-methods url-history
url-cookie generate-lisp-file url-domsuf url-util mailcap url-handlers
url-parse auth-source cl-seq eieio eieio-core cl-macs password-cache
json subr-x map byte-opt gv bytecomp byte-compile url-vars cl-loaddefs
cl-lib rmc iso-transl tooltip cconv eldoc paren electric uniquify
ediff-hook vc-hooks lisp-float-type elisp-mode mwheel term/pgtk-win
pgtk-win term/common-win pgtk-dnd tool-bar dnd fontset image regexp-opt
fringe tabulated-list replace newcomment text-mode lisp-mode prog-mode
register page tab-bar menu-bar rfn-eshadow isearch easymenu timer select
scroll-bar mouse jit-lock font-lock syntax font-core term/tty-colors
frame minibuffer nadvice seq simple cl-generic indonesian philippine
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite emoji-zwj charscript
charprop case-table epa-hook jka-cmpr-hook help abbrev obarray oclosure
cl-preloaded button loaddefs theme-loaddefs faces cus-face macroexp
files window text-properties overlay sha1 md5 base64 format env
code-pages mule custom widget keymap hashtable-print-readable backquote
threads dbusbind inotify dynamic-setting system-font-setting
font-render-setting cairo gtk pgtk lcms2 multi-tty make-network-process
native-compile emacs)

Memory information:
((conses 16 689456 32101)
 (symbols 48 30811 5)
 (strings 32 167646 8527)
 (string-bytes 1 23679638)
 (vectors 16 82271)
 (vector-slots 8 2107107 230233)
 (floats 8 487 403)
 (intervals 56 2195 393)
 (buffers 984 27))
<#secure method=pgpmime mode=sign>

-- 
Julien

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[Michael Albinus]

Julien Roy <julien@jroy.ca> writes:

Hi Julien,

> When I C-x C-f and start typing a TRAMP session to an SSH host, for instance
> `/ssh:`, then hit <tab>, I see a list of known hosts proposed for
> completion. This is the expected behavior. If I then enable fido-mode
> and do the same, instead of having known hosts proposed when hitting
> <tab>, I only get a list of TRAMP methods for completion, eg. `ssh:`,
> `sshx:` and `sshfs:`. I would expect to also have known hosts on
> fido-mode.
>
> To reproduce, open emacs -Q; then:
> C-x C-f /ssh: <tab>
> (This produces the expected behavior of proposing known hosts)
> M-x fido-mode
> C-x C-f /ssh: <tab>
> (Now, only TRAMP methods are proposed for completion)

This is the same as bug#51386, bug#52758, bug#53513, bug#54042. I've merged
your report with them.

Best regards, Michael.

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 1738 bytes --]


>
> When I C-x C-f and start typing a TRAMP session to an SSH host, for 
> instance `/ssh:`, then hit <tab>, I see a list of known hosts proposed 
> for completion. This is the expected behavior. If I then enable 
> fido-mode and do the same, instead of having known hosts proposed when 
> hitting <tab>, I only get a list of TRAMP methods for completion, eg. 
> `ssh:`, `sshx:` and `sshfs:`. I would expect to also have known hosts on 
> fido-mode.
>
> To reproduce, open emacs -Q; then:
> C-x C-f /ssh: <tab>
> (This produces the expected behavior of proposing known hosts)
> M-x fido-mode
> C-x C-f /ssh: <tab>
> (Now, only TRAMP methods are proposed for completion)
>

There are two problems here:

1. fido-mode sets completion-styles to 'flex', and as indicated in the 
docstring of completion-styles "Tramp host name completion (e.g., 
"/ssh:ho<TAB>") currently doesn't work if this list doesn't contain at 
least one of `basic', `emacs22' or `emacs21'."  The workaround is to add 
this to your init file:

(advice-add #'icomplete--fido-mode-setup
 	    :after (lambda () (setq completion-styles '(flex basic))))

You could also ask for that change to be made in core.

2. Doing that is not enough, because of an incompatibility between Tramp 
and the 'flex' and 'substring' mechanisms.  The Tramp manual has a 
footnote which mentions that incompatibility: "Some completion styles, 
like `substring' or `flex', require to type at least one character after 
the trailing `:'."  A one-line patch to fix it was proposed, but rejected, 
a year and a half ago.  It is attached to this email, and you can use it 
locally.  As I said in the bug thread in which that patch was proposed: "I 
expect other bug reports from confused users".

[-- Attachment #2: Improve-handling-of-non-default-completion-styles-wi.patch --]
[-- Type: text/x-diff, Size: 1089 bytes --]

From 5f4b821bbf5123422d43895bee4c847cfadf31b0 Mon Sep 17 00:00:00 2001
From: Gregory Heytings <gregory@heytings.org>
Date: Mon, 6 Sep 2021 20:52:21 +0000
Subject: [PATCH] Improve handling of non-default completion styles with Tramp
 methods

* lisp/net/tramp.el (tramp-completion-handle-file-name-all-completions):
Do not return methods when the filename already contains a complete method.
Fixes bug#50387.
---
 lisp/net/tramp.el | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/lisp/net/tramp.el b/lisp/net/tramp.el
index 4fd7a322d4..259fc1fcfc 100644
--- a/lisp/net/tramp.el
+++ b/lisp/net/tramp.el
@@ -2839,8 +2839,9 @@ tramp-completion-handle-file-name-all-completions
 			       (delq nil all-user-hosts)))))
 
 	    ;; Possible methods.
-	    (setq result
-		  (append result (tramp-get-completion-methods m)))))))
+	    (unless (string-empty-p method)
+	      (setq result
+		    (append result (tramp-get-completion-methods m))))))))
 
     ;; Unify list, add hop, remove nil elements.
     (dolist (elt result)
-- 
2.33.0

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[Michael Albinus]

Gregory Heytings <gregory@heytings.org> writes:

Hi Gregory,

> 2. Doing that is not enough, because of an incompatibility between
> Tramp and the 'flex' and 'substring' mechanisms.  The Tramp manual has
> a footnote which mentions that incompatibility: "Some completion
> styles, like `substring' or `flex', require to type at least one
> character after the trailing `:'."  A one-line patch to fix it was
> proposed, but rejected, a year and a half ago.  It is attached to this
> email, and you can use it locally.  As I said in the bug thread in
> which that patch was proposed: "I expect other bug reports from
> confused users".

I don't deny that there is a problem, and it isn't a surprise that
people report about. But I don't think that Tramp misbehaves, it does
exactly what it is specified to do.

I guess that flex and friends use completion out of the
specification. But I have no knowledge about the completion machinery;
otherwise I would have tried to find the root of the problem. I'm also
not opposed to extend the completion API for use in Tramp. What I'm
opposed to is to apply just an ad-hoc patch, which could have collateral
damages.

I still hope that somebody with more knowledge about the completion
machinery could take a stab at it.

Best regards, Michael.

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[Gregory Heytings]

>> 2. Doing that is not enough, because of an incompatibility between 
>> Tramp and the 'flex' and 'substring' mechanisms.  The Tramp manual has 
>> a footnote which mentions that incompatibility: "Some completion 
>> styles, like `substring' or `flex', require to type at least one 
>> character after the trailing `:'."  A one-line patch to fix it was 
>> proposed, but rejected, a year and a half ago.  It is attached to this 
>> email, and you can use it locally.  As I said in the bug thread in 
>> which that patch was proposed: "I expect other bug reports from 
>> confused users".
>
> I don't deny that there is a problem, and it isn't a surprise that 
> people report about. But I don't think that Tramp misbehaves, it does 
> exactly what it is specified to do.
>

You may remind that we disagreed on that point.  When a connection method 
has already been fully specified by the user, such as "/ssh:", there is no 
reason Tramp should tell the user that there are two other connection 
methods "/sshfs:" and "/sshx:".  IOW, after

C-x C-f /ssh TAB

it makes sense to tell the user that there are three possible methods: 
"/ssh:", "/sshfs:" and "/sshx:".  But after

C-x C-f /ssh: TAB

there is no reason to do that again.  What the user expects at that point 
is a list of hostnames.

>
> I guess that flex and friends use completion out of the specification. 
> But I have no knowledge about the completion machinery; otherwise I 
> would have tried to find the root of the problem. I'm also not opposed 
> to extend the completion API for use in Tramp. What I'm opposed to is to 
> apply just an ad-hoc patch, which could have collateral damages.
>
> I still hope that somebody with more knowledge about the completion 
> machinery could take a stab at it.
>

Why could we not use the ad-hoc patch, with a FIXME note, until somebody 
has the time to find a better solution, instead of letting bug reports 
about that problem accumulate?  Sure, it could possibly have a collateral 
damage, but it could as well have no collateral damage at all.

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 514 bytes --]


>
> Why could we not use the ad-hoc patch, with a FIXME note, until somebody 
> has the time to find a better solution, instead of letting bug reports 
> about that problem accumulate?  Sure, it could possibly have a 
> collateral damage, but it could as well have no collateral damage at 
> all.
>

By the way, to limit the scope of the potential collateral damages, it is 
also possible to use that condition only when completion-styles contain 
'substring' or 'flex'.

What do you think of the attached patch?

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Improve-handling-of-non-default-completion-styles-wi.patch --]
[-- Type: text/x-diff; name=Improve-handling-of-non-default-completion-styles-wi.patch, Size: 3353 bytes --]

From efe26b4dc2693619cc3d569c5da9dd2a43323044 Mon Sep 17 00:00:00 2001
From: Gregory Heytings <gregory@heytings.org>
Date: Fri, 6 Jan 2023 09:48:02 +0000
Subject: [PATCH] Improve handling of non-default completion styles with Tramp
 methods

* lisp/net/tramp.el
(tramp-completion-handle-file-name-all-completions): Do not return
Tramp methods when the filename already contains a complete method
and completion-styles contain 'substring' and/or 'flex'.

* lisp/icomplete.el (icomplete--fido-mode-setup): Add 'basic'
after 'flex' in the completion styles, for cases where 'flex' does
not return anything.  This is required by Tramp.

* doc/misc/tramp.texi (File name completion): Remove the footnote
about the limitation of the 'substring' and 'flex' completion
styles.

Fixes bug#50387, bug#51386, bug#52758, bug#53513, bug#54042 and
bug#60505.
---
 doc/misc/tramp.texi |  5 +----
 lisp/icomplete.el   |  3 ++-
 lisp/net/tramp.el   | 13 +++++++++++--
 3 files changed, 14 insertions(+), 7 deletions(-)

diff --git a/doc/misc/tramp.texi b/doc/misc/tramp.texi
index 56436d3297..c43d9a9991 100644
--- a/doc/misc/tramp.texi
+++ b/doc/misc/tramp.texi
@@ -3497,10 +3497,7 @@ File name completion
 Type @kbd{s h @value{postfixhop}} for the minibuffer completion to
 @samp{@value{prefix}ssh@value{postfixhop}}.  Typing @kbd{@key{TAB}}
 shows host names @value{tramp} extracts from @file{~/.ssh/config}
-@c bug#50387
-file, for example@footnote{Some completion styles, like
-@code{substring} or @code{flex}, require to type at least one
-character after the trailing @samp{@value{postfixhop}}.}.
+file, for example.
 
 @example
 @group
diff --git a/lisp/icomplete.el b/lisp/icomplete.el
index 9fa622017f..e91c2f254a 100644
--- a/lisp/icomplete.el
+++ b/lisp/icomplete.el
@@ -414,7 +414,8 @@ icomplete--fido-mode-setup
                 icomplete-show-matches-on-no-input t
                 icomplete-hide-common-prefix nil
                 icomplete-scroll (not (null icomplete-vertical-mode))
-                completion-styles '(flex)
+                ;; The 'basic' style is required by Tramp.
+                completion-styles '(flex basic)
                 completion-flex-nospace nil
                 completion-ignore-case t
                 read-buffer-completion-ignore-case t
diff --git a/lisp/net/tramp.el b/lisp/net/tramp.el
index f0b17ef393..8e0b0e9c6e 100644
--- a/lisp/net/tramp.el
+++ b/lisp/net/tramp.el
@@ -3022,8 +3022,17 @@ tramp-completion-handle-file-name-all-completions
 			       (delq nil all-user-hosts)))))
 
 	    ;; Possible methods.
-	    (setq result
-		  (append result (tramp-get-completion-methods m)))))))
+	    ;; FIXME: This is a workaround to handle the substring and
+	    ;; flex completion styles in a better way, until a better
+	    ;; fix to handle Tramp methods, possibly in the code of
+	    ;; these completion styles, is implemented.  See
+	    ;; bug#50387, bug#51386, bug#52758, bug#53513, bug#54042
+	    ;; and bug#60505.
+	    (unless (and (string-empty-p method)
+			 (or (memq 'substring completion-styles)
+			     (memq 'flex completion-styles)))
+	      (setq result
+		    (append result (tramp-get-completion-methods m))))))))
 
     ;; Unify list, add hop, remove nil elements.
     (dolist (elt result)
-- 
2.39.0

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[Gregory Heytings]

Michael, what do you think of that patch?  It would be regrettable to 
leave that bug unfixed in Emacs 29.

>
> By the way, to limit the scope of the potential collateral damages, it 
> is also possible to use that condition only when completion-styles 
> contain 'substring' or 'flex'.
>
> What do you think of the attached patch?
>

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[Michael Albinus]

Gregory Heytings <gregory@heytings.org> writes:

> Michael,

Hi Gregory,

> what do you think of that patch?  It would be regrettable to
> leave that bug unfixed in Emacs 29.

Yes, it would be desirable to fix this. But it isn't the end of the
world if this doesn't happen, the problem is already evident in Emacs
28, so we don't have a regression.

>> By the way, to limit the scope of the potential collateral damages,
>> it is also possible to use that condition only when
>> completion-styles contain 'substring' or 'flex'.
>>
>> What do you think of the attached patch?

And what do we want to do if there are more completion-styles like this?
Extend Tramp then?

And there are more use cases like this, which are not working
properly. Think about changing the Tramp syntax to `separate', which
means a remote file name like "[method/user@host]/path/to/file". If you
use default completion, it works like this:

- emacs -Q -l tramp --eval '(tramp-change-syntax (quote separate))'
- Type "/ [ s TAB" and you see all methods
- Continue with "s h / TAB" and you see nonsense :-(
- Continue with "d TAB" and you see all possible host names starting
  with "d"
- Select a host name and type "] TAB TAB" and you see the remote file names

And now with fido:

- emacs -Q -l tramp --eval '(tramp-change-syntax (quote separate))' -f fido-mode
- Type "/ [ s TAB" and you see no completion at all
- Continue with "s h / TAB" and you see the same nonsense :-(
- Continue with "d TAB" and you see no completion at all
- Type host name and type "] TAB TAB" and you see the remote file names

Although the default completion isn't perfect (likely a Tramp bug),
there is much more wrong with fido.

The last days, since you have shown your patch here, I've spent several
hours debugging Emacs completion in order to understand what's
up. Horrible, because Emacs completion machinery is pretty complex and
under-documented (I believe). And honestly, I'm not interested too much
in this machinery.

It is a pity, that *nobody* who knows this in detail tries to
investigate the problem here, although we have already five reports
about.

Instead, Tramp shall apply a patch which is not related there, which
adds further dependencies, which is not working for all use cases, and
which is good for making more trouble in the future. My opinion.

Anyway, I will try to investigate the completion machinery. No promise
that I will succeed.

Best regards, Michael.

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[Gregory Heytings]

Hi Michael,

>> what do you think of that patch?  It would be regrettable to leave that 
>> bug unfixed in Emacs 29.
>
> Yes, it would be desirable to fix this. But it isn't the end of the 
> world if this doesn't happen, the problem is already evident in Emacs 
> 28, so we don't have a regression.
>

It's correct that the same problem is present in Emacs 28, but it is not 
in Emacs 27, so it's a regression.  As I tried to explain in bug#50387, 
the problem is that once Tramp is loaded, Tramp methods are returned when 
completions are requested for a root directory ("/").  This problem is 
even worse since commit d5c6bf9625: prior to that commit only the 'scp' 
and 'scpx' methods were returned (in emacs -Q), now all methods are 
returned.  And this confuses the flex and substring completion mechanisms.

>>> By the way, to limit the scope of the potential collateral damages, it 
>>> is also possible to use that condition only when completion-styles 
>>> contain 'substring' or 'flex'.
>>>
>>> What do you think of the attached patch?
>
> And what do we want to do if there are more completion-styles like this? 
> Extend Tramp then?
>

Not extend Tramp, but add them, if necessary, to the list of 
completion-styles that are handled specially by Tramp at that place. 
Unless of course a better fix has been implemented in the meantime.

>
> And there are more use cases like this, which are not working properly. 
> Think about changing the Tramp syntax to `separate', which means a 
> remote file name like "[method/user@host]/path/to/file". If you use 
> default completion, it works like this:
>
> - emacs -Q -l tramp --eval '(tramp-change-syntax (quote separate))'
> - Type "/ [ s TAB" and you see all methods
> - Continue with "s h / TAB" and you see nonsense :-(
> - Continue with "d TAB" and you see all possible host names starting with "d"
> - Select a host name and type "] TAB TAB" and you see the remote file names
>
> And now with fido:
>
> - emacs -Q -l tramp --eval '(tramp-change-syntax (quote separate))' -f fido-mode
> - Type "/ [ s TAB" and you see no completion at all
> - Continue with "s h / TAB" and you see the same nonsense :-(
> - Continue with "d TAB" and you see no completion at all
> - Type host name and type "] TAB TAB" and you see the remote file names
>
> Although the default completion isn't perfect (likely a Tramp bug), 
> there is much more wrong with fido.
>

Well, that looks like a separate problem, with a non-default syntax, which 
(given the number of hits of "tramp-change-syntax" on Google or on Github) 
is apparently hardly ever used.

Nonetheless, I tried your recipe with the patch applied, and the good news 
is that with it, under Fido, Tramp behaves the same way it behaves under 
the default completions: the available methods are displayed after "/ [ s 
TAB", the same nonsense is displayed after "/ [ ssh/ TAB", and the 
possible host names are displayed after typing a letter followed by TAB.

>
> Instead, Tramp shall apply a patch which is not related there, which 
> adds further dependencies, which is not working for all use cases, and 
> which is good for making more trouble in the future. My opinion.
>

I don't understand what you mean here, sorry.

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[Michael Albinus]

[-- Attachment #1: Type: text/plain, Size: 3902 bytes --]

Gregory Heytings <gregory@heytings.org> writes:

> Hi Michael,

Hi Gregory,

>>> what do you think of that patch?  It would be regrettable to leave
>>> that bug unfixed in Emacs 29.
>>
>> Yes, it would be desirable to fix this. But it isn't the end of the
>> world if this doesn't happen, the problem is already evident in
>> Emacs 28, so we don't have a regression.
>>
>
> It's correct that the same problem is present in Emacs 28, but it is
> not in Emacs 27, so it's a regression.  As I tried to explain in
> bug#50387, the problem is that once Tramp is loaded, Tramp methods are
> returned when completions are requested for a root directory ("/").
> This problem is even worse since commit d5c6bf9625: prior to that
> commit only the 'scp' and 'scpx' methods were returned (in emacs -Q),
> now all methods are returned.  And this confuses the flex and
> substring completion mechanisms.

Yes. But this is not a Tramp fault. Completion styles like flex (and
substring, didn't test) do ignore Tramp file name syntax. They simply
think in terms of file name parts, separated by "/". This is not
appropriate for remote file names.

Tramp knows only file-name-completion and file-name-all-completions. This
is what it is called with; Tramp doesn't know anything about completion
styles. It would be an error to bring this into Tramp, based on these
two functions which are not designed to give the file name handler more
information about completion styles. This is our disagreement.

>>>> By the way, to limit the scope of the potential collateral
>>>> damages, it is also possible to use that condition only when
>>>> completion-styles contain 'substring' or 'flex'.
>>>>
>>>> What do you think of the attached patch?
>>
>> And what do we want to do if there are more completion-styles like
>> this? Extend Tramp then?
>
> Not extend Tramp, but add them, if necessary, to the list of
> completion-styles that are handled specially by Tramp at that
> place. Unless of course a better fix has been implemented in the
> meantime.

And this is exactly my fear: Tramp would depend on unrelated features,
and Tramp must follow. And you can't even guarantee that Tramp knows
all completion styles in question. Any user in the wild can specify her
own completion style.

> Well, that looks like a separate problem, with a non-default syntax,
> which (given the number of hits of "tramp-change-syntax" on Google or
> on Github) is apparently hardly ever used.

Then write a bug report, requesting to remove this Tramp syntax :-)

Seriously, this is the Tramp syntax used by XEmacs years ago. We've
promised the XEmacs converts to keep this syntax, in order to make their
life in GNU Emeacs less alienated.

> Nonetheless, I tried your recipe with the patch applied, and the good
> news is that with it, under Fido, Tramp behaves the same way it
> behaves under the default completions: the available methods are
> displayed after "/ [ s TAB", the same nonsense is displayed after "/ [
> ssh/ TAB", and the possible host names are displayed after typing a
> letter followed by TAB.

So yes, we shall at least hunt the bug displaying nonsense.

>> Instead, Tramp shall apply a patch which is not related there, which
>> adds further dependencies, which is not working for all use cases,
>> and which is good for making more trouble in the future. My opinion.
>
> I don't understand what you mean here, sorry.

Simply that it isn't Tramp's duty to fix the bug we're discussing. It
must be fixed by the flex (and possibly also substring) completion
styles.

I spent another couple of hours debugging completion in
minibuffer.el. The appended patch shows a possible fix for the flex
completion style. I don't claim this shall go into master; it is just a
rawhammer approach due to my limited knowledge about completion
styles. But it might give you an idea how a solution could look like.

Best regards, Michael.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 1996 bytes --]

diff --git a/lisp/minibuffer.el b/lisp/minibuffer.el
index 21d4607e7cf..a002143108f 100644
--- a/lisp/minibuffer.el
+++ b/lisp/minibuffer.el
@@ -3013,8 +3013,11 @@ completion-file-name-table
                                        (substring string 1)
                                        pred action))
        ((eq (car-safe action) 'boundaries)
-        (let ((start (length (file-name-directory string)))
-              (end (string-search "/" (cdr action))))
+        (let ((start (length
+                      (if (string-match-p "\\`/[-[:alnum:]]+:[^:]*\\'" string)
+                          string
+                        (file-name-directory string))))
+              (end (string-search "/\\:" (cdr action))))
           `(boundaries
             ;; if `string' is "C:" in w32, (file-name-directory string)
             ;; returns "C:/", so `start' is 3 rather than 2.
@@ -4205,6 +4208,9 @@ completion-flex--make-flex-pattern
 (defun completion-flex-try-completion (string table pred point)
   "Try to flex-complete STRING in TABLE given PRED and POINT."
   (unless (and completion-flex-nospace (string-search " " string))
+    (if (and (string-match-p "\\`/[-[:alnum:]]+:[^:]*\\'" string)
+             (eq table 'completion-file-name-table))
+        (completion-basic-try-completion string table pred point)
     (pcase-let ((`(,all ,pattern ,prefix ,suffix ,_carbounds)
                  (completion-substring--all-completions
                   string table pred point
@@ -4217,7 +4223,7 @@ completion-flex-try-completion
       ;; contain the substring "config".  FIXME: this still won't
       ;; augment "foo" to "froo" when matching "frodo" and
       ;; "farfromsober".
-      (completion-pcm--merge-try pattern all prefix suffix))))
+      (completion-pcm--merge-try pattern all prefix suffix)))))

 (defun completion-flex-all-completions (string table pred point)
   "Get flex-completions of STRING in TABLE, given PRED and POINT."

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 1800 bytes --]


>> It's correct that the same problem is present in Emacs 28, but it is 
>> not in Emacs 27, so it's a regression.  As I tried to explain in 
>> bug#50387, the problem is that once Tramp is loaded, Tramp methods are 
>> returned when completions are requested for a root directory ("/"). 
>> This problem is even worse since commit d5c6bf9625: prior to that 
>> commit only the 'scp' and 'scpx' methods were returned (in emacs -Q), 
>> now all methods are returned.  And this confuses the flex and substring 
>> completion mechanisms.
>
> Yes. But this is not a Tramp fault.
>

It definitely is Tramp's "fault".

>
> Completion styles like flex (and substring, didn't test) do ignore Tramp 
> file name syntax. They simply think in terms of file name parts, 
> separated by "/". This is not appropriate for remote file names.
>

What filenames are is defined for example by POSIX in its Definitions 
chapter ("Filename", "Pathname", "Pathname Resolution").  It is quite 
clear in those definitions that there is no place for a _method_ or a 
_host name_ in file names.  Something with a method, a host name and a 
path is a different beast: it's an URI.  And it's not a coincidence if 
(absolute) file names start with a slash, and URI do _not_ start with a 
slash.

Tramp decided, for a reason I do not clearly understand, to invent URIs 
starting with a slash.  File name completion mechanisms rightly assume 
that file names correspond to the standard syntax of file names, and there 
is no reason (and, for that matter, no reliable way) to fix the current 
problem at the level of completion styles.

So I have yet another proposal: add a variable to let users (and modes) 
decide whether Tramp methods should be returned when completions for the 
root directory are requested.  Patch attached.

[-- Attachment #2: Improve-handling-of-Tramp-methods-by-completion-styl.patch --]
[-- Type: text/x-diff, Size: 4186 bytes --]

From e049a25599c8abf693b35736e76ba4412cbc7cbc Mon Sep 17 00:00:00 2001
From: Gregory Heytings <gregory@heytings.org>
Date: Wed, 1 Feb 2023 18:06:49 +0000
Subject: [PATCH] Improve handling of Tramp methods by completion styles.

* lisp/net/tramp.el (tramp-methods-in-completions): New defcustom.
(tramp-completion-handle-file-name-all-completions): Use it.

* lisp/icomplete.el (icomplete--fido-mode-setup): Add 'basic'
after 'flex' in the completion styles, for cases where 'flex' does
not return anything.  This is required by Tramp.  Also set the new
defcustom 'tramp-methods-in-completions' to nil.

* doc/misc/tramp.texi (File name completion): Replace the footnote
about the limitation of the 'substring' and 'flex' completion
styles: suggest to use the new defcustom.

Fixes bug#50387, bug#51386, bug#52758, bug#53513, bug#54042 and
bug#60505.
---
 doc/misc/tramp.texi |  7 +++----
 lisp/icomplete.el   |  7 ++++++-
 lisp/net/tramp.el   | 21 +++++++++++++++++++--
 3 files changed, 28 insertions(+), 7 deletions(-)

diff --git a/doc/misc/tramp.texi b/doc/misc/tramp.texi
index 56436d32970..99f317dbbb8 100644
--- a/doc/misc/tramp.texi
+++ b/doc/misc/tramp.texi
@@ -3497,10 +3497,9 @@ File name completion
 Type @kbd{s h @value{postfixhop}} for the minibuffer completion to
 @samp{@value{prefix}ssh@value{postfixhop}}.  Typing @kbd{@key{TAB}}
 shows host names @value{tramp} extracts from @file{~/.ssh/config}
-@c bug#50387
-file, for example@footnote{Some completion styles, like
-@code{substring} or @code{flex}, require to type at least one
-character after the trailing @samp{@value{postfixhop}}.}.
+file, for example@footnote{For some completion styles, like
+@code{substring} or @code{flex}, it is necessary to set
+@code{tramp-methods-in-completions} to @code{t} for this to happen.}.
 
 @example
 @group
diff --git a/lisp/icomplete.el b/lisp/icomplete.el
index 014f38b2024..079e59e28e1 100644
--- a/lisp/icomplete.el
+++ b/lisp/icomplete.el
@@ -414,7 +414,12 @@ icomplete--fido-mode-setup
                 icomplete-show-matches-on-no-input t
                 icomplete-hide-common-prefix nil
                 icomplete-scroll (not (null icomplete-vertical-mode))
-                completion-styles '(flex)
+                ;; The 'basic' style is required by Tramp.  See
+                ;; bug#51386, bug#52758, bug#53513, bug#54042 and
+                ;; bug#60505.
+                completion-styles '(flex basic)
+                ;; Tell Tramp to display its methods lazily.
+                tramp-methods-in-completions nil
                 completion-flex-nospace nil
                 completion-ignore-case t
                 read-buffer-completion-ignore-case t
diff --git a/lisp/net/tramp.el b/lisp/net/tramp.el
index 1916d50af03..533186d1633 100644
--- a/lisp/net/tramp.el
+++ b/lisp/net/tramp.el
@@ -812,6 +812,21 @@ tramp-syntax
   :initialize #'custom-initialize-default
   :set #'tramp-set-syntax)
 
+(defcustom tramp-methods-in-completions t
+  "How Tramp methods are displayed while completing file names.
+
+When t, Tramp methods are displayed unconditionally when
+completing file names, that is, as soon as completions are
+requested for the root directory.
+
+When nil, Tramp methods are displayed lazily when completing file
+names, that is, they are displayed only when no actual file in
+the root directory matches the current user input."
+  :type '(choice (const :tag "Unconditionally" t)
+                 (const :tag "Lazily" nil))
+  :require 'tramp
+  :version "29.1")
+
 (defvar tramp-prefix-format)
 (defvar tramp-prefix-regexp)
 (defvar tramp-method-regexp)
@@ -3022,8 +3037,10 @@ tramp-completion-handle-file-name-all-completions
 			       (delq nil all-user-hosts)))))
 
 	    ;; Possible methods.
-	    (setq result
-		  (append result (tramp-get-completion-methods m)))))))
+	    (unless (and (string-empty-p method)
+			 (not tramp-methods-in-completions))
+	      (setq result
+		    (append result (tramp-get-completion-methods m))))))))
 
     ;; Unify list, add hop, remove nil elements.
     (dolist (elt result)
-- 
2.39.0

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[Michael Albinus]

Gregory Heytings <gregory@heytings.org> writes:

Hi,

>>> It's correct that the same problem is present in Emacs 28, but it
>>> is not in Emacs 27, so it's a regression.  As I tried to explain in
>>> bug#50387, the problem is that once Tramp is loaded, Tramp methods
>>> are returned when completions are requested for a root directory
>>> ("/"). This problem is even worse since commit d5c6bf9625: prior to
>>> that commit only the 'scp' and 'scpx' methods were returned (in
>>> emacs -Q), now all methods are returned.  And this confuses the
>>> flex and substring completion mechanisms.
>>
>> Yes. But this is not a Tramp fault.
>
> It definitely is Tramp's "fault".

Have you even tried to check the PoC patch I've sent?

>> Completion styles like flex (and substring, didn't test) do ignore
>> Tramp file name syntax. They simply think in terms of file name
>> parts, separated by "/". This is not appropriate for remote file
>> names.
>
> What filenames are is defined for example by POSIX in its Definitions
> chapter ("Filename", "Pathname", "Pathname Resolution").  It is quite
> clear in those definitions that there is no place for a _method_ or a
> _host name_ in file names.  Something with a method, a host name and a
> path is a different beast: it's an URI.  And it's not a coincidence if
> (absolute) file names start with a slash, and URI do _not_ start with
> a slash.
>
> Tramp decided, for a reason I do not clearly understand, to invent
> URIs starting with a slash.  File name completion mechanisms rightly
> assume that file names correspond to the standard syntax of file
> names, and there is no reason (and, for that matter, no reliable way)
> to fix the current problem at the level of completion styles.

In that case I recommend you to abstain from using Tramp.

I'm so fed up.

EOT

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[Gregory Heytings]

>
> Have you even tried to check the PoC patch I've sent?
>

I did, of course.  It circumvents (a part of) the bug for the flex 
completion style indeed.  But alas it doesn't really work.  For example:

emacs -Q -l tramp -f fido-mode
C-x C-f /ssh: TAB M-down

inserts a wrong completion candidate in the minibuffer.

More importantly (as I unsuccessfully tried to convey) it circumvents the 
bug in the implementation of a particular completion mechanism, which is 
not the place where the bug lies and should be fixed.  It is not 
reasonable to expect that all existing and future file completion 
mechanisms should implement specific rules to deal with the Tramp syntax 
(or, for that matter, with the syntax of any other package).

>
> In that case I recommend you to abstain from using Tramp.
>

I do not use Tramp (and I do not use Fido, either).  I'm only trying to 
help confused users here.

>
> I'm so fed up.
>

I'm sorry to hear that what I said annoyed you.  Please consider that I 
spent a significant amount of time on this bug, too, and that I proposed 
at least three patches to fix it.

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[Eli Zaretskii]

> Cc: 60505@debbugs.gnu.org, Julien Roy <julien@jroy.ca>
> Date: Wed, 01 Feb 2023 21:27:33 +0000
> From: Gregory Heytings <gregory@heytings.org>
> 
> More importantly (as I unsuccessfully tried to convey) it circumvents the 
> bug in the implementation of a particular completion mechanism, which is 
> not the place where the bug lies and should be fixed.  It is not 
> reasonable to expect that all existing and future file completion 
> mechanisms should implement specific rules to deal with the Tramp syntax 
> (or, for that matter, with the syntax of any other package).

From my POV, it is very reasonable to expect that completion
mechanisms know about Tramp syntax and support it.  Tramp became long
ago an integral part of the Emacs core, so its syntax is as important
to support as any other syntactical aspects of file names in Emacs,
such as the "/:" "quoting".

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

Hi Eli,

>> More importantly (as I unsuccessfully tried to convey) it circumvents the
>> bug in the implementation of a particular completion mechanism, which is
>> not the place where the bug lies and should be fixed.  It is not
>> reasonable to expect that all existing and future file completion
>> mechanisms should implement specific rules to deal with the Tramp syntax
>> (or, for that matter, with the syntax of any other package).
>
> From my POV, it is very reasonable to expect that completion
> mechanisms know about Tramp syntax and support it.  Tramp became long
> ago an integral part of the Emacs core, so its syntax is as important
> to support as any other syntactical aspects of file names in Emacs,
> such as the "/:" "quoting".

I could imagine that the completion machinery offers an API that a
package could register its own idea of a file name syntax. Plus a hook,
for parts of that file name the package is responsible itself. This
would avoid the need to check in the completion machinery, which Tramp
syntax is recent.

The current API, file name handlers for file-name-completion and
file-name-all-completions, is too restricted.

Other packages but Tramp would profit as well. For example, running
"emacs -Q", typing "/: TAB", removes the colon and offers completions
for "/". I don't know whether this is really right; there is a reason
that a user has started with "/:".

Best regards, Michael.

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: Gregory Heytings <gregory@heytings.org>,  60505@debbugs.gnu.org,
>   julien@jroy.ca
> Date: Thu, 02 Feb 2023 09:25:41 +0100
> 
> > From my POV, it is very reasonable to expect that completion
> > mechanisms know about Tramp syntax and support it.  Tramp became long
> > ago an integral part of the Emacs core, so its syntax is as important
> > to support as any other syntactical aspects of file names in Emacs,
> > such as the "/:" "quoting".
> 
> I could imagine that the completion machinery offers an API that a
> package could register its own idea of a file name syntax. Plus a hook,
> for parts of that file name the package is responsible itself. This
> would avoid the need to check in the completion machinery, which Tramp
> syntax is recent.

I won't object to developing such a mechanism.  Stefan, would that
make sense and/or be reasonably practical to implement?

> Other packages but Tramp would profit as well. For example, running
> "emacs -Q", typing "/: TAB", removes the colon and offers completions
> for "/". I don't know whether this is really right; there is a reason
> that a user has started with "/:".

That's true, but completing after typing just "/:" doesn't make a lot
of sense to me; maybe I'm missing something.  OTOH, I also don't
understand why remove the colon in this case.  If we know which code
does that, perhaps we could look into its VCS history and learn
something.

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> I could imagine that the completion machinery offers an API that a
> package could register its own idea of a file name syntax.

The completion code relies on `file-name-directory`,
`directory-file-name`, etc... for that.

The problem as I see it goes as follows:

According to `file-name-directory`, in `/ssh:myhost` the part
`ssh:myhost` is an element of the `/` directory.  For this reason, the
completion machinery would expect (file-name-all-completions "s" "/") to
include "ssh:myhost" in its return list rather than only "ssh:".

Now, it's impractical for Tramp to do that.  So the end result is the
kind of bug reports we're discussing.

One way to fix the problem is for Tramp to "teach" the rest of the
system that `/ssh:` is a kind of directory, in which case the completion
machinery would know that (file-name-all-completions "s" "/") returns
"ssh:" and that "myhost" would be included only in the return value of
(file-name-all-completions "" "/ssh:").

Another is to change Tramp's syntax so that it uses the regular "/"
separator rather than ":".  This would get a similar result but without
touching `file-name-directory` and friends.

We could also consider introducing a new set of (file-name) functions

    completion-file-name-directory, completion-directory-file-name, ...

so the completion code can use its own notion of how a file name gets
separated into parts.  But introducing such a subtle distinction would
likely introduce a lot of bugs&confusion as well.


        Stefan

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[Michael Albinus]

[-- Attachment #1: Type: text/plain, Size: 2854 bytes --]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

Hi Stefan,

>> I could imagine that the completion machinery offers an API that a
>> package could register its own idea of a file name syntax.
>
> The completion code relies on `file-name-directory`,
> `directory-file-name`, etc... for that.
>
> The problem as I see it goes as follows:
>
> According to `file-name-directory`, in `/ssh:myhost` the part
> `ssh:myhost` is an element of the `/` directory.  For this reason, the
> completion machinery would expect (file-name-all-completions "s" "/") to
> include "ssh:myhost" in its return list rather than only "ssh:".
>
> Now, it's impractical for Tramp to do that.  So the end result is the
> kind of bug reports we're discussing.
>
> One way to fix the problem is for Tramp to "teach" the rest of the
> system that `/ssh:` is a kind of directory, in which case the completion
> machinery would know that (file-name-all-completions "s" "/") returns
> "ssh:" and that "myhost" would be included only in the return value of
> (file-name-all-completions "" "/ssh:").

I've played with this idea, and the appended patch makes it work,
indeed. I had to adapt expand-file-name, file-exists-p,
file-name-directory and file-name-nondirectory, but the changes look
simple. It works for me now for the default completion styles (basic
partial-completion emacs22) as well as for fido-mode, with both the
default and simplified Tramp syntax.

It doesn't work (yet) for the separated Tramp syntax, but this isn't our
major problem, and could be fixed later. Could people check how it plays
in their environment?

> Another is to change Tramp's syntax so that it uses the regular "/"
> separator rather than ":".  This would get a similar result but without
> touching `file-name-directory` and friends.

No way. We have had syntax ambiguities for Tramp in the past, and I
don't want to go this painful way, again.

> We could also consider introducing a new set of (file-name) functions
>
>     completion-file-name-directory, completion-directory-file-name, ...
>
> so the completion code can use its own notion of how a file name gets
> separated into parts.

That's what I have done in the patch, adding such functions for the
tramp-completion-file-name-handler implementation. An implementation for
directory-file-name wasn't needed, but we could add if necessary.

> But introducing such a subtle distinction would likely introduce a lot
> of bugs&confusion as well.

Perhaps. So this patch isn't a candidate for Emacs 29.1. I would push it
to the master branch (when there is positive feedback), and I would also
add it to the tramp-2-6-stable branch in the Tramp git repo. By this,
the next Tramp 2.6 release on GNU ELPA would contain this change as
well, and we could get broader feedback.

What do people think?

>         Stefan

Best regards, Michael.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 4437 bytes --]

diff --git a/lisp/net/tramp.el b/lisp/net/tramp.el
index 21dbd40b1d2..0dc7d68983b 100644
--- a/lisp/net/tramp.el
+++ b/lisp/net/tramp.el
@@ -441,6 +441,8 @@ tramp-default-method-alist
 (defconst tramp-default-method-marker "-"
   "Marker for default method in remote file names.")

+(add-to-list 'tramp-methods `(,tramp-default-method-marker))
+
 (defcustom tramp-default-user nil
   "Default user to use for transferring files.
 It is nil by default; otherwise settings in configuration files like
@@ -1414,9 +1416,13 @@ tramp-password-prompt-not-unique
 during direct remote copying with scp.")

 (defconst tramp-completion-file-name-handler-alist
-  '((file-name-all-completions
+  '((expand-file-name . tramp-completion-handle-expand-file-name)
+    (file-exists-p . tramp-completion-handle-file-exists-p)
+    (file-name-all-completions
      . tramp-completion-handle-file-name-all-completions)
-    (file-name-completion . tramp-completion-handle-file-name-completion))
+    (file-name-completion . tramp-completion-handle-file-name-completion)
+    (file-name-directory . tramp-completion-handle-file-name-directory)
+    (file-name-nondirectory . tramp-completion-handle-file-name-nondirectory))
   "Alist of completion handler functions.
 Used for file names matching `tramp-completion-file-name-regexp'.
 Operations not mentioned here will be handled by Tramp's file
@@ -1707,7 +1713,6 @@ tramp-dissect-file-name
 		     :port port :localname localname :hop hop))
 	  ;; The method must be known.
 	  (unless (or nodefault non-essential
-		      (string-equal method tramp-default-method-marker)
 		      (assoc method tramp-methods))
 	    (tramp-user-error
 	     v "Method `%s' is not known." method))
@@ -2941,6 +2946,30 @@ tramp-connectable-p
         (and vec (process-live-p (get-process (tramp-buffer-name vec))))
 	(not non-essential))))

+(defun tramp-completion-handle-expand-file-name (filename &optional directory)
+  "Like `expand-file-name' for partial Tramp files."
+  (if (file-name-absolute-p filename)
+      filename
+    (concat (or directory default-directory "/") filename)))
+
+(defun tramp-completion-handle-file-exists-p (filename)
+  "Like `file-exists-p' for partial Tramp files."
+  ;; We need special handling only when a method is needed.  Then we
+  ;; regard all files "/method:" or "/[method/" as existent, if
+  ;; "method" is a valid Tramp method.
+  (or (and (not (string-empty-p tramp-method-regexp))
+           (string-match
+	    (rx
+	     (regexp tramp-prefix-regexp)
+	     (group (regexp tramp-method-regexp))
+	     (regexp tramp-postfix-method-regexp))
+            filename)
+           ;; Is it a valid method?
+           (member
+            filename
+            (tramp-get-completion-methods (match-string 1 filename))))
+      (tramp-run-real-handler #'file-exists-p (list filename))))
+
 ;; Method, host name and user name completion.
 ;; `tramp-completion-dissect-file-name' returns a list of
 ;; `tramp-file-name' structures.  For all of them we return possible
@@ -3176,6 +3205,27 @@ tramp-get-completion-user-host
   (unless (zerop (+ (length user) (length host)))
     (tramp-completion-make-tramp-file-name method user host nil)))

+(defun tramp-completion-handle-file-name-directory (filename)
+  "Like `file-name-directory' for partial Tramp files."
+  ;; We need special handling only when a method is needed.  Then we
+  ;; return "/method:" or "/[method/", if "method" is a valid Tramp
+  ;; method.
+  (if (and (not (string-empty-p tramp-method-regexp))
+           (string-match
+            (rx (group
+	         (regexp tramp-prefix-regexp)
+                 (group (regexp tramp-method-regexp))
+	         (regexp tramp-postfix-method-regexp)))
+            filename)
+           ;; Is it a valid method?
+           (assoc (match-string 2 filename) tramp-methods))
+      (match-string 1 filename)
+    (tramp-run-real-handler #'file-name-directory (list filename))))
+
+(defun tramp-completion-handle-file-name-nondirectory (filename)
+  "Like `file-name-nondirectory' for partial Tramp files."
+  (tramp-compat-string-replace (file-name-directory filename) "" filename))
+
 (defun tramp-parse-default-user-host (method)
   "Return a list of (user host) tuples allowed to access for METHOD.
 This function is added always in `tramp-get-completion-function'

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> +(defun tramp-completion-handle-expand-file-name (filename &optional directory)
> +  "Like `expand-file-name' for partial Tramp files."
> +  (if (file-name-absolute-p filename)
> +      filename
> +    (concat (or directory default-directory "/") filename)))

Hmm... shouldn't a "/" may need to be added if `directory` or
`default-directory` doesn't end with one (except in cases like `/ssh:`)?

Then again, maybe not.  I don't understand enough of the details of when
`tramp-completion-file-name-handler` is used (IOW what is meant exactly by
"partial Tramp file name").

IIUC the `tramp-completion-file-name*` thingies operate only for
"partial Tramp file name" (so the "completion" part of their name is
technically a misnomer, tho they make sense in practice since this part
of the code is only really important during completion), i.e. file names
that should be handled by Tramp but that are incomplete because they end
before getting to specifying the desired directory/file at the remote
end (or even specifying the remote host's name).

I didn't remember this part of the design, but it sounds good and does
make changes to that part (like the proposed patch) "safer", so it's
probably OK to try it on `master`.

I wonder how this patch interacts with `locate-dominating-file`.


        Stefan

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[Michael Albinus]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

Hi Stefan,

>> +(defun tramp-completion-handle-expand-file-name (filename &optional directory)
>> +  "Like `expand-file-name' for partial Tramp files."
>> +  (if (file-name-absolute-p filename)
>> +      filename
>> +    (concat (or directory default-directory "/") filename)))
>
> Hmm... shouldn't a "/" may need to be added if `directory` or
> `default-directory` doesn't end with one (except in cases like `/ssh:`)?

Rather not. We're speaking about the tramp-completion-file-name-handler,
which is activated for file names matching
"\\`\\(?:\\)?/\\(?:\\(?:\\(-\\|[[:alnum:]]\\{2,\\}\\)\\(?::\\)\\(?:\\([^/:|[:blank:]]+\\)\\(?:@\\)\\)?\\(\\(?:[%._[:alnum:]-]+\\|\\(?:\\[\\)\\(?:\\(?:[[:alnum:]]*:\\)+[.[:alnum:]]*\\)?\\(?:]\\)\\)\\(?:\\(?:#\\)\\(?:[[:digit:]]+\\)\\)?\\)?\\)\\(?:|\\)\\)*\\(?:\\(?:-\\|[[:alnum:]]+\\)\\(?:\\(?::\\)\\(?:[%._[:alnum:]-]+\\)?\\)?\\)?\\'".

It looks complex, but in practice it is everything until
"/method:user@host" without a trailing colon. No slash there.

It's value is

--8<---------------cut here---------------start------------->8---
    (rx
     bos
     ;; `file-name-completion' uses absolute paths for matching.
     ;; This means that on W32 systems, something like
     ;; "/ssh:host:~/path" becomes "c:/ssh:host:~/path".  See also
     ;; `tramp-drop-volume-letter'.
     (? (regexp tramp-volume-letter-regexp))
     ;; We cannot use `tramp-prefix-regexp', because it starts with `bol'.
     (literal tramp-prefix-format)

     ;; Optional multi hops.
     (* (regexp tramp-remote-file-name-spec-regexp)
        (regexp tramp-postfix-hop-regexp))

     ;; Last hop.
     (? (regexp tramp-completion-method-regexp)
	;; Method separator, user name and host name.
	(? (regexp tramp-postfix-method-regexp)
	   ;; This is a little bit lax, but it serves.
	   (? (regexp tramp-host-regexp))))

     eos)
--8<---------------cut here---------------end--------------->8---

> Then again, maybe not.  I don't understand enough of the details of when
> `tramp-completion-file-name-handler` is used (IOW what is meant exactly by
> "partial Tramp file name").

Exactly that.

> I didn't remember this part of the design, but it sounds good and does
> make changes to that part (like the proposed patch) "safer", so it's
> probably OK to try it on `master`.

There is no special design for it. Ordinary file name handler machinery.

> I wonder how this patch interacts with `locate-dominating-file`.

I'll check tomorrow. But since there isn't a slash in such filenames
(except the leading one), I expect it shall work. Needs more testing,
this case and other use cases.

>         Stefan

Best regards, Michael.

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> I didn't remember this part of the design, but it sounds good and does
>> make changes to that part (like the proposed patch) "safer", so it's
>> probably OK to try it on `master`.
> There is no special design for it. Ordinary file name handler machinery.

I was referring to the design choice of approximating "completion
context" by "partial Tramp file name".

>> I wonder how this patch interacts with `locate-dominating-file`.
> I'll check tomorrow. But since there isn't a slash in such filenames
> (except the leading one), I expect it shall work. Needs more testing,
> this case and other use cases.

I was thinking of what happens when `locate-dominating-file` does:

      try /ssh:host:./foo/bar/.git
      try /ssh:host:./foo/.git
      try /ssh:host:./.git
    ? try /ssh:.git      ?
      try /.git


-- Stefan

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[Michael Albinus]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

Hi Stefan,

>>> I didn't remember this part of the design, but it sounds good and does
>>> make changes to that part (like the proposed patch) "safer", so it's
>>> probably OK to try it on `master`.
>> There is no special design for it. Ordinary file name handler machinery.
>
> I was referring to the design choice of approximating "completion
> context" by "partial Tramp file name".

According to the ChangeLog, this was added to Tramp in August 2002. I
don't remember a public discussion about, it was just agreed between Kai
and me.

>>> I wonder how this patch interacts with `locate-dominating-file`.
>> I'll check tomorrow. But since there isn't a slash in such filenames
>> (except the leading one), I expect it shall work. Needs more testing,
>> this case and other use cases.
>
> I was thinking of what happens when `locate-dominating-file` does:
>
>       try /ssh:host:./foo/bar/.git
>       try /ssh:host:./foo/.git
>       try /ssh:host:./.git
>     ? try /ssh:.git      ?
>       try /.git

Don't know. Likely no problem, because

(file-name-directory (directory-file-name "/ssh:host:"))
=> "/ssh:host:"

Perhaps it is a good idea anyway, to add a regexp matching "/ssh:host:"
to locate-dominating-stop-dir-regexp.

> -- Stefan

Best regards, Michael.

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> I was referring to the design choice of approximating "completion
>> context" by "partial Tramp file name".
> According to the ChangeLog, this was added to Tramp in August 2002. I
> don't remember a public discussion about, it was just agreed between Kai
> and me.

In any case, I like it :-)

> Don't know. Likely no problem, because
>
> (file-name-directory (directory-file-name "/ssh:host:"))
> => "/ssh:host:"

Perfect!

> Perhaps it is a good idea anyway, to add a regexp matching "/ssh:host:"
> to locate-dominating-stop-dir-regexp.

I'd rather not if it's not needed.


        Stefan

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[Gregory Heytings]

>
> I've played with this idea, and the appended patch makes it work, 
> indeed. I had to adapt expand-file-name, file-exists-p, 
> file-name-directory and file-name-nondirectory, but the changes look 
> simple. It works for me now for the default completion styles (basic 
> partial-completion emacs22) as well as for fido-mode, with both the 
> default and simplified Tramp syntax.
>

Congratulations, that seems to work correctly indeed.  It fixes the bug in 
Tramp, but not in the way I suggested, so I guess everyone is happy now.

One minor problem:

emacs -Q -l tramp
C-x C-f / TAB

displays "-:" in the completion candidates.

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[Michael Albinus]

Gregory Heytings <gregory@heytings.org> writes:

> One minor problem:
>
> emacs -Q -l tramp
> C-x C-f / TAB
>
> displays "-:" in the completion candidates.

"-" is a valid method, it means "Use the default method".
See (info "(tramp) Default Method")

While working on the patch I've detected, that it isn't offered in file
name completion. So I've fixed this as well, silently.

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[Michael Albinus]

Michael Albinus <michael.albinus@gmx.de> writes:

Hi,

> I've played with this idea, and the appended patch makes it work,
> indeed. I had to adapt expand-file-name, file-exists-p,
> file-name-directory and file-name-nondirectory, but the changes look
> simple. It works for me now for the default completion styles (basic
> partial-completion emacs22) as well as for fido-mode, with both the
> default and simplified Tramp syntax.
>
> It doesn't work (yet) for the separated Tramp syntax, but this isn't our
> major problem, and could be fixed later. Could people check how it plays
> in their environment?

I've improved the patch, and I've committed it to master. It works now
for all different Tramp syntaxes as well as for all completion styles
except `initials' and `shorthand'. Likely, these two are not so relevant
for file name completion.

There's also a new test, tramp-test26-interactive-file-name-completion.
It doesn't cover yet user name completion and multi-hop file name
completion, these are still open. I'll keep the bugs open until this has
been fixed as well.

> Perhaps. So this patch isn't a candidate for Emacs 29.1. I would push it
> to the master branch (when there is positive feedback), and I would also
> add it to the tramp-2-6-stable branch in the Tramp git repo. By this,
> the next Tramp 2.6 release on GNU ELPA would contain this change as
> well, and we could get broader feedback.

The upcoming Tramp 2.6.0.2, being released on GNU ELPA later this month,
will contain the fixes.

Best regards, Michael.

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[Gregory Heytings]

>
> I've improved the patch, and I've committed it to master. It works now 
> for all different Tramp syntaxes as well as for all completion styles 
> except `initials' and `shorthand'.
>

That's great, thanks for your work Michael!

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[Michael Heerdegen]

Michael Albinus <michael.albinus@gmx.de> writes:

> I've improved the patch, and I've committed it to master. It works now
> for all different Tramp syntaxes as well as for all completion styles
> except `initials' and `shorthand'. Likely, these two are not so relevant
> for file name completion.

This breaks Emacs horribly: in emacs -Q (require 'tramp) and M-x dired RET
for example:

Debugger entered--Lisp error: (file-missing "Searching for program" "No such file or directory" "ls")
  (call-process "ls" nil nil nil "--dired")
  ...
  (dired-noselect "~/" nil)

All programs seem to be gone, shells etc.  I thought my Laptop had died!
Rebasing with your commit omitted fixes the problem.


Michael.

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[Michael Albinus]

[-- Attachment #1: Type: text/plain, Size: 904 bytes --]

Michael Heerdegen <michael_heerdegen@web.de> writes:

Hi Michael,

>> I've improved the patch, and I've committed it to master. It works now
>> for all different Tramp syntaxes as well as for all completion styles
>> except `initials' and `shorthand'. Likely, these two are not so relevant
>> for file name completion.
>
> This breaks Emacs horribly: in emacs -Q (require 'tramp) and M-x dired RET
> for example:
>
> Debugger entered--Lisp error: (file-missing "Searching for program" "No such file or directory" "ls")
>   (call-process "ls" nil nil nil "--dired")
>   ...
>   (dired-noselect "~/" nil)

I'm very sorry about this. Strangely, I cannot reproduce the problem
with your recipe.

> All programs seem to be gone, shells etc.  I thought my Laptop had died!
> Rebasing with your commit omitted fixes the problem.

Does the appended patch fixes this for you?

> Michael.

Best regards, Michael.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 601 bytes --]

diff --git a/lisp/net/tramp.el b/lisp/net/tramp.el
index 69812506e48..b75a1816fdb 100644
--- a/lisp/net/tramp.el
+++ b/lisp/net/tramp.el
@@ -2773,7 +2773,7 @@ tramp-completion-file-name-handler
   "Invoke Tramp file name completion handler for OPERATION and ARGS.
 Falls back to normal file name handler if no Tramp file name handler exists."
   (if-let
-      ((fn (and tramp-mode
+      ((fn (and tramp-mode minibuffer-completing-file-name
 		(assoc operation tramp-completion-file-name-handler-alist))))
       (save-match-data (apply (cdr fn) args))
     (tramp-run-real-handler operation args)))

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[Michael Heerdegen]

Michael Albinus <michael.albinus@gmx.de> writes:

> I'm very sorry about this. Strangely, I cannot reproduce the problem
> with your recipe.

Strange indeed.

> index 69812506e48..b75a1816fdb 100644
> --- a/lisp/net/tramp.el
> +++ b/lisp/net/tramp.el
> @@ -2773,7 +2773,7 @@ tramp-completion-file-name-handler
>    "Invoke Tramp file name completion handler for OPERATION and ARGS.
>  Falls back to normal file name handler if no Tramp file name handler exists."
>    (if-let
> -      ((fn (and tramp-mode
> +      ((fn (and tramp-mode minibuffer-completing-file-name
>  		(assoc operation tramp-completion-file-name-handler-alist))))

But this fixes the issue for me.


Thanks,

Michael.

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[Michael Albinus]

Michael Heerdegen <michael_heerdegen@web.de> writes:

Hi Michael,

>> index 69812506e48..b75a1816fdb 100644
>> --- a/lisp/net/tramp.el
>> +++ b/lisp/net/tramp.el
>> @@ -2773,7 +2773,7 @@ tramp-completion-file-name-handler
>>    "Invoke Tramp file name completion handler for OPERATION and ARGS.
>>  Falls back to normal file name handler if no Tramp file name handler exists."
>>    (if-let
>> -      ((fn (and tramp-mode
>> +      ((fn (and tramp-mode minibuffer-completing-file-name
>>  		(assoc operation tramp-completion-file-name-handler-alist))))
>
> But this fixes the issue for me.

Thanks for the feedback, I've pushed it to the master branch.

> Thanks,
>
> Michael.

Best regards, Michael.

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[Michael Heerdegen]

Michael Albinus <michael.albinus@gmx.de> writes:

> Thanks for the feedback, I've pushed it to the master branch.

It still happens sometimes:

| Debugger entered--Lisp error: (file-missing "Searching for program" "No such file or directory" "ls")
|   (call-process "ls" nil nil nil "--dired")
|   (dired-insert-directory "/home/" "-ahl" nil nil t)
|   (dired-readin-insert)
|   (#f(compiled-function () #<bytecode 0x8e6bc1f0877cf80>))
|   (combine-change-calls-1 1 1 #f(compiled-function () #<bytecode 0x8e6bc1f0877cf80>))
|   (dired-readin)
|   (#f(compiled-function (dir-or-list &optional switches mode) #<bytecode -0x5d986f621ad9bc8>) "/home/" nil)
|   (apply #f(compiled-function (dir-or-list &optional switches mode) #<bytecode -0x5d986f621ad9bc8>) ("/home/" nil))
|   (dired-internal-noselect "/home/" nil)
|   (dired-noselect "/home/" nil)
|   (#f(compiled-function (dirname &optional switches) "\"Edit\" directory DIRNAME--delete, rename, print, etc. some files in it.\nOptional second argument SWITCHES specifies the options to be used\nwhen invoking `insert-directory-program', usually `ls', which produces\nthe listing of the directory files and their attributes.\nInteractively, a prefix argument will cause the command to prompt\nfor SWITCHES.\n\nIf DIRNAME is a string, Dired displays a list of files in DIRNAME (which\nmay also have shell wildcards appended to select certain files).\n\nIf DIRNAME is a cons, its first element is taken as the directory name\nand the rest as an explicit list of files to make directory entries for.\nIn this case, SWITCHES are applied to each of the files separately, and\ntherefore switches that control the order of the files in the produced\nlisting have no effect.\n\n\\<dired-mode-map>You can flag files for deletion with \\[dired-flag-file-deletion] and then\ndelete them by typing \\[dired-do-flagged-delete].\nType \\[describe-mode] after entering Dired for more info.\n\nIf DIRNAME is already in a Dired buffer, that buffer is used without refresh." (interactive (dired-read-dir-and-switches "")) #<bytecode -0x14edfe378d0a8e85>) "/home/" nil)
|   (ls-lisp--dired #f(compiled-function (dirname &optional switches) "\"Edit\" directory DIRNAME--delete, rename, print, etc. some files in it.\nOptional second argument SWITCHES specifies the options to be used\nwhen invoking `insert-directory-program', usually `ls', which produces\nthe listing of the directory files and their attributes.\nInteractively, a prefix argument will cause the command to prompt\nfor SWITCHES.\n\nIf DIRNAME is a string, Dired displays a list of files in DIRNAME (which\nmay also have shell wildcards appended to select certain files).\n\nIf DIRNAME is a cons, its first element is taken as the directory name\nand the rest as an explicit list of files to make directory entries for.\nIn this case, SWITCHES are applied to each of the files separately, and\ntherefore switches that control the order of the files in the produced\nlisting have no effect.\n\n\\<dired-mode-map>You can flag files for deletion with \\[dired-flag-file-deletion] and then\ndelete them by typing \\[dired-do-flagged-delete].\nType \\[describe-mode] after entering Dired for more info.\n\nIf DIRNAME is already in a Dired buffer, that buffer is used without refresh." (interactive (dired-read-dir-and-switches "")) #<bytecode -0x14edfe378d0a8e85>) "/home/")
|   (apply ls-lisp--dired #f(compiled-function (dirname &optional switches) "\"Edit\" directory DIRNAME--delete, rename, print, etc. some files in it.\nOptional second argument SWITCHES specifies the options to be used\nwhen invoking `insert-directory-program', usually `ls', which produces\nthe listing of the directory files and their attributes.\nInteractively, a prefix argument will cause the command to prompt\nfor SWITCHES.\n\nIf DIRNAME is a string, Dired displays a list of files in DIRNAME (which\nmay also have shell wildcards appended to select certain files).\n\nIf DIRNAME is a cons, its first element is taken as the directory name\nand the rest as an explicit list of files to make directory entries for.\nIn this case, SWITCHES are applied to each of the files separately, and\ntherefore switches that control the order of the files in the produced\nlisting have no effect.\n\n\\<dired-mode-map>You can flag files for deletion with \\[dired-flag-file-deletion] and then\ndelete them by typing \\[dired-do-flagged-delete].\nType \\[describe-mode] after entering Dired for more info.\n\nIf DIRNAME is already in a Dired buffer, that buffer is used without refresh." (interactive (dired-read-dir-and-switches "")) #<bytecode -0x14edfe378d0a8e85>) "/home/")
|   (dired "/home/")
|   (helm-point-file-in-dired "/home/micha/.")
|   (funcall helm-point-file-in-dired "/home/micha/.")
|   ((closure ((action . helm-point-file-in-dired)) (arg) (setq command-history (cons (list 'funcall (list 'function action) (list 'quote arg)) command-history)) (funcall action arg)) "/home/micha/.")
|   (helm-execute-selection-action-1)
|   (helm-execute-selection-action)
|   (helm-internal helm-source-find-files "/home/micha/" "Find files or url: " nil nil "*helm find files*" nil nil nil)
|   (apply helm-internal (helm-source-find-files "/home/micha/" "Find files or url: " nil nil "*helm find files*" nil nil nil))
|   (helm helm-source-find-files "/home/micha/" "Find files or url: " nil nil "*helm find files*" nil nil nil)
|   (apply helm (helm-source-find-files "/home/micha/" "Find files or url: " nil nil "*helm find files*" nil nil nil))
|   (helm :sources helm-source-find-files :input "/home/micha/" :case-fold-search smart :preselect nil :ff-transformer-show-only-basename nil :default nil :prompt "Find files or url: " :buffer "*helm find files*")
|   (helm-find-files-1 "/home/micha/" nil)
|   (helm-find-files nil)
|   (funcall-interactively helm-find-files nil)
|   (call-interactively helm-find-files nil nil)
|   (command-execute helm-find-files)

Can I help with this in any way?  You can also give me a private phone
call if that would the fastest way of investigation.


Michael.

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[Michael Albinus]

dick <dick.r.chiang@gmail.com> writes:

> It's ironic that the guy who could most benefit from checking EMBA
> before committing a change is also the guy in charge of EMBA.

You cannot "check EMBA before committing a change". EMBA fetches changes
from Emacs' remote git repository.

Best regards, Michael.

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[dick]

EMBA, being a Gitlab product, was designed to accommodate branches, the
idea being you vet your changes on a feature branch before merging to
master.  Something for management to think about since no one, including
the guy in charge of EMBA, seems to do this.

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[Michael Albinus]

dick <dick.r.chiang@gmail.com> writes:

Hi,

> EMBA, being a Gitlab product, was designed to accommodate branches, the
> idea being you vet your changes on a feature branch before merging to
> master.  Something for management to think about since no one, including
> the guy in charge of EMBA, seems to do this.

Here you have a point. I didn't expect so much trouble in this case, and
so I haven't used a branch. Which would have other disadvantages, for
example less people to test.

And a branch to be (pre-)tested on EMBA wouldn't help here. I have added
ERT tests for the new behavior, and they run on EMBA successfully. The
problem is the *interactive* reading of file names - something which
cannot be tested on EMBA.

Best regards, Michael.

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: 60505@debbugs.gnu.org,  Michael Heerdegen <michael_heerdegen@web.de>,
>   Gregory Heytings <gregory@heytings.org>,  Stefan Monnier
>  <monnier@iro.umontreal.ca>,  julien@jroy.ca,  Eli Zaretskii <eliz@gnu.org>
> Date: Wed, 08 Feb 2023 14:08:04 +0100
> 
> dick <dick.r.chiang@gmail.com> writes:
> 
> Hi,
> 
> > EMBA, being a Gitlab product, was designed to accommodate branches, the
> > idea being you vet your changes on a feature branch before merging to
> > master.  Something for management to think about since no one, including
> > the guy in charge of EMBA, seems to do this.
> 
> Here you have a point. I didn't expect so much trouble in this case, and
> so I haven't used a branch. Which would have other disadvantages, for
> example less people to test.
> 
> And a branch to be (pre-)tested on EMBA wouldn't help here. I have added
> ERT tests for the new behavior, and they run on EMBA successfully. The
> problem is the *interactive* reading of file names - something which
> cannot be tested on EMBA.

Michael, please don't sweat over this.  Using temporary branches for
CI-style testing before committing to mainline requires a very
different style and procedures of development than what we have.  We
don't have the resources to use those development procedures on a
routine basis, and the group of people who can be vaguely described as
"the development team" -- those who contribute changes frequently
enough to benefit from test-before-commit workflow -- is too diverse
and too decentralized to expect them to abide by the discipline
required for implementing CI-driven workflows.  dick.r.chiang goes the
easy way of commenting from the peanut gallery (in his usual arrogant
and condescending style) instead of doing what is expected from a
well-meaning community member: volunteer to do this job himself and
find enough volunteers for us to be able to go anywhere near using CI
routinely.

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[dick]

EZ> dick.r.chiang goes the easy way of commenting from the peanut gallery

This was my dialogue with the EMBA guy in November 2021.

Guy: Your patch to gitlab-ci.yml doesn't help here. [Ed. he says "your
blank doesn't help here" a lot]

Me: If you lack the time to make EMBA relevant, let me do it.  I can do it.

Guy: Thanks for the offer, patches welcome! If you like, I could add also an
account for you on emba.gnu.org.

Me: By "account for you on emba.gnu.org," I assume you mean an ssh login
with write privileges to the gitlab installation.  If so, I accept.

Guy: No, I'm speaking about an account of that gitlab instance. OK?

Me: Then never mind, good sir.  Among other issues, I wanted https://emba.gnu.org
to respond within the realm of useabilty (sooner than the current ~10s).  I'm
going to need ssh for that.

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[Michael Albinus]

dick <dick.r.chiang@gmail.com> writes:

Hi,

> EZ> dick.r.chiang goes the easy way of commenting from the peanut gallery
>
> This was my dialogue with the EMBA guy in November 2021.
>
> Guy: Your patch to gitlab-ci.yml doesn't help here. [Ed. he says "your
> blank doesn't help here" a lot]
>
> Me: If you lack the time to make EMBA relevant, let me do it.  I can do it.
>
> Guy: Thanks for the offer, patches welcome! If you like, I could add also an
> account for you on emba.gnu.org.
>
> Me: By "account for you on emba.gnu.org," I assume you mean an ssh login
> with write privileges to the gitlab installation.  If so, I accept.
>
> Guy: No, I'm speaking about an account of that gitlab instance. OK?
>
> Me: Then never mind, good sir.  Among other issues, I wanted https://emba.gnu.org
> to respond within the realm of useabilty (sooner than the current ~10s).  I'm
> going to need ssh for that.

This discussion is from  bug#51399. You have forgotten to quote the rest
of this discussion. (And no, I don't want to discuss this, again.)

Best regards, Michael.

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[dick]

> And a branch to be (pre-)tested on EMBA wouldn't help here.

That's because you've largely missed the point of the Gitlab product.
Register everyone with commit rights to EMBA.  And then hope they're
conscientious enough to kick off a test run before making a nontrivial
change.  This last step would be less of an issue if management ceased
clinging to a neolithic changeset procedure that keeps the dev base
artificially small and old.

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[Michael Albinus]

Michael Heerdegen <michael_heerdegen@web.de> writes:

Hi Michael,

> It still happens sometimes:
>
> | Debugger entered--Lisp error: (file-missing "Searching for program" "No such file or directory" "ls")
> |   (call-process "ls" nil nil nil "--dired")
> |   (dired-insert-directory "/home/" "-ahl" nil nil t)
> |   (dired-readin-insert)
> |   (#f(compiled-function () #<bytecode 0x8e6bc1f0877cf80>))
> |   (combine-change-calls-1 1 1 #f(compiled-function () #<bytecode 0x8e6bc1f0877cf80>))
> |   (dired-readin)
> |   (#f(compiled-function (dir-or-list &optional switches mode) #<bytecode -0x5d986f621ad9bc8>) "/home/" nil)
> |   (apply #f(compiled-function (dir-or-list &optional switches mode) #<bytecode -0x5d986f621ad9bc8>) ("/home/" nil))
> |   (dired-internal-noselect "/home/" nil)
> |   (dired-noselect "/home/" nil)
> |   (#f(compiled-function (dirname &optional switches) "\"Edit\"
> | directory DIRNAME--delete, rename, print, etc. some files in
> | it.\nOptional second argument SWITCHES specifies the options to be
> | used\nwhen invoking `insert-directory-program', usually `ls', which
> | produces\nthe listing of the directory files and their
> | attributes.\nInteractively, a prefix argument will cause the command
> | to prompt\nfor SWITCHES.\n\nIf DIRNAME is a string, Dired displays a
> | list of files in DIRNAME (which\nmay also have shell wildcards
> | appended to select certain files).\n\nIf DIRNAME is a cons, its
> | first element is taken as the directory name\nand the rest as an
> | explicit list of files to make directory entries for.\nIn this case,
> | SWITCHES are applied to each of the files separately, and\ntherefore
> | switches that control the order of the files in the
> | produced\nlisting have no effect.\n\n\\<dired-mode-map>You can flag
> | files for deletion with \\[dired-flag-file-deletion] and
> | then\ndelete them by typing \\[dired-do-flagged-delete].\nType
> | \\[describe-mode] after entering Dired for more info.\n\nIf DIRNAME
> | is already in a Dired buffer, that buffer is used without refresh."
> | (interactive (dired-read-dir-and-switches "")) #<bytecode
> | -0x14edfe378d0a8e85>) "/home/" nil)
> |   (ls-lisp--dired #f(compiled-function (dirname &optional switches)
> | "\"Edit\" directory DIRNAME--delete, rename, print, etc. some files
> | in it.\nOptional second argument SWITCHES specifies the options to
> | be used\nwhen invoking `insert-directory-program', usually `ls',
> | which produces\nthe listing of the directory files and their
> | attributes.\nInteractively, a prefix argument will cause the command
> | to prompt\nfor SWITCHES.\n\nIf DIRNAME is a string, Dired displays a
> | list of files in DIRNAME (which\nmay also have shell wildcards
> | appended to select certain files).\n\nIf DIRNAME is a cons, its
> | first element is taken as the directory name\nand the rest as an
> | explicit list of files to make directory entries for.\nIn this case,
> | SWITCHES are applied to each of the files separately, and\ntherefore
> | switches that control the order of the files in the
> | produced\nlisting have no effect.\n\n\\<dired-mode-map>You can flag
> | files for deletion with \\[dired-flag-file-deletion] and
> | then\ndelete them by typing \\[dired-do-flagged-delete].\nType
> | \\[describe-mode] after entering Dired for more info.\n\nIf DIRNAME
> | is already in a Dired buffer, that buffer is used without refresh."
> | (interactive (dired-read-dir-and-switches "")) #<bytecode
> | -0x14edfe378d0a8e85>) "/home/")
> |   (apply ls-lisp--dired #f(compiled-function (dirname &optional
> | switches) "\"Edit\" directory DIRNAME--delete, rename, print,
> | etc. some files in it.\nOptional second argument SWITCHES specifies
> | the options to be used\nwhen invoking `insert-directory-program',
> | usually `ls', which produces\nthe listing of the directory files and
> | their attributes.\nInteractively, a prefix argument will cause the
> | command to prompt\nfor SWITCHES.\n\nIf DIRNAME is a string, Dired
> | displays a list of files in DIRNAME (which\nmay also have shell
> | wildcards appended to select certain files).\n\nIf DIRNAME is a
> | cons, its first element is taken as the directory name\nand the rest
> | as an explicit list of files to make directory entries for.\nIn this
> | case, SWITCHES are applied to each of the files separately,
> | and\ntherefore switches that control the order of the files in the
> | produced\nlisting have no effect.\n\n\\<dired-mode-map>You can flag
> | files for deletion with \\[dired-flag-file-deletion] and
> | then\ndelete them by typing \\[dired-do-flagged-delete].\nType
> | \\[describe-mode] after entering Dired for more info.\n\nIf DIRNAME
> | is already in a Dired buffer, that buffer is used without refresh."
> | (interactive (dired-read-dir-and-switches "")) #<bytecode
> | -0x14edfe378d0a8e85>) "/home/")
> |   (dired "/home/")
> |   (helm-point-file-in-dired "/home/micha/.")
> |   (funcall helm-point-file-in-dired "/home/micha/.")
> |   ((closure ((action . helm-point-file-in-dired)) (arg) (setq command-history (cons (list 'funcall (list 'function action) (list 'quote arg)) command-history)) (funcall action arg)) "/home/micha/.")
> |   (helm-execute-selection-action-1)
> |   (helm-execute-selection-action)
> |   (helm-internal helm-source-find-files "/home/micha/" "Find files or url: " nil nil "*helm find files*" nil nil nil)
> |   (apply helm-internal (helm-source-find-files "/home/micha/" "Find files or url: " nil nil "*helm find files*" nil nil nil))
> |   (helm helm-source-find-files "/home/micha/" "Find files or url: " nil nil "*helm find files*" nil nil nil)
> |   (apply helm (helm-source-find-files "/home/micha/" "Find files or url: " nil nil "*helm find files*" nil nil nil))
> |   (helm :sources helm-source-find-files :input "/home/micha/" :case-fold-search smart :preselect nil :ff-transformer-show-only-basename nil :default nil :prompt "Find files or url: " :buffer "*helm find files*")
> |   (helm-find-files-1 "/home/micha/" nil)
> |   (helm-find-files nil)
> |   (funcall-interactively helm-find-files nil)
> |   (call-interactively helm-find-files nil nil)
> |   (command-execute helm-find-files)
>
> Can I help with this in any way?  You can also give me a private phone
> call if that would the fastest way of investigation.

I'm still unable to reproduce. I've installed the needed packages from
GNU ELPA (I believe), and then I've applied

# src/emacs -Q -l tramp \
  -l ~/.emacs.d/elpa/helm-20230129.1448/helm-autoloads.el \
  -l ~/.emacs.d/elpa/helm-core-20230117.1925/helm-core-autoloads.el \
  -l ~/.emacs.d/elpa/async-20221228.1315/async-autoloads.el

M-x helm-find-files
=> custom-initialize-reset: In ‘Find Files’ source: ‘helm-find-files-get-candidates’ 
   (user-error "Error: file-notify-error (\"File watching is not available\" \"Too many open files\")")

M-x helm-find-files
/home/albinus/ RET
=> Proper dired listing.

Grrrr. Do you like to run a jitsi session together with me, for debugging?

> Michael.

Best regards, Michael.

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>>> @@ -2773,7 +2773,7 @@ tramp-completion-file-name-handler
>>>    "Invoke Tramp file name completion handler for OPERATION and ARGS.
>>>  Falls back to normal file name handler if no Tramp file name handler exists."
>>>    (if-let
>>> -      ((fn (and tramp-mode
>>> +      ((fn (and tramp-mode minibuffer-completing-file-name
>>>  		(assoc operation tramp-completion-file-name-handler-alist))))

Looks like a "quick&dirty workaround" rather than fix.


        Stefan

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[Michael Albinus]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

Hi Stefan,

>>>> @@ -2773,7 +2773,7 @@ tramp-completion-file-name-handler
>>>>    "Invoke Tramp file name completion handler for OPERATION and ARGS.
>>>>  Falls back to normal file name handler if no Tramp file name handler exists."
>>>>    (if-let
>>>> -      ((fn (and tramp-mode
>>>> +      ((fn (and tramp-mode minibuffer-completing-file-name
>>>>  		(assoc operation tramp-completion-file-name-handler-alist))))
>
> Looks like a "quick&dirty workaround" rather than fix.

Perhaps. But how could I tell Emacs, that this file name handler is
meant for (interactive) file name completion only?

>         Stefan

Best regards, Michael.

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>>>>> @@ -2773,7 +2773,7 @@ tramp-completion-file-name-handler
>>>>>    "Invoke Tramp file name completion handler for OPERATION and ARGS.
>>>>>  Falls back to normal file name handler if no Tramp file name handler exists."
>>>>>    (if-let
>>>>> -      ((fn (and tramp-mode
>>>>> +      ((fn (and tramp-mode minibuffer-completing-file-name
>>>>>  		(assoc operation tramp-completion-file-name-handler-alist))))
>>
>> Looks like a "quick&dirty workaround" rather than fix.
>
> Perhaps. But how could I tell Emacs, that this file name handler is
> meant for (interactive) file name completion only?

You can't.  Instead it should work regardless.


        Stefan

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[Michael Albinus]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

Hi Stefan,

>>>>>> @@ -2773,7 +2773,7 @@ tramp-completion-file-name-handler
>>>>>>    "Invoke Tramp file name completion handler for OPERATION and ARGS.
>>>>>>  Falls back to normal file name handler if no Tramp file name handler exists."
>>>>>>    (if-let
>>>>>> -      ((fn (and tramp-mode
>>>>>> +      ((fn (and tramp-mode minibuffer-completing-file-name
>>>>>>  		(assoc operation tramp-completion-file-name-handler-alist))))
>>>
>>> Looks like a "quick&dirty workaround" rather than fix.
>>
>> Perhaps. But how could I tell Emacs, that this file name handler is
>> meant for (interactive) file name completion only?
>
> You can't.  Instead it should work regardless.

Then we're back at the beginning. The heart of my patch is

(file-name-directory "/ssh:host") => "/ssh:"

And this shall happen only in read-file-name.

>         Stefan

Best regards, Michael.

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> You can't.  Instead it should work regardless.
>
> Then we're back at the beginning. The heart of my patch is
>
> (file-name-directory "/ssh:host") => "/ssh:"

What's the problem with that?

> And this shall happen only in read-file-name.

Why?


        Stefan

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[Michael Albinus]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

Hi Stefan,

>>> You can't.  Instead it should work regardless.
>>
>> Then we're back at the beginning. The heart of my patch is
>>
>> (file-name-directory "/ssh:host") => "/ssh:"
>
> What's the problem with that?
>
>> And this shall happen only in read-file-name.
>
> Why?

Indeed, my patch got trouble when Tramp file name completion was in the
way even if it wasn't asked for. Thanks to Michael, I could debug it in
his environment. Fix pushed to master.

There are still some cases it doesn't work as expected (user name
completion to be mentioned); I'll continue to work on this.

>         Stefan

Best regards, Michael.

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[Michael Albinus]

Version: 29.2

Michael Albinus <michael.albinus@gmx.de> writes:

> There are still some cases it doesn't work as expected (user name
> completion to be mentioned); I'll continue to work on this.

This works now fine, and also multi-hop completion passes the tests. I'm
closing this bug (and the merged ones).

Best regards, Michael.

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[Michael Heerdegen]

Michael Albinus <michael.albinus@gmx.de> writes:

> I'm very sorry about this. Strangely, I cannot reproduce the problem
> with your recipe.

If it matters: maybe it depends on PATH?  Mine currently looks like

  /home/micha/bin:.:/home/micha/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games

And I'm using non-standard shells (fish, elvish).

Michael.

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[Michael Albinus]

Michael Heerdegen <michael_heerdegen@web.de> writes:

Hi Michael,

>> I'm very sorry about this. Strangely, I cannot reproduce the problem
>> with your recipe.
>
> If it matters: maybe it depends on PATH?  Mine currently looks like
>
>   /home/micha/bin:.:/home/micha/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
>
> And I'm using non-standard shells (fish, elvish).

Anything goes, but it doesn't look like it is related. We're speaking
about Tramp file name completion (method or host part), so only
directory names like "/xxx", w/o a slash except the very first one, would matter.

Which directory is meant by "."? Being curious, would it help to remove
it from your PATH temporarily?

> Michael.

Best regards, Michael.

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[Michael Heerdegen]

Michael Albinus <michael.albinus@gmx.de> writes:

> Which directory is meant by "."? Being curious, would it help to remove
> it from your PATH temporarily?

Seems this is not the case.

Michael.

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[Gregory Heytings]

>> More importantly (as I unsuccessfully tried to convey) it circumvents 
>> the bug in the implementation of a particular completion mechanism, 
>> which is not the place where the bug lies and should be fixed.  It is 
>> not reasonable to expect that all existing and future file completion 
>> mechanisms should implement specific rules to deal with the Tramp 
>> syntax (or, for that matter, with the syntax of any other package).
>
> From my POV, it is very reasonable to expect that completion mechanisms 
> know about Tramp syntax and support it.  Tramp became long ago an 
> integral part of the Emacs core, so its syntax is as important to 
> support as any other syntactical aspects of file names in Emacs, such as 
> the "/:" "quoting".
>

The possibility that it would be necessary at some point to add an 
explicit support for the Tramp syntax(es) in the completion mechanisms 
cannot be ruled out, of course.  But this bug (and the related ones) is 
not a reason to do that, because it can easily be fixed inside Tramp.

Did you look at the options that are on the table?

Option 1 is a trivial three-line patch to tramp.el, which adds a 
conditional around an existing statement to give users and modes control 
on the way Tramp methods are displayed (unconditionally with 
tramp-methods-in-completions t, lazily with tramp-methods-in-completions 
nil):

diff --git a/lisp/net/tramp.el b/lisp/net/tramp.el
index 1916d50af03..ca08a0a2cd2 100644
--- a/lisp/net/tramp.el
+++ b/lisp/net/tramp.el
@@ -812,6 +812,8 @@ tramp-syntax
    :initialize #'custom-initialize-default
    :set #'tramp-set-syntax)

+(defcustom tramp-methods-in-completions t)
+
  (defvar tramp-prefix-format)
  (defvar tramp-prefix-regexp)
  (defvar tramp-method-regexp)
@@ -3022,8 +3024,10 @@ tramp-completion-handle-file-name-all-completions
                                (delq nil all-user-hosts)))))

             ;; Possible methods.
-           (setq result
-                 (append result (tramp-get-completion-methods m)))))))
+           (unless (and (string-empty-p method)
+                        (not tramp-methods-in-completions))
+             (setq result
+                   (append result (tramp-get-completion-methods m))))))))

      ;; Unify list, add hop, remove nil elements.
      (dolist (elt result)

Option 2 is a N-lines patch (which doesn't exist yet) to minibuffer.el 
(and possibly other files) to add support for the Tramp syntax in 
completions mechanisms.

Option 3, 4, 5 and 6 were suggested by Stefan in his last three posts: 
change the completion-table API, make Tramp teach the rest of the system 
that its methods are a kind of directory, change Tramp's syntax from e.g. 
'/ssh:' to '/ssh/', introduce a new set of completion file name functions. 
I have no idea how complex any of these solutions are, but it's clear that 
none of them can be implemented easily.

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[Juri Linkov]

> change Tramp's syntax from e.g. '/ssh:' to '/ssh/'

Why the leading slash?  URI components from RFC3986:

         foo://example.com:8042/over/there?name=ferret#nose
         \_/   \______________/\_________/ \_________/ \__/
          |           |            |            |        |
       scheme     authority       path        query   fragment

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[Michael Albinus]

Juri Linkov <juri@linkov.net> writes:

Hi Juri,

>> change Tramp's syntax from e.g. '/ssh:' to '/ssh/'
>
> Why the leading slash?  URI components from RFC3986:
>
>          foo://example.com:8042/over/there?name=ferret#nose
>          \_/   \______________/\_________/ \_________/ \__/
>           |           |            |            |        |
>        scheme     authority       path        query   fragment

This is already supported by url-handler-mode. Do

--8<---------------cut here---------------start------------->8---
M-x url-handler-mode
C-x C-f ssh://user@remote.host/
--8<---------------cut here---------------end--------------->8---

But this has other disadvantages, and it doesn't help in the host name
completion case.

Best regards, Michael.

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Cc: Eli Zaretskii <eliz@gnu.org>,  60505@debbugs.gnu.org,
>   michael.albinus@gmx.de,  Stefan Monnier <monnier@iro.umontreal.ca>,
>   julien@jroy.ca
> Date: Fri, 03 Feb 2023 09:43:14 +0200
> 
> > change Tramp's syntax from e.g. '/ssh:' to '/ssh/'
> 
> Why the leading slash?  URI components from RFC3986:
> 
>          foo://example.com:8042/over/there?name=ferret#nose
>          \_/   \______________/\_________/ \_________/ \__/
>           |           |            |            |        |
>        scheme     authority       path        query   fragment

Tramp file names are not URIs.  If you make them URIs, we will have
breakage all over the place where we are now capable of handling
remote file names.  That way lies madness.

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> Yes. But this is not a Tramp fault. Completion styles like flex (and
> substring, didn't test) do ignore Tramp file name syntax. They simply
> think in terms of file name parts, separated by "/". This is not
> appropriate for remote file names.

Actually, completion styles don't know whether they're completing file
names or not and don't really know that "/" is special in file name syntax.

This is done by `completion-file-name-table` instead.  This one, in turn
tries not to pay attention to "/" either, and to rely on file-name
functions instead (e.g. `file-name-directory`).

> Tramp knows only file-name-completion and file-name-all-completions.

Tramp could also influence the completion behavior via the other
file-name functions.
E.g. defining (file-name-directory "/ssh:foo") => "/ssh:" could fix some
of the flex completion cases discussed here.
[ Note: I'm not actually suggesting that this is the solution.
  It would likely come with its own set of problems.  ]


        Stefan

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> I don't deny that there is a problem, and it isn't a surprise that
> people report about. But I don't think that Tramp misbehaves, it does
> exactly what it is specified to do.
>
> I guess that flex and friends use completion out of the
> specification.

It's neither's fault, really.  It's a shortcoming of the intermediary,
i.e. the completion-table API.

The completion-table API only offers "prefix" completion, so the only
thing "flex" and "substring" can do when asked to complete `/ssh:`
is to ask for all the completions in `/` (because the definition of
"substring" completion means that it should also match `/foossh:bar`
and similarly for "flex").

I have some draft of a new completion-table API where it's possible for
the completion style to propagate more information about which
completions it's looking for, so for example the completion-table might
be asked to return the completions that match `/*ssh:*` or even
`/*s*s*h*:*`.  It's then up to the completion-table to do something
useful with it (of course, the API is designed such that the
completion-tables don't *have* to handle those more complex cases:
there's a general fallback mechanism which reduces those queries to
simpler ones).


        Stefan

bug#60505: 29.0.60; Fido Mode and Tramp Completion

[Julien Roy]

Hi Gregory,

Gregory Heytings <gregory@heytings.org> writes:

>
> There are two problems here:
>
> 1. fido-mode sets completion-styles to 'flex', and as indicated in the
> docstring of completion-styles "Tramp host name completion (e.g.,
> "/ssh:ho<TAB>") currently doesn't work if this list doesn't contain at
> least one of `basic', `emacs22' or `emacs21'."  The workaround is to
> add this to your init file:
>
> (advice-add #'icomplete--fido-mode-setup
> 	    :after (lambda () (setq completion-styles '(flex basic))))
>
> You could also ask for that change to be made in core.
>
> 2. Doing that is not enough, because of an incompatibility between
> Tramp and the 'flex' and 'substring' mechanisms.  The Tramp manual has
> a footnote which mentions that incompatibility: "Some completion
> styles, like `substring' or `flex', require to type at least one
> character after the trailing `:'."  A one-line patch to fix it was
> proposed, but rejected, a year and a half ago.  It is attached to this
> email, and you can use it locally.  As I said in the bug thread in
> which that patch was proposed: "I expect other bug reports from
> confused users".
>
>
<#secure method=pgpmime mode=sign>

I did that, added the snipped to my init file, and patched my
tramp.el. Now, Tramp completion works as expected in fido-mode:
C-x C-f /ssh <tab> : auto-completes tramp methods (/ssh: /sshx: /sshfs:)
C-x C-f /ssh: <tab> : auto-completes with known hosts

Thank you very much!

-- 
Julien

bug#54042: 29.0.50; fido-mode and ssh not listing hosts

[Ergus via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Hi:

Recently I have found that icomplete does not offer host completions
after /ssh:

After some check I got this minimal snippet:

emacs -Q
M-x fido-mode
C-x C-f /ssh:<tab>

Somehow fido mode removes the latest colon (:) and tries to get
completion for /ssh so not host are shown (even in the completions
buffer).

If I choose at this point some of the options (sshx:, ssh:, sshfs:) it a
buffer `ssh:` is created at / (read only of course.)

trying a bit more like:

C-x C-f /ssh:m<tab> (I have many servers starting with m in my ssh config)

It does not show anything as completion but shows [No matches] while
normally in *Completions* appear all the hosts defined in .ssh/config.

I tried with icomplete and the problem seems to be only with fido.


In GNU Emacs 29.0.50 (build 73, x86_64-pc-linux-gnu, GTK+ Version 3.24.31, cairo version 1.17.4)
 of 2022-02-16 built on Ergus
Repository revision: 8737d79be8f82bb12ca12dc1a7de37a4d2875f39
Repository branch: master
System Description: Arch Linux

Configured using:
 'configure --prefix=/home/ergo/.local/ --with-mailutils --with-json
 --with-x-toolkit=gtk3 --with-xft --with-wide-int --with-modules
 --with-cairo --with-harfbuzz --with-native-compilation --with-pgtk'

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
JSON LCMS2 LIBSYSTEMD LIBXML2 MODULES NATIVE_COMP NOTIFY INOTIFY PDUMPER
PGTK PNG RSVG SECCOMP SOUND SQLITE3 THREADS TIFF TOOLKIT_SCROLL_BARS
WEBP XIM GTK3 ZLIB

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  global-auto-revert-mode: t
  xclip-mode: t
  electric-pair-mode: t
  flyspell-mode: t
  company-mode: t
  flycheck-mode: t
  diff-hl-margin-mode: t
  composable-mark-mode: t
  composable-mode: t
  fido-vertical-mode: t
  icomplete-vertical-mode: t
  icomplete-mode: t
  fido-mode: t
  repeat-mode: t
  xterm-mouse-mode: t
  minibuffer-depth-indicate-mode: t
  winner-mode: t
  save-place-mode: t
  delete-selection-mode: t
  savehist-mode: t
  global-display-fill-column-indicator-mode: t
  display-fill-column-indicator-mode: t
  global-display-line-numbers-mode: t
  display-line-numbers-mode: t
  which-key-mode: t
  override-global-mode: t
  eldoc-mode: t
  show-paren-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  context-menu-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  size-indication-mode: t
  column-number-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t

Load-path shadows:
~/gits/emacs_clones/composable/composable-mark hides /home/ergo/.config/emacs/elpa/composable-20201024.1458/composable-mark
~/gits/emacs_clones/composable/composable hides /home/ergo/.config/emacs/elpa/composable-20201024.1458/composable
/home/ergo/.config/emacs/elpa/transient-20220216.2303/transient hides /home/ergo/.local/share/emacs/29.0.50/lisp/transient

Features:
(shadow sort mail-extr autorevert filenotify xclip emacsbug message
mailcap yank-media rmc puny rfc822 mml mml-sec password-cache epa
derived epg rfc6068 epg-config gnus-util text-property-search time-date
mm-decode mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils
mailheader sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr
mail-utils elec-pair flyspell-correct flyspell ispell company-semantic
company-template company-capf company flycheck ansi-color json map
find-func dash pcase diff-hl-margin diff-hl-dired advice dired
dired-loaddefs diff-hl log-view pcvs-util vc-dir ewoc vc vc-dispatcher
diff-mode cape term/tmux term/xterm xterm init composable
composable-mark icomplete repeat xt-mouse edmacro kmacro mb-depth
simple-16-theme winner ring saveplace delsel savehist
display-fill-column-indicator display-line-numbers diminish which-key
cl-extra help-mode use-package use-package-ensure use-package-delight
use-package-diminish use-package-bind-key bind-key easy-mmode
use-package-core disp-table info ede/auto eieio-base cl-seq eieio seq
subr-x byte-opt bytecomp byte-compile cconv eieio-core cl-macs gv
eieio-loaddefs cl-loaddefs cl-lib tex-site rx slime-autoloads early-init
iso-transl tooltip eldoc paren electric uniquify ediff-hook vc-hooks
lisp-float-type elisp-mode mwheel term/pgtk-win pgtk-win term/common-win
tool-bar dnd fontset image regexp-opt fringe tabulated-list replace
newcomment text-mode lisp-mode prog-mode register page tab-bar menu-bar
rfn-eshadow isearch easymenu timer select scroll-bar mouse jit-lock
font-lock syntax font-core term/tty-colors frame minibuffer cl-generic
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite emoji-zwj charscript
charprop case-table epa-hook jka-cmpr-hook help simple abbrev obarray
cl-preloaded nadvice button loaddefs faces cus-face macroexp files
window text-properties overlay sha1 md5 base64 format env code-pages
mule custom widget keymap hashtable-print-readable backquote threads
dbusbind inotify dynamic-setting system-font-setting font-render-setting
cairo gtk pgtk lcms2 multi-tty make-network-process native-compile
emacs)

Memory information:
((conses 16 185154 26978)
 (symbols 48 14286 0)
 (strings 32 51271 5428)
 (string-bytes 1 1740902)
 (vectors 16 21827)
 (vector-slots 8 379766 240772)
 (floats 8 107 1255)
 (intervals 56 762 0)
 (buffers 992 10))

bug#54042: 29.0.50; fido-mode and ssh not listing hosts

[Michael Albinus]

Ergus via "Bug reports for GNU Emacs, the Swiss army knife of text
editors" <bug-gnu-emacs@gnu.org> writes:

> Hi:

Hi,

> Recently I have found that icomplete does not offer host completions
> after /ssh:
>
> After some check I got this minimal snippet:
>
> emacs -Q
> M-x fido-mode
> C-x C-f /ssh:<tab>
>
> Somehow fido mode removes the latest colon (:) and tries to get
> completion for /ssh so not host are shown (even in the completions
> buffer).
>
> If I choose at this point some of the options (sshx:, ssh:, sshfs:) it a
> buffer `ssh:` is created at / (read only of course.)
>
> trying a bit more like:
>
> C-x C-f /ssh:m<tab> (I have many servers starting with m in my ssh config)
>
> It does not show anything as completion but shows [No matches] while
> normally in *Completions* appear all the hosts defined in .ssh/config.
>
> I tried with icomplete and the problem seems to be only with fido.

This is a duplicate of bug#51386, see <https://debbugs.gnu.org/51386>.
I've merged both bugs.

Best regards, Michael.

bug#54042: closed (Re: bug#60505: 29.0.60; Fido Mode and Tramp Completion)

[Ergus via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Hi again:

The initial problem seems to be solved because:

C-x C-f /ssh:<tab>

Opens the *Completions* buffer with right completions...

But

Somehow I am still getting a similar problem remaining. Why in fido
mode?:

C-x C-f /ssh:

does not offer a completion list with the hosts or user names or anything?

Actually:

C-x C-f /ssh:<RET>

In opens dired in the home of a random server from my config but I don't
know how it is selected. What I am missing here?

Best,
Ergus


On Sun, Feb 12, 2023 at 07:28:03PM +0000, GNU bug Tracking System wrote:
>Your bug report
>
>#60505: 29.0.50; fido-mode and ssh not listing hosts
>
>which was filed against the emacs package, has been closed.
>
>The explanation is attached below, along with your original report.
>If you require more details, please reply to 54042@debbugs.gnu.org.
>
>-- 
>60505: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=60505
>GNU Bug Tracking System
>Contact help-debbugs@gnu.org with problems

>From: Michael Albinus <michael.albinus@gmx.de>
>Date: Sun, 12 Feb 2023 20:26:51 +0100
>To: Stefan Monnier <monnier@iro.umontreal.ca>
>Cc: Michael Heerdegen <michael_heerdegen@web.de>, Eli Zaretskii
> <eliz@gnu.org>, 60505-done@debbugs.gnu.org, Gregory Heytings
> <gregory@heytings.org>, julien@jroy.ca
>Subject: Re: bug#60505: 29.0.60; Fido Mode and Tramp Completion
>
>Version: 29.2
>
>Michael Albinus <michael.albinus@gmx.de> writes:
>
>> There are still some cases it doesn't work as expected (user name
>> completion to be mentioned); I'll continue to work on this.
>
>This works now fine, and also multi-hop completion passes the tests. I'm
>closing this bug (and the merged ones).
>
>Best regards, Michael.
>

>From: Ergus <spacibba@aol.com>
>Date: Thu, 17 Feb 2022 15:47:57 +0100
>To: bug-gnu-emacs@gnu.org
>Subject: 29.0.50; fido-mode and ssh not listing hosts
>
>
>Hi:
>
>Recently I have found that icomplete does not offer host completions
>after /ssh:
>
>After some check I got this minimal snippet:
>
>emacs -Q
>M-x fido-mode
>C-x C-f /ssh:<tab>
>
>Somehow fido mode removes the latest colon (:) and tries to get
>completion for /ssh so not host are shown (even in the completions
>buffer).
>
>If I choose at this point some of the options (sshx:, ssh:, sshfs:) it a
>buffer `ssh:` is created at / (read only of course.)
>
>trying a bit more like:
>
>C-x C-f /ssh:m<tab> (I have many servers starting with m in my ssh config)
>
>It does not show anything as completion but shows [No matches] while
>normally in *Completions* appear all the hosts defined in .ssh/config.
>
>I tried with icomplete and the problem seems to be only with fido.
>
>
>In GNU Emacs 29.0.50 (build 73, x86_64-pc-linux-gnu, GTK+ Version 3.24.31, cairo version 1.17.4)
> of 2022-02-16 built on Ergus
>Repository revision: 8737d79be8f82bb12ca12dc1a7de37a4d2875f39
>Repository branch: master
>System Description: Arch Linux
>
>Configured using:
> 'configure --prefix=/home/ergo/.local/ --with-mailutils --with-json
> --with-x-toolkit=gtk3 --with-xft --with-wide-int --with-modules
> --with-cairo --with-harfbuzz --with-native-compilation --with-pgtk'
>
>Configured features:
>ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
>JSON LCMS2 LIBSYSTEMD LIBXML2 MODULES NATIVE_COMP NOTIFY INOTIFY PDUMPER
>PGTK PNG RSVG SECCOMP SOUND SQLITE3 THREADS TIFF TOOLKIT_SCROLL_BARS
>WEBP XIM GTK3 ZLIB
>
>Important settings:
>  value of $LANG: en_US.UTF-8
>  locale-coding-system: utf-8-unix
>
>Major mode: Lisp Interaction
>
>Minor modes in effect:
>  global-auto-revert-mode: t
>  xclip-mode: t
>  electric-pair-mode: t
>  flyspell-mode: t
>  company-mode: t
>  flycheck-mode: t
>  diff-hl-margin-mode: t
>  composable-mark-mode: t
>  composable-mode: t
>  fido-vertical-mode: t
>  icomplete-vertical-mode: t
>  icomplete-mode: t
>  fido-mode: t
>  repeat-mode: t
>  xterm-mouse-mode: t
>  minibuffer-depth-indicate-mode: t
>  winner-mode: t
>  save-place-mode: t
>  delete-selection-mode: t
>  savehist-mode: t
>  global-display-fill-column-indicator-mode: t
>  display-fill-column-indicator-mode: t
>  global-display-line-numbers-mode: t
>  display-line-numbers-mode: t
>  which-key-mode: t
>  override-global-mode: t
>  eldoc-mode: t
>  show-paren-mode: t
>  mouse-wheel-mode: t
>  file-name-shadow-mode: t
>  context-menu-mode: t
>  global-font-lock-mode: t
>  font-lock-mode: t
>  blink-cursor-mode: t
>  auto-composition-mode: t
>  auto-encryption-mode: t
>  auto-compression-mode: t
>  size-indication-mode: t
>  column-number-mode: t
>  line-number-mode: t
>  indent-tabs-mode: t
>  transient-mark-mode: t
>
>Load-path shadows:
>~/gits/emacs_clones/composable/composable-mark hides /home/ergo/.config/emacs/elpa/composable-20201024.1458/composable-mark
>~/gits/emacs_clones/composable/composable hides /home/ergo/.config/emacs/elpa/composable-20201024.1458/composable
>/home/ergo/.config/emacs/elpa/transient-20220216.2303/transient hides /home/ergo/.local/share/emacs/29.0.50/lisp/transient
>
>Features:
>(shadow sort mail-extr autorevert filenotify xclip emacsbug message
>mailcap yank-media rmc puny rfc822 mml mml-sec password-cache epa
>derived epg rfc6068 epg-config gnus-util text-property-search time-date
>mm-decode mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils
>mailheader sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr
>mail-utils elec-pair flyspell-correct flyspell ispell company-semantic
>company-template company-capf company flycheck ansi-color json map
>find-func dash pcase diff-hl-margin diff-hl-dired advice dired
>dired-loaddefs diff-hl log-view pcvs-util vc-dir ewoc vc vc-dispatcher
>diff-mode cape term/tmux term/xterm xterm init composable
>composable-mark icomplete repeat xt-mouse edmacro kmacro mb-depth
>simple-16-theme winner ring saveplace delsel savehist
>display-fill-column-indicator display-line-numbers diminish which-key
>cl-extra help-mode use-package use-package-ensure use-package-delight
>use-package-diminish use-package-bind-key bind-key easy-mmode
>use-package-core disp-table info ede/auto eieio-base cl-seq eieio seq
>subr-x byte-opt bytecomp byte-compile cconv eieio-core cl-macs gv
>eieio-loaddefs cl-loaddefs cl-lib tex-site rx slime-autoloads early-init
>iso-transl tooltip eldoc paren electric uniquify ediff-hook vc-hooks
>lisp-float-type elisp-mode mwheel term/pgtk-win pgtk-win term/common-win
>tool-bar dnd fontset image regexp-opt fringe tabulated-list replace
>newcomment text-mode lisp-mode prog-mode register page tab-bar menu-bar
>rfn-eshadow isearch easymenu timer select scroll-bar mouse jit-lock
>font-lock syntax font-core term/tty-colors frame minibuffer cl-generic
>cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
>korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
>european ethiopic indian cyrillic chinese composite emoji-zwj charscript
>charprop case-table epa-hook jka-cmpr-hook help simple abbrev obarray
>cl-preloaded nadvice button loaddefs faces cus-face macroexp files
>window text-properties overlay sha1 md5 base64 format env code-pages
>mule custom widget keymap hashtable-print-readable backquote threads
>dbusbind inotify dynamic-setting system-font-setting font-render-setting
>cairo gtk pgtk lcms2 multi-tty make-network-process native-compile
>emacs)
>
>Memory information:
>((conses 16 185154 26978)
> (symbols 48 14286 0)
> (strings 32 51271 5428)
> (string-bytes 1 1740902)
> (vectors 16 21827)
> (vector-slots 8 379766 240772)
> (floats 8 107 1255)
> (intervals 56 762 0)
> (buffers 992 10))
>
>

bug#54042: closed (Re: bug#60505: 29.0.60; Fido Mode and Tramp Completion)

[Michael Albinus]

Ergus via "Bug reports for GNU Emacs, the Swiss army knife of text
editors" <bug-gnu-emacs@gnu.org> writes:

> Hi again:

Hi,

> Somehow I am still getting a similar problem remaining. Why in fido
> mode?:
>
> C-x C-f /ssh:
>
> does not offer a completion list with the hosts or user names or
> anything?

Don't know.

> Actually:
>
> C-x C-f /ssh:<RET>
>
> In opens dired in the home of a random server from my config but I don't
> know how it is selected. What I am missing here?

I'm not sure it is Tramp related. When I call 'emacs -Q -f fido-mode'
and do

C-x C-f /s <RET>

it expands the minibuffer to '/afs/'. So it is fido which selects
(randomly?) one of the completion candidates. According to
icomplete-fido-ret, it isn't randomly, but
(car completion-all-sorted-completions) - and this is empty, because above.

That's why fido opens a selected server, instead of showing it in the
minibuffer.

I don't know whether we shall discuss it here in this bug report (and
friends). Perhaps it is worth a new bug report, because it is about
behavior of <RET>, and not od <TAB>. And we definitely need somebody who
knows fido internals.

> Best,
> Ergus

Best regards, Michael.

bug#54042: closed (Re: bug#60505: 29.0.60; Fido Mode and Tramp Completion)

[Michael Albinus]

Michael Albinus <michael.albinus@gmx.de> writes:

Hi,

>> Somehow I am still getting a similar problem remaining. Why in fido
>> mode?:
>>
>> C-x C-f /ssh:
>>
>> does not offer a completion list with the hosts or user names or
>> anything?
>
> Don't know.
>
>> Actually:
>>
>> C-x C-f /ssh:<RET>
>>
>> In opens dired in the home of a random server from my config but I don't
>> know how it is selected. What I am missing here?
>
> I don't know whether we shall discuss it here in this bug report (and
> friends). Perhaps it is worth a new bug report, because it is about
> behavior of <RET>, and not od <TAB>. And we definitely need somebody who
> knows fido internals.

Well, I've pushed a change to master which should improve the
situation. Could you, pls, check?

>> Best,
>> Ergus

Best regards, Michael.

bug#54042: closed (Re: bug#60505: 29.0.60; Fido Mode and Tramp Completion)

[Ergus via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Hi Michael:

It works much better now! There is a detail when opening the connection
that a message appears in the same place than the prompt/cursor... but
that's not part of this issue I think... When I have a reproducible
snippet I will open another bug report.

Very very thanks!!!

Ergus



On Sat, Feb 18, 2023 at 06:25:55PM +0100, Michael Albinus wrote:
>Michael Albinus <michael.albinus@gmx.de> writes:
>
>Hi,
>
>>> Somehow I am still getting a similar problem remaining. Why in fido
>>> mode?:
>>>
>>> C-x C-f /ssh:
>>>
>>> does not offer a completion list with the hosts or user names or
>>> anything?
>>
>> Don't know.
>>
>>> Actually:
>>>
>>> C-x C-f /ssh:<RET>
>>>
>>> In opens dired in the home of a random server from my config but I don't
>>> know how it is selected. What I am missing here?
>>
>> I don't know whether we shall discuss it here in this bug report (and
>> friends). Perhaps it is worth a new bug report, because it is about
>> behavior of <RET>, and not od <TAB>. And we definitely need somebody who
>> knows fido internals.
>
>Well, I've pushed a change to master which should improve the
>situation. Could you, pls, check?
>
>>> Best,
>>> Ergus
>
>Best regards, Michael.

bug#54042: closed (Re: bug#60505: 29.0.60; Fido Mode and Tramp Completion)

[Michael Albinus]

Ergus <spacibba@aol.com> writes:

> Hi Michael:

Hi Ergus,

> It works much better now! There is a detail when opening the connection
> that a message appears in the same place than the prompt/cursor... but
> that's not part of this issue I think... When I have a reproducible
> snippet I will open another bug report.

Does it help to decrease tramp-verbose to 2?

> Very very thanks!!!
>
> Ergus

Best regards, Michael.