bug#35639: 27.0.50; tramp fails to use ssh on LibreCMC (no base64 encoder available, and not mentioned in tramp manual)

bug#35639: 27.0.50; tramp fails to use ssh on LibreCMC (no base64 encoder available, and not mentioned in tramp manual)

[Jeronimo Pellegrini]

Hello. I believe I have found a bug in either tramp.texi or 
tramp-sh.el.

How to reproduce:

[1] set up a vanilla installation of LibreCMC (or OpenWRT, I've
reproduced this on both) on a wireless router. Suppose it's IP
it's 192.168.1.1.

[2] ssh into the router as root and create a file,

cat > test
whatever
^D

[3] from another machine, fire up emacs,

emacs -Q

[4] try to use tramp to access the file using ssh:

C-x X-f  /ssh:root@192.168.1.1:/root/test
(or using the IP address, or host name, of the router, whatever it is)


Emacs will respond with an error, complaining about a "wrong method".

This is because line 3174 of rtamp.el was reached. It is the default
case of a COND expression that has other two cases:
- (or (tramp-local-host-p v)
      (tramp-method-out-of-band-p v size))
- (rem-enc ...)

The rem-enc function should use a base64 encoding program on the
router (but LibreCMC and OpenWRT do not have it!).

If, on the router, I install base64:

# opkg install coreutils-base64

then tramp succeeds.

So, I believe this should be mentioned in the manual
(in the info node Configuration -> Remote Programs
there is a list of required programs, but neither uuencode nor
base64 are mentioned)

I'm willing to send a patch to the manual - however:
reading  lisp/net/tramp-sh.el  I get the impression that tramp
actually *should* be able to encode/decode using busybox/awk, 
which *is* available in LibreCMC (or did I get it wrong?).

So, I'm not sure if this is a tramp bug (can't use awk to encode/decode
files in base64 format) or bug in its manual (doesn't mention that
a base64 binary is needed on the remote side).

Thank you!
J.



In GNU Emacs 27.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.5)
 of 2019-05-08 built on mitra
Repository revision: 8b789755b45e6e10ed2809d7a7b89146b28452fc
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12003000
System Description: Debian GNU/Linux buster/sid

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.
Making completion list...

Configured features:
XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND DBUS GSETTINGS GLIB NOTIFY
INOTIFY LIBSELINUX GNUTLS LIBXML2 FREETYPE XFT ZLIB TOOLKIT_SCROLL_BARS
GTK3 X11 XDBE XIM THREADS LIBSYSTEMD JSON PDUMPER LCMS2 GMP

Important settings:
  value of $LC_CTYPE: en_US.UTF-8
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Fundamental

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny seq byte-opt gv
bytecomp byte-compile cconv dired dired-loaddefs format-spec rfc822 mml
easymenu mml-sec password-cache epa derived epg epg-config gnus-util
rmail rmail-loaddefs text-property-search mm-decode mm-bodies mm-encode
mail-parse rfc2231 mailabbrev gmm-utils mailheader cl-loaddefs cl-lib
sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils
time-date elec-pair mule-util tooltip eldoc electric uniquify ediff-hook
vc-hooks lisp-float-type mwheel term/x-win x-win term/common-win x-dnd
tool-bar dnd fontset image regexp-opt fringe tabulated-list replace
newcomment text-mode elisp-mode lisp-mode prog-mode register page
menu-bar rfn-eshadow isearch timer select scroll-bar mouse jit-lock
font-lock syntax facemenu font-core term/tty-colors frame cl-generic
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite charscript charprop
case-table epa-hook jka-cmpr-hook help simple abbrev obarray minibuffer
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote threads dbusbind
inotify lcms2 dynamic-setting system-font-setting font-render-setting
move-toolbar gtk x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 46847 6151)
 (symbols 48 5968 1)
 (strings 32 15091 1308)
 (string-bytes 1 495765)
 (vectors 16 9664)
 (vector-slots 8 118994 8860)
 (floats 8 17 21)
 (intervals 56 187 0)
 (buffers 992 13))

bug#35639: 27.0.50; tramp fails to use ssh on LibreCMC (no base64 encoder available, and not mentioned in tramp manual)

[Michael Albinus]

Jeronimo Pellegrini <j_p@aleph0.info> writes:

Hi Jeronimo,

> The rem-enc function should use a base64 encoding program on the
> router (but LibreCMC and OpenWRT do not have it!).

Tramp is busy to detect base64 or any other method which can be used for
encoding/decoding. Pls set tramp-verbose to 6, and rerun the test (w/o
an installed base64 on the remote machine). There will be a Tramp debug
buffer.

> reading  lisp/net/tramp-sh.el  I get the impression that tramp
> actually *should* be able to encode/decode using busybox/awk,
> which *is* available in LibreCMC (or did I get it wrong?).

See variable tramp-remote-coding-commands, it contains all the different
encoding/decoding commands Tramp tries to apply. awk is among them.

The debug buffer you'll send shall show us, why it doesn't work.

> So, I'm not sure if this is a tramp bug (can't use awk to encode/decode
> files in base64 format) or bug in its manual (doesn't mention that
> a base64 binary is needed on the remote side).

Tramp shall describe in its manual, what does it need for
encoding/decoding. Agreed.

> Thank you!
> J.

Best regards, Michael.

bug#35639: 27.0.50; tramp fails to use ssh on LibreCMC (no base64 encoder available, and not mentioned in tramp manual)

[Jeronimo Pellegrini]

Michael Albinus <michael.albinus@gmx.de> writes:

Hi,

> Jeronimo Pellegrini <j_p@aleph0.info> writes:
>
> Hi Jeronimo,
>
>> The rem-enc function should use a base64 encoding program on the
>> router (but LibreCMC and OpenWRT do not have it!).
>
> Tramp is busy to detect base64 or any other method which can be used for
> encoding/decoding. Pls set tramp-verbose to 6, and rerun the test (w/o
> an installed base64 on the remote machine). There will be a Tramp debug
> buffer.

Yes. That's where I first saw the ""Wrong method specification" message.
The buffer contents are here:

http://aleph0.info/jp/tramp-debug/tramp-debug-buffer.txt

>> reading  lisp/net/tramp-sh.el  I get the impression that tramp
>> actually *should* be able to encode/decode using busybox/awk,
>> which *is* available in LibreCMC (or did I get it wrong?).
>
> See variable tramp-remote-coding-commands, it contains all the different
> encoding/decoding commands Tramp tries to apply. awk is among them.

Before using tramp it has no content; after trying to connect,
tramp-remote-coding-commands is:

((b64 "base64" "base64 -d -i")
 (b64 "base64" "base64 -d")
 (b64 "openssl enc -base64" "openssl enc -d -base64")
 (b64 "mimencode -b" "mimencode -u -b")
 (b64 "mmencode -b" "mmencode -u -b")
 (b64 "recode data..base64" "recode base64..data")
 (b64 tramp-perl-encode-with-module tramp-perl-decode-with-module)
 (b64 tramp-perl-encode tramp-perl-decode)
 (b64 tramp-awk-encode tramp-awk-decode "test -c /dev/zero && od -v -t x1 -A n </dev/null && busybox awk '{}' </dev/null")
 (uu "uuencode xxx" "uudecode -o /dev/stdout" "test -c /dev/stdout")
 (uu "uuencode xxx" "uudecode -o -")
 (uu "uuencode xxx" "uudecode -p")
 (uu "uuencode xxx" tramp-uudecode)
 (pack tramp-perl-pack tramp-perl-unpack))

Funny -- it lists awk as one method:

 (b64 tramp-awk-encode tramp-awk-decode "test -c /dev/zero && od -v -t x1 -A n </dev/null && busybox awk '{}' </dev/null")

But awk was not used, because 'od' is also not available.
And none of the other binaries exist in LibreCMC or OpenWRT (not after a simple install, at least).

> The debug buffer you'll send shall show us, why it doesn't work.

Yes, it is here:

19:41:26.088945 tramp-sh-handle-file-local-copy (1) # File error: Wrong method specification for ‘ssh’

(this is the only occurrence of the string "error").

As I had mentioned in the first message, this is because line 3174 of
tramp.el was reached. It is the default case of a COND expression that has other two cases:
- (or (tramp-local-host-p v)
      (tramp-method-out-of-band-p v size))
- (rem-enc ...)

What happened is that (rem-enc ...) failed (because tramp could not use
any encoding command), and fell through into the default action, which
is this error.

>> So, I'm not sure if this is a tramp bug (can't use awk to encode/decode
>> files in base64 format) or bug in its manual (doesn't mention that
>> a base64 binary is needed on the remote side).
>
> Tramp shall describe in its manual, what does it need for
> encoding/decoding. Agreed.

Right...
I will later in the week propose a patch to the manual, if that is OK.

J.

bug#35639: 27.0.50; tramp fails to use ssh on LibreCMC (no base64 encoder available, and not mentioned in tramp manual)

[Michael Albinus]

Jeronimo Pellegrini <j_p@aleph0.info> writes:

> Hi,

Hi Jeronimo,

> Before using tramp it has no content; after trying to connect,

Sure. Tramp is autoloaded, its symbols are available only after loading.

> Funny -- it lists awk as one method:
>
>  (b64 tramp-awk-encode tramp-awk-decode "test -c /dev/zero && od -v -t
> x1 -A n </dev/null && busybox awk '{}' </dev/null")
>
> But awk was not used, because 'od' is also not available.

I'm not a heavy awk user. The current awk based implementation has been
contributed by somebody else, IIRC. Do you know another way to use awk
for encoding/decoing base64, without "od"?

The current Tramp implementation is in the two variables
`tramp-awk-encode' and `tramp-awk-decode'. If we could find another
implementation, which runs also on your LibreCMC or OpenWRT machine, we
could add it to Tramp.

> 19:41:26.088945 tramp-sh-handle-file-local-copy (1) # File error:
> Wrong method specification for ‘ssh’
>
> (this is the only occurrence of the string "error").
>
> As I had mentioned in the first message, this is because line 3174 of
> tramp.el was reached. It is the default case of a COND expression that
> has other two cases:
> - (or (tramp-local-host-p v)
>       (tramp-method-out-of-band-p v size))
> - (rem-enc ...)
>
> What happened is that (rem-enc ...) failed (because tramp could not use
> any encoding command), and fell through into the default action, which
> is this error.

Maybe we should enhance the error message.

>> Tramp shall describe in its manual, what does it need for
>> encoding/decoding. Agreed.
>
> Right...
> I will later in the week propose a patch to the manual, if that is OK.

Thanks.

> J.

Best regards, Michael.

bug#35639: 27.0.50; tramp fails to use ssh on LibreCMC (no base64 encoder available, and not mentioned in tramp manual)

[Jeronimo Pellegrini]

Hi Michael,

On Thu, May 09, 2019 at 09:35:13AM +0200, Michael Albinus wrote:
> Jeronimo Pellegrini <j_p@aleph0.info> writes:
> >  (b64 tramp-awk-encode tramp-awk-decode "test -c /dev/zero && od -v -t
> > x1 -A n </dev/null && busybox awk '{}' </dev/null")
> >
> > But awk was not used, because 'od' is also not available.
> 
> I'm not a heavy awk user. The current awk based implementation has been
> contributed by somebody else, IIRC. Do you know another way to use awk
> for encoding/decoing base64, without "od"?

Yes. Instead of  od:
 od -v -t x1 -A n

it would be possible to use hexdump:
 hexdump -v -e '16/1 " %02x" "\n"' 

hexdump is included in busybox, and seems to be at least as fast as
od.

However... hexdump may not be present in all Linux/*BSD/*ix systems.
At least in Debian, it is in package bsdmainutils, which is
not required. (od, on the other hand, is in package coreutils, which 
is required)

> The current Tramp implementation is in the two variables
> `tramp-awk-encode' and `tramp-awk-decode'. If we could find another
> implementation, which runs also on your LibreCMC or OpenWRT machine, we
> could add it to Tramp.

Would it be OK to have two different variables, and two tests for
od+awk and hexdump+awk?

tramp-od-awk-coding-test
tramp-od-awk-{encode,decode}

tramp-hexdump-awk-coding-test
tramp-hexdump-awk-{encode-decode}

Or would it be better if the string in  tramp-awk-encode 
was built from two others, (1) the hex dumper and (2) the awk 
code?
- define a constant tramp-awk-from-hex-to-base64, which does
  only the hex into base64 part of the job

- when connecting, tramp would check for some hex dumper (od 
  or hexdump) this would split tramp-awk-coding-test in two
  others

- tramp-awk-encode would be obtained by concatenating the hex-dumping
  string (either od or hexdump) with " | " and tramp-awk-from-hex-to-base64.

But then tramp-awk-encode would not be a constant anymore.

> > 19:41:26.088945 tramp-sh-handle-file-local-copy (1) # File error:
> > Wrong method specification for ‘ssh’
> >
> > (this is the only occurrence of the string "error").
> >
> > As I had mentioned in the first message, this is because line 3174 of
> > tramp.el was reached. It is the default case of a COND expression that
> > has other two cases:
> > - (or (tramp-local-host-p v)
> >       (tramp-method-out-of-band-p v size))
> > - (rem-enc ...)
> >
> > What happened is that (rem-enc ...) failed (because tramp could not use
> > any encoding command), and fell through into the default action, which
> > is this error.
> 
> Maybe we should enhance the error message.

Perhaps 

(tramp-error v 'file-error "Cannot find any way to encode data using `%s'" method)

would be more precise?

Or, if (tramp-local-host-p v) (tramp-method-out-of-band-p v size) are
false, explicitly write that "encoding binaries were not found on the remote end"?

J.

bug#35639: 27.0.50; tramp fails to use ssh on LibreCMC (no base64 encoder available, and not mentioned in tramp manual)

[Michael Albinus]

Jeronimo Pellegrini <j_p@aleph0.info> writes:

> Hi Michael,

Hi Jeronimo,

> Yes. Instead of  od:
>  od -v -t x1 -A n
>
> it would be possible to use hexdump:
>  hexdump -v -e '16/1 " %02x" "\n"'

Yes, looks good.

> However... hexdump may not be present in all Linux/*BSD/*ix systems.
> At least in Debian, it is in package bsdmainutils, which is
> not required. (od, on the other hand, is in package coreutils, which
> is required)

See, how we do it with the other programs: we test one after the other,
until it succeeds. So it is perfect to have a second entry for awk in
tramp-remote-coding-commands.

> Would it be OK to have two different variables, and two tests for
> od+awk and hexdump+awk?
>
> tramp-od-awk-coding-test
> tramp-od-awk-{encode,decode}
>
> tramp-hexdump-awk-coding-test
> tramp-hexdump-awk-{encode-decode}

Well, decoding does not use od. So it might be sufficient to have

tramp-od-awk-coding-test
tramp-od-awk-encode
tramp-hexdump-awk-coding-test
tramp-hexdump-awk-encode
tramp-awk-decode}


>> > What happened is that (rem-enc ...) failed (because tramp could not use
>> > any encoding command), and fell through into the default action, which
>> > is this error.
>>
>> Maybe we should enhance the error message.
>
> Perhaps
>
> (tramp-error v 'file-error "Cannot find any way to encode data using
> `%s'" method)
>
> would be more precise?
>
> Or, if (tramp-local-host-p v) (tramp-method-out-of-band-p v size) are
> false, explicitly write that "encoding binaries were not found on the
> remote end"?

Yes, something like this. Do you want to prepare such a patch (which
should also include the tramp.texi changes)?

Hmm, looks like this could extend the 15 lines limit, which is the upper
limit for contributions to Emacs/Tramp, w/o having signed the FSF legal
papers (where you declare the copyright of your work be given to the
FSF). Would you like to sign such a paper? It would help also for future
contributions to Emacs or one of its packages.

> J.

Best regards, Michael.

bug#35639: 27.0.50; tramp fails to use ssh on LibreCMC (no base64 encoder available, and not mentioned in tramp manual)

[Jeronimo Pellegrini]

Hello again!

On May 9, 2019 9:05:39 AM GMT-03:00, Michael Albinus <michael.albinus@gmx.de> wrote:
>Jeronimo Pellegrini <j_p@aleph0.info> writes:
>
>> However... hexdump may not be present in all Linux/*BSD/*ix systems.
>> At least in Debian, it is in package bsdmainutils, which is
>> not required. (od, on the other hand, is in package coreutils, which
>> is required)
>
>See, how we do it with the other programs: we test one after the other,
>until it succeeds. So it is perfect to have a second entry for awk in
>tramp-remote-coding-commands.

Good then, it'll be simple.

>Well, decoding does not use od. So it might be sufficient to have
>
>tramp-od-awk-coding-test
>tramp-od-awk-encode
>tramp-hexdump-awk-coding-test
>tramp-hexdump-awk-encode
>tramp-awk-decode}

Ah, all right.

>> Perhaps
>>
>> (tramp-error v 'file-error "Cannot find any way to encode data using
>> `%s'" method)
>>
>> would be more precise?
>>
>> Or, if (tramp-local-host-p v) (tramp-method-out-of-band-p v size) are
>> false, explicitly write that "encoding binaries were not found on the
>> remote end"?
>
>Yes, something like this. Do you want to prepare such a patch (which
>should also include the tramp.texi changes)?

Sure!

The documentation change in the same patch?

>Hmm, looks like this could extend the 15 lines limit, which is the
>upper
>limit for contributions to Emacs/Tramp, w/o having signed the FSF legal
>papers (where you declare the copyright of your work be given to the
>FSF). Would you like to sign such a paper? It would help also for
>future
>contributions to Emacs or one of its packages.

Yes, of course! How do I do that?

J.

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

bug#35639: 27.0.50; tramp fails to use ssh on LibreCMC (no base64 encoder available, and not mentioned in tramp manual)

[Michael Albinus]

Jeronimo Pellegrini <j_p@aleph0.info> writes:

> Hello again!

Hi Jeronimo,

>>Yes, something like this. Do you want to prepare such a patch (which
>>should also include the tramp.texi changes)?
>
> Sure!
>
> The documentation change in the same patch?

Yes, it belongs together.

>>Hmm, looks like this could extend the 15 lines limit, which is the
>>upper
>>limit for contributions to Emacs/Tramp, w/o having signed the FSF legal
>>papers (where you declare the copyright of your work be given to the
>>FSF). Would you like to sign such a paper? It would help also for
>>future
>>contributions to Emacs or one of its packages.
>
> Yes, of course! How do I do that?

I'll send you the papers off-list. In the meantime, you can read about
at <https://www.gnu.org/prep/maintain/html_node/Copyright-Papers.html>.

> J.

Best regards, Michael.

bug#35639: 27.0.50; tramp fails to use ssh on LibreCMC (no base64 encoder available, and not mentioned in tramp manual)

[Jeronimo Pellegrini]

Hi,

On Thu, May 09, 2019 at 02:45:48PM +0200, Michael Albinus wrote:
> Jeronimo Pellegrini <j_p@aleph0.info> writes:
> > The documentation change in the same patch?
> 
> Yes, it belongs together.

Okay, I have it ready. Do I attach it here in the bug report?

And I suppose the patch stays on hold until I finish the
FSF assignment process -- is that right?

J.

bug#35639: 27.0.50; tramp fails to use ssh on LibreCMC (no base64 encoder available, and not mentioned in tramp manual)

[Michael Albinus]

Jeronimo Pellegrini <j_p@aleph0.info> writes:

> Hi,

Hi Jeronimo,

> On Thu, May 09, 2019 at 02:45:48PM +0200, Michael Albinus wrote:
>> Jeronimo Pellegrini <j_p@aleph0.info> writes:
>> > The documentation change in the same patch?
>>
>> Yes, it belongs together.
>
> Okay, I have it ready. Do I attach it here in the bug report?

Yes, pls do.

> And I suppose the patch stays on hold until I finish the
> FSF assignment process -- is that right?

Yes. Unless it is less than 15 changed lines, than we could apply
immediately. Your FSF assignment will help in the future anyway.

> J.

Best regards, Michael.

bug#35639: 27.0.50; tramp fails to use ssh on LibreCMC (no base64 encoder available, and not mentioned in tramp manual)

[Jeronimo Pellegrini]

[-- Attachment #1: Type: text/plain, Size: 320 bytes --]

Hi Michael,

On Thu, May 09, 2019 at 10:02:50PM +0200, Michael Albinus wrote:
> Jeronimo Pellegrini <j_p@aleph0.info> writes:
> > Okay, I have it ready. Do I attach it here in the bug report?
> 
> Yes, pls do.

All right, here it is!

I'm not sure I like the changes to the manual. Do you think it is
OK?

Thank you!
J.

[-- Attachment #2: tramp-use-hexdump.diff --]
[-- Type: text/x-diff, Size: 5193 bytes --]

diff --git a/doc/misc/tramp.texi b/doc/misc/tramp.texi
index 2eb5b45eb2..7bbf1f1771 100644
--- a/doc/misc/tramp.texi
+++ b/doc/misc/tramp.texi
@@ -1960,16 +1960,30 @@ Remote programs
 @section How @value{tramp} finds and uses programs on the remote host
 
 @value{tramp} requires access to and rights to several commands on
-remote hosts: @command{ls}, @command{test}, @command{find} and
+remote hosts: at least @command{ls}, @command{test}, @command{find} and
 @command{cat}.
 
+For inline methods, at least one of the following should be available:
+
+@itemize @bullet
+@item @command{base64}
+@item @command{openssl}
+@item @command{mimencode}
+@item @command{mmencode}
+@item @command{recode}
+@item @command{perl} or @command{perl5}(if the  @command{MIME::Base64} module is available, it will be used)
+@item @command{od} and @command{busybox awk}
+@item @command{hexdump} and @command{busybox awk}
+@item @command{uuencode} and @command{uudecode}
+@end itemize
+
+If none of these are available, @value{tramp} will not work with inline methods,
+but other methods may still be used. If @command{grep} is available, it is
+also used, in order to improve performance.
+
 Besides there are other required programs for @ref{Inline methods} and
 @ref{External methods} of connection.
 
-To improve performance and accuracy of remote file access,
-@value{tramp} uses @command{perl} (or @command{perl5}) and
-@command{grep} when available.
-
 @defopt tramp-remote-path
 @code{tramp-remote-path} specifies which remote directory paths
 @value{tramp} can search for @ref{Remote programs}.
diff --git a/lisp/net/tramp-sh.el b/lisp/net/tramp-sh.el
index b9df403309..7a1d4ca638 100644
--- a/lisp/net/tramp-sh.el
+++ b/lisp/net/tramp-sh.el
@@ -850,7 +850,41 @@ tramp-perl-unpack
   "Perl program to use for decoding a file.
 Escape sequence %s is replaced with name of Perl binary.")
 
-(defconst tramp-awk-encode
+(defconst tramp-hexdump-awk-encode
+  "hexdump -v -e '16/1 \" %%02x\" \"\\n\"' | busybox awk '\\
+BEGIN {
+  b64 = \"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/\"
+  b16 = \"0123456789abcdef\"
+}
+{
+  for (c=1; c<=length($0); c++) {
+    d=index(b16, substr($0,c,1))
+    if (d--) {
+      for (b=1; b<=4; b++) {
+        o=o*2+int(d/8); d=(d*2)%%16
+        if (++obc==6) {
+          printf substr(b64,o+1,1)
+          if (++rc>75) { printf \"\\n\"; rc=0 }
+          obc=0; o=0
+        }
+      }
+    }
+  }
+}
+END {
+  if (obc) {
+    tail=(obc==2) ? \"==\\n\" : \"=\\n\"
+    while (obc++<6) { o=o*2 }
+    printf \"%%c\", substr(b64,o+1,1)
+  } else {
+    tail=\"\\n\"
+  }
+  printf tail
+}'"
+  "Awk/hexdump program to use for encoding a file.
+This string is passed to `format', so percent characters need to be doubled.")
+
+(defconst tramp-od-awk-encode
   "od -v -t x1 -A n | busybox awk '\\
 BEGIN {
   b64 = \"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/\"
@@ -881,7 +915,7 @@ tramp-awk-encode
   }
   printf tail
 }'"
-  "Awk program to use for encoding a file.
+  "Awk/od program to use for encoding a file.
 This string is passed to `format', so percent characters need to be doubled.")
 
 (defconst tramp-awk-decode
@@ -910,11 +944,18 @@ tramp-awk-decode
   "Awk program to use for decoding a file.
 This string is passed to `format', so percent characters need to be doubled.")
 
-(defconst tramp-awk-coding-test
+
+(defconst tramp-hexdump-awk-coding-test
+  "test -c /dev/zero && \
+busybox hexdump -v -e '16/1 \" %02x\" \"\\n\"' < /dev/null && \
+busybox awk '{}' </dev/null"
+  "Test command for checking `tramp-hexdump-awk-encode' and `tramp-awk-decode'.")
+
+(defconst tramp-od-awk-coding-test
   "test -c /dev/zero && \
 od -v -t x1 -A n </dev/null && \
 busybox awk '{}' </dev/null"
-  "Test command for checking `tramp-awk-encode' and `tramp-awk-decode'.")
+  "Test command for checking `tramp-od-awk-encode' and `tramp-awk-decode'.")
 
 (defconst tramp-vc-registered-read-file-names
   "echo \"(\"
@@ -3167,7 +3208,10 @@ tramp-sh-handle-file-local-copy
 
 	   ;; Oops, I don't know what to do.
 	   (t (tramp-error
-	       v 'file-error "Wrong method specification for `%s'" method)))
+	       v 'file-error (concat "Cannot find any way to encode data using `%s'"
+                                     "-- check tramp-remote-path, and also "
+                                     "verify encoding binaries on the remote end.")
+               method)))
 
 	;; Error handling.
 	((error quit)
@@ -4370,8 +4414,9 @@ tramp-remote-coding-commands
     (b64 "recode data..base64" "recode base64..data")
     (b64 tramp-perl-encode-with-module tramp-perl-decode-with-module)
     (b64 tramp-perl-encode tramp-perl-decode)
-    ;; This is painful slow, so we put it on the end.
-    (b64 tramp-awk-encode tramp-awk-decode ,tramp-awk-coding-test)
+    ;; These is painful slow, so we put it on the end.
+    (b64 tramp-od-awk-encode tramp-awk-decode ,tramp-od-awk-coding-test)
+    (b64 tramp-hexdump-awk-encode tramp-awk-decode ,tramp-hexdump-awk-coding-test)
     (uu  "uuencode xxx" "uudecode -o /dev/stdout" "test -c /dev/stdout")
     (uu  "uuencode xxx" "uudecode -o -")
     (uu  "uuencode xxx" "uudecode -p")

bug#35639: 27.0.50; tramp fails to use ssh on LibreCMC (no base64 encoder available, and not mentioned in tramp manual)

[Michael Albinus]

Jeronimo Pellegrini <j_p@aleph0.info> writes:

> Hi Michael,

Hi Jeronimo,

>> > Okay, I have it ready. Do I attach it here in the bug report?
>>
>> Yes, pls do.
>
> All right, here it is!

Thanks.

> I'm not sure I like the changes to the manual. Do you think it is
> OK?

Yes. See my comments, all of them are minor.

> --- a/doc/misc/tramp.texi
> +++ b/doc/misc/tramp.texi

> +@item @command{perl} or @command{perl5}(if the  @command{MIME::Base64} module is available, it will be used)

Use a space between "{perl5}" and "(if". Care about 80 chars per line
limitation.

> +but other methods may still be used. If @command{grep} is available, it is
> +also used, in order to improve performance.

That's an error in the manual, gzip is meant. Pls correct.

> --- a/lisp/net/tramp-sh.el
> +++ b/lisp/net/tramp-sh.el

> +(defconst tramp-hexdump-awk-coding-test
> +  "test -c /dev/zero && \
> +busybox hexdump -v -e '16/1 \" %02x\" \"\\n\"' < /dev/null && \

This must be "hexdump", not "busybox hexdump".

> +  "Test command for checking `tramp-hexdump-awk-encode' and `tramp-awk-decode'.")

Care about 80 chars per line limitation.

>  	   ;; Oops, I don't know what to do.
>  	   (t (tramp-error
> -	       v 'file-error "Wrong method specification for `%s'" method)))
> +	       v 'file-error (concat "Cannot find any way to encode data using `%s'"
> +                                     "-- check tramp-remote-path, and also "
> +                                     "verify encoding binaries on the remote end.")
> +               method)))

This error message is too long, I believe. Just the first part "Cannot
find ... using "%s" should serve.

> +    ;; These is painful slow, so we put it on the end.

";; These are painful slow, so we put them on the end."

> +    (b64 tramp-hexdump-awk-encode tramp-awk-decode ,tramp-hexdump-awk-coding-test)

Care about 80 chars per line limitation.

> Thank you!
> J.

Best regards, Michael.

bug#35639: 27.0.50; tramp fails to use ssh on LibreCMC (no base64 encoder available, and not mentioned in tramp manual)

[Robert Pluim]

>>>>> On Fri, 10 May 2019 10:40:00 +0200, Michael Albinus <michael.albinus@gmx.de> said:
    >> + ;; These is painful slow, so we put it on the end.

    Michael> ";; These are painful slow, so we put them on the end."

Since weʼre nitpicking documentation, it should be "painfully slow"

Robert

bug#35639: 27.0.50; tramp fails to use ssh on LibreCMC (no base64 encoder available, and not mentioned in tramp manual)

[Michael Albinus]

Robert Pluim <rpluim@gmail.com> writes:

>     Michael> ";; These are painful slow, so we put them on the end."
>
> Since weʼre nitpicking documentation, it should be "painfully slow"

Thanks! A painful error, indeed :-)

> Robert

Best regards, Michael.

bug#35639: 27.0.50; tramp fails to use ssh on LibreCMC (no base64 encoder available, and not mentioned in tramp manual)

[Jeronimo Pellegrini]

Hello Michael,
Thank you for your comments!

On Fri, May 10, 2019 at 10:40:00AM +0200, Michael Albinus wrote:
> Jeronimo Pellegrini <j_p@aleph0.info> writes:
> > +(defconst tramp-hexdump-awk-coding-test
> > +  "test -c /dev/zero && \
> > +busybox hexdump -v -e '16/1 \" %02x\" \"\\n\"' < /dev/null && \
> 
> This must be "hexdump", not "busybox hexdump".

The original tramp-awk-encode uses "busybux awk". Is there a particular 
reason for that, or could it also be changed to "awk"?

As far as I can remember, devices with busybox installed will usually
have symlinks for the usual binary names, like

  awk -> busybox,

and so on, so if busybox is called and argv[0] is "awk" it knows
the user's intention. So, using "busybox awk" and "busybox hexdump" 
would guarantee that this encoding scheme works on devices where the 
symlinks are not available (I've never seen one actually); on the other 
hand, it would break the encoding method for computers which do have 
awk and hexdump, but no busybox... What do we do? Duplicate the 
functions again?

tramp-busybox-od-awk-encode
tramp-busybox-hexdump-awk-encode
tramp-busybox-awk-decode

tramp-od-awk-encode
tramp-hexdump-awk-encode
tramp-awk-decode

I feel like that would be a too large amount of duplicated code...
Or is that ok?

Thank you for your patience!
J.

bug#35639: 27.0.50; tramp fails to use ssh on LibreCMC (no base64 encoder available, and not mentioned in tramp manual)

[Michael Albinus]

Jeronimo Pellegrini <j_p@aleph0.info> writes:

> Hello Michael,

Hi Jeronimo,

>> > +(defconst tramp-hexdump-awk-coding-test
>> > +  "test -c /dev/zero && \
>> > +busybox hexdump -v -e '16/1 \" %02x\" \"\\n\"' < /dev/null && \
>>
>> This must be "hexdump", not "busybox hexdump".
>
> The original tramp-awk-encode uses "busybux awk". Is there a particular
> reason for that, or could it also be changed to "awk"?

I don't remember exactly, but I believe there was a reason. Likely, a
simple "awk" didn't work, but "busybox awk" did. On whatever test machine.

> and so on, so if busybox is called and argv[0] is "awk" it knows
> the user's intention. So, using "busybox awk" and "busybox hexdump"
> would guarantee that this encoding scheme works on devices where the
> symlinks are not available (I've never seen one actually); on the other
> hand, it would break the encoding method for computers which do have
> awk and hexdump, but no busybox... What do we do? Duplicate the
> functions again?

The awk methods are intended for very leightweighted machines, like
routers or NAS devices. It is expected that they have installed busybox.

According to <https://busybox.net/downloads/BusyBox.html>, "awk",
"hexdump" and "od" are subcommands of busybox. Unfortunately, not all
busybox installations have all subcommands compiled in. For example, on
my NAS device "awk" and "hexdump" are part of busybox, "od" isn't.

--8<---------------cut here---------------start------------->8---
[~] # busybox --help
BusyBox v1.01 (2019.03.22-03:18+0000) multi-call binary

Usage: busybox [function] [arguments]...
   or: [function] [arguments]...

	BusyBox is a multi-call binary that combines many common Unix
	utilities into a single executable.  Most people will create a
	link to busybox for each function they wish to use and BusyBox
	will act like whatever it was invoked as!

Currently defined functions:
	[, addgroup, adduser, ash, awk, basename, bunzip2, busybox,
	bzcat, cat, chgrp, chmod, chown, chroot, chvt, clear, cmp,
	cp, crond, crontab, cut, date, dc, dd, deallocvt, delgroup,
	deluser, df, dirname, dmesg, dos2unix, du, echo, egrep, env,
	expr, false, fdisk, fgrep, find, free, getty, grep, gunzip,
	gzip, halt, head, hexdump, hostname, hwclock, id, ifconfig,
	init, insmod, install, ip, kill, killall, klogd, linuxrc, ln,
	logger, login, ls, lsmod, md5sum, mkdir, mknod, mktemp, modprobe,
	more, mount, mv, nameif, netstat, nslookup, openvt, passwd,
	pidof, ping, ping6, pivot_root, poweroff, ps, pwd, rdate, readlink,
	reboot, renice, reset, rm, rmdir, rmmod, route, sed, sh, sha1sum,
	sleep, sort, strings, swapoff, swapon, switch_root, sync, sysctl,
	syslogd, tail, tar, tee, telnet, test, tftp, time, top, touch,
	tr, traceroute, true, tty, umount, uname, uniq, unix2dos, unzip,
	uptime, usleep, vi, wc, wget, which, whoami, xargs, yes, zcat
--8<---------------cut here---------------end--------------->8---

Maybe, we shall use just a template for the encoding/decoding functions,
and for all busybox subcommands we need a test, whether they can be
invoked via "busybox command" or "command". There will be sevaeral
combinations to test.

> tramp-busybox-od-awk-encode
> tramp-busybox-hexdump-awk-encode
> tramp-busybox-awk-decode
>
> tramp-od-awk-encode
> tramp-hexdump-awk-encode
> tramp-awk-decode
>
> I feel like that would be a too large amount of duplicated code...
> Or is that ok?

No, too much duplicated code. As said above, provide
tramp-od-awk-encode, tramp-hexdump-awk-encode and tramp-awk-decode, and
make intelleigent tests whether "awk", "hexdump" and "od" need to be
prefixed by "busybox".

> Thank you for your patience!
> J.

Best regards, Michael.

bug#35639: 27.0.50; tramp fails to use ssh on LibreCMC (no base64 encoder available, and not mentioned in tramp manual)

[Jeronimo Pellegrini]

Hi,

On Fri, May 10, 2019 at 12:18:17PM +0200, Michael Albinus wrote:
> Jeronimo Pellegrini <j_p@aleph0.info> writes:
> > What do we do? Duplicate the functions again?
> ( ... )
> 
> Maybe, we shall use just a template for the encoding/decoding functions,
> and for all busybox subcommands we need a test, whether they can be
> invoked via "busybox command" or "command". There will be sevaeral
> combinations to test.

If templating is the way to go, then tramp-od-awk-encode,
tramp-hexdump-awk-encode, tramp-awk-decode would not be constant
strings anymore; a string would need to be build every time tramp
started a connection. Tramp would apply tests, and depending on
what tests succeed, concatenate the proper strings. Is this good?

Or I can try to make the shell code that is sent to the remote
end do the verification itself (I'd test using dash, which I
hope will work on most systems of interest). Something like

{ hexdump < /dev/null && HD=hexdump; } || \
{ busybox hexdump < /dev/null && HD="busybox hexdump"; } &&  \
$HD ...

(but including the proper command line arguments)

This would work on dash, bash, zsh, ksh and hopefully on any shell
that claims to be POSIX-compliant. Won't work on fish and any
other fancy, modern, alternative shells, of course.

Which one is preferable? Templating in emacs-lisp, or creating smart
shell code?

J.

bug#35639: 27.0.50; tramp fails to use ssh on LibreCMC (no base64 encoder available, and not mentioned in tramp manual)

[Jeronimo Pellegrini]

[-- Attachment #1: Type: text/plain, Size: 1803 bytes --]

On Fri, May 10, 2019 at 11:45:46AM -0300, Jeronimo Pellegrini wrote:
> Or I can try to make the shell code that is sent to the remote
> end do the verification itself (I'd test using dash, which I
> hope will work on most systems of interest). Something like
> 
> { hexdump < /dev/null && HD=hexdump; } || \
> { busybox hexdump < /dev/null && HD="busybox hexdump"; } &&  \
> $HD ...

I actually implemented the shell scripting solution.
Only tramp-hexdump-awk-encode is in the email body, but the
full patch is attached.

This seems to work, and requires no extra coding in tramp
(only the constant strings are changed).

I can also do the templating in elisp, if you think it would be 
better.

J.

(defconst tramp-hexdump-awk-encode
  "{ { awk '{}' </dev/null && BA=\"\"  ; } || \
     { busybox awk '{}' </dev/null &&  BA=busybox ; } } && \
   { { hexdump -v -e '16/1 \" %%02x\" \"\\n\"' < /dev/null && BH=\"\" ; } || \
     { busybox hexdump -v -e '16/1 \" %%02x\" \"\\n\"' < /dev/null && BH=busybox ; } } && \
$BH hexdump -v -e '16/1 \" %%02x\" \"\\n\"'| $BA awk '\\
BEGIN {
  b64 = \"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/\"
  b16 = \"0123456789abcdef\"
}
{
  for (c=1; c<=length($0); c++) {
    d=index(b16, substr($0,c,1))
    if (d--) {
      for (b=1; b<=4; b++) {
        o=o*2+int(d/8); d=(d*2)%%16
        if (++obc==6) {
          printf substr(b64,o+1,1)
          if (++rc>75) { printf \"\\n\"; rc=0 }
          obc=0; o=0
        }
      }
    }
  }
}
END {
  if (obc) {
    tail=(obc==2) ? \"==\\n\" : \"=\\n\"
    while (obc++<6) { o=o*2 }
    printf \"%%c\", substr(b64,o+1,1)
  } else {
    tail=\"\\n\"
  }
  printf tail
}'"
  "Awk/hexdump program to use for encoding a file.
This string is passed to `format', so percent characters need to be doubled.")

[-- Attachment #2: tramp-use-hexdump-with-extra-shell-scripting.diff --]
[-- Type: text/x-diff, Size: 6182 bytes --]

diff --git a/doc/misc/tramp.texi b/doc/misc/tramp.texi
index 2eb5b45eb2..ade4d5cb2d 100644
--- a/doc/misc/tramp.texi
+++ b/doc/misc/tramp.texi
@@ -1960,16 +1960,31 @@ Remote programs
 @section How @value{tramp} finds and uses programs on the remote host
 
 @value{tramp} requires access to and rights to several commands on
-remote hosts: @command{ls}, @command{test}, @command{find} and
+remote hosts: at least @command{ls}, @command{test}, @command{find} and
 @command{cat}.
 
+For inline methods, at least one of the following should be available:
+
+@itemize @bullet
+@item @command{base64}
+@item @command{openssl}
+@item @command{mimencode}
+@item @command{mmencode}
+@item @command{recode}
+@item @command{perl} or @command{perl5} (if the  @command{MIME::Base64}
+module is available, it will be used)
+@item @command{od} and @command{busybox awk}
+@item @command{hexdump} and @command{busybox awk}
+@item @command{uuencode} and @command{uudecode}
+@end itemize
+
+If none of these are available, @value{tramp} will not work with inline methods,
+but other methods may still be used. If @command{gzip} is available, it is
+also used, in order to improve performance.
+
 Besides there are other required programs for @ref{Inline methods} and
 @ref{External methods} of connection.
 
-To improve performance and accuracy of remote file access,
-@value{tramp} uses @command{perl} (or @command{perl5}) and
-@command{grep} when available.
-
 @defopt tramp-remote-path
 @code{tramp-remote-path} specifies which remote directory paths
 @value{tramp} can search for @ref{Remote programs}.
diff --git a/lisp/net/tramp-sh.el b/lisp/net/tramp-sh.el
index 95fa61af98..fc2fb51b8a 100644
--- a/lisp/net/tramp-sh.el
+++ b/lisp/net/tramp-sh.el
@@ -850,8 +850,12 @@ tramp-perl-unpack
   "Perl program to use for decoding a file.
 Escape sequence %s is replaced with name of Perl binary.")
 
-(defconst tramp-awk-encode
-  "od -v -t x1 -A n | busybox awk '\\
+(defconst tramp-hexdump-awk-encode
+  "{ { awk '{}' </dev/null && BA=\"\"  ; } || \
+     { busybox awk '{}' </dev/null &&  BA=busybox ; } } && \
+   { { hexdump -v -e '16/1 \" %%02x\" \"\\n\"' < /dev/null && BH=\"\" ; } || \
+     { busybox hexdump -v -e '16/1 \" %%02x\" \"\\n\"' < /dev/null && BH=busybox ; } } && \
+$BH hexdump -v -e '16/1 \" %%02x\" \"\\n\"'| $BA awk '\\
 BEGIN {
   b64 = \"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/\"
   b16 = \"0123456789abcdef\"
@@ -881,11 +885,51 @@ tramp-awk-encode
   }
   printf tail
 }'"
-  "Awk program to use for encoding a file.
+  "Awk/hexdump program to use for encoding a file.
+This string is passed to `format', so percent characters need to be doubled.")
+
+(defconst tramp-od-awk-encode
+  "{ { awk '{}' </dev/null && BA=\"\"  ; } || \
+     { busybox awk '{}' </dev/null &&  BA=busybox  ; } } && \
+   { { od -v -t x1 -A n < /dev/null && BO=\"\" ; } || \
+     { busybox od -v -t x1 -A n < /dev/null && BO=busybox ; } } && \
+$BO od -v -t x1 -A n $BA awk '\\
+BEGIN {
+  b64 = \"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/\"
+  b16 = \"0123456789abcdef\"
+}
+{
+  for (c=1; c<=length($0); c++) {
+    d=index(b16, substr($0,c,1))
+    if (d--) {
+      for (b=1; b<=4; b++) {
+        o=o*2+int(d/8); d=(d*2)%%16
+        if (++obc==6) {
+          printf substr(b64,o+1,1)
+          if (++rc>75) { printf \"\\n\"; rc=0 }
+          obc=0; o=0
+        }
+      }
+    }
+  }
+}
+END {
+  if (obc) {
+    tail=(obc==2) ? \"==\\n\" : \"=\\n\"
+    while (obc++<6) { o=o*2 }
+    printf \"%%c\", substr(b64,o+1,1)
+  } else {
+    tail=\"\\n\"
+  }
+  printf tail
+}'"
+  "Awk/od program to use for encoding a file.
 This string is passed to `format', so percent characters need to be doubled.")
 
 (defconst tramp-awk-decode
-  "busybox awk '\\
+  "{ { awk '{}' </dev/null && B=\"\"  ; } || \
+     { busybox awk '{}' </dev/null &&  B=busybox  ; } } && \
+$B awk '\\
 BEGIN {
   b64 = \"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/\"
 }
@@ -910,11 +954,25 @@ tramp-awk-decode
   "Awk program to use for decoding a file.
 This string is passed to `format', so percent characters need to be doubled.")
 
-(defconst tramp-awk-coding-test
+
+(defconst tramp-hexdump-awk-coding-test
+  "test -c /dev/zero && \
+{ {          hexdump -v -e '16/1 \" %02x\" \"\\n\"' < /dev/null ; } || \
+  { busybox  hexdump -v -e '16/1 \" %02x\" \"\\n\"' < /dev/null ; }    } \
+&& \
+{ {         awk '{}' </dev/null ; } || \
+  { busybox awk '{}' </dev/null ; } } "
+  "Test command for checking `tramp-hexdump-awk-encode' and \
+`tramp-awk-decode'.")
+
+(defconst tramp-od-awk-coding-test
   "test -c /dev/zero && \
-od -v -t x1 -A n </dev/null && \
-busybox awk '{}' </dev/null"
-  "Test command for checking `tramp-awk-encode' and `tramp-awk-decode'.")
+{ {          od -v -t x1 -A n < /dev/null ; } || \
+  { busybox  od -v -t x1 -A n < /dev/null ; }    } \
+&& \
+{ {         awk '{}' </dev/null ; } || \
+  { busybox awk '{}' </dev/null ; } } "
+  "Test command for checking `tramp-od-awk-encode' and `tramp-awk-decode'.")
 
 (defconst tramp-vc-registered-read-file-names
   "echo \"(\"
@@ -3167,7 +3225,8 @@ tramp-sh-handle-file-local-copy
 
 	   ;; Oops, I don't know what to do.
 	   (t (tramp-error
-	       v 'file-error "Wrong method specification for `%s'" method)))
+	       v 'file-error  "Cannot find any way to encode data using `%s'"
+	       method)))
 
 	;; Error handling.
 	((error quit)
@@ -4370,8 +4429,10 @@ tramp-remote-coding-commands
     (b64 "recode data..base64" "recode base64..data")
     (b64 tramp-perl-encode-with-module tramp-perl-decode-with-module)
     (b64 tramp-perl-encode tramp-perl-decode)
-    ;; This is painful slow, so we put it on the end.
-    (b64 tramp-awk-encode tramp-awk-decode ,tramp-awk-coding-test)
+    ;; These is painfully slow, so we put it on the end.
+    (b64 tramp-od-awk-encode tramp-awk-decode ,tramp-od-awk-coding-test)
+    (b64 tramp-hexdump-awk-encode tramp-awk-decode
+	 ,tramp-hexdump-awk-coding-test)
     (uu  "uuencode xxx" "uudecode -o /dev/stdout" "test -c /dev/stdout")
     (uu  "uuencode xxx" "uudecode -o -")
     (uu  "uuencode xxx" "uudecode -p")

bug#35639: 27.0.50; tramp fails to use ssh on LibreCMC (no base64 encoder available, and not mentioned in tramp manual)

[Michael Albinus]

Jeronimo Pellegrini <j_p@aleph0.info> writes:

Hi Jeronimo,

>> Or I can try to make the shell code that is sent to the remote
>> end do the verification itself (I'd test using dash, which I
>> hope will work on most systems of interest). Something like
>>
>> { hexdump < /dev/null && HD=hexdump; } || \
>> { busybox hexdump < /dev/null && HD="busybox hexdump"; } &&  \
>> $HD ...
>
> I actually implemented the shell scripting solution.
> Only tramp-hexdump-awk-encode is in the email body, but the
> full patch is attached.
>
> This seems to work, and requires no extra coding in tramp
> (only the constant strings are changed).

Thanks for this. However, it is not the approach Tramp goes for remote
command detection. The major downsides are, that the whole script must
be transferred to the remote side again and again, and that the
detection code runs again and again. Both decreases the performance.

> I can also do the templating in elisp, if you think it would be
> better.

Yes, please. First, you shall define the functions

tramp-get-remote-busybox
tramp-get-remote-awk
tramp-get-remote-hexdump
tramp-get-remote-od

All of them use connection caching, in order to run the detection only
once, and use the cached result afterwards. tramp-get-remote-busybox is
just a helper function for the other three functions, it could look like

(with-tramp-connection-property vec "busybox"
  (tramp-message vec 5 "Finding a suitable `busybox' command")
  (tramp-find-executable vec "busybox" (tramp-get-remote-path vec)))

See, how other tramp-get-remote-* functions in tramp-sh.el look like.
tramp-get-remote-{awk,hexdump,od} are implemented as

(with-tramp-connection-property vec "awk"
  (tramp-message vec 5 "Finding a suitable `awk' command")
  (or (tramp-find-executable vec "awk" (tramp-get-remote-path vec))
      (let* ((busybox (tramp-get-connection-property vec "busybox" nil))
	     (command (concat busybox " awk {} </dev/null")))
	(and busybox
	     (tramp-send-command-and-check vec command)
	     command))))

In tramp-remote-coding-commands, TEST must be extended not to be only a
string (which is run as command on the remote side), but also a Lisp
form to be evaluated. We would need tramp-hexdump-awk-coding-test and
tramp-od-awk-coding-test, with code like this:

(and (tramp-get-connection-property vec "hexdump" nil)
     (tramp-get-connection-property vec "awk" nil))

and the same for tramp-od-awk-coding-test.

Finally, the encoding/decoding commands in
tramp-{hexcdump,od}-awk-{encode,decode} would need place-holders for
%a (for awk), %h (for hexdump), and %o (for od), like:

"%h -v -e '16/1 \" %%02x\" \"\\n\"'| %a '\\
 ...
 }'"

In tramp-find-inline-encoding, these place-holders must be expanded by
the result of tramp-get-connection-property for the respective command.

All of this sounds a little bit complicate, but it ensures, that every
test on the remote side is run only once, and the result is taken from
the cache afterwards.

Best regards, Michael.

bug#35639: 27.0.50; tramp fails to use ssh on LibreCMC (no base64 encoder available, and not mentioned in tramp manual)

[Jeronimo Pellegrini]

Hi,

I am working on the patch, but due to health issues
it will take a while to answer (some days).

J.

bug#35639: 27.0.50; tramp fails to use ssh on LibreCMC (no base64 encoder available, and not mentioned in tramp manual)

[Michael Albinus]

Jeronimo Pellegrini <j_p@aleph0.info> writes:

> Hi,

Hi Jeronimo,

> I am working on the patch, but due to health issues
> it will take a while to answer (some days).

No problem, take all the time it needs. Your health is much more
important than a little program.

> J.

Get well soon! Michael.

bug#35639: 27.0.50; tramp fails to use ssh on LibreCMC (no base64 encoder available, and not mentioned in tramp manual)

[Michael Albinus]

Michael Albinus <michael.albinus@gmx.de> writes:

Hi Jeronimo,

>> And I suppose the patch stays on hold until I finish the
>> FSF assignment process -- is that right?
>
> Yes. Unless it is less than 15 changed lines, than we could apply
> immediately. Your FSF assignment will help in the future anyway.

This was in May. Do you have feedback from the FSF about your
assignment?

Best regards, Michael.

bug#35639: 27.0.50; tramp fails to use ssh on LibreCMC (no base64 encoder available, and not mentioned in tramp manual)

[Jeronimo Pellegrini via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Hello Michael!

On Fri, Oct 11, 2019 at 03:14:48PM +0200, Michael Albinus wrote:
> Michael Albinus <michael.albinus@gmx.de> writes:
> 
> Hi Jeronimo,
> 
> >> And I suppose the patch stays on hold until I finish the
> >> FSF assignment process -- is that right?
> >
> > Yes. Unless it is less than 15 changed lines, than we could apply
> > immediately. Your FSF assignment will help in the future anyway.
> 
> This was in May. Do you have feedback from the FSF about your
> assignment?

Yes, sorry about the long time without any news -- I underwent 
spine surgery and recovery took over two months.
Also, thinks have been a bt confusing on our side as to how to
get the paperwork; I'll be getting in touch with the FSF this week
in order to get things sorted out.
 
J.

bug#35639: 27.0.50; tramp fails to use ssh on LibreCMC (no base64 encoder available, and not mentioned in tramp manual)

[Michael Albinus]

Jeronimo Pellegrini <j_p@aleph0.info> writes:

> Hello Michael!

Hi Jeronimo,

> Yes, sorry about the long time without any news -- I underwent
> spine surgery and recovery took over two months.

Oh. I wish you all the best in recovery. Take your time, this is much
more important than a little Emacs package ...

> Also, thinks have been a bt confusing on our side as to how to
> get the paperwork; I'll be getting in touch with the FSF this week
> in order to get things sorted out.

Just in case of problems, you could ask here.

> J.

Best regards, Michael.

bug#35639: 27.0.50; tramp fails to use ssh on LibreCMC (no base64 encoder available, and not mentioned in tramp manual)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Yes, sorry about the long time without any news -- I underwent 
  > spine surgery and recovery took over two months.

Will the operation give you the spine necessary to firmly
reject nonfree software?

;-}.

-- 
Dr Richard Stallman
Founder, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#35639: 27.0.50; tramp fails to use ssh on LibreCMC (no base64 encoder available, and not mentioned in tramp manual)

[Michael Albinus]

Michael Albinus <michael.albinus@gmx.de> writes:

Hi Jeronimo,

>> Also, thinks have been a bt confusing on our side as to how to
>> get the paperwork; I'll be getting in touch with the FSF this week
>> in order to get things sorted out.
>
> Just in case of problems, you could ask here.

The emacs-27 branch has been created in git, Emacs 27 will start its
pretest soon. Is there any progress in your paperwork with the FSF?

I would really like to have this solved in Emacs 27. If this is still
stalled, I could try to implement a patch myself. WDYT?

>> J.

Best regards, Michael.

bug#35639: 27.0.50; tramp fails to use ssh on LibreCMC (no base64 encoder available, and not mentioned in tramp manual)

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Date: Fri, 27 Dec 2019 18:26:09 +0100
> Cc: 35639@debbugs.gnu.org
> 
> The emacs-27 branch has been created in git, Emacs 27 will start its
> pretest soon. Is there any progress in your paperwork with the FSF?

It's in the works, but not complete yet.  The last message I've seen
was from May, so it's high time to ping the FSF clerk about this.

bug#35639: 27.0.50; tramp fails to use ssh on LibreCMC (no base64 encoder available, and not mentioned in tramp manual)

[Jeronimo Pellegrini]

Hi Michael,

On Fri, Dec 27, 2019 at 06:26:09PM +0100, Michael Albinus wrote:
> Michael Albinus <michael.albinus@gmx.de> writes:
> 
> Hi Jeronimo,
> 
> >> Also, thinks have been a bt confusing on our side as to how to
> >> get the paperwork; I'll be getting in touch with the FSF this week
> >> in order to get things sorted out.
> >
> > Just in case of problems, you could ask here.
> 
> The emacs-27 branch has been created in git, Emacs 27 will start its
> pretest soon. Is there any progress in your paperwork with the FSF?
> 
> I would really like to have this solved in Emacs 27. If this is still
> stalled, I could try to implement a patch myself. WDYT?

There have been some complications on the University side here, and 
although we do have a plan and could make it work, it would probably take 
some more time (months, likely).
I am sorry that I couldn't get things to work in a mre timely way, and
I hope this didn't cause much trouble.

Jeronimo

bug#35639: 27.0.50; tramp fails to use ssh on LibreCMC (no base64 encoder available, and not mentioned in tramp manual)

[Michael Albinus]

Jeronimo Pellegrini <jeronimo.pellegrini@ufabc.edu.br> writes:

> Hi Michael,

Hi Jeronimo,

>> I would really like to have this solved in Emacs 27. If this is still
>> stalled, I could try to implement a patch myself. WDYT?
>
> There have been some complications on the University side here, and
> although we do have a plan and could make it work, it would probably take
> some more time (months, likely).

I'll prepare a patch along the sketch I've shown in my message from 12
May. Since I have no access to an LibreCMC machine, I would give it to
you for testing. Would this be OK?

> I am sorry that I couldn't get things to work in a mre timely way, and
> I hope this didn't cause much trouble.

No, it doesn't. But it will be useful to sort this out, for the case
you'll offer the next patch (which will be much appreciated).

> Jeronimo

Best regards, Michael.

bug#35639: 27.0.50; tramp fails to use ssh on LibreCMC (no base64 encoder available, and not mentioned in tramp manual)

[Jeronimo Pellegrini via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Hi Michael!
On Wed, Jan 01, 2020 at 10:28:01AM +0100, Michael Albinus wrote:
> Jeronimo Pellegrini <jeronimo.pellegrini@ufabc.edu.br> writes:
> 
> > There have been some complications on the University side here, and
> > although we do have a plan and could make it work, it would probably take
> > some more time (months, likely).
> 
> I'll prepare a patch along the sketch I've shown in my message from 12
> May. Since I have no access to an LibreCMC machine, I would give it to
> you for testing. Would this be OK?

Yes, that would be fine.
I'll be travelling on the January 6th, and will be back on the 15th, and
I won't be able to do the testing in that period, but other than that,
I'll help as much as I can.

> > I am sorry that I couldn't get things to work in a mre timely way, and
> > I hope this didn't cause much trouble.
> 
> No, it doesn't. But it will be useful to sort this out, for the case
> you'll offer the next patch (which will be much appreciated).

I'll see how we can sort things out here...

J.

bug#35639: 27.0.50; tramp fails to use ssh on LibreCMC (no base64 encoder available, and not mentioned in tramp manual)

[Michael Albinus]

[-- Attachment #1: Type: text/plain, Size: 577 bytes --]

Jeronimo Pellegrini <j_p@aleph0.info> writes:

> Hi Michael!

Hi Jeronimo,

>> I'll prepare a patch along the sketch I've shown in my message from 12
>> May. Since I have no access to an LibreCMC machine, I would give it to
>> you for testing. Would this be OK?
>
> Yes, that would be fine.

I've committed the appended patch to the Emacs and Tramp git
repositories. Could you, pls, test with your LibeCMC machine?

If you don't use the git repositories, you might install Tramp 2.4.3
from GNU ELPA, and apply the patch on top of it.

> J.

Thanks, and best regards, Michael.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 8869 bytes --]

diff --git a/lisp/tramp-sh.el b/lisp/tramp-sh.el
index 6cd299ac..6e5b9d24 100644
--- a/lisp/tramp-sh.el
+++ b/lisp/tramp-sh.el
@@ -866,8 +866,12 @@ Escape sequence %s is replaced with name of Perl binary.")
   "Perl program to use for decoding a file.
 Escape sequence %s is replaced with name of Perl binary.")

+(defconst tramp-hexdump-encode "%h -v -e '16/1 \" %%02x\" \"\\n\"'"
+  "`hexdump' program to use for encoding a file.
+This string is passed to `format', so percent characters need to be doubled.")
+
 (defconst tramp-awk-encode
-  "od -v -t x1 -A n | busybox awk '\\
+  "%a '\\
 BEGIN {
   b64 = \"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/\"
   b16 = \"0123456789abcdef\"
@@ -897,11 +901,25 @@ END {
   }
   printf tail
 }'"
-  "Awk program to use for encoding a file.
+  "`awk' program to use for encoding a file.
+This string is passed to `format', so percent characters need to be doubled.")
+
+(defconst tramp-hexdump-awk-encode
+  (format "%s | %s" tramp-hexdump-encode tramp-awk-encode)
+  "`hexdump' / `awk' pipe to use for encoding a file.
+This string is passed to `format', so percent characters need to be doubled.")
+
+(defconst tramp-od-encode "%o -v -t x1 -A n"
+  "`od' program to use for encoding a file.
+This string is passed to `format', so percent characters need to be doubled.")
+
+(defconst tramp-od-awk-encode
+  (format "%s | %s" tramp-od-encode tramp-awk-encode)
+  "`od' / `awk' pipe to use for encoding a file.
 This string is passed to `format', so percent characters need to be doubled.")

 (defconst tramp-awk-decode
-  "busybox awk '\\
+  "%a '\\
 BEGIN {
   b64 = \"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/\"
 }
@@ -926,12 +944,6 @@ BEGIN {
   "Awk program to use for decoding a file.
 This string is passed to `format', so percent characters need to be doubled.")

-(defconst tramp-awk-coding-test
-  "test -c /dev/zero && \
-od -v -t x1 -A n </dev/null && \
-busybox awk '{}' </dev/null"
-  "Test command for checking `tramp-awk-encode' and `tramp-awk-decode'.")
-
 (defconst tramp-vc-registered-read-file-names
   "echo \"(\"
 while read file; do
@@ -4401,7 +4413,7 @@ and end of region, and are expected to replace the region contents
 with the encoded or decoded results, respectively.")

 (defconst tramp-remote-coding-commands
-  `((b64 "base64" "base64 -d -i")
+  '((b64 "base64" "base64 -d -i")
     ;; "-i" is more robust with older base64 from GNU coreutils.
     ;; However, I don't know whether all base64 versions do supports
     ;; this option.
@@ -4412,8 +4424,9 @@ with the encoded or decoded results, respectively.")
     (b64 "recode data..base64" "recode base64..data")
     (b64 tramp-perl-encode-with-module tramp-perl-decode-with-module)
     (b64 tramp-perl-encode tramp-perl-decode)
-    ;; This is painful slow, so we put it on the end.
-    (b64 tramp-awk-encode tramp-awk-decode ,tramp-awk-coding-test)
+    ;; These are painfully slow, so we put them on the end.
+    (b64 tramp-hexdump-awk-encode tramp-awk-decode)
+    (b64 tramp-od-awk-encode tramp-awk-decode)
     (uu  "uuencode xxx" "uudecode -o /dev/stdout" "test -c /dev/stdout")
     (uu  "uuencode xxx" "uudecode -o -")
     (uu  "uuencode xxx" "uudecode -p")
@@ -4439,6 +4452,8 @@ Perl or Shell implementation for this functionality.  This
 program will be transferred to the remote host, and it is
 available as shell function with the same name.  A \"%t\" format
 specifier in the variable value denotes a temporary file.
+\"%a\", \"%h\" and \"%o\" format specifiers are replaced by the
+respective `awk', `hexdump' and `od' commands.

 The optional TEST command can be used for further tests, whether
 ENCODING and DECODING are applicable.")
@@ -4489,11 +4504,6 @@ Goes through the list `tramp-local-coding-commands' and
 		     vec 5 "Checking remote test command `%s'" rem-test)
 		    (unless (tramp-send-command-and-check vec rem-test t)
 		      (throw 'wont-work-remote nil)))
-		  ;; Check if remote perl exists when necessary.
-		  (when (and (symbolp rem-enc)
-			     (string-match-p "perl" (symbol-name rem-enc))
-			     (not (tramp-get-remote-perl vec)))
-		    (throw 'wont-work-remote nil))
 		  ;; Check if remote encoding and decoding commands can be
 		  ;; called remotely with null input and output.  This makes
 		  ;; sure there are no syntax errors and the command is really
@@ -4503,10 +4513,36 @@ Goes through the list `tramp-local-coding-commands' and
 		  ;; redirecting "mimencode" output to /dev/null, then as root
 		  ;; it might change the permissions of /dev/null!
 		  (unless (stringp rem-enc)
-		    (let ((name (symbol-name rem-enc)))
+		    (let ((name (symbol-name rem-enc))
+			  (value (symbol-value rem-enc)))
+		      ;; Check if remote perl exists when necessary.
+		      (and (string-match-p "perl" name)
+			   (not (tramp-get-remote-perl vec))
+			   (throw 'wont-work-remote nil))
+		      ;; Check if remote awk exists when necessary.
+		      (and (string-match-p "\\(^\\|[^%]\\)%a" value)
+			   (not (tramp-get-remote-awk vec))
+			   (throw 'wont-work-remote nil))
+		      ;; Check if remote hexdump exists when necessary.
+		      (and (string-match-p "\\(^\\|[^%]\\)%h" value)
+			   (not (tramp-get-remote-hexdump vec))
+			   (throw 'wont-work-remote nil))
+		      ;; Check if remote od exists when necessary.
+		      (and (string-match-p "\\(^\\|[^%]\\)%o" value)
+			   (not (tramp-get-remote-od vec))
+			   (throw 'wont-work-remote nil))
 		      (while (string-match "-" name)
 			(setq name (replace-match "_" nil t name)))
-		      (tramp-maybe-send-script vec (symbol-value rem-enc) name)
+		      (when (string-match-p "\\(^\\|[^%]\\)%[aho]" value)
+			(setq value
+			      (format-spec
+			       value
+			       (format-spec-make
+				?a (tramp-get-remote-awk vec)
+				?h (tramp-get-remote-hexdump vec)
+				?o (tramp-get-remote-od vec)))
+			      value (replace-regexp-in-string "%" "%%" value)))
+		      (tramp-maybe-send-script vec value name)
 		      (setq rem-enc name)))
 		  (tramp-message
 		   vec 5
@@ -4521,6 +4557,15 @@ Goes through the list `tramp-local-coding-commands' and
 			  tmpfile)
 		      (while (string-match "-" name)
 			(setq name (replace-match "_" nil t name)))
+		      (when (string-match-p "\\(^\\|[^%]\\)%[aho]" value)
+			(setq value
+			      (format-spec
+			       value
+			       (format-spec-make
+				?a (tramp-get-remote-awk vec)
+				?h (tramp-get-remote-hexdump vec)
+				?o (tramp-get-remote-od vec)))
+			      value (replace-regexp-in-string "%" "%%" value)))
 		      (when (string-match-p "\\(^\\|[^%]\\)%t" value)
 			(setq tmpfile
 			      (make-temp-name
@@ -5787,6 +5832,47 @@ ID-FORMAT valid values are `string' and `integer'."
 	tramp-unknown-id-string)
        (t res)))))

+(defun tramp-get-remote-busybox (vec)
+  "Determine remote `busybox' command."
+  (with-tramp-connection-property vec "busybox"
+    (tramp-message vec 5 "Finding a suitable `busybox' command")
+    (tramp-find-executable vec "busybox" (tramp-get-remote-path vec))))
+
+(defun tramp-get-remote-awk (vec)
+  "Determine remote `awk' command."
+  (with-tramp-connection-property vec "awk"
+    (tramp-message vec 5 "Finding a suitable `awk' command")
+    (or (tramp-find-executable vec "awk" (tramp-get-remote-path vec))
+	(let* ((busybox (tramp-get-remote-busybox vec))
+	       (command (format "%s %s" busybox "awk")))
+	  (and busybox
+	       (tramp-send-command-and-check
+		vec (concat command " {} </dev/null"))
+	       command)))))
+
+(defun tramp-get-remote-hexdump (vec)
+  "Determine remote `hexdump' command."
+  (with-tramp-connection-property vec "hexdump"
+    (tramp-message vec 5 "Finding a suitable `hexdump' command")
+    (or (tramp-find-executable vec "hexdump" (tramp-get-remote-path vec))
+	(let* ((busybox (tramp-get-remote-busybox vec))
+	       (command (format "%s %s" busybox "hexdump")))
+	  (and busybox
+	       (tramp-send-command-and-check vec (concat command " </dev/null"))
+	       command)))))
+
+(defun tramp-get-remote-od (vec)
+  "Determine remote `od' command."
+  (with-tramp-connection-property vec "od"
+    (tramp-message vec 5 "Finding a suitable `od' command")
+    (or (tramp-find-executable vec "od" (tramp-get-remote-path vec))
+	(let* ((busybox (tramp-get-remote-busybox vec))
+	       (command (format "%s %s" busybox "od")))
+	  (and busybox
+	       (tramp-send-command-and-check
+		vec (concat command " -A n </dev/null"))
+	       command)))))
+
 (defun tramp-get-env-with-u-option (vec)
   "Check, whether the remote `env' command supports the -u option."
   (with-tramp-connection-property vec "env-u-option"

bug#35639: 27.0.50; tramp fails to use ssh on LibreCMC (no base64 encoder available, and not mentioned in tramp manual)

[Jeronimo Pellegrini via Bug reports for GNU Emacs, the Swiss army knife of text editors]

On Thu, Jan 09, 2020 at 01:58:42PM +0100, Michael Albinus wrote:
> Hi Jeronimo,

Hello Michael!

> 
> I've committed the appended patch to the Emacs and Tramp git
> repositories. Could you, pls, test with your LibeCMC machine?

Yes -- however, I am travelling, and will only be back on the 15th
of January, and until then I won't have access to that mahine.
Is that ok?

> If you don't use the git repositories, you might install Tramp 2.4.3
> from GNU ELPA, and apply the patch on top of it.

I use Emacs from the git repository -- I suppose it already contains
the patch?

J.

bug#35639: 27.0.50; tramp fails to use ssh on LibreCMC (no base64 encoder available, and not mentioned in tramp manual)

[Michael Albinus]

Jeronimo Pellegrini <j_p@aleph0.info> writes:

> Hello Michael!

Hi Jeronimo,

>> I've committed the appended patch to the Emacs and Tramp git
>> repositories. Could you, pls, test with your LibeCMC machine?
>
> Yes -- however, I am travelling, and will only be back on the 15th
> of January, and until then I won't have access to that mahine.
> Is that ok?

I know, take your time. We're already too late for Emacs 27.1, which
will start its pretest soon. So it will be in Emacs 27.2.

If everything goes well, I also plan to release it with Tramp 2.4.3.1,
which is scheduled roughly for end of January.

>> If you don't use the git repositories, you might install Tramp 2.4.3
>> from GNU ELPA, and apply the patch on top of it.
>
> I use Emacs from the git repository -- I suppose it already contains
> the patch?

Yes, in the master branch.

> J.

Best regards, Michael.

bug#35639: 27.0.50; tramp fails to use ssh on LibreCMC (no base64 encoder available, and not mentioned in tramp manual)

[Jeronimo Pellegrini via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Hello!

On Sat, Jan 11, 2020 at 02:23:05PM +0100, Michael Albinus wrote:
> Hi Jeronimo,
> 
> >> I've committed the appended patch to the Emacs and Tramp git
> >> repositories. Could you, pls, test with your LibeCMC machine?

I just tested it with the exact same precedure that I used when I reported 
the bug, and it worked perfectly!

> >> If you don't use the git repositories, you might install Tramp 2.4.3
> >> from GNU ELPA, and apply the patch on top of it.

I compiled Emacs from git (updated today, Jan 16)...

So I believe this is solved! :-)

J.

bug#35639: 27.0.50; tramp fails to use ssh on LibreCMC (no base64 encoder available, and not mentioned in tramp manual)

[Michael Albinus]

Version: 27.2

Jeronimo Pellegrini <j_p@aleph0.info> writes:

> Hello!

Hi Jeronimo,

>> >> I've committed the appended patch to the Emacs and Tramp git
>> >> repositories. Could you, pls, test with your LibeCMC machine?
>
> I just tested it with the exact same precedure that I used when I reported
> the bug, and it worked perfectly!

Thanks for the feedback.

>> >> If you don't use the git repositories, you might install Tramp 2.4.3
>> >> from GNU ELPA, and apply the patch on top of it.
>
> I compiled Emacs from git (updated today, Jan 16)...
>
> So I believe this is solved! :-)

Yep. I'm closing the bug.

> J.

Best regards, Michael.