bug#35055: 27.0.50; async-shell-command truncates output lines

bug#35055: 27.0.50; async-shell-command truncates output lines

[Juri Linkov]

Test case:

1. Open a wide frame and type:

2. `M-& ps aux RET'

3. observe that output lines are truncated at column 80

Another example: `M-& dpkg -l '*emacs*' RET', then try to copy
full package names from the *Async Shell Command* buffer.
Nope, it's impossible because they are truncated.

OTOH, there is no such problem with `M-! ps aux RET'
where lines are not truncated at all.

So the question is: why `M-&' (async-shell-command) limits COLUMNS to 80,
even on wide frames, whereas `M-!' (shell-command) has no limitation.

A third case is `M-x shell' that limits COLUMNS to window-width,
but at least it makes possible to change the window width (e.g.
to make it wider) to influence the number of output columns.

bug#35055: 27.0.50; async-shell-command truncates output lines

[Michael Albinus]

Juri Linkov <juri@linkov.net> writes:

Hi Juri,

> Test case:
>
> 1. Open a wide frame and type:
>
> 2. `M-& ps aux RET'
>
> 3. observe that output lines are truncated at column 80

In my case, output lines are truncated at column 89. In fact the
truncation happens exactly at the size of the *Async Shell Command*
buffer.

> OTOH, there is no such problem with `M-! ps aux RET'
> where lines are not truncated at all.
>
> So the question is: why `M-&' (async-shell-command) limits COLUMNS to 80,
> even on wide frames, whereas `M-!' (shell-command) has no limitation.

Internally, M-! uses `call-process', and M-& uses
`start-file-process'. They have different implementations.

Synchronous processes do not care about the buffer width. So you see
untruncated output.

Asynchronous processes care. They send the information about buffer
dimensions with the function `set-process-window-size', see
(info "(elisp) Process Buffers")

There is the variable `window-adjust-process-window-size-function' which
controls how the dimension information is given to the underlying
process.

I'm not sure whether there exists already a configuration option to
allow asynchronous shell commands using infinite line width, but it
could be implemented this way.

Note that for remote shell commands the situation is even worse, because
it sets the process property `adjust-window-size-function' to nil,
overwriting any setting in `window-adjust-process-window-size-function'.
This affects even synchronous `shell-command' calls, because they are
implemented Tramp internally as asynchronous process.

Best regards, Michael.

bug#35055: 27.0.50; async-shell-command truncates output lines

[Juri Linkov]

>> 1. Open a wide frame and type:
>>
>> 2. `M-& ps aux RET'
>>
>> 3. observe that output lines are truncated at column 80
>
> In my case, output lines are truncated at column 89. In fact the
> truncation happens exactly at the size of the *Async Shell Command*
> buffer.

This is strange.  I tried in `emacs -Q' in a wide frame and
output lines of `ps aux' are always truncated at column 80.

Then I tried to affect the number of columns with the
environment variable COLUMNS, and when using

  M-& COLUMNS=89 ps aux

only then output lines are truncated at column 89.
Trying different values produced different truncation:

  COLUMNS=0 ps aux      - column 80
  COLUMNS=1 ps aux      - column 68
  COLUMNS=5 ps aux      - column 70
  COLUMNS=20 ps aux     - column 80
  COLUMNS=30 ps aux     - column 90
  COLUMNS=40 ps aux     - column 80
  COLUMNS=50 ps aux     - column 100
  COLUMNS=60 ps aux     - column 120
  COLUMNS=70 ps aux     - column 70
  COLUMNS=131071 ps aux - practically unlimited
  COLUMNS=131072 ps aux - again column 80 (looks like maximum reached)

i.e. dependency is not linear.

Also `M-x shell' uses `comint-term-environment' to set COLUMNS
to window-width, but it does this only initially,
so after resizing the shell window, COLUMNS remains
at the old initial value.

>> OTOH, there is no such problem with `M-! ps aux RET'
>> where lines are not truncated at all.
>>
>> So the question is: why `M-&' (async-shell-command) limits COLUMNS to 80,
>> even on wide frames, whereas `M-!' (shell-command) has no limitation.
>
> Internally, M-! uses `call-process', and M-& uses
> `start-file-process'. They have different implementations.
>
> Synchronous processes do not care about the buffer width. So you see
> untruncated output.
>
> Asynchronous processes care. They send the information about buffer
> dimensions with the function `set-process-window-size', see
> (info "(elisp) Process Buffers")
>
> There is the variable `window-adjust-process-window-size-function' which
> controls how the dimension information is given to the underlying
> process.

Thanks for the pointer to process-window-size functions.
I tried to debug them in `emacs -Q', not to affect its default
behavior with customization of async-shell-command-display-buffer
that complicates the issue more.

But it seems `set-process-window-size' has no effect on
the truncated columns.  I see that `set-process-window-size'
in `window--adjust-process-windows' is called with WIDTH
more than 80, but the output of `ps aux' is still truncated
to 80 columns in *Async Shell Command*.

> I'm not sure whether there exists already a configuration option to
> allow asynchronous shell commands using infinite line width, but it
> could be implemented this way.
>
> Note that for remote shell commands the situation is even worse, because
> it sets the process property `adjust-window-size-function' to nil,
> overwriting any setting in `window-adjust-process-window-size-function'.
> This affects even synchronous `shell-command' calls, because they are
> implemented Tramp internally as asynchronous process.

I had truncated output of remote `shell-command' for a long time
and thought that it's impossible to do anything with this,
but now that you mentioned remote shell commands, I tried
to affect their output with a quite large value of COLUMNS
and get untruncated output even on remote.

bug#35055: 27.0.50; async-shell-command truncates output lines

[Michael Albinus]

Juri Linkov <juri@linkov.net> writes:

>>> 1. Open a wide frame and type:
>>>
>>> 2. `M-& ps aux RET'
>>>
>>> 3. observe that output lines are truncated at column 80
>>
>> In my case, output lines are truncated at column 89. In fact the
>> truncation happens exactly at the size of the *Async Shell Command*
>> buffer.
>
> This is strange.  I tried in `emacs -Q' in a wide frame and
> output lines of `ps aux' are always truncated at column 80.

Strange indeed. Today I'm not in office but @home. Now I get the same
behaviour as you.

On both machines I've used a fresh compiled Emacs from master. The only
difference is, that @office I use a Fedora 29 machine with a large
monitor, and @home Ubuntu 18.10 with a smaller monitor. Don't know
whether this makes the difference. Anyway, ...

> Then I tried to affect the number of columns with the
> environment variable COLUMNS, and when using
>
>   M-& COLUMNS=89 ps aux
>
> only then output lines are truncated at column 89.

Same here.

> Trying different values produced different truncation:
>
>   COLUMNS=0 ps aux      - column 80
>   COLUMNS=1 ps aux      - column 68
>   COLUMNS=5 ps aux      - column 70
>   COLUMNS=20 ps aux     - column 80
>   COLUMNS=30 ps aux     - column 90
>   COLUMNS=40 ps aux     - column 80
>   COLUMNS=50 ps aux     - column 100
>   COLUMNS=60 ps aux     - column 120
>   COLUMNS=70 ps aux     - column 70
>   COLUMNS=131071 ps aux - practically unlimited
>   COLUMNS=131072 ps aux - again column 80 (looks like maximum reached)
>
> i.e. dependency is not linear.

Same here. I guess that setting COLUMNS is not reliable for COLUMNS < 80
and > 131072. But at least it is a way to influence the output width of
asynchronous processes.

>> Note that for remote shell commands the situation is even worse, because
>> it sets the process property `adjust-window-size-function' to nil,
>> overwriting any setting in `window-adjust-process-window-size-function'.
>> This affects even synchronous `shell-command' calls, because they are
>> implemented Tramp internally as asynchronous process.
>
> I had truncated output of remote `shell-command' for a long time
> and thought that it's impossible to do anything with this,
> but now that you mentioned remote shell commands, I tried
> to affect their output with a quite large value of COLUMNS
> and get untruncated output even on remote.

That's because Tramp sets process property `adjust-window-size-function'
to `ignore' (and not to nil as I wrote above). This was 2 years ago;
unfortunately, I have neither a bug nor a message reference for that change.

Best regards, Michael.

bug#35055: 27.0.50; async-shell-command truncates output lines

[Juri Linkov]

>>> Note that for remote shell commands the situation is even worse, because
>>> it sets the process property `adjust-window-size-function' to nil,
>>> overwriting any setting in `window-adjust-process-window-size-function'.
>>> This affects even synchronous `shell-command' calls, because they are
>>> implemented Tramp internally as asynchronous process.
>>
>> I had truncated output of remote `shell-command' for a long time
>> and thought that it's impossible to do anything with this,
>> but now that you mentioned remote shell commands, I tried
>> to affect their output with a quite large value of COLUMNS
>> and get untruncated output even on remote.
>
> That's because Tramp sets process property `adjust-window-size-function'
> to `ignore' (and not to nil as I wrote above). This was 2 years ago;
> unfortunately, I have neither a bug nor a message reference for that change.

Then I don't understand why remote shell output is limited to 80 columns
when Tramp sets `adjust-window-size-function' to `ignore', i.e. not to 80.

bug#35055: 27.0.50; async-shell-command truncates output lines

[Juri Linkov]

>>>> Note that for remote shell commands the situation is even worse, because
>>>> it sets the process property `adjust-window-size-function' to nil,
>>>> overwriting any setting in `window-adjust-process-window-size-function'.
>>>> This affects even synchronous `shell-command' calls, because they are
>>>> implemented Tramp internally as asynchronous process.
>>>
>>> I had truncated output of remote `shell-command' for a long time
>>> and thought that it's impossible to do anything with this,
>>> but now that you mentioned remote shell commands, I tried
>>> to affect their output with a quite large value of COLUMNS
>>> and get untruncated output even on remote.
>>
>> That's because Tramp sets process property `adjust-window-size-function'
>> to `ignore' (and not to nil as I wrote above). This was 2 years ago;
>> unfortunately, I have neither a bug nor a message reference for that change.
>
> Then I don't understand why remote shell output is limited to 80 columns
> when Tramp sets `adjust-window-size-function' to `ignore', i.e. not to 80.

We could add a new defcustom process-window-width with a choice of
a numeric value to set COLUMNS env variable in the process, or
a symbol `window-width' to set COLUMNS to the width of the output
buffer's window.  Then it will handle both asynchronous processes
and synchronous remote shell commands as well.

bug#35055: 27.0.50; async-shell-command truncates output lines

[Michael Albinus]

Juri Linkov <juri@linkov.net> writes:

Hi Juri,

>>> That's because Tramp sets process property `adjust-window-size-function'
>>> to `ignore' (and not to nil as I wrote above). This was 2 years ago;
>>> unfortunately, I have neither a bug nor a message reference for that change.
>>
>> Then I don't understand why remote shell output is limited to 80 columns
>> when Tramp sets `adjust-window-size-function' to `ignore', i.e. not to 80.
>
> We could add a new defcustom process-window-width with a choice of
> a numeric value to set COLUMNS env variable in the process, or
> a symbol `window-width' to set COLUMNS to the width of the output
> buffer's window.  Then it will handle both asynchronous processes
> and synchronous remote shell commands as well.

Would be possible. However, I wouldn't speak about COLUMNS of the
underlying process, because this is implementation detail. For remote
processes the process object you see is related to the *local* process
Tramp has opened. But we want to influence the *remote* process, which
has been started from the local process via ssh (for example).

The following code snippet does this already (with an example value of
1024 chars for the width)

--8<---------------cut here---------------start------------->8---
(defun set-fixed-process-window-width ()
  (let ((proc (get-buffer-process (current-buffer))))
    (set-process-window-size proc (window-body-height (selected-window)) 1024)
    ;; Don't change size when Emacs window changes.
    (process-put proc 'adjust-window-size-function #'ignore)))

(add-hook 'shell-mode-hook 'set-fixed-process-window-width)
--8<---------------cut here---------------end--------------->8---

You can test it like this, for example

--8<---------------cut here---------------start------------->8---
(shell-command-to-string "ps aux")
(shell-command-to-string "ps aux &")
--8<---------------cut here---------------end--------------->8---

or in a shell

--8<---------------cut here---------------start------------->8---
M-x shell
## in the *shell* buffer
$ echo $COLUMNS
1024
--8<---------------cut here---------------end--------------->8---

Best regards, Michael.

bug#35055: 27.0.50; async-shell-command truncates output lines

[Juri Linkov]

>>>> That's because Tramp sets process property `adjust-window-size-function'
>>>> to `ignore' (and not to nil as I wrote above). This was 2 years ago;
>>>> unfortunately, I have neither a bug nor a message reference for that change.
>>>
>>> Then I don't understand why remote shell output is limited to 80 columns
>>> when Tramp sets `adjust-window-size-function' to `ignore', i.e. not to 80.
>>
>> We could add a new defcustom process-window-width with a choice of
>> a numeric value to set COLUMNS env variable in the process, or
>> a symbol `window-width' to set COLUMNS to the width of the output
>> buffer's window.  Then it will handle both asynchronous processes
>> and synchronous remote shell commands as well.
>
> Would be possible. However, I wouldn't speak about COLUMNS of the
> underlying process, because this is implementation detail. For remote
> processes the process object you see is related to the *local* process
> Tramp has opened. But we want to influence the *remote* process, which
> has been started from the local process via ssh (for example).
>
> The following code snippet does this already (with an example value of
> 1024 chars for the width)
>
> (defun set-fixed-process-window-width ()
>   (let ((proc (get-buffer-process (current-buffer))))
>     (set-process-window-size proc (window-body-height (selected-window)) 1024)
>     ;; Don't change size when Emacs window changes.
>     (process-put proc 'adjust-window-size-function #'ignore)))
>
> (add-hook 'shell-mode-hook 'set-fixed-process-window-width)
>
>
> You can test it like this, for example
>
> (shell-command-to-string "ps aux")
> (shell-command-to-string "ps aux &")
>
>
> or in a shell
>
> M-x shell
> ## in the *shell* buffer
> $ echo $COLUMNS
> 1024

I didn't know that set-process-window-size changes the
value of COLUMNS.  I thought that COLUMNS is set only by
comint-term-environment.  One thing I still don't understand is
how it would be possible to do the same for the remote process?

bug#35055: 27.0.50; async-shell-command truncates output lines

[Michael Albinus]

Juri Linkov <juri@linkov.net> writes:

Hi Juri,

> I didn't know that set-process-window-size changes the
> value of COLUMNS.  I thought that COLUMNS is set only by
> comint-term-environment.

comint-term-environment doesn't set anything. It just returns env values
you could add to process-environmont, as in comint-exec-1 or compilation-start.

> One thing I still don't understand is
> how it would be possible to do the same for the remote process?

Well, don't know (yet), this needs more testing. At least, we could set
$COLUMNS directly:

(let ((default-directory "/ssh::")
      (process-environment (cons "COLUMNS=1024" process-environment)))
  (shell-command-to-string "ps aux"))

(let ((default-directory "/ssh::")
      (process-environment (cons "COLUMNS=1024" process-environment)))
  (shell-command-to-string "ps aux &"))

(let ((default-directory "/ssh::")
      (process-environment (cons "COLUMNS=1024" process-environment)))
  (shell))

Tramp could add the $COLUMNS setting by its own. Maybe, there exist a
more general solution, let's see.

If you add "COLUMNS=1024" to tramp-remote-process-environment, you would
get this effect already w/o any change in Tramp. Maybe I shall document
it somewhere in the Tramp manual.

Best regards, Michael.

bug#35055: 27.0.50; async-shell-command truncates output lines

[Juri Linkov]

>> One thing I still don't understand is how it would be possible
>> to do the same for the remote process?
>
> Well, don't know (yet), this needs more testing. At least, we could set
> $COLUMNS directly:
>
> (let ((default-directory "/ssh::")
>       (process-environment (cons "COLUMNS=1024" process-environment)))
>   (shell-command-to-string "ps aux"))
>
> (let ((default-directory "/ssh::")
>       (process-environment (cons "COLUMNS=1024" process-environment)))
>   (shell-command-to-string "ps aux &"))
>
> (let ((default-directory "/ssh::")
>       (process-environment (cons "COLUMNS=1024" process-environment)))
>   (shell))
>
> Tramp could add the $COLUMNS setting by its own. Maybe, there exist a
> more general solution, let's see.
>
> If you add "COLUMNS=1024" to tramp-remote-process-environment, you would
> get this effect already w/o any change in Tramp. Maybe I shall document
> it somewhere in the Tramp manual.

It seems it's impossible to set COLUMNS to the width of the output window
neither in Tramp nor in async-shell-command.  shell-mode can do this
using set-process-window-size.  But Tramp can't do the same for the
remote process, and for async-shell-command it's too late to do this
because when it displays a window with the output buffer,
the command is already finished.

bug#35055: 27.0.50; async-shell-command truncates output lines

[Michael Albinus]

Juri Linkov <juri@linkov.net> writes:

Hi Juri,

>> If you add "COLUMNS=1024" to tramp-remote-process-environment, you would
>> get this effect already w/o any change in Tramp. Maybe I shall document
>> it somewhere in the Tramp manual.
>
> It seems it's impossible to set COLUMNS to the width of the output window
> neither in Tramp nor in async-shell-command.  shell-mode can do this
> using set-process-window-size.  But Tramp can't do the same for the
> remote process, and for async-shell-command it's too late to do this
> because when it displays a window with the output buffer,
> the command is already finished.

Again, using tramp-remote-process-environment seems to be the
appropriate place.

Best regards, Michael.

bug#35055: 27.0.50; async-shell-command truncates output lines

[Juri Linkov]

[-- Attachment #1: Type: text/plain, Size: 968 bytes --]

>>> If you add "COLUMNS=1024" to tramp-remote-process-environment, you would
>>> get this effect already w/o any change in Tramp. Maybe I shall document
>>> it somewhere in the Tramp manual.
>>
>> It seems it's impossible to set COLUMNS to the width of the output window
>> neither in Tramp nor in async-shell-command.  shell-mode can do this
>> using set-process-window-size.  But Tramp can't do the same for the
>> remote process, and for async-shell-command it's too late to do this
>> because when it displays a window with the output buffer,
>> the command is already finished.
>
> Again, using tramp-remote-process-environment seems to be the
> appropriate place.

Thanks for the suggestion, I customized it to '("COLUMNS=222")
that is approximately equal to my frame's width, thus `ps`
shows enough output, and `dpkg -l '*emacs*'` is not too wide
('dpkg' tries to fill all available columns).

Should a similar option be added for async-shell-command as well?


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: async-shell-command-width.patch --]
[-- Type: text/x-diff, Size: 1430 bytes --]

diff --git a/lisp/simple.el b/lisp/simple.el
index 857e0fc001..7212686add 100644
--- a/lisp/simple.el
+++ b/lisp/simple.el
@@ -3336,6 +3336,15 @@ async-shell-command-display-buffer
   :group 'shell
   :version "26.1")
 
+(defcustom async-shell-command-width nil
+  "Number of columns available for shell command.
+If nil, use the shell default number (usually 80).
+If a positive integer, use a fixed width."
+  :type '(choice (const :tag "Use system limit" nil)
+                 (integer :tag "Fixed width" :value 80))
+  :group 'shell
+  :version "27.1")
+
 (defcustom shell-command-dont-erase-buffer nil
   "If non-nil, output buffer is not erased between shell commands.
 Also, a non-nil value sets the point in the output buffer
@@ -3599,8 +3608,13 @@ shell-command
 		(with-current-buffer buffer
                   (shell-command--save-pos-or-erase)
 		  (setq default-directory directory)
-                  (setq proc
-                        (start-process-shell-command "Shell" buffer command))
+		  (let ((process-environment
+			 (if (natnump async-shell-command-width)
+			     (cons (format "COLUMNS=%d" async-shell-command-width)
+				   process-environment)
+			   process-environment)))
+		    (setq proc
+			  (start-process-shell-command "Shell" buffer command)))
 		  (setq mode-line-process '(":%s"))
 		  (require 'shell) (shell-mode)
                   (set-process-sentinel proc #'shell-command-sentinel)

bug#35055: 27.0.50; async-shell-command truncates output lines

[Michael Albinus]

Juri Linkov <juri@linkov.net> writes:

Hi Juri,

[sorry for the late reply; it was a busy week]

>> Again, using tramp-remote-process-environment seems to be the
>> appropriate place.
>
> Thanks for the suggestion, I customized it to '("COLUMNS=222")
> that is approximately equal to my frame's width, thus `ps`
> shows enough output, and `dpkg -l '*emacs*'` is not too wide
> ('dpkg' tries to fill all available columns).
>
> Should a similar option be added for async-shell-command as well?
>
> diff --git a/lisp/simple.el b/lisp/simple.el
> index 857e0fc001..7212686add 100644
> --- a/lisp/simple.el
> +++ b/lisp/simple.el
> @@ -3336,6 +3336,15 @@ async-shell-command-display-buffer
>    :group 'shell
>    :version "26.1")
>
> +(defcustom async-shell-command-width nil
> +  "Number of columns available for shell command.
> +If nil, use the shell default number (usually 80).
> +If a positive integer, use a fixed width."
> +  :type '(choice (const :tag "Use system limit" nil)
> +                 (integer :tag "Fixed width" :value 80))
> +  :group 'shell
> +  :version "27.1")
> +
>  (defcustom shell-command-dont-erase-buffer nil
>    "If non-nil, output buffer is not erased between shell commands.
>  Also, a non-nil value sets the point in the output buffer
> @@ -3599,8 +3608,13 @@ shell-command
>  		(with-current-buffer buffer
>                    (shell-command--save-pos-or-erase)
>  		  (setq default-directory directory)
> -                  (setq proc
> -                        (start-process-shell-command "Shell" buffer command))
> +		  (let ((process-environment
> +			 (if (natnump async-shell-command-width)
> +			     (cons (format "COLUMNS=%d" async-shell-command-width)
> +				   process-environment)
> +			   process-environment)))
> +		    (setq proc
> +			  (start-process-shell-command "Shell" buffer command)))
>  		  (setq mode-line-process '(":%s"))
>  		  (require 'shell) (shell-mode)
>                    (set-process-sentinel proc #'shell-command-sentinel)

What about calling this just `shell-command-width'? It wouldn't hurt for
synchronous shell commands, and I could adapt
`tramp-remote-process-environment' accordingly if set. Might be more
convenient for users.

Best regards, Michael.

bug#35055: 27.0.50; async-shell-command truncates output lines

[Juri Linkov]

>> +(defcustom async-shell-command-width nil
>> +  "Number of columns available for shell command.
>> +If nil, use the shell default number (usually 80).
>> +If a positive integer, use a fixed width."
>> +  :type '(choice (const :tag "Use system limit" nil)
>> +                 (integer :tag "Fixed width" :value 80))
>> +  :group 'shell
>> +  :version "27.1")
>> +
>
> What about calling this just `shell-command-width'? It wouldn't hurt for
> synchronous shell commands, and I could adapt
> `tramp-remote-process-environment' accordingly if set. Might be more
> convenient for users.

Yes, this is a more general name.  I pushed to master.

BTW, I had a hard time finding the info about one of the most demanding
Tramp features - an ability to edit files with sudo on a remote server.

I searched for "ssh+sudo" by analogy with the git+ssh:// scheme,
but failed to find anything related among all documented methods in
the Tramp manual, even when searched separately for "ssh" and "sudo".
Eventually I found it in an unexpected chapter "Ad-hoc multi-hops"
that allows to do: C-x C-f /ssh:user@host|sudo::/path <RET>

Would it be possible to make this feature more prominent?

bug#35055: 27.0.50; async-shell-command truncates output lines

[Michael Albinus]

Juri Linkov <juri@linkov.net> writes:

Hi Juri,

>> What about calling this just `shell-command-width'? It wouldn't hurt for
>> synchronous shell commands, and I could adapt
>> `tramp-remote-process-environment' accordingly if set. Might be more
>> convenient for users.
>
> Yes, this is a more general name.  I pushed to master.

Thanks. However, I've meant this also for synchronous
`shell-command'. You have implemented this only for asynchronous.

The synchronous `shell-command' does not suffer from the buffer width
restriction. But there might be use cases to set a fixed width, for
example if you want to get a well-defined output layout.

Would you also mind to document the new user option in the Emacs manual?

> BTW, I had a hard time finding the info about one of the most demanding
> Tramp features - an ability to edit files with sudo on a remote server.
>
> I searched for "ssh+sudo" by analogy with the git+ssh:// scheme,
> but failed to find anything related among all documented methods in
> the Tramp manual, even when searched separately for "ssh" and "sudo".
> Eventually I found it in an unexpected chapter "Ad-hoc multi-hops"
> that allows to do: C-x C-f /ssh:user@host|sudo::/path <RET>

Well, this is not a feature on this own. It is just an example how to
apply ad-hoc multi-hops. That's why it is described in that section.

> Would it be possible to make this feature more prominent?

There is an introduction chapter "Quick Start Guide". I've added some
words about ssh+sudo, and friends.

bug#35055: 27.0.50; async-shell-command truncates output lines

[Juri Linkov]

>>> What about calling this just `shell-command-width'? It wouldn't hurt for
>>> synchronous shell commands, and I could adapt
>>> `tramp-remote-process-environment' accordingly if set. Might be more
>>> convenient for users.
>>
>> Yes, this is a more general name.  I pushed to master.
>
> Thanks. However, I've meant this also for synchronous
> `shell-command'. You have implemented this only for asynchronous.
>
> The synchronous `shell-command' does not suffer from the buffer width
> restriction. But there might be use cases to set a fixed width, for
> example if you want to get a well-defined output layout.

Oh, I don't know what to do in case when only the current limit
on the output of asynchronous commands should be increased, but
the output of synchronous commands needs to be left unlimited.
I have no idea how to customize it in this case.

> Would you also mind to document the new user option in the Emacs manual?

Will do as soon as the details of the new option will be clarified.

>> I searched for "ssh+sudo" by analogy with the git+ssh:// scheme,
>> but failed to find anything related among all documented methods in
>> the Tramp manual, even when searched separately for "ssh" and "sudo".
>> Eventually I found it in an unexpected chapter "Ad-hoc multi-hops"
>> that allows to do: C-x C-f /ssh:user@host|sudo::/path <RET>
>
> Well, this is not a feature on this own. It is just an example how to
> apply ad-hoc multi-hops. That's why it is described in that section.
>
>> Would it be possible to make this feature more prominent?
>
> There is an introduction chapter "Quick Start Guide". I've added some
> words about ssh+sudo, and friends.

Thanks, this is the same place where my search found the first occurrence
of "sudo" and there was no info about using it on a remove host
before your addition.

BTW, I see that in info/dir all manual names are capitalized, except Tramp.
This lower-case name looks a little disparaging.

bug#35055: 27.0.50; async-shell-command truncates output lines

[Michael Albinus]

Juri Linkov <juri@linkov.net> writes:

Hi Juri,

>> Thanks. However, I've meant this also for synchronous
>> `shell-command'. You have implemented this only for asynchronous.
>>
>> The synchronous `shell-command' does not suffer from the buffer width
>> restriction. But there might be use cases to set a fixed width, for
>> example if you want to get a well-defined output layout.
>
> Oh, I don't know what to do in case when only the current limit
> on the output of asynchronous commands should be increased, but
> the output of synchronous commands needs to be left unlimited.
> I have no idea how to customize it in this case.

I don't believe there is a conflict. The main use case will be to
increase the output width of a shell command, in order not to loose
information. People will do this by setting a large value, say
1024. This will be used for both the synchronous and asynchronous case.

And then there's the use case to have a fixed output width for special
commands, in order to get a deterministic layout. This won't be applied
globally, neither for synchronous nor for asynchronous
`shell-command'. Rather, `shell-command-width' will be let-bound.

And we have the advantage, that synchronous `shell-command' behaves
consistently, for both the local and remote case.

So I don't see a problem.

> BTW, I see that in info/dir all manual names are capitalized, except Tramp.
> This lower-case name looks a little disparaging.

Ahh, good catch. I've used @value{tramp} for this, as in the whole
tramp.texi. But this cannot work in @direntry, which is supposed to be
applied outside the document. I've changed it accordingly.

Best regards, Michael.

bug#35055: 27.0.50; async-shell-command truncates output lines

[Juri Linkov]

>>> Thanks. However, I've meant this also for synchronous
>>> `shell-command'. You have implemented this only for asynchronous.
>>>
>>> The synchronous `shell-command' does not suffer from the buffer width
>>> restriction. But there might be use cases to set a fixed width, for
>>> example if you want to get a well-defined output layout.
>>
>> Oh, I don't know what to do in case when only the current limit
>> on the output of asynchronous commands should be increased, but
>> the output of synchronous commands needs to be left unlimited.
>> I have no idea how to customize it in this case.
>
> I don't believe there is a conflict. The main use case will be to
> increase the output width of a shell command, in order not to loose
> information. People will do this by setting a large value, say
> 1024. This will be used for both the synchronous and asynchronous case.

The same value will increase the output width of async shell commands,
but decrease the output width of synchronous shell commands from unlimited.

> And then there's the use case to have a fixed output width for special
> commands, in order to get a deterministic layout. This won't be applied
> globally, neither for synchronous nor for asynchronous
> `shell-command'. Rather, `shell-command-width' will be let-bound.
>
> And we have the advantage, that synchronous `shell-command' behaves
> consistently, for both the local and remote case.
>
> So I don't see a problem.

If output truncation will apply to shell-command-on-region,
especially with its REPLACE arg, this would be a big problem.
After filtering the buffer contents with a shell command,
parts of the buffer will be lost.

bug#35055: 27.0.50; async-shell-command truncates output lines

[Michael Albinus]

Juri Linkov <juri@linkov.net> writes:

Hi Juri,

>> I don't believe there is a conflict. The main use case will be to
>> increase the output width of a shell command, in order not to loose
>> information. People will do this by setting a large value, say
>> 1024. This will be used for both the synchronous and asynchronous case.
>
> The same value will increase the output width of async shell commands,
> but decrease the output width of synchronous shell commands from unlimited.

Yes. You must set a proper value, large enough.

>> And then there's the use case to have a fixed output width for special
>> commands, in order to get a deterministic layout. This won't be applied
>> globally, neither for synchronous nor for asynchronous
>> `shell-command'. Rather, `shell-command-width' will be let-bound.
>>
>> And we have the advantage, that synchronous `shell-command' behaves
>> consistently, for both the local and remote case.
>>
>> So I don't see a problem.
>
> If output truncation will apply to shell-command-on-region,
> especially with its REPLACE arg, this would be a big problem.
> After filtering the buffer contents with a shell command,
> parts of the buffer will be lost.

No. You can always undo the effect on buffer.

Best regards, Michael.

bug#35055: 27.0.50; async-shell-command truncates output lines

[Juri Linkov]

>>> I don't believe there is a conflict. The main use case will be to
>>> increase the output width of a shell command, in order not to loose
>>> information. People will do this by setting a large value, say
>>> 1024. This will be used for both the synchronous and asynchronous case.
>>
>> The same value will increase the output width of async shell commands,
>> but decrease the output width of synchronous shell commands from unlimited.
>
> Yes. You must set a proper value, large enough.
>
>>> And then there's the use case to have a fixed output width for special
>>> commands, in order to get a deterministic layout. This won't be applied
>>> globally, neither for synchronous nor for asynchronous
>>> `shell-command'. Rather, `shell-command-width' will be let-bound.
>>>
>>> And we have the advantage, that synchronous `shell-command' behaves
>>> consistently, for both the local and remote case.
>>>
>>> So I don't see a problem.
>>
>> If output truncation will apply to shell-command-on-region,
>> especially with its REPLACE arg, this would be a big problem.
>> After filtering the buffer contents with a shell command,
>> parts of the buffer will be lost.
>
> No. You can always undo the effect on buffer.

When it's customized to a width more than the window's width,
the user won't notice that the output was truncated and data lost.

We shouldn't cause data loss when unsuspecting users will customize this
option to increase the output width of async or remote shell commands.

There is already data loss in the output of async commands that are truncated.
A new option was intended to reduce data loss by allowing less truncation.

OTOH, currently there is no data loss in synchronous shell commands
that often are used to operate on the buffer's contents by using
shell-command-on-region where output is inserted to a file buffer and saved.

We shouldn't allow data loss in synchronous shell commands
at the cost of reducing data loss in async/remove shells.

There is no need for the new option to be consistent across
both synchronous and async/remote operations.

What we could do is to make clear the scope of the new option
in the documentation:

diff --git a/doc/emacs/misc.texi b/doc/emacs/misc.texi
index 7d7065a441..2f1a33e202 100644
--- a/doc/emacs/misc.texi
+++ b/doc/emacs/misc.texi
@@ -775,6 +775,13 @@ Single Shell
 displayed only when the command generates output, set
 @code{async-shell-command-display-buffer} to @code{nil}.
 
+@vindex shell-command-width
+  The option @code{shell-command-width} defines the number of display
+columns available for output of asynchronous or remote shell commands.
+A positive integer tells the shell to use that number of columns for
+command output.  The default value is @code{nil} that means to use
+the same number of columns as provided by the shell.
+
 @kindex M-|
 @findex shell-command-on-region
   @kbd{M-|} (@code{shell-command-on-region}) is like @kbd{M-!}, but
diff --git a/lisp/simple.el b/lisp/simple.el
index 160c433845..30d111a70b 100644
--- a/lisp/simple.el
+++ b/lisp/simple.el
@@ -3352,7 +3352,7 @@ async-shell-command-display-buffer
   :version "26.1")
 
 (defcustom shell-command-width nil
-  "Number of display columns available for asynchronous shell command output.
+  "Number of display columns available for asynchronous/remote shell command.
 If nil, use the shell default number (usually 80 columns).
 If a positive integer, tell the shell to use that number of columns for
 command output."

bug#35055: 27.0.50; async-shell-command truncates output lines

[Michael Albinus]

Juri Linkov <juri@linkov.net> writes:

> There is no need for the new option to be consistent across
> both synchronous and async/remote operations.

Call it `async-shell-command-width', that's it. Nobody has asked for the
synchronous case.

If somebody needs it for a remote `shell-command', there's still the
option to set COLUMNS in `tramp-remote-process-environment' yourself.

Best regards, Michael.

bug#35055: 27.0.50; async-shell-command truncates output lines

[Juri Linkov]

>> There is no need for the new option to be consistent across
>> both synchronous and async/remote operations.
>
> Call it `async-shell-command-width', that's it. Nobody has asked for the
> synchronous case.

Yes, this is a safer option.

> If somebody needs it for a remote `shell-command', there's still the
> option to set COLUMNS in `tramp-remote-process-environment' yourself.

Or maybe better to add another option `tramp-remote-process-width'?

BTW, recently you added to the Tramp manual the example that I proposed:

  /ssh:user@host|sudo::/path/to/file

But I discovered a warning against using "|sudo::" without the hostname:

https://stackoverflow.com/questions/2177687/open-file-via-ssh-and-sudo-with-emacs

  "Important: be sure to specify the hostname explicitly:
   sudo:remotehost: rather than sudo::"

Is this restriction still valid?  Because when I used "|sudo::"
without the hostname, it works fine.

bug#35055: 27.0.50; async-shell-command truncates output lines

[Michael Albinus]

Juri Linkov <juri@linkov.net> writes:

Hi Juri,

>> If somebody needs it for a remote `shell-command', there's still the
>> option to set COLUMNS in `tramp-remote-process-environment' yourself.
>
> Or maybe better to add another option `tramp-remote-process-width'?

No. Tramp is a library, it shouldn't care who's calling it. If I would
start to add package-specific customisation, other packages could
request the same.

And `tramp-remote-process-width' would compete with `shell-command-width'.

> BTW, recently you added to the Tramp manual the example that I proposed:
>
>   /ssh:user@host|sudo::/path/to/file
>
> But I discovered a warning against using "|sudo::" without the hostname:
>
> https://stackoverflow.com/questions/2177687/open-file-via-ssh-and-sudo-with-emacs
>
>   "Important: be sure to specify the hostname explicitly:
>    sudo:remotehost: rather than sudo::"
>
> Is this restriction still valid?  Because when I used "|sudo::"
> without the hostname, it works fine.

Using the host name of the previous hop as default is a new feature in
Tramp 2.4 (Emacs 27). See etc/NEWS:

--8<---------------cut here---------------start------------->8---
*** For some connection methods, like "su" or "sudo", the host name in
ad-hoc multi-hop file names must match the previous hop.  Default host
names are adjusted to the host name from the previous hop.
--8<---------------cut here---------------end--------------->8---

Best regards, Michael.

bug#35055: 27.0.50; async-shell-command truncates output lines

[Michael Albinus]

Juri Linkov <juri@linkov.net> writes:

Hi Juri,

>>> There is no need for the new option to be consistent across
>>> both synchronous and async/remote operations.
>>
>> Call it `async-shell-command-width', that's it. Nobody has asked for the
>> synchronous case.
>
> Yes, this is a safer option.

Do you still plan to use that name? As said, I believe it is the better
one. Nobody has opposed the last ten days.

And, btw, this option still needs to be documented in the Emacs manual.

Best regards, Michael.

bug#35055: 27.0.50; async-shell-command truncates output lines

[Juri Linkov]

[-- Attachment #1: Type: text/plain, Size: 484 bytes --]

>>>> There is no need for the new option to be consistent across
>>>> both synchronous and async/remote operations.
>>>
>>> Call it `async-shell-command-width', that's it. Nobody has asked for the
>>> synchronous case.
>>
>> Yes, this is a safer option.
>
> Do you still plan to use that name? As said, I believe it is the better
> one. Nobody has opposed the last ten days.
>
> And, btw, this option still needs to be documented in the Emacs manual.

Do you agree with this change?


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: async-shell-command-width.patch --]
[-- Type: text/x-diff, Size: 4108 bytes --]

diff --git a/doc/emacs/misc.texi b/doc/emacs/misc.texi
index 7d7065a441..c61e98df89 100644
--- a/doc/emacs/misc.texi
+++ b/doc/emacs/misc.texi
@@ -775,6 +775,13 @@ Single Shell
 displayed only when the command generates output, set
 @code{async-shell-command-display-buffer} to @code{nil}.
 
+@vindex async-shell-command-width
+  The option @code{async-shell-command-width} defines the number of display
+columns available for output of asynchronous or remote shell commands.
+A positive integer tells the shell to use that number of columns for
+command output.  The default value is @code{nil} that means to use
+the same number of columns as provided by the shell.
+
 @kindex M-|
 @findex shell-command-on-region
   @kbd{M-|} (@code{shell-command-on-region}) is like @kbd{M-!}, but
diff --git a/etc/NEWS b/etc/NEWS
index 9e3559d27e..ebbbea7567 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -1103,7 +1103,7 @@ variable for remote shells.  It still defaults to "/bin/sh".
 ** Single shell commands
 
 ---
-*** 'shell-command-width' defines the number of display columns
+*** 'async-shell-command-width' defines the number of display columns
 available for output of asynchronous or remote shell commands.
 
 ** Pcomplete
diff --git a/lisp/net/tramp.el b/lisp/net/tramp.el
index 2e1a0960d7..c0446b79a7 100644
--- a/lisp/net/tramp.el
+++ b/lisp/net/tramp.el
@@ -3639,10 +3639,10 @@ tramp-handle-shell-command
 
     (if (and (not current-buffer-p) (integerp asynchronous))
 	(let ((tramp-remote-process-environment
-	       ;; `shell-command-width' has been introduced with Emacs 27.1.
-	       (if (natnump (bound-and-true-p shell-command-width))
+	       ;; `async-shell-command-width' has been introduced with Emacs 27.1.
+	       (if (natnump (bound-and-true-p async-shell-command-width))
 		   (cons (format "COLUMNS=%d"
-				 (bound-and-true-p shell-command-width))
+				 (bound-and-true-p async-shell-command-width))
 			 tramp-remote-process-environment)
 		 tramp-remote-process-environment)))
 	  (prog1
diff --git a/lisp/simple.el b/lisp/simple.el
index acea1f9ddc..f93882d9bd 100644
--- a/lisp/simple.el
+++ b/lisp/simple.el
@@ -3353,8 +3353,8 @@ async-shell-command-display-buffer
   :group 'shell
   :version "26.1")
 
-(defcustom shell-command-width nil
-  "Number of display columns available for asynchronous shell command output.
+(defcustom async-shell-command-width nil
+  "Number of display columns available for asynchronous/remote shell command.
 If nil, use the shell default number (usually 80 columns).
 If a positive integer, tell the shell to use that number of columns for
 command output."
@@ -3627,8 +3627,8 @@ shell-command
                   (shell-command--save-pos-or-erase)
 		  (setq default-directory directory)
 		  (let ((process-environment
-			 (if (natnump shell-command-width)
-			     (cons (format "COLUMNS=%d" shell-command-width)
+			 (if (natnump async-shell-command-width)
+			     (cons (format "COLUMNS=%d" async-shell-command-width)
 				   process-environment)
 			   process-environment)))
 		    (setq proc
diff --git a/test/lisp/net/tramp-tests.el b/test/lisp/net/tramp-tests.el
index cba697da18..2c82b99691 100644
--- a/test/lisp/net/tramp-tests.el
+++ b/test/lisp/net/tramp-tests.el
@@ -4192,18 +4192,18 @@ tramp--test-shell-command-to-string-asynchronously
 	;; Cleanup.
 	(ignore-errors (delete-file tmp-name)))
 
-      ;; Test `shell-command-width' of `async-shell-command'.
+      ;; Test `async-shell-command-width' of `async-shell-command'.
       ;; Since Emacs 27.1.
-      (when (and (boundp 'shell-command-width)
+      (when (and (boundp 'async-shell-command-width)
 		 (zerop (call-process "tput" nil nil nil "cols"))
                  (zerop (process-file "tput" nil nil nil "cols")))
-	(let (shell-command-width)
+	(let (async-shell-command-width)
 	  (should
 	   (string-equal
 	    (format "%s\n" (car (process-lines "tput" "cols")))
 	    (tramp--test-shell-command-to-string-asynchronously
 	     "tput cols")))
-	  (setq shell-command-width 1024)
+	  (setq async-shell-command-width 1024)
 	  (should
 	   (string-equal
 	    "1024\n"

bug#35055: 27.0.50; async-shell-command truncates output lines

[Michael Albinus]

Juri Linkov <juri@linkov.net> writes:

Hi Juri,

> Do you agree with this change?

Yes, with the minor remarks:

> +  The option @code{async-shell-command-width} defines the number of display
> +columns available for output of asynchronous or remote shell commands.

Do not mention remote.

> +*** 'async-shell-command-width' defines the number of display columns
>  available for output of asynchronous or remote shell commands.

dito.

> +(defcustom async-shell-command-width nil
> +  "Number of display columns available for asynchronous/remote shell command.

dito.

> +      ;; Test `async-shell-command-width' of `async-shell-command'.

`async-shell-command' shouldn't be mentioned anymore, it's obvious.

Best regards, Michael.

bug#35055: 27.0.50; async-shell-command truncates output lines

[Juri Linkov]

[-- Attachment #1: Type: text/plain, Size: 714 bytes --]

>> Do you agree with this change?
>
> Yes, with the minor remarks:
>
>> +  The option @code{async-shell-command-width} defines the number of display
>> +columns available for output of asynchronous or remote shell commands.
>
> Do not mention remote.
>
>> +*** 'async-shell-command-width' defines the number of display columns
>>  available for output of asynchronous or remote shell commands.
>
> dito.
>
>> +(defcustom async-shell-command-width nil
>> +  "Number of display columns available for asynchronous/remote shell command.
>
> dito.
>
>> +      ;; Test `async-shell-command-width' of `async-shell-command'.
>
> `async-shell-command' shouldn't be mentioned anymore, it's obvious.

I hope now it's right:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: async-shell-command-width.2.patch --]
[-- Type: text/x-diff, Size: 4051 bytes --]

diff --git a/doc/emacs/misc.texi b/doc/emacs/misc.texi
index 7d7065a441..5f74392736 100644
--- a/doc/emacs/misc.texi
+++ b/doc/emacs/misc.texi
@@ -775,6 +775,13 @@ Single Shell
 displayed only when the command generates output, set
 @code{async-shell-command-display-buffer} to @code{nil}.
 
+@vindex async-shell-command-width
+  The option @code{async-shell-command-width} defines the number of display
+columns available for output of asynchronous shell commands.
+A positive integer tells the shell to use that number of columns for
+command output.  The default value is @code{nil} that means to use
+the same number of columns as provided by the shell.
+
 @kindex M-|
 @findex shell-command-on-region
   @kbd{M-|} (@code{shell-command-on-region}) is like @kbd{M-!}, but
diff --git a/etc/NEWS b/etc/NEWS
index 9e3559d27e..5e948452d8 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -1103,8 +1103,8 @@ variable for remote shells.  It still defaults to "/bin/sh".
 ** Single shell commands
 
 ---
-*** 'shell-command-width' defines the number of display columns
-available for output of asynchronous or remote shell commands.
+*** 'async-shell-command-width' defines the number of display columns
+available for output of asynchronous shell commands.
 
 ** Pcomplete
 
diff --git a/lisp/net/tramp.el b/lisp/net/tramp.el
index 2e1a0960d7..c0446b79a7 100644
--- a/lisp/net/tramp.el
+++ b/lisp/net/tramp.el
@@ -3639,10 +3639,10 @@ tramp-handle-shell-command
 
     (if (and (not current-buffer-p) (integerp asynchronous))
 	(let ((tramp-remote-process-environment
-	       ;; `shell-command-width' has been introduced with Emacs 27.1.
-	       (if (natnump (bound-and-true-p shell-command-width))
+	       ;; `async-shell-command-width' has been introduced with Emacs 27.1.
+	       (if (natnump (bound-and-true-p async-shell-command-width))
 		   (cons (format "COLUMNS=%d"
-				 (bound-and-true-p shell-command-width))
+				 (bound-and-true-p async-shell-command-width))
 			 tramp-remote-process-environment)
 		 tramp-remote-process-environment)))
 	  (prog1
diff --git a/lisp/simple.el b/lisp/simple.el
index acea1f9ddc..4454791ad2 100644
--- a/lisp/simple.el
+++ b/lisp/simple.el
@@ -3353,7 +3353,7 @@ async-shell-command-display-buffer
   :group 'shell
   :version "26.1")
 
-(defcustom shell-command-width nil
+(defcustom async-shell-command-width nil
   "Number of display columns available for asynchronous shell command output.
 If nil, use the shell default number (usually 80 columns).
 If a positive integer, tell the shell to use that number of columns for
@@ -3627,8 +3627,8 @@ shell-command
                   (shell-command--save-pos-or-erase)
 		  (setq default-directory directory)
 		  (let ((process-environment
-			 (if (natnump shell-command-width)
-			     (cons (format "COLUMNS=%d" shell-command-width)
+			 (if (natnump async-shell-command-width)
+			     (cons (format "COLUMNS=%d" async-shell-command-width)
 				   process-environment)
 			   process-environment)))
 		    (setq proc
diff --git a/test/lisp/net/tramp-tests.el b/test/lisp/net/tramp-tests.el
index cba697da18..7d3c43408d 100644
--- a/test/lisp/net/tramp-tests.el
+++ b/test/lisp/net/tramp-tests.el
@@ -4192,18 +4192,17 @@ tramp--test-shell-command-to-string-asynchronously
 	;; Cleanup.
 	(ignore-errors (delete-file tmp-name)))
 
-      ;; Test `shell-command-width' of `async-shell-command'.
-      ;; Since Emacs 27.1.
-      (when (and (boundp 'shell-command-width)
+      ;; Test `async-shell-command-width'.  Since Emacs 27.1.
+      (when (and (boundp 'async-shell-command-width)
 		 (zerop (call-process "tput" nil nil nil "cols"))
                  (zerop (process-file "tput" nil nil nil "cols")))
-	(let (shell-command-width)
+	(let (async-shell-command-width)
 	  (should
 	   (string-equal
 	    (format "%s\n" (car (process-lines "tput" "cols")))
 	    (tramp--test-shell-command-to-string-asynchronously
 	     "tput cols")))
-	  (setq shell-command-width 1024)
+	  (setq async-shell-command-width 1024)
 	  (should
 	   (string-equal
 	    "1024\n"

bug#35055: 27.0.50; async-shell-command truncates output lines

[Michael Albinus]

Juri Linkov <juri@linkov.net> writes:

Hi Juri,

> I hope now it's right:

Yes, please push.

I will add some few words to tramp.texi what to do in the synchronous case.

Best regards, Michael.

bug#35055: 27.0.50; async-shell-command truncates output lines

[Juri Linkov]

>> I hope now it's right:
>
> Yes, please push.

Pushed to master and closed.

> I will add some few words to tramp.texi what to do in the synchronous case.

I renamed it in tramp.texi as well.

bug#35055: 27.0.50; async-shell-command truncates output lines

[Michael Albinus]

Juri Linkov <juri@linkov.net> writes:

>> Yes, please push.
>
> Pushed to master and closed.
>
>> I will add some few words to tramp.texi what to do in the synchronous case.
>
> I renamed it in tramp.texi as well.

Thanks for both!

Best regards, Michael.