bug#49714: 28.0.50; TRAMP burns CPU and has insufficient user reporting when using xxxx-sk SSH keys

bug#49714: 28.0.50; TRAMP burns CPU and has insufficient user reporting when using xxxx-sk SSH keys

[Dima Kogan]

Hi.

I have a yubikey, a hardware encryption device. For the purposes of SSH,
the SSH private key is split betweek a file on disk (.ssh/id_ed25519_sk)
and some data on the yubikey. When logging into an SSH server, the
interaction generally looks like this:

  client$ ssh ADDRESS

  Enter passphrase for key '/home/dima/.ssh/id_ed25519_sk':
  [the user enters the passphrase; not echoed to the console]

  Confirm user presence for key ED25519-SK XXXXXXXXXXXXXXX
  [the user physically touches the yubikey with their finger]

  server$
  [we are logged-in]

In TRAMP it looks like this:

   C-x C-f
   /ssh:ADDRESS:

   [Emacs asks for the passphrase. This is good]

   [Now we have a problem]

Emacs now spins the CPU, saying nothing. Internally, ssh is asking for
the user to touch the yubikey, but TRAMP is not communicating this
information. I know I should touch the yubikey, and if I do that, TRAMP
succeeds. There are two bugs here:

1. Emacs should not be spinning the CPU. This is actually an old bug. I
   usually see this with normal SSH connections, but it only happens
   during the login sequence, which isn't very long. In THIS case the
   login sequence could be longer. In any case, we shouldn't be
   spinning.

2. Emacs should relay whatever ssh is saying (i.e. "Confirm user
   presence...") to the user

Thanks!





I'm using emacs built from source on 2021-07-16. I'm pretty sure the 

bug#49714: 28.0.50; TRAMP burns CPU and has insufficient user reporting when using xxxx-sk SSH keys

[Michael Albinus]

Dima Kogan <dima@secretsauce.net> writes:

> Hi.

Hi Dima,

> I have a yubikey, a hardware encryption device. For the purposes of SSH,
> the SSH private key is split betweek a file on disk (.ssh/id_ed25519_sk)
> and some data on the yubikey.

Great! I'm planning for a while to implement support of yubikey, but I
never started to buy a device. I herewith declare you as my tester :-)

> When logging into an SSH server, the
> interaction generally looks like this:
>
>   client$ ssh ADDRESS
>
>   Enter passphrase for key '/home/dima/.ssh/id_ed25519_sk':
>   [the user enters the passphrase; not echoed to the console]
>
>   Confirm user presence for key ED25519-SK XXXXXXXXXXXXXXX
>   [the user physically touches the yubikey with their finger]
>
>   server$
>   [we are logged-in]

Good. This should be sufficient information.

> Emacs now spins the CPU, saying nothing. Internally, ssh is asking for
> the user to touch the yubikey, but TRAMP is not communicating this
> information. I know I should touch the yubikey, and if I do that, TRAMP
> succeeds. There are two bugs here:
>
> 1. Emacs should not be spinning the CPU. This is actually an old bug. I
>    usually see this with normal SSH connections, but it only happens
>    during the login sequence, which isn't very long. In THIS case the
>    login sequence could be longer. In any case, we shouldn't be
>    spinning.

Not so simple. Tramp is in a loop, reading the process output, and
checking, whether there is something it has to handle (for example,
password requests). This loop runs until Tramp detects the remote shell
prompt.

> 2. Emacs should relay whatever ssh is saying (i.e. "Confirm user
>    presence...") to the user

I've implemented this for yubikey, pushed to master. Could you pls try?

Showing everything during the login handshake wouldn't be helpful, Tramp
is doing TRT in the background for the vast majority of actions.

> Thanks!

Best reagards, Michael.

bug#49714: 28.0.50; TRAMP burns CPU and has insufficient user reporting when using xxxx-sk SSH keys

[Dima Kogan]

Michael Albinus <michael.albinus@gmx.de> writes:

> Great! I'm planning for a while to implement support of yubikey, but I
> never started to buy a device. I herewith declare you as my tester :-)

OK. Let's get this running!


>> 1. Emacs should not be spinning the CPU. This is actually an old bug. I
>>    usually see this with normal SSH connections, but it only happens
>>    during the login sequence, which isn't very long. In THIS case the
>>    login sequence could be longer. In any case, we shouldn't be
>>    spinning.
>
> Not so simple. Tramp is in a loop, reading the process output, and
> checking, whether there is something it has to handle (for example,
> password requests). This loop runs until Tramp detects the remote shell
> prompt.

There should be a loop, but emacs shouldn't be using all my CPU cycles
while waiting for user interaction. Emacs can select() on the ssh
process file descriptor, and sleep until the ssh process has stuff to
say.


>> 2. Emacs should relay whatever ssh is saying (i.e. "Confirm user
>>    presence...") to the user
>
> I've implemented this for yubikey, pushed to master. Could you pls try?

I just tried it, and there's no difference in behavior from before.
Debugging suggestions?

bug#49714: 28.0.50; TRAMP burns CPU and has insufficient user reporting when using xxxx-sk SSH keys

[Michael Albinus]

Dima Kogan <dima@secretsauce.net> writes:

Hi Dima,

>> I've implemented this for yubikey, pushed to master. Could you pls try?
>
> I just tried it, and there's no difference in behavior from before.
> Debugging suggestions?

I made an error in my first commit, pls try at least 0c77d1d2ca. If this
doesn't help, set tramp-verbose to 10, and send the Tramp debug buffer
after the test.

Best regards, Michael.

bug#49714: 28.0.50; TRAMP burns CPU and has insufficient user reporting when using xxxx-sk SSH keys

[Dima Kogan]

Michael Albinus <michael.albinus@gmx.de> writes:

> set tramp-verbose to 10, and send the Tramp debug buffer after the
> test.

I have a log. It's large. Do I need to sanitize the credentials out of
it? Don't see anything sensitive at a quick glance.


>> There should be a loop, but emacs shouldn't be using all my CPU cycles
>> while waiting for user interaction. Emacs can select() on the ssh
>> process file descriptor, and sleep until the ssh process has stuff to
>> say.
>
> Well, I'm on Lisp level. I just have accept-process-output, and in my
> loop I check whether there is new output. There's no low level API to
> let Emacs sleep for the ssh process file descriptor.

It just sounds unbelievable that emacs can't do blocking reads from the
lisp level. Let me look at (accept-process-output)

bug#49714: 28.0.50; TRAMP burns CPU and has insufficient user reporting when using xxxx-sk SSH keys

[Michael Albinus]

Dima Kogan <dima@secretsauce.net> writes:

>>> There should be a loop, but emacs shouldn't be using all my CPU cycles
>>> while waiting for user interaction. Emacs can select() on the ssh
>>> process file descriptor, and sleep until the ssh process has stuff to
>>> say.
>>
>> Well, I'm on Lisp level. I just have accept-process-output, and in my
>> loop I check whether there is new output. There's no low level API to
>> let Emacs sleep for the ssh process file descriptor.
>
> It just sounds unbelievable that emacs can't do blocking reads from the
> lisp level. Let me look at (accept-process-output)

accept-process-output could block. But it blocks the whole Emacs then,
which isn't what we want.

bug#49714: 28.0.50; TRAMP burns CPU and has insufficient user reporting when using xxxx-sk SSH keys

[Dima Kogan]

Michael Albinus <michael.albinus@gmx.de> writes:

> Dima Kogan <dima@secretsauce.net> writes:
>
>>>> There should be a loop, but emacs shouldn't be using all my CPU cycles
>>>> while waiting for user interaction. Emacs can select() on the ssh
>>>> process file descriptor, and sleep until the ssh process has stuff to
>>>> say.
>>>
>>> Well, I'm on Lisp level. I just have accept-process-output, and in my
>>> loop I check whether there is new output. There's no low level API to
>>> let Emacs sleep for the ssh process file descriptor.
>>
>> It just sounds unbelievable that emacs can't do blocking reads from the
>> lisp level. Let me look at (accept-process-output)
>
> accept-process-output could block. But it blocks the whole Emacs then,
> which isn't what we want.

But emacs is blocked anyway. At least with the code as it is today,
while TRAMP is spinning the cpu waiting for ssh to respond, emacs is not
responsive to any user input. In a perfect world we'd block on the read,
and then go back to the emacs main loop to do other stuff, but that's
hard for all the reasons you know. If we don't go back to the main loop
(as we don't today), then we don't lose anything by a blocking read. You
know much more about the internals than me; is there other work
happening between the checks during our non-blocking read?

Thanks!

bug#49714: 28.0.50; TRAMP burns CPU and has insufficient user reporting when using xxxx-sk SSH keys

[Michael Albinus]

Dima Kogan <dima@secretsauce.net> writes:

Hi Dima,

> But emacs is blocked anyway. At least with the code as it is today,
> while TRAMP is spinning the cpu waiting for ssh to respond, emacs is not
> responsive to any user input. In a perfect world we'd block on the read,
> and then go back to the emacs main loop to do other stuff, but that's
> hard for all the reasons you know.

That requires a Tramp with thread support. I worked on this years ago,
but this is stalled.

> If we don't go back to the main loop
> (as we don't today), then we don't lose anything by a blocking read. You
> know much more about the internals than me; is there other work
> happening between the checks during our non-blocking read?

Other processes could provide their output, process filters, process
sentinels, timers could run, all these asynchronous tasks Emacs runs in
the background.

> Thanks!

Best regards, Michael.

bug#49714: 28.0.50; TRAMP burns CPU and has insufficient user reporting when using xxxx-sk SSH keys

[Michael Albinus]

[Adding debbugs to Cc]

Dima Kogan <dima@secretsauce.net> writes:

Hi Dima,

I've pushed the recent changes to master (slightly modified).

> Really close now. Everything works generally, but I see one issue:
>
> The state of emacs during the "Confirm user presence" request is odd.
> The "Confirm user presence" text does appear at the bottom. But the
> modeline, point and margins are not rendered, and emacs generally isn't
> responsive. Visually it looks like the program crashed, although it
> hasn't: I can C-g or I can touch the yubikey.

Well, while you see the message, Tramp is looping with
accept-process-output in order to check, whether the yubikey has been
pressed. This is not different from everything else Tramp does, until
the remote shell prompt has been detected. So I don't see what we could
do otherwise ...

Btw, there is a timeout of 30 seconds. When you don't press the yubikey
during this time, Tramp shall cancel the authentication process. Perhaps
you could give this also a short test?

Best regards, Michael.

bug#49714: 28.0.50; TRAMP burns CPU and has insufficient user reporting when using xxxx-sk SSH keys

[Dima Kogan]

Michael Albinus <michael.albinus@gmx.de> writes:

> Well, while you see the message, Tramp is looping with
> accept-process-output in order to check, whether the yubikey has been
> pressed. This is not different from everything else Tramp does, until
> the remote shell prompt has been detected. So I don't see what we
> could do otherwise ...

I understand this argument. However, as a user, I would fully expect
emacs to behave the same way while it's waiting for the passphrase as it
does when waiting for me to touch the yubikey. Currently the passphrase
input stage feels normal, while the wait-for-touch stage feels frozen.
On a deeper level, in both cases we're waiting for some specific data to
come from an OS file descriptor: either something ultimately connected
to the keyboard, or the ssh process. Maybe this isn't worth the effort
to fix, but it is a problem.


> Btw, there is a timeout of 30 seconds. When you don't press the
> yubikey during this time, Tramp shall cancel the authentication
> process. Perhaps you could give this also a short test?

Yes. Let me do that later today. The passphrase query itself (from ssh,
unrelated to emacs) has some timeout too, which maybe expires before 30
sec. I will check.

Thanks so much for implementing this!

bug#49714: 28.0.50; TRAMP burns CPU and has insufficient user reporting when using xxxx-sk SSH keys

[Michael Albinus]

Dima Kogan <dima@secretsauce.net> writes:

Hi Dima,

>> Well, while you see the message, Tramp is looping with
>> accept-process-output in order to check, whether the yubikey has been
>> pressed. This is not different from everything else Tramp does, until
>> the remote shell prompt has been detected. So I don't see what we
>> could do otherwise ...
>
> I understand this argument. However, as a user, I would fully expect
> emacs to behave the same way while it's waiting for the passphrase as it
> does when waiting for me to touch the yubikey. Currently the passphrase
> input stage feels normal, while the wait-for-touch stage feels frozen.
> On a deeper level, in both cases we're waiting for some specific data to
> come from an OS file descriptor: either something ultimately connected
> to the keyboard, or the ssh process. Maybe this isn't worth the effort
> to fix, but it is a problem.

I have no idea how annoying this is. I plan to buy a yubikey next days,
this will give me the chance to play with. And maybe to fix it.

>> Btw, there is a timeout of 30 seconds. When you don't press the
>> yubikey during this time, Tramp shall cancel the authentication
>> process. Perhaps you could give this also a short test?
>
> Yes. Let me do that later today. The passphrase query itself (from ssh,
> unrelated to emacs) has some timeout too, which maybe expires before 30
> sec. I will check.

Thanks!

> Thanks so much for implementing this!

Best regards, Michael.

bug#49714: 28.0.50; TRAMP burns CPU and has insufficient user reporting when using xxxx-sk SSH keys

[Dima Kogan]

Michael Albinus <michael.albinus@gmx.de> writes:

> I have no idea how annoying this is. I plan to buy a yubikey next
> days, this will give me the chance to play with. And maybe to fix it.

Honestly, it's probably something that's jarring and wrong-looking the
first few times, but I bet it'll start to feel normal very quickly. If
there's a mental list of nice-to-have improvements somewhere, this
should be on it.

bug#49714: 28.0.50; TRAMP burns CPU and has insufficient user reporting when using xxxx-sk SSH keys

[Michael Albinus]

Dima Kogan <dima@secretsauce.net> writes:

Hi Dima,

>> Well, while you see the message, Tramp is looping with
>> accept-process-output in order to check, whether the yubikey has been
>> pressed. This is not different from everything else Tramp does, until
>> the remote shell prompt has been detected. So I don't see what we
>> could do otherwise ...
>
> I understand this argument. However, as a user, I would fully expect
> emacs to behave the same way while it's waiting for the passphrase as it
> does when waiting for me to touch the yubikey. Currently the passphrase
> input stage feels normal, while the wait-for-touch stage feels frozen.
> On a deeper level, in both cases we're waiting for some specific data to
> come from an OS file descriptor: either something ultimately connected
> to the keyboard, or the ssh process. Maybe this isn't worth the effort
> to fix, but it is a problem.

I couldn't resist, so I've played with my new yubikey. When it shows the
confirmation request

- I cannot reproduce that the mode line does not render. I've added a
  (redisplay) (due to another problem I've experienced locally), maybe
  this makes it better

- Yes, Emacs is kind of frozen while you see the message. Simply, you
  have to touch the yubikey. What other action do you expect from Emacs?
  It is the same behavior as you see when you have given the
  password/passphrase, and Tramp waits for the shell prompt. There's
  nothing to interact with Emacs during this time.

I've just pushed some cleanups to master, could you pls (re-)check? What
else do you expect from Tramp?

Best regards, Michael.

bug#49714: 28.0.50; TRAMP burns CPU and has insufficient user reporting when using xxxx-sk SSH keys

[Dima Kogan]

Hi Michael. Thanks much for testing. Give me a few days to try out the
latest, and I'll post what I find here. I suspect we're done. Thanks
again!

bug#49714: 28.0.50; TRAMP burns CPU and has insufficient user reporting when using xxxx-sk SSH keys

[Michael Albinus]

Dima Kogan <dima@secretsauce.net> writes:

> Hi Michael.

Hi Dima,

> Thanks much for testing. Give me a few days to try out the
> latest, and I'll post what I find here. I suspect we're done. Thanks
> again!

There's no rush, take your time.

Best regards, Michael.

bug#49714: 28.0.50; TRAMP burns CPU and has insufficient user reporting when using xxxx-sk SSH keys

[Dima Kogan]

Hi Michael. I just updated my emacs to a recent-enough build that has
all the fixes: 0b049fe71d7 from 2021/08/06. I see the issue I described
earlier. Specific steps:

1. emacs -Q '/ssh:whatever:'

2. emacs asks for the passphrase. I enter it

3. emacs says "Confirm user presence..." in the minibuffer

I can touch the yubikey, and emacs completes the sequence. While it's
waiting for the yubikey to be touched, emacs is frozen. The toolbar is
rendered, but the modeline and scroll bars are not. The point is not
rendered. If I switch to another window, and switch back, the toolbar is
redrawn, but nothing else is. Most notably, the "Confirm use
presence..." text in the minibuffer is not redrawn.

I can follow slightly different steps:

1. emacs -Q

2. C-x C-f /ssh:whatever:

3. emacs asks for the passphrase. I enter it

If I do this instead, emacs freezes right away, and I don't ever see the
prompt in the minibuffer at all. Again, it's not "frozen", since
touching the yubikey works properly, but the UI indication is missing.

You don't see this?

Thanks!

bug#49714: 28.0.50; TRAMP burns CPU and has insufficient user reporting when using xxxx-sk SSH keys

[Michael Albinus]

Dima Kogan <dima@secretsauce.net> writes:

> Hi Michael.

Hi Dima,

> I just updated my emacs to a recent-enough build that has
> all the fixes: 0b049fe71d7 from 2021/08/06. I see the issue I described
> earlier. Specific steps:

I'm following your tests.

> 1. emacs -Q '/ssh:whatever:'
>
> 2. emacs asks for the passphrase. I enter it
>
> 3. emacs says "Confirm user presence..." in the minibuffer
>
> I can touch the yubikey, and emacs completes the sequence. While it's
> waiting for the yubikey to be touched, emacs is frozen. The toolbar is
> rendered, but the modeline and scroll bars are not. The point is not
> rendered. If I switch to another window, and switch back, the toolbar is
> redrawn, but nothing else is. Most notably, the "Confirm use
> presence..." text in the minibuffer is not redrawn.

But that's expected. Emacs *is* frozen, it waits for you to touch the
yubikey. Alternatively, you can cancel this operation with C-g, or you
can wait 30 seconds for the timeout.

While Tramp is waiting to complete the connection, you cannot do
anything with Emacs. What else do you expect to do?

> I can follow slightly different steps:
>
> 1. emacs -Q
>
> 2. C-x C-f /ssh:whatever:
>
> 3. emacs asks for the passphrase. I enter it
>
> If I do this instead, emacs freezes right away, and I don't ever see the
> prompt in the minibuffer at all. Again, it's not "frozen", since
> touching the yubikey works properly, but the UI indication is missing.

I see the prompt in the minibuffer. It is the message after your file
name /ssh:whatever: , which is placed in square brackets [ and ].

> You don't see this?

I can send you screenshots, if you mind. Honestly, I don't see anything wrong.

> Thanks!

Best regards, Michael.

bug#49714: 28.0.50; TRAMP burns CPU and has insufficient user reporting when using xxxx-sk SSH keys

[Michael Albinus]

Dima Kogan <dima@secretsauce.net> writes:

> Hi.

Hi Dima,

> From what I see, the rendering is dependent on what the window
> manager does. If you switch to another window to cover up emacs, and
> switch back, is there a redraw? I'm observing no redraw, and that's a
> problem: emacs is frozen, and I don't see the prompt text in the
> minibuffer. Sometimes due to WM quirks I never see the prompt. emacs
> should give control back to the main loop while it waits for user
> interaction (i.e. exactly what it does when asking for the passphrase).

It seems to depend on the window manager, indeed. Running "emacs -Q", I
always see the whole rendered Emacs, including the message in the echo
area. Whether I move another window on top on Emacs, and move it away
afterwards, doesn't matter.

I'm running Fedora 34 with gdm (GNOME) 40.1.

> I realize this might not be a simple thing to implement. Would it be
> simple to wait for keyboard input OR the yubikey touch, whichever comes
> first? If so, we can ask the user to "touch yubikey, or press enter to
> quit". Then the same console input code used for the passphrase input
> could be utilized here.

Not so simple to implement. We have the ssh security key message, which
must be displayed.

I've adapted the code slightly, so it doesn't wait 30 sec for the
confirmation message to appear from ssh. Instead there is a loop, which
waits for 0.1 sec, checks for the message, and calls (redisplay) if it
didn't appear.

Pushed to master. Does it make a difference for you?

> Thanks!

Best regards, Michael.

bug#49714: 28.0.50; TRAMP burns CPU and has insufficient user reporting when using xxxx-sk SSH keys

[Michael Albinus]

Michael Albinus <michael.albinus@gmx.de> writes:

Hi Dima,

>> From what I see, the rendering is dependent on what the window
>> manager does. If you switch to another window to cover up emacs, and
>> switch back, is there a redraw? I'm observing no redraw, and that's a
>> problem: emacs is frozen, and I don't see the prompt text in the
>> minibuffer. Sometimes due to WM quirks I never see the prompt. emacs
>> should give control back to the main loop while it waits for user
>> interaction (i.e. exactly what it does when asking for the passphrase).
>
> It seems to depend on the window manager, indeed. Running "emacs -Q", I
> always see the whole rendered Emacs, including the message in the echo
> area. Whether I move another window on top on Emacs, and move it away
> afterwards, doesn't matter.
>
> I'm running Fedora 34 with gdm (GNOME) 40.1.
>
>> I realize this might not be a simple thing to implement. Would it be
>> simple to wait for keyboard input OR the yubikey touch, whichever comes
>> first? If so, we can ask the user to "touch yubikey, or press enter to
>> quit". Then the same console input code used for the passphrase input
>> could be utilized here.
>
> Not so simple to implement. We have the ssh security key message, which
> must be displayed.
>
> I've adapted the code slightly, so it doesn't wait 30 sec for the
> confirmation message to appear from ssh. Instead there is a loop, which
> waits for 0.1 sec, checks for the message, and calls (redisplay) if it
> didn't appear.
>
> Pushed to master. Does it make a difference for you?

Is there anything else I could do for you? Otherwise, I'd like to close
the bug report.

>> Thanks!

Best regards, Michael.

bug#49714: 28.0.50; TRAMP burns CPU and has insufficient user reporting when using xxxx-sk SSH keys

[Dima Kogan]

Hi Michael. Sorry I haven't replied to this. I think the current code is
good, and I'm closing the bug. If anything else pops up in this area, we
can use a new ticket. Thank you very much!

bug#49714: 28.0.50; TRAMP burns CPU and has insufficient user reporting when using xxxx-sk SSH keys

[Michael Albinus]

Dima Kogan <dima@secretsauce.net> writes:

Hi Dima,

>>> 1. Emacs should not be spinning the CPU. This is actually an old bug. I
>>>    usually see this with normal SSH connections, but it only happens
>>>    during the login sequence, which isn't very long. In THIS case the
>>>    login sequence could be longer. In any case, we shouldn't be
>>>    spinning.
>>
>> Not so simple. Tramp is in a loop, reading the process output, and
>> checking, whether there is something it has to handle (for example,
>> password requests). This loop runs until Tramp detects the remote shell
>> prompt.
>
> There should be a loop, but emacs shouldn't be using all my CPU cycles
> while waiting for user interaction. Emacs can select() on the ssh
> process file descriptor, and sleep until the ssh process has stuff to
> say.

Well, I'm on Lisp level. I just have accept-process-output, and in my
loop I check whether there is new output. There's no low level API to
let Emacs sleep for the ssh process file descriptor.