From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED.blaine.gmane.org!not-for-mail From: Daniel Pittman Newsgroups: gmane.emacs.devel Subject: Re: TRAMP VC optimization fails: non-TRAMP filenames handled incorrectly in async operations. Date: Wed, 27 Mar 2019 16:22:58 +0000 Message-ID: References: <87bm1wmhw6.fsf@gmx.de> Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="000000000000f13632058515dbca" Injection-Info: blaine.gmane.org; posting-host="blaine.gmane.org:195.159.176.226"; logging-data="132363"; mail-complaints-to="usenet@blaine.gmane.org" Cc: emacs-devel To: Michael Albinus Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Wed Mar 27 17:24:21 2019 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([209.51.188.17]) by blaine.gmane.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1h9BLe-000YEJ-Bp for ged-emacs-devel@m.gmane.org; Wed, 27 Mar 2019 17:24:18 +0100 Original-Received: from localhost ([127.0.0.1]:50413 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1h9BLc-0000Gv-QX for ged-emacs-devel@m.gmane.org; Wed, 27 Mar 2019 12:24:16 -0400 Original-Received: from eggs.gnu.org ([209.51.188.92]:33339) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1h9BKz-0000Gj-JH for emacs-devel@gnu.org; Wed, 27 Mar 2019 12:23:39 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1h9BKy-0003Eu-02 for emacs-devel@gnu.org; Wed, 27 Mar 2019 12:23:37 -0400 Original-Received: from mail-io1-xd31.google.com ([2607:f8b0:4864:20::d31]:41011) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1h9BKx-0003De-Io for emacs-devel@gnu.org; Wed, 27 Mar 2019 12:23:35 -0400 Original-Received: by mail-io1-xd31.google.com with SMTP id v10so3935108iom.8 for ; Wed, 27 Mar 2019 09:23:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=2FFsN/+44xjVt5uDU35al5b/Zn6AUxzj04uM2nTYyhc=; b=Q9HCZt9L8qHpEBGhUjZ15MH+C+KwEEes6ecbrdkXsZCU8Dczlb6BRKhYeKC5L4ctYE Q79gW/UReXlfU+7KYSwXyopT4S0NmtL3NlGfVTQ/T/1tlqE1uv2VIXCH4/wIlihuaP31 LUGfFrY/qHyY4wql26bHQkYFZytRSQJPk7CfVou9B1dkqhJxeA/zWSXPWr91l/q1oHaC yfsokUeeeFPFAU7nlPh50S7siAW9v2VgaIk7HMgt3Dplrfe1kXBBUm+OVTyCwRn4HolH nd4FfUKH/FTYzqeOZ7IyQhE6cts0xvTIGcakQRVEnqofh3udpcZPllVyc8akUG0+GTWn wHsA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=2FFsN/+44xjVt5uDU35al5b/Zn6AUxzj04uM2nTYyhc=; b=Z2f/qr0M+QIugXi0xWR7NPSyoIXOVSmiStEvm0tFOmLBTaJfQqQoQW6dwpquua++uq IN3IUAbqJLNE/P+vFH89rBmRHte1+eaKno02c07Gf0gXmeVn9nlMO4CLwblWdZlw3ZGq cmFI9S3PBzMD3UC8V8sWFZVT6muAw80WfHs8tv67lEnQP0s0EFbKJdOTaE5GI/hW3HCP HyggVcxHsaVKSNvT0XvWmZtK7QH0Pb9WI6BtdQqK2JCXsC8KzuQE9AKtibkBDqZlFQ/E k1UIz/XreH8Ibzks+iK7ObDKs9e+pUog9jfKw8hF0UHgpd1aQA4o4c5ep0NnfqmCJFVK 2fng== X-Gm-Message-State: APjAAAX8VQ7GdDdqdwypZSS2W3nFCu+L+islTkRep2zUVoHE3MS7aa1R diKsBu+9ewXSinYAMNbwzjg6IjU+v3Qi01V+QFPjwJp6cctrYA== X-Google-Smtp-Source: APXvYqw5Wr4+oNbijOm6Pt01JXaO8x0RLV8WLK+pv+sr8pjZBUH/lQ60aXE17yE1IY8g27vgbr0jMEIWtjqWJzuhZfY= X-Received: by 2002:a6b:7401:: with SMTP id s1mr20757473iog.55.1553703814345; Wed, 27 Mar 2019 09:23:34 -0700 (PDT) In-Reply-To: <87bm1wmhw6.fsf@gmx.de> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::d31 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.org gmane.emacs.devel:234780 Archived-At: --000000000000f13632058515dbca Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Wed, Mar 27, 2019 at 3:55 PM Michael Albinus wrote: > Daniel Pittman writes: > > > I don't have a patch, but I'll see if I can figure out how to improve > > this. I fear the situation is impossible, however, and that this > > attempt to improve the performance of `vc-registered` is doomed to > > failure unless an async protocol is defined to replace the current, > > synchronous, version. > > Thanks for the very thourogh report, much appreciated! I will check in > parallel to you what could be done. > No problem. > There exists a branch in the git repo, "feature/tramp-thread-safe". This > could solve the issue, if "async", as you've said, means Emacs > threads. Tramp uses mutexes, which should protect its > asynchrosity. However, the work on this branch is stalled due to serious > problems I'm not able to solve myself. > > If you mean something else with "async", then we need even another > approach. Sorry, this response turned out long. I'm afraid I haven't the time to condense further, but a summary: threads are not necessary, I believe, and so we should disable this optimization for now, then bring it back if it is provably safe. The long form with analysis and so forth: I believe that threads are not necessary: Emacs can dispatch, for example, any timer, network connection, or external process, callback when it enters the main wait loop. Since tramp is going there while it awaits feedback from the remote system (as far as I understand both modern tramp, and the event loop) then it can enter essentially arbitrary code. So, if it blocked any other file operations that'd be a deadlock if the code =E2=80=93 for example, the buffer and network connection for `(url-ret= rieve " http://example.com" (lambda (&rest uh-oh) ...))` could call into anything, which in the case of, eg, the package.el system fetching an update to the package list, anticipates being able to write the file locally. I'm confident it was possible to have arbitrary callbacks like that trigger while tramp was awaiting remote data a decade or so ago, when I was last working on it. I presume, but can't be certain, that this remains true today, so most of my concrete analysis comes with "I believe, but could be wrong" attached. It doesn't seem to have changed, though. My best guess is that we should disable that optimization for now, and if desirable, reapproach it. One possibly functional strategy, but that I have not considered all possible angles of, might be to fetch the path and then if `(not (tramp-file-name-p ...))` dispatch to the original file name handlers. I think that would absolutely work as long as none of those callbacks interacted with a tramp path at all, and it .... might, but probably wouldn't, if they did. Historically, I attached a tramp operation to copy the generated server(-start) key to a remote tramp path, since I used a TCP listener, ssh forwarding, and that shared secret to allow remote emacsclient to work. That could have triggered at any point after tramp reconnected, as it advised a fairly low level tramp function. I'm not sure how common (or cared about) that sort of nastly hack is, but I'm not confident that more legitimate ways to do the same could be in use in the wild. --000000000000f13632058515dbca Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
On Wed, Mar 27, 2019 at = 3:55 PM Michael Albinus <micha= el.albinus@gmx.de> wrote:
Daniel Pittman <slippycheeze@google.com> writes:

> I don't have a patch, but I'll see if I can figure out how to = improve
> this.=C2=A0 I fear the situation is impossible, however, and that this=
> attempt to improve the performance of `vc-registered` is doomed to
> failure unless an async protocol is defined to replace the current, > synchronous, version.

Thanks for the very thourogh report, much appreciated! I will check in
parallel to you what could be done.

No = problem.
=C2=A0
There exists a branch in the git repo, "feature/tramp-thread-safe"= ;. This
could solve the issue, if "async", as you've said, means Emac= s
threads. Tramp uses mutexes, which should protect its
asynchrosity. However, the work on this branch is stalled due to serious problems I'm not able to solve myself.

If you mean something else with "async", then we need even anothe= r approach.

Sorry, this response turned out= long.=C2=A0 I'm afraid I haven't the time to condense further, but= a summary: threads are not necessary, I believe, and so we should disable = this optimization for now, then bring it back if it is provably safe.
=

The long form with analysis and so forth:
I'm confident it was possible to have arbitrary callb= acks like that trigger while tramp was awaiting remote data a decade or so = ago, when I was last working on it.=C2=A0 I presume, but can't be certa= in, that this remains true today, so most of my concrete analysis comes wit= h "I believe, but could be wrong" attached.=C2=A0 It doesn't = seem to have changed, though.

My best guess is tha= t we should disable that optimization for now, and if desirable, reapproach= it.

One possibly functional strategy, but that I = have not considered all possible angles of, might be to fetch the path and = then if `(not (tramp-file-name-p ...))` dispatch to the original file name = handlers.=C2=A0 I think that would absolutely work as long as none of those= callbacks interacted with a tramp path at all, and it .... might, but prob= ably wouldn't, if they did.=C2=A0=C2=A0

Histor= ically, I attached a tramp operation to copy the generated server(-start) k= ey to a remote tramp path, since I used a TCP listener, ssh forwarding, and= that shared secret to allow remote emacsclient to work.=C2=A0 That could h= ave triggered at any point after tramp reconnected, as it advised a fairly = low level tramp function.=C2=A0 I'm not sure how common (or cared about= ) that sort of nastly hack is, but I'm not confident that more legitima= te ways to do the same could be in use in the wild.


--000000000000f13632058515dbca--