bug#33135: 27.0.50; (make-thread) TRAMP interactive freeze macOS 10.14

bug#33135: 27.0.50; (make-thread) TRAMP interactive freeze macOS 10.14

[Christian Johansson]

Hi!

I am able to reproduce this on two different computers using macOS 10.14 
but I haven't tested this on any other system.

Steps to reproduce:
1. Open Emacs without any configuration: src/emacs -Q
2. Make sure you are in *scratch* buffer
3. Paste following Emacs Lisp code:

;; FTP URL: ftp.dlptest.com or ftp://ftp.dlptest.com/
;; FTP User: dlpuser@dlptest.com
;; Password: e73jzTRTNqCN9PYAAjjn
(make-thread (lambda()
                (message "Listing directory files..")
                (let ((files (directory-files 
"/ftp:dlpuser@dlptest.com@ftp.dlptest.com:/")))
                  (message "Opened directory")
                  (message "Directory files: %s" files))))

4. Run M-x eval-buffer
5. Emacs asks for FTP password, paste in mini-buffer: e73jzTRTNqCN9PYAAjjn
6. Emacs now hangs, asks for password again and you can't stop it using 
C-g. You get error in terminal:

Christians-Air:emacs christianjohansson$ src/emacs -Q
2018-10-24 12:07:58.383 emacs[627:14815] Failed to initialize color list 
unarchiver: Error Domain=NSCocoaErrorDomain Code=4864 "*** 
-[NSKeyedUnarchiver 
_initForReadingFromData:error:throwLegacyExceptions:]: non-keyed archive 
cannot be decoded by NSKeyedUnarchiver" UserInfo={NSDebugDescription=*** 
-[NSKeyedUnarchiver 
_initForReadingFromData:error:throwLegacyExceptions:]: non-keyed archive 
cannot be decoded by NSKeyedUnarchiver}
2018-10-24 12:08:28.515 emacs[627:15311] WARNING: NSWindow drag regions 
should only be invalidated on the Main Thread! This will throw an 
exception in the future. Called from (
     0   AppKit                              0x00007fff334132e3 
-[NSWindow(NSWindow_Theme) 
_postWindowNeedsToResetDragMarginsUnlessPostingDisabled] + 386
     1   AppKit                              0x00007fff33435422 
-[NSThemeFrame _tileTitlebarAndRedisplay:] + 98
     2   AppKit                              0x00007fff334604dd 
-[NSTitledFrame _titleDidChange] + 217
     3   AppKit                              0x00007fff3345ff89 
-[NSTitledFrame setTitle:] + 730
     4   AppKit                              0x00007fff3345fc00 
-[NSThemeFrame setTitle:] + 50
     5   AppKit                              0x00007fff33423082 
-[NSWindow _dosetTitle:andDefeatWrap:] + 211
     6   emacs                               0x00000001001e7371 
ns_set_name_internal + 161
     7   emacs                               0x000000010004f640 
x_consider_frame_title + 784
     8   emacs                               0x0000000100028ad5 
redisplay_internal + 2949
     9   emacs                               0x00000001000d1e71 
read_char + 2097
     10  emacs                               0x00000001000cfc1d 
read_key_sequence + 1997
     11  emacs                               0x00000001000ce30c 
command_loop_1 + 1228
     12  emacs                               0x000000010014d103 
internal_condition_case + 259
     13  emacs                               0x00000001000dd400 
command_loop_2 + 48
     14  emacs                               0x000000010014c727 
internal_catch + 263
     15  emacs                               0x00000001000cd338 
command_loop + 200
     16  emacs                               0x00000001000cd220 
recursive_edit_1 + 112
     17  emacs                               0x00000001000fbc33 
read_minibuf + 2179
     18  emacs                               0x00000001000fc0ca 
Fread_string + 250
     19  emacs                               0x000000010014f831 
funcall_subr + 353
     20  emacs                               0x000000010014ebc8 Ffuncall 
+ 792
     21  emacs                               0x000000010018ebf6 
exec_byte_code + 1814
     22  emacs                               0x000000010014eb2a Ffuncall 
+ 634
     23  emacs                               0x000000010018ebf6 
exec_byte_code + 1814
     24  emacs                               0x000000010014eb2a Ffuncall 
+ 634
     25  emacs                               0x000000010018ebf6 
exec_byte_code + 1814
     26  emacs                               0x000000010014eb2a Ffuncall 
+ 634
     27  emacs                               0x000000010018ebf6 
exec_byte_code + 1814
     28  emacs                               0x000000010014eb2a Ffuncall 
+ 634
     29  emacs                               0x000000010018ebf6 
exec_byte_code + 1814
     30  emacs                               0x000000010014eb2a Ffuncall 
+ 634
     31  emacs                               0x000000010018ebf6 
exec_byte_code + 1814
     32  emacs                               0x000000010014eb2a Ffuncall 
+ 634
     33  emacs                               0x000000010018ebf6 
exec_byte_code + 1814
     34  emacs                               0x000000010014eb2a Ffuncall 
+ 634
     35  emacs                               0x000000010018ebf6 
exec_byte_code + 1814
     36  emacs                               0x000000010014eb2a Ffuncall 
+ 634
     37  emacs                               0x000000010014ebc8 Ffuncall 
+ 792
     38  emacs                               0x000000010018ebf6 
exec_byte_code + 1814
     39  emacs                               0x000000010014eb2a Ffuncall 
+ 634
     40  emacs                               0x000000010014ebc8 Ffuncall 
+ 792
     41  emacs                               0x000000010018ebf6 
exec_byte_code + 1814
     42  emacs                               0x000000010014eb2a Ffuncall 
+ 634
     43  emacs                               0x000000010014ebc8 Ffuncall 
+ 792
     44  emacs                               0x000000010018ebf6 
exec_byte_code + 1814
     45  emacs                               0x000000010014eb2a Ffuncall 
+ 634
     46  emacs                               0x000000010014b270 call2 + 48
     47  emacs                               0x000000010014f7e7 
funcall_subr + 279
     48  emacs                               0x000000010014ebc8 Ffuncall 
+ 792
     49  emacs                               0x000000010018ebf6 
exec_byte_code + 1814
     50  emacs                               0x000000010014eb2a Ffuncall 
+ 634
     51  emacs                               0x000000010018ebf6 
exec_byte_code + 1814
     52  emacs                               0x000000010014eb2a Ffuncall 
+ 634
     53  emacs                               0x000000010014e740 Fapply + 512
     54  emacs                               0x000000010014ebc8 Ffuncall 
+ 792
     55  emacs                               0x000000010018ebf6 
exec_byte_code + 1814
     56  emacs                               0x000000010014eb2a Ffuncall 
+ 634
     57  emacs                               0x000000010014e740 Fapply + 512
     58  emacs                               0x000000010014ebc8 Ffuncall 
+ 792
     59  emacs                               0x000000010018ebf6 
exec_byte_code + 1814
     60  emacs                               0x000000010014eb2a Ffuncall 
+ 634
     61  emacs                               0x000000010014e740 Fapply + 512
     62  emacs                               0x000000010014ebc8 Ffuncall 
+ 792
     63  emacs                               0x000000010018ebf6 
exec_byte_code + 1814
     64  emacs                               0x000000010014eb2a Ffuncall 
+ 634
     65  emacs                               0x000000010014f53c call5 + 60
     66  emacs                               0x000000010014a921 eval_sub 
+ 2049
     67  emacs                               0x000000010014bc8f Flet + 463
     68  emacs                               0x000000010014a723 eval_sub 
+ 1539
     69  emacs                               0x000000010014fc1d 
funcall_lambda + 813
     70  emacs                               0x000000010014eb2a Ffuncall 
+ 634
     71  emacs                               0x00000001001b69c5 
invoke_thread_function + 53
     72  emacs                               0x000000010014d103 
internal_condition_case + 259
     73  emacs                               0x00000001001b624a 
run_thread + 138
     74  libsystem_pthread.dylib             0x00007fff6316d33d 
_pthread_body + 126
     75  libsystem_pthread.dylib             0x00007fff631702a7 
_pthread_start + 70
     76  libsystem_pthread.dylib             0x00007fff6316c425 
thread_start + 13
)




In GNU Emacs 27.0.50 (build 3, x86_64-apple-darwin17.6.0, NS 
appkit-1561.40 Version 10.13.5 (Build 17F77))
  of 2018-06-30 built on Christians-MacBook-Air.local
Repository revision: ed65ea18152636500399a7b6b75c87bac7d4ef2b
Windowing system distributor 'Apple', version 10.3.1671
System Description:  Mac OS X 10.14

Recent messages:
Source file 
‘/Users/christianjohansson/Documents/emacs/lisp/mail/sendmail.el’ newer 
than byte-compiled file
Source file 
‘/Users/christianjohansson/Documents/emacs/lisp/gnus/message.el’ newer 
than byte-compiled file
Source file 
‘/Users/christianjohansson/Documents/emacs/lisp/mail/mailabbrev.el’ 
newer than byte-compiled file
Source file 
‘/Users/christianjohansson/Documents/emacs/lisp/gnus/gnus-util.el’ newer 
than byte-compiled file
Source file 
‘/Users/christianjohansson/Documents/emacs/lisp/mail/rmail.el’ newer 
than byte-compiled file
Source file ‘/Users/christianjohansson/Documents/emacs/lisp/epg.el’ 
newer than byte-compiled file
Source file 
‘/Users/christianjohansson/Documents/emacs/lisp/epg-config.el’ newer 
than byte-compiled file
Source file ‘/Users/christianjohansson/Documents/emacs/lisp/dired.el’ 
newer than byte-compiled file
Source file 
‘/Users/christianjohansson/Documents/emacs/lisp/emacs-lisp/bytecomp.el’ 
newer than byte-compiled file
Source file 
‘/Users/christianjohansson/Documents/emacs/lisp/emacs-lisp/byte-opt.el’ 
newer than byte-compiled file

Configured features:
RSVG IMAGEMAGICK DBUS NOTIFY ACL GNUTLS LIBXML2 ZLIB TOOLKIT_SCROLL_BARS
NS THREADS

Important settings:
   value of $LC_CTYPE: UTF-8
   value of $LANG: en_SE.UTF-8
   locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
   tooltip-mode: t
   global-eldoc-mode: t
   eldoc-mode: t
   electric-indent-mode: t
   mouse-wheel-mode: t
   tool-bar-mode: t
   menu-bar-mode: t
   file-name-shadow-mode: t
   global-font-lock-mode: t
   font-lock-mode: t
   blink-cursor-mode: t
   auto-composition-mode: t
   auto-encryption-mode: t
   auto-compression-mode: t
   line-number-mode: t
   transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny seq byte-opt gv
bytecomp byte-compile cconv dired dired-loaddefs format-spec rfc822 mml
mml-sec password-cache epa derived epg epg-config gnus-util rmail
rmail-loaddefs mm-decode mm-bodies mm-encode mail-parse rfc2231
mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums
mm-util mail-prsvr mail-utils help-mode easymenu cl-loaddefs cl-lib
apropos elec-pair time-date tooltip eldoc electric uniquify ediff-hook
vc-hooks lisp-float-type mwheel term/ns-win ns-win ucs-normalize
mule-util term/common-win tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode elisp-mode lisp-mode
prog-mode register page menu-bar rfn-eshadow isearch timer select
scroll-bar mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame cl-generic cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932
hebrew greek romanian slovak czech european ethiopic indian cyrillic
chinese composite charscript charprop case-table epa-hook jka-cmpr-hook
help simple abbrev obarray minibuffer cl-preloaded nadvice loaddefs
button faces cus-face macroexp files text-properties overlay sha1 md5
base64 format env code-pages mule custom widget hashtable-print-readable
backquote dbusbind kqueue cocoa ns multi-tty make-network-process emacs)

Memory information:
((conses 16 206501 10044)
  (symbols 48 20198 2)
  (miscs 40 35 169)
  (strings 32 29396 1914)
  (string-bytes 1 776758)
  (vectors 16 35414)
  (vector-slots 8 727972 11800)
  (floats 8 48 68)
  (intervals 56 372 0)
  (buffers 992 13))

bug#33135: 27.0.50; (make-thread) TRAMP interactive freeze macOS 10.14

[Alan Third]

On Wed, Oct 24, 2018 at 12:15:22PM +0200, Christian Johansson wrote:
> Hi!
> 
> I am able to reproduce this on two different computers using macOS 10.14 but
> I haven't tested this on any other system.
> 
> Steps to reproduce:
> 1. Open Emacs without any configuration: src/emacs -Q
> 2. Make sure you are in *scratch* buffer
> 3. Paste following Emacs Lisp code:
> 
> ;; FTP URL: ftp.dlptest.com or ftp://ftp.dlptest.com/
> ;; FTP User: dlpuser@dlptest.com
> ;; Password: e73jzTRTNqCN9PYAAjjn
> (make-thread (lambda()
>                (message "Listing directory files..")
>                (let ((files (directory-files
> "/ftp:dlpuser@dlptest.com@ftp.dlptest.com:/")))
>                  (message "Opened directory")
>                  (message "Directory files: %s" files))))
> 
> 4. Run M-x eval-buffer
> 5. Emacs asks for FTP password, paste in mini-buffer: e73jzTRTNqCN9PYAAjjn
> 6. Emacs now hangs, asks for password again and you can't stop it using C-g.
> You get error in terminal:

The NS port currently doesn’t handle interactive functions from
threads very well. I suspect the problem is being caused by displaying
a message from the background thread.

> Christians-Air:emacs christianjohansson$ src/emacs -Q
> 2018-10-24 12:07:58.383 emacs[627:14815] Failed to initialize color list
> unarchiver: Error Domain=NSCocoaErrorDomain Code=4864 "***
> -[NSKeyedUnarchiver _initForReadingFromData:error:throwLegacyExceptions:]:
> non-keyed archive cannot be decoded by NSKeyedUnarchiver"
> UserInfo={NSDebugDescription=*** -[NSKeyedUnarchiver
> _initForReadingFromData:error:throwLegacyExceptions:]: non-keyed archive
> cannot be decoded by NSKeyedUnarchiver}

This error will be fixed in Emacs 26.2.

> 2018-10-24 12:08:28.515 emacs[627:15311] WARNING: NSWindow drag regions
> should only be invalidated on the Main Thread! This will throw an exception
> in the future. Called from (
>     0   AppKit                              0x00007fff334132e3
> -[NSWindow(NSWindow_Theme)
> _postWindowNeedsToResetDragMarginsUnlessPostingDisabled] + 386
>     1   AppKit                              0x00007fff33435422
> -[NSThemeFrame _tileTitlebarAndRedisplay:] + 98
>     2   AppKit                              0x00007fff334604dd
> -[NSTitledFrame _titleDidChange] + 217
>     3   AppKit                              0x00007fff3345ff89
> -[NSTitledFrame setTitle:] + 730
>     4   AppKit                              0x00007fff3345fc00
> -[NSThemeFrame setTitle:] + 50
>     5   AppKit                              0x00007fff33423082 -[NSWindow
> _dosetTitle:andDefeatWrap:] + 211
>     6   emacs                               0x00000001001e7371
> ns_set_name_internal + 161
>     7   emacs                               0x000000010004f640
> x_consider_frame_title + 784
>     8   emacs                               0x0000000100028ad5
> redisplay_internal + 2949
<snip>
>     73  emacs                               0x00000001001b624a run_thread +
> 138
>     74  libsystem_pthread.dylib             0x00007fff6316d33d _pthread_body
> + 126
>     75  libsystem_pthread.dylib             0x00007fff631702a7
> _pthread_start + 70
>     76  libsystem_pthread.dylib             0x00007fff6316c425 thread_start
> + 13
> )

Yes, we can see redisplay is being called from within the thread, and
that just won’t work currently, I’m afraid.
-- 
Alan Third

bug#33135: 27.0.50; (make-thread) TRAMP interactive freeze macOS 10.14

[Michael Albinus]

Alan Third <alan@idiocy.org> writes:

>> Hi!

Hi,

>> Steps to reproduce:
>> 1. Open Emacs without any configuration: src/emacs -Q
>> 2. Make sure you are in *scratch* buffer
>> 3. Paste following Emacs Lisp code:
>> 
>> ;; FTP URL: ftp.dlptest.com or ftp://ftp.dlptest.com/
>> ;; FTP User: dlpuser@dlptest.com
>> ;; Password: e73jzTRTNqCN9PYAAjjn
>> (make-thread (lambda()
>>                (message "Listing directory files..")
>>                (let ((files (directory-files
>> "/ftp:dlpuser@dlptest.com@ftp.dlptest.com:/")))
>>                  (message "Opened directory")
>>                  (message "Directory files: %s" files))))
>> 
>> 4. Run M-x eval-buffer
>> 5. Emacs asks for FTP password, paste in mini-buffer: e73jzTRTNqCN9PYAAjjn
>> 6. Emacs now hangs, asks for password again and you can't stop it using C-g.
>> You get error in terminal:
>
> The NS port currently doesn’t handle interactive functions from
> threads very well. I suspect the problem is being caused by displaying
> a message from the background thread.

It's not only the NS port. There is a git branch
feature/tramp-thread-safe dedicated to make Tramp thread-safe. However,
this doesn't work well yet, see for example bug#25214 / bug#32426.

Furthermore, the "ftp" method navigates from Tramp to Ange-FTP. I
haven't tested this for being thread-safe; given that nobody has worked
on this I would expect even more problems.

Anyway, if you intend to continue to test this functionality, I
recommend you to use the said branch. There are changes which make file
visiting commands async aware, via a prefix command.  "C-x & C-x C-f ..."
is supposed to visit files asynchronously. This works for both local
files and remote files; but for local files you'll see the difference
for large files only.

Best regards, Michael.

bug#33135: 27.0.50; (make-thread) TRAMP interactive freeze macOS 10.14

[Christian Johansson]

Hi Michael

Alright, I'll check out that branch and continue testing here. 
Asynchronous TRAMP via FTP/SSH without user-interaction seems to work on 
master branch most of the time but I have experienced some occasional 
crashes that you don't get when running synchronously. Will start Emacs 
from my terminal to see if I get any errors there if it crashes

Regards
Christian

On 2018-10-24 13:12, Michael Albinus wrote:
> Alan Third <alan@idiocy.org> writes:
>
>>> Hi!
> Hi,
>
>>> Steps to reproduce:
>>> 1. Open Emacs without any configuration: src/emacs -Q
>>> 2. Make sure you are in *scratch* buffer
>>> 3. Paste following Emacs Lisp code:
>>>
>>> ;; FTP URL: ftp.dlptest.com or ftp://ftp.dlptest.com/
>>> ;; FTP User: dlpuser@dlptest.com
>>> ;; Password: e73jzTRTNqCN9PYAAjjn
>>> (make-thread (lambda()
>>>                 (message "Listing directory files..")
>>>                 (let ((files (directory-files
>>> "/ftp:dlpuser@dlptest.com@ftp.dlptest.com:/")))
>>>                   (message "Opened directory")
>>>                   (message "Directory files: %s" files))))
>>>
>>> 4. Run M-x eval-buffer
>>> 5. Emacs asks for FTP password, paste in mini-buffer: e73jzTRTNqCN9PYAAjjn
>>> 6. Emacs now hangs, asks for password again and you can't stop it using C-g.
>>> You get error in terminal:
>> The NS port currently doesn’t handle interactive functions from
>> threads very well. I suspect the problem is being caused by displaying
>> a message from the background thread.
> It's not only the NS port. There is a git branch
> feature/tramp-thread-safe dedicated to make Tramp thread-safe. However,
> this doesn't work well yet, see for example bug#25214 / bug#32426.
>
> Furthermore, the "ftp" method navigates from Tramp to Ange-FTP. I
> haven't tested this for being thread-safe; given that nobody has worked
> on this I would expect even more problems.
>
> Anyway, if you intend to continue to test this functionality, I
> recommend you to use the said branch. There are changes which make file
> visiting commands async aware, via a prefix command.  "C-x & C-x C-f ..."
> is supposed to visit files asynchronously. This works for both local
> files and remote files; but for local files you'll see the difference
> for large files only.
>
> Best regards, Michael.

bug#33135: 27.0.50; (make-thread) TRAMP interactive freeze macOS 10.14

[Michael Albinus]

Christian Johansson <christian@cvj.se> writes:

> Hi Michael

Hi Christian,

> Alright, I'll check out that branch and continue testing
> here. Asynchronous TRAMP via FTP/SSH without user-interaction seems to
> work on master branch most of the time

The simple case: yes. It will be more interesting if you visit several
(many!) files in parallel. My usual test case is "C-x & C-x C-f
/ssh::~/src/emacs-test/admin/*.el" ("~/src/emacs" is the location of the
Emacs repository on my machine).

You might also open a second frame in parallel, and run "M-x list-threads"
there. This will show you how threads evolve.

If you want to know more about asynchronous file visiting, you might
consult (info "(emacs) Visiting") in the feature/tramp-thread-safe branch.

> but I have experienced some occasional crashes that you don't get when
> running synchronously. Will start Emacs from my terminal to see if I
> get any errors there if it crashes

Yes, pls report.

> Regards
> Christian

Best regards, Michael.

bug#33135: 27.0.50; (make-thread) TRAMP interactive freeze macOS 10.14

[Christian Johansson]

Hi Michael

I haven't experienced a single crash since I started using your threaded 
tramp branch but I sometimes get errors like this, but it doesn't make 
Emacs crash, I can just retry same operation and it will work. It seems 
like sometimes a thread is busy and Emacs tries to access it in some way 
and this error occurs, I have replaced sensitive information with --- below

Debugger entered--Lisp error: (error "Attempt to accept output from 
process *ftp ---...")
   accept-process-output(#<process *ftp ---@---*>)
   ange-ftp-wait-not-busy(#<process *ftp ---@---*>)
   ange-ftp-raw-send-cmd(#<process *ftp ---@---*> "type ascii" nil 
(#f(compiled-function (result line host user cmd msg cont nowait) 
#<bytecode 0x4ffee879>) "---" "---" "type ascii" nil nil nil) nil)

I haven't tried threaded tramp that requires user-interaction yet though 
since I got ~/.authinfo.gpg working with ange-ftp, I'll see if I can 
test that this week

Regards
Christian

On 2018-10-25 10:35, Michael Albinus wrote:
> Christian Johansson <christian@cvj.se> writes:
>
>> Hi Michael
> Hi Christian,
>
>> Alright, I'll check out that branch and continue testing
>> here. Asynchronous TRAMP via FTP/SSH without user-interaction seems to
>> work on master branch most of the time
> The simple case: yes. It will be more interesting if you visit several
> (many!) files in parallel. My usual test case is "C-x & C-x C-f
> /ssh::~/src/emacs-test/admin/*.el" ("~/src/emacs" is the location of the
> Emacs repository on my machine).
>
> You might also open a second frame in parallel, and run "M-x list-threads"
> there. This will show you how threads evolve.
>
> If you want to know more about asynchronous file visiting, you might
> consult (info "(emacs) Visiting") in the feature/tramp-thread-safe branch.
>
>> but I have experienced some occasional crashes that you don't get when
>> running synchronously. Will start Emacs from my terminal to see if I
>> get any errors there if it crashes
> Yes, pls report.
>
>> Regards
>> Christian
> Best regards, Michael.

bug#33135: 27.0.50; (make-thread) TRAMP interactive freeze macOS 10.14

[Michael Albinus]

Christian Johansson <christian@cvj.se> writes:

> Hi Michael

Hi Christian,

> I haven't experienced a single crash since I started using your
> threaded tramp branch but I sometimes get errors like this, but it
> doesn't make Emacs crash, I can just retry same operation and it will
> work. It seems like sometimes a thread is busy and Emacs tries to
> access it in some way and this error occurs, I have replaced sensitive
> information with --- below
>
> Debugger entered--Lisp error: (error "Attempt to accept output from
> process *ftp ---...")
>   accept-process-output(#<process *ftp ---@---*>)
>   ange-ftp-wait-not-busy(#<process *ftp ---@---*>)
>   ange-ftp-raw-send-cmd(#<process *ftp ---@---*> "type ascii" nil
> (#f(compiled-function (result line host user cmd msg cont nowait)
> #<bytecode 0x4ffee879>) "---" "---" "type ascii" nil nil nil) nil)

Looks like the ange-ftp process is not assigned to the proper thread. As
said, ange-ftp is a special case, it wasn't handled yet.

I've pushed a fix to the feature/tramp-thread-safe branch; should work
now. Could you pls retest?

> I haven't tried threaded tramp that requires user-interaction yet
> though since I got ~/.authinfo.gpg working with ange-ftp, I'll see if
> I can test that this week

Likely, this won't work. Nobody has provided a fix for this problem, yet.

> Regards
> Christian

Best regards, Michael.

bug#33135: 27.0.50; (make-thread) TRAMP interactive freeze macOS 10.14

[Christian Johansson]

Alright, I’ll see if I can get the error again. I only got it like 2 times in 8 hours so it’s not frequent.

I tried user-interaction via threaded ange-ftp  tramp on macOS and it seems to work on your branch. I will try it a bit more this week.

Regards
Christian

> 5 nov. 2018 kl. 14:49 skrev Michael Albinus <michael.albinus@gmx.de>:
> 
> Christian Johansson <christian@cvj.se> writes:
> 
>> Hi Michael
> 
> Hi Christian,
> 
>> I haven't experienced a single crash since I started using your
>> threaded tramp branch but I sometimes get errors like this, but it
>> doesn't make Emacs crash, I can just retry same operation and it will
>> work. It seems like sometimes a thread is busy and Emacs tries to
>> access it in some way and this error occurs, I have replaced sensitive
>> information with --- below
>> 
>> Debugger entered--Lisp error: (error "Attempt to accept output from
>> process *ftp ---...")
>>   accept-process-output(#<process *ftp ---@---*>)
>>   ange-ftp-wait-not-busy(#<process *ftp ---@---*>)
>>   ange-ftp-raw-send-cmd(#<process *ftp ---@---*> "type ascii" nil
>> (#f(compiled-function (result line host user cmd msg cont nowait)
>> #<bytecode 0x4ffee879>) "---" "---" "type ascii" nil nil nil) nil)
> 
> Looks like the ange-ftp process is not assigned to the proper thread. As
> said, ange-ftp is a special case, it wasn't handled yet.
> 
> I've pushed a fix to the feature/tramp-thread-safe branch; should work
> now. Could you pls retest?
> 
>> I haven't tried threaded tramp that requires user-interaction yet
>> though since I got ~/.authinfo.gpg working with ange-ftp, I'll see if
>> I can test that this week
> 
> Likely, this won't work. Nobody has provided a fix for this problem, yet.
> 
>> Regards
>> Christian
> 
> Best regards, Michael.

bug#33135: 27.0.50; (make-thread) TRAMP interactive freeze macOS 10.14

[Christian Johansson]

[-- Attachment #1: Type: text/plain, Size: 3332 bytes --]

I was wrong, interactive threaded tramp over ange-ftp on macos doesn’t work. I have got more crashes lately after I updated branch, not sure if errors help but here are some:


 ;; Opening remote file 
emacs(16737,0x700004306000) malloc: *** error for object 0x11cb50885: pointer being freed was not allocated emacs(16737,0x111d105c0) malloc: *** error for object 0x7ffeefbfb610: pointer being freed was not allocated emacs(16737,0x700004306000) malloc: *** set a breakpoint in malloc_error_break to debug emacs(16737,0x111d105c0) malloc: *** set a breakpoint in malloc_error_break to debug Fatal error 6: Abort trapAbort trap: 6 logout Saving session... ...saving history...truncating history files... ...completed.

;; Renaming file over SSH via tramp 
objc[40791]: Invalid or prematurely-freed autorelease pool 0x10200b1c0. Fatal error 6: Abort trapAbort trap: 6 

 ;; another local sudo
emacs(41111,0x70000c892000) malloc: *** error for object 0x70000d093530: pointer being freed was not allocated emacs(41111,0x70000c892000) malloc: *** set a breakpoint in malloc_error_break to debug Fatal error 4: Illegal instructionAbort trap: 6 

;; another remote dired over SSH
emacs(40323,0x1181f95c0) malloc: *** error for object 0x103337305: pointer being freed was not allocated emacs(40323,0x1181f95c0) malloc: *** set a breakpoint in malloc_error_break to debug

> 5 nov. 2018 kl. 16:39 skrev Christian Johansson <christian@cvj.se>:
> 
> Alright, I’ll see if I can get the error again. I only got it like 2 times in 8 hours so it’s not frequent.
> 
> I tried user-interaction via threaded ange-ftp  tramp on macOS and it seems to work on your branch. I will try it a bit more this week.
> 
> Regards
> Christian
> 
>> 5 nov. 2018 kl. 14:49 skrev Michael Albinus <michael.albinus@gmx.de>:
>> 
>> Christian Johansson <christian@cvj.se> writes:
>> 
>>> Hi Michael
>> 
>> Hi Christian,
>> 
>>> I haven't experienced a single crash since I started using your
>>> threaded tramp branch but I sometimes get errors like this, but it
>>> doesn't make Emacs crash, I can just retry same operation and it will
>>> work. It seems like sometimes a thread is busy and Emacs tries to
>>> access it in some way and this error occurs, I have replaced sensitive
>>> information with --- below
>>> 
>>> Debugger entered--Lisp error: (error "Attempt to accept output from
>>> process *ftp ---...")
>>>  accept-process-output(#<process *ftp ---@---*>)
>>>  ange-ftp-wait-not-busy(#<process *ftp ---@---*>)
>>>  ange-ftp-raw-send-cmd(#<process *ftp ---@---*> "type ascii" nil
>>> (#f(compiled-function (result line host user cmd msg cont nowait)
>>> #<bytecode 0x4ffee879>) "---" "---" "type ascii" nil nil nil) nil)
>> 
>> Looks like the ange-ftp process is not assigned to the proper thread. As
>> said, ange-ftp is a special case, it wasn't handled yet.
>> 
>> I've pushed a fix to the feature/tramp-thread-safe branch; should work
>> now. Could you pls retest?
>> 
>>> I haven't tried threaded tramp that requires user-interaction yet
>>> though since I got ~/.authinfo.gpg working with ange-ftp, I'll see if
>>> I can test that this week
>> 
>> Likely, this won't work. Nobody has provided a fix for this problem, yet.
>> 
>>> Regards
>>> Christian
>> 
>> Best regards, Michael.

[-- Attachment #2: Type: text/html, Size: 9683 bytes --]

bug#33135: 27.0.50; (make-thread) TRAMP interactive freeze macOS 10.14

[Michael Albinus]

Christian Johansson <christian@cvj.se> writes:

Hi Christian,

> I was wrong, interactive threaded tramp over ange-ftp on macos doesn’t
> work. I have got more crashes lately after I updated branch, not sure
> if errors help but here are some:

Are these crashes just in case a user interaction with the thread was
needed, or does it happen also when a (remote) file is just visited in a
buffer, without any interaction?

In the latter case, could you pls describe your scenario?

Best regards, Michael.

bug#33135: 27.0.50; (make-thread) TRAMP interactive freeze macOS 10.14

[Christian Johansson]

Hi

Both scenarios cause the crashes, user-interaction for password with 
threaded tramp seems to always crash emacs on macOS. The window doesn't 
close but there is a error in the terminal and emacs gets stuck

In the other scenario I have repeatedly got the error when visiting a 
archive (find-file) on remote ssh tramp host that is also piped into 
another user like this:
(find-file 
"/ssh:user@server.com|sudo:otheruser@server.com:/home/otheruser/myzip.zip")

It crashes when I do it synchronously (unthreaded)

I also think I get the error when visiting directory in dired like 
(dired 
"/ssh:user@server.com|sudo:otheruser@server.com:/home/otheruser/"), also 
unthreaded


On 2018-11-07 08:48, Michael Albinus wrote:
> Christian Johansson <christian@cvj.se> writes:
>
> Hi Christian,
>
>> I was wrong, interactive threaded tramp over ange-ftp on macos doesn’t
>> work. I have got more crashes lately after I updated branch, not sure
>> if errors help but here are some:
> Are these crashes just in case a user interaction with the thread was
> needed, or does it happen also when a (remote) file is just visited in a
> buffer, without any interaction?
>
> In the latter case, could you pls describe your scenario?
>
> Best regards, Michael.

bug#33135: 27.0.50; (make-thread) TRAMP interactive freeze macOS 10.14

[Michael Albinus]

Christian Johansson <christian@cvj.se> writes:

> Hi

Hi Christian,

> In the other scenario I have repeatedly got the error when visiting a
> archive (find-file) on remote ssh tramp host that is also piped into
> another user like this:
> (find-file
> "/ssh:user@server.com|sudo:otheruser@server.com:/home/otheruser/myzip.zip")
>
> It crashes when I do it synchronously (unthreaded)
>
> I also think I get the error when visiting directory in dired like
> (dired
> "/ssh:user@server.com|sudo:otheruser@server.com:/home/otheruser/"),
> also unthreaded

Does it happen only in the feature/tramp-thread-safe branch, or also in
the master branch?

Anyway, I cant reproduce it locally on GNU/Linux, so there is something
wrong in the macOS implementation of threads I fear.

Best regards, Michael.

bug#33135: 27.0.50; (make-thread) TRAMP interactive freeze macOS 10.14

[Christian Johansson]

Hi!

I don’t get the error on master, just this branch. I tried setting execute-file-commands-asynchronously to nil to make sure it’s unthreaded with same result

Regards
Christian



> 9 nov. 2018 kl. 11:12 skrev Michael Albinus <michael.albinus@gmx.de>:
> 
> Christian Johansson <christian@cvj.se> writes:
> 
>> Hi
> 
> Hi Christian,
> 
>> In the other scenario I have repeatedly got the error when visiting a
>> archive (find-file) on remote ssh tramp host that is also piped into
>> another user like this:
>> (find-file
>> "/ssh:user@server.com|sudo:otheruser@server.com:/home/otheruser/myzip.zip")
>> 
>> It crashes when I do it synchronously (unthreaded)
>> 
>> I also think I get the error when visiting directory in dired like
>> (dired
>> "/ssh:user@server.com|sudo:otheruser@server.com:/home/otheruser/"),
>> also unthreaded
> 
> Does it happen only in the feature/tramp-thread-safe branch, or also in
> the master branch?
> 
> Anyway, I cant reproduce it locally on GNU/Linux, so there is something
> wrong in the macOS implementation of threads I fear.
> 
> Best regards, Michael.

bug#33135: 27.0.50; (make-thread) TRAMP interactive freeze macOS 10.14

[Michael Albinus]

Christian Johansson <christian@cvj.se> writes:

> Hi!

Hi Christian,

> I don’t get the error on master, just this branch. I tried setting
> execute-file-commands-asynchronously to nil to make sure it’s
> unthreaded with same result

Strange. Could you bisect the branch in order to find out, when this
problem started to appear? I would do myself, but there's no macOS
machine around.

> Regards
> Christian

Best regards, Michael.

bug#33135: 27.0.50; (make-thread) TRAMP interactive freeze macOS 10.14

[Christian Johansson]

Hi again

 Sure l’ll do that next week

Regards

> 11 nov. 2018 kl. 09:53 skrev Michael Albinus <michael.albinus@gmx.de>:
> 
> Christian Johansson <christian@cvj.se> writes:
> 
>> Hi!
> 
> Hi Christian,
> 
>> I don’t get the error on master, just this branch. I tried setting
>> execute-file-commands-asynchronously to nil to make sure it’s
>> unthreaded with same result
> 
> Strange. Could you bisect the branch in order to find out, when this
> problem started to appear? I would do myself, but there's no macOS
> machine around.
> 
>> Regards
>> Christian
> 
> Best regards, Michael.

bug#33135: 27.0.50; (make-thread) TRAMP interactive freeze macOS 10.14

[Christian Johansson]

Hi Michael

As of latest commit 90ae...5d93 I am no longer able to reproduce the issue, was just about to start bisecting now

Regards
Christian

> 11 nov. 2018 kl. 09:53 skrev Michael Albinus <michael.albinus@gmx.de>:
> 
> Christian Johansson <christian@cvj.se> writes:
> 
>> Hi!
> 
> Hi Christian,
> 
>> I don’t get the error on master, just this branch. I tried setting
>> execute-file-commands-asynchronously to nil to make sure it’s
>> unthreaded with same result
> 
> Strange. Could you bisect the branch in order to find out, when this
> problem started to appear? I would do myself, but there's no macOS
> machine around.
> 
>> Regards
>> Christian
> 
> Best regards, Michael.

bug#33135: 27.0.50; (make-thread) TRAMP interactive freeze macOS 10.14

[Christian Johansson]

Oh wait, false positive. Issue doesn’t come when running without macOS window system (-nw) but only with window system.

Will start bisecting now

Regards
Christian

> 14 nov. 2018 kl. 08:59 skrev Christian Johansson <christian@cvj.se>:
> 
> Hi Michael
> 
> As of latest commit 90ae...5d93 I am no longer able to reproduce the issue, was just about to start bisecting now
> 
> Regards
> Christian
> 
>> 11 nov. 2018 kl. 09:53 skrev Michael Albinus <michael.albinus@gmx.de>:
>> 
>> Christian Johansson <christian@cvj.se> writes:
>> 
>>> Hi!
>> 
>> Hi Christian,
>> 
>>> I don’t get the error on master, just this branch. I tried setting
>>> execute-file-commands-asynchronously to nil to make sure it’s
>>> unthreaded with same result
>> 
>> Strange. Could you bisect the branch in order to find out, when this
>> problem started to appear? I would do myself, but there's no macOS
>> machine around.
>> 
>>> Regards
>>> Christian
>> 
>> Best regards, Michael.

bug#33135: 27.0.50; (make-thread) TRAMP interactive freeze macOS 10.14

[Michael Albinus]

Christian Johansson <christian@cvj.se> writes:

> Oh wait, false positive. Issue doesn’t come when running without macOS window system (-nw) but only with window system.

OK, will wait.

> Will start bisecting now

Thanks!

> Regards
> Christian

Best regards, Michael.

bug#33135: 27.0.50; (make-thread) TRAMP interactive freeze macOS 10.14

[Christian Johansson]

Hi again

The issue didn’t happen without a init file so I narrowed down the issue to this line in my init file:

(when (fboundp ’global-ede-mode) (global-ede-mode nil))

It only happens on this branch and when accessing tramp for some reason. t or nil as argument causes same behavior



> 14 nov. 2018 kl. 09:08 skrev Michael Albinus <michael.albinus@gmx.de>:
> 
> Christian Johansson <christian@cvj.se> writes:
> 
>> Oh wait, false positive. Issue doesn’t come when running without macOS window system (-nw) but only with window system.
> 
> OK, will wait.
> 
>> Will start bisecting now
> 
> Thanks!
> 
>> Regards
>> Christian
> 
> Best regards, Michael.

bug#33135: 27.0.50; (make-thread) TRAMP interactive freeze macOS 10.14

[Michael Albinus]

Christian Johansson <christian@cvj.se> writes:

> Hi again

Hi Christian,

> The issue didn’t happen without a init file so I narrowed down the
> issue to this line in my init file:
>
> (when (fboundp ’global-ede-mode) (global-ede-mode nil))
>
> It only happens on this branch and when accessing tramp for some
> reason. t or nil as argument causes same behavior

Thanks for investigation! So I guess a simple (require 'ede) would
already cause the trouble in  "emacs -Q"?

Do you have a chance to debug or trace ede?

Best regards, Michael.

bug#33135: 27.0.50; (make-thread) TRAMP interactive freeze macOS 10.14

[Christian Johansson]

Hi again

(require ’ede) + open tramp file doesn’t cause bug, only (global-ede-mode) + open tramp file. Local files are fine

I’ll try get a backtrace later, do you recommend GDB for that?

Regards
Chris

> 16 nov. 2018 kl. 12:53 skrev Michael Albinus <michael.albinus@gmx.de>:
> 
> Christian Johansson <christian@cvj.se> writes:
> 
>> Hi again
> 
> Hi Christian,
> 
>> The issue didn’t happen without a init file so I narrowed down the
>> issue to this line in my init file:
>> 
>> (when (fboundp ’global-ede-mode) (global-ede-mode nil))
>> 
>> It only happens on this branch and when accessing tramp for some
>> reason. t or nil as argument causes same behavior
> 
> Thanks for investigation! So I guess a simple (require 'ede) would
> already cause the trouble in  "emacs -Q"?
> 
> Do you have a chance to debug or trace ede?
> 
> Best regards, Michael.

bug#33135: 27.0.50; (make-thread) TRAMP interactive freeze macOS 10.14

[Michael Albinus]

Christian Johansson <christian@cvj.se> writes:

> Hi again

Hi Christian,

> I’ll try get a backtrace later, do you recommend GDB for that?

Yes. But I have no experience to debug threads with GDB. See
<https://debbugs.gnu.org/32537>, where somebody has debugged threaded
Tramp with gdb.

> Regards
> Chris

Best regards, Michael.

bug#33135: 27.0.50; (make-thread) TRAMP interactive freeze macOS 10.14

[Christian Johansson]

Hi again!

Git bisect found that the commit causing this is: 917...933 find-file-asynchronously can alse be regexp, it works before that

Regards
Christian

> 16 nov. 2018 kl. 14:45 skrev Michael Albinus <michael.albinus@gmx.de>:
> 
> Christian Johansson <christian@cvj.se> writes:
> 
>> Hi again
> 
> Hi Christian,
> 
>> I’ll try get a backtrace later, do you recommend GDB for that?
> 
> Yes. But I have no experience to debug threads with GDB. See
> <https://debbugs.gnu.org/32537>, where somebody has debugged threaded
> Tramp with gdb.
> 
>> Regards
>> Chris
> 
> Best regards, Michael.

bug#33135: 27.0.50; (make-thread) TRAMP interactive freeze macOS 10.14

[Michael Albinus]

Christian Johansson <christian@cvj.se> writes:

> Hi Michael

Hi Christian,

> As of latest commit 90ae...5d93 I am no longer able to reproduce the issue, was just about to start bisecting now

Great! Maybe this is due to the last merge with master; some problems
with macOS were reported there.

I'm closing the bug. Feel free to report if there are further problems.

Thanks for your work and testing on the feature/tramp-thread-safe
branch!

> Regards
> Christian

Best regards, Michael.

bug#33135: 27.0.50; (make-thread) TRAMP interactive freeze macOS 10.14

[Christian Johansson]

Alright, I guessed that the threaded user-interaction wasn't completed 
yet, TRAMP without user-interaction seems to work however. Keep up the 
good work. At least it should be easy to test it once it's finished with 
my test. If you access my example without the treading and Emacs saves 
the password it will work inside the thread too I think because there 
will be no need for user-interaction

Regards
Christian

On 2018-10-24 12:53, Alan Third wrote:
> On Wed, Oct 24, 2018 at 12:15:22PM +0200, Christian Johansson wrote:
>> Hi!
>>
>> I am able to reproduce this on two different computers using macOS 10.14 but
>> I haven't tested this on any other system.
>>
>> Steps to reproduce:
>> 1. Open Emacs without any configuration: src/emacs -Q
>> 2. Make sure you are in *scratch* buffer
>> 3. Paste following Emacs Lisp code:
>>
>> ;; FTP URL: ftp.dlptest.com or ftp://ftp.dlptest.com/
>> ;; FTP User: dlpuser@dlptest.com
>> ;; Password: e73jzTRTNqCN9PYAAjjn
>> (make-thread (lambda()
>>                 (message "Listing directory files..")
>>                 (let ((files (directory-files
>> "/ftp:dlpuser@dlptest.com@ftp.dlptest.com:/")))
>>                   (message "Opened directory")
>>                   (message "Directory files: %s" files))))
>>
>> 4. Run M-x eval-buffer
>> 5. Emacs asks for FTP password, paste in mini-buffer: e73jzTRTNqCN9PYAAjjn
>> 6. Emacs now hangs, asks for password again and you can't stop it using C-g.
>> You get error in terminal:
> The NS port currently doesn’t handle interactive functions from
> threads very well. I suspect the problem is being caused by displaying
> a message from the background thread.
>
>> Christians-Air:emacs christianjohansson$ src/emacs -Q
>> 2018-10-24 12:07:58.383 emacs[627:14815] Failed to initialize color list
>> unarchiver: Error Domain=NSCocoaErrorDomain Code=4864 "***
>> -[NSKeyedUnarchiver _initForReadingFromData:error:throwLegacyExceptions:]:
>> non-keyed archive cannot be decoded by NSKeyedUnarchiver"
>> UserInfo={NSDebugDescription=*** -[NSKeyedUnarchiver
>> _initForReadingFromData:error:throwLegacyExceptions:]: non-keyed archive
>> cannot be decoded by NSKeyedUnarchiver}
> This error will be fixed in Emacs 26.2.
>
>> 2018-10-24 12:08:28.515 emacs[627:15311] WARNING: NSWindow drag regions
>> should only be invalidated on the Main Thread! This will throw an exception
>> in the future. Called from (
>>      0   AppKit                              0x00007fff334132e3
>> -[NSWindow(NSWindow_Theme)
>> _postWindowNeedsToResetDragMarginsUnlessPostingDisabled] + 386
>>      1   AppKit                              0x00007fff33435422
>> -[NSThemeFrame _tileTitlebarAndRedisplay:] + 98
>>      2   AppKit                              0x00007fff334604dd
>> -[NSTitledFrame _titleDidChange] + 217
>>      3   AppKit                              0x00007fff3345ff89
>> -[NSTitledFrame setTitle:] + 730
>>      4   AppKit                              0x00007fff3345fc00
>> -[NSThemeFrame setTitle:] + 50
>>      5   AppKit                              0x00007fff33423082 -[NSWindow
>> _dosetTitle:andDefeatWrap:] + 211
>>      6   emacs                               0x00000001001e7371
>> ns_set_name_internal + 161
>>      7   emacs                               0x000000010004f640
>> x_consider_frame_title + 784
>>      8   emacs                               0x0000000100028ad5
>> redisplay_internal + 2949
> <snip>
>>      73  emacs                               0x00000001001b624a run_thread +
>> 138
>>      74  libsystem_pthread.dylib             0x00007fff6316d33d _pthread_body
>> + 126
>>      75  libsystem_pthread.dylib             0x00007fff631702a7
>> _pthread_start + 70
>>      76  libsystem_pthread.dylib             0x00007fff6316c425 thread_start
>> + 13
>> )
> Yes, we can see redisplay is being called from within the thread, and
> that just won’t work currently, I’m afraid.

bug#33135: 27.0.50; (make-thread) TRAMP interactive freeze macOS 10.14

[Lars Ingebrigtsen]

Christian Johansson <christian@cvj.se> writes:

> 10.14 but I haven't tested this on any other system.
>
> Steps to reproduce:
> 1. Open Emacs without any configuration: src/emacs -Q
> 2. Make sure you are in *scratch* buffer
> 3. Paste following Emacs Lisp code:

(I'm going through old bug reports that unfortunately weren't resolved
at the time.)

This was three years ago -- are you still seeing these issues with
recent Emacs versions?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#33135: 27.0.50; (make-thread) TRAMP interactive freeze macOS 10.14

[Christian Johansson]

Yes I think so, I don't use macOS anymore but last time I used multithreading on macOS Emacs dies occasionally. Sometimes I get it on Ubuntu as well but it happens much more rare ans much less deterministically

> 24 apr. 2022 kl. 16:42 skrev Lars Ingebrigtsen <larsi@gnus.org>:
> 
> Christian Johansson <christian@cvj.se> writes:
> 
>> 10.14 but I haven't tested this on any other system.
>> 
>> Steps to reproduce:
>> 1. Open Emacs without any configuration: src/emacs -Q
>> 2. Make sure you are in *scratch* buffer
>> 3. Paste following Emacs Lisp code:
> 
> (I'm going through old bug reports that unfortunately weren't resolved
> at the time.)
> 
> This was three years ago -- are you still seeing these issues with
> recent Emacs versions?
> 
> -- 
> (domestic pets only, the antidote for overdose, milk.)
>   bloggy blog: http://lars.ingebrigtsen.no