trapping process filter error in a thread

trapping process filter error in a thread

[Thien-Thi Nguyen]

[-- Attachment #1.1: Type: text/plain, Size: 399 bytes --]


I'm in the process of writing a dead-link checker and would like
to use Emacs' threads and ‘url-http’ funcs.  I've run into a
problem w/ GNU Emacs 27.1 (Debian), however: When there is a TLS
problem, i am unable to trap the process filter error.  It seems
‘condition-case’ is not the right tool for the job.

Attached here is a fragment that encapsulates the heart of the
problem:

[-- Attachment #1.2: bad.el --]
[-- Type: application/emacs-lisp, Size: 514 bytes --]

[-- Attachment #1.3: Type: text/plain, Size: 92 bytes --]


and a *compilation* buffer capture that shows how the process
filter error is not trapped:

[-- Attachment #1.4: bad.out --]
[-- Type: application/octet-stream, Size: 495 bytes --]

-*- mode: compilation; default-directory: "/tmp/" -*-
Compilation started at Tue Mar 22 09:42:14

for url in https://www.gnu.org/ https://www.gnu.org/WRONG http://www.oecd.org/pisa/test/ ; do echo ; echo url: $url ; emacs -q -Q --batch -l bad.el $url ; done

url: https://www.gnu.org/
"ok"

url: https://www.gnu.org/WRONG
"dead"

url: http://www.oecd.org/pisa/test/
error in process filter: Process www.oecd.org<1> not running

Compilation exited abnormally with code 255 at Tue Mar 22 09:42:17

[-- Attachment #1.5: Type: text/plain, Size: 879 bytes --]


My questions are:

(a) Can anyone else reproduce this on their Emacs?
    (Obviously your Emacs has to be built w/ thread support.)

(b) Does the same code work in different versions of Emacs?

(c) Is there a recommended way to trap process filter errors
    for a noninteractive (perhaps batch) session?

(d) Is there a more idiomatic way to work w/ threads and
    ‘url-http’ (or async network connections, in general)?

Thank-you for any insight into this corner of Emacs Lisp!

-- 
Thien-Thi Nguyen -----------------------------------------------
 (defun responsep (query)               ; (2022) Software Libero
   (pcase (context query)               ;       = Dissenso Etico
     (`(technical ,ml) (correctp ml))
     ...))                              748E A0E8 1CB8 A748 9BFA
--------------------------------------- 6CE4 6703 2224 4C80 7502


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 219 bytes --]

Re: trapping process filter error in a thread

[Emanuel Berg via Users list for the GNU Emacs text editor]

Thien-Thi Nguyen wrote:

> I'm in the process of writing a dead-link checker

Good idea ...

> and would like to use Emacs' threads and ‘url-http’ funcs.
> I've run into a problem w/ GNU Emacs 27.1 (Debian), however:
> When there is a TLS problem, i am unable to trap the process
> filter error. It seems ‘condition-case’ is not the right
> tool for the job.
>
> (defun func ()
>   (let ((noninteractive t))
>     (condition-case nil
>         (if (url-http-file-exists-p url)
>             "ok"
>           "dead")
>       (t "error"))))

Isn't the syntax/usage rather as in e.g.

(defun erc-element-next (&optional prev)
  (interactive "P")
  (when (erc-at-prompt-p)
    (goto-char (point-min)) )
  (condition-case nil
      (if prev
          (erc-button-previous)
        (erc-button-next) )
    (error (progn
             (goto-char (point-max))
             (when prev
               (erc-bol) )))))

> (a) Can anyone else reproduce this on their Emacs?
>     (Obviously your Emacs has to be built w/
>     thread support.)

What are threads in this context and how do you build Emacs
for that

> (c) Is there a recommended way to trap process filter errors
>     for a noninteractive (perhaps batch) session?

What's a process filter?

> Thank-you for any insight into this corner of Emacs Lisp!

You don't say ;)

-- 
underground experts united
https://dataswamp.org/~incal

Re: trapping process filter error in a thread

[Felix Dietrich]

Thien-Thi Nguyen <ttn@gnuvola.org> writes:

> I'm in the process of writing a dead-link checker and would like
> to use Emacs' threads and ‘url-http’ funcs.

You are much braver than I ;).  I am not sure if the url library is
robust enough for that task, but you can certainly try and play around
with it.

> I've run into a problem w/ GNU Emacs 27.1 (Debian), however: When
> there is a TLS problem, i am unable to trap the process filter error.
> It seems ‘condition-case’ is not the right tool for the job.
>
> (defvar url "http://www.oecd.org/pisa/test/")
> (defun func ()
>   (let ((noninteractive t))
>     (condition-case nil
>         (if (url-http-file-exists-p url)
>             "ok"
>           "dead")
>       (t "error"))))
>
> (defvar thread (make-thread 'func))
> (message "%S" (thread-join thread))
>
>
> My questions are:
>
> (a) Can anyone else reproduce this on their Emacs?
>     (Obviously your Emacs has to be built w/ thread support.)

Yes (also Debianʼs 27.1), but it does not appear to have anything to do
with threads: it happens just as well when you call ‘func’ directly.

The issue seems to be that ‘url-http’ tries to ‘process-send-string’ to
a closed connection.  After being 302-redirected from an http to an
https URL, the connection is closed because of the problem with the SSL
certificate: in an interactive session Emacs would ask you whether you
would like to accept the certificated and connect despite its problems;
in batch mode the assumed answer is: “no, do not accept the certificate”
and the connection is closed.  I donʼt know if that could be fixed in
‘url-http’.  The simple ‘failed’ state for the connection also just
signals an error and would crash your script in the same way.  I have
not found a good way to handle these errors with the url library.

> (c) Is there a recommended way to trap process filter errors
>     for a noninteractive (perhaps batch) session?

Fix the process filter ;).

I found the variable ‘command-error-function’ in the Emacs Lisp manual
[1].  “This variable, if non-‘nil’, specifies a function to use to
handle errors that return control to the Emacs command loop.”  To test
it and see what it does, I set it to a dummy handler in your example
script, which appeared to have the desired result: the script exited
indicating success.

    #+begin_src emacs-lisp
      (setq command-error-function
            (lambda (data context caller)
          (message "data: %s\ncontext: %s\ncaller: %s"
               data context caller)))
    #+end_src

> (d) Is there a more idiomatic way to work w/ threads and
>     ‘url-http’ (or async network connections, in general)?

‘url-retrieve’ is the asynchronous counterpart to
‘url-retrieve-synchronously’.  You pass it a function to call once the
retrieval is done.  It would probably suffer from the same issues with
error handling in ‘url-http’ described above.  There is also “emacs-aio”
by Chris Wellons [2] that implements the async-await pattern, but I have
not really used it besides poking at it when I tried to understand how
it works.  I have not used threads in Emacs, so I have nothing on that.

> Thank-you for any insight into this corner of Emacs Lisp!

Here is one that at one point took me a while to find: in batch mode
functions that read from the minibuffer instead read the standard input,
and an Emacs Lisp script can, therefore, read lines from a pipe like so:

    #+begin_src emacs-lisp
      (setq line-count 0)
      (while (setq line-count (1+ line-count)
                   line (ignore-errors (read-string "")))
        (message "%.03i: %s" line-count line))
    #+end_src

Depending on the number of URLs you want to check and their length, this
might safe you a couple of Emacs invocations and initialisations when
you cannot fit all URLs on a single command line.


Footnotes:

[1]  <https://www.gnu.org/software/emacs/manual/html_node/elisp/Processing-of-Errors.html>

     (info "(elisp) Processing of Errors")

[2]  <https://nullprogram.com/blog/2019/03/10/>

     <https://github.com/skeeto/emacs-aio>

-- 
Felix Dietrich

Re: trapping process filter error in a thread

[Thien-Thi Nguyen]

[-- Attachment #1: Type: text/plain, Size: 2321 bytes --]


() Emanuel Berg via Users list for the GNU Emacs text editor <help-gnu-emacs@gnu.org>
() Tue, 22 Mar 2022 20:28:12 +0100

   Isn't the syntax/usage rather as in e.g.

   (defun erc-element-next (&optional prev)
     (interactive "P")
     (when (erc-at-prompt-p)
       (goto-char (point-min)) )
     (condition-case nil
         (if prev
             (erc-button-previous)
           (erc-button-next) )
       (error (progn
                (goto-char (point-max))
                (when prev
                  (erc-bol) )))))

I assume you're talking about ‘condition-case’, right?  I think
it's no problem to specify ‘t’ as the condition instead of
‘error’.  (I've used both forms in my code before w/o issue.)

   > (a) Can anyone else reproduce this on their Emacs?
   >     (Obviously your Emacs has to be built w/
   >     thread support.)

   What are threads in this context and how do you build Emacs
   for that

Threads are the "mini process" abstraction supported by modern
operating systems (in various ways):

- https://en.wikipedia.org/wiki/Thread_%28computing%29

On my system, the pre-packaged Emacs (27.1) has thread support
already included; i did not need to do anything special to
enable it.  However, if you're building from source, i see that
the configure script has this ‘--help’ output excerpt:

  --without-threads       don't compile with elisp threading support

which makes me think that thread support is enabled unless you
explicitly disable it.  (TLDR: Maybe your Emacs already has it!)

   > (c) Is there a recommended way to trap process filter
   > errors for a noninteractive (perhaps batch) session?

   What's a process filter?

It's the function that manages the output of a subprocess:

- (info "(elisp) Filter Functions")

In this context, it also manages network connections (which are
treated using the subprocess framework).

-- 
Thien-Thi Nguyen -----------------------------------------------
 (defun responsep (query)               ; (2022) Software Libero
   (pcase (context query)               ;       = Dissenso Etico
     (`(technical ,ml) (correctp ml))
     ...))                              748E A0E8 1CB8 A748 9BFA
--------------------------------------- 6CE4 6703 2224 4C80 7502


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 219 bytes --]

Re: trapping process filter error in a thread

[Thien-Thi Nguyen]

[-- Attachment #1.1: Type: text/plain, Size: 1369 bytes --]


() Felix Dietrich <felix.dietrich@sperrhaken.name>
() Thu, 24 Mar 2022 03:33:50 +0100

   > would like to use Emacs' threads and ‘url-http’ funcs.

   You are much braver than I ;).  I am not sure if the url
   library is robust enough for that task, but you can certainly
   try and play around with it.

'Tis a fine line between brave and foolish, i'm told.  :-D

   > (a) Can anyone else reproduce this on their Emacs?

   Yes (also Debianʼs 27.1), but it does not appear to have
   anything to do with threads: it happens just as well when you
   call ‘func’ directly.

I think this tells us that error handing wrt threads are as if
they were invoked as a (top-level) command.

   [analysis: chain of failures] I have not found a good way to
   handle these errors with the url library.

Thanks for digging into the details.  After some poking around,
i realize that the url*.el facility has quite a lot of baggage
to deal w/, as well.  It's amazing how much cruft RFCs output!

   > (c) Is there a recommended way to trap process filter
   > errors for a noninteractive (perhaps batch) session?

   Fix the process filter ;).

Haha!

   I found the variable ‘command-error-function’ in the Emacs
   Lisp manual [...].

Good eye!  I have managed to adapt your code fragment to produce
a bad.el (attached here):

[-- Attachment #1.2: bad.el --]
[-- Type: application/emacs-lisp, Size: 878 bytes --]

[-- Attachment #1.3: Type: text/plain, Size: 52 bytes --]

that is able to trap the error (see bad.out, here):

[-- Attachment #1.4: bad.out --]
[-- Type: application/octet-stream, Size: 511 bytes --]

-*- mode: compilation; default-directory: "/tmp/" -*-
Compilation started at Sat Apr  2 19:42:08

for url in https://www.gnu.org/ https://www.gnu.org/WRONG http://www.oecd.org/pisa/test/ ; do printf '\nurl: %s\n' $url ; emacs -batch -Q -l bad.el $url ; done

url: https://www.gnu.org/
"ok"

url: https://www.gnu.org/WRONG
"dead"

url: http://www.oecd.org/pisa/test/
var: (no-catch error (("Process www.oecd.org<1> not running") "error in process filter: "))
"error"

Compilation finished at Sat Apr  2 19:42:11

[-- Attachment #1.5: Type: text/plain, Size: 945 bytes --]


Curiously, there is ‘no-catch’ in the CAR of ‘var’ contents, but
i was not able to find any mention of it in the Elisp reference
manual.

   > (d) Is there a more idiomatic way to work w/ threads and
   >     ‘url-http’ (or async network connections, in general)?

   [url-*.el overview, plus emacs-aio]

I hope to be able to study these Someday.

   > Thank-you for any insight into this corner of Emacs Lisp!

   [‘read-string’ in batch mode]

Thanks for the tip.  It's good to see Emacs can be useful in
many different contexts.

-- 
Thien-Thi Nguyen -----------------------------------------------
 (defun responsep (query)               ; (2022) Software Libero
   (pcase (context query)               ;       = Dissenso Etico
     (`(technical ,ml) (correctp ml))
     ...))                              748E A0E8 1CB8 A748 9BFA
--------------------------------------- 6CE4 6703 2224 4C80 7502


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 219 bytes --]

Re: trapping process filter error in a thread

[Emanuel Berg via Users list for the GNU Emacs text editor]

Thien-Thi Nguyen wrote:

>>> Can anyone else reproduce this on their Emacs? (Obviously
>>> your Emacs has to be built w/ thread support.)
>>
>> What are threads in this context and how do you build Emacs
>> for that
>
> Threads are the "mini process" abstraction supported by
> modern operating systems (in various ways):

Okay, so we are talking Unix. A process is a program in
execution, a thread is a virtual lightweight process
implemented as a process segment.

I thought Emacs maybe had its own threads ...

>   --without-threads       don't compile with elisp threading support
>
> which makes me think that thread support is enabled unless
> you explicitly disable it. (TLDR: Maybe your Emacs already
> has it!)

Hm ... I wonder what kind of support that is exactly?

>> What's a process filter?
>
> It's the function that manages the output of a subprocess

OK!

-- 
underground experts united
https://dataswamp.org/~incal

Re: trapping process filter error in a thread

[Felix Dietrich]

Thien-Thi Nguyen <ttn@gnuvola.org> writes:

> () Felix Dietrich <felix.dietrich@sperrhaken.name>
> () Thu, 24 Mar 2022 03:33:50 +0100
>
> I think this tells us that error handing wrt threads are as if
> they were invoked as a (top-level) command.

With regard to process filters, I would presume.  They run
asynchronously outside the context of whatever code you told Emacs to
execute whenever Emacs is idle or told to accept process input.  It
would be awkward and rather tedious if you had to be aware not only of
the errors your code could signal but also of all the errors any
potentially running process filter might produce – although in your case
and your use of the url library you kind of have to.  In threads, you
should be able to handle clean-up and errors just as you would do
normally: with ‘unwind-protect’ and ‘condition-case’.  (Though, as I had
mentioned, I have not programmed with threads in Emacs, yet; I just
imagine that this is how it works.)


> I have managed to adapt your code fragment to produce a bad.el
> (attached here):
>
>      (throw (car data) (list (cdr data)
>                              context)))))
>
> Curiously, there is ‘no-catch’ in the CAR of ‘var’ contents, but
> i was not able to find any mention of it in the Elisp reference
> manual.

I donʼt understand what you intended with that ‘throw’, but ‘no-catch’
is mentioned under “Nonlocal Exits” as well as in the elisp reference
manualʼs index.  It means that you are missing an accompanying ‘catch’
to your ‘throw’.  Also: in Emacs you ‘signal’ errors; you donʼt throw
them.

    (info "(elisp) Catch and Throw")  [1]
    (info "(elisp) Signaling Errors") [2]

Both are subsections of:

    (info "(elisp) Nonlocal Exits")   [3]


Footnotes:
[1]  <https://www.gnu.org/software/emacs/manual/html_node/elisp/Catch-and-Throw.html>

[2]  <https://www.gnu.org/software/emacs/manual/html_node/elisp/Signaling-Errors.html>

[3]  <https://www.gnu.org/software/emacs/manual/html_node/elisp/Nonlocal-Exits.html>


-- 
Felix Dietrich