using finalizers

using finalizers

[Tomas Hlavaty]

Hi,

I am trying to understand finalizers in Emacs Lisp but I am unable to
triger it.

Here is an example (with lexical-binding):

(defun finito ()
  (let ((x '(a b c)))
    (letrec ((more (make-finalizer close))
             (close (lambda ()
                      (print "close-resource")
                      (setq more nil))))
      (lambda ()
        (when more
          (pop x))))))
(setq foo (finito))
(equal '(a b c nil nil) (mapcar (lambda (x) (funcall foo)) '(1 2 3 4 5)))
(setq foo nil)
(garbage-collect)

After garbage collection, I would expect to see close-resource text in
the *Messages* buffer but there is none.

The interface is different from Common Lisp, where make-finalizer
usually expects object and thunk, not thunk only but I guess the problem
lies somewhere else.

What is the expected example usage of finalizers in Emacs Lisp?
How can I verify, that my finalizer works?

Thanks in advance

Tomas

Re: using finalizers

[LdBeth]

>>>>> In <878rw1pvcw.fsf@logand.com> 
>>>>>	Tomas Hlavaty <tom@logand.com> wrote:
> Hi,

> I am trying to understand finalizers in Emacs Lisp but I am unable to
> triger it.

> Here is an example (with lexical-binding):

> (defun finito ()
>   (let ((x '(a b c)))
>     (letrec ((more (make-finalizer close))
>              (close (lambda ()
>                       (print "close-resource")
>                       (setq more nil))))
>       (lambda ()
>         (when more
>           (pop x))))))
> (setq foo (finito))
> (equal '(a b c nil nil) (mapcar (lambda (x) (funcall foo)) '(1 2 3 4 5)))
> (setq foo nil)
> (garbage-collect)

> After garbage collection, I would expect to see close-resource text in
> the *Messages* buffer but there is none.

You are probably taking the wrong assumption about the effect of letrec

(macroexpand '(letrec ((more (make-finalizer close))
                       (close (lambda ()
                                (setq bar t))))
                (lambda ()
                  (when more
                    (pop x)))))

==> (let (more close)
         (setq more (make-finalizer close))
         (setq close (lambda nil (setq bar t)))
         (lambda nil (when more (pop x))))

So, `close' is nil and the lambda is never set as the finalizer.

I think Common Lisp doesn't have letrec in the standard. And you are
proabaly taking the assumption that ELisp's letrec works as well as
Scheme's one. But it is not.

> What is the expected example usage of finalizers in Emacs Lisp?
> How can I verify, that my finalizer works?

(setq bar nil)
(setq foo (make-finalizer (lambda ()
                            (setq bar t))))

(setq foo nil)
(garbage-collect)

bar ; => t


-- 
LDB

Re: using finalizers

[Stefan Monnier]

> You are probably taking the wrong assumption about the effect of letrec
>
> (macroexpand '(letrec ((more (make-finalizer close))
>                        (close (lambda ()
>                                 (setq bar t))))
>                 (lambda ()
>                   (when more
>                     (pop x)))))
>
> ==> (let (more close)
>          (setq more (make-finalizer close))
>          (setq close (lambda nil (setq bar t)))
>          (lambda nil (when more (pop x))))

Indeed, the code needs to swap `more` and `close`.

> I think Common Lisp doesn't have letrec in the standard. And you are
> proabaly taking the assumption that ELisp's letrec works as well as
> Scheme's one. But it is not.

I don't think Scheme's `letrec` works any differently in this respect,
except that instead of nil, variables get a special "void like" value
before they're initialized.

In the above code, the real bug I can see which we should fix is that
`make-finalizer` should signal an error if its arg is not a function.


        Stefan

Re: using finalizers

[Stefan Monnier]

>> (macroexpand '(letrec ((more (make-finalizer close))
>>                        (close (lambda ()
>>                                 (setq bar t))))
>>                 (lambda ()
>>                   (when more
>>                     (pop x)))))
[...]
> Indeed, the code needs to swap `more` and `close`.

Actually, simpler:

    (letrec ((more (make-finalizer
                    (lambda ()
                      (print "close-resource")
                      (setq more nil)))))
      (lambda ()
        (when more
          (pop x)))))


-- Stefan

Re: using finalizers

[LdBeth]

>>>>> In <jwv35m94fqf.fsf-monnier+emacs@gnu.org> 
>>>>>	Stefan Monnier <monnier@iro.umontreal.ca> wrote:
Stefan> I don't think Scheme's `letrec` works any differently in this respect,
Stefan> except that instead of nil, variables get a special "void like" value
Stefan> before they're initialized.

Right. With that "void" like value, a sane Scheme would probably
signal error at runtime. The simple letrec subr.el macro does not do
that check.

Stefan> In the above code, the real bug I can see which we should fix is that
Stefan> `make-finalizer` should signal an error if its arg is not a function.

There's nothing wrong with the behavior of `make-finalizer`, it just
treats a symbol as a function specifier, just like elisp does not
doing anything to prevent a user from `fset' symbol `nil' and
`(funcall nil)'.

-- 
LDB

Re: using finalizers

[LdBeth]

>>>>> In <tencent_4CC28DA9FBCB31B64701835E87D4CAFF4F05@qq.com> 
>>>>>	LdBeth <andpuke@foxmail.com> wrote:
>>>>> In <jwv35m94fqf.fsf-monnier+emacs@gnu.org> 
>>>>>	Stefan Monnier <monnier@iro.umontreal.ca> wrote:
ldb> There's nothing wrong with the behavior of `make-finalizer`, it just
ldb> treats a symbol as a function specifier, just like elisp does not
ldb> doing anything to prevent a user from `fset' symbol `nil' and
ldb> `(funcall nil)'.

Allow me to make a correction: Emacs 27 would report a error when
`(fset nil (lambda () 1))', but some Common Lisps would allow that,
and `(fset t (lambda () 1))' works in Elisp.

I guess it's because of the difference in some implementation details.
While `(symbolp nil) => t', `nil' is actually a constant stands for
"empty list" that doesn't have the corresponding definition slots.

But still, I think the behavior of `make-finalizer' doesn't need a fix.

-- 
LDB

Re: using finalizers

[Tomas Hlavaty]

On Fri 31 Dec 2021 at 10:46, LdBeth <andpuke@foxmail.com> wrote:
>>>>>> In <878rw1pvcw.fsf@logand.com> Tomas Hlavaty <tom@logand.com> wrote:
> You are probably taking the wrong assumption about the effect of letrec
> So, `close' is nil and the lambda is never set as the finalizer.

You are right, the order was wrong.

>> What is the expected example usage of finalizers in Emacs Lisp?
>> How can I verify, that my finalizer works?
>
> (setq bar nil)
> (setq foo (make-finalizer (lambda () (setq bar t))))
> (setq foo nil)
> (garbage-collect)
> bar ; => t

Nice, that works.

One more question: Because make-finalizer in Emacs Lisp does not take
the object into account, I have to make sure to hold the reference to
the finalizer somehow.  However, this might not be trivial, because a
smart compiler could eliminate the reference.  Is this not an issue?
Shouldn't make-finalizer take the object as an argument and not rely on
me holding the reference explicitly?

Thank you!

Re: using finalizers

[Eli Zaretskii]

> Date: Fri, 31 Dec 2021 10:46:20 +0800
> From: LdBeth <andpuke@foxmail.com>
> Cc: emacs-devel@gnu.org
> 
> > What is the expected example usage of finalizers in Emacs Lisp?
> > How can I verify, that my finalizer works?
> 
> (setq bar nil)
> (setq foo (make-finalizer (lambda ()
>                             (setq bar t))))
> 
> (setq foo nil)
> (garbage-collect)
> 
> bar ; => t

I think that the assumption that a call to garbage-collect will
necessarily call the finalizer is also problematic: there's usually no
simple way to make sure there are no references left to the object,
and GC could have its own ideas when to actually garbage-collect an
object.

Re: using finalizers

[Tomas Hlavaty]

On Fri 31 Dec 2021 at 09:50, Eli Zaretskii <eliz@gnu.org> wrote:
> I think that the assumption that a call to garbage-collect will
> necessarily call the finalizer is also problematic: there's usually no
> simple way to make sure there are no references left to the object,

This breaks my expectations.  Could you explain that more?

I can see these cases where garbage collection might not do its job:

1. imprecise gc

2. program exit or abort

3. a leak to be fixed

What do you have in mind, that could hold extra unexpected references
and not be a leak?

Does it mean that for example thunk-delay leaks even though it releases
the reference to the body thunk?

Does it mean that setting reference to nil in order to dispose an object
(as in my original example) is futile?

> and GC could have its own ideas when to actually garbage-collect an
> object.

I don't think timing is an issue.  I think the expectation from gc is
that it collects the object "eventually as needed".  In the case of
imprecise gc "hopefully eventually as needed".

Re: using finalizers

[LdBeth]

>>>>> In <875yr5p9t3.fsf@logand.com> 
>>>>>	Tomas Hlavaty <tom@logand.com> wrote:

Tomas> One more question: Because make-finalizer in Emacs Lisp does
Tomas> not take the object into account, I have to make sure to hold
Tomas> the reference to the finalizer somehow.  However, this might
Tomas> not be trivial, because a smart compiler could eliminate the
Tomas> reference.  Is this not an issue?

That's being too smart. And I don't think the bytecode compiler can do
the lifetime analaysis right now. It is possible that this could be
changed in future releases of Emacs.

Tomas> Shouldn't make-finalizer take the object as an argument and not
Tomas> rely on me holding the reference explicitly?

You would only need finalizers for complex objects, so ideally you
could use OOP to manage that, probably EIEIO but not limited to that,
and you may even come up with your own.

-- 
LDB

Re: using finalizers

[Tomas Hlavaty]

On Fri 31 Dec 2021 at 18:59, LdBeth <andpuke@foxmail.com> wrote:
>>>>>> In <875yr5p9t3.fsf@logand.com> Tomas Hlavaty <tom@logand.com> wrote:
> Tomas> One more question: Because make-finalizer in Emacs Lisp does
> Tomas> not take the object into account, I have to make sure to hold
> Tomas> the reference to the finalizer somehow.  However, this might
> Tomas> not be trivial, because a smart compiler could eliminate the
> Tomas> reference.  Is this not an issue?
>
> That's being too smart. And I don't think the bytecode compiler can do
> the lifetime analaysis right now. It is possible that this could be
> changed in future releases of Emacs.

ok, so if I understand it correctly, in other words, the make-finalizer
interface relies on the compiler being not very smart

> Tomas> Shouldn't make-finalizer take the object as an argument and not
> Tomas> rely on me holding the reference explicitly?
>
> You would only need finalizers for complex objects, so ideally you
> could use OOP to manage that, probably EIEIO but not limited to that,
> and you may even come up with your own.

What do you mean?

Why not do it "properly" and future-proof the make-finalizer interface,
pass the object explicitly there and leave the rest as implementation
detail instead of leaking the issue to the programmer?

Re: using finalizers

[LdBeth]

>>>>> In <87bl0xm39e.fsf@logand.com> 
>>>>>	Tomas Hlavaty <tom@logand.com> wrote:

ldb> You would only need finalizers for complex objects, so ideally you
ldb> could use OOP to manage that, probably EIEIO but not limited to that,
ldb> and you may even come up with your own.

Tomas> What do you mean?

Tomas> Why not do it "properly" and future-proof the make-finalizer interface,
Tomas> pass the object explicitly there and leave the rest as implementation
Tomas> detail instead of leaking the issue to the programmer?

Isn't it doesn't make sense to associate a primitive data types such
as numbers, symbols, strings with a finalizer? If you need such a
feature on primitive data types, you are probably using this in a way
not intended.

And meawhile it would be problematic to hardcode what kinds of objects
are allowed to have objects. Notice this is a C function so it is not
very flexible.

You can think this is the primitive API that Emacs provides, and
people can build up more higher level ones that suit their particular
needs.

Re: using finalizers

[Tomas Hlavaty]

On Fri 31 Dec 2021 at 19:41, LdBeth <andpuke@foxmail.com> wrote:
>>>>>> In <87bl0xm39e.fsf@logand.com> 
>>>>>>	Tomas Hlavaty <tom@logand.com> wrote:
>
> ldb> You would only need finalizers for complex objects, so ideally you
> ldb> could use OOP to manage that, probably EIEIO but not limited to that,
> ldb> and you may even come up with your own.
>
> Tomas> What do you mean?
>
> Tomas> Why not do it "properly" and future-proof the make-finalizer interface,
> Tomas> pass the object explicitly there and leave the rest as implementation
> Tomas> detail instead of leaking the issue to the programmer?
>
> Isn't it doesn't make sense to associate a primitive data types such
> as numbers, symbols, strings with a finalizer? If you need such a
> feature on primitive data types, you are probably using this in a way
> not intended.
>
> And meawhile it would be problematic to hardcode what kinds of objects
> are allowed to have objects. Notice this is a C function so it is not
> very flexible.

This is irrelevant, I am not talking about that.

> You can think this is the primitive API that Emacs provides, and
> people can build up more higher level ones that suit their particular
> needs.

This primitive API that Emacs provides relies on the implicit assumption
of the compiler being not smart.  I do not see how that is a good idea.

Re: using finalizers

[Eli Zaretskii]

> From: Tomas Hlavaty <tom@logand.com>
> Cc: emacs-devel@gnu.org
> Date: Fri, 31 Dec 2021 10:31:02 +0100
> 
> On Fri 31 Dec 2021 at 09:50, Eli Zaretskii <eliz@gnu.org> wrote:
> > I think that the assumption that a call to garbage-collect will
> > necessarily call the finalizer is also problematic: there's usually no
> > simple way to make sure there are no references left to the object,
> 
> This breaks my expectations.  Could you explain that more?

The main aspect to have in mind is that your Lisp code has no direct
control on when a given Lisp object is no longer referenced by any
other live object.  And even if you take care to do that on the Lisp
level (something that is notoriously hard, if possible), GC also scans
the C stack for anything that looks like a reference to a Lisp object,
and marks any such objects as not (yet) collectible.  And your Lisp
program has no control on what's on the C stack.

> Does it mean that for example thunk-delay leaks even though it releases
> the reference to the body thunk?

If GC doesn't collect an object right away, it doesn't yet mean we
have a leak.  As long as the object is collected "eventually", we are
fine.

> Does it mean that setting reference to nil in order to dispose an object
> (as in my original example) is futile?

It doesn't necessarily guarantee GC will collect the object, yes.

> > and GC could have its own ideas when to actually garbage-collect an
> > object.
> 
> I don't think timing is an issue.  I think the expectation from gc is
> that it collects the object "eventually as needed".

That does happen, but you seem to expect more: you expect that the
object is collected as soon as you do something in Lisp to break the
binding between the object and the variable that was bound to it.

Re: using finalizers

[Rudolf Schlatte]

Tomas Hlavaty <tom@logand.com> writes:
>
> I can see these cases where garbage collection might not do its job:
>
> 1. imprecise gc
>
> 2. program exit or abort
>
> 3. a leak to be fixed

4. the gc is generational and the object is in an old generation

5. the gc is incremental and didn't get around to the object yet

... etc

I'm sure you knew this already, but in general, using gc for non-memory
resource management (e.g., "please close this file when this Lisp object
is GCed") is not a good idea--depending on the GC behavior, you'll run
out of file handles or whatnot.  The RAII pattern in C++
deterministically calls a destructor when a stack-allocated object goes
out of scope; in Lisp, the various `with-foo' macros serve the same
purpose.

Re: using finalizers

[LdBeth]

>>>>> In <878rw1pvcw.fsf@logand.com> 
>>>>>	Tomas Hlavaty <tom@logand.com> wrote:

Tomas> The interface is different from Common Lisp, where make-finalizer
Tomas> usually expects object and thunk, not thunk only but I guess the problem
Tomas> lies somewhere else.

If you mean you are looking for a finalizer interface similar to what
Common Lisp's trivial-garbarge library provides, there was one in
Emacs Lisp before `make-finalizer' was provided.

https://nullprogram.com/blog/2014/01/27/

Both that one and trivial-garbarge are based on weak references.
However that is actually not very good because that impose a lot of
jobs on GC, at least for Emacs the garbage collector is not as good as
many Common Lisp's ones.

Re: using finalizers

[Stefan Monnier]

Rudolf Schlatte [2021-12-31 15:23:32] wrote:

> Tomas Hlavaty <tom@logand.com> writes:
>>
>> I can see these cases where garbage collection might not do its job:
>>
>> 1. imprecise gc
>>
>> 2. program exit or abort
>>
>> 3. a leak to be fixed
>
> 4. the gc is generational and the object is in an old generation
>
> 5. the gc is incremental and didn't get around to the object yet
>
> ... etc
>
> I'm sure you knew this already, but in general, using gc for non-memory
> resource management (e.g., "please close this file when this Lisp object
> is GCed") is not a good idea--depending on the GC behavior, you'll run
> out of file handles or whatnot.  The RAII pattern in C++
> deterministically calls a destructor when a stack-allocated object goes
> out of scope; in Lisp, the various `with-foo' macros serve the same
> purpose.

But the context here is a "bug report" about a finalizer not being
called and it seemed pretty clear that the call to `garbage-collect` was
just there to try and make the bug more apparent, not because the real
ELisp code relies on the finalizer being called right at the first call
to `garbage-collect` after the object became unreachable.


        Stefan

Re: using finalizers

[Stefan Monnier]

> One more question: Because make-finalizer in Emacs Lisp does not take
> the object into account,

Not sure what you mean by "the object" here.

> I have to make sure to hold the reference to the finalizer somehow.
> However, this might not be trivial, because a smart compiler could
> eliminate the reference.

And I definitely don't know what you mean by "the reference", so I have
no idea why a smart compiler would be able to eliminate it.


        Stefan

Re: using finalizers

[Tomas Hlavaty]

On Fri 31 Dec 2021 at 11:21, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> Rudolf Schlatte [2021-12-31 15:23:32] wrote:
>
>> Tomas Hlavaty <tom@logand.com> writes:
>>>
>>> I can see these cases where garbage collection might not do its job:
>>>
>>> 1. imprecise gc
>>>
>>> 2. program exit or abort
>>>
>>> 3. a leak to be fixed
>>
>> 4. the gc is generational and the object is in an old generation
>>
>> 5. the gc is incremental and didn't get around to the object yet
>>
>> ... etc
>>
>> I'm sure you knew this already, but in general, using gc for non-memory
>> resource management (e.g., "please close this file when this Lisp object
>> is GCed") is not a good idea--depending on the GC behavior, you'll run
>> out of file handles or whatnot.  The RAII pattern in C++
>> deterministically calls a destructor when a stack-allocated object goes
>> out of scope; in Lisp, the various `with-foo' macros serve the same
>> purpose.
>
> But the context here is a "bug report" about a finalizer not being
> called and it seemed pretty clear that the call to `garbage-collect` was
> just there to try and make the bug more apparent, not because the real
> ELisp code relies on the finalizer being called right at the first call
> to `garbage-collect` after the object became unreachable.

yes, that was the original issue

Re: using finalizers

[Tomas Hlavaty]

On Fri 31 Dec 2021 at 14:27, Eli Zaretskii <eliz@gnu.org> wrote:
> That does happen, but you seem to expect more: you expect that the
> object is collected as soon as you do something in Lisp to break the
> binding between the object and the variable that was bound to it.

No, that's not what I expect.  I even said that.  What I meant is that I
did not expect some other code not under my control to keep references
to the object forever preventing gc to run the finalizer eventually,
which is how I understood your reply.  I know at least one Lisp which
does keeps extra unexpected references to particular kind of objects
preventing finalizers to run, so that kind of behaviour would not be
unheard of, but I see it as a bug and hope that Emacs does not have that
issue.

On another note, I see that finalizers are rarely used and tested:

./test/src/alloc-tests.el33:  (should (equal (type-of (make-finalizer nil)) 'finalizer)))
./lisp/emacs-lisp/generator.el642:                      (make-finalizer

For example the cps-generate-evaluator function has 70 lines and it is
not obvious to me, that the use of finalizer is not buggy.

Do you have an advice, how are programmers supposed to test finalizers?

My original issue did not run the finalizer because of a bug in my use
of letrec.  Unless there is a good way to actually trigger the
finalizer, there is no good way to test such code paths.  Or shall the
programmers just hope for the best?

Re: using finalizers

[Tomas Hlavaty]

On Fri 31 Dec 2021 at 22:39, LdBeth <andpuke@foxmail.com> wrote:
> If you mean you are looking for a finalizer interface similar to what
> Common Lisp's trivial-garbarge library provides, there was one in
> Emacs Lisp before `make-finalizer' was provided.
>
> https://nullprogram.com/blog/2014/01/27/
>
> Both that one and trivial-garbarge are based on weak references.
> However that is actually not very good because that impose a lot of
> jobs on GC, at least for Emacs the garbage collector is not as good as
> many Common Lisp's ones.

yes, that seems like the thing I was after
thanks for the material for further study

Re: using finalizers

[Eli Zaretskii]

> From: Tomas Hlavaty <tom@logand.com>
> Cc: andpuke@foxmail.com, emacs-devel@gnu.org
> Date: Sat, 01 Jan 2022 18:58:01 +0100
> 
> On Fri 31 Dec 2021 at 14:27, Eli Zaretskii <eliz@gnu.org> wrote:
> > That does happen, but you seem to expect more: you expect that the
> > object is collected as soon as you do something in Lisp to break the
> > binding between the object and the variable that was bound to it.
> 
> No, that's not what I expect.  I even said that.  What I meant is that I
> did not expect some other code not under my control to keep references
> to the object forever preventing gc to run the finalizer eventually,
> which is how I understood your reply.

Not "forever", of course, but depending on the program a reference to
an object could live for quite some time.  Having a reference on the C
stack is the most sinister of these.

> On another note, I see that finalizers are rarely used and tested:
> 
> ./test/src/alloc-tests.el33:  (should (equal (type-of (make-finalizer nil)) 'finalizer)))
> ./lisp/emacs-lisp/generator.el642:                      (make-finalizer
> 
> For example the cps-generate-evaluator function has 70 lines and it is
> not obvious to me, that the use of finalizer is not buggy.
> 
> Do you have an advice, how are programmers supposed to test finalizers?
> 
> My original issue did not run the finalizer because of a bug in my use
> of letrec.  Unless there is a good way to actually trigger the
> finalizer, there is no good way to test such code paths.  Or shall the
> programmers just hope for the best?

I hope Stefan could answer that, but it sounds like he doesn't
understand the issue you are describing (and neither do I, frankly).

Re: using finalizers

[Tomas Hlavaty]

On Sat 01 Jan 2022 at 12:00, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>> One more question: Because make-finalizer in Emacs Lisp does not take
>> the object into account,
>
> Not sure what you mean by "the object" here.

In Common Lisp, the usual semantics is: call this thunk when this object
gets collected.  I pass the object and thunk and make-finalizer does the
rest.

In Emacs Lisp, the semantics is: call this thunk when the returned
finalizer gets collected.  So, if I want to finalize my object, I have
too keep the reference to the finalizer.

>> I have to make sure to hold the reference to the finalizer somehow.
>> However, this might not be trivial, because a smart compiler could
>> eliminate the reference.
>
> And I definitely don't know what you mean by "the reference", so I have
> no idea why a smart compiler would be able to eliminate it.

In Emacs Lisp, my object itself needs to keep the finalizer around,
otherwise it would get finalized.

Re: using finalizers

[Stefan Monnier]

> In Emacs Lisp, the semantics is: call this thunk when the returned
> finalizer gets collected.  So, if I want to finalize my object, I have
> too keep the reference to the finalizer.

That's right: if you want an object to have a finalizer, you need to
save that finalizer somewhere inside the object.
IOW the object literally "has a finalizer" somewhere inside.

There's no doubt that it's different.
Do you have concrete cases where this difference introduces a difficulty?


        Stefan

Re: using finalizers

[Stefan Monnier]

>> No, that's not what I expect.  I even said that.  What I meant is that I
>> did not expect some other code not under my control to keep references
>> to the object forever preventing gc to run the finalizer eventually,
>> which is how I understood your reply.
>
> Not "forever", of course, but depending on the program a reference to
> an object could live for quite some time.  Having a reference on the C
> stack is the most sinister of these.

A common problem during manual experimentation is to have the offending
value stashed in `values`.

>> My original issue did not run the finalizer because of a bug in my use
>> of letrec.  Unless there is a good way to actually trigger the
>> finalizer, there is no good way to test such code paths.  Or shall the
>> programmers just hope for the best?

Calling `garbage-collect` should usually work well enough for tests
(tho this may have to be tweaked if/when we get a fancier GC), tho
there's always the risk that some bitpattern somewhere inside the stack
happens to look like a pointer to your finalizer (or an object that
transitively references it), so no I don't know of a way to *reliably*
get a finalizer to run.

You can minimize the risk of a bitpattern getting in the way by trying
to trigger a GC at a point where the C stack is as small as possible,
but it still can't rule out the problem completely.


        Stefan

Re: using finalizers

[Tomas Hlavaty]

On Fri 31 Dec 2021 at 15:23, Rudolf Schlatte <rudi@constantly.at> wrote:
> I'm sure you knew this already, but in general, using gc for non-memory
> resource management (e.g., "please close this file when this Lisp object
> is GCed") is not a good idea--depending on the GC behavior, you'll run
> out of file handles or whatnot.  The RAII pattern in C++
> deterministically calls a destructor when a stack-allocated object goes
> out of scope; in Lisp, the various `with-foo' macros serve the same
> purpose.

This is another topic.

Somehow it seems very controversial.  Some even violently oppose the
idea of using gc for non-memory resources but I have not found anybody
yet who would help me to understand real deep pros and cons and let me
form my own opinion without forcing his own.

Following stack discipline is great.  Most of the time.  But it comes
with severe restrictions and consequences.  There is a reason why gc was
invented.  For use-cases, which are not covered by with-foo stuff, one
can implement own ad-hoc resource management, or reuse gc.  Thus somehow
there seems to be a need for finalizers.

The negative consequences of following the stack discipline can be seen
in many places.

Lets have a look at the function directory-files.  It is likely the
worst possible API for accessing filesystem.  (Common Lisp has the same
flaw.)  It opens a directory, does its work and closes the directory.
Nice, it is done "properly": open, work, close.  The problem is, that
the amount of work done inside open/close is not under control of the
programmer and is not bound in space and time.  Additionally, the whole
work needs to be done completely, or completely aborted.  It means that
unless the amount of work is trivial, the whole thing and all the things
that are built on top of it are useless.  That sounds a bit extreme
claim.  Lets test it with something real: M-x find-lisp-find-dired
/nix/store [.]service$.  While Emacs blocked, no results seen.  C-g
after several seconds.  That was not very useful.  Lets try M-x
find-dired /nix/store -name '*.service'.  That works nicely.  Why?
Because the directory traversal runs in a second process (the control
loop pushes the results out of the process step by step) and the first
emacs process displays the results step by step as they come (with
directory entry granularity).  How could find-lisp-find-dired be fixed?
Could it be fixed while still using directory-files?  I do not think so.
I guess it can only be fixed by pushing directory entries out from
another thread (like find-dired does) or using a "stream" of directory
entries pulled on demand.  But when should close be called, when such
stream is lazy?  I am sure a reasonable ad-hoc solution could be
implemented.  But what if the directory is traversed recursively?  When
should each close be called?  A case for another ad-hoc solution?  Looks
like finalizers would be great here.

Yes there would be issues, like hitting the open file limit.  But are
those issues show stoppers?  Could there be a useful solution?  Maybe
run gc and try to open the file again?  Or something more sophisticated?
Have somebody explored this area?

Re: using finalizers

[Tomas Hlavaty]

On Sat 01 Jan 2022 at 15:47, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> That's right: if you want an object to have a finalizer, you need to
> save that finalizer somewhere inside the object.
> IOW the object literally "has a finalizer" somewhere inside.
>
> There's no doubt that it's different.
> Do you have concrete cases where this difference introduces a difficulty?

In my original example:

(defun finito ()
  (let ((x '(a b c)))
    (letrec ((more (make-finalizer close))
             (close (lambda ()
                      (print "close-resource")
                      (setq more nil))))
      (lambda ()
        (when more
          (pop x))))))

I use a closure as my object that should have the finalizer "attached"
to.  In order to keep the reference to the finalizer, I have to use it
inside the closure for something (here I stuffed it into the variable
called more).  That is inconvenient.  Second issue is that an advanced
compiler could eliminate even my attempt at using the finalizer for
something if it determines that the finalizer is not actually used for
something.

Re: using finalizers

[Tomas Hlavaty]

On Sat 01 Jan 2022 at 15:55, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>>> My original issue did not run the finalizer because of a bug in my use
>>> of letrec.  Unless there is a good way to actually trigger the
>>> finalizer, there is no good way to test such code paths.  Or shall the
>>> programmers just hope for the best?
>
> Calling `garbage-collect` should usually work well enough for tests

good, thanks

> (tho this may have to be tweaked if/when we get a fancier GC), tho
> there's always the risk that some bitpattern somewhere inside the stack
> happens to look like a pointer to your finalizer (or an object that
> transitively references it), so no I don't know of a way to *reliably*
> get a finalizer to run.
>
> You can minimize the risk of a bitpattern getting in the way by trying
> to trigger a GC at a point where the C stack is as small as possible,
> but it still can't rule out the problem completely.

What does "trigger a GC at a point where the C stack is as small as
possible" mean?  Does it mean trigger gc explicitly from top level?

Re: using finalizers

[Stefan Monnier]

> > Do you have concrete cases where this difference introduces a difficulty?
> In my original example:

I miswrote, by "concrete" I meant a real-word use, no a made-up case.
[ But at least, now I understand what you meant by the fact that the
  behavior depends on the byte-compiler's analysis.  ]

In any case, I don't think we have much experience with the current
`make-finalizer` API (nor with other finalizer APIs) in ELisp, so
I think there's room for tweaks/changes.  But it should be based
on concrete cases in real world ELisp code.


        Stefan

Re: using finalizers

[Stefan Monnier]

> What does "trigger a GC at a point where the C stack is as small as
> possible" mean?  Does it mean trigger gc explicitly from top level?

Something like that, yes.


        Stefan

Re: using finalizers

[Tomas Hlavaty]

On Sat 01 Jan 2022 at 23:55, Tomas Hlavaty <tom@logand.com> wrote:
> On Sat 01 Jan 2022 at 15:47, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>> That's right: if you want an object to have a finalizer, you need to
>> save that finalizer somewhere inside the object.
>> IOW the object literally "has a finalizer" somewhere inside.
>>
>> There's no doubt that it's different.
>> Do you have concrete cases where this difference introduces a difficulty?
>
> In my original example:
>
> (defun finito ()
>   (let ((x '(a b c)))
>     (letrec ((more (make-finalizer close))
>              (close (lambda ()
>                       (print "close-resource")
>                       (setq more nil))))
>       (lambda ()
>         (when more
>           (pop x))))))
>
> I use a closure as my object that should have the finalizer "attached"
> to.  In order to keep the reference to the finalizer, I have to use it
> inside the closure for something (here I stuffed it into the variable
> called more).  That is inconvenient.  Second issue is that an advanced
> compiler could eliminate even my attempt at using the finalizer for
> something if it determines that the finalizer is not actually used for
> something.

For comparison in Common Lisp, my object would not need to reference the
finalizer.  Something like this:

(defun finito ()
  (let* ((x '(a b c))
         (close (lambda () (print "close-resource")))
         (object (lambda () (pop x))))
    (finalize object close)
    object))

Re: using finalizers

[Tomas Hlavaty]

On Sat 01 Jan 2022 at 18:18, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>> > Do you have concrete cases where this difference introduces a
>> > difficulty?

Well, maybe not difficulty but inconvenience and potential future
difficulty.  I found make-finalizer surprising and inconvenient.
But LdBeth pointed out some interesting things I need to read about.

> I miswrote, by "concrete" I meant a real-word use, no a made-up case.

It was a reduced case of:

(defun directory-brook (directory)
  (let (d)
    (letrec ((close (lambda ()
                      (when d
                        (close-directory d)
                        (setq d nil
                              more nil))))
             (more (make-finalizer close)))
      (lambda ()
        (when more
          (unless d
            (setq d (open-directory directory)))
          (let (z (again t))
            (while again
              (setq z (read-directory d))
              (cond
               ((equal z "."))
               ((equal z ".."))
               (z (setq again nil))
               (t (setq again nil) (funcall close))))
            z))))))

There are bits missing but it should give you an idea.

> [ But at least, now I understand what you meant by the fact that the
>   behavior depends on the byte-compiler's analysis.  ]

Great, that was difficult for me to express.

> In any case, I don't think we have much experience with the current
> `make-finalizer` API (nor with other finalizer APIs) in ELisp, so
> I think there's room for tweaks/changes.  But it should be based
> on concrete cases in real world ELisp code.

Understand, thank you.

Re: using finalizers

[Eli Zaretskii]

> From: Tomas Hlavaty <tom@logand.com>
> Date: Sat, 01 Jan 2022 23:36:10 +0100
> 
> Lets have a look at the function directory-files.  It is likely the
> worst possible API for accessing filesystem.  (Common Lisp has the same
> flaw.)  It opens a directory, does its work and closes the directory.
> Nice, it is done "properly": open, work, close.  The problem is, that
> the amount of work done inside open/close is not under control of the
> programmer and is not bound in space and time.  Additionally, the whole
> work needs to be done completely, or completely aborted.  It means that
> unless the amount of work is trivial, the whole thing and all the things
> that are built on top of it are useless.  That sounds a bit extreme
> claim.  Lets test it with something real: M-x find-lisp-find-dired
> /nix/store [.]service$.  While Emacs blocked, no results seen.  C-g
> after several seconds.  That was not very useful.  Lets try M-x
> find-dired /nix/store -name '*.service'.  That works nicely.  Why?
> Because the directory traversal runs in a second process (the control
> loop pushes the results out of the process step by step) and the first
> emacs process displays the results step by step as they come (with
> directory entry granularity).  How could find-lisp-find-dired be fixed?
> Could it be fixed while still using directory-files?  I do not think so.
> I guess it can only be fixed by pushing directory entries out from
> another thread (like find-dired does) or using a "stream" of directory
> entries pulled on demand.

One common paradigm for such use cases is to use a callback: the
function that traverses some collection of objects calls that callback
for every object it finds.  There are usually protocols for the
callback to stop the process and prevent the rest of objects from
being examined/reported.

> But when should close be called, when such stream is lazy?  I am
> sure a reasonable ad-hoc solution could be implemented.  But what if
> the directory is traversed recursively?  When should each close be
> called?  A case for another ad-hoc solution?  Looks like finalizers
> would be great here.

We have the unwind-protect machinery in place for that.  In fact, if
you look at the implementation of directory-files, you will see that
it already uses that machinery, and it most probably actually kicked
in when you typed C-g above.

Re: using finalizers

[Tomas Hlavaty]

On Sun 02 Jan 2022 at 08:54, Eli Zaretskii <eliz@gnu.org> wrote:
> One common paradigm for such use cases is to use a callback: the
> function that traverses some collection of objects calls that callback
> for every object it finds.

There is no such thing in directory-files.

Also using callbacks in directory-files alone would not fix
find-lisp-find-dired.  I think an extra execution thread would be
needed.  Maybe using an extra execution thread could be seen as a work
around for the negative consequences of stack discipline.  If there was
a callback in directory-files, it would allow for pushing directory
items out of the execution thread when encountered, instead of waiting
until the contents of the whole directory has been collected.

> There are usually protocols for the callback to stop the process and
> prevent the rest of objects from being examined/reported.

There is no such thing in directory-files.

I can't see such thing in any other mapping functions in Emacs Lisp.
Could you point me to an example?

Also, such protocols are non-trivial and usually complicate the code
significantly.  See for example how people deal with it JavaScript,
where they have no other choice.

>> But when should close be called, when such stream is lazy?  I am
>> sure a reasonable ad-hoc solution could be implemented.  But what if
>> the directory is traversed recursively?  When should each close be
>> called?  A case for another ad-hoc solution?  Looks like finalizers
>> would be great here.
>
> We have the unwind-protect machinery in place for that.  In fact, if
> you look at the implementation of directory-files, you will see that
> it already uses that machinery, and it most probably actually kicked
> in when you typed C-g above.

I see I did not manage to get the point across.  directory-files is
"eager" and follows stack discipline so unwind-protect there makes
sense.  But the idea of directory-files itself is broken as I
demonstrated previously.  Operating systems get it right and provide
streaming API with open/read/close directory functions.  Also the
brokenness of directory-files propagates to many other points in Emacs
that need to list directories, like find-lisp-find-dired and completion.

But the point here was that in case of lazy sequences, the point in time
where close should be called is not known upfront.  One would have to
split the code into two parts: eager one which would force stack
discipline and lazy one which would do the lazy part.  Maybe doable for
trivial thing like lazy listing of a single directory, but it gets
harder when doing something non-trivial like lazy listing of a directory
recursively.

Re: using finalizers

[Eli Zaretskii]

> From: Tomas Hlavaty <tom@logand.com>
> Cc: rudi@constantly.at, emacs-devel@gnu.org
> Date: Sun, 02 Jan 2022 08:53:31 +0100
> 
> On Sun 02 Jan 2022 at 08:54, Eli Zaretskii <eliz@gnu.org> wrote:
> > One common paradigm for such use cases is to use a callback: the
> > function that traverses some collection of objects calls that callback
> > for every object it finds.
> 
> There is no such thing in directory-files.

It should be easy to add, if we wanted to.  Or we could implement a
separate set of primitives using that, if it were deemed useful.

I fail to see how what's in Emacs now is of any essence for the
purposes of this discussion.  What exactly are you getting at, and how
is what is in Emacs today relevant?  Are you saying that Emacs
couldn't possibly provide such primitives due to some inherent
limitation?  If so, please explain why you think so, because I don't
think I agree.

> Also using callbacks in directory-files alone would not fix
> find-lisp-find-dired.

You'd need a different implementation of that, of course.  Other than
that, what is your point?

> I think an extra execution thread would be needed.

I don't see why.  A callback can easily insert files into the buffer,
and trigger redisplay either directly or via timers and such.  Keep in
mind that when Emacs invokes an async subprocess, the display of what
that subprocess produces is still done in the same single thread where
the Lisp machine runs.  So we already have the capabilities of
"pretending" we have multithreading, when these aspects are involved.

> > There are usually protocols for the callback to stop the process and
> > prevent the rest of objects from being examined/reported.
> 
> There is no such thing in directory-files.

I think we have something like that in lisp/url/.

> Also, such protocols are non-trivial and usually complicate the code
> significantly.

I don't see why this would be non-trivial.  In the simplest case, the
callback returns a special value which tells to stop the iteration.

> > We have the unwind-protect machinery in place for that.  In fact, if
> > you look at the implementation of directory-files, you will see that
> > it already uses that machinery, and it most probably actually kicked
> > in when you typed C-g above.
> 
> I see I did not manage to get the point across.  directory-files is
> "eager" and follows stack discipline so unwind-protect there makes
> sense.  But the idea of directory-files itself is broken as I
> demonstrated previously.  Operating systems get it right and provide
> streaming API with open/read/close directory functions.  Also the
> brokenness of directory-files propagates to many other points in Emacs
> that need to list directories, like find-lisp-find-dired and completion.

Emacs is not an OS (the popular myth about that notwithstanding).  It
provides primitives that are useful for writing editing and
text-processing applications.  What you don't find in Emacs is
probably outside of its scope, or at least is rarely if ever needed.

> But the point here was that in case of lazy sequences, the point in time
> where close should be called is not known upfront.  One would have to
> split the code into two parts: eager one which would force stack
> discipline and lazy one which would do the lazy part.  Maybe doable for
> trivial thing like lazy listing of a single directory, but it gets
> harder when doing something non-trivial like lazy listing of a directory
> recursively.

Where you see something non-trivial, I see just simple use of our
unwinding infrastructure.  There's no problem I see with using that
infrastructure in recursive processing of directories.  If you see
such problems, please tell what they are, specifically, given the
unwind_protect functions we have and the mechanism that actually
implements the unwinding.  In Emacs, many operations could signal an
error or throw, and the user could type C-g at any moment, so this
mechanism is in use all the time, and must handle correctly nested
unwinders.

Re: using finalizers

[Tomas Hlavaty]

On Sun 02 Jan 2022 at 10:33, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Tomas Hlavaty <tom@logand.com>
>> Cc: rudi@constantly.at, emacs-devel@gnu.org
>> Date: Sun, 02 Jan 2022 08:53:31 +0100
>> 
>> On Sun 02 Jan 2022 at 08:54, Eli Zaretskii <eliz@gnu.org> wrote:
>> > One common paradigm for such use cases is to use a callback: the
>> > function that traverses some collection of objects calls that callback
>> > for every object it finds.
>> 
>> There is no such thing in directory-files.
>
> It should be easy to add, if we wanted to.  Or we could implement a
> separate set of primitives using that, if it were deemed useful.
>
> I fail to see how what's in Emacs now is of any essence for the
> purposes of this discussion.  What exactly are you getting at, and how
> is what is in Emacs today relevant?  Are you saying that Emacs
> couldn't possibly provide such primitives due to some inherent
> limitation?  If so, please explain why you think so, because I don't
> think I agree.

I was replying to Rudolf about the use-cases where stack discipline can
have visible negative consequences and demonstrating it with a few
examples in Emacs.  It is also related to the use of finalizers as an
alternative mechanism to stack discipline.

I am saying that there are things that could be improved:

- Emacs Lisp should provide open-directory, read-directory and
  close-directory functions.

I think that is the most important point.

There are other related observations, like:

- find-lisp-find-dired is fundamentally broken.

- the brokenness is due to directory-files being the wrong API.
  It prevents people from writing better code.

- the brokenness propagates to everything which uses directory-files,
  like find-lisp-find-dired, completion (e.g. completion on /nix/store
  blocks emacs completely for significant amount of time).  Now I see
  that even lisp/url uses directory-files thus there are cases where it
  will be unuseable.

- directory-files implies consing all the directory entries except a few
  pre-hardcoded filter use-cases.  It does not allow anything else to be
  more efficient, where one would want to filter out unneeded directory
  entries at the moment they are first encountered.

- return_count argument to directory_files_internal shows a desperate
  attempt to fix patological deficiency:-) return_count is missing from
  directory-files-recursively.  return_count still does not provide
  sufficient control of previously uncontrollable behaviour.

- directory-files-recursively brings yet another different interface for
  traversing file system

- directory-files-recursively seems to be built on different primitive
  file-name-all-completions

I do not understand the implementation of file-name-all-completions yet
but from the observed behaviour, it seems to have the same flaws like
directory-files.

> I don't see why.  A callback can easily insert files into the buffer,
> and trigger redisplay either directly or via timers and such.

But the control is in the hand of the mapping function.

Triggering redisplay is not sufficient for it to behave like in
find-dired, I think.  It needs to handle input and everything as if it
was running in a different execution thread, like find-dired does with
the "mapping function" running in an external process (find program).

> Keep in mind that when Emacs invokes an async subprocess, the display
> of what that subprocess produces is still done in the same single
> thread where the Lisp machine runs.  So we already have the
> capabilities of "pretending" we have multithreading, when these
> aspects are involved.

Yes, it works well when the mapping function (in the case of find-dired,
the find process) runs in a different execution thread/process.

It does not work well, when it runs in the same single execution thread.
See find-lisp-find-dired.

>> > There are usually protocols for the callback to stop the process and
>> > prevent the rest of objects from being examined/reported.
>> 
>> There is no such thing in directory-files.
>
> I think we have something like that in lisp/url/.

Thanks for the hint.  Unfortunatelly I can't find anything from after a
quick search.

I can see that it uses directory-files which means it also inherits its
brokenness in those cases.

>> Also, such protocols are non-trivial and usually complicate the code
>> significantly.
>
> I don't see why this would be non-trivial.  In the simplest case, the
> callback returns a special value which tells to stop the iteration.

The simplest case is also the least interesting.

> Emacs is not an OS (the popular myth about that notwithstanding).  It
> provides primitives that are useful for writing editing and
> text-processing applications.  What you don't find in Emacs is
> probably outside of its scope, or at least is rarely if ever needed.

Directory traversal seems to be essential part of Emacs.

>> But the point here was that in case of lazy sequences, the point in time
>> where close should be called is not known upfront.  One would have to
>> split the code into two parts: eager one which would force stack
>> discipline and lazy one which would do the lazy part.  Maybe doable for
>> trivial thing like lazy listing of a single directory, but it gets
>> harder when doing something non-trivial like lazy listing of a directory
>> recursively.
>
> Where you see something non-trivial, I see just simple use of our
> unwinding infrastructure.  There's no problem I see with using that
> infrastructure in recursive processing of directories.  If you see
> such problems, please tell what they are, specifically, given the
> unwind_protect functions we have and the mechanism that actually
> implements the unwinding.  In Emacs, many operations could signal an
> error or throw, and the user could type C-g at any moment, so this
> mechanism is in use all the time, and must handle correctly nested
> unwinders.

Problem is not with unwind-protect.

Problem is with directory-files and all functions that use it.  I think
I pointed out the problems sufficiently.

One fix could be implementing an alternative to directory-files,
e.g. directory mapping function, plus running it in a extra thread.

Another possibility would be to traverse the directory lazily, but that
brings an issue of eagerly closing directories.

Re: using finalizers

[Eli Zaretskii]

> From: Tomas Hlavaty <tom@logand.com>
> Cc: rudi@constantly.at, emacs-devel@gnu.org
> Date: Sun, 02 Jan 2022 14:10:02 +0100
> 
> - Emacs Lisp should provide open-directory, read-directory and
>   close-directory functions.

If real-life use cases are brought up that cannot be adequately
handled by the existing APIs, we will consider extending them or
adding new APIs.  But the use cases need to come first.

> > I don't see why.  A callback can easily insert files into the buffer,
> > and trigger redisplay either directly or via timers and such.
> 
> But the control is in the hand of the mapping function.

No, the callback can do that under its own control.

> Triggering redisplay is not sufficient for it to behave like in
> find-dired, I think.  It needs to handle input and everything as if it
> was running in a different execution thread, like find-dired does with
> the "mapping function" running in an external process (find program).
> 
> > Keep in mind that when Emacs invokes an async subprocess, the display
> > of what that subprocess produces is still done in the same single
> > thread where the Lisp machine runs.  So we already have the
> > capabilities of "pretending" we have multithreading, when these
> > aspects are involved.
> 
> Yes, it works well when the mapping function (in the case of find-dired,
> the find process) runs in a different execution thread/process.
> 
> It does not work well, when it runs in the same single execution thread.

Like I said: output from the sub-process is processed and displayed by
the same single-thread, without any help from additional threads.  the
subprocess produces output, but it is only displayed by Emacs when we
read that output and process it, which is done by the same single
thread which you say cannot do the job.  So the same can be done
without sub-processes.  If needed, of course.

> See find-lisp-find-dired.

Not relevant: that function was not designed provide output during the
search, so it doesn't.

> >> > There are usually protocols for the callback to stop the process and
> >> > prevent the rest of objects from being examined/reported.
> >> 
> >> There is no such thing in directory-files.
> >
> > I think we have something like that in lisp/url/.
> 
> Thanks for the hint.  Unfortunatelly I can't find anything from after a
> quick search.

Grep for "callback".

> > I don't see why this would be non-trivial.  In the simplest case, the
> > callback returns a special value which tells to stop the iteration.
> 
> The simplest case is also the least interesting.

If it does the job, why isn't it interesting?

> > Emacs is not an OS (the popular myth about that notwithstanding).  It
> > provides primitives that are useful for writing editing and
> > text-processing applications.  What you don't find in Emacs is
> > probably outside of its scope, or at least is rarely if ever needed.
> 
> Directory traversal seems to be essential part of Emacs.

And it does work.

> > Where you see something non-trivial, I see just simple use of our
> > unwinding infrastructure.  There's no problem I see with using that
> > infrastructure in recursive processing of directories.  If you see
> > such problems, please tell what they are, specifically, given the
> > unwind_protect functions we have and the mechanism that actually
> > implements the unwinding.  In Emacs, many operations could signal an
> > error or throw, and the user could type C-g at any moment, so this
> > mechanism is in use all the time, and must handle correctly nested
> > unwinders.
> 
> Problem is not with unwind-protect.

I didn't mean unwind-protect (in Lisp), I meant unwind_protect (in C).
We are talking about primitives that should be implemented in C, yes?

> Problem is with directory-files and all functions that use it.  I think
> I pointed out the problems sufficiently.

And I think I explained why I think your assessment is mistaken.

> One fix could be implementing an alternative to directory-files,
> e.g. directory mapping function, plus running it in a extra thread.

You are welcome to try that, but I don't think it will work in Emacs,
not with the Lisp threads we have.

> Another possibility would be to traverse the directory lazily, but that
> brings an issue of eagerly closing directories.

See above: I see no issue.  They could/should be closed in
unwind_protect functions.

Re: using finalizers

[Eli Zaretskii]

> Date: Sun, 02 Jan 2022 16:42:45 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: rudi@constantly.at, emacs-devel@gnu.org
> 
> > >> > There are usually protocols for the callback to stop the process and
> > >> > prevent the rest of objects from being examined/reported.
> > >> 
> > >> There is no such thing in directory-files.
> > >
> > > I think we have something like that in lisp/url/.
> > 
> > Thanks for the hint.  Unfortunatelly I can't find anything from after a
> > quick search.
> 
> Grep for "callback".

Come to think of it, calling hooks via
run-hook-with-args-until-failure or run-hook-with-args-until-success
implements such a protocol.

Re: using finalizers

[Tomas Hlavaty]

On Sun 02 Jan 2022 at 17:16, Eli Zaretskii <eliz@gnu.org> wrote:
>> Grep for "callback".
>
> Come to think of it, calling hooks via
> run-hook-with-args-until-failure or run-hook-with-args-until-success
> implements such a protocol.

thank you, I'll have a look

Re: using finalizers

[Rudolf Schlatte]

Tomas Hlavaty <tom@logand.com> writes:

> On Fri 31 Dec 2021 at 15:23, Rudolf Schlatte <rudi@constantly.at> wrote:
>> I'm sure you knew this already, but in general, using gc for non-memory
>> resource management (e.g., "please close this file when this Lisp object
>> is GCed") is not a good idea--depending on the GC behavior, you'll run
>> out of file handles or whatnot.  The RAII pattern in C++
>> deterministically calls a destructor when a stack-allocated object goes
>> out of scope; in Lisp, the various `with-foo' macros serve the same
>> purpose.
>
> This is another topic.
>
> Somehow it seems very controversial.  Some even violently oppose the
> idea of using gc for non-memory resources but I have not found anybody
> yet who would help me to understand real deep pros and cons and let me
> form my own opinion without forcing his own.

I apologize for derailing the thread a bit, and acknowledge that I did
not really engage with your concrete problem (this was also pointed out
by Stefan M, thanks for this).  That being said, I'll try to express, in
one exaggerated sentence, the deep cons of using gc for non-memory
resources in the hope that it helps a bit:

”If you rely on the garbage collector to release non-memory resources as
a side effect of reclaiming memory, you will run out of resources if you
increase the main memory of your system.”

Re: using finalizers

[Tomas Hlavaty]

On Sun 02 Jan 2022 at 19:11, Rudolf Schlatte <rudi@constantly.at> wrote:
> That being said, I'll try to express, in one exaggerated sentence, the
> deep cons of using gc for non-memory resources in the hope that it
> helps a bit:
>
> ”If you rely on the garbage collector to release non-memory resources as
> a side effect of reclaiming memory, you will run out of resources if you
> increase the main memory of your system.”

Is that your sentence?  Or could you point me to the source for further
reading?  I agree that the huge inbalance between the amount of
available memory and the amount of available file descriptors is a
problem.  But the sentence does not explain the details.  Is it the
consequence of Lisp running on a hosted operating system with those
constraints?  Did for example lisp machines have the same constraints?
Why?  Could EMFILE and ENFILE trigger gc and then retry open?  sbcl does
finalize file descriptors and iirc some schemes close them automatically
too.

Re: using finalizers

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > ”If you rely on the garbage collector to release non-memory resources as
  > > a side effect of reclaiming memory, you will run out of resources if you
  > > increase the main memory of your system.”

That would be true if the criterion for when to GC is that "memory is
running low".  Increase the total memory size, and it won't run low
very often, so GC will happen less often.

But in Emacs GC is triggered by allocating a certain amount of memory.
Thus, the thing that would hold the non-memory resources too long
is increasing gc-cons-threshold.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)