Re: MPS: out-of-memory

[Pip Cet <pipcet@protonmail.com>]

On Monday, July 8th, 2024 at 18:26, Gerd Möllmann <gerd.moellmann@gmail.com> wrote:
> Pip Cet pipcet@protonmail.com writes:
> 
> > > Yes, I think that's important, and also handling the case of getting
> > > errors codes from MPS in general.
> > 
> > I think we might want to distinguish the two cases: MPS might die,
> > sure, but it looks like we need an out-of-memory mechanism separate
> > from MPS to avoid assertion violations.
> 
> Could you please elaborate which assertions these are?

I meant the MPS assertion which fails when we run out of memory and there's a commit limit. As that appears to happen independently of how the commit limit was set, I don't think we can have a commit limit at this point.

> > > As one can see, igc as a whole is in a certain state, igc_state. The
> > > idea is that the state changes to IGC_STATE_DEAD when something fatal
> > > happens. In that state, malloc is used is used for allocating Lisp
> > > objects instead of MPS. That lets Emacs shut down gracefully without
> > > entering MPS recursively as it was before I added the state.
> > 
> > I've seen that work, and I've seen it not work. Better than nothing
> > :-)
> 
> Right, if you have such a case that I can reproduce, please holler :-)

Sure. What I think I've seen is my code misbehaving and MPS aborting, while leaving memory in an unusable state. I'm not sure that can be avoided, to be honest.

> > > Such state changes are currently done when checking the return code of
> > > an MPS API function and when assertions fail.
> > > 
> > > Seems to work as expected, as I could see in a couple of backtraces from
> > > Ihor I believe showing the set_state (IGC_STATE_DEAD), and from my own
> > > experience. But maybe someone has an idea how to improve it.
> > 
> > If I understood Helmut correctly, he wants a mechanism to avoid
> > thrashing after exceeding the memory limit. Maybe we need a special
> > state for that, in which we stop the GC but continue using our memory
> > while the user quits and goes to buy more RAM?
> 
> Could be. One could for example allocate a block from MPS that we make
> available to MPS as a first aid. I think that would be relatiely easy.

I know that people have different opinions on swap space, particularly in these days of SSDs which can be worn out easily. I'm not sure there's a good way to detect thrashing even when using the Linux kernel, nevermind across all the OSes we want to support.

> Using xzalloc in IGC_STATE_DEAD was just easiest to implement for me. I
> was mainly trying to make the error handling in igc understandable. I
> think introducing the states was not too bad for that. Also for that
> non-error state because of the staticpros, which becomes much clearer
> that way, I believe. Anyway.

I think it's great! I'd just like to extend it by one state so I don't have to go out and buy a new SSD when I mess up :-)

> > Speaking of memory issues in general, I'm currently seeing pure space
> > overflows, after changes which should affect weak objects exclusively.
> > Which is really odd, because weak objects aren't purecopied! Still
> > investigating that one, but it's possible I'll just give up and bump
> > pure space.
> > 
> > In any case, we end up segfaulting because we still try to run GC
> > after pure space overflowed. I'll push a fix for that, but I can't
> > promise to test the case of overflowing pure space very much.
> 
> Once more time glad that I don't have pure space here :-).

Maybe we could at least drop it for MPS builds? :-)  IIRC the arguments for keeping it were mostly about GC performance.

> (I think I've picked what I want to play with next today, BTW.
> https://github.com/actor-framework/actor-framework :-)

Cool, I'll try not to break the branch too much while you're distracted by something else. No promises though.

Pip