unofficial mirror of emacs-devel@gnu.org 
 help / color / mirror / code / Atom feed
From: Pip Cet <pipcet@protonmail.com>
To: "Gerd Möllmann" <gerd.moellmann@gmail.com>
Cc: Eli Zaretskii <eliz@gnu.org>,
	acorallo@gnu.org, eller.helmut@gmail.com, emacs-devel@gnu.org
Subject: Re: MPS: out-of-memory
Date: Mon, 08 Jul 2024 19:44:21 +0000	[thread overview]
Message-ID: <Y0tbYudujKGHq0HEAbyM-44XBcnbBLEqh2Y19UVcHXu7hbQoa1GKvdXEgzxxM8gTBOJsD2I_mNryRhdPdCcFd1JuM2aseScThKl8kHaqpIw=@protonmail.com> (raw)
In-Reply-To: <m2ikxfwxhr.fsf@pro2.fritz.box>

On Monday, July 8th, 2024 at 18:26, Gerd Möllmann <gerd.moellmann@gmail.com> wrote:
> Pip Cet pipcet@protonmail.com writes:
> 
> > > Yes, I think that's important, and also handling the case of getting
> > > errors codes from MPS in general.
> > 
> > I think we might want to distinguish the two cases: MPS might die,
> > sure, but it looks like we need an out-of-memory mechanism separate
> > from MPS to avoid assertion violations.
> 
> Could you please elaborate which assertions these are?

I meant the MPS assertion which fails when we run out of memory and there's a commit limit. As that appears to happen independently of how the commit limit was set, I don't think we can have a commit limit at this point.

> > > As one can see, igc as a whole is in a certain state, igc_state. The
> > > idea is that the state changes to IGC_STATE_DEAD when something fatal
> > > happens. In that state, malloc is used is used for allocating Lisp
> > > objects instead of MPS. That lets Emacs shut down gracefully without
> > > entering MPS recursively as it was before I added the state.
> > 
> > I've seen that work, and I've seen it not work. Better than nothing
> > :-)
> 
> Right, if you have such a case that I can reproduce, please holler :-)

Sure. What I think I've seen is my code misbehaving and MPS aborting, while leaving memory in an unusable state. I'm not sure that can be avoided, to be honest.

> > > Such state changes are currently done when checking the return code of
> > > an MPS API function and when assertions fail.
> > > 
> > > Seems to work as expected, as I could see in a couple of backtraces from
> > > Ihor I believe showing the set_state (IGC_STATE_DEAD), and from my own
> > > experience. But maybe someone has an idea how to improve it.
> > 
> > If I understood Helmut correctly, he wants a mechanism to avoid
> > thrashing after exceeding the memory limit. Maybe we need a special
> > state for that, in which we stop the GC but continue using our memory
> > while the user quits and goes to buy more RAM?
> 
> Could be. One could for example allocate a block from MPS that we make
> available to MPS as a first aid. I think that would be relatiely easy.

I know that people have different opinions on swap space, particularly in these days of SSDs which can be worn out easily. I'm not sure there's a good way to detect thrashing even when using the Linux kernel, nevermind across all the OSes we want to support.

> Using xzalloc in IGC_STATE_DEAD was just easiest to implement for me. I
> was mainly trying to make the error handling in igc understandable. I
> think introducing the states was not too bad for that. Also for that
> non-error state because of the staticpros, which becomes much clearer
> that way, I believe. Anyway.

I think it's great! I'd just like to extend it by one state so I don't have to go out and buy a new SSD when I mess up :-)

> > Speaking of memory issues in general, I'm currently seeing pure space
> > overflows, after changes which should affect weak objects exclusively.
> > Which is really odd, because weak objects aren't purecopied! Still
> > investigating that one, but it's possible I'll just give up and bump
> > pure space.
> > 
> > In any case, we end up segfaulting because we still try to run GC
> > after pure space overflowed. I'll push a fix for that, but I can't
> > promise to test the case of overflowing pure space very much.
> 
> Once more time glad that I don't have pure space here :-).

Maybe we could at least drop it for MPS builds? :-)  IIRC the arguments for keeping it were mostly about GC performance.

> (I think I've picked what I want to play with next today, BTW.
> https://github.com/actor-framework/actor-framework :-)

Cool, I'll try not to break the branch too much while you're distracted by something else. No promises though.

Pip



  reply	other threads:[~2024-07-08 19:44 UTC|newest]

Thread overview: 196+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2024-07-01 20:47 MPS: weak hash tables Gerd Möllmann
2024-07-01 21:16 ` Pip Cet
2024-07-01 23:10   ` Pip Cet
2024-07-02  4:19     ` Gerd Möllmann
2024-07-02  5:47     ` Gerd Möllmann
2024-07-02  6:23       ` Pip Cet
2024-07-02  6:55         ` Gerd Möllmann
2024-07-02  9:15           ` Pip Cet
2024-07-02  9:37             ` Gerd Möllmann
2024-07-02 10:11               ` Gerd Möllmann
2024-07-02 11:36               ` Gerd Möllmann
2024-07-02 13:15                 ` Eli Zaretskii
2024-07-02 13:16                   ` Gerd Möllmann
2024-07-02 13:42                     ` Eli Zaretskii
2024-07-02 15:03                       ` Pip Cet
2024-07-02 15:17                         ` Helmut Eller
2024-07-02 15:35                         ` Eli Zaretskii
2024-07-02 16:34                           ` Pip Cet
2024-07-02 18:20                             ` Eli Zaretskii
2024-07-02 20:16                               ` Pip Cet
2024-07-03  6:30                               ` Gerd Möllmann
2024-07-03 11:23                                 ` Eli Zaretskii
2024-07-03 11:28                                   ` Gerd Möllmann
2024-07-02 13:02             ` Eli Zaretskii
2024-07-02 12:45           ` Eli Zaretskii
2024-07-02 11:23         ` Helmut Eller
2024-07-03  6:11           ` Gerd Möllmann
2024-07-03  6:33             ` Pip Cet
2024-07-03  7:04               ` Gerd Möllmann
2024-07-03  7:24                 ` Helmut Eller
2024-07-03  7:25                 ` Pip Cet
2024-07-03  7:38                   ` Gerd Möllmann
2024-07-03  8:26                     ` Gerd Möllmann
2024-07-03  9:31                       ` Pip Cet
2024-07-03 10:22                         ` Gerd Möllmann
2024-07-03 10:41                           ` Pip Cet
2024-07-03 11:17                             ` Gerd Möllmann
2024-07-03 20:20                           ` Pip Cet
2024-07-04  7:17                             ` Gerd Möllmann
2024-07-04 15:24                               ` Pip Cet
2024-07-04 16:53                                 ` Gerd Möllmann
2024-07-04 20:05                                   ` Pip Cet
2024-07-05  3:50                                     ` Gerd Möllmann
2024-07-05 12:08                                       ` Pip Cet
2024-07-05 12:54                                         ` Gerd Möllmann
2024-07-05 13:27                                         ` Eli Zaretskii
2024-07-05 20:35                                           ` Pip Cet
2024-07-06  6:10                                             ` Eli Zaretskii
2024-07-06  6:31                                               ` Pip Cet
2024-07-06  7:00                                                 ` Eli Zaretskii
2024-07-06  7:40                                                   ` Gerd Möllmann
2024-07-06  9:13                                                   ` Pip Cet
2024-07-06 10:59                                                     ` Eli Zaretskii
2024-07-05 18:14                                         ` Helmut Eller
2024-07-05 19:25                                           ` Pip Cet
2024-07-06  3:39                                             ` Gerd Möllmann
2024-07-06  5:58                                               ` Pip Cet
2024-07-06  6:20                                                 ` Gerd Möllmann
2024-07-06  6:29                                                   ` Pip Cet
2024-07-06  6:51                                                     ` Gerd Möllmann
2024-07-06  6:46                                                 ` Eli Zaretskii
2024-07-06  9:23                                                   ` Pip Cet
2024-07-06 11:03                                                     ` Eli Zaretskii
2024-07-06  3:38                                           ` Gerd Möllmann
2024-07-06  9:47                                             ` Helmut Eller
2024-07-06 10:38                                               ` Gerd Möllmann
2024-07-06 11:13                                               ` Eli Zaretskii
2024-07-06 13:50                                                 ` Helmut Eller
2024-07-06 13:59                                                   ` Eli Zaretskii
2024-07-06 14:38                                                     ` Gerd Möllmann
2024-07-06 16:20                                                     ` Helmut Eller
2024-07-06 16:33                                                       ` Eli Zaretskii
2024-07-06 16:48                                                         ` Helmut Eller
2024-07-06 17:21                                                           ` Eli Zaretskii
2024-07-06 17:59                                                             ` Helmut Eller
2024-07-06 18:14                                                             ` Gerd Möllmann
2024-07-06 18:56                                                               ` Eli Zaretskii
2024-07-06 11:37                                               ` Pip Cet
2024-07-06 11:40                                               ` Gerd Möllmann
2024-07-06 11:57                                               ` Gerd Möllmann
2024-07-06 12:03                                                 ` Eli Zaretskii
2024-07-06 12:16                                                   ` Gerd Möllmann
2024-07-06 12:23                                                     ` Pip Cet
2024-07-06 12:39                                                       ` Gerd Möllmann
2024-07-06 12:30                                                     ` Eli Zaretskii
2024-07-06 12:43                                                       ` Gerd Möllmann
2024-07-06 13:53                                                         ` Eli Zaretskii
2024-07-06 12:36                                                     ` Gerd Möllmann
2024-07-06 14:00                                                       ` Helmut Eller
2024-07-06 14:08                                                         ` Gerd Möllmann
2024-07-06 14:24                                                           ` Gerd Möllmann
2024-07-06 14:44                                                           ` Helmut Eller
2024-07-06 14:52                                                             ` Gerd Möllmann
2024-07-06 15:49                                                               ` Pip Cet
2024-07-06 16:31                                                                 ` Gerd Möllmann
2024-07-06 16:56                                                                   ` Pip Cet
2024-07-06 17:28                                                                     ` Gerd Möllmann
2024-07-06 17:31                                                                       ` Gerd Möllmann
2024-07-06 18:30                                                                       ` Pip Cet
2024-07-06 20:00                                                                         ` Gerd Möllmann
2024-07-06 20:09                                                                           ` Ihor Radchenko
2024-07-07  3:55                                                                             ` Gerd Möllmann
2024-07-07  4:27                                                                             ` Gerd Möllmann
2024-07-07  4:30                                                                               ` Gerd Möllmann
2024-07-07  6:38                                                                               ` Pip Cet
2024-07-07  7:31                                                                                 ` Gerd Möllmann
2024-07-07  7:44                                                                                 ` Helmut Eller
2024-07-07  8:10                                                                                   ` Gerd Möllmann
2024-07-07  8:24                                                                                     ` Gerd Möllmann
2024-07-07  8:47                                                                                       ` Pip Cet
2024-07-07  9:24                                                                                         ` Gerd Möllmann
2024-07-07  9:26                                                                                           ` Gerd Möllmann
2024-07-07 10:47                                                                                           ` Eli Zaretskii
2024-07-07 11:19                                                                                             ` Gerd Möllmann
2024-07-07 14:09                                                                                               ` Eli Zaretskii
2024-07-07 14:15                                                                                                 ` Gerd Möllmann
2024-07-07 14:42                                                                                                   ` Eli Zaretskii
2024-07-07 14:52                                                                                                     ` Gerd Möllmann
2024-07-07 15:34                                                                                                       ` Eli Zaretskii
2024-07-07 15:36                                                                                                         ` Gerd Möllmann
2024-07-07 16:00                                                                                                           ` Eli Zaretskii
2024-07-07 17:08                                                                                                             ` Gerd Möllmann
2024-07-07 17:49                                                                                                               ` Eli Zaretskii
2024-07-07 18:15                                                                                                                 ` Gerd Möllmann
2024-07-07 18:22                                                                                                                   ` Eli Zaretskii
2024-07-07 18:29                                                                                                                     ` Gerd Möllmann
2024-07-07 14:16                                                                                                 ` Gerd Möllmann
2024-07-07 14:18                                                                                                   ` Gerd Möllmann
2024-07-07 10:57                                                                                           ` Pip Cet
2024-07-07 11:35                                                                                             ` Gerd Möllmann
2024-07-07 11:48                                                                                               ` Gerd Möllmann
2024-07-07 14:07                                                                                                 ` Gerd Möllmann
2024-07-07 14:21                                                                                                 ` Pip Cet
2024-07-07 14:27                                                                                                   ` Gerd Möllmann
2024-07-07 15:22                                                                                                   ` Helmut Eller
2024-07-07 15:40                                                                                                     ` Gerd Möllmann
2024-07-07 15:52                                                                                                       ` Helmut Eller
2024-07-07 15:56                                                                                                         ` Gerd Möllmann
2024-07-07 15:57                                                                                                         ` Pip Cet
2024-07-07 16:26                                                                                                           ` Helmut Eller
2024-07-07 17:03                                                                                                             ` Gerd Möllmann
2024-07-07 18:40                                                                                                               ` Gerd Möllmann
2024-07-07 18:53                                                                                                                 ` Helmut Eller
2024-07-07 19:00                                                                                                                   ` Gerd Möllmann
2024-07-07 19:31                                                                                                                     ` Pip Cet
2024-07-07 19:36                                                                                                                       ` Gerd Möllmann
2024-07-08  9:11                                                                                                                     ` MPS: commit limit Gerd Möllmann
2024-07-10  6:46                                                                                                                       ` Helmut Eller
2024-07-10  7:08                                                                                                                         ` Gerd Möllmann
2024-07-16 15:16                                                                                                                       ` Helmut Eller
2024-07-16 15:27                                                                                                                         ` Eli Zaretskii
2024-07-16 15:43                                                                                                                           ` Helmut Eller
2024-07-16 15:54                                                                                                                             ` Eli Zaretskii
2024-07-16 16:29                                                                                                                               ` Helmut Eller
2024-07-16 16:39                                                                                                                                 ` Gerd Möllmann
2024-07-16 16:43                                                                                                                                   ` Pip Cet
2024-07-16 16:56                                                                                                                                     ` Gerd Möllmann
2024-07-16 15:32                                                                                                                         ` Eli Zaretskii
2024-07-16 16:27                                                                                                                           ` Helmut Eller
2024-07-16 18:49                                                                                                                             ` Pip Cet
2024-07-17  6:15                                                                                                                               ` Helmut Eller
2024-07-16 16:32                                                                                                                         ` Pip Cet
2024-07-16 16:48                                                                                                                           ` Helmut Eller
2024-07-08  5:11                                                                                                             ` MPS: weak hash tables Pip Cet
2024-07-08  5:17                                                                                                               ` Gerd Möllmann
2024-07-08  5:37                                                                                                                 ` Pip Cet
2024-07-08  5:43                                                                                                                   ` Gerd Möllmann
2024-07-07  8:49                                                                                       ` Gerd Möllmann
2024-07-08  9:16                                                         ` Andrea Corallo
2024-07-08  9:24                                                           ` Gerd Möllmann
2024-07-08  9:54                                                             ` Andrea Corallo
2024-07-08 10:10                                                               ` Gerd Möllmann
2024-07-08 11:57                                                             ` MPS: out-of-memory Eli Zaretskii
2024-07-08 13:46                                                               ` Gerd Möllmann
2024-07-08 16:45                                                                 ` Pip Cet
2024-07-08 18:26                                                                   ` Gerd Möllmann
2024-07-08 19:44                                                                     ` Pip Cet [this message]
2024-07-09  3:58                                                                       ` Gerd Möllmann
2024-07-08 18:27                                                                   ` Helmut Eller
2024-07-08 18:31                                                                     ` Gerd Möllmann
2024-07-08 19:14                                                                       ` Eli Zaretskii
2024-07-08 19:35                                                                         ` Gerd Möllmann
2024-07-08 19:08                                                                     ` Eli Zaretskii
2024-07-08 19:31                                                                     ` Pip Cet
2024-07-04 15:22                         ` MPS: weak hash tables Helmut Eller
2024-07-04 15:33                           ` Pip Cet
2024-07-04 16:46                             ` Gerd Möllmann
2024-07-04 16:43                           ` Gerd Möllmann
2024-07-02 13:50         ` Mattias Engdegård
2024-07-02  6:57     ` Gerd Möllmann
2024-07-02  7:15       ` Gerd Möllmann
2024-07-02  8:46         ` Ihor Radchenko
2024-07-02  8:59           ` Gerd Möllmann
2024-07-02  9:33             ` Ihor Radchenko
2024-07-02  9:35               ` Pip Cet
2024-07-02 11:03                 ` Ihor Radchenko

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://www.gnu.org/software/emacs/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='Y0tbYudujKGHq0HEAbyM-44XBcnbBLEqh2Y19UVcHXu7hbQoa1GKvdXEgzxxM8gTBOJsD2I_mNryRhdPdCcFd1JuM2aseScThKl8kHaqpIw=@protonmail.com' \
    --to=pipcet@protonmail.com \
    --cc=acorallo@gnu.org \
    --cc=eliz@gnu.org \
    --cc=eller.helmut@gmail.com \
    --cc=emacs-devel@gnu.org \
    --cc=gerd.moellmann@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/emacs.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).