bug#26952: 25.1; loops eating all memory while yanking big rectangle

bug#26952: 25.1; loops eating all memory while yanking big rectangle

[Francesco Potortì]

$ emacs -Q -nw

I find a big text file that you can download from
<http://fly.isti.cnr.it/tmp/bigfile.txt.xz>

Once decompressed, the file is 80 MB long, composed of over 600000 lines
around 150 characters long each

Then I do this:

C-u 29 M-f		--> point is on the tab after "21"
M->
C-b
C-u 30 SPC
M-x kill-rectangle	--> Emacs discards the undo buffer
C-x b aa		--> create a scratch buffer
M-x yank-rectangle

at this point, Emacs freezes and starts growing in size.  On my system,
it started from less than 1GB vmem and grew to over 10GB when I killed
it.  Only kill -9 succeded to kill it.


In GNU Emacs 25.1.1 (x86_64-pc-linux-gnu, X toolkit, Xaw3d scroll bars)
 of 2017-04-23, modified by Debian built on trouble
Windowing system distributor 'The X.Org Foundation', version 11.0.11902000
System Description:	Debian GNU/Linux 9.0 (stretch)

Configured using:
 'configure --build x86_64-linux-gnu --prefix=/usr
 --sharedstatedir=/var/lib --libexecdir=/usr/lib
 --localstatedir=/var/lib --infodir=/usr/share/info
 --mandir=/usr/share/man --with-pop=yes
 --enable-locallisppath=/etc/emacs25:/etc/emacs:/usr/local/share/emacs/25.1/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/25.1/site-lisp:/usr/share/emacs/site-lisp
 --with-sound=alsa --build x86_64-linux-gnu --prefix=/usr
 --sharedstatedir=/var/lib --libexecdir=/usr/lib
 --localstatedir=/var/lib --infodir=/usr/share/info
 --mandir=/usr/share/man --with-pop=yes
 --enable-locallisppath=/etc/emacs25:/etc/emacs:/usr/local/share/emacs/25.1/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/25.1/site-lisp:/usr/share/emacs/site-lisp
 --with-sound=alsa --with-x=yes --with-x-toolkit=lucid
 --with-toolkit-scroll-bars --without-gconf --without-gsettings
 'CFLAGS=-g -O2
 -fdebug-prefix-map=/build/emacs25-d2FC1K/emacs25-25.1+1=. -fstack-protector-strong
 -Wformat -Werror=format-security -Wall' 'CPPFLAGS=-Wdate-time
 -D_FORTIFY_SOURCE=2' LDFLAGS=-Wl,-z,relro'

Configured features:
XAW3D XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS NOTIFY ACL
LIBSELINUX GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB
TOOLKIT_SCROLL_BARS LUCID X11

Important settings:
  value of $LC_COLLATE: it_IT.UTF-8
  value of $LC_CTYPE: it_IT.UTF-8
  value of $LC_NUMERIC: C
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Fundamental

Minor modes in effect:
  TeX-PDF-mode: t
  desktop-save-mode: t
  epa-global-mail-mode: t
  shell-dirtrack-mode: t
  openwith-mode: t
  xterm-mouse-mode: t
  display-time-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t

Recent messages:
pot.bib: parsing reference keys (61%)
pot.bib: parsing reference keys (66%)
pot.bib: parsing reference keys (71%)
pot.bib: parsing reference keys (76%)
pot.bib: parsing reference keys (81%)
pot.bib: parsing reference keys (86%)
pot.bib: parsing reference keys (91%)
pot.bib: parsing reference keys (96%)
pot.bib: parsing reference keys (done)
C-x r y runs the command yank-rectangle

Load-path shadows:
~/elisp/bhl hides /usr/share/emacs/25.1/site-lisp/bhl
~/elisp/bhl hides /usr/share/emacs/site-lisp/bhl
/usr/share/emacs/25.1/site-lisp/debian-startup hides /usr/share/emacs/site-lisp/debian-startup
/usr/share/emacs25/site-lisp/flim/md4 hides /usr/share/emacs/25.1/lisp/md4
/usr/share/emacs25/site-lisp/flim/hex-util hides /usr/share/emacs/25.1/lisp/hex-util
/usr/share/emacs/site-lisp/rst hides /usr/share/emacs/25.1/lisp/textmodes/rst
~/elisp/bibtex hides /usr/share/emacs/25.1/lisp/textmodes/bibtex
/usr/share/emacs25/site-lisp/flim/ntlm hides /usr/share/emacs/25.1/lisp/net/ntlm
/usr/share/emacs25/site-lisp/flim/hmac-md5 hides /usr/share/emacs/25.1/lisp/net/hmac-md5
/usr/share/emacs25/site-lisp/flim/sasl-ntlm hides /usr/share/emacs/25.1/lisp/net/sasl-ntlm
/usr/share/emacs25/site-lisp/flim/sasl-digest hides /usr/share/emacs/25.1/lisp/net/sasl-digest
/usr/share/emacs25/site-lisp/flim/sasl hides /usr/share/emacs/25.1/lisp/net/sasl
/usr/share/emacs25/site-lisp/flim/sasl-cram hides /usr/share/emacs/25.1/lisp/net/sasl-cram
/usr/share/emacs25/site-lisp/flim/hmac-def hides /usr/share/emacs/25.1/lisp/net/hmac-def
/usr/share/emacs25/site-lisp/auctex/tex-fold hides /usr/share/emacs/site-lisp/auctex/tex-fold
/usr/share/emacs25/site-lisp/auctex/context-en hides /usr/share/emacs/site-lisp/auctex/context-en
/usr/share/emacs25/site-lisp/auctex/tex-info hides /usr/share/emacs/site-lisp/auctex/tex-info
/usr/share/emacs25/site-lisp/auctex/plain-tex hides /usr/share/emacs/site-lisp/auctex/plain-tex
/usr/share/emacs25/site-lisp/auctex/tex-mik hides /usr/share/emacs/site-lisp/auctex/tex-mik
/usr/share/emacs25/site-lisp/auctex/texmathp hides /usr/share/emacs/site-lisp/auctex/texmathp
/usr/share/emacs25/site-lisp/auctex/context-nl hides /usr/share/emacs/site-lisp/auctex/context-nl
/usr/share/emacs25/site-lisp/auctex/toolbar-x hides /usr/share/emacs/site-lisp/auctex/toolbar-x
/usr/share/emacs25/site-lisp/auctex/tex hides /usr/share/emacs/site-lisp/auctex/tex
/usr/share/emacs25/site-lisp/auctex/tex-jp hides /usr/share/emacs/site-lisp/auctex/tex-jp
/usr/share/emacs25/site-lisp/auctex/tex-ispell hides /usr/share/emacs/site-lisp/auctex/tex-ispell
/usr/share/emacs25/site-lisp/auctex/bib-cite hides /usr/share/emacs/site-lisp/auctex/bib-cite
/usr/share/emacs25/site-lisp/auctex/multi-prompt hides /usr/share/emacs/site-lisp/auctex/multi-prompt
/usr/share/emacs25/site-lisp/auctex/font-latex hides /usr/share/emacs/site-lisp/auctex/font-latex
/usr/share/emacs25/site-lisp/auctex/prv-emacs hides /usr/share/emacs/site-lisp/auctex/prv-emacs
/usr/share/emacs25/site-lisp/auctex/tex-style hides /usr/share/emacs/site-lisp/auctex/tex-style
/usr/share/emacs25/site-lisp/auctex/context hides /usr/share/emacs/site-lisp/auctex/context
/usr/share/emacs25/site-lisp/auctex/preview hides /usr/share/emacs/site-lisp/auctex/preview
/usr/share/emacs25/site-lisp/auctex/tex-font hides /usr/share/emacs/site-lisp/auctex/tex-font
/usr/share/emacs25/site-lisp/auctex/tex-bar hides /usr/share/emacs/site-lisp/auctex/tex-bar
/usr/share/emacs25/site-lisp/auctex/latex hides /usr/share/emacs/site-lisp/auctex/latex
/usr/share/emacs25/site-lisp/auctex/tex-buf hides /usr/share/emacs/site-lisp/auctex/tex-buf

Features:
(shadow mailalias emacsbug server jka-compr bibtex sh-script executable
image-mode js json map imenu info sgml-mode cc-mode cc-fonts cc-guess
cc-menus cc-cmds cc-styles cc-align cc-engine cc-vars cc-defs latexenc
preview prv-emacs noutline outline vc-dispatcher vc-svn tex-bar
toolbar-x font-latex plain-tex tex-buf latex easy-mmode edmacro kmacro
tex-ispell tex-style tex dbus xml crm tex-mode compile vc-filewise
vc-rcs octave smie generic qp rmailmm message rfc822 mml mml-sec
mm-decode mm-bodies mm-encode mailabbrev gmm-utils mailheader mail-parse
rfc2231 desktop frameset solar cal-dst pot skeleton warnings rmailsum
rmail sendmail rfc2047 rfc2045 ietf-drums mime-compose epa-mail
mail-utils epa derived epg view holidays hol-loaddefs appt diary-lib
diary-loaddefs cal-menu calendar cal-loaddefs tramp tramp-compat
tramp-loaddefs trampver ucs-normalize shell pcomplete comint ring
format-spec advice bhl visual-fill-column switch-to-shell openwith
hi-lock xt-mouse ffap thingatpt url-parse auth-source cl-seq eieio
eieio-core cl-macs gnus-util time-date mm-util help-fns mail-prsvr
password-cache url-vars scroll-in-place filladapt ansi-color time quail
dired-x dired generic-x disp-table finder-inf package epg-config seq
byte-opt gv bytecomp byte-compile cl-extra help-mode easymenu cconv
cl-loaddefs pcase cl-lib debian-el debian-el-loaddefs w3m-load
vm-autoload vm-autoloads vm-version vm-vars vm-init preview-latex
tex-site auto-loads mule-util tooltip eldoc electric uniquify ediff-hook
vc-hooks lisp-float-type mwheel x-win term/common-win x-dnd tool-bar dnd
fontset image regexp-opt fringe tabulated-list newcomment elisp-mode
lisp-mode prog-mode register page menu-bar rfn-eshadow timer select
scroll-bar mouse jit-lock font-lock syntax facemenu font-core frame
cl-generic cham georgian utf-8-lang misc-lang vietnamese tibetan thai
tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek romanian
slovak czech european ethiopic indian cyrillic chinese charscript
case-table epa-hook jka-cmpr-hook help simple abbrev minibuffer
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote dbusbind inotify
dynamic-setting font-render-setting x-toolkit x multi-tty
make-network-process emacs)

Memory information:
((conses 16 472874 45946)
 (symbols 48 37676 0)
 (miscs 40 6718 4170)
 (strings 32 83110 16710)
 (string-bytes 1 2736675)
 (vectors 16 53030)
 (vector-slots 8 905046 9550)
 (floats 8 683 362)
 (intervals 56 17278 1390)
 (buffers 976 116))

bug#26952: 25.1; loops eating all memory while yanking big rectangle

[npostavs]

found 26952 25.2
fixed 26952 26.1
quit

Francesco Potortì <pot@gnu.org> writes:

> $ emacs -Q -nw
>
> I find a big text file that you can download from
> <http://fly.isti.cnr.it/tmp/bigfile.txt.xz>
>
> Once decompressed, the file is 80 MB long, composed of over 600000 lines
> around 150 characters long each
>
> Then I do this:
>
> C-u 29 M-f		--> point is on the tab after "21"
> M->
> C-b
> C-u 30 SPC
> M-x kill-rectangle	--> Emacs discards the undo buffer
> C-x b aa		--> create a scratch buffer
> M-x yank-rectangle
>
> at this point, Emacs freezes and starts growing in size.  On my system,
> it started from less than 1GB vmem and grew to over 10GB when I killed
> it.  Only kill -9 succeded to kill it.

I see the issue also with 25.2, but not with master (no idea what might
have fixed it though).

bug#26952: 25.1; loops eating all memory while yanking big rectangle

[Eli Zaretskii]

> From: npostavs@users.sourceforge.net
> Date: Fri, 19 May 2017 20:24:03 -0400
> Cc: 26952@debbugs.gnu.org
> 
> > $ emacs -Q -nw
> >
> > I find a big text file that you can download from
> > <http://fly.isti.cnr.it/tmp/bigfile.txt.xz>
> >
> > Once decompressed, the file is 80 MB long, composed of over 600000 lines
> > around 150 characters long each
> >
> > Then I do this:
> >
> > C-u 29 M-f		--> point is on the tab after "21"
> > M->
> > C-b
> > C-u 30 SPC
> > M-x kill-rectangle	--> Emacs discards the undo buffer
> > C-x b aa		--> create a scratch buffer
> > M-x yank-rectangle
> >
> > at this point, Emacs freezes and starts growing in size.  On my system,
> > it started from less than 1GB vmem and grew to over 10GB when I killed
> > it.  Only kill -9 succeded to kill it.
> 
> I see the issue also with 25.2, but not with master (no idea what might
> have fixed it though).

Thanks for looking into this.  Can you tell more about where it is
looping in Emacs 25.2?  I'm uneasy about not knowing what fixed this.

bug#26952: 25.1; loops eating all memory while yanking big rectangle

[npostavs]

Eli Zaretskii <eliz@gnu.org> writes:

>> I see the issue also with 25.2, but not with master (no idea what might
>> have fixed it though).
>
> Thanks for looking into this.  Can you tell more about where it is
> looping in Emacs 25.2?  I'm uneasy about not knowing what fixed this.

The following simple loop can trigger the issue (I'm now also limiting
Emacs' memory usage to 1GB with "ulimit -Sv $((1000 * 1024))" so that it
just throws an out of memory error instead of filling my swap and
slowing everything down):

  (let ((str (make-string 150 ?a)))
    (dotimes (_ (* 600 1000))
      (insert str ?\n)))

I think it might be just an inefficient allocater (or this pattern of
allocation happens to hit a pathological case for the allocater).  The
master branch is using the 'hybrid' allocater, while emacs-25 is not.
If I configure 25.2 with REL_ALLOC=yes, then it runs okay.  The only
allocation seems to be from 'enlarge_buffer_text'.

bug#26952: 25.1; loops eating all memory while yanking big rectangle

[Eli Zaretskii]

> From: npostavs@users.sourceforge.net
> Cc: 26952@debbugs.gnu.org,  pot@gnu.org
> Date: Sat, 20 May 2017 13:41:36 -0400
> 
> The following simple loop can trigger the issue (I'm now also limiting
> Emacs' memory usage to 1GB with "ulimit -Sv $((1000 * 1024))" so that it
> just throws an out of memory error instead of filling my swap and
> slowing everything down):
> 
>   (let ((str (make-string 150 ?a)))
>     (dotimes (_ (* 600 1000))
>       (insert str ?\n)))
> 
> I think it might be just an inefficient allocater (or this pattern of
> allocation happens to hit a pathological case for the allocater).  The
> master branch is using the 'hybrid' allocater, while emacs-25 is not.
> If I configure 25.2 with REL_ALLOC=yes, then it runs okay.  The only
> allocation seems to be from 'enlarge_buffer_text'.

Thanks.

So you are saying that inserting 90MB worth of text into a buffer
makes Emacs 25.2 run out of 1GB of memory, due to inefficiencies of
the malloc implementation?  (Here on Windows it produces a 230MB Emacs
session, but the Windows build uses the moral equivalent of mmap for
allocating buffer text.)

Maybe we should release Emacs 25.3 with this single problem fixed?

bug#26952: 25.1; loops eating all memory while yanking big rectangle

[npostavs]

Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> The following simple loop can trigger the issue (I'm now also limiting
>> Emacs' memory usage to 1GB with "ulimit -Sv $((1000 * 1024))" so that it
>> just throws an out of memory error instead of filling my swap and
>> slowing everything down):
>> 
>>   (let ((str (make-string 150 ?a)))
>>     (dotimes (_ (* 600 1000))
>>       (insert str ?\n)))
>> 
>> I think it might be just an inefficient allocater (or this pattern of
>> allocation happens to hit a pathological case for the allocater).  The
>> master branch is using the 'hybrid' allocater, while emacs-25 is not.
>> If I configure 25.2 with REL_ALLOC=yes, then it runs okay.  The only
>> allocation seems to be from 'enlarge_buffer_text'.
>
> Thanks.
>
> So you are saying that inserting 90MB worth of text into a buffer
> makes Emacs 25.2 run out of 1GB of memory, due to inefficiencies of
> the malloc implementation?

When it's inserted in small chunks, yes, I think so.  What seems to
happen is that the buffer gap keeps getting realloc'd to be slightly
bigger, and the deallocated chunks don't get reused.

> (Here on Windows it produces a 230MB Emacs
> session, but the Windows build uses the moral equivalent of mmap for
> allocating buffer text.)

Neither master nor emacs-25 are using mmap (according to configure
output), but I guess the "hybrid" or relocating allocaters are smart
enough to handle this case.

bug#26952: 25.1; loops eating all memory while yanking big rectangle

[Eli Zaretskii]

> From: npostavs@users.sourceforge.net
> Cc: 26952@debbugs.gnu.org,  pot@gnu.org
> Date: Sat, 20 May 2017 15:27:39 -0400
> 
> > So you are saying that inserting 90MB worth of text into a buffer
> > makes Emacs 25.2 run out of 1GB of memory, due to inefficiencies of
> > the malloc implementation?
> 
> When it's inserted in small chunks, yes, I think so.  What seems to
> happen is that the buffer gap keeps getting realloc'd to be slightly
> bigger, and the deallocated chunks don't get reused.
> 
> > (Here on Windows it produces a 230MB Emacs
> > session, but the Windows build uses the moral equivalent of mmap for
> > allocating buffer text.)
> 
> Neither master nor emacs-25 are using mmap (according to configure
> output), but I guess the "hybrid" or relocating allocaters are smart
> enough to handle this case.

I cannot see why.  AFAIK, the only difference between using the hybrid
allocation and not using it is before Emacs is dumped; after that both
use the same system malloc.  So if the hybrid malloc fixed this, the
problem is somehow related to the memory allocation before dumping.
Maybe reallocating a gap that was allocated before dumping somehow
exposes a bug?

Paul, is it feasible to back-port the hybrid allocation to the
emacs-25 branch?  This sounds like a nasty bug, so if we can safely
fix it, I think we ought to release Emacs 25.3 with just this issue
fixed.  WDYT?

bug#26952: 25.1; loops eating all memory while yanking big rectangle

[npostavs]

Eli Zaretskii <eliz@gnu.org> writes:

> I cannot see why.  AFAIK, the only difference between using the hybrid
> allocation and not using it is before Emacs is dumped; after that both
> use the same system malloc.

I was under the impression that it's just the opposite: master is using
hybrid malloc which means to use system malloc *after* dumping, and
emacs-25 is using the custom non-system malloc *after* dumping.  Neither
use the system malloc *before* dumping.

bug#26952: 25.1; loops eating all memory while yanking big rectangle

[Eli Zaretskii]

> From: npostavs@users.sourceforge.net
> Cc: Paul Eggert <eggert@cs.ucla.edu>,  26952@debbugs.gnu.org,  pot@gnu.org
> Date: Sun, 21 May 2017 12:00:00 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I cannot see why.  AFAIK, the only difference between using the hybrid
> > allocation and not using it is before Emacs is dumped; after that both
> > use the same system malloc.
> 
> I was under the impression that it's just the opposite: master is using
> hybrid malloc which means to use system malloc *after* dumping, and
> emacs-25 is using the custom non-system malloc *after* dumping.

By "custom non-system malloc" do you mean gmalloc.c?  Is there
src/gmalloc.o in the build directory?  (I don't have access to a
system when this happens, my GNU/Linux build doesn't have
src/gmalloc.o.)  Does enlarge_buffer_text eventually calls into
gmalloc.c in the problematic build?

I thought we moved to system malloc in Emacs 25.2, and that was my
reading of configure, but maybe I'm confused.

bug#26952: 25.1; loops eating all memory while yanking big rectangle

[npostavs]

Eli Zaretskii <eliz@gnu.org> writes:

> Is there src/gmalloc.o in the build directory?

Yes there is.  Furthermore, in 25.2 configure output shows this:

  Should Emacs use the GNU version of malloc?             yes
  Should Emacs use a relocating allocator for buffers?    no
  Should Emacs use mmap(2) for buffer allocation?         no

In master it says this:

  Should Emacs use the GNU version of malloc?             no (only before dumping)
  Should Emacs use a relocating allocator for buffers?    no
  Should Emacs use mmap(2) for buffer allocation?         no

> Does enlarge_buffer_text eventually calls into gmalloc.c in the
> problematic build?

Yes.

> I thought we moved to system malloc in Emacs 25.2, and that was my
> reading of configure, but maybe I'm confused.

Everytime I try to read the configure code for this, I get more confused :(

bug#26952: 25.1; loops eating all memory while yanking big rectangle

[Paul Eggert]

Eli Zaretskii wrote:
> Paul, is it feasible to back-port the hybrid allocation to the
> emacs-25 branch?

I suppose it would be, yes. Not that I'd relish doing it. I'd rather focus 
efforts on getting the next version out the door. What are the important things 
that prevent us from releasing Emacs 26 from current master?

bug#26952: 25.1; loops eating all memory while yanking big rectangle

[Eli Zaretskii]

> Cc: 26952@debbugs.gnu.org, pot@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Sun, 21 May 2017 12:15:09 -0700
> 
> What are the important things that prevent us from releasing Emacs
> 26 from current master?

None that I know of.  But even if we cut the branch and start the
pretest today, 26.1 is still something like 6 months away, judging by
past experience, perhaps even longer.  Do we want to leave that memory
problem in 25.2 until then?  I thought we could do a quick emergency
release in a week or so, and then proceed with 26.1.

bug#26952: 25.1; loops eating all memory while yanking big rectangle

[npostavs]

Paul Eggert <eggert@cs.ucla.edu> writes:

> Eli Zaretskii wrote:
>> Paul, is it feasible to back-port the hybrid allocation to the
>> emacs-25 branch?
>
> I suppose it would be, yes. Not that I'd relish doing it.

Alternatively, we could set REL_ALLOC=yes by default again (AFAIK, we
did fix the bugs in that configuration in addition to turning it off).

bug#26952: 25.1; loops eating all memory while yanking big rectangle

[Paul Eggert]

Eli Zaretskii wrote:
> even if we cut the branch and start the
> pretest today, 26.1 is still something like 6 months away, judging by
> past experience, perhaps even longer.  Do we want to leave that memory
> problem in 25.2 until then?

I'd prefer that, yes. It'd be better to get 26 out the door. We don't have the 
development resources to maintain multiple versions, especially in an area where 
"Everytime I try to read the configure code for this, I get more confused".

In the meantime, people who have the problem in 25.2 can go back to 25.1.

bug#26952: 25.1; loops eating all memory while yanking big rectangle

[npostavs]

Paul Eggert <eggert@cs.ucla.edu> writes:

> In the meantime, people who have the problem in 25.2 can go back to 25.1.

No, people who have this problem in 25.2 would have #24358 in 25.1, so
going back would mean frequent segfaults.  The OP reported this against
25.1, but that is a 25.1 modified by Debian, which I believe configures
with REL_ALLOC=no (which is the significant difference of 25.2 vs 25.1
with respect to this bug).

bug#26952: 25.1; loops eating all memory while yanking big rectangle

[Paul Eggert]

OK, sorry, I stand corrected: they can go back to 25.1 with REL_ALLOC=no.

bug#26952: 25.1; loops eating all memory while yanking big rectangle

[npostavs]

Paul Eggert <eggert@cs.ucla.edu> writes:

> OK, sorry, I stand corrected: they can go back to 25.1 with REL_ALLOC=no.

No, that's what gives us this bug.  From what I can tell, 25.2 with
REL_ALLOC=yes is the only viable alternative at the moment.

bug#26952: 25.1; loops eating all memory while yanking big rectangle

[Eli Zaretskii]

> From: npostavs@users.sourceforge.net
> Cc: Eli Zaretskii <eliz@gnu.org>,  26952@debbugs.gnu.org,  pot@gnu.org
> Date: Sun, 21 May 2017 16:06:31 -0400
> 
> Paul Eggert <eggert@cs.ucla.edu> writes:
> 
> > Eli Zaretskii wrote:
> >> Paul, is it feasible to back-port the hybrid allocation to the
> >> emacs-25 branch?
> >
> > I suppose it would be, yes. Not that I'd relish doing it.
> 
> Alternatively, we could set REL_ALLOC=yes by default again (AFAIK, we
> did fix the bugs in that configuration in addition to turning it off).

I'd rather not go back there.

bug#26952: 25.1; loops eating all memory while yanking big rectangle

[Eli Zaretskii]

> Cc: npostavs@users.sourceforge.net, 26952@debbugs.gnu.org, pot@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Sun, 21 May 2017 13:20:00 -0700
> 
> Eli Zaretskii wrote:
> > even if we cut the branch and start the
> > pretest today, 26.1 is still something like 6 months away, judging by
> > past experience, perhaps even longer.  Do we want to leave that memory
> > problem in 25.2 until then?
> 
> I'd prefer that, yes. It'd be better to get 26 out the door. We don't have the 
> development resources to maintain multiple versions, especially in an area where 
> "Everytime I try to read the configure code for this, I get more confused".

It's not my intent to maintain multiple versions, any more than we do
now.  If 25.3 is released, its status will be like that of 25.2 now.

> In the meantime, people who have the problem in 25.2 can go back to 25.1.

That version had worse problem in the same area, due to ralloc.c and
its implications.

John, what is your take on this?

bug#26952: 25.1; loops eating all memory while yanking big rectangle

[John Wiegley]

>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:

EZ> John, what is your take on this?

I actually would prefer to get 26 out sooner, than to backport, but I'm not
sure at the moment how big of a headache that would really be.  In general, it
would be nice to accelerate the schedule for major releases. But whether we
have the manpower to really do that is another question.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

bug#26952: 25.1; loops eating all memory while yanking big rectangle

[Eli Zaretskii]

> From: John Wiegley <jwiegley@gmail.com>
> Cc: Paul Eggert <eggert@cs.ucla.edu>,  26952@debbugs.gnu.org,  pot@gnu.org,  npostavs@users.sourceforge.net
> Date: Wed, 24 May 2017 22:31:37 -0700
> 
> >>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:
> 
> EZ> John, what is your take on this?
> 
> I actually would prefer to get 26 out sooner, than to backport, but I'm not
> sure at the moment how big of a headache that would really be.  In general, it
> would be nice to accelerate the schedule for major releases. But whether we
> have the manpower to really do that is another question.

We all want to accelerate the release schedule, but I don't think we
know how.  That's why I don't believe we could have Emacs 26.1 out
earlier than in six months' time, basically what we had with every
other .1 release.  The question is: can we afford leaving users with
this nasty bug for several months, telling them to wait.  I thought we
oughtn't.

But if I'm the only one who is bothered by this, so be it.

bug#26952: 25.1; loops eating all memory while yanking big rectangle

[Paul Eggert]

On 05/25/2017 08:09 AM, Eli Zaretskii wrote:
> I don't believe we could have Emacs 26.1 out
> earlier than in six months' time, basically what we had with every
> other .1 release.


I think 26.1 is a better way to go here, even if we have to wait six 
months for it. Users have already been living with this bug since 25.1's 
release eight months ago, and they can wait a few months more.

We can publish 26.1 earlier if we don't run it through our leisurely six 
month test period. It would not be tested as well as 25.1 was, but the 
same would be true for any 25.3 release, as its changes are not likely 
to be trivial to develop. We could warn users that 26.1 is more 
experimental than 25.1 was, so that users not seriously affected by 
Bug#26952 (i.e., most users) could stick with Emacs 25 if they want to 
be conservative.

bug#26952: 25.1; loops eating all memory while yanking big rectangle

[Francesco Potortì]

Paul Eggert:
>I think 26.1 is a better way to go here, even if we have to wait six 
>months for it. Users have already been living with this bug since 25.1's 
>release eight months ago, and they can wait a few months more.
>
>We can publish 26.1 earlier if we don't run it through our leisurely six 
>month test period. It would not be tested as well as 25.1 was, but the 
>same would be true for any 25.3 release, as its changes are not likely 
>to be trivial to develop. We could warn users that 26.1 is more 
>experimental than 25.1 was, so that users not seriously affected by 
>Bug#26952 (i.e., most users) could stick with Emacs 25 if they want to 
>be conservative.

It's sad.  Until Emacs 22, I could tell my colleagues that Emacs would
never ever crash in practical situations.  I upgraded my old Emacs 22
only recently, because of rmail changes, and I find it much less stable
than I was used to :(

Sorry that I am not able to help.

bug#26952: 25.1; loops eating all memory while yanking big rectangle

[John Wiegley]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

> We all want to accelerate the release schedule, but I don't think we know
> how. That's why I don't believe we could have Emacs 26.1 out earlier than in
> six months' time, basically what we had with every other .1 release. The
> question is: can we afford leaving users with this nasty bug for several
> months, telling them to wait. I thought we oughtn't.

> But if I'm the only one who is bothered by this, so be it.

I can certainly sympathize.

FWIW, I too have noticed Emacs 25.2 being quite unstable. For the longest
time, Emacs 24 would run for months without crashing even once. Now it crashes
pretty regularly 2-4 times a day. I wonder now if I'm hitting a memory ceiling
that is triggering the behavior we're describing...

I'm OK with backporting, btw, and would be happy to test if it fixes my issues
too.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

bug#26952: 25.1; loops eating all memory while yanking big rectangle

[Noam Postavsky]

On Thu, May 25, 2017 at 12:48 PM, John Wiegley <jwiegley@gmail.com> wrote:
>
> FWIW, I too have noticed Emacs 25.2 being quite unstable. For the longest
> time, Emacs 24 would run for months without crashing even once. Now it crashes
> pretty regularly 2-4 times a day. I wonder now if I'm hitting a memory ceiling
> that is triggering the behavior we're describing...

You're on macOS right? This bug and the proposed solution to be
backported only affects GNU/Linux distributions with recent glibc,
AFAIK.
Also, I don't think this bug would trigger crashes, just extreme
system slowness due to swapping.

bug#26952: 25.1; loops eating all memory while yanking big rectangle

[Eli Zaretskii]

> Cc: 26952@debbugs.gnu.org, pot@gnu.org, npostavs@users.sourceforge.net
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Thu, 25 May 2017 09:19:50 -0700
> 
> We can publish 26.1 earlier if we don't run it through our leisurely six 
> month test period.

There's no leisure in our pretests.  We need to wait until the bug
reports come in after each pretest tarball, that's all.  It takes time
for the reports to come in because Emacs is a very large package that
supports many different use patterns.

> It would not be tested as well as 25.1 was, but the same would be
> true for any 25.3 release, as its changes are not likely to be
> trivial to develop. We could warn users that 26.1 is more
> experimental than 25.1 was

We never released unstable Emacs, and I'm not sure we should start.

> so that users not seriously affected by Bug#26952 (i.e., most users)
> could stick with Emacs 25 if they want to be conservative.

IMO, there's nothing in this bug that allows us to assume it is rare.

bug#26952: 25.1; loops eating all memory while yanking big rectangle

[Francesco Potortì]

>On Thu, May 25, 2017 at 12:48 PM, John Wiegley <jwiegley@gmail.com> wrote:
>> FWIW, I too have noticed Emacs 25.2 being quite unstable. For the longest
>> time, Emacs 24 would run for months without crashing even once. Now it crashes
>> pretty regularly 2-4 times a day. I wonder now if I'm hitting a memory ceiling
>> that is triggering the behavior we're describing...
>
>You're on macOS right? This bug and the proposed solution to be
>backported only affects GNU/Linux distributions with recent glibc,
>AFAIK.

I'm on Debian.  I'm curious to know what did I say to make you think I'm
on Mac :)

>Also, I don't think this bug would trigger crashes, just extreme
>system slowness due to swapping.

Well, it eats all my memory.  It is worse than a crash, the first time I
had problems managing to kill Emacs.

bug#26952: 25.1; loops eating all memory while yanking big rectangle

[Noam Postavsky]

On Thu, May 25, 2017 at 1:29 PM, Francesco Potortì <pot@gnu.org> wrote:
>>On Thu, May 25, 2017 at 12:48 PM, John Wiegley <jwiegley@gmail.com> wrote:
>>> FWIW, I too have noticed Emacs 25.2 being quite unstable. For the longest
>>> time, Emacs 24 would run for months without crashing even once. Now it crashes
>>> pretty regularly 2-4 times a day. I wonder now if I'm hitting a memory ceiling
>>> that is triggering the behavior we're describing...
>>
>>You're on macOS right? This bug and the proposed solution to be
>>backported only affects GNU/Linux distributions with recent glibc,
>>AFAIK.
>
> I'm on Debian.  I'm curious to know what did I say to make you think I'm
> on Mac :)

"You're" was referring to John Wiegley there.

>
>>Also, I don't think this bug would trigger crashes, just extreme
>>system slowness due to swapping.
>
> Well, it eats all my memory.  It is worse than a crash,

Yes, it can be. A crash only affects Emacs, the slowness affects the
whole system.

> the first time I
> had problems managing to kill Emacs.

Although as far as I could tell, a normal SIGINT is still enough to
kill Emacs, but it took around 10 or 20 seconds before it actually
took effect (due to the aforementioned system slowness).

bug#26952: 25.1; loops eating all memory while yanking big rectangle

[Eli Zaretskii]

> From: John Wiegley <jwiegley@gmail.com>
> Cc: eggert@cs.ucla.edu,  26952@debbugs.gnu.org,  pot@gnu.org,  npostavs@users.sourceforge.net
> Date: Thu, 25 May 2017 09:48:47 -0700
> 
> I can certainly sympathize.
> 
> FWIW, I too have noticed Emacs 25.2 being quite unstable. For the longest
> time, Emacs 24 would run for months without crashing even once. Now it crashes
> pretty regularly 2-4 times a day. I wonder now if I'm hitting a memory ceiling
> that is triggering the behavior we're describing...
> 
> I'm OK with backporting, btw, and would be happy to test if it fixes my issues
> too.

Paul, could you please do that?

Actually, I see that the hybrid_malloc code does exist on the branch,
so what is needed to be done? configury stuff?

bug#26952: 25.1; loops eating all memory while yanking big rectangle

[Paul Eggert]

On 05/25/2017 10:45 AM, Eli Zaretskii wrote:
>> I'm OK with backporting, btw, and would be happy to test if it fixes my issues
>> too.
> Paul, could you please do that?

I could, if I knew what the "that" is. Could you summarize it, or point 
to the commit to master that you want backported to 25?

bug#26952: 25.1; loops eating all memory while yanking big rectangle

[Eli Zaretskii]

> Cc: 26952@debbugs.gnu.org, pot@gnu.org, npostavs@users.sourceforge.net
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Thu, 25 May 2017 12:24:02 -0700
> 
> On 05/25/2017 10:45 AM, Eli Zaretskii wrote:
> >> I'm OK with backporting, btw, and would be happy to test if it fixes my issues
> >> too.
> > Paul, could you please do that?
> 
> I could, if I knew what the "that" is. Could you summarize it, or point 
> to the commit to master that you want backported to 25?

Sorry.  I will try, but I'm not sure I'll succeed.

What I meant is to make hybrid_malloc the default on the same systems
where it is the default on master, in particular on GNU/Linux systems.
Not sure if this is helpful enough.

Thanks.

bug#26952: 25.1; loops eating all memory while yanking big rectangle

[Paul Eggert]

On 05/25/2017 12:34 PM, Eli Zaretskii wrote:
> What I meant is to make hybrid_malloc the default on the same systems
> where it is the default on master, in particular on GNU/Linux systems.
> Not sure if this is helpful enough.

I'm afraid not. This is partly why I didn't want to backport the fix: I 
don't know what needs to be backported, and figuring that out will be 
nontrivial.

bug#26952: 25.1; loops eating all memory while yanking big rectangle

[Eli Zaretskii]

> Cc: jwiegley@gmail.com, 26952@debbugs.gnu.org, pot@gnu.org,
>  npostavs@users.sourceforge.net
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Thu, 25 May 2017 13:59:33 -0700
> 
> On 05/25/2017 12:34 PM, Eli Zaretskii wrote:
> > What I meant is to make hybrid_malloc the default on the same systems
> > where it is the default on master, in particular on GNU/Linux systems.
> > Not sure if this is helpful enough.
> 
> I'm afraid not. This is partly why I didn't want to backport the fix: I 
> don't know what needs to be backported, and figuring that out will be 
> nontrivial.

Noam, can you please check whether building the emacs-25 branch with
HYBRID_MALLOC defined fix the problem in this bug report?  If it does,
then I think we need to arrange for HYBRID_MALLOC to be defined on ELF
based systems where DOUG_LEA_MALLOC is not defined.  I think d6585a910
on master shows the change in configure.ac which did that.  Bug#22086
holds the relevant discussions.

bug#26952: 25.1; loops eating all memory while yanking big rectangle

[Paul Eggert]

On 05/25/2017 11:25 PM, Eli Zaretskii wrote:
> I think d6585a910
> on master shows the change in configure.ac which did that.  Bug#22086
> holds the relevant discussions.

I was worried that that was the change you wanted (and was hoping it 
would be a different one :-).

We can't merely install that change. We also need at least some of the 
commits that fixed significant problems associated with that change, 
such as 09ece4d, 3d82a8e, a4817d8, a39a03a, dd951c0, 2ee2963, 7fdc3cf, 
a4817d8, e4cd4a7, e1a9f20, 874c59a, 384ffef, dec1390, a39a03a, cb22fce. 
Not all of them are strictly necessary, but it'd be better to install 
the whole batch than start a new development project in this area.

The above list of changes was reasonably easy to generate. The hard part 
will be identifying any followup changes that are also necessary but are 
not immediately obvious. So it's quite possible that if we take this 
approach we will need to publish Emacs 25.3, wait for allocation-related 
bug reports to roll in, and then publish Emacs 25.4 and so forth.

bug#26952: 25.1; loops eating all memory while yanking big rectangle

[Glenn Morris]

Eli Zaretskii wrote:

>> Cc: 26952@debbugs.gnu.org, pot@gnu.org, npostavs@users.sourceforge.net
>> From: Paul Eggert <eggert@cs.ucla.edu>
>> Date: Thu, 25 May 2017 09:19:50 -0700
>> 
>> We can publish 26.1 earlier if we don't run it through our leisurely six 
>> month test period.
>
> There's no leisure in our pretests.  We need to wait until the bug
> reports come in after each pretest tarball, that's all.  It takes time
> for the reports to come in because Emacs is a very large package that
> supports many different use patterns.

Idle speculation:

If a ~ six month testing process doesn't show up the fairly fundamental
memory issues that cropped up in both 25.1 and 25.2 soon after release,
then perhaps it isn't time well spent. Perhaps people would be better
served by more frequent, smaller releases. I have the feeling (not
backed up by any data) that pretest use is declining, with people just
using master or releases. Perhaps switching things around so that
releases come from master (and development happens on a branch) would
help, but maybe not.

bug#26952: 25.1; loops eating all memory while yanking big rectangle

[Paul Eggert]

Glenn Morris wrote:
> I have the feeling (not
> backed up by any data) that pretest use is declining, with people just
> using master or releases

Is there an easy way to tell which fraction of bug reports come from pretests vs 
master vs releases? That might provide some data for how useful pretests etc. 
are, for shaking out bugs.

bug#26952: 25.1; loops eating all memory while yanking big rectangle

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: Paul Eggert <eggert@cs.ucla.edu>,  jwiegley@gmail.com,  26952@debbugs.gnu.org,  npostavs@users.sourceforge.net
> Date: Fri, 26 May 2017 21:15:32 -0400
> 
> If a ~ six month testing process doesn't show up the fairly fundamental
> memory issues that cropped up in both 25.1 and 25.2 soon after release,
> then perhaps it isn't time well spent.

AFAIU, the problem was discovered by Francesco this late because he
only now upgraded from Emacs 22(!) to 25.2.

> Perhaps people would be better served by more frequent, smaller
> releases.

I don't think we know how to do this, either.  The experience of 25.1
and 25.2 shows that it might take a couple of weeks, sometimes more,
from the decision to make a tarball until it finally hits the download
site.  We need to have a much faster turnaround if we want to go your
way.

Another issue is that we will have to be more stringent in our
documentation requirements accompanying each change, if we basically
want to release development snapshots.

> I have the feeling (not backed up by any data) that pretest use is
> declining, with people just using master or releases. Perhaps
> switching things around so that releases come from master (and
> development happens on a branch) would help, but maybe not.

If development happens on a branch, my guess is people who currently
use master will switch to tracking that branch instead, and we are
back at the same problem.

bug#26952: Possible regression of Bug#26952 - "repeated buffer insertion consumes excessive memory"

[Peter Ludemann]

[-- Attachment #1: Type: text/plain, Size: 5638 bytes --]

I'm observing creeping memory increase (at one point, emacs was using over
3.8GB before I restarted it), possibly related to running compilations with
large amounts of output. Seems related to Bug#26952 (which my Bug#38629
seemed to be a duplicate of), and which was fixed in emacs 26.3. It seems
to have reappeared in emacs 28.0.50, but with a difference -- I'm not
seeing the high CPU usage that I previously observed with the high memory
use (perhaps this is because global-auto-revert has been improved in
28.0.50).

What information should I collect to confirm this problem and to help
someone debug it?

In GNU Emacs 28.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.22.30)
 of 2020-01-07 built on wistaria5b
Repository revision: 724af7671590cd91df37f64df6be73f6dca0144d
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.11906000
System Description: Ubuntu 18.04.3 LTS

Recent messages:
Reverting buffer ‘pykythe_utils.pl’.
You can run the command ‘compile’ with C-x C-g g
Reverting buffer ‘TAGS’.
Reverting buffer ‘FILES.js’. [2 times]
Reverting buffer ‘t10.kythe.json-decoded’. [2 times]
Compilation finished
You can run the command ‘garbage-collect’ with M-x ga RET
x is undefined [3 times]
Making completion list...
Quit
Quit
Configured using:
 'configure --prefix=/home/peter/emacs-2019-12-17'

Configured features:
XPM JPEG TIFF GIF PNG RSVG SOUND DBUS GSETTINGS GLIB NOTIFY INOTIFY
GNUTLS LIBXML2 FREETYPE HARFBUZZ M17N_FLT LIBOTF XFT ZLIB
TOOLKIT_SCROLL_BARS GTK3 X11 XDBE XIM MODULES THREADS PDUMPER GMP

Important settings:
  value of $LC_MONETARY: en_CA.UTF-8
  value of $LC_NUMERIC: en_CA.UTF-8
  value of $LC_TIME: en_CA.UTF-8
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: Compilation

Minor modes in effect:
  shell-dirtrack-mode: t
  global-auto-revert-mode: t
  show-paren-mode: t
  display-time-mode: t
  savehist-mode: t
  desktop-save-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
~/emacs/prolog hides
/home/peter/emacs-2019-12-17/share/emacs/28.0.50/lisp/progmodes/prolog

Features:
(shadow mail-extr emacsbug message rmc rfc822 mml mml-sec epa derived
epg epg-config mm-decode mm-bodies mm-encode mail-parse rfc2231
mailabbrev gmm-utils mailheader sendmail sort pulse vc vc-dispatcher
misearch multi-isearch tabify man pcmpl-unix pcmpl-gnu mule-util server
asm-mode conf-mode tar-mode arc-mode archive-mode add-log rst
haskell-mode haskell-cabal haskell-utils haskell-font-lock
haskell-indentation haskell-string haskell-sort-imports haskell-lexeme
rx haskell-align-imports haskell-compat haskell-complete-module
haskell-ghc-support flymake-proc flymake warnings dabbrev
haskell-customize doc-view jka-compr image-mode exif go-mode find-file
ffap etags fileloop generator xref project mhtml-mode css-mode sgml-mode
eww mm-url gnus nnheader gnus-util rmail rmail-loaddefs rfc2047 rfc2045
ietf-drums mail-utils wid-edit mm-util mail-prsvr url-queue url
url-proxy url-privacy url-expand url-methods url-history mailcap shr
text-property-search url-cookie url-domsuf url-util puny svg xml dom
sh-script smie executable markdown-mode color thingatpt noutline outline
dired dired-loaddefs js cc-mode cc-fonts cc-guess cc-menus cc-cmds
cc-styles cc-align cc-engine cc-vars cc-defs make-mode smerge-mode diff
prolog align imenu vc-git diff-mode easy-mmode python tramp-sh tramp
tramp-loaddefs trampver tramp-integration files-x tramp-compat shell
pcomplete parse-time iso8601 time-date ls-lisp format-spec finder-inf
cl-extra help-mode autorevert filenotify grep compile comint ansi-color
ring cus-start cus-load paren time savehist desktop frameset info
package easymenu browse-url url-handlers url-parse auth-source cl-seq
eieio eieio-core cl-macs eieio-loaddefs password-cache json subr-x map
url-vars seq byte-opt gv bytecomp byte-compile cconv cl-loaddefs cl-lib
tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type
mwheel term/x-win x-win term/common-win x-dnd tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode elisp-mode
lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow isearch
timer select scroll-bar mouse jit-lock font-lock syntax facemenu
font-core term/tty-colors frame minibuffer cl-generic cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese composite charscript charprop
case-table epa-hook jka-cmpr-hook help simple abbrev obarray
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote threads dbusbind
inotify dynamic-setting system-font-setting font-render-setting
move-toolbar gtk x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 553627 84742)
 (symbols 48 27664 5)
 (strings 32 109868 12210)
 (string-bytes 1 4407199)
 (vectors 16 47901)
 (vector-slots 8 1347908 158898)
 (floats 8 232 330)
 (intervals 56 48562 536)
 (buffers 1000 503))

[-- Attachment #2: Type: text/html, Size: 6415 bytes --]

bug#39045: Fwd: Possible regression - "repeated buffer insertion consumes excessive memory"

[Peter Ludemann]

[-- Attachment #1: Type: text/plain, Size: 5638 bytes --]

I'm observing creeping memory increase (at one point, emacs was using over
3.8GB before I restarted it), possibly related to running compilations with
large amounts of output. Seems related to Bug#26952 (which my Bug#38629
seemed to be a duplicate of), and which was fixed in emacs 26.3. It seems
to have reappeared in emacs 28.0.50, but with a difference -- I'm not
seeing the high CPU usage that I previously observed with the high memory
use (perhaps this is because global-auto-revert has been improved in
28.0.50).

What information should I collect to confirm this problem and to help
someone debug it?

In GNU Emacs 28.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.22.30)
 of 2020-01-07 built on wistaria5b
Repository revision: 724af7671590cd91df37f64df6be73f6dca0144d
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.11906000
System Description: Ubuntu 18.04.3 LTS

Recent messages:
Reverting buffer ‘pykythe_utils.pl’.
You can run the command ‘compile’ with C-x C-g g
Reverting buffer ‘TAGS’.
Reverting buffer ‘FILES.js’. [2 times]
Reverting buffer ‘t10.kythe.json-decoded’. [2 times]
Compilation finished
You can run the command ‘garbage-collect’ with M-x ga RET
x is undefined [3 times]
Making completion list...
Quit
Quit
Configured using:
 'configure --prefix=/home/peter/emacs-2019-12-17'

Configured features:
XPM JPEG TIFF GIF PNG RSVG SOUND DBUS GSETTINGS GLIB NOTIFY INOTIFY
GNUTLS LIBXML2 FREETYPE HARFBUZZ M17N_FLT LIBOTF XFT ZLIB
TOOLKIT_SCROLL_BARS GTK3 X11 XDBE XIM MODULES THREADS PDUMPER GMP

Important settings:
  value of $LC_MONETARY: en_CA.UTF-8
  value of $LC_NUMERIC: en_CA.UTF-8
  value of $LC_TIME: en_CA.UTF-8
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: Compilation

Minor modes in effect:
  shell-dirtrack-mode: t
  global-auto-revert-mode: t
  show-paren-mode: t
  display-time-mode: t
  savehist-mode: t
  desktop-save-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
~/emacs/prolog hides
/home/peter/emacs-2019-12-17/share/emacs/28.0.50/lisp/progmodes/prolog

Features:
(shadow mail-extr emacsbug message rmc rfc822 mml mml-sec epa derived
epg epg-config mm-decode mm-bodies mm-encode mail-parse rfc2231
mailabbrev gmm-utils mailheader sendmail sort pulse vc vc-dispatcher
misearch multi-isearch tabify man pcmpl-unix pcmpl-gnu mule-util server
asm-mode conf-mode tar-mode arc-mode archive-mode add-log rst
haskell-mode haskell-cabal haskell-utils haskell-font-lock
haskell-indentation haskell-string haskell-sort-imports haskell-lexeme
rx haskell-align-imports haskell-compat haskell-complete-module
haskell-ghc-support flymake-proc flymake warnings dabbrev
haskell-customize doc-view jka-compr image-mode exif go-mode find-file
ffap etags fileloop generator xref project mhtml-mode css-mode sgml-mode
eww mm-url gnus nnheader gnus-util rmail rmail-loaddefs rfc2047 rfc2045
ietf-drums mail-utils wid-edit mm-util mail-prsvr url-queue url
url-proxy url-privacy url-expand url-methods url-history mailcap shr
text-property-search url-cookie url-domsuf url-util puny svg xml dom
sh-script smie executable markdown-mode color thingatpt noutline outline
dired dired-loaddefs js cc-mode cc-fonts cc-guess cc-menus cc-cmds
cc-styles cc-align cc-engine cc-vars cc-defs make-mode smerge-mode diff
prolog align imenu vc-git diff-mode easy-mmode python tramp-sh tramp
tramp-loaddefs trampver tramp-integration files-x tramp-compat shell
pcomplete parse-time iso8601 time-date ls-lisp format-spec finder-inf
cl-extra help-mode autorevert filenotify grep compile comint ansi-color
ring cus-start cus-load paren time savehist desktop frameset info
package easymenu browse-url url-handlers url-parse auth-source cl-seq
eieio eieio-core cl-macs eieio-loaddefs password-cache json subr-x map
url-vars seq byte-opt gv bytecomp byte-compile cconv cl-loaddefs cl-lib
tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type
mwheel term/x-win x-win term/common-win x-dnd tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode elisp-mode
lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow isearch
timer select scroll-bar mouse jit-lock font-lock syntax facemenu
font-core term/tty-colors frame minibuffer cl-generic cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese composite charscript charprop
case-table epa-hook jka-cmpr-hook help simple abbrev obarray
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote threads dbusbind
inotify dynamic-setting system-font-setting font-render-setting
move-toolbar gtk x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 553627 84742)
 (symbols 48 27664 5)
 (strings 32 109868 12210)
 (string-bytes 1 4407199)
 (vectors 16 47901)
 (vector-slots 8 1347908 158898)
 (floats 8 232 330)
 (intervals 56 48562 536)
 (buffers 1000 503))

[-- Attachment #2: Type: text/html, Size: 6379 bytes --]

bug#26952: Possible regression of Bug#26952 - "repeated buffer insertion consumes excessive memory"

[Peter Ludemann]

[-- Attachment #1: Type: text/plain, Size: 38 bytes --]

Opening a new bug for this: bug#39045

[-- Attachment #2: Type: text/html, Size: 131 bytes --]

bug#39045: Fwd: Possible regression - "repeated buffer insertion consumes excessive memory"

[Lars Ingebrigtsen]

Peter Ludemann <peter.ludemann@gmail.com> writes:

> I'm observing creeping memory increase (at one point, emacs was using
> over 3.8GB before I restarted it), possibly related to running
> compilations with large amounts of output. Seems related to Bug#26952
> (which my Bug#38629 seemed to be a duplicate of), and which was fixed
> in emacs 26.3. It seems to have reappeared in emacs 28.0.50, but with
> a difference -- I'm not seeing the high CPU usage that I previously
> observed with the high memory use (perhaps this is because
> global-auto-revert has been improved in 28.0.50).
>
> What information should I collect to confirm this problem and to help
> someone debug it?

(I'm going through old bug reports that unfortunately weren't resolved
at the time.)

Do you still see this problem in Emacs 28.1?  If so, do you have a
recipe to reproduce it, starting from "emacs -Q"?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#39045: Fwd: Possible regression - "repeated buffer insertion consumes excessive memory"

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Do you still see this problem in Emacs 28.1?  If so, do you have a
> recipe to reproduce it, starting from "emacs -Q"?

More information was requested, but no response was given within a
month, so I'm closing this bug report.  If the problem still exists,
please respond to this email and we'll reopen the bug report.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no