unofficial mirror of emacs-devel@gnu.org 
 help / color / mirror / code / Atom feed
* Emacs memory management and sluggishness
@ 2019-07-26 19:20 Óscar Fuentes
  2019-07-27  6:55 ` Eli Zaretskii
  0 siblings, 1 reply; 17+ messages in thread
From: Óscar Fuentes @ 2019-07-26 19:20 UTC (permalink / raw)
  To: emacs-devel

[I sent this message through gmane hours ago and so far it was not
posted, sorry for the duplicate if it finally arrives]

GNU Emacs 27.0.50 (build 1, x86_64-pc-linux-gnu, X toolkit) of
2019-06-29

M-x emacs-runtime says this session is running for 26 days. Light usage.
Right now its RSS is 1.1 GB. I don't open huge files (rarely they are
larger than 200 KB, except for Gnus (?) opening sent-news and sent-mail,
which are about 5MB each). Sometimes I read PDF documents with
PDF-Tools, which amounts to an image-viewer plus the length of the PDF
file, being the largest one 6 MB.

Since a week or two typing normal text (i.e. writing this message) even
in Fundamental Mode shows an annoying lag betwenn the keypress and the
character showing on the screen. Sometimes it affects entire key
sequences, and I'm no fast typist.

So my questions are: do we have a problem with memory fragmentation?
because with the same usage pattern the memory keeps growing.

And could this cause the lag I described above? At this moment, simply
typing text makes Emacs CPU usage go over 50% of a core (on a fast
desktop machine). This does not happen on a fresh instance with the same
configuration. I see the lag increases on correlation with the memory
increase. Apart from Evil-mode, I have no other edition-related modes
apart from what Emacs itself activates, and disabling Evil makes no
difference.

P.S.: profiling indicates that about 80% of the CPU samples are on
Automatic GC. Maybe the cause is running the GC over such large memory
area.

(garbage-collect) returns

((conses 16 4505904 1124616)
 (symbols 48 76877 18)
 (strings 32 340997 83083)
 (string-bytes 1 10456288)
 (vectors 16 688370)
 (vector-slots 8 24140524 2173062)
 (floats 8 1621 1825)
 (intervals 56 112123 94665)
 (buffers 992 449))

I wonder how this ends using 1.1 GB of RSS.

(memory-limit) returns

1874140 (#o7114334, #x1c98dc)

(memory-use-counts) returns

(3617372752 2210912 8232217138 506599 2689836943 19615159 67629016)

gc-elapsed returns

1901.1549781709962

htop says that emacs used 2h38m of CPU so far.



^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: Emacs memory management and sluggishness
  2019-07-26 19:20 Emacs memory management and sluggishness Óscar Fuentes
@ 2019-07-27  6:55 ` Eli Zaretskii
  2019-07-27 12:36   ` Óscar Fuentes
  0 siblings, 1 reply; 17+ messages in thread
From: Eli Zaretskii @ 2019-07-27  6:55 UTC (permalink / raw)
  To: Óscar Fuentes; +Cc: emacs-devel

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Fri, 26 Jul 2019 21:20:17 +0200
> 
> Since a week or two typing normal text (i.e. writing this message) even
> in Fundamental Mode shows an annoying lag betwenn the keypress and the
> character showing on the screen. Sometimes it affects entire key
> sequences, and I'm no fast typist.

Is there any correlation between the lag and the GC cycles?  (Turn on
garbage-collection-messages to see.)  If it does, then there's only
one problem to investigate: why you have frequent GC cycles.

And what is your value of gc-cons-threshold?

> So my questions are: do we have a problem with memory fragmentation?

We shouldn't have.  However, there's a question whether the particular
development snapshot of a month ago you are using had some
memory-related bug.

> ((conses 16 4505904 1124616)
>  (symbols 48 76877 18)
>  (strings 32 340997 83083)
>  (string-bytes 1 10456288)
>  (vectors 16 688370)
>  (vector-slots 8 24140524 2173062)
>  (floats 8 1621 1825)
>  (intervals 56 112123 94665)
>  (buffers 992 449))
> 
> I wonder how this ends using 1.1 GB of RSS.

It doesn't.  But all these measures don't include the memory we
allocate directly, not via Lisp objects.

Also, can you tell how many bytes of text do you have in all your
buffers?  E.g., with this toy program:

  (let ((stext 0))
    (dolist (b (buffer-list))
      (setq stext (+ stext (buffer-size))))
    stext)

Having said all that, it isn't clear to me that an RSS of 1.1 GB per
se is something exceptional in a long-running session.  E.g., the
Emacs 26.2.90 session where I'm typing this, which is running for 44
days, shows an RSS of 360 MB with VM Size of 514 MB, and this is a
32-bit build of Emacs --with-wide-int.  So your 1.1 GB is not
necessarily a sign of a problem.



^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: Emacs memory management and sluggishness
  2019-07-27  6:55 ` Eli Zaretskii
@ 2019-07-27 12:36   ` Óscar Fuentes
  2019-07-27 13:20     ` Eli Zaretskii
  0 siblings, 1 reply; 17+ messages in thread
From: Óscar Fuentes @ 2019-07-27 12:36 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: emacs-devel

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Óscar Fuentes <ofv@wanadoo.es>
>> Date: Fri, 26 Jul 2019 21:20:17 +0200
>> 
>> Since a week or two typing normal text (i.e. writing this message) even
>> in Fundamental Mode shows an annoying lag betwenn the keypress and the
>> character showing on the screen. Sometimes it affects entire key
>> sequences, and I'm no fast typist.
>
> Is there any correlation between the lag and the GC cycles?  (Turn on
> garbage-collection-messages to see.)

Ok. Too bad that the gc messages are not appended to *Messages*.

Indeed, the lag is caused by gc.

> If it does, then there's only
> one problem to investigate: why you have frequent GC cycles.
>
> And what is your value of gc-cons-threshold?

(setq gc-cons-threshold (* 20 1000 1000))

setting it to (* 2 1000 1000)) shortens the gc pauses, but I have the
impression that they occur with the same frecuency.

>> So my questions are: do we have a problem with memory fragmentation?
>
> We shouldn't have.  However, there's a question whether the particular
> development snapshot of a month ago you are using had some
> memory-related bug.
>
>> ((conses 16 4505904 1124616)
>>  (symbols 48 76877 18)
>>  (strings 32 340997 83083)
>>  (string-bytes 1 10456288)
>>  (vectors 16 688370)
>>  (vector-slots 8 24140524 2173062)
>>  (floats 8 1621 1825)
>>  (intervals 56 112123 94665)
>>  (buffers 992 449))
>> 
>> I wonder how this ends using 1.1 GB of RSS.
>
> It doesn't.  But all these measures don't include the memory we
> allocate directly, not via Lisp objects.
>
> Also, can you tell how many bytes of text do you have in all your
> buffers?  E.g., with this toy program:
>
>   (let ((stext 0))
>     (dolist (b (buffer-list))
>       (setq stext (+ stext (buffer-size))))
>     stext)

416484 (#o1455344, #x65ae4)

> Having said all that, it isn't clear to me that an RSS of 1.1 GB per
> se is something exceptional in a long-running session.  E.g., the
> Emacs 26.2.90 session where I'm typing this, which is running for 44
> days, shows an RSS of 360 MB with VM Size of 514 MB, and this is a
> 32-bit build of Emacs --with-wide-int.  So your 1.1 GB is not
> necessarily a sign of a problem.

Until now my experience was similar. Emacs memory usage increased
steadily to about 400 MB but then it stayed there. This is the first time
that I see it over 1 GB. The largest memory usage I observed until now
was 700 MB, but that was visiting very large PDF files and navigating
through many pages.

Since my previous message RSS increased 30 MB.



^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: Emacs memory management and sluggishness
  2019-07-27 12:36   ` Óscar Fuentes
@ 2019-07-27 13:20     ` Eli Zaretskii
  2019-07-27 23:52       ` Óscar Fuentes
  0 siblings, 1 reply; 17+ messages in thread
From: Eli Zaretskii @ 2019-07-27 13:20 UTC (permalink / raw)
  To: Óscar Fuentes; +Cc: emacs-devel

> From: Óscar Fuentes <ofv@wanadoo.es>
> Cc: emacs-devel@gnu.org
> Date: Sat, 27 Jul 2019 14:36:25 +0200
> 
> > Is there any correlation between the lag and the GC cycles?  (Turn on
> > garbage-collection-messages to see.)
> 
> Ok. Too bad that the gc messages are not appended to *Messages*.
> 
> Indeed, the lag is caused by gc.
> 
> > If it does, then there's only
> > one problem to investigate: why you have frequent GC cycles.
> >
> > And what is your value of gc-cons-threshold?
> 
> (setq gc-cons-threshold (* 20 1000 1000))
> 
> setting it to (* 2 1000 1000)) shortens the gc pauses, but I have the
> impression that they occur with the same frecuency.

OK, so it's one down, one to go; and I guess you should consider
lowering the threshold.

> > Having said all that, it isn't clear to me that an RSS of 1.1 GB per
> > se is something exceptional in a long-running session.  E.g., the
> > Emacs 26.2.90 session where I'm typing this, which is running for 44
> > days, shows an RSS of 360 MB with VM Size of 514 MB, and this is a
> > 32-bit build of Emacs --with-wide-int.  So your 1.1 GB is not
> > necessarily a sign of a problem.
> 
> Until now my experience was similar. Emacs memory usage increased
> steadily to about 400 MB but then it stayed there. This is the first time
> that I see it over 1 GB. The largest memory usage I observed until now
> was 700 MB, but that was visiting very large PDF files and navigating
> through many pages.
> 
> Since my previous message RSS increased 30 MB.

So I guess we have a memory leak somewhere.  Not sure if we have good
ways of finding them.  I think I'd suggest to update to the latest
master, and then track your memory footprint, waiting for the problem
to happen and taking notes of what features you invoke.  Once the
problem happens again, it would be good to know which features were
activated and which commands invoked since the last checkpoint, so we
could look into what these features/commands do.



^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: Emacs memory management and sluggishness
  2019-07-27 13:20     ` Eli Zaretskii
@ 2019-07-27 23:52       ` Óscar Fuentes
  2019-07-28  2:38         ` Eli Zaretskii
  0 siblings, 1 reply; 17+ messages in thread
From: Óscar Fuentes @ 2019-07-27 23:52 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: emacs-devel

Eli Zaretskii <eliz@gnu.org> writes:

>> Since my previous message RSS increased 30 MB.
>
> So I guess we have a memory leak somewhere.  Not sure if we have good
> ways of finding them.  I think I'd suggest to update to the latest
> master, and then track your memory footprint, waiting for the problem
> to happen and taking notes of what features you invoke.  Once the
> problem happens again, it would be good to know which features were
> activated and which commands invoked since the last checkpoint, so we
> could look into what these features/commands do.

It occurred to me to take a look at M-x list-timers and it showed about
700 (seven hundred) of them. The most frequent was about 300 repetitions
of

   *       0.05        t aggressive-indent--indent-if-changed

which is created by aggressive-indent-mode, that I use on a subset of my
buffers, then several dozens of

   *       0.50        t pdf-cache--prefetch-start

which comes from pdf-tools, and dozens of

        6467.37        - password-cache-remove

(the Next number varies, of course). So I used cancel-function-timers to
get rid of all the instances of those 3 examples and the lag while
typing text went away. Then, on new Emacs sessions (both with -Q and
with my config) tried to replicate the accumulation of
aggressive-indent--indent-if-changed without success.

Then noticed that everytime I fetch news & email with Gnus 6 new
instances of password-cache-remove are added. I checked that this
behavior is present since at least emacs 26.1. It makes little sense to
me, but those timers will eventually expire. Why the other timers
accumulate, don't know.

Is it possible that the huge RAM usage is related to the existence of
those hundreds of timers?

Now I'll do what you advised: fresh build, fresh instance and watch for
memory (and timers) increase.

Thanks.



^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: Emacs memory management and sluggishness
  2019-07-27 23:52       ` Óscar Fuentes
@ 2019-07-28  2:38         ` Eli Zaretskii
  2019-07-31 20:54           ` Joseph Mingrone
  0 siblings, 1 reply; 17+ messages in thread
From: Eli Zaretskii @ 2019-07-28  2:38 UTC (permalink / raw)
  To: Óscar Fuentes; +Cc: emacs-devel

> From: Óscar Fuentes <ofv@wanadoo.es>
> Cc: emacs-devel@gnu.org
> Date: Sun, 28 Jul 2019 01:52:19 +0200
> 
> It occurred to me to take a look at M-x list-timers and it showed about
> 700 (seven hundred) of them. The most frequent was about 300 repetitions
> of
> 
>    *       0.05        t aggressive-indent--indent-if-changed
> 
> which is created by aggressive-indent-mode, that I use on a subset of my
> buffers, then several dozens of
> 
>    *       0.50        t pdf-cache--prefetch-start
> 
> which comes from pdf-tools, and dozens of
> 
>         6467.37        - password-cache-remove
> 
> (the Next number varies, of course). So I used cancel-function-timers to
> get rid of all the instances of those 3 examples and the lag while
> typing text went away.

Yes, excess number of timers can produce lags.

> Now I'll do what you advised: fresh build, fresh instance and watch for
> memory (and timers) increase.

Suggest to start a bug report and accumulate all the data there.



^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: Emacs memory management and sluggishness
  2019-07-28  2:38         ` Eli Zaretskii
@ 2019-07-31 20:54           ` Joseph Mingrone
  2019-07-31 22:00             ` Óscar Fuentes
  0 siblings, 1 reply; 17+ messages in thread
From: Joseph Mingrone @ 2019-07-31 20:54 UTC (permalink / raw)
  To: emacs-devel

[-- Attachment #1: Type: text/plain, Size: 1039 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:
> Suggest to start a bug report and accumulate all the data there.

Hi,

I am seeing even more aggressive memory usage with recent master branch
commits (at least I think this only started in the last month or so).
Here is a screenshot that shows over 3 GB of
RSS. https://ftfl.ca/misc/emacs_memory.png

This was running commit 06726f6 from today on 12.0-RELEASE-p8.  I run
Gnus, erc, look at an occasional PDF and edit text.  I haven't visited
any extremely large buffers recently.  Garbage collection is happening,
but it does not seem to help.  But, when I call `garbage-collect'
manually there will be a long pause if I haven't called it recently,
then the memory will usually go down to something between 100 and 400
MB.

I just ran your lisp snippet a few times within a minute or so and
without really doing much it reports values between 36195 and 244723 and
it seems to be growing.

If there is an open bug, I will add this information.  If there is not,
I will open one.

Regards,

--
Joseph

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 979 bytes --]

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: Emacs memory management and sluggishness
  2019-07-31 20:54           ` Joseph Mingrone
@ 2019-07-31 22:00             ` Óscar Fuentes
  2019-07-31 22:14               ` Joseph Mingrone
  0 siblings, 1 reply; 17+ messages in thread
From: Óscar Fuentes @ 2019-07-31 22:00 UTC (permalink / raw)
  To: Joseph Mingrone; +Cc: emacs-devel

Joseph Mingrone <jrm@ftfl.ca> writes:

> I am seeing even more aggressive memory usage with recent master branch
> commits (at least I think this only started in the last month or so).
> Here is a screenshot that shows over 3 GB of
> RSS. https://ftfl.ca/misc/emacs_memory.png
>
> This was running commit 06726f6 from today on 12.0-RELEASE-p8.  I run
> Gnus, erc, look at an occasional PDF and edit text.  I haven't visited
> any extremely large buffers recently.  Garbage collection is happening,
> but it does not seem to help.  But, when I call `garbage-collect'
> manually there will be a long pause if I haven't called it recently,
> then the memory will usually go down to something between 100 and 400
> MB.

Do the garbage collection, take note of the used memory, visit one of
those PDF files (preferably one with large, complex & colorful images)
move around for several pages and see if emacs' memory usage grows.



^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: Emacs memory management and sluggishness
  2019-07-31 22:00             ` Óscar Fuentes
@ 2019-07-31 22:14               ` Joseph Mingrone
  2019-07-31 22:19                 ` Óscar Fuentes
  0 siblings, 1 reply; 17+ messages in thread
From: Joseph Mingrone @ 2019-07-31 22:14 UTC (permalink / raw)
  To: Óscar Fuentes; +Cc: emacs-devel

[-- Attachment #1: Type: text/plain, Size: 1103 bytes --]

Óscar Fuentes <ofv@wanadoo.es> writes:

> Joseph Mingrone <jrm@ftfl.ca> writes:

>> I am seeing even more aggressive memory usage with recent master branch
>> commits (at least I think this only started in the last month or so).
>> Here is a screenshot that shows over 3 GB of
>> RSS. https://ftfl.ca/misc/emacs_memory.png

>> This was running commit 06726f6 from today on 12.0-RELEASE-p8.  I run
>> Gnus, erc, look at an occasional PDF and edit text.  I haven't visited
>> any extremely large buffers recently.  Garbage collection is happening,
>> but it does not seem to help.  But, when I call `garbage-collect'
>> manually there will be a long pause if I haven't called it recently,
>> then the memory will usually go down to something between 100 and 400
>> MB.

> Do the garbage collection, take note of the used memory, visit one of
> those PDF files (preferably one with large, complex & colorful images)
> move around for several pages and see if emacs' memory usage grows.

It went from 440 to about 500 MB.  This was a book with 1200
pages and the PDF was about 3 MB.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 979 bytes --]

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: Emacs memory management and sluggishness
  2019-07-31 22:14               ` Joseph Mingrone
@ 2019-07-31 22:19                 ` Óscar Fuentes
  2019-07-31 22:27                   ` Joseph Mingrone
  0 siblings, 1 reply; 17+ messages in thread
From: Óscar Fuentes @ 2019-07-31 22:19 UTC (permalink / raw)
  To: Joseph Mingrone; +Cc: emacs-devel

Joseph Mingrone <jrm@ftfl.ca> writes:

> Óscar Fuentes <ofv@wanadoo.es> writes:
>
>> Joseph Mingrone <jrm@ftfl.ca> writes:
>
>>> I am seeing even more aggressive memory usage with recent master branch
>>> commits (at least I think this only started in the last month or so).
>>> Here is a screenshot that shows over 3 GB of
>>> RSS. https://ftfl.ca/misc/emacs_memory.png
>
>>> This was running commit 06726f6 from today on 12.0-RELEASE-p8.  I run
>>> Gnus, erc, look at an occasional PDF and edit text.  I haven't visited
>>> any extremely large buffers recently.  Garbage collection is happening,
>>> but it does not seem to help.  But, when I call `garbage-collect'
>>> manually there will be a long pause if I haven't called it recently,
>>> then the memory will usually go down to something between 100 and 400
>>> MB.
>
>> Do the garbage collection, take note of the used memory, visit one of
>> those PDF files (preferably one with large, complex & colorful images)
>> move around for several pages and see if emacs' memory usage grows.
>
> It went from 440 to about 500 MB.  This was a book with 1200
> pages and the PDF was about 3 MB.

How many pages of those 1200 you viewed? Do you use PDF-Tools or Doc
View? If you execute `garbage-collect' does the memory go down?

I'm asking all this because one of the possibilities is a leak on the
image handling code.



^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: Emacs memory management and sluggishness
  2019-07-31 22:19                 ` Óscar Fuentes
@ 2019-07-31 22:27                   ` Joseph Mingrone
  2019-08-01  2:43                     ` Eli Zaretskii
  0 siblings, 1 reply; 17+ messages in thread
From: Joseph Mingrone @ 2019-07-31 22:27 UTC (permalink / raw)
  To: Óscar Fuentes; +Cc: emacs-devel

[-- Attachment #1: Type: text/plain, Size: 1846 bytes --]

Óscar Fuentes <ofv@wanadoo.es> writes:

> Joseph Mingrone <jrm@ftfl.ca> writes:

>> Óscar Fuentes <ofv@wanadoo.es> writes:

>>> Joseph Mingrone <jrm@ftfl.ca> writes:

>>>> I am seeing even more aggressive memory usage with recent master branch
>>>> commits (at least I think this only started in the last month or so).
>>>> Here is a screenshot that shows over 3 GB of
>>>> RSS. https://ftfl.ca/misc/emacs_memory.png

>>>> This was running commit 06726f6 from today on 12.0-RELEASE-p8.  I run
>>>> Gnus, erc, look at an occasional PDF and edit text.  I haven't visited
>>>> any extremely large buffers recently.  Garbage collection is happening,
>>>> but it does not seem to help.  But, when I call `garbage-collect'
>>>> manually there will be a long pause if I haven't called it recently,
>>>> then the memory will usually go down to something between 100 and 400
>>>> MB.

>>> Do the garbage collection, take note of the used memory, visit one of
>>> those PDF files (preferably one with large, complex & colorful images)
>>> move around for several pages and see if emacs' memory usage grows.

>> It went from 440 to about 500 MB.  This was a book with 1200
>> pages and the PDF was about 3 MB.

> How many pages of those 1200 you viewed? Do you use PDF-Tools or Doc
> View? If you execute `garbage-collect' does the memory go down?

> I'm asking all this because one of the possibilities is a leak on the
> image handling code.

I just tried it again and held down 'n' to visit all pages and the
memory stayed under 500 MB.  I am using pdf-tools v0.90.  I also just
tried with
https://file-examples.com/wp-content/uploads/2017/10/file-example_PDF_1MB.pdf
and the memory stayed under 400 MB.  I'm somewhat confident that I
encountered memory issues in an Emacs instance without visiting a PDF
file.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 979 bytes --]

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: Emacs memory management and sluggishness
  2019-07-31 22:27                   ` Joseph Mingrone
@ 2019-08-01  2:43                     ` Eli Zaretskii
  2019-08-01  9:46                       ` Joseph Mingrone
  0 siblings, 1 reply; 17+ messages in thread
From: Eli Zaretskii @ 2019-08-01  2:43 UTC (permalink / raw)
  To: Joseph Mingrone; +Cc: ofv, emacs-devel

> From: Joseph Mingrone <jrm@ftfl.ca>
> Date: Wed, 31 Jul 2019 19:27:06 -0300
> Cc: emacs-devel@gnu.org
> 
> > How many pages of those 1200 you viewed? Do you use PDF-Tools or Doc
> > View? If you execute `garbage-collect' does the memory go down?
> 
> > I'm asking all this because one of the possibilities is a leak on the
> > image handling code.
> 
> I just tried it again and held down 'n' to visit all pages and the
> memory stayed under 500 MB.  I am using pdf-tools v0.90.  I also just
> tried with
> https://file-examples.com/wp-content/uploads/2017/10/file-example_PDF_1MB.pdf
> and the memory stayed under 400 MB.  I'm somewhat confident that I
> encountered memory issues in an Emacs instance without visiting a PDF
> file.

What image library is used for that?  Is it ImageMagick or is it
something else?



^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: Emacs memory management and sluggishness
  2019-08-01  2:43                     ` Eli Zaretskii
@ 2019-08-01  9:46                       ` Joseph Mingrone
  2019-08-01 13:49                         ` Eli Zaretskii
  0 siblings, 1 reply; 17+ messages in thread
From: Joseph Mingrone @ 2019-08-01  9:46 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: ofv, emacs-devel

[-- Attachment #1: Type: text/plain, Size: 1218 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Joseph Mingrone <jrm@ftfl.ca>
>> Date: Wed, 31 Jul 2019 19:27:06 -0300
>> Cc: emacs-devel@gnu.org

>> > How many pages of those 1200 you viewed? Do you use PDF-Tools or Doc
>> > View? If you execute `garbage-collect' does the memory go down?

>> > I'm asking all this because one of the possibilities is a leak on the
>> > image handling code.

>> I just tried it again and held down 'n' to visit all pages and the
>> memory stayed under 500 MB.  I am using pdf-tools v0.90.  I also just
>> tried with
>> https://file-examples.com/wp-content/uploads/2017/10/file-example_PDF_1MB.pdf
>> and the memory stayed under 400 MB.  I'm somewhat confident that I
>> encountered memory issues in an Emacs instance without visiting a PDF
>> file.

> What image library is used for that?  Is it ImageMagick or is it
> something else?

From https://github.com/politza/pdf-tools

It uses poppler to render PDFs.  "...there is one feature (following
links of a PDF document by plain keystrokes) which requires
imagemagick’s convert utility."

When I woke this morning, memory usage went from about 200-300 MB to
about 3 GB overnight.  No PDFs were open.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 979 bytes --]

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: Emacs memory management and sluggishness
  2019-08-01  9:46                       ` Joseph Mingrone
@ 2019-08-01 13:49                         ` Eli Zaretskii
  2019-08-10  2:28                           ` Joseph Mingrone
  0 siblings, 1 reply; 17+ messages in thread
From: Eli Zaretskii @ 2019-08-01 13:49 UTC (permalink / raw)
  To: Joseph Mingrone; +Cc: ofv, emacs-devel

> From: Joseph Mingrone <jrm@ftfl.ca>
> Cc: ofv@wanadoo.es,  emacs-devel@gnu.org
> Date: Thu, 01 Aug 2019 06:46:13 -0300
> 
> > What image library is used for that?  Is it ImageMagick or is it
> > something else?
> 
> From https://github.com/politza/pdf-tools
> 
> It uses poppler to render PDFs.  "...there is one feature (following
> links of a PDF document by plain keystrokes) which requires
> imagemagick’s convert utility."
> 
> When I woke this morning, memory usage went from about 200-300 MB to
> about 3 GB overnight.  No PDFs were open.

Is it possible to use some memory-mapping tool to figure out which
part of the Emacs process uses most of those GBytes?  It would be
interesting to see if the data belongs to one of the libraries used by
Emacs or to the Emacs code itself.



^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: Emacs memory management and sluggishness
  2019-08-01 13:49                         ` Eli Zaretskii
@ 2019-08-10  2:28                           ` Joseph Mingrone
  2019-08-10  7:43                             ` Eli Zaretskii
  0 siblings, 1 reply; 17+ messages in thread
From: Joseph Mingrone @ 2019-08-10  2:28 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: ofv, emacs-devel

[-- Attachment #1: Type: text/plain, Size: 1208 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Joseph Mingrone <jrm@ftfl.ca>
>> Cc: ofv@wanadoo.es,  emacs-devel@gnu.org
>> Date: Thu, 01 Aug 2019 06:46:13 -0300

>> > What image library is used for that?  Is it ImageMagick or is it
>> > something else?

>> From https://github.com/politza/pdf-tools

>> It uses poppler to render PDFs.  "...there is one feature (following
>> links of a PDF document by plain keystrokes) which requires
>> imagemagick’s convert utility."

>> When I woke this morning, memory usage went from about 200-300 MB to
>> about 3 GB overnight.  No PDFs were open.

> Is it possible to use some memory-mapping tool to figure out which
> part of the Emacs process uses most of those GBytes?  It would be
> interesting to see if the data belongs to one of the libraries used by
> Emacs or to the Emacs code itself.

Not sure about the memory-mapping tool.

I reverted back to a commit from May 25 and I see a bit of sluggishness
whenever the garbage collector does it's job, but the memory usage does
not explode to several GB and take down the box.  If I can find some
time, I will try to bisect commits to determine what change introduced
the problem.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: Emacs memory management and sluggishness
  2019-08-10  2:28                           ` Joseph Mingrone
@ 2019-08-10  7:43                             ` Eli Zaretskii
  2019-08-11 20:30                               ` Joseph Mingrone
  0 siblings, 1 reply; 17+ messages in thread
From: Eli Zaretskii @ 2019-08-10  7:43 UTC (permalink / raw)
  To: Joseph Mingrone; +Cc: ofv, emacs-devel

> From: Joseph Mingrone <jrm@ftfl.ca>
> Cc: ofv@wanadoo.es,  emacs-devel@gnu.org
> Date: Fri, 09 Aug 2019 23:28:26 -0300
> 
> I reverted back to a commit from May 25 and I see a bit of sluggishness
> whenever the garbage collector does it's job, but the memory usage does
> not explode to several GB and take down the box.

Please state the commit to which you reverted, we have quite a few of
them on any given date.

Also, since you never opened a bug report, could you please show here
the details of your build as they are collected by report-emacs-bug?
Please do that both for the build that does and doesn't exhibit the
memory problems.

> If I can find some time, I will try to bisect commits to determine
> what change introduced the problem.

Thanks, that would help.

There have been changes recently to GC and hash-table management,
maybe they are responsible.  But I find it strange that we don't have
reports about this across the board, which seems to hint that some
local config details or specific use patterns are also needed to
trigger the problem.



^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: Emacs memory management and sluggishness
  2019-08-10  7:43                             ` Eli Zaretskii
@ 2019-08-11 20:30                               ` Joseph Mingrone
  0 siblings, 0 replies; 17+ messages in thread
From: Joseph Mingrone @ 2019-08-11 20:30 UTC (permalink / raw)
  To: emacs-devel

[-- Attachment #1: Type: text/plain, Size: 1238 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Joseph Mingrone <jrm@ftfl.ca>
>> Cc: ofv@wanadoo.es,  emacs-devel@gnu.org
>> Date: Fri, 09 Aug 2019 23:28:26 -0300

>> I reverted back to a commit from May 25 and I see a bit of sluggishness
>> whenever the garbage collector does it's job, but the memory usage does
>> not explode to several GB and take down the box.

> Please state the commit to which you reverted, we have quite a few of
> them on any given date.

> Also, since you never opened a bug report, could you please show here
> the details of your build as they are collected by report-emacs-bug?
> Please do that both for the build that does and doesn't exhibit the
> memory problems.

>> If I can find some time, I will try to bisect commits to determine
>> what change introduced the problem.

> Thanks, that would help.

> There have been changes recently to GC and hash-table management,
> maybe they are responsible.  But I find it strange that we don't have
> reports about this across the board, which seems to hint that some
> local config details or specific use patterns are also needed to
> trigger the problem.

For those interested, there is now a bug report.

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=37006

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 979 bytes --]

^ permalink raw reply	[flat|nested] 17+ messages in thread

end of thread, other threads:[~2019-08-11 20:30 UTC | newest]

Thread overview: 17+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2019-07-26 19:20 Emacs memory management and sluggishness Óscar Fuentes
2019-07-27  6:55 ` Eli Zaretskii
2019-07-27 12:36   ` Óscar Fuentes
2019-07-27 13:20     ` Eli Zaretskii
2019-07-27 23:52       ` Óscar Fuentes
2019-07-28  2:38         ` Eli Zaretskii
2019-07-31 20:54           ` Joseph Mingrone
2019-07-31 22:00             ` Óscar Fuentes
2019-07-31 22:14               ` Joseph Mingrone
2019-07-31 22:19                 ` Óscar Fuentes
2019-07-31 22:27                   ` Joseph Mingrone
2019-08-01  2:43                     ` Eli Zaretskii
2019-08-01  9:46                       ` Joseph Mingrone
2019-08-01 13:49                         ` Eli Zaretskii
2019-08-10  2:28                           ` Joseph Mingrone
2019-08-10  7:43                             ` Eli Zaretskii
2019-08-11 20:30                               ` Joseph Mingrone

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/emacs.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).