Emacs on Android OOMs with pdf-tools. (And an Android howto.)

Emacs on Android OOMs with pdf-tools. (And an Android howto.)

[Vladimir Nikishkin]

Hello everyone.

This is possibly not a problem with emacs, but my setup is so
unconventional that I suspect that people here might get interested even if
unable to help.

I am using Emacs-master on Android 6, re-signed to be used together with
termux.

This is a howto on how I am doing this:


https://lockywolf.net/2024-08-07_Using-an-ebook-instead-of-a-laptop.d/index.html#pdf-tools

The html anchor is already pointing to the problematic place: Emacs OOMs
with pdf-tools (compiled to use poppler from termux).

At the same time, doc-view-mode does not oom even on books with hundreds of
pages.

Does anyone have an idea on how to debug this?

I have progressed a little bit by disabling pdf-tools cache, and trying
imagemagick. But I still haven't managed to get to the end of even a small
pdf of 22 pages.

Thanks!

Re: Emacs on Android OOMs with pdf-tools. (And an Android howto.)

[Stefan Monnier via Users list for the GNU Emacs text editor]

> I have progressed a little bit by disabling pdf-tools cache, and trying
> imagemagick. But I still haven't managed to get to the end of even a small
> pdf of 22 pages.

Can you compare the size of the process (RSS, VMEM, M-x memory-report,
...) "just "before" it crashes with the size of a similar Emacs in (say)
GNU/Linux doing the same thing?


        Stefan

Re: Emacs on Android OOMs with pdf-tools. (And an Android howto.)

[Po Lu]

Stefan Monnier via Users list for the GNU Emacs text editor
<help-gnu-emacs@gnu.org> writes:

>> I have progressed a little bit by disabling pdf-tools cache, and trying
>> imagemagick. But I still haven't managed to get to the end of even a small
>> pdf of 22 pages.

To whomever the OP is (I cannot find his e-mail),

> Can you compare the size of the process (RSS, VMEM, M-x memory-report,
> ...) "just "before" it crashes with the size of a similar Emacs in (say)
> GNU/Linux doing the same thing?

How much memory does your device provide?

Also, I recommend against comparing Emacs's RSS on Android with that on
other systems, since the former includes graphics memory that is
allocated in enormous quantities to foreground user applications and
released as demand increases elsewhere--it's much better simply to refer
to the size of the image cache printed by memory-report.

Re: Emacs on Android OOMs with pdf-tools. (And an Android howto.)

[Lockywolf]

Date: Thu, 22 Aug 2024 08:12:44 +0800

Thanks everyone for the memory-report command, I hadn't known about it
before this case.

>How much memory does your device provide?

The device has 2 Gb of RAM, of which about 1.5 Gb are ordinarily used by
the system and built-in annoyances.
I am trying to _only_ use Emacs on this device, so it should have about
500 Mb left.

But I am pretty sure it is not actually the problem of not enough
memory, as memory-report is clearly showing that something is wrong.

The two reports, before and after the OOM message, are below.
The concerning bit seems to be: "256 MiB  Total Image Cache Size"
256 Mb is a significant number for this device. Below I am also
pasting memory-report with doc-view-mode, which seems to "only" generate
160 Mb of image-cache.

Is it possible to tell Emacs to not generate so much cache?


* Before

Estimated Emacs Memory Usage

     5 MiB  Overall Object Memory Usage
   2.3 MiB  Memory Used By Global Variables
   1.1 MiB  Memory Used By Symbol Plists
   451 KiB  Total Image Cache Size
   237 KiB  Reserved (But Unused) Object Memory
    36 KiB  Total Buffer Memory Usage

Object Storage

   2.2 MiB  Strings
   1.2 MiB  Conses
   1.2 MiB  Vectors
   374 KiB  Symbols
     6 KiB  Intervals
     5 KiB  Buffer-Objects
   1.1 KiB  Floats

Largest Buffers

    11 KiB  andrei
   9.6 KiB  *scratch*
   6.1 KiB  *Messages*
   2.4 KiB   *Minibuf-1*
   2.1 KiB   *code-conversion-work*
   2.1 KiB   *Echo Area 0*
   2.1 KiB   *Echo Area 1*
     203 B  *Memory Report*
     173 B   *Minibuf-0*

Largest Variables

   356 KiB  package-archive-contents
   319 KiB  load-history
   115 KiB  definition-prefixes
   107 KiB  easy-menu-converted-items-table
    79 KiB  org-entities
    64 KiB  widget-global-map
    64 KiB  help-quick-use-map
    64 KiB  global-map
    60 KiB  face--new-frame-defaults
    55 KiB  input-method-alist
    51 KiB  package--builtins
    47 KiB  coding-system-alist
    41 KiB  language-info-alist
    41 KiB  color-name-rgb-alist
    25 KiB  menu-bar-options-menu
    24 KiB  x-colors
    22 KiB  key-translation-map
    21 KiB  org-mode-map
    20 KiB  package--compatibility-table
    19 KiB  minor-mode-map-alist



* After

Estimated Emacs Memory Usage

   256 MiB  Total Image Cache Size
   9.2 MiB  Overall Object Memory Usage
   2.7 MiB  Memory Used By Global Variables
   1.3 MiB  Memory Used By Symbol Plists
   803 KiB  Total Buffer Memory Usage
   245 KiB  Reserved (But Unused) Object Memory

Object Storage

   4.7 MiB  Strings
   2.3 MiB  Vectors
   1.7 MiB  Conses
   485 KiB  Symbols
    12 KiB  Floats
    10 KiB  Intervals
   8.9 KiB  Buffer-Objects

Largest Buffers

   744 KiB  mutual-information-Romaschenko_1710.05984.pdf
    14 KiB  memory-report-before-pdf-tools.txt
    11 KiB  andrei
   9.6 KiB  *scratch*
   6.1 KiB  *Messages*
   2.4 KiB   *Minibuf-1*
   2.2 KiB   *eldoc for major-mode*
   2.1 KiB   *Echo Area 0*
   2.1 KiB   *pdf-info-query--escape*
   2.1 KiB   *code-conversion-work*
   2.1 KiB   *Echo Area 1*
   2.1 KiB   tq-temp-epdfinfo
   2.1 KiB   *pdf-info-query--parse-response*
     203 B  *Memory Report*
     173 B   *Minibuf-0*
      96 B   *epdfinfo*

Largest Variables

   369 KiB  load-history
   356 KiB  package-archive-contents
   148 KiB  easy-menu-converted-items-table
   115 KiB  definition-prefixes
   102 KiB  minor-mode-map-alist
    79 KiB  org-entities
    68 KiB  face--new-frame-defaults
    64 KiB  tablist-edit-column-minor-mode-map
    64 KiB  widget-global-map
    64 KiB  help-quick-use-map
    64 KiB  global-map
    55 KiB  input-method-alist
    51 KiB  package--builtins
    47 KiB  coding-system-alist
    41 KiB  language-info-alist
    41 KiB  color-name-rgb-alist
    27 KiB  custom-face-all
    25 KiB  menu-bar-options-menu
    24 KiB  x-colors
    22 KiB  key-translation-map

* doc-view-mode

Estimated Emacs Memory Usage

   160 MiB  Total Image Cache Size
     6 MiB  Overall Object Memory Usage
   2.3 MiB  Memory Used By Global Variables
   1.1 MiB  Memory Used By Symbol Plists
   568 KiB  Total Buffer Memory Usage
   361 KiB  Reserved (But Unused) Object Memory

Object Storage

   2.3 MiB  Strings
   2.1 MiB  Vectors
   1.2 MiB  Conses
   384 KiB  Symbols
   6.4 KiB  Intervals
   5.5 KiB  Buffer-Objects
   1.4 KiB  Floats

Largest Buffers

   531 KiB  mutual-information-Romaschenko_1710.05984.pdf
    12 KiB  andrei
   9.8 KiB  *scratch*
   6.1 KiB  *Messages*
   2.4 KiB   *Minibuf-1*
   2.1 KiB   *code-conversion-work*
   2.1 KiB   *Echo Area 0*
   2.1 KiB   *Echo Area 1*
     203 B  *Memory Report*
     173 B   *Minibuf-0*

Largest Variables

   356 KiB  package-archive-contents
   323 KiB  load-history
   119 KiB  easy-menu-converted-items-table
   115 KiB  definition-prefixes
    79 KiB  org-entities
    64 KiB  widget-global-map
    64 KiB  help-quick-use-map
    64 KiB  global-map
    60 KiB  face--new-frame-defaults
    55 KiB  input-method-alist
    51 KiB  package--builtins
    47 KiB  coding-system-alist
    41 KiB  language-info-alist
    41 KiB  color-name-rgb-alist
    25 KiB  menu-bar-options-menu
    24 KiB  x-colors
    22 KiB  key-translation-map
    21 KiB  org-mode-map
    20 KiB  package--compatibility-table
    20 KiB  minor-mode-map-alist




-- 
Your sincerely,
Vladimir Nikishkin (MiEr, lockywolf)
(Laptop)

Re: Emacs on Android OOMs with pdf-tools. (And an Android howto.)

[Eli Zaretskii]

> From: Lockywolf <lockywolf@gmail.com>
> Date: Thu, 22 Aug 2024 08:50:42 +0800
> 
> Is it possible to tell Emacs to not generate so much cache?

Try playing with the value of image-cache-eviction-delay.

But how did you get to so many images? what images are they and what
did you do in Emacs to have them all loaded?  Emacs is not meant to be
used with so many images on a device that has such severe memory
limitations.

More importantly, if the OOM mechanism of the OS kicks in while Emacs
still thinks it has more memory to use, it means something is wrong
with how the available memory is being reported to Emacs.  Emacs
should have sensed that it is approaching the limit and signaled the
memory-full signal before that happens.  I think this warrants a bug
report, with a reproducer.

Re: Emacs on Android OOMs with pdf-tools. (And an Android howto.)

[Po Lu]

Lockywolf <lockywolf@gmail.com> writes:

> Date: Thu, 22 Aug 2024 08:12:44 +0800
>
> Thanks everyone for the memory-report command, I hadn't known about it
> before this case.
>
>>How much memory does your device provide?
>
> The device has 2 Gb of RAM, of which about 1.5 Gb are ordinarily used by
> the system and built-in annoyances.
> I am trying to _only_ use Emacs on this device, so it should have about
> 500 Mb left.
>
> But I am pretty sure it is not actually the problem of not enough
> memory, as memory-report is clearly showing that something is wrong.
>
> The two reports, before and after the OOM message, are below.
> The concerning bit seems to be: "256 MiB  Total Image Cache Size"
> 256 Mb is a significant number for this device. Below I am also
> pasting memory-report with doc-view-mode, which seems to "only" generate
> 160 Mb of image-cache.
>
> Is it possible to tell Emacs to not generate so much cache?

Please try Eli's advice, but also please run `adb logcat' (from a PC)
after Emacs signals that memory has been exhausted, and verify that no
line reading "Possible out of memory error" exists which does not
immediately precede a java.lang.OutOfMemoryError, since Emacs assumes by
default that any unknown Java exception is an out-of-memory error.

Re: Emacs on Android OOMs with pdf-tools. (And an Android howto.)

[Lockywolf Laptop]

[-- Attachment #1: Type: text/plain, Size: 9300 bytes --]


(my answer to Po Lu is below)

To Eli:

>if the OOM mechanism of the OS kicks in while Emacs
>still thinks it has more memory to use, it means something is wrong
>with how the available memory is being reported to Emacs.  Emacs
>should have sensed that it is approaching the limit and signaled the
>memory-full signal before that happens.  I think this warrants a bug
>report, with a reproducer.

Emacs doesn't actually crash or doesn't get killed because of OOM.
Rather it produces a message of something like "Memory exhausted, run
C-x C-s, and restart Emacs". Something does get broken at this point,
as, for example, not all faces get rendered, but overall Emacs works.

I am not sure how how bad this is.

>Try playing with the value of image-cache-eviction-delay.

I ended up doing like this in init.el:

#+begin_src elisp
(setf pdf-cache-image-limit 0)
(setq-default pdf-view-display-size 'fit-height)
(setq-default pdf-view-max-image-width 2000)
(setq-default pdf-cache-image-inihibit t)
(advice-add 'pdf-view-redisplay :before #'clear-image-cache)
(advice-add 'pdf-view-redisplay :after #'clear-image-cache)
#+end_src

This works. It's a bit slow, but I can read a whole book of 400 pages
with no issues. Each page generates about 100 Mb of cache, but since
it's evicted after rendering, overall the memory consumption is not that
high, about 16 Mb of image cache.

>But how did you get to so many images? what images are they and what
>did you do in Emacs to have them all loaded?  Emacs is not meant to be
>used with so many images on a device that has such severe memory
>limitations.

Well, this thread is titled "using pdf-tools on Android 6" :).

This is an eBook reader Onyx Boox Max 2, with an eInk display. I am
using it because it has a reflective display that is so-so-so much
better for my strained eyes that I am ready to suffer quite some
hardships to make it applicable to many of my daily tasks.

So I was trying to use pdf-tools to read pdfs on it, with Emacs. Its
native PDF reader works quite okay, the problem is that it's Android
software, and not aimed at power users, so its integration options are
limited. I have written quite a lot about this tool in the howto.

pdf-tools is somehow smarter than doc-view-mode. Doc-view-mode, as far
as I understand, generates a png file with ghostscript out of each page
of the pdf file, whereas pdf-tools uses poppler, and renders much more
than just an image. In particular, it allows for pdf annotations, and
can be integrated with auctex, et cetera.

I have not managed to make pdf-annotating features of pdf-tools work on
this eBook, Emacs just hangs if I try to touch the screen with a pen,
and can be unfrozen with C-g. But so far I didn't really need it, as I
seldom annotate PDFs using PDF built-in annotation features. I prefer to
pass a pdf through mathpix (or some other de-texifying OCR, or even
better, just ask the author to send me the source), and write
my annotations right into the resulting TeX code.


Po Lu <luangruo@yahoo.com> writes:

> Lockywolf <lockywolf@gmail.com> writes:
>
>> Date: Thu, 22 Aug 2024 08:12:44 +0800
>>
>> Thanks everyone for the memory-report command, I hadn't known about it
>> before this case.
>>
>>>How much memory does your device provide?
>>
>> The device has 2 Gb of RAM, of which about 1.5 Gb are ordinarily used by
>> the system and built-in annoyances.
>> I am trying to _only_ use Emacs on this device, so it should have about
>> 500 Mb left.
>>
>> But I am pretty sure it is not actually the problem of not enough
>> memory, as memory-report is clearly showing that something is wrong.
>>
>> The two reports, before and after the OOM message, are below.
>> The concerning bit seems to be: "256 MiB  Total Image Cache Size"
>> 256 Mb is a significant number for this device. Below I am also
>> pasting memory-report with doc-view-mode, which seems to "only" generate
>> 160 Mb of image-cache.
>>
>> Is it possible to tell Emacs to not generate so much cache?
>
> Please try Eli's advice, but also please run `adb logcat' (from a PC)
> after Emacs signals that memory has been exhausted, and verify that no
> line reading "Possible out of memory error" exists which does not
> immediately precede a java.lang.OutOfMemoryError, since Emacs assumes by
> default that any unknown Java exception is an out-of-memory error.

To Po Lu:

Adb logcat produces the following message:

08-22 14:05:09.144 20974 20987 I art     : Starting a blocking GC Alloc
08-22 14:05:09.144 20974 20987 I art     : Starting a blocking GC Alloc
08-22 14:05:09.149 20974 20987 I art     : Starting a blocking GC Alloc
08-22 14:05:09.160 20974 20987 I art     : Alloc partial concurrent mark sweep GC freed 383(124KB) AllocSpace objects, 21(336KB) LOS objects, 8% free, 175MB/191MB, paused 219us total 11.322ms
08-22 14:05:09.160 20974 20987 I art     : Starting a blocking GC Alloc
08-22 14:05:09.171 20974 20987 I art     : Alloc concurrent mark sweep GC freed 141(6KB) AllocSpace objects, 0(0B) LOS objects, 8% free, 175MB/191MB, paused 205us total 10.038ms
08-22 14:05:09.171 20974 20987 I art     : Forcing collection of SoftReferences for 23MB allocation
08-22 14:05:09.171 20974 20987 I art     : Starting a blocking GC Alloc
08-22 14:05:09.181 20974 20987 I art     : Alloc concurrent mark sweep GC freed 5(176B) AllocSpace objects, 0(0B) LOS objects, 8% free, 175MB/191MB, paused 214us total 9.850ms
08-22 14:05:09.181 20974 20987 W art     : Throwing OutOfMemoryError "Failed to allocate a 24687948 byte allocation with 16777056 free bytes and 16MB until OOM"
08-22 14:05:09.181 20974 20987 I art     : Starting a blocking GC Alloc
08-22 14:05:09.181 20974 20987 I art     : Starting a blocking GC Alloc
08-22 14:05:09.184 20974 20987 I art     : Starting a blocking GC Alloc
08-22 14:05:09.194 20974 20987 I art     : Alloc partial concurrent mark sweep GC freed 6(192B) AllocSpace objects, 0(0B) LOS objects, 8% free, 175MB/191MB, paused 207us total 10.227ms
08-22 14:05:09.194 20974 20987 I art     : Starting a blocking GC Alloc
08-22 14:05:09.204 20974 20987 I art     : Alloc concurrent mark sweep GC freed 3(96B) AllocSpace objects, 0(0B) LOS objects, 8% free, 175MB/191MB, paused 198us total 9.826ms
08-22 14:05:09.204 20974 20987 I art     : Forcing collection of SoftReferences for 23MB allocation
08-22 14:05:09.204 20974 20987 I art     : Starting a blocking GC Alloc
08-22 14:05:09.214 20974 20987 I art     : Alloc concurrent mark sweep GC freed 3(96B) AllocSpace objects, 0(0B) LOS objects, 8% free, 175MB/191MB, paused 207us total 9.940ms
08-22 14:05:09.214 20974 20987 W art     : Throwing OutOfMemoryError "Failed to allocate a 24687948 byte allocation with 16777056 free bytes and 16MB until OOM"
08-22 14:05:10.767 20974 20987 I art     : Starting a blocking GC Alloc
08-22 14:05:10.767 20974 20987 I art     : Starting a blocking GC Alloc
08-22 14:05:10.771 20974 20987 I art     : Starting a blocking GC Alloc
08-22 14:05:10.781 20974 20987 I art     : Alloc partial concurrent mark sweep GC freed 43(1712B) AllocSpace objects, 0(0B) LOS objects, 8% free, 175MB/191MB, paused 214us total 10.370ms
08-22 14:05:10.781 20974 20987 I art     : Starting a blocking GC Alloc
08-22 14:05:10.791 20974 20987 I art     : Alloc concurrent mark sweep GC freed 2(64B) AllocSpace objects, 0(0B) LOS objects, 8% free, 175MB/191MB, paused 195us total 9.755ms
08-22 14:05:10.791 20974 20987 I art     : Forcing collection of SoftReferences for 23MB allocation
08-22 14:05:10.791 20974 20987 I art     : Starting a blocking GC Alloc
08-22 14:05:10.801 20974 20987 I art     : Alloc concurrent mark sweep GC freed 3(96B) AllocSpace objects, 0(0B) LOS objects, 8% free, 175MB/191MB, paused 199us total 9.804ms
08-22 14:05:10.801 20974 20987 W art     : Throwing OutOfMemoryError "Failed to allocate a 24687948 byte allocation with 16777056 free bytes and 16MB until OOM"
08-22 14:05:10.801 20974 20987 I art     : Starting a blocking GC Alloc
08-22 14:05:10.801 20974 20987 I art     : Starting a blocking GC Alloc
08-22 14:05:10.804 20974 20987 I art     : Starting a blocking GC Alloc
08-22 14:05:10.814 20974 20987 I art     : Alloc partial concurrent mark sweep GC freed 6(192B) AllocSpace objects, 0(0B) LOS objects, 8% free, 175MB/191MB, paused 206us total 10.249ms
08-22 14:05:10.814 20974 20987 I art     : Starting a blocking GC Alloc
08-22 14:05:10.825 20974 20987 I art     : Alloc concurrent mark sweep GC freed 3(96B) AllocSpace objects, 0(0B) LOS objects, 8% free, 175MB/191MB, paused 200us total 9.893ms
08-22 14:05:10.825 20974 20987 I art     : Forcing collection of SoftReferences for 23MB allocation
08-22 14:05:10.825 20974 20987 I art     : Starting a blocking GC Alloc
08-22 14:05:10.835 20974 20987 I art     : Alloc concurrent mark sweep GC freed 3(96B) AllocSpace objects, 0(0B) LOS objects, 8% free, 175MB/191MB, paused 196us total 9.771ms
08-22 14:05:10.835 20974 20987 W art     : Throwing OutOfMemoryError "Failed to allocate a 24687948 byte allocation with 16777056 free bytes and 16MB until OOM"

I do not know whether this is helpful.


-- 
Your sincerely,
Vladimir Nikishkin (MiEr, lockywolf)
(Laptop)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Emacs on Android OOMs with pdf-tools. (And an Android howto.)

[Eli Zaretskii]

> From: Lockywolf Laptop <lockywolf@gmail.com>
> Cc: help-gnu-emacs@gnu.org, monnier@iro.umontreal.ca
> Date: Thu, 22 Aug 2024 14:06:53 +0800
> 
> >if the OOM mechanism of the OS kicks in while Emacs
> >still thinks it has more memory to use, it means something is wrong
> >with how the available memory is being reported to Emacs.  Emacs
> >should have sensed that it is approaching the limit and signaled the
> >memory-full signal before that happens.  I think this warrants a bug
> >report, with a reproducer.
> 
> Emacs doesn't actually crash or doesn't get killed because of OOM.
> Rather it produces a message of something like "Memory exhausted, run
> C-x C-s, and restart Emacs". Something does get broken at this point,
> as, for example, not all faces get rendered, but overall Emacs works.
> 
> I am not sure how how bad this is.

OK, it means at least Emacs knows about the memory problem and lets
you know, so you could end the session safely.

Thanks.