bug#69519: 1gb file too big for Emacs to handle?

bug#69519: 1gb file too big for Emacs to handle?

[Robert Boyer]

[-- Attachment #1: Type: text/plain, Size: 6245 bytes --]

Dear Emacs bug exterminators,

My problem persists.  I very strongly believe that I have an example of a
bug in Emacs.

I have previously reported this.  I got some advice.  It did not solve my
problem.

Today I got a new, much bigger and better Chromebook, again from Lenovo.
This
one cost me $300, but it has 8gb of core and tons more disk.

On my newest Chromebook I see the following:

> free
               total        used        free      shared  buff/cache
available
Mem:         6736092     1194036     4052888       52208     1489168
5542056
Swap:              0           0           0
>

I do not understand these things at all well, but it looks to me like I now
have 4gb of free space.

So I think it is fair for me to report my real problem, again, almost
exactly
as I did before.

The 1 gb file for which I give a url below does not seem to work in Emacs.

The file enumerates the primes below 10^9, so it would be very handy to
have around.

I can find the file, literally, into an Emacs buffer.

But then I cannot move to the bottom, i.e., using M->.  In fact, in the
attempt to
move to the bottom, things go so badly that Emacs freezes, and I
have to kill Emacs
by 'extraordinary measures'.

Is this file just too big for Emacs to handle?


https://drive.google.com/file/d/1IaRNZ1rUQAZ72A7rJYpescmnJhpuGliA/view?usp=sharing

Please, before you reply, fetch that file, find it literally, and see if
you can move to the bottom with M->.  I'd love to know whether you can do
that.

Thanks,

Bob

I enclose now the info that I believe you like to get with a bug report.

From: bob <bob@penguin>
To: bug-gnu-emacs@gnu.org
Subject: 28.2; 1gb file too big for Emacs to handle?
--text follows this line--




In GNU Emacs 28.2 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.37,
cairo version 1.16.0)
 of 2023-05-13, modified by Debian built on x86-ubc-01
Windowing system distributor 'The X.Org Foundation', version 11.0.12014000
System Description: Debian GNU/Linux 12 (bookworm)

Configured using:
 'configure --build x86_64-linux-gnu --prefix=/usr
 --sharedstatedir=/var/lib --libexecdir=/usr/libexec
 --localstatedir=/var/lib --infodir=/usr/share/info
 --mandir=/usr/share/man --with-libsystemd --with-pop=yes
 --enable-locallisppath=/etc/emacs:/usr/local/share/emacs/28.2/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/28.2/site-lisp:/usr/share/emacs/site-lisp
 --with-sound=alsa --without-gconf --with-mailutils
 --with-native-compilation --build x86_64-linux-gnu --prefix=/usr
 --sharedstatedir=/var/lib --libexecdir=/usr/libexec
 --localstatedir=/var/lib --infodir=/usr/share/info
 --mandir=/usr/share/man --with-libsystemd --with-pop=yes
 --enable-locallisppath=/etc/emacs:/usr/local/share/emacs/28.2/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/28.2/site-lisp:/usr/share/emacs/site-lisp
 --with-sound=alsa --without-gconf --with-mailutils
 --with-native-compilation --with-cairo --with-x=yes
 --with-x-toolkit=gtk3 --with-toolkit-scroll-bars 'CFLAGS=-g -O2
 -ffile-prefix-map=/build/emacs-mPr7Vr/emacs-28.2+1=.
 -fstack-protector-strong -Wformat -Werror=format-security -Wall'
 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2' LDFLAGS=-Wl,-z,relro'

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
JSON LCMS2 LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 M17N_FLT MODULES
NATIVE_COMP NOTIFY INOTIFY PDUMPER PNG RSVG SECCOMP SOUND THREADS TIFF
TOOLKIT_SCROLL_BARS X11 XDBE XIM XPM GTK3 ZLIB

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Shell

Minor modes in effect:
  shell-dirtrack-mode: t
  display-time-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny rfc822 mml mml-sec epa
derived epg rfc6068 epg-config gnus-util text-property-search mm-decode
mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader
sendmail comp comp-cstr warnings rx cl-extra help-mode time-date
cus-start etags fileloop generator xref project dired-aux cus-edit pp
cus-load wid-edit trace sh-script smie executable dired dired-loaddefs
cal-menu calendar cal-loaddefs ange-ftp shell pcomplete comint
ansi-color ring benchmark time rmail rmail-loaddefs rfc2047 rfc2045
ietf-drums mm-util mail-prsvr mail-utils face-remap finder-inf package
browse-url url url-proxy url-privacy url-expand url-methods url-history
url-cookie url-domsuf url-util mailcap url-handlers url-parse
auth-source cl-seq eieio eieio-core cl-macs eieio-loaddefs
password-cache json subr-x map url-vars seq byte-opt gv bytecomp
byte-compile cconv cl-loaddefs cl-lib iso-transl tooltip eldoc paren
electric uniquify ediff-hook vc-hooks lisp-float-type elisp-mode mwheel
term/x-win x-win term/common-win x-dnd tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu
timer select scroll-bar mouse jit-lock font-lock syntax font-core
term/tty-colors frame minibuffer cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite emoji-zwj charscript charprop case-table
epa-hook jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice
button loaddefs faces cus-face macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote threads dbusbind inotify lcms2
dynamic-setting system-font-setting font-render-setting cairo
move-toolbar gtk x-toolkit x multi-tty make-network-process
native-compile emacs)

Memory information:
((conses 16 155179 10068)
 (symbols 48 12889 1)
 (strings 32 39546 3792)
 (string-bytes 1 1274680)
 (vectors 16 25493)
 (vector-slots 8 454855 22628)
 (floats 8 52 30)
 (intervals 56 1484 0)
 (buffers 992 22))

[-- Attachment #2: Type: text/html, Size: 8301 bytes --]

bug#69519: 1gb file too big for Emacs to handle?

[Eli Zaretskii]

> From: Robert Boyer <robertstephenboyer@gmail.com>
> Date: Sat, 2 Mar 2024 22:39:29 -0600
> 
> My problem persists.  I very strongly believe that I have an example of a bug in Emacs.
> 
> I have previously reported this.  I got some advice.  It did not solve my problem.
> 
> Today I got a new, much bigger and better Chromebook, again from Lenovo.  This
> one cost me $300, but it has 8gb of core and tons more disk.
> 
> On my newest Chromebook I see the following:
> 
> > free
>                total        used        free      shared  buff/cache   available
> Mem:         6736092     1194036     4052888       52208     1489168     5542056
> Swap:              0           0           0
> >
> 
> I do not understand these things at all well, but it looks to me like I now
> have 4gb of free space.
> 
> So I think it is fair for me to report my real problem, again, almost exactly
> as I did before.
> 
> The 1 gb file for which I give a url below does not seem to work in Emacs.
> 
> The file enumerates the primes below 10^9, so it would be very handy to
> have around.
> 
> I can find the file, literally, into an Emacs buffer.
> 
> But then I cannot move to the bottom, i.e., using M->.  In fact, in the attempt to
> move to the bottom, things go so badly that Emacs freezes, and I have to kill Emacs
> by 'extraordinary measures'.

How long did you wait?  Unless it's for an hour or so, this could be
just some slow operation.  Does the system page during this (do you
see the hard disk LED light more or less constantly)?  This could be
one reason for the slowness.

How many lines does this file have?  If its lines are very long, this
could be a known slowness in the display engine.

> Is this file just too big for Emacs to handle?

No.

>   https://drive.google.com/file/d/1IaRNZ1rUQAZ72A7rJYpescmnJhpuGliA/view?usp=sharing
> 
> Please, before you reply, fetch that file, find it literally, and see if
> you can move to the bottom with M->.  I'd love to know whether you can do that.

I cannot afford downloading such a huge file, sorry.  Maybe someone
else can.

> In GNU Emacs 28.2 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.37, cairo version 1.16.0)
>  of 2023-05-13, modified by Debian built on x86-ubc-01
> Windowing system distributor 'The X.Org Foundation', version 11.0.12014000
> System Description: Debian GNU/Linux 12 (bookworm)

Emacs 28 is no longer maintained.  But I don't think anything has
changed since then in how we handle large files.  However, if the
lines in the file are very long, you will be better off using Emacs 29
where there are special features for speeding up the display of very
long lines.

bug#69519: 1gb file too big for Emacs to handle?

[Robert Boyer]

[-- Attachment #1: Type: text/plain, Size: 3680 bytes --]

> How long did you wait?

I do not think that matters in this case.  The Emacs process
had gone  dead or crazy, and the operating system said that it was
not responding and offered to 'Force Close' the process, and
so I said go ahead and do it.

> Emacs 28 is no longer maintained.  But I don't think anything has
> changed since then in how we handle large files.  However, if the
> lines in the file are very long, you will be better off using Emacs 29
> where there are special features for speeding up the display of very
> long lines.

Long lines may be the problem!

I am utterly dependent upon Debian concerning the version of Emacs
I use.  If and when I get 29, I'll give the file another try.

Thank you,,

Bob


On Sun, Mar 3, 2024 at 1:39 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Robert Boyer <robertstephenboyer@gmail.com>
> > Date: Sat, 2 Mar 2024 22:39:29 -0600
> >
> > My problem persists.  I very strongly believe that I have an example of
> a bug in Emacs.
> >
> > I have previously reported this.  I got some advice.  It did not solve
> my problem.
> >
> > Today I got a new, much bigger and better Chromebook, again from
> Lenovo.  This
> > one cost me $300, but it has 8gb of core and tons more disk.
> >
> > On my newest Chromebook I see the following:
> >
> > > free
> >                total        used        free      shared  buff/cache
>  available
> > Mem:         6736092     1194036     4052888       52208     1489168
>  5542056
> > Swap:              0           0           0
> > >
> >
> > I do not understand these things at all well, but it looks to me like I
> now
> > have 4gb of free space.
> >
> > So I think it is fair for me to report my real problem, again, almost
> exactly
> > as I did before.
> >
> > The 1 gb file for which I give a url below does not seem to work in
> Emacs.
> >
> > The file enumerates the primes below 10^9, so it would be very handy to
> > have around.
> >
> > I can find the file, literally, into an Emacs buffer.
> >
> > But then I cannot move to the bottom, i.e., using M->.  In fact, in the
> attempt to
> > move to the bottom, things go so badly that Emacs freezes, and I have to
> kill Emacs
> > by 'extraordinary measures'.
>
> How long did you wait?  Unless it's for an hour or so, this could be
> just some slow operation.  Does the system page during this (do you
> see the hard disk LED light more or less constantly)?  This could be
> one reason for the slowness.
>
> How many lines does this file have?  If its lines are very long, this
> could be a known slowness in the display engine.
>
> > Is this file just too big for Emacs to handle?
>
> No.
>
> >
> https://drive.google.com/file/d/1IaRNZ1rUQAZ72A7rJYpescmnJhpuGliA/view?usp=sharing
> >
> > Please, before you reply, fetch that file, find it literally, and see if
> > you can move to the bottom with M->.  I'd love to know whether you can
> do that.
>
> I cannot afford downloading such a huge file, sorry.  Maybe someone
> else can.
>
> > In GNU Emacs 28.2 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.37,
> cairo version 1.16.0)
> >  of 2023-05-13, modified by Debian built on x86-ubc-01
> > Windowing system distributor 'The X.Org Foundation', version
> 11.0.12014000
> > System Description: Debian GNU/Linux 12 (bookworm)
>
> Emacs 28 is no longer maintained.  But I don't think anything has
> changed since then in how we handle large files.  However, if the
> lines in the file are very long, you will be better off using Emacs 29
> where there are special features for speeding up the display of very
> long lines.
>

[-- Attachment #2: Type: text/html, Size: 4740 bytes --]

bug#69519: 1gb file too big for Emacs to handle?

[Basil L. Contovounesios]

I wonder if the View Large Files package would help at all in this case?
https://elpa.gnu.org/packages/vlf.html

-- 
Basil

bug#69519: 1gb file too big for Emacs to handle?

[Bruno Barbier]

Eli Zaretskii <eliz@gnu.org> writes:

>
> I cannot afford downloading such a huge file, sorry.  Maybe someone
> else can.

I downloaded the file (it requires to execute some Javascript from
Google Inc. to get to the real link).

The file extension is ".txt.lisp"; it contains 3 lines, and is about
950MB.  The whole content is on the second lines, the 2 other ones being
only a few char long.

Using GNU Emacs 29.2 (emacs -Q), opening it literally when asked, then
moving to the end (M->) is instantaneous for me.

Hoping this help,

Bruno

bug#69519: 1gb file too big for Emacs to handle?

[Eli Zaretskii]

> From: Bruno Barbier <brubar.cs@gmail.com>
> Cc: 69519@debbugs.gnu.org
> Date: Wed, 06 Mar 2024 09:48:19 +0100
> 
> The file extension is ".txt.lisp"; it contains 3 lines, and is about
> 950MB.  The whole content is on the second lines, the 2 other ones being
> only a few char long.

Thanks, this will certainly hit the issues with very long lines.

> Using GNU Emacs 29.2 (emacs -Q), opening it literally when asked, then
> moving to the end (M->) is instantaneous for me.

Thanks for testing, that's what I would expect with Emacs 29 and
later.

bug#69519: 1gb file too big for Emacs to handle?

[Robert Boyer]

[-- Attachment #1: Type: text/plain, Size: 877 bytes --]

> Using GNU Emacs 29.2 (emacs -Q), opening it literally when asked, then
> moving to the end (M->) is instantaneous for me.

Hurray for you heroes.  I look forward to 29.2

Thank you,

Bob


On Wed, Mar 6, 2024 at 2:48 AM Bruno Barbier <brubar.cs@gmail.com> wrote:

> Eli Zaretskii <eliz@gnu.org> writes:
>
> >
> > I cannot afford downloading such a huge file, sorry.  Maybe someone
> > else can.
>
> I downloaded the file (it requires to execute some Javascript from
> Google Inc. to get to the real link).
>
> The file extension is ".txt.lisp"; it contains 3 lines, and is about
> 950MB.  The whole content is on the second lines, the 2 other ones being
> only a few char long.
>
> Using GNU Emacs 29.2 (emacs -Q), opening it literally when asked, then
> moving to the end (M->) is instantaneous for me.
>
> Hoping this help,
>
> Bruno
>
>

[-- Attachment #2: Type: text/html, Size: 1371 bytes --]

bug#69519: 1gb file too big for Emacs to handle?

[Stefan Kangas]

Robert Boyer <robertstephenboyer@gmail.com> writes:

>> Using GNU Emacs 29.2 (emacs -Q), opening it literally when asked, then
>> moving to the end (M->) is instantaneous for me.
>
> Hurray for you heroes.  I look forward to 29.2

I guess this is fixed in Emacs 29.2, so I'm closing this bug now.

BTW, prefer Emacs 29.3 to 29.2 if at all possible.