bug#19322: 25.0.50; 1.6Mb one line file chocks Emacs

bug#19322: 25.0.50; 1.6Mb one line file chocks Emacs

[Vitalie Spinu]

Hi,

When I am trying to open this 1.6M file:

  http://vitalie.spinu.info/tmp/tmp.data

`emacs -q` locks and the only way to get out is a sigterm. I can open it
with older emacses:

  Emacs 24.3.1 (x86_64-pc-linux-gnu, GTK+ Version 3.10.9),  and
  Emacs 24.3.1 (x86_64-pc-linux-gnu, GTK+ Version 3.10.9)


Thanks,

   Vitalie


----

In GNU Emacs 25.0.50.1 (x86_64-pc-linux-gnu, GTK+ Version 3.12.2)
 of 2014-11-17 on lgw01-08
Windowing system distributor `The X.Org Foundation', version 11.0.11600000
System Description:	Ubuntu 14.10

Configured using:
 `configure --build=x86_64-linux-gnu --prefix=/usr
 '--includedir=${prefix}/include' '--mandir=${prefix}/share/man'
 '--infodir=${prefix}/share/info' --sysconfdir=/etc --localstatedir=/var
 '--libdir=${prefix}/lib/x86_64-linux-gnu'
 '--libexecdir=${prefix}/lib/x86_64-linux-gnu' --disable-maintainer-mode
 --disable-dependency-tracking --prefix=/usr --sharedstatedir=/var/lib
 --program-suffix=-snapshot --with-x=yes --with-x-toolkit=gtk3
 'CFLAGS=-g -O2 -fstack-protector-strong -Wformat
 -Werror=format-security' CPPFLAGS=-D_FORTIFY_SOURCE=2
 'LDFLAGS=-Wl,-Bsymbolic-functions -Wl,-z,relro''

Configured features:
XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GCONF GSETTINGS
NOTIFY LIBSELINUX GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB

Important settings:
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

bug#19322: 25.0.50; 1.6Mb one line file chocks Emacs

[Eli Zaretskii]

> From: Vitalie Spinu <spinuvit@gmail.com>
> Date: Mon, 08 Dec 2014 18:37:31 -0800
> 
> When I am trying to open this 1.6M file:
> 
>   http://vitalie.spinu.info/tmp/tmp.data
> 
> `emacs -q` locks and the only way to get out is a sigterm. I can open it
> with older emacses:
> 
>   Emacs 24.3.1 (x86_64-pc-linux-gnu, GTK+ Version 3.10.9),  and
>   Emacs 24.3.1 (x86_64-pc-linux-gnu, GTK+ Version 3.10.9)

Thanks.  I think I see the problem, and I'm testing a solution.

However, I'm surprised that you see this on a 64-bit host.  Could you
please run an unoptimized build under GDB, put a breakpoint in
memory_full, and see if that breakpoint breaks?  If it does, please
show the backtrace from there.  I'd like to make sure the problem I
see is the same one that affects you.

Also, how much virtual memory (RAM and swap) does your system have at
its disposal?

bug#19322: 25.0.50; 1.6Mb one line file chocks Emacs

[Vitalie Spinu]

Hi Eli,

I have 16GB of RAM and 10GB of Swap. When emacs is stuck it blink
periodically about every 2 seconds as if throwing an error and system
RAM usage goes up and down by about 6GB or so with the same periodicity.

With "--enable-checking 'CFLAGS=-O0 -g3' CPPFLAGS=-DGLYPH_DEBUG=1", the
backtrace is:
   
   (gdb) break memory_full
   Breakpoint 1 at 0x60dc1d: file alloc.c, line 3784.
   (gdb) run
   Starting program: /home/vitoshka/bin/emacs/src/emacs
   [Thread debugging using libthread_db enabled]
   Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
   [New Thread 0x7fffe8d65700 (LWP 6289)]
   
   ** (emacs:6285): WARNING **: Couldn't connect to accessibility bus: Failed to connect to socket /tmp/dbus-wgMsBj3J4m: Connection refused
   [New Thread 0x7fffe3dc3700 (LWP 6290)]
   [New Thread 0x7fffe3068700 (LWP 6293)]
   
   Breakpoint 1, memory_full (nbytes=14854559328) at alloc.c:3784
   3784      bool enough_free_memory = 0;
   (gdb)
   (gdb) backtrace
   #0  memory_full (nbytes=14854559328) at alloc.c:3784
   #1  0x00000000006099ce in xrealloc (block=0x7ffba0368010, size=14854559328) at alloc.c:776
   #2  0x0000000000609c61 in xpalloc (pa=0x7ffba0368010, nitems=0xd357a8, nitems_incr_min=200, nitems_max=1238368963057837, item_size=7448) at alloc.c:882
   #3  0x0000000000515e41 in bidi_cache_ensure_space (idx=1329624) at bidi.c:743
   #4  0x0000000000515ec6 in bidi_cache_iterator_state (bidi_it=0x7fffffff3cd8, resolved=false, update_only=false) at bidi.c:766
   #5  0x000000000051a37c in bidi_find_bracket_pairs (bidi_it=0x7fffffff3cd8) at bidi.c:2480
   #6  0x000000000051afbe in bidi_resolve_brackets (bidi_it=0x7fffffff3cd8) at bidi.c:2719
   #7  0x000000000051b38f in bidi_resolve_neutral (bidi_it=0x7fffffff3cd8) at bidi.c:2820
   #8  0x000000000051bb39 in bidi_type_of_next_char (bidi_it=0x7fffffff3cd8) at bidi.c:3024
   #9  0x000000000051bda9 in bidi_level_of_next_char (bidi_it=0x7fffffff3cd8) at bidi.c:3091
   #10 0x000000000051c3cc in bidi_move_to_visually_next (bidi_it=0x7fffffff3cd8) at bidi.c:3258
   #11 0x0000000000456644 in get_visually_first_element (it=0x7fffffff3340) at xdisp.c:7805
   #12 0x0000000000457aab in next_element_from_buffer (it=0x7fffffff3340) at xdisp.c:8248
   #13 0x0000000000453cf0 in get_next_display_element (it=0x7fffffff3340) at xdisp.c:6934
   #14 0x00000000004836a0 in display_line (it=0x7fffffff3340) at xdisp.c:20257
   #15 0x00000000004754a5 in try_window (window=20399237, pos=..., flags=1) at xdisp.c:17031
   #16 0x0000000000471a67 in redisplay_window (window=20399237, just_this_one_p=false) at xdisp.c:16504
   #17 0x000000000046887d in redisplay_window_0 (window=20399237) at xdisp.c:14329
   #18 0x00000000006345d3 in internal_condition_case_1 (bfun=0x46883b <redisplay_window_0>, arg=20399237, handlers=14393142, hfun=0x46880a <redisplay_window_error>) at eval.c:1369
   #19 0x00000000004687eb in redisplay_windows (window=20399237) at xdisp.c:14309
   #20 0x000000000046754f in redisplay_internal () at xdisp.c:13905
   #21 0x0000000000464fc9 in redisplay () at xdisp.c:13185
   #22 0x000000000058d84e in read_char (commandflag=1, map=61337350, prev_event=14421890, used_mouse_menu=0x7fffffffd4c9, end_time=0x0) at keyboard.c:2593
   #23 0x000000000059dd52 in read_key_sequence (keybuf=0x7fffffffd6b0, bufsize=30, prompt=14421890, dont_downcase_last=false, can_return_switch_frame=true, fix_current_buffer=true,
       prevent_redisplay=false) at keyboard.c:9206
   #24 0x000000000058a76f in command_loop_1 () at keyboard.c:1462
   #25 0x000000000063445f in internal_condition_case (bfun=0x58a35e <command_loop_1>, handlers=14473826, hfun=0x589a85 <cmd_error>) at eval.c:1345
   #26 0x0000000000589fde in command_loop_2 (ignore=14421890) at keyboard.c:1197
   #27 0x000000000063388a in internal_catch (tag=14469458, func=0x589fb8 <command_loop_2>, arg=14421890) at eval.c:1106
   #28 0x0000000000589f8c in command_loop () at keyboard.c:1176
   #29 0x000000000058958d in recursive_edit_1 () at keyboard.c:786
   #30 0x0000000000589775 in Frecursive_edit () at keyboard.c:857
   #31 0x000000000058738a in main (argc=1, argv=0x7fffffffdac8) at emacs.c:1623


Hope it's helpful, 

  Vitalie

bug#19322: 25.0.50; 1.6Mb one line file chocks Emacs

[Eli Zaretskii]

> From: Vitalie Spinu <spinuvit@gmail.com>
> Cc: 19322@debbugs.gnu.org
> Date: Tue, 09 Dec 2014 15:58:36 -0800
> 
> I have 16GB of RAM and 10GB of Swap. When emacs is stuck it blink
> periodically about every 2 seconds as if throwing an error and system
> RAM usage goes up and down by about 6GB or so with the same periodicity.
> 
> With "--enable-checking 'CFLAGS=-O0 -g3' CPPFLAGS=-DGLYPH_DEBUG=1", the
> backtrace is:
>    
>    (gdb) break memory_full
>    Breakpoint 1 at 0x60dc1d: file alloc.c, line 3784.
>    (gdb) run
>    Starting program: /home/vitoshka/bin/emacs/src/emacs
>    [Thread debugging using libthread_db enabled]
>    Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>    [New Thread 0x7fffe8d65700 (LWP 6289)]
>    
>    ** (emacs:6285): WARNING **: Couldn't connect to accessibility bus: Failed to connect to socket /tmp/dbus-wgMsBj3J4m: Connection refused
>    [New Thread 0x7fffe3dc3700 (LWP 6290)]
>    [New Thread 0x7fffe3068700 (LWP 6293)]
>    
>    Breakpoint 1, memory_full (nbytes=14854559328) at alloc.c:3784
>    3784      bool enough_free_memory = 0;
>    (gdb)
>    (gdb) backtrace
>    #0  memory_full (nbytes=14854559328) at alloc.c:3784
>    #1  0x00000000006099ce in xrealloc (block=0x7ffba0368010, size=14854559328) at alloc.c:776

This says Emacs asked for about 14GB of memory, which was rejected.
Any idea why, since you have 26GB of memory?

Also, when did you last synchronize with the repository?  Lately, I
pushed a change that should have reduced the memory Emacs asks in this
case by a factor of 3 on 64-bit hosts.

Anyway, this is indeed the same problem I saw on my system.

> Hope it's helpful, 

It is, thanks.

bug#19322: 25.0.50; 1.6Mb one line file chocks Emacs

[Vitalie Spinu]

 >>> Eli Zaretskii on Wed, 10 Dec 2014 05:48:40 +0200 wrote:

 > This says Emacs asked for about 14GB of memory, which was rejected.
 > Any idea why, since you have 26GB of memory?

Hm, how can that happen? It's just a 1.6Mb file.

 > Also, when did you last synchronize with the repository?

17th of November. I have just pulled it up, and the problem persists but
it is admittedly less severe. It takes "only" 4 GB of RAM and I can see
the content being displayed but the emacs locks anyways.

  Vitalie

bug#19322: 25.0.50; 1.6Mb one line file chocks Emacs

[Eli Zaretskii]

> From: Vitalie Spinu <spinuvit@gmail.com>
> Cc: 19322@debbugs.gnu.org
> Date: Tue, 09 Dec 2014 21:03:45 -0800
> 
>  >>> Eli Zaretskii on Wed, 10 Dec 2014 05:48:40 +0200 wrote:
> 
>  > This says Emacs asked for about 14GB of memory, which was rejected.
>  > Any idea why, since you have 26GB of memory?
> 
> Hm, how can that happen? It's just a 1.6Mb file.

Bidi magic.  Your entire humongous 1,600,000-character line is
enclosed in {..} braces, which triggers the paired brackets resolution
part of the new UBA defined by Unicode 6.3 and implemented on master.
It needs a lot of memory for such a long line.

I will soon push a fix for that.

>  > Also, when did you last synchronize with the repository?
> 
> 17th of November. I have just pulled it up, and the problem persists but
> it is admittedly less severe. It takes "only" 4 GB of RAM and I can see
> the content being displayed but the emacs locks anyways.

Does it lock, or does it just take it a VERY long time to redisplay?
Do you still see the calls to memory_full?

bug#19322: 25.0.50; 1.6Mb one line file chocks Emacs

[Eli Zaretskii]

> Date: Wed, 10 Dec 2014 18:09:03 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 19322@debbugs.gnu.org
> 
> I will soon push a fix for that.

Done in commit f3e16cb on master.

bug#19322: 25.0.50; 1.6Mb one line file chocks Emacs

[Vitalie Spinu]

 >>> Eli Zaretskii on Wed, 10 Dec 2014 19:45:13 +0200 wrote:

 > Done in commit f3e16cb on master.

Confirmed, it's fixed.

Thanks,

   Vitalie

bug#19322: 25.0.50; 1.6Mb one line file chocks Emacs

[Eli Zaretskii]

> From: Vitalie Spinu <spinuvit@gmail.com>
> Cc: 19322@debbugs.gnu.org
> Date: Wed, 10 Dec 2014 18:47:37 -0800
> 
>  >>> Eli Zaretskii on Wed, 10 Dec 2014 19:45:13 +0200 wrote:
> 
>  > Done in commit f3e16cb on master.
> 
> Confirmed, it's fixed.

OK, closing.