"corrupted size vs. prev_size"

"corrupted size vs. prev_size"

[Lars Ingebrigtsen]

After the recent GIF/WebP caching changes, I sometimes get "corrupted
size vs. prev_size" and Emacs then hangs, so I guess I'm messing up
something in the memory allocation/freeing/writing bits somewhere.

But I'm getting it very, very rarely -- like one out of every thirty
times with a bunch of WebP/GIF images, so it's hard to track down.
Connecting to the hung Emacs isn't very informative:

(gdb) info threads
  Id   Target Id                                        Frame 
* 1    Thread 0x7f303a430000 (LWP 84136) "emacs"        0x00007f303d54e1f3 in __pselect (nfds=22, readfds=0x7ffd058d69e0, writefds=0x7ffd058d6a60, 
    exceptfds=0x0, timeout=0x7ffd058d6800, sigmask=<optimized out>)
    at ../sysdeps/unix/sysv/linux/pselect.c:52
  2    Thread 0x7f3039229640 (LWP 84137) "gmain"        0x00007f303d54b87f in __GI___poll (fds=0x561b63028870, nfds=2, timeout=-1)
    at ../sysdeps/unix/sysv/linux/poll.c:29
  3    Thread 0x7f303898c640 (LWP 84138) "gdbus"        0x00007f303d54b87f in __GI___poll (fds=0x561b63452d20, nfds=3, timeout=-1)
    at ../sysdeps/unix/sysv/linux/poll.c:29
  4    Thread 0x7f3033fff640 (LWP 84139) "dconf worker" 0x00007f303d54b87f in __GI___poll (fds=0x561b634be7d0, nfds=1, timeout=-1)
    at ../sysdeps/unix/sysv/linux/poll.c:29

I've stared at the code for some time now to see whether I can track the
problem down that way -- perhaps I've got some lifetime issues;
deallocating webp iterators twice, for instance, or something similar in
the gif code.  Does anybody else see something obviously wonky in that
code in gif_load or webp_load?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: "corrupted size vs. prev_size"

[Andreas Schwab]

On Apr 12 2022, Lars Ingebrigtsen wrote:

> After the recent GIF/WebP caching changes, I sometimes get "corrupted
> size vs. prev_size" and Emacs then hangs, so I guess I'm messing up
> something in the memory allocation/freeing/writing bits somewhere.

Have you tried valgrind yet?

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: "corrupted size vs. prev_size"

[Lars Ingebrigtsen]

Andreas Schwab <schwab@linux-m68k.org> writes:

> Have you tried valgrind yet?

Unfortunately, the animations don't seem to work at all when running
under valgrind.

Hm!  But after a few minutes, where Emacs is apparently not doing much
(at least the display isn't updated), it says:

==90535== Syscall param pselect6(sig) points to unaddressable byte(s)
==90535==    at 0x74981F3: pselect (pselect.c:52)
==90535==    by 0x378F4D: really_call_select (thread.c:612)
==90535==    by 0x37967F: flush_stack_call_func (lisp.h:4189)
==90535==    by 0x37967F: thread_select (thread.c:644)
==90535==    by 0x39C751: xg_select (xgselect.c:182)
==90535==    by 0x351C5B: wait_reading_process_output (process.c:5600)
==90535==    by 0x1654BF: sit_for (dispnew.c:6154)
==90535==    by 0x28111A: read_char (keyboard.c:2837)
==90535==    by 0x281898: read_key_sequence (keyboard.c:9932)
==90535==    by 0x283566: command_loop_1 (keyboard.c:1401)
==90535==    by 0x2FAE56: internal_condition_case (eval.c:1450)
==90535==    by 0x26F475: command_loop_2 (keyboard.c:1142)
==90535==    by 0x2FADB0: internal_catch (eval.c:1180)
==90535==  Address 0x10023b1a70 is 0 bytes inside a block of size 32 in arena "core"
==90535== 



-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: "corrupted size vs. prev_size"

[Po Lu]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> ==90535== Syscall param pselect6(sig) points to unaddressable byte(s)
> ==90535==    at 0x74981F3: pselect (pselect.c:52)
> ==90535==    by 0x378F4D: really_call_select (thread.c:612)
> ==90535==    by 0x37967F: flush_stack_call_func (lisp.h:4189)
> ==90535==    by 0x37967F: thread_select (thread.c:644)
> ==90535==    by 0x39C751: xg_select (xgselect.c:182)
> ==90535==    by 0x351C5B: wait_reading_process_output (process.c:5600)
> ==90535==    by 0x1654BF: sit_for (dispnew.c:6154)
> ==90535==    by 0x28111A: read_char (keyboard.c:2837)
> ==90535==    by 0x281898: read_key_sequence (keyboard.c:9932)
> ==90535==    by 0x283566: command_loop_1 (keyboard.c:1401)
> ==90535==    by 0x2FAE56: internal_condition_case (eval.c:1450)
> ==90535==    by 0x26F475: command_loop_2 (keyboard.c:1142)
> ==90535==    by 0x2FADB0: internal_catch (eval.c:1180)
> ==90535==  Address 0x10023b1a70 is 0 bytes inside a block of size 32 in arena "core"
> ==90535== 

This is a false positive, I guess something changed in pselect and
valgrind wasn't updated for it yet.

Re: "corrupted size vs. prev_size"

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> After the recent GIF/WebP caching changes, I sometimes get "corrupted
> size vs. prev_size" and Emacs then hangs, so I guess I'm messing up
> something in the memory allocation/freeing/writing bits somewhere.

After doing more testing, the problem is definitely related to the
gif_load code, not the webp_load code.

And it only seems to trigger when there's errors in the GIF files (i.e.,
DGifSlurp etc reports that one of the GIFs can't be parsed).  So the
problem is either in the error handling on my side, or it could even be
a bug in the giflib library.  As previously reported, the version of
giflib that's in Debian/bookworm has problems parsing GIFs that earlier
or later versions of giflib doesn't have...

Hm.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: "corrupted size vs. prev_size"

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Hm.

And I've now managed to catch a backtrace in gdb twice.  The error
happens here both times:

__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:49
#1  0x00007ffff52b7546 in __GI_abort () at abort.c:79
#2  0x00007ffff530eeb8 in __libc_message
    (action=action@entry=do_abort, fmt=fmt@entry=0x7ffff542ca78 "%s\n")
    at ../sysdeps/posix/libc_fatal.c:155
#3  0x00007ffff531691a in malloc_printerr
    (str=str@entry=0x7ffff542a714 "corrupted size vs. prev_size")
    at malloc.c:5628
#4  0x00007ffff5317816 in unlink_chunk
    (p=p@entry=0x55557db01da0, av=0x7ffff5463ba0 <main_arena>) at malloc.c:1608
#5  0x00007ffff531806b in _int_free
    (av=0x7ffff5463ba0 <main_arena>, p=0x55557da7efd0, have_lock=<optimized out>) at malloc.c:4575
#6  0x00007ffff531b9b4 in __GI___libc_free (mem=<optimized out>)
    at malloc.c:3309
#7  0x00007ffff7223a0f in  () at /lib/x86_64-linux-gnu/libcairo.so.2
#8  0x00007ffff728349f in cairo_surface_destroy ()
    at /lib/x86_64-linux-gnu/libcairo.so.2
#9  0x00007ffff725f340 in cairo_pattern_destroy ()
    at /lib/x86_64-linux-gnu/libcairo.so.2
#10 0x00005555557de342 in image_clear_image_1
    (flags=7, img=<optimized out>, f=0x555555fde088) at image.c:1684
#11 image_clear_image (img=0x55556eb958d0, f=0x555555fde088) at image.c:1696
#12 gif_clear_image (f=0x555555fde088, img=0x55556eb958d0) at image.c:8610
#13 0x00005555557dc80c in free_image
    (f=f@entry=0x555555fde088, img=0x55556eb958d0) at image.c:1366
#14 0x00005555557def4f in clear_image_cache
    (f=0x555555fde088, filter=filter@entry=0x0) at image.c:1943
#15 0x00005555557e7319 in clear_image_caches (filter=filter@entry=0x0)
    at image.c:1981
#16 0x00005555555f7387 in redisplay_internal () at xdisp.c:16825
#17 0x00005555555f7c45 in redisplay_preserve_echo_area
    (from_where=from_where@entry=8) at xdisp.c:16878
#18 0x00005555556ca78e in detect_input_pending_run_timers
    (do_display=do_display@entry=true) at keyboard.c:10758
#19 0x000055555579d98a in wait_reading_process_output
    (time_limit=time_limit@entry=30, nsecs=nsecs@entry=0, read_kbd=read_kbd@entry=-1, do_display=do_display@entry=true, wait_for_cell=wait_for_cell@entry=0x0, wait_proc=wait_proc@entry=0x0, just_wait_proc=0) at process.c:5695
#20 0x00005555555b14c0 in sit_for
    (timeout=timeout@entry=0x7a, reading=reading@entry=true, display_option=display_option@entry=1) at dispnew.c:6154
#21 0x00005555556cd11b in read_char
    (commandflag=1, map=0x55556e8c9b33, prev_event=0x0, used_mouse_menu=0x7fffffffdb8b, end_time=0x0) at /home/larsi/src/emacs/trunk/src/lisp.h:753
#22 0x00005555556cd899 in read_key_sequence

That may just be because I'm doing out-of-bounds writes somewhere, but
it's odd that it happened the same place both times, perhaps.  Hm.  Does
this suggest anything to anybody?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: "corrupted size vs. prev_size"

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> And I've now managed to catch a backtrace in gdb twice.  The error
> happens here both times:

But now I got a real segfault somewhere else:

#0  0x00005555557210e9 in sweep_conses () at alloc.c:7119
#1  0x0000555555727187 in gc_sweep () at alloc.c:7384
#2  garbage_collect () at alloc.c:6235
#3  0x0000555555727651 in maybe_garbage_collect () at alloc.c:6085
#4  0x0000555555748665 in maybe_gc ()
    at /home/larsi/src/emacs/trunk/src/lisp.h:5522
#5  Ffuncall (nargs=nargs@entry=2, args=args@entry=0x7fffffffd670)
    at eval.c:2868
#6  0x00005555556c3856 in call1 (arg1=0x555579d44a5d, fn=0xf0f0)
    at /home/larsi/src/emacs/trunk/src/lisp.h:3216
#7  timer_check_2 (idle_timers=<optimized out>, timers=<optimized out>)
    at keyboard.c:4591
#8  timer_check () at keyboard.c:4653
#9  0x00005555556c3b65 in readable_events (flags=flags@entry=1)
    at keyboard.c:3484
#10 0x00005555556c47a8 in get_input_pending (flags=flags@entry=1)
    at keyboard.c:7225
#11 0x00005555556cb584 in detect_input_pending_run_timers (do_display=false)
    at keyboard.c:10755
#12 read_char
    (commandflag=1, map=0x55557836f8c3, prev_event=0x0, used_mouse_menu=0x7fffffffdb8b, end_time=0x0) at keyboard.c:2581
#13 0x00005555556cd899 in read_key_sequence

So it's just random out-of-bounds writing.  (In gif_load somewhere,
apparently -- I'm able to trigger the problem with a gif-only test.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: "corrupted size vs. prev_size"

[Lars Ingebrigtsen]

Found it!

It's this bit:

          for (y = 0; y < subimg_height; ++y)
	    for (x = 0; x < subimg_width; ++x)
	      {
		int c = raster[y * subimg_width + x];
		if (transparency_color_index != c || disposal != DISPOSE_DO_NOT)
                  {
		    *(pixmap + x + subimg_left + (y + subimg_top) * width) =
		      pixel_colors[c];
                  }
	      }

On some images, this calculation writes outside the pixmap area.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: "corrupted size vs. prev_size"

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> On some images, this calculation writes outside the pixmap area.

And now fixed -- I had moved the check for whether the frames fit into
an image without adjusting the way it looped over the frames (in images
with invalid frames).

So that should be fixed now.

Eli reported a crash in the webp code, though, so there might be more
problems in this area.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: "corrupted size vs. prev_size"

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Eli reported a crash in the webp code, though, so there might be more
> problems in this area.

I think I spotted the problem in webp_load -- if the data came from a
string (which is the case in eww), then the iterator worked on may have
been garbage-collected.  This should be fixed now.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: "corrupted size vs. prev_size"

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Tue, 12 Apr 2022 12:59:38 +0200
> 
> After the recent GIF/WebP caching changes, I sometimes get "corrupted
> size vs. prev_size" and Emacs then hangs, so I guess I'm messing up
> something in the memory allocation/freeing/writing bits somewhere.
> 
> But I'm getting it very, very rarely -- like one out of every thirty
> times with a bunch of WebP/GIF images, so it's hard to track down.
> Connecting to the hung Emacs isn't very informative:
> 
> (gdb) info threads
>   Id   Target Id                                        Frame 
> * 1    Thread 0x7f303a430000 (LWP 84136) "emacs"        0x00007f303d54e1f3 in __pselect (nfds=22, readfds=0x7ffd058d69e0, writefds=0x7ffd058d6a60, 
>     exceptfds=0x0, timeout=0x7ffd058d6800, sigmask=<optimized out>)
>     at ../sysdeps/unix/sysv/linux/pselect.c:52
>   2    Thread 0x7f3039229640 (LWP 84137) "gmain"        0x00007f303d54b87f in __GI___poll (fds=0x561b63028870, nfds=2, timeout=-1)
>     at ../sysdeps/unix/sysv/linux/poll.c:29
>   3    Thread 0x7f303898c640 (LWP 84138) "gdbus"        0x00007f303d54b87f in __GI___poll (fds=0x561b63452d20, nfds=3, timeout=-1)
>     at ../sysdeps/unix/sysv/linux/poll.c:29
>   4    Thread 0x7f3033fff640 (LWP 84139) "dconf worker" 0x00007f303d54b87f in __GI___poll (fds=0x561b634be7d0, nfds=1, timeout=-1)
>     at ../sysdeps/unix/sysv/linux/poll.c:29
> 
> I've stared at the code for some time now to see whether I can track the
> problem down that way -- perhaps I've got some lifetime issues;
> deallocating webp iterators twice, for instance, or something similar in
> the gif code.  Does anybody else see something obviously wonky in that
> code in gif_load or webp_load?

I didn't see anything obviously wonky, but I can report that I had one
segfault in webp_load.  I couldn't reproduce it no matter how hard I
tried.  Here's the backtrace from that crash, in case it helps (I
doubt that):

  Thread 1 received signal SIGSEGV, Segmentation fault.
  0x61b4c91a in ?? () from D:\usr\bin\libwebp-7.dll
  (gdb) bt
  #0  0x61b4c91a in ?? () from D:\usr\bin\libwebp-7.dll
  #1  0x61b4ce35 in ?? () from D:\usr\bin\libwebp-7.dll
  #2  0x61b4d795 in ?? () from D:\usr\bin\libwebp-7.dll
  #3  0x6bdc18f5 in ?? () from D:\usr\bin\libwebpdemux-2.dll
  #4  0x013d9d13 in webp_load (f=0x7575c68, img=0x7adfd88) at image.c:9467
  #5  0x013cf399 in lookup_image (f=0x7575c68, spec=XIL(0xc000000006791ac0),
      face_id=0) at image.c:2666
  #6  0x0104d011 in handle_single_display_spec (it=0x82ac68,
      spec=XIL(0xc000000006791ac0), object=XIL(0xa000000007697960),
      overlay=XIL(0), position=0x82ad50, bufpos=1, display_replaced=0,
      frame_window_p=true, enable_eval_p=true) at xdisp.c:6012
  #7  0x01049c42 in handle_display_spec (it=0x82ac68,
      spec=XIL(0xc000000006791ac0), object=XIL(0xa000000007697960),
      overlay=XIL(0), position=0x82ad50, bufpos=1, frame_window_p=true)
      at xdisp.c:5475
  #8  0x010490fb in handle_display_prop (it=0x82ac68) at xdisp.c:5383
  #9  0x010449c2 in handle_stop (it=0x82ac68) at xdisp.c:3875
  #10 0x010515db in reseat (it=0x82ac68, pos=..., force_p=true) at xdisp.c:7336
  #11 0x01043c26 in init_iterator (it=0x82ac68, w=0x7575e80, charpos=1,
      bytepos=1, row=0x7635008, base_face_id=DEFAULT_FACE_ID) at xdisp.c:3476
  #12 0x01043cbe in start_display (it=0x82ac68, w=0x7575e80, pos=...)
      at xdisp.c:3492
  #13 0x0107adfc in try_window (window=XIL(0xa000000007575e80), pos=...,
      flags=1) at xdisp.c:20019
  #14 0x01077a6e in redisplay_window (window=XIL(0xa000000007575e80),
      just_this_one_p=false) at xdisp.c:19432
  #15 0x0106f1e3 in redisplay_window_0 (window=XIL(0xa000000007575e80))
      at xdisp.c:17077
  #16 0x0126c924 in internal_condition_case_1 (
      bfun=0x106f18b <redisplay_window_0>, arg=XIL(0xa000000007575e80),
      handlers=XIL(0xc0000000060b349c), hfun=0x106f148 <redisplay_window_error>)
      at eval.c:1474
  #17 0x0106f10a in redisplay_windows (window=XIL(0xa000000007575e80))
      at xdisp.c:17057
  #18 0x0106d96a in redisplay_internal () at xdisp.c:16525
  #19 0x0106e7fb in redisplay_preserve_echo_area (from_where=8) at xdisp.c:16878
  #20 0x0118d249 in detect_input_pending_run_timers (do_display=true)
      at keyboard.c:10758
  #21 0x012fa97a in wait_reading_process_output (time_limit=30, nsecs=0,
      read_kbd=-1, do_display=true, wait_for_cell=XIL(0), wait_proc=0x0,
      just_wait_proc=0) at process.c:5695
  #22 0x01012764 in sit_for (timeout=make_fixnum(30), reading=true,
      display_option=1) at dispnew.c:6154
  #23 0x01171b37 in read_char (commandflag=1, map=XIL(0xc000000007679cb0),
      prev_event=XIL(0), used_mouse_menu=0x82f49f, end_time=0x0)
      at keyboard.c:2837
  #24 0x0118a950 in read_key_sequence (keybuf=0x82f770, prompt=XIL(0),
      dont_downcase_last=false, can_return_switch_frame=true,
      fix_current_buffer=true, prevent_redisplay=false) at keyboard.c:9932
  #25 0x0116c667 in command_loop_1 () at keyboard.c:1401
  #26 0x0126c83a in internal_condition_case (bfun=0x116bf54 <command_loop_1>,
      handlers=XIL(0x90), hfun=0x116af52 <cmd_error>) at eval.c:1450
  #27 0x0116b9c1 in command_loop_2 (handlers=XIL(0x90)) at keyboard.c:1142
  #28 0x0126b6d9 in internal_catch (tag=XIL(0xfb70),
      func=0x116b98a <command_loop_2>, arg=XIL(0x90)) at eval.c:1180
  #29 0x0116b944 in command_loop () at keyboard.c:1120
  #30 0x0116a9b2 in recursive_edit_1 () at keyboard.c:729
  #31 0x0116ac50 in Frecursive_edit () at keyboard.c:812
  #32 0x01165c0a in main (argc=4, argv=0xa42990) at emacs.c:2447

  Lisp Backtrace:
  "redisplay_internal (C function)" (0x0)