bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Michael Heerdegen]

Hello,

Traversing a `stream-of-directory-files' over a huge directory hierarchy
crashes my Emacs.

Here is a recipe: I have a file
"/home/micha/Treasure/today/stream-crash.el" with these contents:

#+begin_src emacs-lisp
;; -*- lexical-binding: t -*-

(require 'stream)

(seq-doseq (_file (stream-of-directory-files
                   "/home/micha" t nil t nil
                   (lambda (file) (and (file-readable-p file) (file-regular-p file)))))
  nil)
#+end_src

Then I

micha> emacs -Q -L /home/micha/software/elpa/packages/stream\
                -l /home/micha/Treasure/today/stream-crash.el

and I get a segfault after ~ 1 minute.

This kind of crash only seems to occur when the traversed directory is
sufficiently large.


Thanks in advance,

Michael.


In GNU Emacs 26.0.91 (build 2, x86_64-pc-linux-gnu, GTK+ Version 3.22.28)
 of 2018-02-26 built on drachen
Repository revision: ea8f9fd7be138708a52aad418d09d5d4ca6b2a7e
Windowing system distributor 'The X.Org Foundation', version 11.0.11906000
System Description:	Debian GNU/Linux testing (buster)

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Eli Zaretskii]

On February 27, 2018 11:22:26 AM GMT+02:00, Michael Heerdegen <michael_heerdegen@web.de> wrote:
> 
> Hello,
> 
> Traversing a `stream-of-directory-files' over a huge directory
> hierarchy
> crashes my Emacs.
> 
> Here is a recipe: I have a file
> "/home/micha/Treasure/today/stream-crash.el" with these contents:
> 
> #+begin_src emacs-lisp
> ;; -*- lexical-binding: t -*-
> 
> (require 'stream)
> 
> (seq-doseq (_file (stream-of-directory-files
>                    "/home/micha" t nil t nil
>   (lambda (file) (and (file-readable-p file) (file-regular-p file)))))
>   nil)
> #+end_src
> 
> Then I
> 
> micha> emacs -Q -L /home/micha/software/elpa/packages/stream\
>                 -l /home/micha/Treasure/today/stream-crash.el
> 
> and I get a segfault after ~ 1 minute.
> 
> This kind of crash only seems to occur when the traversed directory is
> sufficiently large.


I guess it's a stack overflow: that function recurses into subdirectories.

To avoid such problems, the function should be rewritten to work by BFS, not DFS.

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Michael Heerdegen]

Eli Zaretskii <eliz@gnu.org> writes:

> I guess it's a stack overflow: that function recurses into
> subdirectories.
>
> To avoid such problems, the function should be rewritten to work by
> BFS, not DFS.

Here is the backtrace I'm now able to produce.  Looks like the crash
happens in gc:

| [lots of more lines like these cut]
| #104019 0x00000000005e0526 in mark_object (arg=XIL(0x5a15413)) at alloc.c:6624
| #104020 0x00000000005dfab5 in mark_vectorlike (ptr=0x5a584e0) at alloc.c:6227
| #104021 0x00000000005e0526 in mark_object (arg=XIL(0x5bd7cf3)) at alloc.c:6624
| #104022 0x00000000005dfab5 in mark_vectorlike (ptr=0x5873a80) at alloc.c:6227
| #104023 0x00000000005e0526 in mark_object (arg=XIL(0x5bd7cb3)) at alloc.c:6624
| #104024 0x00000000006070ea in mark_specpdl (first=0x58a1348, ptr=0x58a1b90) at eval.c:3868
| #104025 0x00000000006856b2 in mark_one_thread (thread=0xcd5340 <main_thread>) at thread.c:614
| #104026 0x00000000006857c7 in mark_threads_callback (ignore=0x0) at thread.c:649
| #104027 0x00000000005dda03 in flush_stack_call_func (func=0x685781 <mark_threads_callback>, arg=0x0) at alloc.c:5220
| #104028 0x00000000006857f5 in mark_threads () at thread.c:656
| #104029 0x00000000005df2d2 in garbage_collect_1 (end=0x7fffffff63f0) at alloc.c:5997
| #104030 0x00000000005df929 in Fgarbage_collect () at alloc.c:6168
| #104031 0x000000000055746c in maybe_gc () at lisp.h:4749
| #104032 0x00000000006047cb in Ffuncall (nargs=4, args=0x7fffffff64d0) at eval.c:2751
| #104033 0x000000000064cc41 in exec_byte_code (bytestr=XIL(0x42f83c4), vector=XIL(0x3255bb5), maxdepth=make_number(13), args_template=make_number(128), nargs=2, args=0x7fffffff6b38) at bytecode.c:629
| #104034 0x000000000060529c in funcall_lambda (fun=XIL(0x3255c15), nargs=2, arg_vector=0x7fffffff6b38) at eval.c:2970
| #104035 0x00000000006048d4 in Ffuncall (nargs=3, args=0x7fffffff6b30) at eval.c:2771
| #104036 0x000000000060395b in Fapply (nargs=3, args=0x7fffffff6b30) at eval.c:2346
| #104037 0x0000000000604b87 in funcall_subr (subr=0xc3d7c0 <Sapply>, numargs=3, args=0x7fffffff6b30) at eval.c:2824
| #104038 0x0000000000604890 in Ffuncall (nargs=4, args=0x7fffffff6b28) at eval.c:2769
| #104039 0x000000000064cc41 in exec_byte_code (bytestr=XIL(0x42f83e4), vector=XIL(0x15372775), maxdepth=make_number(8), args_template=make_number(256), nargs=0, args=0x7fffffff6ff8) at bytecode.c:629
| #104040 0x000000000060529c in funcall_lambda (fun=XIL(0x153727c5), nargs=0, arg_vector=0x7fffffff6ff8) at eval.c:2970
| #104041 0x00000000006048d4 in Ffuncall (nargs=1, args=0x7fffffff6ff0) at eval.c:2771
| #104042 0x000000000064cc41 in exec_byte_code (bytestr=XIL(0x42b7ad4), vector=XIL(0x930825), maxdepth=make_number(2), args_template=make_number(257), nargs=1, args=0x7fffffff7460) at bytecode.c:629
| #104043 0x000000000060529c in funcall_lambda (fun=XIL(0x3b73d75), nargs=1, arg_vector=0x7fffffff7458) at eval.c:2970
| #104044 0x00000000006048d4 in Ffuncall (nargs=2, args=0x7fffffff7450) at eval.c:2771
| #104045 0x000000000064cc41 in exec_byte_code (bytestr=XIL(0x42f82f4), vector=XIL(0x3255aa5), maxdepth=make_number(3), args_template=make_number(257), nargs=1, args=0x7fffffff78e0) at bytecode.c:629
| #104046 0x000000000060529c in funcall_lambda (fun=XIL(0x3255ab5), nargs=1, arg_vector=0x7fffffff78d8) at eval.c:2970
| #104047 0x00000000006048d4 in Ffuncall (nargs=2, args=0x7fffffff78d0) at eval.c:2771
| #104048 0x000000000064cc41 in exec_byte_code (bytestr=XIL(0x42f83e4), vector=XIL(0x153727f5), maxdepth=make_number(8), args_template=make_number(256), nargs=0, args=0x7fffffff7da8) at bytecode.c:629
| #104049 0x000000000060529c in funcall_lambda (fun=XIL(0x15372845), nargs=0, arg_vector=0x7fffffff7da8) at eval.c:2970
| #104050 0x00000000006048d4 in Ffuncall (nargs=1, args=0x7fffffff7da0) at eval.c:2771
| #104051 0x000000000064cc41 in exec_byte_code (bytestr=XIL(0x42b7ad4), vector=XIL(0x930825), maxdepth=make_number(2), args_template=make_number(257), nargs=1, args=0x7fffffff8210) at bytecode.c:629
| #104052 0x000000000060529c in funcall_lambda (fun=XIL(0x3b73d75), nargs=1, arg_vector=0x7fffffff8208) at eval.c:2970
| #104053 0x00000000006048d4 in Ffuncall (nargs=2, args=0x7fffffff8200) at eval.c:2771
| #104054 0x000000000064cc41 in exec_byte_code (bytestr=XIL(0x42f82f4), vector=XIL(0x3255aa5), maxdepth=make_number(3), args_template=make_number(257), nargs=1, args=0x7fffffff8690) at bytecode.c:629
| #104055 0x000000000060529c in funcall_lambda (fun=XIL(0x3255ab5), nargs=1, arg_vector=0x7fffffff8688) at eval.c:2970
| #104056 0x00000000006048d4 in Ffuncall (nargs=2, args=0x7fffffff8680) at eval.c:2771
| #104057 0x000000000064cc41 in exec_byte_code (bytestr=XIL(0x42f83e4), vector=XIL(0x15372875), maxdepth=make_number(8), args_template=make_number(256), nargs=0, args=0x7fffffff8b58) at bytecode.c:629
| #104058 0x000000000060529c in funcall_lambda (fun=XIL(0x153728c5), nargs=0, arg_vector=0x7fffffff8b58) at eval.c:2970
| #104059 0x00000000006048d4 in Ffuncall (nargs=1, args=0x7fffffff8b50) at eval.c:2771
| #104060 0x000000000064cc41 in exec_byte_code (bytestr=XIL(0x42b7ad4), vector=XIL(0x930825), maxdepth=make_number(2), args_template=make_number(257), nargs=1, args=0x7fffffff8fc0) at bytecode.c:629
| #104061 0x000000000060529c in funcall_lambda (fun=XIL(0x3b73d75), nargs=1, arg_vector=0x7fffffff8fb8) at eval.c:2970
| #104062 0x00000000006048d4 in Ffuncall (nargs=2, args=0x7fffffff8fb0) at eval.c:2771
| #104063 0x000000000064cc41 in exec_byte_code (bytestr=XIL(0x42f82f4), vector=XIL(0x3255aa5), maxdepth=make_number(3), args_template=make_number(257), nargs=1, args=0x7fffffff9440) at bytecode.c:629
| #104064 0x000000000060529c in funcall_lambda (fun=XIL(0x3255ab5), nargs=1, arg_vector=0x7fffffff9438) at eval.c:2970
| #104065 0x00000000006048d4 in Ffuncall (nargs=2, args=0x7fffffff9430) at eval.c:2771
| #104066 0x000000000064cc41 in exec_byte_code (bytestr=XIL(0x42f83e4), vector=XIL(0x153728f5), maxdepth=make_number(8), args_template=make_number(256), nargs=0, args=0x7fffffff9908) at bytecode.c:629
| #104067 0x000000000060529c in funcall_lambda (fun=XIL(0x15372945), nargs=0, arg_vector=0x7fffffff9908) at eval.c:2970
| #104068 0x00000000006048d4 in Ffuncall (nargs=1, args=0x7fffffff9900) at eval.c:2771
| #104069 0x000000000064cc41 in exec_byte_code (bytestr=XIL(0x42b7ad4), vector=XIL(0x930825), maxdepth=make_number(2), args_template=make_number(257), nargs=1, args=0x7fffffff9d70) at bytecode.c:629
| #104070 0x000000000060529c in funcall_lambda (fun=XIL(0x3b73d75), nargs=1, arg_vector=0x7fffffff9d68) at eval.c:2970
| #104071 0x00000000006048d4 in Ffuncall (nargs=2, args=0x7fffffff9d60) at eval.c:2771
| #104072 0x000000000064cc41 in exec_byte_code (bytestr=XIL(0x42f82f4), vector=XIL(0x3255aa5), maxdepth=make_number(3), args_template=make_number(257), nargs=1, args=0x7fffffffa1e8) at bytecode.c:629
| #104073 0x000000000060529c in funcall_lambda (fun=XIL(0x3255ab5), nargs=1, arg_vector=0x7fffffffa1e0) at eval.c:2970
| #104074 0x00000000006048d4 in Ffuncall (nargs=2, args=0x7fffffffa1d8) at eval.c:2771
| #104075 0x000000000064cc41 in exec_byte_code (bytestr=XIL(0x42f8c64), vector=XIL(0x153729a5), maxdepth=make_number(7), args_template=make_number(256), nargs=0, args=0x7fffffffa6a8) at bytecode.c:629
| #104076 0x000000000060529c in funcall_lambda (fun=XIL(0x153729f5), nargs=0, arg_vector=0x7fffffffa6a8) at eval.c:2970
| #104077 0x00000000006048d4 in Ffuncall (nargs=1, args=0x7fffffffa6a0) at eval.c:2771
| #104078 0x000000000064cc41 in exec_byte_code (bytestr=XIL(0x42b7ad4), vector=XIL(0x930825), maxdepth=make_number(2), args_template=make_number(257), nargs=1, args=0x7fffffffab10) at bytecode.c:629
| #104079 0x000000000060529c in funcall_lambda (fun=XIL(0x3b73d75), nargs=1, arg_vector=0x7fffffffab08) at eval.c:2970
| #104080 0x00000000006048d4 in Ffuncall (nargs=2, args=0x7fffffffab00) at eval.c:2771
| #104081 0x000000000064cc41 in exec_byte_code (bytestr=XIL(0x42f82f4), vector=XIL(0x3255aa5), maxdepth=make_number(3), args_template=make_number(257), nargs=1, args=0x7fffffffaf88) at bytecode.c:629
| #104082 0x000000000060529c in funcall_lambda (fun=XIL(0x3255ab5), nargs=1, arg_vector=0x7fffffffaf80) at eval.c:2970
| #104083 0x00000000006048d4 in Ffuncall (nargs=2, args=0x7fffffffaf78) at eval.c:2771
| #104084 0x000000000064cc41 in exec_byte_code (bytestr=XIL(0x42f8c04), vector=XIL(0x331ede5), maxdepth=make_number(5), args_template=make_number(514), nargs=2, args=0x7fffffffb5d0) at bytecode.c:629
| #104085 0x000000000060529c in funcall_lambda (fun=XIL(0x331ee05), nargs=2, arg_vector=0x7fffffffb5c0) at eval.c:2970
| #104086 0x00000000006048d4 in Ffuncall (nargs=3, args=0x7fffffffb5b8) at eval.c:2771
| #104087 0x000000000060390d in Fapply (nargs=4, args=0x7fffffffb5b8) at eval.c:2342
| #104088 0x0000000000604b87 in funcall_subr (subr=0xc3d7c0 <Sapply>, numargs=4, args=0x7fffffffb5b8) at eval.c:2824
| #104089 0x0000000000604890 in Ffuncall (nargs=5, args=0x7fffffffb5b0) at eval.c:2769
| #104090 0x000000000064cc41 in exec_byte_code (bytestr=XIL(0x42fb574), vector=XIL(0x425f285), maxdepth=make_number(15), args_template=make_number(642), nargs=2, args=0x7fffffffba40) at bytecode.c:629
| #104091 0x000000000060529c in funcall_lambda (fun=XIL(0x425f305), nargs=2, arg_vector=0x7fffffffba30) at eval.c:2970
| #104092 0x000000000060500d in apply_lambda (fun=XIL(0x425f305), args=XIL(0x5bd89c3), count=30) at eval.c:2906
| #104093 0x0000000000603602 in eval_sub (form=XIL(0x5bd89d3)) at eval.c:2279
| #104094 0x0000000000632c69 in readevalloop_eager_expand_eval (val=XIL(0x5bd8a73), macroexpand=XIL(0xda5d0)) at lread.c:1850
| #104095 0x00000000006335f0 in readevalloop (readcharfun=XIL(0x5b75ce5), infile0=0x0, sourcename=XIL(0x55c2e14), printflag=false, unibyte=XIL(0), readfun=XIL(0), start=XIL(0), end=XIL(0)) at lread.c:2036
| #104096 0x0000000000633a0a in Feval_buffer (buffer=XIL(0), printflag=XIL(0), filename=XIL(0x56ea3c4), unibyte=XIL(0), do_allow_print=XIL(0)) at lread.c:2103
| #104097 0x0000000000604d25 in funcall_subr (subr=0xc40240 <Seval_buffer>, numargs=0, args=0x7fffffffbfa0) at eval.c:2856
| #104098 0x0000000000604890 in Ffuncall (nargs=1, args=0x7fffffffbf98) at eval.c:2769
| #104099 0x00000000005fc8f7 in Ffuncall_interactively (nargs=1, args=0x7fffffffbf98) at callint.c:252
| #104100 0x0000000000604b87 in funcall_subr (subr=0xc3cfc0 <Sfuncall_interactively>, numargs=1, args=0x7fffffffbf98) at eval.c:2824
| #104101 0x0000000000604890 in Ffuncall (nargs=2, args=0x7fffffffbf90) at eval.c:2769
| #104102 0x00000000005fec50 in Fcall_interactively (function=XIL(0xad320), record_flag=XIL(0xaad0), keys=XIL(0x5a55745)) at callint.c:854
| #104103 0x0000000000604cac in funcall_subr (subr=0xc3d000 <Scall_interactively>, numargs=3, args=0x7fffffffc300) at eval.c:2849
| #104104 0x0000000000604890 in Ffuncall (nargs=4, args=0x7fffffffc2f8) at eval.c:2769
| #104105 0x000000000064cc41 in exec_byte_code (bytestr=XIL(0x9f4bec), vector=XIL(0x9f4c0d), maxdepth=make_number(13), args_template=make_number(1025), nargs=2, args=0x7fffffffc868) at bytecode.c:629
| #104106 0x000000000060529c in funcall_lambda (fun=XIL(0x9f4bbd), nargs=2, arg_vector=0x7fffffffc858) at eval.c:2970
| #104107 0x00000000006048d4 in Ffuncall (nargs=3, args=0x7fffffffc850) at eval.c:2771
| #104108 0x000000000064cc41 in exec_byte_code (bytestr=XIL(0x9f489c), vector=XIL(0x9f48bd), maxdepth=make_number(15), args_template=make_number(769), nargs=3, args=0x7fffffffce00) at bytecode.c:629
| #104109 0x000000000060529c in funcall_lambda (fun=XIL(0x9f485d), nargs=3, arg_vector=0x7fffffffcde8) at eval.c:2970
| #104110 0x00000000006048d4 in Ffuncall (nargs=4, args=0x7fffffffcde0) at eval.c:2771
| #104111 0x0000000000603cdb in Fapply (nargs=2, args=0x7fffffffcfc0) at eval.c:2389
| #104112 0x0000000000604b87 in funcall_subr (subr=0xc3d7c0 <Sapply>, numargs=2, args=0x7fffffffcfc0) at eval.c:2824
| #104113 0x0000000000604890 in Ffuncall (nargs=3, args=0x7fffffffcfb8) at eval.c:2769
| #104114 0x000000000064cc41 in exec_byte_code (bytestr=XIL(0x505da94), vector=XIL(0x5044095), maxdepth=make_number(3), args_template=XIL(0), nargs=0, args=0x0) at bytecode.c:629
| #104115 0x0000000000605615 in funcall_lambda (fun=XIL(0x50c3185), nargs=5, arg_vector=0x5044095) at eval.c:3052
| #104116 0x00000000006048d4 in Ffuncall (nargs=6, args=0x7fffffffd450) at eval.c:2771
| #104117 0x0000000000603cdb in Fapply (nargs=3, args=0x7fffffffd648) at eval.c:2389
| #104118 0x0000000000604b87 in funcall_subr (subr=0xc3d7c0 <Sapply>, numargs=3, args=0x7fffffffd648) at eval.c:2824
| #104119 0x0000000000604890 in Ffuncall (nargs=4, args=0x7fffffffd640) at eval.c:2769
| #104120 0x000000000064cc41 in exec_byte_code (bytestr=XIL(0x14402c4), vector=XIL(0x50441c5), maxdepth=make_number(5), args_template=make_number(128), nargs=4, args=0x7fffffffdbf0) at bytecode.c:629
| #104121 0x000000000060529c in funcall_lambda (fun=XIL(0x50441f5), nargs=4, arg_vector=0x7fffffffdbf0) at eval.c:2970
| #104122 0x00000000006048d4 in Ffuncall (nargs=5, args=0x7fffffffdbe8) at eval.c:2771
| #104123 0x00000000005fc8f7 in Ffuncall_interactively (nargs=5, args=0x7fffffffdbe8) at callint.c:252
| #104124 0x0000000000604b87 in funcall_subr (subr=0xc3cfc0 <Sfuncall_interactively>, numargs=5, args=0x7fffffffdbe8) at eval.c:2824
| #104125 0x0000000000604890 in Ffuncall (nargs=6, args=0x7fffffffdbe0) at eval.c:2769
| #104126 0x0000000000603cdb in Fapply (nargs=3, args=0x7fffffffdd90) at eval.c:2389
| #104127 0x00000000005fcd7d in Fcall_interactively (function=XIL(0xda3c0), record_flag=XIL(0), keys=XIL(0x5a3e0a5)) at callint.c:389
| #104128 0x0000000000604cac in funcall_subr (subr=0xc3d000 <Scall_interactively>, numargs=3, args=0x7fffffffdfe0) at eval.c:2849
| #104129 0x0000000000604890 in Ffuncall (nargs=4, args=0x7fffffffdfd8) at eval.c:2769
| #104130 0x000000000064cc41 in exec_byte_code (bytestr=XIL(0x9f4bec), vector=XIL(0x9f4c0d), maxdepth=make_number(13), args_template=make_number(1025), nargs=1, args=0x7fffffffe520) at bytecode.c:629
| #104131 0x000000000060529c in funcall_lambda (fun=XIL(0x9f4bbd), nargs=1, arg_vector=0x7fffffffe518) at eval.c:2970
| #104132 0x00000000006048d4 in Ffuncall (nargs=2, args=0x7fffffffe510) at eval.c:2771
| #104133 0x00000000006042a0 in call1 (fn=XIL(0x3f00), arg1=XIL(0xda3c0)) at eval.c:2620
| #104134 0x000000000055ec02 in command_loop_1 () at keyboard.c:1482
| #104135 0x00000000006013b2 in internal_condition_case (bfun=0x55e45e <command_loop_1>, handlers=XIL(0x5250), hfun=0x55dc1d <cmd_error>) at eval.c:1332
| #104136 0x000000000055e151 in command_loop_2 (ignore=XIL(0)) at keyboard.c:1110
| #104137 0x0000000000600c94 in internal_catch (tag=XIL(0xc6f0), func=0x55e124 <command_loop_2>, arg=XIL(0)) at eval.c:1097
| #104138 0x000000000055e0ef in command_loop () at keyboard.c:1089
| #104139 0x000000000055d7ef in recursive_edit_1 () at keyboard.c:695
| #104140 0x000000000055d970 in Frecursive_edit () at keyboard.c:766
| #104141 0x000000000055b5f0 in main (argc=1, argv=0x7fffffffe9f8) at emacs.c:1713
| Cannot access memory at address 0x7fffff66ff4f



Michael.

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Michael Heerdegen]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> Here is the backtrace I'm now able to produce.  Looks like the crash
> happens in gc.

Here is a much simpler example that crashes as well:

#+begin_src emacs-lisp
(seq-doseq (_ (stream-range 1 1000000)) nil)
#+end_src

Note that this is executed as a loop due how to streams are implemented,
although the definition of `seq-doseq' looks recursive.  But it seems
that gc has a problem with the large number of conses created when
processing that.


Michael.

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Cc: bug-gnu-emacs@gnu.org,  30626@debbugs.gnu.org
> Date: Tue, 27 Feb 2018 13:08:59 +0100
> 
> #+begin_src emacs-lisp
> (seq-doseq (_ (stream-range 1 1000000)) nil)
> #+end_src
> 
> Note that this is executed as a loop due how to streams are implemented,
> although the definition of `seq-doseq' looks recursive.  But it seems
> that gc has a problem with the large number of conses created when
> processing that.

What can we do instead in such cases?  Stack-overflow protection
cannot work in GC, so you are shooting yourself in the foot by
creating such large recursive structures.  By the time we get to GC,
where the problem will happen, it's too late, because the memory was
already allocated.

Does anyone has a reasonable idea for avoiding the crash in such
programs?

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Noam Postavsky]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Michael Heerdegen <michael_heerdegen@web.de>
>> Cc: bug-gnu-emacs@gnu.org,  30626@debbugs.gnu.org
>> Date: Tue, 27 Feb 2018 13:08:59 +0100
>> 
>> #+begin_src emacs-lisp
>> (seq-doseq (_ (stream-range 1 1000000)) nil)
>> #+end_src
>> 
>> Note that this is executed as a loop due how to streams are implemented,
>> although the definition of `seq-doseq' looks recursive.

Doesn't look recursive to me, it expands to a call to seq-do, which uses
a simple loop.

>> But it seems that gc has a problem with the large number of conses
>> created when processing that.
>
> What can we do instead in such cases?  Stack-overflow protection
> cannot work in GC, so you are shooting yourself in the foot by
> creating such large recursive structures.  By the time we get to GC,
> where the problem will happen, it's too late, because the memory was
> already allocated.
>
> Does anyone has a reasonable idea for avoiding the crash in such
> programs?

I don't have a quick answer for the general case, but I think it's a bug
in stream.el that it's creating such large structures in the first
place.  As far as I understand it, the point of streams is to handle
long lists by encoding them as

    (FIRST-VALUE . FUNCTION-TO-PRODUCE-REST-OF-LIST)
 
so as to avoid allocating large amounts of memory.  Is there an easy way
to find out what the large structures are, and where they are coming
from?

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Michael Heerdegen]

Noam Postavsky <npostavs@gmail.com> writes:

> >> #+begin_src emacs-lisp
> >> (seq-doseq (_ (stream-range 1 1000000)) nil)
> >> #+end_src
> >> 
> >> Note that this is executed as a loop due how to streams are
> >> implemented, although the definition of `seq-doseq' looks
> >> recursive.
>
> Doesn't look recursive to me, it expands to a call to seq-do, which uses
> a simple loop.

I was imprecise, I meant that the streams are defined recursively (most
of the time).  Though, it's a delayed type of recursion.  Anyway, I
think that this doesn't matter here.

> > Does anyone has a reasonable idea for avoiding the crash in such
> > programs?
>
> I don't have a quick answer for the general case, but I think it's a bug
> in stream.el that it's creating such large structures in the first
> place.  As far as I understand it, the point of streams is to handle
> long lists by encoding them as
>
>     (FIRST-VALUE . FUNCTION-TO-PRODUCE-REST-OF-LIST)

Yes, that's exactly how it's implemented.  When requesting more elements
from the stream, that becomes

      (FIRST-VALUE .
        (SECOND-VALUE . FUNCTION-TO-PRODUCE-MORE-REST-OF-LIST))

When we loop over the string, the cons whose car is the FIRST-VALUE,
let's call it cons1, is immediately thrown away, and we continue with

      (SECOND-VALUE . FUNCTION-TO-PRODUCE-MORE-REST-OF-LIST)

etc.

AFAIU the problem is that the cons1 still exists in memory until garbage
collection kicks in.  When that happens, the cons1 points to a largely
recursive cons structure, though this structure is never referenced from
Lisp in this form.
  
> so as to avoid allocating large amounts of memory.  Is there an easy way
> to find out what the large structures are, and where they are coming
> from?

I think I've answered that.  At least, I think so.  What I don't
understand is that when I force the `seq-doseq' to call
`garbace-collect' explicitly every 1000 cycles, or so, it doesn't help:
the crash still happens after generating ~ 70 000 elements, or some
more, but I can't avoid the crash, no matter how often I call gc.  So
I'm not sure whether these long lists are the problem or something else.
The FUNCTION-TO-PRODUCE-MORE-REST-OF-LIST looks harmless when I print
it, even after thousands of iterations, so I would not understand why
that could be problematic.  streams.el implements things in a way that
these rest functions are not deeply nested lambdas.


Michael.

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Cc: Eli Zaretskii <eliz@gnu.org>,  30626@debbugs.gnu.org
> Date: Wed, 28 Feb 2018 11:58:49 +0100
> 
> >     (FIRST-VALUE . FUNCTION-TO-PRODUCE-REST-OF-LIST)
> 
> Yes, that's exactly how it's implemented.  When requesting more elements
> from the stream, that becomes
> 
>       (FIRST-VALUE .
>         (SECOND-VALUE . FUNCTION-TO-PRODUCE-MORE-REST-OF-LIST))
> 
> When we loop over the string, the cons whose car is the FIRST-VALUE,
> let's call it cons1, is immediately thrown away, and we continue with
> 
>       (SECOND-VALUE . FUNCTION-TO-PRODUCE-MORE-REST-OF-LIST)
> 
> etc.

How do you see that the car is immediately thrown away?

> AFAIU the problem is that the cons1 still exists in memory until garbage
> collection kicks in.  When that happens, the cons1 points to a largely
> recursive cons structure, though this structure is never referenced from
> Lisp in this form.

What is "cons1" in this context?

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Michael Heerdegen]

Eli Zaretskii <eliz@gnu.org> writes:

> How do you see that the car is immediately thrown away?

After expansion and dispatch, this is what gets executed:

#+begin_src emacs-lisp
(let ((stream (stream-range 1 1000000)))
  (while (not (stream-empty-p stream))
    (funcall #'ignore (stream-first stream))
    (setq stream (stream-rest stream))))
#+end_src

Creating an element and throwing away the outermost cons alternate.

> > AFAIU the problem is that the cons1 still exists in memory until
> > garbage collection kicks in.  When that happens, the cons1 points to
> > a largely recursive cons structure, though this structure is never
> > referenced from Lisp in this form.

> What is "cons1" in this context?

I said it some lines above: I defined it as name of the "first cons",
i.e. the original stream.  The return value of `stream-range' in the
above example.


Michael.

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Cc: npostavs@gmail.com,  30626@debbugs.gnu.org
> Date: Wed, 28 Feb 2018 17:20:53 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > How do you see that the car is immediately thrown away?
> 
> After expansion and dispatch, this is what gets executed:
> 
> #+begin_src emacs-lisp
> (let ((stream (stream-range 1 1000000)))
>   (while (not (stream-empty-p stream))
>     (funcall #'ignore (stream-first stream))
>     (setq stream (stream-rest stream))))
> #+end_src
> 
> Creating an element and throwing away the outermost cons alternate.

I don't think it's thrown away from the POV of GC.  But you can easily
see what is going on if you trace the GC on the C level.  You should
be able to see which object causes recursion in mark_object.  I didn't
look long enough, but what I did see looks very much like the entire
unwound stream.

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Michael Heerdegen]

Eli Zaretskii <eliz@gnu.org> writes:

> I don't think it's thrown away from the POV of GC.  But you can easily
> see what is going on if you trace the GC on the C level.  You should
> be able to see which object causes recursion in mark_object.  I didn't
> look long enough, but what I did see looks very much like the entire
> unwound stream.

I have an idea what could be going on.  In the stream-range example,
this is how the stream is build:

#+begin_src emacs-lisp
(list stream--identifier
      (let
          (forced val)
        (lambda
          (&optional check)
          (if
              check
              forced
            (unless
                forced
              (setf
               val
               (progn
                 (cons
                  start
                  (stream-range (+ start step) end step))))
              (setf forced t))
            val))))
#+end_src

The inner `stream-range' call results in a closure, and I guess that
this closure includes a reference to the outside VAL, which is the
stream from one step back (though there isn't a lexical reference to the
variable...does that make sense?)

So there could be a chain of references via closure variables back to
the first cons.


Michael.

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Michael Heerdegen]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> The inner `stream-range' call results in a closure, and I guess that
> this closure includes a reference to the outside VAL, which is the
> stream from one step back (though there isn't a lexical reference to the
> variable...does that make sense?)

This is probably only a part of the puzzle.  Some examples:

test1.el: This is more or less the `stream-range' example reimplemented
without dependencies so that it works in emacs -Q:

#+begin_src emacs-lisp
;; -*- lexical-binding: t -*-

(defun my-range (start)
  (let (forced val)
    (lambda ()
      (unless forced
        (setq val (cons start (my-range (1+ start)))
              forced t))
      val)))

(defun my-test ()
  (let ((range-object (my-range 1))
        current-cons)
    (while (< (car (setq current-cons (funcall range-object))) (* 1000 1000))
      (message "%d" (car current-cons))
      (setq range-object (cdr current-cons)))))

(my-test)
#+end_src

Loading this file test1.el crashes Emacs - but if you compile it,
loading the compiled file doesn't crash.  This is what I expected from
my previous thoughts, because only the uncompiled closures include a
reference to the outer VALs.


test2.el:

#+begin_src emacs-lisp
;; -*- lexical-binding: t -*-

(require 'stream)

(let ((stream (stream-range 1 1000000)))
  (stream-flush stream))
#+end_src

This traverses the whole stream ignoring the elements.  Loading this
file crashes Emacs, no matter if compiled or not.  I'm surprised it
doesn't work even when compiled.


test3.el:

#+begin_src emacs-lisp
;; -*- lexical-binding: t -*-

(require 'stream)

(let ((stream (stream-range 1 1000000)))
  (while (not (stream-empty-p stream))
    (cl-callf stream-rest stream)))
#+end_src

This is semantically exactly like test2.el, only the call to
`stream-flush' has been replaced by literally writing out the
definition.  Nonetheless, the compiled file suddenly doesn't crash Emacs
when loaded.  Loading the uncompiled file test3.el still crashes.

Seems that whether we get a crash or not depends on details in the
implementation of lexical-binding.

Michael.

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Cc: npostavs@gmail.com,  30626@debbugs.gnu.org
> Date: Thu, 01 Mar 2018 12:25:43 +0100
> 
> Seems that whether we get a crash or not depends on details in the
> implementation of lexical-binding.

Byte compilation doesn't just produce byte code, it also changes the
code into an equivalent one, but "equivalence" in this context doesn't
include side effects like stack usage.  As an extreme example,
consider a tail-recursive program that the byte compiler converts (or
at least might convert theoretically) into a loop.

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Noam Postavsky]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> #+begin_src emacs-lisp
> ;; -*- lexical-binding: t -*-
>
> (require 'stream)
>
> (let ((stream (stream-range 1 1000000)))
>   (while (not (stream-empty-p stream))
>     (cl-callf stream-rest stream)))
> #+end_src
>
> This is semantically exactly like test2.el, only the call to
> `stream-flush' has been replaced by literally writing out the
> definition.  Nonetheless, the compiled file suddenly doesn't crash Emacs
> when loaded.  Loading the uncompiled file test3.el still crashes.

Aha, but the following also crashes, whether compiled or not:

;; -*- lexical-binding: t -*-

(require 'stream)

(let* ((stream0 (stream-range 1 1000000))
       (stream stream0))
  (while (not (stream-empty-p stream))
    (cl-callf stream-rest stream)))

So the problem is that the initial stream0 object can reach the entire
unfolding stream as it goes, and just holding on to that reference is
enough to keep the whole thing in memory.

Now, I can see that letting stream0 automagically get access to the
unfolded result can be an optimization in some cases, although in this
case it's a pessimization.  It could also affect the semantics if
unfolding the stream has side effects, not sure if stream.el makes
guarantees about that though.

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Michael Heerdegen]

Noam Postavsky <npostavs@gmail.com> writes:

> Aha, but the following also crashes, whether compiled or not:
>
> ;; -*- lexical-binding: t -*-
>
> (require 'stream)
>
> (let* ((stream0 (stream-range 1 1000000))
>        (stream stream0))
>   (while (not (stream-empty-p stream))
>     (cl-callf stream-rest stream)))

I guess you should have some awe if you write something like
(stream-range 1 1000000) and keep in mind what happens with this
gigantic thing.  Though, could the object `stream0' references not
already be garbage-collected when the loop has been entered, since there
is no lexical reference to that variable there?

BTW, it gets even worse if you append streams, since the original
streams are not copied and magically unfolded when you generate elements
from the concatenation.

> Now, I can see that letting stream0 automagically get access to the
> unfolded result can be an optimization in some cases, although in this
> case it's a pessimization.  It could also affect the semantics if
> unfolding the stream has side effects, not sure if stream.el makes
> guarantees about that though.

AFAICT we make no such guarantees at all.  When generating stream
elements has side effects (which is not ideal, but it's not forbidden),
then you must know what you are doing.  In my experience, side effects
often directly correlate with element generation - e.g. for a stream of
search matches, a side effect is to set a variable to the position where
to continue searching.  These kind of side effects are not problematic.

For non-trivial side-effects, dunno, never wanted something like that.
But this problem also concerns other forms of delayed evaluation,
including thunks, and generally everything you can do with closures.


Michael.

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Date: Fri, 02 Mar 2018 16:06:49 +0100
> Cc: Nicolas Petton <nicolas@petton.fr>, 30626@debbugs.gnu.org
> 
> > (let* ((stream0 (stream-range 1 1000000))
> >        (stream stream0))
> >   (while (not (stream-empty-p stream))
> >     (cl-callf stream-rest stream)))
> 
> I guess you should have some awe if you write something like
> (stream-range 1 1000000) and keep in mind what happens with this
> gigantic thing.  Though, could the object `stream0' references not
> already be garbage-collected when the loop has been entered, since there
> is no lexical reference to that variable there?

It's still referenced by the let* form, I think.

But even if it didn't, you cannot rely on it being GC'ed, because
Emacs triggers GC at times which are hard or even impossible to
predict.

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Nicolas Petton]

[-- Attachment #1: Type: text/plain, Size: 894 bytes --]

Noam Postavsky <npostavs@gmail.com> writes:

> Now, I can see that letting stream0 automagically get access to the
> unfolded result can be an optimization in some cases, although in this
> case it's a pessimization.

stream.el is at its core just an implementation of lazy-cons cells, so
not letting stream0 get access to the result would mean changing the
core implementation (I'm not necessarily against it).

If we accept that `seq-elt', and other positional functions of seq.el
should not work on streams, then I could rewrite stream.el to make it a
positioned stream where previous elements are discarded after each
element generation.  However the list of supported functions from seq.el
API would be significantly reduced.

> It could also affect the semantics if unfolding the stream has side
> effects, not sure if stream.el makes guarantees about that though.

No, it doesn't.

Nico

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Nicolas Petton]

[-- Attachment #1: Type: text/plain, Size: 1031 bytes --]

Nicolas Petton <nicolas@petton.fr> writes:

> If we accept that `seq-elt', and other positional functions of seq.el
> should not work on streams, then I could rewrite stream.el to make it a
> positioned stream where previous elements are discarded after each
> element generation.  However the list of supported functions from seq.el
> API would be significantly reduced.

I had something like the following in mind:

  (cl-defstruct nstream current next-function)
  
  (cl-defmethod nstream-next ((stream nstream))
    (setf (nstream-current stream) (funcall (nstream-next-function stream)
                                            (nstream-current stream))))
  
  (defun nstream-range (&optional start end step)
    (unless start (setq start 0))
    (unless step (setq step 1))
    (make-nstream :current start
                  :next-function (lambda (cur)
                           (if (equal cur end)
                               nil
                             (+ cur step)))))

Cheers,
Nico

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Michael Heerdegen]

Nicolas Petton <nicolas@petton.fr> writes:

> I had something like the following in mind:
>
>   (cl-defstruct nstream current next-function)
>   
>   (cl-defmethod nstream-next ((stream nstream))
>     (setf (nstream-current stream) (funcall (nstream-next-function stream)
>                                             (nstream-current stream))))
>   
>   (defun nstream-range (&optional start end step)
>     (unless start (setq start 0))
>     (unless step (setq step 1))
>     (make-nstream :current start
>                   :next-function (lambda (cur)
>                            (if (equal cur end)
>                                nil
>                              (+ cur step)))))

This is an implementation of iterators, not streams.  We already have an
implementation of iterators in Emacs.


Michael.

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Michael Heerdegen]

Nicolas Petton <nicolas@petton.fr> writes:

> stream.el is at its core just an implementation of lazy-cons cells, so
> not letting stream0 get access to the result would mean changing the
> core implementation (I'm not necessarily against it).

Please don't do this.  The semantics of streams is worth keeping.  I
make use of it in el-search and other stuff.


Michael.

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Nicolas Petton]

[-- Attachment #1: Type: text/plain, Size: 214 bytes --]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> Please don't do this.  The semantics of streams is worth keeping.  I
> make use of it in el-search and other stuff.

I'm not saying that I want this :)

Nico

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[John Mastro]

Noam Postavsky <npostavs@gmail.com> wrote:
> Aha, but the following also crashes, whether compiled or not:
>
> ;; -*- lexical-binding: t -*-
>
> (require 'stream)
>
> (let* ((stream0 (stream-range 1 1000000))
>        (stream stream0))
>   (while (not (stream-empty-p stream))
>     (cl-callf stream-rest stream)))
>
> So the problem is that the initial stream0 object can reach the entire
> unfolding stream as it goes, and just holding on to that reference is
> enough to keep the whole thing in memory.
>
> Now, I can see that letting stream0 automagically get access to the
> unfolded result can be an optimization in some cases, although in this
> case it's a pessimization.  It could also affect the semantics if
> unfolding the stream has side effects, not sure if stream.el makes
> guarantees about that though.

Clojure has/had a similar issue. They describe this scenario (having a
live reference to the beginning of the stream, preventing GC from
collecting it) as "holding onto the head" of the stream (in Clojure
they're called lazy seqs).

Their solution was what they call "locals clearing". The compiler tracks
the lifetimes of local bindings and "clears" them (by setting them to
nil/null) after their point of last use, e.g.:

(let* ((stream0 (stream-range 1 1000000))
       (stream stream0))
  (setq stream0 nil) ;; <<< Inserted by compiler
  (while (not (stream-empty-p stream))
    (cl-callf stream-rest stream)))

If the code does reference stream0 later, locals clearing can't help
you, but that's considered a "if it hurts, don't do it" situation.

This probably isn't practical for Emacs, especially since it could only
work for byte-compiled code, but thought the prior art may be of
interest.

        John

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Noam Postavsky]

John Mastro <john.b.mastro@gmail.com> writes:

> (let* ((stream0 (stream-range 1 1000000))
>        (stream stream0))
>   (setq stream0 nil) ;; <<< Inserted by compiler
>   (while (not (stream-empty-p stream))
>     (cl-callf stream-rest stream)))
>
> If the code does reference stream0 later, locals clearing can't help
> you, but that's considered a "if it hurts, don't do it" situation.
>
> This probably isn't practical for Emacs, especially since it could only
> work for byte-compiled code, but thought the prior art may be of
> interest.

Not sure how doable this solution is, but the fact that it works only
for byte-compiled code seems fine to me.  The interpreted case is doomed
to fail anyway since the interpreter doesn't prune redundant variables
from closures.

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Noam Postavsky]

Noam Postavsky <npostavs@gmail.com> writes:

> John Mastro <john.b.mastro@gmail.com> writes:
>
>> (let* ((stream0 (stream-range 1 1000000))
>>        (stream stream0))
>>   (setq stream0 nil) ;; <<< Inserted by compiler
>>   (while (not (stream-empty-p stream))
>>     (cl-callf stream-rest stream)))
>>
>> If the code does reference stream0 later, locals clearing can't help
>> you, but that's considered a "if it hurts, don't do it" situation.
>>
>> This probably isn't practical for Emacs, especially since it could only
>> work for byte-compiled code, but thought the prior art may be of
>> interest.
>
> Not sure how doable this solution is, but the fact that it works only
> for byte-compiled code seems fine to me.  The interpreted case is doomed
> to fail anyway since the interpreter doesn't prune redundant variables
> from closures.

Hmm, I think it won't work by itself though, just doing

    (stream-flush (stream-range 1 1000000))

also crashes, due to the head of the stream being referenced from the C
stack somewhere (I can get the address from gdb, but I can't figure out
how to get to the corresponding C variable from there).

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Eli Zaretskii]

> From: Noam Postavsky <npostavs@gmail.com>
> Date: Sun, 04 Mar 2018 10:56:36 -0500
> Cc: Michael Heerdegen <michael_heerdegen@web.de>,
> 	Nicolas Petton <nicolas@petton.fr>, 30626@debbugs.gnu.org
> 
> Hmm, I think it won't work by itself though, just doing
> 
>     (stream-flush (stream-range 1 1000000))
> 
> also crashes, due to the head of the stream being referenced from the C
> stack somewhere (I can get the address from gdb, but I can't figure out
> how to get to the corresponding C variable from there).

Did you try "info symbol ADDRESS"?  (I'm not sure this will work for
automatic variables, though.)

You could also try "info locals" after "set print address on" and/or
"set print symbol on".

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Noam Postavsky]

[-- Attachment #1: Type: text/plain, Size: 1319 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> also crashes, due to the head of the stream being referenced from the C
>> stack somewhere (I can get the address from gdb, but I can't figure out
>> how to get to the corresponding C variable from there).
>
> Did you try "info symbol ADDRESS"?  (I'm not sure this will work for
> automatic variables, though.)

Doesn't seem to work.  I guess it wouldn't work if the address was in
the middle of an array either.

> You could also try "info locals" after "set print address on" and/or
> "set print symbol on".

Those settings don't seem to help.

I first guessed that the problem is due to saving function arguments
during funcall, so I tried the following to check it:

--- i/src/bytecode.c
+++ w/src/bytecode.c
@@ -387,7 +387,10 @@ exec_byte_code (Lisp_Object bytestr, Lisp_Object vector, Lisp_Object maxdepth,
 			make_number (nargs)));
       ptrdiff_t pushedargs = min (nonrest, nargs);
       for (ptrdiff_t i = 0; i < pushedargs; i++, args++)
-	PUSH (*args);
+        {
+          PUSH (*args);
+          *args = Qnil;
+        }
       if (nonrest < nargs)
 	PUSH (Flist (nargs - nonrest, args));
       else

This did change the backtrace (from starting with mark_specpdl to
mark_stack), meaning I did find one reference, but it still crashes, so
there must be more.


[-- Attachment #2: gdb log excerpts --]
[-- Type: text/plain, Size: 7995 bytes --]

No stack.
Starting program: /home/npostavs/src/emacs/emacs-26/lucid/src/emacs -Q -batch -l elpa/packages/stream/stream.elc -l bug-30626-stream-crash/test5.elc
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
0x000000000060e9b5 in live_cons_holding (m=0x2f6e290, p=0x2f77620) at ../../src/alloc.c:4613
4613		    return make_lisp_ptr (s, Lisp_Cons);
#0  0x000000000060e9b5 in live_cons_holding (m=0x2f6e290, p=0x2f77620) at ../../src/alloc.c:4613
#1  0x000000000060e9ec in live_cons_p (m=0x2f6e290, p=0x2f77620) at ../../src/alloc.c:4622
#2  0x0000000000612f94 in mark_object (arg=XIL(0x2f77623)) at ../../src/alloc.c:6717
#3  0x0000000000611d4f in mark_vectorlike (ptr=0x2f78b50) at ../../src/alloc.c:6227
[...]
#4850 0x0000000000612b42 in mark_object (arg=XIL(0x2efcb83)) at ../../src/alloc.c:6624
#4851 0x0000000000611d4f in mark_vectorlike (ptr=0x2e64c90) at ../../src/alloc.c:6227
#4852 0x0000000000612b42 in mark_object (arg=XIL(0x2e64c95)) at ../../src/alloc.c:6624
#4853 0x000000000060f3ce in mark_maybe_pointer (p=0x2e64c90) at ../../src/alloc.c:4936
#4854 0x000000000060f452 in mark_memory (start=0x7fffffffa520, end=0x7fffffffe868)
    at ../../src/alloc.c:4985
#4855 0x000000000060f493 in mark_stack (bottom=0x7fffffffe868 "a\036h\364\377\177", 
    end=0x7fffffffa520 "0\245\377\377\377\177") at ../../src/alloc.c:5193
#4856 0x00000000006cdd75 in mark_one_thread (thread=0xe103e0 <main_thread>) at ../../src/thread.c:616
#4857 0x00000000006cdf0a in mark_threads_callback (ignore=0x0) at ../../src/thread.c:649
#4858 0x000000000060f4da in flush_stack_call_func (func=0x6cde77 <mark_threads_callback>, arg=0x0)
    at ../../src/alloc.c:5220
#4859 0x00000000006cdf3c in mark_threads () at ../../src/thread.c:656
#4860 0x00000000006114a7 in garbage_collect_1 (end=0x7fffffffa700) at ../../src/alloc.c:5997
#4861 0x0000000000611b94 in Fgarbage_collect () at ../../src/alloc.c:6168
#4862 0x000000000057999d in maybe_gc () at ../../src/lisp.h:4749
#4863 0x000000000063c2cb in Ffuncall (nargs=6, args=0x7fffffffa7f8) at ../../src/eval.c:2751
#4864 0x000000000068d950 in exec_byte_code (bytestr=XIL(0x2e7aad4), vector=XIL(0x2e72715), 
    maxdepth=make_number(18), args_template=make_number(768), nargs=3, args=0x7fffffffad20)
    at ../../src/bytecode.c:632
#4865 0x000000000063cfef in funcall_lambda (fun=XIL(0x1450e45), nargs=3, arg_vector=0x7fffffffad08)
    at ../../src/eval.c:2970
#4866 0x000000000063c402 in Ffuncall (nargs=4, args=0x7fffffffad00) at ../../src/eval.c:2771
#4867 0x000000000068d950 in exec_byte_code (bytestr=XIL(0x2e7b2d4), vector=XIL(0x2fe2945), 
    maxdepth=make_number(7), args_template=make_number(256), nargs=0, args=0x7fffffffb1d8)
    at ../../src/bytecode.c:632
#4868 0x000000000063cfef in funcall_lambda (fun=XIL(0x2fe2985), nargs=0, arg_vector=0x7fffffffb1d8)
    at ../../src/eval.c:2970
#4869 0x000000000063c402 in Ffuncall (nargs=1, args=0x7fffffffb1d0) at ../../src/eval.c:2771
#4870 0x000000000068d950 in exec_byte_code (bytestr=XIL(0x2e769d4), vector=XIL(0x9d00a5), 
    maxdepth=make_number(2), args_template=make_number(257), nargs=1, args=0x7fffffffb670)
    at ../../src/bytecode.c:632
#4871 0x000000000063cfef in funcall_lambda (fun=XIL(0x1626f05), nargs=1, arg_vector=0x7fffffffb668)
    at ../../src/eval.c:2970
#4872 0x000000000063c402 in Ffuncall (nargs=2, args=0x7fffffffb660) at ../../src/eval.c:2771
#4873 0x000000000068d950 in exec_byte_code (bytestr=XIL(0x2e7f7b4), vector=XIL(0x1622fe5), 
    maxdepth=make_number(3), args_template=make_number(257), nargs=1, args=0x7fffffffbb10)
    at ../../src/bytecode.c:632
#4874 0x000000000063cfef in funcall_lambda (fun=XIL(0x1450f35), nargs=1, arg_vector=0x7fffffffbb08)
    at ../../src/eval.c:2970
#4875 0x000000000063c402 in Ffuncall (nargs=2, args=0x7fffffffbb00) at ../../src/eval.c:2771
#4876 0x000000000068d950 in exec_byte_code (bytestr=XIL(0x2ee3c94), vector=XIL(0x2e664f5), 
    maxdepth=make_number(3), args_template=make_number(257), nargs=1, args=0x7fffffffbfa8)
    at ../../src/bytecode.c:632
#4877 0x000000000063cfef in funcall_lambda (fun=XIL(0x2e66515), nargs=1, arg_vector=0x7fffffffbfa0)
    at ../../src/eval.c:2970
#4878 0x000000000063c402 in Ffuncall (nargs=2, args=0x7fffffffbf98) at ../../src/eval.c:2771
#4879 0x000000000068d950 in exec_byte_code (bytestr=XIL(0x2e91054), vector=XIL(0x1699ef5), 
    maxdepth=make_number(4), args_template=XIL(0), nargs=0, args=0x0) at ../../src/bytecode.c:632
#4880 0x000000000068cb20 in Fbyte_code (bytestr=XIL(0x2e91054), vector=XIL(0x1699ef5), 
    maxdepth=make_number(4)) at ../../src/bytecode.c:321
#4881 0x000000000063ac83 in eval_sub (form=XIL(0x2eeb413)) at ../../src/eval.c:2240
#4882 0x0000000000672b34 in readevalloop (readcharfun=XIL(0x68a0), infile0=0x7fffffffc6b0, 
    sourcename=XIL(0x2e90f94), printflag=false, unibyte=XIL(0), readfun=XIL(0), start=XIL(0), 
    end=XIL(0)) at ../../src/lread.c:2038
#4883 0x0000000000670e85 in Fload (file=XIL(0x2e8d884), noerror=XIL(0), nomessage=XIL(0xc090), 
    nosuffix=XIL(0), must_suffix=XIL(0)) at ../../src/lread.c:1425
#4884 0x000000000063c945 in funcall_subr (subr=0xd7a420 <Sload>, numargs=3, args=0x7fffffffc8e8)
    at ../../src/eval.c:2856
#4885 0x000000000063c3be in Ffuncall (nargs=4, args=0x7fffffffc8e0) at ../../src/eval.c:2769
#4886 0x000000000068d950 in exec_byte_code (bytestr=XIL(0xadcc2c), vector=XIL(0xadcc4d), 
    maxdepth=make_number(23), args_template=make_number(257), nargs=1, args=0x7fffffffd248)
    at ../../src/bytecode.c:632
#4887 0x000000000063cfef in funcall_lambda (fun=XIL(0xadcbfd), nargs=1, arg_vector=0x7fffffffd240)
    at ../../src/eval.c:2970
#4888 0x000000000063c402 in Ffuncall (nargs=2, args=0x7fffffffd238) at ../../src/eval.c:2771
#4889 0x000000000068d950 in exec_byte_code (bytestr=XIL(0xad7424), vector=XIL(0xad7445), 
    maxdepth=make_number(21), args_template=make_number(0), nargs=0, args=0x7fffffffde58)
    at ../../src/bytecode.c:632
#4890 0x000000000063cfef in funcall_lambda (fun=XIL(0xad73f5), nargs=0, arg_vector=0x7fffffffde58)
    at ../../src/eval.c:2970
#4891 0x000000000063c402 in Ffuncall (nargs=1, args=0x7fffffffde50) at ../../src/eval.c:2771
#4892 0x000000000068d950 in exec_byte_code (bytestr=XIL(0xad6614), vector=XIL(0xad6635), 
    maxdepth=make_number(12), args_template=make_number(0), nargs=0, args=0x7fffffffe480)
    at ../../src/bytecode.c:632
#4893 0x000000000063cfef in funcall_lambda (fun=XIL(0xad65e5), nargs=0, arg_vector=0x7fffffffe480)
    at ../../src/eval.c:2970
#4894 0x000000000063cc29 in apply_lambda (fun=XIL(0xad65e5), args=XIL(0), count=4)
    at ../../src/eval.c:2906
#4895 0x000000000063ae2a in eval_sub (form=XIL(0x149a8e3)) at ../../src/eval.c:2279
#4896 0x000000000063a19f in Feval (form=XIL(0x149a8e3), lexical=XIL(0)) at ../../src/eval.c:2054
#4897 0x000000000058150f in top_level_2 () at ../../src/keyboard.c:1119
#4898 0x00000000006384ce in internal_condition_case (bfun=0x5814ec <top_level_2>, 
    handlers=XIL(0x5250), hfun=0x580ef1 <cmd_error>) at ../../src/eval.c:1332
#4899 0x0000000000581554 in top_level_1 (ignore=XIL(0)) at ../../src/keyboard.c:1127
#4900 0x00000000006379d7 in internal_catch (tag=XIL(0xc6f0), func=0x581511 <top_level_1>, arg=XIL(0))
    at ../../src/eval.c:1097
#4901 0x000000000058143e in command_loop () at ../../src/keyboard.c:1088
#4902 0x00000000005809db in recursive_edit_1 () at ../../src/keyboard.c:695
#4903 0x0000000000580bd0 in Frecursive_edit () at ../../src/keyboard.c:766
#4904 0x000000000057e76b in main (argc=7, argv=0x7fffffffe9e8) at ../../src/emacs.c:1713
Cannot access memory at address 0x7ffffff7ef7f

(gdb) frame 4854
#4854 0x000000000060f452 in mark_memory (start=0x7fffffffa520, end=0x7fffffffe868)
    at ../../src/alloc.c:4985
4985	      mark_maybe_pointer (*(void **) pp);
(gdb) info locals
pp = 0x7fffffffa968 "\220L\346\002"
(gdb) info symbol 0x7fffffffa968
No symbol matches 0x7fffffffa968.

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Eli Zaretskii]

> From: Noam Postavsky <npostavs@gmail.com>
> Cc: michael_heerdegen@web.de,  john.b.mastro@gmail.com,  nicolas@petton.fr,  30626@debbugs.gnu.org
> Date: Sun, 11 Mar 2018 14:52:22 -0400
> 
> This did change the backtrace (from starting with mark_specpdl to
> mark_stack), meaning I did find one reference, but it still crashes, so
> there must be more.

If you have the address, you could first find the stack frame to which
it belongs, right?  Then go to that stack frame and type "info
locals", which should give you the locals in that frame.  One of them
is your variable.  It could also be a temporary slot, in which case
disassembly of that frame's function should show it.

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Noam Postavsky]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Noam Postavsky <npostavs@gmail.com>
>> Cc: michael_heerdegen@web.de,  john.b.mastro@gmail.com,  nicolas@petton.fr,  30626@debbugs.gnu.org
>> Date: Sun, 11 Mar 2018 14:52:22 -0400
>> 
>> This did change the backtrace (from starting with mark_specpdl to
>> mark_stack), meaning I did find one reference, but it still crashes, so
>> there must be more.
>
> If you have the address, you could first find the stack frame to which
> it belongs, right?

Um, how do I do that part?

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Eli Zaretskii]

> From: Noam Postavsky <npostavs@gmail.com>
> Cc: michael_heerdegen@web.de,  john.b.mastro@gmail.com,  nicolas@petton.fr,  30626@debbugs.gnu.org
> Date: Sun, 11 Mar 2018 17:51:19 -0400
> 
> > If you have the address, you could first find the stack frame to which
> > it belongs, right?
> 
> Um, how do I do that part?

By comparing the address with the value of $bp in each frame, I'd say.

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Noam Postavsky]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Noam Postavsky <npostavs@gmail.com>
>> Cc: michael_heerdegen@web.de,  john.b.mastro@gmail.com,  nicolas@petton.fr,  30626@debbugs.gnu.org
>> Date: Sun, 11 Mar 2018 17:51:19 -0400
>> 
>> > If you have the address, you could first find the stack frame to which
>> > it belongs, right?
>> 
>> Um, how do I do that part?
>
> By comparing the address with the value of $bp in each frame, I'd say.

Hmm, I found a match, but it doesn't make any sense.

#4851 0x0000000000611d4f in mark_vectorlike (ptr=0x2e64c90) at ../../src/alloc.c:6227
#4852 0x0000000000612b42 in mark_object (arg=XIL(0x2e64c95)) at ../../src/alloc.c:6624
#4853 0x000000000060f3ce in mark_maybe_pointer (p=0x2e64c90) at ../../src/alloc.c:4936
#4854 0x000000000060f452 in mark_memory (start=0x7fffffffa520, end=0x7fffffffe868)
    at ../../src/alloc.c:4985
#4855 0x000000000060f493 in mark_stack (bottom=0x7fffffffe868 "a\036h\364\377\177", 
    end=0x7fffffffa520 "0\245\377\377\377\177") at ../../src/alloc.c:5193

(gdb) frame 4854
#4854 0x000000000060f452 in mark_memory (start=0x7fffffffa520, end=0x7fffffffe868)
    at ../../src/alloc.c:4985
4985	      mark_maybe_pointer (*(void **) pp);
(gdb) p pp
$28 = 0x7fffffffa968 "\220L\346\002"

(gdb) frame 4864
#4864 0x000000000068d950 in exec_byte_code (bytestr=XIL(0x2e7aad4), vector=XIL(0x2e72715), 
    maxdepth=make_number(18), args_template=make_number(768), nargs=3, args=0x7fffffffad20)
    at ../../src/bytecode.c:632
632		    TOP = Ffuncall (op + 1, &TOP);
(gdb) p $rbp
$29 = (void *) 0x7fffffffabd0

(gdb) p/x $rbp - $28
$32 = 0x268

(gdb) disas /s
[...]
1180		CASE (Bbuffer_substring):
1181		  {
1182		    Lisp_Object v1 = POP;
   0x000000000068fea4 <+13154>:	mov    -0x40(%rbp),%rax
   0x000000000068fea8 <+13158>:	lea    -0x8(%rax),%rdx
   0x000000000068feac <+13162>:	mov    %rdx,-0x40(%rbp)
   0x000000000068feb0 <+13166>:	mov    (%rax),%rax
   0x000000000068feb3 <+13169>:	mov    %rax,-0x268(%rbp)

1183		    TOP = Fbuffer_substring (TOP, v1);
   0x000000000068feba <+13176>:	mov    -0x268(%rbp),%rdx
   0x000000000068fec1 <+13183>:	mov    -0x40(%rbp),%rax
   0x000000000068fec5 <+13187>:	mov    %rdx,%rsi
   0x000000000068fec8 <+13190>:	mov    (%rax),%rdi
   0x000000000068fecb <+13193>:	callq  0x627e0a <Fbuffer_substring>

It can't be a buffer-substring arg, but that's the only reference to
-0x268(%rbp) in that function.

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Eli Zaretskii]

> From: Noam Postavsky <npostavs@gmail.com>
> Cc: michael_heerdegen@web.de,  john.b.mastro@gmail.com,  nicolas@petton.fr,  30626@debbugs.gnu.org
> Date: Mon, 12 Mar 2018 21:59:57 -0400
> 
> 4985	      mark_maybe_pointer (*(void **) pp);
> (gdb) p pp
> $28 = 0x7fffffffa968 "\220L\346\002"

Should you look at pp or at *pp?

Also note that for Lisp objects that are marked you need to reset
their mark bit before trying to determine their type and value.

If none of the above helps, please walk me through the steps that led
you to look at -0x268(%rbp), because I'm not sure I follow.

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Noam Postavsky]

Eli Zaretskii <eliz@gnu.org> writes:

> Should you look at pp or at *pp?

I think it should be pp, but I'm not sure.  The context:

#4854 0x000000000060f452 in mark_memory (start=0x7fffffffa520, end=0x7fffffffe868)
    at ../../src/alloc.c:4985
#4855 0x000000000060f493 in mark_stack (bottom=0x7fffffffe868 "a\036h\364\377\177", 
    end=0x7fffffffa520 "0\245\377\377\377\177") at ../../src/alloc.c:5193

mark_memory (void *start, void *end)
{
  ...
  for (pp = start; (void *) pp < end; pp += GC_POINTER_ALIGNMENT)
    {
      mark_maybe_pointer (*(void **) pp);
      mark_maybe_object (*(Lisp_Object *) pp);
    }

So the value of pp ranges over stack addresses and *pp would be the
contents of the stack location.

> Also note that for Lisp objects that are marked you need to reset
> their mark bit before trying to determine their type and value.

I think I'm looking for a C variable, and not a Lisp object (although
the C variable presumably contains/points to a Lisp object).

> If none of the above helps, please walk me through the steps that led
> you to look at -0x268(%rbp), because I'm not sure I follow.

Starting with the value of pp, I then go up looking for a close value of
$rbp:

(gdb) p pp
$39 = 0x7fffffffa968 "\220L\346\002"

(gdb) up
#4855 0x000000000060f493 in mark_stack (bottom=0x7fffffffe868 "a\036h\364\377\177", 
    end=0x7fffffffa520 "0\245\377\377\377\177") at ../../src/alloc.c:5193
5193	  mark_memory (bottom, end);
(gdb) p $rbp
$40 = (void *) 0x7fffffffa420
(gdb) up
#4856 0x00000000006cdd75 in mark_one_thread (thread=0xe103e0 <main_thread>) at ../../src/thread.c:616
616	  mark_stack (thread->m_stack_bottom, stack_top);
(gdb) p $rbp
$41 = (void *) 0x7fffffffa470
[...]
(gdb) up 
#4863 0x000000000063c2cb in Ffuncall (nargs=6, args=0x7fffffffa7f8) at ../../src/eval.c:2751
2751	  maybe_gc ();
(gdb) p $rbp
$48 = (void *) 0x7fffffffa780
(gdb) up 
#4864 0x000000000068d950 in exec_byte_code (bytestr=XIL(0x2e7aad4), vector=XIL(0x2e72715), 
    maxdepth=make_number(18), args_template=make_number(768), nargs=3, args=0x7fffffffad20)
    at ../../src/bytecode.c:632
632		    TOP = Ffuncall (op + 1, &TOP);
(gdb) p $rbp
$49 = (void *) 0x7fffffffabd0

Now I see that $rbp is higher than the target address, and the
difference is 0x268, so the target location should be -0x268(%rbp).

(gdb) p $rbp - 0x7fffffffa968
$52 = (void *) 0x268

Except something must be wrong in my reasoning, since the only
ocurrences of -0x268(%rbp) are the buffer-string args, which could only
hold integers or markers (neither of which could further point to long
chains of objects).

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Eli Zaretskii]

> From: Noam Postavsky <npostavs@gmail.com>
> Cc: michael_heerdegen@web.de,  john.b.mastro@gmail.com,  nicolas@petton.fr,  30626@debbugs.gnu.org
> Date: Tue, 13 Mar 2018 20:09:17 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Should you look at pp or at *pp?
> 
> I think it should be pp, but I'm not sure.  The context:
> 
> #4854 0x000000000060f452 in mark_memory (start=0x7fffffffa520, end=0x7fffffffe868)
>     at ../../src/alloc.c:4985
> #4855 0x000000000060f493 in mark_stack (bottom=0x7fffffffe868 "a\036h\364\377\177", 
>     end=0x7fffffffa520 "0\245\377\377\377\177") at ../../src/alloc.c:5193
> 
> mark_memory (void *start, void *end)
> {
>   ...
>   for (pp = start; (void *) pp < end; pp += GC_POINTER_ALIGNMENT)
>     {
>       mark_maybe_pointer (*(void **) pp);
>       mark_maybe_object (*(Lisp_Object *) pp);
>     }
> 
> So the value of pp ranges over stack addresses and *pp would be the
> contents of the stack location.

But the call to mark_maybe_pointer means that we consider pp to be a
pointer (in)to a Lisp object.

Anyway, wouldn't it be easier to look one frame lower?  We have this:

  #4850 0x0000000000612b42 in mark_object (arg=XIL(0x2efcb83)) at ../../src/alloc.c:6624
  #4851 0x0000000000611d4f in mark_vectorlike (ptr=0x2e64c90) at ../../src/alloc.c:6227
  #4852 0x0000000000612b42 in mark_object (arg=XIL(0x2e64c95)) at ../../src/alloc.c:6624
  #4853 0x000000000060f3ce in mark_maybe_pointer (p=0x2e64c90) at ../../src/alloc.c:4936
  #4854 0x000000000060f452 in mark_memory (start=0x7fffffffa520, end=0x7fffffffe868)
      at ../../src/alloc.c:4985
  #4855 0x000000000060f493 in mark_stack (bottom=0x7fffffffe868 "a\036h\364\377\177", 
      end=0x7fffffffa520 "0\245\377\377\377\177") at ../../src/alloc.c:5193

In frame #4852, we have found an object, and we are marking it.  Did
you try looking at that object?  With these caveats:

> > Also note that for Lisp objects that are marked you need to reset
> > their mark bit before trying to determine their type and value.

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Noam Postavsky]

Eli Zaretskii <eliz@gnu.org> writes:

> In frame #4852, we have found an object, and we are marking it.  Did
> you try looking at that object?  With these caveats:
>
>> > Also note that for Lisp objects that are marked you need to reset
>> > their mark bit before trying to determine their type and value.

Okay, I think xpr takes care of that, right?  (I've restarted the debug
sessions a few times, so numbers may not match exactly)

#4853 0x000000000060f429 in mark_maybe_pointer (p=0x2e64c90) at ../../src/alloc.c:4936
4936		mark_object (obj);
(gdb) p obj
$60 = XIL(0x2e64c95)
(gdb) xpr
Lisp_Vectorlike
PVEC_NORMAL_VECTOR
$61 = (struct Lisp_Vector *) 0x2e64c90
{XIL(0x2efcb63), make_number(1000000), XIL(0x2efcb53), XIL(0x2efcb73), XIL(0x2efcb83), 
  XIL(0x20ab5b0), XIL(0xc090)}
(gdb) p $61->contents[0]
$62 = XIL(0x2efcb63)
(gdb) xpr
Lisp_Cons
$63 = (struct Lisp_Cons *) 0x2efcb60
{
  [...]
      car = make_number(2369), 
      [...]
        chain = 0x0
[...]
(gdb) p $61->contents[1]
$64 = make_number(1000000)
(gdb) p $61->contents[2]
$65 = XIL(0x2efcb53)
(gdb) xpr
Lisp_Cons
[...] car = make_number(1), [...] chain = 0x0 [...]
(gdb) p $61->contents[3]
$67 = XIL(0x2efcb73)
(gdb) xpr
[...] car = XIL(0xc090), [...] chain = 0x0 [...]
(gdb) p $61->contents[4]
[...] car = XIL(0x2efc443), [...] chain = 0x0 [...]
(gdb) p $61->contents[5]
[...]
$72 = (struct Lisp_Symbol *) 0x2e97ef0
"stream-range"
(gdb) p $61->contents[6]
[...]
$74 = (struct Lisp_Symbol *) 0xdf89d0 <lispsym+49296>
"t"


It looks like a the lexical environment of a bytecode function, probably
the initial stream, e.g., (stream-range 1 1000000) gives:

(--stream-- #[256 "\211\203\303\242\207\303\242\204 \304\300\242\305\300\242\302\242\\\301\302\242#B\240\210\303\306\240\210\304\242\207" 
              [(1) 1000000 (1) (nil) (nil) stream-range t] 7 "
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

(fn &optional CHECK)"])

Not sure where to go next with this.

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Eli Zaretskii]

> From: Noam Postavsky <npostavs@gmail.com>
> Cc: michael_heerdegen@web.de,  john.b.mastro@gmail.com,  nicolas@petton.fr,  30626@debbugs.gnu.org
> Date: Sat, 17 Mar 2018 11:53:36 -0400
> 
> It looks like a the lexical environment of a bytecode function, probably
> the initial stream, e.g., (stream-range 1 1000000) gives:
> 
> (--stream-- #[256 "\211\203\303\242\207\303\242\204 \304\300\242\305\300\242\302\242\\\301\302\242#B\240\210\303\306\240\210\304\242\207" 
>               [(1) 1000000 (1) (nil) (nil) stream-range t] 7 "
>               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> 
> (fn &optional CHECK)"])
> 
> Not sure where to go next with this.

The goal was to find out which variable holds a reference to the
entire long stream, right?  So it sounds like a pointer to it is kept
in an automatic variable on the stack of exec_byte_code, right?  Which
kinda makes sense, since the stream is still being processed, I think.

Or am I confused?

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Eli Zaretskii]

> Date: Sat, 17 Mar 2018 18:10:24 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: michael_heerdegen@web.de, john.b.mastro@gmail.com, nicolas@petton.fr,
> 	30626@debbugs.gnu.org
> 
> > (--stream-- #[256 "\211\203\303\242\207\303\242\204 \304\300\242\305\300\242\302\242\\\301\302\242#B\240\210\303\306\240\210\304\242\207" 
> >               [(1) 1000000 (1) (nil) (nil) stream-range t] 7 "
> >               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > 
> > (fn &optional CHECK)"])
> > 
> > Not sure where to go next with this.
> 
> The goal was to find out which variable holds a reference to the
> entire long stream, right?  So it sounds like a pointer to it is kept
> in an automatic variable on the stack of exec_byte_code, right?  Which
> kinda makes sense, since the stream is still being processed, I think.
> 
> Or am I confused?

Actually, there's still some mystery: if this object is a 7-element
vector, where do all the other GC frame come from?  Hmm... how
long/deep is each of the cons cells in elements 1 through 4 of the
vector?  If they are deeply nested, then that's the answer we've been
looking for, I think.

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Noam Postavsky]

Eli Zaretskii <eliz@gnu.org> writes:

>> The goal was to find out which variable holds a reference to the
>> entire long stream, right?  So it sounds like a pointer to it is kept
>> in an automatic variable on the stack of exec_byte_code, right?  Which
>> kinda makes sense, since the stream is still being processed, I
>> think.

Yeah, but which variable exactly?  I'd like to find it and add 'X =
Qnil;' to confirm we've found where it is.

> Actually, there's still some mystery: if this object is a 7-element
> vector, where do all the other GC frame come from?  Hmm... how
> long/deep is each of the cons cells in elements 1 through 4 of the
> vector?  If they are deeply nested, then that's the answer we've been
> looking for, I think.

It's a bit confusing because of the indirection: stream-range uses the
stream-cons macro, which uses the stream-make macro, which uses the
thunk-delay macro.  I believe the end result is that the lexical
environment of the resulting closure has access to the next
stream-element in the chain, so the nesting depth is the length of the
stream (i.e., 100000 in the example).  Perhaps this example makes it
clearer:

    (setq print-circle t)

    (let* ((s0 (stream-range 1 2))
           (s1 (stream-rest s0)))
      (list s0 s1))
    ;=>
    ((--stream--
      #[256 "\211\203..."
        [(1) 2 (1) (t)
         ((1 . #1=(--stream--
               #[256 "\211\203..."
                 [(nil) (nil) nil t]
                 3 "\n\n(fn &optional CHECK)"])))
         stream-range t]
        7 "\n\n(fn &optional CHECK)"])
     #1#)

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Eli Zaretskii]

> From: Noam Postavsky <npostavs@gmail.com>
> Cc: michael_heerdegen@web.de,  john.b.mastro@gmail.com,  nicolas@petton.fr,  30626@debbugs.gnu.org
> Date: Sat, 17 Mar 2018 13:28:22 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> The goal was to find out which variable holds a reference to the
> >> entire long stream, right?  So it sounds like a pointer to it is kept
> >> in an automatic variable on the stack of exec_byte_code, right?  Which
> >> kinda makes sense, since the stream is still being processed, I
> >> think.
> 
> Yeah, but which variable exactly?  I'd like to find it and add 'X =
> Qnil;' to confirm we've found where it is.

I don't think you will be able to tell without stepping through the
byte-code interpreter code, keeping track of what it stores where.

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Noam Postavsky]

[-- Attachment #1: Type: text/plain, Size: 1382 bytes --]

Michael Heerdegen <michael_heerdegen@web.de> writes:

>> I don't have a quick answer for the general case, but I think it's a bug
>> in stream.el that it's creating such large structures in the first
>> place.  As far as I understand it, the point of streams is to handle
>> long lists by encoding them as
>>
>>     (FIRST-VALUE . FUNCTION-TO-PRODUCE-REST-OF-LIST)
>
> Yes, that's exactly how it's implemented.  When requesting more elements
> from the stream, that becomes
>
>       (FIRST-VALUE .
>         (SECOND-VALUE . FUNCTION-TO-PRODUCE-MORE-REST-OF-LIST))
>
> When we loop over the string, the cons whose car is the FIRST-VALUE,
> let's call it cons1, is immediately thrown away, and we continue with
>
>       (SECOND-VALUE . FUNCTION-TO-PRODUCE-MORE-REST-OF-LIST)

Coming back to this again.  I think I got lost in the weeds of the
byte-code function objects before.  The core problem is that streams are
not exactly encoded like the above, because even after forcing it you
don't have just a plain SECOND-VALUE stored in the stream.  The original
FUNCTION-TO-PRODUCE-MORE-REST-OF-LIST stays around and keeps referencing
all the code and all the following elements.  So a possible solution is
change the stream to get rid of the lambda part and just leave the
computed value after it's forced.  With the following patch
(stream-flush (stream-range 1 1000000)) succeeds:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: patch --]
[-- Type: text/x-diff, Size: 2937 bytes --]

From 5e7139618f1b4cebbe9785ea5a9303b80d1b2c92 Mon Sep 17 00:00:00 2001
From: Noam Postavsky <npostavs@users.sourceforge.net>
Date: Wed, 24 Apr 2019 22:51:19 -0400
Subject: [PATCH] [WIP] Drop forced lambda's from stream (Bug#30626)

Let the stream id distinguish between forced and unforced stream
values, replacing (think-delay BODY) with just (lambda () BODY).  When
the value is forced, replace the lambda with its result.
---
 packages/stream/stream.el | 30 +++++++++++++++++++++---------
 1 file changed, 21 insertions(+), 9 deletions(-)

diff --git a/packages/stream/stream.el b/packages/stream/stream.el
index 3f6bc4b5b..fa7a3b520 100644
--- a/packages/stream/stream.el
+++ b/packages/stream/stream.el
@@ -65,18 +65,28 @@
 
 (eval-when-compile (require 'cl-lib))
 (require 'seq)
-(require 'thunk)
 
 (eval-and-compile
-  (defconst stream--identifier '--stream--
-    "Symbol internally used to identify streams."))
+  (defconst stream--fresh-identifier '--stream-fresh--
+    "Symbol internally used to streams whose head was not evaluated.")
+  (defconst stream--evald-identifier '--stream-evald--
+    "Symbol internally used to streams whose head was evaluated."))
 
 (defmacro stream-make (&rest body)
   "Return a stream built from BODY.
 BODY must return nil or a cons cell whose cdr is itself a
 stream."
   (declare (debug t))
-  `(list ',stream--identifier (thunk-delay ,@body)))
+  `(list ',stream--fresh-identifier (lambda () ,@body)))
+
+(defun stream-force (stream)
+  (cond
+   ((eq (car-safe stream) stream--evald-identifier)
+    (cadr stream))
+   ((eq (car-safe stream) stream--fresh-identifier)
+    (setf (car stream) stream--evald-identifier)
+    (setf (cadr stream) (funcall (cadr stream))))
+   (t (signal 'wrong-type-argument (list 'streamp stream)))))
 
 (defmacro stream-cons (first rest)
   "Return a stream built from the cons of FIRST and REST.
@@ -159,24 +169,26 @@ (defun stream-range (&optional start end step)
 
 (defun streamp (stream)
   "Return non-nil if STREAM is a stream, nil otherwise."
-  (eq (car-safe stream) stream--identifier))
+  (let ((car (car-safe stream)))
+    (or (eq car stream--fresh-identifier)
+        (eq car stream--evald-identifier))))
 
 (defun stream-empty ()
   "Return a new empty stream."
-  (list stream--identifier (thunk-delay nil)))
+  (list stream--evald-identifier nil))
 
 (defun stream-empty-p (stream)
   "Return non-nil if STREAM is empty, nil otherwise."
-  (null (thunk-force (cadr stream))))
+  (null (cadr (stream-force stream))))
 
 (defun stream-first (stream)
   "Return the first element of STREAM.
 Return nil if STREAM is empty."
-  (car (thunk-force (cadr stream))))
+  (car (stream-force stream)))
 
 (defun stream-rest (stream)
   "Return a stream of all but the first element of STREAM."
-  (or (cdr (thunk-force (cadr stream)))
+  (or (cdr (stream-force stream))
       (stream-empty)))
 
 (defun stream-append (&rest streams)
-- 
2.11.0


[-- Attachment #3: Type: text/plain, Size: 494 bytes --]


The reason I didn't do this in thunk.el is that thunks are just bare
lambdas, so there is no way to get rid it "from the inside", as it were.
Whereas for streams, it's just a matter of list editing.

Some additional things that I thought of changing, but I didn't yet:

- stream--identifier vs just using '--stream-- directly, I don't see
  what's the benefit of indirection here.

- stream-make should use cons instead of list (or maybe a struct?).

- stream-empty should just be a constant.

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Michael Heerdegen]

Noam Postavsky <npostavs@gmail.com> writes:

> Coming back to this again.  I think I got lost in the weeds of the
> byte-code function objects before.  The core problem is that streams
> are not exactly encoded like the above, because even after forcing it
> you don't have just a plain SECOND-VALUE stored in the stream.  The
> original FUNCTION-TO-PRODUCE-MORE-REST-OF-LIST stays around and keeps
> referencing all the code and all the following elements.  So a
> possible solution is change the stream to get rid of the lambda part
> and just leave the computed value after it's forced.  With the
> following patch (stream-flush (stream-range 1 1000000)) succeeds:
>
> [Patch...]

Works for me, and it makes sense.  As a test case I recompiled
el-search.el (it uses streams for several things) with your patch
applied to stream.el, and it worked well.

> Some additional things that I thought of changing, but I didn't yet:

> - stream--identifier vs just using '--stream-- directly, I don't see
>   what's the benefit of indirection here.

A matter of taste I guess.

> - stream-make should use cons instead of list (or maybe a struct?).

I think cons would be ok.  Would a struct make things slower?

> - stream-empty should just be a constant.

Dunno if there are cases where this would be problematic, but I guess we
could do this as well.

Anyway, thanks for looking into this again, I like your solution.  Maybe
Nicolas can also chime in.


Michael.

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Michael Heerdegen]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> > [Patch...]
>
> Works for me, and it makes sense.  As a test case I recompiled
> el-search.el (it uses streams for several things) with your patch
> applied to stream.el, and it worked well.

@Nicolas: If you are short on time, can we just install this patch?
I've tested it for a while and it works well, and I'm quite sure it is
harmless (doesn't change any semantics apart from fixing the crashes).

> > - stream-make should use cons instead of list (or maybe a struct?).
>
> I think cons would be ok.  Would a struct make things slower?
>
> > - stream-empty should just be a constant.
>
> Dunno if there are cases where this would be problematic, but I guess we
> could do this as well.

@Nicolas: Do you want us to care about this or do you want to have a
look yourself?  I don't want to hurry, I just don't want this to be
forgotten.  If you say you have time in four months, it's still ok.


Thanks,

Michael.

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Noam Postavsky]

[-- Attachment #1: Type: text/plain, Size: 2457 bytes --]

tags 30626 + patch
quit

Michael Heerdegen <michael_heerdegen@web.de> writes:

>> > - stream-make should use cons instead of list (or maybe a struct?).
>>
>> I think cons would be ok.  Would a struct make things slower?

A struct might be slower, and cons has the advantage that the print
output is more readable for humans too.  E.g., with this code:

(let ((s (stream-range 1 5)))
  (stream-flush s)
  s)

;; Using cons (patch in this message):
(--stream-evald-- 1 --stream-evald-- 2 --stream-evald-- 3 --stream-evald-- 4 --stream-evald--)

;; Using list (previous patch):
(--stream-evald-- (1 --stream-evald-- (2 --stream-evald-- (3 --stream-evald-- (4 --stream-evald-- nil)))))

;; I guess using a struct would look something like this:
#(--stream-evald-- (1 . #(--stream-evald-- (2 . #(--stream-evald-- (3 . #(--stream-evald-- (4 . #(--stream-evald-- nil)))))))))

;; Using list with thunk (current, v2.2.4)
(--stream-- #[256 "\211\203\007\0\303\242\207\303\242\204 \0\304\300\242\305\300\242\302\242\\\301\302\242#B\240\210\303\306\240\210\304\242\207" [(1) 5 (1) (t) ((1 --stream-- #[256 "\211\203\007\0\303\242\207\303\242\204 \0\304\300\242\305\300\242\302\242\\\301\302\242#B\240\210\303\306\240\210\304\242\207" [(2) 5 (1) (t) ((2 --stream-- #[256 "\211\203\007\0\303\242\207\303\242\204 \0\304\300\242\305\300\242\302\242\\\301\302\242#B\240\210\303\306\240\210\304\242\207" [(3) 5 (1) (t) ((3 --stream-- #[256 "\211\203\007\0\303\242\207\303\242\204 \0\304\300\242\305\300\242\302\242\\\301\302\242#B\240\210\303\306\240\210\304\242\207" [(4) 5 (1) (t) ((4 --stream-- #[256 "\211\203\007\0\300\242\207\300\242\204\024\0\301\302\240\210\300\303\240\210\301\242\207" [(t) (nil) nil t] 3 "

(fn &optional CHECK)"])) stream-range t] 7 "

(fn &optional CHECK)"])) stream-range t] 7 "

(fn &optional CHECK)"])) stream-range t] 7 "

(fn &optional CHECK)"])) stream-range t] 7 "

(fn &optional CHECK)"])

>> > - stream-empty should just be a constant.
>>
>> Dunno if there are cases where this would be problematic, but I guess we
>> could do this as well.

I've done this in the patch below.  Passes all the tests, and I can't
see why it would be problematic.

> @Nicolas: Do you want us to care about this or do you want to have a
> look yourself?  I don't want to hurry, I just don't want this to be
> forgotten.  If you say you have time in four months, it's still ok.

Not getting any response; I'll wait another week for comments and then
push.


[-- Attachment #2: patch --]
[-- Type: text/plain, Size: 5752 bytes --]

From 7b126616c87bf034c933de711befcd80a7ada3bb Mon Sep 17 00:00:00 2001
From: Noam Postavsky <npostavs@users.sourceforge.net>
Date: Wed, 24 Apr 2019 22:51:19 -0400
Subject: [PATCH] Drop forced lambda's from stream (Bug#30626)

Let the stream id distinguish between forced and unforced stream
values.  When the value is forced, replace the lambda with its result.
This lets the lambda and anything it references be garbage collected.

Change the representation of a stream from (--stream-- THUNK)
to (--stream-fresh-- . (lambda () VALUE)) or (--stream-evald .
VALUE).
* packages/stream/stream.el (stream--identifier): Remove.
(stream--fresh-identifier, stream--evald-identifier): New constants to
replace it.
(streamp): Check for new constants.
(stream-make): Use cons and lambda instead of list and thunk-delay.
(stream--force): New function.
(stream-empty-p, stream-first, stream-rest): Use it.
(stream-empty): New constant, return it from the function instead of
creating a new one each time.
* packages/stream/tests/stream-tests.el (stream-to-list): Remove.
(stream-list-test): Use seq-into instead.
---
 packages/stream/stream.el             | 41 +++++++++++++++++++++++++----------
 packages/stream/tests/stream-tests.el | 12 ++--------
 2 files changed, 31 insertions(+), 22 deletions(-)

diff --git a/packages/stream/stream.el b/packages/stream/stream.el
index 3f6bc4b5b..9f73e8b86 100644
--- a/packages/stream/stream.el
+++ b/packages/stream/stream.el
@@ -1,6 +1,6 @@
 ;;; stream.el --- Implementation of streams  -*- lexical-binding: t -*-
 
-;; Copyright (C) 2016 Free Software Foundation, Inc.
+;; Copyright (C) 2016-2019 Free Software Foundation, Inc.
 
 ;; Author: Nicolas Petton <nicolas@petton.fr>
 ;; Keywords: stream, laziness, sequences
@@ -41,7 +41,7 @@
 ;; - ...
 ;;
 ;; All functions are prefixed with "stream-".
-;; All functions are tested in test/automated/stream-tests.el
+;; All functions are tested in tests/stream-tests.el
 ;;
 ;; Here is an example implementation of the Fibonacci numbers
 ;; implemented as in infinite stream:
@@ -65,18 +65,30 @@
 
 (eval-when-compile (require 'cl-lib))
 (require 'seq)
-(require 'thunk)
 
 (eval-and-compile
-  (defconst stream--identifier '--stream--
-    "Symbol internally used to identify streams."))
+  (defconst stream--fresh-identifier '--stream-fresh--
+    "Symbol internally used to streams whose head was not evaluated.")
+  (defconst stream--evald-identifier '--stream-evald--
+    "Symbol internally used to streams whose head was evaluated."))
 
 (defmacro stream-make (&rest body)
   "Return a stream built from BODY.
 BODY must return nil or a cons cell whose cdr is itself a
 stream."
   (declare (debug t))
-  `(list ',stream--identifier (thunk-delay ,@body)))
+  `(cons ',stream--fresh-identifier (lambda () ,@body)))
+
+(defun stream--force (stream)
+  "Evaluate and return the first cons cell of STREAM.
+That value is the one passed to `stream-make'."
+  (cond
+   ((eq (car-safe stream) stream--evald-identifier)
+    (cdr stream))
+   ((eq (car-safe stream) stream--fresh-identifier)
+    (setf (car stream) stream--evald-identifier)
+    (setf (cdr stream) (funcall (cdr stream))))
+   (t (signal 'wrong-type-argument (list 'streamp stream)))))
 
 (defmacro stream-cons (first rest)
   "Return a stream built from the cons of FIRST and REST.
@@ -159,24 +171,29 @@ (defun stream-range (&optional start end step)
 
 (defun streamp (stream)
   "Return non-nil if STREAM is a stream, nil otherwise."
-  (eq (car-safe stream) stream--identifier))
+  (let ((car (car-safe stream)))
+    (or (eq car stream--fresh-identifier)
+        (eq car stream--evald-identifier))))
+
+(defconst stream-empty (cons stream--evald-identifier nil)
+  "The empty stream.")
 
 (defun stream-empty ()
-  "Return a new empty stream."
-  (list stream--identifier (thunk-delay nil)))
+  "Return the empty stream."
+  stream-empty)
 
 (defun stream-empty-p (stream)
   "Return non-nil if STREAM is empty, nil otherwise."
-  (null (thunk-force (cadr stream))))
+  (null (cdr (stream--force stream))))
 
 (defun stream-first (stream)
   "Return the first element of STREAM.
 Return nil if STREAM is empty."
-  (car (thunk-force (cadr stream))))
+  (car (stream--force stream)))
 
 (defun stream-rest (stream)
   "Return a stream of all but the first element of STREAM."
-  (or (cdr (thunk-force (cadr stream)))
+  (or (cdr (stream--force stream))
       (stream-empty)))
 
 (defun stream-append (&rest streams)
diff --git a/packages/stream/tests/stream-tests.el b/packages/stream/tests/stream-tests.el
index 021ed65cf..7487ef69b 100644
--- a/packages/stream/tests/stream-tests.el
+++ b/packages/stream/tests/stream-tests.el
@@ -1,6 +1,6 @@
 ;;; stream-tests.el --- Unit tests for stream.el  -*- lexical-binding: t -*-
 
-;; Copyright (C) 2015, 2017 Free Software Foundation, Inc.
+;; Copyright (C) 2015, 2017-2019 Free Software Foundation, Inc.
 
 ;; Author: Nicolas Petton <nicolas@petton.fr>
 
@@ -29,14 +29,6 @@ (require 'stream)
 (require 'generator)
 (require 'cl-lib)
 
-(defun stream-to-list (stream)
-  "Eagerly traverse STREAM and return a list of its elements."
-  (let (result)
-    (seq-do (lambda (elt)
-                 (push elt result))
-               stream)
-    (reverse result)))
-
 (ert-deftest stream-empty-test ()
   (should (streamp (stream-empty)))
   (should (stream-empty-p (stream-empty))))
@@ -240,7 +232,7 @@ (ert-deftest stream-range-test ()
 
 (ert-deftest stream-list-test ()
   (dolist (list '(nil '(1 2 3) '(a . b)))
-    (should (equal list (stream-to-list (stream list))))))
+    (should (equal list (seq-into (stream list) 'list)))))
 
 (ert-deftest stream-seq-subseq-test ()
   (should (stream-empty-p (seq-subseq (stream-range 2 10) 0 0)))
-- 
2.11.0

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Michael Heerdegen]

Noam Postavsky <npostavs@gmail.com> writes:

> >> > - stream-make should use cons instead of list (or maybe a struct?).
> >>
> >> I think cons would be ok.  Would a struct make things slower?
>
> A struct might be slower, and cons has the advantage that the print
> output is more readable for humans too.  E.g., with this code: [...]

I see no reason not to switch to cons.  Better readable, and less cons
garbage than with `list'.

Would you like to do this?

> >> > - stream-empty should just be a constant.
> >>
> >> Dunno if there are cases where this would be problematic, but I
> >> guess we
> >> could do this as well.
>
> I've done this in the patch below.  Passes all the tests, and I can't
> see why it would be problematic.

Looks good to me.  Also passes my tests (which are all my stream.el
uses, including el-search.el).

> > @Nicolas: Do you want us to care about this or do you want to have a
> > look yourself?  I don't want to hurry, I just don't want this to be
> > forgotten.  If you say you have time in four months, it's still ok.
>
> Not getting any response; I'll wait another week for comments and then
> push.

Ok, let's do this.


Thanks,

Michael.

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Noam Postavsky]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> I see no reason not to switch to cons.  Better readable, and less cons
> garbage than with `list'.
>
> Would you like to do this?

Yes, just to be clear, the patch in my previous message did already
include the switch to cons.

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Michael Heerdegen]

Noam Postavsky <npostavs@gmail.com> writes:

> Yes, just to be clear, the patch in my previous message did already
> include the switch to cons.

Ah - of course - sorry, I was already a bit tired, I read it but wasn't
aware of it.  Sorry, I had spent the time finding a bug in your patch
that turned out was just my oversight of some file to recompile.

Michael.

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Noam Postavsky]

tags 30626 fixed
close 30626 
quit

Pushed to elpa.

b5f4061db 2019-06-03T20:23:03-04:00 "Drop forced lambda's from stream (Bug#30626)"
https://git.savannah.gnu.org/cgit/emacs/elpa.git/commit/?id=b5f4061db4226d1d49fcb0ff53db0424f359e344

> I had spent the time finding a bug in your patch that turned out was
> just my oversight of some file to recompile.

Oh, I hate it when that happens.

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Michael Heerdegen]

Hello,

I had written that

> > (seq-doseq (_ (stream-range 1 1000000)) nil)

crashes.  CC'ing Nicolas, the author of stream.el.

> What can we do instead in such cases?  Stack-overflow protection
> cannot work in GC, so you are shooting yourself in the foot by
> creating such large recursive structures.  By the time we get to GC,
> where the problem will happen, it's too late, because the memory was
> already allocated.
>
> Does anyone has a reasonable idea for avoiding the crash in such
> programs?

I would appreciate any effort to fix that, because it seems that
currently streams are broken by design, and there is no way to fix that
from the Lisp implementation.

Yes, we could implement iterators instead of streams - that's what we
get when we avoid the consing.  But it's something different and not
always an alternative, depending on what you want to do.


Michael.

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Nicolas Petton]

[-- Attachment #1: Type: text/plain, Size: 239 bytes --]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> Hello,

Hi,

> I had written that
>
>> > (seq-doseq (_ (stream-range 1 1000000)) nil)
>
> crashes.  CC'ing Nicolas, the author of stream.el.

Thanks, I'll look into it.

Cheers,
Nico

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Daniel Colascione]

On 02/27/2018 10:08 AM, Eli Zaretskii wrote:
>> From: Michael Heerdegen <michael_heerdegen@web.de>
>> Cc: bug-gnu-emacs@gnu.org,  30626@debbugs.gnu.org
>> Date: Tue, 27 Feb 2018 13:08:59 +0100
>>
>> #+begin_src emacs-lisp
>> (seq-doseq (_ (stream-range 1 1000000)) nil)
>> #+end_src
>>
>> Note that this is executed as a loop due how to streams are implemented,
>> although the definition of `seq-doseq' looks recursive.  But it seems
>> that gc has a problem with the large number of conses created when
>> processing that.
> 
> What can we do instead in such cases?  Stack-overflow protection
> cannot work in GC, so you are shooting yourself in the foot by
> creating such large recursive structures.  By the time we get to GC,
> where the problem will happen, it's too late, because the memory was
> already allocated.
> 
> Does anyone has a reasonable idea for avoiding the crash in such
> programs?

We need to fix GC being deeply recursive once and for all. Tweaking 
stack sizes on various platforms and trying to spot-fix GC for the 
occasional deeply recursive structure is annoying. Here's my proposal:

Turn garbage_collect_1 into a queue-draining loop, initializing the 
object queue with the GC roots before draining it. We'll make 
mark_object put an object on this queue, turning the existing 
mark_object code into a mark_queued_object function.

garbage_collect_1 will just call mark_queued_object in a loop; 
mark_queued_object can call mark_object, but since mark_object just 
enqueues an object and doesn't recurse, we can't exhaust the stack with 
deep object graphs. (We'll repurpose the mark bit to mean that the 
object is on the to-mark queue; by the time we fully drain the queue, 
just before we sweep, the mark bit will have the same meaning it does now.)

We can't allocate memory to hold the queue during GC, so we'll have to 
pre-allocate it. We can implement the queue as a list of queue blocks, 
where each queue block is an array of 16k or so Lisp_Objects. During 
allocation, we'll just make sure we have one Lisp_Object queue-block 
slot for every non-self-representing Lisp object we allocate.

Since we know that we'll have enough queue blocks for the worst GC case, 
we can have mark_object pull queue blocks from a free list, aborting if 
for some reason it ever runs out of queue blocks. (The previous 
paragraph guarantees we won't.) garbage_collect_1 will churn through 
these heap blocks and place each back on the free list after it's called 
mark_queued_object on every Lisp_Object in the queue block.

In this way, in non-pathological cases of GC, we'll end up using the 
same few queue blocks over and over. That's a nice optimization, because 
we can MADV_DONTNEED unused queue blocks so the OS doesn't actually have 
to remember their contents.

In this way, I think we can make the current GC model recursion-proof 
without drastically changing how we allocate Lisp objects. The 
additional memory requirements should be modest: it's basically one 
Lisp_Object per Lisp object allocated.

The naive version of this scheme needs about 4.6MB of overhead on my 
current 20MB Emacs heap, but it should be possible to reduce the 
overhead significantly by taking advantage of the block allocation we do 
for conses and other types --- we can put whole blocks on the queue 
instead of pointers to individual block parts, so we can get away with a 
much smaller queue. Under this approach, the reserved-queue-block scheme 
would impose an overhead of somewhere around 1MB on the same heap. This 
amount of overhead seems reasonable. We may end up actually using less 
memory that we would for recursive mark_object stack invocation.

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Noam Postavsky]

On Thu, Mar 1, 2018 at 5:44 AM, Daniel Colascione <dancol@dancol.org> wrote:

> We need to fix GC being deeply recursive once and for all. Tweaking stack
> sizes on various platforms and trying to spot-fix GC for the occasional
> deeply recursive structure is annoying.

Agreed, but could you please open a new thread for it? AFAICT, this
bug thread is about streams.el functions producing structures
proportional in size to the length of the entire stream, which is a
bug in itself, regardless of whether or not the GC can handle the
result.

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Michael Heerdegen]

Noam Postavsky <npostavs@gmail.com> writes:

> Agreed, but could you please open a new thread for it? AFAICT, this
> bug thread is about streams.el functions producing structures
> proportional in size to the length of the entire stream, which is a
> bug in itself, regardless of whether or not the GC can handle the
> result.

No, it is a feature: we want streams to produce these structures,
streams are delayed lists, not just iterators.  Lists are also
potentially unlimited nested conses, and that's not a bug.  Streams are
the same realized with delayed conses.


Michael.

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Noam Postavsky]

On Thu, Mar 1, 2018 at 11:54 AM, Michael Heerdegen
<michael_heerdegen@web.de> wrote:
> Noam Postavsky <npostavs@gmail.com> writes:
>
>> Agreed, but could you please open a new thread for it? AFAICT, this
>> bug thread is about streams.el functions producing structures
>> proportional in size to the length of the entire stream, which is a
>> bug in itself, regardless of whether or not the GC can handle the
>> result.
>
> No, it is a feature: we want streams to produce these structures,
> streams are delayed lists, not just iterators.  Lists are also
> potentially unlimited nested conses, and that's not a bug.  Streams are
> the same realized with delayed conses.

Maybe I've misunderstood, but is it not the case that iterating over
(stream-range 1 n) should require only a constant amount of memory,
regardless of the value of n?

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Michael Heerdegen]

Noam Postavsky <npostavs@gmail.com> writes:

> Maybe I've misunderstood, but is it not the case that iterating over
> (stream-range 1 n) should require only a constant amount of memory,
> regardless of the value of n?

Depends on your definition of `require'.  Like, for example,

  (dolist (i 1000000) (message "%S" (cons i (1+ i))))

each iteration step creates and discards a new cons (or a constant
number of conses).  Not a exceptional thing in Lisp.  When iterating
over (stream-range 1 n), the garbage collector seems to have problems
with how the garbage is structured.


Michael.

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Noam Postavsky]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> Noam Postavsky <npostavs@gmail.com> writes:
>
>> Maybe I've misunderstood, but is it not the case that iterating over
>> (stream-range 1 n) should require only a constant amount of memory,
>> regardless of the value of n?
>
> Depends on your definition of `require'.  Like, for example,
>
>   (dolist (i 1000000) (message "%S" (cons i (1+ i))))
>
> each iteration step creates and discards a new cons (or a constant
> number of conses).  Not a exceptional thing in Lisp.  When iterating
> over (stream-range 1 n), the garbage collector seems to have problems
> with how the garbage is structured.

Ah, so let me be more precise.  Iterating over (stream-range 1 n) should
require only a constant amount of *reachable* memory at any particular
instant.  So your example above is okay, but the following one would be
not acceptable (I mean, it's fine if some random lisp code does that,
but stream-range should not be creating such long lists, or equivalently
large structures):

    (let ((list nil))
      (dotimes (i 1000000)
        (push i list)
        (message "%S" (car list))))

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Michael Heerdegen]

Noam Postavsky <npostavs@gmail.com> writes:

> Ah, so let me be more precise.  Iterating over (stream-range 1 n) should
> require only a constant amount of *reachable* memory at any particular
> instant.  So your example above is okay, but the following one would be
> not acceptable (I mean, it's fine if some random lisp code does that,
> but stream-range should not be creating such long lists, or equivalently
> large structures):
>
>     (let ((list nil))
>       (dotimes (i 1000000)
>         (push i list)
>         (message "%S" (car list))))

Yes, absolutely agreed.

Using streams, you can logically refer to the complete list of elements
as one object, but the programmer must ensure that the referable list of
generated elements doesn't get too large.

Michael.

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Michael Heerdegen]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> Noam Postavsky <npostavs@gmail.com> writes:
>
> > Maybe I've misunderstood, but is it not the case that iterating over
> > (stream-range 1 n) should require only a constant amount of memory,
> > regardless of the value of n?

But in this regard, we have a problem with how lexical-binding is
implemented for interpreted code.  Nested thunks (as implemented in
"thunk.el") accumulate useless variable bindings - e.g.

(defun test ()
  (thunk-force
   (thunk-delay
    (thunk-force
     (thunk-delay
      (thunk-force
       (thunk-delay
        (lambda () 1))))))))
(test)
==>
#1=(closure
((check)
(#:val . #1#)
(#:forced . t)
(check)
(#:val . #1#)
(#:forced . t)
(check)
(#:val . #1#)
(#:forced . t)
t)
nil 1)

The length of the variable list is equivalent to the number of thunk
wrappers.  I believe that these useless variable lists are responsible
for the crashes of the uncompiled versions of the test files I had
posted.  I think this problem is different from the gc issue.

Streams use nested thunks.  Of course does thunk.el not explicitly add
such variable lists to the result - this is how closures are built in
interpreted code.  For nested thunks these just add up.

BTW, if you byte-compile the above `test' function, then

(disassemble (test))
==>
byte code:
  args: nil
0       constant  1
1       return

and this problem is gone.


Michael.

Let's make the GC safe and iterative (Was: Re: bug#30626)

[Daniel Colascione]

Noam mentioned that I should make a new thread for this proposal, so I'm 
posting an edited version of my original message.

tl;dr: we should be able to make the GC non-recursive with minimal 
overhead, solving the "Emacs crashed because we ran out of stack space 
in GC" problem once and for all.

On 02/27/2018 10:08 AM, Eli Zaretskii wrote:
> What can we do instead in such cases?  Stack-overflow protection
> cannot work in GC, so you are shooting yourself in the foot by
> creating such large recursive structures.  By the time we get to GC,
> where the problem will happen, it's too late, because the memory was
> already allocated.
> 
> Does anyone has a reasonable idea for avoiding the crash in such
> programs?

We need to fix GC being deeply recursive once and for all. Tweaking 
stack sizes on various platforms and trying to spot-fix GC for the 
occasional deeply recursive structure is annoying. Here's my proposal:

I. NAIVE APPROACH

Turn garbage_collect_1 into a queue-draining loop, initializing the 
object queue with the GC roots before draining it. We'll make 
mark_object put an object on this queue, turning the existing 
mark_object code into a mark_queued_object function.

garbage_collect_1 will just call mark_queued_object in a loop; 
mark_queued_object can call mark_object, but since mark_object just 
enqueues an object and doesn't recurse, we can't exhaust the stack with 
deep object graphs. (We'll repurpose the mark bit to mean that the 
object is on the to-mark queue; by the time we fully drain the queue, 
just before we sweep, the mark bit will have the same meaning it does now.)

We can't allocate memory to hold the queue during GC, so we'll have to 
pre-allocate it. We can implement the queue as a list of queue blocks, 
where each queue block is an array of 16k or so Lisp_Objects. During 
allocation, we'll just make sure we have one Lisp_Object queue-block 
slot for every non-self-representing Lisp object we allocate.

Since we know that we'll have enough queue blocks for the worst GC case, 
we can have mark_object pull queue blocks from a free list, aborting if 
for some reason it ever runs out of queue blocks. (The previous 
paragraph guarantees we won't.) garbage_collect_1 will churn through 
these heap blocks and place each back on the free list after it's called 
mark_queued_object on every Lisp_Object in the queue block.

In this way, in non-pathological cases of GC, we'll end up using the 
same few queue blocks over and over. That's a nice optimization, because 
we can MADV_DONTNEED unused queue blocks so the OS doesn't actually have 
to remember their contents.

In this way, I think we can make the current GC model recursion-proof 
without drastically changing how we allocate Lisp objects. The 
additional memory requirements should be modest: it's basically one 
Lisp_Object per Lisp object allocated.

II. ELABORATION

The naive version of this scheme needs about 4.6MB of overhead on my 
current 20MB Emacs heap, but it should be possible to reduce the 
overhead significantly by taking advantage of the block allocation we do 
for conses and other types --- we can put whole blocks on the queue 
instead of pointers to individual block parts, so we can get away with a 
much smaller queue.

It's also interesting to note that we don't need separate queue blocks 
to put a block on the queue, as we do if we want to enqueue individual 
Lisp_Object pointers. Instead, we can add to each block type a pointer 
to the next block *on the to-be-marked queue* and a bitmask yielding the 
positions within that block that we want to mark.

For example, cons_block right now looks like this:

struct cons_block
{
   /* Place `conses' at the beginning, to ease up CONS_INDEX's job.  */
   struct Lisp_Cons conses[CONS_BLOCK_SIZE];
   bits_word gcmarkbits[1 + CONS_BLOCK_SIZE / BITS_PER_BITS_WORD];
   struct cons_block *next;
};

We'd turn it into something like this:

struct cons_block
{
   /* Place `conses' at the beginning, to ease up CONS_INDEX's job.  */
   struct Lisp_Cons conses[CONS_BLOCK_SIZE];
   bits_word gcmarkbits[1 + CONS_BLOCK_SIZE / BITS_PER_BITS_WORD];
   bits_word scan_pending[1 + CONS_BLOCK_SIZE / BITS_PER_BITS_WORD];
   struct cons_block *next;
   struct cons_block *next_scan_pending;
};

When we call mark_object on a cons, we'll look up its cons_block and 
look up the cons in gcmarkbits. If we find the cons mark bit set, we're 
done. Otherwise, we look at the scan_pending bit for the cons cell. If 
_that's_ set, we're also done. If we find the scan_pending bit unset, 
however, we set it, and then look at next_scan_pending. If that's 
non-zero, we know the block as a whole is enqueued for scanning, and 
we're done. If *that's* zero, then we add the whole block to the 
to-be-scanned queue.

We'll modify garbage_collect_1 to drain both the Lisp_Object queue I 
described in the last section (which we still need for big objects like 
buffers) *and* the queue of blocks pending scanning. When we get a cons 
block, we'll scan all the conses with scan_pending bits set to one, set 
their gcmarkbits, and remove the cons block from the queue.

That same cons block might make it back onto the queue later if someone 
calls mark_object for one if its conses we didn't already scan, but 
that's okay. Scanning scan_pending should be very cheap, especially on 
modern CPUs with bit-prefix-scan instructions.

Under this approach, the reserved-queue-block scheme would impose an 
overhead of somewhere around 1MB on the same heap. (I think it'd 
actually be a bit smaller actually.) Conses, strings, and vectors are 
the overwhelming majority of heap-allocated objects, and thanks to block 
packing, we'd get bookkeeping for them for practically free. This amount 
of overhead seems reasonable. I think we may end up actually using less 
memory that we would for recursive mark_object stack invocation.

This scheme interacts well with the portable dumper too. pdumper already 
uses a big bit array to store mark bits; we'd just add another array for 
its scan_pending. We'd basically treat the entire pdumper region as one 
big cons_block for GC purposes.

What do you think? I think this approach solves a longstanding fiddly 
problem with Emacs GC without too much disruption to the internals. It 
also paves the way for concurrent or generational GC if we ever want to 
implement these features.

Re: Let's make the GC safe and iterative (Was: Re: bug#30626)

[Paul Eggert]

On 03/01/2018 03:22 PM, Daniel Colascione wrote:
> What do you think?

Thanks, I like the idea. I suggest implementing the naive version first. 
Although the more-elaborate versions saves virtual memory, it's not 
clear that it'd be an overall performance win in the typical case, so my 
guess is that it's better to start simple.

Re: Let's make the GC safe and iterative (Was: Re: bug#30626)

[Ken Raeburn]

On Mar 1, 2018, at 18:29, Paul Eggert <eggert@cs.ucla.edu> wrote:

> On 03/01/2018 03:22 PM, Daniel Colascione wrote:
>> What do you think?
> 
> Thanks, I like the idea. I suggest implementing the naive version first. Although the more-elaborate versions saves virtual memory, it's not clear that it'd be an overall performance win in the typical case, so my guess is that it's better to start simple.

The more-elaborate version looks like it’s more likely to win on locality of reference as it scans blocks sequentially. That could be significant… especially if anyone is so unfortunate as to still run Emacs on a system small enough that has to page out part of the Lisp memory.

I like the second approach a bit better, but either way, getting away from using the C stack sounds like a big improvement.

Ken

What improvements would be truly useful?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

An improvement in GC wouldn't be a bad thing, but it may not be worth
the effort.  It is likely to lead to many bugs that would be hard to
fix.  Once working, it would not make much difference to users.
It would only permit some operations on larger problems than now.

When I was working at the AI Lab, one of the older programmers told me
that hackers are often eager to make improvements of this sort: which
make the program better in an abstract sense, but not better for
users.  I took that advice to heart.  Now I pass it on.

Changing Emacs to handle indentation and alignment with
variable-width fonts would be an important and useful change.
Certain kinds of use would make sense, which currently don't.

It would be a big step towards making Emacs do the job of
a word processor, which is what I would like to see some day.
Imagine if you could edit nicely formatted documents directly
with Emacs, instead of using LibreOffice?  LibreOffice is
fine to use, it is free software, but it isn't Emacs.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: What improvements would be truly useful?

[John Yates]

Richard,

I too had an early mentor impart similar wisdom, possibly around the
same that time yours did.  Since then I have always tried to keep the
end user experience foremost in my work.

"Many bugs that would be hard to fix" is not a given.  I cannot count
the number times I have replaced recursion with iteration.  With
discipline (no other changes to the algorithm) it is a straight
forward conversion.

I agree that merely being able to handle large files is not a strong
justification.  OTOH, in spite of running on a state of the art
desktop with 64GB RAM, 4GHz+ multi-core processor and high-end gamer
graphics Emacs feels progressively less responsive.  Any work
countering that trend gets my vote.

/john

Re: What improvements would be truly useful?

[Paul Eggert]

On 03/05/2018 06:02 AM, John Yates wrote:
> in spite of running on a state of the art
> desktop with 64GB RAM, 4GHz+ multi-core processor and high-end gamer
> graphics Emacs feels progressively less responsive

I'm feeling that on my older work desktop too (8 GiB RAM, 2.6 GHz 
quad-core, circa-2010 AMD Deneb). However, I suspect it has mostly to do 
with things other than GC.

The main argument to improve GC is robustness, not typical-case 
performance. We're still installing last-minute hacks into Emacs 26 
because of real-world problems here. It would be useful if we didn't 
have to mess with these hacks because we fixed GC to not blow the C stack.

Re: What improvements would be truly useful?

[Stefan Monnier]

> An improvement in GC wouldn't be a bad thing, but it may not be worth
> the effort.

Making the recursion use an explicitly managed stack should be a fairly
simple change, which will fix some crashes.  I agree that it's probably
not worth trying to be too fancy, but it's really just a bug fix.


        Stefan

Re: What improvements would be truly useful?

[Rostislav Svoboda]

> It would be a big step towards making Emacs do the job of
> a word processor, which is what I would like to see some day.
> Imagine if you could edit nicely formatted documents directly
> with Emacs, instead of using LibreOffice?  LibreOffice is
> fine to use, it is free software, but it isn't Emacs.

IMO the future of GUI apps and in our case editors, lies in browser
based frameworks programmable by some lisp dialect. For the moment it
would be a stack based on electron, clojurescript, codemirror, etc.

I know it's a bitter pill to swallow, but let's face it - do we think,
our bellowed Emacs will ever be able to display anything like the
examples from https://threejs.org ?

Bost

Re: What improvements would be truly useful?

[Eli Zaretskii]

> From: Rostislav Svoboda <rostislav.svoboda@gmail.com>
> Date: Mon, 5 Mar 2018 18:32:06 +0100
> Cc: "emacs-devel@gnu.org Development" <emacs-devel@gnu.org>,
> 	Ken Raeburn <raeburn@raeburn.org>, dancol@dancol.org,
> 	Paul Eggert <eggert@cs.ucla.edu>
> 
> I know it's a bitter pill to swallow, but let's face it - do we think,
> our bellowed Emacs will ever be able to display anything like the
> examples from https://threejs.org ?

Can you display something similar in Office?

Re: What improvements would be truly useful?

[Daniel Colascione]

On 03/05/2018 10:04 AM, Eli Zaretskii wrote:
>> From: Rostislav Svoboda <rostislav.svoboda@gmail.com>
>> Date: Mon, 5 Mar 2018 18:32:06 +0100
>> Cc: "emacs-devel@gnu.org Development" <emacs-devel@gnu.org>,
>> 	Ken Raeburn <raeburn@raeburn.org>, dancol@dancol.org,
>> 	Paul Eggert <eggert@cs.ucla.edu>
>>
>> I know it's a bitter pill to swallow, but let's face it - do we think,
>> our bellowed Emacs will ever be able to display anything like the
>> examples from https://threejs.org ?

It can already, with XEmbed --- same way Office does it, with OLE. :-) 
It's a nice trick, but I don't think that sort of pretty display helps 
anyone accomplish a task.

Personally, I don't think word processing is a good focus for Emacs. 
There are two groups of people who want to prepare documents: those who 
want a WYSIWYG system and those who don't. The former group is 
well-served by LibreOffice, which is a free and powerful office suite. 
The latter group is well-served by Emacs with its extensive LaTeX 
integration.

Instead of focusing on areas where we're weak and will realistically 
never catch up with projects dedicated to the task, we should focus on 
improving existing strengths.

1) We should be the best editor around for text and program code. 
There's an opportunity to do much better than the mainstream. 
Conventional IDE groups put a ton of brute force effort into tuning IDEs 
for specific coding styles in specific environments. We can be more 
generic and more flexible, ultimately offering more power and greater 
efficiency for people willing to invest time into learning the system.

a) We should do a better job of integrating interesting ideas like 
undo-tree, ace-jump-mode, yasnippet, helm, and others into the core and 
enabling them by default. I don't think we need to be as conservative as 
we've been historically, and I think there's still a lot of room to 
improve the core editing mechanics.

b) There are long-standing defects that are minor in the scheme of 
things, but that tend to create a poor impression. In particular, 
long-line handling is a sore point, as is support for very large files. 
For long lines: I haven't sufficiently studied what the necessary 
redisplay hacks would look like.

For large files: by moving from a gap buffer to a rope representation 
for buffers, we can partially use memory-mapped backing storage, and 
even when we do need private, modifiable memory for editing, we can 
allocate only when we immediately need and not have to move the gap 
around through humongous amounts of main memory. Such a system would not 
only improve our support for humongous files, but would also make a 
32-bit Emacs capable of editing files larger than its address space.

c) We need a project system. There's been some good work in this area, 
but there's too much fragmentation, which hinders productive 
integration. For example, there's no default keybinding to jump, in C++, 
between an "implementation" and a "header" file, and that's because 
Emacs by default has no idea what either concept means and there are 
something like, what, a dozen(?) different ways to teach it the concept.

d) We need better fontification and indentation. We don't have good 
language coverage, and support for more obscure languages is sometimes 
spotty, limited to fontifying comments, strings, and keywords. Keeping 
up with language development is a constant struggle, and it's easy to 
introduce odd bugs, infloops, and so on in ad-hoc parsing code, 
especially when this code needs to be simultaneously fast, incremental, 
and error tolerant.

I'm now wondering whether the manual approach is wrong. We've been using 
it along with everyone else, but there might be better options these 
days. It's a somewhat radical idea: let's use a machine learning model 
to classify program tokens, then apply manual fontification and 
indentation rules to the resulting token classifications. We'd train the 
model by taking labeled program text (say, from Savannah or GitHub, run 
through a parser), then perturb the program text, rewarding the model 
for retaining token labels under various editing and truncation operations.

In this way, we'd learn an approximate model for understanding even 
damaged program text without having to manually write a lot of code. 
Tons and tons of stuff in cc-mode is heuristics for dealing with damaged 
program text, and I think we could learn this understanding instead. The 
system is equivalent in power to anything we could write by hand: LSTMs 
and other systems are Turing-complete. This way, to add support for a 
new language, you'd just feed Emacs examples. I imagine you might even 
be able to gently correct the system when it misunderstands and improve 
the overall accuracy.

But it's probably a crazy idea. :-)

2) Startup should be instant in all cases. Now that we have a portable 
dumper, we should automatically dump the results of user initialization 
and regenerate the dump when we detect that something's changed. This 
way, users perceive Emacs as a fast, modern system. I know that the 
daemon exists and that it's possible to optimize even a customized 
initialization so that it's fast even without hacks (I do), but users 
shouldn't have to go to the trouble of this kind of manual setup and 
tweaking

3) Mobile support. One of Emacs' strengths is its portability, and this 
portability comes with a much lower footprint than other approaches. 
Desktop and laptop sales have been declining for six years. There are 
lots of tools built on Emacs that would be useful (like gnus, org-mode, 
etc.), except that I can't use them on mobile, which means I end up 
choosing other tools entirely for these tasks.

There is no reason that Emacs couldn't be a good Android citizen. A good 
Java<->elisp bridge would let us transparently use various system APIs. 
While we would probably need mobile-specific GUI code (because the plain 
buffer interface wouldn't be suitable for most tasks, at least without 
mobile-desktop convergence), all the logic and back-end glue would work 
on mobile as well as it works anywhere else, greatly simplifying the 
task of building general-purpose tools like org-mode that really ought 
to work anywhere.

Re: What improvements would be truly useful?

[Eli Zaretskii]

> From: Daniel Colascione <dancol@dancol.org>
> Date: Mon, 5 Mar 2018 11:18:29 -0800
> Cc: eggert@cs.ucla.edu, raeburn@raeburn.org, rms@gnu.org, emacs-devel@gnu.org
> 
> For long lines: I haven't sufficiently studied what the necessary 
> redisplay hacks would look like.

I explained the issues several times, here and elsewhere.  I can
explain again, if someone's interested.

Re: What improvements would be truly useful?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Personally, I don't think word processing is a good focus for Emacs. 

I want to do my word processing in Emacs,
so please stop interfering.  You, personally, can work on other
areas if that area doesn't interest you.

The improvements you suggested for editing programs are also
desirable.  We want to improve what Emacs can do in editing programs.
One of the improvements we need is to have a GNU language server and
connect it to Emacs.  The best proposal for how to make a GNU language
server is to do it by building on GDB.

However, Emacs is also meant for editing textual documents.
To make that easier, we need more support for editing and
saving formatted documents with various fonts.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: What improvements would be truly useful?

[dancol]

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > Personally, I don't think word processing is a good focus for Emacs.
>
> I want to do my word processing in Emacs,
> so please stop interfering.  You, personally, can work on other
> areas if that area doesn't interest you.

You are advocating for your suggested improvements. I am advocating for
mine. Neither of us "interfering" with the other.

Re: What improvements would be truly useful?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > You are advocating for your suggested improvements. I am advocating for
  > mine. 

Emacs is not an independent project, and it isn't governed by its
contributors.  I'm the head of the GNU Project, and that includes
Emacs.

You, as a contributor, can advocate a certain decision, which means
you present arguments why I should approve it.  I don't need to
advocate a decision in that sense.

I leave most technical decisions up to the contributors, including
you, because for most of the questions I don't have any special
preference of my own.  For those questions, I'm happy with whatever
works, and I know the contributors can figure out what works.

However, making progress on Emacs as a word processor is one of my
specific goals.  This is what Emacs needs to do to be useful in all
the ways it should be useful.

It will only take a few more features to make Emacs start to be useful
as a word processor.  Once we can do proper formatting of paragraphs
with variable-width text, with a few kinds of alignment, and we can
save these in files, we will be able to use it for writing letters and
handouts, instead of LibreOffice.

I am really looking forward to this.  LibreOffice is free software,
and it's ethical, but it isn't Emacs.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: What improvements would be truly useful?

[Daniel Colascione]

On 03/06/2018 12:54 PM, Richard Stallman wrote:
> However, making progress on Emacs as a word processor is one of my
> specific goals.  This is what Emacs needs to do to be useful in all
> the ways it should be useful.

If you want to add word processing features to Emacs, I suggest 
implementing the necessary changes features yourself instead of 
suggesting that discussion of other priorities is somehow "interfering".

Code wins arguments. `eval' tends not to work very well when you pass it 
a block of prose about how Emacs should be a word processor.

> It will only take a few more features to make Emacs start to be useful
> as a word processor.  Once we can do proper formatting of paragraphs
> with variable-width text, with a few kinds of alignment, and we can
> save these in files, we will be able to use it for writing letters and
> handouts, instead of LibreOffice.

Sure. It'd be nice if Emacs were suitable for that task. I don't see the 
urgency, considering that plenty of equally free packages exist.

Re: What improvements would be truly useful?

[Stefan Monnier]

> I am really looking forward to this.  LibreOffice is free software,
> and it's ethical, but it isn't Emacs.

IMO the essence of WYSIWYG makes it "not Emacs".  I'd like to be able to
usefully edit .odt documents with Emacs, but not in a WYSIWYG fashion.


        Stefan

Re: What improvements would be truly useful?

[Jefferson Carpenter]

On 3/7/2018 10:45 PM, Stefan Monnier wrote:
>> I am really looking forward to this.  LibreOffice is free software,
>> and it's ethical, but it isn't Emacs.
> 
> IMO the essence of WYSIWYG makes it "not Emacs".  I'd like to be able to
> usefully edit .odt documents with Emacs, but not in a WYSIWYG fashion.
> 
> 
>          Stefan
> 
> 

Agreed.  I like latex-preview-pane's format of "WYSIWYG" editing.  You 
edit the code in one buffer, and see a rendered preview in another 
buffer.  It is not critical that the preview even be an emacs buffer - 
it could be another application.

Jefferson

Re: What improvements would be truly useful?

[Yuri Khan]

On Thu, Mar 8, 2018 at 11:45 AM, Stefan Monnier
<monnier@iro.umontreal.ca> wrote:
> IMO the essence of WYSIWYG makes it "not Emacs".  I'd like to be able to
> usefully edit .odt documents with Emacs, but not in a WYSIWYG fashion.

I was about to say “you sort-of can, after all, .odt is just XML in
zip”. Then I did an (add-to-list 'auto-mode-alist '("\\.odt\\'" .
archive-mode)) and opened the first .odt I could find, and oh
$(DEITY), it’s horrible.

After the first line of xml declaration, the rest of the document is
one single physical line.

Re: What improvements would be truly useful?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Wed, 07 Mar 2018 23:45:05 -0500
> 
> > I am really looking forward to this.  LibreOffice is free software,
> > and it's ethical, but it isn't Emacs.
> 
> IMO the essence of WYSIWYG makes it "not Emacs".  I'd like to be able to
> usefully edit .odt documents with Emacs, but not in a WYSIWYG fashion.

Emacs being Emacs, no one is forced to use any specific Emacs feature
they don't like or want.

Re: What improvements would be truly useful?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > IMO the essence of WYSIWYG makes it "not Emacs".

That is shocking but enigmatic; I don't get any point.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: What improvements would be truly useful?

[Marcin Borkowski]

On 2018-03-08, at 05:45, Stefan Monnier <monnier@iro.umontreal.ca> wrote:

>> I am really looking forward to this.  LibreOffice is free software,
>> and it's ethical, but it isn't Emacs.
>
> IMO the essence of WYSIWYG makes it "not Emacs".  I'd like to be able to
> usefully edit .odt documents with Emacs, but not in a WYSIWYG fashion.

Interesting.  I have a very similar sentiment, though I think it is
quite irrational.

Is mu4e, Org-mode, Calc, or dunnet also "not Emacs"?  If not, how is it
different than a word processor?

I'm not mocking you, I just want to understand the sentiment we both
share.

Best,

-- 
Marcin Borkowski
http://mbork.pl

Re: What improvements would be truly useful?

[Stefan Monnier]

> Interesting.  I have a very similar sentiment, though I think it is
> quite irrational.
> Is mu4e, Org-mode, Calc, or dunnet also "not Emacs"?  If not, how is it
> different than a word processor?

I don't think I can clearly pin-point the problem, but I think the issue
has to do with hiding information: WYSIWYG hides information, in the
sense that there are many different documents with the same visual
appearance, so the user needs to *guess* what is the underlying
document's structure in order to know what his actions will really do.
A typical example is the beginning/end of a bold text which is only
implicitly represented by the fact that some text is bold and other
isn't, but that doesn't let you easily tell whether a new word inserted
between a bold work and a non-bold word will be bold or not.

Of course Emacs can also hide information (as text-properties, as
invisible text, as data stored in buffer-local variables, ...) but most
packages follow a design where as little info as possible is hidden.
Indeed, whenever I hide such information, I think it over many times
because I know there's a very strong chance that users won't like it.

Also when it comes to data that we edit, hiding information either means
hiding it as invisible text (which tends to be fiddly, so it's used
fairly sparingly), or it implies using a complex procedure to `find` and
to `save` the buffer in order to translate between the file contents and
the in-buffer contents.  This complexity is a strong encouragement not
to go down that route.


        Stefan

Re: What improvements would be truly useful?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Of course Emacs can also hide information (as text-properties, as
  > invisible text, as data stored in buffer-local variables, ...) but most
  > packages follow a design where as little info as possible is hidden.
  > Indeed, whenever I hide such information, I think it over many times
  > because I know there's a very strong chance that users won't like it.

That's a valid point, but WYSIWYG text processing is so desirable
that we shouldn't rule it out because of that point.

Perhaps Emacs WYSIWYG should have a mode to show the markup and a mode
to hide the markup, with a character to toggle between them.

Where and how to store the markup data is an independent question.
It could be in invisible characters, or in text properties.
Or some combination of the two.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: What improvements would be truly useful?

[Karl Fogel]

Richard Stallman <rms@gnu.org> writes:
>Perhaps Emacs WYSIWYG should have a mode to show the markup and a mode
>to hide the markup, with a character to toggle between them.

For what it's worth, the venerable proprietary word processor WordPerfect had this feature (under the name "reveal codes").  The feature was widely used, too.  At least, people I knew who normally did not become experts in their software tools nevertheless still still knew about and used "reveal codes" on a regular basis.

Re: What improvements would be truly useful?

[Marcin Borkowski]

On 2018-03-09, at 05:43, Stefan Monnier <monnier@IRO.UMontreal.CA> wrote:

>> Interesting.  I have a very similar sentiment, though I think it is
>> quite irrational.
>> Is mu4e, Org-mode, Calc, or dunnet also "not Emacs"?  If not, how is it
>> different than a word processor?
>
> I don't think I can clearly pin-point the problem, but I think the issue
> has to do with hiding information: WYSIWYG hides information, in the
> sense that there are many different documents with the same visual
> appearance, so the user needs to *guess* what is the underlying
> document's structure in order to know what his actions will really do.
> A typical example is the beginning/end of a bold text which is only
> implicitly represented by the fact that some text is bold and other
> isn't, but that doesn't let you easily tell whether a new word inserted
> between a bold work and a non-bold word will be bold or not.
>
> Of course Emacs can also hide information (as text-properties, as
> invisible text, as data stored in buffer-local variables, ...) but most
> packages follow a design where as little info as possible is hidden.
> Indeed, whenever I hide such information, I think it over many times
> because I know there's a very strong chance that users won't like it.
>
> Also when it comes to data that we edit, hiding information either means
> hiding it as invisible text (which tends to be fiddly, so it's used
> fairly sparingly), or it implies using a complex procedure to `find` and
> to `save` the buffer in order to translate between the file contents and
> the in-buffer contents.  This complexity is a strong encouragement not
> to go down that route.

Thanks, Stefan - this is a very good point.  I guess you managed to nail
it.

Best,

--
Marcin Borkowski
http://mbork.pl

Re: What improvements would be truly useful?

[Marcin Borkowski]

On 2018-03-06, at 00:05, Richard Stallman <rms@gnu.org> wrote:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > Personally, I don't think word processing is a good focus for Emacs.
>
> I want to do my word processing in Emacs,

You already can.  Quite a few people do this, using Org-mode or AUCTeX.

Let me quote the paper about Emacs for TUGboat I mentioned some time
ago.  I wrote there this:

--8<---------------cut here---------------start------------->8---
Emacs, however, is used by many people to edit texts in human languages
(as opposed to computer programs), and has really good support for that
task.  For instance, while many editors have support for movement by
words, Emacs has also movement by sentences.  Another feature which is
a real time-saver is the series of transpose commands: for instance,
transpose-chars swaps the two characters on both sides of the point.
There is a whole chapter in the Emacs manual describing commands for
dealing with text in human languages.
--8<---------------cut here---------------end--------------->8---

One thing that might be nice wrt "word processing" would be making
yanking insert spaces at the right places.  If you have " ipsum" in your
kill ring, and press C-y in this context: "Lorem |dolor sit amet" (where
the bar means the point), you end up with "Lorem  ipsumdolor sit amet"
instead of "Lorem ipsum dolor sit amet".  Obvious, but annoying.  (I
even wrote some code to fix that and I've been using it for a few weeks
now, and I didn't notice any problems.)

This is the power of Emacs: the ability to write small extensions,
helping users with their needs.  I think making that easier (e.g., by
providing better documentation) is one of the best things to improve
Emacs.

Don't get me wrong, Emacs has excellent docs.  But if you want to start
extending it, you are pretty much limited to Robert Chassell's book,
which is great, but way too short and elementary.  (Disclaimer: I am
working on a "next step" Elisp book.)

Best,

--
Marcin Borkowski
http://mbork.pl

Re: What improvements would be truly useful?

[Eli Zaretskii]

> From: Marcin Borkowski <mbork@mbork.pl>
> Cc: Daniel Colascione <dancol@dancol.org>, eggert@cs.ucla.edu, eliz@gnu.org, raeburn@raeburn.org, rostislav.svoboda@gmail.com, emacs-devel@gnu.org
> Date: Tue, 06 Mar 2018 21:01:00 +0100
> 
> > I want to do my word processing in Emacs,
> 
> You already can.  Quite a few people do this, using Org-mode or AUCTeX.

Are you saying that either of these is a WYSIWYG word processor?

Re: What improvements would be truly useful?

[Marcin Borkowski]

On 2018-03-06, at 21:32, Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Marcin Borkowski <mbork@mbork.pl>
>> Cc: Daniel Colascione <dancol@dancol.org>, eggert@cs.ucla.edu, eliz@gnu.org, raeburn@raeburn.org, rostislav.svoboda@gmail.com, emacs-devel@gnu.org
>> Date: Tue, 06 Mar 2018 21:01:00 +0100
>> 
>> > I want to do my word processing in Emacs,
>> 
>> You already can.  Quite a few people do this, using Org-mode or AUCTeX.
>
> Are you saying that either of these is a WYSIWYG word processor?

Did Richard mention WYSIWYG word processing or just word processing?

Also, AUCTeX with pdf-tools is quite close.  On a fast computer, you can
run TeX-command-master and TeX-view on an idle timer.  Not the same, but
close.

Best,

-- 
Marcin Borkowski
http://mbork.pl

Re: What improvements would be truly useful?

[Eli Zaretskii]

> From: Marcin Borkowski <mbork@mbork.pl>
> Cc: rms@gnu.org, dancol@dancol.org, eggert@cs.ucla.edu, raeburn@raeburn.org, rostislav.svoboda@gmail.com, emacs-devel@gnu.org
> Date: Tue, 06 Mar 2018 21:41:00 +0100
> 
> 
> >> > I want to do my word processing in Emacs,
> >> 
> >> You already can.  Quite a few people do this, using Org-mode or AUCTeX.
> >
> > Are you saying that either of these is a WYSIWYG word processor?
> 
> Did Richard mention WYSIWYG word processing or just word processing?

WYSIWYG, of course.  Otherwise it would make no sense to talk about
indentation with variable-pitch fonts and pixel-level justification.

Re: What improvements would be truly useful?

[Marcin Borkowski]

On 2018-03-07, at 18:13, Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Marcin Borkowski <mbork@mbork.pl>
>> Cc: rms@gnu.org, dancol@dancol.org, eggert@cs.ucla.edu, raeburn@raeburn.org, rostislav.svoboda@gmail.com, emacs-devel@gnu.org
>> Date: Tue, 06 Mar 2018 21:41:00 +0100
>> 
>> 
>> >> > I want to do my word processing in Emacs,
>> >> 
>> >> You already can.  Quite a few people do this, using Org-mode or AUCTeX.
>> >
>> > Are you saying that either of these is a WYSIWYG word processor?
>> 
>> Did Richard mention WYSIWYG word processing or just word processing?
>
> WYSIWYG, of course.  Otherwise it would make no sense to talk about
> indentation with variable-pitch fonts and pixel-level justification.

Have you ever seen LyX?

-- 
Marcin Borkowski
http://mbork.pl

Re: What improvements would be truly useful?

[Eli Zaretskii]

> From: Marcin Borkowski <mbork@mbork.pl>
> Cc: rms@gnu.org, dancol@dancol.org, eggert@cs.ucla.edu, raeburn@raeburn.org, rostislav.svoboda@gmail.com, emacs-devel@gnu.org
> Date: Wed, 07 Mar 2018 19:30:07 +0100
> 
> >> > Are you saying that either of these is a WYSIWYG word processor?
> >> 
> >> Did Richard mention WYSIWYG word processing or just word processing?
> >
> > WYSIWYG, of course.  Otherwise it would make no sense to talk about
> > indentation with variable-pitch fonts and pixel-level justification.
> 
> Have you ever seen LyX?

What would it add to the current discussion?

Re: What improvements would be truly useful?

[Marcin Borkowski]

On 2018-03-07, at 20:36, Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Marcin Borkowski <mbork@mbork.pl>
>> Cc: rms@gnu.org, dancol@dancol.org, eggert@cs.ucla.edu, raeburn@raeburn.org, rostislav.svoboda@gmail.com, emacs-devel@gnu.org
>> Date: Wed, 07 Mar 2018 19:30:07 +0100
>>
>> >> > Are you saying that either of these is a WYSIWYG word processor?
>> >>
>> >> Did Richard mention WYSIWYG word processing or just word processing?
>> >
>> > WYSIWYG, of course.  Otherwise it would make no sense to talk about
>> > indentation with variable-pitch fonts and pixel-level justification.
>>
>> Have you ever seen LyX?
>
> What would it add to the current discussion?

Next to nothing, except that variable-pitch fonts and pixel-level
justification do not imply WYSIWYG.

Personally, I don't care for the idea of having a word processor on top
of Emacs, although I admit that it might be interesting.  I think the
resources to do that could be better spent elsewhere.  But if someone
wants to do it - I have nothing against it.  I might even start to use
such an application.

Best,

--
Marcin Borkowski
http://mbork.pl

Re: What improvements would be truly useful?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Did Richard mention WYSIWYG word processing or just word processing?

  > WYSIWYG, of course.  Otherwise it would make no sense to talk about
  > indentation with variable-pitch fonts and pixel-level justification.

I didn't explicitly say WYSIWYG, because I took that for granted
because the other word processors I think of are WYSIWYG.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: What improvements would be truly useful?

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Marcin Borkowski <mbork@mbork.pl> Cc: Daniel Colascione
>> <dancol@dancol.org>, eggert@cs.ucla.edu, eliz@gnu.org,
>> raeburn@raeburn.org, rostislav.svoboda@gmail.com, emacs-devel@gnu.org
>> Date: Tue, 06 Mar 2018 21:01:00 +0100
>> 
>> > I want to do my word processing in Emacs,
>> 
>> You already can.  Quite a few people do this, using Org-mode or AUCTeX.
>
> Are you saying that either of these is a WYSIWYG word processor?

No. But, then Word or LibreOffice are not WYSIWYG either. Both have
extensive modal views of the underlying document (web layout, reading
mode, outline view).

Org-mode is essentially a structure first view of text. AUCTeX with
preview-latex is roughly the same thing. So, both of them have strong
aspects of modern word-processor.

If Emacs were able to turn org-mode into HTML, render that HTML and
then make changes to the text of that HTML, then you'd have something
close to the modal views. Or something like preview-latex which uses
syntax highlighting to provide the WYS aspect but allows you to turn it
on and off.

How easy this would be to implement, I do not know. Emacs does not have
an MVC architecture -- text properties (i.e. visualization) are stored
directly with the contents of a buffer, so switching views is clunky or
difficult (see my own lentic.el for an example).

Phil

Re: What improvements would be truly useful?

[Eli Zaretskii]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Cc: Marcin Borkowski <mbork@mbork.pl>,  eggert@cs.ucla.edu,  rms@gnu.org,  emacs-devel@gnu.org,  raeburn@raeburn.org,  dancol@dancol.org,  rostislav.svoboda@gmail.com
> Date: Fri, 09 Mar 2018 10:56:00 +0000
> 
> > Are you saying that either of these is a WYSIWYG word processor?
> 
> No. But, then Word or LibreOffice are not WYSIWYG either.

This is just a minor semantic issue: let's assume for the purposes of
the current discussion that the definition of a WYSIWYG word processor
is what the Office word processors do.

> Org-mode is essentially a structure first view of text. AUCTeX with
> preview-latex is roughly the same thing. So, both of them have strong
> aspects of modern word-processor.

They both provide the final view of the document only off-line, and
that is the crucial difference, for the purposes of this discussion.

> If Emacs were able to turn org-mode into HTML, render that HTML and
> then make changes to the text of that HTML, then you'd have something
> close to the modal views. Or something like preview-latex which uses
> syntax highlighting to provide the WYS aspect but allows you to turn it
> on and off.

Emacs is capable of displaying text with different typefaces, so it
isn't clear to me why would we need to go through a translator, such
as Org export or LaTeX.

> How easy this would be to implement, I do not know. Emacs does not have
> an MVC architecture -- text properties (i.e. visualization) are stored
> directly with the contents of a buffer, so switching views is clunky or
> difficult (see my own lentic.el for an example).

Actually, MVC is exactly the Emacs architecture.  (And text properties
are stored separately from the buffer contents.)  I wonder what kind
of misunderstanding is at work here.

Re: What improvements would be truly useful?

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: phillip.lord@russet.org.uk (Phillip Lord)
>> Cc: Marcin Borkowski <mbork@mbork.pl>, eggert@cs.ucla.edu,
>> rms@gnu.org, emacs-devel@gnu.org, raeburn@raeburn.org,
>> dancol@dancol.org, rostislav.svoboda@gmail.com
>> Date: Fri, 09 Mar 2018 10:56:00 +0000
>> 
>> > Are you saying that either of these is a WYSIWYG word processor?
>> 
>> No. But, then Word or LibreOffice are not WYSIWYG either.
>
> This is just a minor semantic issue: let's assume for the purposes of
> the current discussion that the definition of a WYSIWYG word processor
> is what the Office word processors do.


I think it is not. There are many different forms of word-processor out
there. Emacs is already one of them, albeit for a specialist user base.


>> Org-mode is essentially a structure first view of text. AUCTeX with
>> preview-latex is roughly the same thing. So, both of them have strong
>> aspects of modern word-processor.
>
> They both provide the final view of the document only off-line, and
> that is the crucial difference, for the purposes of this discussion.

Yes, this is most true, although the idea of the "final view" is not
quite as clear as it might be. Even for latex, the end product can be
more than one thing (PDF and HTML for example). For org-mode, whether
the text view is less final than an HTML rendering is an open question.
 

>> If Emacs were able to turn org-mode into HTML, render that HTML and
>> then make changes to the text of that HTML, then you'd have something
>> close to the modal views. Or something like preview-latex which uses
>> syntax highlighting to provide the WYS aspect but allows you to turn it
>> on and off.
>
> Emacs is capable of displaying text with different typefaces, so it
> isn't clear to me why would we need to go through a translator, such
> as Org export or LaTeX.

Oh, because of your notion of WYS. If the "final document" is supposed
to be in HTML, then having something that looks the similar, but is
never ideal.

>> How easy this would be to implement, I do not know. Emacs does not have
>> an MVC architecture -- text properties (i.e. visualization) are stored
>> directly with the contents of a buffer, so switching views is clunky or
>> difficult (see my own lentic.el for an example).
>
> Actually, MVC is exactly the Emacs architecture.  (And text properties
> are stored separately from the buffer contents.)  I wonder what kind
> of misunderstanding is at work here.

My understanding of, for indirect buffers, are that they share text and
text properties. So you cannot put a different visualisation over two
indirect buffers, if those visualisations use text properties, because
they will interfere with each other. Perhaps it would be possible to
recode indirect buffers so that this were not the case, so that only
text were shared.

Similarly, the text that you seen on screen is pretty much what is in
the buffer. You can do things like put before or after strings in text
properties. But then this will be ignored by everything other than the
visualisation -- you can't search for them, for instance, because the
user level functions operate over the text not the visualisation of it.

And, finally, while Emacs does provide notifications of change to buffer
text both before and after, they are neither guaranteed to be paired nor
consistent. Nor is it apparent to the listening function when they will
not be paired or consistent; which is unfortunate as mostly they are.

Of course, this is not to say there is no abillity to visualise the same
thing twice. You can put a buffer into two windows, with independent
mark and point, selection and so forth. But the existence of these
things is ephemeral. So, for example, when viewing a single buffer,
split the window. Now move point in one window, and then unsplit the
window. The location of mark and point in the second window has now been
lost.

So Emacs does not exactly have an MVC architecture -- it sort of has it.
I do not think, for example, it would be straight-forward to produce
something like preview-latex where you could see the inline images and
equations and latex representation simultaneously and be able to edit
either.


Phil

Re: What improvements would be truly useful?

[Eli Zaretskii]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Cc: mbork@mbork.pl,  eggert@cs.ucla.edu,  rms@gnu.org,  emacs-devel@gnu.org,  raeburn@raeburn.org,  dancol@dancol.org,  rostislav.svoboda@gmail.com
> Date: Sat, 10 Mar 2018 16:02:14 +0000
> 
> >> No. But, then Word or LibreOffice are not WYSIWYG either.
> >
> > This is just a minor semantic issue: let's assume for the purposes of
> > the current discussion that the definition of a WYSIWYG word processor
> > is what the Office word processors do.
> 
> I think it is not.

Like I said, it's a matter of definition.  And I just provided mine
(and I think also Richard's).

> My understanding of, for indirect buffers, are that they share text and
> text properties. So you cannot put a different visualisation over two
> indirect buffers, if those visualisations use text properties, because
> they will interfere with each other.

It is not possible with text properties, but it's possible with
overlays, which can be window-specific.

> Similarly, the text that you seen on screen is pretty much what is in
> the buffer. You can do things like put before or after strings in text
> properties. But then this will be ignored by everything other than the
> visualisation -- you can't search for them, for instance, because the
> user level functions operate over the text not the visualisation of it.

Since we mostly are talking indentations and typefaces, I don't
understand why would we need to use before- or after-strings in this
context.

> And, finally, while Emacs does provide notifications of change to buffer
> text both before and after, they are neither guaranteed to be paired nor
> consistent. Nor is it apparent to the listening function when they will
> not be paired or consistent; which is unfortunate as mostly they are.

Why is that part of the issue?

> So Emacs does not exactly have an MVC architecture -- it sort of has it.

I think your notion of MVC is too specialized.

Re: What improvements would be truly useful?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > If Emacs were able to turn org-mode into HTML, render that HTML and
  > then make changes to the text of that HTML, then you'd have something
  > close to the modal views.

I don't understand that sentence, but what I have in mind is something that
would look and work roughly like LibreOffice.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: What improvements would be truly useful?

[Dmitry Gutov]

On 3/5/18 9:18 PM, Daniel Colascione wrote:

>>> I know it's a bitter pill to swallow, but let's face it - do we think,
>>> our bellowed Emacs will ever be able to display anything like the
>>> examples from https://threejs.org ?
> 
> It can already, with XEmbed --- same way Office does it, with OLE. :-) 
> It's a nice trick, but I don't think that sort of pretty display helps 
> anyone accomplish a task.

Using the built-in SVG support (a tad more practical feature, IMO), 
someone can come pretty close as well.

> a) We should do a better job of integrating interesting ideas like 
> undo-tree, ace-jump-mode, yasnippet, helm, and others into the core and 
> enabling them by default. I don't think we need to be as conservative as 
> we've been historically, and I think there's still a lot of room to 
> improve the core editing mechanics.

I'm all for undo-tree, but as for the rest, including a particular 
workflow by default might make it harder to use competing ones (i.e. 
it's a hard pass for me on Helm).

> b) There are long-standing defects that are minor in the scheme of 
> things, but that tend to create a poor impression.

Long lines are annoying. Large files aren't hugely important, IMO 
(though I've had to deal with a couple over the recent months).

> c) We need a project system.

project.el? It sorely lacks in contributions (ideas and even rough 
prototypes are welcome).

> There's been some good work in this area, 
> but there's too much fragmentation, which hinders productive 
> integration. For example, there's no default keybinding to jump, in C++, 
> between an "implementation" and a "header" file, and that's because 
> Emacs by default has no idea what either concept means and there are 
> something like, what, a dozen(?) different ways to teach it the concept.

One difficulty with that is C++ is just one type of project, so adding 
this kind of command to all projects seems kind of iffy. And yes, 
arguments for picking between dozens different ways to approach this is 
the kind of contribution I'm looking for.

> d) We need better fontification and indentation. We don't have good 
> language coverage, and support for more obscure languages is sometimes 
> spotty, limited to fontifying comments, strings, and keywords. Keeping 
> up with language development is a constant struggle, and it's easy to 
> introduce odd bugs, infloops, and so on in ad-hoc parsing code, 
> especially when this code needs to be simultaneously fast, incremental, 
> and error tolerant.
> 
> I'm now wondering whether the manual approach is wrong. We've been using 
> it along with everyone else, but there might be better options these 
> days. It's a somewhat radical idea: let's use a machine learning model 
> to classify program tokens, then apply manual fontification and 
> indentation rules to the resulting token classifications. We'd train the 
> model by taking labeled program text (say, from Savannah or GitHub, run 
> through a parser), then perturb the program text, rewarding the model 
> for retaining token labels under various editing and truncation operations.

Err, we can start with fontification based on language specification 
like Atom/VS Code and Sublime done. They use JSON, but we don't have to.

In a way, those specs are harder to read than font-lock code, though.
> 3) Mobile support. One of Emacs' strengths is its portability, and this 
> portability comes with a much lower footprint than other approaches. 
> Desktop and laptop sales have been declining for six years. There are 
> lots of tools built on Emacs that would be useful (like gnus, org-mode, 
> etc.), except that I can't use them on mobile, which means I end up 
> choosing other tools entirely for these tasks.
> 
> There is no reason that Emacs couldn't be a good Android citizen. A good 
> Java<->elisp bridge would let us transparently use various system APIs. 
> While we would probably need mobile-specific GUI code (because the plain 
> buffer interface wouldn't be suitable for most tasks, at least without 
> mobile-desktop convergence), all the logic and back-end glue would work 
> on mobile as well as it works anywhere else, greatly simplifying the 
> task of building general-purpose tools like org-mode that really ought 
> to work anywhere.

I've been wondering about having Emacs on my phone for a while. It would 
really need a fitting UI added, though.

Re: What improvements would be truly useful?

[Aaron Ecay]

2018ko martxoak 5an, Rostislav Svoboda-ek idatzi zuen:
> 

[...]

> I know it's a bitter pill to swallow, but let's face it - do we think,
> our bellowed Emacs will ever be able to display anything like the
> examples from https://threejs.org ?

Thanks to the xwidgets support (info "(elisp) Xwidgets"), it is possible
to embed a graphical web browser with full JS support in an emacs buffer.
So itʼs possible to have the best of both worlds: emacsʼs support for
text editing combined with the visualization and interactivity of modern
Javascript libraries.

Iʼm sure that there are ways that graphical programming (for lack of
a better term) in emacs could be made better, including Richardʼs
suggestions that launched this thread.  But we should be proud of
what we already have, rather than pessimistic.

Aaron

PS I donʼt know of any projects using three.js support in xwidgets
specifically; it could turn out that it doesnʼt quite work
(e.g. hardware-accelerated 3D or lack thereof might present an
obstacle).  But anyone can try it and find out, and hopefully if
problems arise figure out how to fix them.

-- 
Aaron Ecay

Re: What improvements would be truly useful?

[Ricardo Wurmus]

Aaron Ecay <aaronecay@gmail.com> writes:

> 2018ko martxoak 5an, Rostislav Svoboda-ek idatzi zuen:
>>
>
> [...]
>
>> I know it's a bitter pill to swallow, but let's face it - do we think,
>> our bellowed Emacs will ever be able to display anything like the
>> examples from https://threejs.org ?
>
> Thanks to the xwidgets support (info "(elisp) Xwidgets"), it is possible
> to embed a graphical web browser with full JS support in an emacs buffer.
> So itʼs possible to have the best of both worlds: emacsʼs support for
> text editing combined with the visualization and interactivity of modern
> Javascript libraries.

Communication with the webkit widget is not easy.  It certainly doesn’t
feel like a first-class citizen in Emacs.  It is possible to send
strings containing a JavaScript programme to the webkit widget and
retrieve simple strings from the widget, but there is no shared state.
All values crossing over the boundary between the Elisp world and the
widget need to be converted first.

We have no way to compile Elisp to JavaScript strings to smoothen out
the rough transition between these two worlds.  (Guile supports
ECMAscript and Elisp, but even in Guile Emacs the webkit widget would be
a foreign body.)

While it is a neat hack to embed an X widget in an Emacs buffer, I don’t
think that we should be using the embedded webkit widget as a display
engine.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: What improvements would be truly useful?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > IMO the future of GUI apps and in our case editors, lies in browser
  > based frameworks programmable by some lisp dialect. For the moment it
  > would be a stack based on electron, clojurescript, codemirror, etc.

I have a feeling we are talking about totally unrelated questions.
I'm talking about extending Emacs to edit more kinds of _documents_.
Documents with more and different layout features, but still _documents_.

A document is a self-contained file of declarative data that you can
display and edit on your computer, without any sort of network
connection.  And if you send it to me, I can display it too
and it looks the same for me as for you.

It is crucial for a document to be declarative.  It is not safe for an
editor to run unrestricted software out of a document.  That's why
Emacs is careful to make sure documents can't run code.

I looked at threejs.org, and it has an array of interesting images
which seem to have something to do with Javascript libraries.  They
may be useful for other purposes, but not for an editor for documents.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: What improvements would be truly useful?

[Phillip Lord]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > IMO the future of GUI apps and in our case editors, lies in browser
>   > based frameworks programmable by some lisp dialect. For the moment it
>   > would be a stack based on electron, clojurescript, codemirror, etc.
>
> I have a feeling we are talking about totally unrelated questions.
> I'm talking about extending Emacs to edit more kinds of _documents_.
> Documents with more and different layout features, but still _documents_.
>
> A document is a self-contained file of declarative data that you can
> display and edit on your computer, without any sort of network
> connection.  And if you send it to me, I can display it too
> and it looks the same for me as for you.


No, it really isn't. A document is a user-centric feature -- it's a
piece of work that stores information in a way that it useful to the
user. The self-contained file notion is a developer issue.  And,
document editing has moved partially (although not wholly) away from the
"your computer". People edit documents in editors built entirely on the
web. To show it to you, I just send the link.

Following your definition, MS Word becomes a document editor and MS Word
Online does not.

Phil

Re: What improvements would be truly useful?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > No, it really isn't. A document is a user-centric feature -- it's a
  > piece of work that stores information in a way that it useful to the
  > user.

I disagree with you there.  If something can't be copied to various places
and looked at, I say it is not a document.  It may be some other kind of thing.

  > The self-contained file notion is a developer issue.

No, it is absolutely crucial to being able to use the document
in the ways we are accustomed to use documents.

  > Following your definition, MS Word becomes a document editor and MS Word
  > Online does not.

I did not state a definition of "document editor" -- it's not part of the point
I want to make -- so I think that is a misunderstanding.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: What improvements would be truly useful?

[Phillip Lord]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > No, it really isn't. A document is a user-centric feature -- it's a
>   > piece of work that stores information in a way that it useful to the
>   > user.
>
> I disagree with you there.  If something can't be copied to various
> places and looked at, I say it is not a document.  It may be some
> other kind of thing.
>
>   > The self-contained file notion is a developer issue.
>
> No, it is absolutely crucial to being able to use the document
> in the ways we are accustomed to use documents.

I have to disagree with this. I mean, it was true 10 years ago, but it
is not now.


>   > Following your definition, MS Word becomes a document editor and MS Word
>   > Online does not.
>
> I did not state a definition of "document editor" -- it's not part of the point
> I want to make -- so I think that is a misunderstanding.

No, I think not. Consider this scenario. I start editing some text on
google docs with a collaborator; after a while we export the doc into
word, share this with Dropbox, then eventually one person takes it over.
Is this text really not a document till we download it?

All of which gets back what I think was the point of the OP; Emacs would
be a better tool if it supported the web. We've been having intense
discussions about widget toolkits; what about a web-delivered Emacs? The
Web and Mobile together make up the majority of computing these days.

http://www.ymacs.org

If you want a good example.

Phil

Re: What improvements would be truly useful?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > No, I think not. Consider this scenario. I start editing some text on
  > google docs with a collaborator; after a while we export the doc into
  > word, share this with Dropbox, then eventually one person takes it over.
  > Is this text really not a document till we download it?

With all due respect, I think you're disputing a misunderstanding.

As far as I know, what people edit with Google Docs is a document.  It
can be copied to a computer (though doing so requires running nonfree
software) and viewed or edited there.

Google Docs is a service for storing documents that is designed for
editing them using nonfree software.

  > All of which gets back what I think was the point of the OP; Emacs would
  > be a better tool if it supported the web. We've been having intense
  > discussions about widget toolkits; what about a web-delivered Emacs?

What exactly would that mean?  Could you describe the scenario by giving
names or letters to each of the computers involved, and saying what job each
computer does?

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: Variable-width font indentation (was: What improvements would be truly useful?)

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Date: Mon, 05 Mar 2018 08:11:38 -0500
> Cc: eggert@cs.ucla.edu, dancol@dancol.org, emacs-devel@gnu.org
> 
> Changing Emacs to handle indentation and alignment with
> variable-width fonts would be an important and useful change.
> Certain kinds of use would make sense, which currently don't.

This has come up several times, and I think I asked at least once what
does it need to entail.  It would be helpful to have the answer to
that, which will have to be the result of looking at the relevant Lisp
code and figuring out which primitives/subroutines will need to be
taught fractional column values.

Emacs already knows how to align text at pixel resolution (which is
easy to express in fractions of the width of the default face's font),
and in fact already does so when it displays TAB characters on GUI
frames.  We have the 'space' display property to do the same with
characters other than a TAB.

So what else is needed?  Can someone who is familiar with indent.el
please tell?

Re: Variable-width font indentation (was: What improvements would be truly useful?)

[dancol]

>> From: Richard Stallman <rms@gnu.org>
>> Date: Mon, 05 Mar 2018 08:11:38 -0500
>> Cc: eggert@cs.ucla.edu, dancol@dancol.org, emacs-devel@gnu.org
>>
>> Changing Emacs to handle indentation and alignment with
>> variable-width fonts would be an important and useful change.
>> Certain kinds of use would make sense, which currently don't.
>
> This has come up several times, and I think I asked at least once what
> does it need to entail.  It would be helpful to have the answer to
> that, which will have to be the result of looking at the relevant Lisp
> code and figuring out which primitives/subroutines will need to be
> taught fractional column values.
>
> Emacs already knows how to align text at pixel resolution (which is
> easy to express in fractions of the width of the default face's font),
> and in fact already does so when it displays TAB characters on GUI
> frames.  We have the 'space' display property to do the same with
> characters other than a TAB.
>
> So what else is needed?  Can someone who is familiar with indent.el
> please tell?

Well, say you want to align the following code:

1: void foo(int a,
2:          int b)
3: {
4:  ...

How is any generic indentation system supposed to know that the "i" on
line 2 is supposed to line up with the "i" on line 1? You'd need to
communicate all indentation intent to the generic system in the form of
positions relative to some anchor. (Presumably, it could cache pixel
offsets per-line.) While that's doable, it's a ton of work, and I don't
think it's worthwhile.

Re: Variable-width font indentation

[Paul Eggert]

On 03/05/2018 11:32 AM, dancol@dancol.org wrote:
> 1: void foo(int a,
> 2:          int b)
> 3: {
> 4:  ...
>
> How is any generic indentation system supposed to know that the "i" on
> line 2 is supposed to line up with the "i" on line 1?

It assumes the input is fixed-width, and that leading white space is 
intended to indent with respect to previous lines. Although it'd be some 
work to get Emacs to do this, I don't offhand see any algorithmic 
problem here.

Re: Variable-width font indentation

[Dmitry Gutov]

On 3/5/18 9:49 PM, Paul Eggert wrote:
> On 03/05/2018 11:32 AM, dancol@dancol.org wrote:
>> 1: void foo(int a,
>> 2:          int b)
>> 3: {
>> 4:  ...
>>
>> How is any generic indentation system supposed to know that the "i" on
>> line 2 is supposed to line up with the "i" on line 1?
> 
> It assumes the input is fixed-width, and that leading white space is 
> intended to indent with respect to previous lines. Although it'd be some 
> work to get Emacs to do this, I don't offhand see any algorithmic 
> problem here.

If the way whitespace at bol is rendered depends on indentation code, 
will the mean that while scrolling a buffer the display engine will have 
to call indentation code?

Re: Variable-width font indentation

[Clément Pit-Claudel]

On 2018-03-06 10:15, Dmitry Gutov wrote:
> On 3/5/18 9:49 PM, Paul Eggert wrote:
>> On 03/05/2018 11:32 AM, dancol@dancol.org wrote:
>>> 1: void foo(int a,
>>> 2:          int b)
>>> 3: {
>>> 4:  ...
>>>
>>> How is any generic indentation system supposed to know that the "i" on
>>> line 2 is supposed to line up with the "i" on line 1?
>>
>> It assumes the input is fixed-width, and that leading white space is intended to indent with respect to previous lines. Although it'd be some work to get Emacs to do this, I don't offhand see any algorithmic problem here.
> 
> If the way whitespace at bol is rendered depends on indentation code, will the mean that while scrolling a buffer the display engine will have to call indentation code?

In Paul's suggestion (the one I implemented), the width of whitespace at bol does not depend on indentation code.

Re: Variable-width font indentation

[Eli Zaretskii]

> Cc: raeburn@raeburn.org, rms@gnu.org, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Tue, 6 Mar 2018 17:15:29 +0200
> 
> If the way whitespace at bol is rendered depends on indentation code, 
> will the mean that while scrolling a buffer the display engine will have 
> to call indentation code?

I very much hope not, as this would be a performance hit.

Re: Variable-width font indentation (was: What improvements would be truly useful?)

[Eli Zaretskii]

> Date: Mon, 5 Mar 2018 11:32:30 -0800
> From: dancol@dancol.org
> Cc: rms@gnu.org,
>  raeburn@raeburn.org,
>  eggert@cs.ucla.edu,
>  dancol@dancol.org,
>  emacs-devel@gnu.org
> 
> Well, say you want to align the following code:
> 
> 1: void foo(int a,
> 2:          int b)
> 3: {
> 4:  ...
> 
> How is any generic indentation system supposed to know that the "i" on
> line 2 is supposed to line up with the "i" on line 1?

How does it know that today, when we only support well the fixed-pitch
fonts?  Or are you talking about a different problem?

> You'd need to communicate all indentation intent to the generic
> system in the form of positions relative to some
> anchor. (Presumably, it could cache pixel offsets per-line.) While
> that's doable, it's a ton of work, and I don't think it's
> worthwhile.

Maybe I still misunderstand you, but the pixel position of a character
on display is trivial to obtain, and I see no reason why we'd need to
cache them.  Again, the current fixed-pitch column-wise indentation
machinery doesn't, AFAIK.

Re: Variable-width font indentation

[Clément Pit-Claudel]

On 2018-03-05 14:58, Eli Zaretskii wrote:
> How does it know that today, when we only support well the
> fixed-pitch fonts?  Or are you talking about a different problem?

A different problem, IIUC.  The problem is the following:

Assume you wrote the following C code, then changed to variable-pitch:

void pr () {
  printf ("hello, world\n");
}

int main (int argc, 
          char** argv) {
  pr ();
  return 0;
}

Depending on which font you use (and the width of a space in that font), the `char** argv' part will move right or left, instead of remaining aligned.  The usual solution is to proceed as Paul suggested:

> It assumes the input is fixed-width, and that leading white space is
> intended to indent with respect to previous lines.

IOW, you scale the spaces preceding `char** argv' to make the `c' of `char' line up with the `i' of int.

Applying this strategy to all lines doesn't work, unfortunately: it indents the `pr ()' and `printf (' lines differently, because the `in' of `int main' and the `vo' of `void' occupy different widths.

I don't know of a good solution to this problem: in general, there's not enough information to distinguish 'indentation' spaces (the two spaces before `printf (', `pr ()', and `return 0') and 'line-up' spaces (the spaces before `char ()').

I hope I'm missing something.

Clément.

Re: Variable-width font indentation

[Paul Eggert]

On 03/05/2018 12:28 PM, Clément Pit-Claudel wrote:
> Applying this strategy to all lines doesn't work, unfortunately: it indents the `pr ()' and `printf (' lines differently, because the `in' of `int main' and the `vo' of `void' occupy different widths.

True, but that's a feature not a defect. The code is easier to read if 
every region is indented individually, as opposed to using exactly the 
same indenting everywhere in the buffer.

Re: Variable-width font indentation

[Daniel Colascione]

On 03/05/2018 02:30 PM, Paul Eggert wrote:
> On 03/05/2018 12:28 PM, Clément Pit-Claudel wrote:
>> Applying this strategy to all lines doesn't work, unfortunately: it 
>> indents the `pr ()' and `printf (' lines differently, because the `in' 
>> of `int main' and the `vo' of `void' occupy different widths.
> 
> True, but that's a feature not a defect. The code is easier to read if 
> every region is indented individually, as opposed to using exactly the 
> same indenting everywhere in the buffer.

In non-lineup styles (e.g., the Linux kernel), different blocks with the 
same notional indentation actually having different offsets from the 
left margin would be very unusual.

Re: Variable-width font indentation

[Clément Pit-Claudel]

On 2018-03-05 17:30, Paul Eggert wrote:
> On 03/05/2018 12:28 PM, Clément Pit-Claudel wrote:
>> Applying this strategy to all lines doesn't work, unfortunately: it
>> indents the `pr ()' and `printf (' lines differently, because the
>> `in' of `int main' and the `vo' of `void' occupy different widths.
> 
> True, but that's a feature not a defect. The code is easier to read
> if every region is indented individually, as opposed to using exactly
> the same indenting everywhere in the buffer.

Hmm.  I hadn't thought of this :)  But I'm not entirely convinced yet.  Consider this snippet:

def x():
    with open("A") as f:
        while y:
            print(y.v())

or this one in elisp:

(when x
  (if y t
    (unless p
      (cl-loop for x in y
               collect x)))) 

In variable-pitch mode, each new indentation level has a slightly-different width, instead of a regular progression of 4 spaces (in Python) and 2 spaces (in Elisp) at each level.

Beyond this, there's the problem of inline line-up spaces.  For an example, consider this, from xdisp.c, which shows "line-up" spaces in full glory:


   +--------------+   redisplay     +----------------+
   | Lisp machine |---------------->| Redisplay code |<--+
   +--------------+   (xdisp.c)     +----------------+   |
	  ^				     |		 |
	  +----------------------------------+           |
	    Block input to prevent this when             |
	    called asynchronously!			 |
							 |
		    note_mouse_highlight (asynchronous)	 |
							 |
				    X mouse events  -----+
							 |
			    expose_frame (asynchronous)	 |
							 |
				   X expose events  -----+

The same problem pops up with org-mode tables, etc.

Clément.

Re: Variable-width font indentation

[Paul Eggert]

On 03/05/2018 05:06 PM, Clément Pit-Claudel wrote:
> In variable-pitch mode, each new indentation level has a slightly-different width, instead of a regular progression of 4 spaces (in Python) and 2 spaces (in Elisp) at each level.

That's fine. Although it's only a small thing, to me it's even a small 
plus, as having every indent level be the same number of pixels 
unnaturally focuses the reader on a distracting regularity that is not 
intrinsic to the code.

> Beyond this, there's the problem of inline line-up spaces.

Yes, for ASCII art I doubt whether any simple automated heuristic would 
work unaided. ASCII art is typically ugly and hard-to-read even in a 
fixed-width font, so it's not much of a loss if it's rendered poorly. If 
there's really a need for it I suppose we could let users hint to Emacs 
to switch to a fixed-width font just for the art; that'd be good enough.

Re: Variable-width font indentation

[Clément Pit-Claudel]

On 2018-03-05 20:30, Paul Eggert wrote:
> On 03/05/2018 05:06 PM, Clément Pit-Claudel wrote:
>> In variable-pitch mode, each new indentation level has a
>> slightly-different width, instead of a regular progression of 4
>> spaces (in Python) and 2 spaces (in Elisp) at each level.
> 
> That's fine. Although it's only a small thing, to me it's even a
> small plus, as having every indent level be the same number of pixels
> unnaturally focuses the reader on a distracting regularity that is
> not intrinsic to the code.

I've been playing with this a bit more; until now, I'd mostly stopped at my previous objection.

The following code gives a preview of what the algorithm that we've been discussing produces:

(defun ~/variable-pitch-indent ()
  (interactive)
  (save-excursion
    (let ((spine "")
          (face `(:foreground ,(face-attribute 'default :background nil))))
      (goto-char (point-min))
      (while (not (eobp))
        (back-to-indentation)
        (unless (eolp)
          (let* ((indentation-amount (current-column))
                 (spine-cut (min (length spine) indentation-amount))
                 (spine-prefix (substring spine 0 spine-cut))
                 (spine-padding (make-string (- indentation-amount spine-cut) ?\s))
                 (spine-end (buffer-substring (point) (point-at-eol))))
            (setq spine
                  (with-temp-buffer
                    ;; HACK: expand spaces
                    (setq-local tab-width 8)
                    (insert spine-prefix spine-padding spine-end)
                    (untabify (point-min) (point-max))
                    (buffer-string)))
            (put-text-property (point-at-bol) (point) 'display
                               (propertize (concat spine-prefix spine-padding)
                                           'face face))))
        (forward-line 1)))))

I've tested it on two files: src/termcap.c and lisp/dabbrev.el.  To try it out, open one of these files in emacs -Q, then run M-x ~/variable-pitch-indent, and then M-x variable-pitch-mode.

I do agree that it doesn't look too bad, and presumably a C implementation of the algorithm above would be very fast, since it could build the "spine" above during redisplay.

Clément.

Re: Variable-width font indentation

[Paul Eggert]

On 03/05/2018 05:40 PM, Clément Pit-Claudel wrote:
> I've tested it on two files: src/termcap.c and lisp/dabbrev.el.  To try it out, open one of these files in emacs -Q, then run M-x ~/variable-pitch-indent, and then M-x variable-pitch-mode.
>
> I do agree that it doesn't look too bad, and presumably a C implementation of the algorithm above would be very fast, since it could build the "spine" above during redisplay.

Thanks, that was fast! Yes, it should work fast.

I saw one easily-fixable minor glitch (on my screen lisp/dabbr.el line 
533 had the wrong indent, presumably since the "user-error" on the 
previous line was in a fatter-than usual font). The biggest trickier 
minor glitch was that indenting of #if and #else subparts were 
annoyingly different because the "#" violates the usual indentation 
rule. Personally I've always thought that we should indent "#" like 
anything else -- the only reason we don't is that it didn't work with 
K&R C and so we got used to bad style -- but even if we stick with the 
bad "#if" indenting it still looks pretty reasonable.

Re: Variable-width font indentation

[Clément Pit-Claudel]

On 2018-03-05 21:04, Paul Eggert wrote:
> On 03/05/2018 05:40 PM, Clément Pit-Claudel wrote:
>> I've tested it on two files: src/termcap.c and lisp/dabbrev.el.  To try it out, open one of these files in emacs -Q, then run M-x ~/variable-pitch-indent, and then M-x variable-pitch-mode.
>>
>> I do agree that it doesn't look too bad, and presumably a C implementation of the algorithm above would be very fast, since it could build the "spine" above during redisplay.
> 
> Thanks, that was fast! Yes, it should work fast.
> 
> I saw one easily-fixable minor glitch (on my screen lisp/dabbr.el line 533 had the wrong indent, presumably since the "user-error" on the previous line was in a fatter-than usual font).

Yes, that's due to the implementation trick that I used, which doesn't preserve faces on the "spine", the string used for indentation.

The biggest trickier minor glitch was that indenting of #if and #else subparts were annoyingly different because the "#" violates the usual indentation rule. Personally I've always thought that we should indent "#" like anything else -- the only reason we don't is that it didn't work with K&R C and so we got used to bad style -- but even if we stick with the bad "#if" indenting it still looks pretty reasonable.

Indeed.  It does look fairly OK, actually — much better than I expected it to be.

I guess the next step would be to make a minor mode to apply this variable-pitch indentation on the fly and see how well it works in real life.

Some difficulties:

* Changing a line will cause all lines below it to get reindented; this will make redisplay costly, and I'm not sure how good it'll look — and we won't be able to tell until we get a C implementation, since an ELisp implementation likely won't update indentation for following lines on the fly.
* I'm not aware of a way to get a specified space whose width equals that of a particular string, without measuring the string beforehand.  Is there a way?

Clément.

Re: Variable-width font indentation

[Eli Zaretskii]

> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Mon, 5 Mar 2018 22:36:55 -0500
> 
> * I'm not aware of a way to get a specified space whose width equals that of a particular string, without measuring the string beforehand.  Is there a way?

What do you mean by "measuring the string"?  Calculating its width?

In your implementation you used 'length', which is definitely not TRT,
especially with variable-width fonts.  I think you can find something
to help you out in shr.el.  One other possibility is to use
font-get-glyphs, but I'm not sure it will help in this case.

And I still think that for programming modes, what we need is to
substitute tab-width and column by suitably computed number of pixels,
perhaps derived from the width of some character.  Other than that,
the indentation code should "just work", as long as it uses :align-to
to get to a given pixel coordinate.

Re: Variable-width font indentation

[Clément Pit-Claudel]

On 2018-03-06 11:47, Eli Zaretskii wrote:
> What do you mean by "measuring the string"?  Calculating its width?

Correct.

> In your implementation you used 'length', which is definitely not TRT,
> especially with variable-width fonts.

I think you misread the implementation :)  But I'm not sure which part caused confusion, so I'm not sure what to explain.

> And I still think that for programming modes, what we need is to
> substitute tab-width and column by suitably computed number of pixels,
> perhaps derived from the width of some character.  Other than that,
> the indentation code should "just work", as long as it uses :align-to
> to get to a given pixel coordinate.

I don't understand this part :/

Re: Variable-width font indentation

[Daniel Colascione]

On 03/05/2018 06:04 PM, Paul Eggert wrote:
> On 03/05/2018 05:40 PM, Clément Pit-Claudel wrote:
>> I've tested it on two files: src/termcap.c and lisp/dabbrev.el.  To 
>> try it out, open one of these files in emacs -Q, then run M-x 
>> ~/variable-pitch-indent, and then M-x variable-pitch-mode.
>>
>> I do agree that it doesn't look too bad, and presumably a C 
>> implementation of the algorithm above would be very fast, since it 
>> could build the "spine" above during redisplay.
> 
> Thanks, that was fast! Yes, it should work fast.
> 
> I saw one easily-fixable minor glitch (on my screen lisp/dabbr.el line 
> 533 had the wrong indent, presumably since the "user-error" on the 
> previous line was in a fatter-than usual font).

Right. Given sufficiently divergent character widgets, it'd be possible 
to trick the eye into thinking two blocks of code at semantically 
different indentation levels are in fact at different levels, or vice 
versa. IMHO, that's dangerous. I'd rather whitelist modes where the 
indentation machinery can reliably provide robust 
alignment-versus-indentation information.

Re: Variable-width font indentation

[Clément Pit-Claudel]

On 2018-03-06 00:48, Daniel Colascione wrote:
> I'd rather whitelist modes where the indentation machinery can
> reliably provide robust alignment-versus-indentation information.

Do you think that'll be enough?  What do we do when we open a file that's already indented in a way that doesn't match what our own indentation function would do?

Or do you mean that we need a new "indentation analyzer" for each new mode that tags spaces as "alignment" spaces vs "indentation" spaces?

One more tricky case, from playing with this a bit more: in OCaml, it's not entirely uncommon to see this:

  match o with
  | None   -> 0 (* Note the extra spaces to line up the arrows *)
  | Some x -> x + 1

Clément.

Re: Variable-width font indentation

[Eli Zaretskii]

> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Mon, 5 Mar 2018 20:40:14 -0500
> 
> The following code gives a preview of what the algorithm that we've been discussing produces:

Thanks, but I still don't see why we couldn't simply "adjust" the
width of white space using the 'space' display property.

Your implementation has a few issues.  First, if you move the cursor
vertically at column zero, do you see "ghost" characters displayed at
cursor position?  Hiding characters by displaying them with foreground
color identical to the background color has its limits ;-)

Also, with this implementation, the braces in the 'if' and 'else'
blocks no longer align.  E.g., try to indent this snippet:

foo ()
{
  if (something)
    {
      do (anything);
    }
  else
    {
      do (something-else);
    }
}

I think this misalignment will be annoying.

> I do agree that it doesn't look too bad, and presumably a C implementation of the algorithm above would be very fast, since it could build the "spine" above during redisplay.

Indenting during redisplay is a bad idea, because it will disable
almost every redisplay optimization.

I actually don't understand why you worry about performance: the
function you wrote is the morel equivalent of C-\, and that one is not
fast with the current implementation.  We never modify indentation of
non-current lines anyway, we expect the user to type TAB or some
electric character to reindent a line, and we expect them to use C-\
to reindent more than one line.  Automatic adjustment of indentation
might be a good feature, but it would be a separate feature.

In any case, I'd suggest to reindent via buffer-modification hooks,
not as part of redisplay.

Re: Variable-width font indentation

[Clément Pit-Claudel]

On 2018-03-06 11:36, Eli Zaretskii wrote:
>> From: Clément Pit-Claudel <cpitclaudel@gmail.com> Date: Mon, 5 Mar
>> 2018 20:40:14 -0500
>> 
>> The following code gives a preview of what the algorithm that we've
>> been discussing produces:
> 
> Thanks, but I still don't see why we couldn't simply "adjust" the 
> width of white space using the 'space' display property.

Because "I'm not aware of a way to get a specified space whose width equals that of a particular string, without measuring the string beforehand.  Is there a way?" :)

IOW, I'm all for it, but I don't know how.

> Your implementation has a few issues.  First, if you move the cursor 
> vertically at column zero, do you see "ghost" characters displayed
> at cursor position?  Hiding characters by displaying them with
> foreground color identical to the background color has its limits
> ;-)

That's a feature :) (Just to be clear: that was the easy way to get a feel for what the algorithm that Paul and I were discussing would produce; I'd have used space properties if I had known a way to get a space of the same with as these ghost characters).

And there are many other issues, of course: we don't want to apply the display property to more than one stretch of space, and the code doesn't deal properly with tabs vs spaces, etc.

> Also, with this implementation, the braces in the 'if' and 'else' 
> blocks no longer align.  E.g., try to indent this snippet:
> 
> foo () { if (something) { do (anything); } else { do
> (something-else); } }
> 
> I think this misalignment will be annoying.


Indeed, this was discussed in this thread; Paul thinks it's not an issue; I'm not convinced (I tend to agree with you) :)

>> I do agree that it doesn't look too bad, and presumably a C
>> implementation of the algorithm above would be very fast, since it
>> could build the "spine" above during redisplay.
> 
> Indenting during redisplay is a bad idea, because it will disable 
> almost every redisplay optimization.

I think we're using "indenting" to mean different things.  (Just to be clear: the process I'm describing doesn't call indent-line-function, nor any of the indentation machinery)

> I actually don't understand why you worry about performance: the 
> function you wrote is the morel equivalent of C-\

C-\ is toggle-input-method on my machine; is that what you meant?

> , and that one is not fast with the current implementation.  We never
> modify indentation of non-current lines anyway, we expect the user to
> type TAB or some electric character to reindent a line, and we expect
> them to use C-\ to reindent more than one line.  Automatic adjustment
> of indentation might be a good feature, but it would be a separate
> feature.

I never meant to adjust indentation (that is, to change the contents of the buffer): just to display existing leading spaces differently based on the contents of previous line. 

> In any case, I'd suggest to reindent via buffer-modification hooks, 
> not as part of redisplay.

Right, and that's why I worry about performance.

Clément.

Re: Variable-width font indentation

[Eli Zaretskii]

> Cc: eggert@cs.ucla.edu, emacs-devel@gnu.org
> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Tue, 6 Mar 2018 15:11:29 -0500
> 
> > Thanks, but I still don't see why we couldn't simply "adjust" the 
> > width of white space using the 'space' display property.
> 
> Because "I'm not aware of a way to get a specified space whose width equals that of a particular string, without measuring the string beforehand.  Is there a way?" :)

The "string" you refer to is already in the buffer on display, right?
If so, you can simply use the likes of posn-at-point to measure
portions of text in pixel units.  Then translate that into
floating-point "column" numbers by dividing by width of some suitable
character (it could be SPC, but that is a bad idea with some fonts, so
maybe something else, like 'e').

> I'd have used space properties if I had known a way to get a space of the same with as these ghost characters).

See above.  And there's font-get-glyphs.

> And there are many other issues, of course: we don't want to apply the display property to more than one stretch of space, and the code doesn't deal properly with tabs vs spaces, etc.

Please elaborate, because I don't understand those issues well enough
to see if and how they can be resolved.

> >> I do agree that it doesn't look too bad, and presumably a C
> >> implementation of the algorithm above would be very fast, since it
> >> could build the "spine" above during redisplay.
> > 
> > Indenting during redisplay is a bad idea, because it will disable 
> > almost every redisplay optimization.
> 
> I think we're using "indenting" to mean different things.  (Just to be clear: the process I'm describing doesn't call indent-line-function, nor any of the indentation machinery)

Then what _do_ you mean by "building the spine during redisplay"?

> > I actually don't understand why you worry about performance: the 
> > function you wrote is the morel equivalent of C-\
> 
> C-\ is toggle-input-method on my machine; is that what you meant?

Sorry, I meant M-C-\, of course.

> > , and that one is not fast with the current implementation.  We never
> > modify indentation of non-current lines anyway, we expect the user to
> > type TAB or some electric character to reindent a line, and we expect
> > them to use C-\ to reindent more than one line.  Automatic adjustment
> > of indentation might be a good feature, but it would be a separate
> > feature.
> 
> I never meant to adjust indentation (that is, to change the contents of the buffer): just to display existing leading spaces differently based on the contents of previous line. 

We never do that with the existing indentation machinery, either.

> > In any case, I'd suggest to reindent via buffer-modification hooks, 
> > not as part of redisplay.
> 
> Right, and that's why I worry about performance.

Then don't automatically reindent, let the user initiate that.

Re: Variable-width font indentation

[Clément Pit-Claudel]

On 2018-03-06 15:40, Eli Zaretskii wrote:
>> Cc: eggert@cs.ucla.edu, emacs-devel@gnu.org
>> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
>> Date: Tue, 6 Mar 2018 15:11:29 -0500
>>
>>> Thanks, but I still don't see why we couldn't simply "adjust" the 
>>> width of white space using the 'space' display property.
>>
>> Because "I'm not aware of a way to get a specified space whose width equals that of a particular string, without measuring the string beforehand.  Is there a way?" :)
> 
> The "string" you refer to is already in the buffer on display, right?
> If so, you can simply use the likes of posn-at-point to measure
> portions of text in pixel units.

Yes and no: it might be offscreen.  But posn-at-point is a useful suggestion, thanks :) 

> Then translate that into
> floating-point "column" numbers by dividing by width of some suitable
> character (it could be SPC, but that is a bad idea with some fonts, so
> maybe something else, like 'e').

I don't think I need that part.

>> I'd have used space properties if I had known a way to get a space of the same with as these ghost characters).
> 
> See above.  And there's font-get-glyphs.

Thanks, I'll look into this.

>> And there are many other issues, of course: we don't want to apply the display property to more than one stretch 
of space, and the code doesn't deal properly with tabs vs spaces, etc.
> 
> Please elaborate, because I don't understand those issues well enough
> to see if and how they can be resolved.

That code was just a rough draft, so it's plenty of corner cases that I didn't think about yet.  That's all :)

>>>> I do agree that it doesn't look too bad, and presumably a C
>>>> implementation of the algorithm above would be very fast, since it
>>>> could build the "spine" above during redisplay.
>>>
>>> Indenting during redisplay is a bad idea, because it will disable 
>>> almost every redisplay optimization.
>>
>> I think we're using "indenting" to mean different things.  (Just to be clear: the process I'm describing doesn't call indent-line-function, nor any of the indentation machinery)
> 
> Then what _do_ you mean by "building the spine during redisplay"?

I meant build a prefix string (which the code I sent called the "spine") to use instead of leading spaces.
The ghost characters that you noticed are the value of the spine on that line.

>>> I actually don't understand why you worry about performance: the 
>>> function you wrote is the morel equivalent of C-\
>>
>> C-\ is toggle-input-method on my machine; is that what you meant?
> 
> Sorry, I meant M-C-\, of course.

Ah, yes, I see. That's not what the code I offered does at all, I think.

>>> , and that one is not fast with the current implementation.  We never
>>> modify indentation of non-current lines anyway, we expect the user to
>>> type TAB or some electric character to reindent a line, and we expect
>>> them to use C-\ to reindent more than one line.  Automatic adjustment
>>> of indentation might be a good feature, but it would be a separate
>>> feature.
>>
>> I never meant to adjust indentation (that is, to change the contents of the buffer): just to display existing leading spaces differently based on the contents of previous line. 
> 
> We never do that with the existing indentation machinery, either.

Wait, never do what?

>>> In any case, I'd suggest to reindent via buffer-modification hooks, 
>>> not as part of redisplay.
>>
>> Right, and that's why I worry about performance.
> 
> Then don't automatically reindent, let the user initiate that.

The feature I'm offering is like font-locking, not like reindenting: it changes the display of existing buffer elements in a way that doesn't modify the underlying text.  Given this, it makes more sense to me to run it automatically (rather than on demand)

Clément. 

Re: Variable-width font indentation

[Eli Zaretskii]

> Cc: eggert@cs.ucla.edu, emacs-devel@gnu.org
> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Wed, 7 Mar 2018 08:15:39 -0500
> 
> > The "string" you refer to is already in the buffer on display, right?
> > If so, you can simply use the likes of posn-at-point to measure
> > portions of text in pixel units.
> 
> Yes and no: it might be offscreen.

Then the text you need to indent below it is also off-screen, and you
don't need to re-indent it, right?

> > Then what _do_ you mean by "building the spine during redisplay"?
> 
> I meant build a prefix string (which the code I sent called the "spine") to use instead of leading spaces.

What would be the advantage of building such a string at redisplay
time?  Consing a short string is very fast, and has the advantage of
being done only when needed (by contrast, display code is many times
invoked for reasons utterly unrelated to actually displaying stuff).

> >> I never meant to adjust indentation (that is, to change the contents of the buffer): just to display existing leading spaces differently based on the contents of previous line. 
> > 
> > We never do that with the existing indentation machinery, either.
> 
> Wait, never do what?

What you say in the sentence I quoted: display existing leading spaces
differently based on the contents of previous line.

> >>> In any case, I'd suggest to reindent via buffer-modification hooks, 
> >>> not as part of redisplay.
> >>
> >> Right, and that's why I worry about performance.
> > 
> > Then don't automatically reindent, let the user initiate that.
> 
> The feature I'm offering is like font-locking, not like reindenting: it changes the display of existing buffer elements in a way that doesn't modify the underlying text.  Given this, it makes more sense to me to run it automatically (rather than on demand)

"Automatically" doesn't necessarily mean "like font-lock".  I think we
should know better than get ourselves once again into an adventure
like font-lock, which even today, with JIT and all the years of
experience and improvements, is still a headache, if not a PITA.  If
there's no other way to do something that's really necessary, fine;
but we should avoid such features as much as possible.

For example, in this case, I'm beginning to think that we should
render SPC characters similarly to what we do with TABs: produce a
stretch glyph of a suitably computed width, and perhaps introduce a
variable space-width (like tab-width) to help Lisp programs with
controlling that.  Note that 'space' display property supports values
that are symbols, in which case it uses the symbol's value -- this
allows Lisp programs to change the value of a buffer-local binding of
the symbol, and have the indentation change at the next redisplay
cycle (a.k.a. "immediately").  Doesn't that provide a better way for
controlling indentation than font-lock style scanning of the visible
portion of the buffer "just in time"?

Re: Variable-width font indentation

[Clément Pit-Claudel]

On 2018-03-07 13:26, Eli Zaretskii wrote:
> Note that 'space' display property supports values
> that are symbols, in which case it uses the symbol's value -- this
> allows Lisp programs to change the value of a buffer-local binding of
> the symbol, and have the indentation change at the next redisplay
> cycle (a.k.a. "immediately").  Doesn't that provide a better way for
> controlling indentation than font-lock style scanning of the visible
> portion of the buffer "just in time"?

Sorry, I missed this email before.
Yes, this sounds pretty good, and I understand the reticence to depending on font-lock.

Re: Variable-width font indentation

[Clément Pit-Claudel]

On 2018-03-05 20:40, Clément Pit-Claudel wrote:
> On 2018-03-05 20:30, Paul Eggert wrote:
>> On 03/05/2018 05:06 PM, Clément Pit-Claudel wrote:
>>> In variable-pitch mode, each new indentation level has a
>>> slightly-different width, instead of a regular progression of 4
>>> spaces (in Python) and 2 spaces (in Elisp) at each level.
>>
>> That's fine. Although it's only a small thing, to me it's even a
>> small plus, as having every indent level be the same number of pixels
>> unnaturally focuses the reader on a distracting regularity that is
>> not intrinsic to the code.
> 
> I've been playing with this a bit more; until now, I'd mostly stopped at my previous objection.
> 
> The following code gives a preview of what the algorithm that we've been discussing produces: [...]

This version is a bit faster, and it uses 'display (space …) instead of ghost strings:

(defun ~/variable-pitch-indent ()
  (interactive)
  (save-excursion
    (let ((spine [])
          (prev-bol nil)
          (prev-eol nil))
      (goto-char (point-min))
      (while (not (eobp))
        (let ((bol (point)))
          (back-to-indentation)
          (unless (eolp)
            (let* ((indentation-amount (current-column))
                   (trunc-spine-to (min (length spine) indentation-amount)))
              (setq spine (seq-subseq spine 0 trunc-spine-to))
              ;; Extend spine based on previous line
              (when (and prev-bol (< (length spine) indentation-amount))
                (let ((beg (+ prev-bol (length spine)))
                      (end (min prev-eol (+ prev-bol indentation-amount))))
                  (setq spine (vconcat spine (~/measure-px-widths beg end)))))
              (setq prev-bol bol prev-eol (point-at-eol))))
          (dotimes (idx (min (length spine) (- (point) bol)))
            (let ((w (aref spine idx)))
              (put-text-property (+ bol idx) (+ bol idx 1)
                                 'display `(space :width (,w)))))
          (forward-line 1))))))

… but it doesn't work with tabs.

Re: Variable-width font indentation

[Eli Zaretskii]

> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Wed, 7 Mar 2018 17:13:09 -0500
> 
> This version is a bit faster, and it uses 'display (space …) instead of ghost strings:
> 
> (defun ~/variable-pitch-indent ()
>   (interactive)
>   (save-excursion
>     (let ((spine [])
>           (prev-bol nil)
>           (prev-eol nil))
>       (goto-char (point-min))
>       (while (not (eobp))
>         (let ((bol (point)))
>           (back-to-indentation)
>           (unless (eolp)
>             (let* ((indentation-amount (current-column))
>                    (trunc-spine-to (min (length spine) indentation-amount)))
>               (setq spine (seq-subseq spine 0 trunc-spine-to))
>               ;; Extend spine based on previous line
>               (when (and prev-bol (< (length spine) indentation-amount))
>                 (let ((beg (+ prev-bol (length spine)))
>                       (end (min prev-eol (+ prev-bol indentation-amount))))
>                   (setq spine (vconcat spine (~/measure-px-widths beg end)))))
>               (setq prev-bol bol prev-eol (point-at-eol))))
>           (dotimes (idx (min (length spine) (- (point) bol)))
>             (let ((w (aref spine idx)))
>               (put-text-property (+ bol idx) (+ bol idx 1)
>                                  'display `(space :width (,w)))))
>           (forward-line 1))))))

It seems you forgot to post ~/measure-px-widths.

Re: Variable-width font indentation

[Clément Pit-Claudel]

On 2018-03-08 08:21, Eli Zaretskii wrote:
>> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
>> Date: Wed, 7 Mar 2018 17:13:09 -0500
>>
>> This version is a bit faster, and it uses 'display (space …) instead of ghost strings:
>> ...
>
> It seems you forgot to post ~/measure-px-widths.

Indeed, apologies :/

(defun ~/measure-px-widths (beg end)
  (let ((widths nil))
    (while (< beg end)
      (push (car (window-text-pixel-size nil beg (1+ beg))) widths)
      (cl-incf beg))
    (nreverse widths)))

(defun ~/variable-pitch-indent ()
  (interactive)
  (save-excursion
    (let ((spine [])
          (prev-bol nil)
          (prev-eol nil))
      (goto-char (point-min))
      (while (not (eobp))
        (let ((bol (point)))
          (back-to-indentation)
          (unless (eolp)
            (let* ((indentation-amount (current-column))
                   (trunc-spine-to (min (length spine) indentation-amount)))
              (setq spine (seq-subseq spine 0 trunc-spine-to))
              ;; Extend spine based on previous line
              (when (and prev-bol (< (length spine) indentation-amount))
                (let ((beg (+ prev-bol (length spine)))
                      (end (min prev-eol (+ prev-bol indentation-amount))))
                  (setq spine (vconcat spine (~/measure-px-widths beg end)))))
              (setq prev-bol bol prev-eol (point-at-eol))))
          (dotimes (idx (min (length spine) (- (point) bol)))
            (let ((w (aref spine idx)))
              (put-text-property (+ bol idx) (+ bol idx 1)
                                 'display `(space :width (,w)))))
          (forward-line 1))))))

Re: Variable-width font indentation

[Eli Zaretskii]

> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Mon, 5 Mar 2018 17:30:24 -0800
> 
> Yes, for ASCII art I doubt whether any simple automated heuristic would 
> work unaided. ASCII art is typically ugly and hard-to-read even in a 
> fixed-width font, so it's not much of a loss if it's rendered poorly. If 
> there's really a need for it I suppose we could let users hint to Emacs 
> to switch to a fixed-width font just for the art; that'd be good enough.

We could have an interactive alignment feature, whereby the user could
move a character at pixel resolution, until it aligns to the user's
liking.  The implementation would modify the :align-to attribute of
the display property we put on whitespace before that character.

Re: Variable-width font indentation

[Daniel Colascione]

On 03/06/2018 08:16 AM, Eli Zaretskii wrote:
>> From: Paul Eggert <eggert@cs.ucla.edu>
>> Date: Mon, 5 Mar 2018 17:30:24 -0800
>>
>> Yes, for ASCII art I doubt whether any simple automated heuristic would
>> work unaided. ASCII art is typically ugly and hard-to-read even in a
>> fixed-width font, so it's not much of a loss if it's rendered poorly. If
>> there's really a need for it I suppose we could let users hint to Emacs
>> to switch to a fixed-width font just for the art; that'd be good enough.
> 
> We could have an interactive alignment feature, whereby the user could
> move a character at pixel resolution, until it aligns to the user's
> liking.  The implementation would modify the :align-to attribute of
> the display property we put on whitespace before that character.

How would you serialize the result?

Re: Variable-width font indentation

[Eli Zaretskii]

> Cc: cpitclaudel@gmail.com, emacs-devel@gnu.org
> From: Daniel Colascione <dancol@dancol.org>
> Date: Tue, 6 Mar 2018 08:38:28 -0800
> 
> > We could have an interactive alignment feature, whereby the user could
> > move a character at pixel resolution, until it aligns to the user's
> > liking.  The implementation would modify the :align-to attribute of
> > the display property we put on whitespace before that character.
> 
> How would you serialize the result?

You mean, save it to a disk file?  I think we should insert the number
of blanks to align stuff with fixed-pitch font, so that it looks well
with those fonts.  If we want to preserve the display properties
across sessions, we could use enriched-text-mode, I think.

Re: Variable-width font indentation

[dancol]

>> Cc: cpitclaudel@gmail.com, emacs-devel@gnu.org
>> From: Daniel Colascione <dancol@dancol.org>
>> Date: Tue, 6 Mar 2018 08:38:28 -0800
>>
>> > We could have an interactive alignment feature, whereby the user could
>> > move a character at pixel resolution, until it aligns to the user's
>> > liking.  The implementation would modify the :align-to attribute of
>> > the display property we put on whitespace before that character.
>>
>> How would you serialize the result?
>
> You mean, save it to a disk file?  I think we should insert the number
> of blanks to align stuff with fixed-pitch font, so that it looks well
> with those fonts.  If we want to preserve the display properties
> across sessions, we could use enriched-text-mode, I think.

I don't think there's any clean way to embed this alignment information in
arbitrary program text though.

Re: Variable-width font indentation

[Eli Zaretskii]

> Date: Tue, 6 Mar 2018 09:55:16 -0800
> From: dancol@dancol.org
> Cc: "Daniel Colascione" <dancol@dancol.org>,
>  eggert@cs.ucla.edu,
>  cpitclaudel@gmail.com,
>  emacs-devel@gnu.org
> 
> > You mean, save it to a disk file?  I think we should insert the number
> > of blanks to align stuff with fixed-pitch font, so that it looks well
> > with those fonts.  If we want to preserve the display properties
> > across sessions, we could use enriched-text-mode, I think.
> 
> I don't think there's any clean way to embed this alignment information in
> arbitrary program text though.

No, it would be necessary to reconstruct it.  But any solution that
uses text properties will have the same problem.  And I don't quite
see how we can align text rendered with variable-pitch font without
something like that.

Re: Variable-width font indentation

[Daniel Colascione]

On 03/06/2018 12:18 PM, Eli Zaretskii wrote:
>> Date: Tue, 6 Mar 2018 09:55:16 -0800
>> From: dancol@dancol.org
>> Cc: "Daniel Colascione" <dancol@dancol.org>,
>>   eggert@cs.ucla.edu,
>>   cpitclaudel@gmail.com,
>>   emacs-devel@gnu.org
>>
>>> You mean, save it to a disk file?  I think we should insert the number
>>> of blanks to align stuff with fixed-pitch font, so that it looks well
>>> with those fonts.  If we want to preserve the display properties
>>> across sessions, we could use enriched-text-mode, I think.
>>
>> I don't think there's any clean way to embed this alignment information in
>> arbitrary program text though.
> 
> No, it would be necessary to reconstruct it.  But any solution that
> uses text properties will have the same problem.  And I don't quite
> see how we can align text rendered with variable-pitch font without
> something like that.

Right. The leading-indentation case is *barely* feasible, if you imagine 
that modes can accurately indent all lines automatically. I don't see a 
path toward internal alignment, since there's no algorithmic way to 
disambiguate spacing. The closest thing would be something like 
align-regexp, and even this facility requires manual intervention.

Re: Variable-width font indentation

[Sam Steingold]

> * Daniel Colascione <qnapby@qnapby.bet> [2018-03-06 08:38:28 -0800]:
>
>> ...
>
> How would you serialize the result?

This is actually the question that bothers me too.
I think we need to separate "visual" indentation for word processing
from "semantic" indentation for coding.
The former should be serialized to something like HTML or LaTeX,
and the latter to plain text as it is done now.

IOW, I think the path to word processing in Emacs lies in html-mode
improvements.

Thanks.

-- 
Sam Steingold (http://sds.podval.org/) on darwin Ns 10.3.1561
http://steingoldpsychology.com http://www.childpsy.net http://mideasttruth.com
http://no2bds.org http://americancensorship.org https://ffii.org
Our business is run on trust.  We trust you will pay in advance.

Re: Variable-width font indentation

[Ricardo Wurmus]

Sam Steingold <sds@gnu.org> writes:

>> * Daniel Colascione <qnapby@qnapby.bet> [2018-03-06 08:38:28 -0800]:
>>
>>> ...
>>
>> How would you serialize the result?
>
> This is actually the question that bothers me too.
> I think we need to separate "visual" indentation for word processing
> from "semantic" indentation for coding.
> The former should be serialized to something like HTML or LaTeX,
> and the latter to plain text as it is done now.

Or to XML, which is used by other word processors, and which has a neat
S-expression representation (sxml).

-- 
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: Variable-width font indentation

[Eli Zaretskii]

> From: Sam Steingold <sds@gnu.org>
> Date: Tue, 06 Mar 2018 13:53:54 -0500
> 
> > * Daniel Colascione <qnapby@qnapby.bet> [2018-03-06 08:38:28 -0800]:
> >
> >> ...
> >
> > How would you serialize the result?
> 
> This is actually the question that bothers me too.
> I think we need to separate "visual" indentation for word processing
> from "semantic" indentation for coding.
> The former should be serialized to something like HTML or LaTeX,
> and the latter to plain text as it is done now.

Take a look at enriched.el (and etc/enriched.txt as the demonstration
of its capabilities): we already have the capability for serializing
this, had it for many years.

Re: Variable-width font indentation

[Daniel Colascione]

On 03/07/2018 09:12 AM, Eli Zaretskii wrote:
>> From: Sam Steingold <sds@gnu.org>
>> Date: Tue, 06 Mar 2018 13:53:54 -0500
>>
>>> * Daniel Colascione <qnapby@qnapby.bet> [2018-03-06 08:38:28 -0800]:
>>>
>>>> ...
>>>
>>> How would you serialize the result?
>>
>> This is actually the question that bothers me too.
>> I think we need to separate "visual" indentation for word processing
>> from "semantic" indentation for coding.
>> The former should be serialized to something like HTML or LaTeX,
>> and the latter to plain text as it is done now.
> 
> Take a look at enriched.el (and etc/enriched.txt as the demonstration
> of its capabilities): we already have the capability for serializing
> this, had it for many years.

That's fine so long as you have a file you want to edit _only_ with 
Emacs. Files that need to be understood by other tools would need 
information about indentation adjustment included in some compatible way 
(maybe a comment), but since other tools would know nothing of this 
information, it would go out of date. Requiring that people edit a 
source file only with Emacs is not realistic.

Re: Variable-width font indentation

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Daniel Colascione <dancol@dancol.org>
> Date: Wed, 7 Mar 2018 11:38:47 -0800
> 
> > Take a look at enriched.el (and etc/enriched.txt as the demonstration
> > of its capabilities): we already have the capability for serializing
> > this, had it for many years.
> 
> That's fine so long as you have a file you want to edit _only_ with 
> Emacs. Files that need to be understood by other tools would need 
> information about indentation adjustment included in some compatible way 
> (maybe a comment), but since other tools would know nothing of this 
> information, it would go out of date. Requiring that people edit a 
> source file only with Emacs is not realistic.

Same thing happens with files produced by Office, no?  Why do we
require Emacs to do something different?

Re: Variable-width font indentation

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Take a look at enriched.el (and etc/enriched.txt as the demonstration
  > > of its capabilities): we already have the capability for serializing
  > > this, had it for many years.

  > That's fine so long as you have a file you want to edit _only_ with 
  > Emacs. Files that need to be understood by other tools would need 
  > information about indentation adjustment included in some compatible way 
  > (maybe a comment), but since other tools would know nothing of this 
  > information, it would go out of date. Requiring that people edit a 
  > source file only with Emacs is not realistic.

I agree.  One of the features we will want
is to make Emacs able to read and write ODT format and RTF,
even if at first it can only handle subsets of their features.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: Variable-width font indentation

[Robert Pluim]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > > Take a look at enriched.el (and etc/enriched.txt as the demonstration
>   > > of its capabilities): we already have the capability for serializing
>   > > this, had it for many years.
>
>   > That's fine so long as you have a file you want to edit _only_ with 
>   > Emacs. Files that need to be understood by other tools would need 
>   > information about indentation adjustment included in some compatible way 
>   > (maybe a comment), but since other tools would know nothing of this 
>   > information, it would go out of date. Requiring that people edit a 
>   > source file only with Emacs is not realistic.
>
> I agree.  One of the features we will want
> is to make Emacs able to read and write ODT format and RTF,
> even if at first it can only handle subsets of their features.

I'm still not clear on the goals here, and this makes things even less
clear. As others have mentioned, LibreOffice exists, so why does Emacs
need to be able to read and write ODT? [1]

Robert
Footnotes: 
[1]  Note that you can already indirectly write to ODT if you start
with org and use an exporter

Re: Variable-width font indentation

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: Daniel Colascione <dancol@dancol.org>,  eliz@gnu.org,  sds@gnu.org,  emacs-devel@gnu.org
> Date: Thu, 08 Mar 2018 13:33:01 +0100
> 
> >   > (maybe a comment), but since other tools would know nothing of this 
> >   > information, it would go out of date. Requiring that people edit a 
> >   > source file only with Emacs is not realistic.
> >
> > I agree.  One of the features we will want
> > is to make Emacs able to read and write ODT format and RTF,
> > even if at first it can only handle subsets of their features.
> 
> I'm still not clear on the goals here, and this makes things even less
> clear. As others have mentioned, LibreOffice exists, so why does Emacs
> need to be able to read and write ODT? [1]

So that we could exchange documents with users of Office.

Re: Variable-width font indentation

[Robert Pluim]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Robert Pluim <rpluim@gmail.com>
>> Cc: Daniel Colascione <dancol@dancol.org>,  eliz@gnu.org,  sds@gnu.org,  emacs-devel@gnu.org
>> Date: Thu, 08 Mar 2018 13:33:01 +0100
>> 
>> >   > (maybe a comment), but since other tools would know nothing of this 
>> >   > information, it would go out of date. Requiring that people edit a 
>> >   > source file only with Emacs is not realistic.
>> >
>> > I agree.  One of the features we will want
>> > is to make Emacs able to read and write ODT format and RTF,
>> > even if at first it can only handle subsets of their features.
>> 
>> I'm still not clear on the goals here, and this makes things even less
>> clear. As others have mentioned, LibreOffice exists, so why does Emacs
>> need to be able to read and write ODT? [1]
>
> So that we could exchange documents with users of Office.

OK. I personally have no such goal (at least not with Emacs involved),
and I can't understand why people would want to, but everyone gets to
scratch their own itch.

Robert

Re: Variable-width font indentation

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I don't see a reason to use variable-width fonts
except when they provide an improvement.  For ASCII art,
a fixed-width font is best.

I don't see a reason to use variable-width fonts for code, but if
people like that, I'm in favor of supporting it.

The place where variable-width fonts, and multiple fonts of different
width, are really important is for editing documents with formatted
text.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: Variable-width font indentation

[Herring, Davis]

> I don't know of a good solution to this problem: in general,
> there's not enough information to distinguish 'indentation'
> spaces (the two spaces before `printf (', `pr ()', and `return 0')
> and 'line-up' spaces (the spaces before `char ()').

This is of course an argument in favor of "smart tabs" <https://www.emacswiki.org/emacs/SmartTabs>: you make each tab occupy some user-specified width, and _then_ count one character per space to do alignment.  That doesn't help if those tabs aren't present, although heuristics to detect certain exact numbers of spaces, comparing to the previous lines and looking for enclosing delimiters, might close enough of the gap between "treat all spaces as ASCII art alignment" and the ideal.

Davis

Re: Variable-width font indentation

[Clément Pit-Claudel]

On 2018-03-05 23:05, Herring, Davis wrote:
>> I don't know of a good solution to this problem: in general,
>> there's not enough information to distinguish 'indentation'
>> spaces (the two spaces before `printf (', `pr ()', and `return 0')
>> and 'line-up' spaces (the spaces before `char ()').
> 
> This is of course an argument in favor of "smart tabs" <https://www.emacswiki.org/emacs/SmartTabs>

No, not really: smart tabs aren't enough AFAICT.  Take the following example:

	match opt with
	| []     -> []
	| x :: t -> x :: f t

You'd want to scale the spaces according to the line *below*, not ebove, and even then you'd have to account for the fact that 'x ' isn't as wide as '[]'.  Smart Tabs are also not very good for languages like ELisp with variable indentation depths.

If anything, I think this is more of an argument in favor of elastic tabs (http://nickgravgaard.com/elastic-tabstops/).

Clément.

Re: Variable-width font indentation

[Herring, Davis]

>         match opt with
>         | []     -> []
>         | x :: t -> x :: f t
> 
> You'd want to scale the spaces according to the line *below*, not ebove,

As per your other message, that's a problem regardless of tab handling.  Yet more heuristics suggest themselves (in this case, looking for text shared between two adjacent lines to guide the alignment), but of course those are becoming a mess.

> and even then you'd have to account for the fact that 'x ' isn't as wide as '[]'.

Maybe I misunderstand -- how is this a problem?  If the number of spaces is correct, it identifies the correct character to match against.

> Smart Tabs are also not very good for languages like ELisp with variable indentation depths.

Which cases are hard?  I just see several places where you would use only spaces (_):

(progn
>-(foo)
>-(bar))

(and
_(foo)
_(bar))

(and (foo)
_____(bar))

> If anything, I think this is more of an argument in favor of
> elastic tabs (http://nickgravgaard.com/elastic-tabstops/).

Those are certainly interesting, but they have their own ambiguities (e.g.,

int>-x>-= 1,
>-y>-=2;
doTheFoo();>-// it's true
doTheBar();>-// so far

where there should not be a "column block" 4 lines tall).  They also of course do not play well with existing rendering (monospace or otherwise).

Davis

Re: Variable-width font indentation

[Clément Pit-Claudel]

On 2018-03-06 09:18, Herring, Davis wrote:
> Maybe I misunderstand -- how is this a problem?  If the number of spaces is correct, it identifies the correct character to match against.

It's a problem if you scale each space to match the width of the character above it (or below it, in this case).  If you instead align to a particular position, you run into the issue of deciding which space to stretch (strech all spaces in the range evenly? stretch a single one? what about cases where the spaces need to be shrunk?)

> (progn
> >-(foo)
> >-(bar))
> 
> (and
> _(foo)
> _(bar))

I have a lot of trouble reading these examples, as the '>' get converted to reply markers by my email client; but looking at the source, I think I understand.  Question: why do you indent the body of a 'progn' differently from an 'and'?

I was thinking of cases like 'if', for example:

  (if test
      true-case
    false-case)

Where do you use tabs? 

> 
> int>-x>-= 1,
> >-y>-=2;
> doTheFoo();>-// it's true
> doTheBar();>-// so far

I don't think these are ambiguities (as in, the spec is clear on what should happen here); but indeed this doesn't render ideally; you'd write it like this instead:

int	x = 1,
	y = 2;

doTheFoo();	// it's true
doTheBar();	// so far

(I personally don't like the concept very much, because it doesn't let you distinguish between a 0-width indentation marker and a non-zero width indentation marker).  That is, it forces you to writ this:

	int f(	int argc;
		char** argv)

instead of 

	int f(int argc;
	      int argv)

Clément.

Re: Variable-width font indentation

[Eli Zaretskii]

> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Mon, 5 Mar 2018 15:28:05 -0500
> Cc: Paul Eggert <eggert@cs.ucla.edu>
> 
> On 2018-03-05 14:58, Eli Zaretskii wrote:
> > How does it know that today, when we only support well the
> > fixed-pitch fonts?  Or are you talking about a different problem?
> 
> A different problem, IIUC.  The problem is the following:
> 
> Assume you wrote the following C code, then changed to variable-pitch:
> 
> void pr () {
>   printf ("hello, world\n");
> }
> 
> int main (int argc, 
>           char** argv) {
>   pr ();
>   return 0;
> }
> 
> Depending on which font you use (and the width of a space in that font), the `char** argv' part will move right or left, instead of remaining aligned.  The usual solution is to proceed as Paul suggested:
> 
> > It assumes the input is fixed-width, and that leading white space is
> > intended to indent with respect to previous lines.
> 
> IOW, you scale the spaces preceding `char** argv' to make the `c' of `char' line up with the `i' of int.

I think I'm still missing something, because I don't understand why we
need to invent/use a strategy different from what we do now in code
indentation.  All we need to do to extend the existing machinery to
variable-pitch fonts is (1) to replace current-column with a function
that returns floats instead of integers (or teach current-column do
that given some optional argument), and (b) place a 'space' display
property on the leading whitespace with an :align-to attribute
computed to align the first non-white character at the "column number"
returned as result of (a) above.  All the rest is already in place,
right?

Also note that Richard was specifically talking about text modes, not
about programming modes.  Indentation in text modes has a slightly
different goal.

Re: Variable-width font indentation

[Clément Pit-Claudel]

On 2018-03-06 11:11, Eli Zaretskii wrote:
> I think I'm still missing something, because I don't understand why we
> need to invent/use a strategy different from what we do now in code
> indentation.  [...]  All the rest is already in place,
> right?

I think I know where the misunderstanding is coming from.

We're thinking of two unrelated things:

* Paul and I have been discussing an language-agnostic heuristic to preserve alignment when changing from monospace to variable-pitch faces.  This intends to preserve the original indentation of the code, by inferring what it would have been if the logic originally used to indent the document had been aware of variable-pitch faces.

* You're describing a language-specific way to get a nice-looking variable-pitch rendering of a document; your way is to reindent the document, essentially.

Does this make sense?

Re: Variable-width font indentation

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org, eggert@cs.ucla.edu
> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Tue, 6 Mar 2018 14:59:21 -0500
> 
> * Paul and I have been discussing an language-agnostic heuristic to preserve alignment when changing from monospace to variable-pitch faces.  This intends to preserve the original indentation of the code, by inferring what it would have been if the logic originally used to indent the document had been aware of variable-pitch faces.

I don't see a need for such a feature, sorry.  We have indentation
already; its only problem is that it doesn't work well with variable
pitch fonts.  That's what we need to try to fix, if we care about the
capability of displaying program code with such fonts.

> * You're describing a language-specific way to get a nice-looking variable-pitch rendering of a document; your way is to reindent the document, essentially.

I was describing a way of teaching the existing indentation code
handle variable-pitch fonts.

Re: Variable-width font indentation

[Paul Eggert]

On 03/06/2018 12:31 PM, Eli Zaretskii wrote:
> We have indentation
> already; its only problem is that it doesn't work well with variable
> pitch fonts.

As I understood it, Clément wrote a quick-and-dirty algorithm to 
simulate better indentation with variable-width fonts. That is, it 
wasn't intended to be something that should go into Emacs, only as a way 
to see what the the user would see if we changed Emacs's display of 
indentation to be along the suggested lines. Unless I'm 
misunderstanding, his work provided a simple proof-of-concept that a 
relatively-inexpensive way of improving indentation would significantly 
improve display with variable-width fonts.

Re: Variable-width font indentation

[Clément Pit-Claudel]

On 2018-03-06 15:31, Eli Zaretskii wrote:
>> Cc: emacs-devel@gnu.org, eggert@cs.ucla.edu
>> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
>> Date: Tue, 6 Mar 2018 14:59:21 -0500
>>
>> * Paul and I have been discussing an language-agnostic heuristic to preserve alignment when changing from monospace to variable-pitch faces.  This intends to preserve the original indentation of the code, by inferring what it would have been if the logic originally used to indent the document had been aware of variable-pitch faces.
> 
> I don't see a need for such a feature, sorry.  We have indentation
> already; its only problem is that it doesn't work well with variable
> pitch fonts.  That's what we need to try to fix, if we care about the
> capability of displaying program code with such fonts.

I'm thoroughly confused: doesn't the algorithm that Paul proposed, and that I show an implementation of, do exactly that?

Do you want to get the indentation code involved because the heuristic isn't perfect?
 
>> * You're describing a language-specific way to get a nice-looking variable-pitch rendering of a document; your way is to reindent the document, essentially.
> 
> I was describing a way of teaching the existing indentation code
> handle variable-pitch fonts.

Right, which is where we differ: the solution Paul and I have been thinking is designed not to rely on exiting indentation facilities. 

In your world, concretely, what happens when I open xdisp.c and press M-x variable-pitch-mode? Does everything look misaligned until I M-x indent-region the whole file?

What about a file from a different project that uses a different indenting convention?

Thanks,
Clément.

Re: Variable-width font indentation

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org, eggert@cs.ucla.edu
> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Tue, 6 Mar 2018 16:47:51 -0500
> 
> > I don't see a need for such a feature, sorry.  We have indentation
> > already; its only problem is that it doesn't work well with variable
> > pitch fonts.  That's what we need to try to fix, if we care about the
> > capability of displaying program code with such fonts.
> 
> I'm thoroughly confused: doesn't the algorithm that Paul proposed, and that I show an implementation of, do exactly that?

Not AFAIU, no.  It is an implementation of a different indentation
method.  I don't see why we should want to go that way, since there's
nothing wrong with the existing algorithm, except for handling of
variable-pitch fonts.

> Do you want to get the indentation code involved because the heuristic isn't perfect?

More like because the existing indentation algorithms are okay, and I
see no need to change them to something else.

> In your world, concretely, what happens when I open xdisp.c and press M-x variable-pitch-mode? Does everything look misaligned until I M-x indent-region the whole file?

Yes.  It isn't different from what happens when you change tab-width.

> What about a file from a different project that uses a different indenting convention?

Not sure how that is relevant.  A different indentation convention
will look differently, of course.

Re: Variable-width font indentation

[Clément Pit-Claudel]

On 2018-03-07 12:22, Eli Zaretskii wrote:
>> Cc: emacs-devel@gnu.org, eggert@cs.ucla.edu
>> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
>> Date: Tue, 6 Mar 2018 16:47:51 -0500
>>
>>> I don't see a need for such a feature, sorry.  We have indentation
>>> already; its only problem is that it doesn't work well with variable
>>> pitch fonts.  That's what we need to try to fix, if we care about the
>>> capability of displaying program code with such fonts.
>>
>> I'm thoroughly confused: doesn't the algorithm that Paul proposed, and that I show an implementation of, do exactly that?
> 
> Not AFAIU, no.  It is an implementation of a different indentation
> method.  I don't see why we should want to go that way, since there's
> nothing wrong with the existing algorithm, except for handling of
> variable-pitch fonts.

Can you clarify what you call an indentation method? I think that's the source of my confusion.
I wouldn't call what I wrote an indentation method.  Maybe I'm not understanding you because of that?

>> Do you want to get the indentation code involved because the heuristic isn't perfect?
> 
> More like because the existing indentation algorithms are okay, and I
> see no need to change them to something else.

Neither do I, and indeed I'm not proposing to change existing indentation algorithms :)

>> In your world, concretely, what happens when I open xdisp.c and press M-x variable-pitch-mode? Does everything look misaligned until I M-x indent-region the whole file?
> 
> Yes.  It isn't different from what happens when you change tab-width.

But changing tab-width doesn't change the contents of the buffer, whereas what you're advocating for does, right?

>> What about a file from a different project that uses a different indenting convention?
> 
> Not sure how that is relevant.  A different indentation convention
> will look differently, of course.

This is relevant because the method that Paul originally proposed (which I'm not even advocating for — I'm not even sure yet whether it's good or bad) does not require changing the buffer or the actual number of spaces present on each line in the buffer.

Maybe I just don't understand what you're advocating for.  AFAIU, you're suggesting that we should teach indentation methods about variable-pitch fonts, so that they indent text to a particular pixel offset instead of a particular column.  Is that correct?

If it is correct, then it solves a problem that's different from the one I was thinking in, which is to display existing code with a variable pitch font, without the indentation function.

Some questions on the solution you propose:

* What happens in major modes that don't provide indentation support?
* What happens in indentation-sensitive languages (Python, F#, Haskell, etc)?
* What happens in comments?
* Do I need to call M-x indent-region every time I switch between variable-pitch and fixed-pitch?
* If I close and reopen a file in variable-pitch mode, do I need to re-run M-x indent-region?
* In variable-pitch mode, what gets saved to disk after I run M-x indent-region?

Thanks,
Clément.

Re: Variable-width font indentation

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org, eggert@cs.ucla.edu
> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Wed, 7 Mar 2018 13:27:35 -0500
> 
> > Not AFAIU, no.  It is an implementation of a different indentation
> > method.  I don't see why we should want to go that way, since there's
> > nothing wrong with the existing algorithm, except for handling of
> > variable-pitch fonts.
> 
> Can you clarify what you call an indentation method? I think that's the source of my confusion.
> I wouldn't call what I wrote an indentation method.  Maybe I'm not understanding you because of that?

I feel that we are somehow bogged down in a miscommunication from
which I see no way out...

An "indentation method" is a method used to determine the indentation
of a line and to insert whitespace in order to achieve that
indentation.

> >> In your world, concretely, what happens when I open xdisp.c and press M-x variable-pitch-mode? Does everything look misaligned until I M-x indent-region the whole file?
> > 
> > Yes.  It isn't different from what happens when you change tab-width.
> 
> But changing tab-width doesn't change the contents of the buffer, whereas what you're advocating for does, right?

No, it doesn't.  I'm advocating a way of changing the width of
whitespace characters on display to account for variable-pitch font,
not any change in the buffer.

> Maybe I just don't understand what you're advocating for.  AFAIU, you're suggesting that we should teach indentation methods about variable-pitch fonts, so that they indent text to a particular pixel offset instead of a particular column.  Is that correct?

Yes, correct.

> If it is correct, then it solves a problem that's different from the one I was thinking in, which is to display existing code with a variable pitch font, without the indentation function.

We cannot possibly DTRT with indentation without an indentation
function.

> * What happens in major modes that don't provide indentation support?

The same as what happens today: the default functions are invoked.

> * What happens in indentation-sensitive languages (Python, F#, Haskell, etc)?

Same as today (the buffer contents will not change from what it is
today).

> * What happens in comments?

Nothing special.

> * Do I need to call M-x indent-region every time I switch between variable-pitch and fixed-pitch?

Not necessarily, not if the idea of controlling the SPC width is
workable.  If it is, then the mode will simply set the value of two
buffer-local variables, calculating them from the font in use.

> * If I close and reopen a file in variable-pitch mode, do I need to re-run M-x indent-region?

See above.

> * In variable-pitch mode, what gets saved to disk after I run M-x indent-region?

Same as what we have now.

Re: Variable-width font indentation

[Clément Pit-Claudel]

On 2018-03-07 15:09, Eli Zaretskii wrote:
>> Cc: emacs-devel@gnu.org, eggert@cs.ucla.edu
>> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
>> Date: Wed, 7 Mar 2018 13:27:35 -0500
>>
>>> Not AFAIU, no.  It is an implementation of a different indentation
>>> method.  I don't see why we should want to go that way, since there's
>>> nothing wrong with the existing algorithm, except for handling of
>>> variable-pitch fonts.
>>
>> Can you clarify what you call an indentation method? I think that's the source of my confusion.
>> I wouldn't call what I wrote an indentation method.  Maybe I'm not understanding you because of that?
> 
> I feel that we are somehow bogged down in a miscommunication from
> which I see no way out...

I think we're making progress, actually :) At least, I'm understanding things better.

> An "indentation method" is a method used to determine the indentation
> of a line and to insert whitespace in order to achieve that
> indentation.

Understood.  Then what I'm offering isn't an indentation method: it doesn't insert whitespace.
But what I'm offering is indeed an imperfect heuristic to determine the indentation of a line.


>> But changing tab-width doesn't change the contents of the buffer, whereas what you're advocating for does, right?
> 
> No, it doesn't.  I'm advocating a way of changing the width of
> whitespace characters on display to account for variable-pitch font,
> not any change in the buffer.

OK, understood.  So for example if I have this snippet:

     01234567
     --------
  0 |x = (y + 
  1 |     z)

…then on line 1 the indentation function would put display properties on each of the 5 spaces leading to `z` to align it with the `y`.  Correct?

That's also what I do, only I compute the width of the spaces without calling the indentation function.

>> If it is correct, then it solves a problem that's different from the one I was thinking in, which is to display existing code with a variable pitch font, without the indentation function.
> 
> We cannot possibly DTRT with indentation without an indentation
> function.

Well, no, we can't DTRT every time without an indentation function.  But we can do fairly good, or at least it looks like that to me given the snippet I posted earlier.

(But note that I said the same thing at first, responding to Paul that his technique couldn't work.)

>> * What happens in major modes that don't provide indentation support?
> 
> The same as what happens today: the default functions are invoked.

I don't understand this part.  Concretely, what will the snippet above look like if it's shown in a prog-mode buffer, in a variable-pitch font?  Will the indentation just be off?

>> * What happens in indentation-sensitive languages (Python, F#, Haskell, etc)?
> 
> Same as today (the buffer contents will not change from what it is
> today).

I meant to ask what the code would look like, not what the buffer contents would change to.

>> * What happens in comments?
> 
> Nothing special.

Same question: what do the contents look like (what text properties do you apply on leading spaces in comments?)
For example, if I have this in a C buffer:

/* FIXME:
   - long
     text
     here
   - more text */

what will this look like in variable-pitch mode?

>> * Do I need to call M-x indent-region every time I switch between variable-pitch and fixed-pitch?
> 
> Not necessarily, not if the idea of controlling the SPC width is
> workable.  If it is, then the mode will simply set the value of two
> buffer-local variables, calculating them from the font in use.

That I don't understand.  Which two values are you thinking of?

As another concrete example, consider this snippet:

(do-stuff a
          b
          c)

won't you need to recompute the width of each leading space every time you change fonts?

>> * If I close and reopen a file in variable-pitch mode, do I need to re-run M-x indent-region?

I don't understand this part either: ow will you determine how to scale each space without calling the indentation function?

>> * In variable-pitch mode, what gets saved to disk after I run M-x indent-region?
> 
> Same as what we have now.

OK.
 

Re: Variable-width font indentation

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org, eggert@cs.ucla.edu
> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Wed, 7 Mar 2018 15:32:53 -0500
> 
> >> But changing tab-width doesn't change the contents of the buffer, whereas what you're advocating for does, right?
> > 
> > No, it doesn't.  I'm advocating a way of changing the width of
> > whitespace characters on display to account for variable-pitch font,
> > not any change in the buffer.
> 
> OK, understood.  So for example if I have this snippet:
> 
>      01234567
>      --------
>   0 |x = (y + 
>   1 |     z)
> 
> …then on line 1 the indentation function would put display properties on each of the 5 spaces leading to `z` to align it with the `y`.  Correct?

Not necessarily display properties, maybe some (most? all?) of the
effect could be achieved by modifying the display width of a space and
a tab according to some buffer-local variable, by the display code
itself, similarly to how we render tab characters now.

> >> * What happens in major modes that don't provide indentation support?
> > 
> > The same as what happens today: the default functions are invoked.
> 
> I don't understand this part.  Concretely, what will the snippet above look like if it's shown in a prog-mode buffer, in a variable-pitch font?  Will the indentation just be off?

When a major mode doesn't provide an indentation function, indent.el
uses the default function, so I'm not sure what problem you see here.

> >> * What happens in indentation-sensitive languages (Python, F#, Haskell, etc)?
> > 
> > Same as today (the buffer contents will not change from what it is
> > today).
> 
> I meant to ask what the code would look like, not what the buffer contents would change to.

Hopefully, the buffer will look like users of those languages expect
it to look.

> >> * What happens in comments?
> > 
> > Nothing special.
> 
> Same question: what do the contents look like (what text properties do you apply on leading spaces in comments?)

I hope we will be able to manage without display properties at all.

> For example, if I have this in a C buffer:
> 
> /* FIXME:
>    - long
>      text
>      here
>    - more text */
> 
> what will this look like in variable-pitch mode?

Hopefully, very similar, although we won't be able to guarantee
column-wise alignment for arbitrary text.  But leading whitespace will
certainly be aligned, something that is not universally true today.

> >> * Do I need to call M-x indent-region every time I switch between variable-pitch and fixed-pitch?
> > 
> > Not necessarily, not if the idea of controlling the SPC width is
> > workable.  If it is, then the mode will simply set the value of two
> > buffer-local variables, calculating them from the font in use.
> 
> That I don't understand.  Which two values are you thinking of?

tab-width and (the hypothetical) space-width.  Actually, only the
latter, since tab-width must be its integral multiple.

> As another concrete example, consider this snippet:
> 
> (do-stuff a
>           b
>           c)
> 
> won't you need to recompute the width of each leading space every time you change fonts?

Hopefully, no.  At least not for approximate alignment.

> >> * If I close and reopen a file in variable-pitch mode, do I need to re-run M-x indent-region?
> 
> I don't understand this part either: ow will you determine how to scale each space without calling the indentation function?

Hopefully, by calculating some representative metrics of the font.

Re: Variable-width font indentation

[Clément Pit-Claudel]

On 2018-03-08 10:39, Eli Zaretskii wrote:
>> From: Clément Pit-Claudel <cpitclaudel@gmail.com>buffer.
>>
>> OK, understood.  So for example if I have this snippet:
>>
>>      01234567
>>      --------
>>   0 |x = (y + 
>>   1 |     z)
>>
>> …then on line 1 the indentation function would put display properties on each of the 5 spaces leading to `z` to align it with the `y`.  Correct?
> 
> Not necessarily display properties, maybe some (most? all?) of the
> effect could be achieved by modifying the display width of a space and
> a tab according to some buffer-local variable, by the display code
> itself, similarly to how we render tab characters now.

Hmm, interesting.  How would that work? For example, if I have these two snippets:

i = (y + 
     z)

w = (y + 
     z)

with monospace fonts 'i' and 'w' have the same width, but with variable-pitch fonts they don't and if you assign a constant width to each space the 'y' and 'z' will line up in at most one of the two examples, right?

>>>> * What happens in major modes that don't provide indentation support?
>>>
>>> The same as what happens today: the default functions are invoked.
>>
>> I don't understand this part.  Concretely, what will the snippet above look like if it's shown in a prog-mode buffer, in a variable-pitch font?  Will the indentation just be off?
> 
> When a major mode doesn't provide an indentation function, indent.el
> uses the default function, so I'm not sure what problem you see here.

I don't know what the default function is, or how it could help us determine proper alignment :/
I thought from earlier emails that you'd use existing indentation functions to determine proper indentation.

>> For example, if I have this in a C buffer:
>>
>> /* FIXME:
>>    - long
>>      text
>>      here
>>    - more text */
>>
>> what will this look like in variable-pitch mode?
> 
> Hopefully, very similar, although we won't be able to guarantee
> column-wise alignment for arbitrary text.  But leading whitespace will
> certainly be aligned, something that is not universally true today.

Ah, that I understand :)
I've been using the following for that purpose for a long time:

  (font-lock-add-keywords nil `(("^[[:space:]]+" 0 '(face fixed-pitch) append)) 'append)

>>>> * Do I need to call M-x indent-region every time I switch between variable-pitch and fixed-pitch?
>>>
>>> Not necessarily, not if the idea of controlling the SPC width is
>>> workable.  If it is, then the mode will simply set the value of two
>>> buffer-local variables, calculating them from the font in use.
>>
>> That I don't understand.  Which two values are you thinking of?
> 
> tab-width and (the hypothetical) space-width.  Actually, only the
> latter, since tab-width must be its integral multiple.

OK, I see.  This won't help with cases like the following, right?

  int main(int argv,
           char** argc)

>> As another concrete example, consider this snippet:
>>
>> (do-stuff a
>>           b
>>           c)
>>
>> won't you need to recompute the width of each leading space every time you change fonts?
> 
> Hopefully, no.  At least not for approximate alignment.

OK. I don't understand how that part works yet.
The solution you propose doesn't get alignment right if I have the following ELisp code, right?

(progn
  (wwwwwwww a
            b
            c)
  (iiiiiiii a
            b
            c))

>>>> * If I close and reopen a file in variable-pitch mode, do I need to re-run M-x indent-region?
>>
>> I don't understand this part either: ow will you determine how to scale each space without calling the indentation function?
> 
> Hopefully, by calculating some representative metrics of the font.

OK.  I think I follow you at this point, although now I'm not sure anymore what role indentation functions will play.  A few emails ago you mentioned teaching indentation functions about variable-pitch, but the solution you're offering now doesn't seem to require that anymore, does it?

Clément.

Re: Variable-width font indentation

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org, eggert@cs.ucla.edu
> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Thu, 8 Mar 2018 11:30:51 -0500
> 
> > Not necessarily display properties, maybe some (most? all?) of the
> > effect could be achieved by modifying the display width of a space and
> > a tab according to some buffer-local variable, by the display code
> > itself, similarly to how we render tab characters now.
> 
> Hmm, interesting.  How would that work? For example, if I have these two snippets:
> 
> i = (y + 
>      z)
> 
> w = (y + 
>      z)
> 
> with monospace fonts 'i' and 'w' have the same width, but with variable-pitch fonts they don't and if you assign a constant width to each space the 'y' and 'z' will line up in at most one of the two examples, right?

Maybe we could find a middle ground, whereby each one of the examples
will approximately align.  If that can be done, and the result is
acceptable, then the problem of recording the text properties in the
file and/or reindenting when the file is revisited goes away.

> >>>> * What happens in major modes that don't provide indentation support?
> >>>
> >>> The same as what happens today: the default functions are invoked.
> >>
> >> I don't understand this part.  Concretely, what will the snippet above look like if it's shown in a prog-mode buffer, in a variable-pitch font?  Will the indentation just be off?
> > 
> > When a major mode doesn't provide an indentation function, indent.el
> > uses the default function, so I'm not sure what problem you see here.
> 
> I don't know what the default function is, or how it could help us determine proper alignment :/

Looking in indent.el, my reading of the code is that the default
function is indent-rigidly.

> I've been using the following for that purpose for a long time:
> 
>   (font-lock-add-keywords nil `(("^[[:space:]]+" 0 '(face fixed-pitch) append)) 'append)

Is it really needed?  With most font back-ends, a tab is displayed as
a stretch glyph whose width is an integral multiple of the width of
the space glyph for the same font.  So they should align even without
the fixed-pitch font trick.

> >>> Not necessarily, not if the idea of controlling the SPC width is
> >>> workable.  If it is, then the mode will simply set the value of two
> >>> buffer-local variables, calculating them from the font in use.
> >>
> >> That I don't understand.  Which two values are you thinking of?
> > 
> > tab-width and (the hypothetical) space-width.  Actually, only the
> > latter, since tab-width must be its integral multiple.
> 
> OK, I see.  This won't help with cases like the following, right?
> 
>   int main(int argv,
>            char** argc)

It might get reasonably close, though.  Maybe this will be good
enough.

> OK.  I think I follow you at this point, although now I'm not sure anymore what role indentation functions will play.  A few emails ago you mentioned teaching indentation functions about variable-pitch, but the solution you're offering now doesn't seem to require that anymore, does it?

That's more relevant for text-derived modes, where indentation levels
are rigid and not determined by previous lines.  There we could do a
better job, I hope.

Re: Variable-width font indentation

[Clément Pit-Claudel]

On 2018-03-08 14:07, Eli Zaretskii wrote:
>> Cc: emacs-devel@gnu.org, eggert@cs.ucla.edu
>> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
>> Date: Thu, 8 Mar 2018 11:30:51 -0500
>>
>>> Not necessarily display properties, maybe some (most? all?) of the
>>> effect could be achieved by modifying the display width of a space and
>>> a tab according to some buffer-local variable, by the display code
>>> itself, similarly to how we render tab characters now.
>>
>> Hmm, interesting.  How would that work? For example, if I have these two snippets:
>>
>> i = (y + 
>>      z)
>>
>> w = (y + 
>>      z)
>>
>> with monospace fonts 'i' and 'w' have the same width, but with variable-pitch fonts they don't and if you assign a constant width to each space the 'y' and 'z' will line up in at most one of the two examples, right?
> 
> Maybe we could find a middle ground, whereby each one of the examples
> will approximately align.  If that can be done, and the result is
> acceptable, then the problem of recording the text properties in the
> file and/or reindenting when the file is revisited goes away.

Maybe :)  I don't know how good it would look, in practice, but it's easy to experiment.
Paul's algorithm is not too bad in that sense: these two examples align pixel-perfectly.

>> I've been using the following for that purpose for a long time:
>>
>>   (font-lock-add-keywords nil `(("^[[:space:]]+" 0 '(face fixed-pitch) append)) 'append)
> 
> Is it really needed?  With most font back-ends, a tab is displayed as
> a stretch glyph whose width is an integral multiple of the width of
> the space glyph for the same font.  So they should align even without
> the fixed-pitch font trick.

Hmm, indeed.  So which problems did you have in mind when you said the following?

> But leading whitespace will certainly be aligned, something that is 
> not universally true today.
>> OK, I see.  This won't help with cases like the following, right?
>>
>>   int main(int argv,
>>            char** argc)
> 
> It might get reasonably close, though.  Maybe this will be good
> enough.

OK

>> OK.  I think I follow you at this point, although now I'm not sure anymore what role indentation functions will play.  A few emails ago you mentioned teaching indentation functions about variable-pitch, but the solution you're offering now doesn't seem to require that anymore, does it?
> 
> That's more relevant for text-derived modes, where indentation levels
> are rigid and not determined by previous lines.  There we could do a
> better job, I hope.

OK.  I tend to think of indentation even in text modes as being somewhat previous-lines dependent (wrapped paragraphs align relative to bullet points, for example), but I agree that it isn't as much as in programming modes.

Re: Variable-width font indentation

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org, eggert@cs.ucla.edu
> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Thu, 8 Mar 2018 14:55:20 -0500
> 
> > Maybe we could find a middle ground, whereby each one of the examples
> > will approximately align.  If that can be done, and the result is
> > acceptable, then the problem of recording the text properties in the
> > file and/or reindenting when the file is revisited goes away.
> 
> Maybe :)  I don't know how good it would look, in practice, but it's easy to experiment.
> Paul's algorithm is not too bad in that sense: these two examples align pixel-perfectly.

Yes, but as I said, I don't think that approach is appropriate for
code indentation, because leading whitespace in code blocks _must_
always align, IMO.  IOW, that approach seems to favor alignment of
multi-line code sequences to alignment of indented blocks, and I think
programming modes should have the opposite preference.

> > Is it really needed?  With most font back-ends, a tab is displayed as
> > a stretch glyph whose width is an integral multiple of the width of
> > the space glyph for the same font.  So they should align even without
> > the fixed-pitch font trick.
> 
> Hmm, indeed.  So which problems did you have in mind when you said the following?
> 
> > But leading whitespace will certainly be aligned, something that is 
> > not universally true today.

At least the w32 font back-end doesn't fulfill that requirement,
because it calculates the font's space width differently (due to
practical problems related to the MS-Windows APIs).  Not sure about
ns.

> > That's more relevant for text-derived modes, where indentation levels
> > are rigid and not determined by previous lines.  There we could do a
> > better job, I hope.
> 
> OK.  I tend to think of indentation even in text modes as being somewhat previous-lines dependent (wrapped paragraphs align relative to bullet points, for example), but I agree that it isn't as much as in programming modes.

IME, indentation of text in word processors I use does not depend on
previous lines at all.

Re: Variable-width font indentation

[Paul Eggert]

On 03/09/2018 12:30 AM, Eli Zaretskii wrote:
> I don't think that approach is appropriate for
> code indentation, because leading whitespace in code blocks_must_
> always align, IMO.

I don't quite follow this objection, since the prototype does align 
leading whitespace in code blocks. That is, in any particular block of 
code consisting of lines prefaced by equivalent white space, the 
approach displays the block with equally-indented lines, so the leading 
whitespace does align.

If the objection is that different blocks of code are indented 
differently (e.g., an if-part might be indented slightly differently 
from the corresponding then-part), then I agree that might irritate some 
users. However, the irritation is quite small to me, and I imagine it 
wouldn't bother other users much either, in the sense that they'll be 
willing to put up with this minor irritation in order to get the 
benefits that the algorithm has elsewhere. So it still sounds like a 
reasonable thing to support, if a user wants to use it.

Of course other display algorithms are possible, but I haven't seen any 
other improvement proposed for variable-pitch fonts that is nearly as 
simple and easy to understand. Simplicity is a virtue here.

Re: Variable-width font indentation

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Fri, 9 Mar 2018 09:52:28 -0800
> 
> If the objection is that different blocks of code are indented 
> differently (e.g., an if-part might be indented slightly differently 
> from the corresponding then-part), then I agree that might irritate some 
> users. However, the irritation is quite small to me, and I imagine it 
> wouldn't bother other users much either

Well, it bothers me a lot, to the degree that I don't think it could
fly.

> Of course other display algorithms are possible, but I haven't seen any 
> other improvement proposed for variable-pitch fonts that is nearly as 
> simple and easy to understand. Simplicity is a virtue here.

Simplicity is a virtue if it gets the job done.  This particular
suggestion doesn't, IMO.

Re: Variable-width font indentation

[Paul Eggert]

On 03/09/2018 10:35 AM, Eli Zaretskii wrote:
> it bothers me a lot, to the degree that I don't think it could
> fly

I've tried it and it flies for me, and it doesn't appear that I'm alone 
in this opinion. Although it's not perfect, it's significantly better 
than what Emacs does now for variable-pitch fonts. So I'm in favor of 
having this feature available as a user option.

Re: Variable-width font indentation

[Eli Zaretskii]

> Cc: cpitclaudel@gmail.com, emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Fri, 9 Mar 2018 12:16:54 -0800
> 
> On 03/09/2018 10:35 AM, Eli Zaretskii wrote:
> > it bothers me a lot, to the degree that I don't think it could
> > fly
> 
> I've tried it and it flies for me, and it doesn't appear that I'm alone 
> in this opinion.

I'm not alone in my opinion, either.

Re: Variable-width font indentation

[Paul Eggert]

Eli Zaretskii wrote:
>> I've tried it and it flies for me, and it doesn't appear that I'm alone
>> in this opinion.
> I'm not alone in my opinion, either.

I've heard other opinions that the approach is not to their taste. But I haven't 
heard anyone else say that the approach should not be available, even as a user 
option.

Re: Variable-width font indentation

[James Cloos]

I've long thought that a good start is to show the initial whitespace in
fixed width.

That doesn't cover aligning things like the right column of *-boxes, but
is a good start and has low-complexity.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 0x997A9F17ED7DAEA6

Re: Variable-width font indentation

[Eli Zaretskii]

> From: James Cloos <cloos@jhcloos.com>
> Cc: Paul Eggert <eggert@cs.ucla.edu>,  cpitclaudel@gmail.com,  emacs-devel@gnu.org
> Date: Fri, 09 Mar 2018 19:09:23 -0500
> 
> I've long thought that a good start is to show the initial whitespace in
> fixed width.
> 
> That doesn't cover aligning things like the right column of *-boxes, but
> is a good start and has low-complexity.

I'm curious: why do you think this would be a good start?  AFAIU, it
will cause misalignment with any font/font size other than the one
where it fits the actual size of the font (assuming that font is
fixed-pitch), so how can it be a step in the right direction?  I think
initial whitespace should rather change its size to match the size of
the font---then it will produce nicely indented display with any font.

Or maybe I misunderstand what you mean by "fixed width".  You did mean
fixed width in pixels, yes?  If not, then what did you mean?

Re: Variable-width font indentation

[James Cloos]

>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:

EZ> Or maybe I misunderstand what you mean by "fixed width".  You did mean
EZ> fixed width in pixels, yes?  If not, then what did you mean?

Yes.  By fixed-width I mean the font the buffer would use were variable-
pitch-mode not in effect.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 0x997A9F17ED7DAEA6

Re: Variable-width font indentation

[Eli Zaretskii]

> From: James Cloos <cloos@jhcloos.com>
> Cc: eggert@cs.ucla.edu,  cpitclaudel@gmail.com,  emacs-devel@gnu.org
> Date: Sat, 10 Mar 2018 14:48:33 -0500
> 
> >>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:
> 
> EZ> Or maybe I misunderstand what you mean by "fixed width".  You did mean
> EZ> fixed width in pixels, yes?  If not, then what did you mean?
> 
> Yes.  By fixed-width I mean the font the buffer would use were variable-
> pitch-mode not in effect.

Then I really don't understand why you think it's a good start.  I
think it's a step in the wrong direction, because it will make things
worse than they are now: what we have now makes sure the initial
whitespace changes in sync on all lines of the buffer, so that the
first non-whitespace characters on each line stay aligned.

Re: Variable-width font indentation

[James Cloos]

>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:

EZ> Then I really don't understand why you think it's a good start.  I
EZ> think it's a step in the wrong direction, because it will make things
EZ> worse than they are now: what we have now makes sure the initial
EZ> whitespace changes in sync on all lines of the buffer, so that the
EZ> first non-whitespace characters on each line stay aligned.

I have not re-compiled in a while, nor did I read the whole thread.
Perhaps things have progressed since the version I currently have.

Consistent and not-too-small indenting is important.  Basing indents on
the width of the fixed pitch space and tab stops on a multiple of that
would help, compared to what the version I have does.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 0x997A9F17ED7DAEA6

Re: Variable-width font indentation

[Eli Zaretskii]

> From: James Cloos <cloos@jhcloos.com>
> Cc: eggert@cs.ucla.edu,  cpitclaudel@gmail.com,  emacs-devel@gnu.org
> Date: Sat, 10 Mar 2018 17:18:12 -0500
> 
> >>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:
> 
> EZ> Then I really don't understand why you think it's a good start.  I
> EZ> think it's a step in the wrong direction, because it will make things
> EZ> worse than they are now: what we have now makes sure the initial
> EZ> whitespace changes in sync on all lines of the buffer, so that the
> EZ> first non-whitespace characters on each line stay aligned.
> 
> I have not re-compiled in a while, nor did I read the whole thread.
> Perhaps things have progressed since the version I currently have.
> 
> Consistent and not-too-small indenting is important.  Basing indents on
> the width of the fixed pitch space and tab stops on a multiple of that
> would help, compared to what the version I have does.

I don't think anything's happened in this regard lately.  A tab was
always rendered on GUI frames as a stretch whose width is an integral
multiple of the space width of the font.  Except in the Windows port,
the space width is actually the width of the font glyph that displays
the space character.  So I'm not sure I understand what problems you
describe.

Can you provide a concrete example where the behavior you describe
produces bad results?

Re: Variable-width font indentation

[Clément Pit-Claudel]

On 2018-03-09 03:30, Eli Zaretskii wrote:
>> Cc: emacs-devel@gnu.org, eggert@cs.ucla.edu
>> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
>> Date: Thu, 8 Mar 2018 14:55:20 -0500
>>
>>> Maybe we could find a middle ground, whereby each one of the examples
>>> will approximately align.  If that can be done, and the result is
>>> acceptable, then the problem of recording the text properties in the
>>> file and/or reindenting when the file is revisited goes away.
>>
>> Maybe :)  I don't know how good it would look, in practice, but it's easy to experiment.
>> Paul's algorithm is not too bad in that sense: these two examples align pixel-perfectly.
> 
> Yes, but as I said, I don't think that approach is appropriate for
> code indentation, because leading whitespace in code blocks _must_
> always align, IMO.  IOW, that approach seems to favor alignment of
> multi-line code sequences to alignment of indented blocks, and I think
> programming modes should have the opposite preference.

Indeed :) I was mostly of the same opinion, but I find the actual results pretty OK.

>>> That's more relevant for text-derived modes, where indentation levels
>>> are rigid and not determined by previous lines.  There we could do a
>>> better job, I hope.
>>
>> OK.  I tend to think of indentation even in text modes as being somewhat previous-lines dependent (wrapped paragraphs align relative to bullet points, for example), but I agree that it isn't as much as in programming modes.
> 
> IME, indentation of text in word processors I use does not depend on
> previous lines at all.

Except in lists, right? For example it's not uncommon to wrap like this:

* Some long text here
  that wraps around:

  i.   Point 1
       … continued
  ii.  Point 2
       … continued
  iii. Point 3
       … continued

Clément

Re: Variable-width font indentation

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org, eggert@cs.ucla.edu
> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Fri, 9 Mar 2018 14:48:33 -0500
> 
> > IME, indentation of text in word processors I use does not depend on
> > previous lines at all.
> 
> Except in lists, right?

No, in lists as well.

> For example it's not uncommon to wrap like this:
> 
> * Some long text here
>   that wraps around:
> 
>   i.   Point 1
>        … continued
>   ii.  Point 2
>        … continued
>   iii. Point 3
>        … continued

What I meant was that if you, for example, enlarge the font, the
indentation doesn't get larger in pixels.  (I'm not sure we should
follow that extreme example, but it does drive the point home, I
think.)

Re: Variable-width font indentation

[Clément Pit-Claudel]

On 2018-03-09 16:18, Eli Zaretskii wrote:
>> Cc: emacs-devel@gnu.org, eggert@cs.ucla.edu
>> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
>> Date: Fri, 9 Mar 2018 14:48:33 -0500
>>
>>> IME, indentation of text in word processors I use does not depend on
>>> previous lines at all.
>>
>> Except in lists, right?
> 
> No, in lists as well.

I don't understand: in the example below, the widths of the bullets and the numbers (i, ii, …) affects the indentation of the following lines, right?

In any case, I agree it's a fairly limited issue.  I don't think it matters much.

>> For example it's not uncommon to wrap like this:
>>
>> * Some long text here
>>   that wraps around:
>>
>>   i.   Point 1
>>        … continued
>>   ii.  Point 2
>>        … continued
>>   iii. Point 3
>>        … continued
> 
> What I meant was that if you, for example, enlarge the font, the
> indentation doesn't get larger in pixels.  (I'm not sure we should
> follow that extreme example, but it does drive the point home, I
> think.)

Re: Variable-width font indentation

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org, eggert@cs.ucla.edu
> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Fri, 9 Mar 2018 17:54:11 -0500
> 
> >>> IME, indentation of text in word processors I use does not depend on
> >>> previous lines at all.
> >>
> >> Except in lists, right?
> > 
> > No, in lists as well.
> 
> I don't understand: in the example below, the widths of the bullets and the numbers (i, ii, …) affects the indentation of the following lines, right?

No, it doesn't.  It surprised me to see that in MS Word, but the
evidence is clear.  Not sure we should follow suit, but it does tell
us how far this could be taken.

> In any case, I agree it's a fairly limited issue.  I don't think it matters much.

Agreed.

Re: Variable-width font indentation

[Yuri Khan]

On Sat, Mar 10, 2018 at 2:48 AM, Clément Pit-Claudel
<cpitclaudel@gmail.com> wrote:

> Except in lists, right? For example it's not uncommon to wrap like this:
>
> * Some long text here
>   that wraps around:
>
>   i.   Point 1
>        … continued
>   ii.  Point 2
>        … continued
>   iii. Point 3
>        … continued

The way word processors implement this is:

1. Specify a left margin of, say, 0.5in. This makes each line start at
that offset.

2. Specify a first line indentation of, say, -0.25in. This way the
list marker starts at 0.25in, immediately followed by the rest of the
first line. Each subsequent line starts at the left margin 0.5in.

3. To make the item text of the first line align with the rest of the
paragraph, put a tabulation between the marker and the item text.
Align that tabulation at 0.5in.


The way HTML+CSS implement this is (roughly):

1. Specify a left margin for the whole list, say, 0.5in.

2. Treat list markers as special absolutely positioned elements. Put
them at, say, (-0.25in across, +0in down) from the start of list item.
(Absolutely positioned elements do not affect the layout of the
surrounding text, so list item text is aligned normally.)


In both cases, if the list marker becomes really long and no longer
fits its allotted space of 0.25in, there will be display artifacts. In
Word-alikes, the item text will start at the *next* tab stop. In
HTML+CSS, item text will visually overlap the marker.

Re: Variable-width font indentation

[Eli Zaretskii]

> From: Yuri Khan <yuri.v.khan@gmail.com>
> Date: Sat, 10 Mar 2018 11:27:39 +0700
> Cc: Eli Zaretskii <eliz@gnu.org>, Paul Eggert <eggert@cs.ucla.edu>, 
> 	Emacs developers <emacs-devel@gnu.org>
> 
> In both cases, if the list marker becomes really long and no longer
> fits its allotted space of 0.25in, there will be display artifacts. In
> Word-alikes, the item text will start at the *next* tab stop.

My point was that the _subsequent_ lines still stay at the same 0.5"
offset, no matter what happens on the first line.  Which may or may
not make sense, but it's good to know what "prior art" is in this
case.

Re: Variable-width font indentation

[Yuri Khan]

On Sat, Mar 10, 2018 at 3:39 PM, Eli Zaretskii <eliz@gnu.org> wrote:

>> In both cases, if the list marker becomes really long and no longer
>> fits its allotted space of 0.25in, there will be display artifacts. In
>> Word-alikes, the item text will start at the *next* tab stop.
>
> My point was that the _subsequent_ lines still stay at the same 0.5"
> offset, no matter what happens on the first line.  Which may or may
> not make sense, but it's good to know what "prior art" is in this
> case.

Yes. I’m actually confirming your claim that indentation of subsequent
lines is not affected by the width of the marker.

Re: Variable-width font indentation

[Eli Zaretskii]

> From: Yuri Khan <yuri.v.khan@gmail.com>
> Date: Sat, 10 Mar 2018 18:42:57 +0700
> Cc: Clément Pit-Claudel <cpitclaudel@gmail.com>, 
> 	Paul Eggert <eggert@cs.ucla.edu>, Emacs developers <emacs-devel@gnu.org>
> 
> > My point was that the _subsequent_ lines still stay at the same 0.5"
> > offset, no matter what happens on the first line.  Which may or may
> > not make sense, but it's good to know what "prior art" is in this
> > case.
> 
> Yes. I’m actually confirming your claim that indentation of subsequent
> lines is not affected by the width of the marker.

Ah, okay.  Thanks.

Re: Variable-width font indentation

[Clément Pit-Claudel]

On 2018-03-09 23:27, Yuri Khan wrote:
> 1. Specify a left margin of, say, 0.5in. This makes each line start at
> that offset.

Of course; I didn't realize we were talking about teaching Emacs to do this kind of advanced positioning (I thought we were still discussing pure text documents)

Re: Variable-width font indentation

[Stefan Monnier]

>> In your world, concretely, what happens when I open xdisp.c and press M-x
>> variable-pitch-mode? Does everything look misaligned until I M-x
>> indent-region the whole file?
> Yes.  It isn't different from what happens when you change tab-width.

It's different in one important way:
- it's considered normal to impose that everyone who looks/edits this
  file needs to have tad-width set to a particular value (8,
  traditionally).
- it's not considered normal to impose that everyone who looks/edits
  this file needs to use the same *font*.
Also, you can easily circumvent the tag-width issue by not using TABs in
your file, whereas it's hard to have a file whose appearance is not
affected by the choice of font.


        Stefan

Re: Variable-width font indentation

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Wed, 07 Mar 2018 23:50:45 -0500
> 
> >> In your world, concretely, what happens when I open xdisp.c and press M-x
> >> variable-pitch-mode? Does everything look misaligned until I M-x
> >> indent-region the whole file?
> > Yes.  It isn't different from what happens when you change tab-width.
> 
> It's different in one important way:
> - it's considered normal to impose that everyone who looks/edits this
>   file needs to have tad-width set to a particular value (8,
>   traditionally).
> - it's not considered normal to impose that everyone who looks/edits
>   this file needs to use the same *font*.

The above doesn't force any particular font, just a fixed-pitch one, a
limitation we already have today.  So we will be no worse.

> Also, you can easily circumvent the tag-width issue by not using TABs in
> your file, whereas it's hard to have a file whose appearance is not
> affected by the choice of font.

Yes, using variable-pitch fonts in code where indentation is important
has its disadvantages, and always will.  Any solution will have the
same issues, just under different use cases and situations.  That's
why this is not the main use case where we should support
variable-pitch fonts well, the main use case is in text-related modes,
where in many/most cases the indentation is rigid or changes
proportionally with the width of some canonical character.

Re: Variable-width font indentation

[Paul Eggert]

On 03/08/2018 05:44 AM, Eli Zaretskii wrote:
> Yes, using variable-pitch fonts in code where indentation is important
> has its disadvantages, and always will.  Any solution will have the
> same issues, just under different use cases and situations.  That's
> why this is not the main use case

Although it has disadvantages, it shouldn't be that hard to do it better 
than Emacs does it now (along the lines that Clément prototyped), and if 
Emacs had these improvements the benefits of variable-pitch fonts would 
outweigh the costs in many practical programming applications. 
Programming in variable-width fonts makes more-efficient use of valuable 
screen real estate, so it's worth doing when it works reasonably well, 
as it often would if Emacs supported it better.

Re: Variable-width font indentation

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Thu, 8 Mar 2018 14:03:21 -0800
> 
> On 03/08/2018 05:44 AM, Eli Zaretskii wrote:
> > Yes, using variable-pitch fonts in code where indentation is important
> > has its disadvantages, and always will.  Any solution will have the
> > same issues, just under different use cases and situations.  That's
> > why this is not the main use case
> 
> Although it has disadvantages, it shouldn't be that hard to do it better 
> than Emacs does it now (along the lines that Clément prototyped), and if 
> Emacs had these improvements the benefits of variable-pitch fonts would 
> outweigh the costs in many practical programming applications. 

I agree (although I don't think we should use the basic idea of
calculating the indentation that the prototype used).

> Programming in variable-width fonts makes more-efficient use of valuable 
> screen real estate, so it's worth doing when it works reasonably well, 
> as it often would if Emacs supported it better.

Agreed.

Do people think adding a per-buffer space-width variable would be a
good step in this direction?  It should be easy to add, I think.

Re: Variable-width font indentation

[Clément Pit-Claudel]

On 2018-03-09 03:34, Eli Zaretskii wrote:
> Do people think adding a per-buffer space-width variable would be a
> good step in this direction?  It should be easy to add, I think.

It does sound convenient.  Will it apply only to leading whitespace, or also to inter-word whitespace?

Also, do we need it on the C side?  It sounds fairly easy to achieve with existing tools like font-lock + display properties.

Clément.

Re: Variable-width font indentation

[Eli Zaretskii]

> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Fri, 9 Mar 2018 11:05:13 -0500
> 
> On 2018-03-09 03:34, Eli Zaretskii wrote:
> > Do people think adding a per-buffer space-width variable would be a
> > good step in this direction?  It should be easy to add, I think.
> 
> It does sound convenient.  Will it apply only to leading whitespace, or also to inter-word whitespace?

I'd say it should apply everywhere.  It would be confusing to have SPC
take a different number of pixels depending on where it is.

> Also, do we need it on the C side?  It sounds fairly easy to achieve with existing tools like font-lock + display properties.

Display properties have the disadvantage that they cannot be easily
saved.  Basically, having this in the display code is convenient for
the same reason the TAB display is handled in C.

Re: Variable-width font alignment

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
>> Date: Fri, 9 Mar 2018 11:05:13 -0500
>> 
>> On 2018-03-09 03:34, Eli Zaretskii wrote:
>> > Do people think adding a per-buffer space-width variable would be a
>> > good step in this direction?  It should be easy to add, I think.
>> 
>> It does sound convenient. Will it apply only to leading whitespace,
>> or also to inter-word whitespace?
>
> I'd say it should apply everywhere.  It would be confusing to have SPC
> take a different number of pixels depending on where it is.

Right. Unless we need something else to fix the intra-line align
function - ie align '=>' in 

            Trace                    => Verbosity > 1,
            Put_Parse_Table          => Verbosity > 0,
            Ignore_Unused_Tokens     => Verbosity > 1,
            Ignore_Unknown_Conflicts => Verbosity > 1);

It seems to me that implementing this in a variable width font requires some
sort of align-region-local tab setting, which is a change to current align
algorithms (which just insert spaces).

I use this feature extensively in Ada code, so I would not switch to
variable width font without it.

One step in align.el `align-areas' computes an `align-column'. In
variable-pitch-mode, we could convert that into a tab setting by
computing the pixel column of that character column in the line with the
least whitespace/longest text ("Ignore_Unknown_Conflicts =>" above).

I could try doing this. What is the best approach to implementing an
align-region-local tab setting?

-- 
-- Stephe

Re: Variable-width font alignment

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Date: Sat, 10 Mar 2018 02:12:57 -0600
> 
>             Trace                    => Verbosity > 1,
>             Put_Parse_Table          => Verbosity > 0,
>             Ignore_Unused_Tokens     => Verbosity > 1,
>             Ignore_Unknown_Conflicts => Verbosity > 1);
> 
> It seems to me that implementing this in a variable width font requires some
> sort of align-region-local tab setting, which is a change to current align
> algorithms (which just insert spaces).

Whatever we do to adapt indentation to variable-pitch fonts, we _must_
still begin by inserting spaces so that the same text looks aligned
with fixed-pitch fonts.  Otherwise, we will be unable to let other
people view our code/documents in other editors which use fixed-pitch
fonts, or even in Emacs with fixed-pitch fonts.

The features that align text when variable-pitch fonts are in use must
work _on_top_ of the "basic" alignment that just uses spaces and tabs
as we do today.

> One step in align.el `align-areas' computes an `align-column'. In
> variable-pitch-mode, we could convert that into a tab setting by
> computing the pixel column of that character column in the line with the
> least whitespace/longest text ("Ignore_Unknown_Conflicts =>" above).
> 
> I could try doing this. What is the best approach to implementing an
> align-region-local tab setting?

Any local change in whitespace width would need to use 'display' text
properties, such as '(space :width (N))'.  The problem with that is
that you cannot easily record this when you save the file, so these
properties will have to be recomputed anew each time the file is
visited.  Which will take us down the path of font-lock and JIT lock,
something that I'd like to avoid.  We could also record that in a
separate file, but I'm not sure such meta-data approach will be
acceptable.

Another idea would be to modify align-areas such that it attempts to
find an align-column that will align given text without changing the
width of space and tab, by looking for some kind of LCD for the widths
of the text strings involved in this.  Not sure if this is workable,
though.

Re: Variable-width font alignment

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stephen Leake <stephen_leake@stephe-leake.org>
>> Date: Sat, 10 Mar 2018 02:12:57 -0600
>> 
>>             Trace                    => Verbosity > 1,
>>             Put_Parse_Table          => Verbosity > 0,
>>             Ignore_Unused_Tokens     => Verbosity > 1,
>>             Ignore_Unknown_Conflicts => Verbosity > 1);
>> 
>> It seems to me that implementing this in a variable width font requires some
>> sort of align-region-local tab setting, which is a change to current align
>> algorithms (which just insert spaces).
>
> Whatever we do to adapt indentation to variable-pitch fonts, we _must_
> still begin by inserting spaces so that the same text looks aligned
> with fixed-pitch fonts.  Otherwise, we will be unable to let other
> people view our code/documents in other editors which use fixed-pitch
> fonts, or even in Emacs with fixed-pitch fonts.
>
> The features that align text when variable-pitch fonts are in use must
> work _on_top_ of the "basic" alignment that just uses spaces and tabs
> as we do today.

Right.

>> One step in align.el `align-areas' computes an `align-column'. In
>> variable-pitch-mode, we could convert that into a tab setting by
>> computing the pixel column of that character column in the line with the
>> least whitespace/longest text ("Ignore_Unknown_Conflicts =>" above).
>> 
>> I could try doing this. What is the best approach to implementing an
>> align-region-local tab setting?
>
> Any local change in whitespace width would need to use 'display' text
> properties, such as '(space :width (N))'.  

Ok; that doesn't need an explicit tab.

> The problem with that is that you cannot easily record this when you
> save the file, so these properties will have to be recomputed anew
> each time the file is visited. 

Right. For example, that happens now in info files; formatting is
applied to the links via text properties.

> Which will take us down the path of font-lock and JIT lock, something
> that I'd like to avoid. 

That's how align.el works now. We could modify it to only set the text
properties if the fixed-width spaces are correct, or when in a read-only
buffer, or in variable-pitch-mode.

On the other hand, the ada-mode currently in GELPA computes indentation
and faces from a parse of the source text. It works very well, as long
as the source is syntactically correct. I'm working on implementing
error recovery in the parser; it works fairly well at the moment, and
I'm still improving it.

It should be possible to compute alignments from the parse as well; that
would elminate some of the minor bugs in current ada-mode alignment. That
would still require a parse when visiting the file.

> We could also record that in a separate file, but I'm not sure such
> meta-data approach will be acceptable.

Computing the align properties from the source has to be fast enough to
be acceptable while editing (as align.el is now), so it will be
acceptable on visiting.

-- 
-- Stephe

Re: Variable-width font alignment

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Date: Sat, 10 Mar 2018 11:30:53 -0600
> 
> Computing the align properties from the source has to be fast enough to
> be acceptable while editing (as align.el is now), so it will be
> acceptable on visiting.

Yes, if this is feasible.  Will it be fast enough to handle large
sources files, such as xdisp.c, for example?

Re: Variable-width font indentation

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Paul Eggert <eggert@cs.ucla.edu>
>
>> Programming in variable-width fonts makes more-efficient use of valuable 
>> screen real estate, so it's worth doing when it works reasonably well, 
>> as it often would if Emacs supported it better.
>
> Agreed.
>
> Do people think adding a per-buffer space-width variable would be a
> good step in this direction?  It should be easy to add, I think.

+1

-- 
-- Stephe

Re: Variable-width font indentation

[Stefan Monnier]

> I don't know of a good solution to this problem: in general, there's not
> enough information to distinguish 'indentation' spaces (the two spaces
> before `printf (', `pr ()', and `return 0') and 'line-up' spaces (the spaces
> before `char ()').

Indeed, you'd need to ask the indenter for the why this particular was
chosen (where the answer will usually be of the form "indentation =
OFFSET + column_of_position POS").

Of course, the additional problem is that the indenter is not
necessarily Emacs's indent-line-function.


        Stefan

Re: Variable-width font indentation

[Clément Pit-Claudel]

On 2018-03-06 11:35, Stefan Monnier wrote:
> Of course, the additional problem is that the indenter is not
> necessarily Emacs's indent-line-function.

Exactly

Re: Variable-width font indentation (was: What improvements would be truly useful?)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > This has come up several times, and I think I asked at least once what
  > does it need to entail.

It means, for one thing, that M-q can fill text with a smooth
right margin.

Has that been implemented yet?

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: Variable-width font indentation

[Stefan Monnier]

>   > This has come up several times, and I think I asked at least once what
>   > does it need to entail.
> It means, for one thing, that M-q can fill text with a smooth
> right margin.
> Has that been implemented yet?

AFAIK `shr` does that, yes.


        Stefan

Re: Variable-width font indentation

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Tue, 06 Mar 2018 11:04:41 -0500
> 
> >   > This has come up several times, and I think I asked at least once what
> >   > does it need to entail.
> > It means, for one thing, that M-q can fill text with a smooth
> > right margin.
> > Has that been implemented yet?
> 
> AFAIK `shr` does that, yes.

No, AFAIK it doesn't.  What it does is modify the fill algorithm to
measure the actual width of text, without assuming fixed-pitch font.
But it doesn't produce smooth right margin by inserting stretch
glyphs.

Re: Variable-width font indentation

[Stefan Monnier]

> No, AFAIK it doesn't.  What it does is modify the fill algorithm to
> measure the actual width of text, without assuming fixed-pitch font.
> But it doesn't produce smooth right margin by inserting stretch
> glyphs.

Don't know if it does do it or not, indeed, but AFAIU the algorithm it
uses does give it the pixel info needed to do full-justification.


        Stefan

Re: Variable-width font indentation

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > No, AFAIK it doesn't.  What it does is modify the fill algorithm to
  > measure the actual width of text, without assuming fixed-pitch font.
  > But it doesn't produce smooth right margin by inserting stretch
  > glyphs.

Given that this part has already been implemented, 
we should make M-q do it too.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: Variable-width font indentation

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > It means, for one thing, that M-q can fill text with a smooth
  > > right margin.
  > > Has that been implemented yet?

  > AFAIK `shr` does that, yes.

I don't think it does THIS job.

shr is a facility that renders HTML.
I'm talking about using M-q to do filling
that looks good with variable-width fonts.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: Variable-width font indentation

[Yuri Khan]

On Wed, Mar 7, 2018 at 3:55 AM, Richard Stallman <rms@gnu.org> wrote:

> shr is a facility that renders HTML.
> I'm talking about using M-q to do filling
> that looks good with variable-width fonts.

For word processing purposes, you should not need manual or
semi-automated filling such as M-q provides. Proper word processing
operates two level of abstraction higher.

Level +0: you directly affect where words end up using spaces and line
feeds. This is the level that M-q works on.

Level +1: you specify a paper size for the document or a sections,
and, for each paragraph, a left margin, a right margin, a font, and a
paragraph alignment (left, center, right, or justified). The layout
engine automatically chops the paragraph into lines for display and
print purposes according to these settings.

Level +2: you do not specify individual formatting settings on each
paragraph; instead, you specify a paragraph style that specifies all
these things. (Like Emacs faces provide a level of indirection for
face attributes.)

So: you don’t fill a paragraph with a left margin of 30 spaces and a
right margin at position 72. You designate the paragraph as Addressee,
and the Addressee style specifies a left margin of 50% paper width and
inherits its right margin of 1 inch from the Default style.

All of which is, of course, achievable with HTML + CSS (or a moral
equivalent). WYSIWYG editing for that is possible but 20 years of
observation show that users of WYSIWYG systems tend to produce (at
best) level +1 documents rather than +2.

Re: Variable-width font indentation (was: What improvements would be truly useful?)

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> CC: raeburn@raeburn.org, eggert@cs.ucla.edu, dancol@dancol.org,
> 	emacs-devel@gnu.org
> Date: Mon, 05 Mar 2018 18:05:11 -0500
> 
> It means, for one thing, that M-q can fill text with a smooth
> right margin.
> 
> Has that been implemented yet?

No, not yet.  But unless I'm missing something, doing that would be
relatively simple: we need a variant of justify-current-line which
will (a) compute the number of blanks as a float number, and (b) add a
'display' property to each blank with an :align-to attribute computed
(in pixels) to produce the smooth right margin.

Patches to that effect are welcome.

Re: Variable-width font indentation (was: What improvements would be truly useful?)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > It means, for one thing, that M-q can fill text with a smooth
  > > right margin.
  > > 
  > > Has that been implemented yet?

  > No, not yet.  But unless I'm missing something, doing that would be
  > relatively simple: we need a variant of justify-current-line which
  > will (a) compute the number of blanks as a float number, and (b) add a
  > 'display' property to each blank with an :align-to attribute computed
  > (in pixels) to produce the smooth right margin.

If it is that easy, it means that we already have done more
of this job than I thought we did.  That would be good news.

Would someone please implement this?  Whenever the buffer
has characters of different widths, it ought to fill them
with attention to their widths.

Once this is done, we can see what word-processing feature to add next.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: What improvements would be truly useful?

[Daniele Nicolodi]

On 3/5/18 6:11 AM, Richard Stallman wrote:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
> An improvement in GC wouldn't be a bad thing, but it may not be worth
> the effort.  It is likely to lead to many bugs that would be hard to
> fix.  Once working, it would not make much difference to users.
> It would only permit some operations on larger problems than now.
> 
> When I was working at the AI Lab, one of the older programmers told me
> that hackers are often eager to make improvements of this sort: which
> make the program better in an abstract sense, but not better for
> users.  I took that advice to heart.  Now I pass it on.
> 
> Changing Emacs to handle indentation and alignment with
> variable-width fonts would be an important and useful change.
> Certain kinds of use would make sense, which currently don't.
> 
> It would be a big step towards making Emacs do the job of
> a word processor, which is what I would like to see some day.
> Imagine if you could edit nicely formatted documents directly
> with Emacs, instead of using LibreOffice?  LibreOffice is
> fine to use, it is free software, but it isn't Emacs.

Richard,

with all the due respect for your role in starting the Emacs project and
the Free software movement, I don't think that dismissing contributions
to Emacs on the base of their relative usefulness to implementing new
features that you think are important, does any good to Emacs.

Hackers work on Emacs because they like it, and scratch their own itch.
I don't think that anyone is in the position to suggest how they should
be spending their time. Even more so if (as far as I know) there is no
agreed upon roadmap for the project.

I think that what will help to keep Emacs relevant in the long run is
modernizing its structure and code base, moving away from the baroque
architecture that developed as a result of its very long history (and
some less than optimally future proof design decisions).

Personally, I find that better support for display with variable width
fonts will not improve my use of Emacs. What would improve my user
experience would be improvements to the display engine that will allow a
better document viewing capabilities in something like pdf-tools (like
continuous scrolling).

The only effective (and not detrimental to the project) way to steer
hacker attentions to features you find important is to pay them for
their work. With your popularity it should not be impossible to rise
enough money to do so.

Cheers,
Dan

Re: What improvements would be truly useful?

[Eli Zaretskii]

> From: Daniele Nicolodi <daniele@grinta.net>
> Date: Mon, 5 Mar 2018 11:51:09 -0700
> 
> I think that what will help to keep Emacs relevant in the long run is
> modernizing its structure and code base, moving away from the baroque
> architecture that developed as a result of its very long history (and
> some less than optimally future proof design decisions).

What exactly are you talking about here?  AFAIK, the Emacs
architecture didn't change since its inception, so its long history
has no relevance here.  But maybe I'm missing something or
misunderstanding what you intended to convey.

> What would improve my user experience would be improvements to the
> display engine that will allow a better document viewing
> capabilities in something like pdf-tools (like continuous
> scrolling).

AFAIU, that's already possible from the display engine POV, what's
missing is application-level Lisp code that would take advantage of
the infrastructure.

Re: What improvements would be truly useful?

[Daniele Nicolodi]

On 05/03/2018 12:22, Eli Zaretskii wrote:
>> From: Daniele Nicolodi <daniele@grinta.net>
>> Date: Mon, 5 Mar 2018 11:51:09 -0700
>>
>> I think that what will help to keep Emacs relevant in the long run is
>> modernizing its structure and code base, moving away from the baroque
>> architecture that developed as a result of its very long history (and
>> some less than optimally future proof design decisions).
> 
> What exactly are you talking about here?  AFAIK, the Emacs
> architecture didn't change since its inception, so its long history
> has no relevance here.  But maybe I'm missing something or
> misunderstanding what you intended to convey.

I'm far from being familiar with the Emacs codebase thus I may be
reporting something that it is not completely true, however: Emacs was
born as a console only application, the graphical user interface seems
to be duct taped on.  Also, GTK support seems a bit of an hack that
requires layering violations (reaching down to the X primitives) to
work.  Being GTK the only modern toolkit supported on Linux (as far as I
know) and the only way to get nartive Wayland support, some radical
cleanup in that area would probably be a good thing.
>> What would improve my user experience would be improvements to the
>> display engine that will allow a better document viewing
>> capabilities in something like pdf-tools (like continuous
>> scrolling).
> 
> AFAIU, that's already possible from the display engine POV, what's
> missing is application-level Lisp code that would take advantage of
> the infrastructure.

Reading the pdf-tools bug report on the matter, I had the impression
that implementing this was not possible because of some Emacs
limitations.  I'm happy if that's not the case.  I'll go and investigate
more.

Cheers,
Dan

Re: What improvements would be truly useful?

[Eli Zaretskii]

> From: Daniele Nicolodi <daniele@grinta.net>
> Date: Mon, 5 Mar 2018 23:32:28 -0700
> 
> I'm far from being familiar with the Emacs codebase thus I may be
> reporting something that it is not completely true, however: Emacs was
> born as a console only application, the graphical user interface seems
> to be duct taped on.

I object to the "duct taped" derogation, and invite you to study the
relevant code before you form your opinions.  Besides, Emacs still
supports text-mode terminals, and moreover, supports text-mode and GUI
frames in the same session (a very important feature), so some degree
of compatibility to a console is still a requirement.

> Also, GTK support seems a bit of an hack that
> requires layering violations (reaching down to the X primitives) to
> work.  Being GTK the only modern toolkit supported on Linux (as far as I
> know) and the only way to get nartive Wayland support, some radical
> cleanup in that area would probably be a good thing.

It's true that GTK support was added in a not very clean way, but I
don't think we can throw away support for the other toolkits just yet,
because they are still being used.

Re: What improvements would be truly useful?

[Daniel Colascione]

On 03/06/2018 08:52 AM, Eli Zaretskii wrote:
>> From: Daniele Nicolodi <daniele@grinta.net>
>> Date: Mon, 5 Mar 2018 23:32:28 -0700
>>
>> I'm far from being familiar with the Emacs codebase thus I may be
>> reporting something that it is not completely true, however: Emacs was
>> born as a console only application, the graphical user interface seems
>> to be duct taped on.
> 
> I object to the "duct taped" derogation, and invite you to study the
> relevant code before you form your opinions.  Besides, Emacs still
> supports text-mode terminals, and moreover, supports text-mode and GUI
> frames in the same session (a very important feature), so some degree
> of compatibility to a console is still a requirement.
> 
>> Also, GTK support seems a bit of an hack that
>> requires layering violations (reaching down to the X primitives) to
>> work.  Being GTK the only modern toolkit supported on Linux (as far as I
>> know) and the only way to get nartive Wayland support, some radical
>> cleanup in that area would probably be a good thing.
> 
> It's true that GTK support was added in a not very clean way, but I
> don't think we can throw away support for the other toolkits just yet,
> because they are still being used.

Eli is right. The current approach is fine, however it got there.

Eventually, we'll have to support Wayland, but I disagree that "radical 
cleanups" will necessarily be needed. As I wrote a while ago, the right 
way forward is a pure GTK+ (as opposed to a mixed X11/GTK+) back-end, 
letting GTK+ handle running on Wayland, and this kind of window system 
could be most accommodated in the existing model. It'll end up looking a 
lot like the NS port.

More radical, I think, would be multiple-window-system support. For 
that, more abstraction in the frame rendering will be required.

Re: What improvements would be truly useful?

[Daniele Nicolodi]

On 06/03/2018 09:52, Eli Zaretskii wrote:
>> From: Daniele Nicolodi <daniele@grinta.net>
>> Date: Mon, 5 Mar 2018 23:32:28 -0700
>>
>> I'm far from being familiar with the Emacs codebase thus I may be
>> reporting something that it is not completely true, however: Emacs was
>> born as a console only application, the graphical user interface seems
>> to be duct taped on.
> 
> I object to the "duct taped" derogation, and invite you to study the
> relevant code before you form your opinions.  Besides, Emacs still
> supports text-mode terminals, and moreover, supports text-mode and GUI
> frames in the same session (a very important feature), so some degree
> of compatibility to a console is still a requirement.

I didn't intend "duct taped" as derogatory.  I meant to say that it has
been added in a way that works and is effective, but it is not elegant
or nice to look at.  I'm an experimental physicist and all complex
experiments have some literal duct tape solution that makes them work,
that was often supposed to be a quick fix to get things going, and that
ends to survive generations of PhD students.

>> Also, GTK support seems a bit of an hack that
>> requires layering violations (reaching down to the X primitives) to
>> work.  Being GTK the only modern toolkit supported on Linux (as far as I
>> know) and the only way to get nartive Wayland support, some radical
>> cleanup in that area would probably be a good thing.
> 
> It's true that GTK support was added in a not very clean way, but I
> don't think we can throw away support for the other toolkits just yet,
> because they are still being used.

I don't think that cleaning up the GTK support requires removing support
for any other toolkit, and I'm far from suggesting something like that.

Cheers,
Dan

Re: What improvements would be truly useful?

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Daniele Nicolodi <daniele@grinta.net>
> Date: Tue, 6 Mar 2018 22:45:25 -0700
> 
> >> Also, GTK support seems a bit of an hack that
> >> requires layering violations (reaching down to the X primitives) to
> >> work.  Being GTK the only modern toolkit supported on Linux (as far as I
> >> know) and the only way to get nartive Wayland support, some radical
> >> cleanup in that area would probably be a good thing.
> > 
> > It's true that GTK support was added in a not very clean way, but I
> > don't think we can throw away support for the other toolkits just yet,
> > because they are still being used.
> 
> I don't think that cleaning up the GTK support requires removing support
> for any other toolkit, and I'm far from suggesting something like that.

Then how should we understand your suggestion for "radical cleanup",
right after asserting that GTK is the only modern toolkit supported by
GNU/Linux?

Re: What improvements would be truly useful?

[Daniele Nicolodi]

On 3/7/18 10:45 AM, Eli Zaretskii wrote:
>> Cc: emacs-devel@gnu.org
>> From: Daniele Nicolodi <daniele@grinta.net>
>> Date: Tue, 6 Mar 2018 22:45:25 -0700
>>
>>>> Also, GTK support seems a bit of an hack that
>>>> requires layering violations (reaching down to the X primitives) to
>>>> work.  Being GTK the only modern toolkit supported on Linux (as far as I
>>>> know) and the only way to get nartive Wayland support, some radical
>>>> cleanup in that area would probably be a good thing.
>>>
>>> It's true that GTK support was added in a not very clean way, but I
>>> don't think we can throw away support for the other toolkits just yet,
>>> because they are still being used.
>>
>> I don't think that cleaning up the GTK support requires removing support
>> for any other toolkit, and I'm far from suggesting something like that.
> 
> Then how should we understand your suggestion for "radical cleanup",
> right after asserting that GTK is the only modern toolkit supported by
> GNU/Linux?

Radical cleanup of the GTK integration.

Re: What improvements would be truly useful?

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Daniele Nicolodi <daniele@grinta.net>
> Date: Wed, 7 Mar 2018 10:49:39 -0700
> 
> > Then how should we understand your suggestion for "radical cleanup",
> > right after asserting that GTK is the only modern toolkit supported by
> > GNU/Linux?
> 
> Radical cleanup of the GTK integration.

If you have ideas for how to achieve that, I'd be interested to hear
them.

Thanks.

Re: What improvements would be truly useful?

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 729 bytes --]

Eli Zaretskii <eliz@gnu.org> schrieb am Mi., 7. März 2018 um 19:06 Uhr:

> > Cc: emacs-devel@gnu.org
> > From: Daniele Nicolodi <daniele@grinta.net>
> > Date: Wed, 7 Mar 2018 10:49:39 -0700
> >
> > > Then how should we understand your suggestion for "radical cleanup",
> > > right after asserting that GTK is the only modern toolkit supported by
> > > GNU/Linux?
> >
> > Radical cleanup of the GTK integration.
>
> If you have ideas for how to achieve that, I'd be interested to hear
> them.
>

By adding a gtkterm.c, that does the same as w32term.c, just for GTK,
without referring to X.org in any way?
The cleanup would then consist in removing all GTK functionality from
xterm.c once gtkterm.c is stable.

[-- Attachment #2: Type: text/html, Size: 1188 bytes --]

Re: What improvements would be truly useful?

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Wed, 07 Mar 2018 21:45:46 +0000
> Cc: Daniele Nicolodi <daniele@grinta.net>, emacs-devel@gnu.org
> 
>  If you have ideas for how to achieve that, I'd be interested to hear
>  them.
> 
> By adding a gtkterm.c, that does the same as w32term.c, just for GTK, without referring to X.org in any way?
> The cleanup would then consist in removing all GTK functionality from xterm.c once gtkterm.c is stable.

(I guess you meant removing GTK-related portions of xfns.c as well?)

If you look at these two files, you will see that the GTK-specific
snippets in them are a few and far between.  Most of the code is
generic.  So I think making separate gtkterm.c/gtkfns.c will risk
duplicating a lot of code, something we already have with w32 and ns
ports, and such duplication is undesirable.  But maybe I'm missing
something.

My understanding of this sub-thread was that the idea was to separate
GTK _architecturally_, not just making it yet another display
back-end.

Re: What improvements would be truly useful?

[Alan Third]

On Thu, Mar 08, 2018 at 03:20:58PM +0200, Eli Zaretskii wrote:
> > From: Philipp Stephani <p.stephani2@gmail.com>
> > Date: Wed, 07 Mar 2018 21:45:46 +0000
> > Cc: Daniele Nicolodi <daniele@grinta.net>, emacs-devel@gnu.org
> >
> > By adding a gtkterm.c, that does the same as w32term.c, just for GTK, without referring to X.org in any way?
> > The cleanup would then consist in removing all GTK functionality from xterm.c once gtkterm.c is stable.
> 
> (I guess you meant removing GTK-related portions of xfns.c as well?)
> 
> If you look at these two files, you will see that the GTK-specific
> snippets in them are a few and far between.  Most of the code is
> generic.

That’s part of the problem with how Emacs deals with GTK, it does many
things the X way rather than the GTK way.

For example, mouse and keyboard events are dealt with using X code,
but GTK provides it’s own APIs for dealing with them. I thought
implementing touch events would be easy because GTK already does all
the heavy lifting, but Emacs doesn’t use GTK for events.

I strongly suspect a Wayland compatible GTK Emacs would have to remove
a lot of (all?) X code and replace it with GTK code.

> My understanding of this sub-thread was that the idea was to separate
> GTK _architecturally_, not just making it yet another display
> back-end.

I don’t think I understand the difference.
-- 
Alan Third

Re: What improvements would be truly useful?

[Daniel Colascione]

On 03/08/2018 08:24 AM, Alan Third wrote:
> On Thu, Mar 08, 2018 at 03:20:58PM +0200, Eli Zaretskii wrote:
>>> From: Philipp Stephani <p.stephani2@gmail.com>
>>> Date: Wed, 07 Mar 2018 21:45:46 +0000
>>> Cc: Daniele Nicolodi <daniele@grinta.net>, emacs-devel@gnu.org
>>>
>>> By adding a gtkterm.c, that does the same as w32term.c, just for GTK, without referring to X.org in any way?
>>> The cleanup would then consist in removing all GTK functionality from xterm.c once gtkterm.c is stable.
>>
>> (I guess you meant removing GTK-related portions of xfns.c as well?)
>>
>> If you look at these two files, you will see that the GTK-specific
>> snippets in them are a few and far between.  Most of the code is
>> generic.
> 
> That’s part of the problem with how Emacs deals with GTK, it does many
> things the X way rather than the GTK way.
> 
> For example, mouse and keyboard events are dealt with using X code,
> but GTK provides it’s own APIs for dealing with them. I thought
> implementing touch events would be easy because GTK already does all
> the heavy lifting, but Emacs doesn’t use GTK for events.
> 
> I strongly suspect a Wayland compatible GTK Emacs would have to remove
> a lot of (all?) X code and replace it with GTK code.
> 
>> My understanding of this sub-thread was that the idea was to separate
>> GTK _architecturally_, not just making it yet another display
>> back-end.
> 
> I don’t think I understand the difference.

Emacs can be compiled with support for many different window systems, 
such as X11, MS-Windows, and Cocoa/NS. Currently, having chosen the X11 
window system, you can go on to choose which toolkit you want to use: 
Motif, GTK2, GTK3, Lucid, and so on.

The proposal is to promote GTK support from an X11 toolkit to a 
top-level window system, excising, in the process, the GTK display 
back-end of any mention of X11.

I think it's an excellent project and one important to the future 
maintainability of the project. As much as it saddens* me to say so, X11 
is dying, while GTK will continue to be a decent abstraction that will 
hit the happy, conventional path of GNU/Linux input and display 
infrastructure. Emacs X11 integration being unusual and dubiously legal 
has caused compatibility problems in the past, particularly with systems 
like NX.

As part of this effort, I'd love to see more abstraction in the window 
system selection. All the xterm.c functions that we carefully replicate 
in every window system really should be a table of frame-specific 
function pointers, and we should be able to compile a single Emacs 
binary with support for all window systems enabled simultaneously.

* IMHO, Wayland should be been a series of new X protocol extensions, 
not a greenfield project.

Re: What improvements would be truly useful?

[Yuri Khan]

On Tue, Mar 6, 2018 at 2:22 AM, Eli Zaretskii <eliz@gnu.org> wrote:

>> I think that what will help to keep Emacs relevant in the long run is
>> modernizing its structure and code base, moving away from the baroque
>> architecture that developed as a result of its very long history (and
>> some less than optimally future proof design decisions).
>
> What exactly are you talking about here?  AFAIK, the Emacs
> architecture didn't change since its inception, so its long history
> has no relevance here.  But maybe I'm missing something or
> misunderstanding what you intended to convey.

One architectural decision that I would call baroque is the input model.

In Emacs, the input model is that of the character terminal: the input
is a sequence of characters. The repertoire of Unicode characters is
extended with special values for function and editing keys, and for
keys modified with Ctrl, Super, Hyper, and Shift where necessary (and
Meta is represented as an ESC prefix), but still it is a sequence of
characters without explicit ties to the keys. Even mouse clicks are
processed as a special kind of characters.

This is inevitable when running in a terminal emulator where the
underlying abstraction works this way, but in a GUI build this leads
to inability to distinguish, for example, the ‘.’ character as coming
from the bottom-row right ring finger in US English layout, or from
the bottom-row right pinky in Russian layout, or from the decimal
separator key on the keypad. Bindings such as M-{ are unnecessarily
hard for international users whose keyboard layout does not have a {
character at all or puts it into third or fourth level.

Another example was described in Daniel Colascione’s article/rant
“Buttery Smooth Emacs”.

Re: What improvements would be truly useful?

[Eli Zaretskii]

> From: Yuri Khan <yuri.v.khan@gmail.com>
> Date: Tue, 6 Mar 2018 14:48:39 +0700
> Cc: Daniele Nicolodi <daniele@grinta.net>, Emacs developers <emacs-devel@gnu.org>
> 
> One architectural decision that I would call baroque is the input model.

Then we have different interpretations of what "architecture" means in
this context.  To me, architecture of a program such as Emacs is its
model-view-controller structure, the single-threaded operation, etc.
Input model is a lower-level design and implementation layer.

> In Emacs, the input model is that of the character terminal: the input
> is a sequence of characters. The repertoire of Unicode characters is
> extended with special values for function and editing keys, and for
> keys modified with Ctrl, Super, Hyper, and Shift where necessary (and
> Meta is represented as an ESC prefix), but still it is a sequence of
> characters without explicit ties to the keys. Even mouse clicks are
> processed as a special kind of characters.

Unless I misunderstand what you mean by "character", the above is
simply not true.  In Emacs, input is a stream of events.  Several
kinds of events are supported, see enum event_kind in termhooks.h.
Only some of them are characters, most are not.  The relevant variable
in keyboard.c is frequently called 'c', so one could think it's a
character, but in fact it is a Lisp object, and can be anything.  In
particular, a mouse click doesn't look at all like a character, it is
actually a cons cell.

> This is inevitable when running in a terminal emulator where the
> underlying abstraction works this way, but in a GUI build this leads
> to inability to distinguish, for example, the ‘.’ character as coming
> from the bottom-row right ring finger in US English layout, or from
> the bottom-row right pinky in Russian layout, or from the decimal
> separator key on the keypad.

That is a different issue, unrelated to what kind of input events are
recognized and how they are represented.  Emacs does support function
keys on X and other GUI environments, so it could potentially support
any abstraction for keys, if the underlying GUI APIs allow that on the
X message level.  We even install a low-level keyboard hook on Windows
to salvage some of the keys that Windows insists on intercepting ahead
of the application.

IOW, I see no architectural limitations here, only implementation that
might be missing or hard to come up with.

> Another example was described in Daniel Colascione’s article/rant
> “Buttery Smooth Emacs”.

Daniel's blog is a good read for someone who is familiar with the
internals, but I don't recommend to form your opinions about the Emacs
internals and their problems based solely on that blog.  For starters,
some of what is written there is no longer true these days.

Re: What improvements would be truly useful?

[Juri Linkov]

> What would improve my user experience would be improvements to the
> display engine that will allow a better document viewing capabilities
> in something like pdf-tools (like continuous scrolling).

Continuous scrolling can be easily implemented with a simple hack
to insert adjacent images for the previous/next pdf pages in
the same DocView buffer.

But what would be much more strategically advantageous is to employ
the Blink rendering engine as an alternative Emacs display engine.

Re: What improvements would be truly useful?

[Óscar Fuentes]

Juri Linkov <juri@linkov.net> writes:

>> What would improve my user experience would be improvements to the
>> display engine that will allow a better document viewing capabilities
>> in something like pdf-tools (like continuous scrolling).
>
> Continuous scrolling can be easily implemented with a simple hack
> to insert adjacent images for the previous/next pdf pages in
> the same DocView buffer.

I've looked into implementing this feature and it is not as simple as it
seems. The hack you mention is precisely that: a hack, and a very cranky
one. AFAIK there is no way of implementing continous scrolling with
adjacent images if you care at all about robustness and user experience.
And the fact that people is asking for this feature for years and, to
this day, nobody implemented it, seems to confirm my impressions.

Re: What improvements would be truly useful?

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Tue, 06 Mar 2018 00:46:27 +0100
> 
> I've looked into implementing this feature and it is not as simple as it
> seems. The hack you mention is precisely that: a hack, and a very cranky
> one. AFAIK there is no way of implementing continous scrolling with
> adjacent images if you care at all about robustness and user experience.

Can you explain the obstacles that prevent implementing this?  We have
vscroll support in the display engine, so I think the implementation
should be quite easy.  But maybe I'm missing something.

> And the fact that people is asking for this feature for years and, to
> this day, nobody implemented it, seems to confirm my impressions.

That could be because people are unaware of some crucial feature, or
because no one with the know-how had enough motivation to code this,
or for any number of other reasons.  Not necessarily because it's
impossible.

Continuous image scrolling (was: What improvements would be truly useful?)

[Juri Linkov]

>>> What would improve my user experience would be improvements to the
>>> display engine that will allow a better document viewing capabilities
>>> in something like pdf-tools (like continuous scrolling).
>>
>> Continuous scrolling can be easily implemented with a simple hack
>> to insert adjacent images for the previous/next pdf pages in
>> the same DocView buffer.
>
> I've looked into implementing this feature and it is not as simple as it
> seems. The hack you mention is precisely that: a hack, and a very cranky
> one. AFAIK there is no way of implementing continous scrolling with
> adjacent images if you care at all about robustness and user experience.
> And the fact that people is asking for this feature for years and, to
> this day, nobody implemented it, seems to confirm my impressions.

I just realized we already have continuous image scrolling mode:

1. visiting a PDF file generates PNG images in the temporary directory
2. C-x d /tmp/docview1000/... RET
3. M-x iimage-mode RET
4. enjoy continuous image scrolling mode directly in dired
   (the same logic could be later incorporated into doc-view.el)

I tried this with a directory containing ca 1000 pages/images
and there is no performance degradation while continuously scrolling
all these 1000 images.

Re: Continuous image scrolling

[Óscar Fuentes]

Juri Linkov <juri@linkov.net> writes:

>>>> What would improve my user experience would be improvements to the
>>>> display engine that will allow a better document viewing capabilities
>>>> in something like pdf-tools (like continuous scrolling).
>>>
>>> Continuous scrolling can be easily implemented with a simple hack
>>> to insert adjacent images for the previous/next pdf pages in
>>> the same DocView buffer.
>>
>> I've looked into implementing this feature and it is not as simple as it
>> seems. The hack you mention is precisely that: a hack, and a very cranky
>> one. AFAIK there is no way of implementing continous scrolling with
>> adjacent images if you care at all about robustness and user experience.
>> And the fact that people is asking for this feature for years and, to
>> this day, nobody implemented it, seems to confirm my impressions.
>
> I just realized we already have continuous image scrolling mode:
>
> 1. visiting a PDF file generates PNG images in the temporary directory
> 2. C-x d /tmp/docview1000/... RET
> 3. M-x iimage-mode RET
> 4. enjoy continuous image scrolling mode directly in dired
>    (the same logic could be later incorporated into doc-view.el)
>
> I tried this with a directory containing ca 1000 pages/images
> and there is no performance degradation while continuously scrolling
> all these 1000 images.

PDF-Tools (which is vastly superior to the built-in PDF viewer) does not
work this way. It generates pages on demand. It is an absurd waste of
resources to visit a book and generate one png file for each page.
PDF-Tools solves this and does much more.

So, if the solution depends on having pre-generated images, it is worse
than not having continuous scroll, IMAO.

OTOH, I tried what you suggest on a directory with lots of images. The
result is quite crude. For example, it seems that the curso height
corresponds to the image height, i.e. it is not possible to finely
scroll the sequence of images up and down, you go one image at the time.

The problems I faced while trying to implement continous document scroll
for PDF-Tools revolved around making the scroll actually continous: if
you scroll down from page P-1 to page P, remove page P-2 and append page
P+1 without producing visible jumps.

Re: Continuous image scrolling

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Tue, 03 Apr 2018 03:59:23 +0200
> 
> The problems I faced while trying to implement continous document scroll
> for PDF-Tools revolved around making the scroll actually continous: if
> you scroll down from page P-1 to page P, remove page P-2 and append page
> P+1 without producing visible jumps.

Sounds like some simple enough changes to the logic of
line-move-visual and its subroutines.  Patches are welcome.

Re: Continuous image scrolling

[Juri Linkov]

>> I just realized we already have continuous image scrolling mode:
>>
>> 1. visiting a PDF file generates PNG images in the temporary directory
>> 2. C-x d /tmp/docview1000/... RET
>> 3. M-x iimage-mode RET
>> 4. enjoy continuous image scrolling mode directly in dired
>>    (the same logic could be later incorporated into doc-view.el)
>>
>> I tried this with a directory containing ca 1000 pages/images
>> and there is no performance degradation while continuously scrolling
>> all these 1000 images.
>
> PDF-Tools (which is vastly superior to the built-in PDF viewer) does not
> work this way. It generates pages on demand. It is an absurd waste of
> resources to visit a book and generate one png file for each page.
> PDF-Tools solves this and does much more.
>
> So, if the solution depends on having pre-generated images, it is worse
> than not having continuous scroll, IMAO.

This is an optimization, but a significant one, indeed,
if you are not going to read the whole PDF at once.

> OTOH, I tried what you suggest on a directory with lots of images. The
> result is quite crude. For example, it seems that the curso height
> corresponds to the image height,

It's possible to hide the cursor like we do in image-mode.

> i.e. it is not possible to finely scroll the sequence of images up and
> down, you go one image at the time.

It's because next-line goes to the next file in Dired by default,
this is why I disabled this inconvenience in my ~/.emacs by using
normal cursor motion keybindings:

  (define-key dired-mode-map [remap next-line] nil)
  (define-key dired-mode-map [remap previous-line] nil)

> The problems I faced while trying to implement continous document scroll
> for PDF-Tools revolved around making the scroll actually continous: if
> you scroll down from page P-1 to page P, remove page P-2

I see no need to remove page P-2.  For optimization it's better to keep it,
so that when the user returns to that page, it's already here.

> and append page P+1 without producing visible jumps.

Based on the simplistic approach like used by iimage-mode,
it should be easy to insert placeholders for all pages,
and use font-lock machinery to replace placeholders with
generated images when the user navigates to other parts
of the PDF buffer.

RE: Continuous image scrolling

[Drew Adams]

> use font-lock machinery to replace placeholders with
> generated images when the user navigates to other parts
> of the PDF buffer.

Interesting idea!  But would that be responsive, for
the user?  Would the images be generated on the fly?
Would they be automatically deleted and garbaged
afterward?

Re: Continuous image scrolling

[Juri Linkov]

>> use font-lock machinery to replace placeholders with
>> generated images when the user navigates to other parts
>> of the PDF buffer.
>
> Interesting idea!  But would that be responsive, for
> the user?  Would the images be generated on the fly?
> Would they be automatically deleted and garbaged
> afterward?

I think it will be no less responsive than other PDF readers
that also generate images on the fly.  But the main problem is
how to pre-render these placeholders to avoid visible jumps on
inserting new images.  Maybe we need to scan the whole PDF document
and get dimensions of all pages, then use them for placeholders.

Re: Continuous image scrolling

[Stefan Monnier]

> I see no need to remove page P-2.  For optimization it's better to keep it,
> so that when the user returns to that page, it's already here.

Reminds me: I've had Emacs crash on me when flipping though large PDF
documents with doc-view.  I haven't been able to reproduce it, but
I have the impression that we have a problem there somewhere.
[ It's not directly related to the above in the sense that doc-view
  only displays one page at any given time anyway.  ]


        Stefan

Re: Continuous image scrolling

[Clément Pit-Claudel]

On 2018-04-03 15:59, Juri Linkov wrote:
>> i.e. it is not possible to finely scroll the sequence of images up and
>> down, you go one image at the time.

> It's because next-line goes to the next file in Dired by default,
> this is why I disabled this inconvenience in my ~/.emacs by using
> normal cursor motion keybindings:

Maybe I'm misunderstanding the original request, but if not then I don't think this is the issue.

AFAIU, one thing that is not too easy to implement today in ELisp is continuous ("smooth") scrolling over multiple images, as shown in e.g. evince: if I drag the scroll bars or press the up and down keys, the current page and the following one move up and down by a bit.

We usually call this pixel-scroll in Emacs, and we have an implementation of it based on set-window-vscroll in lisp/pixel-scroll.el.  This is also what Docview and PDFTools use to scroll images that don't fit in the selected window.  But I don't think that trick works for sequences of images, at least not straight out of the box.

The main issue, IIUC, is that images occupy one (tall) line, and Emacs tries to always keep the line containing the point in full view, which means that set-window-vscroll does not always work if the point is on a line containing an image.

You can see what I mean concretely with the following example. Open a new buffer, and run (insert (propertize " " 'display '(image :type jpeg :file "/path/to/some/large/picture.jpg" :scale 1))).  Then insert 5 lines of "A".  Your buffer will look like this:

|----------------|
|  Picture here  |
|                |
|----------------|
A
A
A
A

Put the point on the third A and run (set-window-vscroll (selected-window) 8 t); note how the images moves a bit up.  That's the kind of scrolling we want, which is encouraging :)

But now put the point on the picture's line, and try the same experiment; notice how the picture doesn't move, because we try to keep the line in full view. (if the image is larger than the window, it does work, and that's how docview scrolls single pages. I don't know if there's a way to get that behavior when the image fits in the window).  That behavior becomes an issue when you have sequences of images, because you can't leave the point on the line of the first image, but moving the point to the line of the second image scrolls it entirely into view.

The two techniques to work around this that I'm aware of are (1) cutting the images into thin horizontal strips (and showing one strip per line), and (2) inserting a thin text line between each image on which to place the point while scrolling.  I don't know if either of these would work for pdftools.  Hopefully there's a third, simpler technique to achieve this?

Clément.

Re: Continuous image scrolling

[Eli Zaretskii]

> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Tue, 3 Apr 2018 17:57:35 -0400
> 
> AFAIU, one thing that is not too easy to implement today in ELisp is continuous ("smooth") scrolling over multiple images, as shown in e.g. evince: if I drag the scroll bars or press the up and down keys, the current page and the following one move up and down by a bit.

It could/should be easy to implement in Lisp because the current
behavior is implemented in Lisp.  At least, I won't believe assertions
such as the above until Someone™ digs into the code in
line-move-visual and its subroutines, where the current behavior is
implemented, and presents clear evidence that some code or feature
coded in C gets in the way.

> The main issue, IIUC, is that images occupy one (tall) line, and Emacs tries to always keep the line containing the point in full view, which means that set-window-vscroll does not always work if the point is on a line containing an image.

Yes, but most, if not all, of the logic supporting this is in
simple.el.

> But now put the point on the picture's line, and try the same experiment; notice how the picture doesn't move, because we try to keep the line in full view. (if the image is larger than the window, it does work, and that's how docview scrolls single pages. I don't know if there's a way to get that behavior when the image fits in the window).  That behavior becomes an issue when you have sequences of images, because you can't leave the point on the line of the first image, but moving the point to the line of the second image scrolls it entirely into view.
> 
> The two techniques to work around this that I'm aware of are (1) cutting the images into thin horizontal strips (and showing one strip per line), and (2) inserting a thin text line between each image on which to place the point while scrolling.  I don't know if either of these would work for pdftools.  Hopefully there's a third, simpler technique to achieve this?

It makes little sense to me to work around the behavior we ourselves
implemented.  Instead, I invite volunteers to take a good look at the
code and try changing it to lift the limitations that currently get in
the way.

The current code behaves like it does because it was _programmed_ to
behave that way.  It explicitly uses window-vscroll only when the
current line cannot be fully shown in the window for some reason.  But
if you look at the relevant code, you will see that this condition is
programmed there, and is specifically tested to determine when to use
window-vscroll and when to scroll to the next/previous screen line.
So by changing that logic, perhaps controlled by some defcustom, it
should be possible to get what DocView and PDF-tools want to
accomplish.

Let me remind you that similar misconceptions were around regarding
pixel-level scrolling of text, until Tak came up with his package that
did just that, entirely in Lisp.

I hope someone will step forward and do the necessary research of what
is needed, because my gut feeling is that the features sought out here
are within our reach.  (I can help with advice, because I'm
responsible for at least some of the code in that area.)

Any volunteers?

Re: Continuous image scrolling

[Juri Linkov]

> But now put the point on the picture's line, and try the same experiment;
> notice how the picture doesn't move, because we try to keep the line in
> full view. (if the image is larger than the window, it does work, and
> that's how docview scrolls single pages. I don't know if there's a way to
> get that behavior when the image fits in the window).

I tried only with images larger than the window height where multi-image
scrolling works fine.  But now I see that scrolling smaller images is not
so smooth.  I don't know if such narrow images are common in PDF documents.

Re: Continuous image scrolling

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Date: Wed, 04 Apr 2018 22:50:43 +0300
> Cc: emacs-devel@gnu.org
> 
> I tried only with images larger than the window height where multi-image
> scrolling works fine.  But now I see that scrolling smaller images is not
> so smooth.  I don't know if such narrow images are common in PDF documents.

With today's large monitors and user tendency to maximize the frame,
it's quite possible, I think.

Re: Continuous image scrolling

[Tak Kunihiro]

I think with the following setup,

M-x pixel-scroll-mode
(setq pixel-resolution-fine-flag t)

a buffer with large images, as you shown, can be scrolled by height of typical line.

Re: Continuous image scrolling

[Clément Pit-Claudel]

On 2018-04-04 19:31, Tak Kunihiro wrote:
> I think with the following setup,
> 
> M-x pixel-scroll-mode
> (setq pixel-resolution-fine-flag t)
> 
> a buffer with large images, as you shown, can be scrolled by height of typical line.
I think I must be doing something wrong :/

I used this snippet to insert 10 copies of a large picture in a new buffer:

   (dotimes (_ 10) 
      (insert (propertize " " 'display '(image :type jpeg :file "some-large-picture.jpg" :scale 1)) "\n"))

and then I turned pixel-scroll-mode on with the setting you mentioned, but Emacs still scrolls by full images.

Clément.

Re: Continuous image scrolling

[Tak Kunihiro]

>> I think with the following setup,
>>
>> M-x pixel-scroll-mode
>> (setq pixel-resolution-fine-flag t)
>>
>> a buffer with large images, as you shown, can be scrolled by height of typical line.
>
> I think I must be doing something wrong :/
> 
> I used this snippet to insert 10 copies of a large picture in a new buffer:
> 
>    (dotimes (_ 10)
>       (insert (propertize " " 'display '(image :type jpeg :file "some-large-picture.jpg" :scale 1)) "\n"))
> 
> and then I turned pixel-scroll-mode on with the setting you mentioned, but Emacs still scrolls by full images.

Let me assume height of an image is less than height of visible area of
a window.

M-x pixel-scroll-mode
(setq pixel-resolution-fine-flag t)
M-: (pixel-scroll-up 1)

The (pixel-scroll-up 1) scrolls a buffer up by (frame-char-height) with
re-displaying each pixel.

A function that scrolls a buffer up like (pixel-scroll-up 1) but without
re-displaying each pixel, should be written.  The function serves as
alternative of (scroll-up).

Re: Continuous image scrolling

[Clément Pit-Claudel]

On 2018-04-04 23:54, Tak Kunihiro wrote:
>>> I think with the following setup,
>>>
>>> M-x pixel-scroll-mode
>>> (setq pixel-resolution-fine-flag t)
>>>
>>> a buffer with large images, as you shown, can be scrolled by height of typical line.
>>
>> I think I must be doing something wrong :/
>>
>> I used this snippet to insert 10 copies of a large picture in a new buffer:
>>
>>    (dotimes (_ 10)
>>       (insert (propertize " " 'display '(image :type jpeg :file "some-large-picture.jpg" :scale 1)) "\n"))
>>
>> and then I turned pixel-scroll-mode on with the setting you mentioned, but Emacs still scrolls by full images.
> 
> Let me assume height of an image is less than height of visible area of
> a window.
> 
> M-x pixel-scroll-mode
> (setq pixel-resolution-fine-flag t)
> M-: (pixel-scroll-up 1)
> 
> The (pixel-scroll-up 1) scrolls a buffer up by (frame-char-height) with
> re-displaying each pixel.
> 
> A function that scrolls a buffer up like (pixel-scroll-up 1) but without
> re-displaying each pixel, should be written.  The function serves as
> alternative of (scroll-up).

I don't understand. I followed your instructions, but M-: (pixel-scroll-up 1) scrolls up by a full image.

Re: Continuous image scrolling

[Tak Kunihiro]

Can you try following on 26.0.91 or later?

$ emacs -Q
M-x about-emacs
M-x pixel-scroll-mode
M-: (setq pixel-resolution-fine-flag t)
M-: (pixel-scroll-up 1)

Re: Continuous image scrolling

[Clément Pit-Claudel]

On 2018-04-05 18:59, Tak Kunihiro wrote:
> Can you try following on 26.0.91 or later?
> 
> $ emacs -Q
> M-x about-emacs
> M-x pixel-scroll-mode
> M-: (setq pixel-resolution-fine-flag t)
> M-: (pixel-scroll-up 1)

That works fine.  But about-emacs doesn't show a buffer with multiple large images in a row, right?

Re: Continuous image scrolling

[Tak Kunihiro]

I found following only works when height of an image is less than half
of height of a window.

(progn
  (pixel-scroll-mode 1)
  (setq pixel-resolution-fine-flag t)
  (dotimes (_ 10)
    (insert (propertize " " 'display '(image :type jpeg :file "some-large-picture.jpg" :scale 1)) "\n"))
  (dotimes (_ 100)
    (pixel-scroll-down 1)
    (sit-for 0.3)))

Re: Continuous image scrolling

[John Wiegley]

>>>>> "ÓF" == Óscar Fuentes <ofv@wanadoo.es> writes:

ÓF> PDF-Tools (which is vastly superior to the built-in PDF viewer) does not
ÓF> work this way. It generates pages on demand. It is an absurd waste of
ÓF> resources to visit a book and generate one png file for each page.
ÓF> PDF-Tools solves this and does much more.

It really is quite good, I use it as my main PDF reader on the Mac now.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: What improvements would be truly useful?

[John Wiegley]

>>>>> "DN" == Daniele Nicolodi <daniele@grinta.net> writes:

DN> Personally, I find that better support for display with variable width
DN> fonts will not improve my use of Emacs. What would improve my user
DN> experience would be improvements to the display engine that will allow a
DN> better document viewing capabilities in something like pdf-tools (like
DN> continuous scrolling).

I have to agree with Daniele here. I don't use Emacs for writing such
documents, nor would having variable width support by itself grant all the
other features such users have come to expect from modern Word Processors.

Time spent on a better GC could help the *overall* user experience, in the
form of generally better latency, even if no one would be able to point to the
GC as the reason for their better experience with Emacs.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: What improvements would be truly useful?

[daniel sutton]

[-- Attachment #1: Type: text/plain, Size: 1244 bytes --]

I also really like Daniel's passion and ideas. I think he has been
contributing new ideas and plenty of code to back it up. Seems like
something to be encouraged.

On Mon, Mar 5, 2018 at 4:09 PM, John Wiegley <johnw@gnu.org> wrote:

> >>>>> "DN" == Daniele Nicolodi <daniele@grinta.net> writes:
>
> DN> Personally, I find that better support for display with variable width
> DN> fonts will not improve my use of Emacs. What would improve my user
> DN> experience would be improvements to the display engine that will allow
> a
> DN> better document viewing capabilities in something like pdf-tools (like
> DN> continuous scrolling).
>
> I have to agree with Daniele here. I don't use Emacs for writing such
> documents, nor would having variable width support by itself grant all the
> other features such users have come to expect from modern Word Processors.
>
> Time spent on a better GC could help the *overall* user experience, in the
> form of generally better latency, even if no one would be able to point to
> the
> GC as the reason for their better experience with Emacs.
>
> --
> John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
> http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2
>
>

[-- Attachment #2: Type: text/html, Size: 1834 bytes --]

Re: What improvements would be truly useful?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

   I don't use Emacs for writing such
  > documents, nor would having variable width support by itself grant all the
  > other features such users have come to expect from modern Word Processors.

You're correct in observing that to make Emacs work as a word
processor requires adding a number of features.  A single added
feature won't reach that point.  A few features, well chosen, WILL
reach that point, and then we will see people using Emacs as a word
processor.

Thus, I picked one of these features, one which we certainly need,
and asked people to implement it in a user-usable way.

When that's done, we can add another word-processing feature,
and another, and soon people will start using Emacs for some
simple cases of word processing.

Some simple cases, such as letters, are not very many features
away.  I am sure we could get there this year.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: What improvements would be truly useful?

[Toon Claes]

[-- Attachment #1: Type: text/plain, Size: 628 bytes --]

IMHO, one thing that I would find interesting is proper support
for multiple major modes.

I know there has been some talking about this in the past:

https://lists.gnu.org/archive/html/emacs-devel/2016-04/msg00585.html

But I'm not sure what the outcome was of that very long thread.

I only know later on some improvements were made to have html, css, and
js modes working together: 

https://lists.gnu.org/archive/html/emacs-devel/2017-01/msg00775.html

But still having great multiple major mode support--with islands having
their own major and minor modes--would make Emacs a more modern editor.

-- Toon

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: What improvements would be truly useful?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > But still having great multiple major mode support--with islands having
  > their own major and minor modes--would make Emacs a more modern editor.

I strongly support the request.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: What improvements would be truly useful?

[Dmitry Gutov]

On 3/8/18 1:44 PM, Toon Claes wrote:
> IMHO, one thing that I would find interesting is proper support
> for multiple major modes.
> 
> I know there has been some talking about this in the past:
> 
> https://lists.gnu.org/archive/html/emacs-devel/2016-04/msg00585.html
> 
> But I'm not sure what the outcome was of that very long thread.

That one reached a dead end, for now.

> I only know later on some improvements were made to have html, css, and
> js modes working together:
> 
> https://lists.gnu.org/archive/html/emacs-devel/2017-01/msg00775.html
> 
> But still having great multiple major mode support--with islands having
> their own major and minor modes--would make Emacs a more modern editor.

mmm-mode (a very old package providing features like that) is soon going 
to be in ELPA. If anyone on this mailing list wanted to contribute, but 
stopped because it wasn't a part of Emacs or ELPA, that will be their 
chance.

Re: Let's make the GC safe and iterative

[Stefan Monnier]

> We need to fix GC being deeply recursive once and for all. Tweaking stack
> sizes on various platforms and trying to spot-fix GC for the occasional
> deeply recursive structure is annoying. Here's my proposal:

I'm OK with making the GC loop without recursing on the C stack, but
I have two comments:
1- Don't use a queue: use a stack.  Mark&sweep naturally work with
   a stack whereas stop&copy naturally works with a queue, so in most cases
   the choice is implicit/accidental, but experience shows that if we
   can choose, the stack is the better option (e.g. there's been
   various works that try to tweak stop&copy to use a stack rather than
   a queue), for reasons of locality.
2- Why do you say "We can't allocate memory to hold the queue during GC"?
   We very well can, and doing it should make things simpler (and make
   sure we don't preallocate way too much memory).

If instead of a queue we use a stack and we just push Lisp_Object values
onto that stack, the resulting space use can't be worse than what we
have now (it's just going to use our malloc-managed stack instead of the
C stack).


        Stefan

Re: Let's make the GC safe and iterative

[Daniel Colascione]

On 03/01/2018 03:38 PM, Stefan Monnier wrote:
>> We need to fix GC being deeply recursive once and for all. Tweaking stack
>> sizes on various platforms and trying to spot-fix GC for the occasional
>> deeply recursive structure is annoying. Here's my proposal:
> 
> I'm OK with making the GC loop without recursing on the C stack, but
> I have two comments:
> 1- Don't use a queue: use a stack.  Mark&sweep naturally work with
>     a stack whereas stop&copy naturally works with a queue, so in most cases
>     the choice is implicit/accidental, but experience shows that if we
>     can choose, the stack is the better option (e.g. there's been
>     various works that try to tweak stop&copy to use a stack rather than
>     a queue), for reasons of locality.

Sure. Using a stack instead of a queue is a minor detail; the basic 
algorithm is the same.

Within blocks, we still need to be queue-based though: we'll scan the 
scan_pending bitmap once; if more bits behind the scan_pending read 
position become set during the scan, we won't know until the next time 
we examine the block. But we *can* make sure that we re-examine this 
block immediately after we finish the scan_pending bit-scan.

> 2- Why do you say "We can't allocate memory to hold the queue during GC"?
>     We very well can, and doing it should make things simpler (and make
>     sure we don't preallocate way too much memory).

We can try, but we need a plan if allocation fails. I don't want Emacs 
to be one of those programs that just aborts if malloc fails. Too many 
other programs get it wrong these days. I don't think the worst-case 
preallocation overhead is severe enough that we have to give up malloc 
robustness.

> If instead of a queue we use a stack and we just push Lisp_Object values
> onto that stack, the resulting space use can't be worse than what we
> have now (it's just going to use our malloc-managed stack instead of the
> C stack).

Sure, except that stack use is ephemeral, and the space we allocate for 
the stack gets used for lisp too. OTOH, we need to preallocate these 
linkage structures and keep them allocated between GCs. But like I said, 
I don't think the overhead is worth worrying about.

Re: Let's make the GC safe and iterative

[Stefan Monnier]

> Sure. Using a stack instead of a queue is a minor detail; the basic
> algorithm is the same.

The point is that a stack will be better locality-wise and (more
importantly) that it should be as good as what we have now w.r.t memory
use (actually, it should be significantly better, because at each
"recursive" step we only push one Lisp_Object on our stack whereas the
current code pushes an activation frame which is at least twice as large
since it also needs a return address).

So I think it does affect the algorithm: I'm not convinced we need to do
anything special for objects allocated from blocks.

>> 2- Why do you say "We can't allocate memory to hold the queue during GC"?
>>     We very well can, and doing it should make things simpler (and make
>>     sure we don't preallocate way too much memory).
> We can try, but we need a plan if allocation fails.

We can just abort the GC (only requires resetting the markbits).
It's not like we GC when our memory is full: we just GC on a regular
basis for "prophylactic" reasons.

> I don't think the worst-case preallocation overhead is severe enough
> that we have to give up malloc robustness.

It sounds pretty significant to me.  I could live with it if it were
indispensable but I find it difficult to justify.

> Sure, except that stack use is ephemeral,

Our manually-managed stack can have the same property.

> and the space we allocate for the stack gets used for lisp too.

Yes, that's a downside of such a change w.r.t average use of the
combined stack space, but I don't think it affect the maximal combined
stack space, so I'm not worried.  This said, we could also move the Lisp
stack to that same manually-managed stack (at least for the bytecode
stacks).


        Stefan

bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Cc: bug-gnu-emacs@gnu.org, 30626@debbugs.gnu.org
> Date: Tue, 27 Feb 2018 12:39:47 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I guess it's a stack overflow: that function recurses into
> > subdirectories.
> >
> > To avoid such problems, the function should be rewritten to work by
> > BFS, not DFS.
> 
> Here is the backtrace I'm now able to produce.  Looks like the crash
> happens in gc:

GC is deeply-recursive, and you have exacerbated that by using up a
lot of stack.