Let's make the GC safe and iterative (Was: Re: bug#30626)

Let's make the GC safe and iterative (Was: Re: bug#30626)

[Daniel Colascione]

Noam mentioned that I should make a new thread for this proposal, so I'm 
posting an edited version of my original message.

tl;dr: we should be able to make the GC non-recursive with minimal 
overhead, solving the "Emacs crashed because we ran out of stack space 
in GC" problem once and for all.

On 02/27/2018 10:08 AM, Eli Zaretskii wrote:
> What can we do instead in such cases?  Stack-overflow protection
> cannot work in GC, so you are shooting yourself in the foot by
> creating such large recursive structures.  By the time we get to GC,
> where the problem will happen, it's too late, because the memory was
> already allocated.
> 
> Does anyone has a reasonable idea for avoiding the crash in such
> programs?

We need to fix GC being deeply recursive once and for all. Tweaking 
stack sizes on various platforms and trying to spot-fix GC for the 
occasional deeply recursive structure is annoying. Here's my proposal:

I. NAIVE APPROACH

Turn garbage_collect_1 into a queue-draining loop, initializing the 
object queue with the GC roots before draining it. We'll make 
mark_object put an object on this queue, turning the existing 
mark_object code into a mark_queued_object function.

garbage_collect_1 will just call mark_queued_object in a loop; 
mark_queued_object can call mark_object, but since mark_object just 
enqueues an object and doesn't recurse, we can't exhaust the stack with 
deep object graphs. (We'll repurpose the mark bit to mean that the 
object is on the to-mark queue; by the time we fully drain the queue, 
just before we sweep, the mark bit will have the same meaning it does now.)

We can't allocate memory to hold the queue during GC, so we'll have to 
pre-allocate it. We can implement the queue as a list of queue blocks, 
where each queue block is an array of 16k or so Lisp_Objects. During 
allocation, we'll just make sure we have one Lisp_Object queue-block 
slot for every non-self-representing Lisp object we allocate.

Since we know that we'll have enough queue blocks for the worst GC case, 
we can have mark_object pull queue blocks from a free list, aborting if 
for some reason it ever runs out of queue blocks. (The previous 
paragraph guarantees we won't.) garbage_collect_1 will churn through 
these heap blocks and place each back on the free list after it's called 
mark_queued_object on every Lisp_Object in the queue block.

In this way, in non-pathological cases of GC, we'll end up using the 
same few queue blocks over and over. That's a nice optimization, because 
we can MADV_DONTNEED unused queue blocks so the OS doesn't actually have 
to remember their contents.

In this way, I think we can make the current GC model recursion-proof 
without drastically changing how we allocate Lisp objects. The 
additional memory requirements should be modest: it's basically one 
Lisp_Object per Lisp object allocated.

II. ELABORATION

The naive version of this scheme needs about 4.6MB of overhead on my 
current 20MB Emacs heap, but it should be possible to reduce the 
overhead significantly by taking advantage of the block allocation we do 
for conses and other types --- we can put whole blocks on the queue 
instead of pointers to individual block parts, so we can get away with a 
much smaller queue.

It's also interesting to note that we don't need separate queue blocks 
to put a block on the queue, as we do if we want to enqueue individual 
Lisp_Object pointers. Instead, we can add to each block type a pointer 
to the next block *on the to-be-marked queue* and a bitmask yielding the 
positions within that block that we want to mark.

For example, cons_block right now looks like this:

struct cons_block
{
   /* Place `conses' at the beginning, to ease up CONS_INDEX's job.  */
   struct Lisp_Cons conses[CONS_BLOCK_SIZE];
   bits_word gcmarkbits[1 + CONS_BLOCK_SIZE / BITS_PER_BITS_WORD];
   struct cons_block *next;
};

We'd turn it into something like this:

struct cons_block
{
   /* Place `conses' at the beginning, to ease up CONS_INDEX's job.  */
   struct Lisp_Cons conses[CONS_BLOCK_SIZE];
   bits_word gcmarkbits[1 + CONS_BLOCK_SIZE / BITS_PER_BITS_WORD];
   bits_word scan_pending[1 + CONS_BLOCK_SIZE / BITS_PER_BITS_WORD];
   struct cons_block *next;
   struct cons_block *next_scan_pending;
};

When we call mark_object on a cons, we'll look up its cons_block and 
look up the cons in gcmarkbits. If we find the cons mark bit set, we're 
done. Otherwise, we look at the scan_pending bit for the cons cell. If 
_that's_ set, we're also done. If we find the scan_pending bit unset, 
however, we set it, and then look at next_scan_pending. If that's 
non-zero, we know the block as a whole is enqueued for scanning, and 
we're done. If *that's* zero, then we add the whole block to the 
to-be-scanned queue.

We'll modify garbage_collect_1 to drain both the Lisp_Object queue I 
described in the last section (which we still need for big objects like 
buffers) *and* the queue of blocks pending scanning. When we get a cons 
block, we'll scan all the conses with scan_pending bits set to one, set 
their gcmarkbits, and remove the cons block from the queue.

That same cons block might make it back onto the queue later if someone 
calls mark_object for one if its conses we didn't already scan, but 
that's okay. Scanning scan_pending should be very cheap, especially on 
modern CPUs with bit-prefix-scan instructions.

Under this approach, the reserved-queue-block scheme would impose an 
overhead of somewhere around 1MB on the same heap. (I think it'd 
actually be a bit smaller actually.) Conses, strings, and vectors are 
the overwhelming majority of heap-allocated objects, and thanks to block 
packing, we'd get bookkeeping for them for practically free. This amount 
of overhead seems reasonable. I think we may end up actually using less 
memory that we would for recursive mark_object stack invocation.

This scheme interacts well with the portable dumper too. pdumper already 
uses a big bit array to store mark bits; we'd just add another array for 
its scan_pending. We'd basically treat the entire pdumper region as one 
big cons_block for GC purposes.

What do you think? I think this approach solves a longstanding fiddly 
problem with Emacs GC without too much disruption to the internals. It 
also paves the way for concurrent or generational GC if we ever want to 
implement these features.

Re: Let's make the GC safe and iterative (Was: Re: bug#30626)

[Paul Eggert]

On 03/01/2018 03:22 PM, Daniel Colascione wrote:
> What do you think?

Thanks, I like the idea. I suggest implementing the naive version first. 
Although the more-elaborate versions saves virtual memory, it's not 
clear that it'd be an overall performance win in the typical case, so my 
guess is that it's better to start simple.

Re: Let's make the GC safe and iterative

[Stefan Monnier]

> We need to fix GC being deeply recursive once and for all. Tweaking stack
> sizes on various platforms and trying to spot-fix GC for the occasional
> deeply recursive structure is annoying. Here's my proposal:

I'm OK with making the GC loop without recursing on the C stack, but
I have two comments:
1- Don't use a queue: use a stack.  Mark&sweep naturally work with
   a stack whereas stop&copy naturally works with a queue, so in most cases
   the choice is implicit/accidental, but experience shows that if we
   can choose, the stack is the better option (e.g. there's been
   various works that try to tweak stop&copy to use a stack rather than
   a queue), for reasons of locality.
2- Why do you say "We can't allocate memory to hold the queue during GC"?
   We very well can, and doing it should make things simpler (and make
   sure we don't preallocate way too much memory).

If instead of a queue we use a stack and we just push Lisp_Object values
onto that stack, the resulting space use can't be worse than what we
have now (it's just going to use our malloc-managed stack instead of the
C stack).


        Stefan

Re: Let's make the GC safe and iterative

[Daniel Colascione]

On 03/01/2018 03:38 PM, Stefan Monnier wrote:
>> We need to fix GC being deeply recursive once and for all. Tweaking stack
>> sizes on various platforms and trying to spot-fix GC for the occasional
>> deeply recursive structure is annoying. Here's my proposal:
> 
> I'm OK with making the GC loop without recursing on the C stack, but
> I have two comments:
> 1- Don't use a queue: use a stack.  Mark&sweep naturally work with
>     a stack whereas stop&copy naturally works with a queue, so in most cases
>     the choice is implicit/accidental, but experience shows that if we
>     can choose, the stack is the better option (e.g. there's been
>     various works that try to tweak stop&copy to use a stack rather than
>     a queue), for reasons of locality.

Sure. Using a stack instead of a queue is a minor detail; the basic 
algorithm is the same.

Within blocks, we still need to be queue-based though: we'll scan the 
scan_pending bitmap once; if more bits behind the scan_pending read 
position become set during the scan, we won't know until the next time 
we examine the block. But we *can* make sure that we re-examine this 
block immediately after we finish the scan_pending bit-scan.

> 2- Why do you say "We can't allocate memory to hold the queue during GC"?
>     We very well can, and doing it should make things simpler (and make
>     sure we don't preallocate way too much memory).

We can try, but we need a plan if allocation fails. I don't want Emacs 
to be one of those programs that just aborts if malloc fails. Too many 
other programs get it wrong these days. I don't think the worst-case 
preallocation overhead is severe enough that we have to give up malloc 
robustness.

> If instead of a queue we use a stack and we just push Lisp_Object values
> onto that stack, the resulting space use can't be worse than what we
> have now (it's just going to use our malloc-managed stack instead of the
> C stack).

Sure, except that stack use is ephemeral, and the space we allocate for 
the stack gets used for lisp too. OTOH, we need to preallocate these 
linkage structures and keep them allocated between GCs. But like I said, 
I don't think the overhead is worth worrying about.

Re: Let's make the GC safe and iterative

[Stefan Monnier]

> Sure. Using a stack instead of a queue is a minor detail; the basic
> algorithm is the same.

The point is that a stack will be better locality-wise and (more
importantly) that it should be as good as what we have now w.r.t memory
use (actually, it should be significantly better, because at each
"recursive" step we only push one Lisp_Object on our stack whereas the
current code pushes an activation frame which is at least twice as large
since it also needs a return address).

So I think it does affect the algorithm: I'm not convinced we need to do
anything special for objects allocated from blocks.

>> 2- Why do you say "We can't allocate memory to hold the queue during GC"?
>>     We very well can, and doing it should make things simpler (and make
>>     sure we don't preallocate way too much memory).
> We can try, but we need a plan if allocation fails.

We can just abort the GC (only requires resetting the markbits).
It's not like we GC when our memory is full: we just GC on a regular
basis for "prophylactic" reasons.

> I don't think the worst-case preallocation overhead is severe enough
> that we have to give up malloc robustness.

It sounds pretty significant to me.  I could live with it if it were
indispensable but I find it difficult to justify.

> Sure, except that stack use is ephemeral,

Our manually-managed stack can have the same property.

> and the space we allocate for the stack gets used for lisp too.

Yes, that's a downside of such a change w.r.t average use of the
combined stack space, but I don't think it affect the maximal combined
stack space, so I'm not worried.  This said, we could also move the Lisp
stack to that same manually-managed stack (at least for the bytecode
stacks).


        Stefan

Re: Let's make the GC safe and iterative (Was: Re: bug#30626)

[Ken Raeburn]

On Mar 1, 2018, at 18:29, Paul Eggert <eggert@cs.ucla.edu> wrote:

> On 03/01/2018 03:22 PM, Daniel Colascione wrote:
>> What do you think?
> 
> Thanks, I like the idea. I suggest implementing the naive version first. Although the more-elaborate versions saves virtual memory, it's not clear that it'd be an overall performance win in the typical case, so my guess is that it's better to start simple.

The more-elaborate version looks like it’s more likely to win on locality of reference as it scans blocks sequentially. That could be significant… especially if anyone is so unfortunate as to still run Emacs on a system small enough that has to page out part of the Lisp memory.

I like the second approach a bit better, but either way, getting away from using the C stack sounds like a big improvement.

Ken

What improvements would be truly useful?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

An improvement in GC wouldn't be a bad thing, but it may not be worth
the effort.  It is likely to lead to many bugs that would be hard to
fix.  Once working, it would not make much difference to users.
It would only permit some operations on larger problems than now.

When I was working at the AI Lab, one of the older programmers told me
that hackers are often eager to make improvements of this sort: which
make the program better in an abstract sense, but not better for
users.  I took that advice to heart.  Now I pass it on.

Changing Emacs to handle indentation and alignment with
variable-width fonts would be an important and useful change.
Certain kinds of use would make sense, which currently don't.

It would be a big step towards making Emacs do the job of
a word processor, which is what I would like to see some day.
Imagine if you could edit nicely formatted documents directly
with Emacs, instead of using LibreOffice?  LibreOffice is
fine to use, it is free software, but it isn't Emacs.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: What improvements would be truly useful?

[John Yates]

Richard,

I too had an early mentor impart similar wisdom, possibly around the
same that time yours did.  Since then I have always tried to keep the
end user experience foremost in my work.

"Many bugs that would be hard to fix" is not a given.  I cannot count
the number times I have replaced recursion with iteration.  With
discipline (no other changes to the algorithm) it is a straight
forward conversion.

I agree that merely being able to handle large files is not a strong
justification.  OTOH, in spite of running on a state of the art
desktop with 64GB RAM, 4GHz+ multi-core processor and high-end gamer
graphics Emacs feels progressively less responsive.  Any work
countering that trend gets my vote.

/john

Re: What improvements would be truly useful?

[Stefan Monnier]

> An improvement in GC wouldn't be a bad thing, but it may not be worth
> the effort.

Making the recursion use an explicitly managed stack should be a fairly
simple change, which will fix some crashes.  I agree that it's probably
not worth trying to be too fancy, but it's really just a bug fix.


        Stefan

Re: What improvements would be truly useful?

[Paul Eggert]

On 03/05/2018 06:02 AM, John Yates wrote:
> in spite of running on a state of the art
> desktop with 64GB RAM, 4GHz+ multi-core processor and high-end gamer
> graphics Emacs feels progressively less responsive

I'm feeling that on my older work desktop too (8 GiB RAM, 2.6 GHz 
quad-core, circa-2010 AMD Deneb). However, I suspect it has mostly to do 
with things other than GC.

The main argument to improve GC is robustness, not typical-case 
performance. We're still installing last-minute hacks into Emacs 26 
because of real-world problems here. It would be useful if we didn't 
have to mess with these hacks because we fixed GC to not blow the C stack.

Re: What improvements would be truly useful?

[Rostislav Svoboda]

> It would be a big step towards making Emacs do the job of
> a word processor, which is what I would like to see some day.
> Imagine if you could edit nicely formatted documents directly
> with Emacs, instead of using LibreOffice?  LibreOffice is
> fine to use, it is free software, but it isn't Emacs.

IMO the future of GUI apps and in our case editors, lies in browser
based frameworks programmable by some lisp dialect. For the moment it
would be a stack based on electron, clojurescript, codemirror, etc.

I know it's a bitter pill to swallow, but let's face it - do we think,
our bellowed Emacs will ever be able to display anything like the
examples from https://threejs.org ?

Bost

Re: Variable-width font indentation (was: What improvements would be truly useful?)

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Date: Mon, 05 Mar 2018 08:11:38 -0500
> Cc: eggert@cs.ucla.edu, dancol@dancol.org, emacs-devel@gnu.org
> 
> Changing Emacs to handle indentation and alignment with
> variable-width fonts would be an important and useful change.
> Certain kinds of use would make sense, which currently don't.

This has come up several times, and I think I asked at least once what
does it need to entail.  It would be helpful to have the answer to
that, which will have to be the result of looking at the relevant Lisp
code and figuring out which primitives/subroutines will need to be
taught fractional column values.

Emacs already knows how to align text at pixel resolution (which is
easy to express in fractions of the width of the default face's font),
and in fact already does so when it displays TAB characters on GUI
frames.  We have the 'space' display property to do the same with
characters other than a TAB.

So what else is needed?  Can someone who is familiar with indent.el
please tell?

Re: What improvements would be truly useful?

[Eli Zaretskii]

> From: Rostislav Svoboda <rostislav.svoboda@gmail.com>
> Date: Mon, 5 Mar 2018 18:32:06 +0100
> Cc: "emacs-devel@gnu.org Development" <emacs-devel@gnu.org>,
> 	Ken Raeburn <raeburn@raeburn.org>, dancol@dancol.org,
> 	Paul Eggert <eggert@cs.ucla.edu>
> 
> I know it's a bitter pill to swallow, but let's face it - do we think,
> our bellowed Emacs will ever be able to display anything like the
> examples from https://threejs.org ?

Can you display something similar in Office?

Re: What improvements would be truly useful?

[Daniele Nicolodi]

On 3/5/18 6:11 AM, Richard Stallman wrote:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
> An improvement in GC wouldn't be a bad thing, but it may not be worth
> the effort.  It is likely to lead to many bugs that would be hard to
> fix.  Once working, it would not make much difference to users.
> It would only permit some operations on larger problems than now.
> 
> When I was working at the AI Lab, one of the older programmers told me
> that hackers are often eager to make improvements of this sort: which
> make the program better in an abstract sense, but not better for
> users.  I took that advice to heart.  Now I pass it on.
> 
> Changing Emacs to handle indentation and alignment with
> variable-width fonts would be an important and useful change.
> Certain kinds of use would make sense, which currently don't.
> 
> It would be a big step towards making Emacs do the job of
> a word processor, which is what I would like to see some day.
> Imagine if you could edit nicely formatted documents directly
> with Emacs, instead of using LibreOffice?  LibreOffice is
> fine to use, it is free software, but it isn't Emacs.

Richard,

with all the due respect for your role in starting the Emacs project and
the Free software movement, I don't think that dismissing contributions
to Emacs on the base of their relative usefulness to implementing new
features that you think are important, does any good to Emacs.

Hackers work on Emacs because they like it, and scratch their own itch.
I don't think that anyone is in the position to suggest how they should
be spending their time. Even more so if (as far as I know) there is no
agreed upon roadmap for the project.

I think that what will help to keep Emacs relevant in the long run is
modernizing its structure and code base, moving away from the baroque
architecture that developed as a result of its very long history (and
some less than optimally future proof design decisions).

Personally, I find that better support for display with variable width
fonts will not improve my use of Emacs. What would improve my user
experience would be improvements to the display engine that will allow a
better document viewing capabilities in something like pdf-tools (like
continuous scrolling).

The only effective (and not detrimental to the project) way to steer
hacker attentions to features you find important is to pay them for
their work. With your popularity it should not be impossible to rise
enough money to do so.

Cheers,
Dan

Re: What improvements would be truly useful?

[Aaron Ecay]

2018ko martxoak 5an, Rostislav Svoboda-ek idatzi zuen:
> 

[...]

> I know it's a bitter pill to swallow, but let's face it - do we think,
> our bellowed Emacs will ever be able to display anything like the
> examples from https://threejs.org ?

Thanks to the xwidgets support (info "(elisp) Xwidgets"), it is possible
to embed a graphical web browser with full JS support in an emacs buffer.
So itʼs possible to have the best of both worlds: emacsʼs support for
text editing combined with the visualization and interactivity of modern
Javascript libraries.

Iʼm sure that there are ways that graphical programming (for lack of
a better term) in emacs could be made better, including Richardʼs
suggestions that launched this thread.  But we should be proud of
what we already have, rather than pessimistic.

Aaron

PS I donʼt know of any projects using three.js support in xwidgets
specifically; it could turn out that it doesnʼt quite work
(e.g. hardware-accelerated 3D or lack thereof might present an
obstacle).  But anyone can try it and find out, and hopefully if
problems arise figure out how to fix them.

-- 
Aaron Ecay

Re: What improvements would be truly useful?

[Daniel Colascione]

On 03/05/2018 10:04 AM, Eli Zaretskii wrote:
>> From: Rostislav Svoboda <rostislav.svoboda@gmail.com>
>> Date: Mon, 5 Mar 2018 18:32:06 +0100
>> Cc: "emacs-devel@gnu.org Development" <emacs-devel@gnu.org>,
>> 	Ken Raeburn <raeburn@raeburn.org>, dancol@dancol.org,
>> 	Paul Eggert <eggert@cs.ucla.edu>
>>
>> I know it's a bitter pill to swallow, but let's face it - do we think,
>> our bellowed Emacs will ever be able to display anything like the
>> examples from https://threejs.org ?

It can already, with XEmbed --- same way Office does it, with OLE. :-) 
It's a nice trick, but I don't think that sort of pretty display helps 
anyone accomplish a task.

Personally, I don't think word processing is a good focus for Emacs. 
There are two groups of people who want to prepare documents: those who 
want a WYSIWYG system and those who don't. The former group is 
well-served by LibreOffice, which is a free and powerful office suite. 
The latter group is well-served by Emacs with its extensive LaTeX 
integration.

Instead of focusing on areas where we're weak and will realistically 
never catch up with projects dedicated to the task, we should focus on 
improving existing strengths.

1) We should be the best editor around for text and program code. 
There's an opportunity to do much better than the mainstream. 
Conventional IDE groups put a ton of brute force effort into tuning IDEs 
for specific coding styles in specific environments. We can be more 
generic and more flexible, ultimately offering more power and greater 
efficiency for people willing to invest time into learning the system.

a) We should do a better job of integrating interesting ideas like 
undo-tree, ace-jump-mode, yasnippet, helm, and others into the core and 
enabling them by default. I don't think we need to be as conservative as 
we've been historically, and I think there's still a lot of room to 
improve the core editing mechanics.

b) There are long-standing defects that are minor in the scheme of 
things, but that tend to create a poor impression. In particular, 
long-line handling is a sore point, as is support for very large files. 
For long lines: I haven't sufficiently studied what the necessary 
redisplay hacks would look like.

For large files: by moving from a gap buffer to a rope representation 
for buffers, we can partially use memory-mapped backing storage, and 
even when we do need private, modifiable memory for editing, we can 
allocate only when we immediately need and not have to move the gap 
around through humongous amounts of main memory. Such a system would not 
only improve our support for humongous files, but would also make a 
32-bit Emacs capable of editing files larger than its address space.

c) We need a project system. There's been some good work in this area, 
but there's too much fragmentation, which hinders productive 
integration. For example, there's no default keybinding to jump, in C++, 
between an "implementation" and a "header" file, and that's because 
Emacs by default has no idea what either concept means and there are 
something like, what, a dozen(?) different ways to teach it the concept.

d) We need better fontification and indentation. We don't have good 
language coverage, and support for more obscure languages is sometimes 
spotty, limited to fontifying comments, strings, and keywords. Keeping 
up with language development is a constant struggle, and it's easy to 
introduce odd bugs, infloops, and so on in ad-hoc parsing code, 
especially when this code needs to be simultaneously fast, incremental, 
and error tolerant.

I'm now wondering whether the manual approach is wrong. We've been using 
it along with everyone else, but there might be better options these 
days. It's a somewhat radical idea: let's use a machine learning model 
to classify program tokens, then apply manual fontification and 
indentation rules to the resulting token classifications. We'd train the 
model by taking labeled program text (say, from Savannah or GitHub, run 
through a parser), then perturb the program text, rewarding the model 
for retaining token labels under various editing and truncation operations.

In this way, we'd learn an approximate model for understanding even 
damaged program text without having to manually write a lot of code. 
Tons and tons of stuff in cc-mode is heuristics for dealing with damaged 
program text, and I think we could learn this understanding instead. The 
system is equivalent in power to anything we could write by hand: LSTMs 
and other systems are Turing-complete. This way, to add support for a 
new language, you'd just feed Emacs examples. I imagine you might even 
be able to gently correct the system when it misunderstands and improve 
the overall accuracy.

But it's probably a crazy idea. :-)

2) Startup should be instant in all cases. Now that we have a portable 
dumper, we should automatically dump the results of user initialization 
and regenerate the dump when we detect that something's changed. This 
way, users perceive Emacs as a fast, modern system. I know that the 
daemon exists and that it's possible to optimize even a customized 
initialization so that it's fast even without hacks (I do), but users 
shouldn't have to go to the trouble of this kind of manual setup and 
tweaking

3) Mobile support. One of Emacs' strengths is its portability, and this 
portability comes with a much lower footprint than other approaches. 
Desktop and laptop sales have been declining for six years. There are 
lots of tools built on Emacs that would be useful (like gnus, org-mode, 
etc.), except that I can't use them on mobile, which means I end up 
choosing other tools entirely for these tasks.

There is no reason that Emacs couldn't be a good Android citizen. A good 
Java<->elisp bridge would let us transparently use various system APIs. 
While we would probably need mobile-specific GUI code (because the plain 
buffer interface wouldn't be suitable for most tasks, at least without 
mobile-desktop convergence), all the logic and back-end glue would work 
on mobile as well as it works anywhere else, greatly simplifying the 
task of building general-purpose tools like org-mode that really ought 
to work anywhere.

Re: What improvements would be truly useful?

[Eli Zaretskii]

> From: Daniele Nicolodi <daniele@grinta.net>
> Date: Mon, 5 Mar 2018 11:51:09 -0700
> 
> I think that what will help to keep Emacs relevant in the long run is
> modernizing its structure and code base, moving away from the baroque
> architecture that developed as a result of its very long history (and
> some less than optimally future proof design decisions).

What exactly are you talking about here?  AFAIK, the Emacs
architecture didn't change since its inception, so its long history
has no relevance here.  But maybe I'm missing something or
misunderstanding what you intended to convey.

> What would improve my user experience would be improvements to the
> display engine that will allow a better document viewing
> capabilities in something like pdf-tools (like continuous
> scrolling).

AFAIU, that's already possible from the display engine POV, what's
missing is application-level Lisp code that would take advantage of
the infrastructure.

Re: Variable-width font indentation (was: What improvements would be truly useful?)

[dancol]

>> From: Richard Stallman <rms@gnu.org>
>> Date: Mon, 05 Mar 2018 08:11:38 -0500
>> Cc: eggert@cs.ucla.edu, dancol@dancol.org, emacs-devel@gnu.org
>>
>> Changing Emacs to handle indentation and alignment with
>> variable-width fonts would be an important and useful change.
>> Certain kinds of use would make sense, which currently don't.
>
> This has come up several times, and I think I asked at least once what
> does it need to entail.  It would be helpful to have the answer to
> that, which will have to be the result of looking at the relevant Lisp
> code and figuring out which primitives/subroutines will need to be
> taught fractional column values.
>
> Emacs already knows how to align text at pixel resolution (which is
> easy to express in fractions of the width of the default face's font),
> and in fact already does so when it displays TAB characters on GUI
> frames.  We have the 'space' display property to do the same with
> characters other than a TAB.
>
> So what else is needed?  Can someone who is familiar with indent.el
> please tell?

Well, say you want to align the following code:

1: void foo(int a,
2:          int b)
3: {
4:  ...

How is any generic indentation system supposed to know that the "i" on
line 2 is supposed to line up with the "i" on line 1? You'd need to
communicate all indentation intent to the generic system in the form of
positions relative to some anchor. (Presumably, it could cache pixel
offsets per-line.) While that's doable, it's a ton of work, and I don't
think it's worthwhile.

Re: Variable-width font indentation

[Paul Eggert]

On 03/05/2018 11:32 AM, dancol@dancol.org wrote:
> 1: void foo(int a,
> 2:          int b)
> 3: {
> 4:  ...
>
> How is any generic indentation system supposed to know that the "i" on
> line 2 is supposed to line up with the "i" on line 1?

It assumes the input is fixed-width, and that leading white space is 
intended to indent with respect to previous lines. Although it'd be some 
work to get Emacs to do this, I don't offhand see any algorithmic 
problem here.

Re: Variable-width font indentation (was: What improvements would be truly useful?)

[Eli Zaretskii]

> Date: Mon, 5 Mar 2018 11:32:30 -0800
> From: dancol@dancol.org
> Cc: rms@gnu.org,
>  raeburn@raeburn.org,
>  eggert@cs.ucla.edu,
>  dancol@dancol.org,
>  emacs-devel@gnu.org
> 
> Well, say you want to align the following code:
> 
> 1: void foo(int a,
> 2:          int b)
> 3: {
> 4:  ...
> 
> How is any generic indentation system supposed to know that the "i" on
> line 2 is supposed to line up with the "i" on line 1?

How does it know that today, when we only support well the fixed-pitch
fonts?  Or are you talking about a different problem?

> You'd need to communicate all indentation intent to the generic
> system in the form of positions relative to some
> anchor. (Presumably, it could cache pixel offsets per-line.) While
> that's doable, it's a ton of work, and I don't think it's
> worthwhile.

Maybe I still misunderstand you, but the pixel position of a character
on display is trivial to obtain, and I see no reason why we'd need to
cache them.  Again, the current fixed-pitch column-wise indentation
machinery doesn't, AFAIK.

Re: What improvements would be truly useful?

[Eli Zaretskii]

> From: Daniel Colascione <dancol@dancol.org>
> Date: Mon, 5 Mar 2018 11:18:29 -0800
> Cc: eggert@cs.ucla.edu, raeburn@raeburn.org, rms@gnu.org, emacs-devel@gnu.org
> 
> For long lines: I haven't sufficiently studied what the necessary 
> redisplay hacks would look like.

I explained the issues several times, here and elsewhere.  I can
explain again, if someone's interested.

Re: Variable-width font indentation

[Clément Pit-Claudel]

On 2018-03-05 14:58, Eli Zaretskii wrote:
> How does it know that today, when we only support well the
> fixed-pitch fonts?  Or are you talking about a different problem?

A different problem, IIUC.  The problem is the following:

Assume you wrote the following C code, then changed to variable-pitch:

void pr () {
  printf ("hello, world\n");
}

int main (int argc, 
          char** argv) {
  pr ();
  return 0;
}

Depending on which font you use (and the width of a space in that font), the `char** argv' part will move right or left, instead of remaining aligned.  The usual solution is to proceed as Paul suggested:

> It assumes the input is fixed-width, and that leading white space is
> intended to indent with respect to previous lines.

IOW, you scale the spaces preceding `char** argv' to make the `c' of `char' line up with the `i' of int.

Applying this strategy to all lines doesn't work, unfortunately: it indents the `pr ()' and `printf (' lines differently, because the `in' of `int main' and the `vo' of `void' occupy different widths.

I don't know of a good solution to this problem: in general, there's not enough information to distinguish 'indentation' spaces (the two spaces before `printf (', `pr ()', and `return 0') and 'line-up' spaces (the spaces before `char ()').

I hope I'm missing something.

Clément.

Re: What improvements would be truly useful?

[Ricardo Wurmus]

Aaron Ecay <aaronecay@gmail.com> writes:

> 2018ko martxoak 5an, Rostislav Svoboda-ek idatzi zuen:
>>
>
> [...]
>
>> I know it's a bitter pill to swallow, but let's face it - do we think,
>> our bellowed Emacs will ever be able to display anything like the
>> examples from https://threejs.org ?
>
> Thanks to the xwidgets support (info "(elisp) Xwidgets"), it is possible
> to embed a graphical web browser with full JS support in an emacs buffer.
> So itʼs possible to have the best of both worlds: emacsʼs support for
> text editing combined with the visualization and interactivity of modern
> Javascript libraries.

Communication with the webkit widget is not easy.  It certainly doesn’t
feel like a first-class citizen in Emacs.  It is possible to send
strings containing a JavaScript programme to the webkit widget and
retrieve simple strings from the widget, but there is no shared state.
All values crossing over the boundary between the Elisp world and the
widget need to be converted first.

We have no way to compile Elisp to JavaScript strings to smoothen out
the rough transition between these two worlds.  (Guile supports
ECMAscript and Elisp, but even in Guile Emacs the webkit widget would be
a foreign body.)

While it is a neat hack to embed an X widget in an Emacs buffer, I don’t
think that we should be using the embedded webkit widget as a display
engine.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: What improvements would be truly useful?

[Juri Linkov]

> What would improve my user experience would be improvements to the
> display engine that will allow a better document viewing capabilities
> in something like pdf-tools (like continuous scrolling).

Continuous scrolling can be easily implemented with a simple hack
to insert adjacent images for the previous/next pdf pages in
the same DocView buffer.

But what would be much more strategically advantageous is to employ
the Blink rendering engine as an alternative Emacs display engine.

Re: What improvements would be truly useful?

[John Wiegley]

>>>>> "DN" == Daniele Nicolodi <daniele@grinta.net> writes:

DN> Personally, I find that better support for display with variable width
DN> fonts will not improve my use of Emacs. What would improve my user
DN> experience would be improvements to the display engine that will allow a
DN> better document viewing capabilities in something like pdf-tools (like
DN> continuous scrolling).

I have to agree with Daniele here. I don't use Emacs for writing such
documents, nor would having variable width support by itself grant all the
other features such users have come to expect from modern Word Processors.

Time spent on a better GC could help the *overall* user experience, in the
form of generally better latency, even if no one would be able to point to the
GC as the reason for their better experience with Emacs.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Variable-width font indentation

[Paul Eggert]

On 03/05/2018 12:28 PM, Clément Pit-Claudel wrote:
> Applying this strategy to all lines doesn't work, unfortunately: it indents the `pr ()' and `printf (' lines differently, because the `in' of `int main' and the `vo' of `void' occupy different widths.

True, but that's a feature not a defect. The code is easier to read if 
every region is indented individually, as opposed to using exactly the 
same indenting everywhere in the buffer.

Re: Variable-width font indentation

[Daniel Colascione]

On 03/05/2018 02:30 PM, Paul Eggert wrote:
> On 03/05/2018 12:28 PM, Clément Pit-Claudel wrote:
>> Applying this strategy to all lines doesn't work, unfortunately: it 
>> indents the `pr ()' and `printf (' lines differently, because the `in' 
>> of `int main' and the `vo' of `void' occupy different widths.
> 
> True, but that's a feature not a defect. The code is easier to read if 
> every region is indented individually, as opposed to using exactly the 
> same indenting everywhere in the buffer.

In non-lineup styles (e.g., the Linux kernel), different blocks with the 
same notional indentation actually having different offsets from the 
left margin would be very unusual.

Re: What improvements would be truly useful?

[daniel sutton]

[-- Attachment #1: Type: text/plain, Size: 1244 bytes --]

I also really like Daniel's passion and ideas. I think he has been
contributing new ideas and plenty of code to back it up. Seems like
something to be encouraged.

On Mon, Mar 5, 2018 at 4:09 PM, John Wiegley <johnw@gnu.org> wrote:

> >>>>> "DN" == Daniele Nicolodi <daniele@grinta.net> writes:
>
> DN> Personally, I find that better support for display with variable width
> DN> fonts will not improve my use of Emacs. What would improve my user
> DN> experience would be improvements to the display engine that will allow
> a
> DN> better document viewing capabilities in something like pdf-tools (like
> DN> continuous scrolling).
>
> I have to agree with Daniele here. I don't use Emacs for writing such
> documents, nor would having variable width support by itself grant all the
> other features such users have come to expect from modern Word Processors.
>
> Time spent on a better GC could help the *overall* user experience, in the
> form of generally better latency, even if no one would be able to point to
> the
> GC as the reason for their better experience with Emacs.
>
> --
> John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
> http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2
>
>

[-- Attachment #2: Type: text/html, Size: 1834 bytes --]

Re: What improvements would be truly useful?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > IMO the future of GUI apps and in our case editors, lies in browser
  > based frameworks programmable by some lisp dialect. For the moment it
  > would be a stack based on electron, clojurescript, codemirror, etc.

I have a feeling we are talking about totally unrelated questions.
I'm talking about extending Emacs to edit more kinds of _documents_.
Documents with more and different layout features, but still _documents_.

A document is a self-contained file of declarative data that you can
display and edit on your computer, without any sort of network
connection.  And if you send it to me, I can display it too
and it looks the same for me as for you.

It is crucial for a document to be declarative.  It is not safe for an
editor to run unrestricted software out of a document.  That's why
Emacs is careful to make sure documents can't run code.

I looked at threejs.org, and it has an array of interesting images
which seem to have something to do with Javascript libraries.  They
may be useful for other purposes, but not for an editor for documents.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: Variable-width font indentation (was: What improvements would be truly useful?)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > This has come up several times, and I think I asked at least once what
  > does it need to entail.

It means, for one thing, that M-q can fill text with a smooth
right margin.

Has that been implemented yet?

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: What improvements would be truly useful?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Personally, I don't think word processing is a good focus for Emacs. 

I want to do my word processing in Emacs,
so please stop interfering.  You, personally, can work on other
areas if that area doesn't interest you.

The improvements you suggested for editing programs are also
desirable.  We want to improve what Emacs can do in editing programs.
One of the improvements we need is to have a GNU language server and
connect it to Emacs.  The best proposal for how to make a GNU language
server is to do it by building on GDB.

However, Emacs is also meant for editing textual documents.
To make that easier, we need more support for editing and
saving formatted documents with various fonts.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: What improvements would be truly useful?

[dancol]

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > Personally, I don't think word processing is a good focus for Emacs.
>
> I want to do my word processing in Emacs,
> so please stop interfering.  You, personally, can work on other
> areas if that area doesn't interest you.

You are advocating for your suggested improvements. I am advocating for
mine. Neither of us "interfering" with the other.

Re: What improvements would be truly useful?

[Óscar Fuentes]

Juri Linkov <juri@linkov.net> writes:

>> What would improve my user experience would be improvements to the
>> display engine that will allow a better document viewing capabilities
>> in something like pdf-tools (like continuous scrolling).
>
> Continuous scrolling can be easily implemented with a simple hack
> to insert adjacent images for the previous/next pdf pages in
> the same DocView buffer.

I've looked into implementing this feature and it is not as simple as it
seems. The hack you mention is precisely that: a hack, and a very cranky
one. AFAIK there is no way of implementing continous scrolling with
adjacent images if you care at all about robustness and user experience.
And the fact that people is asking for this feature for years and, to
this day, nobody implemented it, seems to confirm my impressions.

Re: Variable-width font indentation

[Clément Pit-Claudel]

On 2018-03-05 17:30, Paul Eggert wrote:
> On 03/05/2018 12:28 PM, Clément Pit-Claudel wrote:
>> Applying this strategy to all lines doesn't work, unfortunately: it
>> indents the `pr ()' and `printf (' lines differently, because the
>> `in' of `int main' and the `vo' of `void' occupy different widths.
> 
> True, but that's a feature not a defect. The code is easier to read
> if every region is indented individually, as opposed to using exactly
> the same indenting everywhere in the buffer.

Hmm.  I hadn't thought of this :)  But I'm not entirely convinced yet.  Consider this snippet:

def x():
    with open("A") as f:
        while y:
            print(y.v())

or this one in elisp:

(when x
  (if y t
    (unless p
      (cl-loop for x in y
               collect x)))) 

In variable-pitch mode, each new indentation level has a slightly-different width, instead of a regular progression of 4 spaces (in Python) and 2 spaces (in Elisp) at each level.

Beyond this, there's the problem of inline line-up spaces.  For an example, consider this, from xdisp.c, which shows "line-up" spaces in full glory:


   +--------------+   redisplay     +----------------+
   | Lisp machine |---------------->| Redisplay code |<--+
   +--------------+   (xdisp.c)     +----------------+   |
	  ^				     |		 |
	  +----------------------------------+           |
	    Block input to prevent this when             |
	    called asynchronously!			 |
							 |
		    note_mouse_highlight (asynchronous)	 |
							 |
				    X mouse events  -----+
							 |
			    expose_frame (asynchronous)	 |
							 |
				   X expose events  -----+

The same problem pops up with org-mode tables, etc.

Clément.

Re: Variable-width font indentation

[Paul Eggert]

On 03/05/2018 05:06 PM, Clément Pit-Claudel wrote:
> In variable-pitch mode, each new indentation level has a slightly-different width, instead of a regular progression of 4 spaces (in Python) and 2 spaces (in Elisp) at each level.

That's fine. Although it's only a small thing, to me it's even a small 
plus, as having every indent level be the same number of pixels 
unnaturally focuses the reader on a distracting regularity that is not 
intrinsic to the code.

> Beyond this, there's the problem of inline line-up spaces.

Yes, for ASCII art I doubt whether any simple automated heuristic would 
work unaided. ASCII art is typically ugly and hard-to-read even in a 
fixed-width font, so it's not much of a loss if it's rendered poorly. If 
there's really a need for it I suppose we could let users hint to Emacs 
to switch to a fixed-width font just for the art; that'd be good enough.

Re: Variable-width font indentation

[Clément Pit-Claudel]

On 2018-03-05 20:30, Paul Eggert wrote:
> On 03/05/2018 05:06 PM, Clément Pit-Claudel wrote:
>> In variable-pitch mode, each new indentation level has a
>> slightly-different width, instead of a regular progression of 4
>> spaces (in Python) and 2 spaces (in Elisp) at each level.
> 
> That's fine. Although it's only a small thing, to me it's even a
> small plus, as having every indent level be the same number of pixels
> unnaturally focuses the reader on a distracting regularity that is
> not intrinsic to the code.

I've been playing with this a bit more; until now, I'd mostly stopped at my previous objection.

The following code gives a preview of what the algorithm that we've been discussing produces:

(defun ~/variable-pitch-indent ()
  (interactive)
  (save-excursion
    (let ((spine "")
          (face `(:foreground ,(face-attribute 'default :background nil))))
      (goto-char (point-min))
      (while (not (eobp))
        (back-to-indentation)
        (unless (eolp)
          (let* ((indentation-amount (current-column))
                 (spine-cut (min (length spine) indentation-amount))
                 (spine-prefix (substring spine 0 spine-cut))
                 (spine-padding (make-string (- indentation-amount spine-cut) ?\s))
                 (spine-end (buffer-substring (point) (point-at-eol))))
            (setq spine
                  (with-temp-buffer
                    ;; HACK: expand spaces
                    (setq-local tab-width 8)
                    (insert spine-prefix spine-padding spine-end)
                    (untabify (point-min) (point-max))
                    (buffer-string)))
            (put-text-property (point-at-bol) (point) 'display
                               (propertize (concat spine-prefix spine-padding)
                                           'face face))))
        (forward-line 1)))))

I've tested it on two files: src/termcap.c and lisp/dabbrev.el.  To try it out, open one of these files in emacs -Q, then run M-x ~/variable-pitch-indent, and then M-x variable-pitch-mode.

I do agree that it doesn't look too bad, and presumably a C implementation of the algorithm above would be very fast, since it could build the "spine" above during redisplay.

Clément.

Re: Variable-width font indentation

[Paul Eggert]

On 03/05/2018 05:40 PM, Clément Pit-Claudel wrote:
> I've tested it on two files: src/termcap.c and lisp/dabbrev.el.  To try it out, open one of these files in emacs -Q, then run M-x ~/variable-pitch-indent, and then M-x variable-pitch-mode.
>
> I do agree that it doesn't look too bad, and presumably a C implementation of the algorithm above would be very fast, since it could build the "spine" above during redisplay.

Thanks, that was fast! Yes, it should work fast.

I saw one easily-fixable minor glitch (on my screen lisp/dabbr.el line 
533 had the wrong indent, presumably since the "user-error" on the 
previous line was in a fatter-than usual font). The biggest trickier 
minor glitch was that indenting of #if and #else subparts were 
annoyingly different because the "#" violates the usual indentation 
rule. Personally I've always thought that we should indent "#" like 
anything else -- the only reason we don't is that it didn't work with 
K&R C and so we got used to bad style -- but even if we stick with the 
bad "#if" indenting it still looks pretty reasonable.

Re: Variable-width font indentation

[Clément Pit-Claudel]

On 2018-03-05 21:04, Paul Eggert wrote:
> On 03/05/2018 05:40 PM, Clément Pit-Claudel wrote:
>> I've tested it on two files: src/termcap.c and lisp/dabbrev.el.  To try it out, open one of these files in emacs -Q, then run M-x ~/variable-pitch-indent, and then M-x variable-pitch-mode.
>>
>> I do agree that it doesn't look too bad, and presumably a C implementation of the algorithm above would be very fast, since it could build the "spine" above during redisplay.
> 
> Thanks, that was fast! Yes, it should work fast.
> 
> I saw one easily-fixable minor glitch (on my screen lisp/dabbr.el line 533 had the wrong indent, presumably since the "user-error" on the previous line was in a fatter-than usual font).

Yes, that's due to the implementation trick that I used, which doesn't preserve faces on the "spine", the string used for indentation.

The biggest trickier minor glitch was that indenting of #if and #else subparts were annoyingly different because the "#" violates the usual indentation rule. Personally I've always thought that we should indent "#" like anything else -- the only reason we don't is that it didn't work with K&R C and so we got used to bad style -- but even if we stick with the bad "#if" indenting it still looks pretty reasonable.

Indeed.  It does look fairly OK, actually — much better than I expected it to be.

I guess the next step would be to make a minor mode to apply this variable-pitch indentation on the fly and see how well it works in real life.

Some difficulties:

* Changing a line will cause all lines below it to get reindented; this will make redisplay costly, and I'm not sure how good it'll look — and we won't be able to tell until we get a C implementation, since an ELisp implementation likely won't update indentation for following lines on the fly.
* I'm not aware of a way to get a specified space whose width equals that of a particular string, without measuring the string beforehand.  Is there a way?

Clément.

Re: What improvements would be truly useful?

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Tue, 06 Mar 2018 00:46:27 +0100
> 
> I've looked into implementing this feature and it is not as simple as it
> seems. The hack you mention is precisely that: a hack, and a very cranky
> one. AFAIK there is no way of implementing continous scrolling with
> adjacent images if you care at all about robustness and user experience.

Can you explain the obstacles that prevent implementing this?  We have
vscroll support in the display engine, so I think the implementation
should be quite easy.  But maybe I'm missing something.

> And the fact that people is asking for this feature for years and, to
> this day, nobody implemented it, seems to confirm my impressions.

That could be because people are unaware of some crucial feature, or
because no one with the know-how had enough motivation to code this,
or for any number of other reasons.  Not necessarily because it's
impossible.

Re: Variable-width font indentation

[Herring, Davis]

> I don't know of a good solution to this problem: in general,
> there's not enough information to distinguish 'indentation'
> spaces (the two spaces before `printf (', `pr ()', and `return 0')
> and 'line-up' spaces (the spaces before `char ()').

This is of course an argument in favor of "smart tabs" <https://www.emacswiki.org/emacs/SmartTabs>: you make each tab occupy some user-specified width, and _then_ count one character per space to do alignment.  That doesn't help if those tabs aren't present, although heuristics to detect certain exact numbers of spaces, comparing to the previous lines and looking for enclosing delimiters, might close enough of the gap between "treat all spaces as ASCII art alignment" and the ideal.

Davis

Re: Variable-width font indentation

[Daniel Colascione]

On 03/05/2018 06:04 PM, Paul Eggert wrote:
> On 03/05/2018 05:40 PM, Clément Pit-Claudel wrote:
>> I've tested it on two files: src/termcap.c and lisp/dabbrev.el.  To 
>> try it out, open one of these files in emacs -Q, then run M-x 
>> ~/variable-pitch-indent, and then M-x variable-pitch-mode.
>>
>> I do agree that it doesn't look too bad, and presumably a C 
>> implementation of the algorithm above would be very fast, since it 
>> could build the "spine" above during redisplay.
> 
> Thanks, that was fast! Yes, it should work fast.
> 
> I saw one easily-fixable minor glitch (on my screen lisp/dabbr.el line 
> 533 had the wrong indent, presumably since the "user-error" on the 
> previous line was in a fatter-than usual font).

Right. Given sufficiently divergent character widgets, it'd be possible 
to trick the eye into thinking two blocks of code at semantically 
different indentation levels are in fact at different levels, or vice 
versa. IMHO, that's dangerous. I'd rather whitelist modes where the 
indentation machinery can reliably provide robust 
alignment-versus-indentation information.

Re: What improvements would be truly useful?

[Daniele Nicolodi]

On 05/03/2018 12:22, Eli Zaretskii wrote:
>> From: Daniele Nicolodi <daniele@grinta.net>
>> Date: Mon, 5 Mar 2018 11:51:09 -0700
>>
>> I think that what will help to keep Emacs relevant in the long run is
>> modernizing its structure and code base, moving away from the baroque
>> architecture that developed as a result of its very long history (and
>> some less than optimally future proof design decisions).
> 
> What exactly are you talking about here?  AFAIK, the Emacs
> architecture didn't change since its inception, so its long history
> has no relevance here.  But maybe I'm missing something or
> misunderstanding what you intended to convey.

I'm far from being familiar with the Emacs codebase thus I may be
reporting something that it is not completely true, however: Emacs was
born as a console only application, the graphical user interface seems
to be duct taped on.  Also, GTK support seems a bit of an hack that
requires layering violations (reaching down to the X primitives) to
work.  Being GTK the only modern toolkit supported on Linux (as far as I
know) and the only way to get nartive Wayland support, some radical
cleanup in that area would probably be a good thing.
>> What would improve my user experience would be improvements to the
>> display engine that will allow a better document viewing
>> capabilities in something like pdf-tools (like continuous
>> scrolling).
> 
> AFAIU, that's already possible from the display engine POV, what's
> missing is application-level Lisp code that would take advantage of
> the infrastructure.

Reading the pdf-tools bug report on the matter, I had the impression
that implementing this was not possible because of some Emacs
limitations.  I'm happy if that's not the case.  I'll go and investigate
more.

Cheers,
Dan

Re: What improvements would be truly useful?

[Yuri Khan]

On Tue, Mar 6, 2018 at 2:22 AM, Eli Zaretskii <eliz@gnu.org> wrote:

>> I think that what will help to keep Emacs relevant in the long run is
>> modernizing its structure and code base, moving away from the baroque
>> architecture that developed as a result of its very long history (and
>> some less than optimally future proof design decisions).
>
> What exactly are you talking about here?  AFAIK, the Emacs
> architecture didn't change since its inception, so its long history
> has no relevance here.  But maybe I'm missing something or
> misunderstanding what you intended to convey.

One architectural decision that I would call baroque is the input model.

In Emacs, the input model is that of the character terminal: the input
is a sequence of characters. The repertoire of Unicode characters is
extended with special values for function and editing keys, and for
keys modified with Ctrl, Super, Hyper, and Shift where necessary (and
Meta is represented as an ESC prefix), but still it is a sequence of
characters without explicit ties to the keys. Even mouse clicks are
processed as a special kind of characters.

This is inevitable when running in a terminal emulator where the
underlying abstraction works this way, but in a GUI build this leads
to inability to distinguish, for example, the ‘.’ character as coming
from the bottom-row right ring finger in US English layout, or from
the bottom-row right pinky in Russian layout, or from the decimal
separator key on the keypad. Bindings such as M-{ are unnecessarily
hard for international users whose keyboard layout does not have a {
character at all or puts it into third or fourth level.

Another example was described in Daniel Colascione’s article/rant
“Buttery Smooth Emacs”.

Re: Variable-width font indentation

[Clément Pit-Claudel]

On 2018-03-06 00:48, Daniel Colascione wrote:
> I'd rather whitelist modes where the indentation machinery can
> reliably provide robust alignment-versus-indentation information.

Do you think that'll be enough?  What do we do when we open a file that's already indented in a way that doesn't match what our own indentation function would do?

Or do you mean that we need a new "indentation analyzer" for each new mode that tags spaces as "alignment" spaces vs "indentation" spaces?

One more tricky case, from playing with this a bit more: in OCaml, it's not entirely uncommon to see this:

  match o with
  | None   -> 0 (* Note the extra spaces to line up the arrows *)
  | Some x -> x + 1

Clément.

Re: Variable-width font indentation

[Clément Pit-Claudel]

On 2018-03-05 23:05, Herring, Davis wrote:
>> I don't know of a good solution to this problem: in general,
>> there's not enough information to distinguish 'indentation'
>> spaces (the two spaces before `printf (', `pr ()', and `return 0')
>> and 'line-up' spaces (the spaces before `char ()').
> 
> This is of course an argument in favor of "smart tabs" <https://www.emacswiki.org/emacs/SmartTabs>

No, not really: smart tabs aren't enough AFAICT.  Take the following example:

	match opt with
	| []     -> []
	| x :: t -> x :: f t

You'd want to scale the spaces according to the line *below*, not ebove, and even then you'd have to account for the fact that 'x ' isn't as wide as '[]'.  Smart Tabs are also not very good for languages like ELisp with variable indentation depths.

If anything, I think this is more of an argument in favor of elastic tabs (http://nickgravgaard.com/elastic-tabstops/).

Clément.

Re: Variable-width font indentation

[Herring, Davis]

>         match opt with
>         | []     -> []
>         | x :: t -> x :: f t
> 
> You'd want to scale the spaces according to the line *below*, not ebove,

As per your other message, that's a problem regardless of tab handling.  Yet more heuristics suggest themselves (in this case, looking for text shared between two adjacent lines to guide the alignment), but of course those are becoming a mess.

> and even then you'd have to account for the fact that 'x ' isn't as wide as '[]'.

Maybe I misunderstand -- how is this a problem?  If the number of spaces is correct, it identifies the correct character to match against.

> Smart Tabs are also not very good for languages like ELisp with variable indentation depths.

Which cases are hard?  I just see several places where you would use only spaces (_):

(progn
>-(foo)
>-(bar))

(and
_(foo)
_(bar))

(and (foo)
_____(bar))

> If anything, I think this is more of an argument in favor of
> elastic tabs (http://nickgravgaard.com/elastic-tabstops/).

Those are certainly interesting, but they have their own ambiguities (e.g.,

int>-x>-= 1,
>-y>-=2;
doTheFoo();>-// it's true
doTheBar();>-// so far

where there should not be a "column block" 4 lines tall).  They also of course do not play well with existing rendering (monospace or otherwise).

Davis

Re: Variable-width font indentation

[Clément Pit-Claudel]

On 2018-03-06 09:18, Herring, Davis wrote:
> Maybe I misunderstand -- how is this a problem?  If the number of spaces is correct, it identifies the correct character to match against.

It's a problem if you scale each space to match the width of the character above it (or below it, in this case).  If you instead align to a particular position, you run into the issue of deciding which space to stretch (strech all spaces in the range evenly? stretch a single one? what about cases where the spaces need to be shrunk?)

> (progn
> >-(foo)
> >-(bar))
> 
> (and
> _(foo)
> _(bar))

I have a lot of trouble reading these examples, as the '>' get converted to reply markers by my email client; but looking at the source, I think I understand.  Question: why do you indent the body of a 'progn' differently from an 'and'?

I was thinking of cases like 'if', for example:

  (if test
      true-case
    false-case)

Where do you use tabs? 

> 
> int>-x>-= 1,
> >-y>-=2;
> doTheFoo();>-// it's true
> doTheBar();>-// so far

I don't think these are ambiguities (as in, the spec is clear on what should happen here); but indeed this doesn't render ideally; you'd write it like this instead:

int	x = 1,
	y = 2;

doTheFoo();	// it's true
doTheBar();	// so far

(I personally don't like the concept very much, because it doesn't let you distinguish between a 0-width indentation marker and a non-zero width indentation marker).  That is, it forces you to writ this:

	int f(	int argc;
		char** argv)

instead of 

	int f(int argc;
	      int argv)

Clément.

Re: Variable-width font indentation

[Dmitry Gutov]

On 3/5/18 9:49 PM, Paul Eggert wrote:
> On 03/05/2018 11:32 AM, dancol@dancol.org wrote:
>> 1: void foo(int a,
>> 2:          int b)
>> 3: {
>> 4:  ...
>>
>> How is any generic indentation system supposed to know that the "i" on
>> line 2 is supposed to line up with the "i" on line 1?
> 
> It assumes the input is fixed-width, and that leading white space is 
> intended to indent with respect to previous lines. Although it'd be some 
> work to get Emacs to do this, I don't offhand see any algorithmic 
> problem here.

If the way whitespace at bol is rendered depends on indentation code, 
will the mean that while scrolling a buffer the display engine will have 
to call indentation code?

Re: Variable-width font indentation

[Clément Pit-Claudel]

On 2018-03-06 10:15, Dmitry Gutov wrote:
> On 3/5/18 9:49 PM, Paul Eggert wrote:
>> On 03/05/2018 11:32 AM, dancol@dancol.org wrote:
>>> 1: void foo(int a,
>>> 2:          int b)
>>> 3: {
>>> 4:  ...
>>>
>>> How is any generic indentation system supposed to know that the "i" on
>>> line 2 is supposed to line up with the "i" on line 1?
>>
>> It assumes the input is fixed-width, and that leading white space is intended to indent with respect to previous lines. Although it'd be some work to get Emacs to do this, I don't offhand see any algorithmic problem here.
> 
> If the way whitespace at bol is rendered depends on indentation code, will the mean that while scrolling a buffer the display engine will have to call indentation code?

In Paul's suggestion (the one I implemented), the width of whitespace at bol does not depend on indentation code.

Re: What improvements would be truly useful?

[Dmitry Gutov]

On 3/5/18 9:18 PM, Daniel Colascione wrote:

>>> I know it's a bitter pill to swallow, but let's face it - do we think,
>>> our bellowed Emacs will ever be able to display anything like the
>>> examples from https://threejs.org ?
> 
> It can already, with XEmbed --- same way Office does it, with OLE. :-) 
> It's a nice trick, but I don't think that sort of pretty display helps 
> anyone accomplish a task.

Using the built-in SVG support (a tad more practical feature, IMO), 
someone can come pretty close as well.

> a) We should do a better job of integrating interesting ideas like 
> undo-tree, ace-jump-mode, yasnippet, helm, and others into the core and 
> enabling them by default. I don't think we need to be as conservative as 
> we've been historically, and I think there's still a lot of room to 
> improve the core editing mechanics.

I'm all for undo-tree, but as for the rest, including a particular 
workflow by default might make it harder to use competing ones (i.e. 
it's a hard pass for me on Helm).

> b) There are long-standing defects that are minor in the scheme of 
> things, but that tend to create a poor impression.

Long lines are annoying. Large files aren't hugely important, IMO 
(though I've had to deal with a couple over the recent months).

> c) We need a project system.

project.el? It sorely lacks in contributions (ideas and even rough 
prototypes are welcome).

> There's been some good work in this area, 
> but there's too much fragmentation, which hinders productive 
> integration. For example, there's no default keybinding to jump, in C++, 
> between an "implementation" and a "header" file, and that's because 
> Emacs by default has no idea what either concept means and there are 
> something like, what, a dozen(?) different ways to teach it the concept.

One difficulty with that is C++ is just one type of project, so adding 
this kind of command to all projects seems kind of iffy. And yes, 
arguments for picking between dozens different ways to approach this is 
the kind of contribution I'm looking for.

> d) We need better fontification and indentation. We don't have good 
> language coverage, and support for more obscure languages is sometimes 
> spotty, limited to fontifying comments, strings, and keywords. Keeping 
> up with language development is a constant struggle, and it's easy to 
> introduce odd bugs, infloops, and so on in ad-hoc parsing code, 
> especially when this code needs to be simultaneously fast, incremental, 
> and error tolerant.
> 
> I'm now wondering whether the manual approach is wrong. We've been using 
> it along with everyone else, but there might be better options these 
> days. It's a somewhat radical idea: let's use a machine learning model 
> to classify program tokens, then apply manual fontification and 
> indentation rules to the resulting token classifications. We'd train the 
> model by taking labeled program text (say, from Savannah or GitHub, run 
> through a parser), then perturb the program text, rewarding the model 
> for retaining token labels under various editing and truncation operations.

Err, we can start with fontification based on language specification 
like Atom/VS Code and Sublime done. They use JSON, but we don't have to.

In a way, those specs are harder to read than font-lock code, though.
> 3) Mobile support. One of Emacs' strengths is its portability, and this 
> portability comes with a much lower footprint than other approaches. 
> Desktop and laptop sales have been declining for six years. There are 
> lots of tools built on Emacs that would be useful (like gnus, org-mode, 
> etc.), except that I can't use them on mobile, which means I end up 
> choosing other tools entirely for these tasks.
> 
> There is no reason that Emacs couldn't be a good Android citizen. A good 
> Java<->elisp bridge would let us transparently use various system APIs. 
> While we would probably need mobile-specific GUI code (because the plain 
> buffer interface wouldn't be suitable for most tasks, at least without 
> mobile-desktop convergence), all the logic and back-end glue would work 
> on mobile as well as it works anywhere else, greatly simplifying the 
> task of building general-purpose tools like org-mode that really ought 
> to work anywhere.

I've been wondering about having Emacs on my phone for a while. It would 
really need a fitting UI added, though.

Re: Variable-width font indentation

[Stefan Monnier]

>   > This has come up several times, and I think I asked at least once what
>   > does it need to entail.
> It means, for one thing, that M-q can fill text with a smooth
> right margin.
> Has that been implemented yet?

AFAIK `shr` does that, yes.


        Stefan

Re: Variable-width font indentation

[Eli Zaretskii]

> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Mon, 5 Mar 2018 15:28:05 -0500
> Cc: Paul Eggert <eggert@cs.ucla.edu>
> 
> On 2018-03-05 14:58, Eli Zaretskii wrote:
> > How does it know that today, when we only support well the
> > fixed-pitch fonts?  Or are you talking about a different problem?
> 
> A different problem, IIUC.  The problem is the following:
> 
> Assume you wrote the following C code, then changed to variable-pitch:
> 
> void pr () {
>   printf ("hello, world\n");
> }
> 
> int main (int argc, 
>           char** argv) {
>   pr ();
>   return 0;
> }
> 
> Depending on which font you use (and the width of a space in that font), the `char** argv' part will move right or left, instead of remaining aligned.  The usual solution is to proceed as Paul suggested:
> 
> > It assumes the input is fixed-width, and that leading white space is
> > intended to indent with respect to previous lines.
> 
> IOW, you scale the spaces preceding `char** argv' to make the `c' of `char' line up with the `i' of int.

I think I'm still missing something, because I don't understand why we
need to invent/use a strategy different from what we do now in code
indentation.  All we need to do to extend the existing machinery to
variable-pitch fonts is (1) to replace current-column with a function
that returns floats instead of integers (or teach current-column do
that given some optional argument), and (b) place a 'space' display
property on the leading whitespace with an :align-to attribute
computed to align the first non-white character at the "column number"
returned as result of (a) above.  All the rest is already in place,
right?

Also note that Richard was specifically talking about text modes, not
about programming modes.  Indentation in text modes has a slightly
different goal.

Re: Variable-width font indentation (was: What improvements would be truly useful?)

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> CC: raeburn@raeburn.org, eggert@cs.ucla.edu, dancol@dancol.org,
> 	emacs-devel@gnu.org
> Date: Mon, 05 Mar 2018 18:05:11 -0500
> 
> It means, for one thing, that M-q can fill text with a smooth
> right margin.
> 
> Has that been implemented yet?

No, not yet.  But unless I'm missing something, doing that would be
relatively simple: we need a variant of justify-current-line which
will (a) compute the number of blanks as a float number, and (b) add a
'display' property to each blank with an :align-to attribute computed
(in pixels) to produce the smooth right margin.

Patches to that effect are welcome.

Re: Variable-width font indentation

[Eli Zaretskii]

> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Mon, 5 Mar 2018 17:30:24 -0800
> 
> Yes, for ASCII art I doubt whether any simple automated heuristic would 
> work unaided. ASCII art is typically ugly and hard-to-read even in a 
> fixed-width font, so it's not much of a loss if it's rendered poorly. If 
> there's really a need for it I suppose we could let users hint to Emacs 
> to switch to a fixed-width font just for the art; that'd be good enough.

We could have an interactive alignment feature, whereby the user could
move a character at pixel resolution, until it aligns to the user's
liking.  The implementation would modify the :align-to attribute of
the display property we put on whitespace before that character.

Re: Variable-width font indentation

[Stefan Monnier]

> I don't know of a good solution to this problem: in general, there's not
> enough information to distinguish 'indentation' spaces (the two spaces
> before `printf (', `pr ()', and `return 0') and 'line-up' spaces (the spaces
> before `char ()').

Indeed, you'd need to ask the indenter for the why this particular was
chosen (where the answer will usually be of the form "indentation =
OFFSET + column_of_position POS").

Of course, the additional problem is that the indenter is not
necessarily Emacs's indent-line-function.


        Stefan

Re: Variable-width font indentation

[Eli Zaretskii]

> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Mon, 5 Mar 2018 20:40:14 -0500
> 
> The following code gives a preview of what the algorithm that we've been discussing produces:

Thanks, but I still don't see why we couldn't simply "adjust" the
width of white space using the 'space' display property.

Your implementation has a few issues.  First, if you move the cursor
vertically at column zero, do you see "ghost" characters displayed at
cursor position?  Hiding characters by displaying them with foreground
color identical to the background color has its limits ;-)

Also, with this implementation, the braces in the 'if' and 'else'
blocks no longer align.  E.g., try to indent this snippet:

foo ()
{
  if (something)
    {
      do (anything);
    }
  else
    {
      do (something-else);
    }
}

I think this misalignment will be annoying.

> I do agree that it doesn't look too bad, and presumably a C implementation of the algorithm above would be very fast, since it could build the "spine" above during redisplay.

Indenting during redisplay is a bad idea, because it will disable
almost every redisplay optimization.

I actually don't understand why you worry about performance: the
function you wrote is the morel equivalent of C-\, and that one is not
fast with the current implementation.  We never modify indentation of
non-current lines anyway, we expect the user to type TAB or some
electric character to reindent a line, and we expect them to use C-\
to reindent more than one line.  Automatic adjustment of indentation
might be a good feature, but it would be a separate feature.

In any case, I'd suggest to reindent via buffer-modification hooks,
not as part of redisplay.

Re: Variable-width font indentation

[Daniel Colascione]

On 03/06/2018 08:16 AM, Eli Zaretskii wrote:
>> From: Paul Eggert <eggert@cs.ucla.edu>
>> Date: Mon, 5 Mar 2018 17:30:24 -0800
>>
>> Yes, for ASCII art I doubt whether any simple automated heuristic would
>> work unaided. ASCII art is typically ugly and hard-to-read even in a
>> fixed-width font, so it's not much of a loss if it's rendered poorly. If
>> there's really a need for it I suppose we could let users hint to Emacs
>> to switch to a fixed-width font just for the art; that'd be good enough.
> 
> We could have an interactive alignment feature, whereby the user could
> move a character at pixel resolution, until it aligns to the user's
> liking.  The implementation would modify the :align-to attribute of
> the display property we put on whitespace before that character.

How would you serialize the result?

Re: Variable-width font indentation

[Eli Zaretskii]

> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Mon, 5 Mar 2018 22:36:55 -0500
> 
> * I'm not aware of a way to get a specified space whose width equals that of a particular string, without measuring the string beforehand.  Is there a way?

What do you mean by "measuring the string"?  Calculating its width?

In your implementation you used 'length', which is definitely not TRT,
especially with variable-width fonts.  I think you can find something
to help you out in shr.el.  One other possibility is to use
font-get-glyphs, but I'm not sure it will help in this case.

And I still think that for programming modes, what we need is to
substitute tab-width and column by suitably computed number of pixels,
perhaps derived from the width of some character.  Other than that,
the indentation code should "just work", as long as it uses :align-to
to get to a given pixel coordinate.

Re: What improvements would be truly useful?

[Eli Zaretskii]

> From: Daniele Nicolodi <daniele@grinta.net>
> Date: Mon, 5 Mar 2018 23:32:28 -0700
> 
> I'm far from being familiar with the Emacs codebase thus I may be
> reporting something that it is not completely true, however: Emacs was
> born as a console only application, the graphical user interface seems
> to be duct taped on.

I object to the "duct taped" derogation, and invite you to study the
relevant code before you form your opinions.  Besides, Emacs still
supports text-mode terminals, and moreover, supports text-mode and GUI
frames in the same session (a very important feature), so some degree
of compatibility to a console is still a requirement.

> Also, GTK support seems a bit of an hack that
> requires layering violations (reaching down to the X primitives) to
> work.  Being GTK the only modern toolkit supported on Linux (as far as I
> know) and the only way to get nartive Wayland support, some radical
> cleanup in that area would probably be a good thing.

It's true that GTK support was added in a not very clean way, but I
don't think we can throw away support for the other toolkits just yet,
because they are still being used.

Re: What improvements would be truly useful?

[Daniel Colascione]

On 03/06/2018 08:52 AM, Eli Zaretskii wrote:
>> From: Daniele Nicolodi <daniele@grinta.net>
>> Date: Mon, 5 Mar 2018 23:32:28 -0700
>>
>> I'm far from being familiar with the Emacs codebase thus I may be
>> reporting something that it is not completely true, however: Emacs was
>> born as a console only application, the graphical user interface seems
>> to be duct taped on.
> 
> I object to the "duct taped" derogation, and invite you to study the
> relevant code before you form your opinions.  Besides, Emacs still
> supports text-mode terminals, and moreover, supports text-mode and GUI
> frames in the same session (a very important feature), so some degree
> of compatibility to a console is still a requirement.
> 
>> Also, GTK support seems a bit of an hack that
>> requires layering violations (reaching down to the X primitives) to
>> work.  Being GTK the only modern toolkit supported on Linux (as far as I
>> know) and the only way to get nartive Wayland support, some radical
>> cleanup in that area would probably be a good thing.
> 
> It's true that GTK support was added in a not very clean way, but I
> don't think we can throw away support for the other toolkits just yet,
> because they are still being used.

Eli is right. The current approach is fine, however it got there.

Eventually, we'll have to support Wayland, but I disagree that "radical 
cleanups" will necessarily be needed. As I wrote a while ago, the right 
way forward is a pure GTK+ (as opposed to a mixed X11/GTK+) back-end, 
letting GTK+ handle running on Wayland, and this kind of window system 
could be most accommodated in the existing model. It'll end up looking a 
lot like the NS port.

More radical, I think, would be multiple-window-system support. For 
that, more abstraction in the frame rendering will be required.

Re: What improvements would be truly useful?

[Eli Zaretskii]

> From: Yuri Khan <yuri.v.khan@gmail.com>
> Date: Tue, 6 Mar 2018 14:48:39 +0700
> Cc: Daniele Nicolodi <daniele@grinta.net>, Emacs developers <emacs-devel@gnu.org>
> 
> One architectural decision that I would call baroque is the input model.

Then we have different interpretations of what "architecture" means in
this context.  To me, architecture of a program such as Emacs is its
model-view-controller structure, the single-threaded operation, etc.
Input model is a lower-level design and implementation layer.

> In Emacs, the input model is that of the character terminal: the input
> is a sequence of characters. The repertoire of Unicode characters is
> extended with special values for function and editing keys, and for
> keys modified with Ctrl, Super, Hyper, and Shift where necessary (and
> Meta is represented as an ESC prefix), but still it is a sequence of
> characters without explicit ties to the keys. Even mouse clicks are
> processed as a special kind of characters.

Unless I misunderstand what you mean by "character", the above is
simply not true.  In Emacs, input is a stream of events.  Several
kinds of events are supported, see enum event_kind in termhooks.h.
Only some of them are characters, most are not.  The relevant variable
in keyboard.c is frequently called 'c', so one could think it's a
character, but in fact it is a Lisp object, and can be anything.  In
particular, a mouse click doesn't look at all like a character, it is
actually a cons cell.

> This is inevitable when running in a terminal emulator where the
> underlying abstraction works this way, but in a GUI build this leads
> to inability to distinguish, for example, the ‘.’ character as coming
> from the bottom-row right ring finger in US English layout, or from
> the bottom-row right pinky in Russian layout, or from the decimal
> separator key on the keypad.

That is a different issue, unrelated to what kind of input events are
recognized and how they are represented.  Emacs does support function
keys on X and other GUI environments, so it could potentially support
any abstraction for keys, if the underlying GUI APIs allow that on the
X message level.  We even install a low-level keyboard hook on Windows
to salvage some of the keys that Windows insists on intercepting ahead
of the application.

IOW, I see no architectural limitations here, only implementation that
might be missing or hard to come up with.

> Another example was described in Daniel Colascione’s article/rant
> “Buttery Smooth Emacs”.

Daniel's blog is a good read for someone who is familiar with the
internals, but I don't recommend to form your opinions about the Emacs
internals and their problems based solely on that blog.  For starters,
some of what is written there is no longer true these days.

Re: Variable-width font indentation

[Eli Zaretskii]

> Cc: raeburn@raeburn.org, rms@gnu.org, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Tue, 6 Mar 2018 17:15:29 +0200
> 
> If the way whitespace at bol is rendered depends on indentation code, 
> will the mean that while scrolling a buffer the display engine will have 
> to call indentation code?

I very much hope not, as this would be a performance hit.

Re: Variable-width font indentation

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Tue, 06 Mar 2018 11:04:41 -0500
> 
> >   > This has come up several times, and I think I asked at least once what
> >   > does it need to entail.
> > It means, for one thing, that M-q can fill text with a smooth
> > right margin.
> > Has that been implemented yet?
> 
> AFAIK `shr` does that, yes.

No, AFAIK it doesn't.  What it does is modify the fill algorithm to
measure the actual width of text, without assuming fixed-pitch font.
But it doesn't produce smooth right margin by inserting stretch
glyphs.

Re: Variable-width font indentation

[Eli Zaretskii]

> Cc: cpitclaudel@gmail.com, emacs-devel@gnu.org
> From: Daniel Colascione <dancol@dancol.org>
> Date: Tue, 6 Mar 2018 08:38:28 -0800
> 
> > We could have an interactive alignment feature, whereby the user could
> > move a character at pixel resolution, until it aligns to the user's
> > liking.  The implementation would modify the :align-to attribute of
> > the display property we put on whitespace before that character.
> 
> How would you serialize the result?

You mean, save it to a disk file?  I think we should insert the number
of blanks to align stuff with fixed-pitch font, so that it looks well
with those fonts.  If we want to preserve the display properties
across sessions, we could use enriched-text-mode, I think.

Re: Variable-width font indentation

[dancol]

>> Cc: cpitclaudel@gmail.com, emacs-devel@gnu.org
>> From: Daniel Colascione <dancol@dancol.org>
>> Date: Tue, 6 Mar 2018 08:38:28 -0800
>>
>> > We could have an interactive alignment feature, whereby the user could
>> > move a character at pixel resolution, until it aligns to the user's
>> > liking.  The implementation would modify the :align-to attribute of
>> > the display property we put on whitespace before that character.
>>
>> How would you serialize the result?
>
> You mean, save it to a disk file?  I think we should insert the number
> of blanks to align stuff with fixed-pitch font, so that it looks well
> with those fonts.  If we want to preserve the display properties
> across sessions, we could use enriched-text-mode, I think.

I don't think there's any clean way to embed this alignment information in
arbitrary program text though.

Re: Variable-width font indentation

[Stefan Monnier]

> No, AFAIK it doesn't.  What it does is modify the fill algorithm to
> measure the actual width of text, without assuming fixed-pitch font.
> But it doesn't produce smooth right margin by inserting stretch
> glyphs.

Don't know if it does do it or not, indeed, but AFAIU the algorithm it
uses does give it the pixel info needed to do full-justification.


        Stefan

Re: Variable-width font indentation

[Sam Steingold]

> * Daniel Colascione <qnapby@qnapby.bet> [2018-03-06 08:38:28 -0800]:
>
>> ...
>
> How would you serialize the result?

This is actually the question that bothers me too.
I think we need to separate "visual" indentation for word processing
from "semantic" indentation for coding.
The former should be serialized to something like HTML or LaTeX,
and the latter to plain text as it is done now.

IOW, I think the path to word processing in Emacs lies in html-mode
improvements.

Thanks.

-- 
Sam Steingold (http://sds.podval.org/) on darwin Ns 10.3.1561
http://steingoldpsychology.com http://www.childpsy.net http://mideasttruth.com
http://no2bds.org http://americancensorship.org https://ffii.org
Our business is run on trust.  We trust you will pay in advance.

Re: Variable-width font indentation

[Clément Pit-Claudel]

On 2018-03-06 11:47, Eli Zaretskii wrote:
> What do you mean by "measuring the string"?  Calculating its width?

Correct.

> In your implementation you used 'length', which is definitely not TRT,
> especially with variable-width fonts.

I think you misread the implementation :)  But I'm not sure which part caused confusion, so I'm not sure what to explain.

> And I still think that for programming modes, what we need is to
> substitute tab-width and column by suitably computed number of pixels,
> perhaps derived from the width of some character.  Other than that,
> the indentation code should "just work", as long as it uses :align-to
> to get to a given pixel coordinate.

I don't understand this part :/

Re: Variable-width font indentation

[Clément Pit-Claudel]

On 2018-03-06 11:11, Eli Zaretskii wrote:
> I think I'm still missing something, because I don't understand why we
> need to invent/use a strategy different from what we do now in code
> indentation.  [...]  All the rest is already in place,
> right?

I think I know where the misunderstanding is coming from.

We're thinking of two unrelated things:

* Paul and I have been discussing an language-agnostic heuristic to preserve alignment when changing from monospace to variable-pitch faces.  This intends to preserve the original indentation of the code, by inferring what it would have been if the logic originally used to indent the document had been aware of variable-pitch faces.

* You're describing a language-specific way to get a nice-looking variable-pitch rendering of a document; your way is to reindent the document, essentially.

Does this make sense?

Re: Variable-width font indentation

[Clément Pit-Claudel]

On 2018-03-06 11:35, Stefan Monnier wrote:
> Of course, the additional problem is that the indenter is not
> necessarily Emacs's indent-line-function.

Exactly

Re: What improvements would be truly useful?

[Marcin Borkowski]

On 2018-03-06, at 00:05, Richard Stallman <rms@gnu.org> wrote:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > Personally, I don't think word processing is a good focus for Emacs.
>
> I want to do my word processing in Emacs,

You already can.  Quite a few people do this, using Org-mode or AUCTeX.

Let me quote the paper about Emacs for TUGboat I mentioned some time
ago.  I wrote there this:

--8<---------------cut here---------------start------------->8---
Emacs, however, is used by many people to edit texts in human languages
(as opposed to computer programs), and has really good support for that
task.  For instance, while many editors have support for movement by
words, Emacs has also movement by sentences.  Another feature which is
a real time-saver is the series of transpose commands: for instance,
transpose-chars swaps the two characters on both sides of the point.
There is a whole chapter in the Emacs manual describing commands for
dealing with text in human languages.
--8<---------------cut here---------------end--------------->8---

One thing that might be nice wrt "word processing" would be making
yanking insert spaces at the right places.  If you have " ipsum" in your
kill ring, and press C-y in this context: "Lorem |dolor sit amet" (where
the bar means the point), you end up with "Lorem  ipsumdolor sit amet"
instead of "Lorem ipsum dolor sit amet".  Obvious, but annoying.  (I
even wrote some code to fix that and I've been using it for a few weeks
now, and I didn't notice any problems.)

This is the power of Emacs: the ability to write small extensions,
helping users with their needs.  I think making that easier (e.g., by
providing better documentation) is one of the best things to improve
Emacs.

Don't get me wrong, Emacs has excellent docs.  But if you want to start
extending it, you are pretty much limited to Robert Chassell's book,
which is great, but way too short and elementary.  (Disclaimer: I am
working on a "next step" Elisp book.)

Best,

--
Marcin Borkowski
http://mbork.pl

Re: Variable-width font indentation

[Clément Pit-Claudel]

On 2018-03-06 11:36, Eli Zaretskii wrote:
>> From: Clément Pit-Claudel <cpitclaudel@gmail.com> Date: Mon, 5 Mar
>> 2018 20:40:14 -0500
>> 
>> The following code gives a preview of what the algorithm that we've
>> been discussing produces:
> 
> Thanks, but I still don't see why we couldn't simply "adjust" the 
> width of white space using the 'space' display property.

Because "I'm not aware of a way to get a specified space whose width equals that of a particular string, without measuring the string beforehand.  Is there a way?" :)

IOW, I'm all for it, but I don't know how.

> Your implementation has a few issues.  First, if you move the cursor 
> vertically at column zero, do you see "ghost" characters displayed
> at cursor position?  Hiding characters by displaying them with
> foreground color identical to the background color has its limits
> ;-)

That's a feature :) (Just to be clear: that was the easy way to get a feel for what the algorithm that Paul and I were discussing would produce; I'd have used space properties if I had known a way to get a space of the same with as these ghost characters).

And there are many other issues, of course: we don't want to apply the display property to more than one stretch of space, and the code doesn't deal properly with tabs vs spaces, etc.

> Also, with this implementation, the braces in the 'if' and 'else' 
> blocks no longer align.  E.g., try to indent this snippet:
> 
> foo () { if (something) { do (anything); } else { do
> (something-else); } }
> 
> I think this misalignment will be annoying.


Indeed, this was discussed in this thread; Paul thinks it's not an issue; I'm not convinced (I tend to agree with you) :)

>> I do agree that it doesn't look too bad, and presumably a C
>> implementation of the algorithm above would be very fast, since it
>> could build the "spine" above during redisplay.
> 
> Indenting during redisplay is a bad idea, because it will disable 
> almost every redisplay optimization.

I think we're using "indenting" to mean different things.  (Just to be clear: the process I'm describing doesn't call indent-line-function, nor any of the indentation machinery)

> I actually don't understand why you worry about performance: the 
> function you wrote is the morel equivalent of C-\

C-\ is toggle-input-method on my machine; is that what you meant?

> , and that one is not fast with the current implementation.  We never
> modify indentation of non-current lines anyway, we expect the user to
> type TAB or some electric character to reindent a line, and we expect
> them to use C-\ to reindent more than one line.  Automatic adjustment
> of indentation might be a good feature, but it would be a separate
> feature.

I never meant to adjust indentation (that is, to change the contents of the buffer): just to display existing leading spaces differently based on the contents of previous line. 

> In any case, I'd suggest to reindent via buffer-modification hooks, 
> not as part of redisplay.

Right, and that's why I worry about performance.

Clément.

Re: Variable-width font indentation

[Eli Zaretskii]

> Date: Tue, 6 Mar 2018 09:55:16 -0800
> From: dancol@dancol.org
> Cc: "Daniel Colascione" <dancol@dancol.org>,
>  eggert@cs.ucla.edu,
>  cpitclaudel@gmail.com,
>  emacs-devel@gnu.org
> 
> > You mean, save it to a disk file?  I think we should insert the number
> > of blanks to align stuff with fixed-pitch font, so that it looks well
> > with those fonts.  If we want to preserve the display properties
> > across sessions, we could use enriched-text-mode, I think.
> 
> I don't think there's any clean way to embed this alignment information in
> arbitrary program text though.

No, it would be necessary to reconstruct it.  But any solution that
uses text properties will have the same problem.  And I don't quite
see how we can align text rendered with variable-pitch font without
something like that.

Re: Variable-width font indentation

[Daniel Colascione]

On 03/06/2018 12:18 PM, Eli Zaretskii wrote:
>> Date: Tue, 6 Mar 2018 09:55:16 -0800
>> From: dancol@dancol.org
>> Cc: "Daniel Colascione" <dancol@dancol.org>,
>>   eggert@cs.ucla.edu,
>>   cpitclaudel@gmail.com,
>>   emacs-devel@gnu.org
>>
>>> You mean, save it to a disk file?  I think we should insert the number
>>> of blanks to align stuff with fixed-pitch font, so that it looks well
>>> with those fonts.  If we want to preserve the display properties
>>> across sessions, we could use enriched-text-mode, I think.
>>
>> I don't think there's any clean way to embed this alignment information in
>> arbitrary program text though.
> 
> No, it would be necessary to reconstruct it.  But any solution that
> uses text properties will have the same problem.  And I don't quite
> see how we can align text rendered with variable-pitch font without
> something like that.

Right. The leading-indentation case is *barely* feasible, if you imagine 
that modes can accurately indent all lines automatically. I don't see a 
path toward internal alignment, since there's no algorithmic way to 
disambiguate spacing. The closest thing would be something like 
align-regexp, and even this facility requires manual intervention.

Re: Variable-width font indentation

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org, eggert@cs.ucla.edu
> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Tue, 6 Mar 2018 14:59:21 -0500
> 
> * Paul and I have been discussing an language-agnostic heuristic to preserve alignment when changing from monospace to variable-pitch faces.  This intends to preserve the original indentation of the code, by inferring what it would have been if the logic originally used to indent the document had been aware of variable-pitch faces.

I don't see a need for such a feature, sorry.  We have indentation
already; its only problem is that it doesn't work well with variable
pitch fonts.  That's what we need to try to fix, if we care about the
capability of displaying program code with such fonts.

> * You're describing a language-specific way to get a nice-looking variable-pitch rendering of a document; your way is to reindent the document, essentially.

I was describing a way of teaching the existing indentation code
handle variable-pitch fonts.

Re: What improvements would be truly useful?

[Eli Zaretskii]

> From: Marcin Borkowski <mbork@mbork.pl>
> Cc: Daniel Colascione <dancol@dancol.org>, eggert@cs.ucla.edu, eliz@gnu.org, raeburn@raeburn.org, rostislav.svoboda@gmail.com, emacs-devel@gnu.org
> Date: Tue, 06 Mar 2018 21:01:00 +0100
> 
> > I want to do my word processing in Emacs,
> 
> You already can.  Quite a few people do this, using Org-mode or AUCTeX.

Are you saying that either of these is a WYSIWYG word processor?

Re: Variable-width font indentation

[Eli Zaretskii]

> Cc: eggert@cs.ucla.edu, emacs-devel@gnu.org
> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Tue, 6 Mar 2018 15:11:29 -0500
> 
> > Thanks, but I still don't see why we couldn't simply "adjust" the 
> > width of white space using the 'space' display property.
> 
> Because "I'm not aware of a way to get a specified space whose width equals that of a particular string, without measuring the string beforehand.  Is there a way?" :)

The "string" you refer to is already in the buffer on display, right?
If so, you can simply use the likes of posn-at-point to measure
portions of text in pixel units.  Then translate that into
floating-point "column" numbers by dividing by width of some suitable
character (it could be SPC, but that is a bad idea with some fonts, so
maybe something else, like 'e').

> I'd have used space properties if I had known a way to get a space of the same with as these ghost characters).

See above.  And there's font-get-glyphs.

> And there are many other issues, of course: we don't want to apply the display property to more than one stretch of space, and the code doesn't deal properly with tabs vs spaces, etc.

Please elaborate, because I don't understand those issues well enough
to see if and how they can be resolved.

> >> I do agree that it doesn't look too bad, and presumably a C
> >> implementation of the algorithm above would be very fast, since it
> >> could build the "spine" above during redisplay.
> > 
> > Indenting during redisplay is a bad idea, because it will disable 
> > almost every redisplay optimization.
> 
> I think we're using "indenting" to mean different things.  (Just to be clear: the process I'm describing doesn't call indent-line-function, nor any of the indentation machinery)

Then what _do_ you mean by "building the spine during redisplay"?

> > I actually don't understand why you worry about performance: the 
> > function you wrote is the morel equivalent of C-\
> 
> C-\ is toggle-input-method on my machine; is that what you meant?

Sorry, I meant M-C-\, of course.

> > , and that one is not fast with the current implementation.  We never
> > modify indentation of non-current lines anyway, we expect the user to
> > type TAB or some electric character to reindent a line, and we expect
> > them to use C-\ to reindent more than one line.  Automatic adjustment
> > of indentation might be a good feature, but it would be a separate
> > feature.
> 
> I never meant to adjust indentation (that is, to change the contents of the buffer): just to display existing leading spaces differently based on the contents of previous line. 

We never do that with the existing indentation machinery, either.

> > In any case, I'd suggest to reindent via buffer-modification hooks, 
> > not as part of redisplay.
> 
> Right, and that's why I worry about performance.

Then don't automatically reindent, let the user initiate that.

Re: What improvements would be truly useful?

[Marcin Borkowski]

On 2018-03-06, at 21:32, Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Marcin Borkowski <mbork@mbork.pl>
>> Cc: Daniel Colascione <dancol@dancol.org>, eggert@cs.ucla.edu, eliz@gnu.org, raeburn@raeburn.org, rostislav.svoboda@gmail.com, emacs-devel@gnu.org
>> Date: Tue, 06 Mar 2018 21:01:00 +0100
>> 
>> > I want to do my word processing in Emacs,
>> 
>> You already can.  Quite a few people do this, using Org-mode or AUCTeX.
>
> Are you saying that either of these is a WYSIWYG word processor?

Did Richard mention WYSIWYG word processing or just word processing?

Also, AUCTeX with pdf-tools is quite close.  On a fast computer, you can
run TeX-command-master and TeX-view on an idle timer.  Not the same, but
close.

Best,

-- 
Marcin Borkowski
http://mbork.pl

Re: What improvements would be truly useful?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

   I don't use Emacs for writing such
  > documents, nor would having variable width support by itself grant all the
  > other features such users have come to expect from modern Word Processors.

You're correct in observing that to make Emacs work as a word
processor requires adding a number of features.  A single added
feature won't reach that point.  A few features, well chosen, WILL
reach that point, and then we will see people using Emacs as a word
processor.

Thus, I picked one of these features, one which we certainly need,
and asked people to implement it in a user-usable way.

When that's done, we can add another word-processing feature,
and another, and soon people will start using Emacs for some
simple cases of word processing.

Some simple cases, such as letters, are not very many features
away.  I am sure we could get there this year.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: Variable-width font indentation

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I don't see a reason to use variable-width fonts
except when they provide an improvement.  For ASCII art,
a fixed-width font is best.

I don't see a reason to use variable-width fonts for code, but if
people like that, I'm in favor of supporting it.

The place where variable-width fonts, and multiple fonts of different
width, are really important is for editing documents with formatted
text.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: What improvements would be truly useful?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > You are advocating for your suggested improvements. I am advocating for
  > mine. 

Emacs is not an independent project, and it isn't governed by its
contributors.  I'm the head of the GNU Project, and that includes
Emacs.

You, as a contributor, can advocate a certain decision, which means
you present arguments why I should approve it.  I don't need to
advocate a decision in that sense.

I leave most technical decisions up to the contributors, including
you, because for most of the questions I don't have any special
preference of my own.  For those questions, I'm happy with whatever
works, and I know the contributors can figure out what works.

However, making progress on Emacs as a word processor is one of my
specific goals.  This is what Emacs needs to do to be useful in all
the ways it should be useful.

It will only take a few more features to make Emacs start to be useful
as a word processor.  Once we can do proper formatting of paragraphs
with variable-width text, with a few kinds of alignment, and we can
save these in files, we will be able to use it for writing letters and
handouts, instead of LibreOffice.

I am really looking forward to this.  LibreOffice is free software,
and it's ethical, but it isn't Emacs.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: Variable-width font indentation

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > It means, for one thing, that M-q can fill text with a smooth
  > > right margin.
  > > Has that been implemented yet?

  > AFAIK `shr` does that, yes.

I don't think it does THIS job.

shr is a facility that renders HTML.
I'm talking about using M-q to do filling
that looks good with variable-width fonts.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: Variable-width font indentation (was: What improvements would be truly useful?)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > It means, for one thing, that M-q can fill text with a smooth
  > > right margin.
  > > 
  > > Has that been implemented yet?

  > No, not yet.  But unless I'm missing something, doing that would be
  > relatively simple: we need a variant of justify-current-line which
  > will (a) compute the number of blanks as a float number, and (b) add a
  > 'display' property to each blank with an :align-to attribute computed
  > (in pixels) to produce the smooth right margin.

If it is that easy, it means that we already have done more
of this job than I thought we did.  That would be good news.

Would someone please implement this?  Whenever the buffer
has characters of different widths, it ought to fill them
with attention to their widths.

Once this is done, we can see what word-processing feature to add next.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: Variable-width font indentation

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > No, AFAIK it doesn't.  What it does is modify the fill algorithm to
  > measure the actual width of text, without assuming fixed-pitch font.
  > But it doesn't produce smooth right margin by inserting stretch
  > glyphs.

Given that this part has already been implemented, 
we should make M-q do it too.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: Variable-width font indentation

[Paul Eggert]

On 03/06/2018 12:31 PM, Eli Zaretskii wrote:
> We have indentation
> already; its only problem is that it doesn't work well with variable
> pitch fonts.

As I understood it, Clément wrote a quick-and-dirty algorithm to 
simulate better indentation with variable-width fonts. That is, it 
wasn't intended to be something that should go into Emacs, only as a way 
to see what the the user would see if we changed Emacs's display of 
indentation to be along the suggested lines. Unless I'm 
misunderstanding, his work provided a simple proof-of-concept that a 
relatively-inexpensive way of improving indentation would significantly 
improve display with variable-width fonts.

Re: What improvements would be truly useful?

[Daniel Colascione]

On 03/06/2018 12:54 PM, Richard Stallman wrote:
> However, making progress on Emacs as a word processor is one of my
> specific goals.  This is what Emacs needs to do to be useful in all
> the ways it should be useful.

If you want to add word processing features to Emacs, I suggest 
implementing the necessary changes features yourself instead of 
suggesting that discussion of other priorities is somehow "interfering".

Code wins arguments. `eval' tends not to work very well when you pass it 
a block of prose about how Emacs should be a word processor.

> It will only take a few more features to make Emacs start to be useful
> as a word processor.  Once we can do proper formatting of paragraphs
> with variable-width text, with a few kinds of alignment, and we can
> save these in files, we will be able to use it for writing letters and
> handouts, instead of LibreOffice.

Sure. It'd be nice if Emacs were suitable for that task. I don't see the 
urgency, considering that plenty of equally free packages exist.

Re: Variable-width font indentation

[Clément Pit-Claudel]

On 2018-03-06 15:31, Eli Zaretskii wrote:
>> Cc: emacs-devel@gnu.org, eggert@cs.ucla.edu
>> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
>> Date: Tue, 6 Mar 2018 14:59:21 -0500
>>
>> * Paul and I have been discussing an language-agnostic heuristic to preserve alignment when changing from monospace to variable-pitch faces.  This intends to preserve the original indentation of the code, by inferring what it would have been if the logic originally used to indent the document had been aware of variable-pitch faces.
> 
> I don't see a need for such a feature, sorry.  We have indentation
> already; its only problem is that it doesn't work well with variable
> pitch fonts.  That's what we need to try to fix, if we care about the
> capability of displaying program code with such fonts.

I'm thoroughly confused: doesn't the algorithm that Paul proposed, and that I show an implementation of, do exactly that?

Do you want to get the indentation code involved because the heuristic isn't perfect?
 
>> * You're describing a language-specific way to get a nice-looking variable-pitch rendering of a document; your way is to reindent the document, essentially.
> 
> I was describing a way of teaching the existing indentation code
> handle variable-pitch fonts.

Right, which is where we differ: the solution Paul and I have been thinking is designed not to rely on exiting indentation facilities. 

In your world, concretely, what happens when I open xdisp.c and press M-x variable-pitch-mode? Does everything look misaligned until I M-x indent-region the whole file?

What about a file from a different project that uses a different indenting convention?

Thanks,
Clément.

Re: Variable-width font indentation

[Ricardo Wurmus]

Sam Steingold <sds@gnu.org> writes:

>> * Daniel Colascione <qnapby@qnapby.bet> [2018-03-06 08:38:28 -0800]:
>>
>>> ...
>>
>> How would you serialize the result?
>
> This is actually the question that bothers me too.
> I think we need to separate "visual" indentation for word processing
> from "semantic" indentation for coding.
> The former should be serialized to something like HTML or LaTeX,
> and the latter to plain text as it is done now.

Or to XML, which is used by other word processors, and which has a neat
S-expression representation (sxml).

-- 
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: What improvements would be truly useful?

[Daniele Nicolodi]

On 06/03/2018 09:52, Eli Zaretskii wrote:
>> From: Daniele Nicolodi <daniele@grinta.net>
>> Date: Mon, 5 Mar 2018 23:32:28 -0700
>>
>> I'm far from being familiar with the Emacs codebase thus I may be
>> reporting something that it is not completely true, however: Emacs was
>> born as a console only application, the graphical user interface seems
>> to be duct taped on.
> 
> I object to the "duct taped" derogation, and invite you to study the
> relevant code before you form your opinions.  Besides, Emacs still
> supports text-mode terminals, and moreover, supports text-mode and GUI
> frames in the same session (a very important feature), so some degree
> of compatibility to a console is still a requirement.

I didn't intend "duct taped" as derogatory.  I meant to say that it has
been added in a way that works and is effective, but it is not elegant
or nice to look at.  I'm an experimental physicist and all complex
experiments have some literal duct tape solution that makes them work,
that was often supposed to be a quick fix to get things going, and that
ends to survive generations of PhD students.

>> Also, GTK support seems a bit of an hack that
>> requires layering violations (reaching down to the X primitives) to
>> work.  Being GTK the only modern toolkit supported on Linux (as far as I
>> know) and the only way to get nartive Wayland support, some radical
>> cleanup in that area would probably be a good thing.
> 
> It's true that GTK support was added in a not very clean way, but I
> don't think we can throw away support for the other toolkits just yet,
> because they are still being used.

I don't think that cleaning up the GTK support requires removing support
for any other toolkit, and I'm far from suggesting something like that.

Cheers,
Dan

Re: Variable-width font indentation

[Yuri Khan]

On Wed, Mar 7, 2018 at 3:55 AM, Richard Stallman <rms@gnu.org> wrote:

> shr is a facility that renders HTML.
> I'm talking about using M-q to do filling
> that looks good with variable-width fonts.

For word processing purposes, you should not need manual or
semi-automated filling such as M-q provides. Proper word processing
operates two level of abstraction higher.

Level +0: you directly affect where words end up using spaces and line
feeds. This is the level that M-q works on.

Level +1: you specify a paper size for the document or a sections,
and, for each paragraph, a left margin, a right margin, a font, and a
paragraph alignment (left, center, right, or justified). The layout
engine automatically chops the paragraph into lines for display and
print purposes according to these settings.

Level +2: you do not specify individual formatting settings on each
paragraph; instead, you specify a paragraph style that specifies all
these things. (Like Emacs faces provide a level of indirection for
face attributes.)

So: you don’t fill a paragraph with a left margin of 30 spaces and a
right margin at position 72. You designate the paragraph as Addressee,
and the Addressee style specifies a left margin of 50% paper width and
inherits its right margin of 1 inch from the Default style.

All of which is, of course, achievable with HTML + CSS (or a moral
equivalent). WYSIWYG editing for that is possible but 20 years of
observation show that users of WYSIWYG systems tend to produce (at
best) level +1 documents rather than +2.

Re: Variable-width font indentation

[Clément Pit-Claudel]

On 2018-03-06 15:40, Eli Zaretskii wrote:
>> Cc: eggert@cs.ucla.edu, emacs-devel@gnu.org
>> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
>> Date: Tue, 6 Mar 2018 15:11:29 -0500
>>
>>> Thanks, but I still don't see why we couldn't simply "adjust" the 
>>> width of white space using the 'space' display property.
>>
>> Because "I'm not aware of a way to get a specified space whose width equals that of a particular string, without measuring the string beforehand.  Is there a way?" :)
> 
> The "string" you refer to is already in the buffer on display, right?
> If so, you can simply use the likes of posn-at-point to measure
> portions of text in pixel units.

Yes and no: it might be offscreen.  But posn-at-point is a useful suggestion, thanks :) 

> Then translate that into
> floating-point "column" numbers by dividing by width of some suitable
> character (it could be SPC, but that is a bad idea with some fonts, so
> maybe something else, like 'e').

I don't think I need that part.

>> I'd have used space properties if I had known a way to get a space of the same with as these ghost characters).
> 
> See above.  And there's font-get-glyphs.

Thanks, I'll look into this.

>> And there are many other issues, of course: we don't want to apply the display property to more than one stretch 
of space, and the code doesn't deal properly with tabs vs spaces, etc.
> 
> Please elaborate, because I don't understand those issues well enough
> to see if and how they can be resolved.

That code was just a rough draft, so it's plenty of corner cases that I didn't think about yet.  That's all :)

>>>> I do agree that it doesn't look too bad, and presumably a C
>>>> implementation of the algorithm above would be very fast, since it
>>>> could build the "spine" above during redisplay.
>>>
>>> Indenting during redisplay is a bad idea, because it will disable 
>>> almost every redisplay optimization.
>>
>> I think we're using "indenting" to mean different things.  (Just to be clear: the process I'm describing doesn't call indent-line-function, nor any of the indentation machinery)
> 
> Then what _do_ you mean by "building the spine during redisplay"?

I meant build a prefix string (which the code I sent called the "spine") to use instead of leading spaces.
The ghost characters that you noticed are the value of the spine on that line.

>>> I actually don't understand why you worry about performance: the 
>>> function you wrote is the morel equivalent of C-\
>>
>> C-\ is toggle-input-method on my machine; is that what you meant?
> 
> Sorry, I meant M-C-\, of course.

Ah, yes, I see. That's not what the code I offered does at all, I think.

>>> , and that one is not fast with the current implementation.  We never
>>> modify indentation of non-current lines anyway, we expect the user to
>>> type TAB or some electric character to reindent a line, and we expect
>>> them to use C-\ to reindent more than one line.  Automatic adjustment
>>> of indentation might be a good feature, but it would be a separate
>>> feature.
>>
>> I never meant to adjust indentation (that is, to change the contents of the buffer): just to display existing leading spaces differently based on the contents of previous line. 
> 
> We never do that with the existing indentation machinery, either.

Wait, never do what?

>>> In any case, I'd suggest to reindent via buffer-modification hooks, 
>>> not as part of redisplay.
>>
>> Right, and that's why I worry about performance.
> 
> Then don't automatically reindent, let the user initiate that.

The feature I'm offering is like font-locking, not like reindenting: it changes the display of existing buffer elements in a way that doesn't modify the underlying text.  Given this, it makes more sense to me to run it automatically (rather than on demand)

Clément. 

Re: Variable-width font indentation

[Eli Zaretskii]

> From: Sam Steingold <sds@gnu.org>
> Date: Tue, 06 Mar 2018 13:53:54 -0500
> 
> > * Daniel Colascione <qnapby@qnapby.bet> [2018-03-06 08:38:28 -0800]:
> >
> >> ...
> >
> > How would you serialize the result?
> 
> This is actually the question that bothers me too.
> I think we need to separate "visual" indentation for word processing
> from "semantic" indentation for coding.
> The former should be serialized to something like HTML or LaTeX,
> and the latter to plain text as it is done now.

Take a look at enriched.el (and etc/enriched.txt as the demonstration
of its capabilities): we already have the capability for serializing
this, had it for many years.

Re: What improvements would be truly useful?

[Eli Zaretskii]

> From: Marcin Borkowski <mbork@mbork.pl>
> Cc: rms@gnu.org, dancol@dancol.org, eggert@cs.ucla.edu, raeburn@raeburn.org, rostislav.svoboda@gmail.com, emacs-devel@gnu.org
> Date: Tue, 06 Mar 2018 21:41:00 +0100
> 
> 
> >> > I want to do my word processing in Emacs,
> >> 
> >> You already can.  Quite a few people do this, using Org-mode or AUCTeX.
> >
> > Are you saying that either of these is a WYSIWYG word processor?
> 
> Did Richard mention WYSIWYG word processing or just word processing?

WYSIWYG, of course.  Otherwise it would make no sense to talk about
indentation with variable-pitch fonts and pixel-level justification.

Re: Variable-width font indentation

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org, eggert@cs.ucla.edu
> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Tue, 6 Mar 2018 16:47:51 -0500
> 
> > I don't see a need for such a feature, sorry.  We have indentation
> > already; its only problem is that it doesn't work well with variable
> > pitch fonts.  That's what we need to try to fix, if we care about the
> > capability of displaying program code with such fonts.
> 
> I'm thoroughly confused: doesn't the algorithm that Paul proposed, and that I show an implementation of, do exactly that?

Not AFAIU, no.  It is an implementation of a different indentation
method.  I don't see why we should want to go that way, since there's
nothing wrong with the existing algorithm, except for handling of
variable-pitch fonts.

> Do you want to get the indentation code involved because the heuristic isn't perfect?

More like because the existing indentation algorithms are okay, and I
see no need to change them to something else.

> In your world, concretely, what happens when I open xdisp.c and press M-x variable-pitch-mode? Does everything look misaligned until I M-x indent-region the whole file?

Yes.  It isn't different from what happens when you change tab-width.

> What about a file from a different project that uses a different indenting convention?

Not sure how that is relevant.  A different indentation convention
will look differently, of course.

Re: What improvements would be truly useful?

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Daniele Nicolodi <daniele@grinta.net>
> Date: Tue, 6 Mar 2018 22:45:25 -0700
> 
> >> Also, GTK support seems a bit of an hack that
> >> requires layering violations (reaching down to the X primitives) to
> >> work.  Being GTK the only modern toolkit supported on Linux (as far as I
> >> know) and the only way to get nartive Wayland support, some radical
> >> cleanup in that area would probably be a good thing.
> > 
> > It's true that GTK support was added in a not very clean way, but I
> > don't think we can throw away support for the other toolkits just yet,
> > because they are still being used.
> 
> I don't think that cleaning up the GTK support requires removing support
> for any other toolkit, and I'm far from suggesting something like that.

Then how should we understand your suggestion for "radical cleanup",
right after asserting that GTK is the only modern toolkit supported by
GNU/Linux?

Re: What improvements would be truly useful?

[Daniele Nicolodi]

On 3/7/18 10:45 AM, Eli Zaretskii wrote:
>> Cc: emacs-devel@gnu.org
>> From: Daniele Nicolodi <daniele@grinta.net>
>> Date: Tue, 6 Mar 2018 22:45:25 -0700
>>
>>>> Also, GTK support seems a bit of an hack that
>>>> requires layering violations (reaching down to the X primitives) to
>>>> work.  Being GTK the only modern toolkit supported on Linux (as far as I
>>>> know) and the only way to get nartive Wayland support, some radical
>>>> cleanup in that area would probably be a good thing.
>>>
>>> It's true that GTK support was added in a not very clean way, but I
>>> don't think we can throw away support for the other toolkits just yet,
>>> because they are still being used.
>>
>> I don't think that cleaning up the GTK support requires removing support
>> for any other toolkit, and I'm far from suggesting something like that.
> 
> Then how should we understand your suggestion for "radical cleanup",
> right after asserting that GTK is the only modern toolkit supported by
> GNU/Linux?

Radical cleanup of the GTK integration.

Re: What improvements would be truly useful?

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Daniele Nicolodi <daniele@grinta.net>
> Date: Wed, 7 Mar 2018 10:49:39 -0700
> 
> > Then how should we understand your suggestion for "radical cleanup",
> > right after asserting that GTK is the only modern toolkit supported by
> > GNU/Linux?
> 
> Radical cleanup of the GTK integration.

If you have ideas for how to achieve that, I'd be interested to hear
them.

Thanks.

Re: Variable-width font indentation

[Eli Zaretskii]

> Cc: eggert@cs.ucla.edu, emacs-devel@gnu.org
> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Wed, 7 Mar 2018 08:15:39 -0500
> 
> > The "string" you refer to is already in the buffer on display, right?
> > If so, you can simply use the likes of posn-at-point to measure
> > portions of text in pixel units.
> 
> Yes and no: it might be offscreen.

Then the text you need to indent below it is also off-screen, and you
don't need to re-indent it, right?

> > Then what _do_ you mean by "building the spine during redisplay"?
> 
> I meant build a prefix string (which the code I sent called the "spine") to use instead of leading spaces.

What would be the advantage of building such a string at redisplay
time?  Consing a short string is very fast, and has the advantage of
being done only when needed (by contrast, display code is many times
invoked for reasons utterly unrelated to actually displaying stuff).

> >> I never meant to adjust indentation (that is, to change the contents of the buffer): just to display existing leading spaces differently based on the contents of previous line. 
> > 
> > We never do that with the existing indentation machinery, either.
> 
> Wait, never do what?

What you say in the sentence I quoted: display existing leading spaces
differently based on the contents of previous line.

> >>> In any case, I'd suggest to reindent via buffer-modification hooks, 
> >>> not as part of redisplay.
> >>
> >> Right, and that's why I worry about performance.
> > 
> > Then don't automatically reindent, let the user initiate that.
> 
> The feature I'm offering is like font-locking, not like reindenting: it changes the display of existing buffer elements in a way that doesn't modify the underlying text.  Given this, it makes more sense to me to run it automatically (rather than on demand)

"Automatically" doesn't necessarily mean "like font-lock".  I think we
should know better than get ourselves once again into an adventure
like font-lock, which even today, with JIT and all the years of
experience and improvements, is still a headache, if not a PITA.  If
there's no other way to do something that's really necessary, fine;
but we should avoid such features as much as possible.

For example, in this case, I'm beginning to think that we should
render SPC characters similarly to what we do with TABs: produce a
stretch glyph of a suitably computed width, and perhaps introduce a
variable space-width (like tab-width) to help Lisp programs with
controlling that.  Note that 'space' display property supports values
that are symbols, in which case it uses the symbol's value -- this
allows Lisp programs to change the value of a buffer-local binding of
the symbol, and have the indentation change at the next redisplay
cycle (a.k.a. "immediately").  Doesn't that provide a better way for
controlling indentation than font-lock style scanning of the visible
portion of the buffer "just in time"?

Re: Variable-width font indentation

[Clément Pit-Claudel]

On 2018-03-07 12:22, Eli Zaretskii wrote:
>> Cc: emacs-devel@gnu.org, eggert@cs.ucla.edu
>> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
>> Date: Tue, 6 Mar 2018 16:47:51 -0500
>>
>>> I don't see a need for such a feature, sorry.  We have indentation
>>> already; its only problem is that it doesn't work well with variable
>>> pitch fonts.  That's what we need to try to fix, if we care about the
>>> capability of displaying program code with such fonts.
>>
>> I'm thoroughly confused: doesn't the algorithm that Paul proposed, and that I show an implementation of, do exactly that?
> 
> Not AFAIU, no.  It is an implementation of a different indentation
> method.  I don't see why we should want to go that way, since there's
> nothing wrong with the existing algorithm, except for handling of
> variable-pitch fonts.

Can you clarify what you call an indentation method? I think that's the source of my confusion.
I wouldn't call what I wrote an indentation method.  Maybe I'm not understanding you because of that?

>> Do you want to get the indentation code involved because the heuristic isn't perfect?
> 
> More like because the existing indentation algorithms are okay, and I
> see no need to change them to something else.

Neither do I, and indeed I'm not proposing to change existing indentation algorithms :)

>> In your world, concretely, what happens when I open xdisp.c and press M-x variable-pitch-mode? Does everything look misaligned until I M-x indent-region the whole file?
> 
> Yes.  It isn't different from what happens when you change tab-width.

But changing tab-width doesn't change the contents of the buffer, whereas what you're advocating for does, right?

>> What about a file from a different project that uses a different indenting convention?
> 
> Not sure how that is relevant.  A different indentation convention
> will look differently, of course.

This is relevant because the method that Paul originally proposed (which I'm not even advocating for — I'm not even sure yet whether it's good or bad) does not require changing the buffer or the actual number of spaces present on each line in the buffer.

Maybe I just don't understand what you're advocating for.  AFAIU, you're suggesting that we should teach indentation methods about variable-pitch fonts, so that they indent text to a particular pixel offset instead of a particular column.  Is that correct?

If it is correct, then it solves a problem that's different from the one I was thinking in, which is to display existing code with a variable pitch font, without the indentation function.

Some questions on the solution you propose:

* What happens in major modes that don't provide indentation support?
* What happens in indentation-sensitive languages (Python, F#, Haskell, etc)?
* What happens in comments?
* Do I need to call M-x indent-region every time I switch between variable-pitch and fixed-pitch?
* If I close and reopen a file in variable-pitch mode, do I need to re-run M-x indent-region?
* In variable-pitch mode, what gets saved to disk after I run M-x indent-region?

Thanks,
Clément.

Re: What improvements would be truly useful?

[Marcin Borkowski]

On 2018-03-07, at 18:13, Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Marcin Borkowski <mbork@mbork.pl>
>> Cc: rms@gnu.org, dancol@dancol.org, eggert@cs.ucla.edu, raeburn@raeburn.org, rostislav.svoboda@gmail.com, emacs-devel@gnu.org
>> Date: Tue, 06 Mar 2018 21:41:00 +0100
>> 
>> 
>> >> > I want to do my word processing in Emacs,
>> >> 
>> >> You already can.  Quite a few people do this, using Org-mode or AUCTeX.
>> >
>> > Are you saying that either of these is a WYSIWYG word processor?
>> 
>> Did Richard mention WYSIWYG word processing or just word processing?
>
> WYSIWYG, of course.  Otherwise it would make no sense to talk about
> indentation with variable-pitch fonts and pixel-level justification.

Have you ever seen LyX?

-- 
Marcin Borkowski
http://mbork.pl

Re: What improvements would be truly useful?

[Eli Zaretskii]

> From: Marcin Borkowski <mbork@mbork.pl>
> Cc: rms@gnu.org, dancol@dancol.org, eggert@cs.ucla.edu, raeburn@raeburn.org, rostislav.svoboda@gmail.com, emacs-devel@gnu.org
> Date: Wed, 07 Mar 2018 19:30:07 +0100
> 
> >> > Are you saying that either of these is a WYSIWYG word processor?
> >> 
> >> Did Richard mention WYSIWYG word processing or just word processing?
> >
> > WYSIWYG, of course.  Otherwise it would make no sense to talk about
> > indentation with variable-pitch fonts and pixel-level justification.
> 
> Have you ever seen LyX?

What would it add to the current discussion?

Re: Variable-width font indentation

[Daniel Colascione]

On 03/07/2018 09:12 AM, Eli Zaretskii wrote:
>> From: Sam Steingold <sds@gnu.org>
>> Date: Tue, 06 Mar 2018 13:53:54 -0500
>>
>>> * Daniel Colascione <qnapby@qnapby.bet> [2018-03-06 08:38:28 -0800]:
>>>
>>>> ...
>>>
>>> How would you serialize the result?
>>
>> This is actually the question that bothers me too.
>> I think we need to separate "visual" indentation for word processing
>> from "semantic" indentation for coding.
>> The former should be serialized to something like HTML or LaTeX,
>> and the latter to plain text as it is done now.
> 
> Take a look at enriched.el (and etc/enriched.txt as the demonstration
> of its capabilities): we already have the capability for serializing
> this, had it for many years.

That's fine so long as you have a file you want to edit _only_ with 
Emacs. Files that need to be understood by other tools would need 
information about indentation adjustment included in some compatible way 
(maybe a comment), but since other tools would know nothing of this 
information, it would go out of date. Requiring that people edit a 
source file only with Emacs is not realistic.

Re: Variable-width font indentation

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org, eggert@cs.ucla.edu
> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Wed, 7 Mar 2018 13:27:35 -0500
> 
> > Not AFAIU, no.  It is an implementation of a different indentation
> > method.  I don't see why we should want to go that way, since there's
> > nothing wrong with the existing algorithm, except for handling of
> > variable-pitch fonts.
> 
> Can you clarify what you call an indentation method? I think that's the source of my confusion.
> I wouldn't call what I wrote an indentation method.  Maybe I'm not understanding you because of that?

I feel that we are somehow bogged down in a miscommunication from
which I see no way out...

An "indentation method" is a method used to determine the indentation
of a line and to insert whitespace in order to achieve that
indentation.

> >> In your world, concretely, what happens when I open xdisp.c and press M-x variable-pitch-mode? Does everything look misaligned until I M-x indent-region the whole file?
> > 
> > Yes.  It isn't different from what happens when you change tab-width.
> 
> But changing tab-width doesn't change the contents of the buffer, whereas what you're advocating for does, right?

No, it doesn't.  I'm advocating a way of changing the width of
whitespace characters on display to account for variable-pitch font,
not any change in the buffer.

> Maybe I just don't understand what you're advocating for.  AFAIU, you're suggesting that we should teach indentation methods about variable-pitch fonts, so that they indent text to a particular pixel offset instead of a particular column.  Is that correct?

Yes, correct.

> If it is correct, then it solves a problem that's different from the one I was thinking in, which is to display existing code with a variable pitch font, without the indentation function.

We cannot possibly DTRT with indentation without an indentation
function.

> * What happens in major modes that don't provide indentation support?

The same as what happens today: the default functions are invoked.

> * What happens in indentation-sensitive languages (Python, F#, Haskell, etc)?

Same as today (the buffer contents will not change from what it is
today).

> * What happens in comments?

Nothing special.

> * Do I need to call M-x indent-region every time I switch between variable-pitch and fixed-pitch?

Not necessarily, not if the idea of controlling the SPC width is
workable.  If it is, then the mode will simply set the value of two
buffer-local variables, calculating them from the font in use.

> * If I close and reopen a file in variable-pitch mode, do I need to re-run M-x indent-region?

See above.

> * In variable-pitch mode, what gets saved to disk after I run M-x indent-region?

Same as what we have now.

Re: Variable-width font indentation

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Daniel Colascione <dancol@dancol.org>
> Date: Wed, 7 Mar 2018 11:38:47 -0800
> 
> > Take a look at enriched.el (and etc/enriched.txt as the demonstration
> > of its capabilities): we already have the capability for serializing
> > this, had it for many years.
> 
> That's fine so long as you have a file you want to edit _only_ with 
> Emacs. Files that need to be understood by other tools would need 
> information about indentation adjustment included in some compatible way 
> (maybe a comment), but since other tools would know nothing of this 
> information, it would go out of date. Requiring that people edit a 
> source file only with Emacs is not realistic.

Same thing happens with files produced by Office, no?  Why do we
require Emacs to do something different?

Re: Variable-width font indentation

[Clément Pit-Claudel]

On 2018-03-07 15:09, Eli Zaretskii wrote:
>> Cc: emacs-devel@gnu.org, eggert@cs.ucla.edu
>> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
>> Date: Wed, 7 Mar 2018 13:27:35 -0500
>>
>>> Not AFAIU, no.  It is an implementation of a different indentation
>>> method.  I don't see why we should want to go that way, since there's
>>> nothing wrong with the existing algorithm, except for handling of
>>> variable-pitch fonts.
>>
>> Can you clarify what you call an indentation method? I think that's the source of my confusion.
>> I wouldn't call what I wrote an indentation method.  Maybe I'm not understanding you because of that?
> 
> I feel that we are somehow bogged down in a miscommunication from
> which I see no way out...

I think we're making progress, actually :) At least, I'm understanding things better.

> An "indentation method" is a method used to determine the indentation
> of a line and to insert whitespace in order to achieve that
> indentation.

Understood.  Then what I'm offering isn't an indentation method: it doesn't insert whitespace.
But what I'm offering is indeed an imperfect heuristic to determine the indentation of a line.


>> But changing tab-width doesn't change the contents of the buffer, whereas what you're advocating for does, right?
> 
> No, it doesn't.  I'm advocating a way of changing the width of
> whitespace characters on display to account for variable-pitch font,
> not any change in the buffer.

OK, understood.  So for example if I have this snippet:

     01234567
     --------
  0 |x = (y + 
  1 |     z)

…then on line 1 the indentation function would put display properties on each of the 5 spaces leading to `z` to align it with the `y`.  Correct?

That's also what I do, only I compute the width of the spaces without calling the indentation function.

>> If it is correct, then it solves a problem that's different from the one I was thinking in, which is to display existing code with a variable pitch font, without the indentation function.
> 
> We cannot possibly DTRT with indentation without an indentation
> function.

Well, no, we can't DTRT every time without an indentation function.  But we can do fairly good, or at least it looks like that to me given the snippet I posted earlier.

(But note that I said the same thing at first, responding to Paul that his technique couldn't work.)

>> * What happens in major modes that don't provide indentation support?
> 
> The same as what happens today: the default functions are invoked.

I don't understand this part.  Concretely, what will the snippet above look like if it's shown in a prog-mode buffer, in a variable-pitch font?  Will the indentation just be off?

>> * What happens in indentation-sensitive languages (Python, F#, Haskell, etc)?
> 
> Same as today (the buffer contents will not change from what it is
> today).

I meant to ask what the code would look like, not what the buffer contents would change to.

>> * What happens in comments?
> 
> Nothing special.

Same question: what do the contents look like (what text properties do you apply on leading spaces in comments?)
For example, if I have this in a C buffer:

/* FIXME:
   - long
     text
     here
   - more text */

what will this look like in variable-pitch mode?

>> * Do I need to call M-x indent-region every time I switch between variable-pitch and fixed-pitch?
> 
> Not necessarily, not if the idea of controlling the SPC width is
> workable.  If it is, then the mode will simply set the value of two
> buffer-local variables, calculating them from the font in use.

That I don't understand.  Which two values are you thinking of?

As another concrete example, consider this snippet:

(do-stuff a
          b
          c)

won't you need to recompute the width of each leading space every time you change fonts?

>> * If I close and reopen a file in variable-pitch mode, do I need to re-run M-x indent-region?

I don't understand this part either: ow will you determine how to scale each space without calling the indentation function?

>> * In variable-pitch mode, what gets saved to disk after I run M-x indent-region?
> 
> Same as what we have now.

OK.
 

Re: What improvements would be truly useful?

[Marcin Borkowski]

On 2018-03-07, at 20:36, Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Marcin Borkowski <mbork@mbork.pl>
>> Cc: rms@gnu.org, dancol@dancol.org, eggert@cs.ucla.edu, raeburn@raeburn.org, rostislav.svoboda@gmail.com, emacs-devel@gnu.org
>> Date: Wed, 07 Mar 2018 19:30:07 +0100
>>
>> >> > Are you saying that either of these is a WYSIWYG word processor?
>> >>
>> >> Did Richard mention WYSIWYG word processing or just word processing?
>> >
>> > WYSIWYG, of course.  Otherwise it would make no sense to talk about
>> > indentation with variable-pitch fonts and pixel-level justification.
>>
>> Have you ever seen LyX?
>
> What would it add to the current discussion?

Next to nothing, except that variable-pitch fonts and pixel-level
justification do not imply WYSIWYG.

Personally, I don't care for the idea of having a word processor on top
of Emacs, although I admit that it might be interesting.  I think the
resources to do that could be better spent elsewhere.  But if someone
wants to do it - I have nothing against it.  I might even start to use
such an application.

Best,

--
Marcin Borkowski
http://mbork.pl

Re: What improvements would be truly useful?

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 729 bytes --]

Eli Zaretskii <eliz@gnu.org> schrieb am Mi., 7. März 2018 um 19:06 Uhr:

> > Cc: emacs-devel@gnu.org
> > From: Daniele Nicolodi <daniele@grinta.net>
> > Date: Wed, 7 Mar 2018 10:49:39 -0700
> >
> > > Then how should we understand your suggestion for "radical cleanup",
> > > right after asserting that GTK is the only modern toolkit supported by
> > > GNU/Linux?
> >
> > Radical cleanup of the GTK integration.
>
> If you have ideas for how to achieve that, I'd be interested to hear
> them.
>

By adding a gtkterm.c, that does the same as w32term.c, just for GTK,
without referring to X.org in any way?
The cleanup would then consist in removing all GTK functionality from
xterm.c once gtkterm.c is stable.

[-- Attachment #2: Type: text/html, Size: 1188 bytes --]

Re: Variable-width font indentation

[Clément Pit-Claudel]

On 2018-03-05 20:40, Clément Pit-Claudel wrote:
> On 2018-03-05 20:30, Paul Eggert wrote:
>> On 03/05/2018 05:06 PM, Clément Pit-Claudel wrote:
>>> In variable-pitch mode, each new indentation level has a
>>> slightly-different width, instead of a regular progression of 4
>>> spaces (in Python) and 2 spaces (in Elisp) at each level.
>>
>> That's fine. Although it's only a small thing, to me it's even a
>> small plus, as having every indent level be the same number of pixels
>> unnaturally focuses the reader on a distracting regularity that is
>> not intrinsic to the code.
> 
> I've been playing with this a bit more; until now, I'd mostly stopped at my previous objection.
> 
> The following code gives a preview of what the algorithm that we've been discussing produces: [...]

This version is a bit faster, and it uses 'display (space …) instead of ghost strings:

(defun ~/variable-pitch-indent ()
  (interactive)
  (save-excursion
    (let ((spine [])
          (prev-bol nil)
          (prev-eol nil))
      (goto-char (point-min))
      (while (not (eobp))
        (let ((bol (point)))
          (back-to-indentation)
          (unless (eolp)
            (let* ((indentation-amount (current-column))
                   (trunc-spine-to (min (length spine) indentation-amount)))
              (setq spine (seq-subseq spine 0 trunc-spine-to))
              ;; Extend spine based on previous line
              (when (and prev-bol (< (length spine) indentation-amount))
                (let ((beg (+ prev-bol (length spine)))
                      (end (min prev-eol (+ prev-bol indentation-amount))))
                  (setq spine (vconcat spine (~/measure-px-widths beg end)))))
              (setq prev-bol bol prev-eol (point-at-eol))))
          (dotimes (idx (min (length spine) (- (point) bol)))
            (let ((w (aref spine idx)))
              (put-text-property (+ bol idx) (+ bol idx 1)
                                 'display `(space :width (,w)))))
          (forward-line 1))))))

… but it doesn't work with tabs.

Re: What improvements would be truly useful?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Did Richard mention WYSIWYG word processing or just word processing?

  > WYSIWYG, of course.  Otherwise it would make no sense to talk about
  > indentation with variable-pitch fonts and pixel-level justification.

I didn't explicitly say WYSIWYG, because I took that for granted
because the other word processors I think of are WYSIWYG.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: Variable-width font indentation

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Take a look at enriched.el (and etc/enriched.txt as the demonstration
  > > of its capabilities): we already have the capability for serializing
  > > this, had it for many years.

  > That's fine so long as you have a file you want to edit _only_ with 
  > Emacs. Files that need to be understood by other tools would need 
  > information about indentation adjustment included in some compatible way 
  > (maybe a comment), but since other tools would know nothing of this 
  > information, it would go out of date. Requiring that people edit a 
  > source file only with Emacs is not realistic.

I agree.  One of the features we will want
is to make Emacs able to read and write ODT format and RTF,
even if at first it can only handle subsets of their features.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: What improvements would be truly useful?

[Stefan Monnier]

> I am really looking forward to this.  LibreOffice is free software,
> and it's ethical, but it isn't Emacs.

IMO the essence of WYSIWYG makes it "not Emacs".  I'd like to be able to
usefully edit .odt documents with Emacs, but not in a WYSIWYG fashion.


        Stefan

Re: Variable-width font indentation

[Stefan Monnier]

>> In your world, concretely, what happens when I open xdisp.c and press M-x
>> variable-pitch-mode? Does everything look misaligned until I M-x
>> indent-region the whole file?
> Yes.  It isn't different from what happens when you change tab-width.

It's different in one important way:
- it's considered normal to impose that everyone who looks/edits this
  file needs to have tad-width set to a particular value (8,
  traditionally).
- it's not considered normal to impose that everyone who looks/edits
  this file needs to use the same *font*.
Also, you can easily circumvent the tag-width issue by not using TABs in
your file, whereas it's hard to have a file whose appearance is not
affected by the choice of font.


        Stefan

Re: What improvements would be truly useful?

[Jefferson Carpenter]

On 3/7/2018 10:45 PM, Stefan Monnier wrote:
>> I am really looking forward to this.  LibreOffice is free software,
>> and it's ethical, but it isn't Emacs.
> 
> IMO the essence of WYSIWYG makes it "not Emacs".  I'd like to be able to
> usefully edit .odt documents with Emacs, but not in a WYSIWYG fashion.
> 
> 
>          Stefan
> 
> 

Agreed.  I like latex-preview-pane's format of "WYSIWYG" editing.  You 
edit the code in one buffer, and see a rendered preview in another 
buffer.  It is not critical that the preview even be an emacs buffer - 
it could be another application.

Jefferson

Re: What improvements would be truly useful?

[Yuri Khan]

On Thu, Mar 8, 2018 at 11:45 AM, Stefan Monnier
<monnier@iro.umontreal.ca> wrote:
> IMO the essence of WYSIWYG makes it "not Emacs".  I'd like to be able to
> usefully edit .odt documents with Emacs, but not in a WYSIWYG fashion.

I was about to say “you sort-of can, after all, .odt is just XML in
zip”. Then I did an (add-to-list 'auto-mode-alist '("\\.odt\\'" .
archive-mode)) and opened the first .odt I could find, and oh
$(DEITY), it’s horrible.

After the first line of xml declaration, the rest of the document is
one single physical line.

Re: What improvements would be truly useful?

[Toon Claes]

[-- Attachment #1: Type: text/plain, Size: 628 bytes --]

IMHO, one thing that I would find interesting is proper support
for multiple major modes.

I know there has been some talking about this in the past:

https://lists.gnu.org/archive/html/emacs-devel/2016-04/msg00585.html

But I'm not sure what the outcome was of that very long thread.

I only know later on some improvements were made to have html, css, and
js modes working together: 

https://lists.gnu.org/archive/html/emacs-devel/2017-01/msg00775.html

But still having great multiple major mode support--with islands having
their own major and minor modes--would make Emacs a more modern editor.

-- Toon

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Variable-width font indentation

[Robert Pluim]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > > Take a look at enriched.el (and etc/enriched.txt as the demonstration
>   > > of its capabilities): we already have the capability for serializing
>   > > this, had it for many years.
>
>   > That's fine so long as you have a file you want to edit _only_ with 
>   > Emacs. Files that need to be understood by other tools would need 
>   > information about indentation adjustment included in some compatible way 
>   > (maybe a comment), but since other tools would know nothing of this 
>   > information, it would go out of date. Requiring that people edit a 
>   > source file only with Emacs is not realistic.
>
> I agree.  One of the features we will want
> is to make Emacs able to read and write ODT format and RTF,
> even if at first it can only handle subsets of their features.

I'm still not clear on the goals here, and this makes things even less
clear. As others have mentioned, LibreOffice exists, so why does Emacs
need to be able to read and write ODT? [1]

Robert
Footnotes: 
[1]  Note that you can already indirectly write to ODT if you start
with org and use an exporter

Re: What improvements would be truly useful?

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Wed, 07 Mar 2018 21:45:46 +0000
> Cc: Daniele Nicolodi <daniele@grinta.net>, emacs-devel@gnu.org
> 
>  If you have ideas for how to achieve that, I'd be interested to hear
>  them.
> 
> By adding a gtkterm.c, that does the same as w32term.c, just for GTK, without referring to X.org in any way?
> The cleanup would then consist in removing all GTK functionality from xterm.c once gtkterm.c is stable.

(I guess you meant removing GTK-related portions of xfns.c as well?)

If you look at these two files, you will see that the GTK-specific
snippets in them are a few and far between.  Most of the code is
generic.  So I think making separate gtkterm.c/gtkfns.c will risk
duplicating a lot of code, something we already have with w32 and ns
ports, and such duplication is undesirable.  But maybe I'm missing
something.

My understanding of this sub-thread was that the idea was to separate
GTK _architecturally_, not just making it yet another display
back-end.

Re: Variable-width font indentation

[Eli Zaretskii]

> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Wed, 7 Mar 2018 17:13:09 -0500
> 
> This version is a bit faster, and it uses 'display (space …) instead of ghost strings:
> 
> (defun ~/variable-pitch-indent ()
>   (interactive)
>   (save-excursion
>     (let ((spine [])
>           (prev-bol nil)
>           (prev-eol nil))
>       (goto-char (point-min))
>       (while (not (eobp))
>         (let ((bol (point)))
>           (back-to-indentation)
>           (unless (eolp)
>             (let* ((indentation-amount (current-column))
>                    (trunc-spine-to (min (length spine) indentation-amount)))
>               (setq spine (seq-subseq spine 0 trunc-spine-to))
>               ;; Extend spine based on previous line
>               (when (and prev-bol (< (length spine) indentation-amount))
>                 (let ((beg (+ prev-bol (length spine)))
>                       (end (min prev-eol (+ prev-bol indentation-amount))))
>                   (setq spine (vconcat spine (~/measure-px-widths beg end)))))
>               (setq prev-bol bol prev-eol (point-at-eol))))
>           (dotimes (idx (min (length spine) (- (point) bol)))
>             (let ((w (aref spine idx)))
>               (put-text-property (+ bol idx) (+ bol idx 1)
>                                  'display `(space :width (,w)))))
>           (forward-line 1))))))

It seems you forgot to post ~/measure-px-widths.

Re: What improvements would be truly useful?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Wed, 07 Mar 2018 23:45:05 -0500
> 
> > I am really looking forward to this.  LibreOffice is free software,
> > and it's ethical, but it isn't Emacs.
> 
> IMO the essence of WYSIWYG makes it "not Emacs".  I'd like to be able to
> usefully edit .odt documents with Emacs, but not in a WYSIWYG fashion.

Emacs being Emacs, no one is forced to use any specific Emacs feature
they don't like or want.

Re: Variable-width font indentation

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Wed, 07 Mar 2018 23:50:45 -0500
> 
> >> In your world, concretely, what happens when I open xdisp.c and press M-x
> >> variable-pitch-mode? Does everything look misaligned until I M-x
> >> indent-region the whole file?
> > Yes.  It isn't different from what happens when you change tab-width.
> 
> It's different in one important way:
> - it's considered normal to impose that everyone who looks/edits this
>   file needs to have tad-width set to a particular value (8,
>   traditionally).
> - it's not considered normal to impose that everyone who looks/edits
>   this file needs to use the same *font*.

The above doesn't force any particular font, just a fixed-pitch one, a
limitation we already have today.  So we will be no worse.

> Also, you can easily circumvent the tag-width issue by not using TABs in
> your file, whereas it's hard to have a file whose appearance is not
> affected by the choice of font.

Yes, using variable-pitch fonts in code where indentation is important
has its disadvantages, and always will.  Any solution will have the
same issues, just under different use cases and situations.  That's
why this is not the main use case where we should support
variable-pitch fonts well, the main use case is in text-related modes,
where in many/most cases the indentation is rigid or changes
proportionally with the width of some canonical character.

Re: Variable-width font indentation

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: Daniel Colascione <dancol@dancol.org>,  eliz@gnu.org,  sds@gnu.org,  emacs-devel@gnu.org
> Date: Thu, 08 Mar 2018 13:33:01 +0100
> 
> >   > (maybe a comment), but since other tools would know nothing of this 
> >   > information, it would go out of date. Requiring that people edit a 
> >   > source file only with Emacs is not realistic.
> >
> > I agree.  One of the features we will want
> > is to make Emacs able to read and write ODT format and RTF,
> > even if at first it can only handle subsets of their features.
> 
> I'm still not clear on the goals here, and this makes things even less
> clear. As others have mentioned, LibreOffice exists, so why does Emacs
> need to be able to read and write ODT? [1]

So that we could exchange documents with users of Office.

Re: Variable-width font indentation

[Clément Pit-Claudel]

On 2018-03-08 08:21, Eli Zaretskii wrote:
>> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
>> Date: Wed, 7 Mar 2018 17:13:09 -0500
>>
>> This version is a bit faster, and it uses 'display (space …) instead of ghost strings:
>> ...
>
> It seems you forgot to post ~/measure-px-widths.

Indeed, apologies :/

(defun ~/measure-px-widths (beg end)
  (let ((widths nil))
    (while (< beg end)
      (push (car (window-text-pixel-size nil beg (1+ beg))) widths)
      (cl-incf beg))
    (nreverse widths)))

(defun ~/variable-pitch-indent ()
  (interactive)
  (save-excursion
    (let ((spine [])
          (prev-bol nil)
          (prev-eol nil))
      (goto-char (point-min))
      (while (not (eobp))
        (let ((bol (point)))
          (back-to-indentation)
          (unless (eolp)
            (let* ((indentation-amount (current-column))
                   (trunc-spine-to (min (length spine) indentation-amount)))
              (setq spine (seq-subseq spine 0 trunc-spine-to))
              ;; Extend spine based on previous line
              (when (and prev-bol (< (length spine) indentation-amount))
                (let ((beg (+ prev-bol (length spine)))
                      (end (min prev-eol (+ prev-bol indentation-amount))))
                  (setq spine (vconcat spine (~/measure-px-widths beg end)))))
              (setq prev-bol bol prev-eol (point-at-eol))))
          (dotimes (idx (min (length spine) (- (point) bol)))
            (let ((w (aref spine idx)))
              (put-text-property (+ bol idx) (+ bol idx 1)
                                 'display `(space :width (,w)))))
          (forward-line 1))))))

Re: Variable-width font indentation

[Robert Pluim]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Robert Pluim <rpluim@gmail.com>
>> Cc: Daniel Colascione <dancol@dancol.org>,  eliz@gnu.org,  sds@gnu.org,  emacs-devel@gnu.org
>> Date: Thu, 08 Mar 2018 13:33:01 +0100
>> 
>> >   > (maybe a comment), but since other tools would know nothing of this 
>> >   > information, it would go out of date. Requiring that people edit a 
>> >   > source file only with Emacs is not realistic.
>> >
>> > I agree.  One of the features we will want
>> > is to make Emacs able to read and write ODT format and RTF,
>> > even if at first it can only handle subsets of their features.
>> 
>> I'm still not clear on the goals here, and this makes things even less
>> clear. As others have mentioned, LibreOffice exists, so why does Emacs
>> need to be able to read and write ODT? [1]
>
> So that we could exchange documents with users of Office.

OK. I personally have no such goal (at least not with Emacs involved),
and I can't understand why people would want to, but everyone gets to
scratch their own itch.

Robert

Re: Variable-width font indentation

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org, eggert@cs.ucla.edu
> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Wed, 7 Mar 2018 15:32:53 -0500
> 
> >> But changing tab-width doesn't change the contents of the buffer, whereas what you're advocating for does, right?
> > 
> > No, it doesn't.  I'm advocating a way of changing the width of
> > whitespace characters on display to account for variable-pitch font,
> > not any change in the buffer.
> 
> OK, understood.  So for example if I have this snippet:
> 
>      01234567
>      --------
>   0 |x = (y + 
>   1 |     z)
> 
> …then on line 1 the indentation function would put display properties on each of the 5 spaces leading to `z` to align it with the `y`.  Correct?

Not necessarily display properties, maybe some (most? all?) of the
effect could be achieved by modifying the display width of a space and
a tab according to some buffer-local variable, by the display code
itself, similarly to how we render tab characters now.

> >> * What happens in major modes that don't provide indentation support?
> > 
> > The same as what happens today: the default functions are invoked.
> 
> I don't understand this part.  Concretely, what will the snippet above look like if it's shown in a prog-mode buffer, in a variable-pitch font?  Will the indentation just be off?

When a major mode doesn't provide an indentation function, indent.el
uses the default function, so I'm not sure what problem you see here.

> >> * What happens in indentation-sensitive languages (Python, F#, Haskell, etc)?
> > 
> > Same as today (the buffer contents will not change from what it is
> > today).
> 
> I meant to ask what the code would look like, not what the buffer contents would change to.

Hopefully, the buffer will look like users of those languages expect
it to look.

> >> * What happens in comments?
> > 
> > Nothing special.
> 
> Same question: what do the contents look like (what text properties do you apply on leading spaces in comments?)

I hope we will be able to manage without display properties at all.

> For example, if I have this in a C buffer:
> 
> /* FIXME:
>    - long
>      text
>      here
>    - more text */
> 
> what will this look like in variable-pitch mode?

Hopefully, very similar, although we won't be able to guarantee
column-wise alignment for arbitrary text.  But leading whitespace will
certainly be aligned, something that is not universally true today.

> >> * Do I need to call M-x indent-region every time I switch between variable-pitch and fixed-pitch?
> > 
> > Not necessarily, not if the idea of controlling the SPC width is
> > workable.  If it is, then the mode will simply set the value of two
> > buffer-local variables, calculating them from the font in use.
> 
> That I don't understand.  Which two values are you thinking of?

tab-width and (the hypothetical) space-width.  Actually, only the
latter, since tab-width must be its integral multiple.

> As another concrete example, consider this snippet:
> 
> (do-stuff a
>           b
>           c)
> 
> won't you need to recompute the width of each leading space every time you change fonts?

Hopefully, no.  At least not for approximate alignment.

> >> * If I close and reopen a file in variable-pitch mode, do I need to re-run M-x indent-region?
> 
> I don't understand this part either: ow will you determine how to scale each space without calling the indentation function?

Hopefully, by calculating some representative metrics of the font.

Re: What improvements would be truly useful?

[Alan Third]

On Thu, Mar 08, 2018 at 03:20:58PM +0200, Eli Zaretskii wrote:
> > From: Philipp Stephani <p.stephani2@gmail.com>
> > Date: Wed, 07 Mar 2018 21:45:46 +0000
> > Cc: Daniele Nicolodi <daniele@grinta.net>, emacs-devel@gnu.org
> >
> > By adding a gtkterm.c, that does the same as w32term.c, just for GTK, without referring to X.org in any way?
> > The cleanup would then consist in removing all GTK functionality from xterm.c once gtkterm.c is stable.
> 
> (I guess you meant removing GTK-related portions of xfns.c as well?)
> 
> If you look at these two files, you will see that the GTK-specific
> snippets in them are a few and far between.  Most of the code is
> generic.

That’s part of the problem with how Emacs deals with GTK, it does many
things the X way rather than the GTK way.

For example, mouse and keyboard events are dealt with using X code,
but GTK provides it’s own APIs for dealing with them. I thought
implementing touch events would be easy because GTK already does all
the heavy lifting, but Emacs doesn’t use GTK for events.

I strongly suspect a Wayland compatible GTK Emacs would have to remove
a lot of (all?) X code and replace it with GTK code.

> My understanding of this sub-thread was that the idea was to separate
> GTK _architecturally_, not just making it yet another display
> back-end.

I don’t think I understand the difference.
-- 
Alan Third

Re: Variable-width font indentation

[Clément Pit-Claudel]

On 2018-03-08 10:39, Eli Zaretskii wrote:
>> From: Clément Pit-Claudel <cpitclaudel@gmail.com>buffer.
>>
>> OK, understood.  So for example if I have this snippet:
>>
>>      01234567
>>      --------
>>   0 |x = (y + 
>>   1 |     z)
>>
>> …then on line 1 the indentation function would put display properties on each of the 5 spaces leading to `z` to align it with the `y`.  Correct?
> 
> Not necessarily display properties, maybe some (most? all?) of the
> effect could be achieved by modifying the display width of a space and
> a tab according to some buffer-local variable, by the display code
> itself, similarly to how we render tab characters now.

Hmm, interesting.  How would that work? For example, if I have these two snippets:

i = (y + 
     z)

w = (y + 
     z)

with monospace fonts 'i' and 'w' have the same width, but with variable-pitch fonts they don't and if you assign a constant width to each space the 'y' and 'z' will line up in at most one of the two examples, right?

>>>> * What happens in major modes that don't provide indentation support?
>>>
>>> The same as what happens today: the default functions are invoked.
>>
>> I don't understand this part.  Concretely, what will the snippet above look like if it's shown in a prog-mode buffer, in a variable-pitch font?  Will the indentation just be off?
> 
> When a major mode doesn't provide an indentation function, indent.el
> uses the default function, so I'm not sure what problem you see here.

I don't know what the default function is, or how it could help us determine proper alignment :/
I thought from earlier emails that you'd use existing indentation functions to determine proper indentation.

>> For example, if I have this in a C buffer:
>>
>> /* FIXME:
>>    - long
>>      text
>>      here
>>    - more text */
>>
>> what will this look like in variable-pitch mode?
> 
> Hopefully, very similar, although we won't be able to guarantee
> column-wise alignment for arbitrary text.  But leading whitespace will
> certainly be aligned, something that is not universally true today.

Ah, that I understand :)
I've been using the following for that purpose for a long time:

  (font-lock-add-keywords nil `(("^[[:space:]]+" 0 '(face fixed-pitch) append)) 'append)

>>>> * Do I need to call M-x indent-region every time I switch between variable-pitch and fixed-pitch?
>>>
>>> Not necessarily, not if the idea of controlling the SPC width is
>>> workable.  If it is, then the mode will simply set the value of two
>>> buffer-local variables, calculating them from the font in use.
>>
>> That I don't understand.  Which two values are you thinking of?
> 
> tab-width and (the hypothetical) space-width.  Actually, only the
> latter, since tab-width must be its integral multiple.

OK, I see.  This won't help with cases like the following, right?

  int main(int argv,
           char** argc)

>> As another concrete example, consider this snippet:
>>
>> (do-stuff a
>>           b
>>           c)
>>
>> won't you need to recompute the width of each leading space every time you change fonts?
> 
> Hopefully, no.  At least not for approximate alignment.

OK. I don't understand how that part works yet.
The solution you propose doesn't get alignment right if I have the following ELisp code, right?

(progn
  (wwwwwwww a
            b
            c)
  (iiiiiiii a
            b
            c))

>>>> * If I close and reopen a file in variable-pitch mode, do I need to re-run M-x indent-region?
>>
>> I don't understand this part either: ow will you determine how to scale each space without calling the indentation function?
> 
> Hopefully, by calculating some representative metrics of the font.

OK.  I think I follow you at this point, although now I'm not sure anymore what role indentation functions will play.  A few emails ago you mentioned teaching indentation functions about variable-pitch, but the solution you're offering now doesn't seem to require that anymore, does it?

Clément.

Re: What improvements would be truly useful?

[Daniel Colascione]

On 03/08/2018 08:24 AM, Alan Third wrote:
> On Thu, Mar 08, 2018 at 03:20:58PM +0200, Eli Zaretskii wrote:
>>> From: Philipp Stephani <p.stephani2@gmail.com>
>>> Date: Wed, 07 Mar 2018 21:45:46 +0000
>>> Cc: Daniele Nicolodi <daniele@grinta.net>, emacs-devel@gnu.org
>>>
>>> By adding a gtkterm.c, that does the same as w32term.c, just for GTK, without referring to X.org in any way?
>>> The cleanup would then consist in removing all GTK functionality from xterm.c once gtkterm.c is stable.
>>
>> (I guess you meant removing GTK-related portions of xfns.c as well?)
>>
>> If you look at these two files, you will see that the GTK-specific
>> snippets in them are a few and far between.  Most of the code is
>> generic.
> 
> That’s part of the problem with how Emacs deals with GTK, it does many
> things the X way rather than the GTK way.
> 
> For example, mouse and keyboard events are dealt with using X code,
> but GTK provides it’s own APIs for dealing with them. I thought
> implementing touch events would be easy because GTK already does all
> the heavy lifting, but Emacs doesn’t use GTK for events.
> 
> I strongly suspect a Wayland compatible GTK Emacs would have to remove
> a lot of (all?) X code and replace it with GTK code.
> 
>> My understanding of this sub-thread was that the idea was to separate
>> GTK _architecturally_, not just making it yet another display
>> back-end.
> 
> I don’t think I understand the difference.

Emacs can be compiled with support for many different window systems, 
such as X11, MS-Windows, and Cocoa/NS. Currently, having chosen the X11 
window system, you can go on to choose which toolkit you want to use: 
Motif, GTK2, GTK3, Lucid, and so on.

The proposal is to promote GTK support from an X11 toolkit to a 
top-level window system, excising, in the process, the GTK display 
back-end of any mention of X11.

I think it's an excellent project and one important to the future 
maintainability of the project. As much as it saddens* me to say so, X11 
is dying, while GTK will continue to be a decent abstraction that will 
hit the happy, conventional path of GNU/Linux input and display 
infrastructure. Emacs X11 integration being unusual and dubiously legal 
has caused compatibility problems in the past, particularly with systems 
like NX.

As part of this effort, I'd love to see more abstraction in the window 
system selection. All the xterm.c functions that we carefully replicate 
in every window system really should be a table of frame-specific 
function pointers, and we should be able to compile a single Emacs 
binary with support for all window systems enabled simultaneously.

* IMHO, Wayland should be been a series of new X protocol extensions, 
not a greenfield project.

Re: Variable-width font indentation

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org, eggert@cs.ucla.edu
> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Thu, 8 Mar 2018 11:30:51 -0500
> 
> > Not necessarily display properties, maybe some (most? all?) of the
> > effect could be achieved by modifying the display width of a space and
> > a tab according to some buffer-local variable, by the display code
> > itself, similarly to how we render tab characters now.
> 
> Hmm, interesting.  How would that work? For example, if I have these two snippets:
> 
> i = (y + 
>      z)
> 
> w = (y + 
>      z)
> 
> with monospace fonts 'i' and 'w' have the same width, but with variable-pitch fonts they don't and if you assign a constant width to each space the 'y' and 'z' will line up in at most one of the two examples, right?

Maybe we could find a middle ground, whereby each one of the examples
will approximately align.  If that can be done, and the result is
acceptable, then the problem of recording the text properties in the
file and/or reindenting when the file is revisited goes away.

> >>>> * What happens in major modes that don't provide indentation support?
> >>>
> >>> The same as what happens today: the default functions are invoked.
> >>
> >> I don't understand this part.  Concretely, what will the snippet above look like if it's shown in a prog-mode buffer, in a variable-pitch font?  Will the indentation just be off?
> > 
> > When a major mode doesn't provide an indentation function, indent.el
> > uses the default function, so I'm not sure what problem you see here.
> 
> I don't know what the default function is, or how it could help us determine proper alignment :/

Looking in indent.el, my reading of the code is that the default
function is indent-rigidly.

> I've been using the following for that purpose for a long time:
> 
>   (font-lock-add-keywords nil `(("^[[:space:]]+" 0 '(face fixed-pitch) append)) 'append)

Is it really needed?  With most font back-ends, a tab is displayed as
a stretch glyph whose width is an integral multiple of the width of
the space glyph for the same font.  So they should align even without
the fixed-pitch font trick.

> >>> Not necessarily, not if the idea of controlling the SPC width is
> >>> workable.  If it is, then the mode will simply set the value of two
> >>> buffer-local variables, calculating them from the font in use.
> >>
> >> That I don't understand.  Which two values are you thinking of?
> > 
> > tab-width and (the hypothetical) space-width.  Actually, only the
> > latter, since tab-width must be its integral multiple.
> 
> OK, I see.  This won't help with cases like the following, right?
> 
>   int main(int argv,
>            char** argc)

It might get reasonably close, though.  Maybe this will be good
enough.

> OK.  I think I follow you at this point, although now I'm not sure anymore what role indentation functions will play.  A few emails ago you mentioned teaching indentation functions about variable-pitch, but the solution you're offering now doesn't seem to require that anymore, does it?

That's more relevant for text-derived modes, where indentation levels
are rigid and not determined by previous lines.  There we could do a
better job, I hope.

Re: Variable-width font indentation

[Clément Pit-Claudel]

On 2018-03-08 14:07, Eli Zaretskii wrote:
>> Cc: emacs-devel@gnu.org, eggert@cs.ucla.edu
>> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
>> Date: Thu, 8 Mar 2018 11:30:51 -0500
>>
>>> Not necessarily display properties, maybe some (most? all?) of the
>>> effect could be achieved by modifying the display width of a space and
>>> a tab according to some buffer-local variable, by the display code
>>> itself, similarly to how we render tab characters now.
>>
>> Hmm, interesting.  How would that work? For example, if I have these two snippets:
>>
>> i = (y + 
>>      z)
>>
>> w = (y + 
>>      z)
>>
>> with monospace fonts 'i' and 'w' have the same width, but with variable-pitch fonts they don't and if you assign a constant width to each space the 'y' and 'z' will line up in at most one of the two examples, right?
> 
> Maybe we could find a middle ground, whereby each one of the examples
> will approximately align.  If that can be done, and the result is
> acceptable, then the problem of recording the text properties in the
> file and/or reindenting when the file is revisited goes away.

Maybe :)  I don't know how good it would look, in practice, but it's easy to experiment.
Paul's algorithm is not too bad in that sense: these two examples align pixel-perfectly.

>> I've been using the following for that purpose for a long time:
>>
>>   (font-lock-add-keywords nil `(("^[[:space:]]+" 0 '(face fixed-pitch) append)) 'append)
> 
> Is it really needed?  With most font back-ends, a tab is displayed as
> a stretch glyph whose width is an integral multiple of the width of
> the space glyph for the same font.  So they should align even without
> the fixed-pitch font trick.

Hmm, indeed.  So which problems did you have in mind when you said the following?

> But leading whitespace will certainly be aligned, something that is 
> not universally true today.
>> OK, I see.  This won't help with cases like the following, right?
>>
>>   int main(int argv,
>>            char** argc)
> 
> It might get reasonably close, though.  Maybe this will be good
> enough.

OK

>> OK.  I think I follow you at this point, although now I'm not sure anymore what role indentation functions will play.  A few emails ago you mentioned teaching indentation functions about variable-pitch, but the solution you're offering now doesn't seem to require that anymore, does it?
> 
> That's more relevant for text-derived modes, where indentation levels
> are rigid and not determined by previous lines.  There we could do a
> better job, I hope.

OK.  I tend to think of indentation even in text modes as being somewhat previous-lines dependent (wrapped paragraphs align relative to bullet points, for example), but I agree that it isn't as much as in programming modes.

Re: What improvements would be truly useful?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > IMO the essence of WYSIWYG makes it "not Emacs".

That is shocking but enigmatic; I don't get any point.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: What improvements would be truly useful?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > But still having great multiple major mode support--with islands having
  > their own major and minor modes--would make Emacs a more modern editor.

I strongly support the request.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: What improvements would be truly useful?

[Marcin Borkowski]

On 2018-03-08, at 05:45, Stefan Monnier <monnier@iro.umontreal.ca> wrote:

>> I am really looking forward to this.  LibreOffice is free software,
>> and it's ethical, but it isn't Emacs.
>
> IMO the essence of WYSIWYG makes it "not Emacs".  I'd like to be able to
> usefully edit .odt documents with Emacs, but not in a WYSIWYG fashion.

Interesting.  I have a very similar sentiment, though I think it is
quite irrational.

Is mu4e, Org-mode, Calc, or dunnet also "not Emacs"?  If not, how is it
different than a word processor?

I'm not mocking you, I just want to understand the sentiment we both
share.

Best,

-- 
Marcin Borkowski
http://mbork.pl

Re: Variable-width font indentation

[Paul Eggert]

On 03/08/2018 05:44 AM, Eli Zaretskii wrote:
> Yes, using variable-pitch fonts in code where indentation is important
> has its disadvantages, and always will.  Any solution will have the
> same issues, just under different use cases and situations.  That's
> why this is not the main use case

Although it has disadvantages, it shouldn't be that hard to do it better 
than Emacs does it now (along the lines that Clément prototyped), and if 
Emacs had these improvements the benefits of variable-pitch fonts would 
outweigh the costs in many practical programming applications. 
Programming in variable-width fonts makes more-efficient use of valuable 
screen real estate, so it's worth doing when it works reasonably well, 
as it often would if Emacs supported it better.

Re: What improvements would be truly useful?

[Stefan Monnier]

> Interesting.  I have a very similar sentiment, though I think it is
> quite irrational.
> Is mu4e, Org-mode, Calc, or dunnet also "not Emacs"?  If not, how is it
> different than a word processor?

I don't think I can clearly pin-point the problem, but I think the issue
has to do with hiding information: WYSIWYG hides information, in the
sense that there are many different documents with the same visual
appearance, so the user needs to *guess* what is the underlying
document's structure in order to know what his actions will really do.
A typical example is the beginning/end of a bold text which is only
implicitly represented by the fact that some text is bold and other
isn't, but that doesn't let you easily tell whether a new word inserted
between a bold work and a non-bold word will be bold or not.

Of course Emacs can also hide information (as text-properties, as
invisible text, as data stored in buffer-local variables, ...) but most
packages follow a design where as little info as possible is hidden.
Indeed, whenever I hide such information, I think it over many times
because I know there's a very strong chance that users won't like it.

Also when it comes to data that we edit, hiding information either means
hiding it as invisible text (which tends to be fiddly, so it's used
fairly sparingly), or it implies using a complex procedure to `find` and
to `save` the buffer in order to translate between the file contents and
the in-buffer contents.  This complexity is a strong encouragement not
to go down that route.


        Stefan

Re: Variable-width font indentation

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org, eggert@cs.ucla.edu
> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Thu, 8 Mar 2018 14:55:20 -0500
> 
> > Maybe we could find a middle ground, whereby each one of the examples
> > will approximately align.  If that can be done, and the result is
> > acceptable, then the problem of recording the text properties in the
> > file and/or reindenting when the file is revisited goes away.
> 
> Maybe :)  I don't know how good it would look, in practice, but it's easy to experiment.
> Paul's algorithm is not too bad in that sense: these two examples align pixel-perfectly.

Yes, but as I said, I don't think that approach is appropriate for
code indentation, because leading whitespace in code blocks _must_
always align, IMO.  IOW, that approach seems to favor alignment of
multi-line code sequences to alignment of indented blocks, and I think
programming modes should have the opposite preference.

> > Is it really needed?  With most font back-ends, a tab is displayed as
> > a stretch glyph whose width is an integral multiple of the width of
> > the space glyph for the same font.  So they should align even without
> > the fixed-pitch font trick.
> 
> Hmm, indeed.  So which problems did you have in mind when you said the following?
> 
> > But leading whitespace will certainly be aligned, something that is 
> > not universally true today.

At least the w32 font back-end doesn't fulfill that requirement,
because it calculates the font's space width differently (due to
practical problems related to the MS-Windows APIs).  Not sure about
ns.

> > That's more relevant for text-derived modes, where indentation levels
> > are rigid and not determined by previous lines.  There we could do a
> > better job, I hope.
> 
> OK.  I tend to think of indentation even in text modes as being somewhat previous-lines dependent (wrapped paragraphs align relative to bullet points, for example), but I agree that it isn't as much as in programming modes.

IME, indentation of text in word processors I use does not depend on
previous lines at all.

Re: Variable-width font indentation

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Thu, 8 Mar 2018 14:03:21 -0800
> 
> On 03/08/2018 05:44 AM, Eli Zaretskii wrote:
> > Yes, using variable-pitch fonts in code where indentation is important
> > has its disadvantages, and always will.  Any solution will have the
> > same issues, just under different use cases and situations.  That's
> > why this is not the main use case
> 
> Although it has disadvantages, it shouldn't be that hard to do it better 
> than Emacs does it now (along the lines that Clément prototyped), and if 
> Emacs had these improvements the benefits of variable-pitch fonts would 
> outweigh the costs in many practical programming applications. 

I agree (although I don't think we should use the basic idea of
calculating the indentation that the prototype used).

> Programming in variable-width fonts makes more-efficient use of valuable 
> screen real estate, so it's worth doing when it works reasonably well, 
> as it often would if Emacs supported it better.

Agreed.

Do people think adding a per-buffer space-width variable would be a
good step in this direction?  It should be easy to add, I think.

Re: What improvements would be truly useful?

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Marcin Borkowski <mbork@mbork.pl> Cc: Daniel Colascione
>> <dancol@dancol.org>, eggert@cs.ucla.edu, eliz@gnu.org,
>> raeburn@raeburn.org, rostislav.svoboda@gmail.com, emacs-devel@gnu.org
>> Date: Tue, 06 Mar 2018 21:01:00 +0100
>> 
>> > I want to do my word processing in Emacs,
>> 
>> You already can.  Quite a few people do this, using Org-mode or AUCTeX.
>
> Are you saying that either of these is a WYSIWYG word processor?

No. But, then Word or LibreOffice are not WYSIWYG either. Both have
extensive modal views of the underlying document (web layout, reading
mode, outline view).

Org-mode is essentially a structure first view of text. AUCTeX with
preview-latex is roughly the same thing. So, both of them have strong
aspects of modern word-processor.

If Emacs were able to turn org-mode into HTML, render that HTML and
then make changes to the text of that HTML, then you'd have something
close to the modal views. Or something like preview-latex which uses
syntax highlighting to provide the WYS aspect but allows you to turn it
on and off.

How easy this would be to implement, I do not know. Emacs does not have
an MVC architecture -- text properties (i.e. visualization) are stored
directly with the contents of a buffer, so switching views is clunky or
difficult (see my own lentic.el for an example).

Phil

Re: What improvements would be truly useful?

[Phillip Lord]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > IMO the future of GUI apps and in our case editors, lies in browser
>   > based frameworks programmable by some lisp dialect. For the moment it
>   > would be a stack based on electron, clojurescript, codemirror, etc.
>
> I have a feeling we are talking about totally unrelated questions.
> I'm talking about extending Emacs to edit more kinds of _documents_.
> Documents with more and different layout features, but still _documents_.
>
> A document is a self-contained file of declarative data that you can
> display and edit on your computer, without any sort of network
> connection.  And if you send it to me, I can display it too
> and it looks the same for me as for you.


No, it really isn't. A document is a user-centric feature -- it's a
piece of work that stores information in a way that it useful to the
user. The self-contained file notion is a developer issue.  And,
document editing has moved partially (although not wholly) away from the
"your computer". People edit documents in editors built entirely on the
web. To show it to you, I just send the link.

Following your definition, MS Word becomes a document editor and MS Word
Online does not.

Phil

Re: What improvements would be truly useful?

[Eli Zaretskii]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Cc: Marcin Borkowski <mbork@mbork.pl>,  eggert@cs.ucla.edu,  rms@gnu.org,  emacs-devel@gnu.org,  raeburn@raeburn.org,  dancol@dancol.org,  rostislav.svoboda@gmail.com
> Date: Fri, 09 Mar 2018 10:56:00 +0000
> 
> > Are you saying that either of these is a WYSIWYG word processor?
> 
> No. But, then Word or LibreOffice are not WYSIWYG either.

This is just a minor semantic issue: let's assume for the purposes of
the current discussion that the definition of a WYSIWYG word processor
is what the Office word processors do.

> Org-mode is essentially a structure first view of text. AUCTeX with
> preview-latex is roughly the same thing. So, both of them have strong
> aspects of modern word-processor.

They both provide the final view of the document only off-line, and
that is the crucial difference, for the purposes of this discussion.

> If Emacs were able to turn org-mode into HTML, render that HTML and
> then make changes to the text of that HTML, then you'd have something
> close to the modal views. Or something like preview-latex which uses
> syntax highlighting to provide the WYS aspect but allows you to turn it
> on and off.

Emacs is capable of displaying text with different typefaces, so it
isn't clear to me why would we need to go through a translator, such
as Org export or LaTeX.

> How easy this would be to implement, I do not know. Emacs does not have
> an MVC architecture -- text properties (i.e. visualization) are stored
> directly with the contents of a buffer, so switching views is clunky or
> difficult (see my own lentic.el for an example).

Actually, MVC is exactly the Emacs architecture.  (And text properties
are stored separately from the buffer contents.)  I wonder what kind
of misunderstanding is at work here.

Re: Variable-width font indentation

[Clément Pit-Claudel]

On 2018-03-09 03:34, Eli Zaretskii wrote:
> Do people think adding a per-buffer space-width variable would be a
> good step in this direction?  It should be easy to add, I think.

It does sound convenient.  Will it apply only to leading whitespace, or also to inter-word whitespace?

Also, do we need it on the C side?  It sounds fairly easy to achieve with existing tools like font-lock + display properties.

Clément.

Re: Variable-width font indentation

[Clément Pit-Claudel]

On 2018-03-07 13:26, Eli Zaretskii wrote:
> Note that 'space' display property supports values
> that are symbols, in which case it uses the symbol's value -- this
> allows Lisp programs to change the value of a buffer-local binding of
> the symbol, and have the indentation change at the next redisplay
> cycle (a.k.a. "immediately").  Doesn't that provide a better way for
> controlling indentation than font-lock style scanning of the visible
> portion of the buffer "just in time"?

Sorry, I missed this email before.
Yes, this sounds pretty good, and I understand the reticence to depending on font-lock.

Re: Variable-width font indentation

[Eli Zaretskii]

> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Fri, 9 Mar 2018 11:05:13 -0500
> 
> On 2018-03-09 03:34, Eli Zaretskii wrote:
> > Do people think adding a per-buffer space-width variable would be a
> > good step in this direction?  It should be easy to add, I think.
> 
> It does sound convenient.  Will it apply only to leading whitespace, or also to inter-word whitespace?

I'd say it should apply everywhere.  It would be confusing to have SPC
take a different number of pixels depending on where it is.

> Also, do we need it on the C side?  It sounds fairly easy to achieve with existing tools like font-lock + display properties.

Display properties have the disadvantage that they cannot be easily
saved.  Basically, having this in the display code is convenient for
the same reason the TAB display is handled in C.

Re: Variable-width font indentation

[Paul Eggert]

On 03/09/2018 12:30 AM, Eli Zaretskii wrote:
> I don't think that approach is appropriate for
> code indentation, because leading whitespace in code blocks_must_
> always align, IMO.

I don't quite follow this objection, since the prototype does align 
leading whitespace in code blocks. That is, in any particular block of 
code consisting of lines prefaced by equivalent white space, the 
approach displays the block with equally-indented lines, so the leading 
whitespace does align.

If the objection is that different blocks of code are indented 
differently (e.g., an if-part might be indented slightly differently 
from the corresponding then-part), then I agree that might irritate some 
users. However, the irritation is quite small to me, and I imagine it 
wouldn't bother other users much either, in the sense that they'll be 
willing to put up with this minor irritation in order to get the 
benefits that the algorithm has elsewhere. So it still sounds like a 
reasonable thing to support, if a user wants to use it.

Of course other display algorithms are possible, but I haven't seen any 
other improvement proposed for variable-pitch fonts that is nearly as 
simple and easy to understand. Simplicity is a virtue here.

Re: Variable-width font indentation

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Fri, 9 Mar 2018 09:52:28 -0800
> 
> If the objection is that different blocks of code are indented 
> differently (e.g., an if-part might be indented slightly differently 
> from the corresponding then-part), then I agree that might irritate some 
> users. However, the irritation is quite small to me, and I imagine it 
> wouldn't bother other users much either

Well, it bothers me a lot, to the degree that I don't think it could
fly.

> Of course other display algorithms are possible, but I haven't seen any 
> other improvement proposed for variable-pitch fonts that is nearly as 
> simple and easy to understand. Simplicity is a virtue here.

Simplicity is a virtue if it gets the job done.  This particular
suggestion doesn't, IMO.

Re: Variable-width font indentation

[Clément Pit-Claudel]

On 2018-03-09 03:30, Eli Zaretskii wrote:
>> Cc: emacs-devel@gnu.org, eggert@cs.ucla.edu
>> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
>> Date: Thu, 8 Mar 2018 14:55:20 -0500
>>
>>> Maybe we could find a middle ground, whereby each one of the examples
>>> will approximately align.  If that can be done, and the result is
>>> acceptable, then the problem of recording the text properties in the
>>> file and/or reindenting when the file is revisited goes away.
>>
>> Maybe :)  I don't know how good it would look, in practice, but it's easy to experiment.
>> Paul's algorithm is not too bad in that sense: these two examples align pixel-perfectly.
> 
> Yes, but as I said, I don't think that approach is appropriate for
> code indentation, because leading whitespace in code blocks _must_
> always align, IMO.  IOW, that approach seems to favor alignment of
> multi-line code sequences to alignment of indented blocks, and I think
> programming modes should have the opposite preference.

Indeed :) I was mostly of the same opinion, but I find the actual results pretty OK.

>>> That's more relevant for text-derived modes, where indentation levels
>>> are rigid and not determined by previous lines.  There we could do a
>>> better job, I hope.
>>
>> OK.  I tend to think of indentation even in text modes as being somewhat previous-lines dependent (wrapped paragraphs align relative to bullet points, for example), but I agree that it isn't as much as in programming modes.
> 
> IME, indentation of text in word processors I use does not depend on
> previous lines at all.

Except in lists, right? For example it's not uncommon to wrap like this:

* Some long text here
  that wraps around:

  i.   Point 1
       … continued
  ii.  Point 2
       … continued
  iii. Point 3
       … continued

Clément

Re: Variable-width font indentation

[Paul Eggert]

On 03/09/2018 10:35 AM, Eli Zaretskii wrote:
> it bothers me a lot, to the degree that I don't think it could
> fly

I've tried it and it flies for me, and it doesn't appear that I'm alone 
in this opinion. Although it's not perfect, it's significantly better 
than what Emacs does now for variable-pitch fonts. So I'm in favor of 
having this feature available as a user option.

Re: Variable-width font indentation

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org, eggert@cs.ucla.edu
> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Fri, 9 Mar 2018 14:48:33 -0500
> 
> > IME, indentation of text in word processors I use does not depend on
> > previous lines at all.
> 
> Except in lists, right?

No, in lists as well.

> For example it's not uncommon to wrap like this:
> 
> * Some long text here
>   that wraps around:
> 
>   i.   Point 1
>        … continued
>   ii.  Point 2
>        … continued
>   iii. Point 3
>        … continued

What I meant was that if you, for example, enlarge the font, the
indentation doesn't get larger in pixels.  (I'm not sure we should
follow that extreme example, but it does drive the point home, I
think.)

Re: Variable-width font indentation

[Eli Zaretskii]

> Cc: cpitclaudel@gmail.com, emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Fri, 9 Mar 2018 12:16:54 -0800
> 
> On 03/09/2018 10:35 AM, Eli Zaretskii wrote:
> > it bothers me a lot, to the degree that I don't think it could
> > fly
> 
> I've tried it and it flies for me, and it doesn't appear that I'm alone 
> in this opinion.

I'm not alone in my opinion, either.

Re: Variable-width font indentation

[Paul Eggert]

Eli Zaretskii wrote:
>> I've tried it and it flies for me, and it doesn't appear that I'm alone
>> in this opinion.
> I'm not alone in my opinion, either.

I've heard other opinions that the approach is not to their taste. But I haven't 
heard anyone else say that the approach should not be available, even as a user 
option.

Re: What improvements would be truly useful?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Of course Emacs can also hide information (as text-properties, as
  > invisible text, as data stored in buffer-local variables, ...) but most
  > packages follow a design where as little info as possible is hidden.
  > Indeed, whenever I hide such information, I think it over many times
  > because I know there's a very strong chance that users won't like it.

That's a valid point, but WYSIWYG text processing is so desirable
that we shouldn't rule it out because of that point.

Perhaps Emacs WYSIWYG should have a mode to show the markup and a mode
to hide the markup, with a character to toggle between them.

Where and how to store the markup data is an independent question.
It could be in invisible characters, or in text properties.
Or some combination of the two.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: What improvements would be truly useful?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > No, it really isn't. A document is a user-centric feature -- it's a
  > piece of work that stores information in a way that it useful to the
  > user.

I disagree with you there.  If something can't be copied to various places
and looked at, I say it is not a document.  It may be some other kind of thing.

  > The self-contained file notion is a developer issue.

No, it is absolutely crucial to being able to use the document
in the ways we are accustomed to use documents.

  > Following your definition, MS Word becomes a document editor and MS Word
  > Online does not.

I did not state a definition of "document editor" -- it's not part of the point
I want to make -- so I think that is a misunderstanding.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: What improvements would be truly useful?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > If Emacs were able to turn org-mode into HTML, render that HTML and
  > then make changes to the text of that HTML, then you'd have something
  > close to the modal views.

I don't understand that sentence, but what I have in mind is something that
would look and work roughly like LibreOffice.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: What improvements would be truly useful?

[Karl Fogel]

Richard Stallman <rms@gnu.org> writes:
>Perhaps Emacs WYSIWYG should have a mode to show the markup and a mode
>to hide the markup, with a character to toggle between them.

For what it's worth, the venerable proprietary word processor WordPerfect had this feature (under the name "reveal codes").  The feature was widely used, too.  At least, people I knew who normally did not become experts in their software tools nevertheless still still knew about and used "reveal codes" on a regular basis.

Re: Variable-width font indentation

[Clément Pit-Claudel]

On 2018-03-09 16:18, Eli Zaretskii wrote:
>> Cc: emacs-devel@gnu.org, eggert@cs.ucla.edu
>> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
>> Date: Fri, 9 Mar 2018 14:48:33 -0500
>>
>>> IME, indentation of text in word processors I use does not depend on
>>> previous lines at all.
>>
>> Except in lists, right?
> 
> No, in lists as well.

I don't understand: in the example below, the widths of the bullets and the numbers (i, ii, …) affects the indentation of the following lines, right?

In any case, I agree it's a fairly limited issue.  I don't think it matters much.

>> For example it's not uncommon to wrap like this:
>>
>> * Some long text here
>>   that wraps around:
>>
>>   i.   Point 1
>>        … continued
>>   ii.  Point 2
>>        … continued
>>   iii. Point 3
>>        … continued
> 
> What I meant was that if you, for example, enlarge the font, the
> indentation doesn't get larger in pixels.  (I'm not sure we should
> follow that extreme example, but it does drive the point home, I
> think.)

Re: Variable-width font indentation

[James Cloos]

I've long thought that a good start is to show the initial whitespace in
fixed width.

That doesn't cover aligning things like the right column of *-boxes, but
is a good start and has low-complexity.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 0x997A9F17ED7DAEA6

Re: Variable-width font indentation

[Yuri Khan]

On Sat, Mar 10, 2018 at 2:48 AM, Clément Pit-Claudel
<cpitclaudel@gmail.com> wrote:

> Except in lists, right? For example it's not uncommon to wrap like this:
>
> * Some long text here
>   that wraps around:
>
>   i.   Point 1
>        … continued
>   ii.  Point 2
>        … continued
>   iii. Point 3
>        … continued

The way word processors implement this is:

1. Specify a left margin of, say, 0.5in. This makes each line start at
that offset.

2. Specify a first line indentation of, say, -0.25in. This way the
list marker starts at 0.25in, immediately followed by the rest of the
first line. Each subsequent line starts at the left margin 0.5in.

3. To make the item text of the first line align with the rest of the
paragraph, put a tabulation between the marker and the item text.
Align that tabulation at 0.5in.


The way HTML+CSS implement this is (roughly):

1. Specify a left margin for the whole list, say, 0.5in.

2. Treat list markers as special absolutely positioned elements. Put
them at, say, (-0.25in across, +0in down) from the start of list item.
(Absolutely positioned elements do not affect the layout of the
surrounding text, so list item text is aligned normally.)


In both cases, if the list marker becomes really long and no longer
fits its allotted space of 0.25in, there will be display artifacts. In
Word-alikes, the item text will start at the *next* tab stop. In
HTML+CSS, item text will visually overlap the marker.

Re: Variable-width font indentation

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Paul Eggert <eggert@cs.ucla.edu>
>
>> Programming in variable-width fonts makes more-efficient use of valuable 
>> screen real estate, so it's worth doing when it works reasonably well, 
>> as it often would if Emacs supported it better.
>
> Agreed.
>
> Do people think adding a per-buffer space-width variable would be a
> good step in this direction?  It should be easy to add, I think.

+1

-- 
-- Stephe

Re: Variable-width font indentation

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org, eggert@cs.ucla.edu
> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Fri, 9 Mar 2018 17:54:11 -0500
> 
> >>> IME, indentation of text in word processors I use does not depend on
> >>> previous lines at all.
> >>
> >> Except in lists, right?
> > 
> > No, in lists as well.
> 
> I don't understand: in the example below, the widths of the bullets and the numbers (i, ii, …) affects the indentation of the following lines, right?

No, it doesn't.  It surprised me to see that in MS Word, but the
evidence is clear.  Not sure we should follow suit, but it does tell
us how far this could be taken.

> In any case, I agree it's a fairly limited issue.  I don't think it matters much.

Agreed.

Re: Variable-width font alignment

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
>> Date: Fri, 9 Mar 2018 11:05:13 -0500
>> 
>> On 2018-03-09 03:34, Eli Zaretskii wrote:
>> > Do people think adding a per-buffer space-width variable would be a
>> > good step in this direction?  It should be easy to add, I think.
>> 
>> It does sound convenient. Will it apply only to leading whitespace,
>> or also to inter-word whitespace?
>
> I'd say it should apply everywhere.  It would be confusing to have SPC
> take a different number of pixels depending on where it is.

Right. Unless we need something else to fix the intra-line align
function - ie align '=>' in 

            Trace                    => Verbosity > 1,
            Put_Parse_Table          => Verbosity > 0,
            Ignore_Unused_Tokens     => Verbosity > 1,
            Ignore_Unknown_Conflicts => Verbosity > 1);

It seems to me that implementing this in a variable width font requires some
sort of align-region-local tab setting, which is a change to current align
algorithms (which just insert spaces).

I use this feature extensively in Ada code, so I would not switch to
variable width font without it.

One step in align.el `align-areas' computes an `align-column'. In
variable-pitch-mode, we could convert that into a tab setting by
computing the pixel column of that character column in the line with the
least whitespace/longest text ("Ignore_Unknown_Conflicts =>" above).

I could try doing this. What is the best approach to implementing an
align-region-local tab setting?

-- 
-- Stephe

Re: Variable-width font indentation

[Eli Zaretskii]

> From: James Cloos <cloos@jhcloos.com>
> Cc: Paul Eggert <eggert@cs.ucla.edu>,  cpitclaudel@gmail.com,  emacs-devel@gnu.org
> Date: Fri, 09 Mar 2018 19:09:23 -0500
> 
> I've long thought that a good start is to show the initial whitespace in
> fixed width.
> 
> That doesn't cover aligning things like the right column of *-boxes, but
> is a good start and has low-complexity.

I'm curious: why do you think this would be a good start?  AFAIU, it
will cause misalignment with any font/font size other than the one
where it fits the actual size of the font (assuming that font is
fixed-pitch), so how can it be a step in the right direction?  I think
initial whitespace should rather change its size to match the size of
the font---then it will produce nicely indented display with any font.

Or maybe I misunderstand what you mean by "fixed width".  You did mean
fixed width in pixels, yes?  If not, then what did you mean?

Re: Variable-width font indentation

[Eli Zaretskii]

> From: Yuri Khan <yuri.v.khan@gmail.com>
> Date: Sat, 10 Mar 2018 11:27:39 +0700
> Cc: Eli Zaretskii <eliz@gnu.org>, Paul Eggert <eggert@cs.ucla.edu>, 
> 	Emacs developers <emacs-devel@gnu.org>
> 
> In both cases, if the list marker becomes really long and no longer
> fits its allotted space of 0.25in, there will be display artifacts. In
> Word-alikes, the item text will start at the *next* tab stop.

My point was that the _subsequent_ lines still stay at the same 0.5"
offset, no matter what happens on the first line.  Which may or may
not make sense, but it's good to know what "prior art" is in this
case.

Re: Variable-width font alignment

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Date: Sat, 10 Mar 2018 02:12:57 -0600
> 
>             Trace                    => Verbosity > 1,
>             Put_Parse_Table          => Verbosity > 0,
>             Ignore_Unused_Tokens     => Verbosity > 1,
>             Ignore_Unknown_Conflicts => Verbosity > 1);
> 
> It seems to me that implementing this in a variable width font requires some
> sort of align-region-local tab setting, which is a change to current align
> algorithms (which just insert spaces).

Whatever we do to adapt indentation to variable-pitch fonts, we _must_
still begin by inserting spaces so that the same text looks aligned
with fixed-pitch fonts.  Otherwise, we will be unable to let other
people view our code/documents in other editors which use fixed-pitch
fonts, or even in Emacs with fixed-pitch fonts.

The features that align text when variable-pitch fonts are in use must
work _on_top_ of the "basic" alignment that just uses spaces and tabs
as we do today.

> One step in align.el `align-areas' computes an `align-column'. In
> variable-pitch-mode, we could convert that into a tab setting by
> computing the pixel column of that character column in the line with the
> least whitespace/longest text ("Ignore_Unknown_Conflicts =>" above).
> 
> I could try doing this. What is the best approach to implementing an
> align-region-local tab setting?

Any local change in whitespace width would need to use 'display' text
properties, such as '(space :width (N))'.  The problem with that is
that you cannot easily record this when you save the file, so these
properties will have to be recomputed anew each time the file is
visited.  Which will take us down the path of font-lock and JIT lock,
something that I'd like to avoid.  We could also record that in a
separate file, but I'm not sure such meta-data approach will be
acceptable.

Another idea would be to modify align-areas such that it attempts to
find an align-column that will align given text without changing the
width of space and tab, by looking for some kind of LCD for the widths
of the text strings involved in this.  Not sure if this is workable,
though.

Re: Variable-width font indentation

[Yuri Khan]

On Sat, Mar 10, 2018 at 3:39 PM, Eli Zaretskii <eliz@gnu.org> wrote:

>> In both cases, if the list marker becomes really long and no longer
>> fits its allotted space of 0.25in, there will be display artifacts. In
>> Word-alikes, the item text will start at the *next* tab stop.
>
> My point was that the _subsequent_ lines still stay at the same 0.5"
> offset, no matter what happens on the first line.  Which may or may
> not make sense, but it's good to know what "prior art" is in this
> case.

Yes. I’m actually confirming your claim that indentation of subsequent
lines is not affected by the width of the marker.

Re: Variable-width font indentation

[Clément Pit-Claudel]

On 2018-03-09 23:27, Yuri Khan wrote:
> 1. Specify a left margin of, say, 0.5in. This makes each line start at
> that offset.

Of course; I didn't realize we were talking about teaching Emacs to do this kind of advanced positioning (I thought we were still discussing pure text documents)

Re: Variable-width font indentation

[Eli Zaretskii]

> From: Yuri Khan <yuri.v.khan@gmail.com>
> Date: Sat, 10 Mar 2018 18:42:57 +0700
> Cc: Clément Pit-Claudel <cpitclaudel@gmail.com>, 
> 	Paul Eggert <eggert@cs.ucla.edu>, Emacs developers <emacs-devel@gnu.org>
> 
> > My point was that the _subsequent_ lines still stay at the same 0.5"
> > offset, no matter what happens on the first line.  Which may or may
> > not make sense, but it's good to know what "prior art" is in this
> > case.
> 
> Yes. I’m actually confirming your claim that indentation of subsequent
> lines is not affected by the width of the marker.

Ah, okay.  Thanks.

Re: What improvements would be truly useful?

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: phillip.lord@russet.org.uk (Phillip Lord)
>> Cc: Marcin Borkowski <mbork@mbork.pl>, eggert@cs.ucla.edu,
>> rms@gnu.org, emacs-devel@gnu.org, raeburn@raeburn.org,
>> dancol@dancol.org, rostislav.svoboda@gmail.com
>> Date: Fri, 09 Mar 2018 10:56:00 +0000
>> 
>> > Are you saying that either of these is a WYSIWYG word processor?
>> 
>> No. But, then Word or LibreOffice are not WYSIWYG either.
>
> This is just a minor semantic issue: let's assume for the purposes of
> the current discussion that the definition of a WYSIWYG word processor
> is what the Office word processors do.


I think it is not. There are many different forms of word-processor out
there. Emacs is already one of them, albeit for a specialist user base.


>> Org-mode is essentially a structure first view of text. AUCTeX with
>> preview-latex is roughly the same thing. So, both of them have strong
>> aspects of modern word-processor.
>
> They both provide the final view of the document only off-line, and
> that is the crucial difference, for the purposes of this discussion.

Yes, this is most true, although the idea of the "final view" is not
quite as clear as it might be. Even for latex, the end product can be
more than one thing (PDF and HTML for example). For org-mode, whether
the text view is less final than an HTML rendering is an open question.
 

>> If Emacs were able to turn org-mode into HTML, render that HTML and
>> then make changes to the text of that HTML, then you'd have something
>> close to the modal views. Or something like preview-latex which uses
>> syntax highlighting to provide the WYS aspect but allows you to turn it
>> on and off.
>
> Emacs is capable of displaying text with different typefaces, so it
> isn't clear to me why would we need to go through a translator, such
> as Org export or LaTeX.

Oh, because of your notion of WYS. If the "final document" is supposed
to be in HTML, then having something that looks the similar, but is
never ideal.

>> How easy this would be to implement, I do not know. Emacs does not have
>> an MVC architecture -- text properties (i.e. visualization) are stored
>> directly with the contents of a buffer, so switching views is clunky or
>> difficult (see my own lentic.el for an example).
>
> Actually, MVC is exactly the Emacs architecture.  (And text properties
> are stored separately from the buffer contents.)  I wonder what kind
> of misunderstanding is at work here.

My understanding of, for indirect buffers, are that they share text and
text properties. So you cannot put a different visualisation over two
indirect buffers, if those visualisations use text properties, because
they will interfere with each other. Perhaps it would be possible to
recode indirect buffers so that this were not the case, so that only
text were shared.

Similarly, the text that you seen on screen is pretty much what is in
the buffer. You can do things like put before or after strings in text
properties. But then this will be ignored by everything other than the
visualisation -- you can't search for them, for instance, because the
user level functions operate over the text not the visualisation of it.

And, finally, while Emacs does provide notifications of change to buffer
text both before and after, they are neither guaranteed to be paired nor
consistent. Nor is it apparent to the listening function when they will
not be paired or consistent; which is unfortunate as mostly they are.

Of course, this is not to say there is no abillity to visualise the same
thing twice. You can put a buffer into two windows, with independent
mark and point, selection and so forth. But the existence of these
things is ephemeral. So, for example, when viewing a single buffer,
split the window. Now move point in one window, and then unsplit the
window. The location of mark and point in the second window has now been
lost.

So Emacs does not exactly have an MVC architecture -- it sort of has it.
I do not think, for example, it would be straight-forward to produce
something like preview-latex where you could see the inline images and
equations and latex representation simultaneously and be able to edit
either.


Phil

Re: Variable-width font alignment

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stephen Leake <stephen_leake@stephe-leake.org>
>> Date: Sat, 10 Mar 2018 02:12:57 -0600
>> 
>>             Trace                    => Verbosity > 1,
>>             Put_Parse_Table          => Verbosity > 0,
>>             Ignore_Unused_Tokens     => Verbosity > 1,
>>             Ignore_Unknown_Conflicts => Verbosity > 1);
>> 
>> It seems to me that implementing this in a variable width font requires some
>> sort of align-region-local tab setting, which is a change to current align
>> algorithms (which just insert spaces).
>
> Whatever we do to adapt indentation to variable-pitch fonts, we _must_
> still begin by inserting spaces so that the same text looks aligned
> with fixed-pitch fonts.  Otherwise, we will be unable to let other
> people view our code/documents in other editors which use fixed-pitch
> fonts, or even in Emacs with fixed-pitch fonts.
>
> The features that align text when variable-pitch fonts are in use must
> work _on_top_ of the "basic" alignment that just uses spaces and tabs
> as we do today.

Right.

>> One step in align.el `align-areas' computes an `align-column'. In
>> variable-pitch-mode, we could convert that into a tab setting by
>> computing the pixel column of that character column in the line with the
>> least whitespace/longest text ("Ignore_Unknown_Conflicts =>" above).
>> 
>> I could try doing this. What is the best approach to implementing an
>> align-region-local tab setting?
>
> Any local change in whitespace width would need to use 'display' text
> properties, such as '(space :width (N))'.  

Ok; that doesn't need an explicit tab.

> The problem with that is that you cannot easily record this when you
> save the file, so these properties will have to be recomputed anew
> each time the file is visited. 

Right. For example, that happens now in info files; formatting is
applied to the links via text properties.

> Which will take us down the path of font-lock and JIT lock, something
> that I'd like to avoid. 

That's how align.el works now. We could modify it to only set the text
properties if the fixed-width spaces are correct, or when in a read-only
buffer, or in variable-pitch-mode.

On the other hand, the ada-mode currently in GELPA computes indentation
and faces from a parse of the source text. It works very well, as long
as the source is syntactically correct. I'm working on implementing
error recovery in the parser; it works fairly well at the moment, and
I'm still improving it.

It should be possible to compute alignments from the parse as well; that
would elminate some of the minor bugs in current ada-mode alignment. That
would still require a parse when visiting the file.

> We could also record that in a separate file, but I'm not sure such
> meta-data approach will be acceptable.

Computing the align properties from the source has to be fast enough to
be acceptable while editing (as align.el is now), so it will be
acceptable on visiting.

-- 
-- Stephe

Re: What improvements would be truly useful?

[Eli Zaretskii]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Cc: mbork@mbork.pl,  eggert@cs.ucla.edu,  rms@gnu.org,  emacs-devel@gnu.org,  raeburn@raeburn.org,  dancol@dancol.org,  rostislav.svoboda@gmail.com
> Date: Sat, 10 Mar 2018 16:02:14 +0000
> 
> >> No. But, then Word or LibreOffice are not WYSIWYG either.
> >
> > This is just a minor semantic issue: let's assume for the purposes of
> > the current discussion that the definition of a WYSIWYG word processor
> > is what the Office word processors do.
> 
> I think it is not.

Like I said, it's a matter of definition.  And I just provided mine
(and I think also Richard's).

> My understanding of, for indirect buffers, are that they share text and
> text properties. So you cannot put a different visualisation over two
> indirect buffers, if those visualisations use text properties, because
> they will interfere with each other.

It is not possible with text properties, but it's possible with
overlays, which can be window-specific.

> Similarly, the text that you seen on screen is pretty much what is in
> the buffer. You can do things like put before or after strings in text
> properties. But then this will be ignored by everything other than the
> visualisation -- you can't search for them, for instance, because the
> user level functions operate over the text not the visualisation of it.

Since we mostly are talking indentations and typefaces, I don't
understand why would we need to use before- or after-strings in this
context.

> And, finally, while Emacs does provide notifications of change to buffer
> text both before and after, they are neither guaranteed to be paired nor
> consistent. Nor is it apparent to the listening function when they will
> not be paired or consistent; which is unfortunate as mostly they are.

Why is that part of the issue?

> So Emacs does not exactly have an MVC architecture -- it sort of has it.

I think your notion of MVC is too specialized.

Re: Variable-width font alignment

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Date: Sat, 10 Mar 2018 11:30:53 -0600
> 
> Computing the align properties from the source has to be fast enough to
> be acceptable while editing (as align.el is now), so it will be
> acceptable on visiting.

Yes, if this is feasible.  Will it be fast enough to handle large
sources files, such as xdisp.c, for example?

Re: Variable-width font indentation

[James Cloos]

>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:

EZ> Or maybe I misunderstand what you mean by "fixed width".  You did mean
EZ> fixed width in pixels, yes?  If not, then what did you mean?

Yes.  By fixed-width I mean the font the buffer would use were variable-
pitch-mode not in effect.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 0x997A9F17ED7DAEA6

Re: Variable-width font indentation

[Eli Zaretskii]

> From: James Cloos <cloos@jhcloos.com>
> Cc: eggert@cs.ucla.edu,  cpitclaudel@gmail.com,  emacs-devel@gnu.org
> Date: Sat, 10 Mar 2018 14:48:33 -0500
> 
> >>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:
> 
> EZ> Or maybe I misunderstand what you mean by "fixed width".  You did mean
> EZ> fixed width in pixels, yes?  If not, then what did you mean?
> 
> Yes.  By fixed-width I mean the font the buffer would use were variable-
> pitch-mode not in effect.

Then I really don't understand why you think it's a good start.  I
think it's a step in the wrong direction, because it will make things
worse than they are now: what we have now makes sure the initial
whitespace changes in sync on all lines of the buffer, so that the
first non-whitespace characters on each line stay aligned.

Re: What improvements would be truly useful?

[Phillip Lord]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > No, it really isn't. A document is a user-centric feature -- it's a
>   > piece of work that stores information in a way that it useful to the
>   > user.
>
> I disagree with you there.  If something can't be copied to various
> places and looked at, I say it is not a document.  It may be some
> other kind of thing.
>
>   > The self-contained file notion is a developer issue.
>
> No, it is absolutely crucial to being able to use the document
> in the ways we are accustomed to use documents.

I have to disagree with this. I mean, it was true 10 years ago, but it
is not now.


>   > Following your definition, MS Word becomes a document editor and MS Word
>   > Online does not.
>
> I did not state a definition of "document editor" -- it's not part of the point
> I want to make -- so I think that is a misunderstanding.

No, I think not. Consider this scenario. I start editing some text on
google docs with a collaborator; after a while we export the doc into
word, share this with Dropbox, then eventually one person takes it over.
Is this text really not a document till we download it?

All of which gets back what I think was the point of the OP; Emacs would
be a better tool if it supported the web. We've been having intense
discussions about widget toolkits; what about a web-delivered Emacs? The
Web and Mobile together make up the majority of computing these days.

http://www.ymacs.org

If you want a good example.

Phil

Re: Variable-width font indentation

[James Cloos]

>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:

EZ> Then I really don't understand why you think it's a good start.  I
EZ> think it's a step in the wrong direction, because it will make things
EZ> worse than they are now: what we have now makes sure the initial
EZ> whitespace changes in sync on all lines of the buffer, so that the
EZ> first non-whitespace characters on each line stay aligned.

I have not re-compiled in a while, nor did I read the whole thread.
Perhaps things have progressed since the version I currently have.

Consistent and not-too-small indenting is important.  Basing indents on
the width of the fixed pitch space and tab stops on a multiple of that
would help, compared to what the version I have does.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 0x997A9F17ED7DAEA6

Re: Variable-width font indentation

[Eli Zaretskii]

> From: James Cloos <cloos@jhcloos.com>
> Cc: eggert@cs.ucla.edu,  cpitclaudel@gmail.com,  emacs-devel@gnu.org
> Date: Sat, 10 Mar 2018 17:18:12 -0500
> 
> >>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:
> 
> EZ> Then I really don't understand why you think it's a good start.  I
> EZ> think it's a step in the wrong direction, because it will make things
> EZ> worse than they are now: what we have now makes sure the initial
> EZ> whitespace changes in sync on all lines of the buffer, so that the
> EZ> first non-whitespace characters on each line stay aligned.
> 
> I have not re-compiled in a while, nor did I read the whole thread.
> Perhaps things have progressed since the version I currently have.
> 
> Consistent and not-too-small indenting is important.  Basing indents on
> the width of the fixed pitch space and tab stops on a multiple of that
> would help, compared to what the version I have does.

I don't think anything's happened in this regard lately.  A tab was
always rendered on GUI frames as a stretch whose width is an integral
multiple of the space width of the font.  Except in the Windows port,
the space width is actually the width of the font glyph that displays
the space character.  So I'm not sure I understand what problems you
describe.

Can you provide a concrete example where the behavior you describe
produces bad results?

Re: What improvements would be truly useful?

[Marcin Borkowski]

On 2018-03-09, at 05:43, Stefan Monnier <monnier@IRO.UMontreal.CA> wrote:

>> Interesting.  I have a very similar sentiment, though I think it is
>> quite irrational.
>> Is mu4e, Org-mode, Calc, or dunnet also "not Emacs"?  If not, how is it
>> different than a word processor?
>
> I don't think I can clearly pin-point the problem, but I think the issue
> has to do with hiding information: WYSIWYG hides information, in the
> sense that there are many different documents with the same visual
> appearance, so the user needs to *guess* what is the underlying
> document's structure in order to know what his actions will really do.
> A typical example is the beginning/end of a bold text which is only
> implicitly represented by the fact that some text is bold and other
> isn't, but that doesn't let you easily tell whether a new word inserted
> between a bold work and a non-bold word will be bold or not.
>
> Of course Emacs can also hide information (as text-properties, as
> invisible text, as data stored in buffer-local variables, ...) but most
> packages follow a design where as little info as possible is hidden.
> Indeed, whenever I hide such information, I think it over many times
> because I know there's a very strong chance that users won't like it.
>
> Also when it comes to data that we edit, hiding information either means
> hiding it as invisible text (which tends to be fiddly, so it's used
> fairly sparingly), or it implies using a complex procedure to `find` and
> to `save` the buffer in order to translate between the file contents and
> the in-buffer contents.  This complexity is a strong encouragement not
> to go down that route.

Thanks, Stefan - this is a very good point.  I guess you managed to nail
it.

Best,

--
Marcin Borkowski
http://mbork.pl

Re: What improvements would be truly useful?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > No, I think not. Consider this scenario. I start editing some text on
  > google docs with a collaborator; after a while we export the doc into
  > word, share this with Dropbox, then eventually one person takes it over.
  > Is this text really not a document till we download it?

With all due respect, I think you're disputing a misunderstanding.

As far as I know, what people edit with Google Docs is a document.  It
can be copied to a computer (though doing so requires running nonfree
software) and viewed or edited there.

Google Docs is a service for storing documents that is designed for
editing them using nonfree software.

  > All of which gets back what I think was the point of the OP; Emacs would
  > be a better tool if it supported the web. We've been having intense
  > discussions about widget toolkits; what about a web-delivered Emacs?

What exactly would that mean?  Could you describe the scenario by giving
names or letters to each of the computers involved, and saying what job each
computer does?

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: What improvements would be truly useful?

[Dmitry Gutov]

On 3/8/18 1:44 PM, Toon Claes wrote:
> IMHO, one thing that I would find interesting is proper support
> for multiple major modes.
> 
> I know there has been some talking about this in the past:
> 
> https://lists.gnu.org/archive/html/emacs-devel/2016-04/msg00585.html
> 
> But I'm not sure what the outcome was of that very long thread.

That one reached a dead end, for now.

> I only know later on some improvements were made to have html, css, and
> js modes working together:
> 
> https://lists.gnu.org/archive/html/emacs-devel/2017-01/msg00775.html
> 
> But still having great multiple major mode support--with islands having
> their own major and minor modes--would make Emacs a more modern editor.

mmm-mode (a very old package providing features like that) is soon going 
to be in ELPA. If anyone on this mailing list wanted to contribute, but 
stopped because it wasn't a part of Emacs or ELPA, that will be their 
chance.

Continuous image scrolling (was: What improvements would be truly useful?)

[Juri Linkov]

>>> What would improve my user experience would be improvements to the
>>> display engine that will allow a better document viewing capabilities
>>> in something like pdf-tools (like continuous scrolling).
>>
>> Continuous scrolling can be easily implemented with a simple hack
>> to insert adjacent images for the previous/next pdf pages in
>> the same DocView buffer.
>
> I've looked into implementing this feature and it is not as simple as it
> seems. The hack you mention is precisely that: a hack, and a very cranky
> one. AFAIK there is no way of implementing continous scrolling with
> adjacent images if you care at all about robustness and user experience.
> And the fact that people is asking for this feature for years and, to
> this day, nobody implemented it, seems to confirm my impressions.

I just realized we already have continuous image scrolling mode:

1. visiting a PDF file generates PNG images in the temporary directory
2. C-x d /tmp/docview1000/... RET
3. M-x iimage-mode RET
4. enjoy continuous image scrolling mode directly in dired
   (the same logic could be later incorporated into doc-view.el)

I tried this with a directory containing ca 1000 pages/images
and there is no performance degradation while continuously scrolling
all these 1000 images.

Re: Continuous image scrolling

[Óscar Fuentes]

Juri Linkov <juri@linkov.net> writes:

>>>> What would improve my user experience would be improvements to the
>>>> display engine that will allow a better document viewing capabilities
>>>> in something like pdf-tools (like continuous scrolling).
>>>
>>> Continuous scrolling can be easily implemented with a simple hack
>>> to insert adjacent images for the previous/next pdf pages in
>>> the same DocView buffer.
>>
>> I've looked into implementing this feature and it is not as simple as it
>> seems. The hack you mention is precisely that: a hack, and a very cranky
>> one. AFAIK there is no way of implementing continous scrolling with
>> adjacent images if you care at all about robustness and user experience.
>> And the fact that people is asking for this feature for years and, to
>> this day, nobody implemented it, seems to confirm my impressions.
>
> I just realized we already have continuous image scrolling mode:
>
> 1. visiting a PDF file generates PNG images in the temporary directory
> 2. C-x d /tmp/docview1000/... RET
> 3. M-x iimage-mode RET
> 4. enjoy continuous image scrolling mode directly in dired
>    (the same logic could be later incorporated into doc-view.el)
>
> I tried this with a directory containing ca 1000 pages/images
> and there is no performance degradation while continuously scrolling
> all these 1000 images.

PDF-Tools (which is vastly superior to the built-in PDF viewer) does not
work this way. It generates pages on demand. It is an absurd waste of
resources to visit a book and generate one png file for each page.
PDF-Tools solves this and does much more.

So, if the solution depends on having pre-generated images, it is worse
than not having continuous scroll, IMAO.

OTOH, I tried what you suggest on a directory with lots of images. The
result is quite crude. For example, it seems that the curso height
corresponds to the image height, i.e. it is not possible to finely
scroll the sequence of images up and down, you go one image at the time.

The problems I faced while trying to implement continous document scroll
for PDF-Tools revolved around making the scroll actually continous: if
you scroll down from page P-1 to page P, remove page P-2 and append page
P+1 without producing visible jumps.

Re: Continuous image scrolling

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Tue, 03 Apr 2018 03:59:23 +0200
> 
> The problems I faced while trying to implement continous document scroll
> for PDF-Tools revolved around making the scroll actually continous: if
> you scroll down from page P-1 to page P, remove page P-2 and append page
> P+1 without producing visible jumps.

Sounds like some simple enough changes to the logic of
line-move-visual and its subroutines.  Patches are welcome.

Re: Continuous image scrolling

[Juri Linkov]

>> I just realized we already have continuous image scrolling mode:
>>
>> 1. visiting a PDF file generates PNG images in the temporary directory
>> 2. C-x d /tmp/docview1000/... RET
>> 3. M-x iimage-mode RET
>> 4. enjoy continuous image scrolling mode directly in dired
>>    (the same logic could be later incorporated into doc-view.el)
>>
>> I tried this with a directory containing ca 1000 pages/images
>> and there is no performance degradation while continuously scrolling
>> all these 1000 images.
>
> PDF-Tools (which is vastly superior to the built-in PDF viewer) does not
> work this way. It generates pages on demand. It is an absurd waste of
> resources to visit a book and generate one png file for each page.
> PDF-Tools solves this and does much more.
>
> So, if the solution depends on having pre-generated images, it is worse
> than not having continuous scroll, IMAO.

This is an optimization, but a significant one, indeed,
if you are not going to read the whole PDF at once.

> OTOH, I tried what you suggest on a directory with lots of images. The
> result is quite crude. For example, it seems that the curso height
> corresponds to the image height,

It's possible to hide the cursor like we do in image-mode.

> i.e. it is not possible to finely scroll the sequence of images up and
> down, you go one image at the time.

It's because next-line goes to the next file in Dired by default,
this is why I disabled this inconvenience in my ~/.emacs by using
normal cursor motion keybindings:

  (define-key dired-mode-map [remap next-line] nil)
  (define-key dired-mode-map [remap previous-line] nil)

> The problems I faced while trying to implement continous document scroll
> for PDF-Tools revolved around making the scroll actually continous: if
> you scroll down from page P-1 to page P, remove page P-2

I see no need to remove page P-2.  For optimization it's better to keep it,
so that when the user returns to that page, it's already here.

> and append page P+1 without producing visible jumps.

Based on the simplistic approach like used by iimage-mode,
it should be easy to insert placeholders for all pages,
and use font-lock machinery to replace placeholders with
generated images when the user navigates to other parts
of the PDF buffer.

RE: Continuous image scrolling

[Drew Adams]

> use font-lock machinery to replace placeholders with
> generated images when the user navigates to other parts
> of the PDF buffer.

Interesting idea!  But would that be responsive, for
the user?  Would the images be generated on the fly?
Would they be automatically deleted and garbaged
afterward?

Re: Continuous image scrolling

[Stefan Monnier]

> I see no need to remove page P-2.  For optimization it's better to keep it,
> so that when the user returns to that page, it's already here.

Reminds me: I've had Emacs crash on me when flipping though large PDF
documents with doc-view.  I haven't been able to reproduce it, but
I have the impression that we have a problem there somewhere.
[ It's not directly related to the above in the sense that doc-view
  only displays one page at any given time anyway.  ]


        Stefan

Re: Continuous image scrolling

[Clément Pit-Claudel]

On 2018-04-03 15:59, Juri Linkov wrote:
>> i.e. it is not possible to finely scroll the sequence of images up and
>> down, you go one image at the time.

> It's because next-line goes to the next file in Dired by default,
> this is why I disabled this inconvenience in my ~/.emacs by using
> normal cursor motion keybindings:

Maybe I'm misunderstanding the original request, but if not then I don't think this is the issue.

AFAIU, one thing that is not too easy to implement today in ELisp is continuous ("smooth") scrolling over multiple images, as shown in e.g. evince: if I drag the scroll bars or press the up and down keys, the current page and the following one move up and down by a bit.

We usually call this pixel-scroll in Emacs, and we have an implementation of it based on set-window-vscroll in lisp/pixel-scroll.el.  This is also what Docview and PDFTools use to scroll images that don't fit in the selected window.  But I don't think that trick works for sequences of images, at least not straight out of the box.

The main issue, IIUC, is that images occupy one (tall) line, and Emacs tries to always keep the line containing the point in full view, which means that set-window-vscroll does not always work if the point is on a line containing an image.

You can see what I mean concretely with the following example. Open a new buffer, and run (insert (propertize " " 'display '(image :type jpeg :file "/path/to/some/large/picture.jpg" :scale 1))).  Then insert 5 lines of "A".  Your buffer will look like this:

|----------------|
|  Picture here  |
|                |
|----------------|
A
A
A
A

Put the point on the third A and run (set-window-vscroll (selected-window) 8 t); note how the images moves a bit up.  That's the kind of scrolling we want, which is encouraging :)

But now put the point on the picture's line, and try the same experiment; notice how the picture doesn't move, because we try to keep the line in full view. (if the image is larger than the window, it does work, and that's how docview scrolls single pages. I don't know if there's a way to get that behavior when the image fits in the window).  That behavior becomes an issue when you have sequences of images, because you can't leave the point on the line of the first image, but moving the point to the line of the second image scrolls it entirely into view.

The two techniques to work around this that I'm aware of are (1) cutting the images into thin horizontal strips (and showing one strip per line), and (2) inserting a thin text line between each image on which to place the point while scrolling.  I don't know if either of these would work for pdftools.  Hopefully there's a third, simpler technique to achieve this?

Clément.

Re: Continuous image scrolling

[Eli Zaretskii]

> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Tue, 3 Apr 2018 17:57:35 -0400
> 
> AFAIU, one thing that is not too easy to implement today in ELisp is continuous ("smooth") scrolling over multiple images, as shown in e.g. evince: if I drag the scroll bars or press the up and down keys, the current page and the following one move up and down by a bit.

It could/should be easy to implement in Lisp because the current
behavior is implemented in Lisp.  At least, I won't believe assertions
such as the above until Someone™ digs into the code in
line-move-visual and its subroutines, where the current behavior is
implemented, and presents clear evidence that some code or feature
coded in C gets in the way.

> The main issue, IIUC, is that images occupy one (tall) line, and Emacs tries to always keep the line containing the point in full view, which means that set-window-vscroll does not always work if the point is on a line containing an image.

Yes, but most, if not all, of the logic supporting this is in
simple.el.

> But now put the point on the picture's line, and try the same experiment; notice how the picture doesn't move, because we try to keep the line in full view. (if the image is larger than the window, it does work, and that's how docview scrolls single pages. I don't know if there's a way to get that behavior when the image fits in the window).  That behavior becomes an issue when you have sequences of images, because you can't leave the point on the line of the first image, but moving the point to the line of the second image scrolls it entirely into view.
> 
> The two techniques to work around this that I'm aware of are (1) cutting the images into thin horizontal strips (and showing one strip per line), and (2) inserting a thin text line between each image on which to place the point while scrolling.  I don't know if either of these would work for pdftools.  Hopefully there's a third, simpler technique to achieve this?

It makes little sense to me to work around the behavior we ourselves
implemented.  Instead, I invite volunteers to take a good look at the
code and try changing it to lift the limitations that currently get in
the way.

The current code behaves like it does because it was _programmed_ to
behave that way.  It explicitly uses window-vscroll only when the
current line cannot be fully shown in the window for some reason.  But
if you look at the relevant code, you will see that this condition is
programmed there, and is specifically tested to determine when to use
window-vscroll and when to scroll to the next/previous screen line.
So by changing that logic, perhaps controlled by some defcustom, it
should be possible to get what DocView and PDF-tools want to
accomplish.

Let me remind you that similar misconceptions were around regarding
pixel-level scrolling of text, until Tak came up with his package that
did just that, entirely in Lisp.

I hope someone will step forward and do the necessary research of what
is needed, because my gut feeling is that the features sought out here
are within our reach.  (I can help with advice, because I'm
responsible for at least some of the code in that area.)

Any volunteers?

Re: Continuous image scrolling

[John Wiegley]

>>>>> "ÓF" == Óscar Fuentes <ofv@wanadoo.es> writes:

ÓF> PDF-Tools (which is vastly superior to the built-in PDF viewer) does not
ÓF> work this way. It generates pages on demand. It is an absurd waste of
ÓF> resources to visit a book and generate one png file for each page.
ÓF> PDF-Tools solves this and does much more.

It really is quite good, I use it as my main PDF reader on the Mac now.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Continuous image scrolling

[Juri Linkov]

> But now put the point on the picture's line, and try the same experiment;
> notice how the picture doesn't move, because we try to keep the line in
> full view. (if the image is larger than the window, it does work, and
> that's how docview scrolls single pages. I don't know if there's a way to
> get that behavior when the image fits in the window).

I tried only with images larger than the window height where multi-image
scrolling works fine.  But now I see that scrolling smaller images is not
so smooth.  I don't know if such narrow images are common in PDF documents.

Re: Continuous image scrolling

[Juri Linkov]

>> use font-lock machinery to replace placeholders with
>> generated images when the user navigates to other parts
>> of the PDF buffer.
>
> Interesting idea!  But would that be responsive, for
> the user?  Would the images be generated on the fly?
> Would they be automatically deleted and garbaged
> afterward?

I think it will be no less responsive than other PDF readers
that also generate images on the fly.  But the main problem is
how to pre-render these placeholders to avoid visible jumps on
inserting new images.  Maybe we need to scan the whole PDF document
and get dimensions of all pages, then use them for placeholders.

Re: Continuous image scrolling

[Tak Kunihiro]

I think with the following setup,

M-x pixel-scroll-mode
(setq pixel-resolution-fine-flag t)

a buffer with large images, as you shown, can be scrolled by height of typical line.

Re: Continuous image scrolling

[Clément Pit-Claudel]

On 2018-04-04 19:31, Tak Kunihiro wrote:
> I think with the following setup,
> 
> M-x pixel-scroll-mode
> (setq pixel-resolution-fine-flag t)
> 
> a buffer with large images, as you shown, can be scrolled by height of typical line.
I think I must be doing something wrong :/

I used this snippet to insert 10 copies of a large picture in a new buffer:

   (dotimes (_ 10) 
      (insert (propertize " " 'display '(image :type jpeg :file "some-large-picture.jpg" :scale 1)) "\n"))

and then I turned pixel-scroll-mode on with the setting you mentioned, but Emacs still scrolls by full images.

Clément.

Re: Continuous image scrolling

[Tak Kunihiro]

>> I think with the following setup,
>>
>> M-x pixel-scroll-mode
>> (setq pixel-resolution-fine-flag t)
>>
>> a buffer with large images, as you shown, can be scrolled by height of typical line.
>
> I think I must be doing something wrong :/
> 
> I used this snippet to insert 10 copies of a large picture in a new buffer:
> 
>    (dotimes (_ 10)
>       (insert (propertize " " 'display '(image :type jpeg :file "some-large-picture.jpg" :scale 1)) "\n"))
> 
> and then I turned pixel-scroll-mode on with the setting you mentioned, but Emacs still scrolls by full images.

Let me assume height of an image is less than height of visible area of
a window.

M-x pixel-scroll-mode
(setq pixel-resolution-fine-flag t)
M-: (pixel-scroll-up 1)

The (pixel-scroll-up 1) scrolls a buffer up by (frame-char-height) with
re-displaying each pixel.

A function that scrolls a buffer up like (pixel-scroll-up 1) but without
re-displaying each pixel, should be written.  The function serves as
alternative of (scroll-up).

Re: Continuous image scrolling

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Date: Wed, 04 Apr 2018 22:50:43 +0300
> Cc: emacs-devel@gnu.org
> 
> I tried only with images larger than the window height where multi-image
> scrolling works fine.  But now I see that scrolling smaller images is not
> so smooth.  I don't know if such narrow images are common in PDF documents.

With today's large monitors and user tendency to maximize the frame,
it's quite possible, I think.

Re: Continuous image scrolling

[Clément Pit-Claudel]

On 2018-04-04 23:54, Tak Kunihiro wrote:
>>> I think with the following setup,
>>>
>>> M-x pixel-scroll-mode
>>> (setq pixel-resolution-fine-flag t)
>>>
>>> a buffer with large images, as you shown, can be scrolled by height of typical line.
>>
>> I think I must be doing something wrong :/
>>
>> I used this snippet to insert 10 copies of a large picture in a new buffer:
>>
>>    (dotimes (_ 10)
>>       (insert (propertize " " 'display '(image :type jpeg :file "some-large-picture.jpg" :scale 1)) "\n"))
>>
>> and then I turned pixel-scroll-mode on with the setting you mentioned, but Emacs still scrolls by full images.
> 
> Let me assume height of an image is less than height of visible area of
> a window.
> 
> M-x pixel-scroll-mode
> (setq pixel-resolution-fine-flag t)
> M-: (pixel-scroll-up 1)
> 
> The (pixel-scroll-up 1) scrolls a buffer up by (frame-char-height) with
> re-displaying each pixel.
> 
> A function that scrolls a buffer up like (pixel-scroll-up 1) but without
> re-displaying each pixel, should be written.  The function serves as
> alternative of (scroll-up).

I don't understand. I followed your instructions, but M-: (pixel-scroll-up 1) scrolls up by a full image.

Re: Continuous image scrolling

[Tak Kunihiro]

Can you try following on 26.0.91 or later?

$ emacs -Q
M-x about-emacs
M-x pixel-scroll-mode
M-: (setq pixel-resolution-fine-flag t)
M-: (pixel-scroll-up 1)

Re: Continuous image scrolling

[Clément Pit-Claudel]

On 2018-04-05 18:59, Tak Kunihiro wrote:
> Can you try following on 26.0.91 or later?
> 
> $ emacs -Q
> M-x about-emacs
> M-x pixel-scroll-mode
> M-: (setq pixel-resolution-fine-flag t)
> M-: (pixel-scroll-up 1)

That works fine.  But about-emacs doesn't show a buffer with multiple large images in a row, right?

Re: Continuous image scrolling

[Tak Kunihiro]

I found following only works when height of an image is less than half
of height of a window.

(progn
  (pixel-scroll-mode 1)
  (setq pixel-resolution-fine-flag t)
  (dotimes (_ 10)
    (insert (propertize " " 'display '(image :type jpeg :file "some-large-picture.jpg" :scale 1)) "\n"))
  (dotimes (_ 100)
    (pixel-scroll-down 1)
    (sit-for 0.3)))