RTL lines

RTL lines

[Alexandre Garreau]

Is there a var to make any line whose first either rtl or ltr char is RTL 
be displayed RTL (so for instance an org-mode definition list (with ::) 
would be displayed RTL with the bullets at right, if the defining terms (at 
the beginning, just after the */+/-/etc.) are in arabic/hebrew/etc.)?

Another idea I have is is there a way to do that per-paragraph (for 
instance for a defun that would use a such language)?

and arbitrarily (for direction-agnostic languages such as some extreme-
oriental ones)?

and does emacs support TTB/BTT (top to bottom, bottom to top), because I 
feel that would be a better use of human peripheral vision and physical 
ergonomical limit of line-length?

Re: RTL lines

[Eli Zaretskii]

> From: Alexandre Garreau <galex-713@galex-713.eu>
> Date: Mon, 18 Oct 2021 14:40:42 +0200
> 
> Is there a var to make any line whose first either rtl or ltr char is RTL 
> be displayed RTL (so for instance an org-mode definition list (with ::) 
> would be displayed RTL with the bullets at right, if the defining terms (at 
> the beginning, just after the */+/-/etc.) are in arabic/hebrew/etc.)?

Set bidi-paragraph-direction to the nil value.  It's a per-buffer
variable.

> Another idea I have is is there a way to do that per-paragraph (for 
> instance for a defun that would use a such language)?

The above will do that per paragraph.  That is, each paragraph will be
LTR or RTL depending on the first "strong directional" character in
that paragraph.

(This all is supposed to be well documented in the Emacs manual.
Hint, hint...)

> and arbitrarily (for direction-agnostic languages such as some extreme-
> oriental ones)?

I don't think I understand what kind of "arbitrariness" you have in
mind here.

> and does emacs support TTB/BTT (top to bottom, bottom to top)

No.  if you want to submit patches to that effect,l they will be most
welcome, though.

Verticality and future of display engine and lines (bis) [Was: Re: RTL lines]

[Alexandre Garreau]

I just coincidentally ended on a webpage with a vertical script (which 
surprisingly to me displayed correctly under firefox):

https://fr.wikipedia.org/wiki/Sakhaline

However copypasting that into emacs, or viewing the page with Eww makes 
that text incorrectly LTR.  Hence I suppose emacs doesn’t yet support 
viewing and editing TTB/BBT text as such…

Have there been an attempt at that? Anyway, what would a such change 
require and how much would it overturn the current display engine? would a 
solution to that problem possibly cover up with the problem I raised in 
the other almost-homonym thread?

Le lundi 18 octobre 2021, 14:40:42 CEST Alexandre Garreau a écrit :
> Is there a var to make any line whose first either rtl or ltr char is
> RTL be displayed RTL (so for instance an org-mode definition list (with
> ::) would be displayed RTL with the bullets at right, if the defining
> terms (at the beginning, just after the */+/-/etc.) are in
> arabic/hebrew/etc.)?
> 
> Another idea I have is is there a way to do that per-paragraph (for
> instance for a defun that would use a such language)?
> 
> and arbitrarily (for direction-agnostic languages such as some extreme-
> oriental ones)?
> 
> and does emacs support TTB/BTT (top to bottom, bottom to top), because I
> feel that would be a better use of human peripheral vision and physical
> ergonomical limit of line-length?

Re: Verticality and future of display engine and lines (bis) [Was: Re: RTL lines]

[Eli Zaretskii]

> From: Alexandre Garreau <galex-713@galex-713.eu>
> Date: Wed, 27 Oct 2021 16:50:17 +0200
> 
> I just coincidentally ended on a webpage with a vertical script (which 
> surprisingly to me displayed correctly under firefox):
> 
> https://fr.wikipedia.org/wiki/Sakhaline
> 
> However copypasting that into emacs, or viewing the page with Eww makes 
> that text incorrectly LTR.  Hence I suppose emacs doesn’t yet support 
> viewing and editing TTB/BBT text as such…
> 
> Have there been an attempt at that?

No.

> Anyway, what would a such change require and how much would it
> overturn the current display engine?

The low level of the display engine will need to acquire a set of
routines that work very differently from what we have there now.

And I don't even understand what is being expected from this.  E.g.,
what happens when the user scrolls the window up or down?  Do all the
columns change like a snake?

> would a solution to that problem possibly cover up with the problem
> I raised in the other almost-homonym thread?

Which is what?

Re: Verticality and future of display engine and lines (bis) [Was: Re: RTL lines]

[Alexandre Garreau]

[-- Attachment #1: Type: text/plain, Size: 6486 bytes --]

Le mercredi 27 octobre 2021, 18:18:21 CEST Eli Zaretskii a écrit :
> > From: Alexandre Garreau <galex-713@galex-713.eu>
> > Date: Wed, 27 Oct 2021 16:50:17 +0200
> > 
> > I just coincidentally ended on a webpage with a vertical script (which
> > surprisingly to me displayed correctly under firefox):
> > 
> > https://fr.wikipedia.org/wiki/Sakhaline
> > 
> > However copypasting that into emacs, or viewing the page with Eww
> > makes
> > that text incorrectly LTR.  Hence I suppose emacs doesn’t yet support
> > viewing and editing TTB/BBT text as such…
> > 
> > Have there been an attempt at that?
> 
> No.

I feel like although RTL support is considered as a minority concern in 
IT, TTB/BBT is something several orders of magnitude more esoteric…

There have been literally only *once* I saw it mentioned anywhere, and it 
was in the spec of XML-FO (yet not a lot of technical details were given 
about the precise implementation, especially about mixing).

> > Anyway, what would a such change require and how much would it
> > overturn the current display engine?
> 
> The low level of the display engine will need to acquire a set of
> routines that work very differently from what we have there now.
> 
> And I don't even understand what is being expected from this.  E.g.,
> what happens when the user scrolls the window up or down?  Do all the
> columns change like a snake?

I expect a full TTB buffer to behave exactly as a LTR one but turned 90° 
and portrait-like instead of landscape-like (like a screen usually is)

Hence to see the rest of buffer, instead of scrolling up/down, the user 
would scroll left/right.

Scrolling up/down would not be needed (and the up/down scrollbar shouldn’t 
be displayed) unless the (vertical) lines are too long to fit on the screen 
(which shouldn’t happen because by default they get wrapped): then you 
would just scroll as little as the length of lines requires.  That’s just 
like when in LTR/RTL lines are too long and not wrapped and then you get a 
right/left scrollbar.  This is the current behavior.  It just happens to 
support only one direction: left/right.

The most complex scenario is when you mix LTR/RTL and TTB/BTT: then you 
will get very high spacing between lines of the containing text, to leave 
space for the contained text.  The wikipedia page I linked initially in 
the OP, under firefox ESR 78, displays a such example, here attached so you 
can see it’s not an image and even supports hybrid LTR/TTB text selection.

Interestingly, this is an already existing behavior within emacs, and that 
question arises under any powerful-enough unicode-supporting display: if 
you add a lot of dı̇̇̇̇̇̇̇̇̇̇̇̇̇̇̇̇acritics (like here) on a character, the line becomes 
taller to contain it (most of other software than emacs actually either 
fails (most often) to display more than one diacritic (erroneously, since 
this is a necessary display feature for some writing systems (such as 
vietnamese, afair)), or (rarely, or only with Qt in textareas) 
overdisplays the diacritics over the above lines, screenshot (Qt/KMail, 
GTK/Emacs) attached).  Interestingly, other GTK software than emacs just 
fails to display several diacritic one on top of each other, and just 
surimpress them all at the same place (so it’s unreadable), so emacs 
performs just better than bare gtk alone.

Overdisplaying is not necessarily a bad thing (although this is not 
current emacs behavior), because it keeps the typographical gray 
consistent.  However, it makes both sets of glyphs unreadable, so it’s 
unpractical: what would be interesting is not to display totally the 
contained text most of the time (maybe by making it progressively clear 
from the baseline up to the top of the next line, so the user still 
understands the text normally continues down), and when you move to TTB 
text or the char/word/line with diacritics, to put an (partially) opaque 
background under the TTB text or the char with diacritics, so at least one 
of the portions of text is readable at every time, without perturbing the 
beautiful typographical grey of the typesetting.

Another useful thing would be to wrap the TTB text very heavily, like 
after some chars, or to display it very small (at least most of the time, 
as for the last suggestion)

The final useful thing is the current behavior: simply not to display it 
TTB, but LTR.  I think that’s the most practical, although the less 
faithful.  We could however generalize into “always use the direction of 
containing text for the contained text”, or rather, in our simple case 
with our simple buffers: “select only one text direction for one buffer”.

That’s akin to the preferred solution of some friend of mine who simply 
did set up his XeTeX config so that when he includes latin text into arabic 
text, it’s written RTL, and when he includes arabic text into latin text, 
it’s written LTR.  Although that’s counter-intuitive and a little 
confusing to native reader of both languages for the included text, he 
deems that more practical as anyway such included portions of text are 
usually very small (usually one word (for linguistic reference), sometimes 
small sentences (typically quotes)), and “having to find the other part of 
the included text portion to begin reading it and then going back to it 
after being over” is considered as tiring by him (but well, he’s also a 
adept of boustrophedon, although he still didn’t manage to set up any 
software to support such directionality).

Btw it would be nice if emacs supported such tweaking of directionality 
(although my friend wouldn’t benefit from it since he’s a user of vim, but 
I’m pretty sure that would be a point in advertising emacs to him).

> > would a solution to that problem possibly cover up with the problem
> > I raised in the other almost-homonym thread?
> 
> Which is what?

“Future of display engines and lines”, where I started talking about 
multicolumn and that continued about sub-buffers, evolving the display 
engine, how web engines wouldn’t last long enough, etc.

If we found a solution to that problem (inserting multiple columns or 
subbuffers inside a window, having something more like a DOM/tree like 
within TeX, etc.), would support of TTB/BTT become easier, or naturally 
come along?

[-- Attachment #2: Sakhaline.png --]
[-- Type: image/png, Size: 123705 bytes --]

[-- Attachment #3: emacs.png --]
[-- Type: image/png, Size: 14556 bytes --]

[-- Attachment #4: qt.png --]
[-- Type: image/png, Size: 21207 bytes --]

Re: Verticality and future of display engine and lines (bis) [Was: Re: RTL lines]

[Eli Zaretskii]

> From: Alexandre Garreau <galex-713@galex-713.eu>
> Cc: help-gnu-emacs@gnu.org
> Date: Wed, 27 Oct 2021 20:46:45 +0200
> 
> > > Anyway, what would a such change require and how much would it
> > > overturn the current display engine?
> > 
> > The low level of the display engine will need to acquire a set of
> > routines that work very differently from what we have there now.
> > 
> > And I don't even understand what is being expected from this.  E.g.,
> > what happens when the user scrolls the window up or down?  Do all the
> > columns change like a snake?
> 
> I expect a full TTB buffer to behave exactly as a LTR one but turned 90° 
> and portrait-like instead of landscape-like (like a screen usually is)
> 
> Hence to see the rest of buffer, instead of scrolling up/down, the user 
> would scroll left/right.

So you are not only talking about a revolution in the display engine,
you are also talking about a revolution in scrolling commands.

> Interestingly, this is an already existing behavior within emacs, and that 
> question arises under any powerful-enough unicode-supporting display: if 
> you add a lot of dı̇̇̇̇̇̇̇̇̇̇̇̇̇̇̇̇acritics (like here) on a character, the line becomes 
> taller to contain it (most of other software than emacs actually either 
> fails (most often) to display more than one diacritic (erroneously, since 
> this is a necessary display feature for some writing systems (such as 
> vietnamese, afair)), or (rarely, or only with Qt in textareas) 
> overdisplays the diacritics over the above lines, screenshot (Qt/KMail, 
> GTK/Emacs) attached).

I don't see any excess height of the lines, I think if you see that in
Emacs, you have a faulty font or something.

> Interestingly, other GTK software than emacs just 
> fails to display several diacritic one on top of each other, and just 
> surimpress them all at the same place (so it’s unreadable), so emacs 
> performs just better than bare gtk alone.

Showing just one diacritic is TRT in this case.  Anything else is a
display bug.

> Btw it would be nice if emacs supported such tweaking of directionality 
> (although my friend wouldn’t benefit from it since he’s a user of vim, but 
> I’m pretty sure that would be a point in advertising emacs to him).

You can have this with special bidirectional formatting control
characters, like LRO and RLO.  Emacs supports them.

> > > would a solution to that problem possibly cover up with the problem
> > > I raised in the other almost-homonym thread?
> > 
> > Which is what?
> 
> “Future of display engines and lines”, where I started talking about 
> multicolumn and that continued about sub-buffers, evolving the display 
> engine, how web engines wouldn’t last long enough, etc.

No, that's an entirely different problem that would need an entirely
different solution.

Re: RTL lines

[Alexandre Garreau]

Pardon me, I think I missed that mail too, and caused you to loose time 
because of that (given the relatedness of many conversations now) :/

I really should read all emacs manual, I even have it printed…

Le lundi 18 octobre 2021, 15:18:11 CEST Eli Zaretskii a écrit :
> > From: Alexandre Garreau <galex-713@galex-713.eu>
> > Date: Mon, 18 Oct 2021 14:40:42 +0200
> > 
> > Is there a var to make any line whose first either rtl or ltr char is
> > RTL be displayed RTL (so for instance an org-mode definition list
> > (with ::) would be displayed RTL with the bullets at right, if the
> > defining terms (at the beginning, just after the */+/-/etc.) are in
> > arabic/hebrew/etc.)?
> Set bidi-paragraph-direction to the nil value.  It's a per-buffer
> variable.

Thank you!

> > Another idea I have is is there a way to do that per-paragraph (for
> > instance for a defun that would use a such language)?
> 
> The above will do that per paragraph.  That is, each paragraph will be
> LTR or RTL depending on the first "strong directional" character in
> that paragraph.

Thinking to it, yes, a paragraph is the most meaningful semantical unit to 
do that.  I guess in any file where lines can be filled (like, except csv 
files for instance) there wouldn’t be sense to do that…

…but I thought the definition of a paragraph was contextual, and determined 
what M-q would do… but here I notice it isn’t: it’s barely “any portion of 
text separated by a blank line”, so many bullets within org-mode (whose 
each would be filled/wrapped separately by M-q), and even several 
*sections* in org/outline-mode would be considered as a paragraph, until 
the next blank line, even in the middle of a subsection…

> (This all is supposed to be well documented in the Emacs manual.
> Hint, hint...)

Emacs is very big and so is its manual, but yes I should get more of an 
habit of searching within it, instead of simply limiting myself to doing 
C-h (maybe both kind of documentations could be hyperlinked? maybe they 
already are…)

> > and arbitrarily (for direction-agnostic languages such as some
> > extreme-
> > oriental ones)?
> 
> I don't think I understand what kind of "arbitrariness" you have in
> mind here.

Like “I want chinese to be written TTB” (well, let’s say japanese and LTR, 
given what you say thereafter).

> > and does emacs support TTB/BTT (top to bottom, bottom to top)
> 
> No.  if you want to submit patches to that effect,l they will be most
> welcome, though.

Thank you very much for the encouragement, if I succeed getting again into 
hacking I certainly will.

Re: Verticality and future of display engine and lines (bis) [Was: Re: RTL lines]

[Alexandre Garreau]

Le mercredi 27 octobre 2021, 20:58:47 CEST Eli Zaretskii a écrit :
> > From: Alexandre Garreau <galex-713@galex-713.eu>
> > Cc: help-gnu-emacs@gnu.org
> > Date: Wed, 27 Oct 2021 20:46:45 +0200
> > 
> > > > Anyway, what would a such change require and how much would it
> > > > overturn the current display engine?
> > > 
> > > The low level of the display engine will need to acquire a set of
> > > routines that work very differently from what we have there now.
> > > 
> > > And I don't even understand what is being expected from this.  E.g.,
> > > what happens when the user scrolls the window up or down?  Do all
> > > the
> > > columns change like a snake?
> > 
> > I expect a full TTB buffer to behave exactly as a LTR one but turned
> > 90° and portrait-like instead of landscape-like (like a screen
> > usually is)
> > 
> > Hence to see the rest of buffer, instead of scrolling up/down, the
> > user
> > would scroll left/right.
> 
> So you are not only talking about a revolution in the display engine,
> you are also talking about a revolution in scrolling commands.

That’s less than half a dozen of commands and the change is trivial, it’s 
just level of abstraction.

> > Interestingly, this is an already existing behavior within emacs, and
> > that question arises under any powerful-enough unicode-supporting
> > display: if you add a lot of dı̇̇̇̇̇̇̇̇̇̇̇̇̇̇̇̇acritics (like here)
> > on a character, the line becomes taller to contain it (most of other
> > software than emacs actually either fails (most often) to display
> > more than one diacritic (erroneously, since this is a necessary
> > display feature for some writing systems (such as vietnamese,
> > afair)), or (rarely, or only with Qt in textareas) overdisplays the
> > diacritics over the above lines, screenshot (Qt/KMail, GTK/Emacs)
> > attached).
> 
> I don't see any excess height of the lines, I think if you see that in
> Emacs, you have a faulty font or something.

Did you really look at the screenshots? don’t you see all the blank 
between the lines?

> > Interestingly, other GTK software than emacs just
> > fails to display several diacritic one on top of each other, and just
> > surimpress them all at the same place (so it’s unreadable), so emacs
> > performs just better than bare gtk alone.
> 
> Showing just one diacritic is TRT in this case.  Anything else is a
> display bug.

So how do you display languages which *need* several diacritics?

> > Btw it would be nice if emacs supported such tweaking of
> > directionality
> > (although my friend wouldn’t benefit from it since he’s a user of vim,
> > but I’m pretty sure that would be a point in advertising emacs to
> > him).
> You can have this with special bidirectional formatting control
> characters, like LRO and RLO.  Emacs supports them.

Yes I know but I mean to have that systematically without having to 
manually enter them at each script change (and avoiding to put them in a 
possibly shared file with people with different opinions regarding 
directionality)

> > > > would a solution to that problem possibly cover up with the
> > > > problem
> > > > I raised in the other almost-homonym thread?
> > > 
> > > Which is what?
> > 
> > “Future of display engines and lines”, where I started talking about
> > multicolumn and that continued about sub-buffers, evolving the display
> > engine, how web engines wouldn’t last long enough, etc.
> 
> No, that's an entirely different problem that would need an entirely
> different solution.

Ok.

Re: RTL lines

[Eli Zaretskii]

> From: Alexandre Garreau <galex-713@galex-713.eu>
> Cc: Eli Zaretskii <eliz@gnu.org>
> Date: Wed, 27 Oct 2021 21:26:18 +0200
> 
> I really should read all emacs manual, I even have it printed…

That's a large job.  But you don't need to read the manual in its
entirety, you can read it subject by subject, based on subjects that
are of interest, and using 'i' (Info-index) to find where a subject is
described.

> > The above will do that per paragraph.  That is, each paragraph will be
> > LTR or RTL depending on the first "strong directional" character in
> > that paragraph.
> 
> Thinking to it, yes, a paragraph is the most meaningful semantical unit to 
> do that.  I guess in any file where lines can be filled (like, except csv 
> files for instance) there wouldn’t be sense to do that…

That's not our invention, that's what the Unicode Bidirectional
Algorithm (which Emacs implements) mandates.

> …but I thought the definition of a paragraph was contextual, and determined 
> what M-q would do… but here I notice it isn’t: it’s barely “any portion of 
> text separated by a blank line”, so many bullets within org-mode (whose 
> each would be filled/wrapped separately by M-q), and even several 
> *sections* in org/outline-mode would be considered as a paragraph, until 
> the next blank line, even in the middle of a subsection…

The reordering of bidirectional text for display has its own separate
definition of what is a paragraph, see bidi-paragraph-start-re and
bidi-paragraph-separate-re.

> > (This all is supposed to be well documented in the Emacs manual.
> > Hint, hint...)
> 
> Emacs is very big and so is its manual, but yes I should get more of an 
> habit of searching within it, instead of simply limiting myself to doing 
> C-h (maybe both kind of documentations could be hyperlinked? maybe they 
> already are…)

No need to read everything, just type "i bidi RET", and you will land
in the right spot.

Re: RTL lines

[Alexandre Garreau]

Le jeudi 28 octobre 2021, 08:04:02 CEST Eli Zaretskii a écrit :
> > From: Alexandre Garreau <galex-713@galex-713.eu>
> > Cc: Eli Zaretskii <eliz@gnu.org>
> > Date: Wed, 27 Oct 2021 21:26:18 +0200
> > 
> > I really should read all emacs manual, I even have it printed…
> 
> That's a large job.  But you don't need to read the manual in its
> entirety, you can read it subject by subject, based on subjects that
> are of interest, and using 'i' (Info-index) to find where a subject is
> described.

I know I know, however reading everything is an important goal I have as 
I’m curious and I’d like to see how emacs works for most important stuff, 
the most possible.  It looks doable.

> > > The above will do that per paragraph.  That is, each paragraph will
> > > be
> > > LTR or RTL depending on the first "strong directional" character in
> > > that paragraph.
> > 
> > Thinking to it, yes, a paragraph is the most meaningful semantical
> > unit to do that.  I guess in any file where lines can be filled
> > (like, except csv files for instance) there wouldn’t be sense to do
> > that…
> 
> That's not our invention, that's what the Unicode Bidirectional
> Algorithm (which Emacs implements) mandates.

Oh ok, but do they define what a paragraph is?

> > …but I thought the definition of a paragraph was contextual, and
> > determined what M-q would do… but here I notice it isn’t: it’s barely
> > “any portion of text separated by a blank line”, so many bullets
> > within org-mode (whose each would be filled/wrapped separately by
> > M-q), and even several *sections* in org/outline-mode would be
> > considered as a paragraph, until the next blank line, even in the
> > middle of a subsection…
> 
> The reordering of bidirectional text for display has its own separate
> definition of what is a paragraph, see bidi-paragraph-start-re and
> bidi-paragraph-separate-re.

Oh! but why isn’t that kept synchronized with the normal emacs definition 
of a paragraph (such as with the outline/org-mode one? when they’re 
respectively enabled)

> > > (This all is supposed to be well documented in the Emacs manual.
> > > Hint, hint...)
> > 
> > Emacs is very big and so is its manual, but yes I should get more of
> > an
> > habit of searching within it, instead of simply limiting myself to
> > doing C-h (maybe both kind of documentations could be hyperlinked?
> > maybe they already are…)
> 
> No need to read everything, just type "i bidi RET", and you will land
> in the right spot.

I didn’t imagine it was sectionned like that, I imagined that there would 
be some information about internationalization scattered among it, dunno 
why I imagined that… these time I tend to imagine that manuals/books 
seldom section things the same way I would…

Re: Verticality and future of display engine and lines (bis) [Was: Re: RTL lines]

[Eli Zaretskii]

> From: Alexandre Garreau <galex-713@galex-713.eu>
> Date: Wed, 27 Oct 2021 21:54:11 +0200
> 
> > So you are not only talking about a revolution in the display engine,
> > you are also talking about a revolution in scrolling commands.
> 
> That’s less than half a dozen of commands and the change is trivial, it’s 
> just level of abstraction.

I guess you are unaware, or perhaps forgot, that the display engine
itself scrolls the window when it finds that necessary.

> > I don't see any excess height of the lines, I think if you see that in
> > Emacs, you have a faulty font or something.
> 
> Did you really look at the screenshots? don’t you see all the blank 
> between the lines?

I'm talking about what I see in my Emacs session where I read your
email.  If any Emacs session displays that as you describe, that's
either a font configuration problem or some rendering bug that isn't
present in my build of Emacs.

> > Showing just one diacritic is TRT in this case.  Anything else is a
> > display bug.
> 
> So how do you display languages which *need* several diacritics?

Not every arbitrary combination of the diacritics on the same base
character is valid.  Lumping the same diacritic several times on the
same base character certainly isn't.  For valid combination of the
diacritics, we will display them the way the font and the shaping
engine (HarfBuzz) tell us to do.

IOW, find a real-life case with several diacritics on the same
character, and then let's talk.  The example you provided makes no
sense in real life, and the correct reaction of the Emacs display
engine to that nonsensical example is to show just one diacritic.

> > > Btw it would be nice if emacs supported such tweaking of
> > > directionality
> > > (although my friend wouldn’t benefit from it since he’s a user of vim,
> > > but I’m pretty sure that would be a point in advertising emacs to
> > > him).
> > You can have this with special bidirectional formatting control
> > characters, like LRO and RLO.  Emacs supports them.
> 
> Yes I know but I mean to have that systematically without having to 
> manually enter them at each script change (and avoiding to put them in a 
> possibly shared file with people with different opinions regarding 
> directionality)

Using those formatting controls _is_ the systematic way of changing
the text directionality.  Emacs is not a playground for arbitrarily
messing with the text layout, it follows relevant standards of
layout.  And in those standards, the _only_ way of controlling the
text directionality is by using directionality controls and
overrides.

You could, of course, go overboard and modify the basic properties of
the characters, which Emacs holds in char-tables.  But that is not
recommended, and I'm not even sure I understand why would you like to
do something like that.

Re: RTL lines

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1075 bytes --]

On Thu, Oct 28, 2021 at 08:08:53AM +0200, Alexandre Garreau wrote:

[...]

> Oh ok, but do they define what a paragraph is?

This is... complicated. See [0], especially [1], and the references
from there. My take (but note that I'm an analphabet here) is that
there usually will be a surrounding level of "markup" (e.g. XML,
HTML, something else) which decides what a paragraph is. The usual
convention of an empty line separating paragraphs is fine as that
"surrounding level", too. Unicode also has a "paragraph separator",
\u2029, in case you need one.

However that separation is done, the BIDI algorithm doesn't care.
It works exclusively whithin one paragraph, whatever happens in one
paragraph doesn't touch the others, and thus it wants to be fed
single paragraphs.

Read the refs. Very instructive (and you'll learn to appreciate a
bit more what a labour of love Eli has been doing before he became
our maintainer).

Cheers

[0] https://www.unicode.org/reports/tr9/
[1] https://www.unicode.org/reports/tr9/#Basic_Display_Algorithm

 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Verticality and future of display engine and lines (bis) [Was: Re: RTL lines]

[Alexandre Garreau]

Le jeudi 28 octobre 2021, 08:53:21 CEST Eli Zaretskii a écrit :
> > From: Alexandre Garreau <galex-713@galex-713.eu>
> > Date: Wed, 27 Oct 2021 21:54:11 +0200
> > 
> > > So you are not only talking about a revolution in the display
> > > engine,
> > > you are also talking about a revolution in scrolling commands.
> > 
> > That’s less than half a dozen of commands and the change is trivial,
> > it’s just level of abstraction.
> 
> I guess you are unaware, or perhaps forgot, that the display engine
> itself scrolls the window when it finds that necessary.

oh… but still that’s a few functions right…? I hope… like “scroll down” 
and “scroll up” calls that should be conditionally changed to “scroll 
forward” and “scroll backward”

> > > I don't see any excess height of the lines, I think if you see that
> > > in
> > > Emacs, you have a faulty font or something.
> > 
> > Did you really look at the screenshots? don’t you see all the blank
> > between the lines?
> 
> I'm talking about what I see in my Emacs session where I read your
> email.  If any Emacs session displays that as you describe, that's
> either a font configuration problem or some rendering bug that isn't
> present in my build of Emacs.

But did you look at the screenshots? doesn’t your gnus support simple and 
direct display of mime attachments?

> > > Showing just one diacritic is TRT in this case.  Anything else is a
> > > display bug.
> > 
> > So how do you display languages which *need* several diacritics?
> 
> Not every arbitrary combination of the diacritics on the same base
> character is valid.  Lumping the same diacritic several times on the
> same base character certainly isn't.  For valid combination of the
> diacritics, we will display them the way the font and the shaping
> engine (HarfBuzz) tell us to do.

oh ok

> IOW, find a real-life case with several diacritics on the same
> character, and then let's talk.  The example you provided makes no
> sense in real life, and the correct reaction of the Emacs display
> engine to that nonsensical example is to show just one diacritic.
>
> > > > Btw it would be nice if emacs supported such tweaking of
> > > > directionality
> > > > (although my friend wouldn’t benefit from it since he’s a user of
> > > > vim,
> > > > but I’m pretty sure that would be a point in advertising emacs to
> > > > him).
> > > 
> > > You can have this with special bidirectional formatting control
> > > characters, like LRO and RLO.  Emacs supports them.
> > 
> > Yes I know but I mean to have that systematically without having to
> > manually enter them at each script change (and avoiding to put them in
> > a possibly shared file with people with different opinions regarding
> > directionality)
> 
> Using those formatting controls _is_ the systematic way of changing
> the text directionality.  Emacs is not a playground for arbitrarily
> messing with the text layout, it follows relevant standards of
> layout.  And in those standards, the _only_ way of controlling the
> text directionality is by using directionality controls and
> overrides.
> 
> You could, of course, go overboard and modify the basic properties of
> the characters, which Emacs holds in char-tables.

Ohhhhh thank you! I should then look into that

> But that is not
> recommended,

well as you said that’s non-standard

> and I'm not even sure I understand why would you like to
> do something like that.

I already said it was to make reading more comfortable without having to 
lump from one part of the text to another, read, and yet again go there to 
resume reading, so the direction of reading is always consistent (and 
ideally to have even less jumping, one would need to use boustrophedon as 
a script direction, but afaik no existing software supports that)

Re: Verticality and future of display engine and lines (bis) [Was: Re: RTL lines]

[Alexandre Garreau]

[-- Attachment #1: Type: text/plain, Size: 820 bytes --]

forgot that:

Le jeudi 28 octobre 2021, 08:53:21 CEST Eli Zaretskii a écrit :
> IOW, find a real-life case with several diacritics on the same
> character, and then let's talk.  The example you provided makes no
> sense in real life, and the correct reaction of the Emacs display
> engine to that nonsensical example is to show just one diacritic.

“Vietnamese (Vietnamese: tiếng Việt) is an Austroasiatic language” from 
https://en.wikipedia.org/wiki/Vietnamese_language (which firefox displays 
just fine, screenshot attached).

For some reason, for me, within emacs, this displays bad: the e is 
oversmall, while using combining diacritics displays right: “tiếng Việt”.

Whatever the way, Qt displays that incorrectly just the same.

Screenshots attached (but do you look at them?)

[-- Attachment #2: firefox.png --]
[-- Type: image/png, Size: 6388 bytes --]

[-- Attachment #3: emacs.png --]
[-- Type: image/png, Size: 8713 bytes --]

[-- Attachment #4: qt.png --]
[-- Type: image/png, Size: 5721 bytes --]

Re: RTL lines

[Eli Zaretskii]

> From: Alexandre Garreau <galex-713@galex-713.eu>
> Cc: help-gnu-emacs@gnu.org
> Date: Thu, 28 Oct 2021 08:08:53 +0200
> 
> > > Thinking to it, yes, a paragraph is the most meaningful semantical
> > > unit to do that.  I guess in any file where lines can be filled
> > > (like, except csv files for instance) there wouldn’t be sense to do
> > > that…
> > 
> > That's not our invention, that's what the Unicode Bidirectional
> > Algorithm (which Emacs implements) mandates.
> 
> Oh ok, but do they define what a paragraph is?

Yes, they do.

> > The reordering of bidirectional text for display has its own separate
> > definition of what is a paragraph, see bidi-paragraph-start-re and
> > bidi-paragraph-separate-re.
> 
> Oh! but why isn’t that kept synchronized with the normal emacs definition 
> of a paragraph (such as with the outline/org-mode one? when they’re 
> respectively enabled)

Because in some important modes that "normal" definition makes no
sense for bidirectional reordering.  For example, texinfo-mode.

Re: Verticality and future of display engine and lines (bis) [Was: Re: RTL lines]

[Eli Zaretskii]

> From: Alexandre Garreau <galex-713@galex-713.eu>
> Cc: Eli Zaretskii <eliz@gnu.org>
> Date: Thu, 28 Oct 2021 09:12:38 +0200
> 
> > I guess you are unaware, or perhaps forgot, that the display engine
> > itself scrolls the window when it finds that necessary.
> 
> oh… but still that’s a few functions right…? I hope… like “scroll down” 
> and “scroll up” calls that should be conditionally changed to “scroll 
> forward” and “scroll backward”

I guess you think that scrolling, in whatever direction, is a simple
business.  It isn't; what those "few functions" do has a lot of
implicit assumptions, most of them will be wrong with the change of
direction.  Someone(TM) will have to come up with the necessary logic
that doesn't exist, and make it support all the scroll-related
features we have, like scroll-step, scroll-conservatively,
scroll-up/down-aggressively, scroll-preserve-screen-position, etc.
And then the low-level code which scrolls the screen by moving pixels
will have to be rethought as well.

> > > Did you really look at the screenshots? don’t you see all the blank
> > > between the lines?
> > 
> > I'm talking about what I see in my Emacs session where I read your
> > email.  If any Emacs session displays that as you describe, that's
> > either a font configuration problem or some rendering bug that isn't
> > present in my build of Emacs.
> 
> But did you look at the screenshots? doesn’t your gnus support simple and 
> direct display of mime attachments?

(I don't use Gnus.)  Of course, I looked at them.  Why do you ask?
they look like display bugs to me, as I said.

> > and I'm not even sure I understand why would you like to
> > do something like that.
> 
> I already said it was to make reading more comfortable without having to 
> lump from one part of the text to another, read, and yet again go there to 
> resume reading, so the direction of reading is always consistent (and 
> ideally to have even less jumping, one would need to use boustrophedon as 
> a script direction, but afaik no existing software supports that)

I cannot imagine it will be easy to read an RTL text that wasn't
reordered for display.  You'd have to read it one character at a time,
something that is extremely slow.

Re: Verticality and future of display engine and lines (bis) [Was: Re: RTL lines]

[Eli Zaretskii]

> From: Alexandre Garreau <galex-713@galex-713.eu>
> Cc: Eli Zaretskii <eliz@gnu.org>
> Date: Thu, 28 Oct 2021 09:18:55 +0200
> 
> “Vietnamese (Vietnamese: tiếng Việt) is an Austroasiatic language” from 
> https://en.wikipedia.org/wiki/Vietnamese_language (which firefox displays 
> just fine, screenshot attached).
> 
> For some reason, for me, within emacs, this displays bad: the e is 
> oversmall, while using combining diacritics displays right: “tiếng Việt”.

Your default font doesn't support Vietnamese; find a better font, if
you want Vietnamese text to be displayed without this ugliness.

Re: Verticality and future of display engine and lines (bis) [Was: Re: RTL lines]

[Alexandre Garreau]

Le jeudi 28 octobre 2021, 11:28:40 CEST Eli Zaretskii a écrit :
> > From: Alexandre Garreau <galex-713@galex-713.eu>
> > Cc: Eli Zaretskii <eliz@gnu.org>
> > Date: Thu, 28 Oct 2021 09:18:55 +0200
> > 
> > “Vietnamese (Vietnamese: tiếng Việt) is an Austroasiatic language”
> > from
> > https://en.wikipedia.org/wiki/Vietnamese_language (which firefox
> > displays just fine, screenshot attached).
> > 
> > For some reason, for me, within emacs, this displays bad: the e is
> > oversmall, while using combining diacritics displays right: “tiếng
> > Việt”.
> Your default font doesn't support Vietnamese; find a better font, if
> you want Vietnamese text to be displayed without this ugliness.

Still not a user problem, as one should not have to manually search and 
gather fonts before to start reading/seeing/discovering a certain script.  
A such font should be installed by default.  But of course I can conceive 
this is not emacs’ job but rather, dunno, most probably X’s one I guess 
(more globally: debian’s one in my case).

Re: Verticality and future of display engine and lines (bis) [Was: Re: RTL lines]

[Alexandre Garreau]

Le jeudi 28 octobre 2021, 11:23:58 CEST Eli Zaretskii a écrit :
> > From: Alexandre Garreau <galex-713@galex-713.eu>
> > Cc: Eli Zaretskii <eliz@gnu.org>
> > Date: Thu, 28 Oct 2021 09:12:38 +0200
> > 
> > > I guess you are unaware, or perhaps forgot, that the display engine
> > > itself scrolls the window when it finds that necessary.
> > 
> > oh… but still that’s a few functions right…? I hope… like “scroll
> > down”
> > and “scroll up” calls that should be conditionally changed to “scroll
> > forward” and “scroll backward”
> 
> I guess you think that scrolling, in whatever direction, is a simple
> business.  It isn't; what those "few functions" do has a lot of
> implicit assumptions, most of them will be wrong with the change of
> direction.  Someone(TM) will have to come up with the necessary logic
> that doesn't exist, and make it support all the scroll-related
> features we have, like scroll-step, scroll-conservatively,
> scroll-up/down-aggressively, scroll-preserve-screen-position, etc.
> And then the low-level code which scrolls the screen by moving pixels
> will have to be rethought as well.
> 
> > > > Did you really look at the screenshots? don’t you see all the
> > > > blank
> > > > between the lines?
> > > 
> > > I'm talking about what I see in my Emacs session where I read your
> > > email.  If any Emacs session displays that as you describe, that's
> > > either a font configuration problem or some rendering bug that isn't
> > > present in my build of Emacs.
> > 
> > But did you look at the screenshots? doesn’t your gnus support simple
> > and direct display of mime attachments?
> 
> (I don't use Gnus.)

Sorry, in the absence of user-agent I wrongly assumed.

> Of course, I looked at them.  Why do you ask?
> they look like display bugs to me, as I said.

Oh ok I wanted to be sure, because I wasn’t (since to me it didn’t), now 
I’m sure they can look like bugs.

But about the mongolian script (you looked at that screenshot too right?), 
how would you imagine it to be displayed, ideally? did what firefox did 
look like a display bug too, to you?

> > > and I'm not even sure I understand why would you like to
> > > do something like that.
> > 
> > I already said it was to make reading more comfortable without having
> > to lump from one part of the text to another, read, and yet again go
> > there to resume reading, so the direction of reading is always
> > consistent (and ideally to have even less jumping, one would need to
> > use boustrophedon as a script direction, but afaik no existing
> > software supports that)
> I cannot imagine it will be easy to read an RTL text that wasn't
> reordered for display.  You'd have to read it one character at a time,
> something that is extremely slow.

Oh yes indeed because when we learn to learn fast we learn to recognize 
words instead of characters.

But I guess you can get used to that (and somewhat faster than learning 
each word twice), otherwise boustrophedon wouldn’t have been a practical 
script direction to be used at first with many scripts (for instance greek 
and other more ancient scripts, at times where the direction of writing of 
occidental languages wasn’t fixed yet).

Supporting that, I actually can read latin mirrored, because I initially 
learnt to write in that sense (now I’m much slower at that than with 
normal writing, because I’ve come to read a lot more than to write, but it 
may be that in a couple decades I’ve growed faster at reading that sense), 
and I wouldn’t mind getting again more accustomed to it.  But I guess as 
that can’t be done efficiently in elisp, emacs is not made for that, since 
that’s no more than an esoteric usecase (we could also argue about the 
esoterism of the inclusion of many extinct scripts in unicode, and that 
they shouldn’t be supported (although ancient boustrophedonian greek 
script is not included, while it’s more recent than certain extinct 
ancient scripts that are included))

Re: Verticality and future of display engine and lines (bis) [Was: Re: RTL lines]

[Eli Zaretskii]

> From: Alexandre Garreau <galex-713@galex-713.eu>
> Cc: Eli Zaretskii <eliz@gnu.org>
> Date: Thu, 28 Oct 2021 16:28:39 +0200
> 
> > > For some reason, for me, within emacs, this displays bad: the e is
> > > oversmall, while using combining diacritics displays right: “tiếng
> > > Việt”.
> > Your default font doesn't support Vietnamese; find a better font, if
> > you want Vietnamese text to be displayed without this ugliness.
> 
> Still not a user problem, as one should not have to manually search and 
> gather fonts before to start reading/seeing/discovering a certain script.  
> A such font should be installed by default.

Then take it up with your distro's maintainers.

It's impossible to compose glyphs from different fonts, so if you want
such cases to be handled, you need to make sure you have the suitable
fonts installed and available to Emacs.  There's no way around that.

Re: Verticality and future of display engine and lines (bis) [Was: Re: RTL lines]

[Eli Zaretskii]

> From: Alexandre Garreau <galex-713@galex-713.eu>
> Cc: Eli Zaretskii <eliz@gnu.org>
> Date: Thu, 28 Oct 2021 16:40:39 +0200
> 
> But about the mongolian script (you looked at that screenshot too right?), 
> how would you imagine it to be displayed, ideally? did what firefox did 
> look like a display bug too, to you?

You mean, the TTB script?  I don't know.

> Supporting that, I actually can read latin mirrored, because I initially 
> learnt to write in that sense

One can read it, but it's inconvenient.