Basic WYSIWYG printing in GNU Emacs (Arabic & Hebrew)

Basic WYSIWYG printing in GNU Emacs (Arabic & Hebrew)

[Anand Tamariya]

[-- Attachment #1: Type: text/plain, Size: 428 bytes --]

Here is my next iteration on basic WYSIWYG printing in GNU Emacs for
printing RTL languages like Arabic and Hebrew.

   -

   Run M-x ps-print-buffer-with-faces to send output to printer.
   -

   Run C-u M-x ps-print-buffer-with-faces to send output to file (PS
   format).

*Code:* https://gitlab.com/atamariya/emacs/tree/dev

*Screenshots:*
https://lifeofpenguin.blogspot.com/2021/06/basic-wysiwyg-printing-in-gnu-emacs.html

[-- Attachment #2: Type: text/html, Size: 1316 bytes --]

Re: Basic WYSIWYG printing in GNU Emacs (Arabic & Hebrew)

[Eli Zaretskii]

> From: Anand Tamariya <atamariya@gmail.com>
> Date: Tue, 10 Aug 2021 11:11:26 +0530
> 
> Here is my next iteration on basic WYSIWYG printing in GNU Emacs for printing RTL languages like Arabic
> and Hebrew.

Does it also start RTL text at the right edge of the paper?  And how
does it reorder characters for printing?

(Sorry for asking these questions, but I'm still unable to see
anything useful on the GitLab URL you show -- when I go there, I see
only the navigation tree on the left and an empty pane on the right.)

Re: Basic WYSIWYG printing in GNU Emacs (Arabic & Hebrew)

[Anand Tamariya]

[-- Attachment #1: Type: text/plain, Size: 879 bytes --]

On Tue, Aug 10, 2021 at 5:31 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Anand Tamariya <atamariya@gmail.com>
> > Date: Tue, 10 Aug 2021 11:11:26 +0530
> >
> > Here is my next iteration on basic WYSIWYG printing in GNU Emacs for
> printing RTL languages like Arabic
> > and Hebrew.
>
> Does it also start RTL text at the right edge of the paper?  And how
> does it reorder characters for printing?
>
> (Sorry for asking these questions, but I'm still unable to see
> anything useful on the GitLab URL you show -- when I go there, I see
> only the navigation tree on the left and an empty pane on the right.)
>
> The code uses window positioning information for characters from Emacs and
then scales the content to fit the print width.

It uses following API to determine BIDI classification and processes
accordingly:
(get-char-code-property (char-after from) 'bidi-class)

[-- Attachment #2: Type: text/html, Size: 1351 bytes --]

Re: Basic WYSIWYG printing in GNU Emacs (Arabic & Hebrew)

[Eli Zaretskii]

> From: Anand Tamariya <atamariya@gmail.com>
> Date: Wed, 11 Aug 2021 09:18:23 +0530
> Cc: emacs-devel@gnu.org
> 
>  Does it also start RTL text at the right edge of the paper?  And how
>  does it reorder characters for printing?
> 
>  (Sorry for asking these questions, but I'm still unable to see
>  anything useful on the GitLab URL you show -- when I go there, I see
>  only the navigation tree on the left and an empty pane on the right.)
> 
> The code uses window positioning information for characters from Emacs and then scales the content to fit
> the print width. 

So this needs the buffer text be displayed in some window, before it
could be printed?

> It uses following API to determine BIDI classification and processes accordingly:
> (get-char-code-property (char-after from) 'bidi-class)

Hmm... not sure this is enough, except in the simplest cases.  The
bidi-class of a character is just its basic attribute; application of
the full Unicode Bidirectional Algorithm can and does change the
effective attributes of some characters, before they are displayed.

Is this test performed on each character or only at the beginning of a
visual line?

And I see that if you detect RTL characters, you just reverse the text
you get?  That is also very simplistic and will work correctly only if
the text includes only strong RTL characters, not numbers and no
embedded LTR characters.

Apologies if I didn't understand what the code does correctly; I only
had a cursory glance on it, and there are not so many comments
explaining what it does.

Thanks.

Re: Basic WYSIWYG printing in GNU Emacs (Arabic & Hebrew)

[Anand Tamariya]

[-- Attachment #1: Type: text/plain, Size: 1021 bytes --]

> So this needs the buffer text be displayed in some window, before it
> could be printed?
>
Yes. Plain text file would not have any font/ display information otherwise.

> It uses following API to determine BIDI classification and processes
> accordingly:
> > (get-char-code-property (char-after from) 'bidi-class)
>
> Hmm... not sure this is enough, except in the simplest cases.  The
> bidi-class of a character is just its basic attribute; application of
> the full Unicode Bidirectional Algorithm can and does change the
> effective attributes of some characters, before they are displayed.
>
Do you have some sample text for this use case? I can run a quick test.

Is this test performed on each character or only at the beginning of a
> visual line?
>
This happens at every point where the font changes.

Apologies if I didn't understand what the code does correctly; I only
> had a cursory glance on it, and there are not so many comments
> explaining what it does.
>
No worries. I'm happy to answer any queries.

[-- Attachment #2: Type: text/html, Size: 1784 bytes --]

Re: Basic WYSIWYG printing in GNU Emacs (Arabic & Hebrew)

[Eli Zaretskii]

> From: Anand Tamariya <atamariya@gmail.com>
> Date: Wed, 11 Aug 2021 21:29:53 +0530
> Cc: emacs-devel@gnu.org
> 
>  > It uses following API to determine BIDI classification and processes accordingly:
>  > (get-char-code-property (char-after from) 'bidi-class)
> 
>  Hmm... not sure this is enough, except in the simplest cases.  The
>  bidi-class of a character is just its basic attribute; application of
>  the full Unicode Bidirectional Algorithm can and does change the
>  effective attributes of some characters, before they are displayed.
> 
> Do you have some sample text for this use case? I can run a quick test.

You could try TUTORIAL.he, for example.

>  Is this test performed on each character or only at the beginning of a
>  visual line?
> 
> This happens at every point where the font changes. 

So if the entire buffer uses just a single font, like TUTORIAL.he
does, you do the test only once, at the first character of the buffer
text?  But that could produce incorrect results, because the text
further into the buffer could have both RTL and LTR paragraphs
intermixed, and the correct display will show each of these at their
correct base direction.  For example, most of the paragraphs in
TUTORIAL.he have right-to-left base direction, but the last paragraph,
with the Local Variables, is left-to-right, so its display starts at
the left edge of the window.

Re: Basic WYSIWYG printing in GNU Emacs (Arabic & Hebrew)

[Anand Tamariya]

[-- Attachment #1.1: Type: text/plain, Size: 1179 bytes --]

> So if the entire buffer uses just a single font, like TUTORIAL.he
> does, you do the test only once, at the first character of the buffer
> text?  But that could produce incorrect results, because the text
> further into the buffer could have both RTL and LTR paragraphs
> intermixed, and the correct display will show each of these at their
> correct base direction.  For example, most of the paragraphs in
> TUTORIAL.he have right-to-left base direction, but the last paragraph,
> with the Local Variables, is left-to-right, so its display starts at
> the left edge of the window.
>
Intermixed paragraphs are fine though mixed text is slightly problematic
(see attachment).

The window positioning information can only be obtained for the part
> of the buffer text actually visible in a window; for buffer positions
> outside of the viewport posn-at-point will give you nil.  How do you
> work around this limitation to allow printing text of the entire
> buffer?

You are correct with your doubts. And unfortunately, I don't have all the
answers. That's why in all my communication I've always maintained "Basic"
WYSIWYG. Maybe somebody else can improve upon the algorithm.

[-- Attachment #1.2: Type: text/html, Size: 1638 bytes --]

[-- Attachment #2: rtl1.png --]
[-- Type: image/png, Size: 72045 bytes --]

Re: Basic WYSIWYG printing in GNU Emacs (Arabic & Hebrew)

[Eli Zaretskii]

> From: Anand Tamariya <atamariya@gmail.com>
> Date: Thu, 12 Aug 2021 10:54:41 +0530
> Cc: emacs-devel@gnu.org
> 
>  So if the entire buffer uses just a single font, like TUTORIAL.he
>  does, you do the test only once, at the first character of the buffer
>  text?  But that could produce incorrect results, because the text
>  further into the buffer could have both RTL and LTR paragraphs
>  intermixed, and the correct display will show each of these at their
>  correct base direction.  For example, most of the paragraphs in
>  TUTORIAL.he have right-to-left base direction, but the last paragraph,
>  with the Local Variables, is left-to-right, so its display starts at
>  the left edge of the window.
> 
> Intermixed paragraphs are fine though mixed text is slightly problematic (see attachment).

Yes, the problems with rendering mixed text are to be expected with
such simplistic handling of bidirectional text.

>  The window positioning information can only be obtained for the part
>  of the buffer text actually visible in a window; for buffer positions
>  outside of the viewport posn-at-point will give you nil.  How do you
>  work around this limitation to allow printing text of the entire
>  buffer?
> 
> You are correct with your doubts. And unfortunately, I don't have all the answers. That's why in all my
> communication I've always maintained "Basic" WYSIWYG. Maybe somebody else can improve upon the
> algorithm.

OK, thanks.

I think a better way of doing this job is to implement a PS display
back-end, similar to the X, w32, and NS back-ends we have now (see
xterm.c, w32term.c, nsterm.m), but one which would emit PS code to
print stuff on a paper page of known dimensions, instead of showing it
on the screen.  Then the bidi reordering will be taken care of by the
display code without any additional efforts, and we will be free from
the limitations of having the buffer displayed in its entirety in a
window etc.  Unfortunately, this means most of the job will have to be
in C, not in Lisp.

Since you know much more than I do about PostScript and PS printers,
may I ask you a question: if one sends RTL text embedded in a PS
program to a PS printer, does the printer have the capability to
reorder RTL text when it prints it?  That is, if you send a string of
RTL characters that way, are they printed in reverse order?  Or do PS
printers need the text in the "visual" order, already reordered for
display?

Re: Basic WYSIWYG printing in GNU Emacs (Arabic & Hebrew)

[Anand Tamariya]

[-- Attachment #1: Type: text/plain, Size: 627 bytes --]

>
> Since you know much more than I do about PostScript and PS printers,
> may I ask you a question: if one sends RTL text embedded in a PS
> program to a PS printer, does the printer have the capability to
> reorder RTL text when it prints it?  That is, if you send a string of
> RTL characters that way, are they printed in reverse order?  Or do PS
> printers need the text in the "visual" order, already reordered for
> display?
>
Two things to remember about PS printer:
- It's like dumb terminal which places characters left to right unless
specified otherwise.
- It doesn't understand unicode, let alone RTL re-ordering.

[-- Attachment #2: Type: text/html, Size: 900 bytes --]

Re: Basic WYSIWYG printing in GNU Emacs (Arabic & Hebrew)

[Eli Zaretskii]

> From: Anand Tamariya <atamariya@gmail.com>
> Date: Thu, 12 Aug 2021 14:21:58 +0530
> Cc: emacs-devel@gnu.org
> 
>  Since you know much more than I do about PostScript and PS printers,
>  may I ask you a question: if one sends RTL text embedded in a PS
>  program to a PS printer, does the printer have the capability to
>  reorder RTL text when it prints it?  That is, if you send a string of
>  RTL characters that way, are they printed in reverse order?  Or do PS
>  printers need the text in the "visual" order, already reordered for
>  display?
> 
> Two things to remember about PS printer:
> - It's like dumb terminal which places characters left to right unless specified otherwise.
> - It doesn't understand unicode, let alone RTL re-ordering.

Thanks.  Then maybe we should expose to Lisp the Emacs bidi reordering
engine, in the form of a function that would accept a string and
return that string reordered for display according to the UBA.  Let me
know if that would help you improve the ps-print machinery.

Re: Basic WYSIWYG printing in GNU Emacs (Arabic & Hebrew)

[Anand Tamariya]

[-- Attachment #1: Type: text/plain, Size: 335 bytes --]

> Thanks.  Then maybe we should expose to Lisp the Emacs bidi reordering
> engine, in the form of a function that would accept a string and
> return that string reordered for display according to the UBA.  Let me
> know if that would help you improve the ps-print machinery.
>
Can't say anything just yet - need to think this through.

[-- Attachment #2: Type: text/html, Size: 558 bytes --]

Re: Basic WYSIWYG printing in GNU Emacs (Arabic & Hebrew)

[Eli Zaretskii]

> Date: Wed, 11 Aug 2021 14:37:24 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> > From: Anand Tamariya <atamariya@gmail.com>
> > Date: Wed, 11 Aug 2021 09:18:23 +0530
> > Cc: emacs-devel@gnu.org
> > 
> >  Does it also start RTL text at the right edge of the paper?  And how
> >  does it reorder characters for printing?
> > 
> >  (Sorry for asking these questions, but I'm still unable to see
> >  anything useful on the GitLab URL you show -- when I go there, I see
> >  only the navigation tree on the left and an empty pane on the right.)
> > 
> > The code uses window positioning information for characters from Emacs and then scales the content to fit
> > the print width. 
> 
> So this needs the buffer text be displayed in some window, before it
> could be printed?

The window positioning information can only be obtained for the part
of the buffer text actually visible in a window; for buffer positions
outside of the viewport posn-at-point will give you nil.  How do you
work around this limitation to allow printing text of the entire
buffer?