HTML rendering

HTML rendering

[Stefan Monnier]

I don't use EWW very much, but I often use SHR via Gnus and I must say,
I really like the new proportional-font rendering.

Thanks Lars,


        Stefan

Re: HTML rendering

[Katsumi Yamaoka]

[-- Attachment #1: Type: text/plain, Size: 592 bytes --]

On Thu, 02 Jul 2015 12:03:20 -0400, Stefan Monnier wrote:
> I don't use EWW very much, but I often use SHR via Gnus and I must say,
> I really like the new proportional-font rendering.

> Thanks Lars,

I've set shr-use-fonts to nil unwillingly because mm-shr fills
Japanese text into lines that are always longer than the window
width.  An example reproducing it is below.  Even for English
text (filled shorter than the window width), lines can get too
long depending on the font Emacs chooses.  Is there a way to help
it?  Setting shr-width is helpless since mm-shr overrides it.

Thanks.


[-- Attachment #2: Type: application/emacs-lisp, Size: 732 bytes --]

(let ((shr-use-fonts t)
      (temp (get-buffer-create "*shr-temp*"))
      (test (get-buffer-create "*shr-test*"))
      handle)
  (with-current-buffer temp
    (erase-buffer)
    (set-buffer-multibyte nil)
    (insert (encode-coding-string
	     "寿限無寿限無五劫の擦り切れ海砂利水魚の水行末雲来末風来末食う寝る処に住む処やぶら小路の藪柑子パイポパイポパイポのシューリンガンシューリンガンのグーリンダイグーリンダイのポンポコピーのポンポコナーの長久命の長助\n"
	     'utf-8)))
  (setq handle `(,temp ("text/html" (charset . "utf-8"))
		       nil nil nil nil nil nil))
  (pop-to-buffer test)
  (erase-buffer)
  (mm-shr handle))

Re: HTML rendering

[Eli Zaretskii]

> Date: Fri, 03 Jul 2015 10:45:14 +0900
> From: Katsumi Yamaoka <yamaoka@jpl.org>
> Cc: Lars Magne Ingebrigtsen <larsi@gnus.org>,
> 	Stefan Monnier <monnier@iro.umontreal.ca>
> 
> I've set shr-use-fonts to nil unwillingly because mm-shr fills
> Japanese text into lines that are always longer than the window
> width.  An example reproducing it is below.

Please in the future include in the example any libraries that need to
be loaded for it to work, for the benefit of those who try the example
in "emacs -Q".

> Even for English text (filled shorter than the window width), lines
> can get too long depending on the font Emacs chooses.  Is there a
> way to help it?  Setting shr-width is helpless since mm-shr
> overrides it.

You are supposed to set shr-width to nil, I think.  Did you try?

Anyway, looking at the example, it sounds like some off-by-one error
to me: always only one glyph exceeds the window width.  I suggest to
take a look at the filling calculations.

Re: HTML rendering

[Katsumi Yamaoka]

On Fri, 03 Jul 2015 09:24:19 +0300, Eli Zaretskii wrote:
> Please in the future include in the example any libraries that need to
> be loaded for it to work, for the benefit of those who try the example
> in "emacs -Q".

It requires (require 'gnus-art) to make it work with "emacs -Q",
sorry.

>> Even for English text (filled shorter than the window width), lines
>> can get too long depending on the font Emacs chooses.  Is there a
>> way to help it?  Setting shr-width is helpless since mm-shr
>> overrides it.

> You are supposed to set shr-width to nil, I think.  Did you try?

I tried 64 or something, that's smaller than the window width.
Though the mm-shr function overrides it with nil if shr-use-fonts
is non-nil.  And the default width

(- (window-pixel-width) (* (frame-fringe-width) 2))

will be used (see shr-insert-document).

The functions that fill text lines are shr-fill-line,
shr-vertical-motion, and so forth.  I think the cause filling
Japanese text fails is that shr-vertical-motion uses

(frame-char-width)

to decide the fill position.  It returns 8 in my case, however
the width of Japanses characters is 16.

The most easy way to fix this problem would be to remove let-
binding of shr-width in mm-shr.  Users who need it should set
shr-width of course[1].  Otherwise, it might be also good that
mm-shr binds shr-width to something other than nil.

Thanks.

[1] Setting shr-width to 64 leads long Japanese text to be filled
  in 66-character-width lines.

Re: HTML rendering

[Eli Zaretskii]

> Date: Mon, 06 Jul 2015 09:54:35 +0900
> From: Katsumi Yamaoka <yamaoka@jpl.org>
> Cc: emacs-devel@gnu.org, larsi@gnus.org, monnier@iro.umontreal.ca
> 
> I think the cause filling Japanese text fails is that
> shr-vertical-motion uses
> 
> (frame-char-width)
> 
> to decide the fill position.  It returns 8 in my case, however
> the width of Japanses characters is 16.

But then how to explain that only 1 character was overflowing the line
length?  If your explanation were correct, you'd have half the
characters spilled to the next line.

Re: HTML rendering

[Katsumi Yamaoka]

On Mon, 06 Jul 2015 05:38:08 +0300, Eli Zaretskii wrote:
> But then how to explain that only 1 character was overflowing the line
> length?  If your explanation were correct, you'd have half the
> characters spilled to the next line.

If you meant one Japanese character overflowing, simply move it
to the next line like the fill functions do, except for KINSOKU
characters.

FYI:
・Japanese text doesn't use space to separate words normally.
・Japanese text may be folded even at the middle of a word.
・KINSOKU processing is required.  cf. international/kinsoku.el
  Eg, the characters "。", "、", "ッ", "ゃ", "ゅ", "ょ", "ー",
  etc. should not be placed at the bol.  If fill-column is 6 or 7,
  "あいうえお" will be filled into "あいう\nえお", but
  "イーマックス" will be filled into "イーマッ\nクス".
  (I've added those rules to shr.el years ago.)

Re: HTML rendering

[Eli Zaretskii]

> Date: Mon, 06 Jul 2015 13:37:19 +0900
> From: Katsumi Yamaoka <yamaoka@jpl.org>
> Cc: emacs-devel@gnu.org, larsi@gnus.org, monnier@iro.umontreal.ca
> 
> > But then how to explain that only 1 character was overflowing the line
> > length?  If your explanation were correct, you'd have half the
> > characters spilled to the next line.
> 
> If you meant one Japanese character overflowing, simply move it
> to the next line like the fill functions do, except for KINSOKU
> characters.

No, that's not what I meant.  I meant that the problem cannot be
incorrect accounting for wide characters, because then there's no way
we can explain why lines are always exactly one character too long.

Anyway, going back to what you said before:

> The functions that fill text lines are shr-fill-line,
> shr-vertical-motion, and so forth.  I think the cause filling
> Japanese text fails is that shr-vertical-motion uses
> 
> (frame-char-width)
> 
> to decide the fill position.  It returns 8 in my case, however
> the width of Japanses characters is 16.

No, I don't think this is the root cause of the problem.
shr-vertical-motion calls vertical-motion, which internally does all
its calculations in pixels, by converting the columns into pixels
using the frame's canonical character width.  So dividing by the value
returned by frame-char-width is exactly the right thing.

> FYI:
> ・Japanese text doesn't use space to separate words normally.
> ・Japanese text may be folded even at the middle of a word.
> ・KINSOKU processing is required.  cf. international/kinsoku.el

I think this is the root cause of the problem: when the line is too
long, shr-fill-line attempts to find a place to break it, by looking
for whitespace characters.  But in your example, there are no such
characters, so shr simply punts and does nothing.

Lars, why don't you use functions in fill.el to do that?  They already
know about kinsoku, so should DTRT without you having to reinvent the
wheel.  Is something missing in fill.el to support this use case?

Re: HTML rendering

[Lars Magne Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> I think this is the root cause of the problem: when the line is too
> long, shr-fill-line attempts to find a place to break it, by looking
> for whitespace characters.  But in your example, there are no such
> characters, so shr simply punts and does nothing.

shr-fill-line uses kinsoku to find fill points...

There is a bug in shr in the way it computes available widths when using
table layouts -- there seems to be a off-by-one error somewhere that I
haven't been able to chase down.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: HTML rendering

[Michael Heerdegen]

Katsumi Yamaoka <yamaoka@jpl.org> writes:

> I've set shr-use-fonts to nil unwillingly because mm-shr fills
> Japanese text into lines that are always longer than the window
> width.

FWIW, I read German newspaper articles in eww with shr-use-fonts t, and
saw the same: the longest lines were always a bit too long, so that the
last character was out of view (using truncate-lines non-nil).

I think I didn't change any related variables, especially not shr-width.


Michael.

Re: HTML rendering

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Date: Mon, 06 Jul 2015 14:32:10 +0200
> Cc: Lars Magne Ingebrigtsen <larsi@gnus.org>,
> 	Stefan Monnier <monnier@iro.umontreal.ca>, emacs-devel@gnu.org
> 
> Katsumi Yamaoka <yamaoka@jpl.org> writes:
> 
> > I've set shr-use-fonts to nil unwillingly because mm-shr fills
> > Japanese text into lines that are always longer than the window
> > width.
> 
> FWIW, I read German newspaper articles in eww with shr-use-fonts t, and
> saw the same: the longest lines were always a bit too long, so that the
> last character was out of view (using truncate-lines non-nil).

A detailed bug report will be greatly appreciated, as usual.

If you can spare a few moments, please see what goes wrong inside
shr-fill-line in your case.  I'm guessing it's something different
from Yamaoka-san's case, i.e. you do have whitespace characters in
those lines.

Re: HTML rendering

[Michael Heerdegen]

Eli Zaretskii <eliz@gnu.org> writes:

> A detailed bug report will be greatly appreciated, as usual.

Yip.  It's just that I'm on vacation, and I'm not sure it's not
related to my config.

> If you can spare a few moments, please see what goes wrong inside
> shr-fill-line in your case.

I can surely spare some moments, though it can take some days.


Michael.

Re: HTML rendering

[Michael Heerdegen]

Eli Zaretskii <eliz@gnu.org> writes:

> A detailed bug report will be greatly appreciated, as usual.

Reported as bug#21012.

> If you can spare a few moments, please see what goes wrong inside
> shr-fill-line in your case.  I'm guessing it's something different
> from Yamaoka-san's case, i.e. you do have whitespace characters in
> those lines.

Seems it's related.  At least, my provisional fixes fix his example
case too.


Regards,

Michael.

Re: HTML rendering

[Lars Magne Ingebrigtsen]

Katsumi Yamaoka <yamaoka@jpl.org> writes:

> I've set shr-use-fonts to nil unwillingly because mm-shr fills
> Japanese text into lines that are always longer than the window
> width.  An example reproducing it is below. 

Your test case was filled correctly on this machine.

I suspect that there is a bug somewhere in `window-text-pixel-size' for
certain fonts.  For instance, it doesn't return the correct width for
the "emdash" character in the font I use on this machine.

I haven't found time to investigate this yet, so I haven't filed a bug
report.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: HTML rendering

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Date: Sun, 30 Aug 2015 13:47:26 +0200
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, emacs-devel@gnu.org
> 
> Katsumi Yamaoka <yamaoka@jpl.org> writes:
> 
> > I've set shr-use-fonts to nil unwillingly because mm-shr fills
> > Japanese text into lines that are always longer than the window
> > width.  An example reproducing it is below. 
> 
> Your test case was filled correctly on this machine.
> 
> I suspect that there is a bug somewhere in `window-text-pixel-size' for
> certain fonts.  For instance, it doesn't return the correct width for
> the "emdash" character in the font I use on this machine.
> 
> I haven't found time to investigate this yet, so I haven't filed a bug
> report.

Quite some water went under the bridge since then.  See bug#21012.  We
arrived at a patch there, but it looks like it wasn't committed yet.
Michael?

Re: HTML rendering

[Michael Heerdegen]

Eli Zaretskii <eliz@gnu.org> writes:

> Quite some water went under the bridge since then.  See bug#21012.  We
> arrived at a patch there, but it looks like it wasn't committed yet.
> Michael?

Yes, my fault, sorry!  I wanted to read and memorize all the committing
related rules before I'm committing anything.  Unfortunately I didn't do
that yet.

I'll commit the patch in the next three days - I promise that.


Sorry again,

Michael.

Re: HTML rendering

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Date: Sun, 30 Aug 2015 17:45:51 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Quite some water went under the bridge since then.  See bug#21012.  We
> > arrived at a patch there, but it looks like it wasn't committed yet.
> > Michael?
> 
> Yes, my fault, sorry!  I wanted to read and memorize all the committing
> related rules before I'm committing anything.  Unfortunately I didn't do
> that yet.
> 
> I'll commit the patch in the next three days - I promise that.

There's no rush, take your time.  I was just surprised not to see this
in the repository, but no harm done.

> Sorry again,

And nothing to be sorry about.

Thanks.

Re: HTML rendering

[Lars Magne Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> I don't use EWW very much, but I often use SHR via Gnus and I must say,
> I really like the new proportional-font rendering.

Thanks!

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no