"Readability" feature in eww

"Readability" feature in eww

[Lars Magne Ingebrigtsen]

It occurred to me that it would probably make sense for eww to have a
"readability" feature a couple of hours ago, so I implemented a take on
it and committed it.

The `R' command in eww will try to find the parts of the current page
where most of the text is, and only display that part.  This makes all
the menus and stuff disappear, and you don't have to page forever to
find the actual article on newspaper sites.

This is a heuristic, of course, so it can be tweaked endlessly.  The
current algorithm just gives most words a positive score, HTML markup a
negative score, and words inside <a> tags a negative score.  For such a
simple algorithm, it seems to give pretty good results.

But tweaking is necessary for it to be ... better.  If anybody has ideas
for tweaks or better algorithms, please be my guest and have at it.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: "Readability" feature in eww

[Yoni Rabkin]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> It occurred to me that it would probably make sense for eww to have a
> "readability" feature a couple of hours ago, so I implemented a take on
> it and committed it.
>
> The `R' command in eww will try to find the parts of the current page
> where most of the text is, and only display that part.  This makes all
> the menus and stuff disappear, and you don't have to page forever to
> find the actual article on newspaper sites.
>
> This is a heuristic, of course, so it can be tweaked endlessly.  The
> current algorithm just gives most words a positive score, HTML markup a
> negative score, and words inside <a> tags a negative score.  For such a
> simple algorithm, it seems to give pretty good results.
>
> But tweaking is necessary for it to be ... better.  If anybody has ideas
> for tweaks or better algorithms, please be my guest and have at it.

One thing I've found significantly increases readability for me is far
simpler: I disable color in `eww-tag-body'. Rendering color, especially
background color, doesn't necessarily make sense when you use a purely
textual display.

I always work with a dark background, so when Eww renders a full white
background and, say, light blue text the page becomes unreadable.

I could send in a patch which makes color rending in the body optional;
it's definitely how I will be using Eww going forward.

-- 
   "Cut your own wood and it will warm you twice"

Colour selection in shr (was: "Readability" feature in eww)

[Lars Magne Ingebrigtsen]

Yoni Rabkin <yoni@rabkins.net> writes:

> One thing I've found significantly increases readability for me is far
> simpler: I disable color in `eww-tag-body'. Rendering color, especially
> background color, doesn't necessarily make sense when you use a purely
> textual display.
>
> I always work with a dark background, so when Eww renders a full white
> background and, say, light blue text the page becomes unreadable.

If shr selects colours that are difficult to read, that's a bug.  It's
supposed to ensure sufficient colour distance to make things legible.
Do you have an example URL?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Colour selection in shr

[Yoni Rabkin]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Yoni Rabkin <yoni@rabkins.net> writes:
>
>> One thing I've found significantly increases readability for me is far
>> simpler: I disable color in `eww-tag-body'. Rendering color, especially
>> background color, doesn't necessarily make sense when you use a purely
>> textual display.
>>
>> I always work with a dark background, so when Eww renders a full white
>> background and, say, light blue text the page becomes unreadable.
>
> If shr selects colours that are difficult to read, that's a bug.  It's
> supposed to ensure sufficient colour distance to make things legible.
> Do you have an example URL?

For instance, see:
http://www.classicalnotes.net/classics3/appassionata.html

This renders for me with a bright white background and blue text. But
note that for me, anything which changes the background color of my
window breaks the way I use GNU/Emacs.

Let me motivate my above patch request/suggestion:

"Difficult" in this case is subjective and dependent on the way my eyes
work, or in my case
(http://en.wikipedia.org/wiki/Astigmatism_%28eye%29#Based_on_axis_of_the_principal_meridians),
the particular way my eyes don't work. I'm sure that a healthy person
can see the text, but for me the bright white background makes the text
effectively invisible (the background white "bleeds" into the letters).

In conclusion: I can't expect shr to select colors that work with my
eyes, and I don't know to say that there is any problem with how shr
selects colors. But I would like eww to allow me to say "I've set up
Emacs to work with my eyes, please don't touch the background and
foreground colors I've set up."

-- 
   "Cut your own wood and it will warm you twice"

Re: Colour selection in shr

[Lars Magne Ingebrigtsen]

Yoni Rabkin <yoni@rabkins.net> writes:

> For instance, see:
> http://www.classicalnotes.net/classics3/appassionata.html

Ok; looks reasonable to me.  The luminance could perhaps be pushed up a
bit...

But as you say:

> "Difficult" in this case is subjective and dependent on the way my eyes
> work, or in my case
> (http://en.wikipedia.org/wiki/Astigmatism_%28eye%29#Based_on_axis_of_the_principal_meridians),
> the particular way my eyes don't work. I'm sure that a healthy person
> can see the text, but for me the bright white background makes the text
> effectively invisible (the background white "bleeds" into the letters).

Emacs should definitely provide a way to switch colourisation off
completely for the users that want/need that.  And, come to think of it,
it should probably switch off that on TTYs, too, if the user hasn't
specified whether the background is `light' or `dark'.

A patch to shr.el to provide a defcustom to switch off everything
colourful would be appreciated.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Colour selection in shr

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Date: Mon, 03 Nov 2014 18:26:22 +0100
> Cc: emacs-devel@gnu.org
> 
> Emacs should definitely provide a way to switch colourisation off
> completely for the users that want/need that.  And, come to think of it,
> it should probably switch off that on TTYs, too, if the user hasn't
> specified whether the background is `light' or `dark'.

On some TTYs, Emacs can determine the background automatically.

Re: Colour selection in shr

[Achim Gratz]

Eli Zaretskii writes:
>> Emacs should definitely provide a way to switch colourisation off
>> completely for the users that want/need that.  And, come to think of it,
>> it should probably switch off that on TTYs, too, if the user hasn't
>> specified whether the background is `light' or `dark'.
>
> On some TTYs, Emacs can determine the background automatically.

Which are those?  Or are you talking about that hardcoded regex that
decides which terminals are 'light (which is wrong more often than not
and not customizable either).


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Terratec KOMPLEXER:
http://Synth.Stromeko.net/Downloads.html#KomplexerWaves

Re: Colour selection in shr

[Eli Zaretskii]

> From: Achim Gratz <Stromeko@nexgo.de>
> Date: Mon, 03 Nov 2014 19:20:09 +0100
> 
> > On some TTYs, Emacs can determine the background automatically.
> 
> Which are those?

xterm, for example.  See lisp/term/xterm.el (search for
"xterm--report-background-handler") for what I had in mind.

Re: Colour selection in shr

[Lars Magne Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> xterm, for example.  See lisp/term/xterm.el (search for
> "xterm--report-background-handler") for what I had in mind.

Is there a, like, top-level function that will return the background
colour of the terminal, if the terminal supports that?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Colour selection in shr

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Cc: Achim Gratz <Stromeko@nexgo.de>,  emacs-devel@gnu.org
> Date: Mon, 03 Nov 2014 20:23:51 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > xterm, for example.  See lisp/term/xterm.el (search for
> > "xterm--report-background-handler") for what I had in mind.
> 
> Is there a, like, top-level function that will return the background
> colour of the terminal, if the terminal supports that?

I must be missing something, because the obvious answer is this:

   (face-background 'default)

Re: Colour selection in shr

[Lars Magne Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> I must be missing something, because the obvious answer is this:
>
>    (face-background 'default)

Something may be obvious to you that's not obvious to others.  >"?

Anyway, I just tried that with emacs -nw -q in an xterm on Debian Stale,
and the result was "unspecified-bg".

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Colour selection in shr

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Cc: Stromeko@nexgo.de,  emacs-devel@gnu.org
> Date: Mon, 03 Nov 2014 20:42:26 +0100
> 
> >    (face-background 'default)
> 
> Something may be obvious to you that's not obvious to others.  >"?
> 
> Anyway, I just tried that with emacs -nw -q in an xterm on Debian Stale,
> and the result was "unspecified-bg".

Which AFAIU means your xterm is too old, and doesn't support the
(relatively modern) interface used by Emacs to interrogate the
background color.

Or maybe Debian (who are known for customizing the heck out of Emacs
behind your back) did something to xterm.el, like set
xterm-extra-capabilities to some non-default value...

Re: Colour selection in shr

[Lars Magne Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Or maybe Debian (who are known for customizing the heck out of Emacs
> behind your back) did something to xterm.el, like set
> xterm-extra-capabilities to some non-default value...

This is with Emacs from trunk.

Perhaps the xterm is too old:

[larsi@stories ~/src/emacs/trunk/lisp]$ xterm -v       
XTerm(278)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Colour selection in shr

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Cc: Stromeko@nexgo.de,  emacs-devel@gnu.org
> Date: Mon, 03 Nov 2014 21:34:56 +0100
> 
> Perhaps the xterm is too old:
> 
> [larsi@stories ~/src/emacs/trunk/lisp]$ xterm -v       
> XTerm(278)

According to my reading of lisp/term/xterm.el, version 242 or later
should support that.  But I'm not an expert on this.

Re: Colour selection in shr

[Yoni Rabkin]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Yoni Rabkin <yoni@rabkins.net> writes:
>
>> For instance, see:
>> http://www.classicalnotes.net/classics3/appassionata.html
>
> Ok; looks reasonable to me.  The luminance could perhaps be pushed up a
> bit...
>
> But as you say:
>
>> "Difficult" in this case is subjective and dependent on the way my eyes
>> work, or in my case
>> (http://en.wikipedia.org/wiki/Astigmatism_%28eye%29#Based_on_axis_of_the_principal_meridians),
>> the particular way my eyes don't work. I'm sure that a healthy person
>> can see the text, but for me the bright white background makes the text
>> effectively invisible (the background white "bleeds" into the letters).
>
> Emacs should definitely provide a way to switch colourisation off
> completely for the users that want/need that.  And, come to think of it,
> it should probably switch off that on TTYs, too, if the user hasn't
> specified whether the background is `light' or `dark'.
>
> A patch to shr.el to provide a defcustom to switch off everything
> colourful would be appreciated.

A patch to shr.el won't work. When called through Eww, title tags are
colorized by shr.el but body tags are colored by eww.el. So turning off
colorization in shr.el would still leave `eww-colorize-region' in eww
working.

So I'm hesitating a moment just in case you are thinking of moving all
of the Eww's coloration into either eww.el or shr.el.

-- 
   "Cut your own wood and it will warm you twice"

Re: Colour selection in shr

[Lars Magne Ingebrigtsen]

Yoni Rabkin <yoni@rabkins.net> writes:

> A patch to shr.el won't work. When called through Eww, title tags are
> colorized by shr.el but body tags are colored by eww.el. So turning off
> colorization in shr.el would still leave `eww-colorize-region' in eww
> working.

I don't know what you mean by "title tags" and "body tags" here.

Anyway, eww can do stuff based on shr settings.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Colour selection in shr

[Yoni Rabkin]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Yoni Rabkin <yoni@rabkins.net> writes:
>
>> A patch to shr.el won't work. When called through Eww, title tags are
>> colorized by shr.el but body tags are colored by eww.el. So turning off
>> colorization in shr.el would still leave `eww-colorize-region' in eww
>> working.
>
> I don't know what you mean by "title tags" and "body tags" here.
>
> Anyway, eww can do stuff based on shr settings.

A defcustom in shr.el which stops shr from coloring stuff won't make eww
stop coloring everything. This is because eww.el:eww-colorize-region
colors stuff independently of shr.

To repeat: eww.el:eww-colorize-region is eww's own private copy of
shr.el:shr-colorize-region.

-- 
   "Cut your own wood and it will warm you twice"

Re: Colour selection in shr

[Lars Magne Ingebrigtsen]

Yoni Rabkin <yoni@rabkins.net> writes:

> A defcustom in shr.el which stops shr from coloring stuff won't make eww
> stop coloring everything.

Why not?  eww can look at shr variables just as much as shr can.

> To repeat: eww.el:eww-colorize-region is eww's own private copy of
> shr.el:shr-colorize-region.

Looks like a something left over for hysterical raisins.  I've now
removed the eww version of the function.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Colour selection in shr

[Yoni Rabkin]

[-- Attachment #1: Type: text/plain, Size: 533 bytes --]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Yoni Rabkin <yoni@rabkins.net> writes:
>
>> A defcustom in shr.el which stops shr from coloring stuff won't make eww
>> stop coloring everything.
>
> Why not?  eww can look at shr variables just as much as shr can.
>
>> To repeat: eww.el:eww-colorize-region is eww's own private copy of
>> shr.el:shr-colorize-region.
>
> Looks like a something left over for hysterical raisins.  I've now
> removed the eww version of the function.

In which case, can we use something like this?


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-inhibit-coloration.patch --]
[-- Type: text/x-diff, Size: 1491 bytes --]

From 7b69e65059c91d93ebb82d22b8d082ad35522cb7 Mon Sep 17 00:00:00 2001
From: Yoni Rabkin <yrk@gnu.org>
Date: Mon, 3 Nov 2014 20:59:08 -0500
Subject: [PATCH] Add option to inhibit coloration.

---
 lisp/ChangeLog  |    3 +++
 lisp/net/shr.el |    7 +++++++
 2 files changed, 10 insertions(+)

diff --git a/lisp/ChangeLog b/lisp/ChangeLog
index ac78e34..9d104ed 100644
--- a/lisp/ChangeLog
+++ b/lisp/ChangeLog
@@ -9,6 +9,9 @@
 
 	* net/shr.el: Ditto.
 
+	* net/shr.el: Add option to inhibit color rendering (patch by Yoni
+	Rabkin yrk@gnu.org).
+
 2014-11-03  Stefan Monnier  <monnier@iro.umontreal.ca>
 
 	* emacs-lisp/edebug.el (edebug-safe-prin1-to-string): Assume that
diff --git a/lisp/net/shr.el b/lisp/net/shr.el
index 5db0324..ccaba20 100644
--- a/lisp/net/shr.el
+++ b/lisp/net/shr.el
@@ -106,6 +106,12 @@ Alternative suggestions are:
   :group 'shr
   :type 'boolean)
 
+(defcustom shr-inhibit-coloration nil
+  "Non nil means that colors will not be rendered."
+  :version "24.4"
+  :group 'shr
+  :type 'boolean)
+
 (defvar shr-content-function nil
   "If bound, this should be a function that will return the content.
 This is used for cid: URLs, and the function is called with the
@@ -920,6 +926,7 @@ ones, in case fg and bg are nil."
 
 (defun shr-colorize-region (start end fg &optional bg)
   (when (and (not shr-inhibit-decoration)
+	     (not shr-inhibit-coloration)
 	     (or fg bg))
     (let ((new-colors (shr-color-check fg bg)))
       (when new-colors
-- 
1.7.9.5


[-- Attachment #3: Type: text/plain, Size: 56 bytes --]



-- 
   "Cut your own wood and it will warm you twice"

Re: Colour selection in shr

[Stefan Monnier]

> +(defcustom shr-inhibit-coloration nil

Again: why not use Emacs's standard way to specify this,
i.e. `font-lock-mode'?


        Stefan

Re: Colour selection in shr

[Lars Magne Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> +(defcustom shr-inhibit-coloration nil
>
> Again: why not use Emacs's standard way to specify this,
> i.e. `font-lock-mode'?

Again, I have no idea what you mean by that.  How would you use
`font-lock-mode' to put colours on an HTML document based on CSS?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Colour selection in shr

[Stefan Monnier]

> Again, I have no idea what you mean by that.  How would you use
> `font-lock-mode' to put colours on an HTML document based on CSS?

I'm not suggesting to use font-lock-keywords to put the colors.
I'm suggesting to use the `font-lock-mode' setting (a boolean variable)
to decide whether to do highlighting or not.

You could even do it dynamically/automatically by using the
`font-lock-face' instead of the `face' property.


        Stefan

Re: Colour selection in shr

[Lars Magne Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Again, I have no idea what you mean by that.  How would you use
>> `font-lock-mode' to put colours on an HTML document based on CSS?
>
> I'm not suggesting to use font-lock-keywords to put the colors.
> I'm suggesting to use the `font-lock-mode' setting (a boolean variable)
> to decide whether to do highlighting or not.

Oh, just look at the `font-lock-mode' variable?  Sure, if that's the
way that variable is meant to be used.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Colour selection in shr

[Yoni Rabkin]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Again, I have no idea what you mean by that.  How would you use
>> `font-lock-mode' to put colours on an HTML document based on CSS?
>
> I'm not suggesting to use font-lock-keywords to put the colors.
> I'm suggesting to use the `font-lock-mode' setting (a boolean variable)
> to decide whether to do highlighting or not.
>
> You could even do it dynamically/automatically by using the
> `font-lock-face' instead of the `face' property.

Doesn't that mean I have to set font-lock-mode to nil in each buffer
which uses shr/eww?

I thought that font-lock-mode is buffer local, and what I'm looking for
is to disable colorization in any future use of shr/eww in my emacs
instance.

-- 
   "Cut your own wood and it will warm you twice"

Re: Colour selection in shr

[Stefan Monnier]

> Doesn't that mean I have to set font-lock-mode to nil in each buffer
> which uses shr/eww?

Yes.  You'll most likely want to do it from a hook.


        Stefan

Re: Colour selection in shr

[Wolfgang Jenkner]

On Tue, Nov 04 2014, Stefan Monnier wrote:

>> Again, I have no idea what you mean by that.  How would you use
>> `font-lock-mode' to put colours on an HTML document based on CSS?
>
> I'm not suggesting to use font-lock-keywords to put the colors.
> I'm suggesting to use the `font-lock-mode' setting (a boolean variable)
> to decide whether to do highlighting or not.
>
> You could even do it dynamically/automatically by using the
> `font-lock-face' instead of the `face' property.

The thread was about turning off colours while preserving other face
attributes, I think.  But using `font-lock-face' for colour settings and
`face' for the rest doesn't work for this, viz. with the following code
snippet, "bar" appears underlined but not coloured.

(with-current-buffer (generate-new-buffer "*test*")
  (insert "foobarquux")
  (put-text-property 1 7 'face '(:underline t))
  (put-text-property 4 11 'font-lock-face '(:foreground "red"))
  (font-lock-mode)
  (pop-to-buffer (current-buffer)))

Wolfgang

Re: Colour selection in shr

[Stefan Monnier]

> Emacs should definitely provide a way to switch colourisation off
> completely for the users that want/need that.

Doing it via `font-lock-mode' would seem like the obvious way.


        Stefan

Re: Colour selection in shr

[Lars Magne Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Emacs should definitely provide a way to switch colourisation off
>> completely for the users that want/need that.
>
> Doing it via `font-lock-mode' would seem like the obvious way.

Switching off shr colourisation via `font-lock-mode' is the obvious way?

I think you'll have to expand that thought a bit...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Colour selection in shr

[Yoni Rabkin]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>>> Emacs should definitely provide a way to switch colourisation off
>>> completely for the users that want/need that.
>>
>> Doing it via `font-lock-mode' would seem like the obvious way.
>
> Switching off shr colourisation via `font-lock-mode' is the obvious way?
>
> I think you'll have to expand that thought a bit...

This discussion, at large, went places that don't help me directly in
any way. So in the meantime I'll be setting shr-colorize-region to a NOP
on my machine until that breaks something.

-- 
   "Cut your own wood and it will warm you twice"

Re: Colour selection in shr

[Lars Magne Ingebrigtsen]

Yoni Rabkin <yoni@rabkins.net> writes:

> This discussion, at large, went places that don't help me directly in
> any way. So in the meantime I'll be setting shr-colorize-region to a NOP
> on my machine until that breaks something.

I think we (some of us, at least >"?) agreed that there should be a way
for the user to say "all colours in this buffer should go away", and
that the most practical way to do this would be to introduce a
buffer-local variable (for instance, `inhibit-color') that would be
checked by the Emacs redisplay.  If it's set, then all face `foreground'
and `background' specs would just be ignored.

This would do exactly what people who need it (because of visual
problems) or want it (because they just do) require, without touching on
the other markup issues (for instance, we don't want links to lose their
underlining).

If someone were to create a patch to do that, I'd apply it.  >"?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: "Readability" feature in eww

[Rüdiger Sonderfeld]

On Monday 03 November 2014 01:41:14 Lars Magne Ingebrigtsen wrote:
> This is a heuristic, of course, so it can be tweaked endlessly.  The
> current algorithm just gives most words a positive score, HTML markup a
> negative score, and words inside <a> tags a negative score.  For such a
> simple algorithm, it seems to give pretty good results.
> 
> But tweaking is necessary for it to be ... better.  If anybody has ideas
> for tweaks or better algorithms, please be my guest and have at it.

HTML5 has introduced tags such as <main> and <article>, which can be used to 
identify the important parts.  I'm not sure how widespread their use thus far 
is (I think org-mode supports it already if one sets the HTML5 export option).  
But at least adding them to the heuristic might help.

E.g., https://developer.mozilla.org/en-US/docs/Web/HTML/Element/main

Regards,
Rüdiger

Re: "Readability" feature in eww

[Rasmus]

Rüdiger Sonderfeld <ruediger@c-plusplus.de> writes:

> On Monday 03 November 2014 01:41:14 Lars Magne Ingebrigtsen wrote:
>> This is a heuristic, of course, so it can be tweaked endlessly.  The
>> current algorithm just gives most words a positive score, HTML markup a
>> negative score, and words inside <a> tags a negative score.  For such a
>> simple algorithm, it seems to give pretty good results.
>> 
>> But tweaking is necessary for it to be ... better.  If anybody has ideas
>> for tweaks or better algorithms, please be my guest and have at it.
>
> HTML5 has introduced tags such as <main> and <article>, which can be used to 
> identify the important parts.  I'm not sure how widespread their use thus far 
> is
> 
> (I think org-mode supports it already if one sets the HTML5 export option).  

Indeed, but html5 is not default.  As far as I remember you'd have to
wrap your article part in #+begin_article ⋯ #+end_article.  There was
a dicussion at some point, and there were some good html5-reasons why
the body is not wrappend in article by default.

—Rasmus

-- 
. . . The proofs are technical in nature and provides no real understanding

Re: "Readability" feature in eww

[Lars Magne Ingebrigtsen]

Rüdiger Sonderfeld <ruediger@c-plusplus.de> writes:

> HTML5 has introduced tags such as <main> and <article>, which can be
> used to identify the important parts.  I'm not sure how widespread
> their use thus far is (I think org-mode supports it already if one
> sets the HTML5 export option).  But at least adding them to the
> heuristic might help.

I kinda doubt that.  Even if you disregard the fact that semantic markup
will never happen because of reasons, newspapers (and the like) have
different goals than their readers have.

We want to read the article.  The web site owners want to 1) present us
with as many ads as possible, and 2) lots of new links that we can click
on, so that they can (see 1).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: "Readability" feature in eww

[Rasmus]

Sounds great with the Readability key!

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> But tweaking is necessary for it to be ... better.  If anybody has ideas
> for tweaks or better algorithms, please be my guest and have at it.

I would love a hook so that I could do something like:

  (while (<= (point) (point-max))
         (fill-paragraph) (forward-paragraph))

Since by default line length seems to be a function of Emacs's window
width.

A built-in algorithm would maybe be better as it could try to
determine if whatever-at-point is a text paragraph (versus a link or a
table or whatever).

Thanks,
Rasmus

-- 
To err is human. To screw up 10⁶ times per second, you need a computer

Re: "Readability" feature in eww

[Lars Magne Ingebrigtsen]

Rasmus <rasmus@gmx.us> writes:

> I would love a hook so that I could do something like:
>
>   (while (<= (point) (point-max))
>          (fill-paragraph) (forward-paragraph))
>
> Since by default line length seems to be a function of Emacs's window
> width.

I'm not sure what you mean.  shr fills paragraphs automatically.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: "Readability" feature in eww

[Rasmus]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Rasmus <rasmus@gmx.us> writes:
>
>> I would love a hook so that I could do something like:
>>
>>   (while (<= (point) (point-max))
>>          (fill-paragraph) (forward-paragraph))
>>
>> Since by default line length seems to be a function of Emacs's window
>> width.
>
> I'm not sure what you mean.  shr fills paragraphs automatically.

Right, I see that in emacs -q.  I must have some variable in my
init.el that disables this.  I have no user variables starting with
neither eww- nor shr-, so I will have to dig through it.

—Rasmus

-- 
Lasciate ogni speranza o voi che entrate: siete nella mani di'machellaio

Re: "Readability" feature in eww

[Óscar Fuentes]

Rasmus <rasmus@gmx.us> writes:

> I would love a hook so that I could do something like:
>
>   (while (<= (point) (point-max))
>          (fill-paragraph) (forward-paragraph))
>
> Since by default line length seems to be a function of Emacs's window
> width.

Something like this hopefully would fix
http://debbugs.gnu.org/cgi/bugreport.cgi?bug=17414

Re: "Readability" feature in eww

[raman]

Nice:-) Incidentally I've used tricks like this with Emacspeak to read
Web content efficiently for a long time, so I can vouch for its
efficacy.

Also, see some of what I implemented atop eww described here:

http://emacspeak.blogspot.com/2014/05/emacspeak-eww-updates-for-complete.html
Code is in
emacspeak-eww.el in emacspeak.googlecode.com/svn

I basically implemented the ability  to filter the parsed HTML tree by
elements,  attributes such as id, class and role, and the ability to
navigate via key elements.

Re: "Readability" feature in eww

[David Engster]

Lars Magne Ingebrigtsen writes:
> The `R' command in eww will try to find the parts of the current page
> where most of the text is, and only display that part.  This makes all
> the menus and stuff disappear, and you don't have to page forever to
> find the actual article on newspaper sites.
>
> This is a heuristic, of course, so it can be tweaked endlessly.  The
> current algorithm just gives most words a positive score, HTML markup a
> negative score, and words inside <a> tags a negative score.  For such a
> simple algorithm, it seems to give pretty good results.
>
> But tweaking is necessary for it to be ... better.  If anybody has ideas
> for tweaks or better algorithms, please be my guest and have at it.

I've looked into this a bit years ago when I was coding on emacs-w3m's
'shimbun' feature for Gnus. I took a peek at the algorithm which was
used for the 'boilerplate' library[1], but never got around implementing
it. Since I mostly needed it for reading blogs, I coded a quick solution
which looks at the 'generator' meta-tag and extracts the main content
for CMS like Wordpress, Typepad or Blogspot/Blogger, which was already
enough for me.

It'd be great if you could make this extraction method flexible, similar
to the 'washing' feature from Gnus, so that users could hook their own
methods for extracting the main content into eww. The user would provide
an extraction function and the corresponding regexp that matches against
the URL, or optionally also against the source to match things like the
'generator' meta-tag.

-David

[1] http://www.l3s.de/~kohlschuetter/publications/wsdm187-kohlschuetter.pdf

Re: "Readability" feature in eww

[Lars Magne Ingebrigtsen]

David Engster <deng@randomsample.de> writes:

> It'd be great if you could make this extraction method flexible, similar
> to the 'washing' feature from Gnus, so that users could hook their own
> methods for extracting the main content into eww. The user would provide
> an extraction function and the corresponding regexp that matches against
> the URL, or optionally also against the source to match things like the
> 'generator' meta-tag.

Well, the best is if we find a solution that works out of the box,
because then all users can just, like, use it.  >"?

The current heuristics are probably too simple, but I've been going
through bunches of pages, and it already seems to basically do what
you'd expect it to do, which surprises me, sort of.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: "Readability" feature in eww

[David Engster]

Lars Magne Ingebrigtsen writes:
> David Engster <deng@randomsample.de> writes:
>
>> It'd be great if you could make this extraction method flexible, similar
>> to the 'washing' feature from Gnus, so that users could hook their own
>> methods for extracting the main content into eww. The user would provide
>> an extraction function and the corresponding regexp that matches against
>> the URL, or optionally also against the source to match things like the
>> 'generator' meta-tag.
>
> Well, the best is if we find a solution that works out of the box,
> because then all users can just, like, use it.  >"?
>
> The current heuristics are probably too simple, but I've been going
> through bunches of pages, and it already seems to basically do what
> you'd expect it to do, which surprises me, sort of.

Well, users might want to even automatically load another version of the
page. This is actually quite common in emacs-w3m's shimbun library. For
instance, many sites break their articles among several pages to
generate more clicks, but also provide a printable version where
everything is on one, which the shimbun then automatically loads.

-David

Re: "Readability" feature in eww

[Lars Magne Ingebrigtsen]

David Engster <deng@randomsample.de> writes:

> Well, users might want to even automatically load another version of the
> page. This is actually quite common in emacs-w3m's shimbun library. For
> instance, many sites break their articles among several pages to
> generate more clicks, but also provide a printable version where
> everything is on one, which the shimbun then automatically loads.

How does shimbun find the one-page version of the article?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: "Readability" feature in eww

[David Engster]

Lars Magne Ingebrigtsen writes:
> David Engster <deng@randomsample.de> writes:
>
>> Well, users might want to even automatically load another version of the
>> page. This is actually quite common in emacs-w3m's shimbun library. For
>> instance, many sites break their articles among several pages to
>> generate more clicks, but also provide a printable version where
>> everything is on one, which the shimbun then automatically loads.
>
> How does shimbun find the one-page version of the article?

There's no easy way, you have to put that into code. There are many
shimbuns for all kinds of different sites. This is why I think it'd be
good if that 'readability' feature would be customizable, based on the
URL.

-David

Re: "Readability" feature in eww

[raman]

In the emacspeak-eww module, I implemented the following -- if it's more
generally useful to eww, I'd be happy to contribute it  to eww.
http://emacspeak.googlecode.com/svn/trunk/lisp/emacspeak-eww.el

That file uses folding-mode -- and the section pertaining to filtering
could likely just be ripped out from there and moved to eww.el --

Filtering:

1. Filter by element or elements
2. Filter by class value or values
3. Filter by id value / values
4. Filter by "role" value/values (role was introduced for Accessibility
in HTML and gets used increasingly eg role=main vs role=nav

The above lets me filter pages interactively; once I find a suitable
recipe that works, I usually code it up as a per-site wizard --

--Raman

Re: "Readability" feature in eww

[Yuri Khan]

On Tue, Nov 4, 2014 at 9:49 PM, Lars Magne Ingebrigtsen <larsi@gnus.org> wrote:

> How does shimbun find the one-page version of the article?

In an ideal world, it would be referenced with a <link rel="alternate"
type="text/html" media="print" href="…">. Practically, nobody cares :(