inclusion of emacs-w3m

inclusion of emacs-w3m

[Ivan Kanis]

Hello,

emacs-w3m is used with Gnus, newsticker and probably other packages.
Wouldn't it be useful to include it in emacs?
-- 
Ivan Kanis
http://ivan.kanis.fr

Knowledge is of two kinds. We know a subject ourselves or we know
where we can find information upon it.
    -- Samuel Johnson

Re: inclusion of emacs-w3m

[Lars Ingebrigtsen]

Ivan Kanis <ivan.kanis@googlemail.com> writes:

> emacs-w3m is used with Gnus, newsticker and probably other packages.
> Wouldn't it be useful to include it in emacs?

The main problem with emacs-w3m is that it depends on the external w3m
program, so it's an additional hassle.

Now that Emacs proper has an HTML parser (via libxml2), an HTML renderer
(via shr.el), and an HTTP library (via url*.el), writing a totally
Emacs-based web browser should be pretty easy for somebody who has some
time to spare.

Which, unfortunately, rules me out, for the time being.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  http://lars.ingebrigtsen.no  *  Sent from my Emacs

Re: inclusion of emacs-w3m

[Ivan Kanis]

Lars Ingebrigtsen <larsi@gnus.org> a écrit

> Ivan Kanis <ivan.kanis@googlemail.com> writes:
>
>> emacs-w3m is used with Gnus, newsticker and probably other packages.
>> Wouldn't it be useful to include it in emacs?
>
> The main problem with emacs-w3m is that it depends on the external w3m
> program, so it's an additional hassle.

Well w3m is available on Un*x, Windows and Mac so I think the external
binary is not much of a hassle.

> Now that Emacs proper has an HTML parser (via libxml2), an HTML renderer
> (via shr.el), and an HTTP library (via url*.el), writing a totally
> Emacs-based web browser should be pretty easy for somebody who has some
> time to spare.

I don't think it's that easy.
-- 
Ivan Kanis
http://ivan.kanis.fr

L'ennui c'est que j'ai raison. Il faut avouer que ce n'est pas gai.
    -- Jean Paulhan

Re: inclusion of emacs-w3m

[Óscar Fuentes]

Ivan Kanis <ivan.kanis@googlemail.com> writes:

> Well w3m is available on Un*x, Windows and Mac so I think the external
> binary is not much of a hassle.

Just for the record: the w3m FAQ says that it requires Cygwin on
Windows, and that is a hassle.

Re: inclusion of emacs-w3m

[Stefan Monnier]

> emacs-w3m is used with Gnus, newsticker and probably other packages.
> Wouldn't it be useful to include it in Emacs?

Yes.  Would you like to help us collect the needed paperwork?


        Stefan

Re: inclusion of emacs-w3m

[Lars Ingebrigtsen]

Ivan Kanis <ivan.kanis@googlemail.com> writes:

>> Now that Emacs proper has an HTML parser (via libxml2), an HTML renderer
>> (via shr.el), and an HTTP library (via url*.el), writing a totally
>> Emacs-based web browser should be pretty easy for somebody who has some
>> time to spare.
>
> I don't think it's that easy.

Why not?  You just need to add some bookmarking stuff and a cookie
editor, and you're done.

Oh, and forms support.  And stuff.  It shouldn't take anybody more than
a month to implement a full-featured Emacs browser.  (Well.  As
full-featured as a totally non-JS browser will ever be.)

-- 
(domestic pets only, the antidote for overdose, milk.)
  http://lars.ingebrigtsen.no  *  Lars Magne Ingebrigtsen

Re: inclusion of emacs-w3m

[Andrey Kotlarski]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Ivan Kanis <ivan.kanis@googlemail.com> writes:
>
>>> Now that Emacs proper has an HTML parser (via libxml2), an HTML renderer
>>> (via shr.el), and an HTTP library (via url*.el), writing a totally
>>> Emacs-based web browser should be pretty easy for somebody who has some
>>> time to spare.
>>
>> I don't think it's that easy.
>
> Why not?  You just need to add some bookmarking stuff and a cookie
> editor, and you're done.

There is w3, though unmaintained for years.  Until some threading
support comes to Elisp, such packages would suffer.

> Oh, and forms support.  And stuff.  It shouldn't take anybody more than
> a month to implement a full-featured Emacs browser.  (Well.  As
> full-featured as a totally non-JS browser will ever be.)

If the rumours of Guile Elisp nearing working integration with GNU/Emacs
are right and as Guile has support for Javascript too (not to mention
native threads), that could open some doors.


-- 
It is better to keep your mouth shut and be thought a fool than to
open it and remove all doubt.
		-- Mark Twain

Re: inclusion of emacs-w3m

[Tim Cross]

On 6 September 2012 07:44, Andrey Kotlarski <m00naticus@gmail.com> wrote:
> Lars Ingebrigtsen <larsi@gnus.org> writes:
>
>> Ivan Kanis <ivan.kanis@googlemail.com> writes:
>>
>>>> Now that Emacs proper has an HTML parser (via libxml2), an HTML renderer
>>>> (via shr.el), and an HTTP library (via url*.el), writing a totally
>>>> Emacs-based web browser should be pretty easy for somebody who has some
>>>> time to spare.
>>>
>>> I don't think it's that easy.
>>
>> Why not?  You just need to add some bookmarking stuff and a cookie
>> editor, and you're done.
>
> There is w3, though unmaintained for years.  Until some threading
> support comes to Elisp, such packages would suffer.
>
>> Oh, and forms support.  And stuff.  It shouldn't take anybody more than
>> a month to implement a full-featured Emacs browser.  (Well.  As
>> full-featured as a totally non-JS browser will ever be.)
>
> If the rumours of Guile Elisp nearing working integration with GNU/Emacs
> are right and as Guile has support for Javascript too (not to mention
> native threads), that could open some doors.
>
>
> --
> It is better to keep your mouth shut and be thought a fool than to
> open it and remove all doubt.
>                 -- Mark Twain
>
>

I think Lars' suggestion is a good one. There are lots of very cool
things you can do with an all elisp web browser. The emacspeak package
has made some very interesting use of w3, but the lack of work on w3
is beginning to cause some problems.

w3m is useful, but limited compared to what you can do with a browser
all written in elisp. I've looked at the w3 code base a number of
times, but the work required to re-factor it is quite daunting. The
addition of a parser and renderer in emacs certainly offers some
excellent opportunities. If I had more knowledge re: DOM, CSS etc, it
would be something I'd love to have time to look at. Would certainly
try to assist anyone who was thinking of developing something in this
are where I can. Assuming a browser with no javascript support, the
challenge is likely to be how to implement a workable subset of CSS
support to enable a reasonable rendering of pages. This is one area
where w3 seems to be quite broken.

The Guile approach would appear to offer some really great
opportunities, but after so many years, I will wait until we actually
see an emacs with support for guile which is as integrated as elisp
is.

In the meantime, my suggestion would be for any package maintainers
who have requirements for basic display of html content to consider
using native elisp packages over relying on things like w3m.el. While
w3m.el gives a solution to things like viewing HTML formatted email,
native elsip support would allow the sort of customizations we have
grown to love.

If the appropriate paperwork and licensing can be worked out to add
w3m.el as an elpa package, I don't see any problems with that - it
would provide at lest an option for those on platforms where it makes
sense.

-- 
Tim Cross

Re: inclusion of emacs-w3m

[Stefan Schlee]

[-- Attachment #1: Type: text/plain, Size: 1032 bytes --]

This posting may miss the spirit of this thread but anyhow:

Have you tried using Conkeror (http://www.conkeror.org/)?

As far as I understand this is based on xulrunner, but it behaves like Emacs.

Because it is based on xulrunner it is a fully featured HTML-browser (Java Script, CSS, cookies, forms etc. support).  Normally I have an Emacs instance and Conkeror instance open, and because it behaves like Emacs I sometimes forget that I am interacting with the browser.

Wouldn´t it be much more in the UNIX-kind of spirit to not reinvent the wheel but to use and combine the tools all ready existing?  Which means would it not be better to work on a really good interface between Emacs and Conkerer and relieve yourself of the work involving the maintenance of top notch Java Script, CSS interpreters etc..  For me it would be enough to have excellent access to the html-source, the DOM-tree, Java Script code, CSS files, cookies etc. in an Emacs buffers, and have the page rendered by Conkeror.

Kind regards Stefan

[-- Attachment #2: Type: text/html, Size: 1214 bytes --]

Re: inclusion of emacs-w3m

[Tim Cross]

On 6 September 2012 18:33, Stefan Schlee <stefan_schlee@yahoo.com> wrote:
> This posting may miss the spirit of this thread but anyhow:
>
> Have you tried using Conkeror (http://www.conkeror.org/)?
>
> As far as I understand this is based on xulrunner, but it behaves like
> Emacs.
>
> Because it is based on xulrunner it is a fully featured HTML-browser (Java
> Script, CSS, cookies, forms etc. support).  Normally I have an Emacs
> instance and Conkeror instance open, and because it behaves like Emacs I
> sometimes forget that I am interacting with the browser.
>
> Wouldn´t it be much more in the UNIX-kind of spirit to not reinvent the
> wheel but to use and combine the tools all ready existing?  Which means
> would it not be better to work on a really good interface between Emacs and
> Conkerer and relieve yourself of the work involving the maintenance of top
> notch Java Script, CSS interpreters etc..  For me it would be enough to have
> excellent access to the html-source, the DOM-tree, Java Script code, CSS
> files, cookies etc. in an Emacs buffers, and have the page rendered by
> Conkeror.
>
> Kind regards Stefan

That approach can have some benefits. A while back, I used some elisp
packages and firefox plugins that gave that sort of close interface -
essentially, access to the lower level stuff in emacs buffers and
reliance on firefox to do the rendering. You even had arepl that you
could interact with from within emacs that would manipulate firefox.
It was quite good, but not as good in some situations as the sort of
stuff you can do when everything is in elisp. If you have a look at
some of the stuff T. V. Raman has done with emacspeak and w3 you can
sort of get an idea of the potential.

To some extent, it gives you an interactive environment where you can
explore with new ideas regarding how you interact with HTML based
content. I may be a little outside the normal requirement space for
what most people want in this area. For me, rendering of HTML, CSS etc
is only part of the picture. What I would really like is the ability
to really impact/affect how the content is processed and handled. I
want an interactive environment where I can work on new content
handlers, content transformations via xslt, object extraction etc, but
do it in a convenient environment I am comfortable with i.e.
emacs/elisp.

Tim

-- 
Tim Cross

Re: inclusion of emacs-w3m

[joakim]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Ivan Kanis <ivan.kanis@googlemail.com> writes:
>
>> emacs-w3m is used with Gnus, newsticker and probably other packages.
>> Wouldn't it be useful to include it in emacs?
>
> The main problem with emacs-w3m is that it depends on the external w3m
> program, so it's an additional hassle.
>
> Now that Emacs proper has an HTML parser (via libxml2), an HTML renderer
> (via shr.el), and an HTTP library (via url*.el), writing a totally
> Emacs-based web browser should be pretty easy for somebody who has some
> time to spare.
>
> Which, unfortunately, rules me out, for the time being.  :-)

Can I shamelessly plug my emacs xwidget branch here?
It has webkit integration and stuff. One of my long term ideas is to
make it possible to expose the DOM so that SHR can render the same tree
webkit sees. There are some initial efforts in that direction in the
branch. There are some pros and cons to this of course.



-- 
Joakim Verona

Re: inclusion of emacs-w3m

[Ivan Kanis]

Stefan Monnier <monnier@iro.umontreal.ca> wrote:

>> emacs-w3m is used with Gnus, newsticker and probably other packages.
>> Wouldn't it be useful to include it in Emacs?
>
> Yes.  Would you like to help us collect the needed paperwork?

Yes I can. Could you, please, give me a copy of what I need to send out?

Take care,
-- 
Ivan Kanis
http://ivan.kanis.fr

Our whole economy is base on planned obsolescence.
    -- Brooks Stevens

Re: inclusion of emacs-w3m

[Thien-Thi Nguyen]

[-- Attachment #1: Type: text/plain, Size: 3348 bytes --]

() Lars Ingebrigtsen <larsi@gnus.org>
() Wed, 05 Sep 2012 16:27:50 +0200

   Oh, and forms support.  And stuff.  It shouldn't take anybody more
   than a month to implement a full-featured Emacs browser.  (Well.  As
   full-featured as a totally non-JS browser will ever be.)

Don't forget decent table rendering (especially, nested tables), one of
w3m's distinguishing features.  From /usr/share/doc/w3m/STORY.html:

  Table rendering algorithm in w3m                                                            
                                                                                            
  HTML table rendering is difficult.  Tabular environment of LaTeX is not
  very difficult, which makes the width of a column either a specified
  value or the maximum width to put items into it.  On the other hand,
  HTML table renderer has to decide the width of a column so that the
  entire table can fit into the display appropriately, and fold the
  contents of the table according to the column width.  Inappropriate
  column width decision makes the table ugly.  Moreover, table can be
  nested, which makes the algorithm more complicated.
                                                                                            
  1. First, calculate the maximum and minimum width of each column.  The
     maximum width is the width required to display the column without
     folding the contents.  Generally, it is the length of paragraph
     delimited by <BR> or <P>.  The minimum width is the lower limit to
     display the contents.  If the column contains the word
     `internationalization', the minimum width will be 20.  If the column
     contains <pre>..</pre>, the maximum width of the preformatted text
     will be the minimum width of the column.

  2. If the width of the column is specified by WIDTH attribute, fix the
     column width using that value.  If the specified width is smaller
     than the minimum width of the column, fix the column width to the
     minimum width.

  3. Calculate the sum of the maximum width (or fixed width) of each
     column and check if the sum exceeds the screen width.  If it is
     smaller than screen width, these values are used for width of each
     column.

  4. If the sum is larger than the screen width, determine the widths of
     each column according to the following steps.

     1. Let W be the screen width subtracted by the sum of widths of
        fixed-width columns.

     2. Distribute W into the columns whose width are not decided, in
        proportion to the logarithm of the maximum width of each column.

     3. If the distributed width of a column is smaller than the minimum
        width, then fix the width of the column to the minimum width,
        and do the distribution again.
                                    
So, maybe another week or two for the Inspired One...  who will it be?

[cc trimmed]
-- 
Thien-Thi Nguyen ..................................... GPG key: 4C807502
.                  NB: ttn at glug dot org is not me                   .
.                 (and has not been since 2007 or so)                  .
.                        ACCEPT NO SUBSTITUTES                         .
........... please send technical questions to mailing lists ...........

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: inclusion of emacs-w3m

[chad]

On 11 Sep 2012, at 03:38, Thien-Thi Nguyen <ttn@gnuvola.org> wrote:

> Don't forget decent table rendering (especially, nested tables), one of
> w3m's distinguishing features.  From /usr/share/doc/w3m/STORY.html:

Table rending is certainly a PITA, but there are 2.5 big things
working on the side of an emacs-based HTML renderer:

1.) Terrible table hacks like nested tables and table-based layout are
vastly less common than they were when w3m was created (largely to
overcome lynx's poor table handling).

2.) The Emacs display engine *can* scroll side-to-side, whereas w3m
lived in a world where things off the right side of the tty were
forever out of sight.

2.5) shr.el already renders tables (so one could get started without
solving this problem first).

*Chad

Re: inclusion of emacs-w3m

[Andreas Schwab]

chad <yandros@MIT.EDU> writes:

> 2.) The Emacs display engine *can* scroll side-to-side,

As does w3m.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: inclusion of emacs-w3m

[chad]

On 11 Sep 2012, at 10:14, Andreas Schwab <schwab@linux-m68k.org> wrote:

> chad <yandros@MIT.EDU> writes:
> 
>> 2.) The Emacs display engine *can* scroll side-to-side,
> 
> As does w3m.

It couldn't when the table engine was written, which is
presumably when the document describing the algorithm was
written.

*Chad

Re: inclusion of emacs-w3m

[Wojciech Meyer]

chad <yandros@MIT.EDU> writes:

> On 11 Sep 2012, at 10:14, Andreas Schwab <schwab@linux-m68k.org> wrote:
>
>> chad <yandros@MIT.EDU> writes:
>> 
>>> 2.) The Emacs display engine *can* scroll side-to-side,
>> 
>> As does w3m.
>
> It couldn't when the table engine was written, which is
> presumably when the document describing the algorithm was
> written.
>

Width of the screen is surely one single constant number.

-- 
Wojciech Meyer
http://danmey.org

Re: inclusion of emacs-w3m

[Lars Magne Ingebrigtsen]

Thien-Thi Nguyen <ttn@gnuvola.org> writes:

> Don't forget decent table rendering (especially, nested tables), one of
> w3m's distinguishing features.

shr.el renders tables.  

> So, maybe another week or two for the Inspired One...  who will it be?

The algorithm is rather slow for deeply nested tables, though.  It
basically does a search of the entire "table space" to find the best
layout, and that's not the most efficient way to do it.  If you have
tables nested 20 deep (which you see in real life), it's not ideal.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: inclusion of emacs-w3m

[Tekk]

Do you see it in modern pages though? general consensus for years has been 
to avoid using tables for handling the presentation structure.

On Thu, 27 Sep 2012, Lars Magne Ingebrigtsen wrote:

> Thien-Thi Nguyen <ttn@gnuvola.org> writes:
>
>> Don't forget decent table rendering (especially, nested tables), one of
>> w3m's distinguishing features.
>
> shr.el renders tables.
>
>> So, maybe another week or two for the Inspired One...  who will it be?
>
> The algorithm is rather slow for deeply nested tables, though.  It
> basically does a search of the entire "table space" to find the best
> layout, and that's not the most efficient way to do it.  If you have
> tables nested 20 deep (which you see in real life), it's not ideal.
>
> -- 
> (domestic pets only, the antidote for overdose, milk.)
>  bloggy blog http://lars.ingebrigtsen.no/
>
>

Re: inclusion of emacs-w3m

[Lars Magne Ingebrigtsen]

Tekk <danny@parlementum.net> writes:

> Do you see it in modern pages though? general consensus for years has
> been to avoid using tables for handling the presentation structure.

That must be a consensus amongst people who don't do much web design.
:-)

<table> is the only way to do the sensible gridbag layout that most
other layout engines support.  But apparently CSS4 is finally going to
support this most basic of all layout methodologies.

Until then, <table> is where it's at.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: inclusion of emacs-w3m

[Stephen J. Turnbull]

Lars Magne Ingebrigtsen writes:
 > Tekk <danny@parlementum.net> writes:
 > 
 > > Do you see it in modern pages though? general consensus for years has
 > > been to avoid using tables for handling the presentation structure.
 > 
 > That must be a consensus amongst people who don't do much web design.
 > :-)

That's true.  But it's also a consensus among people who do so much
web design that they need to chase away customers with sky-high prices
that mortal managers (and their employers) can't afford to pay.  So we
rarely see their products.

Unfortunately, both are far outnumbered by wage slaves in the middle
who never heard of, let alone use, the "float" property.

 > Until then, <table> is where it's at.

I wish I could contest that.  But even some of the designers I know
who do know better say that they are slaves to their design tools for
their bread and butter.  Few clients are willing to pay for the time
it takes to do CSS Zen Garden work, even when it's provably more
usable.  "Everybody I know uses IE at 1024x768 so who cares if it's
even readable in anything else?" :-(

Re: inclusion of emacs-w3m

[Thien-Thi Nguyen]

[-- Attachment #1: Type: text/plain, Size: 1540 bytes --]

() Lars Magne Ingebrigtsen <larsi@gnus.org>
() Thu, 27 Sep 2012 23:31:00 +0200

   > Don't forget decent table rendering (especially, nested tables),
   > one of w3m's distinguishing features.

   shr.el renders tables.  

   > So, maybe another week or two for the Inspired One...  who will it
   > be?

   The algorithm is rather slow for deeply nested tables, though.  It
   basically does a search of the entire "table space" to find the best
   layout, and that's not the most efficient way to do it.

Sorry, i don't quite follow -- are you talking about the algorithm in
shr.el or the one in w3m?  I get the impression (being mostly ignorant
re table-layout algorithms in general and too lazy at the moment to look
at source code) that w3m's is somewhat directed.  In STORY.html, the w3m
author characterizes it as "fair", rather than "perfect", so perhaps its
heuristics can be tweaked (further) to better adapt it to an Emacs Lisp
implementation, if need be.  If only Someone would muster a try...

[-: It's now September.  All you (under)grad students looking to Do
    Something Interesting -- what are you waiting for?!  :-]

-- 
Thien-Thi Nguyen ..................................... GPG key: 4C807502
.                  NB: ttn at glug dot org is not me                   .
.                 (and has not been since 2007 or so)                  .
.                        ACCEPT NO SUBSTITUTES                         .
........... please send technical questions to mailing lists ...........


[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: inclusion of emacs-w3m

[Ted Zlatanov]

On Thu, 06 Sep 2012 14:30:44 +0200 joakim@verona.se wrote: 

j> Lars Ingebrigtsen <larsi@gnus.org> writes:
>> Now that Emacs proper has an HTML parser (via libxml2), an HTML renderer
>> (via shr.el), and an HTTP library (via url*.el), writing a totally
>> Emacs-based web browser should be pretty easy for somebody who has some
>> time to spare.

j> Can I shamelessly plug my emacs xwidget branch here?
j> It has webkit integration and stuff. One of my long term ideas is to
j> make it possible to expose the DOM so that SHR can render the same tree
j> webkit sees. There are some initial efforts in that direction in the
j> branch. There are some pros and cons to this of course.

I think an Emacs-based web browser should support various rendering
backends, including xwidget, but should have a fallback mode that works
in older Emacsen as well.

I thought shr.el, especially the CSS parsing it does, would be a really
hard project but Lars proved it possible and even simple.  So I believe
him an Emacs-based web browser is possible, even though it seems hard
because of our experience with w3.

Ted