Any interest in using HTML for locally-installed Texinfo documentation?

Any interest in using HTML for locally-installed Texinfo documentation?

[Gavin Smith]

Dear Guix developers,

I hope I am not intruding by advertising a project that may be of
interest to you.

Documentation for GNU packages and others is often installed in the
Info format, a plain text format.  Using a plaintext based format for
documentation does not take advantage of bitmapped displays that have
been available for decades.  It does not allow styling of text or
reflowing of text.  Much information is lost in the conversion from
Texinfo to Info and any attempt in, for example, Emacs to re-add this
information is unreliable.

Nonetheless, Info viewers have continued to have advantages over web
browsers.  They are fast, and have features for searching the manual
with index lookup.  They allow the use of keyboard commands.

In attempt to bring some of the benefits of the Info viewers to HTML
documentation in web browsers, in 2017, as part of Google Summer of
Code, Matthieu Lirzin worked on a JavaScript interface that works with
the HTML that texi2any produces.  His work is substantially complete.
A manual with this interface added is at
https://www.gnu.org/software/texinfo/manual/texinfo-html/Overview.html.
All the important keyboard commands that work in the Info viewers are
implemented, including index lookup.

The code he produced is in the js/ subdirectory of the Texinfo git
repository, and also available at
https://alpha.gnu.org/gnu/texinfo/texinfo-js-0.0.90.tar.gz

I believe this work has great potential to increase the ease of
accessing documentation, including documentation locally installed on
a user's own computer.  When a user is using a bitmapped display (e.g.
with X11), this could become the default way that they access
documentation.

I am contacting you because the distribution level may be the best
place to push this forward.  There are two reasons:
* The distribution could take care of installation of HTML
documentation files (at the moment, there is no standard place to
install these, and Automake does not support installing HTML files
generated from Texinfo).
* It could also take responsibility for checking web browser
compatibility.  Even if we don't use the JavaScript interface for
documentation on the GNU website due to browser compatibility
concerns, an OS distribution would have control over which browser was
used to view documentation.

Although I have little knowledge of Guix, it is the natural choice of
operating system distribution to contact about this possibility, as
both Texinfo and Guix are GNU projects.

If there is nobody who wants to take this forward within Guix, then
suggestions would also be welcome on how to otherwise push this
forward.

Best wishes,

Gavin

Re: Any interest in using HTML for locally-installed Texinfo documentation?

[sirgazil]

Hello, Gavin :)

El 1/04/19 a las 7:55 a. m., Gavin Smith escribió:
> Dear Guix developers,
> 
> I hope I am not intruding by advertising a project that may be of
> interest to you.
> 
> Documentation for GNU packages and others is often installed in the
> Info format, a plain text format.  Using a plaintext based format for
> documentation does not take advantage of bitmapped displays that have
> been available for decades.  It does not allow styling of text or
> reflowing of text.  Much information is lost in the conversion from
> Texinfo to Info and any attempt in, for example, Emacs to re-add this
> information is unreliable.
> 
> Nonetheless, Info viewers have continued to have advantages over web
> browsers.  They are fast, and have features for searching the manual
> with index lookup.  They allow the use of keyboard commands.
> 
> In attempt to bring some of the benefits of the Info viewers to HTML
> documentation in web browsers, in 2017, as part of Google Summer of
> Code, Matthieu Lirzin worked on a JavaScript interface that works with
> the HTML that texi2any produces.  His work is substantially complete.
> A manual with this interface added is at
> https://www.gnu.org/software/texinfo/manual/texinfo-html/Overview.html.
> All the important keyboard commands that work in the Info viewers are
> implemented, including index lookup.
> 
> The code he produced is in the js/ subdirectory of the Texinfo git
> repository, and also available at
> https://alpha.gnu.org/gnu/texinfo/texinfo-js-0.0.90.tar.gz
> 
> I believe this work has great potential to increase the ease of
> accessing documentation, including documentation locally installed on
> a user's own computer.  When a user is using a bitmapped display (e.g.
> with X11), this could become the default way that they access
> documentation.
> 
> I am contacting you because the distribution level may be the best
> place to push this forward.  There are two reasons:
> * The distribution could take care of installation of HTML
> documentation files (at the moment, there is no standard place to
> install these, and Automake does not support installing HTML files
> generated from Texinfo).
> * It could also take responsibility for checking web browser
> compatibility.  Even if we don't use the JavaScript interface for
> documentation on the GNU website due to browser compatibility
> concerns, an OS distribution would have control over which browser was
> used to view documentation.
> 
> Although I have little knowledge of Guix, it is the natural choice of
> operating system distribution to contact about this possibility, as
> both Texinfo and Guix are GNU projects.
> 
> If there is nobody who wants to take this forward within Guix, then
> suggestions would also be welcome on how to otherwise push this
> forward.


I didn't know this project existed, I wanted something like this.

As a high-level user of computers, I'd like to see this kind of 
documentation available both in the desktop and on the Web. I support it.

Thanks for the information,


-- 
Luis Felipe López Acevedo
http://sirgazil.bitbucket.io/

Re: Any interest in using HTML for locally-installed Texinfo documentation?

[Ludovic Courtès]

Hello Gavin,

Gavin Smith <gavinsmith0123@gmail.com> skribis:

> Documentation for GNU packages and others is often installed in the
> Info format, a plain text format.  Using a plaintext based format for
> documentation does not take advantage of bitmapped displays that have
> been available for decades.  It does not allow styling of text or
> reflowing of text.  Much information is lost in the conversion from
> Texinfo to Info and any attempt in, for example, Emacs to re-add this
> information is unreliable.
>
> Nonetheless, Info viewers have continued to have advantages over web
> browsers.  They are fast, and have features for searching the manual
> with index lookup.  They allow the use of keyboard commands.
>
> In attempt to bring some of the benefits of the Info viewers to HTML
> documentation in web browsers, in 2017, as part of Google Summer of
> Code, Matthieu Lirzin worked on a JavaScript interface that works with
> the HTML that texi2any produces.  His work is substantially complete.
> A manual with this interface added is at
> https://www.gnu.org/software/texinfo/manual/texinfo-html/Overview.html.
> All the important keyboard commands that work in the Info viewers are
> implemented, including index lookup.

Nice!  Org-info.js achieved something similar (info "(org) JavaScript
support") and I agree that it’s a great improvement.

I hope we can eventually upload manuals on gnu.org that take advantage
of this features when viewed with browsers that support JavaScript
(“progressive enhancement” as they call it.)  We could change Gnulib’s
‘gendocs.sh’ to do the right thing.  I’d suggest improving the CSS to
make the document less dense, but that’s a minor issue.

(For some reason ‘i’ does open the index search box for me, but then
hitting enter doesn’t produce any effect.  The other navigation commands
work fine, though.)

> I believe this work has great potential to increase the ease of
> accessing documentation, including documentation locally installed on
> a user's own computer.  When a user is using a bitmapped display (e.g.
> with X11), this could become the default way that they access
> documentation.

I hear the argument; it’s true that not everyone uses Emacs or is
familiar with the standalone Info reader.  Rendering of Info manuals in
Emacs is not bad, but a modern browser can do a better job.

Yet I’m not completely sold to the everything in the browser approach,
and everything in JavaScript.  In an ideal world (for me), we’d rather
provide a local documentation viewer that renders Texinfo directly.
TTN’s IXIN experiment was a step in the right direction IMO, but I
understand this approach is not something that’s happening now.

When talking about ease of access, we can’t ignore keyword searches.
How would you do ‘info -k’?  How would you even simply point your
browser to a specific manual?  What about inter-manual cross-references?
Would we need a mechanism similar to ‘hxmlxref.cnf’ but that would
browse local manuals?  What would be the recommended solution for Emacs
and console users?

> I am contacting you because the distribution level may be the best
> place to push this forward.  There are two reasons:
> * The distribution could take care of installation of HTML
> documentation files (at the moment, there is no standard place to
> install these, and Automake does not support installing HTML files
> generated from Texinfo).
> * It could also take responsibility for checking web browser
> compatibility.  Even if we don't use the JavaScript interface for
> documentation on the GNU website due to browser compatibility
> concerns, an OS distribution would have control over which browser was
> used to view documentation.

I think we could do this in Guix when we have answers to the questions
above.  :-)

There’s a side issue, which is that HTML documentation tends to take
quite a lot of space, but we’ll see whether that’s a problem.

> Although I have little knowledge of Guix, it is the natural choice of
> operating system distribution to contact about this possibility, as
> both Texinfo and Guix are GNU projects.

A good idea!  We should also consider working on adjusting policies and
practices in GNU, too.  At that point, it’ll be easier to reach out to
other distros.

Thank you,
Ludo’.

Re: Any interest in using HTML for locally-installed Texinfo documentation?

[Gavin Smith]

On Tue, Apr 02, 2019 at 11:37:51AM +0200, Ludovic Courtès wrote:
> (For some reason ‘i’ does open the index search box for me, but then
> hitting enter doesn’t produce any effect.  The other navigation commands
> work fine, though.)

It works on Firefox 53, at least.

> Yet I’m not completely sold to the everything in the browser approach,
> and everything in JavaScript.  In an ideal world (for me), we’d rather
> provide a local documentation viewer that renders Texinfo directly.
> TTN’s IXIN experiment was a step in the right direction IMO, but I
> understand this approach is not something that’s happening now.

To my knowledge, no program has ever been produced that does anything 
useful with IXIN.

Using JavaScript within a web browser has big drawbacks due to its 
"sandboxed" nature.  (You can't access environment variables, for 
example.)  However, we'd want to avoid having to re-implement too much 
of the web browser; for example, input file parsing, text layout and font 
rendering.

One thought is that there may be other "layout engines" that could be 
used, such as those in various GUI toolkits.

> When talking about ease of access, we can’t ignore keyword searches.
> How would you do ‘info -k’?

I don't know.  You would have to have some way of finding all the 
installed manuals.  This may be difficult with a standard web browser 
due to security restrictions.

> How would you even simply point your
> browser to a specific manual?

Maybe there could be a command for this within the browser.  There could 
also be a command-line program that would launch the documentation 
browser.

> What about inter-manual cross-references?
> Would we need a mechanism similar to ‘htmlxref.cnf’ but that would
> browse local manuals?

Good question.  The inter-manual links in locally-installed HTML files 
would have to be recognizable.  They could look like

<a href="../texinfo/index.html#Top">Texinfo</a>

instead of

<a href="https://www.gnu.org/software/texinfo/manual/texinfo/html_node/index.html#Top">Texinfo</a>

as there were would no way of resolving the second link to a 
locally-installed file (it is not clear from the URL what the name of 
the manual even is).  It would be quite simple to get texi2any to 
output this instead.  (It can currently be done by passing '-c 
HTMLXREF=empty' where 'empty' is an empty file, but a better interface 
could be devised.)

Then inter-manual links would all work, assuming the installed manuals 
are all subdirectories of the same directory (e.g. 
/usr/share/info/html).

This doesn't address the issue of multiple installed versions of the 
same manual or manuals in different "prefix hierarchies".  I imagine the 
Info browser would interpret the "../" string specially in a link and 
go looking through a search path for the referenced manual.  Again, this 
may be difficult to implement in standard web browsers due to security 
restrictions.

> What would be the recommended solution for Emacs
> and console users?

Info files would carry on as an option.

I'm getting the feeling that we need a web browser, or something like 
it, which can integrate with the operating system a lot more, without
sandboxing or security restrictions.

Re: Any interest in using HTML for locally-installed Texinfo documentation?

[Per Bothner]

On 4/2/19 8:02 AM, Gavin Smith wrote:
> Using JavaScript within a web browser has big drawbacks due to its
> "sandboxed" nature.  (You can't access environment variables, for
> example.)  However, we'd want to avoid having to re-implement too much
> of the web browser; for example, input file parsing, text layout and font
> rendering.
Both Electron and QtWebEngine have mechanisms for communicating between
the "browser" window and the main application, which is not sandboxed.
(For QtWebEngine the main application is regular non-sandboxed C++ code.)
For a desktop browser like Firefox, one could pass important environment
variables in the URL.  For bi-directional communication between a
sandboxed desktop browser and the C/C++ wrapper program one can always
use WebSockets or XmlHttpRequest ("AJAX").

Note below I'm talking in terms of an ideal long-term roadmap.
Medium-term one would prioritize, of course.

I'm assuming you would prefer to not require either Qt or Electron as
a dependency.  In that case my recommendation is two separate commands,
supplied by two separate distribution packages:

(1) A plain 'info' command can parse command-lines, find the right
html file, and then either open a window in the default desktop browser,
or display the file using 'emacs -nw'.  (If old-style info files are
available, it could also display those directly.) If qtinfo or electroninfo are
installed, they could be used to open a window instead.

Cross-manual links and some other functionality may be restricted when
using a standard desktop browser.

(2) Either 'qtinfo' or 'electroninfo' or both, which is an alternative
or wrapper for info using QtWebEngine or Electron, respectively.
GtkWebView may also be worth considering, but I have no experience with it.

> This doesn't address the issue of multiple installed versions of the
> same manual or manuals in different "prefix hierarchies".  I imagine the
> Info browser would interpret the "../" string specially in a link and
> go looking through a search path for the referenced manual.  Again, this
> may be difficult to implement in standard web browsers due to security
> restrictions.

It can be done in a standard web browsers using WebSockets or
XmlHttpRequest (AJAX), but that's more work.

> I'm getting the feeling that we need a web browser, or something like 
> it, which can integrate with the operating system a lot more, without
> sandboxing or security restrictions.

I (tentatively) recommend using QtWebEngine (i.e. 'qtinfo').  Electron is easier
to write and less verbose, but it is heavy duty and packaging may be a pain.
(Though there are other good reasons to package Electron.)  Qt is more
established in the GNU/Linux environment, and is easier to integrate
with C/C++ code, including re-using code from existing info.
A Gtk equivalent would have similar benefits.

If curious, you can look at the Qt code for DomTerm at:
https://github.com/PerBothner/DomTerm/tree/master/qtdomterm
This handles creating a QtWebEngine window, menus, and bi-directional
communication with the JavaScript in the QtWebEngine window.  It's not
super-compact, but it's quite reasonable. (The main C code for
process/pty management is in the lws-term directory.)
-- 
	--Per Bothner
per@bothner.com   http://per.bothner.com/

Re: Any interest in using HTML for locally-installed Texinfo documentation?

[Gavin Smith]

On Tue, Apr 02, 2019 at 09:46:05AM -0700, Per Bothner wrote:
> Both Electron and QtWebEngine have mechanisms for communicating between
> the "browser" window and the main application, which is not sandboxed.
> (For QtWebEngine the main application is regular non-sandboxed C++ code.)
> For a desktop browser like Firefox, one could pass important environment
> variables in the URL.  For bi-directional communication between a
> sandboxed desktop browser and the C/C++ wrapper program one can always
> use WebSockets or XmlHttpRequest ("AJAX").

I've started work on a documentation browser using QtWebEngine.  The 
work can be seen in the qt-info branch of the Texinfo Git repository:

http://git.savannah.gnu.org/cgit/texinfo.git/tree/js/docbrowser?h=qt-info

To build this, qmake must be installed.  It can be built by opening 
docbrowser.pro with qtcreator.  Running qmake would probably also work:
I have a Makefile inside the build directory build-docbrowser-Desktop-Debug/
(inside the js/ directory, at the same level as docbrowser/) with a 
comment:

#############################################################################
# Makefile for building: docbrowser
# Generated by qmake (3.1) (Qt 5.9.6)
# Project:  ../docbrowser/docbrowser.pro
# Template: app
# Command: /usr/bin/qmake-qt5 -o Makefile ../docbrowser/docbrowser.pro -spec linux-g++ CONFIG+=debug CONFIG+=qml_debug
#############################################################################



It's at a very early stage of development, but shows how bidirectional
communication with info.js can work.  This uses QWebChannel for bi-directional
communication: I believe domterm uses libwebsockets which is not specific to
Qt.

The QTINFO_DATADIR environment variable must be set to the "js" directory of
the sources.  The program can find manuals under the "examples" subdirectory.

The index search doesn't work because of this bug:

https://bugreports.qt.io/browse/QTBUG-54433

The drop-down list for indices would have to be implemented on the C++ 
side.

I don't know how far I am going to push this, but should do a bit more 
on it in the next few days.  (Development is slow for me because I am not very
familiar with Qt, C++, or JavaScript.)  The next things for me to work on are
proper handling of inter-manual links, as well as injecting info.js into the
HTML pages (instead of requiring it to be referenced in the HTML files in a
<script> tag).

> Note below I'm talking in terms of an ideal long-term roadmap.
> Medium-term one would prioritize, of course.
> 
> I'm assuming you would prefer to not require either Qt or Electron as
> a dependency.  In that case my recommendation is two separate commands,
> supplied by two separate distribution packages:
> 
> (1) A plain 'info' command can parse command-lines, find the right
> html file, and then either open a window in the default desktop browser,
> or display the file using 'emacs -nw'.  (If old-style info files are
> available, it could also display those directly.) If qtinfo or electroninfo are
> installed, they could be used to open a window instead.

This can be considered if a desktop browser is ever finished.

> > This doesn't address the issue of multiple installed versions of the
> > same manual or manuals in different "prefix hierarchies".  I imagine the
> > Info browser would interpret the "../" string specially in a link and
> > go looking through a search path for the referenced manual.  Again, this
> > may be difficult to implement in standard web browsers due to security
> > restrictions.
> 
> It can be done in a standard web browsers using WebSockets or
> XmlHttpRequest (AJAX), but that's more work.

The current state of the work intercepts links to external manuals (but 
doesn't do much with them yet).

> If curious, you can look at the Qt code for DomTerm at:
> https://github.com/PerBothner/DomTerm/tree/master/qtdomterm
> This handles creating a QtWebEngine window, menus, and bi-directional
> communication with the JavaScript in the QtWebEngine window.  It's not
> super-compact, but it's quite reasonable. (The main C code for
> process/pty management is in the lws-term directory.)

I never got this to work, but it was useful to be able to look at the source
code.

Re: Any interest in using HTML for locally-installed Texinfo documentation?

[Ludovic Courtès]

Hello,

Gavin Smith <gavinsmith0123@gmail.com> skribis:

> On Tue, Apr 02, 2019 at 09:46:05AM -0700, Per Bothner wrote:
>> Both Electron and QtWebEngine have mechanisms for communicating between
>> the "browser" window and the main application, which is not sandboxed.
>> (For QtWebEngine the main application is regular non-sandboxed C++ code.)
>> For a desktop browser like Firefox, one could pass important environment
>> variables in the URL.  For bi-directional communication between a
>> sandboxed desktop browser and the C/C++ wrapper program one can always
>> use WebSockets or XmlHttpRequest ("AJAX").
>
> I've started work on a documentation browser using QtWebEngine.  The 
> work can be seen in the qt-info branch of the Texinfo Git repository:
>
> http://git.savannah.gnu.org/cgit/texinfo.git/tree/js/docbrowser?h=qt-info

Neat!

From a “social” viewpoint, I think WebKitGTK would be more appropriate,
GTK+/GNOME being affiliated with GNU.

Also, QtWebEngine relies on bits of Chromium, which is a real challenge
from a software freedom viewpoint and from a security viewpoint, to the
point that we ended up removing it from our Qt builds in Guix:

  https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/qt.scm#n143
  https://lists.gnu.org/archive/html/guix-devel/2015-06/msg00302.html

Regardless, thanks for moving forward!

Ludo’.

Re: Any interest in using HTML for locally-installed Texinfo documentation?

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 448 bytes --]

Ludovic Courtès <ludo@gnu.org> writes:

> Also, QtWebEngine relies on bits of Chromium, which is a real challenge
> from a software freedom viewpoint and from a security viewpoint, to the
> point that we ended up removing it from our Qt builds in Guix:

Actually Marius Bakke posted a patch on March 11th adding the
QtWebengine.
At first glance it does not seem quite as hairy as Chromium.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Any interest in using HTML for locally-installed Texinfo documentation?

[Gavin Smith]

On Mon, Apr 08, 2019 at 05:12:17PM +0200, Ludovic Courtès wrote:
> Neat!
> 
> From a “social” viewpoint, I think WebKitGTK would be more appropriate,
> GTK+/GNOME being affiliated with GNU.

It's the "social" viewpoint that will make it a success.  We don't just
need a proof-of-concept or something that is 95% complete, we need something 
that is reliable and complete, and that people will find easy to install 
and use, and also that we can continue to develop and maintain.

I'm still trying to implement very basic functionality in the Qt code.
I will probably push on with it and see how far I get.  If GTK is used 
instead, the needed functionality would be present, and the Qt code 
would be a prototype.

I expect it would be easier to integrate GTK-based code with an automake 
build system (Qt has its own build system tool, qmake).  I understand 
from reading various blogs and so forth that GTK has its own problems, 
too.  However, it is still plausible that it could be used.

> Also, QtWebEngine relies on bits of Chromium, which is a real challenge
> from a software freedom viewpoint and from a security viewpoint, to the
> point that we ended up removing it from our Qt builds in Guix:
> 
>   https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/qt.scm#n143
>   https://lists.gnu.org/archive/html/guix-devel/2015-06/msg00302.html

Those messages don't give a very promising picture.

I like the idea of a simple help browser with few dependencies, as it 
means you still be able to get some help even if more complicated/novel 
things are broken.

Re: Any interest in using HTML for locally-installed Texinfo documentation?

[Eli Zaretskii]

> From: Gavin Smith <gavinsmith0123@gmail.com>
> Date: Tue, 9 Apr 2019 00:46:09 +0100
> Cc: guix-devel@gnu.org, Texinfo <bug-texinfo@gnu.org>
> 
> I'm still trying to implement very basic functionality in the Qt code.
> I will probably push on with it and see how far I get.  If GTK is used 
> instead, the needed functionality would be present, and the Qt code 
> would be a prototype.
> 
> I expect it would be easier to integrate GTK-based code with an automake 
> build system (Qt has its own build system tool, qmake).  I understand 
> from reading various blogs and so forth that GTK has its own problems, 
> too.  However, it is still plausible that it could be used.
> 
> > Also, QtWebEngine relies on bits of Chromium, which is a real challenge
> > from a software freedom viewpoint and from a security viewpoint, to the
> > point that we ended up removing it from our Qt builds in Guix:
> > 
> >   https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/qt.scm#n143
> >   https://lists.gnu.org/archive/html/guix-devel/2015-06/msg00302.html
> 
> Those messages don't give a very promising picture.

In case people aren't aware: AFAIK there's also a legal problem with
Qt itself, in that its license is GPLv3, without the "or later"
clause.

Re: Any interest in using HTML for locally-installed Texinfo documentation?

[Gavin Smith]

On Mon, Apr 08, 2019 at 05:12:17PM +0200, Ludovic Courtès wrote:
> > I've started work on a documentation browser using QtWebEngine.  The 
> > work can be seen in the qt-info branch of the Texinfo Git repository:
> >
> > http://git.savannah.gnu.org/cgit/texinfo.git/tree/js/docbrowser?h=qt-info
> 
> Neat!
> 
> From a “social” viewpoint, I think WebKitGTK would be more appropriate,
> GTK+/GNOME being affiliated with GNU.
> 
> Also, QtWebEngine relies on bits of Chromium, which is a real challenge
> from a software freedom viewpoint and from a security viewpoint, to the
> point that we ended up removing it from our Qt builds in Guix:

I've moved forward enough with Qt and QtWebEngine that I'm confident 
that it could be used for all the required features: path search for 
manuals and index search.  However, I'm not really happy with the hybrid 
architecture with one part of it in JavaScript, the other part in C++, 
communicating via a web socket.  (The JavaScript part itself 
is in two or three parts that send objects to each other via the "Message API"
to evade browser security restrictions.)  It's an overly fiddly 
architecture.  I have found there is potential for race conditions to 
exist and also performance problems with JavaScript, especially when 
manipulating the DOM.  The existing JavaScript code is useful as a prototype, 
but I think really the entire thing should be written in one language, 
not two.  Apparently with QtWebKit applications had full access to the 
DOM from the C++ side, but with QtWebEngine you are forced to use 
JavaScript.

Re: Any interest in using HTML for locally-installed Texinfo documentation?

[Pronaip]

Not directly connected, but have yall seen how rust packages its documentation? Personally I am REALLY not a fan of everything running in a separate browser, one browser is more than enough. But as long as there is one browser, why not generate a searchable index that uses JS the way Rust did it?
And if people can't run JS, they can just grep through the folder with all the HTML files.

Re: Any interest in using HTML for locally-installed Texinfo documentation?

[Gavin Smith]

Link to thread archive:
https://lists.gnu.org/archive/html/bug-texinfo/2019-04/msg00001.html

On Sat, Apr 13, 2019 at 5:18 PM Gavin Smith <gavinsmith0123@gmail.com> wrote:
> I've moved forward enough with Qt and QtWebEngine that I'm confident
> that it could be used for all the required features: path search for
> manuals and index search.  However, I'm not really happy with the hybrid
> architecture with one part of it in JavaScript, the other part in C++,
> communicating via a web socket.  (The JavaScript part itself
> is in two or three parts that send objects to each other via the "Message API"
> to evade browser security restrictions.)  It's an overly fiddly
> architecture.  I have found there is potential for race conditions to
> exist and also performance problems with JavaScript, especially when
> manipulating the DOM.  The existing JavaScript code is useful as a prototype,
> but I think really the entire thing should be written in one language,
> not two.  Apparently with QtWebKit applications had full access to the
> DOM from the C++ side, but with QtWebEngine you are forced to use
> JavaScript.

I'm replying to give an update on the current situation.

I haven't done any work on this in several months, so I will try to
remember the best I can.

I started another line of development, using the WebKitGTK engine.
http://git.savannah.gnu.org/cgit/texinfo.git/log/?h=webkitgtk-info

WebKitGTK seemed to be the best option for a lightweight embedded
web-browser. I looked into other options, such as the Gecko engine
used inside Thunderbird, but apparently it is not supported any more
to embed it in other programs. WebKitGTK allowed access to the DOM
tree of the documents without the complication of communicating with
embedded JavaScript, as was needed with QtWebEngine.

One annoyance is that there has to be a split architecture where the
DOM access has to be done in a separate process. Apparently this was
different in WebKitGTK version 1, but this was changed. One
recommended way around this is to communicate through dbus. I don't
know if we should depend on dbus, as I think a help system should
minimise dependencies (so you can still get help if your system is
borked). I implemented a Unix socket to pass data, although am not
sure how reliable my code is yet. Another option is to develop based
on WebKitGTK version 1 instead.

I found the documentation worse for GTK+ than for Qt and that it took
longer to get things working. However, compilation speed for C is much
faster, so trying to work things out by trial and error is less
painful. Also, running the program with valgrind showed many issues -
I don't know if this is my code's fault or a general issue with GTK+.
The GTK+ program integrated painlessly with an Automake build system,
whereas Qt code most naturally uses qmake (although qmake is quite
good and it doesn't appear that you are locked into using qmake or Qt
Designer in any way). The layout designer in Qt Designer appears to me
to be superior to Glade for GTK+, but I don't anticipate a need to
have complex widget layouts.

Text search is not implemented, but WebKitGTK does appear to have
special support for this, so it could be supported.

Texinfo 6.7 has been released which marks links to index pages with
the rel="index" attribute. This should make it easier to identify and
load the indices for a manual.

Start up time is an issue. The command-line "info" program starts up
almost instantaneously, whereas starting up a program with WebKitGTK
has a delay of a noticeable fraction of a second - even more so if
there is more than one WebKitGTK instance (for example, to load
indices in the background). I tend to start and stop many "info"
processes freely, rather than keeping the same process running all the
time. There is probably not much that can be done here.

I may be able to get an initial prototype that other people could try
ready in a few days. As ever, if anyone is interested in helping
with/taking over this project, please let me know.

Re: Any interest in using HTML for locally-installed Texinfo documentation?

[P]

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Tuesday, October 15, 2019 7:27 PM, Gavin Smith <gavinsmith0123@gmail.com> wrote:
> WebKitGTK seemed to be the best option for a lightweight embedded
> web-browser. I looked into other options, such as the Gecko engine
> used inside Thunderbird, but apparently it is not supported any more
> to embed it in other programs. WebKitGTK allowed access to the DOM
> tree of the documents without the complication of communicating with
> embedded JavaScript, as was needed with QtWebEngine.
>

Out of curiosity, have you looked at smaller browser engines, such as Dillo or Netsurf?

Re: Any interest in using HTML for locally-installed Texinfo documentation?

[Gavin Smith]

On Tue, Oct 15, 2019 at 08:20:14PM +0000, P wrote:
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> On Tuesday, October 15, 2019 7:27 PM, Gavin Smith <gavinsmith0123@gmail.com> wrote:
> > WebKitGTK seemed to be the best option for a lightweight embedded
> > web-browser. I looked into other options, such as the Gecko engine
> > used inside Thunderbird, but apparently it is not supported any more
> > to embed it in other programs. WebKitGTK allowed access to the DOM
> > tree of the documents without the complication of communicating with
> > embedded JavaScript, as was needed with QtWebEngine.
> >
> 
> Out of curiosity, have you looked at smaller browser engines, such as Dillo or Netsurf?

I hadn't looked at either of them.  Do you know if they can be embedded 
inside another program and allow access to the DOM tree of the web page?

Re: Any interest in using HTML for locally-installed Texinfo documentation?

[Per Bothner]

On 10/15/19 1:20 PM, P wrote:
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> On Tuesday, October 15, 2019 7:27 PM, Gavin Smith <gavinsmith0123@gmail.com> wrote:
>> WebKitGTK seemed to be the best option for a lightweight embedded
>> web-browser. I looked into other options, such as the Gecko engine
>> used inside Thunderbird, but apparently it is not supported any more
>> to embed it in other programs. WebKitGTK allowed access to the DOM
>> tree of the documents without the complication of communicating with
>> embedded JavaScript, as was needed with QtWebEngine.
>>
> 
> Out of curiosity, have you looked at smaller browser engines, such as Dillo or Netsurf?

Dillo does not support JavaScript or frames, which would seem to preclude
(or at least complicate) the kind of functionality we are hoping for.

-- 
	--Per Bothner
per@bothner.com   http://per.bothner.com/

Re: Any interest in using HTML for locally-installed Texinfo documentation?

[Gavin Smith]

On Tue, Oct 15, 2019 at 01:40:17PM -0700, Per Bothner wrote:
> On 10/15/19 1:20 PM, P wrote:
> > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> > On Tuesday, October 15, 2019 7:27 PM, Gavin Smith <gavinsmith0123@gmail.com> wrote:
> > > WebKitGTK seemed to be the best option for a lightweight embedded
> > > web-browser. I looked into other options, such as the Gecko engine
> > > used inside Thunderbird, but apparently it is not supported any more
> > > to embed it in other programs. WebKitGTK allowed access to the DOM
> > > tree of the documents without the complication of communicating with
> > > embedded JavaScript, as was needed with QtWebEngine.
> > > 
> > 
> > Out of curiosity, have you looked at smaller browser engines, such as Dillo or Netsurf?
> 
> Dillo does not support JavaScript or frames, which would seem to preclude
> (or at least complicate) the kind of functionality we are hoping for.

JavaScript should not be necessary if there is DOM access from the C/C++ 
side, as the case with WebKitGTK (although it is not as easy as it could 
be).  Frames should not be necessary either: for a table of contents 
side bar, I imagine this would be done as a widget outside of the 
embedded browser.

Re: Any interest in using HTML for locally-installed Texinfo documentation?

[Per Bothner]

On 10/15/19 2:00 PM, Gavin Smith wrote:

> JavaScript should not be necessary if there is DOM access from the C/C++
> side, as the case with WebKitGTK (although it is not as easy as it could
> be).  Frames should not be necessary either: for a table of contents
> side bar, I imagine this would be done as a widget outside of the
> embedded browser.

That's ok if you don't mind implementing and maintaining two separate
implementations, one for online documentation access and one for local documentation.
I would recommend against that.
-- 
	--Per Bothner
per@bothner.com   http://per.bothner.com/

Re: Any interest in using HTML for locally-installed Texinfo documentation?

[Gavin Smith]

On Tue, Oct 15, 2019 at 02:09:59PM -0700, Per Bothner wrote:
> On 10/15/19 2:00 PM, Gavin Smith wrote:
> 
> > JavaScript should not be necessary if there is DOM access from the C/C++
> > side, as the case with WebKitGTK (although it is not as easy as it could
> > be).  Frames should not be necessary either: for a table of contents
> > side bar, I imagine this would be done as a widget outside of the
> > embedded browser.
> 
> That's ok if you don't mind implementing and maintaining two separate
> implementations, one for online documentation access and one for local documentation.
> I would recommend against that.

I think there should be two different implementations.  Otherwise the 
local documentation reader will likely have a large amount of glue code 
and a confusing and unnecessarily complicated architecture.  Moreover, 
the existing JavaScript code does not deal with multiple manuals at all.

Re: Any interest in using HTML for locally-installed Texinfo documentation?

[Ricardo Wurmus]

Hi Gavin,

you may not even need to write all of this from scratch.  GNOME has Yelp
which uses WebKitGTK and which already supports info documents.  Perhaps
this could be enhanced instead?

-- 
Ricardo

Re: Any interest in using HTML for locally-installed Texinfo documentation?

[Ludovic Courtès]

Hi Gavin,

Gavin Smith <gavinsmith0123@gmail.com> skribis:

> I started another line of development, using the WebKitGTK engine.
> http://git.savannah.gnu.org/cgit/texinfo.git/log/?h=webkitgtk-info

[...]

> I may be able to get an initial prototype that other people could try
> ready in a few days. As ever, if anyone is interested in helping
> with/taking over this project, please let me know.

Thanks for the update, it looks promising.  Let us know when you’d like
people to give it a try!

Ludo’.

Re: Any interest in using HTML for locally-installed Texinfo documentation?

[Gavin Smith]

On Sat, Oct 19, 2019 at 9:31 PM Ludovic Courtès <ludo@gnu.org> wrote:
> > I started another line of development, using the WebKitGTK engine.
> > http://git.savannah.gnu.org/cgit/texinfo.git/log/?h=webkitgtk-info
>
> [...]
>
> > I may be able to get an initial prototype that other people could try
> > ready in a few days. As ever, if anyone is interested in helping
> > with/taking over this project, please let me know.
>
> Thanks for the update, it looks promising.  Let us know when you’d like
> people to give it a try!

The work is available on the webkitgit-info branch of the texinfo git
repository. I think it is developed to a point where it shows that a
browser for locally installed HTML documentation is clearly possible
with WebKitGTK. There are some notes in the README file on how to
build manuals for use with the browser.

I tried to upload a demo video to
https://goblinrefuge.com/mediagoblin/u/gavin/ but it hasn't appeared
yet.

Re: Any interest in using HTML for locally-installed Texinfo documentation?

[Gavin Smith]

On Tue, Oct 22, 2019 at 8:00 PM Gavin Smith <gavinsmith0123@gmail.com> wrote:
> I tried to upload a demo video to
> https://goblinrefuge.com/mediagoblin/u/gavin/ but it hasn't appeared
> yet.

That doesn't seem to be working, so I've uploaded the video to
https://www.gnu.org/software/texinfo/video/demo.webm.

Re: Any interest in using HTML for locally-installed Texinfo documentation?

[Ludovic Courtès]

Hi Gavin,

Gavin Smith <gavinsmith0123@gmail.com> skribis:

> On Sat, Oct 19, 2019 at 9:31 PM Ludovic Courtès <ludo@gnu.org> wrote:
>> > I started another line of development, using the WebKitGTK engine.
>> > http://git.savannah.gnu.org/cgit/texinfo.git/log/?h=webkitgtk-info
>>
>> [...]
>>
>> > I may be able to get an initial prototype that other people could try
>> > ready in a few days. As ever, if anyone is interested in helping
>> > with/taking over this project, please let me know.
>>
>> Thanks for the update, it looks promising.  Let us know when you’d like
>> people to give it a try!
>
> The work is available on the webkitgit-info branch of the texinfo git
> repository. I think it is developed to a point where it shows that a
> browser for locally installed HTML documentation is clearly possible
> with WebKitGTK. There are some notes in the README file on how to
> build manuals for use with the browser.

[...]

> https://www.gnu.org/software/texinfo/video/demo.webm.

This looks very nice already!  It seems to me that the core features one
would want are there: use of local copies of the manual, index search,
browsing commands, etc.

Does the reader fall back to an on-line copy of manuals that are
unavailable locally?  That would be nice, though it should probably
first ask for user consent.

I’d love to see an appropriate CSS applied by default to all the locally
installed manual.  Perhaps the WebKitGTK code could “force” a CSS to
each HTML page?

In the future, it’d be great to have syntax highlighting like we have at
<https://guix.gnu.org/manual/devel/en/html_node/Using-the-Configuration-System.html>,
but… I guess that’s another story.  :-)

What would be the next steps for you?  Do you plan to have this new
reader released as part of the next Texinfo release, or as a separate
package?

Thanks,
Ludo’.

Re: Any interest in using HTML for locally-installed Texinfo documentation?

[Gavin Smith]

On Sun, Nov 03, 2019 at 03:04:27PM +0100, Ludovic Courtès wrote:
> > The work is available on the webkitgit-info branch of the texinfo git
> > repository. I think it is developed to a point where it shows that a
> > browser for locally installed HTML documentation is clearly possible
> > with WebKitGTK. There are some notes in the README file on how to
> > build manuals for use with the browser.
> 
> [...]
> 
> > https://www.gnu.org/software/texinfo/video/demo.webm.
> 
> This looks very nice already!  It seems to me that the core features one
> would want are there: use of local copies of the manual, index search,
> browsing commands, etc.
> 
> Does the reader fall back to an on-line copy of manuals that are
> unavailable locally?  That would be nice, though it should probably
> first ask for user consent.

It doesn't do that yet.  It would have to look at an htmlxref.cnf file or 
equivalent, as the URL for the remote manual should not be in the locally 
installed documentation.

> I’d love to see an appropriate CSS applied by default to all the locally
> installed manual.  Perhaps the WebKitGTK code could “force” a CSS to
> each HTML page?

It is possible using webkit_web_view_new_with_user_content_manager.

> In the future, it’d be great to have syntax highlighting like we have at
> <https://guix.gnu.org/manual/devel/en/html_node/Using-the-Configuration-System.html>,
> but… I guess that’s another story.  :-)

How is that done? Are the HTML file post-processed somehow?

> What would be the next steps for you?  Do you plan to have this new
> reader released as part of the next Texinfo release, or as a separate
> package?

It would probably be for a separate package.  At the moment the program 
is called "infog" standing for "Info GTK".

There are various things that need to be done before it is ready for 
release:
* Allow installing the program, so that it can be run via PATH
* Handle external links in a web browser (using some kind of user 
desktop default)
* I'd like to make the index search completions in a separate pane 
rather than a pop-up menu, as in the "devhelp" program.
* Perhaps support for tabs
* The program uses a deprecated API in the WebKitGTK library to access 
the DOM of pages.  Allegedly it is possible to use JavaScript to do the 
same thing, but the documentation is not that helpful on how to do this.
* There is no text search facility in pages
* Standardize a location for installing HTML manuals.  What the GNU 
Coding Standards currently says about "htmldir" is insufficient, as a 
manual may have a different name to the package it is part of.
* It would be nice if the text input for a new window could be done as 
some kind of pop-over widget rather than in a separate dialog box.

I only have a few hours a week to spend on this, so it could take me 
some time to get through it.

I have been looking at tweaking the output of texi2any so the HTML looks 
better in this browser, including using mini-tables of contents instead 
of menus, and the table of contents linking to the top of a page rather 
than to an anchor a little down the page.

Re: Any interest in using HTML for locally-installed Texinfo documentation?

[Ludovic Courtès]

Hello,

Gavin Smith <gavinsmith0123@gmail.com> skribis:

> On Sun, Nov 03, 2019 at 03:04:27PM +0100, Ludovic Courtès wrote:

[...]

>> Does the reader fall back to an on-line copy of manuals that are
>> unavailable locally?  That would be nice, though it should probably
>> first ask for user consent.
>
> It doesn't do that yet.  It would have to look at an htmlxref.cnf file or 
> equivalent, as the URL for the remote manual should not be in the locally 
> installed documentation.

Sounds like a plan.

>> I’d love to see an appropriate CSS applied by default to all the locally
>> installed manual.  Perhaps the WebKitGTK code could “force” a CSS to
>> each HTML page?
>
> It is possible using webkit_web_view_new_with_user_content_manager.

Neat.

>> In the future, it’d be great to have syntax highlighting like we have at
>> <https://guix.gnu.org/manual/devel/en/html_node/Using-the-Configuration-System.html>,
>> but… I guess that’s another story.  :-)
>
> How is that done? Are the HTML file post-processed somehow?

Yes, it’s a bit ugly: we post-process the HTML in search of

  <pre class="lisp">

blocks (which correspond to @lisp) pass them through
guile-syntax-highlight.  There’s a bit of CSS for the rainbow
parentheses.  See
<https://git.savannah.gnu.org/cgit/guix.git/tree/doc/build.scm#n206>.

>> What would be the next steps for you?  Do you plan to have this new
>> reader released as part of the next Texinfo release, or as a separate
>> package?
>
> It would probably be for a separate package.  At the moment the program 
> is called "infog" standing for "Info GTK".

OK.

> There are various things that need to be done before it is ready for 
> release:
> * Allow installing the program, so that it can be run via PATH
> * Handle external links in a web browser (using some kind of user 
> desktop default)
> * I'd like to make the index search completions in a separate pane 
> rather than a pop-up menu, as in the "devhelp" program.
> * Perhaps support for tabs
> * The program uses a deprecated API in the WebKitGTK library to access 
> the DOM of pages.  Allegedly it is possible to use JavaScript to do the 
> same thing, but the documentation is not that helpful on how to do this.
> * There is no text search facility in pages
> * Standardize a location for installing HTML manuals.  What the GNU 
> Coding Standards currently says about "htmldir" is insufficient, as a 
> manual may have a different name to the package it is part of.
> * It would be nice if the text input for a new window could be done as 
> some kind of pop-over widget rather than in a separate dialog box.

Good.  I don’t think any of these are a showstopper, except perhaps the
bit about standardizing HTML installation (and getting distros to
actually do that!).  Other than that, your program is already useful as
it is, IMO.

> I only have a few hours a week to spend on this, so it could take me 
> some time to get through it.
>
> I have been looking at tweaking the output of texi2any so the HTML looks 
> better in this browser, including using mini-tables of contents instead 
> of menus, and the table of contents linking to the top of a page rather 
> than to an anchor a little down the page.

Nice.

Ludo’.

Re: Any interest in using HTML for locally-installed Texinfo documentation?

[Ludovic Courtès]

Gavin Smith <gavinsmith0123@gmail.com> skribis:

> On Tue, Apr 02, 2019 at 11:37:51AM +0200, Ludovic Courtès wrote:
>> (For some reason ‘i’ does open the index search box for me, but then
>> hitting enter doesn’t produce any effect.  The other navigation commands
>> work fine, though.)
>
> It works on Firefox 53, at least.

That’s with IceCat 60.6.1.

> Using JavaScript within a web browser has big drawbacks due to its 
> "sandboxed" nature.  (You can't access environment variables, for 
> example.)  However, we'd want to avoid having to re-implement too much 
> of the web browser; for example, input file parsing, text layout and font 
> rendering.
>
> One thought is that there may be other "layout engines" that could be 
> used, such as those in various GUI toolkits.

Yes, the GTK+ stacks has everything we need to display hypertext
content nicely, I believe.

>> When talking about ease of access, we can’t ignore keyword searches.
>> How would you do ‘info -k’?
>
> I don't know.  You would have to have some way of finding all the 
> installed manuals.

One option would be to have the option of letting ‘info’ parse HTML
files or a pre-built keyword database.

>> How would you even simply point your
>> browser to a specific manual?
>
> Maybe there could be a command for this within the browser.  There could 
> also be a command-line program that would launch the documentation 
> browser.

Sounds good.

>> What about inter-manual cross-references?
>> Would we need a mechanism similar to ‘htmlxref.cnf’ but that would
>> browse local manuals?
>
> Good question.  The inter-manual links in locally-installed HTML files 
> would have to be recognizable.  They could look like
>
> <a href="../texinfo/index.html#Top">Texinfo</a>
>
> instead of
>
> <a href="https://www.gnu.org/software/texinfo/manual/texinfo/html_node/index.html#Top">Texinfo</a>

Hmm, I’m skeptical.  :-)

And we haven’t talked about $INFOPATH yet.

>> What would be the recommended solution for Emacs
>> and console users?
>
> Info files would carry on as an option.

OK.

> I'm getting the feeling that we need a web browser, or something like 
> it, which can integrate with the operating system a lot more, without
> sandboxing or security restrictions.

Yelp apparently tried to address this very issue.  Perhaps we could
also check what’s missing to make it work correctly, or to make it work
better.

One issue that Ricardo mentioned is text reflowing.  To allow the UI to
do that, we need to feed it with some markup language and not Info:
HTML, XML, Texinfo, etc.  Then it would have enough information to
provide a rich interface comparable to what Texinfo-JS gives us.

Sorry to answer with new questions!

Ludo’.

Re: Any interest in using HTML for locally-installed Texinfo documentation?

[Gavin Smith]

On Wed, Apr 03, 2019 at 11:21:32PM +0200, Ludovic Courtès wrote:
> > One thought is that there may be other "layout engines" that could be 
> > used, such as those in various GUI toolkits.
> 
> Yes, the GTK+ stacks has everything we need to display hypertext
> content nicely, I believe.

OK, so embedding a full web browser might not be necessary.

> >> When talking about ease of access, we can’t ignore keyword searches.
> >> How would you do ‘info -k’?
> >
> > I don't know.  You would have to have some way of finding all the 
> > installed manuals.
> 
> One option would be to have the option of letting ‘info’ parse HTML
> files or a pre-built keyword database.

This is possible, assuming the code for this is not in JavaScript.

> >> What about inter-manual cross-references?
> >> Would we need a mechanism similar to ‘htmlxref.cnf’ but that would
> >> browse local manuals?
> >
> > Good question.  The inter-manual links in locally-installed HTML files 
> > would have to be recognizable.  They could look like
> >
> > <a href="../texinfo/index.html#Top">Texinfo</a>
> >
> > instead of
> >
> > <a href="https://www.gnu.org/software/texinfo/manual/texinfo/html_node/index.html#Top">Texinfo</a>
> 
> Hmm, I’m skeptical.  :-)

Some other appropriate syntax could be devised.

> And we haven’t talked about $INFOPATH yet.

I anticipate that the help program would intercept links to external 
manuals and interpret them in terms of INFOPATH or an equivalent 
environment variable.

Re: Any interest in using HTML for locally-installed Texinfo documentation?

[Per Bothner]

On 4/2/19 2:37 AM, Ludovic Courtès wrote:
> Yet I’m not completely sold to the everything in the browser approach,
> and everything in JavaScript.  In an ideal world (for me), we’d rather
> provide a local documentation viewer

  I don't think we're aiming for "everything in the browser".  A closer
approximation  is "everything using html+javascript".  I.e. using html
as the file type and JavaScript as the UI implementation language.
That does not require a traditional desktop browser: You can write
a nice desktop application using Electron, or Qt (QtWebEngine),
or Gtk (WebKitGTK), or Java (JavaFX/WebView).  You use one of these
toolkits to create a top-level window, with whatever "chrome" (window
frame, menus etc), but most of UI would be in JavaScript.  (I do have
a nice pure-JavaScript implementation of menus (menubar and popup), BTW.)

I have a lot of experience doing something similar for the DomTerm terminal
emulator (https://domterm.org): Display management, escape sequence
parsing, keyboard command processing are all handled by JavasScript.
This JavaScript can run in a regular browser (Firefox and Chrome and
been tested most) or using an Electron or Qt wrapper.  It works very
well - using Electron or Qt it looks and acts just like a regular terminal
emulator.  You start it up with a 'domterm' command, which depending on the
command-line switches forks a pty and creates an Electron/Qt/browser window.

> that renders Texinfo directly.

That's a lot of work, and I see little benefit to it.

> When talking about ease of access, we can’t ignore keyword searches.
> How would you do ‘info -k’?  How would you even simply point your
> browser to a specific manual?  What about inter-manual cross-references?

You can still have an 'info' command, which would parse the command-line,
find the appropriate html file, and then start up an Electron/Qt/browser
window.

If running under DomTerm or similar, the 'info' command can even re-use the
existing terminal window.  See the output from 'domterm help' in the
top-right pane of the first screenshot at http://domterm.org/index.html .

> Would we need a mechanism similar to ‘hxmlxref.cnf’ but that would
> browse local manuals?  What would be the recommended solution for Emacs
> and console users?

I think the best approach for Emacs is a hybrid of eww and info modes:
Instead of reading an info file, it would read an html file, which would
be displayed using eww.  However, the keybindings and search/navigation logic
would be based on that of info mode.

On a plain terminal, info could either create a fresh window, or it
could delegate to 'emacs -nw'.

> There’s a side issue, which is that HTML documentation tends to take
> quite a lot of space, but we’ll see whether that’s a problem.

It does require some more space, but it should compress fairly well.
What I do for the Kawa manual is generate an 'epub' archive, which is
basically a zip archive, with compression.  It is fairly simple for a
web server to extract a zip member and send it to a browser directly
as a gzip-compressed file, without actually decompressing the file
(until it gets to the browser).  I contributed support for this to
https://libwebsockets.org/, which is a compact C-language http server.
DomTerm uses this to "serve" the JavaScript files to the browser,
and a revamped 'info' program could do the same.
-- 
	--Per Bothner
per@bothner.com   http://per.bothner.com/

Re: Any interest in using HTML for locally-installed Texinfo documentation?

[Ludovic Courtès]

Hi Per,

Per Bothner <per@bothner.com> skribis:

> On 4/2/19 2:37 AM, Ludovic Courtès wrote:
>> Yet I’m not completely sold to the everything in the browser approach,
>> and everything in JavaScript.  In an ideal world (for me), we’d rather
>> provide a local documentation viewer
>
>  I don't think we're aiming for "everything in the browser".  A closer
> approximation  is "everything using html+javascript".

Yeah, that’s what I meant.  :-)

I find things like DOMTerm very impressive, and it’s true that
HTML/JS/CSS nowadays constitute an unequaled UI framework (to the point
that GNOME Shell is also written in JS + CSS.)

That would be a good argument in favor of doing things this way.  Yet, I
have to say that this is not a direction that I like, technically and
otherwise (we’re talking about code bases orders of magnitudes bigger
than all of Texinfo including info-stnd, and code bases under the
control of a couple of companies.)

>> that renders Texinfo directly.
>
> That's a lot of work, and I see little benefit to it.

I was mentioning this because it’s an experiment that Andy Wingo did
about 15 years (?!) ago.  Andy wrote the Texinfo parser that’s now part
of Guile, and then had a Guile-GTK program that used a tree widget to
show the contents, had clickable links, text would reflow, etc.  (See
<https://wingolog.org/archives/2004/07/25/literate-programming-with-guile-lib>.
Unfortunately the screenshot has disappeared.)

That said, it surely is quite a bit of work, but I think it’s an option
we could consider.

>> When talking about ease of access, we can’t ignore keyword searches.
>> How would you do ‘info -k’?  How would you even simply point your
>> browser to a specific manual?  What about inter-manual cross-references?
>
> You can still have an 'info' command, which would parse the command-line,
> find the appropriate html file, and then start up an Electron/Qt/browser
> window.

Sounds like a plan.

>> Would we need a mechanism similar to ‘hxmlxref.cnf’ but that would
>> browse local manuals?  What would be the recommended solution for Emacs
>> and console users?
>
> I think the best approach for Emacs is a hybrid of eww and info modes:
> Instead of reading an info file, it would read an html file, which would
> be displayed using eww.  However, the keybindings and search/navigation logic
> would be based on that of info mode.
>
> On a plain terminal, info could either create a fresh window, or it
> could delegate to 'emacs -nw'.

Yes.

>> There’s a side issue, which is that HTML documentation tends to take
>> quite a lot of space, but we’ll see whether that’s a problem.
>
> It does require some more space, but it should compress fairly well.
> What I do for the Kawa manual is generate an 'epub' archive, which is
> basically a zip archive, with compression.  It is fairly simple for a
> web server to extract a zip member and send it to a browser directly
> as a gzip-compressed file, without actually decompressing the file
> (until it gets to the browser).  I contributed support for this to
> https://libwebsockets.org/, which is a compact C-language http server.
> DomTerm uses this to "serve" the JavaScript files to the browser,
> and a revamped 'info' program could do the same.

A simpler solution might be to use ‘Content-Encoding: gzip’.

Thanks,
Ludo’.

Re: Any interest in using HTML for locally-installed Texinfo documentation?

[Per Bothner]

On 4/3/19 2:11 PM, Ludovic Courtès wrote:

>> What I do for the Kawa manual is generate an 'epub' archive, which is
>> basically a zip archive, with compression.  It is fairly simple for a
>> web server to extract a zip member and send it to a browser directly
>> as a gzip-compressed file, without actually decompressing the file
>> (until it gets to the browser).  I contributed support for this to
>> https://libwebsockets.org/, which is a compact C-language http server.
>> DomTerm uses this to "serve" the JavaScript files to the browser,
>> and a revamped 'info' program could do the same.
> 
> A simpler solution might be to use ‘Content-Encoding: gzip’.

That is what libwebsockets does given a zip archive (and a browser
that can handle Content-Encoding: gzip).  There is a little bit of header munging,
but it turns out the the compression used for a member of a zip archive is
exactly the same as used by ‘Content-Encoding: gzip’.  So the web server can
extract the compressed data from the zip archive and send it directly to the
browser without having to decompress and then re-compress it.

DomTerm does the same for its JavaScript and css files: They're installed as a
zip archive, and the domterm command (using libwebsockets) starts up a browser window
with a URL pointing back at itself.  When files are requested it can send them
to the browser as ‘Content-Encoding: gzip’, without having to uncompress them first.
-- 
	--Per Bothner
per@bothner.com   http://per.bothner.com/

Re: Any interest in using HTML for locally-installed Texinfo documentation?

[Gavin Smith]

On Wed, Apr 03, 2019 at 11:11:36PM +0200, Ludovic Courtès wrote:
> I find things like DOMTerm very impressive, and it’s true that
> HTML/JS/CSS nowadays constitute an unequaled UI framework (to the point
> that GNOME Shell is also written in JS + CSS.)
> 
> That would be a good argument in favor of doing things this way.  Yet, I
> have to say that this is not a direction that I like, technically and
> otherwise (we’re talking about code bases orders of magnitudes bigger
> than all of Texinfo including info-stnd, and code bases under the
> control of a couple of companies.)

We need something that can render text and has hyperlinks.  When 
the user clicks a hyperlink, the program should have control over what 
happens.  It doesn't sound like a very complicated problem.  

Maybe there is a lightweight solution that doesn't require embedding a 
full web browser in the program.  For example, in Qt there is the QLabel 
widget which appears to support links in text: 
https://doc.qt.io/qt-5/qlabel.html#signals

It may be possible to change the embedded web browser to a different 
one if a company controlling it takes it in a bad direction.

Ideally, we'd avoid tying a graphical help browser into large frameworks 
like Chromium.  However, I think that an embedded browser (as in 
DomTerm) is a good place to start, reusing the existing JavaScript code.
Maybe simpler solutions could be explored later.

I've been trying to learn about Qt and it certainly is quite complex 
(taking many hours to compile some of its development tools).

> I was mentioning this because it’s an experiment that Andy Wingo did
> about 15 years (?!) ago.  Andy wrote the Texinfo parser that’s now part
> of Guile, and then had a Guile-GTK program that used a tree widget to
> show the contents, had clickable links, text would reflow, etc.  (See
> <https://wingolog.org/archives/2004/07/25/literate-programming-with-guile-lib>.
> Unfortunately the screenshot has disappeared.)
> 
> That said, it surely is quite a bit of work, but I think it’s an option
> we could consider.

This is an interesting idea.  How far developed was the program?  Is it 
still extant?  texi2any has its own Texinfo parser, written in Perl and 
C.

Re: Any interest in using HTML for locally-installed Texinfo documentation?

[Ludovic Courtès]

Hi Gavin,

Gavin Smith <gavinsmith0123@gmail.com> skribis:

> On Wed, Apr 03, 2019 at 11:11:36PM +0200, Ludovic Courtès wrote:

[...]

>> I was mentioning this because it’s an experiment that Andy Wingo did
>> about 15 years (?!) ago.  Andy wrote the Texinfo parser that’s now part
>> of Guile, and then had a Guile-GTK program that used a tree widget to
>> show the contents, had clickable links, text would reflow, etc.  (See
>> <https://wingolog.org/archives/2004/07/25/literate-programming-with-guile-lib>.
>> Unfortunately the screenshot has disappeared.)
>> 
>> That said, it surely is quite a bit of work, but I think it’s an option
>> we could consider.
>
> This is an interesting idea.  How far developed was the program?  Is it 
> still extant?

This was a pure Guile approach.  The parser itself has been part of
Guile since 2.0; it’s good but incomplete (usually it cannot parse
complete manuals.)

The viewer was based on Guile-GNOME, which was in the GTK+2 era I
believe.  It definitely wouldn’t be usable as-is today.  Andy, do you
remember more?  :-)

Thanks,
Ludo’.

Re: Any interest in using HTML for locally-installed Texinfo documentation?

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

> I hear the argument; it’s true that not everyone uses Emacs or is
> familiar with the standalone Info reader.  Rendering of Info manuals in
> Emacs is not bad, but a modern browser can do a better job.
>
> Yet I’m not completely sold to the everything in the browser approach,
> and everything in JavaScript.  In an ideal world (for me), we’d rather
> provide a local documentation viewer that renders Texinfo directly.

As far as I know GNOME’s Yelp is a frontend to different kinds of
documentation and it does support Info files.

It may not work out of the box without setting some environment
variables first, but I remember viewing Info manuals in Yelp not too
long ago when I first learned that it supports Info.

--
Ricardo

Re: Any interest in using HTML for locally-installed Texinfo documentation?

[Ricardo Wurmus]

Ricardo Wurmus <rekado@elephly.net> writes:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> I hear the argument; it’s true that not everyone uses Emacs or is
>> familiar with the standalone Info reader.  Rendering of Info manuals in
>> Emacs is not bad, but a modern browser can do a better job.
>>
>> Yet I’m not completely sold to the everything in the browser approach,
>> and everything in JavaScript.  In an ideal world (for me), we’d rather
>> provide a local documentation viewer that renders Texinfo directly.
>
> As far as I know GNOME’s Yelp is a frontend to different kinds of
> documentation and it does support Info files.
>
> It may not work out of the box without setting some environment
> variables first, but I remember viewing Info manuals in Yelp not too
> long ago when I first learned that it supports Info.

It actually does seem to just work.  Try this:

   yelp info:guix

-- 
Ricardo

Re: Any interest in using HTML for locally-installed Texinfo documentation?

[sirgazil]

El 2/04/19 a las 3:27 p. m., Ricardo Wurmus escribió:
> 
> Ricardo Wurmus <rekado@elephly.net> writes:
> 
>> Ludovic Courtès <ludo@gnu.org> writes:
>>
>>> I hear the argument; it’s true that not everyone uses Emacs or is
>>> familiar with the standalone Info reader.  Rendering of Info manuals in
>>> Emacs is not bad, but a modern browser can do a better job.
>>>
>>> Yet I’m not completely sold to the everything in the browser approach,
>>> and everything in JavaScript.  In an ideal world (for me), we’d rather
>>> provide a local documentation viewer that renders Texinfo directly.
>>
>> As far as I know GNOME’s Yelp is a frontend to different kinds of
>> documentation and it does support Info files.
>>
>> It may not work out of the box without setting some environment
>> variables first, but I remember viewing Info manuals in Yelp not too
>> long ago when I first learned that it supports Info.
> 
> It actually does seem to just work.  Try this:
> 
>     yelp info:guix


It works for me too (even searching). But I see some things missing,
although I'm using Yelp 3.18 from the host distro:

* Clicking any index result in an error:
   error on line 1114 at column 428: PCDATA invalid Char value 8
* Info manuals are not in the list of manuals available.

Being able to use Yelp would be great, in my opinion. After all, GNOME 
is the GNU Desktop.


-- 
Luis Felipe López Acevedo
http://sirgazil.bitbucket.io/

Re: Any interest in using HTML for locally-installed Texinfo documentation?

[Per Bothner]

On 4/2/19 1:12 PM, Ricardo Wurmus wrote:
> As far as I know GNOME’s Yelp is a frontend to different kinds of
> documentation and it does support Info files.

That reads *info* files.  We're talking about reading *html* files.
See Gavin's original message for why we want to use html.
-- 
	--Per Bothner
per@bothner.com   http://per.bothner.com/

Re: Any interest in using HTML for locally-installed Texinfo documentation?

[sirgazil]

El 2/04/19 a las 5:10 p. m., Per Bothner escribió:
> On 4/2/19 1:12 PM, Ricardo Wurmus wrote:
>> As far as I know GNOME’s Yelp is a frontend to different kinds of
>> documentation and it does support Info files.
> 
> That reads *info* files.  We're talking about reading *html* files.
> See Gavin's original message for why we want to use html.

Isn't it more about "increase the ease of
accessing documentation, including documentation locally installed on
a user's own computer.  When a user is using a bitmapped display (e.g.
with X11), this could become the default way that they access
documentation."?

I find Yelp easy to use as a desktop user. I think it would be great to 
easily access info manuals or info manuals translated to docbook or 
mallard from the Yelp viewer. It would make GNU documentation easier to 
discover on the desktop.

I think Matthieu's JavaScript interface would still be useful for the 
HTML documentation online.


-- 
Luis Felipe López Acevedo
http://sirgazil.bitbucket.io/

Re: Any interest in using HTML for locally-installed Texinfo documentation?

[Gavin Smith]

On Tue, Apr 02, 2019 at 06:09:40PM -0500, sirgazil wrote:
> El 2/04/19 a las 5:10 p. m., Per Bothner escribió:
> >On 4/2/19 1:12 PM, Ricardo Wurmus wrote:
> >>As far as I know GNOME’s Yelp is a frontend to different kinds of
> >>documentation and it does support Info files.
> >
> >That reads *info* files.  We're talking about reading *html* files.
> >See Gavin's original message for why we want to use html.
> 
> Isn't it more about "increase the ease of
> accessing documentation, including documentation locally installed on
> a user's own computer.  When a user is using a bitmapped display (e.g.
> with X11), this could become the default way that they access
> documentation."?

Variation of fonts and text reflowing, as I said in my original message.

Re: Any interest in using HTML for locally-installed Texinfo documentation?

[sirgazil]

El 3/04/19 a las 3:43 a. m., Gavin Smith escribió:
> On Tue, Apr 02, 2019 at 06:09:40PM -0500, sirgazil wrote:
>> El 2/04/19 a las 5:10 p. m., Per Bothner escribió:
>>> On 4/2/19 1:12 PM, Ricardo Wurmus wrote:
>>>> As far as I know GNOME’s Yelp is a frontend to different kinds of
>>>> documentation and it does support Info files.
>>>
>>> That reads *info* files.  We're talking about reading *html* files.
>>> See Gavin's original message for why we want to use html.
>>
>> Isn't it more about "increase the ease of
>> accessing documentation, including documentation locally installed on
>> a user's own computer.  When a user is using a bitmapped display (e.g.
>> with X11), this could become the default way that they access
>> documentation."?
> 
> Variation of fonts and text reflowing, as I said in my original message.

Sorry, I don't understand. Documents in Yelp seem to adapt to some 
extent to the screen width (text reflows, for example). Videos an images 
don't adapt well to the screen width in the version I'm using, and info 
documents seem to have a fixed width.

As for fonts, Yelp seems to use the same fonts for the kind of documents 
it supports. Isn't it desirable to present all documents uniformly?

However, Yelp seems to use WebKit (I'm not sure), and GNOME and GTK 
components are being modified to adapt to different screen sizes to 
support mobile devices. So problems of adaptability of the content to 
the size of the screen will likely disappear...


-- 
Luis Felipe López Acevedo
http://sirgazil.bitbucket.io/

Re: Any interest in using HTML for locally-installed Texinfo documentation?

[Per Bothner]

On 4/3/19 7:23 AM, sirgazil wrote:

> Sorry, I don't understand. Documents in Yelp seem to adapt to some extent to the screen width (text reflows, for example). Videos an images don't adapt well to the screen width in the version I'm using, and info documents seem to have a fixed width.

Yes, that's what we're talking about: info documents.
This is not a matter of improving Yelp: the info file format inherently
does not support reflow, or mixing a variable-width font (for main text)
with a fixed-width font (for code).

That is why we're talking about having makeinfo convert texinfo to html
instead of info, and installing html in distributions.
-- 
	--Per Bothner
per@bothner.com   http://per.bothner.com/

Re: Any interest in using HTML for locally-installed Texinfo documentation?

[Ricardo Wurmus]

Per Bothner <per@bothner.com> writes:

> On 4/2/19 1:12 PM, Ricardo Wurmus wrote:
>> As far as I know GNOME’s Yelp is a frontend to different kinds of
>> documentation and it does support Info files.
>
> That reads *info* files.  We're talking about reading *html* files.
> See Gavin's original message for why we want to use html.

FWIW Yelp also reads HTML files (and many more formats), though I
haven’t yet tried this successfully with our Guix manual.

Yelp unfortunately doesn’t utilize the Info manual’s index, which is a
great loss in my opinion.  The info files are rendered with hard line
breaks; the info format seems to make it hard to support variable width
paragraphs, syntax highlighting in code snippets / examples, etc,
because it doesn’t seem to retain markup.

Yelp has a few bugs when rendering Info files; links to other Info
documents don’t seem to work (e.g. references to the Guile manual in the
Guix manual), for example, and images are not shown either.

--
Ricardo

Re: Any interest in using HTML for locally-installed Texinfo documentation?

[George Clemmer]

Hi Gavin,

Gavin Smith <gavinsmith0123@gmail.com> writes:
[...]
> A manual with this interface added is at
> https://www.gnu.org/software/texinfo/manual/texinfo-html/Overview.html.
> All the important keyboard commands that work in the Info viewers are
> implemented, including index lookup.

Nice! I like seeing Info commands work on WWW-posted doc. This would be
useful to an Emacs/Info user for packages that aren't installed or when
their Info system is broken. To reach the more numerous "non Emacs/Info
users", these commands need to be advertised and promoted. Is there a
plan for this?

WRT the current implementation ...

In Emacs and Info, typing 'h' shows Info keyboard commands. It didn't do
anything visiting this site with CHROME and Safari.

It would be nice if a second 'C-s' advanced to the next match without
requiring <RET>.

'C-s' on the site is so slow that I thought it was broken at first :(

HTH - George

Re: Any interest in using HTML for locally-installed Texinfo documentation?

[Gavin Smith]

On Tue, Apr 02, 2019 at 05:02:19PM -0400, George Clemmer wrote:
> Gavin Smith <gavinsmith0123@gmail.com> writes:
> [...]
> > A manual with this interface added is at
> > https://www.gnu.org/software/texinfo/manual/texinfo-html/Overview.html.
> > All the important keyboard commands that work in the Info viewers are
> > implemented, including index lookup.
> 
> Nice! I like seeing Info commands work on WWW-posted doc. This would be
> useful to an Emacs/Info user for packages that aren't installed or when
> their Info system is broken. To reach the more numerous "non Emacs/Info
> users", these commands need to be advertised and promoted. Is there a
> plan for this?

Not really.  That's why I wrote to this list.

> WRT the current implementation ...
> 
> In Emacs and Info, typing 'h' shows Info keyboard commands. It didn't do
> anything visiting this site with CHROME and Safari.
> 
> It would be nice if a second 'C-s' advanced to the next match without
> requiring <RET>.
> 
> 'C-s' on the site is so slow that I thought it was broken at first :(

Thanks for the feedback.  C-s will be slow because it is retrieving each 
HTML page over a network.