bug#53205: 28.0.90; [PATCH] GNU ELPA: Provide more control over linked documentation

bug#53205: 28.0.90; [PATCH] GNU ELPA: Provide more control over linked documentation

[Y. E. via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[-- Attachment #1: Type: text/plain, Size: 242 bytes --]

The introductory text is longish, so if no additional context is needed,
please see the following attached patches:

- 0001-Provide-more-control-over-linked-documentation.patch

- 0001-README-doc-html-resources-Document-new-properties.patch


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Provide more control over linked documentation --]
[-- Type: text/x-patch, Size: 3082 bytes --]

From 834f01f267f2f8ab677d0de33e12034c06dbabd6 Mon Sep 17 00:00:00 2001
From: YugaEgo <yet@ego.team>
Date: Wed, 12 Jan 2022 12:24:09 +0200
Subject: [PATCH] Provide more control over linked documentation

* README: (:doc-html, :resources): Document new properties.
* elpa-admin.el (elpaa--html-insert-resources): New function.
(elpaa--html-make-pkg): Use new function.
(elpaa--build-Info-1): Do not generate HTML doc if :doc-html is
'ignore'.
---
 README        | 13 +++++++++++++
 elpa-admin.el | 12 +++++++++++-
 2 files changed, 24 insertions(+), 1 deletion(-)

diff --git a/README b/README
index a0e34ff78c..b83e8dfd1a 100644
--- a/README
+++ b/README
@@ -146,6 +146,19 @@ Name of the documentation file for the package.
 It can be either an Info file, a Texinfo file, or an Org file.
 FILE can also be a list of those.
 
+** =:doc-html=
+By default, if =:doc= specification contains a Texinfo file, then HTML
+documentation is generated from it; the link to the generated HTML
+file is added to the package page.  To disable this behavior, set
+=:doc-html= to ~ignore~.
+
+** =:resoures LINKS=
+Enables 'Additional Resources' section on the package page.  This must
+be an association list of the titles and URLs of online resources.
+For example:
+
+    (("User Manual" . "https://example.tld/manual.html"))
+
 ** =:make TARGET=
 Indicates that we should run ~make TARGET~ in order to build some files.
 This is run before processing =:doc=, so it can be used to generate
diff --git a/elpa-admin.el b/elpa-admin.el
index d570c3c6aa..c09e22241d 100644
--- a/elpa-admin.el
+++ b/elpa-admin.el
@@ -1469,6 +1469,13 @@ arbitrary code."
 	  ))
       (insert "</dd>\n"))))
 
+(defun elpaa--html-insert-resources (pkg-spec)
+  (when-let ((links (elpaa--spec-get pkg-spec :resources)))
+    (insert "<dt>Additional Resources</dt> <dd>\n")
+    (dolist (link links)
+      (insert "<a href=\"" (cdr link) "\">" (car link) "</a>\n"))
+    (insert "</dd>\n")))
+
 (defun elpaa--html-make-pkg (pkg pkg-spec files srcdir)
   (let* ((name (symbol-name (car pkg)))
          (latest (package-version-join (aref (cdr pkg) 0)))
@@ -1517,6 +1524,7 @@ arbitrary code."
            (elpaa--get-prop "URL" name srcdir mainsrcfile)))
       (insert (format "<dt>Badge</dt><dd><img src=\"%s.svg\"/></dd>\n" (elpaa--html-quote name)))
       (elpaa--html-insert-docs pkg-spec)
+      (elpaa--html-insert-resources pkg-spec)
       (insert "</dl>")
       (insert (format "<p>To install this package, run in Emacs:</p>
                        <pre>M-x <span class=\"kw\">package-install</span> RET <span class=\"kw\">%s</span> RET</pre>"
@@ -2106,7 +2114,9 @@ relative to elpa root."
            t "makeinfo" "--no-split" docfile "-o" info-file)
           (message "%s" (buffer-string)))
 
-	(when html-dir (elpaa--html-build-doc pkg-spec docfile html-dir))
+        (let ((doc-html (elpaa--spec-get pkg-spec :doc-html)))
+          (when (and html-dir (not (equal doc-html 'ignore)))
+            (elpaa--html-build-doc pkg-spec docfile html-dir)))
 
         (setq docfile info-file)))
 
-- 
2.34.1


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: README file changes for the 'main' branch --]
[-- Type: text/x-patch, Size: 1205 bytes --]

From ab2394e3c3816deb85aaa241d0b341b2fc7a5fea Mon Sep 17 00:00:00 2001
From: YugaEgo <yet@ego.team>
Date: Wed, 12 Jan 2022 12:31:30 +0200
Subject: [PATCH] * README (:doc-html, :resources): Document new properties

---
 README | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/README b/README
index f05e9e58ac..eafe517fac 100644
--- a/README
+++ b/README
@@ -184,6 +184,19 @@ Name of the documentation file for the package.
 It can be either an Info file, a Texinfo file, or an Org file.
 FILE can also be a list of those.
 
+** =:doc-html=
+By default, if =:doc= specification contains a Texinfo file, then HTML
+documentation is generated from it; the link to the generated HTML
+file is added to the package page.  To disable this behavior, set
+=:doc-html= to ~ignore~.
+
+** =:resoures LINKS=
+Enables 'Additional Resources' section on the package page.  This must
+be an association list of the titles and URLs of online resources.
+For example:
+
+    (("User Manual" . "https://example.tld/manual.html"))
+
 ** =:make TARGET=
 Indicates that we should run ~make TARGET~ in order to build some files.
 This is run before processing =:doc=, so it can be used to generate
-- 
2.34.1


[-- Attachment #4: Type: text/plain, Size: 2534 bytes --]



I've been locally testing configuration for Company documentation (to
add it to GNU ELPA) and noticed that an HTML version of the manual is
automatically generated from the '.texi' file. The generated HTML file
is linked under the label 'Manual' on the package page and served from
the 'archive-devel/doc/company/' directory.

I was pleased to see that automatic generation but there were two issues
(namely, images and styles).

The user manual for Company is shipped with the images; and it is
currently impossible to configure the addition of them to the
'elpaa--doc-subdirectory'.

My initial idea for fixing this issue was to add a property
':doc-asset', which could be configured to '(FROM . TO)' directories
names, then used for moving images to the 'archive-devel/doc/company/'
folder, as shown below:

#+BEGIN_SRC diff
diff --git a/elpa-admin.el b/elpa-admin.el
index d94cc7fd73..23bb582ceb 100644
--- a/elpa-admin.el
+++ b/elpa-admin.el
@@ -1999,7 +1999,11 @@ directory; one of archive, archive-devel."
 	    (expand-file-name elpaa--doc-subdirectory tarball-dir)))))
     (when html-dir
       (when (not (file-readable-p html-dir)) ;FIXME: Why bother testing?
-	(make-directory html-dir t)))
+	    (make-directory html-dir t))
+      (when-let ((assets (elpaa--spec-get pkg-spec :doc-assets)))
+        (cl-loop for (from . to) in assets
+                 do (rename-file (expand-file-name from dir)
+                                 (expand-file-name to html-dir) t))))

     (plist-put (cdr pkg-spec) :internal--html-docs nil)
     (dolist (f docfiles)
-- 
2.34.1
#+END_SRC

That worked fine during my tests but thinking about how to apply CSS
styles to the manual (including to control the size of the shown
images), I've got the second idea: Company already has an online version
of the user manual with all the proper styles and images in place, so
why not link directly to it?

So, the suggested patch provides a way to do that. It introduces and
uses two new properties: ':doc-html' and ':resources'.

With it, for example, Company setup could be configured this way:

#+BEGIN_SRC lisp
("company"
  :url "https://github.com/company-mode/company-mode.git"
  :shell-command "mkdir images; mv doc/images/small images/"
  :doc "doc/company.texi"
  :doc-html ignore
  :resources (("User Manual" . "https://company-mode.github.io/manual/index.html"))
  :auto-sync t)
#+END_SRC


I also attach a version with a bit different (IMO less robust/clear)
approach. It introduces/uses one new property instead: ':doc-links'.



[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #5: Add :doc-links property to specifications --]
[-- Type: text/x-patch, Size: 4592 bytes --]

From c470c6a669c395d82247d79fa3be0084727bb8e3 Mon Sep 17 00:00:00 2001
From: YugaEgo <yet@ego.team>
Date: Tue, 11 Jan 2022 03:05:37 +0200
Subject: [PATCH] Add ':doc-links' property to specifications

Provide control over documentation linked from package page.

* elpa-admin.el (elpaa--html-insert-docs, elpaa--build-Info-1):
Utilize new property.
* README (:doc-links): Usage instructions.
---
 README        | 10 ++++++++++
 elpa-admin.el | 51 +++++++++++++++++++++++++++++++--------------------
 2 files changed, 41 insertions(+), 20 deletions(-)

diff --git a/README b/README
index a0e34ff78c..dd5d9aef41 100644
--- a/README
+++ b/README
@@ -146,6 +146,16 @@ Name of the documentation file for the package.
 It can be either an Info file, a Texinfo file, or an Org file.
 FILE can also be a list of those.
 
+** =:doc-links LINKS=
+Allows configuring documentation links for the package page.  By
+default, if =:doc= specification contains a Texinfo file, then HTML
+documentation is generated from it; the link to the generated HTML
+file is added to the package page.  To disable this behavior, set
+=:doc-links= to ~ignore~ or to an association list of the titles and
+URLs of an existing documentation.  For example:
+
+    (("User Manual" . "https://example.tld/manual.html"))
+
 ** =:make TARGET=
 Indicates that we should run ~make TARGET~ in order to build some files.
 This is run before processing =:doc=, so it can be used to generate
diff --git a/elpa-admin.el b/elpa-admin.el
index 363919d9ab..6c25be5483 100644
--- a/elpa-admin.el
+++ b/elpa-admin.el
@@ -1449,25 +1449,35 @@ arbitrary code."
   (concat (file-name-base docfile) ".html"))
 
 (defun elpaa--html-insert-docs (pkg-spec)
-  (let ((docfiles (reverse (plist-get (cdr pkg-spec) :internal--html-docs)))
-	;; `html-dir' is relative to the tarball directory, so html
-	;; references on mirrors work.  It does not include the
-	;; package name, so cross references among package docs work.
-	(html-dir (when elpaa--doc-subdirectory
-	            (file-name-as-directory elpaa--doc-subdirectory))))
-    (when (and docfiles html-dir
-	       ;; FIXME: This dir is shared, so it will always exist.
-	       ;; Should we use (expand-file-name pkg html-dir) instead?
-               (file-readable-p html-dir)) ;; html doc files were built
-      (insert "<dt>Manual</dt> <dd>\n")
-      (dolist (doc docfiles)
-	(let ((html-file (concat html-dir (cdr doc))))
-	  (insert "<a href=\"" html-file "\">"
-	          (car doc)
-	          "</a>\n")
-	  ;; FIXME: get link text from info direntry?
-	  ))
-      (insert "</dd>\n"))))
+  (let ((doc-links (elpaa--spec-get pkg-spec :doc-links)))
+    (unless (equal doc-links 'ignore)
+      (let ((docfiles (or doc-links
+                          (reverse (elpaa--spec-get
+                                    pkg-spec :internal--html-docs))))
+            ;; `html-dir' is relative to the tarball directory, so
+            ;; html references on mirrors work.  It does not include
+            ;; the package name, so cross references among package
+            ;; docs work.
+            (html-dir (when (and (not doc-links) elpaa--doc-subdirectory)
+                        (file-name-as-directory elpaa--doc-subdirectory))))
+        (when (and docfiles
+                   (or doc-links
+                       ;; FIXME: This dir is shared, so it will always
+                       ;; exist.  Should we use (expand-file-name pkg
+                       ;; html-dir) instead?
+                       (and html-dir
+                            ;; html doc files were built
+                            (file-readable-p html-dir))))
+          (insert "<dt>Documentation</dt> <dd>\n")
+          (dolist (doc docfiles)
+            (let ((html-file (if doc-links (cdr doc)
+                               (concat html-dir (cdr doc)))))
+              (insert "<a href=\"" html-file "\">"
+                      (car doc)
+                      "</a>\n")
+              ;; FIXME: get link text from info direntry?
+              ))
+          (insert "</dd>\n"))))))
 
 (defun elpaa--html-make-pkg (pkg pkg-spec files srcdir)
   (let* ((name (symbol-name (car pkg)))
@@ -2106,7 +2116,8 @@ relative to elpa root."
            t "makeinfo" "--no-split" docfile "-o" info-file)
           (message "%s" (buffer-string)))
 
-	(when html-dir (elpaa--html-build-doc pkg-spec docfile html-dir))
+        (when (and html-dir (not (elpaa--spec-get pkg-spec :doc-links)))
+          (elpaa--html-build-doc pkg-spec docfile html-dir))
 
         (setq docfile info-file)))
 
-- 
2.34.1

bug#53205: 28.0.90; [PATCH] GNU ELPA: Provide more control over linked documentation

[Lars Ingebrigtsen]

Y. E. <yet@ego.team> writes:

> My initial idea for fixing this issue was to add a property
> ':doc-asset', which could be configured to '(FROM . TO)' directories
> names, then used for moving images to the 'archive-devel/doc/company/'
> folder, as shown below:

I think that sounds OK...

> That worked fine during my tests but thinking about how to apply CSS
> styles to the manual (including to control the size of the shown
> images), I've got the second idea: Company already has an online version
> of the user manual with all the proper styles and images in place, so
> why not link directly to it?

[...]

>   :doc-html ignore
>   :resources (("User Manual" . "https://company-mode.github.io/manual/index.html"))

Well, the user wants to see the version of the manual that corresponds
to the version of the package they have installed, so just redirecting
them to an external web page doesn't seem very useful?  But perhaps I'm
misunderstanding the proposal.  (And I'm adding Stefan to the CCs;
perhaps he has some comments.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#53205: 28.0.90; [PATCH] GNU ELPA: Provide more control over linked documentation

[Y. E. via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>Lars Ingebrigtsen <larsi@gnus.org> writes:

>> Y. E. <yet@ego.team> writes:
>>
>> My initial idea for fixing this issue was to add a property
>> ':doc-asset', which could be configured to '(FROM . TO)' directories
>> names, then used for moving images to the 'archive-devel/doc/company/'
>> folder, as shown below:
>
> I think that sounds OK...

Fine by me, though with a workaround to apply custom styles. The
generated plain HTML outputs images as is, which doesn't look well in
the described case.

>
>> That worked fine during my tests but thinking about how to apply CSS
>> styles to the manual (including to control the size of the shown
>> images), I've got the second idea: Company already has an online version
>> of the user manual with all the proper styles and images in place, so
>> why not link directly to it?
>
> [...]
>
>>   :doc-html ignore
>>   :resources (("User Manual" . "https://company-mode.github.io/manual/index.html"))
>
> Well, the user wants to see the version of the manual that corresponds
> to the version of the package they have installed, so just redirecting
> them to an external web page doesn't seem very useful?  But perhaps I'm
> misunderstanding the proposal.  (And I'm adding Stefan to the CCs;
> perhaps he has some comments.)

I agree, this is a good point. Currently, the most recent version of the
manual is displayed at Company website. But it's planned to output the
stable version there in the future. So, yes, it would be nice/convenient
to have a development version generated at GNU ELPA.

bug#53205: 28.0.90; [PATCH] GNU ELPA: Provide more control over linked documentation

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Hi,

Yay!  More people working on `elpa-admin.el`!

> +** =:doc-html=
> +By default, if =:doc= specification contains a Texinfo file, then HTML
> +documentation is generated from it; the link to the generated HTML
> +file is added to the package page.  To disable this behavior, set
> +=:doc-html= to ~ignore~.

Hmm... why would you want to disable it?

> +** =:resoures LINKS=
> +Enables 'Additional Resources' section on the package page.  This must
> +be an association list of the titles and URLs of online resources.
> +For example:
> +
> +    (("User Manual" . "https://example.tld/manual.html"))

Could you give more context as for why we want this?
More specifically, the `:readme` file can already contain links to other
places, so I'm not sure why we need special support for it here
(especially since the :readme file is much more flexible and can be
changed without write access to `elpa.git`).

> I was pleased to see that automatic generation but there were two issues
> (namely, images and styles).
>
> The user manual for Company is shipped with the images; and it is
> currently impossible to configure the addition of them to the
> 'elpaa--doc-subdirectory'.

Ah... yes, this is indeed a serious problem with the current code.
But we should fix it rather than disable the generation of the
HTML manual.
[ Note that the problem also applies to Org manuals, and to the
  `:readme`s.  ]

> My initial idea for fixing this issue was to add a property
> ':doc-asset', which could be configured to '(FROM . TO)' directories
> names, then used for moving images to the 'archive-devel/doc/company/'
> folder, as shown below:

That sounds like a good plan.  

> That worked fine during my tests but thinking about how to apply CSS
> styles to the manual (including to control the size of the shown
> images), I've got the second idea: Company already has an online version
> of the user manual with all the proper styles and images in place, so
> why not link directly to it?

There are various reasons why this would not be a good solution.
One Lars already mentioned the issue of the version that may not match,
another one is that some packages don't have such a separate web page.

> I also attach a version with a bit different (IMO less robust/clear)
> approach.  It introduces/uses one new property instead: ':doc-links'.

FWIW, I like this approach better than the first patch, tho I'd still
much prefer to get the "local HTML doc" working better.


        Stefan

bug#53205: 28.0.90; [PATCH] GNU ELPA: Provide more control over linked documentation

[Y. E. via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Ah... yes, this is indeed a serious problem with the current code.
> But we should fix it rather than disable the generation of the
> HTML manual.
> [ Note that the problem also applies to Org manuals, and to the
>   `:readme`s.  ]

I'm aware of the issues reported at
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=53069 (though the second
example images are loaded fine with my setup). I looked at it because of
the images load issue coincidence. It seemed to me that changes of a
different type than discussed here would be required for that bug.

>
>> My initial idea for fixing this issue was to add a property
>> ':doc-asset', which could be configured to '(FROM . TO)' directories
>> names, then used for moving images to the 'archive-devel/doc/company/'
>> folder, as shown below:
>
> That sounds like a good plan.

After getting back to this plan, appears there's one more issue for
which I don't see a good strategy to handle.

In the final part of the 'elpaa--html-build-doc' function, it creates a
symlink to the actual HTML file; this symlink is displayed at the
package page. Naturally, putting images the suggested way ('doc-assets'
diff) works fine for the actual HTML file, but not for the symlinked.

Creating a symlinked directory, with the name(s) only one particular
package needs, in the shared '/doc/' directory doesn't look like a good
idea to me. Neither does requiring packages to comply with some naming
for the ELPA needs only.

Any advice on which approach could be suitable to handle this?


> I'd still much prefer to get the "local HTML doc" working better.

This sounds good to me.

Thinking about improving images output for Company local HTML doc, may
it become possible to allow adding small quantity per-package page
styles directly to the ELPA repository?

Having an HTML "package-name wrapper" could allow doing something like:

#pkg-company img {
    max-width: 300px;
}

bug#53205: 28.0.90; [PATCH] GNU ELPA: Provide more control over linked documentation

[Y. E. via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> Y. E. <yet@ego.team> writes:

> Thinking about improving images output for Company local HTML doc, may
> it become possible to allow adding small quantity per-package page
> styles directly to the ELPA repository?
>
> Having an HTML "package-name wrapper" could allow doing something like:
>
> #pkg-company img {
>     max-width: 300px;
> }

Sorry, my mind was confused at that moment, so I shifted the focus from
the local HTML doc styles to the main package page styles. Even though
the suggested stays valid for the main page styles, the local HTML doc
styles would have to be included to the generated HTML first, of course.

I meant to suggest the following approach:

#+BEGIN_SRC diff
diff --git a/elpa-admin.el b/elpa-admin.el
index d570c3c6aa..cad33482cb 100644
--- a/elpa-admin.el
+++ b/elpa-admin.el
@@ -2047,7 +2047,12 @@ directory; one of archive, archive-devel."
 	  (concat (make-temp-name (expand-file-name "doc")) ".html")))
     (with-temp-buffer
       (elpaa--call-sandboxed
-       t "makeinfo" "--no-split" "--html" docfile "-o" tmpfile)
+       t "makeinfo" "--no-split" "--html" docfile
+       "-c"  (format "AFTER_BODY_OPEN=<div id=\"pkg-%s\">"
+                     (car pkg-spec))
+       "-c" "PRE_BODY_CLOSE=</div>"
+       "--css-ref" "../../html/doc-html.css" ; TODO: Use robust path.
+       "-o" tmpfile)
       (message "%s" (buffer-string)))
     (rename-file tmpfile html-file t)
     ;; FIXME: Use `push' in Emacs≥28
#+END_SRC

'<elpa>/html/doc-html.css' then could contain:

#+BEGIN_SRC css
#pkg-company img {
    /* Until images output is fixed, hide them. */
    display: none;
    /* max-width: 360px; */
    /* margin-bottom: 0.8em; */
}
#+END_SRC

Would it be fine to follow this route?

bug#53205: 28.0.90; [PATCH] GNU ELPA: Provide more control over linked documentation

[Lars Ingebrigtsen]

Y. E. <yet@ego.team> writes:

>      (with-temp-buffer
>        (elpaa--call-sandboxed
> -       t "makeinfo" "--no-split" "--html" docfile "-o" tmpfile)
> +       t "makeinfo" "--no-split" "--html" docfile
> +       "-c"  (format "AFTER_BODY_OPEN=<div id=\"pkg-%s\">"
> +                     (car pkg-spec))
> +       "-c" "PRE_BODY_CLOSE=</div>"
> +       "--css-ref" "../../html/doc-html.css" ; TODO: Use robust path.
> +       "-o" tmpfile)
>        (message "%s" (buffer-string)))
>      (rename-file tmpfile html-file t)
>      ;; FIXME: Use `push' in Emacs≥28
> #+END_SRC
>
> '<elpa>/html/doc-html.css' then could contain:
>
> #+BEGIN_SRC css
> #pkg-company img {
>     /* Until images output is fixed, hide them. */
>     display: none;
>     /* max-width: 360px; */
>     /* margin-bottom: 0.8em; */
> }
> #+END_SRC
>
> Would it be fine to follow this route?

This was half a year ago, but there was apparently no followup here.
Stefan, do you have any comments?