[PATCH] A patch to generate paginated packages page

[PATCH] A patch to generate paginated packages page

[Alex Sassmannshausen]

Hello,

Following from repeated discussions on the mailing list about the
packages page, here a rough attempt at fixing the issue.

The patch below adds/modifies procedures so we can generate a packages
page for each "starting character" of package names.

The idea would be that we have a page for all packages starting with
"A", with "B", etc.

There will also still be a page with all packages on it.

The logic is basically there I think — but the CSS will need some
tweaking, and I'm also not 100% sure on testing it.

I wanted to get it out here for comments first of all.

What do you think?

Cheers,

Alex

Alex Sassmannshausen (1):
  Generate multiple paginated packages pages.

 website/www.scm          | 12 +++++++-
 website/www/packages.scm | 74 +++++++++++++++++++++++++++++++++++++-----------
 2 files changed, 69 insertions(+), 17 deletions(-)

-- 
2.10.1

[PATCH] Generate multiple paginated packages pages.

[Alex Sassmannshausen]

* website/www.scm (%web-pages): Add prototype code for generating our
  packages pages.
* website/www/packages.scm (all-packages): Re-factor to
  `packages-by-grouping`.
  (paginated-packages-page): New procedure.
  (packages-page): Tweak for use by `paginated-packages-page` as well as
  standalone.
  (issues-page): Use `packages-by-grouping`.
---
 website/www.scm          | 12 +++++++-
 website/www/packages.scm | 74 +++++++++++++++++++++++++++++++++++++-----------
 2 files changed, 69 insertions(+), 17 deletions(-)

diff --git a/website/www.scm b/website/www.scm
index 459629f..489260e 100644
--- a/website/www.scm
+++ b/website/www.scm
@@ -293,7 +293,17 @@ Distribution.")
     ("download/index.html" ,download-page)
     ("help/index.html" ,help-page)
     ("security/index.html" ,security-page)
-    ;; ("packages/index.html" ,packages-page) ; Need Guix
+    ;; Paged packages pages!                     Need Guix
+    ;; Not 100% if this how the website is supposed to work.  Would
+    ;; appreciate comment on this.
+    ;; ,@(map (lambda (grouping)
+    ;;          `(,(string-append "packages/" grouping ".html")
+    ;;            (paginated-packages-page ,grouping)))
+    ;;        (cons "0-9" (map string '(#\a #\b #\c #\d #\e #\f #\g #\h
+    ;;                                  #\i #\j #\k #\l #\m #\n #\o #\p
+    ;;                                  #\q #\r #\s #\t #\u #\v #\w #\x
+    ;;                                  #\y #\z))))
+    ;; ("packages/index.html" ,packages-page)
     ;; ("packages/issues.html" ,issues-page)
     ))
 
diff --git a/website/www/packages.scm b/website/www/packages.scm
index ccafa28..9d39bc6 100644
--- a/website/www/packages.scm
+++ b/website/www/packages.scm
@@ -438,18 +438,39 @@ PACKAGES."
 ;;; Pages.
 ;;;
 
-(define (all-packages)
-  "Return the list of all package objects, sorted by name."
-  (sort (fold-packages (lambda (package lst)
-                         (cons (or (package-replacement package)
-                                   package)
-                               lst))
-                       '())
-        (lambda (p1 p2)
-          (string<? (package-name p1)
-                    (package-name p2)))))
-
-(define (packages-page)
+(define packages-by-grouping
+  (let ((packages (sort (fold-packages (lambda (package lst)
+                             (cons (or (package-replacement package)
+                                       package)
+                                   lst))
+                           '())
+            (lambda (p1 p2)
+              (string<? (package-name p1)
+                        (package-name p2))))))
+    (lambda* (#:optional (grouping 'all))
+      "Return an alphabetically sorted list of Guix packages, limited
+to those matching GROUPING.  GROUPING can be 'all for all packages,
+the string '0-9' for all packages starting with digits, or a string of
+a single, lower-case letter for a list of all packages starting with
+that letter."
+      (match grouping
+        ('all packages)
+        ("0-9" (filter (compose (cut char-set-contains? char-set:digit <>)
+                                first string->list package-name)
+                       packages))
+        (letter (filter (lambda (package)
+                          (string=? (string-take (package-name package) 1)
+                                    letter))
+                    packages))))))
+
+(define (paginated-packages-page grouping)
+  "Return a packages page that contains only content for the packages
+that match GROUPING (either the string '0-9' or a string of one
+letter)."
+  (packages-page (string-upcase grouping) (packages-by-grouping grouping)))
+
+(define* (packages-page #:optional (grouping "All")
+                        (packages (packages-by-grouping)))
   `(html (@ (lang "en"))
 	 ,(html-page-header "Packages" #:css "packages.css" #:js "packages.js")
 	 (body
@@ -458,17 +479,38 @@ PACKAGES."
 
 	  (div (@ (id "content-box"))
 	       (article
-		(h1 "Packages")
+		(h1 ,(string-append "Packages [" grouping "]"))
 		(p "GNU Guix provides "
                    ,(number* (fold-packages (lambda (p n) (+ 1 n)) 0))
                    " packages transparently "
 		   (a (@ (href "http://hydra.gnu.org/jobset/gnu/master#tabs-status"))
 		      "available as pre-built binaries")
-		   ". This is a complete list of the packages.  Our "
+		   ". These pages provide a complete list of the packages.
+  Our "
 		   (a (@ (href "http://hydra.gnu.org/jobset/gnu/master"))
 		      "continuous integration system")
 		   " shows their current build status.")
-		,(packages->sxml (all-packages))
+                ;; fixme: Ensure these pages work.
+                (p "You can browse packages indexed by their first letter, or 
+you can view "
+                   (a (@ (href "/software/guix/packages/all"))
+                      "all packages on a single page."))
+                (ul
+                 ,@(map (lambda (grouping)
+                          `(li (@ (id ,(string-append grouping "-link"))
+                                  (class "package-index-link"))
+                               (a (@ (href ,(string-append "/software/guix/packages/"
+                                                           grouping ".html")))
+
+                                  ,(string-upcase grouping))))
+                        (cons "0-9"
+                              (map string
+                                   '(#\a #\b #\c #\d #\e #\f #\g #\h
+                                     #\i #\j #\k #\l #\m #\n #\o #\p
+                                     #\q #\r #\s #\t #\u #\v #\w #\x
+                                     #\y #\z)))))
+
+                ,(packages->sxml packages)
 
                 (p "Updated " ,(date->string (current-date) "~B ~e, ~Y")
                    ".")))
@@ -492,7 +534,7 @@ reported by "
                                  "manual/html_node/Invoking-guix-lint.html")))
                       (code "guix lint")) ".")
 
-		,(packages->issue-sxml (all-packages)
+		,(packages->issue-sxml (packages-by-grouping)
                                        #:checkers checkers)
 
                 (p "Updated " ,(date->string (current-date) "~B ~e, ~Y")
-- 
2.10.1

Re: [PATCH] Generate multiple paginated packages pages.

[Ludovic Courtès]

Hello Alex!

Alex Sassmannshausen <alex.sassmannshausen@gmail.com> skribis:

> * website/www.scm (%web-pages): Add prototype code for generating our
>   packages pages.
> * website/www/packages.scm (all-packages): Re-factor to
>   `packages-by-grouping`.
>   (paginated-packages-page): New procedure.
>   (packages-page): Tweak for use by `paginated-packages-page` as well as
>   standalone.
>   (issues-page): Use `packages-by-grouping`.

Neat!

I had in mind something that would involve JS code in the browser and
would allow people to make generic searches.  However, what you suggest
has the advantage of not requiring JS at all and it’s convenient as
well, so I guess we should go for it.  :-)

> diff --git a/website/www.scm b/website/www.scm
> index 459629f..489260e 100644
> --- a/website/www.scm
> +++ b/website/www.scm
> @@ -293,7 +293,17 @@ Distribution.")
>      ("download/index.html" ,download-page)
>      ("help/index.html" ,help-page)
>      ("security/index.html" ,security-page)
> -    ;; ("packages/index.html" ,packages-page) ; Need Guix
> +    ;; Paged packages pages!                     Need Guix
> +    ;; Not 100% if this how the website is supposed to work.  Would
> +    ;; appreciate comment on this.
> +    ;; ,@(map (lambda (grouping)
> +    ;;          `(,(string-append "packages/" grouping ".html")
> +    ;;            (paginated-packages-page ,grouping)))
> +    ;;        (cons "0-9" (map string '(#\a #\b #\c #\d #\e #\f #\g #\h
> +    ;;                                  #\i #\j #\k #\l #\m #\n #\o #\p
> +    ;;                                  #\q #\r #\s #\t #\u #\v #\w #\x
> +    ;;                                  #\y #\z))))

Yes this should work.

> -(define (all-packages)
> -  "Return the list of all package objects, sorted by name."
> -  (sort (fold-packages (lambda (package lst)
> -                         (cons (or (package-replacement package)
> -                                   package)
> -                               lst))
> -                       '())
> -        (lambda (p1 p2)
> -          (string<? (package-name p1)
> -                    (package-name p2)))))
> -
> -(define (packages-page)
> +(define packages-by-grouping
> +  (let ((packages (sort (fold-packages (lambda (package lst)
> +                             (cons (or (package-replacement package)
> +                                       package)
> +                                   lst))
> +                           '())
> +            (lambda (p1 p2)
> +              (string<? (package-name p1)
> +                        (package-name p2))))))

I think we should keep ‘all-packages’ instead of inlining it here.

> +                ;; fixme: Ensure these pages work.
> +                (p "You can browse packages indexed by their first letter, or 
> +you can view "
> +                   (a (@ (href "/software/guix/packages/all"))
> +                      "all packages on a single page."))

What about generating a list of links instead, like:

  (map (lambda (group)
         `(a (@ (href ,(group-file-name group))) ,(group-name group)))
       %groups)

where:

  (define %groups
    ;; List of package groups.
    (cons "0-9" (map string '(#\a …))))

  (define (group-file-name group)
    (match group
      ("0-9" "0-9.html")
      …))

  …

?

With these changes, that’s OK for me.

I’ll adjust the cron job on hydra.gnu.org to generate all the pages when
it’s committed.

Thank you!

Ludo’.

[PATCH] Revised generate multiple package pages.

[Alex Sassmannshausen]

Hello,

Here a revised patch generating multiple package pages.

Since the last version I've:
* added CSS styling
* rebased on master
* incorporated ludo's feedback
* tested it locally

Local testing seems to work a charm, except for the appearance of #f in
the software table, the emission of warnings during site compilation
and the failure of the Javascript folding scripts.

All of those problems seemed to happen on master too, so I concluded
they were artifacts from local testing.

Finally, please note that the paginated pages page is wrapped in an
additional lambda.  This is to make the commented section in (www)
work, but might be incorrect in how those pages are actually generated
using cron scripts…

Best wishes,

Alex

Alex Sassmannshausen (1):
  Generate multiple paginated packages pages.

 website/static/base/css/packages.css |  8 ++-
 website/www.scm                      |  8 ++-
 website/www/packages.scm             | 96 ++++++++++++++++++++++++++++--------
 3 files changed, 89 insertions(+), 23 deletions(-)

-- 
2.10.2

[PATCH] Generate multiple paginated packages pages.

[Alex Sassmannshausen]

* website/www.scm (%web-pages): Add code for generating our packages
  pages.
* website/www/packages.scm (packages-by-grouping): New procedure.
  (paginated-packages-page): New procedure.
  (packages-page): Tweak for use by `paginated-packages-page` as well as
  standalone.
* website/static/base/css/packages.css (li.package-index-link): Add
  styling.
---
 website/static/base/css/packages.css |  8 ++-
 website/www.scm                      |  8 ++-
 website/www/packages.scm             | 96 ++++++++++++++++++++++++++++--------
 3 files changed, 89 insertions(+), 23 deletions(-)

diff --git a/website/static/base/css/packages.css b/website/static/base/css/packages.css
index 177f416..d218c51 100644
--- a/website/static/base/css/packages.css
+++ b/website/static/base/css/packages.css
@@ -2,6 +2,12 @@
 
 @import url("article.css");
 
+li.package-index-link {
+    list-style: none;
+    display: inline;
+    margin: 0 0.3em;
+}
+
 a {
     transition: all 0.3s;
 }
@@ -82,4 +88,4 @@ img.status-icon {
     position: absolute;
     top: 0px;
     left: 0px;
-}
\ No newline at end of file
+}
diff --git a/website/www.scm b/website/www.scm
index 59e917a..7ca6a78 100644
--- a/website/www.scm
+++ b/website/www.scm
@@ -27,6 +27,7 @@
   #:use-module (www about)
   #:use-module (www contribute)
   #:use-module (www help)
+  #:use-module (www packages)
   #:use-module (www security)
   #:use-module (www news)
   #:use-module (haunt post)
@@ -293,7 +294,12 @@ Distribution.")
     ("download/index.html" ,download-page)
     ("help/index.html" ,help-page)
     ("security/index.html" ,security-page)
-    ;; ("packages/index.html" ,packages-page) ; Need Guix
+    ;; ,@(map (lambda (group)
+    ;;          `(,(string-append "packages/" group ".html")
+    ;;            ,(paginated-packages-page group)))
+    ;;        %groups)
+    ;; ("packages/index.html" ,(paginated-packages-page "0-9"))
+    ;; ("packages/all.html" ,packages-page)
     ;; ("packages/issues.html" ,issues-page)
     ))
 
diff --git a/website/www/packages.scm b/website/www/packages.scm
index 9f345ae..63cea3a 100644
--- a/website/www/packages.scm
+++ b/website/www/packages.scm
@@ -44,7 +44,9 @@
   #:use-module (srfi srfi-26)
   #:use-module (texinfo)
   #:use-module (texinfo html)
-  #:export (packages-page
+  #:export (%groups
+            packages-page
+            paginated-packages-page
             issues-page))
 
 (define lookup-gnu-package
@@ -441,6 +443,21 @@ PACKAGES."
 ;;; Pages.
 ;;;
 
+(define %groups
+  ;; List of package groups.
+  (cons "0-9"
+        (map string
+             '(#\a #\b #\c #\d #\e #\f #\g #\h
+               #\i #\j #\k #\l #\m #\n #\o #\p
+               #\q #\r #\s #\t #\u #\v #\w #\x
+               #\y #\z))))
+
+(define (group-file-name group)
+  (string-append "/packages/" group ".html"))
+
+(define (group-name group)
+  (string-upcase group))
+
 (define (all-packages)
   "Return the list of all package objects, sorted by name."
   (sort (fold-packages (lambda (package lst)
@@ -452,29 +469,66 @@ PACKAGES."
           (string<? (package-name p1)
                     (package-name p2)))))
 
-(define (packages-page)
+(define packages-by-grouping
+  (lambda* (#:optional (grouping 'all))
+    "Return an alphabetically sorted list of Guix packages, limited
+to those matching GROUPING.  GROUPING can be 'all for all packages,
+the string '0-9' for all packages starting with digits, or a string of
+a single, lower-case letter for a list of all packages starting with
+that letter."
+    (match grouping
+      ('all (all-packages))
+      ("0-9" (filter (compose (cut char-set-contains? char-set:digit <>)
+                              first string->list package-name)
+                     (all-packages)))
+      (letter (filter (lambda (package)
+                        (string=? (string-take (package-name package) 1)
+                                  letter))
+                      (all-packages))))))
+
+(define (paginated-packages-page grouping)
+  "Return a packages page that contains only content for the packages
+that match GROUPING (either the string '0-9' or a string of one
+letter)."
+  (lambda ()
+    (packages-page (string-upcase grouping) (packages-by-grouping grouping))))
+
+(define* (packages-page #:optional (grouping "All")
+                        (packages (all-packages)))
   `(html (@ (lang "en"))
-	 ,(html-page-header "Packages" #:css "packages.css" #:js "packages.js")
-	 (body
-	  ,(html-page-description)
-	  ,(html-page-links)
-
-	  (div (@ (id "content-box"))
-	       (article
-		(h1 "Packages")
-		(p "GNU Guix provides "
+         ,(html-page-header "Packages" #:css "packages.css" #:js "packages.js")
+         (body
+          ,(html-page-description)
+          ,(html-page-links)
+
+          (div (@ (id "content-box"))
+               (article
+                (h1 ,(string-append "Packages [" grouping "]"))
+                (p "GNU Guix provides "
                    ,(number* (fold-packages (lambda (p n) (+ 1 n)) 0))
                    " packages transparently "
-		   (a (@ (href "http://hydra.gnu.org/jobset/gnu/master#tabs-status"))
-		      "available as pre-built binaries")
-		   ". This is a complete list of the packages.  Our "
-		   (a (@ (href "http://hydra.gnu.org/jobset/gnu/master"))
-		      "continuous integration system")
-		   " shows their current build status "
-		   "(Updated " ,(date->string (current-date) "~B ~e, ~Y") ").")
-		,(packages->sxml (all-packages))))
-
-	  ,(html-page-footer))))
+                   (a (@ (href "http://hydra.gnu.org/jobset/gnu/master#tabs-status"))
+                      "available as pre-built binaries")
+                   ". These pages provide a complete list of the packages.
+  Our "
+                   (a (@ (href "http://hydra.gnu.org/jobset/gnu/master"))
+                      "continuous integration system")
+                   " shows their current build status "
+                   "(Updated " ,(date->string (current-date) "~B ~e, ~Y") ").")
+                (p "You can browse packages indexed by their first letter, or
+you can view "
+                   (a (@ (href "/packages/all.html"))
+                      "all packages on a single page."))
+                (ul
+                 ,@(map (lambda (group)
+                          `(li (@ (id ,(string-append group "-link"))
+                                  (class "package-index-link"))
+                               (a (@ (href ,(group-file-name group)))
+                                  ,(group-name group))))
+                        %groups))
+                ,(packages->sxml packages)))
+
+          ,(html-page-footer))))
 
 (define* (issues-page #:key (checkers %issue-checkers))
   `(html
-- 
2.10.2

Re: [PATCH] Generate multiple paginated packages pages.

[Ludovic Courtès]

Alex Sassmannshausen <alex.sassmannshausen@gmail.com> skribis:

> * website/www.scm (%web-pages): Add code for generating our packages
>   pages.
> * website/www/packages.scm (packages-by-grouping): New procedure.
>   (paginated-packages-page): New procedure.
>   (packages-page): Tweak for use by `paginated-packages-page` as well as
>   standalone.
> * website/static/base/css/packages.css (li.package-index-link): Add
>   styling.

Sure, please push!

I thought we had agreed on that one already, but now I see this message
as unread in my inbox.  Go figure!

Thank you Alex!

Ludo’.

Re: [PATCH] Generate multiple paginated packages pages.

[Alex Sassmannshausen]

Ludovic Courtès writes:

> Alex Sassmannshausen <alex.sassmannshausen@gmail.com> skribis:
>
>> * website/www.scm (%web-pages): Add code for generating our packages
>>   pages.
>> * website/www/packages.scm (packages-by-grouping): New procedure.
>>   (paginated-packages-page): New procedure.
>>   (packages-page): Tweak for use by `paginated-packages-page` as well as
>>   standalone.
>> * website/static/base/css/packages.css (li.package-index-link): Add
>>   styling.
>
> Sure, please push!

Done.

> I thought we had agreed on that one already, but now I see this message
> as unread in my inbox.  Go figure!

Ah, indeed, we had agreed on the previous version of the patch.  Anyway,
thanks for reconfirming.

Alex

Re: [PATCH] Generate multiple paginated packages pages.

[Ludovic Courtès]

Hello!

It’s on-line now!

  https://www.gnu.org/software/guix/packages/

Note that I removed the all-in-one package page because the poor CVS
server at gnu.org simply can’t handle it.

I’ve updated the cron job so if everything goes well it’ll be updated
every night again.

Thanks Alex!

Ludo’.

Re: [PATCH] Generate multiple paginated packages pages.

[Jan Synáček]

On Thu, Dec 8, 2016 at 11:00 AM, Ludovic Courtès <ludo@gnu.org> wrote:
> Hello!
>
> It’s on-line now!
>
>   https://www.gnu.org/software/guix/packages/

Just a cosmetic issue that I have noticed. In the expanded package
description, hovering over the check marks next to architectures says
"unknown" for both green and red check mark. Not sure if it's supposed
to be like that.

Anyways, great work!

Cheers,
-- 
Jan Synáček

Re: [PATCH] Generate multiple paginated packages pages.

[Alex Sassmannshausen]

Ludovic Courtès writes:

> Hello!
>
> It’s on-line now!
>
>   https://www.gnu.org/software/guix/packages/

Neat! Nice to see it there :-)

> Note that I removed the all-in-one package page because the poor CVS
> server at gnu.org simply can’t handle it.

Sure — I figure this way of browsing is more user friendly anyway…

> I’ve updated the cron job so if everything goes well it’ll be updated
> every night again.
>
> Thanks Alex!
>
> Ludo’.

Re: [PATCH] Generate multiple paginated packages pages.

[John Darrington]

[-- Attachment #1: Type: text/plain, Size: 958 bytes --]

On Thu, Dec 08, 2016 at 12:29:22PM +0100, Alex Sassmannshausen wrote:
     
     > Note that I removed the all-in-one package page because the poor CVS
     > server at gnu.org simply can???t handle it.
     
     Sure ??? I figure this way of browsing is more user friendly anyway???
     
Well yes and no.

It's more friendly in the respect that it no longer takes 2 minutes to load.

But the down side is that I can no longer load the page and then use my 
browser's "Find in Page" button to search for a package---Now I have to
know what the package is called before I can search for it.

Maybe sometime in the future a "browse by category" or "search for packages"
feature will be available?

J'
     

-- 
Avoid eavesdropping.  Send strong encrypted email.
PGP Public key ID: 1024D/2DE827B3 
fingerprint = 8797 A26D 0854 2EAB 0285  A290 8A67 719C 2DE8 27B3
See http://sks-keyservers.net or any PGP keyserver for public key.


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [PATCH] Generate multiple paginated packages pages.

[Alex Sassmannshausen]

Heya

John Darrington writes:

> On Thu, Dec 08, 2016 at 12:29:22PM +0100, Alex Sassmannshausen wrote:
>      
>      > Note that I removed the all-in-one package page because the poor CVS
>      > server at gnu.org simply can???t handle it.
>      
>      Sure ??? I figure this way of browsing is more user friendly anyway???
>      
> Well yes and no.
>
> It's more friendly in the respect that it no longer takes 2 minutes to load.
>
> But the down side is that I can no longer load the page and then use my 
> browser's "Find in Page" button to search for a package---Now I have to
> know what the package is called before I can search for it.

Fair point!  I think Luis Felipe is thinking of a more comprehensive
approach to the package pages — mine was more of a quick fix to solve an
urgent problem.

> Maybe sometime in the future a "browse by category" or "search for packages"
> feature will be available?
>
> J'