bug#28159: Updater needs to support HTTP(S) servers

bug#28159: Updater needs to support HTTP(S) servers

[Hartmut Goebel]

[-- Attachment #1: Type: text/plain, Size: 3039 bytes --]

Hi,

our updater currently only supports FTP servers, but more and more
projects shutdown the FTP service and provide HTTP(S) servers only (e.g
the Linux kernel). For other projects, the main distribution point has
changed to HTTP and the mirrors still providing FTP at lagging (e.g.
KDE, see [1]).

A common case is to simply use Apache to serve the directories, but it
will deliver a HTML view on the directory contents (using mod_autoindex
[3]).

In [2] Ludo wrote:

    So we need a way to list the latest releases somehow.  If they publish
    JSON, XML, or some other structured info format, that’s fine too.  But
    HTTP alone is not good: we’d have to infer the information from HTML
    pages, which sounds fragile.

IMHO we can not expect project and mirror sites to provide these
additional data. Most projects simply will not do since this would
require the server to generate some data-files n the fly.

OTOH, I assume the delivered directory index pages to be well-formed
(X)HTML. Thus parsing the HTML should be quite simple: We only need to
pattern-match "<A>" tags, or – if guile has some decent one – a 
xml/html-parser use this to query the data.

Only relative links without slash (except a trailing one) have to be
handled. Links with a trailing slash can be assumed to be a directories.
(Since auto-index only works if URL is pointing to a directory and the
directory is marked by a training slash we can assume the generated
links for directories will all have the trailing slash.) At least this
would be a good start which could be refined if necessary.

Please note tha I'm not suggesting to write a general-purpose parser,
but aiming for auto-index html-pages only.

Some things I already found out:

  * Directory-listings generated by mod_autoindex can be provided as a
    simple list by passing the query-parameter "F=0" in the URL [4].
    There are other query parameters for sorting and pattern matching.
  * nginx's "ngx_http_autoindex_module" [6] seem to not use query
    parameters, but can be configured (on the server-side) to provide
    the content as XML or json. The "fancy_index" module [7] si
    documented to "Allow choosing to sort elements", but [7] does not
    state how and if "fancy" can be switched off.
  * Lighttp supports some of these options [5].

[1] http://lists.gnu.org/archive/html/guix-devel/2017-05/msg00237.html
[2] http://lists.gnu.org/archive/html/guix-devel/2017-05/msg00292.html
[3] https://httpd.apache.org/docs/2.4/mod/mod_autoindex.html
[4] https://httpd.apache.org/docs/2.4/mod/mod_autoindex.html#query
[5]
https://redmine.lighttpd.net/projects/1/wiki/Docs_ModDirlisting#Table-sorting
[6] http://nginx.org/en/docs/http/ngx_http_autoindex_module.html
[7] https://www.nginx.com/resources/wiki/modules/fancy_index/

-- 

Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |


[-- Attachment #2: 0xBF773B65.asc --]
[-- Type: application/pgp-keys, Size: 14855 bytes --]

bug#28159: Updater needs to support HTTP(S) servers

[Ludovic Courtès]

Hi Hartmut,

Hartmut Goebel <h.goebel@crazy-compilers.com> skribis:

> our updater currently only supports FTP servers,

More precisely, several updaters rely on FTP (gnu, kernel.org, kde,
etc. see (guix gnu-maintenance)), but others rely on structured data
retrieved over HTTP(S) (pypi, cran, elpa, etc.)

> but more and more projects shutdown the FTP service and provide
> HTTP(S) servers only (e.g the Linux kernel). For other projects, the
> main distribution point has changed to HTTP and the mirrors still
> providing FTP at lagging (e.g.  KDE, see [1]).

The FTP updater had the advantage of being simple and fairly generic,
but here we’ll probably have to go for project specific methods.

So I would suggest picking one updater, say kde, and implementing it
using whatever metadata can be retrieved from kde.org.

This should be simpler than trying to figure out a generic method that
will work for every software project.

HTH!

Ludo’.

bug#28159: Updater needs to support HTTP(S) servers

[Hartmut Goebel]

Am 22.08.2017 um 10:57 schrieb Ludovic Courtès:
> So I would suggest picking one updater, say kde, and implementing it
> using whatever metadata can be retrieved from kde.org.

I'm not sure if I understood what you mean with "whatever metadata can
be retrieved from kde.org".

By change, download.kde.org indeed provides a "ls-lR" and "ls-lR.bz"
file at the top-level. I was not aware of this up to just now. Using
this might be an option (It is lagging a bit, though I think this is
acceptable. From what I've ssen I guess it is generated each hour if
some file changed.)

So for kde we might find a simpler solution. But in the long-run IMHO we
need a simple html parser.

I'm not skilled enough in scheme/guile to write such a parser, sorry.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

bug#28159: Updater needs to support HTTP(S) servers

[Ludovic Courtès]

Hartmut Goebel <h.goebel@crazy-compilers.com> skribis:

> Am 22.08.2017 um 10:57 schrieb Ludovic Courtès:
>> So I would suggest picking one updater, say kde, and implementing it
>> using whatever metadata can be retrieved from kde.org.
>
> I'm not sure if I understood what you mean with "whatever metadata can
> be retrieved from kde.org".

I mean using package metadata provided by kde.org (maybe they have a
JSON representation of the package graph or something?), or the ‘ls-lR’
files at worst.

> By change, download.kde.org indeed provides a "ls-lR" and "ls-lR.bz"
> file at the top-level. I was not aware of this up to just now. Using
> this might be an option (It is lagging a bit, though I think this is
> acceptable. From what I've ssen I guess it is generated each hour if
> some file changed.)

Sounds good.

> So for kde we might find a simpler solution. But in the long-run IMHO we
> need a simple html parser.

In some cases yes, but maybe not in all cases.  I also suspect that
something that attempts to extract the latest release number from a home
page may be brittle.

> I'm not skilled enough in scheme/guile to write such a parser, sorry.

This can be done along these lines:

--8<---------------cut here---------------start------------->8---
scheme@(guile-user)> ,use(sxml simple)
scheme@(guile-user)> ,use(web client)
scheme@(guile-user)> ,use(sxml match)
scheme@(guile-user)> (define page (xml->sxml (call-with-values 
						 (lambda ()
						   (http-get "http://www.gnu.org/software/guix/guix.html" #:streaming? #t))
					       (lambda (response port)
						 port))))
--8<---------------cut here---------------end--------------->8---

… where ‘page’ is the SXML representation of the web page.  The
difficulty is to browse this page (using ‘match’ or ‘sxml-match’.)

HTH,
Ludo’.

bug#28159: Updater needs to support HTTP(S) servers

[Ludovic Courtès]

Hello,

I just learned that ftp://ftp.gnu.org will be retired on Nov. 1st, 2017,
so we’ll have to implement a replacement for the ‘gnu’ updater at least.

At worst, we’ll parse HTML index files like the one at
<https://ftp.gnu.org/gnu/guile/>, but I’m trying to see if the FSF
sysadmin could generate an ‘ls-lR’ file or similar.

Ludo’.

bug#28159: Updater needs to support HTTP(S) servers

[Hartmut Goebel]

Hi,
> I just learned that ftp://ftp.gnu.org will be retired on Nov. 1st, 2017,
> so we’ll have to implement a replacement for the ‘gnu’ updater at least.

By change, also this server provides a `ls-lrRt.txt.gz` file.
Unfurtunaly is as a slightly different (date-) format than the one at
kde.org:

kde:

    drwxr-xr-x   3 ftpadmin packager       6 2000-10-01 14:07 adm


gnu:

    drwxr-xr-x   2 root root      4096 Aug  2  2003 third-party


Also by chance ftp.gnu.org also provides a file `find.txt.gz`, listing
all files, including the full path:

    ./video/Stephen_Fry-Happy_Birthday_GNU-nq_600px_425kbit.ogv
    ./old-gnu/g77/g77-0.5.21.tar.gz
    ./old-gnu/guile
    ./old-gnu/guile/guile-www-1.0.1.tar.gz
    ./old-gnu/guile/guile-1.3.2.tar.gz


> At worst, we’ll parse HTML index files like the one at
> <https://ftp.gnu.org/gnu/guile/>,

This is what Ihis bug is about :-) Please mind the query-parameters one
can pass to apache: <https://ftp.gnu.org/gnu/guile/?F=0> is much more terse.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

bug#28159: Updater needs to support HTTP(S) servers

[Ludovic Courtès]

Hi Hartmut,

Hartmut Goebel <h.goebel@crazy-compilers.com> skribis:

> Also by chance ftp.gnu.org also provides a file `find.txt.gz`, listing
> all files, including the full path:
>
>     ./video/Stephen_Fry-Happy_Birthday_GNU-nq_600px_425kbit.ogv
>     ./old-gnu/g77/g77-0.5.21.tar.gz
>     ./old-gnu/guile
>     ./old-gnu/guile/guile-www-1.0.1.tar.gz
>     ./old-gnu/guile/guile-1.3.2.tar.gz

This one is nice and smaller than ‘ls-lR’.  I reimplemented the GNU
updater in terms of this file, and kept the previous FTP-based updater
around for GNU packages not hosted on ftp.gnu.org:

  https://git.savannah.gnu.org/cgit/guix.git/commit/?id=100b216d8a4218daec4a79024d62d54b52dc07be

“guix refresh -t gnu” is now much faster.

The next step may be to have a more-or-less generic updater based on
‘ls-lR’ files.

Thanks,
Ludo’.

bug#28159: Updater needs to support HTTP(S) servers

[Ludovic Courtès]

Hello sysadmins!

How frequently is ftp.gnu.org/find.txt.gz updated?  It seems to be less
than once a day.

Could we arrange to have it regenerated every time a new file is
uploaded?

I suppose uploads aren’t this frequent, but regenerating ‘find.txt.gz’
right after an upload would ensure that it’s always current.

Thanks in advance!

Ludo’.

PS: For context, see <https://debbugs.gnu.org/cgi/bugreport.cgi?bug=28159#20>.
PPS: Please reply to all.

bug#28159: [gnu.org #1238656] Re: bug#28159: Updater needs to support HTTP(S) servers

[=?UTF-8?B?UnViw6luIFJvZHLDrWd1ZXogUMOpcmV6?= via RT]

On Fri Sep 08 04:31:05 2017, ludo@gnu.org wrote:
> Hello sysadmins!

Hi Ludo

> How frequently is ftp.gnu.org/find.txt.gz updated?  It seems to be
> less
> than once a day.

It is run by cron.daily

> Could we arrange to have it regenerated every time a new file is
> uploaded?
> 
> I suppose uploads aren’t this frequent, but regenerating ‘find.txt.gz’
> right after an upload would ensure that it’s always current.

I've modified the cron script to apply that change, now on the look for the next upload to see if it worked.

Regards,
-- 
Ruben Rodriguez | Senior Systems Administrator, Free Software Foundation
GPG Key: 05EF 1D2F FE61 747D 1FC8  27C3 7FAC 7D26 472F 4409
https://fsf.org | https://gnu.org

bug#28159: [gnu.org #1238656] Re: bug#28159: Updater needs to support HTTP(S) servers

[Ludovic Courtès]

Hi Rubén,

"Rubén Rodríguez Pérez via RT" <sysadmin@gnu.org> skribis:

> On Fri Sep 08 04:31:05 2017, ludo@gnu.org wrote:

[...]

>> Could we arrange to have it regenerated every time a new file is
>> uploaded?
>> 
>> I suppose uploads aren’t this frequent, but regenerating ‘find.txt.gz’
>> right after an upload would ensure that it’s always current.
>
> I've modified the cron script to apply that change, now on the look for the next upload to see if it worked.

Awesome, thanks!

Ludo’.

bug#28159: [gnu.org #1238656] Re: bug#28159: Updater needs to support HTTP(S) servers

[Rubén Rodríguez Pérez via RT]

On Fri Sep 08 04:31:05 2017, ludo@gnu.org wrote:
> Hello sysadmins!

Hi Ludo

> How frequently is ftp.gnu.org/find.txt.gz updated?  It seems to be
> less
> than once a day.

It is run by cron.daily

> Could we arrange to have it regenerated every time a new file is
> uploaded?
> 
> I suppose uploads aren’t this frequent, but regenerating ‘find.txt.gz’
> right after an upload would ensure that it’s always current.

I've modified the cron script to apply that change, now on the look for the next upload to see if it worked.

Regards,
-- 
Ruben Rodriguez | Senior Systems Administrator, Free Software Foundation
GPG Key: 05EF 1D2F FE61 747D 1FC8  27C3 7FAC 7D26 472F 4409
https://fsf.org | https://gnu.org

bug#28159: Updater needs to support HTTP(S) servers

[Ludovic Courtès]

ludo@gnu.org (Ludovic Courtès) skribis:

> This one is nice and smaller than ‘ls-lR’.  I reimplemented the GNU
> updater in terms of this file, and kept the previous FTP-based updater
> around for GNU packages not hosted on ftp.gnu.org:
>
>   https://git.savannah.gnu.org/cgit/guix.git/commit/?id=100b216d8a4218daec4a79024d62d54b52dc07be
>
> “guix refresh -t gnu” is now much faster.

Commit c1d8b3b3b5af8282328b87dd7a8d09357cbb0af7 rewrites the GNOME
updater in terms of the ‘cache.json’ files that can be found in each
package directory at <https://ftp.gnome.org/pub/gnome/sources>.

Ludo’.

bug#28159: Updater needs to support HTTP(S) servers

[Ludovic Courtès]

ludo@gnu.org (Ludovic Courtès) skribis:

> ludo@gnu.org (Ludovic Courtès) skribis:
>
>> This one is nice and smaller than ‘ls-lR’.  I reimplemented the GNU
>> updater in terms of this file, and kept the previous FTP-based updater
>> around for GNU packages not hosted on ftp.gnu.org:
>>
>>   https://git.savannah.gnu.org/cgit/guix.git/commit/?id=100b216d8a4218daec4a79024d62d54b52dc07be
>>
>> “guix refresh -t gnu” is now much faster.
>
> Commit c1d8b3b3b5af8282328b87dd7a8d09357cbb0af7 rewrites the GNOME
> updater in terms of the ‘cache.json’ files that can be found in each
> package directory at <https://ftp.gnome.org/pub/gnome/sources>.

Commit 5230dce154a8861d806fcd667f2d424def571ed6 rewrites the kernel.org
updater so that it’s based on an analysis of HTML directory listings
such as <https://cdn.kernel.org/pub/software/scm/git/>.

Ludo’.

bug#28159: Updater needs to support HTTP(S) servers

[Hartmut Goebel]

Am 22.08.17 um 10:57 schrieb Ludovic Courtès:
> More precisely, several updaters rely on FTP (gnu, kernel.org, kde,
> etc. see (guix gnu-maintenance)), but others rely on structured data
> retrieved over HTTP(S) (pypi, cran, elpa, etc.)

For the records: KDE no longer relies on FTP access. It now fetches the
ls-lR.bz2 file list using HTTPS from download.kde.org, converts it into
a list of file paths and caches the list.

See http://debbugs.gnu.org/cgi/bugreport.cgi?bug=36919
commit 4eb69bf0d33810886ee118f38989cef696e4c868

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

bug#28159: Closing bug #28159? Updater needs to support HTTP(S) servers

[Brice Waegeneire]

Hello Guix,

It looks like now most of the major updaters that relied on FTP (GNU,
kernel.org, KDE and Gnbome) now support HTTP(S). I think we can close 
this
bug.

Ludovic Courtès wrote on Tue Aug 22 10:57:20+0200 2017:
> More precisely, several updaters rely on FTP (gnu, kernel.org, kde,
> etc. see (guix gnu-maintenance)), but others rely on structured data
> retrieved over HTTP(S) (pypi, cran, elpa, etc.)

Ludovic Courtès wrote on Sun Sep 03 23:40:18+0200 2017:
> This one is nice and smaller than ‘ls-lR’.  I reimplemented the GNU
> updater in terms of this file, and kept the previous FTP-based updater
> around for GNU packages not hosted on ftp.gnu.org:

Ludovic Courtès wrote on Tue Sep 26 00:39:54+0200 2017:
> Commit c1d8b3b3b5af8282328b87dd7a8d09357cbb0af7 rewrites the GNOME
> updater in terms of the ‘cache.json’ files that can be found in each
> package directory at <https://ftp.gnome.org/pub/gnome/sources>.

Ludovic Courtès wrote on Sat Nov 10 23:38:16+0100 2018:
> Commit 5230dce154a8861d806fcd667f2d424def571ed6 rewrites the kernel.org
> updater so that it’s based on an analysis of HTML directory listings
> such as <https://cdn.kernel.org/pub/software/scm/git/>.

Hartmut Goebel wrote on Tue Sep 10 19:25:58+0200 2019:
> For the records: KDE no longer relies on FTP access. It now fetches the
> ls-lR.bz2 file list using HTTPS from download.kde.org, converts it into
> a list of file paths and caches the list.

- Brice

bug#28159: Closing bug #28159? Updater needs to support HTTP(S) servers

[Ludovic Courtès]

Hi Brice,

Brice Waegeneire <brice@waegenei.re> skribis:

> It looks like now most of the major updaters that relied on FTP (GNU,
> kernel.org, KDE and Gnbome) now support HTTP(S). I think we can close
> this
> bug.

Yup.  There’s still the ‘gnu-ftp’ and the ‘xorg’ updaters which,
according to ‘guix refresh --list-updaters’, account for 2.2% of the
packages.  We can change them later when it becomes necessary.

Closing, thank you!

Ludo’.