From: Leo Prikler <leo.prikler@student.tugraz.at>
To: Maxime Devos <maximedevos@telenet.be>, 49828@debbugs.gnu.org
Subject: [bug#49828] [PATCH 06/20] guix: Add ContentDB importer.
Date: Thu, 05 Aug 2021 18:41:00 +0200 [thread overview]
Message-ID: <74fcdc1aec1e92afec81c84930cff6ac831f72b1.camel@student.tugraz.at> (raw)
In-Reply-To: <20210802155019.6122-6-maximedevos@telenet.be>
Hi,
Am Montag, den 02.08.2021, 17:50 +0200 schrieb Maxime Devos:
> * guix/import/contentdb.scm: New file.
> * guix/scripts/import/contentdb.scm: New file.
> * tests/contentdb.scm: New file.
> * Makefile.am (MODULES, SCM_TESTS): Register them.
> * po/guix/POTFILES.in: Likewise.
> * doc/guix.texi (Invoking guix import): Document it.
> [...]
> diff --git a/doc/guix.texi b/doc/guix.texi
> index 43c248234d..d06c9b73c5 100644
> --- a/doc/guix.texi
> +++ b/doc/guix.texi
> @@ -11313,6 +11313,31 @@ and generate package expressions for all
> those packages that are not yet
> in Guix.
> @end table
>
> +@item contentdb
> +@cindex ContentDB
> +Import metadata from @uref{https://content.minetest.net, ContentDB}.
> +Information is taken from the JSON-formatted metadata provided
> through
> +@uref{https://content.minetest.net/help/api/, ContentDB's API} and
> +includes most relevant information, including dependencies. There
> are
> +some caveats, however. The license information on ContentDB does
> not
> +distinguish between GPLvN-only and GPLvN-or-later. The commit id is
> +sometimes missing. The descriptions are in the Markdown format, but
> +Guix uses Texinfo instead. Texture packs and subgames are
> unsupported.
What is the "commit id"? Is it the hash? A tag? Anything that
resolves to a commit?
Also, since ContentDB sounds fairly generic (a database of content?),
perhaps we ought to call this the "minetest" importer instead?
> [...]
> +;; The ContentDB API is documented at
> +;; <https://content.minetest.net>;.
> +
> +(define %contentdb-api
> + (make-parameter "https://content.minetest.net/api/"))
> +
> +(define (string-or-false x)
> + (and (string? x) x))
> +
> +(define (natural-or-false x)
> + (and (exact-integer? x) (>= x 0) x))
> +
> +;; Descriptions on ContentDB use carriage returns, but Guix doesn't.
> +(define (delete-cr text)
> + (string-delete #\cr text))
> +
> +;; Minetest package.
> +;;
> +;; API endpoint: /packages/AUTHOR/NAME/
> +(define-json-mapping <package> make-package package?
> + json->package
> + (author package-author) ; string
> + (creation-date package-creation-date ; string
> + "created_at")
> + (downloads package-downloads) ; integer
> + (forums package-forums "forums" natural-or-false) ;
> natural | #f
This comment and some others like it seem to simply be repeating
already present information. Is there a use for them? Should we
instead provide a third argument on every field to verify/enforce the
type?
> + (issue-tracker package-issue-tracker "issue_tracker") ; string
> + (license package-license) ; string
> + (long-description package-long-description "long_description") ;
> string
> + (maintainers package-maintainers ; list of strings
> + "maintainers" vector->list)
> + (media-license package-media-license "media_license") ; string
> + (name package-name) ; string
> + (provides package-provides ; list of strings
> + "provides" vector->list)
> + (release package-release) ; integer
> + (repository package-repository "repo" string-or-false) ;
> string | #f
> + (score package-score) ; flonum
> + (screenshots package-screenshots "screenshots" vector->list)
> ; list of strings
> + (short-description package-short-description "short_description")
> ; string
> + (state package-state) ; string
> + (tags package-tags "tags" vector->list) ; list of
> strings
> + (thumbnail package-thumbnail) ; string
> + (title package-title) ; string
> + (type package-type) ; string
> + (url package-url) ; string
> + (website package-website "website" string-or-false)) ;
> string | #f
> +
> +(define-json-mapping <release> make-release release?
> + json->release
> + (commit release-commit "commit" string-or-false) ;
> string | #f
> + (downloads release-downloads) ; integer
> + (id release-id) ; integer
> + (max-minetest-version release-max-minetest-version) ; string | #f
> + (min-minetest-version release-min-minetest-version) ; string | #f
> + (release-date release-data) ; string
> + (title release-title) ; string
> + (url release-url)) ; string
> +
> +(define-json-mapping <dependency> make-dependency dependency?
> + json->dependency
> + (optional? dependency-optional? "is_optional") ; #t | #f
Also known as "boolean".
> + (name dependency-name) ; string
> + (packages dependency-packages "packages" vector->list)) ; list of
> strings
> +
> +(define (contentdb-fetch author name)
> + "Return a <package> record for package NAME by AUTHOR, or #f on
> failure."
> + (and=> (json-fetch
> + (string-append (%contentdb-api) "packages/" author "/"
> name "/"))
> + json->package))
Is there a reason for author and name to be separate keys? For me it
makes more sense to take AUTHOR/NAME as a singular search string from
users and then perform queries based on that. If ContentDB allows
searching, we might also resolve NAME to a singular package where
possible and otherwise error out, telling the user to choose one.
> [...]
> +
> +(define (important-dependencies dependencies author name)
> + (define dependency-list
> + (assoc-ref dependencies (string-append author "/" name)))
> + (filter-map
> + (lambda (dependency)
> + (and (not (dependency-optional? dependency))
> + ;; "default" must be provided by the 'subgame' in use
> + ;; and does not refer to a specific minetest mod.
> + ;; "doors", "bucket" ... are provided by the default
> minetest
> + ;; subgame.
> + (not (member (dependency-name dependency)
> + '("default" "doors" "beds" "bucket" "doors"
> "farming"
> + "flowers" "stairs" "xpanes")))
> + ;; Dependencies often have only one implementation.
> + (let* ((/name (string-append "/" (dependency-name
> dependency)))
> + (likewise-named-implementations
> + (filter (cut string-suffix? /name <>)
> + (dependency-packages dependency)))
> + (implementation
> + (and (not (null? likewise-named-implementations))
> + (first likewise-named-implementations))))
> + (and implementation
> + (apply cons (string-split implementation #\/))))))
> + dependency-list))
What exactly does the likewise-named-implementations bit do here?
> +(define (contentdb-recursive-import author name)
> + ;; recursive-import expects upstream package names to be strings,
> + ;; so do some conversions.
> + (define (split-author/name author/name)
> + (string-split author/name #\/))
+1 for my author/name splitting, as it's already required for recursive
imports.
> + (define (author+name->author/name author+name)
> + (string-append (car author+name) "/" (cdr author+name)))
> + (define* (contentdb->guix-package* author/name #:key repo version)
> + (receive (package . maybe-dependencies)
> + (apply contentdb->guix-package (split-author/name
> author/name))
> + (and package
> + (receive (dependencies)
> + (apply values maybe-dependencies)
> + (values package
> + (map author+name->author/name
> dependencies))))))
> + (recursive-import (author+name->author/name (cons author name))
> + #:repo->guix-package contentdb->guix-package*
> + #:guix-name
> + (lambda (author/name)
> + (contentdb->package-name
> + (second (split-author/name author/name))))))
> +
> +;; A list of license names is available at
> +;; <https://content.minetest.net/api/licenses/>;.
> +(define (string->license str)
> + "Convert the string STR into a license object."
> + (match str
> + ("GPLv3" license:gpl3)
> + ("GPLv2" license:gpl2)
> + ("ISC" license:isc)
> + ;; "MIT" means the Expat license on ContentDB,
> + ;; see <
> https://github.com/minetest/contentdb/issues/326#issuecomment-890143784>
> ;.
> + ("MIT" license:expat)
> + ("CC BY-SA 3.0" license:cc-by-sa3.0)
> + ("CC BY-SA 4.0" license:cc-by-sa4.0)
> + ("LGPLv2.1" license:lgpl2.1)
> + ("LGPLv3" license:lgpl3)
> + ("MPL 2.0" license:mpl2.0)
> + ("ZLib" license:zlib)
> + ("Unlicense" license:unlicense)
> + (_ #f)))
The link mentions, that ContentDB now supports all SPDX identifiers.
Do we have a SPDX->Guix converter lying around in some other importer
that we could use as default case here (especially w.r.t. "or later")
WDYT?
next prev parent reply other threads:[~2021-08-05 16:42 UTC|newest]
Thread overview: 71+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-08-02 15:46 [bug#49828] [PATCH 00/20] Add minetest mods Maxime Devos
2021-08-02 15:50 ` [bug#49828] [PATCH 01/20] gnu: minetest: Respect --without-tests Maxime Devos
2021-08-02 15:50 ` [bug#49828] [PATCH 02/20] gnu: minetest: Search for mods in MINETEST_MOD_PATH Maxime Devos
2021-08-02 17:28 ` Leo Prikler
2021-08-02 17:53 ` Maxime Devos
2021-08-02 18:47 ` Leo Prikler
2021-08-03 11:09 ` Maxime Devos
2021-08-03 11:10 ` Maxime Devos
2021-08-03 11:54 ` Leo Prikler
2021-08-02 15:50 ` [bug#49828] [PATCH 03/20] gnu: minetest: New package module Maxime Devos
2021-08-02 15:50 ` [bug#49828] [PATCH 04/20] build-system: Add 'minetest-mod-build-system' Maxime Devos
2021-08-02 15:50 ` [bug#49828] [PATCH 05/20] build-system: minetest: Don't retain references to "bash-minimal" Maxime Devos
2021-08-03 9:17 ` Leo Prikler
2021-08-03 11:59 ` Maxime Devos
2021-08-03 12:28 ` Leo Prikler
2021-08-05 11:01 ` Maxime Devos
2021-08-05 12:04 ` Leo Prikler
2021-08-05 13:16 ` Maxime Devos
2021-08-05 13:42 ` Leo Prikler
2021-08-05 14:41 ` Maxime Devos
2021-08-05 15:15 ` Leo Prikler
2021-08-02 15:50 ` [bug#49828] [PATCH 06/20] guix: Add ContentDB importer Maxime Devos
2021-08-05 16:41 ` Leo Prikler [this message]
2021-08-07 18:31 ` Maxime Devos
2021-08-07 19:47 ` Leo Prikler
2021-08-09 20:00 ` [bug#49828] [PATCH 06/20] guix: Add ContentDB importer. (XXX Don't send yet) Maxime Devos
2021-08-09 20:04 ` Maxime Devos
2021-08-09 21:45 ` [bug#49828] [PATCH 06/20] guix: Add ContentDB importer. (XXX Yes send now) Leo Prikler
2021-08-10 11:02 ` [bug#49828] [PATCH 06/20] guix: Add ContentDB importer Maxime Devos
2021-08-10 12:16 ` Leo Prikler
2021-08-10 15:07 ` [bug#49828] [PATCH v3 00/20] Add minetest mods Maxime Devos
2021-08-10 15:07 ` [bug#49828] [PATCH v3 01/20] gnu: minetest: Respect --without-tests Maxime Devos
2021-08-10 15:07 ` [bug#49828] [PATCH v3 02/20] gnu: minetest: Search for mods in MINETEST_MOD_PATH Maxime Devos
2021-08-20 11:45 ` bug#49828: " Leo Prikler
2021-08-10 15:07 ` [bug#49828] [PATCH v3 03/20] gnu: minetest: New package module Maxime Devos
2021-08-10 15:07 ` [bug#49828] [PATCH v3 04/20] build-system: Add 'minetest-mod-build-system' Maxime Devos
2021-08-10 15:07 ` [bug#49828] [PATCH v3 05/20] import/utils: Recognise GPL-3.0-or-later and friends Maxime Devos
2021-08-10 15:07 ` [bug#49828] [PATCH v3 06/20] guix: Add ContentDB importer Maxime Devos
2021-08-10 15:07 ` [bug#49828] [PATCH v3 07/20] gnu: Add minetest-mesecons Maxime Devos
2021-08-10 15:07 ` [bug#49828] [PATCH v3 08/20] gnu: Add minetest-basic-materials Maxime Devos
2021-08-10 15:07 ` [bug#49828] [PATCH v3 09/20] gnu: Add minetest-unifieddyes Maxime Devos
2021-08-10 15:07 ` [bug#49828] [PATCH v3 10/20] gnu: Add minetest-pipeworks Maxime Devos
2021-08-10 15:07 ` [bug#49828] [PATCH v3 11/20] gnu: Add minetest-coloredwood Maxime Devos
2021-08-10 15:07 ` [bug#49828] [PATCH v3 12/20] gnu: Add minetest-ethereal Maxime Devos
2021-08-10 15:07 ` [bug#49828] [PATCH v3 13/20] gnu: Add minetest-technic Maxime Devos
2021-08-10 15:07 ` [bug#49828] [PATCH v3 14/20] gnu: Add minetest-throwing Maxime Devos
2021-08-10 15:07 ` [bug#49828] [PATCH v3 15/20] gnu: Add minetest-throwing-arrows Maxime Devos
2021-08-10 15:07 ` [bug#49828] [PATCH v3 16/20] gnu: Add minetest-unified-inventory Maxime Devos
2021-08-10 15:07 ` [bug#49828] [PATCH v3 17/20] gnu: Add minetest-worldedit Maxime Devos
2021-08-10 15:07 ` [bug#49828] [PATCH v3 18/20] gnu: Add minetest-mobs Maxime Devos
2021-08-10 15:07 ` [bug#49828] [PATCH v3 19/20] gnu: Add minetest-mobs-animal Maxime Devos
2021-08-10 15:07 ` [bug#49828] [PATCH v3 20/20] gnu: Add minetest-homedecor-modpack Maxime Devos
2021-08-02 15:50 ` [bug#49828] [PATCH 07/20] gnu: Add minetest-mesecons Maxime Devos
2021-08-02 15:50 ` [bug#49828] [PATCH 08/20] gnu: Add minetest-basic-materials Maxime Devos
2021-08-02 15:50 ` [bug#49828] [PATCH 09/20] gnu: Add minetest-unifieddyes Maxime Devos
2021-08-02 15:50 ` [bug#49828] [PATCH 10/20] gnu: Add minetest-pipeworks Maxime Devos
2021-08-02 15:50 ` [bug#49828] [PATCH 11/20] gnu: Add minetest-coloredwood Maxime Devos
2021-08-02 15:50 ` [bug#49828] [PATCH 12/20] gnu: Add minetest-ethereal Maxime Devos
2021-08-02 15:50 ` [bug#49828] [PATCH 13/20] gnu: Add minetest-technic Maxime Devos
2021-08-02 15:50 ` [bug#49828] [PATCH 14/20] gnu: Add minetest-throwing Maxime Devos
2021-08-02 15:50 ` [bug#49828] [PATCH 15/20] gnu: Add minetest-throwing-arrows Maxime Devos
2021-08-02 15:50 ` [bug#49828] [PATCH 16/20] gnu: Add minetest-unified-inventory Maxime Devos
2021-08-02 15:50 ` [bug#49828] [PATCH 17/20] gnu: Add minetest-worldedit Maxime Devos
2021-08-02 15:50 ` [bug#49828] [PATCH 18/20] gnu: Add minetest-mobs Maxime Devos
2021-08-02 15:50 ` [bug#49828] [PATCH 19/20] gnu: Add minetest-mobs-animal Maxime Devos
2021-08-02 15:50 ` [bug#49828] [PATCH 20/20] gnu: Add minetest-homedecor-modpack Maxime Devos
2021-08-02 17:14 ` [bug#49828] [PATCH 01/20] gnu: minetest: Respect --without-tests Leo Prikler
2021-08-02 17:18 ` Maxime Devos
2021-08-02 17:22 ` Leo Prikler
2021-08-05 12:46 ` [bug#49828] [PATCH 00/20] Add minetest mods Andrew Ward
2021-08-05 21:10 ` Maxime Devos
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=74fcdc1aec1e92afec81c84930cff6ac831f72b1.camel@student.tugraz.at \
--to=leo.prikler@student.tugraz.at \
--cc=49828@debbugs.gnu.org \
--cc=maximedevos@telenet.be \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this external index
https://git.savannah.gnu.org/cgit/guix.git
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.