unofficial mirror of guix-devel@gnu.org 
 help / color / mirror / code / Atom feed
* New build system: copy-build-system
@ 2020-01-26 20:27 Pierre Neidhardt
  2020-01-27  0:45 ` Jesse Gibbons
                   ` (2 more replies)
  0 siblings, 3 replies; 24+ messages in thread
From: Pierre Neidhardt @ 2020-01-26 20:27 UTC (permalink / raw)
  To: guix-devel

[-- Attachment #1: Type: text/plain, Size: 775 bytes --]

Hi!

We've got quite a few packages that simply copy some files from their
source to the output.  Examples include `debian-archive-keyring`,
`0ad-data`, `gcide`, or `rofi-pass`.

Those packages end up being tedious to write because we need to rely on
the trivial-build-system, also known as "the least trivial of them all!" :)

What about adding a "copy-build-system" which has one argument, say an
`#:install-plan` that takes a list of source-destination pairs?

This would drastically simplify the definition of the aforementioned
packages, basically summing up dozens of boilerplate lines into an
effectively trivial file mapping.

If this sounds like a good idea, I'd be happy to send a patch!

Cheers!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: New build system: copy-build-system
  2020-01-26 20:27 New build system: copy-build-system Pierre Neidhardt
@ 2020-01-27  0:45 ` Jesse Gibbons
  2020-01-27  4:14   ` Jack Hill
  2020-01-27 14:27 ` zimoun
  2020-01-27 19:03 ` New build system: copy-build-system Ricardo Wurmus
  2 siblings, 1 reply; 24+ messages in thread
From: Jesse Gibbons @ 2020-01-27  0:45 UTC (permalink / raw)
  To: Pierre Neidhardt, guix-devel

On Sun, 2020-01-26 at 21:27 +0100, Pierre Neidhardt wrote:
> 	Error verifying signature: Failed to execute gpg.
> Hi!
> 
> We've got quite a few packages that simply copy some files from their
> source to the output.  Examples include `debian-archive-keyring`,
> `0ad-data`, `gcide`, or `rofi-pass`.
> 
> Those packages end up being tedious to write because we need to rely
> on
> the trivial-build-system, also known as "the least trivial of them
> all!" :)
> 
> What about adding a "copy-build-system" which has one argument, say
> an
> `#:install-plan` that takes a list of source-destination pairs?
> 
> This would drastically simplify the definition of the aforementioned
> packages, basically summing up dozens of boilerplate lines into an
> effectively trivial file mapping.
> 
> If this sounds like a good idea, I'd be happy to send a patch!
> 
> Cheers!
> 
That would be great!

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: New build system: copy-build-system
  2020-01-27  0:45 ` Jesse Gibbons
@ 2020-01-27  4:14   ` Jack Hill
  0 siblings, 0 replies; 24+ messages in thread
From: Jack Hill @ 2020-01-27  4:14 UTC (permalink / raw)
  To: Jesse Gibbons; +Cc: guix-devel



On Sun, 26 Jan 2020, Jesse Gibbons wrote:

> On Sun, 2020-01-26 at 21:27 +0100, Pierre Neidhardt wrote:
>> 	Error verifying signature: Failed to execute gpg.
>> Hi!
>>
>> We've got quite a few packages that simply copy some files from their
>> source to the output.  Examples include `debian-archive-keyring`,
>> `0ad-data`, `gcide`, or `rofi-pass`.
>>
>> Those packages end up being tedious to write because we need to rely
>> on
>> the trivial-build-system, also known as "the least trivial of them
>> all!" :)
>>
>> What about adding a "copy-build-system" which has one argument, say
>> an
>> `#:install-plan` that takes a list of source-destination pairs?
>>
>> This would drastically simplify the definition of the aforementioned
>> packages, basically summing up dozens of boilerplate lines into an
>> effectively trivial file mapping.
>>
>> If this sounds like a good idea, I'd be happy to send a patch!
>>
>> Cheers!
>>
> That would be great!

I agree that this would be great as well. In addition to tedium, 
trivial-build-system can make modifications to package definitions, which 
I consider a really cool guix feature and a win for software freedom, 
more tricky. See 
https://lists.gnu.org/archive/html/help-guix/2019-12/msg00139.html for an 
issue I had with it (sorry Marius, I haven't had time to play with 
switching the build system of matcha-theme, but it's still on my todo list 
:)).

Best,
Jack

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: New build system: copy-build-system
  2020-01-26 20:27 New build system: copy-build-system Pierre Neidhardt
  2020-01-27  0:45 ` Jesse Gibbons
@ 2020-01-27 14:27 ` zimoun
  2020-01-27 14:44   ` Pierre Neidhardt
  2020-01-27 19:03 ` New build system: copy-build-system Ricardo Wurmus
  2 siblings, 1 reply; 24+ messages in thread
From: zimoun @ 2020-01-27 14:27 UTC (permalink / raw)
  To: Pierre Neidhardt; +Cc: Guix Devel

Hi Pierre,

On Sun, 26 Jan 2020 at 21:27, Pierre Neidhardt <mail@ambrevar.xyz> wrote:

> Those packages end up being tedious to write because we need to rely on
> the trivial-build-system, also known as "the least trivial of them all!" :)

I agree!


> What about adding a "copy-build-system" which has one argument, say an
> `#:install-plan` that takes a list of source-destination pairs?

I agree that something is missing.


> This would drastically simplify the definition of the aforementioned
> packages, basically summing up dozens of boilerplate lines into an
> effectively trivial file mapping.

I agree too.


> If this sounds like a good idea, I'd be happy to send a patch!

Related to this idea, I would point to [1] a previous discussion about
"pure" data package distribution.

[1] https://lists.gnu.org/archive/html/guix-devel/2018-02/msg00183.html


For a concrete example, let consider the Debian package [2]
'astrometry-data-2mass-00' which is roughly 13.9GB of data. A big set
of index used by astronomer. The Debian package calls 'curl' under the
hood to fetch the data from the http://data.astrometry.net; see [3].

[2] https://packages.debian.org/fr/stretch/astrometry-data-2mass-00
[3] https://salsa.debian.org/debian-astro-team/astrometry-data-2mass/blob/master/debian/astrometry-data-2mass-00.postinst


Well, the questions are:
 - hash?
 - a copy on Berlin as substitute?

I am just taking the occasion to discussion the hypothetical
possibility to discuss if 'copy-build-system' could be way to
distribute such data set.

Other said, the 'copy-build-system' could:

 - fetch the data from an archive, such http://data.astrometry.net
 - fetch the resulting <hash> of /gnu/store/<hash>-name-version from susbtitutes


Cheers,
simon

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: New build system: copy-build-system
  2020-01-27 14:27 ` zimoun
@ 2020-01-27 14:44   ` Pierre Neidhardt
  2020-01-27 15:03     ` zimoun
  0 siblings, 1 reply; 24+ messages in thread
From: Pierre Neidhardt @ 2020-01-27 14:44 UTC (permalink / raw)
  To: zimoun; +Cc: Guix Devel

[-- Attachment #1: Type: text/plain, Size: 835 bytes --]

zimoun <zimon.toutoune@gmail.com> writes:

> Well, the questions are:
>  - hash?
>  - a copy on Berlin as substitute?

What are the questions exactly?  I don't understand what you are asking.

> I am just taking the occasion to discussion the hypothetical
> possibility to discuss if 'copy-build-system' could be way to
> distribute such data set.
>
> Other said, the 'copy-build-system' could:
>
>  - fetch the data from an archive, such http://data.astrometry.net
>  - fetch the resulting <hash> of /gnu/store/<hash>-name-version from susbtitutes

Hmmm, maybe you are asking for something else, because the
copy-build-system, as a build system, has nothing to do with the fetcher
which is responsible for fetching the data over the network.

Did I misunderstand?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: New build system: copy-build-system
  2020-01-27 14:44   ` Pierre Neidhardt
@ 2020-01-27 15:03     ` zimoun
  2020-01-27 15:18       ` Pierre Neidhardt
  2020-01-27 16:45       ` How to deal with very large sources Ricardo Wurmus
  0 siblings, 2 replies; 24+ messages in thread
From: zimoun @ 2020-01-27 15:03 UTC (permalink / raw)
  To: Pierre Neidhardt; +Cc: Guix Devel

On Mon, 27 Jan 2020 at 15:45, Pierre Neidhardt <mail@ambrevar.xyz> wrote:
>
> zimoun <zimon.toutoune@gmail.com> writes:
>
> > Well, the questions are:
> >  - hash?
> >  - a copy on Berlin as substitute?
>
> What are the questions exactly?  I don't understand what you are asking.

Considering "large" data, the questions are:

How to compute the hash?
Does Berlin have a copy of large data set?


> > I am just taking the occasion to discussion the hypothetical
> > possibility to discuss if 'copy-build-system' could be way to
> > distribute such data set.

[...]

> Hmmm, maybe you are asking for something else, because the
> copy-build-system, as a build system, has nothing to do with the fetcher
> which is responsible for fetching the data over the network.

I know that it was not your initial intention with 'copy-build-sytem'. :-)


You want something like that:

--8<---------------cut here---------------start------------->8---
(define-public debian-archive-keyring
  (package
    (name "debian-archive-keyring")
    (version "2019.1")
    (source ...)
    (build-system copy-build-system)
    (arguments
     '(#:install-plan
       (lambda* (#:key outputs #:allow-other-keys)
                (let* ((out (assoc-ref outputs "out"))
                       (apt (string-append out "/etc/apt/trusted.gpg.d/"))
                       (key (string-append out "/share/keyrings/")))
                  (install-file "keyrings/debian-archive-keyring.gpg" key)
                  (install-file "keyrings/debian-archive-removed-keys.gpg" key)
                  (for-each (lambda (file)
                              (install-file file apt))
                            (find-files "trusted.gpg" "\\.gpg$")))
                #t)))))
--8<---------------cut here---------------end--------------->8---

instead of the current:

--8<---------------cut here---------------start------------->8---
    (build-system gnu-build-system)
    (arguments
     '(#:test-target "verify-results"
       #:parallel-build? #f ; has race conditions
       #:phases
       (modify-phases %standard-phases
         (delete 'configure) ; no configure script
         (replace 'install
--8<---------------cut here---------------end--------------->8---


I agree that it is a good idea.


The aim of my comment was:
 1. Point out a previous discussion about the packages of data only;
which is what we are talking about, right?
 2. Discuss if your proposal can fit some data distribution issue;
discussed earlier at point 1..


Cheers,
simon

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: New build system: copy-build-system
  2020-01-27 15:03     ` zimoun
@ 2020-01-27 15:18       ` Pierre Neidhardt
  2020-01-27 15:36         ` zimoun
  2020-01-27 16:45       ` How to deal with very large sources Ricardo Wurmus
  1 sibling, 1 reply; 24+ messages in thread
From: Pierre Neidhardt @ 2020-01-27 15:18 UTC (permalink / raw)
  To: zimoun; +Cc: Guix Devel

[-- Attachment #1: Type: text/plain, Size: 1530 bytes --]

zimoun <zimon.toutoune@gmail.com> writes:

> You want something like that:
>
> --8<---------------cut here---------------start------------->8---
> (define-public debian-archive-keyring
>   (package
>     (name "debian-archive-keyring")
>     (version "2019.1")
>     (source ...)
>     (build-system copy-build-system)
>     (arguments
>      '(#:install-plan
>        (lambda* (#:key outputs #:allow-other-keys)
>                 (let* ((out (assoc-ref outputs "out"))
>                        (apt (string-append out "/etc/apt/trusted.gpg.d/"))
>                        (key (string-append out "/share/keyrings/")))
>                   (install-file "keyrings/debian-archive-keyring.gpg" key)
>                   (install-file "keyrings/debian-archive-removed-keys.gpg" key)
>                   (for-each (lambda (file)
>                               (install-file file apt))
>                             (find-files "trusted.gpg" "\\.gpg$")))
>                 #t)))))
> --8<---------------cut here---------------end--------------->8---

Actually we could simplify the install plan with something like

--8<---------------cut here---------------start------------->8---
#:install-plan
(("source1" "target1")
 ("source-dir2/" "target-dir2" #:exclude ("foo")))
--8<---------------cut here---------------end--------------->8---

Regarding the data fetching, if I understand your point I think it
should be handled in a separate issue.

Cheers!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: New build system: copy-build-system
  2020-01-27 15:18       ` Pierre Neidhardt
@ 2020-01-27 15:36         ` zimoun
  2020-01-27 15:51           ` Pierre Neidhardt
  0 siblings, 1 reply; 24+ messages in thread
From: zimoun @ 2020-01-27 15:36 UTC (permalink / raw)
  To: Pierre Neidhardt; +Cc: Guix Devel

On Mon, 27 Jan 2020 at 16:18, Pierre Neidhardt <mail@ambrevar.xyz> wrote:
>
> zimoun <zimon.toutoune@gmail.com> writes:

> Actually we could simplify the install plan with something like
>
> --8<---------------cut here---------------start------------->8---
> #:install-plan
> (("source1" "target1")
>  ("source-dir2/" "target-dir2" #:exclude ("foo")))
> --8<---------------cut here---------------end--------------->8---

Yes. For sure. :-)

> Regarding the data fetching, if I understand your point I think it
> should be handled in a separate issue.

Maybe. :-)
It is just the occasion to pop out again the issue.
And why not discuss how to deal with large data set. I mean, if I use
my typical french bad faith in grumpy fashion, then I do not see why
the "small" data set are included in the store and the "large" not.
What does fix the definition of "small" and "large"?


And for example, the 'copy-build-system' could:

 - fetch the data from an archive, such http://data.astrometry.net or
IPFS or Zenodo or <name-it>
 - fetch the resulting <hash> of /gnu/store/<hash>-name-version from susbtitutes

Other said design a build-system to manage large dataset as regular packages.
Yes, it is out of scope of your initial idea. :-)


Cheers,
simon

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: New build system: copy-build-system
  2020-01-27 15:36         ` zimoun
@ 2020-01-27 15:51           ` Pierre Neidhardt
  2020-01-27 16:17             ` zimoun
  0 siblings, 1 reply; 24+ messages in thread
From: Pierre Neidhardt @ 2020-01-27 15:51 UTC (permalink / raw)
  To: zimoun; +Cc: Guix Devel

[-- Attachment #1: Type: text/plain, Size: 370 bytes --]

zimoun <zimon.toutoune@gmail.com> writes:

> And for example, the 'copy-build-system' could:
>
>  - fetch the data from an archive, such http://data.astrometry.net or
> IPFS or Zenodo or <name-it>
>  - fetch the resulting <hash> of /gnu/store/<hash>-name-version from susbtitutes

Build systems don't fetch data.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: New build system: copy-build-system
  2020-01-27 15:51           ` Pierre Neidhardt
@ 2020-01-27 16:17             ` zimoun
  2020-01-27 18:18               ` Julien Lepiller
  0 siblings, 1 reply; 24+ messages in thread
From: zimoun @ 2020-01-27 16:17 UTC (permalink / raw)
  To: Pierre Neidhardt; +Cc: Guix Devel

On Mon, 27 Jan 2020 at 16:51, Pierre Neidhardt <mail@ambrevar.xyz> wrote:
>
> zimoun <zimon.toutoune@gmail.com> writes:
>
> > And for example, the 'copy-build-system' could:
> >
> >  - fetch the data from an archive, such http://data.astrometry.net or
> > IPFS or Zenodo or <name-it>
> >  - fetch the resulting <hash> of /gnu/store/<hash>-name-version from susbtitutes
>
> Build systems don't fetch data.

I know (isolated blabla). :-)

"The package using the 'copy-build-system'" is a better wording? ;-)

I do not know exactly how a derivation is computed and what is used to
compute the hash used in the store. Ok, first I have to do my
homework. :-)
But one does not want to hash several tens of GB.

Whatever, I just have the feeling that what you are proposing (with
small tweaks) could improve the situation for "large" data set. I do
not know...

Cheers,
simon

ps:
Well, maybe we should work in the same office... it will be easier and
faster than exchanging all these emails and I know yours has an
awesome view :-D

^ permalink raw reply	[flat|nested] 24+ messages in thread

* How to deal with very large sources
  2020-01-27 15:03     ` zimoun
  2020-01-27 15:18       ` Pierre Neidhardt
@ 2020-01-27 16:45       ` Ricardo Wurmus
  1 sibling, 0 replies; 24+ messages in thread
From: Ricardo Wurmus @ 2020-01-27 16:45 UTC (permalink / raw)
  To: zimoun; +Cc: guix-devel


zimoun <zimon.toutoune@gmail.com> writes:

> On Mon, 27 Jan 2020 at 15:45, Pierre Neidhardt <mail@ambrevar.xyz> wrote:
>>
>> zimoun <zimon.toutoune@gmail.com> writes:
>>
>> > Well, the questions are:
>> >  - hash?
>> >  - a copy on Berlin as substitute?
>>
>> What are the questions exactly?  I don't understand what you are asking.
>
> Considering "large" data, the questions are:
>
> How to compute the hash?
> Does Berlin have a copy of large data set?

We can mark packages as not substitutable, so the sources would be
downloaded on the users’ machines.  Whoever packages things would have
to compute the hash first, though.

-- 
Ricardo

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: New build system: copy-build-system
  2020-01-27 16:17             ` zimoun
@ 2020-01-27 18:18               ` Julien Lepiller
  0 siblings, 0 replies; 24+ messages in thread
From: Julien Lepiller @ 2020-01-27 18:18 UTC (permalink / raw)
  To: zimoun, Pierre Neidhardt; +Cc: Guix Devel

Le 27 janvier 2020 11:17:10 GMT-05:00, zimoun <zimon.toutoune@gmail.com> a écrit :
>On Mon, 27 Jan 2020 at 16:51, Pierre Neidhardt <mail@ambrevar.xyz>
>wrote:
>>
>> zimoun <zimon.toutoune@gmail.com> writes:
>>
>> > And for example, the 'copy-build-system' could:
>> >
>> >  - fetch the data from an archive, such http://data.astrometry.net
>or
>> > IPFS or Zenodo or <name-it>
>> >  - fetch the resulting <hash> of /gnu/store/<hash>-name-version
>from susbtitutes
>>
>> Build systems don't fetch data.
>
>I know (isolated blabla). :-)
>
>"The package using the 'copy-build-system'" is a better wording? ;-)
>
>I do not know exactly how a derivation is computed and what is used to
>compute the hash used in the store. Ok, first I have to do my
>homework. :-)
>But one does not want to hash several tens of GB.
>
>Whatever, I just have the feeling that what you are proposing (with
>small tweaks) could improve the situation for "large" data set. I do
>not know...
>
>Cheers,
>simon
>
>ps:
>Well, maybe we should work in the same office... it will be easier and
>faster than exchanging all these emails and I know yours has an
>awesome view :-D

From my own experiments, computing the hash of a source is pretty fast, even on a large archive (a few seconds for a 5.5 GB file, using guix hash the-file). However, guix download the-file will fail when you don't have (roughly) 3 times the file's size in free memory. So a large source would be whatever your RAM is / 3 or so.

We should improve that, but that's another issue :)

Also, the file size has nothing to do with derivation computation, which only relies on the declared sha256 of the source (known statically).

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: New build system: copy-build-system
  2020-01-26 20:27 New build system: copy-build-system Pierre Neidhardt
  2020-01-27  0:45 ` Jesse Gibbons
  2020-01-27 14:27 ` zimoun
@ 2020-01-27 19:03 ` Ricardo Wurmus
  2020-01-28 10:32   ` Ludovic Courtès
  2 siblings, 1 reply; 24+ messages in thread
From: Ricardo Wurmus @ 2020-01-27 19:03 UTC (permalink / raw)
  To: Pierre Neidhardt; +Cc: guix-devel


Hi Pierre,

> We've got quite a few packages that simply copy some files from their
> source to the output.  Examples include `debian-archive-keyring`,
> `0ad-data`, `gcide`, or `rofi-pass`.
>
> Those packages end up being tedious to write because we need to rely on
> the trivial-build-system, also known as "the least trivial of them all!" :)
>
> What about adding a "copy-build-system" which has one argument, say an
> `#:install-plan` that takes a list of source-destination pairs?

This sounds like a good idea.  We did something similar when the
font-build-system was added to simplify the tedium of copying files.

--
Ricardo

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: New build system: copy-build-system
  2020-01-27 19:03 ` New build system: copy-build-system Ricardo Wurmus
@ 2020-01-28 10:32   ` Ludovic Courtès
  2020-01-28 13:39     ` Pierre Neidhardt
  0 siblings, 1 reply; 24+ messages in thread
From: Ludovic Courtès @ 2020-01-28 10:32 UTC (permalink / raw)
  To: Ricardo Wurmus; +Cc: guix-devel

Hi,

Ricardo Wurmus <rekado@elephly.net> skribis:

>> We've got quite a few packages that simply copy some files from their
>> source to the output.  Examples include `debian-archive-keyring`,
>> `0ad-data`, `gcide`, or `rofi-pass`.
>>
>> Those packages end up being tedious to write because we need to rely on
>> the trivial-build-system, also known as "the least trivial of them all!" :)
>>
>> What about adding a "copy-build-system" which has one argument, say an
>> `#:install-plan` that takes a list of source-destination pairs?
>
> This sounds like a good idea.  We did something similar when the
> font-build-system was added to simplify the tedium of copying files.

Perhaps some of these can just be plain <origin> records if they are
only used internally (not as packages on the command line).

For the other cases, ‘copy-build-system’ may prove useful!

Ludo’.

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: New build system: copy-build-system
  2020-01-28 10:32   ` Ludovic Courtès
@ 2020-01-28 13:39     ` Pierre Neidhardt
  2020-01-28 23:04       ` Ludovic Courtès
  0 siblings, 1 reply; 24+ messages in thread
From: Pierre Neidhardt @ 2020-01-28 13:39 UTC (permalink / raw)
  To: Ludovic Courtès, Ricardo Wurmus; +Cc: guix-devel

[-- Attachment #1: Type: text/plain, Size: 1945 bytes --]

Regarding the "install-plan", I suggest the following syntax:

(SOURCE TARGET [FILTERS])

In the above, FILTERS are optional.

- When SOURCE matches a file or directory without trailing slash, install it to
  TARGET.
  - If TARGET has a trailing slash, install SOURCE basename beneath TARGET.
  - Otherwise install SOURCE as TARGET.

- When SOURCE is a directory with a trailing slash, or when FILTERS are used,
  the trailing slash of TARGET is implied.
  - Without FILTERS, install the full SOURCE _content_ to TARGET.
    The paths relative to SOURCE are preserved within TARGET.
  - With FILTERS among `#:include`, `#:include-regexp`, `#:exclude`,
    `#:exclude-regexp`:
    - With `#:include`, install only the exact subpaths in the list.
    - With `#:include-regexp`, install subpaths matching the regexps in the list.
    - The `#:exclude*` FILTERS work similarly.  Without `#:include*` flags,
      install every subpath but the files matching the `#:exlude*` filters.
      If both `#:include*` and `#:exclude*` are specified, the exclusion is done
      on the inclusion list.

Examples:

- `("foo/bar" "share/my-app/")`: Install bar to "share/my-app/bar".
- `("foo/bar" "share/my-app/baz")`: Install bar to "share/my-app/baz".
- `("foo/" "share/my-app")`: Install the content of foo inside "share/my-app",
  e.g. install "foo/sub/file" to "share/my-app/sub/file".
- `("foo/" "share/my-app" #:include ("sub/file"))`: Install only "foo/sub/file" to "share/my-app/sub/file".
- `("foo/sub" "share/my-app" #:include ("file"))`: Install "foo/sub/file" to "share/my-app/file".

Maybe instead of having two #:include and #:include-regexp we could have just
one #:include in which the list elements have a "marker" that tells whether they
are regexps or not.

Clojure uses #"..." to mark regexps. Does Guile have something similar?
Any other suggestion?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: New build system: copy-build-system
  2020-01-28 13:39     ` Pierre Neidhardt
@ 2020-01-28 23:04       ` Ludovic Courtès
  2020-02-14 12:54         ` Pierre Neidhardt
  0 siblings, 1 reply; 24+ messages in thread
From: Ludovic Courtès @ 2020-01-28 23:04 UTC (permalink / raw)
  To: Pierre Neidhardt; +Cc: guix-devel

Hi Pierre,

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

> Examples:
>
> - `("foo/bar" "share/my-app/")`: Install bar to "share/my-app/bar".
> - `("foo/bar" "share/my-app/baz")`: Install bar to "share/my-app/baz".
> - `("foo/" "share/my-app")`: Install the content of foo inside "share/my-app",
>   e.g. install "foo/sub/file" to "share/my-app/sub/file".
> - `("foo/" "share/my-app" #:include ("sub/file"))`: Install only "foo/sub/file" to "share/my-app/sub/file".
> - `("foo/sub" "share/my-app" #:include ("file"))`: Install "foo/sub/file" to "share/my-app/file".

Looks like it could work.

I think the general approach should be: pick a handful of packages that
you’d like to rewrite to the new build system, provide the options that
support it (but not more), and verify that the resulting package
definitions are indeed shorter.

> Maybe instead of having two #:include and #:include-regexp we could have just
> one #:include in which the list elements have a "marker" that tells whether they
> are regexps or not.

Probably the former, so that strings are either always or never
interpreted as regexps.  (Guile doesn’t have special “markers” for
regexps.)

Ludo’.

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: New build system: copy-build-system
  2020-01-28 23:04       ` Ludovic Courtès
@ 2020-02-14 12:54         ` Pierre Neidhardt
  2020-02-14 13:22           ` Julien Lepiller
  2020-02-17  3:46           ` Jesse Gibbons
  0 siblings, 2 replies; 24+ messages in thread
From: Pierre Neidhardt @ 2020-02-14 12:54 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: guix-devel

[-- Attachment #1: Type: text/plain, Size: 204 bytes --]

I've just sent a patch: 39599@debbugs.gnu.org.

Ideally, it would need a bit more testing with a package that has
subdirectories.
Any good suggestion?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: New build system: copy-build-system
  2020-02-14 12:54         ` Pierre Neidhardt
@ 2020-02-14 13:22           ` Julien Lepiller
  2020-02-17  3:46           ` Jesse Gibbons
  1 sibling, 0 replies; 24+ messages in thread
From: Julien Lepiller @ 2020-02-14 13:22 UTC (permalink / raw)
  To: guix-devel, Pierre Neidhardt, Ludovic Courtès

Le 14 février 2020 07:54:48 GMT-05:00, Pierre Neidhardt <mail@ambrevar.xyz> a écrit :
>I've just sent a patch: 39599@debbugs.gnu.org.
>
>Ideally, it would need a bit more testing with a package that has
>subdirectories.
>Any good suggestion?

Maybe onee of the vim plugins?

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: New build system: copy-build-system
  2020-02-14 12:54         ` Pierre Neidhardt
  2020-02-14 13:22           ` Julien Lepiller
@ 2020-02-17  3:46           ` Jesse Gibbons
  2020-02-17  7:56             ` Pierre Neidhardt
  1 sibling, 1 reply; 24+ messages in thread
From: Jesse Gibbons @ 2020-02-17  3:46 UTC (permalink / raw)
  To: Pierre Neidhardt, Ludovic Courtès; +Cc: guix-devel

[-- Attachment #1: Type: text/plain, Size: 1277 bytes --]

Hi Pierre,
On Fri, 2020-02-14 at 13:54 +0100, Pierre Neidhardt wrote:
> 	Error verifying signature: Failed to execute gpg.
> I've just sent a patch: 39599@debbugs.gnu.org.
> 
> Ideally, it would need a bit more testing with a package that has
> subdirectories.
> Any good suggestion?
> 
Thank you for making this amazing time-saver!

Attached is an early version of a good candidate. It's a wrapper script
for clojure taken from from clojure's brew installer.

src/main/resources/clojure should be moved to bin/
src/main/resources/cl
j should be moved to bin/

doc/clojure.1 should be moved to share/man/

And unless there's a sufficient reason not to,
epl.html could be moved to share/doc/clojure-1.10.1/


Everything else could probably be ignored.

Since I'm patching one of the scripts to play better with guix, the
source will become a tarball. I would rather not worry about extracting
everything, moving some things, and deleting what is not used.

If your patches are pushed before you get this, I will make the package
myself. But it's a complex yet useful example of a package that would
be simpler with the copy-build-system, so you can check how easy it is
to change from trivial-build-system and test how well copy-build-system 
works with subdirectories.

-Jesse

[-- Attachment #2: Type: text/plain, Size: 2908 bytes --]

;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2018 Alex Vong <alexvong1995@gmail.com>
;;; Copyright © 2018 Pierre Neidhardt <mail@ambrevar.xyz>
;;; Copyright © 2019 Tobias Geerinckx-Rice <me@tobias.gr>
;;; Copyright © 2020 Ludovic Courtès <ludo@gnu.org>
;;;
;;; This file is part of GNU Guix.
;;;
;;; GNU Guix is free software; you can redistribute it and/or modify it
;;; under the terms of the GNU General Public License as published by
;;; the Free Software Foundation; either version 3 of the License, or (at
;;; your option) any later version.
;;;
;;; GNU Guix is distributed in the hope that it will be useful, but
;;; WITHOUT ANY WARRANTY; without even the implied warranty of
;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;;; GNU General Public License for more details.
;;;
;;; You should have received a copy of the GNU General Public License
;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.

(define-module (clojure)
  #:use-module (gnu packages)
  #:use-module ((guix licenses) #:prefix license:)
  #:use-module (guix packages)
  #:use-module (guix download)
  #:use-module (guix git-download)
  #:use-module (guix build-system ant)
  #:use-module (guix build-system clojure)
  #:use-module (guix build-system trivial)
  #:use-module (ice-9 match))

(define-public clojure-wrapper
       (package
        (name "clojure-wrapper")
        (version "1.10.1.507")
        (source
         (origin
          (method git-fetch)
          (uri (git-reference
                (url "https://github.com/clojure/brew-install.git")
                (commit version)))
          (file-name (git-file-name name version))
          (sha256
           (base32 "1zipz22pszv4vls4qhxkia8gm86s1wkahr0jdbqhc46mpd8n54fz"))))
        (build-system trivial-build-system)
        (arguments
         `(#:modules ((guix build utils))
           #:builder (begin
                     (use-modules (guix build utils))
                     (let* ((out (assoc-ref %outputs "out"))
                            (bin-dir (string-append out "/bin/"))
                            (source (assoc-ref %build-inputs "source"))
                            (script-dir
                             (string-append source "/src/main/resources/"))
                                         (scripts '("clojure" "clj")))
		       (mkdir out)
		       (mkdir bin-dir)
                           (map (lambda (script)
				  (copy-file (string-append script-dir script)
					     (string-append bin-dir script)))
                                scripts)))))
		     
        (synopsis "Clojure launch scripts")
        (description "Scripts to launch clojure from the command line.
Without these scripts a user would need to run jar with the clojure jar's
location. Who would want to do that?")
        (home-page "https://clojure.org/")
        (license license:epl1.0)))


^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: New build system: copy-build-system
  2020-02-17  3:46           ` Jesse Gibbons
@ 2020-02-17  7:56             ` Pierre Neidhardt
  2020-02-17 16:52               ` Jesse Gibbons
  0 siblings, 1 reply; 24+ messages in thread
From: Pierre Neidhardt @ 2020-02-17  7:56 UTC (permalink / raw)
  To: Jesse Gibbons, Ludovic Courtès; +Cc: guix-devel

[-- Attachment #1: Type: text/plain, Size: 364 bytes --]

Hi Jesse,

thanks for sharing, I'll include it with the patch then.

> Since I'm patching one of the scripts to play better with guix, the
> source will become a tarball. I would rather not worry about extracting
> everything, moving some things, and deleting what is not used.

What do you mean with this?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: New build system: copy-build-system
  2020-02-17  7:56             ` Pierre Neidhardt
@ 2020-02-17 16:52               ` Jesse Gibbons
  2020-02-17 17:10                 ` Pierre Neidhardt
  0 siblings, 1 reply; 24+ messages in thread
From: Jesse Gibbons @ 2020-02-17 16:52 UTC (permalink / raw)
  To: Pierre Neidhardt, Ludovic Courtès; +Cc: guix-devel

On Mon, 2020-02-17 at 08:56 +0100, Pierre Neidhardt wrote:
> 	Error verifying signature: Failed to execute gpg.
> Hi Jesse,
> 
> thanks for sharing, I'll include it with the patch then.
> 
> > Since I'm patching one of the scripts to play better with guix, the
> > source will become a tarball. I would rather not worry about
> > extracting
> > everything, moving some things, and deleting what is not used.
> 
> What do you mean with this?
> 
Sorry if I confused you.

The package I shared will need to be adjusted because I'm not sure how
to configure the copy-build-system. It was also in an early stage, and
the source itself will need some changes for the package to work. I
only sent it to help you test copy-build-system, since you asked for
suggestions.

The package as I sent it, even if changed to use the copy-build-system, 
is not ready to be sent to to guix because it will not find java or the
clojure jar built by guix. I already have a patch to correct this, and
intend to send it with the rest of the package when I can call
"clojure" from bash in an pure environment with the clojure package and
enter a clojure repl.

IIUC, copy-build-system is based on gnu-build-system, so it should work
the same with tarballs and directories. That means it should be a time
saver; I won't need to worry about changing things if I need to add a
patch to make a package work with what guix does. Thank you so much for
this build system :)

-Jesse

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: New build system: copy-build-system
  2020-02-17 16:52               ` Jesse Gibbons
@ 2020-02-17 17:10                 ` Pierre Neidhardt
  2020-02-19  1:33                   ` Alex Griffin
  0 siblings, 1 reply; 24+ messages in thread
From: Pierre Neidhardt @ 2020-02-17 17:10 UTC (permalink / raw)
  To: Jesse Gibbons, Ludovic Courtès; +Cc: guix-devel

[-- Attachment #1: Type: text/plain, Size: 109 bytes --]

OK, then I'll remove clojure-wrapper for the time being! :)

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: New build system: copy-build-system
  2020-02-17 17:10                 ` Pierre Neidhardt
@ 2020-02-19  1:33                   ` Alex Griffin
  2020-02-19 14:01                     ` Pierre Neidhardt
  0 siblings, 1 reply; 24+ messages in thread
From: Alex Griffin @ 2020-02-19  1:33 UTC (permalink / raw)
  To: Pierre Neidhardt; +Cc: guix-devel

I was looking to package [PaperWM][1][2] today, a tiling window manager extension for gnome-shell. To my surprise, it's an excellent candidate for copy-build-system! So I think I will wait for this to merge first. Good job with this!
-- 
Alex Griffin

[1]: https://github.com/paperwm/PaperWM
[2]: https://jvns.ca/blog/2020/01/05/paperwm/

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: New build system: copy-build-system
  2020-02-19  1:33                   ` Alex Griffin
@ 2020-02-19 14:01                     ` Pierre Neidhardt
  0 siblings, 0 replies; 24+ messages in thread
From: Pierre Neidhardt @ 2020-02-19 14:01 UTC (permalink / raw)
  To: Alex Griffin; +Cc: guix-devel

[-- Attachment #1: Type: text/plain, Size: 195 bytes --]

If you've got time, could you try my patch with PaperWM then to see if
it does the job for you?
I'd like to get some feedback before merging :)

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

^ permalink raw reply	[flat|nested] 24+ messages in thread

end of thread, other threads:[~2020-02-19 14:01 UTC | newest]

Thread overview: 24+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-01-26 20:27 New build system: copy-build-system Pierre Neidhardt
2020-01-27  0:45 ` Jesse Gibbons
2020-01-27  4:14   ` Jack Hill
2020-01-27 14:27 ` zimoun
2020-01-27 14:44   ` Pierre Neidhardt
2020-01-27 15:03     ` zimoun
2020-01-27 15:18       ` Pierre Neidhardt
2020-01-27 15:36         ` zimoun
2020-01-27 15:51           ` Pierre Neidhardt
2020-01-27 16:17             ` zimoun
2020-01-27 18:18               ` Julien Lepiller
2020-01-27 16:45       ` How to deal with very large sources Ricardo Wurmus
2020-01-27 19:03 ` New build system: copy-build-system Ricardo Wurmus
2020-01-28 10:32   ` Ludovic Courtès
2020-01-28 13:39     ` Pierre Neidhardt
2020-01-28 23:04       ` Ludovic Courtès
2020-02-14 12:54         ` Pierre Neidhardt
2020-02-14 13:22           ` Julien Lepiller
2020-02-17  3:46           ` Jesse Gibbons
2020-02-17  7:56             ` Pierre Neidhardt
2020-02-17 16:52               ` Jesse Gibbons
2020-02-17 17:10                 ` Pierre Neidhardt
2020-02-19  1:33                   ` Alex Griffin
2020-02-19 14:01                     ` Pierre Neidhardt

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/guix.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).