Making local development easy

Making local development easy

[Christopher Baines]

[-- Attachment #1.1: Type: text/plain, Size: 1583 bytes --]

Today I had a go at writing a dummy python library, and packaging it,
and then installing it within an environment, with the aim of
experimenting more with how Guix works, and to explore how Guix can be
used as a developer (with unpublished local packages).

There were several slight issues with this, but I got some help on IRC
to work around these.

The first, is that the hash is required, which I only had to compute
once, but if I wanted to change the package, I would have to update
this, which is prohibitive to local development. As an improvement to
this, could the hash be optional, and if it does not exist, be
calculated when the build is performed?

The second issue was getting the data to the store, as first I tried a
relative path, but that did not work, and ended up having to expose the
repository using a HTTP server, and access it over the loopback
interface. This again is rather prohibitive to local development. To
improve on this, could guix build do the insertion in to the store, such
that relative paths would work? This would allow just writing a
"guix.scm" file, which just uses (git-reference (url "./") (commit "HEAD")).

I'm not sure how this use case is perceived, but it seems to me that
supporting supporting use of Guix locally would be helpful, both in
terms of helping users just get things done, and supporting those who
also wish to contribute to Guix.

Any thoughts on this? I'm still learning about all of the Guix
internals, so I'm not sure how feasible the changes I mention above are.

Thanks,

Chris


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: Making local development easy

[Alex Sassmannshausen]

Hello,

Christopher Baines writes:

> Today I had a go at writing a dummy python library, and packaging it,
> and then installing it within an environment, with the aim of
> experimenting more with how Guix works, and to explore how Guix can be
> used as a developer (with unpublished local packages).
>
> There were several slight issues with this, but I got some help on IRC
> to work around these.

I've definitely gone down the same route of wanting to use Guix in local
development.  At the moment I'm using
  guix environment -l project-guix.scm
to spawn a shell with the full working environment for my project — this
is awesome, but I think it's slightly orthogenal to what you describe.

> The first, is that the hash is required, which I only had to compute
> once, but if I wanted to change the package, I would have to update
> this, which is prohibitive to local development. As an improvement to
> this, could the hash be optional, and if it does not exist, be
> calculated when the build is performed?

From my perspective, I think silently calculating a hash on the fly if
it is not provided would be problematic: it might lead to laziness in
completing the hash, which would undermine the security model of Guix
(if I understand correctly).

But maybe an explicit flag setting the declaration to "dev-mode", might
be useful?

> The second issue was getting the data to the store, as first I tried a
> relative path, but that did not work, and ended up having to expose the
> repository using a HTTP server, and access it over the loopback
> interface. This again is rather prohibitive to local development. To
> improve on this, could guix build do the insertion in to the store, such
> that relative paths would work? This would allow just writing a
> "guix.scm" file, which just uses (git-reference (url "./") (commit "HEAD")).

This, I think, would be great.  Specifically, accepting a "file:///"
protocol for (uri)/(git-reference (url)) would be pretty useful…

My 2¢…

Alex

Re: Making local development easy

[Ricardo Wurmus]

Christopher Baines <mail@cbaines.net> writes:

> The second issue was getting the data to the store, as first I tried a
> relative path, but that did not work, and ended up having to expose the
> repository using a HTTP server, and access it over the loopback
> interface. This again is rather prohibitive to local development. To
> improve on this, could guix build do the insertion in to the store, such
> that relative paths would work? This would allow just writing a
> "guix.scm" file, which just uses (git-reference (url "./") (commit "HEAD")).

Maybe I’m misunderstanding the problem, but do you know that you can use
“file://” URIs?  I’m doing this for local copies of tarballs that I
don’t plan to distribute.

~~ Ricardo

Re: Making local development easy

[Alex Sassmannshausen]

Ricardo Wurmus writes:

> Christopher Baines <mail@cbaines.net> writes:
>
>> The second issue was getting the data to the store, as first I tried a
>> relative path, but that did not work, and ended up having to expose the
>> repository using a HTTP server, and access it over the loopback
>> interface. This again is rather prohibitive to local development. To
>> improve on this, could guix build do the insertion in to the store, such
>> that relative paths would work? This would allow just writing a
>> "guix.scm" file, which just uses (git-reference (url "./") (commit "HEAD")).
>
> Maybe I’m misunderstanding the problem, but do you know that you can use
> “file://” URIs?  I’m doing this for local copies of tarballs that I
> don’t plan to distribute.

OMG, awesome.  Note to self: spend more time reading manuals.

Alex

Re: Making local development easy

[Andy Wingo]

On Tue 24 May 2016 20:25, Christopher Baines <mail@cbaines.net> writes:

> The first, is that the hash is required, which I only had to compute
> once, but if I wanted to change the package, I would have to update
> this, which is prohibitive to local development. As an improvement to
> this, could the hash be optional, and if it does not exist, be
> calculated when the build is performed?

Yes I hate this!!!!  I don't know what the right solution is, but for
local development somethign is needed :)

I have nothing constructive to add but these things you mention are very
good points.

Andy

Re: Making local development easy

[Christopher Baines]

[-- Attachment #1.1: Type: text/plain, Size: 942 bytes --]

On 25/05/16 10:23, Alex Sassmannshausen wrote:
>> The first, is that the hash is required, which I only had to compute
>> once, but if I wanted to change the package, I would have to update
>> this, which is prohibitive to local development. As an improvement to
>> this, could the hash be optional, and if it does not exist, be
>> calculated when the build is performed?
> 
> From my perspective, I think silently calculating a hash on the fly if
> it is not provided would be problematic: it might lead to laziness in
> completing the hash, which would undermine the security model of Guix
> (if I understand correctly).
> 
> But maybe an explicit flag setting the declaration to "dev-mode", might
> be useful?

In the use case I described, where you are packaging a local resource, I
don't think this is relevant, as you implicitly trust your own machine.
If you don't for some reason, you can just specify the hash.



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: Making local development easy

[Christopher Baines]

[-- Attachment #1.1: Type: text/plain, Size: 1010 bytes --]

On 25/05/16 11:29, Ricardo Wurmus wrote:
> Christopher Baines <mail@cbaines.net> writes:
>> The second issue was getting the data to the store, as first I tried a
>> relative path, but that did not work, and ended up having to expose the
>> repository using a HTTP server, and access it over the loopback
>> interface. This again is rather prohibitive to local development. To
>> improve on this, could guix build do the insertion in to the store, such
>> that relative paths would work? This would allow just writing a
>> "guix.scm" file, which just uses (git-reference (url "./") (commit "HEAD")).
> 
> Maybe I’m misunderstanding the problem, but do you know that you can use
> “file://” URIs?  I’m doing this for local copies of tarballs that I
> don’t plan to distribute.

I did try this, but it did not work. davexunit on IRC said that this was
due to the isolation in the build process. I was trying to reference
/tmp, where did you place the repository to make this work?



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: Making local development easy

[Leo Famulari]

On Wed, May 25, 2016 at 10:23:20AM +0200, Alex Sassmannshausen wrote:
> Christopher Baines writes:
> > The first, is that the hash is required, which I only had to compute
> > once, but if I wanted to change the package, I would have to update
> > this, which is prohibitive to local development. As an improvement to
> > this, could the hash be optional, and if it does not exist, be
> > calculated when the build is performed?
> 
> From my perspective, I think silently calculating a hash on the fly if
> it is not provided would be problematic: it might lead to laziness in
> completing the hash, which would undermine the security model of Guix
> (if I understand correctly).
> 
> But maybe an explicit flag setting the declaration to "dev-mode", might
> be useful?

Perhaps I'm too paranoid, but I'd rather not see this implemented in
Guix. It would create the necessary elements for a "downgrade attack"
[0], where an attacker exploits some bug to enable the "dev-mode" when
the user doesn't intend it.

Why not write an external script that will recalculate the hash and
rewrite the package definition for you?

Re: Making local development easy

[Ricardo Wurmus]

Christopher Baines <mail@cbaines.net> writes:

> On 25/05/16 11:29, Ricardo Wurmus wrote:
>> Christopher Baines <mail@cbaines.net> writes:
>>> The second issue was getting the data to the store, as first I tried a
>>> relative path, but that did not work, and ended up having to expose the
>>> repository using a HTTP server, and access it over the loopback
>>> interface. This again is rather prohibitive to local development. To
>>> improve on this, could guix build do the insertion in to the store, such
>>> that relative paths would work? This would allow just writing a
>>> "guix.scm" file, which just uses (git-reference (url "./") (commit "HEAD")).
>> 
>> Maybe I’m misunderstanding the problem, but do you know that you can use
>> “file://” URIs?  I’m doing this for local copies of tarballs that I
>> don’t plan to distribute.
>
> I did try this, but it did not work. davexunit on IRC said that this was
> due to the isolation in the build process. I was trying to reference
> /tmp, where did you place the repository to make this work?

I’m using something like this and it works just right:

(source (origin
          (method url-fetch)
          (uri (string-append "file:///srv/some/tarball-v"
                              version ".tar.gz"))
          (sha256
           (base32
            "..."))))

I used /srv but also /gnu and other locations.

~~ Ricardo

Re: Making local development easy

[Ludovic Courtès]

Ricardo Wurmus <rekado@elephly.net> skribis:

> Christopher Baines <mail@cbaines.net> writes:
>
>> On 25/05/16 11:29, Ricardo Wurmus wrote:
>>> Christopher Baines <mail@cbaines.net> writes:
>>>> The second issue was getting the data to the store, as first I tried a
>>>> relative path, but that did not work, and ended up having to expose the
>>>> repository using a HTTP server, and access it over the loopback
>>>> interface. This again is rather prohibitive to local development. To
>>>> improve on this, could guix build do the insertion in to the store, such
>>>> that relative paths would work? This would allow just writing a
>>>> "guix.scm" file, which just uses (git-reference (url "./") (commit "HEAD")).
>>> 
>>> Maybe I’m misunderstanding the problem, but do you know that you can use
>>> “file://” URIs?  I’m doing this for local copies of tarballs that I
>>> don’t plan to distribute.
>>
>> I did try this, but it did not work. davexunit on IRC said that this was
>> due to the isolation in the build process. I was trying to reference
>> /tmp, where did you place the repository to make this work?
>
> I’m using something like this and it works just right:
>
> (source (origin
>           (method url-fetch)
>           (uri (string-append "file:///srv/some/tarball-v"
>                               version ".tar.gz"))
>           (sha256
>            (base32
>             "..."))))
>
> I used /srv but also /gnu and other locations.

Even shorter:

  (source "/some/file/out/there")

or:

  (source ".")

Also, in some cases, --with-source might work for you (info "(guix)
Package Transformation Options"):

  guix build my-package --with-source=my-package-0.1.3rc2.tar.gz

Ludo’.

Re: Making local development easy

[Thompson, David]

On Thu, May 26, 2016 at 4:07 AM, Ludovic Courtès <ludo@gnu.org> wrote:

> Even shorter:
>
>   (source "/some/file/out/there")
>
> or:
>
>   (source ".")

I think there is a bug here because I've tried this at various points
in time and it has never worked.  A pattern matcher somewhere fails
because a string isn't considered an acceptable source object.

- Dave

Re: Making local development easy

[Ludovic Courtès]

"Thompson, David" <dthompson2@worcester.edu> skribis:

> On Thu, May 26, 2016 at 4:07 AM, Ludovic Courtès <ludo@gnu.org> wrote:
>
>> Even shorter:
>>
>>   (source "/some/file/out/there")
>>
>> or:
>>
>>   (source ".")
>
> I think there is a bug here because I've tried this at various points
> in time and it has never worked.  A pattern matcher somewhere fails
> because a string isn't considered an acceptable source object.

I haven’t experience this bug (and see "package-source-derivation, file"
in tests/packages.scm).  However, I’m happy to say that since commit
da675305ddf2ba574e309e515d18ae1f778297be, it is possible to use a
‘local-file’ there:

  (package
    ;; …
    (source (local-file "foo/bar" #:recursive? #t)))

See ‘current-guix’ in (gnu packages package-management) for an example.

Ludo’.