From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:470:142:3::10]:59381) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hZkrY-0003wg-8r for guix-patches@gnu.org; Sat, 08 Jun 2019 19:35:06 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hZkrW-0006Yl-E3 for guix-patches@gnu.org; Sat, 08 Jun 2019 19:35:04 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:40352) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1hZkrW-0006YW-9i for guix-patches@gnu.org; Sat, 08 Jun 2019 19:35:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1hZkrW-0007CC-4k for guix-patches@gnu.org; Sat, 08 Jun 2019 19:35:02 -0400 Subject: [bug#35318] [PATCH] Update cargo-build-system to expand package inputs Resent-Message-ID: Content-Type: multipart/alternative; boundary="Apple-Mail=_036D0402-C7D4-4314-AA67-5843146420CC" Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) From: Ivan Petkov In-Reply-To: <87ftoj9as8.fsf@gmail.com> Date: Sat, 8 Jun 2019 16:33:50 -0700 Message-Id: References: <87ftpsnhal.fsf@gnu.org> <2C03880B-F90E-4949-9637-DC918B6D40A0@gmail.com> <074E4899-C4FF-4A76-8E97-093378D2F8D5@gmail.com> <87pnojvqdu.fsf@gnu.org> <6A0A0108-A722-4D73-85B7-E61AB8230026@gmail.com> <87ftoj9as8.fsf@gmail.com> List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+kyle=kyleam.com@gnu.org Sender: "Guix-patches" To: Chris Marusich Cc: 35318@debbugs.gnu.org --Apple-Mail=_036D0402-C7D4-4314-AA67-5843146420CC Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hi Chris, > On Jun 8, 2019, at 11:44 AM, Chris Marusich = wrote: >=20 > Sorry for taking so long to review this. In short, I think these > changes are good, and if nobody has more feedback in the next few = days, > I will merge it and we can see how it works in practice! Thanks for the feedback, and thanks for the wonderful pros and cons = summary! > Cons: >=20 > - Because the "dependencies" and "dev-dependencies" are specified = as > package arguments instead of any kind of "input", they won't show > up in some of the graphs produced by "guix package". However, in > theory "guix graph" could be taught to display nodes for crate > "dependencies" and "dev-dependencies", too. Totally correct, we can iterate on this in the future. > - Everyone who defines a Guix package for a crate must make sure = the > origin's file name ends in ".cargo", since the cargo-build-system > now assumes that any input ending in ".cargo" is a crate that > should be extracted into the build's crate-dir. This is a little > brittle, and I wish we had a better way to check this, but I = can't > think of a better way at the moment. Since you've updated the > importer to always add this, it probably won't be much of an = issue > in practice, since that's the default way new crates will be = added > to Guix. Going forward, maybe we can avoid this by just checking > the inputs to see which ones are gzipped tarballs containing > Cargo.toml files. We discussed an alternative solution to this offline, namely we can ask tar to check if a Cargo.toml file exists at the top level without fully = unpacking the archive. If it exists, we can assume the tarball is a crate source = and only then unpack it in the vendor directory. This will make things less brittle as the `.crate` convention won=E2=80=99= t be necessary (a potential downside would be that if there happens to be some = arbitrary input which happens to be a tarball which happens to have a Cargo.toml, it = will get included in the vendor directory without us knowing about it. I think = the chances of this happening in practice are virtually zero, so we can iterate on = this if it ever becomes a problem). I=E2=80=99m going to try to update the patch series to include this = change over the next few days (along with Ludo=E2=80=99s naming-related feedback). If I = don=E2=80=99t get a chance to finish this, we can always merge it in later! > Limitations imposed by Rust/Cargo itself: >=20 > - I'm not a Rust or Cargo expert, but my current understanding is > that it isn't feasible to save the artifacts produced when > building a crate for re-use when building another crate. In the > world of C, it is common to produce a library, and then link > against that library when building other software later. In = Guix, > when building a C library, we install the built artifacts (e.g., > .so files) into the output, so that those artifacts can be used = as > the input for another package's build. It seems that, by Cargo's > design, it isn't currently feasible to do the same sort of thing > with Cargo: that is, it isn't feasible to build artifacts, = install > them somewhere for later use, and then later re-use them in > another Cargo build. I'd be glad to learn that I'm mistaken, but > currently that is my understanding. Your understanding here is correct. > - Related to this, I doubt that a Rust programmer will be able to > invoke a command like "guix environment my-crate" (even if we > teach it to understand crate "dependencies" and > "dev-dependencies") to make all the dependencies required to = build > my-crate available. If a Rust programmer wants to hack on > my-crate, they'll probably still just use "cargo" to do it = without > using Guix at all. Is there any way to avoid this and make it > possible to get the dependencies used by Guix in the build, so > that a Rust programmer can hack around using precisely those > dependencies? If this were C or Python, you could do that using > "guix environment," but I'm not sure how this could work with = Rust > crates. Correct again, a programmer will not be able to run `guix environment = my-crate`, however, I don=E2=80=99t think anyone will do this in practice, since = you can=E2=80=99t =E2=80=9Cpoint=E2=80=9D cargo at that source closure/outputs anyway. For hacking on Rust crates, I=E2=80=99d imagine a Rust programmer would = simply stick with cargo and the crates.io ecosystem proper. I don=E2=80=99= t ever see guix as being a replacement for crates.io (it would be impossible to keep up = with the package changes, even if we automate things, crates aren=E2=80=99t guaranteed to even build, etc.). I=E2=80=99d imagine that crates should only be imported into guix if = they=E2=80=99re necessary for supporting an actual application or service written in Rust, but not used for = day-to-day development. =E2=80=94Ivan --Apple-Mail=_036D0402-C7D4-4314-AA67-5843146420CC Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Hi = Chris,

On Jun 8, 2019, at 11:44 AM, Chris Marusich = <cmmarusich@gmail.com> wrote:

Sorry for taking so long to review this. =  In short, I think these
changes are good, and if = nobody has more feedback in the next few days,
I will = merge it and we can see how it works in practice!

Thanks = for the feedback, and thanks for the wonderful pros and cons = summary!

Cons:

=    - Because the "dependencies" and "dev-dependencies" = are specified as
     package = arguments instead of any kind of "input", they won't show
=      up in some of the graphs produced by "guix = package".  However, in
=      theory "guix graph" could be taught to = display nodes for crate
=      "dependencies" and "dev-dependencies", = too.

Totally correct, we can iterate on this in the = future.

   - Everyone who defines a = Guix package for a crate must make sure the
=      origin's file name ends in ".cargo", since = the cargo-build-system
     now = assumes that any input ending in ".cargo" is a crate that
=      should be extracted into the build's = crate-dir.  This is a little
=      brittle, and I wish we had a better way to = check this, but I can't
=      think of a better way at the moment. =  Since you've updated the
=      importer to always add this, it probably = won't be much of an issue
=      in practice, since that's the default way = new crates will be added
     to = Guix.  Going forward, maybe we can avoid this by just checking
     the inputs to see which ones = are gzipped tarballs containing
=      Cargo.toml files.

We = discussed an alternative solution to this offline, namely we can = ask
tar to check if a Cargo.toml file exists at the top level = without fully unpacking
the archive. If it exists, we can = assume the tarball is a crate source and only
then unpack it = in the vendor directory.

This will = make things less brittle as the `.crate` convention won=E2=80=99t be = necessary
(a potential downside would be that if there happens = to be some arbitrary input
which happens to be a tarball which = happens to have a Cargo.toml, it will get
included in the = vendor directory without us knowing about it. I think the = chances
of this happening in practice are virtually zero, so = we can iterate on this if it ever
becomes a = problem).

I=E2=80=99m going to try = to update the patch series to include this change over the = next
few days (along with Ludo=E2=80=99s naming-related = feedback). If I don=E2=80=99t get a chance to
finish this, we = can always merge it in later!

Limitations = imposed by Rust/Cargo itself:

=    - I'm not a Rust or Cargo expert, but my current = understanding is
     that it = isn't feasible to save the artifacts produced when
=      building a crate for re-use when building = another crate.  In the
=      world of C, it is common to produce a = library, and then link
=      against that library when building other = software later.  In Guix,
=      when building a C library, we install the = built artifacts (e.g.,
     .so = files) into the output, so that those artifacts can be used as
     the input for another = package's build.  It seems that, by Cargo's
=      design, it isn't currently feasible to do = the same sort of thing
     with = Cargo: that is, it isn't feasible to build artifacts, install
     them somewhere for later use, = and then later re-use them in
=      another Cargo build.  I'd be glad to = learn that I'm mistaken, but
=      currently that is my understanding.

Your = understanding here is correct.

=    - Related to this, I doubt that a Rust programmer will = be able to
     invoke a command = like "guix environment my-crate" (even if we
=      teach it to understand crate = "dependencies" and
=      "dev-dependencies") to make all the = dependencies required to build
=      my-crate available.  If a Rust = programmer wants to hack on
=      my-crate, they'll probably still just use = "cargo" to do it without
=      using Guix at all.  Is there any way = to avoid this and make it
=      possible to get the dependencies used by = Guix in the build, so
     that = a Rust programmer can hack around using precisely those
=      dependencies?  If this were C or = Python, you could do that using
=      "guix environment," but I'm not sure how = this could work with Rust
=      crates.

Correct = again, a programmer will not be able to run `guix environment = my-crate`,
however, I don=E2=80=99t think anyone will do this = in practice, since you can=E2=80=99t =E2=80=9Cpoint=E2=80=9D = cargo
at that source closure/outputs anyway.

For hacking on Rust crates, I=E2=80=99d imagine a = Rust programmer would simply stick with
cargo and the crates.io ecosystem = proper. I don=E2=80=99t ever see guix as being a = replacement
for crates.io (it would be impossible to keep up with = the package changes, even if we
automate things, crates = aren=E2=80=99t guaranteed to even build, etc.).

I=E2=80=99d imagine that crates should only be = imported into guix if they=E2=80=99re necessary for = supporting
an actual application or service written in Rust, = but not used for day-to-day development.

=E2=80=94Ivan


= --Apple-Mail=_036D0402-C7D4-4314-AA67-5843146420CC--