* Idea: a meta language for (language) build systems - npm, Racket, Rust cargo @ 2021-05-30 8:38 Pjotr Prins 2021-05-30 19:17 ` Konrad Hinsen 0 siblings, 1 reply; 21+ messages in thread From: Pjotr Prins @ 2021-05-30 8:38 UTC (permalink / raw) To: guix-devel Maybe this is a crazy idea: It appears to me that every language out there today is creating their own build system. We know that creating build-system package support for GNU Guix is complicated by: 1. Live updates over the internet 2. Circular dependencies 3. Tests requiring internet access Now we never got to truely solving npm, for example. And despite heroic efforts: Go, Rust, Racket are often in stages of disarray. The irony is that, when we leverage GNU Guix infrastructure, the software packages themselves can be really simple. All we need is a source ball and a little metadata. From that GNU Guix can do the hard stuff and handle the builds and dependencies. This is why our C build system is so effective and simple packages translate well to GNU Guix. My idea is this: rather than generating 'imports' from native build packages for every target - which is kinda laborious in its way - we could translate an existing Rust/Node/Racket package into an intermediate simple package tar ball - call it a sexp-pack ;). This repackaged sexp-pack can only reference other sexp-packs which gets rid of (1). The sexp-pack generator can identify and help resolve circular dependencies which solves (2). At this intermediate stage we can also also patch sources which helps with (3). We would have a sexp-pack for cargo and another for npm. As sexp-packs are generated from source, the GNU Guix turn around and packaging time can potentially be sped up. And once a hosted sexp-pack exists it is stable. It may be silly to create another layer (of indirection). But I think that breaking up the packaging process this way will help us to actually fix npm and cargo support in a reasonable time frame. Also it works the other way, I would no longer need cargo or Racket packages ;) Pj. ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Idea: a meta language for (language) build systems - npm, Racket, Rust cargo 2021-05-30 8:38 Idea: a meta language for (language) build systems - npm, Racket, Rust cargo Pjotr Prins @ 2021-05-30 19:17 ` Konrad Hinsen 2021-05-31 17:47 ` Pjotr Prins 0 siblings, 1 reply; 21+ messages in thread From: Konrad Hinsen @ 2021-05-30 19:17 UTC (permalink / raw) To: Pjotr Prins, guix-devel Hi Pjotr, > Maybe this is a crazy idea: Not crazy enough for me ;-) Today's mess with language and platform specific build systems is indeed a problem, and I have been thinking as well about how Guix could help with this beyond what it already does. Guix' big achievement, beyond being a practically useful tool, is demonstrating what it takes to have a full system build system. Everyone's life would be easier if software packaging were made more compatible with how Guix (and Nix of course) work today. And one way to get the rest of the world move towards this goal would be to help them: propose and implement changes to their build systems that would make integration with Guix easier. So... > My idea is this: rather than generating 'imports' from native build > packages for every target - which is kinda laborious in its way - we > could translate an existing Rust/Node/Racket package into an > intermediate simple package tar ball - call it a sexp-pack ;). This How about pushing all the other package manager towards producing sexp-packs, and helping them to get there? Cheers, Konrad ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Idea: a meta language for (language) build systems - npm, Racket, Rust cargo 2021-05-30 19:17 ` Konrad Hinsen @ 2021-05-31 17:47 ` Pjotr Prins 2021-06-01 5:50 ` Adriano Peluso ` (3 more replies) 0 siblings, 4 replies; 21+ messages in thread From: Pjotr Prins @ 2021-05-31 17:47 UTC (permalink / raw) To: Konrad Hinsen; +Cc: guix-devel On Sun, May 30, 2021 at 09:17:20PM +0200, Konrad Hinsen wrote: > How about pushing all the other package manager towards producing > sexp-packs, and helping them to get there? I have a feeling they won't be that interested ;). My thoughts are that every software package simply consists of files that need to be compiled (if not interpreted) and be copied in place. As Guix takes care of the first and the last - the issue centers around building. The idea is to dress down these language specific builders, such as cargo, so you don't have all the included complexity. A sexp-pack would represent the most simple build instructions to build a package on its own. Now, of course the current guix-builders solve that too. But, what I am proposing is to split out the actual build step into a package definition, so as to present something simpler to Guix. I found a cargo -> ninja converter. It is that kind of idea. Guix would use ninja with rustc instead of cargo. A stripped down cargo could potentially work too - but cargo is a complex beast. A simplified build step would make it easier to troubleshoot these packages. See what I mean? Pj. ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Idea: a meta language for (language) build systems - npm, Racket, Rust cargo 2021-05-31 17:47 ` Pjotr Prins @ 2021-06-01 5:50 ` Adriano Peluso 2021-06-01 6:24 ` Leo Prikler ` (2 subsequent siblings) 3 siblings, 0 replies; 21+ messages in thread From: Adriano Peluso @ 2021-06-01 5:50 UTC (permalink / raw) To: Pjotr Prins, Konrad Hinsen; +Cc: guix-devel Il giorno lun, 31/05/2021 alle 19.47 +0200, Pjotr Prins ha scritto: > On Sun, May 30, 2021 at 09:17:20PM +0200, Konrad Hinsen wrote: > > How about pushing all the other package manager towards producing > > sexp-packs, and helping them to get there? > > I have a feeling they won't be that interested ;). > > My thoughts are that every software package simply consists of files > that need to be compiled (if not interpreted) and be copied in place. > > As Guix takes care of the first and the last - the issue centers > around building. The idea is to dress down these language specific > builders, such as cargo, so you don't have all the included > complexity. > > A sexp-pack would represent the most simple build instructions to > build a package on its own. Now, of course the current guix-builders > solve that too. But, what I am proposing is to split out the actual > build step into a package definition, so as to present something > simpler to Guix. > > I found a cargo -> ninja converter. It is that kind of idea. Guix > would use ninja with rustc instead of cargo. A stripped down cargo > could potentially work too - but cargo is a complex beast. > > A simplified build step would make it easier to troubleshoot these > packages. > > See what I mean? You mean to break the chain introducing an intermediate step With intermediate step, I mean a new kind of entity with its own format I like the idea Many times, breaking long chains makes them more understandable to me It's the old idea of composability, if you want It's like refactoring a huge procedure in several smaller ones It also resembles a sort of ETL process or some sort of "data science" task of "massaging" data for some later analisys phase In a way, doing this would in itself push language communities to think about their approach (of releasing blobs) I doubt they would start such an initiative themselves, they wouldn't have begun releasing blobs in the first place ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Idea: a meta language for (language) build systems - npm, Racket, Rust cargo 2021-05-31 17:47 ` Pjotr Prins 2021-06-01 5:50 ` Adriano Peluso @ 2021-06-01 6:24 ` Leo Prikler 2021-06-01 7:23 ` Pjotr Prins 2021-06-01 8:59 ` Adriano Peluso 2021-06-01 7:56 ` Konrad Hinsen 2021-06-08 13:11 ` Ludovic Courtès 3 siblings, 2 replies; 21+ messages in thread From: Leo Prikler @ 2021-06-01 6:24 UTC (permalink / raw) To: Pjotr Prins, Konrad Hinsen; +Cc: guix-devel Am Montag, den 31.05.2021, 19:47 +0200 schrieb Pjotr Prins: > On Sun, May 30, 2021 at 09:17:20PM +0200, Konrad Hinsen wrote: > > How about pushing all the other package manager towards producing > > sexp-packs, and helping them to get there? > > I have a feeling they won't be that interested ;). I do think some of them are interested, but you're right in that the industry as a whole is not. > My thoughts are that every software package simply consists of files > that need to be compiled (if not interpreted) and be copied in place. That is the case, but surely you aren't suggesting to implement all build systems in terms of copy-build-system, right? :P > As Guix takes care of the first and the last - the issue centers > around building. The idea is to dress down these language specific > builders, such as cargo, so you don't have all the included > complexity. The thing with complex systems is that you will still have complexity no matter how you look at them and cargo is such a system. The only way to not see that complexity is to ignore it because your build succeeds, and I'm sure Rust, npm, etc. users (read: victims) are physically trained to clear their mind from such heathen thought. > A sexp-pack would represent the most simple build instructions to > build a package on its own. Now, of course the current guix-builders > solve that too. But, what I am proposing is to split out the actual > build step into a package definition, so as to present something > simpler to Guix. I don't think this would be simpler to Guix, you'd just create even more packages, that actually aren't usable. > I found a cargo -> ninja converter. It is that kind of idea. Guix > would use ninja with rustc instead of cargo. A stripped down cargo > could potentially work too - but cargo is a complex beast. The problem with rust+cargo is not just that cargo isn't make or ninja (although that is certainly a part of it), it is also that any make/ninja port would have to somehow take care of all the other things that cargo does. It doesn't shake the ginormous dependency trees for example. This is not to say, that rustc+ninja is not worth pursuing. If the output of rustc+ninja had some nice property, e.g. it was a shared library, that could be reused, we might still want to rewrite rust- build-system in terms of it. In a similar manner, Guix people are currently looking towards Rome to perhaps simplify node-build-system or erect a new build system next to it. Small rant w.r.t. #:skip-build?, this flag is a hack and everyone involved knows it and we ought to find a way to actually build rust crates in a manner that doesn't require complete source closure for the next rust-thingy to use it. > A simplified build step would make it easier to troubleshoot these > packages. I think I'd personally rather deal with the output of make or ninja over that provided by rust/cargo 9 times out of 10 if that is what you're going for, but different strokes for different folks. We don't want to alienate all the people actually using Rust on Guix by taking away their favourite error message :P Regards, Leo ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Idea: a meta language for (language) build systems - npm, Racket, Rust cargo 2021-06-01 6:24 ` Leo Prikler @ 2021-06-01 7:23 ` Pjotr Prins 2021-06-01 8:12 ` Leo Prikler 2021-06-01 8:59 ` Adriano Peluso 1 sibling, 1 reply; 21+ messages in thread From: Pjotr Prins @ 2021-06-01 7:23 UTC (permalink / raw) To: Leo Prikler; +Cc: guix-devel On Tue, Jun 01, 2021 at 08:24:51AM +0200, Leo Prikler wrote: > > I have a feeling they won't be that interested ;). > I do think some of them are interested, but you're right in that the > industry as a whole is not. Industry appears to go the other way. There are several initiatives to speed up Rust through caching and storing compiled items in the cloud(?!). > The thing with complex systems is that you will still have complexity > no matter how you look at them and cargo is such a system. The only Particularly where it comes to cross-platform building and things like CUDA and webassembly. I don't think we should try and duplicate all of cargo though. > I don't think this would be simpler to Guix, you'd just create even > more packages, that actually aren't usable. We'd have to see. Adriano made a great point about composability. Truth is *I* have been annoyed by build systems for ages - one of the first attractions of Guix was successfully getting rid of Ruby bundler when we wrote the Ruby build system. Guix goes a long way towards simplifying the actual requirements for a build system because it handles dependencies and 'flavours'. So, we can come up with something simple. When I was coding in D I had similar thoughts - I managed to avoid the D build system completely. Now I want to achieve the same with Rust! > that cargo does. It doesn't shake the ginormous dependency trees for > example. No. That issue sits partly with developers though Cargo and npm make it far too easy to pull in 500 dependencies ;). I always remember Joe Armstrong, who said that part of the longevity of Erlang is that it had virtually no dependencies. Long running projects do well to think about that statement. There is also the security angle... Simplifying packages is not the remit of GNU Guix. But sexp-pack (or should we name it sixp-pack to avoid criticism?) can simplify by throwing out needless dependencies. I don't buy it when an 'hello world' program requires 100+ dependencies. And I have already seen that with Rust. If we manage to scale down and influence developers - we may influence the industry. Who knows :) > This is not to say, that rustc+ninja is not worth pursuing. If the > output of rustc+ninja had some nice property, e.g. it was a shared > library, that could be reused, we might still want to rewrite rust- > build-system in terms of it. In a similar manner, Guix people are The cargo->ninja converter was just an example. I kinda like ninja because it is really simple minded and minimalistic. We could come up with a scheme->ninja generator (I don't like cmake or meson so much). ninja comes into play when you want to do incremental builds - which is important for developers. Another reason to introduce a new layer. > currently looking towards Rome to perhaps simplify node-build-system or > erect a new build system next to it. I really respect the accumulated knowledge we have in the Guix community for build systems. That is one reason for bringing it up on the mailing list rather than trying to just hack something. > Small rant w.r.t. #:skip-build?, this flag is a hack and everyone > involved knows it and we ought to find a way to actually build rust > crates in a manner that doesn't require complete source closure for the > next rust-thingy to use it. The real problem is that all(?) sources need to be visible, similar to .h include files in C, right? Rust code inlining optimizations too, so it needs the source view. To me the solution sounds to include the necessary source code with the deployed package, or at least have a -dev version for builds. Efraim suggest looking at what Debian does. They somehow include the full source space. > > A simplified build step would make it easier to troubleshoot these > > packages. > I think I'd personally rather deal with the output of make or ninja > over that provided by rust/cargo 9 times out of 10 if that is what > you're going for, but different strokes for different folks. We don't > want to alienate all the people actually using Rust on Guix by taking > away their favourite error message :P We can still have both ;). It is true that sixp-pack would be an alt-verse, unless we ascertain cargo picks up deployed -dev packages properly. Another option is to hack and partly disable cargo - so it only builds. I know we are resistant to changing upstream packagers, but if it is the easier way we might want to consider it. Pj. ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Idea: a meta language for (language) build systems - npm, Racket, Rust cargo 2021-06-01 7:23 ` Pjotr Prins @ 2021-06-01 8:12 ` Leo Prikler 0 siblings, 0 replies; 21+ messages in thread From: Leo Prikler @ 2021-06-01 8:12 UTC (permalink / raw) To: Pjotr Prins; +Cc: guix-devel Am Dienstag, den 01.06.2021, 09:23 +0200 schrieb Pjotr Prins: > On Tue, Jun 01, 2021 at 08:24:51AM +0200, Leo Prikler wrote: > > > I have a feeling they won't be that interested ;). > > I do think some of them are interested, but you're right in that > > the > > industry as a whole is not. > > Industry appears to go the other way. There are several initiatives > to speed up Rust through caching and storing compiled items in the > cloud(?!). We can go even faster: Build all rust stuff itself in the cloud and never locally. That way, we can also ensure, that there's no pesky people trying to inject other packages by building offline. > > The thing with complex systems is that you will still have > > complexity no matter how you look at them and cargo is such a > > system. The only > > Particularly where it comes to cross-platform building > and things like CUDA and webassembly. I don't think we should > try and duplicate all of cargo though. Neither do I. > > I don't think this would be simpler to Guix, you'd just create even > > more packages, that actually aren't usable. > > We'd have to see. Adriano made a great point about composability. > > Truth is *I* have been annoyed by build systems for ages - one of the > first attractions of Guix was successfully getting rid of Ruby > bundler when we wrote the Ruby build system. Guix goes a long way > towards simplifying the actual requirements for a build system > because it handles dependencies and 'flavours'. So, we can come up > with something simple. When I was coding in D I had similar thoughts > - I managed to avoid the D build system completely. Now I want to > achieve the same with Rust! That is a noble goal, but I'm not convinced your proposed solution does that. It rather appears to me as though it pushes the actual problem elsewhere. > > that cargo does. It doesn't shake the ginormous dependency trees > > for example. > > No. That issue sits partly with developers though Cargo and npm make > it far too easy to pull in 500 dependencies ;). I always remember Joe > Armstrong, who said that part of the longevity of Erlang is that it > had virtually no dependencies. Long running projects do well to think > about that statement. There is also the security angle... The issue is very much in the system as it encourages and often requires such behaviour. Basically, language package managers don't know how to be good package managers. I see the same problem with the proposed package management solutions towards Erlang. Ever heard about Elixir? > Simplifying packages is not the remit of GNU Guix. But sexp-pack > (or should we name it sixp-pack to avoid criticism?) can simplify by > throwing out needless dependencies. I don't buy it when an 'hello > world' program requires 100+ dependencies. And I have already seen > that with Rust. If we manage to scale down and influence developers - > we may influence the industry. Who knows :) How would your macho build system be any different from Guix in that regard, though? Guix already has all the notions of inputs, outputs, etc. that you need. Especially with Rust, which already has packages that DON'T DO ANYTHING DURING BUILD, I struggle to see how macho build system would be a useful abstraction. Now you might ask yourself "why do I even need to package Rust library #2147483647, can't I just stuff everything into a computed origin and yolo my way out?", but that's exactly the kind of thing we DON'T want to do. > > This is not to say, that rustc+ninja is not worth pursuing. If the > > output of rustc+ninja had some nice property, e.g. it was a shared > > library, that could be reused, we might still want to rewrite rust- > > build-system in terms of it. In a similar manner, Guix people are > > The cargo->ninja converter was just an example. I kinda like ninja > because it is really simple minded and minimalistic. We could come up > with a scheme->ninja generator (I don't like cmake or meson so much). > ninja comes into play when you want to do incremental builds - which > is important for developers. Another reason to introduce a new layer. GWL also has incremental builds. What's your point? > > Small rant w.r.t. #:skip-build?, this flag is a hack and everyone > > involved knows it and we ought to find a way to actually build rust > > crates in a manner that doesn't require complete source closure for > > the next rust-thingy to use it. > > The real problem is that all(?) sources need to be visible, similar > to .h include files in C, right? Rust code inlining optimizations > too, so it needs the source view. To me the solution sounds to > include the necessary source code with the deployed package, or at > least have a -dev version for builds. Efraim suggest looking at what > Debian does. They somehow include the full source space. Assuming, that master does not yet copy all sources to output, a patch that does that already sits in the MLs. No, it does not solve the issues I've pointed out, particularly not #:skip-build? > > > A simplified build step would make it easier to troubleshoot > > > these > > > packages. > > I think I'd personally rather deal with the output of make or ninja > > over that provided by rust/cargo 9 times out of 10 if that is what > > you're going for, but different strokes for different folks. We > > don't > > want to alienate all the people actually using Rust on Guix by > > taking > > away their favourite error message :P > > We can still have both ;). It is true that sixp-pack would be an > alt-verse, unless we ascertain cargo picks up deployed -dev packages > properly. We don't do -dev packages. At best, we might want to have outputs for static libs or binaries or stuff like that. No -dev packages. > Another option is to hack and partly disable cargo - so it only > builds. I know we are resistant to changing upstream packagers, but > if it is the easier way we might want to consider it. We don't need to hack cargo so that it only builds, we have already configured it, so that inside a build environment anything that would result in it not building but rather doing something else instead conjures the error message Rust devs believe to be the most appropriate for it not being able to do that. I don't think such a change is meaningful outside of `guix build'. We might still need to do it to comply with the FSDG, but that's a different issue. Regards, Leo ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Idea: a meta language for (language) build systems - npm, Racket, Rust cargo 2021-06-01 6:24 ` Leo Prikler 2021-06-01 7:23 ` Pjotr Prins @ 2021-06-01 8:59 ` Adriano Peluso 2021-06-01 9:11 ` Leo Prikler 1 sibling, 1 reply; 21+ messages in thread From: Adriano Peluso @ 2021-06-01 8:59 UTC (permalink / raw) To: Leo Prikler, Pjotr Prins; +Cc: guix-devel Il giorno mar, 01/06/2021 alle 08.24 +0200, Leo Prikler ha scritto: > > A sexp-pack would represent the most simple build instructions to > > build a package on its own. Now, of course the current guix- > > builders > > solve that too. But, what I am proposing is to split out the actual > > build step into a package definition, so as to present something > > simpler to Guix. > I don't think this would be simpler to Guix, you'd just create even > more packages, that actually aren't usable. The output could be a collection of .tar.gz files distributed through ipfs, bittorrent, syncthing or rsync Not necessarily packages in the way Guix intends them I understand there's already some work going on to reproduce tarballs in a format convenient to Guix (maybe with proper hashes and metadata ?) for when they get erased by distributors ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Idea: a meta language for (language) build systems - npm, Racket, Rust cargo 2021-06-01 8:59 ` Adriano Peluso @ 2021-06-01 9:11 ` Leo Prikler 2021-06-01 10:52 ` Adriano Peluso 0 siblings, 1 reply; 21+ messages in thread From: Leo Prikler @ 2021-06-01 9:11 UTC (permalink / raw) To: randomlooser, Pjotr Prins; +Cc: guix-devel Am Dienstag, den 01.06.2021, 10:59 +0200 schrieb Adriano Peluso: > Il giorno mar, 01/06/2021 alle 08.24 +0200, Leo Prikler ha scritto: > > > A sexp-pack would represent the most simple build instructions to > > > build a package on its own. Now, of course the current guix- > > > builders solve that too. But, what I am proposing is to split out > > > the actual build step into a package definition, so as to present > > > something simpler to Guix. > > I don't think this would be simpler to Guix, you'd just create even > > more packages, that actually aren't usable. > > The output could be a collection of .tar.gz files distributed through > ipfs, bittorrent, syncthing or rsync > > Not necessarily packages in the way Guix intends them > > I understand there's already some work going on to reproduce tarballs > in a format convenient to Guix (maybe with proper hashes and metadata > ?) for when they get erased by distributors Well, ideally Guix would have have ipfs-fetch, bittorrent-fetch etc. as methods or fallbacks, but this doesn't solve the problem that's posed here. You can't just pull the complete source closure of e.g. Fractal over the ether and pretend it's just one package. We already drop all vendored dependencies from tarballs, that aren't created by Rust et al., this does the exact opposite. Regards, Leo ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Idea: a meta language for (language) build systems - npm, Racket, Rust cargo 2021-06-01 9:11 ` Leo Prikler @ 2021-06-01 10:52 ` Adriano Peluso 2021-06-01 11:03 ` Leo Prikler 0 siblings, 1 reply; 21+ messages in thread From: Adriano Peluso @ 2021-06-01 10:52 UTC (permalink / raw) To: Leo Prikler, Pjotr Prins; +Cc: guix-devel Il giorno mar, 01/06/2021 alle 11.11 +0200, Leo Prikler ha scritto: > > The output could be a collection of .tar.gz files distributed > > through > > ipfs, bittorrent, syncthing or rsync > > > > Not necessarily packages in the way Guix intends them > > > > I understand there's already some work going on to reproduce > > tarballs > > in a format convenient to Guix (maybe with proper hashes and > > metadata > > ?) for when they get erased by distributors > Well, ideally Guix would have have ipfs-fetch, bittorrent-fetch etc. > as > methods or fallbacks, but this doesn't solve the problem that's posed > here. You can't just pull the complete source closure of e.g. > Fractal > over the ether and pretend it's just one package. Probably the Fractal package will depend on some others, so it's gonna be a collection 🤷️ Doesn't that happen already for traditional tarballs ? > We already drop all > vendored dependencies from tarballs, that aren't created by Rust et > al., this does the exact opposite. I'm not sure I understand This does the opposite ? How so ? ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Idea: a meta language for (language) build systems - npm, Racket, Rust cargo 2021-06-01 10:52 ` Adriano Peluso @ 2021-06-01 11:03 ` Leo Prikler 2021-06-01 11:28 ` Adriano Peluso 0 siblings, 1 reply; 21+ messages in thread From: Leo Prikler @ 2021-06-01 11:03 UTC (permalink / raw) To: randomlooser, Pjotr Prins; +Cc: guix-devel Am Dienstag, den 01.06.2021, 12:52 +0200 schrieb Adriano Peluso: > Il giorno mar, 01/06/2021 alle 11.11 +0200, Leo Prikler ha scritto: > > > The output could be a collection of .tar.gz files distributed > > > through > > > ipfs, bittorrent, syncthing or rsync > > > > > > Not necessarily packages in the way Guix intends them > > > > > > I understand there's already some work going on to reproduce > > > tarballs > > > in a format convenient to Guix (maybe with proper hashes and > > > metadata > > > ?) for when they get erased by distributors > > Well, ideally Guix would have have ipfs-fetch, bittorrent-fetch > > etc. > > as > > methods or fallbacks, but this doesn't solve the problem that's > > posed > > here. You can't just pull the complete source closure of e.g. > > Fractal > > over the ether and pretend it's just one package. > > Probably the Fractal package will depend on some others, so it's > gonna be a collection 🤷️ > > Doesn't that happen already for traditional tarballs ? We don't stuff tarball collections into packages. We stuff inputs into packages and one input equals one tarball. > > We already drop all > > vendored dependencies from tarballs, that aren't created by Rust et > > al., this does the exact opposite. > > I'm not sure I understand > > This does the opposite ? > > How so ? Let's assume we form this sexp-pack and use it as input to some package. What happens? Regards, Leo ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Idea: a meta language for (language) build systems - npm, Racket, Rust cargo 2021-06-01 11:03 ` Leo Prikler @ 2021-06-01 11:28 ` Adriano Peluso 2021-06-01 15:22 ` Leo Prikler 0 siblings, 1 reply; 21+ messages in thread From: Adriano Peluso @ 2021-06-01 11:28 UTC (permalink / raw) To: Leo Prikler, Pjotr Prins; +Cc: guix-devel Il giorno mar, 01/06/2021 alle 13.03 +0200, Leo Prikler ha scritto: > Am Dienstag, den 01.06.2021, 12:52 +0200 schrieb Adriano Peluso: > > Il giorno mar, 01/06/2021 alle 11.11 +0200, Leo Prikler ha scritto: > > > > The output could be a collection of .tar.gz files distributed > > > > through > > > > ipfs, bittorrent, syncthing or rsync > > > > > > > > Not necessarily packages in the way Guix intends them > > > > > > > > I understand there's already some work going on to reproduce > > > > tarballs > > > > in a format convenient to Guix (maybe with proper hashes and > > > > metadata > > > > ?) for when they get erased by distributors > > > Well, ideally Guix would have have ipfs-fetch, bittorrent-fetch > > > etc. > > > as > > > methods or fallbacks, but this doesn't solve the problem that's > > > posed > > > here. You can't just pull the complete source closure of e.g. > > > Fractal > > > over the ether and pretend it's just one package. > > > > Probably the Fractal package will depend on some others, so it's > > gonna be a collection 🤷️ > > > > Doesn't that happen already for traditional tarballs ? > We don't stuff tarball collections into packages. We stuff inputs > into > packages and one input equals one tarball. > > > > We already drop all > > > vendored dependencies from tarballs, that aren't created by Rust > > > et > > > al., this does the exact opposite. > > > > I'm not sure I understand > > > > This does the opposite ? > > > > How so ? > Let's assume we form this sexp-pack and use it as input to some > package. What happens? we wouldn't use a sexp-pack as an input to a guix package in the same way as, as you noticed, we don't feed tarballs as inputs to guix packages A Guix package doesn't depend on some tarballs. It depends on some other Guix packages In the same way, a Guix package wouldn't depend on a sexp-pack. You can think of sexp-pack as an alternative format to tarball With, maybe, some more metadata For example, a sexp-pack could contain a hash of itself and hashes of other _sexp-packs_ it depends on Similarly to how, for example, python packages on pypi express dependecies on other packages in pypi The difference is that, as far as I understand, python packages (those in pypi, not those in Guix) express dependencies in a somewhat loose way Guix packages are stricter sexp-packs could be stricter too, bringing part of the data reconciliation outside of Guix A Guix importer could recursively import sexp-packs the same way the python importer... I'm assuming that a Rust package can be built in a sane way, with dependencies properly sorted out. I know that's possible for javascript packages, I'm not sure about Rust Such a data/packages collection could be used by mainstream linux distributions too, as far as I understand 🤷️ ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Idea: a meta language for (language) build systems - npm, Racket, Rust cargo 2021-06-01 11:28 ` Adriano Peluso @ 2021-06-01 15:22 ` Leo Prikler 0 siblings, 0 replies; 21+ messages in thread From: Leo Prikler @ 2021-06-01 15:22 UTC (permalink / raw) To: randomlooser, Pjotr Prins; +Cc: guix-devel Am Dienstag, den 01.06.2021, 13:28 +0200 schrieb Adriano Peluso: > Il giorno mar, 01/06/2021 alle 13.03 +0200, Leo Prikler ha scritto: > > Am Dienstag, den 01.06.2021, 12:52 +0200 schrieb Adriano Peluso: > > > Il giorno mar, 01/06/2021 alle 11.11 +0200, Leo Prikler ha > > > scritto: > > > > > [...] > > > Probably the Fractal package will depend on some others, so it's > > > gonna be a collection 🤷️ > > > > > > Doesn't that happen already for traditional tarballs ? > > We don't stuff tarball collections into packages. We stuff inputs > > into > > packages and one input equals one tarball. > > > > > > We already drop all > > > > vendored dependencies from tarballs, that aren't created by > > > > Rust > > > > et > > > > al., this does the exact opposite. > > > > > > I'm not sure I understand > > > > > > This does the opposite ? > > > > > > How so ? > > Let's assume we form this sexp-pack and use it as input to some > > package. What happens? > > we wouldn't use a sexp-pack as an input to a guix package in the same > way as, as you noticed, we don't feed tarballs as inputs to guix > packages > > A Guix package doesn't depend on some tarballs. It depends on some > other Guix packages That's only half-true. You can stuff origins into inputs and people do that. > In the same way, a Guix package wouldn't depend on a sexp-pack. > > You can think of sexp-pack as an alternative format to tarball > > With, maybe, some more metadata Tarball + metadata is a standard of package management.[1] > For example, a sexp-pack could contain a hash of itself and hashes of > other _sexp-packs_ it depends on > > Similarly to how, for example, python packages on pypi express > dependecies on other packages in pypi > > The difference is that, as far as I understand, python packages > (those in pypi, not those in Guix) express dependencies in a somewhat > loose way > > Guix packages are stricter > > sexp-packs could be stricter too, bringing part of the data > reconciliation outside of Guix I don't see the benefit here. At best, sexp-packs are actual packages, at worst they're closer to origins. When bringing "dependencies" into the mix, one might want to implement them as packages with near-empty build, that have propagated-inputs. I'm really struggling to see the use case here. Packages are useful abstractions to both developers and maintainers – the former need not really care about the build step if all they want is an environment. > A Guix importer could recursively import sexp-packs the same way the > python importer... You do know we can already import packages from JSON, right? > I'm assuming that a Rust package can be built in a sane way, with > dependencies properly sorted out. > > I know that's possible for javascript packages, I'm not sure about > Rust The mathematical term you're seeking is "conjecture". > Such a data/packages collection could be used by mainstream linux > distributions too, as far as I understand 🤷️ Guess what, distros already use it. Not the hypothetical data collection you want to establish, the ones that actually exist as published by pip, cpan, npm, cargo… Whatever fancy intermediate format you want to think up, the fact that this won't make anyone's lives easier follows from the data processing inequality. In other words, how you decide to lay out that data *does not matter* as long as it can somehow be parsed (and remember that your own tools will also need to parse it) and you don't omit anything – at best importing data from your layout will be as good as importing it from the original source. In order to improve this situation, you would either need to improve the tools in question (e.g. make it so that cargo actually produces shared libraries) or replace them with technologies, that have the properties we want (perhaps cargo by ninja). If you furthermore want a ubiquitous solution, guess what, that'd be Guix itself. So you're looking for something that's either similar to Guix or more accommodating to Guix, that will get accepted by some language community. Note, that you can't really sell that solution on the same grounds as Guix – as otherwise Guix would already be used instead – and you will also have to compete against whatever technology already exists, e.g. cargo or npm. W.r.t. npm, I think an npm importer would already be a great improvement. IIUC the biggest problem with npm is the sheer number of inputs. We might also try to deduplicate dependencies, but alas, let's not get ahead of ourselves. Regards, Leo [1] https://xkcd.com/927/ ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Idea: a meta language for (language) build systems - npm, Racket, Rust cargo 2021-05-31 17:47 ` Pjotr Prins 2021-06-01 5:50 ` Adriano Peluso 2021-06-01 6:24 ` Leo Prikler @ 2021-06-01 7:56 ` Konrad Hinsen 2021-06-01 23:35 ` Joshua Branson 2021-06-08 13:11 ` Ludovic Courtès 3 siblings, 1 reply; 21+ messages in thread From: Konrad Hinsen @ 2021-06-01 7:56 UTC (permalink / raw) To: Pjotr Prins; +Cc: guix-devel Hi Pjotr, > I have a feeling they won't be that interested ;). Depends on who and when. My idea is not to send them a mail saying, "look, we have a great idea, you should us it!". It's rather to open issues on their build systems to suggest features and actually help with the implementations. Good old proselytism ;-) It's a long-term strategy, not a quick fix project. > Industry appears to go the other way. There are several initiatives to > speed up Rust through caching and storing compiled items in the > cloud(?!). Then it's time to intervene before it's too late ;-) There is a clear momentum building up for reproducibility. Few people argue nowadays that it doesn't matter. It's more like "would be nice, but it's too difficult/complicated". That's where Guix can help: as a demonstration of possibility. And once your builds are reproducible, you can safely cache them in the cloud! The next message to get across is the need for cross-language build systems. The idea of one language, one platform doesn't work any more. Real software systems need to mix languages, even runtimes. And it's not just X + C any more either. Again, Guix serves as the proof of concept, even for people who dislike some specific implementation aspect of Guix. Cheers, Konrad. ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Idea: a meta language for (language) build systems - npm, Racket, Rust cargo 2021-06-01 7:56 ` Konrad Hinsen @ 2021-06-01 23:35 ` Joshua Branson 0 siblings, 0 replies; 21+ messages in thread From: Joshua Branson @ 2021-06-01 23:35 UTC (permalink / raw) To: Konrad Hinsen; +Cc: Pjotr Prins, guix-devel Konrad Hinsen <konrad.hinsen@fastmail.net> writes: > Hi Pjotr, > > > The next message to get across is the need for cross-language build > systems. The idea of one language, one platform doesn't work any more. > Real software systems need to mix languages, even runtimes. And it's not > just X + C any more either. Again, Guix serves as the proof of concept, > even for people who dislike some specific implementation aspect of Guix. Thinking about "cross-language build systems" reminds me of Zig cc https://andrewkelley.me/post/zig-cc-powerful-drop-in-replacement-gcc-clang.html #+BEGIN SRC sh andy@ark ~/tmp> cat hello.c #include <stdio.h> int main(int argc, char **argv) { fprintf(stderr, "Hello, World!\n"); return 0; } andy@ark ~/tmp> zig cc -o hello.exe hello.c -target x86_64-windows-gnu andy@ark ~/tmp> wine64 hello.exe Hello, World! andy@ark ~/tmp> zig cc -o hello hello.c -target mipsel-linux-musl andy@ark ~/tmp> qemu-mipsel ./hello Hello, World! andy@ark ~/tmp> zig cc -o hello hello.c -target aarch64-linux-gnu andy@ark ~/tmp> qemu-aarch64 -L ~/Downloads/glibc/multi-2.31/install/glibcs/aarch64-linux-gnu ./hello Hello, World! #+END_SRC I think that's pretty cool! > > Cheers, > Konrad. > -- Joshua Branson (joshuaBPMan in #guix) Sent from Emacs and Gnus https://gnucode.me https://video.hardlimit.com/accounts/joshua_branson/video-channels https://propernaming.org "You can have whatever you want, as long as you help enough other people get what they want." - Zig Ziglar ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Idea: a meta language for (language) build systems - npm, Racket, Rust cargo 2021-05-31 17:47 ` Pjotr Prins ` (2 preceding siblings ...) 2021-06-01 7:56 ` Konrad Hinsen @ 2021-06-08 13:11 ` Ludovic Courtès 2021-06-14 14:29 ` Pjotr Prins 3 siblings, 1 reply; 21+ messages in thread From: Ludovic Courtès @ 2021-06-08 13:11 UTC (permalink / raw) To: Pjotr Prins; +Cc: guix-devel Hi! (I’m late to the party…) Pjotr Prins <pjotr.public12@thebird.nl> skribis: > As Guix takes care of the first and the last - the issue centers > around building. The idea is to dress down these language specific > builders, such as cargo, so you don't have all the included > complexity. > > A sexp-pack would represent the most simple build instructions to > build a package on its own. Now, of course the current guix-builders > solve that too. But, what I am proposing is to split out the actual > build step into a package definition, so as to present something > simpler to Guix. > > I found a cargo -> ninja converter. It is that kind of idea. Guix > would use ninja with rustc instead of cargo. A stripped down cargo > could potentially work too - but cargo is a complex beast. In a way, aren’t importers addressing this use case? In the end, they let everyone develop a package manager/build system that suits their needs, while still allowing us to have our cake. WDYT? Ludo’. ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Idea: a meta language for (language) build systems - npm, Racket, Rust cargo 2021-06-08 13:11 ` Ludovic Courtès @ 2021-06-14 14:29 ` Pjotr Prins 2021-06-14 22:04 ` Leo Prikler 2021-06-20 15:46 ` Ludovic Courtès 0 siblings, 2 replies; 21+ messages in thread From: Pjotr Prins @ 2021-06-14 14:29 UTC (permalink / raw) To: Ludovic Courtès; +Cc: guix-devel On Tue, Jun 08, 2021 at 03:11:31PM +0200, Ludovic Courtès wrote: > (I’m late to the party…) Never late :) > > I found a cargo -> ninja converter. It is that kind of idea. Guix > > would use ninja with rustc instead of cargo. A stripped down cargo > > could potentially work too - but cargo is a complex beast. > > In a way, aren’t importers addressing this use case? Importers and build-systems, even guix.scm, don't really create a rapid and dynamic builder for developers to use when developing software. At the same time the builders, such as cargo, are heavy, bloated and misguided. And hard to embed in GNU Guix for multiple reasons. There is a missing link in my opinion. Though it is not a Guix problem (I agree). > In the end, they let everyone develop a package manager/build system > that suits their needs, while still allowing us to have our cake. If you strip out dependency management and cross-platform compiling there is not much left to do for cmake, autotools, cargo etc. Everything Guix is good at. Ninja resolves the build graph, you only need something to generate the build graph for source files. A mini-Guix of sorts for source files. Eelco, at the time, wrote Maak - a builder that would work on NixOS. He wrote it in Haskell, but it is similar thinking. https://en.wikipedia.org/wiki/Maak Ninja fits the bill - but it is something that needs to be generated. Minimalism and simplicity is what I want. I hate it when software starts thinking for me ;). https://github.com/rust-lang/cargo/issues/6584 We need to create another builder that replaces cargo - or maybe just simplifies cargo. I am not programming in Rust or Racket, right now, but when I do (again), I'll try to generate ninja files to define a build as a starting point in a simple way. One thing we can do with Guix is create a parallel build system. An alt-verse for builders ;) Pj. ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Idea: a meta language for (language) build systems - npm, Racket, Rust cargo 2021-06-14 14:29 ` Pjotr Prins @ 2021-06-14 22:04 ` Leo Prikler 2021-06-15 8:48 ` Pjotr Prins 2021-06-15 10:08 ` Ricardo Wurmus 2021-06-20 15:46 ` Ludovic Courtès 1 sibling, 2 replies; 21+ messages in thread From: Leo Prikler @ 2021-06-14 22:04 UTC (permalink / raw) To: Pjotr Prins, Ludovic Courtès; +Cc: guix-devel Am Montag, den 14.06.2021, 16:29 +0200 schrieb Pjotr Prins: > On Tue, Jun 08, 2021 at 03:11:31PM +0200, Ludovic Courtès wrote: > > (I’m late to the party…) > > Never late :) > > > > I found a cargo -> ninja converter. It is that kind of idea. Guix > > > would use ninja with rustc instead of cargo. A stripped down > > > cargo > > > could potentially work too - but cargo is a complex beast. > > > > In a way, aren’t importers addressing this use case? > > Importers and build-systems, even guix.scm, don't really create a > rapid and dynamic builder for developers to use when developing > software. At the same time the builders, such as cargo, are heavy, > bloated and misguided. And hard to embed in GNU Guix for multiple > reasons. > > There is a missing link in my opinion. Though it is not a > Guix problem (I agree). This is really something that's true rather for cargo than for most other builders, though. gnu-build-system, meson-build-system etc. are – in my opinion – appropriately sized and a hypothetical cargo-ninja- build-system could just reuse most of meson-build-system but creating the ninja files from cargo rather than meson (i.e. swapping the configure phase). As for "rapid development", `guix environment` provides the things you need in an environment (heh), that is simple to set up. The missing link when that is already given is to set up the build system that you'd use anyway. I think most modern tools have a --generate-me-some- generic-hello-world-template, whereas for the GNU Build System you simply copy amhello. > > In the end, they let everyone develop a package manager/build > > system > > that suits their needs, while still allowing us to have our cake. > > If you strip out dependency management and cross-platform compiling > there is not much left to do for cmake, autotools, cargo etc. > Everything Guix is good at. Ninja resolves the build graph, you only > need something to generate the build graph for source files. A > mini-Guix of sorts for source files. Eelco, at the time, wrote Maak - > a builder that would work on NixOS. He wrote it in Haskell, but it is > similar thinking. There never was that much to do for a build system. All they're supposed to do is to simplify a sequence of compiler and linker calls, so that you can write `make all' instead of having to type out 500 invocations of GCC. They were never meant to be package managers and those who think they are will be doomed to reinvent the wheel over and over again. They instead provide an interface to package managers, that interface being e.g. the locations in which they will generally look for certain headers (though reasonable build systems and package managers use pkg-config to solve this issue as well). > https://en.wikipedia.org/wiki/Maak > > Ninja fits the bill - but it is something that needs to be generated. > > Minimalism and simplicity is what I want. I hate it when software > starts thinking for me ;). > > https://github.com/rust-lang/cargo/issues/6584 > > We need to create another builder that replaces cargo - or maybe just > simplifies cargo. I am not programming in Rust or Racket, right now, > but when I do (again), I'll try to generate ninja files to define a > build as a starting point in a simple way. One thing we can do with > Guix is create a parallel build system. An alt-verse for builders ;) How build systems proliferate Situation: There are 14 competing build systems Developer: 14?! Ridiculous! We need to develop one universal standard, that covers everyone's use cases. Developer with a ponytail: Yeah! Soon: There are 15 competing build systems Guix as a build system, with or without GWL, is fine as long as it's just fun and games, but it comes with the serious downside of not being compatible with any other distro out there. Perhaps that works out for those, who pray towards our lord and saviour Flatpak, but if you want to be able to ship to traditional distros through normal channels, you will need a reasonable build system like make or ninja and perhaps something that generates you the nitty gritty stuff like Autotools, Meson, etc. What we need in order to make packages written in Rust less of a PITA is more widespread usage of generic build systems like make or meson. We won't get there with cargo cult programming. On a somewhat related note, I find it amusing, that Rust folk dread the usage of Rust without cargo, as if the Rust standard library on its own was somehow worthless or something :P Regards, Leo ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Idea: a meta language for (language) build systems - npm, Racket, Rust cargo 2021-06-14 22:04 ` Leo Prikler @ 2021-06-15 8:48 ` Pjotr Prins 2021-06-15 10:08 ` Ricardo Wurmus 1 sibling, 0 replies; 21+ messages in thread From: Pjotr Prins @ 2021-06-15 8:48 UTC (permalink / raw) To: Leo Prikler; +Cc: guix-devel On Tue, Jun 15, 2021 at 12:04:59AM +0200, Leo Prikler wrote: > Soon: There are 15 competing build systems Very true ;) > Guix as a build system, with or without GWL, is fine as long as it's > just fun and games, but it comes with the serious downside of not being > compatible with any other distro out there. Perhaps that works out for Maybe. I can still support cargo - just don't want to deal with it... A Debian target would not be too hard. > What we need in order to make packages written in Rust less of a PITA > is more widespread usage of generic build systems like make or meson. > We won't get there with cargo cult programming. On a somewhat related > note, I find it amusing, that Rust folk dread the usage of Rust without > cargo, as if the Rust standard library on its own was somehow worthless > or something :P Meson may be good enough (indeed). I have one project that uses meson effectively and it is not bad. But I had to look up a lot of things - it is sometimes counterintuitive and hardly simple. Sure beats cmake and autotools. Still, it could be simpler. Pj. ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Idea: a meta language for (language) build systems - npm, Racket, Rust cargo 2021-06-14 22:04 ` Leo Prikler 2021-06-15 8:48 ` Pjotr Prins @ 2021-06-15 10:08 ` Ricardo Wurmus 1 sibling, 0 replies; 21+ messages in thread From: Ricardo Wurmus @ 2021-06-15 10:08 UTC (permalink / raw) To: Leo Prikler; +Cc: guix-devel Leo Prikler <leo.prikler@student.tugraz.at> writes: > Guix as a build system, with or without GWL, is fine as long as > it's > just fun and games […] I keep seeing GWL in unexpected places, so I feel the need to state that the GWL is not meant to help anyone *build* software. It’s meant to let you *combine* applications and *run* them on an HPC cluster (or on a bunch of physical or virtual machines) to process raw data. The intended use case is scientific workflows. Don’t try to use the GWL as a build system; it will end in tears. -- Ricardo ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Idea: a meta language for (language) build systems - npm, Racket, Rust cargo 2021-06-14 14:29 ` Pjotr Prins 2021-06-14 22:04 ` Leo Prikler @ 2021-06-20 15:46 ` Ludovic Courtès 1 sibling, 0 replies; 21+ messages in thread From: Ludovic Courtès @ 2021-06-20 15:46 UTC (permalink / raw) To: Pjotr Prins; +Cc: guix-devel Hi! Pjotr Prins <pjotr.public12@thebird.nl> skribis: > If you strip out dependency management and cross-platform compiling > there is not much left to do for cmake, autotools, cargo etc. > Everything Guix is good at. Ninja resolves the build graph, you only > need something to generate the build graph for source files. A > mini-Guix of sorts for source files. Eelco, at the time, wrote Maak - > a builder that would work on NixOS. He wrote it in Haskell, but it is > similar thinking. > > https://en.wikipedia.org/wiki/Maak It is an inspiration for hacks like <https://guix.gnu.org/en/blog/2021/reproducible-data-processing-pipelines/>. This is the kind of thing you’d typically do using Make, but where using Guix is advantageous. Now, as for Maak, it needs a library of helpers to be readily usable—“rules” to build .o from .c, libraries, etc. Is that the kind of thing you have in mind? Then again, once you have that, you’re still very far from replacing or integrating with tools like Cargo. I guess it’s all about finding the right spot where things can be usefully glued together! Ludo’. ^ permalink raw reply [flat|nested] 21+ messages in thread
end of thread, other threads:[~2021-06-20 15:46 UTC | newest] Thread overview: 21+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-05-30 8:38 Idea: a meta language for (language) build systems - npm, Racket, Rust cargo Pjotr Prins 2021-05-30 19:17 ` Konrad Hinsen 2021-05-31 17:47 ` Pjotr Prins 2021-06-01 5:50 ` Adriano Peluso 2021-06-01 6:24 ` Leo Prikler 2021-06-01 7:23 ` Pjotr Prins 2021-06-01 8:12 ` Leo Prikler 2021-06-01 8:59 ` Adriano Peluso 2021-06-01 9:11 ` Leo Prikler 2021-06-01 10:52 ` Adriano Peluso 2021-06-01 11:03 ` Leo Prikler 2021-06-01 11:28 ` Adriano Peluso 2021-06-01 15:22 ` Leo Prikler 2021-06-01 7:56 ` Konrad Hinsen 2021-06-01 23:35 ` Joshua Branson 2021-06-08 13:11 ` Ludovic Courtès 2021-06-14 14:29 ` Pjotr Prins 2021-06-14 22:04 ` Leo Prikler 2021-06-15 8:48 ` Pjotr Prins 2021-06-15 10:08 ` Ricardo Wurmus 2021-06-20 15:46 ` Ludovic Courtès
Code repositories for project(s) associated with this public inbox https://git.savannah.gnu.org/cgit/guix.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).