* Tiny Guix (and containers) @ 2017-10-25 8:18 Pjotr Prins 2017-10-25 11:12 ` Pjotr Prins ` (2 more replies) 0 siblings, 3 replies; 17+ messages in thread From: Pjotr Prins @ 2017-10-25 8:18 UTC (permalink / raw) To: guix-devel I have been playing around with containers to create something for a book: https://gitlab.com/pjotrp/guix-notes/blob/master/CONTAINERS.org these are (Docker) images that can be run by anyone - which is great! Only (minor) issue is the containers are 1.4GB in size. These are the biggies in this image: wrk@lario /gnu/store [env]# du -sh *|grep -e "[[:digit:]][[:digit:]]M" 36M 0s5manjvfa0gmsv2r71rchky7ab70g1d-icu4c-58.2 36M 15plffwjdv8ii6vbsnvrvy6irbfsz4x4-gfortran-5.4.0-lib 22M 3x53yv4v144c9xp02rs64z7j597kkqax-gcc-5.4.0-lib 16M 42d5rjrdkln6nwvzwdc8dyd4w6iy3n5j-coreutils-8.27 13M 5c9qygcdircjg1dkaw6rnw59c6v4akca-python-cython-0.27 90M 5sv5zy2kgg6iaqyv8zw49w4243j0xkd0-gcc-5.4.0 19M 5vjvxngriskrsrg5lv162i927crciy9x-bioperl-minimal-1.7.0 40M 9c5ip78j3cr6ywx9vjicn545pmq24802-book-evolutionary-genomics-0.2.1-53a7aef 92M 9shcsn90rpzacysklhfiidc9qkcn3f1l-ruby-2.4.2 16M azbfh3i72lbaqvhgg5m7p6ymmqq0ii6q-glib-2.52.3 59M bzn4wyrbdmfc1bd7lq05db5psfl5f8x8-perl-5.26.0 28M d4hwf3w2ca9lmf7nrgk2jsr9mhxcmb40-vim-8.0.1207 15M d5khpb89l7w65qvwry6ikx57sknnnl9p-python-biopython-1.70 46M d8gkn84yqacjr80pzicz1ka3y2s1f2x0-guile-2.2.2 63M f7j1vq0ncc1znrn0cyy07gps3r0h1iaa-openblas-0.2.19 10M fqxyqlpkbm4fc9x91yyqs7zbhm495b2s-ruby-ffi-1.9.18 65M ks27x0mf95gir0cdgb9h573xbava6v1k-python-3.5.3 11M kwzs8k97qy7avxxldnzavlii9zphh3d6-libxml2-2.9.4 13M l95wcgysigi5mkmzgrr9k0irrszzd2m8-profile 41M n6nvxlk2j8ysffjh3jphn1k5silnakh6-glibc-2.25 34M nnykzgwfy8mwh2gmxm715sjxykg8qjwn-binutils-2.28 11M nyd19c9ccykji2lg9jhxfzv6zd67ar55-lapack-3.7.1 423M p4h18j5dnn7sgajni9pfas0jm4dgdnv6-emboss-6.5.7 22M pc96dryzfghnih9lqrjfa24n205ds7ir-python-numpy-1.12.0 31M sq8yhawx0x3pyvvqrxx8hmw4rnp4mz03-bioruby-1.5.1 45M y8g28w9icanll84llw6xvdyjrw0qvnp9-r-minimal-3.4.2 15M zb662xs7i06l079n2wr5mn56862v6wfm-r-biostrings-2.44.2 14M zbywrj6klakskj0sppq56viqh9l56jl0-util-linux-2.30.1 It is obvious emboss could be trimmed down ;). But I think none of these packages when trimmed down ought to be larger than 15M. Not even gcc. Separately I have been using Travis-Ci for (integration) testing of a C++ project (soon to adding D) which we are working on https://travis-ci.org/genetics-statistics/GEMMA Now it takes forever to set up the image - a tiny Guix Docker container would be great here and probably get attention (most Travis testing time goes up in downloading stuff!). To users it makes a huge difference if they have to wait 10 minutes or 10 seconds. Then there is my idea of creating (mail) servers using Guix. It would be great to go tiny there too. And in the embedded world they may consider Guix when it is tiny. My question is, who else would be interested and how to best to go about this in Guix. It makes no sense to write separate packages (I think) because the maintenance effort would go up hugely. But tiny is also not exactly a subset of existing packages. I mean guix package -i glibc:tiny could be a target. Splitting packages in bioruby:doc, bioruby:dev and bioruby:lib would serve that purpose too. We know Debian struggled with this and we also know that OpenWRT/LEDE went the NIH way. What lessons are there? But, really, I think when talking embedded systems and containers we all want tiny. Even HPC can benefit. Tiny containers may be an attractive proposition. Pj. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Tiny Guix (and containers) 2017-10-25 8:18 Tiny Guix (and containers) Pjotr Prins @ 2017-10-25 11:12 ` Pjotr Prins 2017-10-26 7:02 ` Ricardo Wurmus 2017-10-31 14:11 ` Dave Love 2 siblings, 0 replies; 17+ messages in thread From: Pjotr Prins @ 2017-10-25 11:12 UTC (permalink / raw) To: Pjotr Prins; +Cc: guix-devel Another route is to create a container using just the essential dependencies, like I did here: https://gitlab.com/pjotrp/guix-relocatable-binary-packages/blob/master/README.org Binary with dependencies distribution down to 18Mb. Of course locales won't work ;) And yes, I am still using relocatable packages to distribute specific software for HPC... It is rather nice. Pj. -- ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Tiny Guix (and containers) 2017-10-25 8:18 Tiny Guix (and containers) Pjotr Prins 2017-10-25 11:12 ` Pjotr Prins @ 2017-10-26 7:02 ` Ricardo Wurmus 2017-10-26 10:42 ` Pjotr Prins 2017-10-27 0:35 ` Ludovic Courtès 2017-10-31 14:11 ` Dave Love 2 siblings, 2 replies; 17+ messages in thread From: Ricardo Wurmus @ 2017-10-26 7:02 UTC (permalink / raw) To: Pjotr Prins; +Cc: guix-devel Pjotr Prins <pjotr.public12@thebird.nl> writes: > 22M 3x53yv4v144c9xp02rs64z7j597kkqax-gcc-5.4.0-lib According to “du”, this is 32M on my disk. The “lib” subdir contains both shared libraries as well as ar archives for static linking; together they weigh in at 12MB. We may want to move them to a separate output. The package also contains lots of header files: 6.3M /gnu/store/3x53yv4v144c9xp02rs64z7j597kkqax-gcc-5.4.0-lib/lib/gcc/x86_64-unknown-linux-gnu/5.4.0/plugin/include/ Not sure what to do with those without making the use of GCC a hassle. > 41M n6nvxlk2j8ysffjh3jphn1k5silnakh6-glibc-2.25 This package still contains a lot of locale data. The directory “share/i18n/locales/” takes up 6.7M, and “share/locale” takes up another 4.3M. All the .a files under “lib” take up 8.7M. > 34M nnykzgwfy8mwh2gmxm715sjxykg8qjwn-binutils-2.28 “share/locale” is 9.4M. This is a cross-cutting concern. We don’t have a way to globally filter locales to only requested locales. Even if we split them each into a separate output — how would you specify that you want the “de_DE” locale in each package and not install the rest? There seems to be some duplication with these directories: /gnu/store/nnykzgwfy8mwh2gmxm715sjxykg8qjwn-binutils-2.28/x86_64-unknown-linux-gnu/bin/ /gnu/store/nnykzgwfy8mwh2gmxm715sjxykg8qjwn-binutils-2.28/bin/ But the binaries seem to be hardlinked, so they don’t take up extra space. > Now it takes forever to set up the image Have you tried disabling compression? This could be a lot faster. I found that tar with gzip compression is terribly slow to copy things from the store into a compressed tar archive. Disabling compression speeds this up considerably, even though it is still rather slow. -- Ricardo GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC https://elephly.net ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Tiny Guix (and containers) 2017-10-26 7:02 ` Ricardo Wurmus @ 2017-10-26 10:42 ` Pjotr Prins 2017-10-26 10:48 ` Ricardo Wurmus 2017-10-27 8:25 ` Hartmut Goebel 2017-10-27 0:35 ` Ludovic Courtès 1 sibling, 2 replies; 17+ messages in thread From: Pjotr Prins @ 2017-10-26 10:42 UTC (permalink / raw) To: Ricardo Wurmus; +Cc: guix-devel Hi Ricardo, Thanks for responding. I think this discussion belongs here. On Thu, Oct 26, 2017 at 09:02:56AM +0200, Ricardo Wurmus wrote: > > Pjotr Prins <pjotr.public12@thebird.nl> writes: > > > 22M 3x53yv4v144c9xp02rs64z7j597kkqax-gcc-5.4.0-lib > > According to “du”, this is 32M on my disk. The “lib” subdir contains > both shared libraries as well as ar archives for static linking; > together they weigh in at 12MB. We may want to move them to a separate > output. Yes, I think that is what we should head for eventually. I vaguely remember a discussion about this on this ML and people were against separate outputs for doc, include, static-lib etc. What are you all thinking now? Does it make sense to have the base package as small as possible and split out the rest? I know it is extra work. > The package also contains lots of header files: > > 6.3M /gnu/store/3x53yv4v144c9xp02rs64z7j597kkqax-gcc-5.4.0-lib/lib/gcc/x86_64-unknown-linux-gnu/5.4.0/plugin/include/ > > Not sure what to do with those without making the use of GCC a hassle. Yes, that goes with gcc. No point keeping those separate. > > 41M n6nvxlk2j8ysffjh3jphn1k5silnakh6-glibc-2.25 > > This package still contains a lot of locale data. The directory > “share/i18n/locales/” takes up 6.7M, and “share/locale” takes up another > 4.3M. All the .a files under “lib” take up 8.7M. > > > 34M nnykzgwfy8mwh2gmxm715sjxykg8qjwn-binutils-2.28 > > “share/locale” is 9.4M. This is a cross-cutting concern. We don’t have > a way to globally filter locales to only requested locales. Even if we > split them each into a separate output — how would you specify that you > want the “de_DE” locale in each package and not install the rest? Yes, it is a specific target discussion. Same goes for compilation without debug information and optimizing for size. Tiny = tiny. Ludo wrote at some point that target optimizations are on the roadmap. It would be rather good to have. Maybe in the form of channels. > There seems to be some duplication with these directories: > > /gnu/store/nnykzgwfy8mwh2gmxm715sjxykg8qjwn-binutils-2.28/x86_64-unknown-linux-gnu/bin/ > /gnu/store/nnykzgwfy8mwh2gmxm715sjxykg8qjwn-binutils-2.28/bin/ > > But the binaries seem to be hardlinked, so they don’t take up extra > space. > > > Now it takes forever to set up the image > > Have you tried disabling compression? This could be a lot faster. I > found that tar with gzip compression is terribly slow to copy things > from the store into a compressed tar archive. Disabling compression > speeds this up considerably, even though it is still rather slow. That will help :). But I was talking about Docker images. Making them small would be helpful for a lot of people out there. It takes forever to download and run Docker tests, for example, I hear people complain. Pj. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Tiny Guix (and containers) 2017-10-26 10:42 ` Pjotr Prins @ 2017-10-26 10:48 ` Ricardo Wurmus 2017-10-27 8:25 ` Hartmut Goebel 1 sibling, 0 replies; 17+ messages in thread From: Ricardo Wurmus @ 2017-10-26 10:48 UTC (permalink / raw) To: Pjotr Prins; +Cc: guix-devel Pjotr Prins <pjotr.public12@thebird.nl> writes: >> > Now it takes forever to set up the image >> >> Have you tried disabling compression? This could be a lot faster. I >> found that tar with gzip compression is terribly slow to copy things >> from the store into a compressed tar archive. Disabling compression >> speeds this up considerably, even though it is still rather slow. > > That will help :). But I was talking about Docker images. Making them > small would be helpful for a lot of people out there. It takes forever > to download and run Docker tests, for example, I hear people complain. I know but there are two use cases here, no? One is to build an image repeatedly for Travis CI, the other is a “release” image. For the CI image one could disable compression to speed this up. But clearly, that’s a tradeoff between the time it takes to download the uncompressed blob and building a compressed blob. -- Ricardo GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC https://elephly.net ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Tiny Guix (and containers) 2017-10-26 10:42 ` Pjotr Prins 2017-10-26 10:48 ` Ricardo Wurmus @ 2017-10-27 8:25 ` Hartmut Goebel 2017-10-28 20:06 ` Ludovic Courtès 1 sibling, 1 reply; 17+ messages in thread From: Hartmut Goebel @ 2017-10-27 8:25 UTC (permalink / raw) To: guix-devel Hello, Am 26.10.2017 um 12:42 schrieb Pjotr Prins: > Yes, I think that is what we should head for eventually. I vaguely > remember a discussion about this on this ML and people were against > separate outputs for doc, include, static-lib etc. What are you all > thinking now? Does it make sense to have the base package as small as > possible and split out the rest? I'm in favor of (automatically?) splitting of "development" packages, including the headers and the static libs (much like the "-devel" or "-dev" packages in other distributions. One does not need them on a production system and they are just wasting space. When Guix needs to build a package, it automatically installs the ":devel" output of all it's inputs. We could do the same for "docs", but "docs" may not be easy to determine, except for man-pages and info-files. Regarding localization-files I'm unsure if for the average package this is worth the effort. But for big packages this could be worth the effort. Maybe we could even make them "noarch" packages, thus savine space and build time. Here is a list of "langpacks" my distribution offers (I used pt_BR since this is easy to search in the package-repo): aspell-pt_BR childsplay-sounds-pt_BR firefox-pt_BR gcompris-sounds-pt_BR gnome-getting-started-docs-pt_BR kde-l10n-handbooks-pt_BR kde-l10n-pt_BR kompozer-myspell-pt_BR kompozer-pt_BR libreoffice-langpack-pt_BR man-pages-pt_BR thunderbird-pt_BR -- Regards Hartmut Goebel | Hartmut Goebel | h.goebel@crazy-compilers.com | | www.crazy-compilers.com | compilers which you thought are impossible | ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Tiny Guix (and containers) 2017-10-27 8:25 ` Hartmut Goebel @ 2017-10-28 20:06 ` Ludovic Courtès 2017-10-31 14:17 ` Dave Love 0 siblings, 1 reply; 17+ messages in thread From: Ludovic Courtès @ 2017-10-28 20:06 UTC (permalink / raw) To: Hartmut Goebel; +Cc: guix-devel Hi, Hartmut Goebel <h.goebel@crazy-compilers.com> skribis: > Am 26.10.2017 um 12:42 schrieb Pjotr Prins: >> Yes, I think that is what we should head for eventually. I vaguely >> remember a discussion about this on this ML and people were against >> separate outputs for doc, include, static-lib etc. What are you all >> thinking now? Does it make sense to have the base package as small as >> possible and split out the rest? > > I'm in favor of (automatically?) splitting of "development" packages, > including the headers and the static libs (much like the "-devel" or > "-dev" packages in other distributions. One does not need them on a > production system and they are just wasting space. When Guix needs to > build a package, it automatically installs the ":devel" output of all > it's inputs. We could do that (the Nixpkgs folks did exactly that a while back), but it won’t help that much. What does help is running ‘guix size’, looking at specific packages that are problematic, then finding a solution for these packages—and often enough the solution is non-trivial. But yeah, I think we should track packages that are big or have a surprisingly build closure, and try to fix that incrementally! > Regarding localization-files I'm unsure if for the average package this > is worth the effort. But for big packages this could be worth the > effort. Maybe we could even make them "noarch" packages, thus savine > space and build time. For things like Binutils, libc, or LO, it surely makes a difference. For an FHS distro it’s easy to keep locale data separate. In our case, I’m not sure how to do it, as discussed earlier. Thanks, Ludo’. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Tiny Guix (and containers) 2017-10-28 20:06 ` Ludovic Courtès @ 2017-10-31 14:17 ` Dave Love 2017-11-05 16:02 ` Ludovic Courtès 0 siblings, 1 reply; 17+ messages in thread From: Dave Love @ 2017-10-31 14:17 UTC (permalink / raw) To: Ludovic Courtès; +Cc: guix-devel ludo@gnu.org (Ludovic Courtès) writes: > Hi, > > Hartmut Goebel <h.goebel@crazy-compilers.com> skribis: >> I'm in favor of (automatically?) splitting of "development" packages, >> including the headers and the static libs (much like the "-devel" or >> "-dev" packages in other distributions. One does not need them on a >> production system and they are just wasting space. When Guix needs to >> build a package, it automatically installs the ":devel" output of all >> it's inputs. I've been arguing for making sub-packages similar to other distributions, but I'd expect to specify something like :devel as the input to builds. I'm not sure that it would even work generally to infer it (e.g. when cross-compiling). > We could do that (the Nixpkgs folks did exactly that a while back), but > it won’t help that much. It looks to me as if it would often help significantly, e.g. when a pkg-config file, or something else sucks in a load of stuff that's irrelevant for running the package. (Separating :lib and needing that for building means you need to know something about the packaging rather than just using "devel", say.) > What does help is running ‘guix size’, looking > at specific packages that are problematic, then finding a solution for > these packages—and often enough the solution is non-trivial. I think it often will reflect the lack of separation of development files, documentation, etc., and inclusion of static libraries, for instance. Boost is a case in point. There's also the issue of multiple copies/versions of packages in the closure, which seems problematic for more than just size reasons. > But yeah, I think we should track packages that are big or have a > surprisingly build closure, and try to fix that incrementally! Shouldn't that be done as a matter of course? I don't remember if it's part of Fedora or Debian packaging guidelines but packagers should check dependencies for sanity when packaging, and there's some detection of unnecessary linkage. Guix' Qt dependencies are a striking example which looks hard to resolve. Generally I get surprised at what typically gets built -- at great length! -- on updates or installation. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Tiny Guix (and containers) 2017-10-31 14:17 ` Dave Love @ 2017-11-05 16:02 ` Ludovic Courtès 2017-11-06 15:45 ` Dave Love 0 siblings, 1 reply; 17+ messages in thread From: Ludovic Courtès @ 2017-11-05 16:02 UTC (permalink / raw) To: Dave Love; +Cc: guix-devel Heya, Dave Love <fx@gnu.org> skribis: > ludo@gnu.org (Ludovic Courtès) writes: > >> Hi, >> >> Hartmut Goebel <h.goebel@crazy-compilers.com> skribis: > >>> I'm in favor of (automatically?) splitting of "development" packages, >>> including the headers and the static libs (much like the "-devel" or >>> "-dev" packages in other distributions. One does not need them on a >>> production system and they are just wasting space. When Guix needs to >>> build a package, it automatically installs the ":devel" output of all >>> it's inputs. > > I've been arguing for making sub-packages similar to other > distributions, but I'd expect to specify something like :devel as the > input to builds. I'm not sure that it would even work generally to > infer it (e.g. when cross-compiling). OK. >> We could do that (the Nixpkgs folks did exactly that a while back), but >> it won’t help that much. > > It looks to me as if it would often help significantly, e.g. when a > pkg-config file, or something else sucks in a load of stuff that's > irrelevant for running the package. (Separating :lib and needing that > for building means you need to know something about the packaging rather > than just using "devel", say.) Right, good point. The nice thing with “lib” and “doc” is that it has a direct mapping to the GNU directory classification (libdir, docdir, etc.) Now, we could depart from it and go with “devel”, for the reasons you give. Let’s experiment and see how it goes! >> What does help is running ‘guix size’, looking >> at specific packages that are problematic, then finding a solution for >> these packages—and often enough the solution is non-trivial. > > I think it often will reflect the lack of separation of development > files, documentation, etc., and inclusion of static libraries, for > instance. Boost is a case in point. There's also the issue of multiple > copies/versions of packages in the closure, which seems problematic for > more than just size reasons. Boost is also a special case, being a mostly-header library. :-) But yeah, I get your point. >> But yeah, I think we should track packages that are big or have a >> surprisingly build closure, and try to fix that incrementally! > > Shouldn't that be done as a matter of course? I don't remember if it's > part of Fedora or Debian packaging guidelines but packagers should check > dependencies for sanity when packaging, and there's some detection of > unnecessary linkage. Guix' Qt dependencies are a striking example which > looks hard to resolve. Generally I get surprised at what typically gets > built -- at great length! -- on updates or installation. No argument here! The guidelines at <https://www.gnu.org/software/guix/manual/html_node/Submitting-Patches.html> also mention the closure size. We can clearly do better though, and what you’ve been doing with Open MPI et al. is a step in the right direction IMO. I would very much welcome experiments introducing “devel” outputs. Perhaps as a first step we could make it opt-in. We would add support for that in gnu-build-system.scm, and then modify key packages to use “devel” instead of “lib”, for example. Incremental changes like this can be tractable. Thoughts? Ludo’. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Tiny Guix (and containers) 2017-11-05 16:02 ` Ludovic Courtès @ 2017-11-06 15:45 ` Dave Love 2017-11-07 10:19 ` Ludovic Courtès 0 siblings, 1 reply; 17+ messages in thread From: Dave Love @ 2017-11-06 15:45 UTC (permalink / raw) To: Ludovic Courtès; +Cc: guix-devel Ludovic Courtès <ludo@gnu.org> writes: >> It looks to me as if it would often help significantly, e.g. when a >> pkg-config file, or something else sucks in a load of stuff that's >> irrelevant for running the package. (Separating :lib and needing that >> for building means you need to know something about the packaging rather >> than just using "devel", say.) > > Right, good point. > > The nice thing with “lib” and “doc” is that it has a direct mapping to > the GNU directory classification (libdir, docdir, etc.) Sure, though there's typically a distinction between lib and, say, lib64, in other distributions, where lib has other than linkable libraries (e.g. in Fedora, openmpi is mostly under the prefix /usr/lib64/openmpi). > Now, we could depart from it and go with “devel”, for the reasons you > give. Let’s experiment and see how it goes! Good to hear as an experimentalist! I wonder how much practical experience people have with conventional packaging and the resulting trades-off, e.g. as Debian, Fedora, etc. maintainers. I think it helps to understand that reasonably well. I'm happy to explain to the extent I can if it helps. I'm more familiar with Fedora, but then Debian is usually easier. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Tiny Guix (and containers) 2017-11-06 15:45 ` Dave Love @ 2017-11-07 10:19 ` Ludovic Courtès 0 siblings, 0 replies; 17+ messages in thread From: Ludovic Courtès @ 2017-11-07 10:19 UTC (permalink / raw) To: Dave Love; +Cc: guix-devel Dave Love <fx@gnu.org> skribis: > Ludovic Courtès <ludo@gnu.org> writes: > >>> It looks to me as if it would often help significantly, e.g. when a >>> pkg-config file, or something else sucks in a load of stuff that's >>> irrelevant for running the package. (Separating :lib and needing that >>> for building means you need to know something about the packaging rather >>> than just using "devel", say.) >> >> Right, good point. >> >> The nice thing with “lib” and “doc” is that it has a direct mapping to >> the GNU directory classification (libdir, docdir, etc.) > > Sure, though there's typically a distinction between lib and, say, > lib64, I’m talking about the classification, not about specific choices like lib vs. lib64. >> Now, we could depart from it and go with “devel”, for the reasons you >> give. Let’s experiment and see how it goes! > > Good to hear as an experimentalist! :-) > I wonder how much practical experience people have with conventional > packaging and the resulting trades-off, e.g. as Debian, Fedora, > etc. maintainers. I think it helps to understand that reasonably well. > I'm happy to explain to the extent I can if it helps. I'm more familiar > with Fedora, but then Debian is usually easier. I think your expertise is most welcome here. Not everything will have a direct mapping to Guix, but surely we can build upon the experience of other distros. For information on what Nixpkgs does with some of its packages, see also: https://nixos.org/nixpkgs/manual/#chap-multiple-output AFAIK this remains an opt-in mechanism, pretty much like in Guix. Their early experience can be read here: https://nixos.org/nix-dev/2016-April/020154.html Ludo’. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Tiny Guix (and containers) 2017-10-26 7:02 ` Ricardo Wurmus 2017-10-26 10:42 ` Pjotr Prins @ 2017-10-27 0:35 ` Ludovic Courtès 1 sibling, 0 replies; 17+ messages in thread From: Ludovic Courtès @ 2017-10-27 0:35 UTC (permalink / raw) To: Ricardo Wurmus; +Cc: guix-devel Ricardo Wurmus <rekado@elephly.net> skribis: >> 34M nnykzgwfy8mwh2gmxm715sjxykg8qjwn-binutils-2.28 > > “share/locale” is 9.4M. This is a cross-cutting concern. We don’t have > a way to globally filter locales to only requested locales. Even if we > split them each into a separate output — how would you specify that you > want the “de_DE” locale in each package and not install the rest? There’s also the problem that packages typically hardcode $localedir in binaries, which means we would need a special mechanism to decouple locale data from packages. > There seems to be some duplication with these directories: > > /gnu/store/nnykzgwfy8mwh2gmxm715sjxykg8qjwn-binutils-2.28/x86_64-unknown-linux-gnu/bin/ > /gnu/store/nnykzgwfy8mwh2gmxm715sjxykg8qjwn-binutils-2.28/bin/ > > But the binaries seem to be hardlinked, so they don’t take up extra > space. But they do take up extra space when transmitted over the wire as nars, and in Docker/tarball packs. In general we should replace hard links with symlinks because of that (Mesa has the same problem.) >> Now it takes forever to set up the image > > Have you tried disabling compression? This could be a lot faster. I > found that tar with gzip compression is terribly slow to copy things > from the store into a compressed tar archive. Disabling compression > speeds this up considerably, even though it is still rather slow. In general one wants compression though, no? I’d be curious to see how we can make it faster! Ludo’. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Tiny Guix (and containers) 2017-10-25 8:18 Tiny Guix (and containers) Pjotr Prins 2017-10-25 11:12 ` Pjotr Prins 2017-10-26 7:02 ` Ricardo Wurmus @ 2017-10-31 14:11 ` Dave Love 2017-11-03 12:08 ` Pjotr Prins 2017-11-05 15:55 ` Ludovic Courtès 2 siblings, 2 replies; 17+ messages in thread From: Dave Love @ 2017-10-31 14:11 UTC (permalink / raw) To: Pjotr Prins; +Cc: guix-devel Pjotr Prins <pjotr.public12@thebird.nl> writes: > But, really, I think when talking embedded systems and containers we > all want tiny. Even HPC can benefit. Tiny containers may be an > attractive proposition. Yes, containers aside, dependencies in Guix are one of the reasons I'm currently unconvinced by its trades-off for HPC use; the closure a single package can be comparable with the whole compute node image I used to maintain with rpm. Even then, you don't generally have debugging info available. I suppose the general point is that Guix seems to have rejected experience from other distributions, and it's not clear to me why. (I don't mean it should necessarily follow them, of course.) Is there any summary of the rationale for different decisions like not splitting packages into development and runtime and other components, packaging static libraries, and discarding debugging information, for instance? ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Tiny Guix (and containers) 2017-10-31 14:11 ` Dave Love @ 2017-11-03 12:08 ` Pjotr Prins 2017-11-06 15:10 ` Dave Love 2017-11-05 15:55 ` Ludovic Courtès 1 sibling, 1 reply; 17+ messages in thread From: Pjotr Prins @ 2017-11-03 12:08 UTC (permalink / raw) To: Dave Love; +Cc: guix-devel On Tue, Oct 31, 2017 at 02:11:36PM +0000, Dave Love wrote: > Pjotr Prins <pjotr.public12@thebird.nl> writes: > > > But, really, I think when talking embedded systems and containers we > > all want tiny. Even HPC can benefit. Tiny containers may be an > > attractive proposition. > > Yes, containers aside, dependencies in Guix are one of the reasons I'm > currently unconvinced by its trades-off for HPC use; the closure a > single package can be comparable with the whole compute node image I > used to maintain with rpm. Even then, you don't generally have > debugging info available. For most software we just need the binary executable and its shared libs. That is exactly what I package with my tool: For gemma the binary closure is only 18Mb. See https://github.com/genetics-statistics/GEMMA/issues/90 That is small. And it runs on HPC. Maybe this is the way forward for HPC. For many HPC tools we can create such very small tarballs. You don't even need my relocatable installer if you have permission for /gnu/store (but then NFS would be a better idea). Great thing about Guix packages is that we can figure out what the dependencies are and strip it down to those. There is no interference. When it works, it works always! Once you have a Guix package and closure it is quite easy to compile it into something small and relocatable. That is what I have been experimenting with and I like the result. I am going to deploy GEMMA on a KNL supercomputer this month. > I suppose the general point is that Guix seems to have rejected > experience from other distributions, and it's not clear to me why. (I > don't mean it should necessarily follow them, of course.) Is there any > summary of the rationale for different decisions like not splitting > packages into development and runtime and other components, packaging > static libraries, and discarding debugging information, for instance? In my opinion Guix is both young and mature at the same time. Splitting packages is hard work and I think if enough people want to do it it will happen (eventually). Like Ludo suggests, we could start with the low hanging fruit. In the first E-mail I wrote this would be of great interest to embedded systems and HPC. I may try my stripping method on ARM and MIPS coming year. Just for the heck of it. Pj. -- ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Tiny Guix (and containers) 2017-11-03 12:08 ` Pjotr Prins @ 2017-11-06 15:10 ` Dave Love 0 siblings, 0 replies; 17+ messages in thread From: Dave Love @ 2017-11-06 15:10 UTC (permalink / raw) To: Pjotr Prins; +Cc: guix-devel Pjotr Prins <pjotr.public12@thebird.nl> writes: > For most software we just need the binary executable and its shared > libs. That is exactly what I package with my tool: > > For gemma the binary closure is only 18Mb. See > https://github.com/genetics-statistics/GEMMA/issues/90 > > That is small. And it runs on HPC. Maybe this is the way forward for > HPC. For many HPC tools we can create such very small tarballs. Small size is probably mostly an issue with ramfs nodes as long as you don't hammer a shared filesystem. The issue is probably more the dependencies, and what happens when you try to rebuild. Also it can be relevant if you want to version a whole system image or install on local disk. > You > don't even need my relocatable installer if you have permission for > /gnu/store (but then NFS would be a better idea). Great thing > about Guix packages is that we can figure out what the dependencies > are and strip it down to those. There is no interference. When it > works, it works always! But OS packages maintain dependencies, and you can supply dummy provides to avoid sucking in too much from the base system. There are advantages to their typical dependencies on the basis of interfaces. For what it's worth, you don't necessarily get what you need with Guix. For instance, the openmpi package now doesn't depend on (a version of) gfortran, so you need to know what to install with it to build MPI Fortran code (or profile it). > Once you have a Guix package and closure it is quite easy to compile > it into something small and relocatable. That is what I have been > experimenting with and I like the result. I am going to deploy GEMMA > on a KNL supercomputer this month. For interest, why specifically does the closure matter in that case? (Something that's relevant to KNL and not in Guix is the memkind librray.) >> I suppose the general point is that Guix seems to have rejected >> experience from other distributions, and it's not clear to me why. (I >> don't mean it should necessarily follow them, of course.) Is there any >> summary of the rationale for different decisions like not splitting >> packages into development and runtime and other components, packaging >> static libraries, and discarding debugging information, for instance? > > In my opinion Guix is both young and mature at the same time. > Splitting packages is hard work and I think if enough people want to > do it it will happen (eventually). Like Ludo suggests, we could start > with the low hanging fruit. Yes, on the status. I hope that problems I see can be fixed with engineering effort, and I've tackled some large closures to understand a bit of what goes on. The major effort doesn't isn't an issue in other distributions which conventionally do split packages, though there's often scope for reducing dependencies. Also they normally shouldn't end up with multiple versions of the same library, or multiple ones supporting the same interface, though the latter is currently the case with linear algebra in Fedora. I mean to be constructive, and I obviously think it's worth persisting with, but there do seem to be real issues to tackle, especially if Guix is going to fulfil the promise for user-built software. I hope it's useful to raise issues. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Tiny Guix (and containers) 2017-10-31 14:11 ` Dave Love 2017-11-03 12:08 ` Pjotr Prins @ 2017-11-05 15:55 ` Ludovic Courtès 2017-11-06 15:11 ` Dave Love 1 sibling, 1 reply; 17+ messages in thread From: Ludovic Courtès @ 2017-11-05 15:55 UTC (permalink / raw) To: Dave Love; +Cc: guix-devel Dave Love <fx@gnu.org> skribis: > Pjotr Prins <pjotr.public12@thebird.nl> writes: > >> But, really, I think when talking embedded systems and containers we >> all want tiny. Even HPC can benefit. Tiny containers may be an >> attractive proposition. > > Yes, containers aside, dependencies in Guix are one of the reasons I'm > currently unconvinced by its trades-off for HPC use; the closure a > single package can be comparable with the whole compute node image I > used to maintain with rpm. Even then, you don't generally have > debugging info available. > > I suppose the general point is that Guix seems to have rejected > experience from other distributions, and it's not clear to me why. (I > don't mean it should necessarily follow them, of course.) Is there any > summary of the rationale for different decisions like not splitting > packages into development and runtime and other components, packaging > static libraries, and discarding debugging information, for instance? The main reason, as you have noticed, is that multiple outputs don’t always work out-of-the-box: the GC scanner does not handle circular dependencies among outputs of a given derivation, which makes things more difficult. Another reason is that splitting is just part of the story. Often, it’s not moving out 10 KB of header files that really helps; rather, it’s stripping the dependency graph. Some people also argued that having everything in one package made things easier for them as users, though that one is more questionable to me. Ludo’. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Tiny Guix (and containers) 2017-11-05 15:55 ` Ludovic Courtès @ 2017-11-06 15:11 ` Dave Love 0 siblings, 0 replies; 17+ messages in thread From: Dave Love @ 2017-11-06 15:11 UTC (permalink / raw) To: Ludovic Courtès; +Cc: guix-devel Ludovic Courtès <ludo@gnu.org> writes: > Another reason is that splitting is just part of the story. Often, it’s > not moving out 10 KB of header files that really helps; rather, it’s > stripping the dependency graph. Oh, sure, and that probably isn't the reason for convention of splitting development stuff out (though keeping the number of files down may have some advantage). ^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2017-11-07 10:19 UTC | newest] Thread overview: 17+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2017-10-25 8:18 Tiny Guix (and containers) Pjotr Prins 2017-10-25 11:12 ` Pjotr Prins 2017-10-26 7:02 ` Ricardo Wurmus 2017-10-26 10:42 ` Pjotr Prins 2017-10-26 10:48 ` Ricardo Wurmus 2017-10-27 8:25 ` Hartmut Goebel 2017-10-28 20:06 ` Ludovic Courtès 2017-10-31 14:17 ` Dave Love 2017-11-05 16:02 ` Ludovic Courtès 2017-11-06 15:45 ` Dave Love 2017-11-07 10:19 ` Ludovic Courtès 2017-10-27 0:35 ` Ludovic Courtès 2017-10-31 14:11 ` Dave Love 2017-11-03 12:08 ` Pjotr Prins 2017-11-06 15:10 ` Dave Love 2017-11-05 15:55 ` Ludovic Courtès 2017-11-06 15:11 ` Dave Love
Code repositories for project(s) associated with this public inbox https://git.savannah.gnu.org/cgit/guix.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).