qt: monolithic or modular?

qt: monolithic or modular?

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1047 bytes --]

I try very hard to not build qt on my laptop, mostly because of the long build time (7 hours on hydra [0]). Currently we download and use the big download of qt[1] and frankly I'd rather not. Qt does also ship in smaller bits[2], 32 if I counted correctly. I propose we package the submodules and over time we go through the packages that use qt and switch out the monolithic qt for just the parts that the program actually uses. It makes it less daunting to build, should make the closures smaller, and means that if a submodule fails to build on an architecture then they only lose that module, not all of qt.

[0] http://hydra.gnu.org/build/1114596
[1] https://download.qt.io/official_releases/qt/5.6/5.6.0/single/qt-everywhere-opensource-src-5.6.0.tar.xz
[2] https://download.qt.io/official_releases/qt/5.6/5.6.0/submodules/

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: qt: monolithic or modular?

[Ludovic Courtès]

Efraim Flashner <efraim@flashner.co.il> skribis:

> I try very hard to not build qt
> on my laptop, mostly because of the long build time (7 hours on hydra
> [0]). Currently we download and use the big download of qt[1] and
> frankly I'd rather not. Qt does also ship in smaller bits[2], 32 if I
> counted correctly. I propose we package the submodules and over time
> we go through the packages that use qt and switch out the monolithic
> qt for just the parts that the program actually uses. It makes it less
> daunting to build, should make the closures smaller, and means that if
> a submodule fails to build on an architecture then they only lose that
> module, not all of qt.

I’m all for it!  It may also simplify unbundling as discussed in
<http://bugs.gnu.org/21288>.  (We’ll need a Qt updaters and/or importer
at some point.)

The difficulty is to make sure Qt finds its modules at run time, but the
fact that Nixpkgs uses this approach¹ suggests that it should work.

If you’re willing to work on this, I would suggest creating a branch,
which we’ll build on Hydra when is appropriate.

Thanks,
Ludo’.

¹ https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/libraries/qt-5/5.4/default.nix

Re: qt: monolithic or modular?

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1858 bytes --]

On Tue, Apr 05, 2016 at 11:38:18AM +0200, Ludovic Courtès wrote:
> Efraim Flashner <efraim@flashner.co.il> skribis:
> 
> > I try very hard to not build qt
> > on my laptop, mostly because of the long build time (7 hours on hydra
> > [0]). Currently we download and use the big download of qt[1] and
> > frankly I'd rather not. Qt does also ship in smaller bits[2], 32 if I
> > counted correctly. I propose we package the submodules and over time
> > we go through the packages that use qt and switch out the monolithic
> > qt for just the parts that the program actually uses. It makes it less
> > daunting to build, should make the closures smaller, and means that if
> > a submodule fails to build on an architecture then they only lose that
> > module, not all of qt.
> 
> I’m all for it!  It may also simplify unbundling as discussed in
> <http://bugs.gnu.org/21288>.  (We’ll need a Qt updaters and/or importer
> at some point.)
> 
> The difficulty is to make sure Qt finds its modules at run time, but the
> fact that Nixpkgs uses this approach¹ suggests that it should work.
> 
> If you’re willing to work on this, I would suggest creating a branch,
> which we’ll build on Hydra when is appropriate.
> 
> Thanks,
> Ludo’.
> 
> ¹ https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/libraries/qt-5/5.4/default.nix

quick followup: qtbase has the full config and makefiles, a bunch of the
other modules don't. It was suggested to me on irc that after compiling
qtbase I should try running qmake in the other folders to compile them,
but I haven't had a chance to progress that far yet.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: qt: monolithic or modular?

[Andreas Enge]

On Tue, Apr 05, 2016 at 07:22:20AM +0300, Efraim Flashner wrote:
> I try very hard to not build qt on my laptop, mostly because of the long build time (7 hours on hydra [0]). Currently we download and use the big download of qt[1] and frankly I'd rather not. Qt does also ship in smaller bits[2], 32 if I counted correctly. I propose we package the submodules and over time we go through the packages that use qt and switch out the monolithic qt for just the parts that the program actually uses. It makes it less daunting to build, should make the closures smaller, and means that if a submodule fails to build on an architecture then they only lose that module, not all of qt.

This would be very welcome indeed! It would also make debugging easier (for
instance, Qt-5 currently does not build on arm). If I understand correctly, 
we could in the beginning keep the modules in parallel with a monolithic build
(thanks to different names qt and qtbase) and switch over one package using Qt
after the other?

Since you mentioned incomplete build systems for some modules, would it not
be enough to add qtbase (which includes qmake, I suppose?) to their inputs?

Andreas

Re: qt: monolithic or modular?

[Efraim Flashner]

[-- Attachment #1.1: Type: text/plain, Size: 2531 bytes --]

On Tue, Apr 05, 2016 at 07:22:20AM +0300, Efraim Flashner wrote:
> I try very hard to not build qt on my laptop, mostly because of the long build time (7 hours on hydra [0]). Currently we download and use the big download of qt[1] and frankly I'd rather not. Qt does also ship in smaller bits[2], 32 if I counted correctly. I propose we package the submodules and over time we go through the packages that use qt and switch out the monolithic qt for just the parts that the program actually uses. It makes it less daunting to build, should make the closures smaller, and means that if a submodule fails to build on an architecture then they only lose that module, not all of qt.
> 
> [0] http://hydra.gnu.org/build/1114596
> [1] https://download.qt.io/official_releases/qt/5.6/5.6.0/single/qt-everywhere-opensource-src-5.6.0.tar.xz
> [2] https://download.qt.io/official_releases/qt/5.6/5.6.0/submodules/
> 

Finally got around to building qtbase out, took me 6 hours total on my
machine. So since hydra[1] says it takes 7:15 it's a bit shorter. I
haven't had a chance yet to try out qmake on the other modules or to try
to optimize the build yet. One of the things I did want to try was
replacing python2 with python-wrapper and enabling parallel-builds.

I opted for straight out copying qt-5's build rather than inheriting so
it'll be easier to remove it if/when we're ready, and I updated the
license based on the text shown during build-time.

I've attached what I have so far if anyone else wants to take a look at
it while I'm working on it.

Also very worthy of note, qt-5.5.1 is listed at 288 MB, and qtbase-5.6.0
is all of 85 MB.

efraim@debian-netbook:~$ du -sch
/gnu/store/r9bpiyz2w5bkavnx3s1ffxpgc51wa9z5-qtbase-5.6.0/*
6.6M    /gnu/store/r9bpiyz2w5bkavnx3s1ffxpgc51wa9z5-qtbase-5.6.0/bin
560K    /gnu/store/r9bpiyz2w5bkavnx3s1ffxpgc51wa9z5-qtbase-5.6.0/doc
25M     /gnu/store/r9bpiyz2w5bkavnx3s1ffxpgc51wa9z5-qtbase-5.6.0/examples
20M     /gnu/store/r9bpiyz2w5bkavnx3s1ffxpgc51wa9z5-qtbase-5.6.0/include
30M     /gnu/store/r9bpiyz2w5bkavnx3s1ffxpgc51wa9z5-qtbase-5.6.0/lib
2.5M    /gnu/store/r9bpiyz2w5bkavnx3s1ffxpgc51wa9z5-qtbase-5.6.0/mkspecs
2.6M    /gnu/store/r9bpiyz2w5bkavnx3s1ffxpgc51wa9z5-qtbase-5.6.0/plugins
85M total

[1] http://hydra.gnu.org/build/1202329

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #1.2: 0001-gnu-Add-qtbase.patch --]
[-- Type: text/plain, Size: 7336 bytes --]

From 8d324a49e93b51daf2b756c7b749fd9be295824d Mon Sep 17 00:00:00 2001
From: Efraim Flashner <efraim@flashner.co.il>
Date: Wed, 18 May 2016 15:15:28 +0300
Subject: [PATCH] gnu: Add qtbase.

* gnu/packages/qt.scm (qtbase): New variable.
---
 gnu/packages/qt.scm | 142 +++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 140 insertions(+), 2 deletions(-)

diff --git a/gnu/packages/qt.scm b/gnu/packages/qt.scm
index 097ec06..bdd298f 100644
--- a/gnu/packages/qt.scm
+++ b/gnu/packages/qt.scm
@@ -2,7 +2,7 @@
 ;;; Copyright © 2013, 2014, 2015 Andreas Enge <andreas@enge.fr>
 ;;; Copyright © 2015 Sou Bunnbu <iyzsong@gmail.com>
 ;;; Copyright © 2015 Ludovic Courtès <ludo@gnu.org>
-;;; Copyright © 2015 Efraim Flashner <efraim@flashner.co.il>
+;;; Copyright © 2015, 2016 Efraim Flashner <efraim@flashner.co.il>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -20,7 +20,7 @@
 ;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
 
 (define-module (gnu packages qt)
-  #:use-module ((guix licenses) #:select (bsd-3 gpl2 gpl3 lgpl2.1 lgpl2.1+ x11-style))
+  #:use-module ((guix licenses) #:select (bsd-3 gpl2 gpl3 lgpl2.1 lgpl2.1+ lgpl3 x11-style))
   #:use-module (guix packages)
   #:use-module (guix download)
   #:use-module (guix build utils)
@@ -201,6 +201,144 @@ developers using C++ or QML, a CSS & JavaScript like language.")
     ;; Qt 5: assembler error; see <http://hydra.gnu.org/build/112526>.
     (supported-systems (delete "mips64el-linux" %supported-systems))))
 
+(define-public qtbase
+  (package
+    (name "qtbase")
+    (version "5.6.0")
+    (source (origin
+             (method url-fetch)
+             (uri (string-append "https://download.qt.io/official_releases/qt/"
+                                 (version-major+minor version) "/" version
+                                 "/submodules/" name "-opensource-src-"
+                                 version ".tar.xz"))
+             (sha256
+               (base32
+                 "0ynnvcs5idivzldsq5ciqg9myg82b3l3906l4vjv54lyamf8mykf"))
+             (modules '((guix build utils)))
+             (snippet
+              '(begin
+                ;; Remove one of the two bundled harfbuzz copies in addition
+                ;; to passing "-system-harfbuzz".
+                (delete-file-recursively "src/3rdparty/harfbuzz-ng")
+                ;; Remove the bundled sqlite copy in addition to
+                ;; passing "-system-sqlite".
+                (delete-file-recursively "src/3rdparty/sqlite")))))
+    (build-system gnu-build-system)
+    (propagated-inputs
+     `(("mesa" ,mesa)))
+    (inputs
+     `(("alsa-lib" ,alsa-lib)
+       ("dbus" ,dbus)
+       ("cups" ,cups)
+       ("expat" ,expat)
+       ("fontconfig" ,fontconfig)
+       ("freetype" ,freetype)
+       ("glib" ,glib)
+       ("harfbuzz" ,harfbuzz)
+       ("icu4c" ,icu4c)
+       ("libjpeg" ,libjpeg)
+       ("libmng" ,libmng)
+       ("libpci" ,pciutils)
+       ("libpng" ,libpng)
+       ("libx11" ,libx11)
+       ("libxcomposite" ,libxcomposite)
+       ("libxcursor" ,libxcursor)
+       ("libxfixes" ,libxfixes)
+       ("libxi" ,libxi)
+       ("libxinerama" ,libxinerama)
+       ("libxkbcommon" ,libxkbcommon)
+       ("libxml2" ,libxml2)
+       ("libxrandr" ,libxrandr)
+       ("libxrender" ,libxrender)
+       ("libxslt" ,libxslt)
+       ("libxtst" ,libxtst)
+       ("mtdev" ,mtdev)
+       ("mysql" ,mysql)
+       ("nss" ,nss)
+       ("openssl" ,openssl)
+       ("postgresql" ,postgresql)
+       ("pulseaudio" ,pulseaudio)
+       ("pcre" ,pcre)
+       ("sqlite" ,sqlite)
+       ("udev" ,eudev)
+       ("unixodbc" ,unixodbc)
+       ("xcb-util" ,xcb-util)
+       ("xcb-util-image" ,xcb-util-image)
+       ("xcb-util-keysyms" ,xcb-util-keysyms)
+       ("xcb-util-renderutil" ,xcb-util-renderutil)
+       ("xcb-util-wm" ,xcb-util-wm)
+       ("zlib" ,zlib)))
+    (native-inputs
+     `(("bison" ,bison)
+       ("flex" ,flex)
+       ("gperf" ,gperf)
+       ("perl" ,perl)
+       ("pkg-config" ,pkg-config)
+       ("python" ,python-2)
+       ("ruby" ,ruby)
+       ("which" ,(@ (gnu packages base) which))))
+    (arguments
+     `(;; FIXME: Disabling parallel building is a quick hack to avoid the
+       ;; failure described in
+       ;; https://lists.gnu.org/archive/html/guix-devel/2016-01/msg00837.html
+       ;; A more structural fix is needed.
+       #:parallel-build? #f
+       #:phases
+       (modify-phases %standard-phases
+         (add-after 'configure 'patch-bin-sh
+           (lambda _
+             (substitute* '("config.status"
+                            "configure"
+                            "mkspecs/features/qt_functions.prf"
+                            "qmake/library/qmakebuiltins.cpp")
+                          (("/bin/sh") (which "sh")))
+             #t))
+         (replace 'configure
+           (lambda* (#:key outputs #:allow-other-keys)
+             (let ((out (assoc-ref outputs "out")))
+               (substitute* "configure"
+                 (("/bin/pwd") (which "pwd")))
+               (substitute* "src/corelib/global/global.pri"
+                 (("/bin/ls") (which "ls")))
+               ;; do not pass "--enable-fast-install", which makes the
+               ;; configure process fail
+               (zero? (system*
+                       "./configure"
+                       "-verbose"
+                       "-prefix" out
+                       "-opensource"
+                       "-confirm-license"
+                       ;; Most "-system-..." are automatic, but some use
+                       ;; the bundled copy by default.
+                       "-system-sqlite"
+                       "-system-harfbuzz"
+                       ;; explicitly link with openssl instead of dlopening it
+                       "-openssl-linked"
+                       ;; explicitly link with dbus instead of dlopening it
+                       "-dbus-linked"
+                       ;; drop special machine instructions not supported
+                       ;; on all instances of the target
+                       ,@(if (string-prefix? "x86_64"
+                                             (or (%current-target-system)
+                                                 (%current-system)))
+                             '()
+                             '("-no-sse2"))
+                       "-no-sse3"
+                       "-no-ssse3"
+                       "-no-sse4.1"
+                       "-no-sse4.2"
+                       "-no-avx"
+                       "-no-avx2"
+                       "-no-mips_dsp"
+                       "-no-mips_dspr2"))))))))
+    (home-page "https://www.qt.io/")
+    (synopsis "Cross-platform GUI library")
+    (description "Qt is a cross-platform application and UI framework for
+developers using C++ or QML, a CSS & JavaScript like language.")
+    (license (list lgpl2.1 lgpl3))
+    ;; Qt 5: assembler error; see <http://hydra.gnu.org/build/112526>.
+    (supported-systems (delete "mips64el-linux" %supported-systems))))
+
 (define-public qt-4
   (package (inherit qt)
     (version "4.8.7")
-- 
2.7.4


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: qt: monolithic or modular?

[Ludovic Courtès]

Efraim Flashner <efraim@flashner.co.il> skribis:

> On Tue, Apr 05, 2016 at 07:22:20AM +0300, Efraim Flashner wrote:
>> I try very hard to not build qt on my laptop, mostly because of the long build time (7 hours on hydra [0]). Currently we download and use the big download of qt[1] and frankly I'd rather not. Qt does also ship in smaller bits[2], 32 if I counted correctly. I propose we package the submodules and over time we go through the packages that use qt and switch out the monolithic qt for just the parts that the program actually uses. It makes it less daunting to build, should make the closures smaller, and means that if a submodule fails to build on an architecture then they only lose that module, not all of qt.
>> 
>> [0] http://hydra.gnu.org/build/1114596
>> [1] https://download.qt.io/official_releases/qt/5.6/5.6.0/single/qt-everywhere-opensource-src-5.6.0.tar.xz
>> [2] https://download.qt.io/official_releases/qt/5.6/5.6.0/submodules/
>> 
>
> Finally got around to building qtbase out, took me 6 hours total on my
> machine. So since hydra[1] says it takes 7:15 it's a bit shorter. I
> haven't had a chance yet to try out qmake on the other modules or to try
> to optimize the build yet. One of the things I did want to try was
> replacing python2 with python-wrapper and enabling parallel-builds.
>
> I opted for straight out copying qt-5's build rather than inheriting so
> it'll be easier to remove it if/when we're ready, and I updated the
> license based on the text shown during build-time.
>
> I've attached what I have so far if anyone else wants to take a look at
> it while I'm working on it.

Nice!  I gather that some applications require more than just qtbase, so
we’d need to have all the different Qt components before we can fully
switch to this model, right?

Now, this could be done incrementally: we could start moving packages
that need nothing beyond qtbase to this new package, and so on.

Thoughts?

> Also very worthy of note, qt-5.5.1 is listed at 288 MB, and qtbase-5.6.0
> is all of 85 MB.
>
> efraim@debian-netbook:~$ du -sch
> /gnu/store/r9bpiyz2w5bkavnx3s1ffxpgc51wa9z5-qtbase-5.6.0/*
> 6.6M    /gnu/store/r9bpiyz2w5bkavnx3s1ffxpgc51wa9z5-qtbase-5.6.0/bin
> 560K    /gnu/store/r9bpiyz2w5bkavnx3s1ffxpgc51wa9z5-qtbase-5.6.0/doc
> 25M     /gnu/store/r9bpiyz2w5bkavnx3s1ffxpgc51wa9z5-qtbase-5.6.0/examples
> 20M     /gnu/store/r9bpiyz2w5bkavnx3s1ffxpgc51wa9z5-qtbase-5.6.0/include
> 30M     /gnu/store/r9bpiyz2w5bkavnx3s1ffxpgc51wa9z5-qtbase-5.6.0/lib
> 2.5M    /gnu/store/r9bpiyz2w5bkavnx3s1ffxpgc51wa9z5-qtbase-5.6.0/mkspecs
> 2.6M    /gnu/store/r9bpiyz2w5bkavnx3s1ffxpgc51wa9z5-qtbase-5.6.0/plugins
> 85M total

It would be awesome if this would use the various configure flags that
our qt4 package is using, such that we don’t have weird top-level
directories such as ’mkspecs’ and ‘plugins’.

If would be worth trying to move doc/ and examples/ to a “doc” output,
but as noted in a comment in qt4, moving the examples may not be
possible.

Could you give it a try?

Thanks,
Ludo’.

Re: qt: monolithic or modular?

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 4203 bytes --]

On Thu, May 19, 2016 at 02:15:07PM +0200, Ludovic Courtès wrote:
> Efraim Flashner <efraim@flashner.co.il> skribis:
> 
> > On Tue, Apr 05, 2016 at 07:22:20AM +0300, Efraim Flashner wrote:
> >> I try very hard to not build qt on my laptop, mostly because of the long build time (7 hours on hydra [0]). Currently we download and use the big download of qt[1] and frankly I'd rather not. Qt does also ship in smaller bits[2], 32 if I counted correctly. I propose we package the submodules and over time we go through the packages that use qt and switch out the monolithic qt for just the parts that the program actually uses. It makes it less daunting to build, should make the closures smaller, and means that if a submodule fails to build on an architecture then they only lose that module, not all of qt.
> >> 
> >> [0] http://hydra.gnu.org/build/1114596
> >> [1] https://download.qt.io/official_releases/qt/5.6/5.6.0/single/qt-everywhere-opensource-src-5.6.0.tar.xz
> >> [2] https://download.qt.io/official_releases/qt/5.6/5.6.0/submodules/
> >> 
> >
> > Finally got around to building qtbase out, took me 6 hours total on my
> > machine. So since hydra[1] says it takes 7:15 it's a bit shorter. I
> > haven't had a chance yet to try out qmake on the other modules or to try
> > to optimize the build yet. One of the things I did want to try was
> > replacing python2 with python-wrapper and enabling parallel-builds.
> >
> > I opted for straight out copying qt-5's build rather than inheriting so
> > it'll be easier to remove it if/when we're ready, and I updated the
> > license based on the text shown during build-time.
> >
> > I've attached what I have so far if anyone else wants to take a look at
> > it while I'm working on it.
> 
> Nice!  I gather that some applications require more than just qtbase, so
> we’d need to have all the different Qt components before we can fully
> switch to this model, right?
> 
> Now, this could be done incrementally: we could start moving packages
> that need nothing beyond qtbase to this new package, and so on.
> 
> Thoughts?

That was my plan. Looking at nix's qt-5 it doesn't look like they even
build all of the packages so hopefully it shouldn't be too long. My
original hope that the base would take significantly less time didn't
pan out, but this is still good. I'm now hoping that since qtbase took
forever the other modules should be quick.

> 
> > Also very worthy of note, qt-5.5.1 is listed at 288 MB, and qtbase-5.6.0
> > is all of 85 MB.
> >
> > efraim@debian-netbook:~$ du -sch
> > /gnu/store/r9bpiyz2w5bkavnx3s1ffxpgc51wa9z5-qtbase-5.6.0/*
> > 6.6M    /gnu/store/r9bpiyz2w5bkavnx3s1ffxpgc51wa9z5-qtbase-5.6.0/bin
> > 560K    /gnu/store/r9bpiyz2w5bkavnx3s1ffxpgc51wa9z5-qtbase-5.6.0/doc
> > 25M     /gnu/store/r9bpiyz2w5bkavnx3s1ffxpgc51wa9z5-qtbase-5.6.0/examples
> > 20M     /gnu/store/r9bpiyz2w5bkavnx3s1ffxpgc51wa9z5-qtbase-5.6.0/include
> > 30M     /gnu/store/r9bpiyz2w5bkavnx3s1ffxpgc51wa9z5-qtbase-5.6.0/lib
> > 2.5M    /gnu/store/r9bpiyz2w5bkavnx3s1ffxpgc51wa9z5-qtbase-5.6.0/mkspecs
> > 2.6M    /gnu/store/r9bpiyz2w5bkavnx3s1ffxpgc51wa9z5-qtbase-5.6.0/plugins
> > 85M total
> 
> It would be awesome if this would use the various configure flags that
> our qt4 package is using, such that we don’t have weird top-level
> directories such as ’mkspecs’ and ‘plugins’.
> 
> If would be worth trying to move doc/ and examples/ to a “doc” output,
> but as noted in a comment in qt4, moving the examples may not be
> possible.
> 
> Could you give it a try?
> 

mkspecs we actually might need, it has qmake.conf files for tons of
different architecture/OS options. Examples hopefully we can split out,
but reading the note for qt-4 I'm not too optimistic. Docs don't look to
me like they're big enough to bother with. Plugins has interface .so
files for things like different sql databases and other stuff.

> Thanks,
> Ludo’.

:)

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: qt: monolithic or modular?

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 4685 bytes --]

On Thu, May 19, 2016 at 04:17:50PM +0300, Efraim Flashner wrote:
> On Thu, May 19, 2016 at 02:15:07PM +0200, Ludovic Courtès wrote:
> > Efraim Flashner <efraim@flashner.co.il> skribis:
> > 
> > > On Tue, Apr 05, 2016 at 07:22:20AM +0300, Efraim Flashner wrote:
> > >> I try very hard to not build qt on my laptop, mostly because of the long build time (7 hours on hydra [0]). Currently we download and use the big download of qt[1] and frankly I'd rather not. Qt does also ship in smaller bits[2], 32 if I counted correctly. I propose we package the submodules and over time we go through the packages that use qt and switch out the monolithic qt for just the parts that the program actually uses. It makes it less daunting to build, should make the closures smaller, and means that if a submodule fails to build on an architecture then they only lose that module, not all of qt.
> > >> 
> > >> [0] http://hydra.gnu.org/build/1114596
> > >> [1] https://download.qt.io/official_releases/qt/5.6/5.6.0/single/qt-everywhere-opensource-src-5.6.0.tar.xz
> > >> [2] https://download.qt.io/official_releases/qt/5.6/5.6.0/submodules/
> > >> 
> > >
> > > Finally got around to building qtbase out, took me 6 hours total on my
> > > machine. So since hydra[1] says it takes 7:15 it's a bit shorter. I
> > > haven't had a chance yet to try out qmake on the other modules or to try
> > > to optimize the build yet. One of the things I did want to try was
> > > replacing python2 with python-wrapper and enabling parallel-builds.
> > >
> > > I opted for straight out copying qt-5's build rather than inheriting so
> > > it'll be easier to remove it if/when we're ready, and I updated the
> > > license based on the text shown during build-time.
> > >
> > > I've attached what I have so far if anyone else wants to take a look at
> > > it while I'm working on it.
> > 
> > Nice!  I gather that some applications require more than just qtbase, so
> > we’d need to have all the different Qt components before we can fully
> > switch to this model, right?
> > 
> > Now, this could be done incrementally: we could start moving packages
> > that need nothing beyond qtbase to this new package, and so on.
> > 
> > Thoughts?
> 
> That was my plan. Looking at nix's qt-5 it doesn't look like they even
> build all of the packages so hopefully it shouldn't be too long. My
> original hope that the base would take significantly less time didn't
> pan out, but this is still good. I'm now hoping that since qtbase took
> forever the other modules should be quick.
> 
> > 
> > > Also very worthy of note, qt-5.5.1 is listed at 288 MB, and qtbase-5.6.0
> > > is all of 85 MB.
> > >
> > > efraim@debian-netbook:~$ du -sch
> > > /gnu/store/r9bpiyz2w5bkavnx3s1ffxpgc51wa9z5-qtbase-5.6.0/*
> > > 6.6M    /gnu/store/r9bpiyz2w5bkavnx3s1ffxpgc51wa9z5-qtbase-5.6.0/bin
> > > 560K    /gnu/store/r9bpiyz2w5bkavnx3s1ffxpgc51wa9z5-qtbase-5.6.0/doc
> > > 25M     /gnu/store/r9bpiyz2w5bkavnx3s1ffxpgc51wa9z5-qtbase-5.6.0/examples
> > > 20M     /gnu/store/r9bpiyz2w5bkavnx3s1ffxpgc51wa9z5-qtbase-5.6.0/include
> > > 30M     /gnu/store/r9bpiyz2w5bkavnx3s1ffxpgc51wa9z5-qtbase-5.6.0/lib
> > > 2.5M    /gnu/store/r9bpiyz2w5bkavnx3s1ffxpgc51wa9z5-qtbase-5.6.0/mkspecs
> > > 2.6M    /gnu/store/r9bpiyz2w5bkavnx3s1ffxpgc51wa9z5-qtbase-5.6.0/plugins
> > > 85M total
> > 
> > It would be awesome if this would use the various configure flags that
> > our qt4 package is using, such that we don’t have weird top-level
> > directories such as ’mkspecs’ and ‘plugins’.
> > 
> > If would be worth trying to move doc/ and examples/ to a “doc” output,
> > but as noted in a comment in qt4, moving the examples may not be
> > possible.
> > 
> > Could you give it a try?
> > 
> 
> mkspecs we actually might need, it has qmake.conf files for tons of
> different architecture/OS options. Examples hopefully we can split out,
> but reading the note for qt-4 I'm not too optimistic. Docs don't look to
> me like they're big enough to bother with. Plugins has interface .so
> files for things like different sql databases and other stuff.
> 
> > Thanks,
> > Ludo’.
> 
> :)
> 

Actually I just checked ./configure --help again, and I can pass the
flag '-nomake examples' and it shouldn't make the examples at all. I'm
not sure how much compile time it'll save, but it should save us the
25MB of examples that we didn't really want.


-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: qt: monolithic or modular?

[Ludovic Courtès]

Efraim Flashner <efraim@flashner.co.il> skribis:

> On Thu, May 19, 2016 at 02:15:07PM +0200, Ludovic Courtès wrote:
>> Efraim Flashner <efraim@flashner.co.il> skribis:
>> 
>> > On Tue, Apr 05, 2016 at 07:22:20AM +0300, Efraim Flashner wrote:
>> >> I try very hard to not build qt on my laptop, mostly because of the long build time (7 hours on hydra [0]). Currently we download and use the big download of qt[1] and frankly I'd rather not. Qt does also ship in smaller bits[2], 32 if I counted correctly. I propose we package the submodules and over time we go through the packages that use qt and switch out the monolithic qt for just the parts that the program actually uses. It makes it less daunting to build, should make the closures smaller, and means that if a submodule fails to build on an architecture then they only lose that module, not all of qt.
>> >> 
>> >> [0] http://hydra.gnu.org/build/1114596
>> >> [1] https://download.qt.io/official_releases/qt/5.6/5.6.0/single/qt-everywhere-opensource-src-5.6.0.tar.xz
>> >> [2] https://download.qt.io/official_releases/qt/5.6/5.6.0/submodules/
>> >> 
>> >
>> > Finally got around to building qtbase out, took me 6 hours total on my
>> > machine. So since hydra[1] says it takes 7:15 it's a bit shorter. I
>> > haven't had a chance yet to try out qmake on the other modules or to try
>> > to optimize the build yet. One of the things I did want to try was
>> > replacing python2 with python-wrapper and enabling parallel-builds.
>> >
>> > I opted for straight out copying qt-5's build rather than inheriting so
>> > it'll be easier to remove it if/when we're ready, and I updated the
>> > license based on the text shown during build-time.
>> >
>> > I've attached what I have so far if anyone else wants to take a look at
>> > it while I'm working on it.
>> 
>> Nice!  I gather that some applications require more than just qtbase, so
>> we’d need to have all the different Qt components before we can fully
>> switch to this model, right?
>> 
>> Now, this could be done incrementally: we could start moving packages
>> that need nothing beyond qtbase to this new package, and so on.
>> 
>> Thoughts?
>
> That was my plan. Looking at nix's qt-5 it doesn't look like they even
> build all of the packages so hopefully it shouldn't be too long. My
> original hope that the base would take significantly less time didn't
> pan out, but this is still good. I'm now hoping that since qtbase took
> forever the other modules should be quick.

Cool.

>> 
>> > Also very worthy of note, qt-5.5.1 is listed at 288 MB, and qtbase-5.6.0
>> > is all of 85 MB.
>> >
>> > efraim@debian-netbook:~$ du -sch
>> > /gnu/store/r9bpiyz2w5bkavnx3s1ffxpgc51wa9z5-qtbase-5.6.0/*
>> > 6.6M    /gnu/store/r9bpiyz2w5bkavnx3s1ffxpgc51wa9z5-qtbase-5.6.0/bin
>> > 560K    /gnu/store/r9bpiyz2w5bkavnx3s1ffxpgc51wa9z5-qtbase-5.6.0/doc
>> > 25M     /gnu/store/r9bpiyz2w5bkavnx3s1ffxpgc51wa9z5-qtbase-5.6.0/examples
>> > 20M     /gnu/store/r9bpiyz2w5bkavnx3s1ffxpgc51wa9z5-qtbase-5.6.0/include
>> > 30M     /gnu/store/r9bpiyz2w5bkavnx3s1ffxpgc51wa9z5-qtbase-5.6.0/lib
>> > 2.5M    /gnu/store/r9bpiyz2w5bkavnx3s1ffxpgc51wa9z5-qtbase-5.6.0/mkspecs
>> > 2.6M    /gnu/store/r9bpiyz2w5bkavnx3s1ffxpgc51wa9z5-qtbase-5.6.0/plugins
>> > 85M total
>> 
>> It would be awesome if this would use the various configure flags that
>> our qt4 package is using, such that we don’t have weird top-level
>> directories such as ’mkspecs’ and ‘plugins’.
>> 
>> If would be worth trying to move doc/ and examples/ to a “doc” output,
>> but as noted in a comment in qt4, moving the examples may not be
>> possible.
>> 
>> Could you give it a try?
>> 
>
> mkspecs we actually might need, it has qmake.conf files for tons of
> different architecture/OS options.

Sure; I’m not suggesting to remove ‘mkspecs’, just to move it in a more
standard place, as done in qt4:

            "-importdir" (string-append out "/lib/qt-4"
                                        "/imports")

> Examples hopefully we can split out, but reading the note for qt-4 I'm
> not too optimistic. Docs don't look to me like they're big enough to
> bother with. Plugins has interface .so files for things like different
> sql databases and other stuff.

Ludo’.

Re: qt: monolithic or modular?

[Ludovic Courtès]

Efraim Flashner <efraim@flashner.co.il> skribis:

> Actually I just checked ./configure --help again, and I can pass the
> flag '-nomake examples' and it shouldn't make the examples at all. I'm
> not sure how much compile time it'll save, but it should save us the
> 25MB of examples that we didn't really want.

That sounds like a good idea.  We could probably create a separate
‘qt-examples’ package if someone needs them.

Ludo’.

Re: qt: monolithic or modular?

[Andreas Enge]

On Fri, May 20, 2016 at 01:59:04PM +0200, Ludovic Courtès wrote:
> Efraim Flashner <efraim@flashner.co.il> skribis:
> > Actually I just checked ./configure --help again, and I can pass the
> > flag '-nomake examples' and it shouldn't make the examples at all. I'm
> > not sure how much compile time it'll save, but it should save us the
> > 25MB of examples that we didn't really want.
> That sounds like a good idea.  We could probably create a separate
> ‘qt-examples’ package if someone needs them.

Is that not what our different outputs are for? Concerning the docs, I concur
with Ephraim that we should keep them in the same output, as their size is
negligible. For the examples, passing "-nomake examples" sounds like a good
option to me for now, with a comment in the package; and if someone complains
later, we can add a different output.

Thanks, Ephraim, for working on this!

Andreas

Re: qt: monolithic or modular?

[Andreas Enge]

Hi Efraim,

as you have noticed, I have created a wip-qt branch, mainly to test building
qtbase on mips. I removed the removal of mips; but it turns out that an input
is missing anyway. So I think we can include mips for the time being, it does
no harm and may serve as a reminder. It also builds on arm, which is a progress
compared to our current monolithic qt. So maybe the packages that depend on qt
will finally be available on arm again.

One thing I did not yet have time to do was to check whether all inputs were
referenced in the result; I would have expected that the base package would
require less inputs than the whole.

After that, I think we can push to master, as the new package is completely
independent of the old one, and nothing can break. I think it will serve no
purpose to keep the separate wip-qt branch, as each evaluation is quite
costly; this one took about 3,5 hours, and I just created the branch since
hydra was idle (which will hopefully change with the building of core-updates).
My apologies, since your patch did not apply any more, I copied and pasted
by hand and forgot to add you as a coauthor. We can correct this on master.

What do you think?

Andreas

Re: qt: monolithic or modular?

[Andreas Enge]

Hi again,

in the end I dropped pciutils and udev, which were not referenced in the
output of qtbase, and pushed to master.

Thanks!

Andreas

Re: qt: monolithic or modular?

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 3657 bytes --]

On Fri, Jun 03, 2016 at 10:30:32PM +0200, Andreas Enge wrote:
> Hi Efraim,
> 
> as you have noticed, I have created a wip-qt branch, mainly to test building
> qtbase on mips. I removed the removal of mips; but it turns out that an input
> is missing anyway. So I think we can include mips for the time being, it does
> no harm and may serve as a reminder. It also builds on arm, which is a progress
> compared to our current monolithic qt. So maybe the packages that depend on qt
> will finally be available on arm again.
> 
> One thing I did not yet have time to do was to check whether all inputs were
> referenced in the result; I would have expected that the base package would
> require less inputs than the whole.
> 
> After that, I think we can push to master, as the new package is completely
> independent of the old one, and nothing can break. I think it will serve no
> purpose to keep the separate wip-qt branch, as each evaluation is quite
> costly; this one took about 3,5 hours, and I just created the branch since
> hydra was idle (which will hopefully change with the building of core-updates).
> My apologies, since your patch did not apply any more, I copied and pasted
> by hand and forgot to add you as a coauthor. We can correct this on master.
> 
> What do you think?
> 
> Andreas
> 

To be fair, it was only the arm build that took 3.5 hours, i686 and
x86_64 both took about an hour. I've added to the wip-qt branch with
some of the "round 1"/direct dependencies on qtbase. I haven't had a
chance yet to try redirecting some of our qt-dependant packages on the
new system but I'm guessing it'll be soon.

As for the wip-qt branch, now with qtbase already in master I think it
makes sense to rebase the branch on master (and let me squash a commit
or two) and we can see about switching packages from qt to qtbase.

I wanted to make a nice ascii-art chart, but I'll just post the output
and reword it after:

nixpkgs/pkgs/development/libraries/qt-5/5.6$ grep qtInputs -R
qtgraphicaleffects.nix:  qtInputs = [ qtdeclarative ];
qtimageformats.nix:  qtInputs = [ qtbase ];
default.nix:      propagatedBuildInputs = args.qtInputs ++
(args.propagatedBuildInputs or []);
qtsvg.nix:  qtInputs = [ qtbase ];
qttranslations.nix:  qtInputs = [ qttools ];
qtconnectivity.nix:  qtInputs = [ qtbase qtdeclarative ];
qtwebsockets.nix:  qtInputs = [ qtbase qtdeclarative ];
qtlocation.nix:  qtInputs = [ qtbase qtmultimedia ];
qtsensors.nix:  qtInputs = [ qtbase qtdeclarative ];
qttools.nix:  qtInputs = [ qtbase qtdeclarative ];
qtdoc.nix:  qtInputs = [ qtdeclarative ];
qtdeclarative/default.nix:  qtInputs = [ qtbase qtsvg qtxmlpatterns ];
qtscript/default.nix:  qtInputs = [ qtbase qttools ];
qtenginio.nix:  qtInputs = [ qtdeclarative ];
qtserialport/default.nix:  qtInputs = [ qtbase ];
qtx11extras.nix:  qtInputs = [ qtbase ];
qtmultimedia.nix:  qtInputs = [ qtbase qtdeclarative ];
qtxmlpatterns.nix:  qtInputs = [ qtbase ];
qtquickcontrols.nix:  qtInputs = [ qtdeclarative ];

round 0:
qtbase

round 1:
qtsvg, qtimageformats, qtx11extras, qtxmlpatterns, qtserialport

round 2:
qtdeclarative

round 3:
qtgraphicaleffects, qtconnectivity, qtwebsockets, qtsensors, qttools,
qtdoc, qtenginio, qtmultimedia, qtquickcontrols

round 4:
qttranslations, qtlocation

and that gets us to about where nix has broken down the internal qt
dependencies for us.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: qt: monolithic or modular?

[Andreas Enge]

On Sun, Jun 05, 2016 at 10:24:18AM +0300, Efraim Flashner wrote:
> To be fair, it was only the arm build that took 3.5 hours, i686 and
> x86_64 both took about an hour.

What took 3,5 hours was the "evaluation" on hydra: The time between the
moment I made hydra look at wip-qt, and the moment the first compilation
of a package started, and this at a time where hydra had nothing else
to do. So I do not think it makes sense to continue working in the wip-qt
branch on hydra and to evaluate it ever so often, even more so since so far
nothing depends on it and we will for the time being keep the monolithic
qt-5; you may as well work locally as for other packages.

By the way, dropping the examples made for a factor of 2 in the compilation
time; a nice improvement!

Andreas

Re: qt: monolithic or modular?

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1211 bytes --]

On Sun, Jun 05, 2016 at 11:15:59AM +0200, Andreas Enge wrote:
> On Sun, Jun 05, 2016 at 10:24:18AM +0300, Efraim Flashner wrote:
> > To be fair, it was only the arm build that took 3.5 hours, i686 and
> > x86_64 both took about an hour.
> 
> What took 3,5 hours was the "evaluation" on hydra: The time between the
> moment I made hydra look at wip-qt, and the moment the first compilation
> of a package started, and this at a time where hydra had nothing else
> to do. So I do not think it makes sense to continue working in the wip-qt
> branch on hydra and to evaluate it ever so often, even more so since so far
> nothing depends on it and we will for the time being keep the monolithic
> qt-5; you may as well work locally as for other packages.
> 

Ok, I wasn't aware that it took hydra that long to actually start
working on the next package set.

> By the way, dropping the examples made for a factor of 2 in the compilation
> time; a nice improvement!
> 
> Andreas
> 

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: qt: monolithic or modular?

[Andreas Enge]

On Sun, Jun 05, 2016 at 01:24:23PM +0300, Efraim Flashner wrote:
> Ok, I wasn't aware that it took hydra that long to actually start
> working on the next package set.

This is not normal and should be fixed once the new machine will have
arrived, and Mathieu's GSoC project is finished :-)

Andreas