Upstreaming my Android channel

Upstreaming my Android channel

[Julien Lepiller]

Hi Guix!

I have been working on updating Android packages for some time now. I
think it's at a point I'm confident I can share and move them upstream.
My work is currently in a separate channel at
https://framagit.org/tyreunom/guix-android. I have been working on the
SDK and tools, which we already have. We currently are limited to
Android 7 versions, because later Android versions do not use the same
build system anymore.

Since then, Android uses the Soong build system, based on blueprint.
It's pretty complex, and specific. I tried to build it, but it assumes
too many things to be useful. Instead, I "reimplemented" it by
following what a Nix contributor did in soongnix:
https://github.com/danielfullmer/soongnix

I created a soong-build-system that has a few modules for cc, art and
java types of packages. It supports building most packages from the SDK
and other Android tools: aapt, appt2, adb, aidl, apksigner, dexdump,
dmtracedump, dx, etc1tool, fastboot, hprof-conv, libaapt2_jni,
split-select, zipalign.

I have an importer that is a bit different from the existing importers.
First, it needs to checkout ~30GB of git repositories (and that's with
--depth=1 for all of them!), explore the set of Android.bp files (used
by soong) and create one package for each package in these Android.bp
files. That creates packages that can be a bit smaller than expected.
Instead of printing, the importer creates two files: one for the source
definitions (many packages share the same source) and one for the
packages and their dependencies.

Since I have all of that in a channel, the importer doesn't work from
the command-line, but I have a relatively simple API that you can use
to import packages:

```
,use (android import repo)
(define manifest (get-manifest))

;; Fetch repositories that form the android distribution.
;; This is very big (> 28GB) and will take some time.  If it fails
;; because of a network issue, restart that command again until
;; everything is downloaded.
;;
;; You can also add an optional destination directory where the
;; sources are to be downloaded, e.g. on a different file-system if
;; you lack space on the file system your home directory is on.
(define root (fetch-manifest-repositories manifest))

;; Finds all Android.bp files from which to import, takes ~10s on warm
;; cache, significantly longer on cold cache and spiny disks.
(define bp-files (get-all-bp-files root))

;; Creates the shared structure: a map of modules and variables to the
;; files that define them, so we can find them, without actually parsing
;; everything.  Will take ~5mins on an HDD, could be faster on SSD.
(define bp-maps (get-bp-maps bp-files))

;; Run the import.  The last two arguments are the names of files that
;; will be generated.
(import-recursively manifest root bp-maps '("adb" "fastboot")
                    "sources.scm" "packages.scm")
```

That would import adb and fastboot

You can see the result of importing a bit more than that at
https://framagit.org/tyreunom/guix-android/-/blob/master/android/packages/android-tools.scm
and
https://framagit.org/tyreunom/guix-android/-/blob/master/android/packages/android-sources.scm

It's pretty great but missing a few things. These packages don't have a
home page, synopsis, description, ... and I don't see a way to easily
import that. Although I could manually add them, the work will have to
be done again at each update. Every month, Google releases a new
Android revision (not necessarily a major version) that modifies some
sources, and can impact the set of dependencies we need to build.

I was aiming at having an importer that was able to recreate all the
files with no human intervention. I think it's the only sane way to
take care of all those packages.

Currently I think all of android/build, android/build-system,
android/import, android/packages/android-{headers,tools,sources}.scm
and android/packages/{bison,clang}.scm can be upstreamed. The rest is
wip or broken packages.

Do you have any ideas how we could plan upstreaming all this?
Especially, how could the importer fit in the current Guix
infrastructure?

Thoughts?

Upstreaming my Android channel

[Nathan Dehnel]

Exciting!

Re: Upstreaming my Android channel

[Maxim Cournoyer]

Hi Julien,

Julien Lepiller <julien@lepiller.eu> writes:

> Hi Guix!
>
> I have been working on updating Android packages for some time now. I
> think it's at a point I'm confident I can share and move them upstream.
> My work is currently in a separate channel at
> https://framagit.org/tyreunom/guix-android. I have been working on the
> SDK and tools, which we already have. We currently are limited to
> Android 7 versions, because later Android versions do not use the same
> build system anymore.
>
> Since then, Android uses the Soong build system, based on blueprint.
> It's pretty complex, and specific. I tried to build it, but it assumes
> too many things to be useful. Instead, I "reimplemented" it by
> following what a Nix contributor did in soongnix:
> https://github.com/danielfullmer/soongnix
>
> I created a soong-build-system that has a few modules for cc, art and
> java types of packages. It supports building most packages from the SDK
> and other Android tools: aapt, appt2, adb, aidl, apksigner, dexdump,
> dmtracedump, dx, etc1tool, fastboot, hprof-conv, libaapt2_jni,
> split-select, zipalign.

This sounds good!

> I have an importer that is a bit different from the existing importers.
> First, it needs to checkout ~30GB of git repositories (and that's with
> --depth=1 for all of them!), explore the set of Android.bp files (used
> by soong) and create one package for each package in these Android.bp
> files. That creates packages that can be a bit smaller than expected.
> Instead of printing, the importer creates two files: one for the source
> definitions (many packages share the same source) and one for the
> packages and their dependencies.

30 GiB!  Wow.  Is there no lighter means to get at the needed data?
Perhaps some tarball snapshot with test data filtered or similar?  I
hope this data is at least gets reused (cached) between invocations?
But I'd expect even just updating such a large set of git repository
would be slow... potentially slowing each invocation of the importer?
Or do you handle this?

> Since I have all of that in a channel, the importer doesn't work from
> the command-line, but I have a relatively simple API that you can use
> to import packages:
>
> ```
> ,use (android import repo)
> (define manifest (get-manifest))
>
> ;; Fetch repositories that form the android distribution.
> ;; This is very big (> 28GB) and will take some time.  If it fails
> ;; because of a network issue, restart that command again until
> ;; everything is downloaded.
> ;;
> ;; You can also add an optional destination directory where the
> ;; sources are to be downloaded, e.g. on a different file-system if
> ;; you lack space on the file system your home directory is on.
> (define root (fetch-manifest-repositories manifest))
>
> ;; Finds all Android.bp files from which to import, takes ~10s on warm
> ;; cache, significantly longer on cold cache and spiny disks.
> (define bp-files (get-all-bp-files root))
>
> ;; Creates the shared structure: a map of modules and variables to the
> ;; files that define them, so we can find them, without actually parsing
> ;; everything.  Will take ~5mins on an HDD, could be faster on SSD.
> (define bp-maps (get-bp-maps bp-files))
>
> ;; Run the import.  The last two arguments are the names of files that
> ;; will be generated.
> (import-recursively manifest root bp-maps '("adb" "fastboot")
>                     "sources.scm" "packages.scm")
> ```
>
> That would import adb and fastboot

Cool!

> You can see the result of importing a bit more than that at
> https://framagit.org/tyreunom/guix-android/-/blob/master/android/packages/android-tools.scm
> and
> https://framagit.org/tyreunom/guix-android/-/blob/master/android/packages/android-sources.scm
>
> It's pretty great but missing a few things. These packages don't have a
> home page, synopsis, description, ... and I don't see a way to easily
> import that. Although I could manually add them, the work will have to
> be done again at each update. Every month, Google releases a new
> Android revision (not necessarily a major version) that modifies some
> sources, and can impact the set of dependencies we need to build.
>
> I was aiming at having an importer that was able to recreate all the
> files with no human intervention. I think it's the only sane way to
> take care of all those packages.

Perhaps there's another web site (like pkg.go.dev for Go packages)
holding such metadata that we could query to fill the missing blanks?

> Currently I think all of android/build, android/build-system,
> android/import, android/packages/android-{headers,tools,sources}.scm
> and android/packages/{bison,clang}.scm can be upstreamed. The rest is
> wip or broken packages.
>
> Do you have any ideas how we could plan upstreaming all this?
> Especially, how could the importer fit in the current Guix
> infrastructure?

If the quirks (30 GiB checkout for one) are well documented, also
perhaps directly interactively (warning: the importer needs to fetch
about 30 GiB of sources; hit enter to proceed), I don't see why the
importer couldn't sit right next to the others!

Thank you!

Maxim