Re: (Re-) Designing extractong-downaloder

[pukkamustard <pukkamustard@posteo.net>]

Hi Hartmut,

Hartmut Goebel <h.goebel@crazy-compilers.com> writes:

> I'm about pick up work on „extracting downloader“ and the rebar build
> system (for erlang),

I'm very much looking forward to this!

> The basic idea behind „extracting downloader“ is as follows: Packages
> provided by hex.pm (the distribution repository for erlang and elixir
> packages) are tar-archives containing some meta-data files and the
> actual source (contents.tar.gz), see example below, So the ideas was
> to only store the contents.tar.gz (instead of requiring an additional
> unpacking step).

Why use the source from hex.pm at all? Would it be possible to just
fetch the hex.pm archive when importing a package, read the
metadata.config file and then try and use upstream source (e.g. GitHub)?

The hex.pm metadata.config file does not seem to exactly specify the
upstream source. We would need some heuristics to figure this out. But
maybe we could find a heuristic that works well enough? This would solve
the double-archive problem.

For packages where the heuristics fails we fallback and use the source
as provided from hex.pm (unextracted) and use an additional build phase
to do the double extraction? If this only affects a few packages then
storing the source double-archived does not seem so bad.

Thanks,
pukkamustard