Large git repositories

Large git repositories

[Andreas Enge]

Hello,

I am trying to package the droid font family. It appears to be available
only via the android git at:
   https://android.googlesource.com/platform/frameworks/base/+/android-4.4.4_r2.0.1/data/fonts/
where
   https://android.googlesource.com/platform/frameworks/base.git
defines the git repository.

This page contains a tgz download link:
   https://android.googlesource.com/platform/frameworks/base/+archive/android-4.4.4_r2.0.1/data/fonts.tar.gz
Unfortunately, the file seems to be generated on the fly and have a
different hash value each time.

The git-reference method seems to clone all of the repository, which has
about 1.5 GB. I tried several methods by hand, such as
   git clone -b android-4.4.4_r2.0.1 https://android.googlesource.com/platform/frameworks/base.git data/fonts
or
   git init; git fetch https://android.googlesource.com/platform/frameworks/base.git android-4.4.4_r2.0.1
but all of them apparently download the full repository.

I also tried
   git archive --remote=https://android.googlesource.com/platform/frameworks/base.git android-4.4.4_r2.0.1 data/fonts
but the git server apparently does not support this:
fatal: Operation not supported by protocol.
(and I am also not sure that the output would have a fixed hash).

Do you see any solution to this problem?

Andreas

Re: Large git repositories

[Andreas Enge]

It looks like:
   --depth=1 --single-branch
could be useful parameters for "git clone".

Andreas

Re: Large git repositories

[Andreas Enge]

Finally, I also tried a git "sparse checkout" to only obtain the data/fonts/
subdirectory. It does not seem to do what I need, as it still claims it will
download 1.5 GB. My guess is that it filters the files after the download.

Andreas

Re: Large git repositories

[Ludovic Courtès]

Andreas Enge <andreas@enge.fr> skribis:

> It looks like:
>    --depth=1 --single-branch
> could be useful parameters for "git clone".

Yes, that’s what came to mind.

If it happens to be useful, we could extend ‘git-reference’ with a list
of extra parameters to pass to ‘git’.

Ludo’.

Re: Large git repositories

[Amirouche Boubekki]

[-- Attachment #1: Type: text/plain, Size: 1957 bytes --]

On Mon Jan 12 2015 at 11:03:48 PM Andreas Enge <andreas@enge.fr> wrote:

> Hello,
>
> I am trying to package the droid font family. It appears to be available
> only via the android git at:
>    https://android.googlesource.com/platform/frameworks/base/+
> /android-4.4.4_r2.0.1/data/fonts/
> where
>    https://android.googlesource.com/platform/frameworks/base.git
> defines the git repository.
>
> This page contains a tgz download link:
>    https://android.googlesource.com/platform/frameworks/base/+
> archive/android-4.4.4_r2.0.1/data/fonts.tar.gz
> Unfortunately, the file seems to be generated on the fly and have a
> different hash value each time.
>
> The git-reference method seems to clone all of the repository, which has
> about 1.5 GB. I tried several methods by hand, such as
>    git clone -b android-4.4.4_r2.0.1 https://android.googlesource.
> com/platform/frameworks/base.git data/fonts
> or
>    git init; git fetch https://android.googlesource.
> com/platform/frameworks/base.git android-4.4.4_r2.0.1
> but all of them apparently download the full repository.
>
> I also tried
>    git archive --remote=https://android.googlesource.com/platform/
> frameworks/base.git android-4.4.4_r2.0.1 data/fonts
> but the git server apparently does not support this:
> fatal: Operation not supported by protocol.
> (and I am also not sure that the output would have a fixed hash).
>
> Do you see any solution to this problem?
>

Google web fonts has droid font available in zip format via http [1],
not sure if the link is works across updates.

There is also the mercurial repository hosting all web fonts [2]. It's
quite big 4,1G and AFAIK there is no “hg clone --depth=1”


[1]
http://www.google.com/fonts#UsePlace:use/Collection:Droid+Sans:400,700|Droid+Sans+Mono|Droid+Serif:400,700,400italic,700italic

[2] https://code.google.com/p/googlefontdirectory/source/checkout

Andreas
>
>
>

[-- Attachment #2: Type: text/html, Size: 3397 bytes --]

Re: Large git repositories

[Andreas Enge]

Hello! Sorry for my late reply, I have been working on something else in the
meantime.

On Tue, Jan 13, 2015 at 01:31:51PM +0000, Amirouche Boubekki wrote:
> Google web fonts has droid font available in zip format via http [1],
> not sure if the link is works across updates.

Thanks, that could be interesting (assuming that zips do not contain time stamps).

On Thu, Jan 22, 2015 at 01:33:57PM +0000, Amirouche Boubekki wrote:
> This is again regarding the issue of downloading droid font in particular.
> This can be done via github with links like the following:
> https://github.com/android/platform_frameworks_base/raw/
> feef9887e8f8eb6f64fc1b4552c02efb5755cdc1/data/fonts/DroidSans.ttf

Nice, thanks! What does the "feef..." stand for? I assumed it would be the
git commit hash, but there is none of this form. The problem here is that
we need about 20 different files for the full family including different
scripts.

On Tue, Jan 13, 2015 at 11:32:25AM +0100, Ludovic Courtès wrote:
> Andreas Enge <andreas@enge.fr> skribis:
> > It looks like:
> >    --depth=1 --single-branch
> > could be useful parameters for "git clone".
> Yes, that’s what came to mind.
> If it happens to be useful, we could extend ‘git-reference’ with a list
> of extra parameters to pass to ‘git’.

Actually, I have something else in mind. Currently, we accept commit hashes
or branch/tag names. How about adding an additional parameter "branch?"
(in various places, when I looked at the code I think I understood where
and would be willing to propose a patch)? If it is set, we can directly
specify the branch/tag at clone, without the need for an additional checkout,
and could also pass the parameters "--depth=1 --single-branch".

Alternatively, we could also automatically distinguish commit hashes from
branch/tag names, for instance, by imposing that always the long commit hash
is used and assuming that a branch/tag name never equals a hexidecimal string
of the corresponding length. But I prefer the additional parameter.

In our case, we could then git clone the 1.5 GB once and for all and delete
everything outside data/fonts (and in particular the huge .git). The result
would be the source for a variety of android fonts (maybe to be placed into
font-android.scm or the like).

What do you think?

Andreas

Re: Large git repositories

[Ludovic Courtès]

Andreas Enge <andreas@enge.fr> skribis:

> On Tue, Jan 13, 2015 at 11:32:25AM +0100, Ludovic Courtès wrote:
>> Andreas Enge <andreas@enge.fr> skribis:
>> > It looks like:
>> >    --depth=1 --single-branch
>> > could be useful parameters for "git clone".
>> Yes, that’s what came to mind.
>> If it happens to be useful, we could extend ‘git-reference’ with a list
>> of extra parameters to pass to ‘git’.
>
> Actually, I have something else in mind. Currently, we accept commit hashes
> or branch/tag names.

It’s already possible, in the ‘commit’ field, to use anything that ‘git
checkout’ would accept, which includes a commit hash, a tag, or a branch
name.

> How about adding an additional parameter "branch?"  (in various
> places, when I looked at the code I think I understood where and would
> be willing to propose a patch)? If it is set, we can directly specify
> the branch/tag at clone, without the need for an additional checkout,
> and could also pass the parameters "--depth=1 --single-branch".
>
> Alternatively, we could also automatically distinguish commit hashes from
> branch/tag names, for instance, by imposing that always the long commit hash
> is used and assuming that a branch/tag name never equals a hexidecimal string
> of the corresponding length. But I prefer the additional parameter.

I see, that makes sense.

> In our case, we could then git clone the 1.5 GB once and for all and delete
> everything outside data/fonts (and in particular the huge .git). The result
> would be the source for a variety of android fonts (maybe to be placed into
> font-android.scm or the like).
>
> What do you think?

Those changes you propose would be fine, but still, it’s extra
complexity and I’m worried that it would be used essentially in a just
one place.  Is there really no other way to get these fonts?

Thanks,
Ludo’.