Re: Move to git is imminent - awaiting Stefan's approval

Re: Move to git is imminent - awaiting Stefan's approval

[Angelo Graziosi]

Eli Zaretskii wrote:
> 8. There's the emacs-bzr-version whose value gets copied into the bug
>     reports.  This should be replaced by the suitable git equivalent,
>     or else the bug reports (of which we have quite a few each day)
>     will not identify the version correctly.

Perhaps I have a similar question (or the same?)

Usually I build emacs trunk weekly. My local copy of trunk has been 
created, the first time, with

$ bzr checkout --lightweight http://bzr.savannah.gnu.org/r/emacs/trunk 
emacs-trunk


and then updating "daily" it with

$ cd emacs-trunk
$ bzr up

which prints each time the trunk revision number, e.g.

[...]
All changes applied successfully. 

Updated to revision 115890 of branch 
http://bzr.savannah.gnu.org/r/emacs/trunk


When I find a problem ("bug"), I flag it to this list saying something 
like this

"In trunk rev. 115890 there is this problem... BLA BLA BLA..."


Now trying

$ git clone git://git.savannah.gnu.org/emacs.git emacs.git


I don't get any revision number.. So in my future bug report how the 
trunk will be identified?

Beside this, my local copy of BZR repository has dimension

$ du -s emacs-trunk/
126M	emacs-trunk/

instead the git repository has dimension

$ du -s emacs.git/
1.1G	emacs.git/

it contains an emacs.git/.git directory of about 930M which probably 
doesn't need for the Emacs build (bootstrap).

This is very annoying...


Ciao,
  Angelo.

Re: Move to git is imminent - awaiting Stefan's approval

[David Kastrup]

Angelo Graziosi <angelo.graziosi@alice.it> writes:

> Usually I build emacs trunk weekly. My local copy of trunk has been
> created, the first time, with
>
> $ bzr checkout --lightweight http://bzr.savannah.gnu.org/r/emacs/trunk
> emacs-trunk
>
>
> and then updating "daily" it with
>
> $ cd emacs-trunk
> $ bzr up

"lightweight"

> When I find a problem ("bug"), I flag it to this list saying something
> like this
>
> "In trunk rev. 115890 there is this problem... BLA BLA BLA..."
>
>
> Now trying
>
> $ git clone git://git.savannah.gnu.org/emacs.git emacs.git
>
>
> I don't get any revision number.. So in my future bug report how the
> trunk will be identified?

The commit ids are unique.  It's usually easiest to quote the entire
SHA128 via copy&paste, but the first 16 digits or so are usually
sufficient.

> Beside this, my local copy of BZR repository has dimension
>
> $ du -s emacs-trunk/
> 126M	emacs-trunk/

"lightweight"

> instead the git repository has dimension
>
> $ du -s emacs.git/
> 1.1G	emacs.git/
>
> it contains an emacs.git/.git directory of about 930M which probably
> doesn't need for the Emacs build (bootstrap).

For the build?  No.  git clone has the following options:

        --depth <depth>
           Create a shallow clone with a history truncated to the specified
           number of revisions. A shallow repository has a number of
           limitations (you cannot clone or fetch from it, nor push from nor
           into it), but is adequate if you are only interested in the recent
           history of a large project with a long history, and would want to
           send in fixes as patches.

       --[no-]single-branch
           Clone only the history leading to the tip of a single branch,
           either specified by the --branch option or the primary branch
           remote’s HEAD points at. When creating a shallow clone with the
           --depth option, this is the default, unless --no-single-branch is
           given to fetch the histories near the tips of all branches. Further
           fetches into the resulting repository will only update the
           remote-tracking branch for the branch this option was used for the
           initial cloning. If the HEAD at the remote did not point at any
           branch when --single-branch clone was made, no remote-tracking
           branch is created.

In addition, once you have a local repository, something like

git clone existing_workdir new_workdir

will do the cloning by using hard links, taking almost no time and no
space.  I often do that for one-off testing of stuff while my "main"
workdir is busy compiling stuff.

-- 
David Kastrup

Re: Move to git is imminent - awaiting Stefan's approval

[Nix]

On 6 Jan 2014, David Kastrup stated:

> The commit ids are unique.  It's usually easiest to quote the entire
> SHA128 via copy&paste, but the first 16 digits or so are usually
> sufficient.

You can just use 'git log --abbrev-commit' to get commit IDs abbreviated
(enough for nonambiguity and no further).

If you want to enable this forever, git 1.7.6+ lets you say

git config log.abbrevCommit true

(add --global or --system to make it apply to all repositories for this
user, or all repositories on the system, respectively).

> In addition, once you have a local repository, something like
>
> git clone existing_workdir new_workdir
>
> will do the cloning by using hard links, taking almost no time and no
> space.  I often do that for one-off testing of stuff while my "main"
> workdir is busy compiling stuff.

In addition in addition, git-new-workdir (a script in git's contrib/
directory, likely to be installed by default eventually in some form or
other) will let you check out more than one branch at once on the same
repository.

-- 
NULL && (void)

Re: Move to git is imminent - awaiting Stefan's approval

[Andreas Schwab]

Angelo Graziosi <angelo.graziosi@alice.it> writes:

> Now trying
>
> $ git clone git://git.savannah.gnu.org/emacs.git emacs.git
>
>
> I don't get any revision number.. So in my future bug report how the trunk
> will be identified?

A good way to identify a git branch is to use git describe.  However it
prefers to have an annotated (release) tag available to base on, which
we currently don't have (bzr doesn't have the concept of annotated
tags).

> Beside this, my local copy of BZR repository has dimension
>
> $ du -s emacs-trunk/
> 126M	emacs-trunk/
>
> instead the git repository has dimension
>
> $ du -s emacs.git/
> 1.1G	emacs.git/

By default git downloads the history of all branches.  You can use
--single-branch to only clone a single branch.  This is the default if
you create a shallow clone.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Move to git is imminent - awaiting Stefan's approval

[Angelo Graziosi]

Il 07/01/2014 0.01, Andreas Schwab ha scritto:
> Angelo Graziosi <angelo.graziosi@alice.it> writes:
>
>> Now trying
>>
>> $ git clone git://git.savannah.gnu.org/emacs.git emacs.git
>>
>>
>> I don't get any revision number.. So in my future bug report how the trunk
>> will be identified?
>
> A good way to identify a git branch is to use git describe.  However it
> prefers to have an annotated (release) tag available to base on, which
> we currently don't have (bzr doesn't have the concept of annotated
> tags).
>
>> Beside this, my local copy of BZR repository has dimension
>>
>> $ du -s emacs-trunk/
>> 126M	emacs-trunk/
>>
>> instead the git repository has dimension
>>
>> $ du -s emacs.git/
>> 1.1G	emacs.git/
>
> By default git downloads the history of all branches.  You can use
> --single-branch to only clone a single branch.  This is the default if
> you create a shallow clone.

David Kastrup wrote:

> For the build?  No.  git clone has the following options:
>
>         --depth <depth>
...
> --[no-]single-branch
...

OK, trying

$ git clone --single-branch git://git.savannah.gnu.org/emacs.git emacs.git

the local repository has dimension 1.1G, so probably I need a more 
explicit example...

Instead, trying

$ git clone --depth 1 git://git.savannah.gnu.org/emacs.git emacs.git

the local repository has dimension 160M which is a little greater than 
my current BZR (checkout --lightweight) repository (126M), but acceptable...

Suppose now I want identify which "revision" introduced a bug. I am able 
to do this with a binary search using the BZR command

$ bzr up -r ARG

How can I do this with the "minimal" clone done with GIT?


I hope someone creates a Wiki page with all these doubts clarified with 
explicit examples... :-)


Thanks,
  Angelo.

Git tips for Emacs development (was Re: Move to git is imminent - awaiting Stefan's approval)

[Nic Ferrier]

Angelo Graziosi <angelo.graziosi@alice.it> writes:

> Instead, trying
>
> $ git clone --depth 1 git://git.savannah.gnu.org/emacs.git emacs.git
>
> the local repository has dimension 160M which is a little greater than 
> my current BZR (checkout --lightweight) repository (126M), but acceptable...
>
> Suppose now I want identify which "revision" introduced a bug. I am able 
> to do this with a binary search using the BZR command
>
> $ bzr up -r ARG

What does ARG do in this context?

> How can I do this with the "minimal" clone done with GIT?
>
>
> I hope someone creates a Wiki page with all these doubts clarified with 
> explicit examples... :-)

http://www.emacswiki.org/emacs/GitForEmacsDevs

is the start of one.

Do add questions.


Nic

Re: Git tips for Emacs development (was Re: Move to git is imminent - awaiting Stefan's approval)

[Eli Zaretskii]

> From: Nic Ferrier <nferrier@ferrier.me.uk>
> Date: Tue, 07 Jan 2014 10:37:44 +0000
> Cc: Andreas Schwab <schwab@linux-m68k.org>, emacs <emacs-devel@gnu.org>
> 
> > $ bzr up -r ARG
> 
> What does ARG do in this context?

It's the revision to which the tree will be updated.

Re: Move to git is imminent - awaiting Stefan's approval

[Sven Axelsson]

On 7 January 2014 11:24, Angelo Graziosi <angelo.graziosi@alice.it> wrote:
>
>
> OK, trying
>
> $ git clone --single-branch git://git.savannah.gnu.org/emacs.git emacs.git
>
> the local repository has dimension 1.1G, so probably I need a more explicit example...

This clones only the main branch (master). Since this is where almost
all Emacs history is located it won't make much difference size wise.

> Instead, trying
>
> $ git clone --depth 1 git://git.savannah.gnu.org/emacs.git emacs.git
>
> the local repository has dimension 160M which is a little greater than my current BZR (checkout --lightweight) repository (126M), but acceptable...
>
> Suppose now I want identify which "revision" introduced a bug. I am able to do this with a binary search using the BZR command
>
> $ bzr up -r ARG
>
> How can I do this with the "minimal" clone done with GIT?

Git does not have any concept like Bazaar's lightweight checkouts -
nor does any other dvcd afaik. Depth 1 truncates all history except
for the most recent commit. From `man git clone`:

A shallow repository has a number of limitations (you cannot clone or
fetch from it, nor push from nor into it), but is adequate if you are
only interested in the recent history of a large project with a long
history, and would want to send in fixes as patches.


-- 
Sven Axelsson
++++++++++[>++++++++++>+++++++++++>++++++++++>++++++
>++++<<<<<-]>++++.+.++++.>+++++.>+.<<-.>>+.>++++.<<.
+++.>-.<<++.>>----.<++.>>>++++++.<<<<.>>++++.<----.

Re: Move to git is imminent - awaiting Stefan's approval

[Stephen Berman]

On Tue, 7 Jan 2014 13:27:22 +0100 Sven Axelsson <sven.axelsson@gmail.com> wrote:

> On 7 January 2014 11:24, Angelo Graziosi <angelo.graziosi@alice.it> wrote:
>>
>>
>> OK, trying
>>
>> $ git clone --single-branch git://git.savannah.gnu.org/emacs.git emacs.git
>>
>> the local repository has dimension 1.1G, so probably I need a more explicit
>> example...
>
> This clones only the main branch (master). Since this is where almost
> all Emacs history is located it won't make much difference size wise.

Is it correct that main or master is the same thing as the Emacs trunk
branch?  If so, I wonder why the git repository containing just this is
so large: my bzr Emacs repository only contains the trunk (as a bound
branch), several local branches of it, and the emacs-24 branch; the
.bzr/ directory of the repository, containing all the history, is only
428 MiB, not 1.1 GiB (the trunk directory itself contains an additional
129.3 MiB, but the total is still half the size of the git repo).  Why
is the git repository so much larger than the bzr repository?

Steve Berman

Re: Move to git is imminent - awaiting Stefan's approval

[Andreas Schwab]

Stephen Berman <stephen.berman@gmx.net> writes:

> Why is the git repository so much larger than the bzr repository?

The git repositories on savannah appear to be packed rather badly.  A
reasonably well packed emacs repository should have around 350MB.

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: Move to git is imminent - awaiting Stefan's approval

[Stephen Berman]

On Tue, 07 Jan 2014 15:57:05 +0100 Andreas Schwab <schwab@suse.de> wrote:

> Stephen Berman <stephen.berman@gmx.net> writes:
>
>> Why is the git repository so much larger than the bzr repository?
>
> The git repositories on savannah appear to be packed rather badly.  A
> reasonably well packed emacs repository should have around 350MB.

Ok.  If the packing can be improved, I think it would be good for that
to happen before finalizing the move to git.

Steve Berman

Re: Move to git is imminent - awaiting Stefan's approval

[Eli Zaretskii]

> From: Andreas Schwab <schwab@suse.de>
> Date: Tue, 07 Jan 2014 15:57:05 +0100
> Cc: Angelo Graziosi <angelo.graziosi@alice.it>,
> 	Sven Axelsson <sven.axelsson@gmail.com>, emacs <emacs-devel@gnu.org>
> 
> The git repositories on savannah appear to be packed rather badly.  A
> reasonably well packed emacs repository should have around 350MB.

Can repacking be done locally, or does it have to be on savannah?  If
the former, how to do it?

Re: Move to git is imminent - awaiting Stefan's approval

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

> Can repacking be done locally, or does it have to be on savannah?

You can always repack locally.  It's just that during clone, you inherit
the packing of the remote, since existing deltas are not recomputed.

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: Move to git is imminent - awaiting Stefan's approval

[Angelo Graziosi]

Il 07/01/2014 17.35, Andreas Schwab ha scritto:
> Eli Zaretskii <eliz@gnu.org> writes:
>
>> Can repacking be done locally, or does it have to be on savannah?
>
> You can always repack locally.  It's just that during clone, you inherit
> the packing of the remote, since existing deltas are not recomputed.

OK, but Eli asked how do it... So how?


Thanks,
   Angelo.

Re: Move to git is imminent - awaiting Stefan's approval

[Andreas Schwab]

Angelo Graziosi <angelo.graziosi@alice.it> writes:

> Il 07/01/2014 17.35, Andreas Schwab ha scritto:
>> Eli Zaretskii <eliz@gnu.org> writes:
>>
>>> Can repacking be done locally, or does it have to be on savannah?
>>
>> You can always repack locally.  It's just that during clone, you inherit
>> the packing of the remote, since existing deltas are not recomputed.
>
> OK, but Eli asked how do it... So how?

See git-gc(1)

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: Move to git is imminent - awaiting Stefan's approval

[Eli Zaretskii]

> From: Andreas Schwab <schwab@suse.de>
> Cc: Eli Zaretskii <eliz@gnu.org>,  stephen.berman@gmx.net,  sven.axelsson@gmail.com,  emacs-devel@gnu.org
> Date: Tue, 07 Jan 2014 17:41:43 +0100
> 
> Angelo Graziosi <angelo.graziosi@alice.it> writes:
> 
> > Il 07/01/2014 17.35, Andreas Schwab ha scritto:
> >> Eli Zaretskii <eliz@gnu.org> writes:
> >>
> >>> Can repacking be done locally, or does it have to be on savannah?
> >>
> >> You can always repack locally.  It's just that during clone, you inherit
> >> the packing of the remote, since existing deltas are not recomputed.
> >
> > OK, but Eli asked how do it... So how?
> 
> See git-gc(1)

Thanks.  "git gc" gets me from 1.18GB to 1.1GB.  I tried --aggressive,
but that ran out of memory (trying to allocate 1GB) after some time
and errored out.

Re: Move to git is imminent - awaiting Stefan's approval

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

> Thanks.  "git gc" gets me from 1.18GB to 1.1GB.  I tried --aggressive,
> but that ran out of memory (trying to allocate 1GB) after some time
> and errored out.

That's strange:

oscar@qcore:~/dev/emacs/emacs$ time git gc
Counting objects: 734772, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (145032/145032), done.
Writing objects: 100% (734772/734772), done.
Total 734772 (delta 589118), reused 734312 (delta 588658)
Checking connectivity: 734772, done.

oscar@qcore:~/dev/emacs/emacs$ du -sh .git
289M    .git
oscar@qcore:~/dev/emacs/emacs$ git branch -a | wc
    127     130    4667


Maybe you instructed git to download extra info (such as the meta-data
the fast-import/export process uses and is useful for relating git
commits to their corresponding bzr commits.)

Re: Move to git is imminent - awaiting Stefan's approval

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Tue, 07 Jan 2014 18:23:48 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Thanks.  "git gc" gets me from 1.18GB to 1.1GB.  I tried --aggressive,
> > but that ran out of memory (trying to allocate 1GB) after some time
> > and errored out.
> 
> That's strange:
> 
> oscar@qcore:~/dev/emacs/emacs$ time git gc
> Counting objects: 734772, done.
> Delta compression using up to 4 threads.
> Compressing objects: 100% (145032/145032), done.
> Writing objects: 100% (734772/734772), done.
> Total 734772 (delta 589118), reused 734312 (delta 588658)
> Checking connectivity: 734772, done.
> 
> oscar@qcore:~/dev/emacs/emacs$ du -sh .git
> 289M    .git
> oscar@qcore:~/dev/emacs/emacs$ git branch -a | wc
>     127     130    4667
> 
> 
> Maybe you instructed git to download extra info (such as the meta-data
> the fast-import/export process uses and is useful for relating git
> commits to their corresponding bzr commits.)

Nope, just "git clone" followed by pull's.

I now tried on fencepost.gnu.org, a 64-bit GNU/Linux system, with
similar results: gc goes from 1.78GB down to 1.6GB.  --aggressive also
fails there, funnily enough.

Re: Move to git is imminent - awaiting Stefan's approval

[Stephen J. Turnbull]

Eli Zaretskii writes:

 > > See git-gc(1)

git-gc does not necessarily repack existing packs according to the
doc.

Try git-repack.

Or wait for me to try it ... it's past my bedtime, will report
tomorrow if somebody doesn't beat me to it.

Re: Move to git is imminent - awaiting Stefan's approval

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Cc: Andreas Schwab <schwab@suse.de>,
>     stephen.berman@gmx.net,
>     emacs-devel@gnu.org,
>     sven.axelsson@gmail.com,
>     angelo.graziosi@alice.it
> Date: Wed, 08 Jan 2014 02:24:02 +0900
> 
> Eli Zaretskii writes:
> 
>  > > See git-gc(1)
> 
> git-gc does not necessarily repack existing packs according to the
> doc.
> 
> Try git-repack.

I thought "git gc" runs git-repack, no?

Re: Move to git is imminent - awaiting Stefan's approval

[Sven Axelsson]

On 7 January 2014 19:05, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: "Stephen J. Turnbull" <stephen@xemacs.org>
>> Cc: Andreas Schwab <schwab@suse.de>,
>>     stephen.berman@gmx.net,
>>     emacs-devel@gnu.org,
>>     sven.axelsson@gmail.com,
>>     angelo.graziosi@alice.it
>> Date: Wed, 08 Jan 2014 02:24:02 +0900
>>
>> Eli Zaretskii writes:
>>
>>  > > See git-gc(1)
>>
>> git-gc does not necessarily repack existing packs according to the
>> doc.
>>
>> Try git-repack.
>
> I thought "git gc" runs git-repack, no?

No. git-repack is often a very expensive operation, so it is never run
automatically, which git-gc can be.

-- 
Sven Axelsson
++++++++++[>++++++++++>+++++++++++>++++++++++>++++++
>++++<<<<<-]>++++.+.++++.>+++++.>+.<<-.>>+.>++++.<<.
+++.>-.<<++.>>----.<++.>>>++++++.<<<<.>>++++.<----.

Re: Move to git is imminent - awaiting Stefan's approval

[Stephen J. Turnbull]

Sven Axelsson writes:
 > On 7 January 2014 19:05, Eli Zaretskii <eliz@gnu.org> wrote:
 > >> From: "Stephen J. Turnbull" <stephen@xemacs.org>

 > >> git-gc does not necessarily repack existing packs according to the
 > >> doc.
 > >>
 > >> Try git-repack.
 > >
 > > I thought "git gc" runs git-repack, no?

It kinda pisses me off that you complain about the excessive detail of
git documentation, and then ask others to tell you about the details
you could read in the documentation.

 > No. git-repack is often a very expensive operation, so it is never
 > run automatically, which git-gc can be.

Specifically, according to the help, git-gc removes garbage (in the
usual sense of a tracing collector) and "compresses" (ie, packs) loose
objects using delta compression (and then the result is deflated as
with loose objects, I believe).  It will repack the packs into *one*
pack (presumably with better delta compression) if the number of packs
is greater than the value of gc.autopacklimit (in .git/config;
default: 50).

Re: Move to git is imminent - awaiting Stefan's approval

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,
>     schwab@suse.de,
>     stephen.berman@gmx.net,
>     angelo.graziosi@alice.it,
>     emacs <emacs-devel@gnu.org>
> Date: Wed, 08 Jan 2014 03:49:24 +0900
> 
> Sven Axelsson writes:
>  > On 7 January 2014 19:05, Eli Zaretskii <eliz@gnu.org> wrote:
>  > >> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> 
>  > >> git-gc does not necessarily repack existing packs according to the
>  > >> doc.
>  > >>
>  > >> Try git-repack.
>  > >
>  > > I thought "git gc" runs git-repack, no?
> 
> It kinda pisses me off that you complain about the excessive detail of
> git documentation, and then ask others to tell you about the details
> you could read in the documentation.

I did read the documentation.  The docs of gc mentions git-repack, so
I thought the former runs the latter.  What part did I miss?

> Specifically, according to the help, git-gc removes garbage (in the
> usual sense of a tracing collector) and "compresses" (ie, packs) loose
> objects using delta compression (and then the result is deflated as
> with loose objects, I believe).  It will repack the packs into *one*
> pack (presumably with better delta compression) if the number of packs
> is greater than the value of gc.autopacklimit (in .git/config;
> default: 50).

And how was I supposed to know from this whether git-repack will or
won't be run, what with the word "repack" explicitly in this text (and
in other parts of that man page)?

Don't bother answering if you are still pissed off.

Re: Move to git is imminent - awaiting Stefan's approval

[Stephen J. Turnbull]

Eli Zaretskii writes:

 > I did read the documentation.  The docs of gc mentions git-repack, so
 > I thought the former runs the latter.  What part did I miss?

The fact that the documentation clearly puts conditions on when git-gc
runs git-repack.

 > And how was I supposed to know from this whether git-repack will or
 > won't be run, what with the word "repack" explicitly in this text (and
 > in other parts of that man page)?

Huh?  If there's a condition on whether it runs, then presumably
sometimes it doesn't run.  That's all that I deduced from it, and
advised you run git-repack manually.

Re: Move to git is imminent - awaiting Stefan's approval

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Cc: schwab@suse.de,
>     stephen.berman@gmx.net,
>     emacs-devel@gnu.org,
>     sven.axelsson@gmail.com,
>     angelo.graziosi@alice.it
> Date: Wed, 08 Jan 2014 12:47:29 +0900
> 
> Eli Zaretskii writes:
>  > I did read the documentation.  The docs of gc mentions git-repack, so
>  > I thought the former runs the latter.  What part did I miss?
> 
> The fact that the documentation clearly puts conditions on when git-gc
> runs git-repack.

I wouldn't qualify the gc manpage as "clear" on this matter, see
below.  But given what it does say, I _guessed_ that "git gc" _might_
run git-repack, and asked whether that wasn't so.  Why that pissed you
off I don't know.  You seem to have deduced a different conclusion,
but IMO that's just one more evidence that the man page is not really
clear on this matter.

>  > And how was I supposed to know from this whether git-repack will or
>  > won't be run, what with the word "repack" explicitly in this text (and
>  > in other parts of that man page)?
> 
> Huh?  If there's a condition on whether it runs, then presumably
> sometimes it doesn't run.

AFAIU, you refer to this part of your previous message:

> Specifically, according to the help, git-gc removes garbage (in the
> usual sense of a tracing collector) and "compresses" (ie, packs) loose
> objects using delta compression (and then the result is deflated as
> with loose objects, I believe).  It will repack the packs into *one*
> pack (presumably with better delta compression) if the number of packs
> is greater than the value of gc.autopacklimit (in .git/config;
> default: 50).

However, this comes from the part in the man page that describes the
"--auto" option to gc.  I didn't use and didn't mean to use that
option, so the above is not necessarily relevant to "git gc".

But even if this description is relevant when --auto is not used,
the above says that repacking will sometimes happen.

As to what happens regardless of any options, the man page says just
this:

  DESCRIPTION

    Runs a number of housekeeping tasks within the current repository,
    such as compressing file revisions (to reduce disk space and
    increase performance) and removing unreachable objects which may
    have been created from prior invocations of git add.

This says "such as", which doesn't imply an exhaustive list of tasks.
Moreover, it never says that "compressing file revisions" and
"removing unreachable objects" is not done by git-repack.  So the
above description does not contradict my conclusion, and definitely
doesn't present exact description of the conditions under which gc
will run repack.  Perhaps if I knew exactly what repack does, I could
have filled the voids, but I don't.

Finally, the "Configuration" section has this, among the rest:

  The optional configuration variable gc.packrefs determines if git gc
  runs git pack-refs. This can be set to "notbare" to enable it within
  all non-bare repos or it can be set to a boolean value. This
  defaults to true.

  The optional configuration variable gc.aggressiveWindow controls how
  much time is spent optimizing the delta compression of the objects
  in the repository when the --aggressive option is specified. The
  larger the value, the more time is spent optimizing the delta
  compression. See the documentation for the --window' option in
  git-repack(1) for more details. This defaults to 250.

Both of these configuration settings explicitly mention packing and/or
repacking.  Again, without knowing exactly what repack does, there's
no clarity here whether pack-refs is related to repack in any way,
shape or form (looking in the sources resolves this, see below).

Given these hints, IMO it was a reasonable guess that gc does run
repack at least under some conditions, when --auto is not used.  Since
these conditions are not defined completely by the man page, it was my
guess that gc, with or without the --aggressive option, might run
repack.  As it turns out, the second part is surely true.

> You read the source?  I find that hard to believe.

I didn't read the source in this case, only the documentation.

I don't understand the "hard to believe" part, though: I do have the
git's git repo on my disk, and you could easily imagine that I'm not
new to reading sources.

Having now looked in the sources, this is what I see:

	if (pack_refs && run_command_v_opt(pack_refs_cmd.argv, RUN_GIT_CMD))
		return error(FAILED_RUN, pack_refs_cmd.argv[0]);

	if (run_command_v_opt(reflog.argv, RUN_GIT_CMD))
		return error(FAILED_RUN, reflog.argv[0]);

	if (run_command_v_opt(repack.argv, RUN_GIT_CMD))
		return error(FAILED_RUN, repack.argv[0]);

	if (prune_expire) {
		argv_array_push(&prune, prune_expire);
		if (quiet)
			argv_array_push(&prune, "--no-progress");
		if (run_command_v_opt(prune.argv, RUN_GIT_CMD))
			return error(FAILED_RUN, prune.argv[0]);
	}

	if (run_command_v_opt(rerere.argv, RUN_GIT_CMD))
		return error(FAILED_RUN, rerere.argv[0]);

My reading of this is that "git gc" runs, in sequence, pack-refs
(conditioned on gc.packrefs), reflog, repack, prune (conditioned on
the --prune option), and finally rerere.  IOW, "git gc" does run
repack.  The only difference between --aggressive and non-aggressive
modes is here:

	if (aggressive) {
		argv_array_push(&repack, "-f");
		argv_array_push(&repack, "--depth=250");
		if (aggressive_window > 0)
			argv_array_pushf(&repack, "--window=%d", aggressive_window);
	}

That is, --aggressive just passes the -f switch to git-repack, which
tells it not to reuse deltas (i.e. recompute them anew, AFAIU).

> (The doc for --aggressive says nothing about repack, only that it
> optimizes harder.)

See above: the description of aggressiveWindow setting quite clearly
says that --aggressive does run repack.  And in fact it does, indeed.

> > Stephen can now be pissed off by himself.
> 
> Yup.  And I won't waste time on suppressing my irritation while
> helping with GitForEmacsDevs

Too bad.

Re: Move to git is imminent - awaiting Stefan's approval

[Thien-Thi Nguyen]

[-- Attachment #1: Type: text/plain, Size: 684 bytes --]

() Eli Zaretskii <eliz@gnu.org>
() Tue, 07 Jan 2014 19:08:04 +0200

   Thanks.  "git gc" gets me from 1.18GB to 1.1GB.  I tried
   --aggressive, but that ran out of memory (trying to allocate 1GB)
   after some time and errored out.

I had the same problem in a similar circumstance (git-bzr).  See:

 http://lists.gnu.org/archive/html/emacs-devel/2014-01/msg00329.html

for the resolution (briefly: use "git repack" w/ ‘--window-memory’).

-- 
Thien-Thi Nguyen
   GPG key: 4C807502
   (if you're human and you know it)
      read my lisp: (responsep (questions 'technical)
                               (not (via 'mailing-list)))
                     => nil

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: Move to git is imminent - awaiting Stefan's approval

[Sven Axelsson]

On 7 January 2014 19:02, Thien-Thi Nguyen <ttn@gnu.org> wrote:
> () Eli Zaretskii <eliz@gnu.org>
> () Tue, 07 Jan 2014 19:08:04 +0200
>
>    Thanks.  "git gc" gets me from 1.18GB to 1.1GB.  I tried
>    --aggressive, but that ran out of memory (trying to allocate 1GB)
>    after some time and errored out.
>
> I had the same problem in a similar circumstance (git-bzr).  See:
>
>  http://lists.gnu.org/archive/html/emacs-devel/2014-01/msg00329.html
>
> for the resolution (briefly: use "git repack" w/ ‘--window-memory’).
>

Using `git repack -a -d -f --window=250 --depth=250` as mentioned
earlier in the conversation brings the .git folder down to 211 Mb for
me.

-- 
Sven Axelsson
++++++++++[>++++++++++>+++++++++++>++++++++++>++++++
>++++<<<<<-]>++++.+.++++.>+++++.>+.<<-.>>+.>++++.<<.
+++.>-.<<++.>>----.<++.>>>++++++.<<<<.>>++++.<----.

Re: Move to git is imminent - awaiting Stefan's approval

[Andreas Schwab]

Sven Axelsson <sven.axelsson@gmail.com> writes:

> Using `git repack -a -d -f --window=250 --depth=250` as mentioned
> earlier in the conversation brings the .git folder down to 211 Mb for
> me.

That's basically what git gc --aggressive does.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Move to git is imminent - awaiting Stefan's approval

[Sven Axelsson]

On 7 January 2014 19:47, Andreas Schwab <schwab@linux-m68k.org> wrote:
> Sven Axelsson <sven.axelsson@gmail.com> writes:
>
>> Using `git repack -a -d -f --window=250 --depth=250` as mentioned
>> earlier in the conversation brings the .git folder down to 211 Mb for
>> me.
>
> That's basically what git gc --aggressive does.

Looking at the source for git-gc I see that you are indeed correct.
Specifically, `git gc --aggressive` will run `git repack -d -l -f
--depth=250 --window=250`, among other things. Maybe this setting is
more appropriate: using the -a flag can give a better pack, but may
cause extra network traffic when fetching the repo over a dumb
protocol. All this can be found in tha man pages.

-- 
Sven Axelsson
++++++++++[>++++++++++>+++++++++++>++++++++++>++++++
>++++<<<<<-]>++++.+.++++.>+++++.>+.<<-.>>+.>++++.<<.
+++.>-.<<++.>>----.<++.>>>++++++.<<<<.>>++++.<----.

Re: Move to git is imminent - awaiting Stefan's approval

[Eli Zaretskii]

> Date: Tue, 7 Jan 2014 20:07:06 +0100
> From: Sven Axelsson <sven.axelsson@gmail.com>
> Cc: emacs <emacs-devel@gnu.org>
> 
> >> Using `git repack -a -d -f --window=250 --depth=250` as mentioned
> >> earlier in the conversation brings the .git folder down to 211 Mb for
> >> me.
> >
> > That's basically what git gc --aggressive does.
> 
> Looking at the source for git-gc I see that you are indeed correct.
> Specifically, `git gc --aggressive` will run `git repack -d -l -f
> --depth=250 --window=250`, among other things. Maybe this setting is
> more appropriate: using the -a flag can give a better pack, but may
> cause extra network traffic when fetching the repo over a dumb
> protocol. All this can be found in tha man pages.

Which was why I interpreted them to mean that gc runs repack.

Stephen can now be pissed off by himself.

Re: Move to git is imminent - awaiting Stefan's approval

[Stephen J. Turnbull]

Eli Zaretskii writes:

 > Which was why I interpreted them to mean that gc runs repack.

You read the source?  I find that hard to believe.  (The doc for
--aggressive says nothing about repack, only that it optimizes
harder.)

 > Stephen can now be pissed off by himself.

Yup.  And I won't waste time on suppressing my irritation while
helping with GitForEmacsDevs, since you're the main person I would
(normally) care to make such an effort for.

Re: Move to git is imminent - awaiting Stefan's approval

[Angelo Graziosi]

Il 07/01/2014 18.08, Eli Zaretskii ha scritto:
> Thanks.  "git gc" gets me from 1.18GB to 1.1GB.  I tried --aggressive,
> but that ran out of memory (trying to allocate 1GB) after some time
> and errored out.
>

Here (on Cygwin) it worked... Firstly, I did

$ git clone git://git.savannah.gnu.org/emacs.git emacs.git

and its size was

$ du -s emacs.git/
1,1G	emacs.git/

then I did

$ cd emacs.git/
$ git gc --aggressive
Counting objects: 735546, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (734446/734446), done.
Writing objects: 100% (735546/735546), done.
Total 735546 (delta 589814), reused 143177 (delta 0)
Checking connectivity: 735546, done.

which took almost 3 hours to be completed.

And the size is:

$ cd ..

$ du -s emacs.git/
327M	emacs.git/

which seems very good!


Ciao,
   Angelo.

Re: Move to git is imminent - awaiting Stefan's approval

[Eli Zaretskii]

> Date: Tue, 07 Jan 2014 21:08:51 +0100
> From: Angelo Graziosi <angelo.graziosi@alice.it>
> CC: stephen.berman@gmx.net, sven.axelsson@gmail.com, emacs-devel@gnu.org
> 
> $ cd emacs.git/
> $ git gc --aggressive
> Counting objects: 735546, done.
> Delta compression using up to 2 threads.
> Compressing objects: 100% (734446/734446), done.
> Writing objects: 100% (735546/735546), done.
> Total 735546 (delta 589814), reused 143177 (delta 0)
> Checking connectivity: 735546, done.
> 
> which took almost 3 hours to be completed.
> 
> And the size is:
> 
> $ cd ..
> 
> $ du -s emacs.git/
> 327M	emacs.git/
> 
> which seems very good!

I guess this was on Windows 7, right?  I succeeded to do this on a
64-bit Windows 7 as well, and it took about 2 hours.  And now I
understand why git ran out of memory on Windows XP: the memory
footprint of git-pack-objects goes up to 3GB, whereas Windows XP has
only 3GB address space for all the applications.

So I guess we will need to repack the repository on savannah as a
prerequisite for switching, and then make sure it is repacked
regularly afterwards.  Otherwise, people with low memory and/or slow
CPUs will need many hours to do this on their local machines.

Re: Move to git is imminent - awaiting Stefan's approval

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

> So I guess we will need to repack the repository on savannah as a
> prerequisite for switching, and then make sure it is repacked
> regularly afterwards.

I think what should be done is to increase the default configuration
value of pack.window for the git repos on savannah.  That should help
keeping the packs better packed in the long run.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Move to git is imminent - awaiting Stefan's approval

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: Angelo Graziosi <angelo.graziosi@alice.it>,  stephen.berman@gmx.net,  sven.axelsson@gmail.com,  emacs-devel@gnu.org
> Date: Wed, 08 Jan 2014 19:34:56 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > So I guess we will need to repack the repository on savannah as a
> > prerequisite for switching, and then make sure it is repacked
> > regularly afterwards.
> 
> I think what should be done is to increase the default configuration
> value of pack.window for the git repos on savannah.  That should help
> keeping the packs better packed in the long run.

Whatever the experts say, I know nowhere near enough about that.  The
goal is to let people clone 330MB as opposed to 1.1GB.

Thanks.

Re: Move to git is imminent - awaiting Stefan's approval

[Angelo Graziosi]

Il 08/01/2014 19.42, Eli Zaretskii ha scritto:

> Whatever the experts say, I know nowhere near enough about that.  The
> goal is to let people clone 330MB as opposed to 1.1GB.

Indeed.. Why should one clone 1.1 GiB and the repack all in 330 MiB? and 
shouldn't the remote repository "cleaned" (repacked etc..) regularly?

Ciao,
  Angelo.

Re: Move to git is imminent - awaiting Stefan's approval

[Stefan Monnier]

> Indeed.. Why should one clone 1.1 GiB and the repack all in 330 MiB? and
> shouldn't the remote repository "cleaned" (repacked etc..) regularly?

Because Git is so much more efficient,


        Stefan

Re: Move to git is imminent - awaiting Stefan's approval

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Andreas Schwab <schwab@linux-m68k.org>,  stephen.berman@gmx.net,  sven.axelsson@gmail.com,  emacs-devel@gnu.org
> Date: Wed, 08 Jan 2014 19:05:47 -0500
> 
> > Indeed.. Why should one clone 1.1 GiB and the repack all in 330 MiB? and
> > shouldn't the remote repository "cleaned" (repacked etc..) regularly?
> 
> Because Git is so much more efficient,

A joke, I presume.

But if not: pulling 1.1GB through the net instead of just a third or
forth of that is still a painful experience in some parts of the
world, and git can do nothing at all about that, even though it
compresses.

There really is no excuse for having such a badly packed repository on
Savannah.

Re: Move to git is imminent - awaiting Stefan's approval

[Bob Proulx]

Andreas Schwab wrote:
> Eli Zaretskii writes:
> > So I guess we will need to repack the repository on savannah as a
> > prerequisite for switching, and then make sure it is repacked
> > regularly afterwards.
> 
> I think what should be done is to increase the default configuration
> value of pack.window for the git repos on savannah.  That should help
> keeping the packs better packed in the long run.

All of the repositories are repacked monthly on the 4th.  As you can
see it is a heavy operation and therefore not run every day.  But this
thread seems to have exposed a deficiency in the current repacking.
Currently the script does:

  git --git-dir=$dir gc -q

Obvious from the discussion here that is not sufficient.  However:

Angelo Graziosi wrote:
> $ git gc --aggressive
> Counting objects: 735546, done.
> Delta compression using up to 2 threads.
> Compressing objects: 100% (734446/734446), done.
> Writing objects: 100% (735546/735546), done.
> Total 735546 (delta 589814), reused 143177 (delta 0)
> Checking connectivity: 735546, done.
> 
> which took almost 3 hours to be completed.

Eli Zaretskii wrote:
> And now I understand why git ran out of memory on Windows XP: the
> memory footprint of git-pack-objects goes up to 3GB, whereas Windows
> XP has only 3GB address space for all the applications.

Three hours and 3G of memory for this one repack!  I worry that
turning --agressive on globally will cause the server to fall over.
Also apparently careful tuning of threads is needed.  However the disk
and network savings look significant.

I have a couple of questions if someone already has the clone
available for a test.  Does a clone from the agressively repacked to
327M repository remain similarly compacted when cloned again from it?

If an agressively repacked repository is again repacked but this time
without the --agressive option does the size stay around 327M or does
it get expanded on the subsequent pass?  (Wondering if we can
periodically run 'git gc --agressive' on the larger git repositories
at a niced background task priority but not too often and still
achieve a good benefit for the time between agressive repacks.)

Bob

Re: Move to git is imminent - awaiting Stefan's approval

[Andreas Schwab]

Bob Proulx <bob@proulx.com> writes:

> Three hours and 3G of memory for this one repack!  I worry that
> turning --agressive on globally will cause the server to fall over.

IMHO --aggressive is too aggressive, and not useful for on-going
maintenance.  Incresing pack.window to 50 or so should already help
enough to improve the packing.

> If an agressively repacked repository is again repacked but this time
> without the --agressive option does the size stay around 327M or does
> it get expanded on the subsequent pass?

Unless you run repack with -f (ie. gc --aggresive) existing deltas are
reused, and only newly added objects are deltified.

> (Wondering if we can periodically run 'git gc --agressive' on the
> larger git repositories at a niced background task priority but not
> too often and still achieve a good benefit for the time between
> agressive repacks.)

Another option is to touch a <pack>.keep file for the largest pack so
that it is never touched again.  New objects will then be added to a
separate pack even after git gc.  If that large pack is already well
packed this should save some processing time.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Move to git is imminent - awaiting Stefan's approval

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Date: Sun, 12 Jan 2014 11:11:21 +0100
> Cc: emacs-devel@gnu.org
> 
> Increasing pack.window to 50 or so should already help enough to
> improve the packing.

Would it make sense to make such a change in our local clones as well?

Re: Move to git is imminent - awaiting Stefan's approval

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

>> Increasing pack.window to 50 or so should already help enough to
>> improve the packing.
>
> Would it make sense to make such a change in our local clones as well?

Yes, setting that in your personal .gitconfig is probably a good idea.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Move to git is imminent - awaiting Stefan's approval

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: bob@proulx.com,  emacs-devel@gnu.org
> Date: Sun, 12 Jan 2014 18:09:29 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Increasing pack.window to 50 or so should already help enough to
> >> improve the packing.
> >
> > Would it make sense to make such a change in our local clones as well?
> 
> Yes, setting that in your personal .gitconfig is probably a good idea.

Thanks.

Re: Move to git is imminent - awaiting Stefan's approval

[Bob Proulx]

Andreas Schwab wrote:
> Bob Proulx <bob@proulx.com> writes:
> > Three hours and 3G of memory for this one repack!  I worry that
> > turning --agressive on globally will cause the server to fall over.
> 
> IMHO --aggressive is too aggressive, and not useful for on-going
> maintenance.  Incresing pack.window to 50 or so should already help
> enough to improve the packing.

It sounds like I should schedule a full agressive repack once.  Once
being the operative word.

  git repack -a -d -f --window=250 --depth=250

And then as you suggest above ensure that the regularly routine
ongoing maintenance 'git gc' runs with a large enough pack window.

I see that the git-gc default is --window=250 so at that point nothing
more should need to be done.  The normal 'git gc' routine maintenance
should be okay at that point.  Is that right?  Seems like it.  We can
tweak this further as needed.

Certain times of day the vcs system is quite heavily loaded.  The vcs
system is a VM on a dom0 also hosting many other VMs.  At some times
of day the entire dom0 is very heavily I/O limited.  This has been an
ongoing problem and discussion.  It is definitely an ongoing problem.
I will run this during a less busy dom0 time.

I spent some time researching this problem and found this note and
some information in the entire thread useful and interesting.  It is a
long thread but there are some good gems in there.

  http://article.gmane.org/gmane.comp.gcc.devel/94613

> > If an agressively repacked repository is again repacked but this time
> > without the --agressive option does the size stay around 327M or does
> > it get expanded on the subsequent pass?
> 
> Unless you run repack with -f (ie. gc --aggresive) existing deltas are
> reused, and only newly added objects are deltified.

Sounds good.  I will note that -f forces a full (--no-reuse-delta)
repack of everything and should be avoided.

> > (Wondering if we can periodically run 'git gc --agressive' on the
> > larger git repositories at a niced background task priority but not
> > too often and still achieve a good benefit for the time between
> > agressive repacks.)
> 
> Another option is to touch a <pack>.keep file for the largest pack so
> that it is never touched again.  New objects will then be added to a
> separate pack even after git gc.  If that large pack is already well
> packed this should save some processing time.

That seems like a useful additional tweak for a large stage such as
this.  If nothing else it will help out the backup since that file
won't be changing on a routine basis and will remain static for the
purposes of backup transfer.  This could be applied to several of the
large repositories.

It seems that on the client side after a new clone that this tweak is
not propagated.  It seems that if there are multiple packs on the
server side that they are combined into a single pack file.  But
without being repacked.  Therefore downstream clients that wish this
would need to do it manually after a clone.

Bob

Re: Move to git is imminent - awaiting Stefan's approval

[Andreas Schwab]

Bob Proulx <bob@proulx.com> writes:

> I see that the git-gc default is --window=250

Only if you add --aggressive (which uses gc.aggressiveWindow to override
the default for git repack, which is pack.window).

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Move to git is imminent - awaiting Stefan's approval

[Bob Proulx]

Andreas Schwab wrote:
> Bob Proulx <bob@proulx.com> writes:
> > I see that the git-gc default is --window=250
> 
> Only if you add --aggressive (which uses gc.aggressiveWindow to override
> the default for git repack, which is pack.window).

Ah...  Good clarification.  I have updated pack.window and set the new
value to 250.

Bob

Re: Move to git is imminent - awaiting Stefan's approval

[Bob Proulx]

Bob Proulx wrote:
> It sounds like I should schedule a full agressive repack once.  Once
> being the operative word.
> 
>   git repack -a -d -f --window=250 --depth=250

This has been done and the repository is now down to 285M.  I set the
number of pack.threads=2 and it ran for 104 minutes to completion.

(I had experimented letting it use all available 7-cores but that ran
94 minutes and ran out of memory at the 3G 32-bit limit.  The progress
indicator took 15 minutes to get to 90% and then spent the next 80
minutes getting no further than 92%.)

> > Another option is to touch a <pack>.keep file for the largest pack so
> > that it is never touched again.  New objects will then be added to a
> > separate pack even after git gc.  If that large pack is already well
> > packed this should save some processing time.
> 
> That seems like a useful additional tweak for a large stage such as
> this.  If nothing else it will help out the backup since that file
> won't be changing on a routine basis and will remain static for the
> purposes of backup transfer.  This could be applied to several of the
> large repositories.
> 
> It seems that on the client side after a new clone that this tweak is
> not propagated.  It seems that if there are multiple packs on the
> server side that they are combined into a single pack file.  But
> without being repacked.  Therefore downstream clients that wish this
> would need to do it manually after a clone.

I also touched the .keep file for the resulting pack file so that it
would remain static.

I checked out a local clone and the size was 329M.

Bob

P.S. I saw the other discussion thread talking about possibly redoing
the repository.  If that happens let me know and I will repack just as
above again.

Re: Move to git is imminent - awaiting Stefan's approval

[Achim Gratz]

Bob Proulx writes:
> I have a couple of questions if someone already has the clone
> available for a test.  Does a clone from the agressively repacked to
> 327M repository remain similarly compacted when cloned again from it?

Once the whole repository is repacked it keeps those pack decisions,
also across clones.  The pack strategy for new commits would typically
be near-optimal unless someone fast-imported a large branch again (in
that case it should preferrably be locally repacked before getting
pushed to Savannah).

> If an agressively repacked repository is again repacked but this time
> without the --agressive option does the size stay around 327M or does
> it get expanded on the subsequent pass?

I don't think a plain gc will expand any old packs, in any case the
benefits of "git gc --agressive" are supposed to last for a long time.

>(Wondering if we can periodically run 'git gc --agressive' on the
>larger git repositories at a niced background task priority but not too
>often and still achieve a good benefit for the time between agressive
>repacks.)

Just a manual "git gc --aggressive" once in a while should be enough.
If you are going to do it as a cron-job, I'm not sure if it's a good
idea to nice it.  I think you'd rather want to limit the number of
threads it uses, which also limits the memory footprint, which is
probably the bigger problem anyway and should probably be tuned further
depending on how beefy the server is.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for Waldorf Q V3.00R3 and Q+ V3.54R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: Move to git is imminent - awaiting Stefan's approval

[Bob Proulx]

Achim Gratz wrote:
> Bob Proulx writes:
> >(Wondering if we can periodically run 'git gc --agressive' on the
> >larger git repositories at a niced background task priority but not too
> >often and still achieve a good benefit for the time between agressive
> >repacks.)
> 
> Just a manual "git gc --aggressive" once in a while should be enough.
> If you are going to do it as a cron-job, I'm not sure if it's a good
> idea to nice it.

Due to the performance issues some of the Savannah admins have started
nicing background tasks as to hint to the scheduler.  Hoping that
interactive tasks get more priority.  Because at some times of day
there has been a problem with the server being slow enough to cause
timeouts.  Especially the web frontend to vcs.  Therefore if it isn't
a time critical task then running it niced is a hint to the scheduler.

The problem is that as a VM all of the system performance information
is fake data and can't show the actual system state.  And since we are
currently I/O limited at times the system is often blocked waiting for
I/O in which case the cpu priority doesn't matter.  There is more cpu
resource to go around than backplane disk I/O.  But from the VM itself
there isn't any useful performance data.  Only the FSF admins have
access to the dom0 and we count on them for real performance
information from it.  Unfortunately the FSF admin time to work on
server performance problems has been limited.

> I think you'd rather want to limit the number of threads it uses,
> which also limits the memory footprint, which is probably the bigger
> problem anyway and should probably be tuned further depending on how
> beefy the server is.

The dom0 server is an 8-core AMD Opteron running Xen for virtualization.
It has sufficient RAM.  But IMNHO too much I/O from all of the VMs
running on it.  The vcs VM is allocated seven xen cores.

The hint about threads is a good one and I will keep an eye on it.

Bob

Re: Move to git is imminent - awaiting Stefan's approval

[Eli Zaretskii]

> Date: Sat, 11 Jan 2014 23:27:22 -0700
> From: Bob Proulx <bob@proulx.com>
> 
> Eli Zaretskii wrote:
> > And now I understand why git ran out of memory on Windows XP: the
> > memory footprint of git-pack-objects goes up to 3GB, whereas Windows
> > XP has only 3GB address space for all the applications.
> 
> Three hours and 3G of memory for this one repack!

It was only 1 hour in my case (I guess Angelo has a slower machine).

Also, limiting the number of threads to 2 or even 1 radically lowers
the required memory usage.

But that doesn't mean you need necessarily run "gc --agressive",
please listen to Andreas on this matter.

> I have a couple of questions if someone already has the clone
> available for a test.  Does a clone from the agressively repacked to
> 327M repository remain similarly compacted when cloned again from it?

It grows slowly after that.  E.g., in the 3 days since then it grew by
12MB.  I don't know whether this will level out later, a few days is
not enough to tell.

> If an agressively repacked repository is again repacked but this time
> without the --agressive option does the size stay around 327M or does
> it get expanded on the subsequent pass?

No, it doesn't expand, at least not on my system.  In fact, "git gc"
just slashed the storage by 10MB out of 12MB that it gained since I
ran "gc --agressive".

Again, this is just to give you some numbers, I don't want to suggest
what should be done on Savannah.

Re: Move to git is imminent - awaiting Stefan's approval

[Angelo Graziosi]

Il 08/01/2014 18.32, Eli Zaretskii ha scritto:
>> Date: Tue, 07 Jan 2014 21:08:51 +0100
>> From: Angelo Graziosi <angelo.graziosi@alice.it>
>> CC: stephen.berman@gmx.net, sven.axelsson@gmail.com, emacs-devel@gnu.org
>>
>> $ cd emacs.git/
>> $ git gc --aggressive
>> Counting objects: 735546, done.
>> Delta compression using up to 2 threads.
>> Compressing objects: 100% (734446/734446), done.
>> Writing objects: 100% (735546/735546), done.
>> Total 735546 (delta 589814), reused 143177 (delta 0)
>> Checking connectivity: 735546, done.
>>
>> which took almost 3 hours to be completed.
>>
>> And the size is:
>>
>> $ cd ..
>>
>> $ du -s emacs.git/
>> 327M	emacs.git/
>>
>> which seems very good!
>
> I guess this was on Windows 7, right?  I succeeded to do this on a

NO!! It is on Win XP 32 bit (SP3) with AMD Athlon 64 X2 Dual Core 
Processor 3800+ and 1.7GiB of RAM.

On the same machine I have Kubuntu 12.04 and "git gc --aggressive" works 
there too..

BTW, I noticed that running "git pull" to have the local repository 
updated, the size increases of about 1 MiB.

> 64-bit Windows 7 as well, and it took about 2 hours.  And now I
> understand why git ran out of memory on Windows XP: the memory
> footprint of git-pack-objects goes up to 3GB, whereas Windows XP has
> only 3GB address space for all the applications.

Here, on XP, Task Manager shows a memory usage under 1500 MiB (about 1.4 
GiB)

In any case this

> So I guess we will need to repack the repository on savannah as a
> prerequisite for switching, and then make sure it is repacked
> regularly afterwards.  Otherwise, people with low memory and/or slow
> CPUs will need many hours to do this on their local machines.
>

should be mandatory..

Ciao,
  Angelo.

Re: Move to git is imminent - awaiting Stefan's approval

[Eli Zaretskii]

> Date: Wed, 08 Jan 2014 22:55:21 +0100
> From: Angelo Graziosi <angelo.graziosi@alice.it>
> CC: schwab@suse.de, stephen.berman@gmx.net, sven.axelsson@gmail.com, 
>  emacs-devel@gnu.org
> 
> Il 08/01/2014 18.32, Eli Zaretskii ha scritto:
> >> Date: Tue, 07 Jan 2014 21:08:51 +0100
> >> From: Angelo Graziosi <angelo.graziosi@alice.it>
> >> CC: stephen.berman@gmx.net, sven.axelsson@gmail.com, emacs-devel@gnu.org
> >>
> >> $ cd emacs.git/
> >> $ git gc --aggressive
> >> Counting objects: 735546, done.
> >> Delta compression using up to 2 threads.
> >> Compressing objects: 100% (734446/734446), done.
> >> Writing objects: 100% (735546/735546), done.
> >> Total 735546 (delta 589814), reused 143177 (delta 0)
> >> Checking connectivity: 735546, done.
> >>
> >> which took almost 3 hours to be completed.
> >>
> >> And the size is:
> >>
> >> $ cd ..
> >>
> >> $ du -s emacs.git/
> >> 327M	emacs.git/
> >>
> >> which seems very good!
> >
> > I guess this was on Windows 7, right?  I succeeded to do this on a
> 
> NO!! It is on Win XP 32 bit (SP3) with AMD Athlon 64 X2 Dual Core 
> Processor 3800+ and 1.7GiB of RAM.

The secret that unlocked this mystery was this:

> >> Delta compression using up to 2 threads.
                                   ^^^^^^^^^
On my machine, which is a Core i7, this originally said "8 threads".
So I did this: 

  git config --global pack.threads "2"

then ran "git gc --aggressive" again.  That again ran out of memory,
but much later in the process.  The next step was obvious:

  git config --global pack.threads "1"

With that, "git gc --aggressive" churned away for a little bit more
than an hour, and brought me to this:

  $ du -sh .
  323M    .

> > 64-bit Windows 7 as well, and it took about 2 hours.  And now I
> > understand why git ran out of memory on Windows XP: the memory
> > footprint of git-pack-objects goes up to 3GB, whereas Windows XP has
> > only 3GB address space for all the applications.
> 
> Here, on XP, Task Manager shows a memory usage under 1500 MiB (about 1.4 
> GiB)

With 1 thread, it never goes above 850MB.

Re: Move to git is imminent - awaiting Stefan's approval

[Angelo Graziosi]

Il 07/01/2014 15.41, Stephen Berman ha scritto:

> Is it correct that main or master is the same thing as the Emacs trunk
> branch?  If so, I wonder why the git repository containing just this is
> so large: my bzr Emacs repository only contains the trunk (as a bound
> branch), several local branches of it, and the emacs-24 branch; the
> .bzr/ directory of the repository, containing all the history, is only
> 428 MiB, not 1.1 GiB (the trunk directory itself contains an additional
> 129.3 MiB, but the total is still half the size of the git repo).  Why
> is the git repository so much larger than the bzr repository?

As I noticed before 
(http://lists.gnu.org/archive/html/emacs-devel/2014-01/msg00515.html), 
it contains emacs.git/.git directory of about 930 MiB...

Ciao,
  Angelo.

Re: Move to git is imminent - awaiting Stefan's approval

[Eli Zaretskii]

> Date: Tue, 7 Jan 2014 13:27:22 +0100
> From: Sven Axelsson <sven.axelsson@gmail.com>
> Cc: Andreas Schwab <schwab@linux-m68k.org>, emacs <emacs-devel@gnu.org>
> 
> Depth 1 truncates all history except for the most recent
> commit.

I think it keeps the 2 last commits, try "git log" in such a clone.

> From `man git clone`:
> 
> A shallow repository has a number of limitations (you cannot clone or
> fetch from it, nor push from nor into it), but is adequate if you are
> only interested in the recent history of a large project with a long
> history, and would want to send in fixes as patches.

For these reasons, this kind of repository is not recommended for
"Emacs devs".

Re: Move to git is imminent - awaiting Stefan's approval

[Antonio Nikishaev]

Sven Axelsson <sven.axelsson@gmail.com> writes:

>> Instead, trying
>>
>> $ git clone --depth 1 git://git.savannah.gnu.org/emacs.git emacs.git
>>
>> the local repository has dimension 160M which is a little greater
>> than my current BZR (checkout --lightweight) repository (126M), but
>> acceptable...
>>
>> Suppose now I want identify which "revision" introduced a bug. I am
>> able to do this with a binary search using the BZR command
>>
>> $ bzr up -r ARG
>>
>> How can I do this with the "minimal" clone done with GIT?
>
> Git does not have any concept like Bazaar's lightweight checkouts -
> nor does any other dvcd afaik.

Darcs has clever lazy repos. (Everything works. If the stuff which is
not yet in the repository is needed, it will be downloaded.)

Just FYI

Re: Move to git is imminent - awaiting Stefan's approval

[David Kastrup]

Angelo Graziosi <angelo.graziosi@alice.it> writes:

> Instead, trying
>
> $ git clone --depth 1 git://git.savannah.gnu.org/emacs.git emacs.git
>
> the local repository has dimension 160M which is a little greater than
> my current BZR (checkout --lightweight) repository (126M), but
> acceptable...
>
> Suppose now I want identify which "revision" introduced a bug. I am
> able to do this with a binary search using the BZR command
>
> $ bzr up -r ARG
>
> How can I do this with the "minimal" clone done with GIT?

Uh, not at all?  Either you fetch previous revisions and can make use of
them, or not.  I think there is some command for extending a shallow
clone, but of course that will have the respective impact on the
repository size.

-- 
David Kastrup

Re: Move to git is imminent - awaiting Stefan's approval

[Eli Zaretskii]

> Date: Tue, 07 Jan 2014 11:24:12 +0100
> From: Angelo Graziosi <angelo.graziosi@alice.it>
> Cc: emacs <emacs-devel@gnu.org>
> 
> >> $ du -s emacs-trunk/
> >> 126M	emacs-trunk/
> >>
> >> instead the git repository has dimension
> >>
> >> $ du -s emacs.git/
> >> 1.1G	emacs.git/
> >
> > By default git downloads the history of all branches.  You can use
> > --single-branch to only clone a single branch.  This is the default if
> > you create a shallow clone.
> 
> David Kastrup wrote:
> 
> > For the build?  No.  git clone has the following options:
> >
> >         --depth <depth>
> ...
> > --[no-]single-branch
> ...
> 
> OK, trying
> 
> $ git clone --single-branch git://git.savannah.gnu.org/emacs.git emacs.git
> 
> the local repository has dimension 1.1G, so probably I need a more 
> explicit example...

The command you tried is correct, it's just that it doesn't save too
much disk space because the trunk has almost all of the revisions.

> Instead, trying
> 
> $ git clone --depth 1 git://git.savannah.gnu.org/emacs.git emacs.git
> 
> the local repository has dimension 160M which is a little greater than 
> my current BZR (checkout --lightweight) repository (126M), but acceptable...

This is not recommended at all, as such a shallow repository is
severely limited: you cannot clone from it, nor push from or to it.
You also have almost no history, and git does not support bzr-like
operation whereby commands in lightweight checkouts that need history
information go to the server.  So you will be unable to walk through
history and bisect.

> Suppose now I want identify which "revision" introduced a bug. I am able 
> to do this with a binary search using the BZR command
> 
> $ bzr up -r ARG
> 
> How can I do this with the "minimal" clone done with GIT?

You don't, see above.  With a full repository, or at least one that
was cloned with --single-branch, the equivalent of "bzr up" with git
is "git checkout", and the revision which you want is specified by
either is SHA1 signature or relatively, as in HEAD~20.

Re: Move to git is imminent - awaiting Stefan's approval

[Angelo Graziosi]

Il 07/01/2014 0.01, Andreas Schwab ha scritto:
> A good way to identify a git branch is to use git describe.  However it
> prefers to have an annotated (release) tag available to base on, which
> we currently don't have (bzr doesn't have the concept of annotated
> tags).

Just for completeness...

I see that I can download the trunk source with

$ wget 
http://git.savannah.gnu.org/cgit/emacs.git/snapshot/emacs-trunk.tar.gz

but how I can identify it?

After unpacking it, the tree should contain at least a text file with a 
revision number or the SHA sum of the last commit... or not?


Ciao,
  Angelo.

Re: Move to git is imminent - awaiting Stefan's approval

[Andreas Schwab]

Angelo Graziosi <angelo.graziosi@alice.it> writes:

> I see that I can download the trunk source with
>
> $ wget
> http://git.savannah.gnu.org/cgit/emacs.git/snapshot/emacs-trunk.tar.gz
>
> but how I can identify it?

Normally git get-tar-commit-id should do it, but the archive wasn't
produced by git archive, or it was created from a tree ID instead of a
commit ID.  That looks like a bug in cgit.

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: Move to git is imminent - awaiting Stefan's approval

[Angelo Graziosi]

Il 07/01/2014 15.24, Andreas Schwab ha scritto:
> Angelo Graziosi <angelo.graziosi@alice.it> writes:
>
>> I see that I can download the trunk source with
>>
>> $ wget
>> http://git.savannah.gnu.org/cgit/emacs.git/snapshot/emacs-trunk.tar.gz
>>
>> but how I can identify it?
>
> Normally git get-tar-commit-id should do it, but the archive wasn't
> produced by git archive, or it was created from a tree ID instead of a
> commit ID.  That looks like a bug in cgit.

I have downloaded

1) 
http://git.savannah.gnu.org/cgit/emacs.git/snapshot/emacs-e42fcf1482e2c9e9d93f459985d13dbbabc61956.tar.gz

2) http://git.savannah.gnu.org/cgit/emacs.git/snapshot/emacs-trunk.tar.gz

which correspond to the same trunk but I cannot identify the trunk: the 
command "git get-tar-commit-id" hangs.

If I have understood, with GIT the trunk is identified with the SHA1 
string like "e42fcf1482e2c9e9d93f459985d13dbbabc61956", so that in the 
first case I know the trunk "revision". Anyway, I would prefer that that 
string is stored inside the tree given by those tar-balls. For example 
in a text file, LAST-COMMIT-ID, so that one can know it with

$ cat emacs-e42fcf1482e2c9e9d93f459985d13dbbabc61956/LAST-COMMIT-ID

or

$ cat emacs-trunk/LAST-COMMIT-ID


In this way, I can create a build script with this skeleton:

...
wget http://git.savannah.gnu.org/cgit/emacs.git/snapshot/emacs-trunk.tar.gz

tar -xf emacs-trunk.tar.gz

rev=`cat emacs-trunk/LAST-COMMIT-ID`

autogen.sh
configure...
make bootstrap
make install DESTDIR=...

tar -cf emacs-$rev-bin.tar.xz $INSTDIR


Ciao,
  Angelo.

Re: Move to git is imminent - awaiting Stefan's approval

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Date: Tue, 07 Jan 2014 00:01:45 +0100
> Cc: emacs <emacs-devel@gnu.org>
> 
> Angelo Graziosi <angelo.graziosi@alice.it> writes:
> 
> > Now trying
> >
> > $ git clone git://git.savannah.gnu.org/emacs.git emacs.git
> >
> >
> > I don't get any revision number.. So in my future bug report how the trunk
> > will be identified?
> 
> A good way to identify a git branch is to use git describe.  However it
> prefers to have an annotated (release) tag available to base on, which
> we currently don't have

Indeed:

  $ git describe
  fatal: No annotated tags can describe 'e1d32fe4408170b992528a2da4b917a63e86fbc4'

I think Angelo wants one of the following, which will all produce
exactly the same output:

  git log --pretty='format:%H' @^..
  git rev-list  @^..
  git rev-parse --verify HEAD

Re: Move to git is imminent - awaiting Stefan's approval

[Rüdiger Sonderfeld]

On Tuesday 07 January 2014 18:14:45 Eli Zaretskii wrote:
>   $ git describe
>   fatal: No annotated tags can describe
> 'e1d32fe4408170b992528a2da4b917a63e86fbc4'
> 
> I think Angelo wants one of the following, which will all produce
> exactly the same output:
> 
>   git log --pretty='format:%H' @^..
>   git rev-list  @^..
>   git rev-parse --verify HEAD

Or

  git describe --always
  git show --pretty=oneline
  git show --oneline

(Produces a different output though)

Regards
Rüdiger