Re: Move to git is imminent - awaiting Stefan's approval

Re: Move to git is imminent - awaiting Stefan's approval

[Angelo Graziosi]

Eli Zaretskii wrote:
> 8. There's the emacs-bzr-version whose value gets copied into the bug
>     reports.  This should be replaced by the suitable git equivalent,
>     or else the bug reports (of which we have quite a few each day)
>     will not identify the version correctly.

Perhaps I have a similar question (or the same?)

Usually I build emacs trunk weekly. My local copy of trunk has been 
created, the first time, with

$ bzr checkout --lightweight http://bzr.savannah.gnu.org/r/emacs/trunk 
emacs-trunk


and then updating "daily" it with

$ cd emacs-trunk
$ bzr up

which prints each time the trunk revision number, e.g.

[...]
All changes applied successfully. 

Updated to revision 115890 of branch 
http://bzr.savannah.gnu.org/r/emacs/trunk


When I find a problem ("bug"), I flag it to this list saying something 
like this

"In trunk rev. 115890 there is this problem... BLA BLA BLA..."


Now trying

$ git clone git://git.savannah.gnu.org/emacs.git emacs.git


I don't get any revision number.. So in my future bug report how the 
trunk will be identified?

Beside this, my local copy of BZR repository has dimension

$ du -s emacs-trunk/
126M	emacs-trunk/

instead the git repository has dimension

$ du -s emacs.git/
1.1G	emacs.git/

it contains an emacs.git/.git directory of about 930M which probably 
doesn't need for the Emacs build (bootstrap).

This is very annoying...


Ciao,
  Angelo.

Re: Move to git is imminent - awaiting Stefan's approval

[David Kastrup]

Angelo Graziosi <angelo.graziosi@alice.it> writes:

> Usually I build emacs trunk weekly. My local copy of trunk has been
> created, the first time, with
>
> $ bzr checkout --lightweight http://bzr.savannah.gnu.org/r/emacs/trunk
> emacs-trunk
>
>
> and then updating "daily" it with
>
> $ cd emacs-trunk
> $ bzr up

"lightweight"

> When I find a problem ("bug"), I flag it to this list saying something
> like this
>
> "In trunk rev. 115890 there is this problem... BLA BLA BLA..."
>
>
> Now trying
>
> $ git clone git://git.savannah.gnu.org/emacs.git emacs.git
>
>
> I don't get any revision number.. So in my future bug report how the
> trunk will be identified?

The commit ids are unique.  It's usually easiest to quote the entire
SHA128 via copy&paste, but the first 16 digits or so are usually
sufficient.

> Beside this, my local copy of BZR repository has dimension
>
> $ du -s emacs-trunk/
> 126M	emacs-trunk/

"lightweight"

> instead the git repository has dimension
>
> $ du -s emacs.git/
> 1.1G	emacs.git/
>
> it contains an emacs.git/.git directory of about 930M which probably
> doesn't need for the Emacs build (bootstrap).

For the build?  No.  git clone has the following options:

        --depth <depth>
           Create a shallow clone with a history truncated to the specified
           number of revisions. A shallow repository has a number of
           limitations (you cannot clone or fetch from it, nor push from nor
           into it), but is adequate if you are only interested in the recent
           history of a large project with a long history, and would want to
           send in fixes as patches.

       --[no-]single-branch
           Clone only the history leading to the tip of a single branch,
           either specified by the --branch option or the primary branch
           remote’s HEAD points at. When creating a shallow clone with the
           --depth option, this is the default, unless --no-single-branch is
           given to fetch the histories near the tips of all branches. Further
           fetches into the resulting repository will only update the
           remote-tracking branch for the branch this option was used for the
           initial cloning. If the HEAD at the remote did not point at any
           branch when --single-branch clone was made, no remote-tracking
           branch is created.

In addition, once you have a local repository, something like

git clone existing_workdir new_workdir

will do the cloning by using hard links, taking almost no time and no
space.  I often do that for one-off testing of stuff while my "main"
workdir is busy compiling stuff.

-- 
David Kastrup

Re: Move to git is imminent - awaiting Stefan's approval

[Nix]

On 6 Jan 2014, David Kastrup stated:

> The commit ids are unique.  It's usually easiest to quote the entire
> SHA128 via copy&paste, but the first 16 digits or so are usually
> sufficient.

You can just use 'git log --abbrev-commit' to get commit IDs abbreviated
(enough for nonambiguity and no further).

If you want to enable this forever, git 1.7.6+ lets you say

git config log.abbrevCommit true

(add --global or --system to make it apply to all repositories for this
user, or all repositories on the system, respectively).

> In addition, once you have a local repository, something like
>
> git clone existing_workdir new_workdir
>
> will do the cloning by using hard links, taking almost no time and no
> space.  I often do that for one-off testing of stuff while my "main"
> workdir is busy compiling stuff.

In addition in addition, git-new-workdir (a script in git's contrib/
directory, likely to be installed by default eventually in some form or
other) will let you check out more than one branch at once on the same
repository.

-- 
NULL && (void)

Re: Move to git is imminent - awaiting Stefan's approval

[Andreas Schwab]

Angelo Graziosi <angelo.graziosi@alice.it> writes:

> Now trying
>
> $ git clone git://git.savannah.gnu.org/emacs.git emacs.git
>
>
> I don't get any revision number.. So in my future bug report how the trunk
> will be identified?

A good way to identify a git branch is to use git describe.  However it
prefers to have an annotated (release) tag available to base on, which
we currently don't have (bzr doesn't have the concept of annotated
tags).

> Beside this, my local copy of BZR repository has dimension
>
> $ du -s emacs-trunk/
> 126M	emacs-trunk/
>
> instead the git repository has dimension
>
> $ du -s emacs.git/
> 1.1G	emacs.git/

By default git downloads the history of all branches.  You can use
--single-branch to only clone a single branch.  This is the default if
you create a shallow clone.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Move to git is imminent - awaiting Stefan's approval

[Angelo Graziosi]

Il 07/01/2014 0.01, Andreas Schwab ha scritto:
> Angelo Graziosi <angelo.graziosi@alice.it> writes:
>
>> Now trying
>>
>> $ git clone git://git.savannah.gnu.org/emacs.git emacs.git
>>
>>
>> I don't get any revision number.. So in my future bug report how the trunk
>> will be identified?
>
> A good way to identify a git branch is to use git describe.  However it
> prefers to have an annotated (release) tag available to base on, which
> we currently don't have (bzr doesn't have the concept of annotated
> tags).
>
>> Beside this, my local copy of BZR repository has dimension
>>
>> $ du -s emacs-trunk/
>> 126M	emacs-trunk/
>>
>> instead the git repository has dimension
>>
>> $ du -s emacs.git/
>> 1.1G	emacs.git/
>
> By default git downloads the history of all branches.  You can use
> --single-branch to only clone a single branch.  This is the default if
> you create a shallow clone.

David Kastrup wrote:

> For the build?  No.  git clone has the following options:
>
>         --depth <depth>
...
> --[no-]single-branch
...

OK, trying

$ git clone --single-branch git://git.savannah.gnu.org/emacs.git emacs.git

the local repository has dimension 1.1G, so probably I need a more 
explicit example...

Instead, trying

$ git clone --depth 1 git://git.savannah.gnu.org/emacs.git emacs.git

the local repository has dimension 160M which is a little greater than 
my current BZR (checkout --lightweight) repository (126M), but acceptable...

Suppose now I want identify which "revision" introduced a bug. I am able 
to do this with a binary search using the BZR command

$ bzr up -r ARG

How can I do this with the "minimal" clone done with GIT?


I hope someone creates a Wiki page with all these doubts clarified with 
explicit examples... :-)


Thanks,
  Angelo.

Git tips for Emacs development (was Re: Move to git is imminent - awaiting Stefan's approval)

[Nic Ferrier]

Angelo Graziosi <angelo.graziosi@alice.it> writes:

> Instead, trying
>
> $ git clone --depth 1 git://git.savannah.gnu.org/emacs.git emacs.git
>
> the local repository has dimension 160M which is a little greater than 
> my current BZR (checkout --lightweight) repository (126M), but acceptable...
>
> Suppose now I want identify which "revision" introduced a bug. I am able 
> to do this with a binary search using the BZR command
>
> $ bzr up -r ARG

What does ARG do in this context?

> How can I do this with the "minimal" clone done with GIT?
>
>
> I hope someone creates a Wiki page with all these doubts clarified with 
> explicit examples... :-)

http://www.emacswiki.org/emacs/GitForEmacsDevs

is the start of one.

Do add questions.


Nic

Re: Git tips for Emacs development (was Re: Move to git is imminent - awaiting Stefan's approval)

[Eli Zaretskii]

> From: Nic Ferrier <nferrier@ferrier.me.uk>
> Date: Tue, 07 Jan 2014 10:37:44 +0000
> Cc: Andreas Schwab <schwab@linux-m68k.org>, emacs <emacs-devel@gnu.org>
> 
> > $ bzr up -r ARG
> 
> What does ARG do in this context?

It's the revision to which the tree will be updated.

Re: Move to git is imminent - awaiting Stefan's approval

[Sven Axelsson]

On 7 January 2014 11:24, Angelo Graziosi <angelo.graziosi@alice.it> wrote:
>
>
> OK, trying
>
> $ git clone --single-branch git://git.savannah.gnu.org/emacs.git emacs.git
>
> the local repository has dimension 1.1G, so probably I need a more explicit example...

This clones only the main branch (master). Since this is where almost
all Emacs history is located it won't make much difference size wise.

> Instead, trying
>
> $ git clone --depth 1 git://git.savannah.gnu.org/emacs.git emacs.git
>
> the local repository has dimension 160M which is a little greater than my current BZR (checkout --lightweight) repository (126M), but acceptable...
>
> Suppose now I want identify which "revision" introduced a bug. I am able to do this with a binary search using the BZR command
>
> $ bzr up -r ARG
>
> How can I do this with the "minimal" clone done with GIT?

Git does not have any concept like Bazaar's lightweight checkouts -
nor does any other dvcd afaik. Depth 1 truncates all history except
for the most recent commit. From `man git clone`:

A shallow repository has a number of limitations (you cannot clone or
fetch from it, nor push from nor into it), but is adequate if you are
only interested in the recent history of a large project with a long
history, and would want to send in fixes as patches.


-- 
Sven Axelsson
++++++++++[>++++++++++>+++++++++++>++++++++++>++++++
>++++<<<<<-]>++++.+.++++.>+++++.>+.<<-.>>+.>++++.<<.
+++.>-.<<++.>>----.<++.>>>++++++.<<<<.>>++++.<----.

Re: Move to git is imminent - awaiting Stefan's approval

[Stephen Berman]

On Tue, 7 Jan 2014 13:27:22 +0100 Sven Axelsson <sven.axelsson@gmail.com> wrote:

> On 7 January 2014 11:24, Angelo Graziosi <angelo.graziosi@alice.it> wrote:
>>
>>
>> OK, trying
>>
>> $ git clone --single-branch git://git.savannah.gnu.org/emacs.git emacs.git
>>
>> the local repository has dimension 1.1G, so probably I need a more explicit
>> example...
>
> This clones only the main branch (master). Since this is where almost
> all Emacs history is located it won't make much difference size wise.

Is it correct that main or master is the same thing as the Emacs trunk
branch?  If so, I wonder why the git repository containing just this is
so large: my bzr Emacs repository only contains the trunk (as a bound
branch), several local branches of it, and the emacs-24 branch; the
.bzr/ directory of the repository, containing all the history, is only
428 MiB, not 1.1 GiB (the trunk directory itself contains an additional
129.3 MiB, but the total is still half the size of the git repo).  Why
is the git repository so much larger than the bzr repository?

Steve Berman

Re: Move to git is imminent - awaiting Stefan's approval

[Andreas Schwab]

Stephen Berman <stephen.berman@gmx.net> writes:

> Why is the git repository so much larger than the bzr repository?

The git repositories on savannah appear to be packed rather badly.  A
reasonably well packed emacs repository should have around 350MB.

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: Move to git is imminent - awaiting Stefan's approval

[Stephen Berman]

On Tue, 07 Jan 2014 15:57:05 +0100 Andreas Schwab <schwab@suse.de> wrote:

> Stephen Berman <stephen.berman@gmx.net> writes:
>
>> Why is the git repository so much larger than the bzr repository?
>
> The git repositories on savannah appear to be packed rather badly.  A
> reasonably well packed emacs repository should have around 350MB.

Ok.  If the packing can be improved, I think it would be good for that
to happen before finalizing the move to git.

Steve Berman

Re: Move to git is imminent - awaiting Stefan's approval

[Eli Zaretskii]

> From: Andreas Schwab <schwab@suse.de>
> Date: Tue, 07 Jan 2014 15:57:05 +0100
> Cc: Angelo Graziosi <angelo.graziosi@alice.it>,
> 	Sven Axelsson <sven.axelsson@gmail.com>, emacs <emacs-devel@gnu.org>
> 
> The git repositories on savannah appear to be packed rather badly.  A
> reasonably well packed emacs repository should have around 350MB.

Can repacking be done locally, or does it have to be on savannah?  If
the former, how to do it?

Re: Move to git is imminent - awaiting Stefan's approval

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

> Can repacking be done locally, or does it have to be on savannah?

You can always repack locally.  It's just that during clone, you inherit
the packing of the remote, since existing deltas are not recomputed.

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: Move to git is imminent - awaiting Stefan's approval

[Angelo Graziosi]

Il 07/01/2014 17.35, Andreas Schwab ha scritto:
> Eli Zaretskii <eliz@gnu.org> writes:
>
>> Can repacking be done locally, or does it have to be on savannah?
>
> You can always repack locally.  It's just that during clone, you inherit
> the packing of the remote, since existing deltas are not recomputed.

OK, but Eli asked how do it... So how?


Thanks,
   Angelo.

Re: Move to git is imminent - awaiting Stefan's approval

[Andreas Schwab]

Angelo Graziosi <angelo.graziosi@alice.it> writes:

> Il 07/01/2014 17.35, Andreas Schwab ha scritto:
>> Eli Zaretskii <eliz@gnu.org> writes:
>>
>>> Can repacking be done locally, or does it have to be on savannah?
>>
>> You can always repack locally.  It's just that during clone, you inherit
>> the packing of the remote, since existing deltas are not recomputed.
>
> OK, but Eli asked how do it... So how?

See git-gc(1)

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: Move to git is imminent - awaiting Stefan's approval

[Eli Zaretskii]

> From: Andreas Schwab <schwab@suse.de>
> Cc: Eli Zaretskii <eliz@gnu.org>,  stephen.berman@gmx.net,  sven.axelsson@gmail.com,  emacs-devel@gnu.org
> Date: Tue, 07 Jan 2014 17:41:43 +0100
> 
> Angelo Graziosi <angelo.graziosi@alice.it> writes:
> 
> > Il 07/01/2014 17.35, Andreas Schwab ha scritto:
> >> Eli Zaretskii <eliz@gnu.org> writes:
> >>
> >>> Can repacking be done locally, or does it have to be on savannah?
> >>
> >> You can always repack locally.  It's just that during clone, you inherit
> >> the packing of the remote, since existing deltas are not recomputed.
> >
> > OK, but Eli asked how do it... So how?
> 
> See git-gc(1)

Thanks.  "git gc" gets me from 1.18GB to 1.1GB.  I tried --aggressive,
but that ran out of memory (trying to allocate 1GB) after some time
and errored out.

Re: Move to git is imminent - awaiting Stefan's approval

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

> Thanks.  "git gc" gets me from 1.18GB to 1.1GB.  I tried --aggressive,
> but that ran out of memory (trying to allocate 1GB) after some time
> and errored out.

That's strange:

oscar@qcore:~/dev/emacs/emacs$ time git gc
Counting objects: 734772, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (145032/145032), done.
Writing objects: 100% (734772/734772), done.
Total 734772 (delta 589118), reused 734312 (delta 588658)
Checking connectivity: 734772, done.

oscar@qcore:~/dev/emacs/emacs$ du -sh .git
289M    .git
oscar@qcore:~/dev/emacs/emacs$ git branch -a | wc
    127     130    4667


Maybe you instructed git to download extra info (such as the meta-data
the fast-import/export process uses and is useful for relating git
commits to their corresponding bzr commits.)

Re: Move to git is imminent - awaiting Stefan's approval

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Tue, 07 Jan 2014 18:23:48 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Thanks.  "git gc" gets me from 1.18GB to 1.1GB.  I tried --aggressive,
> > but that ran out of memory (trying to allocate 1GB) after some time
> > and errored out.
> 
> That's strange:
> 
> oscar@qcore:~/dev/emacs/emacs$ time git gc
> Counting objects: 734772, done.
> Delta compression using up to 4 threads.
> Compressing objects: 100% (145032/145032), done.
> Writing objects: 100% (734772/734772), done.
> Total 734772 (delta 589118), reused 734312 (delta 588658)
> Checking connectivity: 734772, done.
> 
> oscar@qcore:~/dev/emacs/emacs$ du -sh .git
> 289M    .git
> oscar@qcore:~/dev/emacs/emacs$ git branch -a | wc
>     127     130    4667
> 
> 
> Maybe you instructed git to download extra info (such as the meta-data
> the fast-import/export process uses and is useful for relating git
> commits to their corresponding bzr commits.)

Nope, just "git clone" followed by pull's.

I now tried on fencepost.gnu.org, a 64-bit GNU/Linux system, with
similar results: gc goes from 1.78GB down to 1.6GB.  --aggressive also
fails there, funnily enough.

Re: Move to git is imminent - awaiting Stefan's approval

[Stephen J. Turnbull]

Eli Zaretskii writes:

 > > See git-gc(1)

git-gc does not necessarily repack existing packs according to the
doc.

Try git-repack.

Or wait for me to try it ... it's past my bedtime, will report
tomorrow if somebody doesn't beat me to it.

Re: Move to git is imminent - awaiting Stefan's approval

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Cc: Andreas Schwab <schwab@suse.de>,
>     stephen.berman@gmx.net,
>     emacs-devel@gnu.org,
>     sven.axelsson@gmail.com,
>     angelo.graziosi@alice.it
> Date: Wed, 08 Jan 2014 02:24:02 +0900
> 
> Eli Zaretskii writes:
> 
>  > > See git-gc(1)
> 
> git-gc does not necessarily repack existing packs according to the
> doc.
> 
> Try git-repack.

I thought "git gc" runs git-repack, no?

Re: Move to git is imminent - awaiting Stefan's approval

[Sven Axelsson]

On 7 January 2014 19:05, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: "Stephen J. Turnbull" <stephen@xemacs.org>
>> Cc: Andreas Schwab <schwab@suse.de>,
>>     stephen.berman@gmx.net,
>>     emacs-devel@gnu.org,
>>     sven.axelsson@gmail.com,
>>     angelo.graziosi@alice.it
>> Date: Wed, 08 Jan 2014 02:24:02 +0900
>>
>> Eli Zaretskii writes:
>>
>>  > > See git-gc(1)
>>
>> git-gc does not necessarily repack existing packs according to the
>> doc.
>>
>> Try git-repack.
>
> I thought "git gc" runs git-repack, no?

No. git-repack is often a very expensive operation, so it is never run
automatically, which git-gc can be.

-- 
Sven Axelsson
++++++++++[>++++++++++>+++++++++++>++++++++++>++++++
>++++<<<<<-]>++++.+.++++.>+++++.>+.<<-.>>+.>++++.<<.
+++.>-.<<++.>>----.<++.>>>++++++.<<<<.>>++++.<----.

Re: Move to git is imminent - awaiting Stefan's approval

[Stephen J. Turnbull]

Sven Axelsson writes:
 > On 7 January 2014 19:05, Eli Zaretskii <eliz@gnu.org> wrote:
 > >> From: "Stephen J. Turnbull" <stephen@xemacs.org>

 > >> git-gc does not necessarily repack existing packs according to the
 > >> doc.
 > >>
 > >> Try git-repack.
 > >
 > > I thought "git gc" runs git-repack, no?

It kinda pisses me off that you complain about the excessive detail of
git documentation, and then ask others to tell you about the details
you could read in the documentation.

 > No. git-repack is often a very expensive operation, so it is never
 > run automatically, which git-gc can be.

Specifically, according to the help, git-gc removes garbage (in the
usual sense of a tracing collector) and "compresses" (ie, packs) loose
objects using delta compression (and then the result is deflated as
with loose objects, I believe).  It will repack the packs into *one*
pack (presumably with better delta compression) if the number of packs
is greater than the value of gc.autopacklimit (in .git/config;
default: 50).

Re: Move to git is imminent - awaiting Stefan's approval

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,
>     schwab@suse.de,
>     stephen.berman@gmx.net,
>     angelo.graziosi@alice.it,
>     emacs <emacs-devel@gnu.org>
> Date: Wed, 08 Jan 2014 03:49:24 +0900
> 
> Sven Axelsson writes:
>  > On 7 January 2014 19:05, Eli Zaretskii <eliz@gnu.org> wrote:
>  > >> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> 
>  > >> git-gc does not necessarily repack existing packs according to the
>  > >> doc.
>  > >>
>  > >> Try git-repack.
>  > >
>  > > I thought "git gc" runs git-repack, no?
> 
> It kinda pisses me off that you complain about the excessive detail of
> git documentation, and then ask others to tell you about the details
> you could read in the documentation.

I did read the documentation.  The docs of gc mentions git-repack, so
I thought the former runs the latter.  What part did I miss?

> Specifically, according to the help, git-gc removes garbage (in the
> usual sense of a tracing collector) and "compresses" (ie, packs) loose
> objects using delta compression (and then the result is deflated as
> with loose objects, I believe).  It will repack the packs into *one*
> pack (presumably with better delta compression) if the number of packs
> is greater than the value of gc.autopacklimit (in .git/config;
> default: 50).

And how was I supposed to know from this whether git-repack will or
won't be run, what with the word "repack" explicitly in this text (and
in other parts of that man page)?

Don't bother answering if you are still pissed off.

Re: Move to git is imminent - awaiting Stefan's approval

[Stephen J. Turnbull]

Eli Zaretskii writes:

 > I did read the documentation.  The docs of gc mentions git-repack, so
 > I thought the former runs the latter.  What part did I miss?

The fact that the documentation clearly puts conditions on when git-gc
runs git-repack.

 > And how was I supposed to know from this whether git-repack will or
 > won't be run, what with the word "repack" explicitly in this text (and
 > in other parts of that man page)?

Huh?  If there's a condition on whether it runs, then presumably
sometimes it doesn't run.  That's all that I deduced from it, and
advised you run git-repack manually.

Re: Move to git is imminent - awaiting Stefan's approval

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Cc: schwab@suse.de,
>     stephen.berman@gmx.net,
>     emacs-devel@gnu.org,
>     sven.axelsson@gmail.com,
>     angelo.graziosi@alice.it
> Date: Wed, 08 Jan 2014 12:47:29 +0900
> 
> Eli Zaretskii writes:
>  > I did read the documentation.  The docs of gc mentions git-repack, so
>  > I thought the former runs the latter.  What part did I miss?
> 
> The fact that the documentation clearly puts conditions on when git-gc
> runs git-repack.

I wouldn't qualify the gc manpage as "clear" on this matter, see
below.  But given what it does say, I _guessed_ that "git gc" _might_
run git-repack, and asked whether that wasn't so.  Why that pissed you
off I don't know.  You seem to have deduced a different conclusion,
but IMO that's just one more evidence that the man page is not really
clear on this matter.

>  > And how was I supposed to know from this whether git-repack will or
>  > won't be run, what with the word "repack" explicitly in this text (and
>  > in other parts of that man page)?
> 
> Huh?  If there's a condition on whether it runs, then presumably
> sometimes it doesn't run.

AFAIU, you refer to this part of your previous message:

> Specifically, according to the help, git-gc removes garbage (in the
> usual sense of a tracing collector) and "compresses" (ie, packs) loose
> objects using delta compression (and then the result is deflated as
> with loose objects, I believe).  It will repack the packs into *one*
> pack (presumably with better delta compression) if the number of packs
> is greater than the value of gc.autopacklimit (in .git/config;
> default: 50).

However, this comes from the part in the man page that describes the
"--auto" option to gc.  I didn't use and didn't mean to use that
option, so the above is not necessarily relevant to "git gc".

But even if this description is relevant when --auto is not used,
the above says that repacking will sometimes happen.

As to what happens regardless of any options, the man page says just
this:

  DESCRIPTION

    Runs a number of housekeeping tasks within the current repository,
    such as compressing file revisions (to reduce disk space and
    increase performance) and removing unreachable objects which may
    have been created from prior invocations of git add.

This says "such as", which doesn't imply an exhaustive list of tasks.
Moreover, it never says that "compressing file revisions" and
"removing unreachable objects" is not done by git-repack.  So the
above description does not contradict my conclusion, and definitely
doesn't present exact description of the conditions under which gc
will run repack.  Perhaps if I knew exactly what repack does, I could
have filled the voids, but I don't.

Finally, the "Configuration" section has this, among the rest:

  The optional configuration variable gc.packrefs determines if git gc
  runs git pack-refs. This can be set to "notbare" to enable it within
  all non-bare repos or it can be set to a boolean value. This
  defaults to true.

  The optional configuration variable gc.aggressiveWindow controls how
  much time is spent optimizing the delta compression of the objects
  in the repository when the --aggressive option is specified. The
  larger the value, the more time is spent optimizing the delta
  compression. See the documentation for the --window' option in
  git-repack(1) for more details. This defaults to 250.

Both of these configuration settings explicitly mention packing and/or
repacking.  Again, without knowing exactly what repack does, there's
no clarity here whether pack-refs is related to repack in any way,
shape or form (looking in the sources resolves this, see below).

Given these hints, IMO it was a reasonable guess that gc does run
repack at least under some conditions, when --auto is not used.  Since
these conditions are not defined completely by the man page, it was my
guess that gc, with or without the --aggressive option, might run
repack.  As it turns out, the second part is surely true.

> You read the source?  I find that hard to believe.

I didn't read the source in this case, only the documentation.

I don't understand the "hard to believe" part, though: I do have the
git's git repo on my disk, and you could easily imagine that I'm not
new to reading sources.

Having now looked in the sources, this is what I see:

	if (pack_refs && run_command_v_opt(pack_refs_cmd.argv, RUN_GIT_CMD))
		return error(FAILED_RUN, pack_refs_cmd.argv[0]);

	if (run_command_v_opt(reflog.argv, RUN_GIT_CMD))
		return error(FAILED_RUN, reflog.argv[0]);

	if (run_command_v_opt(repack.argv, RUN_GIT_CMD))
		return error(FAILED_RUN, repack.argv[0]);

	if (prune_expire) {
		argv_array_push(&prune, prune_expire);
		if (quiet)
			argv_array_push(&prune, "--no-progress");
		if (run_command_v_opt(prune.argv, RUN_GIT_CMD))
			return error(FAILED_RUN, prune.argv[0]);
	}

	if (run_command_v_opt(rerere.argv, RUN_GIT_CMD))
		return error(FAILED_RUN, rerere.argv[0]);

My reading of this is that "git gc" runs, in sequence, pack-refs
(conditioned on gc.packrefs), reflog, repack, prune (conditioned on
the --prune option), and finally rerere.  IOW, "git gc" does run
repack.  The only difference between --aggressive and non-aggressive
modes is here:

	if (aggressive) {
		argv_array_push(&repack, "-f");
		argv_array_push(&repack, "--depth=250");
		if (aggressive_window > 0)
			argv_array_pushf(&repack, "--window=%d", aggressive_window);
	}

That is, --aggressive just passes the -f switch to git-repack, which
tells it not to reuse deltas (i.e. recompute them anew, AFAIU).

> (The doc for --aggressive says nothing about repack, only that it
> optimizes harder.)

See above: the description of aggressiveWindow setting quite clearly
says that --aggressive does run repack.  And in fact it does, indeed.

> > Stephen can now be pissed off by himself.
> 
> Yup.  And I won't waste time on suppressing my irritation while
> helping with GitForEmacsDevs

Too bad.

Re: Move to git is imminent - awaiting Stefan's approval

[Thien-Thi Nguyen]

[-- Attachment #1: Type: text/plain, Size: 684 bytes --]

() Eli Zaretskii <eliz@gnu.org>
() Tue, 07 Jan 2014 19:08:04 +0200

   Thanks.  "git gc" gets me from 1.18GB to 1.1GB.  I tried
   --aggressive, but that ran out of memory (trying to allocate 1GB)
   after some time and errored out.

I had the same problem in a similar circumstance (git-bzr).  See:

 http://lists.gnu.org/archive/html/emacs-devel/2014-01/msg00329.html

for the resolution (briefly: use "git repack" w/ ‘--window-memory’).

-- 
Thien-Thi Nguyen
   GPG key: 4C807502
   (if you're human and you know it)
      read my lisp: (responsep (questions 'technical)
                               (not (via 'mailing-list)))
                     => nil

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: Move to git is imminent - awaiting Stefan's approval

[Sven Axelsson]

On 7 January 2014 19:02, Thien-Thi Nguyen <ttn@gnu.org> wrote:
> () Eli Zaretskii <eliz@gnu.org>
> () Tue, 07 Jan 2014 19:08:04 +0200
>
>    Thanks.  "git gc" gets me from 1.18GB to 1.1GB.  I tried
>    --aggressive, but that ran out of memory (trying to allocate 1GB)
>    after some time and errored out.
>
> I had the same problem in a similar circumstance (git-bzr).  See:
>
>  http://lists.gnu.org/archive/html/emacs-devel/2014-01/msg00329.html
>
> for the resolution (briefly: use "git repack" w/ ‘--window-memory’).
>

Using `git repack -a -d -f --window=250 --depth=250` as mentioned
earlier in the conversation brings the .git folder down to 211 Mb for
me.

-- 
Sven Axelsson
++++++++++[>++++++++++>+++++++++++>++++++++++>++++++
>++++<<<<<-]>++++.+.++++.>+++++.>+.<<-.>>+.>++++.<<.
+++.>-.<<++.>>----.<++.>>>++++++.<<<<.>>++++.<----.

Re: Move to git is imminent - awaiting Stefan's approval

[Andreas Schwab]

Sven Axelsson <sven.axelsson@gmail.com> writes:

> Using `git repack -a -d -f --window=250 --depth=250` as mentioned
> earlier in the conversation brings the .git folder down to 211 Mb for
> me.

That's basically what git gc --aggressive does.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Move to git is imminent - awaiting Stefan's approval

[Sven Axelsson]

On 7 January 2014 19:47, Andreas Schwab <schwab@linux-m68k.org> wrote:
> Sven Axelsson <sven.axelsson@gmail.com> writes:
>
>> Using `git repack -a -d -f --window=250 --depth=250` as mentioned
>> earlier in the conversation brings the .git folder down to 211 Mb for
>> me.
>
> That's basically what git gc --aggressive does.

Looking at the source for git-gc I see that you are indeed correct.
Specifically, `git gc --aggressive` will run `git repack -d -l -f
--depth=250 --window=250`, among other things. Maybe this setting is
more appropriate: using the -a flag can give a better pack, but may
cause extra network traffic when fetching the repo over a dumb
protocol. All this can be found in tha man pages.

-- 
Sven Axelsson
++++++++++[>++++++++++>+++++++++++>++++++++++>++++++
>++++<<<<<-]>++++.+.++++.>+++++.>+.<<-.>>+.>++++.<<.
+++.>-.<<++.>>----.<++.>>>++++++.<<<<.>>++++.<----.

Re: Move to git is imminent - awaiting Stefan's approval

[Eli Zaretskii]

> Date: Tue, 7 Jan 2014 20:07:06 +0100
> From: Sven Axelsson <sven.axelsson@gmail.com>
> Cc: emacs <emacs-devel@gnu.org>
> 
> >> Using `git repack -a -d -f --window=250 --depth=250` as mentioned
> >> earlier in the conversation brings the .git folder down to 211 Mb for
> >> me.
> >
> > That's basically what git gc --aggressive does.
> 
> Looking at the source for git-gc I see that you are indeed correct.
> Specifically, `git gc --aggressive` will run `git repack -d -l -f
> --depth=250 --window=250`, among other things. Maybe this setting is
> more appropriate: using the -a flag can give a better pack, but may
> cause extra network traffic when fetching the repo over a dumb
> protocol. All this can be found in tha man pages.

Which was why I interpreted them to mean that gc runs repack.

Stephen can now be pissed off by himself.

Re: Move to git is imminent - awaiting Stefan's approval

[Stephen J. Turnbull]

Eli Zaretskii writes:

 > Which was why I interpreted them to mean that gc runs repack.

You read the source?  I find that hard to believe.  (The doc for
--aggressive says nothing about repack, only that it optimizes
harder.)

 > Stephen can now be pissed off by himself.

Yup.  And I won't waste time on suppressing my irritation while
helping with GitForEmacsDevs, since you're the main person I would
(normally) care to make such an effort for.

Re: Move to git is imminent - awaiting Stefan's approval

[Angelo Graziosi]

Il 07/01/2014 18.08, Eli Zaretskii ha scritto:
> Thanks.  "git gc" gets me from 1.18GB to 1.1GB.  I tried --aggressive,
> but that ran out of memory (trying to allocate 1GB) after some time
> and errored out.
>

Here (on Cygwin) it worked... Firstly, I did

$ git clone git://git.savannah.gnu.org/emacs.git emacs.git

and its size was

$ du -s emacs.git/
1,1G	emacs.git/

then I did

$ cd emacs.git/
$ git gc --aggressive
Counting objects: 735546, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (734446/734446), done.
Writing objects: 100% (735546/735546), done.
Total 735546 (delta 589814), reused 143177 (delta 0)
Checking connectivity: 735546, done.

which took almost 3 hours to be completed.

And the size is:

$ cd ..

$ du -s emacs.git/
327M	emacs.git/

which seems very good!


Ciao,
   Angelo.

Re: Move to git is imminent - awaiting Stefan's approval

[Eli Zaretskii]

> Date: Tue, 07 Jan 2014 21:08:51 +0100
> From: Angelo Graziosi <angelo.graziosi@alice.it>
> CC: stephen.berman@gmx.net, sven.axelsson@gmail.com, emacs-devel@gnu.org
> 
> $ cd emacs.git/
> $ git gc --aggressive
> Counting objects: 735546, done.
> Delta compression using up to 2 threads.
> Compressing objects: 100% (734446/734446), done.
> Writing objects: 100% (735546/735546), done.
> Total 735546 (delta 589814), reused 143177 (delta 0)
> Checking connectivity: 735546, done.
> 
> which took almost 3 hours to be completed.
> 
> And the size is:
> 
> $ cd ..
> 
> $ du -s emacs.git/
> 327M	emacs.git/
> 
> which seems very good!

I guess this was on Windows 7, right?  I succeeded to do this on a
64-bit Windows 7 as well, and it took about 2 hours.  And now I
understand why git ran out of memory on Windows XP: the memory
footprint of git-pack-objects goes up to 3GB, whereas Windows XP has
only 3GB address space for all the applications.

So I guess we will need to repack the repository on savannah as a
prerequisite for switching, and then make sure it is repacked
regularly afterwards.  Otherwise, people with low memory and/or slow
CPUs will need many hours to do this on their local machines.

Re: Move to git is imminent - awaiting Stefan's approval

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

> So I guess we will need to repack the repository on savannah as a
> prerequisite for switching, and then make sure it is repacked
> regularly afterwards.

I think what should be done is to increase the default configuration
value of pack.window for the git repos on savannah.  That should help
keeping the packs better packed in the long run.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Move to git is imminent - awaiting Stefan's approval

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: Angelo Graziosi <angelo.graziosi@alice.it>,  stephen.berman@gmx.net,  sven.axelsson@gmail.com,  emacs-devel@gnu.org
> Date: Wed, 08 Jan 2014 19:34:56 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > So I guess we will need to repack the repository on savannah as a
> > prerequisite for switching, and then make sure it is repacked
> > regularly afterwards.
> 
> I think what should be done is to increase the default configuration
> value of pack.window for the git repos on savannah.  That should help
> keeping the packs better packed in the long run.

Whatever the experts say, I know nowhere near enough about that.  The
goal is to let people clone 330MB as opposed to 1.1GB.

Thanks.

Re: Move to git is imminent - awaiting Stefan's approval

[Angelo Graziosi]

Il 08/01/2014 19.42, Eli Zaretskii ha scritto:

> Whatever the experts say, I know nowhere near enough about that.  The
> goal is to let people clone 330MB as opposed to 1.1GB.

Indeed.. Why should one clone 1.1 GiB and the repack all in 330 MiB? and 
shouldn't the remote repository "cleaned" (repacked etc..) regularly?

Ciao,
  Angelo.

Re: Move to git is imminent - awaiting Stefan's approval

[Stefan Monnier]

> Indeed.. Why should one clone 1.1 GiB and the repack all in 330 MiB? and
> shouldn't the remote repository "cleaned" (repacked etc..) regularly?

Because Git is so much more efficient,


        Stefan

Re: Move to git is imminent - awaiting Stefan's approval

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Andreas Schwab <schwab@linux-m68k.org>,  stephen.berman@gmx.net,  sven.axelsson@gmail.com,  emacs-devel@gnu.org
> Date: Wed, 08 Jan 2014 19:05:47 -0500
> 
> > Indeed.. Why should one clone 1.1 GiB and the repack all in 330 MiB? and
> > shouldn't the remote repository "cleaned" (repacked etc..) regularly?
> 
> Because Git is so much more efficient,

A joke, I presume.

But if not: pulling 1.1GB through the net instead of just a third or
forth of that is still a painful experience in some parts of the
world, and git can do nothing at all about that, even though it
compresses.

There really is no excuse for having such a badly packed repository on
Savannah.

Re: Move to git is imminent - awaiting Stefan's approval

[Bob Proulx]

Andreas Schwab wrote:
> Eli Zaretskii writes:
> > So I guess we will need to repack the repository on savannah as a
> > prerequisite for switching, and then make sure it is repacked
> > regularly afterwards.
> 
> I think what should be done is to increase the default configuration
> value of pack.window for the git repos on savannah.  That should help
> keeping the packs better packed in the long run.

All of the repositories are repacked monthly on the 4th.  As you can
see it is a heavy operation and therefore not run every day.  But this
thread seems to have exposed a deficiency in the current repacking.
Currently the script does:

  git --git-dir=$dir gc -q

Obvious from the discussion here that is not sufficient.  However:

Angelo Graziosi wrote:
> $ git gc --aggressive
> Counting objects: 735546, done.
> Delta compression using up to 2 threads.
> Compressing objects: 100% (734446/734446), done.
> Writing objects: 100% (735546/735546), done.
> Total 735546 (delta 589814), reused 143177 (delta 0)
> Checking connectivity: 735546, done.
> 
> which took almost 3 hours to be completed.

Eli Zaretskii wrote:
> And now I understand why git ran out of memory on Windows XP: the
> memory footprint of git-pack-objects goes up to 3GB, whereas Windows
> XP has only 3GB address space for all the applications.

Three hours and 3G of memory for this one repack!  I worry that
turning --agressive on globally will cause the server to fall over.
Also apparently careful tuning of threads is needed.  However the disk
and network savings look significant.

I have a couple of questions if someone already has the clone
available for a test.  Does a clone from the agressively repacked to
327M repository remain similarly compacted when cloned again from it?

If an agressively repacked repository is again repacked but this time
without the --agressive option does the size stay around 327M or does
it get expanded on the subsequent pass?  (Wondering if we can
periodically run 'git gc --agressive' on the larger git repositories
at a niced background task priority but not too often and still
achieve a good benefit for the time between agressive repacks.)

Bob

Re: Move to git is imminent - awaiting Stefan's approval

[Andreas Schwab]

Bob Proulx <bob@proulx.com> writes:

> Three hours and 3G of memory for this one repack!  I worry that
> turning --agressive on globally will cause the server to fall over.

IMHO --aggressive is too aggressive, and not useful for on-going
maintenance.  Incresing pack.window to 50 or so should already help
enough to improve the packing.

> If an agressively repacked repository is again repacked but this time
> without the --agressive option does the size stay around 327M or does
> it get expanded on the subsequent pass?

Unless you run repack with -f (ie. gc --aggresive) existing deltas are
reused, and only newly added objects are deltified.

> (Wondering if we can periodically run 'git gc --agressive' on the
> larger git repositories at a niced background task priority but not
> too often and still achieve a good benefit for the time between
> agressive repacks.)

Another option is to touch a <pack>.keep file for the largest pack so
that it is never touched again.  New objects will then be added to a
separate pack even after git gc.  If that large pack is already well
packed this should save some processing time.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Move to git is imminent - awaiting Stefan's approval

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Date: Sun, 12 Jan 2014 11:11:21 +0100
> Cc: emacs-devel@gnu.org
> 
> Increasing pack.window to 50 or so should already help enough to
> improve the packing.

Would it make sense to make such a change in our local clones as well?

Re: Move to git is imminent - awaiting Stefan's approval

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

>> Increasing pack.window to 50 or so should already help enough to
>> improve the packing.
>
> Would it make sense to make such a change in our local clones as well?

Yes, setting that in your personal .gitconfig is probably a good idea.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Move to git is imminent - awaiting Stefan's approval

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: bob@proulx.com,  emacs-devel@gnu.org
> Date: Sun, 12 Jan 2014 18:09:29 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Increasing pack.window to 50 or so should already help enough to
> >> improve the packing.
> >
> > Would it make sense to make such a change in our local clones as well?
> 
> Yes, setting that in your personal .gitconfig is probably a good idea.

Thanks.

Re: Move to git is imminent - awaiting Stefan's approval

[Bob Proulx]

Andreas Schwab wrote:
> Bob Proulx <bob@proulx.com> writes:
> > Three hours and 3G of memory for this one repack!  I worry that
> > turning --agressive on globally will cause the server to fall over.
> 
> IMHO --aggressive is too aggressive, and not useful for on-going
> maintenance.  Incresing pack.window to 50 or so should already help
> enough to improve the packing.

It sounds like I should schedule a full agressive repack once.  Once
being the operative word.

  git repack -a -d -f --window=250 --depth=250

And then as you suggest above ensure that the regularly routine
ongoing maintenance 'git gc' runs with a large enough pack window.

I see that the git-gc default is --window=250 so at that point nothing
more should need to be done.  The normal 'git gc' routine maintenance
should be okay at that point.  Is that right?  Seems like it.  We can
tweak this further as needed.

Certain times of day the vcs system is quite heavily loaded.  The vcs
system is a VM on a dom0 also hosting many other VMs.  At some times
of day the entire dom0 is very heavily I/O limited.  This has been an
ongoing problem and discussion.  It is definitely an ongoing problem.
I will run this during a less busy dom0 time.

I spent some time researching this problem and found this note and
some information in the entire thread useful and interesting.  It is a
long thread but there are some good gems in there.

  http://article.gmane.org/gmane.comp.gcc.devel/94613

> > If an agressively repacked repository is again repacked but this time
> > without the --agressive option does the size stay around 327M or does
> > it get expanded on the subsequent pass?
> 
> Unless you run repack with -f (ie. gc --aggresive) existing deltas are
> reused, and only newly added objects are deltified.

Sounds good.  I will note that -f forces a full (--no-reuse-delta)
repack of everything and should be avoided.

> > (Wondering if we can periodically run 'git gc --agressive' on the
> > larger git repositories at a niced background task priority but not
> > too often and still achieve a good benefit for the time between
> > agressive repacks.)
> 
> Another option is to touch a <pack>.keep file for the largest pack so
> that it is never touched again.  New objects will then be added to a
> separate pack even after git gc.  If that large pack is already well
> packed this should save some processing time.

That seems like a useful additional tweak for a large stage such as
this.  If nothing else it will help out the backup since that file
won't be changing on a routine basis and will remain static for the
purposes of backup transfer.  This could be applied to several of the
large repositories.

It seems that on the client side after a new clone that this tweak is
not propagated.  It seems that if there are multiple packs on the
server side that they are combined into a single pack file.  But
without being repacked.  Therefore downstream clients that wish this
would need to do it manually after a clone.

Bob

Re: Move to git is imminent - awaiting Stefan's approval

[Andreas Schwab]

Bob Proulx <bob@proulx.com> writes:

> I see that the git-gc default is --window=250

Only if you add --aggressive (which uses gc.aggressiveWindow to override
the default for git repack, which is pack.window).

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Move to git is imminent - awaiting Stefan's approval

[Bob Proulx]

Andreas Schwab wrote:
> Bob Proulx <bob@proulx.com> writes:
> > I see that the git-gc default is --window=250
> 
> Only if you add --aggressive (which uses gc.aggressiveWindow to override
> the default for git repack, which is pack.window).

Ah...  Good clarification.  I have updated pack.window and set the new
value to 250.

Bob

Re: Move to git is imminent - awaiting Stefan's approval

[Bob Proulx]

Bob Proulx wrote:
> It sounds like I should schedule a full agressive repack once.  Once
> being the operative word.
> 
>   git repack -a -d -f --window=250 --depth=250

This has been done and the repository is now down to 285M.  I set the
number of pack.threads=2 and it ran for 104 minutes to completion.

(I had experimented letting it use all available 7-cores but that ran
94 minutes and ran out of memory at the 3G 32-bit limit.  The progress
indicator took 15 minutes to get to 90% and then spent the next 80
minutes getting no further than 92%.)

> > Another option is to touch a <pack>.keep file for the largest pack so
> > that it is never touched again.  New objects will then be added to a
> > separate pack even after git gc.  If that large pack is already well
> > packed this should save some processing time.
> 
> That seems like a useful additional tweak for a large stage such as
> this.  If nothing else it will help out the backup since that file
> won't be changing on a routine basis and will remain static for the
> purposes of backup transfer.  This could be applied to several of the
> large repositories.
> 
> It seems that on the client side after a new clone that this tweak is
> not propagated.  It seems that if there are multiple packs on the
> server side that they are combined into a single pack file.  But
> without being repacked.  Therefore downstream clients that wish this
> would need to do it manually after a clone.

I also touched the .keep file for the resulting pack file so that it
would remain static.

I checked out a local clone and the size was 329M.

Bob

P.S. I saw the other discussion thread talking about possibly redoing
the repository.  If that happens let me know and I will repack just as
above again.

Re: Move to git is imminent - awaiting Stefan's approval

[Achim Gratz]

Bob Proulx writes:
> I have a couple of questions if someone already has the clone
> available for a test.  Does a clone from the agressively repacked to
> 327M repository remain similarly compacted when cloned again from it?

Once the whole repository is repacked it keeps those pack decisions,
also across clones.  The pack strategy for new commits would typically
be near-optimal unless someone fast-imported a large branch again (in
that case it should preferrably be locally repacked before getting
pushed to Savannah).

> If an agressively repacked repository is again repacked but this time
> without the --agressive option does the size stay around 327M or does
> it get expanded on the subsequent pass?

I don't think a plain gc will expand any old packs, in any case the
benefits of "git gc --agressive" are supposed to last for a long time.

>(Wondering if we can periodically run 'git gc --agressive' on the
>larger git repositories at a niced background task priority but not too
>often and still achieve a good benefit for the time between agressive
>repacks.)

Just a manual "git gc --aggressive" once in a while should be enough.
If you are going to do it as a cron-job, I'm not sure if it's a good
idea to nice it.  I think you'd rather want to limit the number of
threads it uses, which also limits the memory footprint, which is
probably the bigger problem anyway and should probably be tuned further
depending on how beefy the server is.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for Waldorf Q V3.00R3 and Q+ V3.54R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: Move to git is imminent - awaiting Stefan's approval

[Bob Proulx]

Achim Gratz wrote:
> Bob Proulx writes:
> >(Wondering if we can periodically run 'git gc --agressive' on the
> >larger git repositories at a niced background task priority but not too
> >often and still achieve a good benefit for the time between agressive
> >repacks.)
> 
> Just a manual "git gc --aggressive" once in a while should be enough.
> If you are going to do it as a cron-job, I'm not sure if it's a good
> idea to nice it.

Due to the performance issues some of the Savannah admins have started
nicing background tasks as to hint to the scheduler.  Hoping that
interactive tasks get more priority.  Because at some times of day
there has been a problem with the server being slow enough to cause
timeouts.  Especially the web frontend to vcs.  Therefore if it isn't
a time critical task then running it niced is a hint to the scheduler.

The problem is that as a VM all of the system performance information
is fake data and can't show the actual system state.  And since we are
currently I/O limited at times the system is often blocked waiting for
I/O in which case the cpu priority doesn't matter.  There is more cpu
resource to go around than backplane disk I/O.  But from the VM itself
there isn't any useful performance data.  Only the FSF admins have
access to the dom0 and we count on them for real performance
information from it.  Unfortunately the FSF admin time to work on
server performance problems has been limited.

> I think you'd rather want to limit the number of threads it uses,
> which also limits the memory footprint, which is probably the bigger
> problem anyway and should probably be tuned further depending on how
> beefy the server is.

The dom0 server is an 8-core AMD Opteron running Xen for virtualization.
It has sufficient RAM.  But IMNHO too much I/O from all of the VMs
running on it.  The vcs VM is allocated seven xen cores.

The hint about threads is a good one and I will keep an eye on it.

Bob

Re: Move to git is imminent - awaiting Stefan's approval

[Eli Zaretskii]

> Date: Sat, 11 Jan 2014 23:27:22 -0700
> From: Bob Proulx <bob@proulx.com>
> 
> Eli Zaretskii wrote:
> > And now I understand why git ran out of memory on Windows XP: the
> > memory footprint of git-pack-objects goes up to 3GB, whereas Windows
> > XP has only 3GB address space for all the applications.
> 
> Three hours and 3G of memory for this one repack!

It was only 1 hour in my case (I guess Angelo has a slower machine).

Also, limiting the number of threads to 2 or even 1 radically lowers
the required memory usage.

But that doesn't mean you need necessarily run "gc --agressive",
please listen to Andreas on this matter.

> I have a couple of questions if someone already has the clone
> available for a test.  Does a clone from the agressively repacked to
> 327M repository remain similarly compacted when cloned again from it?

It grows slowly after that.  E.g., in the 3 days since then it grew by
12MB.  I don't know whether this will level out later, a few days is
not enough to tell.

> If an agressively repacked repository is again repacked but this time
> without the --agressive option does the size stay around 327M or does
> it get expanded on the subsequent pass?

No, it doesn't expand, at least not on my system.  In fact, "git gc"
just slashed the storage by 10MB out of 12MB that it gained since I
ran "gc --agressive".

Again, this is just to give you some numbers, I don't want to suggest
what should be done on Savannah.

Re: Move to git is imminent - awaiting Stefan's approval

[Angelo Graziosi]

Il 08/01/2014 18.32, Eli Zaretskii ha scritto:
>> Date: Tue, 07 Jan 2014 21:08:51 +0100
>> From: Angelo Graziosi <angelo.graziosi@alice.it>
>> CC: stephen.berman@gmx.net, sven.axelsson@gmail.com, emacs-devel@gnu.org
>>
>> $ cd emacs.git/
>> $ git gc --aggressive
>> Counting objects: 735546, done.
>> Delta compression using up to 2 threads.
>> Compressing objects: 100% (734446/734446), done.
>> Writing objects: 100% (735546/735546), done.
>> Total 735546 (delta 589814), reused 143177 (delta 0)
>> Checking connectivity: 735546, done.
>>
>> which took almost 3 hours to be completed.
>>
>> And the size is:
>>
>> $ cd ..
>>
>> $ du -s emacs.git/
>> 327M	emacs.git/
>>
>> which seems very good!
>
> I guess this was on Windows 7, right?  I succeeded to do this on a

NO!! It is on Win XP 32 bit (SP3) with AMD Athlon 64 X2 Dual Core 
Processor 3800+ and 1.7GiB of RAM.

On the same machine I have Kubuntu 12.04 and "git gc --aggressive" works 
there too..

BTW, I noticed that running "git pull" to have the local repository 
updated, the size increases of about 1 MiB.

> 64-bit Windows 7 as well, and it took about 2 hours.  And now I
> understand why git ran out of memory on Windows XP: the memory
> footprint of git-pack-objects goes up to 3GB, whereas Windows XP has
> only 3GB address space for all the applications.

Here, on XP, Task Manager shows a memory usage under 1500 MiB (about 1.4 
GiB)

In any case this

> So I guess we will need to repack the repository on savannah as a
> prerequisite for switching, and then make sure it is repacked
> regularly afterwards.  Otherwise, people with low memory and/or slow
> CPUs will need many hours to do this on their local machines.
>

should be mandatory..

Ciao,
  Angelo.

Re: Move to git is imminent - awaiting Stefan's approval

[Eli Zaretskii]

> Date: Wed, 08 Jan 2014 22:55:21 +0100
> From: Angelo Graziosi <angelo.graziosi@alice.it>
> CC: schwab@suse.de, stephen.berman@gmx.net, sven.axelsson@gmail.com, 
>  emacs-devel@gnu.org
> 
> Il 08/01/2014 18.32, Eli Zaretskii ha scritto:
> >> Date: Tue, 07 Jan 2014 21:08:51 +0100
> >> From: Angelo Graziosi <angelo.graziosi@alice.it>
> >> CC: stephen.berman@gmx.net, sven.axelsson@gmail.com, emacs-devel@gnu.org
> >>
> >> $ cd emacs.git/
> >> $ git gc --aggressive
> >> Counting objects: 735546, done.
> >> Delta compression using up to 2 threads.
> >> Compressing objects: 100% (734446/734446), done.
> >> Writing objects: 100% (735546/735546), done.
> >> Total 735546 (delta 589814), reused 143177 (delta 0)
> >> Checking connectivity: 735546, done.
> >>
> >> which took almost 3 hours to be completed.
> >>
> >> And the size is:
> >>
> >> $ cd ..
> >>
> >> $ du -s emacs.git/
> >> 327M	emacs.git/
> >>
> >> which seems very good!
> >
> > I guess this was on Windows 7, right?  I succeeded to do this on a
> 
> NO!! It is on Win XP 32 bit (SP3) with AMD Athlon 64 X2 Dual Core 
> Processor 3800+ and 1.7GiB of RAM.

The secret that unlocked this mystery was this:

> >> Delta compression using up to 2 threads.
                                   ^^^^^^^^^
On my machine, which is a Core i7, this originally said "8 threads".
So I did this: 

  git config --global pack.threads "2"

then ran "git gc --aggressive" again.  That again ran out of memory,
but much later in the process.  The next step was obvious:

  git config --global pack.threads "1"

With that, "git gc --aggressive" churned away for a little bit more
than an hour, and brought me to this:

  $ du -sh .
  323M    .

> > 64-bit Windows 7 as well, and it took about 2 hours.  And now I
> > understand why git ran out of memory on Windows XP: the memory
> > footprint of git-pack-objects goes up to 3GB, whereas Windows XP has
> > only 3GB address space for all the applications.
> 
> Here, on XP, Task Manager shows a memory usage under 1500 MiB (about 1.4 
> GiB)

With 1 thread, it never goes above 850MB.

Re: Move to git is imminent - awaiting Stefan's approval

[Angelo Graziosi]

Il 07/01/2014 15.41, Stephen Berman ha scritto:

> Is it correct that main or master is the same thing as the Emacs trunk
> branch?  If so, I wonder why the git repository containing just this is
> so large: my bzr Emacs repository only contains the trunk (as a bound
> branch), several local branches of it, and the emacs-24 branch; the
> .bzr/ directory of the repository, containing all the history, is only
> 428 MiB, not 1.1 GiB (the trunk directory itself contains an additional
> 129.3 MiB, but the total is still half the size of the git repo).  Why
> is the git repository so much larger than the bzr repository?

As I noticed before 
(http://lists.gnu.org/archive/html/emacs-devel/2014-01/msg00515.html), 
it contains emacs.git/.git directory of about 930 MiB...

Ciao,
  Angelo.

Re: Move to git is imminent - awaiting Stefan's approval

[Eli Zaretskii]

> Date: Tue, 7 Jan 2014 13:27:22 +0100
> From: Sven Axelsson <sven.axelsson@gmail.com>
> Cc: Andreas Schwab <schwab@linux-m68k.org>, emacs <emacs-devel@gnu.org>
> 
> Depth 1 truncates all history except for the most recent
> commit.

I think it keeps the 2 last commits, try "git log" in such a clone.

> From `man git clone`:
> 
> A shallow repository has a number of limitations (you cannot clone or
> fetch from it, nor push from nor into it), but is adequate if you are
> only interested in the recent history of a large project with a long
> history, and would want to send in fixes as patches.

For these reasons, this kind of repository is not recommended for
"Emacs devs".

Re: Move to git is imminent - awaiting Stefan's approval

[Antonio Nikishaev]

Sven Axelsson <sven.axelsson@gmail.com> writes:

>> Instead, trying
>>
>> $ git clone --depth 1 git://git.savannah.gnu.org/emacs.git emacs.git
>>
>> the local repository has dimension 160M which is a little greater
>> than my current BZR (checkout --lightweight) repository (126M), but
>> acceptable...
>>
>> Suppose now I want identify which "revision" introduced a bug. I am
>> able to do this with a binary search using the BZR command
>>
>> $ bzr up -r ARG
>>
>> How can I do this with the "minimal" clone done with GIT?
>
> Git does not have any concept like Bazaar's lightweight checkouts -
> nor does any other dvcd afaik.

Darcs has clever lazy repos. (Everything works. If the stuff which is
not yet in the repository is needed, it will be downloaded.)

Just FYI

Re: Move to git is imminent - awaiting Stefan's approval

[David Kastrup]

Angelo Graziosi <angelo.graziosi@alice.it> writes:

> Instead, trying
>
> $ git clone --depth 1 git://git.savannah.gnu.org/emacs.git emacs.git
>
> the local repository has dimension 160M which is a little greater than
> my current BZR (checkout --lightweight) repository (126M), but
> acceptable...
>
> Suppose now I want identify which "revision" introduced a bug. I am
> able to do this with a binary search using the BZR command
>
> $ bzr up -r ARG
>
> How can I do this with the "minimal" clone done with GIT?

Uh, not at all?  Either you fetch previous revisions and can make use of
them, or not.  I think there is some command for extending a shallow
clone, but of course that will have the respective impact on the
repository size.

-- 
David Kastrup

Re: Move to git is imminent - awaiting Stefan's approval

[Eli Zaretskii]

> Date: Tue, 07 Jan 2014 11:24:12 +0100
> From: Angelo Graziosi <angelo.graziosi@alice.it>
> Cc: emacs <emacs-devel@gnu.org>
> 
> >> $ du -s emacs-trunk/
> >> 126M	emacs-trunk/
> >>
> >> instead the git repository has dimension
> >>
> >> $ du -s emacs.git/
> >> 1.1G	emacs.git/
> >
> > By default git downloads the history of all branches.  You can use
> > --single-branch to only clone a single branch.  This is the default if
> > you create a shallow clone.
> 
> David Kastrup wrote:
> 
> > For the build?  No.  git clone has the following options:
> >
> >         --depth <depth>
> ...
> > --[no-]single-branch
> ...
> 
> OK, trying
> 
> $ git clone --single-branch git://git.savannah.gnu.org/emacs.git emacs.git
> 
> the local repository has dimension 1.1G, so probably I need a more 
> explicit example...

The command you tried is correct, it's just that it doesn't save too
much disk space because the trunk has almost all of the revisions.

> Instead, trying
> 
> $ git clone --depth 1 git://git.savannah.gnu.org/emacs.git emacs.git
> 
> the local repository has dimension 160M which is a little greater than 
> my current BZR (checkout --lightweight) repository (126M), but acceptable...

This is not recommended at all, as such a shallow repository is
severely limited: you cannot clone from it, nor push from or to it.
You also have almost no history, and git does not support bzr-like
operation whereby commands in lightweight checkouts that need history
information go to the server.  So you will be unable to walk through
history and bisect.

> Suppose now I want identify which "revision" introduced a bug. I am able 
> to do this with a binary search using the BZR command
> 
> $ bzr up -r ARG
> 
> How can I do this with the "minimal" clone done with GIT?

You don't, see above.  With a full repository, or at least one that
was cloned with --single-branch, the equivalent of "bzr up" with git
is "git checkout", and the revision which you want is specified by
either is SHA1 signature or relatively, as in HEAD~20.

Re: Move to git is imminent - awaiting Stefan's approval

[Angelo Graziosi]

Il 07/01/2014 0.01, Andreas Schwab ha scritto:
> A good way to identify a git branch is to use git describe.  However it
> prefers to have an annotated (release) tag available to base on, which
> we currently don't have (bzr doesn't have the concept of annotated
> tags).

Just for completeness...

I see that I can download the trunk source with

$ wget 
http://git.savannah.gnu.org/cgit/emacs.git/snapshot/emacs-trunk.tar.gz

but how I can identify it?

After unpacking it, the tree should contain at least a text file with a 
revision number or the SHA sum of the last commit... or not?


Ciao,
  Angelo.

Re: Move to git is imminent - awaiting Stefan's approval

[Andreas Schwab]

Angelo Graziosi <angelo.graziosi@alice.it> writes:

> I see that I can download the trunk source with
>
> $ wget
> http://git.savannah.gnu.org/cgit/emacs.git/snapshot/emacs-trunk.tar.gz
>
> but how I can identify it?

Normally git get-tar-commit-id should do it, but the archive wasn't
produced by git archive, or it was created from a tree ID instead of a
commit ID.  That looks like a bug in cgit.

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: Move to git is imminent - awaiting Stefan's approval

[Angelo Graziosi]

Il 07/01/2014 15.24, Andreas Schwab ha scritto:
> Angelo Graziosi <angelo.graziosi@alice.it> writes:
>
>> I see that I can download the trunk source with
>>
>> $ wget
>> http://git.savannah.gnu.org/cgit/emacs.git/snapshot/emacs-trunk.tar.gz
>>
>> but how I can identify it?
>
> Normally git get-tar-commit-id should do it, but the archive wasn't
> produced by git archive, or it was created from a tree ID instead of a
> commit ID.  That looks like a bug in cgit.

I have downloaded

1) 
http://git.savannah.gnu.org/cgit/emacs.git/snapshot/emacs-e42fcf1482e2c9e9d93f459985d13dbbabc61956.tar.gz

2) http://git.savannah.gnu.org/cgit/emacs.git/snapshot/emacs-trunk.tar.gz

which correspond to the same trunk but I cannot identify the trunk: the 
command "git get-tar-commit-id" hangs.

If I have understood, with GIT the trunk is identified with the SHA1 
string like "e42fcf1482e2c9e9d93f459985d13dbbabc61956", so that in the 
first case I know the trunk "revision". Anyway, I would prefer that that 
string is stored inside the tree given by those tar-balls. For example 
in a text file, LAST-COMMIT-ID, so that one can know it with

$ cat emacs-e42fcf1482e2c9e9d93f459985d13dbbabc61956/LAST-COMMIT-ID

or

$ cat emacs-trunk/LAST-COMMIT-ID


In this way, I can create a build script with this skeleton:

...
wget http://git.savannah.gnu.org/cgit/emacs.git/snapshot/emacs-trunk.tar.gz

tar -xf emacs-trunk.tar.gz

rev=`cat emacs-trunk/LAST-COMMIT-ID`

autogen.sh
configure...
make bootstrap
make install DESTDIR=...

tar -cf emacs-$rev-bin.tar.xz $INSTDIR


Ciao,
  Angelo.

Re: Move to git is imminent - awaiting Stefan's approval

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Date: Tue, 07 Jan 2014 00:01:45 +0100
> Cc: emacs <emacs-devel@gnu.org>
> 
> Angelo Graziosi <angelo.graziosi@alice.it> writes:
> 
> > Now trying
> >
> > $ git clone git://git.savannah.gnu.org/emacs.git emacs.git
> >
> >
> > I don't get any revision number.. So in my future bug report how the trunk
> > will be identified?
> 
> A good way to identify a git branch is to use git describe.  However it
> prefers to have an annotated (release) tag available to base on, which
> we currently don't have

Indeed:

  $ git describe
  fatal: No annotated tags can describe 'e1d32fe4408170b992528a2da4b917a63e86fbc4'

I think Angelo wants one of the following, which will all produce
exactly the same output:

  git log --pretty='format:%H' @^..
  git rev-list  @^..
  git rev-parse --verify HEAD

Re: Move to git is imminent - awaiting Stefan's approval

[Rüdiger Sonderfeld]

On Tuesday 07 January 2014 18:14:45 Eli Zaretskii wrote:
>   $ git describe
>   fatal: No annotated tags can describe
> 'e1d32fe4408170b992528a2da4b917a63e86fbc4'
> 
> I think Angelo wants one of the following, which will all produce
> exactly the same output:
> 
>   git log --pretty='format:%H' @^..
>   git rev-list  @^..
>   git rev-parse --verify HEAD

Or

  git describe --always
  git show --pretty=oneline
  git show --oneline

(Produces a different output though)

Regards
Rüdiger

RE: Move to git is imminent - awaiting Stefan's approval

[Drew Adams]

>  5. Have the procedures and the recommended workflows described on
>     the wiki, similar to what was done with bzr migration.

Only a wiki description?

Isn't there an Info manual for developing Emacs using its chosen dev
tools, procedures, guidelines, etc.?  If not, shouldn't there be?

Re: Move to git is imminent - awaiting Stefan's approval

[Eli Zaretskii]

> Date: Mon, 6 Jan 2014 09:41:25 -0800 (PST)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: emacs-devel@gnu.org
> 
> >  5. Have the procedures and the recommended workflows described on
> >     the wiki, similar to what was done with bzr migration.
> 
> Only a wiki description?

That's what we have for bzr, so we should have at least that much.

> Isn't there an Info manual for developing Emacs using its chosen dev
> tools, procedures, guidelines, etc.?

No.

> If not, shouldn't there be?

It should.  And I also should have $5M on my bank account (it's New
year, right?).  But I don't, and we don't.  Volunteers are welcome
(for both jobs).

Move to git is imminent - awaiting Stefan's approval

[Eric S. Raymond]

The thread on the git move proposal is showing signs of exhaustion and
terminal topic drift.  We can ask Karl Fogel for a poll count but I
think we all know how that would turn out; the pro-git vote has
been overwhelming.

I have asked Andreas Schwab and he reports the Savannah git mirror
ready for production use. Having examined it, I agree. Andreas and I
have discussed the switchover and believe it should consist of the
following steps:

1. Andreas will turn off bzr commit mirroring.

2. Andreas will enter a small documentation commit recording the changeover.

3. Andreas will announce on the dev list that the git repo is live for
   developer pushes.

4. I will do the work required to update /etc and /admin for git use
   over the following few days.

I am now recommending that Andreas perform steps 1-3 at his convenience, no
sooner than 2014-01-06T11:59:00  and no later than 2014-01-07T11:59:00 -
that is, sometime tomorrow.

Consideration of workflow changes is explicitly deferred. Developers
should continue writing ChangeLog entries as per established custom.

I want to extend a thank you to everyone who participated in the
debate, *including* the anti-git dissenters, for generally sticking to
real issues and maintaining civility.

Stefan, please either (a) confirm that you as the official maintainer
are authorizing this, or (b) put a hold on it and explain what 
additional steps or conditions you deem necessary before we move.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

A ``decay in the social contract'' is detectable; there is a growing
feeling, particularly among middle-income taxpayers, that they are not
getting back, from society and government, their money's worth for
taxes paid. The tendency is for taxpayers to try to take more control
of their finances...	-- IRS Strategic Plan, (May 1984)

Re: Move to git is imminent - awaiting Stefan's approval

[Jay Belanger]

> 1. Andreas will turn off bzr commit mirroring.
>
> 2. Andreas will enter a small documentation commit recording the changeover.
>
> 3. Andreas will announce on the dev list that the git repo is live for
>    developer pushes.
>
> 4. I will do the work required to update /etc and /admin for git use
>    over the following few days.

When we changed to bzr, the "Bzr for Emacs Devs" page on the wiki
was invaluable.  For the changeover, could someone (please?) do
something similar for git? 

Thanks,
Jay

Re: Move to git is imminent - awaiting Stefan's approval

[Eric S. Raymond]

Jay Belanger <jay.p.belanger@gmail.com>:
> When we changed to bzr, the "Bzr for Emacs Devs" page on the wiki
> was invaluable.  For the changeover, could someone (please?) do
> something similar for git? 

I'll put that on my todo list.  But updating the in-tree stuff comes first.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: Move to git is imminent - awaiting Stefan's approval

[Jay Belanger]

>> When we changed to bzr, the "Bzr for Emacs Devs" page on the wiki
>> was invaluable.  For the changeover, could someone (please?) do
>> something similar for git? 
>
> I'll put that on my todo list.  But updating the in-tree stuff comes first.

Thanks, and of course.

Re: Move to git is imminent - awaiting Stefan's approval

[Stephen J. Turnbull]

Jay Belanger writes:

 > When we changed to bzr, the "Bzr for Emacs Devs" page on the wiki
 > was invaluable.  For the changeover, could someone (please?) do
 > something similar for git? 

http://www.python.org/dev/peps/pep-0374/ is a start.

Re: Move to git is imminent - awaiting Stefan's approval

[Eric S. Raymond]

Stephen J. Turnbull <stephen@xemacs.org>:
> Jay Belanger writes:
> 
>  > When we changed to bzr, the "Bzr for Emacs Devs" page on the wiki
>  > was invaluable.  For the changeover, could someone (please?) do
>  > something similar for git? 
> 
> http://www.python.org/dev/peps/pep-0374/ is a start.

I'm working this problem, which is why I've been quiet the last day.

Results to be announced shortly.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: Move to git is imminent - awaiting Stefan's approval

[Eli Zaretskii]

> From: esr@thyrsus.com (Eric S. Raymond)
> Date: Mon,  6 Jan 2014 11:51:08 -0500 (EST)
> 
> I have asked Andreas Schwab and he reports the Savannah git mirror
> ready for production use. Having examined it, I agree. Andreas and I
> have discussed the switchover and believe it should consist of the
> following steps:
> 
> 1. Andreas will turn off bzr commit mirroring.
> 
> 2. Andreas will enter a small documentation commit recording the changeover.
> 
> 3. Andreas will announce on the dev list that the git repo is live for
>    developer pushes.
> 
> 4. I will do the work required to update /etc and /admin for git use
>    over the following few days.
> 
> I am now recommending that Andreas perform steps 1-3 at his convenience, no
> sooner than 2014-01-06T11:59:00  and no later than 2014-01-07T11:59:00 -
> that is, sometime tomorrow.

The above is not enough.  I would suggest to have the following
additional steps, before the switch:

 5. Have the procedures and the recommended workflows described on the
    wiki, similar to what was done with bzr migration.

 6. Describe (and test if needed) the procedure for migrating local
    bzr branches into git without losing history (yes, I have a couple
    in the works), and describe that on the wiki as well.

 7. What about the mail messages to emacs-diffs mailing list?  That
    should be working as well, and support pushes to non-trunk
    branches.

 8. There's the emacs-bzr-version whose value gets copied into the bug
    reports.  This should be replaced by the suitable git equivalent,
    or else the bug reports (of which we have quite a few each day)
    will not identify the version correctly.

I suggest to leave some time, certainly more than one day, for others
to come up with additional activities that need to be completed before
the switch.  I see no special reason to rush.

Re: Move to git is imminent - awaiting Stefan's approval

[Stefan Monnier]

> The above is not enough.  I would suggest to have the following
> additional steps, before the switch:

Sounds right.

I'd also like:

- Improve vc-git.el so that it can automatically enable smerge-mode when
  opening a conflicted file and (probably conditional on a config var)
  mark the file as "not conflicted any more" when saving with no
  remaining diff3 markers.
  This currently works in vc-bzr.el (and vc-svn.el as well, IIRC).

- Improve vc-git.el with vc-git-conflicted-files so that
  vc-find-conflicted-files works for Git as well.

- Release 24.4 (i.e. I'd rather not deal with the move before we
  release 24.4).

- The $5M mentioned by Eli sounds good as well.  Tho somehow I have the
  impression this isn't gonna happen.

> - He's not bringing any new argument that has not been discussed
> before; the only difference is that Richard and Stefan seem to accept
> now that Bazaar is beyond hope.

Side note: I've considered Bzr dead for a while now (hence the move to
Git for `elpa').  FWIW, my vote is on "have better things to do with my
time than worry about which VCS we use".
IOW the main change so far is that Richard does not insist on us using
Bzr any more.


        Stefan

Re: Move to git is imminent - awaiting Stefan's approval

[Óscar Fuentes]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

[snip]

> I'd also like:
>
> - Improve vc-git.el so that it can automatically enable smerge-mode when
>   opening a conflicted file

We discussed a patch for this. IIRC it had the inconvenience of
requiring two git calls for each file, although in theory it was
possible to cache the results for the entire tree.

>   and (probably conditional on a config var)
>   mark the file as "not conflicted any more" when saving with no
>   remaining diff3 markers.
>   This currently works in vc-bzr.el (and vc-svn.el as well, IIRC).

The config var is important.

Here, conflict is considered solved only after testing the changes (does
it compile? does it pass the regression tests?.)

[snip]

Re: Move to git is imminent - awaiting Stefan's approval

[Stefan Monnier]

>> and (probably conditional on a config var)
[...]
> The config var is important.

I know some people are very ticklish about it, which is why I mentioned
the config var.  I think the default should be to have the feature
enabled.

> Here, conflict is considered solved only after testing the changes (does
> it compile? does it pass the regression tests?.)

When Git resolves concurrent changes automatically, it doesn't mark the
file as "there was an unchecked change in here", so for the same reason
I think the default behavior of vc-git + smerge-mode should be to remove
the "conflicted" mark on files after the user has removed all
diff3 markers.


        Stefan

Re: Move to git is imminent - awaiting Stefan's approval

[Leo Liu]

On 2014-01-07 05:09 +0800, Stefan Monnier wrote:
> FWIW, my vote is on "have better things to do with my time than worry
> about which VCS we use".

Pragmatism triumphs ;)

Leo

Re: Move to git is imminent - awaiting Stefan's approval

[Thierry Volpiatto]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> The above is not enough.  I would suggest to have the following
>> additional steps, before the switch:
>
> Sounds right.
>
> I'd also like:
>
> - Improve vc-git.el so that it can automatically enable smerge-mode when
>   opening a conflicted file and (probably conditional on a config var)
>   mark the file as "not conflicted any more" when saving with no
>   remaining diff3 markers.
>   This currently works in vc-bzr.el (and vc-svn.el as well, IIRC).
>
> - Improve vc-git.el with vc-git-conflicted-files so that
>   vc-find-conflicted-files works for Git as well.

You can use git-mergetool with ediff-merge for this.

http://stackoverflow.com/questions/1817370/using-ediff-as-git-mergetool

When git fail to merge and signal a conflict, run git-mergetool in e.g
eshell will bring you a new frame with ediff-merge to resolve the
conflict manually, always worked fine for me.

-- 
Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997 

Re: Move to git is imminent - awaiting Stefan's approval

[Stefan Monnier]

> You can use git-mergetool with ediff-merge for this.

I find ediff too heavy, so it would have to be smerge-mode instead.

But, while I'd be happy to take patches to better support git-mergetool
(if needed), I prefer using the vc-find-conflicted-files workflow.
It could probably run git-mergetool internally.


        Stefan

Re: Move to git is imminent - awaiting Stefan's approval

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Tue, 07 Jan 2014 08:45:47 -0500
> Cc: emacs-devel@gnu.org
> 
> while I'd be happy to take patches to better support git-mergetool
> (if needed), I prefer using the vc-find-conflicted-files workflow.

Same here.  It's so dead easy that it's hard to beat: just visit the
offending file (which is automatically put in SMerge mode), edit to
resolve the conflict and remove the conflict markers, then save -- and
the VC back-end will automatically do whatever it takes to finish
conflict resolution.

Re: Move to git is imminent - awaiting Stefan's approval

[Barry Warsaw]

[-- Attachment #1: Type: text/plain, Size: 480 bytes --]

On Jan 06, 2014, at 04:09 PM, Stefan Monnier wrote:

>- Improve vc-git.el so that it can automatically enable smerge-mode when
>  opening a conflicted file and (probably conditional on a config var)
>  mark the file as "not conflicted any more" when saving with no
>  remaining diff3 markers.
>  This currently works in vc-bzr.el (and vc-svn.el as well, IIRC).

Yes please!  It's one huge feature I'm missing when working in git repos as
opposed to bzr repos.

-Barry

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Move to git is imminent - awaiting Stefan's approval

[Stefan Monnier]

> Yes please!  It's one huge feature I'm missing when working in git repos as
> opposed to bzr repos.

I'm so glad to see people like it!


        Stefan "who had no idea!"

Re: Move to git is imminent - awaiting Stefan's approval

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Wed, 08 Jan 2014 19:04:34 -0500
> Cc: emacs-devel@gnu.org
> 
> > Yes please!  It's one huge feature I'm missing when working in git repos as
> > opposed to bzr repos.
> 
> I'm so glad to see people like it!

Count me in as well.  Great feature, indeed: seamless, does it job
silently and well.  When I first saw it, and "bzr conflicts" suddenly
said nothing at all after saving a fixed file, it was one of those WTF
moments.

Re: Move to git is imminent - awaiting Stefan's approval

[David Engster]

Eli Zaretskii writes:
>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Date: Wed, 08 Jan 2014 19:04:34 -0500
>> Cc: emacs-devel@gnu.org
>> 
>> > Yes please!  It's one huge feature I'm missing when working in git repos as
>> > opposed to bzr repos.
>> 
>> I'm so glad to see people like it!
>
> Count me in as well.  Great feature, indeed: seamless, does it job
> silently and well.  When I first saw it, and "bzr conflicts" suddenly
> said nothing at all after saving a fixed file, it was one of those WTF
> moments.

I always thought it was a Bazaar feature...

-David

Re: Move to git is imminent - awaiting Stefan's approval

[Juanma Barranquero]

On Thu, Jan 9, 2014 at 7:32 AM, Eli Zaretskii <eliz@gnu.org> wrote:

> Great feature, indeed: seamless, does it job
> silently and well.  When I first saw it, and "bzr conflicts" suddenly
> said nothing at all after saving a fixed file, it was one of those WTF
> moments.

My thoughts exactly.

    J

Re: Move to git is imminent - awaiting Stefan's approval

[Juanma Barranquero]

On Mon, Jan 6, 2014 at 5:51 PM, Eric S. Raymond <esr@thyrsus.com> wrote:

> I am now recommending that Andreas perform steps 1-3 at his convenience, no
> sooner than 2014-01-06T11:59:00  and no later than 2014-01-07T11:59:00 -
> that is, sometime tomorrow.

Why? Where's the hurry? We're in a freeze, we should be concentrating
on fixing bugs, not learning new ways and new procedures...

    J

Re: Move to git is imminent - awaiting Stefan's approval

[Bastien]

Juanma Barranquero <lekktu@gmail.com> writes:

> On Mon, Jan 6, 2014 at 5:51 PM, Eric S. Raymond <esr@thyrsus.com> wrote:
>
>> I am now recommending that Andreas perform steps 1-3 at his convenience, no
>> sooner than 2014-01-06T11:59:00  and no later than 2014-01-07T11:59:00 -
>> that is, sometime tomorrow.
>
> Why? Where's the hurry? We're in a freeze, we should be concentrating
> on fixing bugs, not learning new ways and new procedures...

Agreed.  The discussion itself has been distracting, even if useful.
Also, a more formal count would help to let those who are less vocal
express their views/vote.

PS: I appreciate Jordi's points and have been testing hg more lately,
so maybe my previous vote will be git-or-hg.  Need to test more.

-- 
 Bastien

Re: Move to git is imminent - awaiting Stefan's approval

[Jarek Czekalski]

W dniu 01/06/2014 07:42 PM, Bastien pisze:
> Agreed.  The discussion itself has been distracting, even if useful.
> Also, a more formal count would help to let those who are less vocal
> express their views/vote.

Sure, let's remember how this thread started. Karl Fogel wrote:
> I'm mainly posting
> this so there's a place for people to follow up to express their
> preference, so we can quickly get a sense of whether moving to git is
> the obvious call for the group as a whole, not just for those of us who
> have been been expressing that preference for some time.

Now Eric turns this thread into voting:
> We can ask Karl Fogel for a poll count but I
> think we all know how that would turn out; the pro-git vote has
> been overwhelming.

If the decision is to be made by a poll, I would like to know:
- who votes
- if not only admins, what are the final positions of the admins (each of
them)

Please give this info in a separate thread, that would be easy to catch.
Please add a vote "both are ok for me" to the poll

I would like to support admins with my vote, if it counts.

Thanks,
Jarek

PS. I have problems posting to devel, forgive me if this arrives twice.



--
View this message in context: http://emacs.1067599.n5.nabble.com/Move-to-git-is-imminent-awaiting-Stefan-s-approval-tp308660p308684.html
Sent from the Emacs - Dev mailing list archive at Nabble.com.

RE: Move to git is imminent - awaiting Stefan's approval

[Drew Adams]

> > I am now recommending that Andreas perform steps 1-3 at his
> > convenience, no sooner than 2014-01-06T11:59:00  and no
> > later than 2014-01-07T11:59:00 - that is, sometime tomorrow.
> 
> Why?  Where's the hurry?  We're in a freeze, we should be
> concentrating on fixing bugs, not learning new ways and
> new procedures...

+1  Juanma gets the "Emperor's New Clothes" badge for pointing
    out something important that should be obvious but is
    seemingly ignored or overlooked.

    So much energy in this thread and its offshoots...
    So many 24.3.50 bugs...

Re: Move to git is imminent - awaiting Stefan's approval

[Eric S. Raymond]

Juanma Barranquero <lekktu@gmail.com>:
> Why? Where's the hurry? We're in a freeze, we should be concentrating
> on fixing bugs, not learning new ways and new procedures...

This *is* fixing a bug.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

RE: Move to git is imminent - awaiting Stefan's approval

[Drew Adams]

> > Why? Where's the hurry? We're in a freeze, we should be
> > concentrating on fixing bugs, not learning new ways and
> > new procedures...
> 
> This *is* fixing a bug.

It may be.  For Emacs 24.4?  What's the bug number, if so?

In any case, there are thousands of other bugs as well.
Is this one such a high priority for Emacs 24.4?

Re: Move to git is imminent - awaiting Stefan's approval

[Eric S. Raymond]

Drew Adams <drew.adams@oracle.com>:
> In any case, there are thousands of other bugs as well.
> Is this one such a high priority for Emacs 24.4?

I think it is, and that's why I'm putting effort into it.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: Move to git is imminent - awaiting Stefan's approval

[Juanma Barranquero]

On Mon, Jan 6, 2014 at 8:42 PM, Eric S. Raymond <esr@thyrsus.com> wrote:

> This *is* fixing a bug.

This is an *opinion*. Opinion are not bugs. They aren't even facts.

   J

Re: Move to git is imminent - awaiting Stefan's approval

[Stephen J. Turnbull]

Eric S. Raymond writes:

 > This *is* fixing a bug.
 > -- 
 > 		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Nonsense.  It can and should wait for the release to be complete.  It
would be different if Emacs released from a branch rather than the
trunk, but it doesn't.

Re: Move to git is imminent - awaiting Stefan's approval

[joakim]

esr@thyrsus.com (Eric S. Raymond) writes:

> The thread on the git move proposal is showing signs of exhaustion and
> terminal topic drift.  We can ask Karl Fogel for a poll count but I
> think we all know how that would turn out; the pro-git vote has
> been overwhelming.
>
> I have asked Andreas Schwab and he reports the Savannah git mirror
> ready for production use. Having examined it, I agree. Andreas and I
> have discussed the switchover and believe it should consist of the
> following steps:
>
> 1. Andreas will turn off bzr commit mirroring.
>
> 2. Andreas will enter a small documentation commit recording the changeover.
>
> 3. Andreas will announce on the dev list that the git repo is live for
>    developer pushes.
>
> 4. I will do the work required to update /etc and /admin for git use
>    over the following few days.
>

What about branches? I have a long-lived branch in the bzr repo.

In my case I can just drop the branch and make a new git repo, but I
think we need a statement on branch handling nevertheless.


> I am now recommending that Andreas perform steps 1-3 at his convenience, no
> sooner than 2014-01-06T11:59:00  and no later than 2014-01-07T11:59:00 -
> that is, sometime tomorrow.
>
> Consideration of workflow changes is explicitly deferred. Developers
> should continue writing ChangeLog entries as per established custom.
>
> I want to extend a thank you to everyone who participated in the
> debate, *including* the anti-git dissenters, for generally sticking to
> real issues and maintaining civility.
>
> Stefan, please either (a) confirm that you as the official maintainer
> are authorizing this, or (b) put a hold on it and explain what 
> additional steps or conditions you deem necessary before we move.

-- 
Joakim Verona

Re: Move to git is imminent - awaiting Stefan's approval

[Andreas Schwab]

joakim@verona.se writes:

> What about branches? I have a long-lived branch in the bzr repo.

All branches are carried over.

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: Move to git is imminent - awaiting Stefan's approval

[joakim]

Andreas Schwab <schwab@suse.de> writes:

> joakim@verona.se writes:
>
>> What about branches? I have a long-lived branch in the bzr repo.
>
> All branches are carried over.

Fantastic!

> Andreas.

-- 
Joakim Verona

Re: Move to git is imminent - awaiting Stefan's approval

[Martin Geisler]

Eric S. Raymond <esr <at> thyrsus.com> writes:

> 
> The thread on the git move proposal is showing signs of exhaustion and
> terminal topic drift.  We can ask Karl Fogel for a poll count but I
> think we all know how that would turn out; the pro-git vote has
> been overwhelming.

I'm a Mercurial developer, but I wish you luck on the transition! In any
community, people should use the tool they like best and feel most
productive with.

However, let me just add that Facebook recently announced that they're
switching from Subversion to Mercurial after evaluating the performance of
Git. Their conclusion was that Git could not be improved as easily as
Mercurial so they went ahead and made Mercurial fast for their infrastructure:

  https://code.facebook.com/posts/218678814984400/scaling-mercurial-at-facebook/

Also, I would like to hear what graphical tools people use with Git that
matches the cross-platform TortoiseHg tool? It is invaluable in showing the
DAG and quickly browsing revisions:

  http://tortoisehg.bitbucket.org/screenshots.html

(The screenshots are mostly from Windows, but I use it daily on Linux.)

-- 
Martin Geisler

Re: Move to git is imminent - awaiting Stefan's approval

[Stefan Monnier]

> Also, I would like to hear what graphical tools people use with Git that

You do realize you're talking to "emacs-devel", right?


        Stefan ;-)

Re: Move to git is imminent - awaiting Stefan's approval

[Martin Geisler]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Also, I would like to hear what graphical tools people use with Git that
>
> You do realize you're talking to "emacs-devel", right?

Heh, yeah :) I use Emacs and I always use it in X11, so it's a graphical
tool for me.

When I started with Git, the first thing I wondered was where all the
nice graphical tools were and that prompted my question. I don't need a
graphical tool for making commits, but I much prefer a graphical tool
for running annotate and for browsing the history.

-- 
Martin Geisler

Re: Move to git is imminent - awaiting Stefan's approval

[Stefan Monnier]

>>> Also, I would like to hear what graphical tools people use with Git that
>> You do realize you're talking to "emacs-devel", right?
> Heh, yeah :) I use Emacs and I always use it in X11, so it's a graphical
> tool for me.

Exactly, so you have the answer: "people" use Emacs for that.


        Stefan

Re: Move to git is imminent - awaiting Stefan's approval

[Rüdiger Sonderfeld]

On Wednesday 15 January 2014 16:57:56 Martin Geisler wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
> >> Also, I would like to hear what graphical tools people use with Git that
> > 
> > You do realize you're talking to "emacs-devel", right?
> 
> Heh, yeah :) I use Emacs and I always use it in X11, so it's a graphical
> tool for me.
> 
> When I started with Git, the first thing I wondered was where all the
> nice graphical tools were and that prompted my question. I don't need a
> graphical tool for making commits, but I much prefer a graphical tool
> for running annotate and for browsing the history.

Magit is probably one of the most popular tools for using git in GNU Emacs.  
It is very nice to use and frequently seems to be the envy from users of other  
editors and users of other version control systems.  And honestly I never saw 
a better integration of a version control system into an editor myself.

(And not to forget vc.el which is shipped with GNU Emacs and provides a nice 
interface to all kinds of version control systems, including git and hg.)

Regards,
Rüdiger

Re: Move to git is imminent - awaiting Stefan's approval

[Yuri Khan]

On Thu, Jan 16, 2014 at 12:23 AM, Martin Geisler <martin@geisler.net> wrote:
>
> Also, I would like to hear what graphical tools people use with Git that
> matches the cross-platform TortoiseHg tool? It is invaluable in showing the
> DAG and quickly browsing revisions:

I use Git GUI and Gitk, which are included out-of-the-box in Windows
and are in separate packages on Debian/Ubuntu. Gitk is for browsing
the DAG (incl. filters by various conditions, e.g. only revisions
which modify a given file) and manipulating branches (branch, reset,
cherry-pick) while Git GUI is a commit tool (good for selective
staging).

Git GUI can also act as a frontend to git blame (when invoked as git
gui blame <filename>), but I usually just use vc-annotate from Emacs
for this.