* Re: Move to git is imminent - awaiting Stefan's approval
@ 2014-01-06 22:14 Angelo Graziosi
2014-01-06 22:57 ` David Kastrup
2014-01-06 23:01 ` Andreas Schwab
0 siblings, 2 replies; 62+ messages in thread
From: Angelo Graziosi @ 2014-01-06 22:14 UTC (permalink / raw
To: emacs
Eli Zaretskii wrote:
> 8. There's the emacs-bzr-version whose value gets copied into the bug
> reports. This should be replaced by the suitable git equivalent,
> or else the bug reports (of which we have quite a few each day)
> will not identify the version correctly.
Perhaps I have a similar question (or the same?)
Usually I build emacs trunk weekly. My local copy of trunk has been
created, the first time, with
$ bzr checkout --lightweight http://bzr.savannah.gnu.org/r/emacs/trunk
emacs-trunk
and then updating "daily" it with
$ cd emacs-trunk
$ bzr up
which prints each time the trunk revision number, e.g.
[...]
All changes applied successfully.
Updated to revision 115890 of branch
http://bzr.savannah.gnu.org/r/emacs/trunk
When I find a problem ("bug"), I flag it to this list saying something
like this
"In trunk rev. 115890 there is this problem... BLA BLA BLA..."
Now trying
$ git clone git://git.savannah.gnu.org/emacs.git emacs.git
I don't get any revision number.. So in my future bug report how the
trunk will be identified?
Beside this, my local copy of BZR repository has dimension
$ du -s emacs-trunk/
126M emacs-trunk/
instead the git repository has dimension
$ du -s emacs.git/
1.1G emacs.git/
it contains an emacs.git/.git directory of about 930M which probably
doesn't need for the Emacs build (bootstrap).
This is very annoying...
Ciao,
Angelo.
^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-06 22:14 Move to git is imminent - awaiting Stefan's approval Angelo Graziosi @ 2014-01-06 22:57 ` David Kastrup 2014-01-11 22:34 ` Nix 2014-01-06 23:01 ` Andreas Schwab 1 sibling, 1 reply; 62+ messages in thread From: David Kastrup @ 2014-01-06 22:57 UTC (permalink / raw To: emacs-devel Angelo Graziosi <angelo.graziosi@alice.it> writes: > Usually I build emacs trunk weekly. My local copy of trunk has been > created, the first time, with > > $ bzr checkout --lightweight http://bzr.savannah.gnu.org/r/emacs/trunk > emacs-trunk > > > and then updating "daily" it with > > $ cd emacs-trunk > $ bzr up "lightweight" > When I find a problem ("bug"), I flag it to this list saying something > like this > > "In trunk rev. 115890 there is this problem... BLA BLA BLA..." > > > Now trying > > $ git clone git://git.savannah.gnu.org/emacs.git emacs.git > > > I don't get any revision number.. So in my future bug report how the > trunk will be identified? The commit ids are unique. It's usually easiest to quote the entire SHA128 via copy&paste, but the first 16 digits or so are usually sufficient. > Beside this, my local copy of BZR repository has dimension > > $ du -s emacs-trunk/ > 126M emacs-trunk/ "lightweight" > instead the git repository has dimension > > $ du -s emacs.git/ > 1.1G emacs.git/ > > it contains an emacs.git/.git directory of about 930M which probably > doesn't need for the Emacs build (bootstrap). For the build? No. git clone has the following options: --depth <depth> Create a shallow clone with a history truncated to the specified number of revisions. A shallow repository has a number of limitations (you cannot clone or fetch from it, nor push from nor into it), but is adequate if you are only interested in the recent history of a large project with a long history, and would want to send in fixes as patches. --[no-]single-branch Clone only the history leading to the tip of a single branch, either specified by the --branch option or the primary branch remote’s HEAD points at. When creating a shallow clone with the --depth option, this is the default, unless --no-single-branch is given to fetch the histories near the tips of all branches. Further fetches into the resulting repository will only update the remote-tracking branch for the branch this option was used for the initial cloning. If the HEAD at the remote did not point at any branch when --single-branch clone was made, no remote-tracking branch is created. In addition, once you have a local repository, something like git clone existing_workdir new_workdir will do the cloning by using hard links, taking almost no time and no space. I often do that for one-off testing of stuff while my "main" workdir is busy compiling stuff. -- David Kastrup ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-06 22:57 ` David Kastrup @ 2014-01-11 22:34 ` Nix 0 siblings, 0 replies; 62+ messages in thread From: Nix @ 2014-01-11 22:34 UTC (permalink / raw To: David Kastrup; +Cc: emacs-devel On 6 Jan 2014, David Kastrup stated: > The commit ids are unique. It's usually easiest to quote the entire > SHA128 via copy&paste, but the first 16 digits or so are usually > sufficient. You can just use 'git log --abbrev-commit' to get commit IDs abbreviated (enough for nonambiguity and no further). If you want to enable this forever, git 1.7.6+ lets you say git config log.abbrevCommit true (add --global or --system to make it apply to all repositories for this user, or all repositories on the system, respectively). > In addition, once you have a local repository, something like > > git clone existing_workdir new_workdir > > will do the cloning by using hard links, taking almost no time and no > space. I often do that for one-off testing of stuff while my "main" > workdir is busy compiling stuff. In addition in addition, git-new-workdir (a script in git's contrib/ directory, likely to be installed by default eventually in some form or other) will let you check out more than one branch at once on the same repository. -- NULL && (void) ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-06 22:14 Move to git is imminent - awaiting Stefan's approval Angelo Graziosi 2014-01-06 22:57 ` David Kastrup @ 2014-01-06 23:01 ` Andreas Schwab 2014-01-07 10:24 ` Angelo Graziosi ` (2 more replies) 1 sibling, 3 replies; 62+ messages in thread From: Andreas Schwab @ 2014-01-06 23:01 UTC (permalink / raw To: Angelo Graziosi; +Cc: emacs Angelo Graziosi <angelo.graziosi@alice.it> writes: > Now trying > > $ git clone git://git.savannah.gnu.org/emacs.git emacs.git > > > I don't get any revision number.. So in my future bug report how the trunk > will be identified? A good way to identify a git branch is to use git describe. However it prefers to have an annotated (release) tag available to base on, which we currently don't have (bzr doesn't have the concept of annotated tags). > Beside this, my local copy of BZR repository has dimension > > $ du -s emacs-trunk/ > 126M emacs-trunk/ > > instead the git repository has dimension > > $ du -s emacs.git/ > 1.1G emacs.git/ By default git downloads the history of all branches. You can use --single-branch to only clone a single branch. This is the default if you create a shallow clone. Andreas. -- Andreas Schwab, schwab@linux-m68k.org GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5 "And now for something completely different." ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-06 23:01 ` Andreas Schwab @ 2014-01-07 10:24 ` Angelo Graziosi 2014-01-07 10:37 ` Git tips for Emacs development (was Re: Move to git is imminent - awaiting Stefan's approval) Nic Ferrier ` (3 more replies) 2014-01-07 13:37 ` Angelo Graziosi 2014-01-07 16:14 ` Eli Zaretskii 2 siblings, 4 replies; 62+ messages in thread From: Angelo Graziosi @ 2014-01-07 10:24 UTC (permalink / raw To: Andreas Schwab; +Cc: emacs Il 07/01/2014 0.01, Andreas Schwab ha scritto: > Angelo Graziosi <angelo.graziosi@alice.it> writes: > >> Now trying >> >> $ git clone git://git.savannah.gnu.org/emacs.git emacs.git >> >> >> I don't get any revision number.. So in my future bug report how the trunk >> will be identified? > > A good way to identify a git branch is to use git describe. However it > prefers to have an annotated (release) tag available to base on, which > we currently don't have (bzr doesn't have the concept of annotated > tags). > >> Beside this, my local copy of BZR repository has dimension >> >> $ du -s emacs-trunk/ >> 126M emacs-trunk/ >> >> instead the git repository has dimension >> >> $ du -s emacs.git/ >> 1.1G emacs.git/ > > By default git downloads the history of all branches. You can use > --single-branch to only clone a single branch. This is the default if > you create a shallow clone. David Kastrup wrote: > For the build? No. git clone has the following options: > > --depth <depth> ... > --[no-]single-branch ... OK, trying $ git clone --single-branch git://git.savannah.gnu.org/emacs.git emacs.git the local repository has dimension 1.1G, so probably I need a more explicit example... Instead, trying $ git clone --depth 1 git://git.savannah.gnu.org/emacs.git emacs.git the local repository has dimension 160M which is a little greater than my current BZR (checkout --lightweight) repository (126M), but acceptable... Suppose now I want identify which "revision" introduced a bug. I am able to do this with a binary search using the BZR command $ bzr up -r ARG How can I do this with the "minimal" clone done with GIT? I hope someone creates a Wiki page with all these doubts clarified with explicit examples... :-) Thanks, Angelo. ^ permalink raw reply [flat|nested] 62+ messages in thread
* Git tips for Emacs development (was Re: Move to git is imminent - awaiting Stefan's approval) 2014-01-07 10:24 ` Angelo Graziosi @ 2014-01-07 10:37 ` Nic Ferrier 2014-01-07 16:05 ` Eli Zaretskii 2014-01-07 12:27 ` Move to git is imminent - awaiting Stefan's approval Sven Axelsson ` (2 subsequent siblings) 3 siblings, 1 reply; 62+ messages in thread From: Nic Ferrier @ 2014-01-07 10:37 UTC (permalink / raw To: Angelo Graziosi; +Cc: Andreas Schwab, emacs Angelo Graziosi <angelo.graziosi@alice.it> writes: > Instead, trying > > $ git clone --depth 1 git://git.savannah.gnu.org/emacs.git emacs.git > > the local repository has dimension 160M which is a little greater than > my current BZR (checkout --lightweight) repository (126M), but acceptable... > > Suppose now I want identify which "revision" introduced a bug. I am able > to do this with a binary search using the BZR command > > $ bzr up -r ARG What does ARG do in this context? > How can I do this with the "minimal" clone done with GIT? > > > I hope someone creates a Wiki page with all these doubts clarified with > explicit examples... :-) http://www.emacswiki.org/emacs/GitForEmacsDevs is the start of one. Do add questions. Nic ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Git tips for Emacs development (was Re: Move to git is imminent - awaiting Stefan's approval) 2014-01-07 10:37 ` Git tips for Emacs development (was Re: Move to git is imminent - awaiting Stefan's approval) Nic Ferrier @ 2014-01-07 16:05 ` Eli Zaretskii 0 siblings, 0 replies; 62+ messages in thread From: Eli Zaretskii @ 2014-01-07 16:05 UTC (permalink / raw To: Nic Ferrier; +Cc: emacs-devel, schwab, angelo.graziosi > From: Nic Ferrier <nferrier@ferrier.me.uk> > Date: Tue, 07 Jan 2014 10:37:44 +0000 > Cc: Andreas Schwab <schwab@linux-m68k.org>, emacs <emacs-devel@gnu.org> > > > $ bzr up -r ARG > > What does ARG do in this context? It's the revision to which the tree will be updated. ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-07 10:24 ` Angelo Graziosi 2014-01-07 10:37 ` Git tips for Emacs development (was Re: Move to git is imminent - awaiting Stefan's approval) Nic Ferrier @ 2014-01-07 12:27 ` Sven Axelsson 2014-01-07 14:41 ` Stephen Berman ` (2 more replies) 2014-01-07 12:29 ` David Kastrup 2014-01-07 16:11 ` Eli Zaretskii 3 siblings, 3 replies; 62+ messages in thread From: Sven Axelsson @ 2014-01-07 12:27 UTC (permalink / raw To: Angelo Graziosi; +Cc: Andreas Schwab, emacs On 7 January 2014 11:24, Angelo Graziosi <angelo.graziosi@alice.it> wrote: > > > OK, trying > > $ git clone --single-branch git://git.savannah.gnu.org/emacs.git emacs.git > > the local repository has dimension 1.1G, so probably I need a more explicit example... This clones only the main branch (master). Since this is where almost all Emacs history is located it won't make much difference size wise. > Instead, trying > > $ git clone --depth 1 git://git.savannah.gnu.org/emacs.git emacs.git > > the local repository has dimension 160M which is a little greater than my current BZR (checkout --lightweight) repository (126M), but acceptable... > > Suppose now I want identify which "revision" introduced a bug. I am able to do this with a binary search using the BZR command > > $ bzr up -r ARG > > How can I do this with the "minimal" clone done with GIT? Git does not have any concept like Bazaar's lightweight checkouts - nor does any other dvcd afaik. Depth 1 truncates all history except for the most recent commit. From `man git clone`: A shallow repository has a number of limitations (you cannot clone or fetch from it, nor push from nor into it), but is adequate if you are only interested in the recent history of a large project with a long history, and would want to send in fixes as patches. -- Sven Axelsson ++++++++++[>++++++++++>+++++++++++>++++++++++>++++++ >++++<<<<<-]>++++.+.++++.>+++++.>+.<<-.>>+.>++++.<<. +++.>-.<<++.>>----.<++.>>>++++++.<<<<.>>++++.<----. ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-07 12:27 ` Move to git is imminent - awaiting Stefan's approval Sven Axelsson @ 2014-01-07 14:41 ` Stephen Berman 2014-01-07 14:57 ` Andreas Schwab 2014-01-07 15:00 ` Angelo Graziosi 2014-01-07 16:17 ` Eli Zaretskii 2014-01-10 6:40 ` Antonio Nikishaev 2 siblings, 2 replies; 62+ messages in thread From: Stephen Berman @ 2014-01-07 14:41 UTC (permalink / raw To: Sven Axelsson; +Cc: emacs, Andreas Schwab, Angelo Graziosi On Tue, 7 Jan 2014 13:27:22 +0100 Sven Axelsson <sven.axelsson@gmail.com> wrote: > On 7 January 2014 11:24, Angelo Graziosi <angelo.graziosi@alice.it> wrote: >> >> >> OK, trying >> >> $ git clone --single-branch git://git.savannah.gnu.org/emacs.git emacs.git >> >> the local repository has dimension 1.1G, so probably I need a more explicit >> example... > > This clones only the main branch (master). Since this is where almost > all Emacs history is located it won't make much difference size wise. Is it correct that main or master is the same thing as the Emacs trunk branch? If so, I wonder why the git repository containing just this is so large: my bzr Emacs repository only contains the trunk (as a bound branch), several local branches of it, and the emacs-24 branch; the .bzr/ directory of the repository, containing all the history, is only 428 MiB, not 1.1 GiB (the trunk directory itself contains an additional 129.3 MiB, but the total is still half the size of the git repo). Why is the git repository so much larger than the bzr repository? Steve Berman ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-07 14:41 ` Stephen Berman @ 2014-01-07 14:57 ` Andreas Schwab 2014-01-07 15:05 ` Stephen Berman 2014-01-07 16:25 ` Eli Zaretskii 2014-01-07 15:00 ` Angelo Graziosi 1 sibling, 2 replies; 62+ messages in thread From: Andreas Schwab @ 2014-01-07 14:57 UTC (permalink / raw To: Stephen Berman; +Cc: Angelo Graziosi, Sven Axelsson, emacs Stephen Berman <stephen.berman@gmx.net> writes: > Why is the git repository so much larger than the bzr repository? The git repositories on savannah appear to be packed rather badly. A reasonably well packed emacs repository should have around 350MB. Andreas. -- Andreas Schwab, SUSE Labs, schwab@suse.de GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-07 14:57 ` Andreas Schwab @ 2014-01-07 15:05 ` Stephen Berman 2014-01-07 16:25 ` Eli Zaretskii 1 sibling, 0 replies; 62+ messages in thread From: Stephen Berman @ 2014-01-07 15:05 UTC (permalink / raw To: Andreas Schwab; +Cc: Angelo Graziosi, Sven Axelsson, emacs On Tue, 07 Jan 2014 15:57:05 +0100 Andreas Schwab <schwab@suse.de> wrote: > Stephen Berman <stephen.berman@gmx.net> writes: > >> Why is the git repository so much larger than the bzr repository? > > The git repositories on savannah appear to be packed rather badly. A > reasonably well packed emacs repository should have around 350MB. Ok. If the packing can be improved, I think it would be good for that to happen before finalizing the move to git. Steve Berman ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-07 14:57 ` Andreas Schwab 2014-01-07 15:05 ` Stephen Berman @ 2014-01-07 16:25 ` Eli Zaretskii 2014-01-07 16:35 ` Andreas Schwab 1 sibling, 1 reply; 62+ messages in thread From: Eli Zaretskii @ 2014-01-07 16:25 UTC (permalink / raw To: Andreas Schwab Cc: stephen.berman, emacs-devel, sven.axelsson, angelo.graziosi > From: Andreas Schwab <schwab@suse.de> > Date: Tue, 07 Jan 2014 15:57:05 +0100 > Cc: Angelo Graziosi <angelo.graziosi@alice.it>, > Sven Axelsson <sven.axelsson@gmail.com>, emacs <emacs-devel@gnu.org> > > The git repositories on savannah appear to be packed rather badly. A > reasonably well packed emacs repository should have around 350MB. Can repacking be done locally, or does it have to be on savannah? If the former, how to do it? ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-07 16:25 ` Eli Zaretskii @ 2014-01-07 16:35 ` Andreas Schwab 2014-01-07 16:37 ` Angelo Graziosi 0 siblings, 1 reply; 62+ messages in thread From: Andreas Schwab @ 2014-01-07 16:35 UTC (permalink / raw To: Eli Zaretskii; +Cc: stephen.berman, emacs-devel, sven.axelsson, angelo.graziosi Eli Zaretskii <eliz@gnu.org> writes: > Can repacking be done locally, or does it have to be on savannah? You can always repack locally. It's just that during clone, you inherit the packing of the remote, since existing deltas are not recomputed. Andreas. -- Andreas Schwab, SUSE Labs, schwab@suse.de GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-07 16:35 ` Andreas Schwab @ 2014-01-07 16:37 ` Angelo Graziosi 2014-01-07 16:41 ` Andreas Schwab 0 siblings, 1 reply; 62+ messages in thread From: Angelo Graziosi @ 2014-01-07 16:37 UTC (permalink / raw To: Andreas Schwab, Eli Zaretskii; +Cc: stephen.berman, sven.axelsson, emacs-devel Il 07/01/2014 17.35, Andreas Schwab ha scritto: > Eli Zaretskii <eliz@gnu.org> writes: > >> Can repacking be done locally, or does it have to be on savannah? > > You can always repack locally. It's just that during clone, you inherit > the packing of the remote, since existing deltas are not recomputed. OK, but Eli asked how do it... So how? Thanks, Angelo. ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-07 16:37 ` Angelo Graziosi @ 2014-01-07 16:41 ` Andreas Schwab 2014-01-07 17:08 ` Eli Zaretskii 0 siblings, 1 reply; 62+ messages in thread From: Andreas Schwab @ 2014-01-07 16:41 UTC (permalink / raw To: Angelo Graziosi; +Cc: Eli Zaretskii, stephen.berman, sven.axelsson, emacs-devel Angelo Graziosi <angelo.graziosi@alice.it> writes: > Il 07/01/2014 17.35, Andreas Schwab ha scritto: >> Eli Zaretskii <eliz@gnu.org> writes: >> >>> Can repacking be done locally, or does it have to be on savannah? >> >> You can always repack locally. It's just that during clone, you inherit >> the packing of the remote, since existing deltas are not recomputed. > > OK, but Eli asked how do it... So how? See git-gc(1) Andreas. -- Andreas Schwab, SUSE Labs, schwab@suse.de GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-07 16:41 ` Andreas Schwab @ 2014-01-07 17:08 ` Eli Zaretskii 2014-01-07 17:23 ` Óscar Fuentes ` (3 more replies) 0 siblings, 4 replies; 62+ messages in thread From: Eli Zaretskii @ 2014-01-07 17:08 UTC (permalink / raw To: Andreas Schwab Cc: stephen.berman, emacs-devel, sven.axelsson, angelo.graziosi > From: Andreas Schwab <schwab@suse.de> > Cc: Eli Zaretskii <eliz@gnu.org>, stephen.berman@gmx.net, sven.axelsson@gmail.com, emacs-devel@gnu.org > Date: Tue, 07 Jan 2014 17:41:43 +0100 > > Angelo Graziosi <angelo.graziosi@alice.it> writes: > > > Il 07/01/2014 17.35, Andreas Schwab ha scritto: > >> Eli Zaretskii <eliz@gnu.org> writes: > >> > >>> Can repacking be done locally, or does it have to be on savannah? > >> > >> You can always repack locally. It's just that during clone, you inherit > >> the packing of the remote, since existing deltas are not recomputed. > > > > OK, but Eli asked how do it... So how? > > See git-gc(1) Thanks. "git gc" gets me from 1.18GB to 1.1GB. I tried --aggressive, but that ran out of memory (trying to allocate 1GB) after some time and errored out. ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-07 17:08 ` Eli Zaretskii @ 2014-01-07 17:23 ` Óscar Fuentes 2014-01-07 17:56 ` Eli Zaretskii 2014-01-07 17:24 ` Stephen J. Turnbull ` (2 subsequent siblings) 3 siblings, 1 reply; 62+ messages in thread From: Óscar Fuentes @ 2014-01-07 17:23 UTC (permalink / raw To: emacs-devel Eli Zaretskii <eliz@gnu.org> writes: > Thanks. "git gc" gets me from 1.18GB to 1.1GB. I tried --aggressive, > but that ran out of memory (trying to allocate 1GB) after some time > and errored out. That's strange: oscar@qcore:~/dev/emacs/emacs$ time git gc Counting objects: 734772, done. Delta compression using up to 4 threads. Compressing objects: 100% (145032/145032), done. Writing objects: 100% (734772/734772), done. Total 734772 (delta 589118), reused 734312 (delta 588658) Checking connectivity: 734772, done. oscar@qcore:~/dev/emacs/emacs$ du -sh .git 289M .git oscar@qcore:~/dev/emacs/emacs$ git branch -a | wc 127 130 4667 Maybe you instructed git to download extra info (such as the meta-data the fast-import/export process uses and is useful for relating git commits to their corresponding bzr commits.) ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-07 17:23 ` Óscar Fuentes @ 2014-01-07 17:56 ` Eli Zaretskii 0 siblings, 0 replies; 62+ messages in thread From: Eli Zaretskii @ 2014-01-07 17:56 UTC (permalink / raw To: Óscar Fuentes; +Cc: emacs-devel > From: Óscar Fuentes <ofv@wanadoo.es> > Date: Tue, 07 Jan 2014 18:23:48 +0100 > > Eli Zaretskii <eliz@gnu.org> writes: > > > Thanks. "git gc" gets me from 1.18GB to 1.1GB. I tried --aggressive, > > but that ran out of memory (trying to allocate 1GB) after some time > > and errored out. > > That's strange: > > oscar@qcore:~/dev/emacs/emacs$ time git gc > Counting objects: 734772, done. > Delta compression using up to 4 threads. > Compressing objects: 100% (145032/145032), done. > Writing objects: 100% (734772/734772), done. > Total 734772 (delta 589118), reused 734312 (delta 588658) > Checking connectivity: 734772, done. > > oscar@qcore:~/dev/emacs/emacs$ du -sh .git > 289M .git > oscar@qcore:~/dev/emacs/emacs$ git branch -a | wc > 127 130 4667 > > > Maybe you instructed git to download extra info (such as the meta-data > the fast-import/export process uses and is useful for relating git > commits to their corresponding bzr commits.) Nope, just "git clone" followed by pull's. I now tried on fencepost.gnu.org, a 64-bit GNU/Linux system, with similar results: gc goes from 1.78GB down to 1.6GB. --aggressive also fails there, funnily enough. ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-07 17:08 ` Eli Zaretskii 2014-01-07 17:23 ` Óscar Fuentes @ 2014-01-07 17:24 ` Stephen J. Turnbull 2014-01-07 18:05 ` Eli Zaretskii 2014-01-07 18:02 ` Thien-Thi Nguyen 2014-01-07 20:08 ` Angelo Graziosi 3 siblings, 1 reply; 62+ messages in thread From: Stephen J. Turnbull @ 2014-01-07 17:24 UTC (permalink / raw To: Eli Zaretskii Cc: Andreas Schwab, stephen.berman, sven.axelsson, angelo.graziosi, emacs-devel Eli Zaretskii writes: > > See git-gc(1) git-gc does not necessarily repack existing packs according to the doc. Try git-repack. Or wait for me to try it ... it's past my bedtime, will report tomorrow if somebody doesn't beat me to it. ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-07 17:24 ` Stephen J. Turnbull @ 2014-01-07 18:05 ` Eli Zaretskii 2014-01-07 18:17 ` Sven Axelsson 0 siblings, 1 reply; 62+ messages in thread From: Eli Zaretskii @ 2014-01-07 18:05 UTC (permalink / raw To: Stephen J. Turnbull Cc: schwab, stephen.berman, sven.axelsson, angelo.graziosi, emacs-devel > From: "Stephen J. Turnbull" <stephen@xemacs.org> > Cc: Andreas Schwab <schwab@suse.de>, > stephen.berman@gmx.net, > emacs-devel@gnu.org, > sven.axelsson@gmail.com, > angelo.graziosi@alice.it > Date: Wed, 08 Jan 2014 02:24:02 +0900 > > Eli Zaretskii writes: > > > > See git-gc(1) > > git-gc does not necessarily repack existing packs according to the > doc. > > Try git-repack. I thought "git gc" runs git-repack, no? ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-07 18:05 ` Eli Zaretskii @ 2014-01-07 18:17 ` Sven Axelsson 2014-01-07 18:49 ` Stephen J. Turnbull 0 siblings, 1 reply; 62+ messages in thread From: Sven Axelsson @ 2014-01-07 18:17 UTC (permalink / raw To: Eli Zaretskii Cc: schwab, Stephen J. Turnbull, stephen.berman, angelo.graziosi, emacs On 7 January 2014 19:05, Eli Zaretskii <eliz@gnu.org> wrote: >> From: "Stephen J. Turnbull" <stephen@xemacs.org> >> Cc: Andreas Schwab <schwab@suse.de>, >> stephen.berman@gmx.net, >> emacs-devel@gnu.org, >> sven.axelsson@gmail.com, >> angelo.graziosi@alice.it >> Date: Wed, 08 Jan 2014 02:24:02 +0900 >> >> Eli Zaretskii writes: >> >> > > See git-gc(1) >> >> git-gc does not necessarily repack existing packs according to the >> doc. >> >> Try git-repack. > > I thought "git gc" runs git-repack, no? No. git-repack is often a very expensive operation, so it is never run automatically, which git-gc can be. -- Sven Axelsson ++++++++++[>++++++++++>+++++++++++>++++++++++>++++++ >++++<<<<<-]>++++.+.++++.>+++++.>+.<<-.>>+.>++++.<<. +++.>-.<<++.>>----.<++.>>>++++++.<<<<.>>++++.<----. ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-07 18:17 ` Sven Axelsson @ 2014-01-07 18:49 ` Stephen J. Turnbull 2014-01-07 19:06 ` Eli Zaretskii 0 siblings, 1 reply; 62+ messages in thread From: Stephen J. Turnbull @ 2014-01-07 18:49 UTC (permalink / raw To: Sven Axelsson Cc: schwab, Eli Zaretskii, stephen.berman, emacs, angelo.graziosi Sven Axelsson writes: > On 7 January 2014 19:05, Eli Zaretskii <eliz@gnu.org> wrote: > >> From: "Stephen J. Turnbull" <stephen@xemacs.org> > >> git-gc does not necessarily repack existing packs according to the > >> doc. > >> > >> Try git-repack. > > > > I thought "git gc" runs git-repack, no? It kinda pisses me off that you complain about the excessive detail of git documentation, and then ask others to tell you about the details you could read in the documentation. > No. git-repack is often a very expensive operation, so it is never > run automatically, which git-gc can be. Specifically, according to the help, git-gc removes garbage (in the usual sense of a tracing collector) and "compresses" (ie, packs) loose objects using delta compression (and then the result is deflated as with loose objects, I believe). It will repack the packs into *one* pack (presumably with better delta compression) if the number of packs is greater than the value of gc.autopacklimit (in .git/config; default: 50). ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-07 18:49 ` Stephen J. Turnbull @ 2014-01-07 19:06 ` Eli Zaretskii 2014-01-08 3:47 ` Stephen J. Turnbull 0 siblings, 1 reply; 62+ messages in thread From: Eli Zaretskii @ 2014-01-07 19:06 UTC (permalink / raw To: Stephen J. Turnbull Cc: schwab, stephen.berman, emacs-devel, sven.axelsson, angelo.graziosi > From: "Stephen J. Turnbull" <stephen@xemacs.org> > Cc: Eli Zaretskii <eliz@gnu.org>, > schwab@suse.de, > stephen.berman@gmx.net, > angelo.graziosi@alice.it, > emacs <emacs-devel@gnu.org> > Date: Wed, 08 Jan 2014 03:49:24 +0900 > > Sven Axelsson writes: > > On 7 January 2014 19:05, Eli Zaretskii <eliz@gnu.org> wrote: > > >> From: "Stephen J. Turnbull" <stephen@xemacs.org> > > > >> git-gc does not necessarily repack existing packs according to the > > >> doc. > > >> > > >> Try git-repack. > > > > > > I thought "git gc" runs git-repack, no? > > It kinda pisses me off that you complain about the excessive detail of > git documentation, and then ask others to tell you about the details > you could read in the documentation. I did read the documentation. The docs of gc mentions git-repack, so I thought the former runs the latter. What part did I miss? > Specifically, according to the help, git-gc removes garbage (in the > usual sense of a tracing collector) and "compresses" (ie, packs) loose > objects using delta compression (and then the result is deflated as > with loose objects, I believe). It will repack the packs into *one* > pack (presumably with better delta compression) if the number of packs > is greater than the value of gc.autopacklimit (in .git/config; > default: 50). And how was I supposed to know from this whether git-repack will or won't be run, what with the word "repack" explicitly in this text (and in other parts of that man page)? Don't bother answering if you are still pissed off. ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-07 19:06 ` Eli Zaretskii @ 2014-01-08 3:47 ` Stephen J. Turnbull 2014-01-08 15:23 ` Eli Zaretskii 0 siblings, 1 reply; 62+ messages in thread From: Stephen J. Turnbull @ 2014-01-08 3:47 UTC (permalink / raw To: Eli Zaretskii Cc: schwab, stephen.berman, sven.axelsson, angelo.graziosi, emacs-devel Eli Zaretskii writes: > I did read the documentation. The docs of gc mentions git-repack, so > I thought the former runs the latter. What part did I miss? The fact that the documentation clearly puts conditions on when git-gc runs git-repack. > And how was I supposed to know from this whether git-repack will or > won't be run, what with the word "repack" explicitly in this text (and > in other parts of that man page)? Huh? If there's a condition on whether it runs, then presumably sometimes it doesn't run. That's all that I deduced from it, and advised you run git-repack manually. ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-08 3:47 ` Stephen J. Turnbull @ 2014-01-08 15:23 ` Eli Zaretskii 0 siblings, 0 replies; 62+ messages in thread From: Eli Zaretskii @ 2014-01-08 15:23 UTC (permalink / raw To: Stephen J. Turnbull Cc: schwab, stephen.berman, sven.axelsson, angelo.graziosi, emacs-devel > From: "Stephen J. Turnbull" <stephen@xemacs.org> > Cc: schwab@suse.de, > stephen.berman@gmx.net, > emacs-devel@gnu.org, > sven.axelsson@gmail.com, > angelo.graziosi@alice.it > Date: Wed, 08 Jan 2014 12:47:29 +0900 > > Eli Zaretskii writes: > > I did read the documentation. The docs of gc mentions git-repack, so > > I thought the former runs the latter. What part did I miss? > > The fact that the documentation clearly puts conditions on when git-gc > runs git-repack. I wouldn't qualify the gc manpage as "clear" on this matter, see below. But given what it does say, I _guessed_ that "git gc" _might_ run git-repack, and asked whether that wasn't so. Why that pissed you off I don't know. You seem to have deduced a different conclusion, but IMO that's just one more evidence that the man page is not really clear on this matter. > > And how was I supposed to know from this whether git-repack will or > > won't be run, what with the word "repack" explicitly in this text (and > > in other parts of that man page)? > > Huh? If there's a condition on whether it runs, then presumably > sometimes it doesn't run. AFAIU, you refer to this part of your previous message: > Specifically, according to the help, git-gc removes garbage (in the > usual sense of a tracing collector) and "compresses" (ie, packs) loose > objects using delta compression (and then the result is deflated as > with loose objects, I believe). It will repack the packs into *one* > pack (presumably with better delta compression) if the number of packs > is greater than the value of gc.autopacklimit (in .git/config; > default: 50). However, this comes from the part in the man page that describes the "--auto" option to gc. I didn't use and didn't mean to use that option, so the above is not necessarily relevant to "git gc". But even if this description is relevant when --auto is not used, the above says that repacking will sometimes happen. As to what happens regardless of any options, the man page says just this: DESCRIPTION Runs a number of housekeeping tasks within the current repository, such as compressing file revisions (to reduce disk space and increase performance) and removing unreachable objects which may have been created from prior invocations of git add. This says "such as", which doesn't imply an exhaustive list of tasks. Moreover, it never says that "compressing file revisions" and "removing unreachable objects" is not done by git-repack. So the above description does not contradict my conclusion, and definitely doesn't present exact description of the conditions under which gc will run repack. Perhaps if I knew exactly what repack does, I could have filled the voids, but I don't. Finally, the "Configuration" section has this, among the rest: The optional configuration variable gc.packrefs determines if git gc runs git pack-refs. This can be set to "notbare" to enable it within all non-bare repos or it can be set to a boolean value. This defaults to true. The optional configuration variable gc.aggressiveWindow controls how much time is spent optimizing the delta compression of the objects in the repository when the --aggressive option is specified. The larger the value, the more time is spent optimizing the delta compression. See the documentation for the --window' option in git-repack(1) for more details. This defaults to 250. Both of these configuration settings explicitly mention packing and/or repacking. Again, without knowing exactly what repack does, there's no clarity here whether pack-refs is related to repack in any way, shape or form (looking in the sources resolves this, see below). Given these hints, IMO it was a reasonable guess that gc does run repack at least under some conditions, when --auto is not used. Since these conditions are not defined completely by the man page, it was my guess that gc, with or without the --aggressive option, might run repack. As it turns out, the second part is surely true. > You read the source? I find that hard to believe. I didn't read the source in this case, only the documentation. I don't understand the "hard to believe" part, though: I do have the git's git repo on my disk, and you could easily imagine that I'm not new to reading sources. Having now looked in the sources, this is what I see: if (pack_refs && run_command_v_opt(pack_refs_cmd.argv, RUN_GIT_CMD)) return error(FAILED_RUN, pack_refs_cmd.argv[0]); if (run_command_v_opt(reflog.argv, RUN_GIT_CMD)) return error(FAILED_RUN, reflog.argv[0]); if (run_command_v_opt(repack.argv, RUN_GIT_CMD)) return error(FAILED_RUN, repack.argv[0]); if (prune_expire) { argv_array_push(&prune, prune_expire); if (quiet) argv_array_push(&prune, "--no-progress"); if (run_command_v_opt(prune.argv, RUN_GIT_CMD)) return error(FAILED_RUN, prune.argv[0]); } if (run_command_v_opt(rerere.argv, RUN_GIT_CMD)) return error(FAILED_RUN, rerere.argv[0]); My reading of this is that "git gc" runs, in sequence, pack-refs (conditioned on gc.packrefs), reflog, repack, prune (conditioned on the --prune option), and finally rerere. IOW, "git gc" does run repack. The only difference between --aggressive and non-aggressive modes is here: if (aggressive) { argv_array_push(&repack, "-f"); argv_array_push(&repack, "--depth=250"); if (aggressive_window > 0) argv_array_pushf(&repack, "--window=%d", aggressive_window); } That is, --aggressive just passes the -f switch to git-repack, which tells it not to reuse deltas (i.e. recompute them anew, AFAIU). > (The doc for --aggressive says nothing about repack, only that it > optimizes harder.) See above: the description of aggressiveWindow setting quite clearly says that --aggressive does run repack. And in fact it does, indeed. > > Stephen can now be pissed off by himself. > > Yup. And I won't waste time on suppressing my irritation while > helping with GitForEmacsDevs Too bad. ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-07 17:08 ` Eli Zaretskii 2014-01-07 17:23 ` Óscar Fuentes 2014-01-07 17:24 ` Stephen J. Turnbull @ 2014-01-07 18:02 ` Thien-Thi Nguyen 2014-01-07 18:06 ` Sven Axelsson 2014-01-07 20:08 ` Angelo Graziosi 3 siblings, 1 reply; 62+ messages in thread From: Thien-Thi Nguyen @ 2014-01-07 18:02 UTC (permalink / raw To: emacs-devel [-- Attachment #1: Type: text/plain, Size: 684 bytes --] () Eli Zaretskii <eliz@gnu.org> () Tue, 07 Jan 2014 19:08:04 +0200 Thanks. "git gc" gets me from 1.18GB to 1.1GB. I tried --aggressive, but that ran out of memory (trying to allocate 1GB) after some time and errored out. I had the same problem in a similar circumstance (git-bzr). See: http://lists.gnu.org/archive/html/emacs-devel/2014-01/msg00329.html for the resolution (briefly: use "git repack" w/ ‘--window-memory’). -- Thien-Thi Nguyen GPG key: 4C807502 (if you're human and you know it) read my lisp: (responsep (questions 'technical) (not (via 'mailing-list))) => nil [-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --] ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-07 18:02 ` Thien-Thi Nguyen @ 2014-01-07 18:06 ` Sven Axelsson 2014-01-07 18:47 ` Andreas Schwab 0 siblings, 1 reply; 62+ messages in thread From: Sven Axelsson @ 2014-01-07 18:06 UTC (permalink / raw To: emacs On 7 January 2014 19:02, Thien-Thi Nguyen <ttn@gnu.org> wrote: > () Eli Zaretskii <eliz@gnu.org> > () Tue, 07 Jan 2014 19:08:04 +0200 > > Thanks. "git gc" gets me from 1.18GB to 1.1GB. I tried > --aggressive, but that ran out of memory (trying to allocate 1GB) > after some time and errored out. > > I had the same problem in a similar circumstance (git-bzr). See: > > http://lists.gnu.org/archive/html/emacs-devel/2014-01/msg00329.html > > for the resolution (briefly: use "git repack" w/ ‘--window-memory’). > Using `git repack -a -d -f --window=250 --depth=250` as mentioned earlier in the conversation brings the .git folder down to 211 Mb for me. -- Sven Axelsson ++++++++++[>++++++++++>+++++++++++>++++++++++>++++++ >++++<<<<<-]>++++.+.++++.>+++++.>+.<<-.>>+.>++++.<<. +++.>-.<<++.>>----.<++.>>>++++++.<<<<.>>++++.<----. ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-07 18:06 ` Sven Axelsson @ 2014-01-07 18:47 ` Andreas Schwab 2014-01-07 19:07 ` Sven Axelsson 0 siblings, 1 reply; 62+ messages in thread From: Andreas Schwab @ 2014-01-07 18:47 UTC (permalink / raw To: Sven Axelsson; +Cc: emacs Sven Axelsson <sven.axelsson@gmail.com> writes: > Using `git repack -a -d -f --window=250 --depth=250` as mentioned > earlier in the conversation brings the .git folder down to 211 Mb for > me. That's basically what git gc --aggressive does. Andreas. -- Andreas Schwab, schwab@linux-m68k.org GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5 "And now for something completely different." ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-07 18:47 ` Andreas Schwab @ 2014-01-07 19:07 ` Sven Axelsson 2014-01-07 19:16 ` Eli Zaretskii 0 siblings, 1 reply; 62+ messages in thread From: Sven Axelsson @ 2014-01-07 19:07 UTC (permalink / raw To: Andreas Schwab; +Cc: emacs On 7 January 2014 19:47, Andreas Schwab <schwab@linux-m68k.org> wrote: > Sven Axelsson <sven.axelsson@gmail.com> writes: > >> Using `git repack -a -d -f --window=250 --depth=250` as mentioned >> earlier in the conversation brings the .git folder down to 211 Mb for >> me. > > That's basically what git gc --aggressive does. Looking at the source for git-gc I see that you are indeed correct. Specifically, `git gc --aggressive` will run `git repack -d -l -f --depth=250 --window=250`, among other things. Maybe this setting is more appropriate: using the -a flag can give a better pack, but may cause extra network traffic when fetching the repo over a dumb protocol. All this can be found in tha man pages. -- Sven Axelsson ++++++++++[>++++++++++>+++++++++++>++++++++++>++++++ >++++<<<<<-]>++++.+.++++.>+++++.>+.<<-.>>+.>++++.<<. +++.>-.<<++.>>----.<++.>>>++++++.<<<<.>>++++.<----. ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-07 19:07 ` Sven Axelsson @ 2014-01-07 19:16 ` Eli Zaretskii 2014-01-08 3:52 ` Stephen J. Turnbull 0 siblings, 1 reply; 62+ messages in thread From: Eli Zaretskii @ 2014-01-07 19:16 UTC (permalink / raw To: Sven Axelsson; +Cc: schwab, emacs-devel > Date: Tue, 7 Jan 2014 20:07:06 +0100 > From: Sven Axelsson <sven.axelsson@gmail.com> > Cc: emacs <emacs-devel@gnu.org> > > >> Using `git repack -a -d -f --window=250 --depth=250` as mentioned > >> earlier in the conversation brings the .git folder down to 211 Mb for > >> me. > > > > That's basically what git gc --aggressive does. > > Looking at the source for git-gc I see that you are indeed correct. > Specifically, `git gc --aggressive` will run `git repack -d -l -f > --depth=250 --window=250`, among other things. Maybe this setting is > more appropriate: using the -a flag can give a better pack, but may > cause extra network traffic when fetching the repo over a dumb > protocol. All this can be found in tha man pages. Which was why I interpreted them to mean that gc runs repack. Stephen can now be pissed off by himself. ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-07 19:16 ` Eli Zaretskii @ 2014-01-08 3:52 ` Stephen J. Turnbull 0 siblings, 0 replies; 62+ messages in thread From: Stephen J. Turnbull @ 2014-01-08 3:52 UTC (permalink / raw To: Eli Zaretskii; +Cc: schwab, Sven Axelsson, emacs-devel Eli Zaretskii writes: > Which was why I interpreted them to mean that gc runs repack. You read the source? I find that hard to believe. (The doc for --aggressive says nothing about repack, only that it optimizes harder.) > Stephen can now be pissed off by himself. Yup. And I won't waste time on suppressing my irritation while helping with GitForEmacsDevs, since you're the main person I would (normally) care to make such an effort for. ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-07 17:08 ` Eli Zaretskii ` (2 preceding siblings ...) 2014-01-07 18:02 ` Thien-Thi Nguyen @ 2014-01-07 20:08 ` Angelo Graziosi 2014-01-08 17:32 ` Eli Zaretskii 3 siblings, 1 reply; 62+ messages in thread From: Angelo Graziosi @ 2014-01-07 20:08 UTC (permalink / raw To: Eli Zaretskii, Andreas Schwab; +Cc: stephen.berman, sven.axelsson, emacs-devel Il 07/01/2014 18.08, Eli Zaretskii ha scritto: > Thanks. "git gc" gets me from 1.18GB to 1.1GB. I tried --aggressive, > but that ran out of memory (trying to allocate 1GB) after some time > and errored out. > Here (on Cygwin) it worked... Firstly, I did $ git clone git://git.savannah.gnu.org/emacs.git emacs.git and its size was $ du -s emacs.git/ 1,1G emacs.git/ then I did $ cd emacs.git/ $ git gc --aggressive Counting objects: 735546, done. Delta compression using up to 2 threads. Compressing objects: 100% (734446/734446), done. Writing objects: 100% (735546/735546), done. Total 735546 (delta 589814), reused 143177 (delta 0) Checking connectivity: 735546, done. which took almost 3 hours to be completed. And the size is: $ cd .. $ du -s emacs.git/ 327M emacs.git/ which seems very good! Ciao, Angelo. ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-07 20:08 ` Angelo Graziosi @ 2014-01-08 17:32 ` Eli Zaretskii 2014-01-08 18:34 ` Andreas Schwab 2014-01-08 21:55 ` Angelo Graziosi 0 siblings, 2 replies; 62+ messages in thread From: Eli Zaretskii @ 2014-01-08 17:32 UTC (permalink / raw To: Angelo Graziosi; +Cc: schwab, stephen.berman, sven.axelsson, emacs-devel > Date: Tue, 07 Jan 2014 21:08:51 +0100 > From: Angelo Graziosi <angelo.graziosi@alice.it> > CC: stephen.berman@gmx.net, sven.axelsson@gmail.com, emacs-devel@gnu.org > > $ cd emacs.git/ > $ git gc --aggressive > Counting objects: 735546, done. > Delta compression using up to 2 threads. > Compressing objects: 100% (734446/734446), done. > Writing objects: 100% (735546/735546), done. > Total 735546 (delta 589814), reused 143177 (delta 0) > Checking connectivity: 735546, done. > > which took almost 3 hours to be completed. > > And the size is: > > $ cd .. > > $ du -s emacs.git/ > 327M emacs.git/ > > which seems very good! I guess this was on Windows 7, right? I succeeded to do this on a 64-bit Windows 7 as well, and it took about 2 hours. And now I understand why git ran out of memory on Windows XP: the memory footprint of git-pack-objects goes up to 3GB, whereas Windows XP has only 3GB address space for all the applications. So I guess we will need to repack the repository on savannah as a prerequisite for switching, and then make sure it is repacked regularly afterwards. Otherwise, people with low memory and/or slow CPUs will need many hours to do this on their local machines. ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-08 17:32 ` Eli Zaretskii @ 2014-01-08 18:34 ` Andreas Schwab 2014-01-08 18:42 ` Eli Zaretskii 2014-01-12 6:27 ` Bob Proulx 2014-01-08 21:55 ` Angelo Graziosi 1 sibling, 2 replies; 62+ messages in thread From: Andreas Schwab @ 2014-01-08 18:34 UTC (permalink / raw To: Eli Zaretskii; +Cc: stephen.berman, emacs-devel, sven.axelsson, Angelo Graziosi Eli Zaretskii <eliz@gnu.org> writes: > So I guess we will need to repack the repository on savannah as a > prerequisite for switching, and then make sure it is repacked > regularly afterwards. I think what should be done is to increase the default configuration value of pack.window for the git repos on savannah. That should help keeping the packs better packed in the long run. Andreas. -- Andreas Schwab, schwab@linux-m68k.org GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5 "And now for something completely different." ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-08 18:34 ` Andreas Schwab @ 2014-01-08 18:42 ` Eli Zaretskii 2014-01-08 21:59 ` Angelo Graziosi 2014-01-12 6:27 ` Bob Proulx 1 sibling, 1 reply; 62+ messages in thread From: Eli Zaretskii @ 2014-01-08 18:42 UTC (permalink / raw To: Andreas Schwab Cc: stephen.berman, emacs-devel, sven.axelsson, angelo.graziosi > From: Andreas Schwab <schwab@linux-m68k.org> > Cc: Angelo Graziosi <angelo.graziosi@alice.it>, stephen.berman@gmx.net, sven.axelsson@gmail.com, emacs-devel@gnu.org > Date: Wed, 08 Jan 2014 19:34:56 +0100 > > Eli Zaretskii <eliz@gnu.org> writes: > > > So I guess we will need to repack the repository on savannah as a > > prerequisite for switching, and then make sure it is repacked > > regularly afterwards. > > I think what should be done is to increase the default configuration > value of pack.window for the git repos on savannah. That should help > keeping the packs better packed in the long run. Whatever the experts say, I know nowhere near enough about that. The goal is to let people clone 330MB as opposed to 1.1GB. Thanks. ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-08 18:42 ` Eli Zaretskii @ 2014-01-08 21:59 ` Angelo Graziosi 2014-01-09 0:05 ` Stefan Monnier 0 siblings, 1 reply; 62+ messages in thread From: Angelo Graziosi @ 2014-01-08 21:59 UTC (permalink / raw To: Eli Zaretskii, Andreas Schwab; +Cc: stephen.berman, sven.axelsson, emacs-devel Il 08/01/2014 19.42, Eli Zaretskii ha scritto: > Whatever the experts say, I know nowhere near enough about that. The > goal is to let people clone 330MB as opposed to 1.1GB. Indeed.. Why should one clone 1.1 GiB and the repack all in 330 MiB? and shouldn't the remote repository "cleaned" (repacked etc..) regularly? Ciao, Angelo. ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-08 21:59 ` Angelo Graziosi @ 2014-01-09 0:05 ` Stefan Monnier 2014-01-09 6:35 ` Eli Zaretskii 0 siblings, 1 reply; 62+ messages in thread From: Stefan Monnier @ 2014-01-09 0:05 UTC (permalink / raw To: Angelo Graziosi Cc: Eli Zaretskii, stephen.berman, Andreas Schwab, sven.axelsson, emacs-devel > Indeed.. Why should one clone 1.1 GiB and the repack all in 330 MiB? and > shouldn't the remote repository "cleaned" (repacked etc..) regularly? Because Git is so much more efficient, Stefan ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-09 0:05 ` Stefan Monnier @ 2014-01-09 6:35 ` Eli Zaretskii 0 siblings, 0 replies; 62+ messages in thread From: Eli Zaretskii @ 2014-01-09 6:35 UTC (permalink / raw To: Stefan Monnier Cc: emacs-devel, stephen.berman, schwab, sven.axelsson, angelo.graziosi > From: Stefan Monnier <monnier@iro.umontreal.ca> > Cc: Eli Zaretskii <eliz@gnu.org>, Andreas Schwab <schwab@linux-m68k.org>, stephen.berman@gmx.net, sven.axelsson@gmail.com, emacs-devel@gnu.org > Date: Wed, 08 Jan 2014 19:05:47 -0500 > > > Indeed.. Why should one clone 1.1 GiB and the repack all in 330 MiB? and > > shouldn't the remote repository "cleaned" (repacked etc..) regularly? > > Because Git is so much more efficient, A joke, I presume. But if not: pulling 1.1GB through the net instead of just a third or forth of that is still a painful experience in some parts of the world, and git can do nothing at all about that, even though it compresses. There really is no excuse for having such a badly packed repository on Savannah. ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-08 18:34 ` Andreas Schwab 2014-01-08 18:42 ` Eli Zaretskii @ 2014-01-12 6:27 ` Bob Proulx 2014-01-12 10:11 ` Andreas Schwab ` (2 more replies) 1 sibling, 3 replies; 62+ messages in thread From: Bob Proulx @ 2014-01-12 6:27 UTC (permalink / raw To: emacs-devel Andreas Schwab wrote: > Eli Zaretskii writes: > > So I guess we will need to repack the repository on savannah as a > > prerequisite for switching, and then make sure it is repacked > > regularly afterwards. > > I think what should be done is to increase the default configuration > value of pack.window for the git repos on savannah. That should help > keeping the packs better packed in the long run. All of the repositories are repacked monthly on the 4th. As you can see it is a heavy operation and therefore not run every day. But this thread seems to have exposed a deficiency in the current repacking. Currently the script does: git --git-dir=$dir gc -q Obvious from the discussion here that is not sufficient. However: Angelo Graziosi wrote: > $ git gc --aggressive > Counting objects: 735546, done. > Delta compression using up to 2 threads. > Compressing objects: 100% (734446/734446), done. > Writing objects: 100% (735546/735546), done. > Total 735546 (delta 589814), reused 143177 (delta 0) > Checking connectivity: 735546, done. > > which took almost 3 hours to be completed. Eli Zaretskii wrote: > And now I understand why git ran out of memory on Windows XP: the > memory footprint of git-pack-objects goes up to 3GB, whereas Windows > XP has only 3GB address space for all the applications. Three hours and 3G of memory for this one repack! I worry that turning --agressive on globally will cause the server to fall over. Also apparently careful tuning of threads is needed. However the disk and network savings look significant. I have a couple of questions if someone already has the clone available for a test. Does a clone from the agressively repacked to 327M repository remain similarly compacted when cloned again from it? If an agressively repacked repository is again repacked but this time without the --agressive option does the size stay around 327M or does it get expanded on the subsequent pass? (Wondering if we can periodically run 'git gc --agressive' on the larger git repositories at a niced background task priority but not too often and still achieve a good benefit for the time between agressive repacks.) Bob ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-12 6:27 ` Bob Proulx @ 2014-01-12 10:11 ` Andreas Schwab 2014-01-12 16:11 ` Eli Zaretskii 2014-01-12 21:13 ` Bob Proulx 2014-01-12 10:29 ` Achim Gratz 2014-01-12 16:07 ` Eli Zaretskii 2 siblings, 2 replies; 62+ messages in thread From: Andreas Schwab @ 2014-01-12 10:11 UTC (permalink / raw To: Bob Proulx; +Cc: emacs-devel Bob Proulx <bob@proulx.com> writes: > Three hours and 3G of memory for this one repack! I worry that > turning --agressive on globally will cause the server to fall over. IMHO --aggressive is too aggressive, and not useful for on-going maintenance. Incresing pack.window to 50 or so should already help enough to improve the packing. > If an agressively repacked repository is again repacked but this time > without the --agressive option does the size stay around 327M or does > it get expanded on the subsequent pass? Unless you run repack with -f (ie. gc --aggresive) existing deltas are reused, and only newly added objects are deltified. > (Wondering if we can periodically run 'git gc --agressive' on the > larger git repositories at a niced background task priority but not > too often and still achieve a good benefit for the time between > agressive repacks.) Another option is to touch a <pack>.keep file for the largest pack so that it is never touched again. New objects will then be added to a separate pack even after git gc. If that large pack is already well packed this should save some processing time. Andreas. -- Andreas Schwab, schwab@linux-m68k.org GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5 "And now for something completely different." ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-12 10:11 ` Andreas Schwab @ 2014-01-12 16:11 ` Eli Zaretskii 2014-01-12 17:09 ` Andreas Schwab 2014-01-12 21:13 ` Bob Proulx 1 sibling, 1 reply; 62+ messages in thread From: Eli Zaretskii @ 2014-01-12 16:11 UTC (permalink / raw To: Andreas Schwab; +Cc: bob, emacs-devel > From: Andreas Schwab <schwab@linux-m68k.org> > Date: Sun, 12 Jan 2014 11:11:21 +0100 > Cc: emacs-devel@gnu.org > > Increasing pack.window to 50 or so should already help enough to > improve the packing. Would it make sense to make such a change in our local clones as well? ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-12 16:11 ` Eli Zaretskii @ 2014-01-12 17:09 ` Andreas Schwab 2014-01-12 17:41 ` Eli Zaretskii 0 siblings, 1 reply; 62+ messages in thread From: Andreas Schwab @ 2014-01-12 17:09 UTC (permalink / raw To: Eli Zaretskii; +Cc: bob, emacs-devel Eli Zaretskii <eliz@gnu.org> writes: >> Increasing pack.window to 50 or so should already help enough to >> improve the packing. > > Would it make sense to make such a change in our local clones as well? Yes, setting that in your personal .gitconfig is probably a good idea. Andreas. -- Andreas Schwab, schwab@linux-m68k.org GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5 "And now for something completely different." ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-12 17:09 ` Andreas Schwab @ 2014-01-12 17:41 ` Eli Zaretskii 0 siblings, 0 replies; 62+ messages in thread From: Eli Zaretskii @ 2014-01-12 17:41 UTC (permalink / raw To: Andreas Schwab; +Cc: bob, emacs-devel > From: Andreas Schwab <schwab@linux-m68k.org> > Cc: bob@proulx.com, emacs-devel@gnu.org > Date: Sun, 12 Jan 2014 18:09:29 +0100 > > Eli Zaretskii <eliz@gnu.org> writes: > > >> Increasing pack.window to 50 or so should already help enough to > >> improve the packing. > > > > Would it make sense to make such a change in our local clones as well? > > Yes, setting that in your personal .gitconfig is probably a good idea. Thanks. ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-12 10:11 ` Andreas Schwab 2014-01-12 16:11 ` Eli Zaretskii @ 2014-01-12 21:13 ` Bob Proulx 2014-01-12 21:24 ` Andreas Schwab 2014-01-15 5:15 ` Bob Proulx 1 sibling, 2 replies; 62+ messages in thread From: Bob Proulx @ 2014-01-12 21:13 UTC (permalink / raw To: emacs-devel Andreas Schwab wrote: > Bob Proulx <bob@proulx.com> writes: > > Three hours and 3G of memory for this one repack! I worry that > > turning --agressive on globally will cause the server to fall over. > > IMHO --aggressive is too aggressive, and not useful for on-going > maintenance. Incresing pack.window to 50 or so should already help > enough to improve the packing. It sounds like I should schedule a full agressive repack once. Once being the operative word. git repack -a -d -f --window=250 --depth=250 And then as you suggest above ensure that the regularly routine ongoing maintenance 'git gc' runs with a large enough pack window. I see that the git-gc default is --window=250 so at that point nothing more should need to be done. The normal 'git gc' routine maintenance should be okay at that point. Is that right? Seems like it. We can tweak this further as needed. Certain times of day the vcs system is quite heavily loaded. The vcs system is a VM on a dom0 also hosting many other VMs. At some times of day the entire dom0 is very heavily I/O limited. This has been an ongoing problem and discussion. It is definitely an ongoing problem. I will run this during a less busy dom0 time. I spent some time researching this problem and found this note and some information in the entire thread useful and interesting. It is a long thread but there are some good gems in there. http://article.gmane.org/gmane.comp.gcc.devel/94613 > > If an agressively repacked repository is again repacked but this time > > without the --agressive option does the size stay around 327M or does > > it get expanded on the subsequent pass? > > Unless you run repack with -f (ie. gc --aggresive) existing deltas are > reused, and only newly added objects are deltified. Sounds good. I will note that -f forces a full (--no-reuse-delta) repack of everything and should be avoided. > > (Wondering if we can periodically run 'git gc --agressive' on the > > larger git repositories at a niced background task priority but not > > too often and still achieve a good benefit for the time between > > agressive repacks.) > > Another option is to touch a <pack>.keep file for the largest pack so > that it is never touched again. New objects will then be added to a > separate pack even after git gc. If that large pack is already well > packed this should save some processing time. That seems like a useful additional tweak for a large stage such as this. If nothing else it will help out the backup since that file won't be changing on a routine basis and will remain static for the purposes of backup transfer. This could be applied to several of the large repositories. It seems that on the client side after a new clone that this tweak is not propagated. It seems that if there are multiple packs on the server side that they are combined into a single pack file. But without being repacked. Therefore downstream clients that wish this would need to do it manually after a clone. Bob ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-12 21:13 ` Bob Proulx @ 2014-01-12 21:24 ` Andreas Schwab 2014-01-13 3:07 ` Bob Proulx 2014-01-15 5:15 ` Bob Proulx 1 sibling, 1 reply; 62+ messages in thread From: Andreas Schwab @ 2014-01-12 21:24 UTC (permalink / raw To: emacs-devel Bob Proulx <bob@proulx.com> writes: > I see that the git-gc default is --window=250 Only if you add --aggressive (which uses gc.aggressiveWindow to override the default for git repack, which is pack.window). Andreas. -- Andreas Schwab, schwab@linux-m68k.org GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5 "And now for something completely different." ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-12 21:24 ` Andreas Schwab @ 2014-01-13 3:07 ` Bob Proulx 0 siblings, 0 replies; 62+ messages in thread From: Bob Proulx @ 2014-01-13 3:07 UTC (permalink / raw To: emacs-devel Andreas Schwab wrote: > Bob Proulx <bob@proulx.com> writes: > > I see that the git-gc default is --window=250 > > Only if you add --aggressive (which uses gc.aggressiveWindow to override > the default for git repack, which is pack.window). Ah... Good clarification. I have updated pack.window and set the new value to 250. Bob ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-12 21:13 ` Bob Proulx 2014-01-12 21:24 ` Andreas Schwab @ 2014-01-15 5:15 ` Bob Proulx 1 sibling, 0 replies; 62+ messages in thread From: Bob Proulx @ 2014-01-15 5:15 UTC (permalink / raw To: emacs-devel Bob Proulx wrote: > It sounds like I should schedule a full agressive repack once. Once > being the operative word. > > git repack -a -d -f --window=250 --depth=250 This has been done and the repository is now down to 285M. I set the number of pack.threads=2 and it ran for 104 minutes to completion. (I had experimented letting it use all available 7-cores but that ran 94 minutes and ran out of memory at the 3G 32-bit limit. The progress indicator took 15 minutes to get to 90% and then spent the next 80 minutes getting no further than 92%.) > > Another option is to touch a <pack>.keep file for the largest pack so > > that it is never touched again. New objects will then be added to a > > separate pack even after git gc. If that large pack is already well > > packed this should save some processing time. > > That seems like a useful additional tweak for a large stage such as > this. If nothing else it will help out the backup since that file > won't be changing on a routine basis and will remain static for the > purposes of backup transfer. This could be applied to several of the > large repositories. > > It seems that on the client side after a new clone that this tweak is > not propagated. It seems that if there are multiple packs on the > server side that they are combined into a single pack file. But > without being repacked. Therefore downstream clients that wish this > would need to do it manually after a clone. I also touched the .keep file for the resulting pack file so that it would remain static. I checked out a local clone and the size was 329M. Bob P.S. I saw the other discussion thread talking about possibly redoing the repository. If that happens let me know and I will repack just as above again. ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-12 6:27 ` Bob Proulx 2014-01-12 10:11 ` Andreas Schwab @ 2014-01-12 10:29 ` Achim Gratz 2014-01-12 21:29 ` Bob Proulx 2014-01-12 16:07 ` Eli Zaretskii 2 siblings, 1 reply; 62+ messages in thread From: Achim Gratz @ 2014-01-12 10:29 UTC (permalink / raw To: emacs-devel Bob Proulx writes: > I have a couple of questions if someone already has the clone > available for a test. Does a clone from the agressively repacked to > 327M repository remain similarly compacted when cloned again from it? Once the whole repository is repacked it keeps those pack decisions, also across clones. The pack strategy for new commits would typically be near-optimal unless someone fast-imported a large branch again (in that case it should preferrably be locally repacked before getting pushed to Savannah). > If an agressively repacked repository is again repacked but this time > without the --agressive option does the size stay around 327M or does > it get expanded on the subsequent pass? I don't think a plain gc will expand any old packs, in any case the benefits of "git gc --agressive" are supposed to last for a long time. >(Wondering if we can periodically run 'git gc --agressive' on the >larger git repositories at a niced background task priority but not too >often and still achieve a good benefit for the time between agressive >repacks.) Just a manual "git gc --aggressive" once in a while should be enough. If you are going to do it as a cron-job, I'm not sure if it's a good idea to nice it. I think you'd rather want to limit the number of threads it uses, which also limits the memory footprint, which is probably the bigger problem anyway and should probably be tuned further depending on how beefy the server is. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ SD adaptations for Waldorf Q V3.00R3 and Q+ V3.54R2: http://Synth.Stromeko.net/Downloads.html#WaldorfSDada ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-12 10:29 ` Achim Gratz @ 2014-01-12 21:29 ` Bob Proulx 0 siblings, 0 replies; 62+ messages in thread From: Bob Proulx @ 2014-01-12 21:29 UTC (permalink / raw To: emacs-devel Achim Gratz wrote: > Bob Proulx writes: > >(Wondering if we can periodically run 'git gc --agressive' on the > >larger git repositories at a niced background task priority but not too > >often and still achieve a good benefit for the time between agressive > >repacks.) > > Just a manual "git gc --aggressive" once in a while should be enough. > If you are going to do it as a cron-job, I'm not sure if it's a good > idea to nice it. Due to the performance issues some of the Savannah admins have started nicing background tasks as to hint to the scheduler. Hoping that interactive tasks get more priority. Because at some times of day there has been a problem with the server being slow enough to cause timeouts. Especially the web frontend to vcs. Therefore if it isn't a time critical task then running it niced is a hint to the scheduler. The problem is that as a VM all of the system performance information is fake data and can't show the actual system state. And since we are currently I/O limited at times the system is often blocked waiting for I/O in which case the cpu priority doesn't matter. There is more cpu resource to go around than backplane disk I/O. But from the VM itself there isn't any useful performance data. Only the FSF admins have access to the dom0 and we count on them for real performance information from it. Unfortunately the FSF admin time to work on server performance problems has been limited. > I think you'd rather want to limit the number of threads it uses, > which also limits the memory footprint, which is probably the bigger > problem anyway and should probably be tuned further depending on how > beefy the server is. The dom0 server is an 8-core AMD Opteron running Xen for virtualization. It has sufficient RAM. But IMNHO too much I/O from all of the VMs running on it. The vcs VM is allocated seven xen cores. The hint about threads is a good one and I will keep an eye on it. Bob ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-12 6:27 ` Bob Proulx 2014-01-12 10:11 ` Andreas Schwab 2014-01-12 10:29 ` Achim Gratz @ 2014-01-12 16:07 ` Eli Zaretskii 2 siblings, 0 replies; 62+ messages in thread From: Eli Zaretskii @ 2014-01-12 16:07 UTC (permalink / raw To: Bob Proulx; +Cc: emacs-devel > Date: Sat, 11 Jan 2014 23:27:22 -0700 > From: Bob Proulx <bob@proulx.com> > > Eli Zaretskii wrote: > > And now I understand why git ran out of memory on Windows XP: the > > memory footprint of git-pack-objects goes up to 3GB, whereas Windows > > XP has only 3GB address space for all the applications. > > Three hours and 3G of memory for this one repack! It was only 1 hour in my case (I guess Angelo has a slower machine). Also, limiting the number of threads to 2 or even 1 radically lowers the required memory usage. But that doesn't mean you need necessarily run "gc --agressive", please listen to Andreas on this matter. > I have a couple of questions if someone already has the clone > available for a test. Does a clone from the agressively repacked to > 327M repository remain similarly compacted when cloned again from it? It grows slowly after that. E.g., in the 3 days since then it grew by 12MB. I don't know whether this will level out later, a few days is not enough to tell. > If an agressively repacked repository is again repacked but this time > without the --agressive option does the size stay around 327M or does > it get expanded on the subsequent pass? No, it doesn't expand, at least not on my system. In fact, "git gc" just slashed the storage by 10MB out of 12MB that it gained since I ran "gc --agressive". Again, this is just to give you some numbers, I don't want to suggest what should be done on Savannah. ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-08 17:32 ` Eli Zaretskii 2014-01-08 18:34 ` Andreas Schwab @ 2014-01-08 21:55 ` Angelo Graziosi 2014-01-09 19:44 ` Eli Zaretskii 1 sibling, 1 reply; 62+ messages in thread From: Angelo Graziosi @ 2014-01-08 21:55 UTC (permalink / raw To: Eli Zaretskii; +Cc: schwab, stephen.berman, sven.axelsson, emacs-devel Il 08/01/2014 18.32, Eli Zaretskii ha scritto: >> Date: Tue, 07 Jan 2014 21:08:51 +0100 >> From: Angelo Graziosi <angelo.graziosi@alice.it> >> CC: stephen.berman@gmx.net, sven.axelsson@gmail.com, emacs-devel@gnu.org >> >> $ cd emacs.git/ >> $ git gc --aggressive >> Counting objects: 735546, done. >> Delta compression using up to 2 threads. >> Compressing objects: 100% (734446/734446), done. >> Writing objects: 100% (735546/735546), done. >> Total 735546 (delta 589814), reused 143177 (delta 0) >> Checking connectivity: 735546, done. >> >> which took almost 3 hours to be completed. >> >> And the size is: >> >> $ cd .. >> >> $ du -s emacs.git/ >> 327M emacs.git/ >> >> which seems very good! > > I guess this was on Windows 7, right? I succeeded to do this on a NO!! It is on Win XP 32 bit (SP3) with AMD Athlon 64 X2 Dual Core Processor 3800+ and 1.7GiB of RAM. On the same machine I have Kubuntu 12.04 and "git gc --aggressive" works there too.. BTW, I noticed that running "git pull" to have the local repository updated, the size increases of about 1 MiB. > 64-bit Windows 7 as well, and it took about 2 hours. And now I > understand why git ran out of memory on Windows XP: the memory > footprint of git-pack-objects goes up to 3GB, whereas Windows XP has > only 3GB address space for all the applications. Here, on XP, Task Manager shows a memory usage under 1500 MiB (about 1.4 GiB) In any case this > So I guess we will need to repack the repository on savannah as a > prerequisite for switching, and then make sure it is repacked > regularly afterwards. Otherwise, people with low memory and/or slow > CPUs will need many hours to do this on their local machines. > should be mandatory.. Ciao, Angelo. ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-08 21:55 ` Angelo Graziosi @ 2014-01-09 19:44 ` Eli Zaretskii 0 siblings, 0 replies; 62+ messages in thread From: Eli Zaretskii @ 2014-01-09 19:44 UTC (permalink / raw To: Angelo Graziosi; +Cc: emacs-devel > Date: Wed, 08 Jan 2014 22:55:21 +0100 > From: Angelo Graziosi <angelo.graziosi@alice.it> > CC: schwab@suse.de, stephen.berman@gmx.net, sven.axelsson@gmail.com, > emacs-devel@gnu.org > > Il 08/01/2014 18.32, Eli Zaretskii ha scritto: > >> Date: Tue, 07 Jan 2014 21:08:51 +0100 > >> From: Angelo Graziosi <angelo.graziosi@alice.it> > >> CC: stephen.berman@gmx.net, sven.axelsson@gmail.com, emacs-devel@gnu.org > >> > >> $ cd emacs.git/ > >> $ git gc --aggressive > >> Counting objects: 735546, done. > >> Delta compression using up to 2 threads. > >> Compressing objects: 100% (734446/734446), done. > >> Writing objects: 100% (735546/735546), done. > >> Total 735546 (delta 589814), reused 143177 (delta 0) > >> Checking connectivity: 735546, done. > >> > >> which took almost 3 hours to be completed. > >> > >> And the size is: > >> > >> $ cd .. > >> > >> $ du -s emacs.git/ > >> 327M emacs.git/ > >> > >> which seems very good! > > > > I guess this was on Windows 7, right? I succeeded to do this on a > > NO!! It is on Win XP 32 bit (SP3) with AMD Athlon 64 X2 Dual Core > Processor 3800+ and 1.7GiB of RAM. The secret that unlocked this mystery was this: > >> Delta compression using up to 2 threads. ^^^^^^^^^ On my machine, which is a Core i7, this originally said "8 threads". So I did this: git config --global pack.threads "2" then ran "git gc --aggressive" again. That again ran out of memory, but much later in the process. The next step was obvious: git config --global pack.threads "1" With that, "git gc --aggressive" churned away for a little bit more than an hour, and brought me to this: $ du -sh . 323M . > > 64-bit Windows 7 as well, and it took about 2 hours. And now I > > understand why git ran out of memory on Windows XP: the memory > > footprint of git-pack-objects goes up to 3GB, whereas Windows XP has > > only 3GB address space for all the applications. > > Here, on XP, Task Manager shows a memory usage under 1500 MiB (about 1.4 > GiB) With 1 thread, it never goes above 850MB. ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-07 14:41 ` Stephen Berman 2014-01-07 14:57 ` Andreas Schwab @ 2014-01-07 15:00 ` Angelo Graziosi 1 sibling, 0 replies; 62+ messages in thread From: Angelo Graziosi @ 2014-01-07 15:00 UTC (permalink / raw To: Stephen Berman, Sven Axelsson; +Cc: Andreas Schwab, emacs Il 07/01/2014 15.41, Stephen Berman ha scritto: > Is it correct that main or master is the same thing as the Emacs trunk > branch? If so, I wonder why the git repository containing just this is > so large: my bzr Emacs repository only contains the trunk (as a bound > branch), several local branches of it, and the emacs-24 branch; the > .bzr/ directory of the repository, containing all the history, is only > 428 MiB, not 1.1 GiB (the trunk directory itself contains an additional > 129.3 MiB, but the total is still half the size of the git repo). Why > is the git repository so much larger than the bzr repository? As I noticed before (http://lists.gnu.org/archive/html/emacs-devel/2014-01/msg00515.html), it contains emacs.git/.git directory of about 930 MiB... Ciao, Angelo. ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-07 12:27 ` Move to git is imminent - awaiting Stefan's approval Sven Axelsson 2014-01-07 14:41 ` Stephen Berman @ 2014-01-07 16:17 ` Eli Zaretskii 2014-01-10 6:40 ` Antonio Nikishaev 2 siblings, 0 replies; 62+ messages in thread From: Eli Zaretskii @ 2014-01-07 16:17 UTC (permalink / raw To: Sven Axelsson; +Cc: emacs-devel, schwab, angelo.graziosi > Date: Tue, 7 Jan 2014 13:27:22 +0100 > From: Sven Axelsson <sven.axelsson@gmail.com> > Cc: Andreas Schwab <schwab@linux-m68k.org>, emacs <emacs-devel@gnu.org> > > Depth 1 truncates all history except for the most recent > commit. I think it keeps the 2 last commits, try "git log" in such a clone. > From `man git clone`: > > A shallow repository has a number of limitations (you cannot clone or > fetch from it, nor push from nor into it), but is adequate if you are > only interested in the recent history of a large project with a long > history, and would want to send in fixes as patches. For these reasons, this kind of repository is not recommended for "Emacs devs". ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-07 12:27 ` Move to git is imminent - awaiting Stefan's approval Sven Axelsson 2014-01-07 14:41 ` Stephen Berman 2014-01-07 16:17 ` Eli Zaretskii @ 2014-01-10 6:40 ` Antonio Nikishaev 2 siblings, 0 replies; 62+ messages in thread From: Antonio Nikishaev @ 2014-01-10 6:40 UTC (permalink / raw To: emacs-devel Sven Axelsson <sven.axelsson@gmail.com> writes: >> Instead, trying >> >> $ git clone --depth 1 git://git.savannah.gnu.org/emacs.git emacs.git >> >> the local repository has dimension 160M which is a little greater >> than my current BZR (checkout --lightweight) repository (126M), but >> acceptable... >> >> Suppose now I want identify which "revision" introduced a bug. I am >> able to do this with a binary search using the BZR command >> >> $ bzr up -r ARG >> >> How can I do this with the "minimal" clone done with GIT? > > Git does not have any concept like Bazaar's lightweight checkouts - > nor does any other dvcd afaik. Darcs has clever lazy repos. (Everything works. If the stuff which is not yet in the repository is needed, it will be downloaded.) Just FYI ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-07 10:24 ` Angelo Graziosi 2014-01-07 10:37 ` Git tips for Emacs development (was Re: Move to git is imminent - awaiting Stefan's approval) Nic Ferrier 2014-01-07 12:27 ` Move to git is imminent - awaiting Stefan's approval Sven Axelsson @ 2014-01-07 12:29 ` David Kastrup 2014-01-07 16:11 ` Eli Zaretskii 3 siblings, 0 replies; 62+ messages in thread From: David Kastrup @ 2014-01-07 12:29 UTC (permalink / raw To: emacs-devel Angelo Graziosi <angelo.graziosi@alice.it> writes: > Instead, trying > > $ git clone --depth 1 git://git.savannah.gnu.org/emacs.git emacs.git > > the local repository has dimension 160M which is a little greater than > my current BZR (checkout --lightweight) repository (126M), but > acceptable... > > Suppose now I want identify which "revision" introduced a bug. I am > able to do this with a binary search using the BZR command > > $ bzr up -r ARG > > How can I do this with the "minimal" clone done with GIT? Uh, not at all? Either you fetch previous revisions and can make use of them, or not. I think there is some command for extending a shallow clone, but of course that will have the respective impact on the repository size. -- David Kastrup ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-07 10:24 ` Angelo Graziosi ` (2 preceding siblings ...) 2014-01-07 12:29 ` David Kastrup @ 2014-01-07 16:11 ` Eli Zaretskii 3 siblings, 0 replies; 62+ messages in thread From: Eli Zaretskii @ 2014-01-07 16:11 UTC (permalink / raw To: Angelo Graziosi; +Cc: schwab, emacs-devel > Date: Tue, 07 Jan 2014 11:24:12 +0100 > From: Angelo Graziosi <angelo.graziosi@alice.it> > Cc: emacs <emacs-devel@gnu.org> > > >> $ du -s emacs-trunk/ > >> 126M emacs-trunk/ > >> > >> instead the git repository has dimension > >> > >> $ du -s emacs.git/ > >> 1.1G emacs.git/ > > > > By default git downloads the history of all branches. You can use > > --single-branch to only clone a single branch. This is the default if > > you create a shallow clone. > > David Kastrup wrote: > > > For the build? No. git clone has the following options: > > > > --depth <depth> > ... > > --[no-]single-branch > ... > > OK, trying > > $ git clone --single-branch git://git.savannah.gnu.org/emacs.git emacs.git > > the local repository has dimension 1.1G, so probably I need a more > explicit example... The command you tried is correct, it's just that it doesn't save too much disk space because the trunk has almost all of the revisions. > Instead, trying > > $ git clone --depth 1 git://git.savannah.gnu.org/emacs.git emacs.git > > the local repository has dimension 160M which is a little greater than > my current BZR (checkout --lightweight) repository (126M), but acceptable... This is not recommended at all, as such a shallow repository is severely limited: you cannot clone from it, nor push from or to it. You also have almost no history, and git does not support bzr-like operation whereby commands in lightweight checkouts that need history information go to the server. So you will be unable to walk through history and bisect. > Suppose now I want identify which "revision" introduced a bug. I am able > to do this with a binary search using the BZR command > > $ bzr up -r ARG > > How can I do this with the "minimal" clone done with GIT? You don't, see above. With a full repository, or at least one that was cloned with --single-branch, the equivalent of "bzr up" with git is "git checkout", and the revision which you want is specified by either is SHA1 signature or relatively, as in HEAD~20. ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-06 23:01 ` Andreas Schwab 2014-01-07 10:24 ` Angelo Graziosi @ 2014-01-07 13:37 ` Angelo Graziosi 2014-01-07 14:24 ` Andreas Schwab 2014-01-07 16:14 ` Eli Zaretskii 2 siblings, 1 reply; 62+ messages in thread From: Angelo Graziosi @ 2014-01-07 13:37 UTC (permalink / raw To: Andreas Schwab; +Cc: emacs Il 07/01/2014 0.01, Andreas Schwab ha scritto: > A good way to identify a git branch is to use git describe. However it > prefers to have an annotated (release) tag available to base on, which > we currently don't have (bzr doesn't have the concept of annotated > tags). Just for completeness... I see that I can download the trunk source with $ wget http://git.savannah.gnu.org/cgit/emacs.git/snapshot/emacs-trunk.tar.gz but how I can identify it? After unpacking it, the tree should contain at least a text file with a revision number or the SHA sum of the last commit... or not? Ciao, Angelo. ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-07 13:37 ` Angelo Graziosi @ 2014-01-07 14:24 ` Andreas Schwab 2014-01-09 12:18 ` Angelo Graziosi 0 siblings, 1 reply; 62+ messages in thread From: Andreas Schwab @ 2014-01-07 14:24 UTC (permalink / raw To: Angelo Graziosi; +Cc: emacs Angelo Graziosi <angelo.graziosi@alice.it> writes: > I see that I can download the trunk source with > > $ wget > http://git.savannah.gnu.org/cgit/emacs.git/snapshot/emacs-trunk.tar.gz > > but how I can identify it? Normally git get-tar-commit-id should do it, but the archive wasn't produced by git archive, or it was created from a tree ID instead of a commit ID. That looks like a bug in cgit. Andreas. -- Andreas Schwab, SUSE Labs, schwab@suse.de GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-07 14:24 ` Andreas Schwab @ 2014-01-09 12:18 ` Angelo Graziosi 0 siblings, 0 replies; 62+ messages in thread From: Angelo Graziosi @ 2014-01-09 12:18 UTC (permalink / raw To: Andreas Schwab; +Cc: emacs Il 07/01/2014 15.24, Andreas Schwab ha scritto: > Angelo Graziosi <angelo.graziosi@alice.it> writes: > >> I see that I can download the trunk source with >> >> $ wget >> http://git.savannah.gnu.org/cgit/emacs.git/snapshot/emacs-trunk.tar.gz >> >> but how I can identify it? > > Normally git get-tar-commit-id should do it, but the archive wasn't > produced by git archive, or it was created from a tree ID instead of a > commit ID. That looks like a bug in cgit. I have downloaded 1) http://git.savannah.gnu.org/cgit/emacs.git/snapshot/emacs-e42fcf1482e2c9e9d93f459985d13dbbabc61956.tar.gz 2) http://git.savannah.gnu.org/cgit/emacs.git/snapshot/emacs-trunk.tar.gz which correspond to the same trunk but I cannot identify the trunk: the command "git get-tar-commit-id" hangs. If I have understood, with GIT the trunk is identified with the SHA1 string like "e42fcf1482e2c9e9d93f459985d13dbbabc61956", so that in the first case I know the trunk "revision". Anyway, I would prefer that that string is stored inside the tree given by those tar-balls. For example in a text file, LAST-COMMIT-ID, so that one can know it with $ cat emacs-e42fcf1482e2c9e9d93f459985d13dbbabc61956/LAST-COMMIT-ID or $ cat emacs-trunk/LAST-COMMIT-ID In this way, I can create a build script with this skeleton: ... wget http://git.savannah.gnu.org/cgit/emacs.git/snapshot/emacs-trunk.tar.gz tar -xf emacs-trunk.tar.gz rev=`cat emacs-trunk/LAST-COMMIT-ID` autogen.sh configure... make bootstrap make install DESTDIR=... tar -cf emacs-$rev-bin.tar.xz $INSTDIR Ciao, Angelo. ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-06 23:01 ` Andreas Schwab 2014-01-07 10:24 ` Angelo Graziosi 2014-01-07 13:37 ` Angelo Graziosi @ 2014-01-07 16:14 ` Eli Zaretskii 2014-01-07 17:38 ` Rüdiger Sonderfeld 2 siblings, 1 reply; 62+ messages in thread From: Eli Zaretskii @ 2014-01-07 16:14 UTC (permalink / raw To: Andreas Schwab; +Cc: emacs-devel, angelo.graziosi > From: Andreas Schwab <schwab@linux-m68k.org> > Date: Tue, 07 Jan 2014 00:01:45 +0100 > Cc: emacs <emacs-devel@gnu.org> > > Angelo Graziosi <angelo.graziosi@alice.it> writes: > > > Now trying > > > > $ git clone git://git.savannah.gnu.org/emacs.git emacs.git > > > > > > I don't get any revision number.. So in my future bug report how the trunk > > will be identified? > > A good way to identify a git branch is to use git describe. However it > prefers to have an annotated (release) tag available to base on, which > we currently don't have Indeed: $ git describe fatal: No annotated tags can describe 'e1d32fe4408170b992528a2da4b917a63e86fbc4' I think Angelo wants one of the following, which will all produce exactly the same output: git log --pretty='format:%H' @^.. git rev-list @^.. git rev-parse --verify HEAD ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Move to git is imminent - awaiting Stefan's approval 2014-01-07 16:14 ` Eli Zaretskii @ 2014-01-07 17:38 ` Rüdiger Sonderfeld 0 siblings, 0 replies; 62+ messages in thread From: Rüdiger Sonderfeld @ 2014-01-07 17:38 UTC (permalink / raw To: emacs-devel, Eli Zaretskii; +Cc: Andreas Schwab, angelo.graziosi On Tuesday 07 January 2014 18:14:45 Eli Zaretskii wrote: > $ git describe > fatal: No annotated tags can describe > 'e1d32fe4408170b992528a2da4b917a63e86fbc4' > > I think Angelo wants one of the following, which will all produce > exactly the same output: > > git log --pretty='format:%H' @^.. > git rev-list @^.. > git rev-parse --verify HEAD Or git describe --always git show --pretty=oneline git show --oneline (Produces a different output though) Regards Rüdiger ^ permalink raw reply [flat|nested] 62+ messages in thread
end of thread, other threads:[~2014-01-15 5:15 UTC | newest] Thread overview: 62+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2014-01-06 22:14 Move to git is imminent - awaiting Stefan's approval Angelo Graziosi 2014-01-06 22:57 ` David Kastrup 2014-01-11 22:34 ` Nix 2014-01-06 23:01 ` Andreas Schwab 2014-01-07 10:24 ` Angelo Graziosi 2014-01-07 10:37 ` Git tips for Emacs development (was Re: Move to git is imminent - awaiting Stefan's approval) Nic Ferrier 2014-01-07 16:05 ` Eli Zaretskii 2014-01-07 12:27 ` Move to git is imminent - awaiting Stefan's approval Sven Axelsson 2014-01-07 14:41 ` Stephen Berman 2014-01-07 14:57 ` Andreas Schwab 2014-01-07 15:05 ` Stephen Berman 2014-01-07 16:25 ` Eli Zaretskii 2014-01-07 16:35 ` Andreas Schwab 2014-01-07 16:37 ` Angelo Graziosi 2014-01-07 16:41 ` Andreas Schwab 2014-01-07 17:08 ` Eli Zaretskii 2014-01-07 17:23 ` Óscar Fuentes 2014-01-07 17:56 ` Eli Zaretskii 2014-01-07 17:24 ` Stephen J. Turnbull 2014-01-07 18:05 ` Eli Zaretskii 2014-01-07 18:17 ` Sven Axelsson 2014-01-07 18:49 ` Stephen J. Turnbull 2014-01-07 19:06 ` Eli Zaretskii 2014-01-08 3:47 ` Stephen J. Turnbull 2014-01-08 15:23 ` Eli Zaretskii 2014-01-07 18:02 ` Thien-Thi Nguyen 2014-01-07 18:06 ` Sven Axelsson 2014-01-07 18:47 ` Andreas Schwab 2014-01-07 19:07 ` Sven Axelsson 2014-01-07 19:16 ` Eli Zaretskii 2014-01-08 3:52 ` Stephen J. Turnbull 2014-01-07 20:08 ` Angelo Graziosi 2014-01-08 17:32 ` Eli Zaretskii 2014-01-08 18:34 ` Andreas Schwab 2014-01-08 18:42 ` Eli Zaretskii 2014-01-08 21:59 ` Angelo Graziosi 2014-01-09 0:05 ` Stefan Monnier 2014-01-09 6:35 ` Eli Zaretskii 2014-01-12 6:27 ` Bob Proulx 2014-01-12 10:11 ` Andreas Schwab 2014-01-12 16:11 ` Eli Zaretskii 2014-01-12 17:09 ` Andreas Schwab 2014-01-12 17:41 ` Eli Zaretskii 2014-01-12 21:13 ` Bob Proulx 2014-01-12 21:24 ` Andreas Schwab 2014-01-13 3:07 ` Bob Proulx 2014-01-15 5:15 ` Bob Proulx 2014-01-12 10:29 ` Achim Gratz 2014-01-12 21:29 ` Bob Proulx 2014-01-12 16:07 ` Eli Zaretskii 2014-01-08 21:55 ` Angelo Graziosi 2014-01-09 19:44 ` Eli Zaretskii 2014-01-07 15:00 ` Angelo Graziosi 2014-01-07 16:17 ` Eli Zaretskii 2014-01-10 6:40 ` Antonio Nikishaev 2014-01-07 12:29 ` David Kastrup 2014-01-07 16:11 ` Eli Zaretskii 2014-01-07 13:37 ` Angelo Graziosi 2014-01-07 14:24 ` Andreas Schwab 2014-01-09 12:18 ` Angelo Graziosi 2014-01-07 16:14 ` Eli Zaretskii 2014-01-07 17:38 ` Rüdiger Sonderfeld
Code repositories for project(s) associated with this external index https://git.savannah.gnu.org/cgit/emacs.git https://git.savannah.gnu.org/cgit/emacs/org-mode.git This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.