From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: John Wiegley Newsgroups: gmane.emacs.devel Subject: Re: Git mirrors Date: Wed, 12 Oct 2011 19:08:07 -0500 Message-ID: References: <8762k095n4.fsf@lifelogs.com> <871uuksdxi.fsf@lifelogs.com> <83lissxj9a.fsf@gnu.org> <87k48by7ex.fsf@lifelogs.com> <87r52jfbrw.fsf@wanadoo.es> <87k48bf1q9.fsf@wanadoo.es> NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Trace: dough.gmane.org 1318464502 11655 80.91.229.12 (13 Oct 2011 00:08:22 GMT) X-Complaints-To: usenet@dough.gmane.org NNTP-Posting-Date: Thu, 13 Oct 2011 00:08:22 +0000 (UTC) Cc: Andreas Schwab , emacs-devel@gnu.org To: =?utf-8?Q?=C3=93scar?= Fuentes Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Thu Oct 13 02:08:18 2011 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([140.186.70.17]) by lo.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1RE8qO-0006bs-1z for ged-emacs-devel@m.gmane.org; Thu, 13 Oct 2011 02:08:16 +0200 Original-Received: from localhost ([::1]:40390 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RE8qN-0005fc-3z for ged-emacs-devel@m.gmane.org; Wed, 12 Oct 2011 20:08:15 -0400 Original-Received: from eggs.gnu.org ([140.186.70.92]:44681) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RE8qK-0005fM-C3 for emacs-devel@gnu.org; Wed, 12 Oct 2011 20:08:13 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1RE8qJ-00017O-0R for emacs-devel@gnu.org; Wed, 12 Oct 2011 20:08:12 -0400 Original-Received: from mail-iy0-f169.google.com ([209.85.210.169]:58695) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RE8qI-00017K-Qw for emacs-devel@gnu.org; Wed, 12 Oct 2011 20:08:10 -0400 Original-Received: by iagj37 with SMTP id j37so732176iag.0 for ; Wed, 12 Oct 2011 17:08:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=to:cc:subject:references:from:date:in-reply-to:message-id :user-agent:mime-version:content-type:content-transfer-encoding; bh=qz03ldogGjj+dSgj43HFZVeaT8nFyet5IaeViUSd3EQ=; b=IUYXeKHuRprspzfws1vw1QNNi4JcIY/r5lY2Jj/qJPfbw2UGia/EV5RWIkRH/fqO/b +xi8LFmbrmXLDfC9FbN2hxnOQ615B7xwiBAHiMb/K2fVPLnf4vX2V7EcLb6BR1Ka1Xq0 T3x/KuPn4OmVEpeqiW3UAWaoGBDFQSfGHeN5Y= Original-Received: by 10.42.189.6 with SMTP id dc6mr2594904icb.16.1318464490157; Wed, 12 Oct 2011 17:08:10 -0700 (PDT) Original-Received: from vulcan.localhost (c-98-215-105-167.hsd1.il.comcast.net. [98.215.105.167]) by mx.google.com with ESMTPS id bu33sm9480441ibb.11.2011.10.12.17.08.08 (version=TLSv1/SSLv3 cipher=OTHER); Wed, 12 Oct 2011 17:08:09 -0700 (PDT) Original-Received: by vulcan.localhost (Postfix, from userid 501) id ED21D55516B7; Wed, 12 Oct 2011 19:08:07 -0500 (CDT) In-Reply-To: <87k48bf1q9.fsf@wanadoo.es> (=?utf-8?Q?=22=C3=93scar?= Fuentes"'s message of "Wed, 12 Oct 2011 03:16:46 +0200") User-Agent: Gnus/5.110018 (No Gnus v0.18) Emacs/23.3 (darwin) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 2) X-Received-From: 209.85.210.169 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Xref: news.gmane.org gmane.emacs.devel:145083 Archived-At: >>>>> =C3=93scar Fuentes writes: > AFAIK, the only one keeping the savannah git mirror up to date is Andreas > Schwab and he is doing that manually. I'm not sure what's the actual proc= ess > he is using, but if it is fast-import/fast-export the worst thing that co= uld > happen is a failed push because the other part pushed first. In that case= a > pull should fix things. But he has the authoritative word on this. Sure enough, I'm stepping on whoever else is running an automated mirroring process. Just about every hour I'm re-deleting branches that they are re-pushing: ,---- | =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D | Wed Oct 12 18:00:01 CDT 2011 | =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D | receiving incremental file list |=20 | sent 357 bytes received 20797 bytes 475.37 bytes/sec | total size is 376714147 speedup is 17808.18 | To savannah:/srv/git/emacs.git | - [deleted] old-branches/cedet-branch | - [deleted] old-branches/emacs-unicode | - [deleted] old-branches/gerd_defvaralias | - [deleted] old-branches/gnus-5_10-branch | - [deleted] old-branches/multi-tty | - [deleted] old-branches/pending | - [deleted] old-branches/rmail-mbox-branch | - [deleted] other-branches/gerd_0001 | - [deleted] other-branches/gerd_int | - [deleted] other-branches/miles-orphaned-changes | - [deleted] other-branches/thomas-posix1996 | * [new branch] tmp -> tmp `---- Who do I talk to so that they disable their mirror, or at least refine it? Below is the script I'm now using. It relies upon a Python script, also attached. John ------------------------------------------------------------------------ #!/bin/bash # Many thanks to Andreas Schwab for this script. It # is currently maintained by John Wiegley . test last-sync-start -nt last-sync-ready || touch -r last-sync-start last-s= ync touch last-sync-start echo =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D date echo =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D rsync -av --del --delete-excluded \ --exclude=3Dobsolete_packs \ --exclude=3Dlock/ \ bzr.sv.gnu.org::bzr/emacs/ emacs.bzr/ dc emacs.bzr find * -path '*/.bzr/branch/last-revision' | while read r; do test $r -ot ../last-sync && continue b=3D${r%/.bzr/*} b2=3D$b test $b =3D trunk && b2=3Dmaster test $b =3D master && b2=3Dbzr/master echo $b bzr fast-export --plain --marks=3D../bzr-marks -b $b2 $b | ( cd ../emacs.git git fast-import --quiet --export-marks=3D../git-marks --import-marks=3D= ../git-marks ) done dc ../emacs.git git push --mirror savannah:/srv/git/emacs.git cd .. bin/correlate | gzip -c > commit-map.txt.gz touch last-sync-ready exit 0 ### emacs-mirror.sh ends here ------------------------------------------------------------------------ #!/usr/bin/env python # This is bin/correlate import re import os def load_marks(path, inserter): marks =3D open(path) for line in marks.readlines(): match =3D re.match(':([0-9]+) (.+)', line) assert match ident, info =3D match.groups() inserter(table, ident, info) marks.close() def insert_at(table, key, value, pos): if key not in table: table[key] =3D [None, None] table[key][pos] =3D value table =3D {} load_marks('bzr-marks', lambda table, ident, info: insert_at(table, ident, info, 0)) load_marks('git-marks', lambda table, ident, info: insert_at(table, ident, info, 1)) for key in table: print table[key][0], table[key][1]