Switching to bzr: what Emacs developers should know?

Switching to bzr: what Emacs developers should know?

[Bastien]

Can someone describe the bzr workflow for Emacs developers?

Will the switch to bzr affect the way upstream packages like Gnus
or Org are integrated in Emacs development?

Would there be any advantage of switching to bzr for these packages?

-- 
 Bastien

Re: Switching to bzr: what Emacs developers should know?

[B Smith-Mannschott]

On Sat, Aug 8, 2009 at 18:24, Bastien<bastienguerry@googlemail.com> wrote:
> Can someone describe the bzr workflow for Emacs developers?

Presumably, you've seen this, yes?:
http://www.emacswiki.org/emacs/BzrForEmacsDevs

> Will the switch to bzr affect the way upstream packages like Gnus
> or Org are integrated in Emacs development?
>
> Would there be any advantage of switching to bzr for these packages?

Well, the wiki page currently only says:

[quote]
Merging Upstream Packages (e.g., Gnus) Into Emacs With Bazaar

TODO: TBD (but frankly, this should be easy, as it fits perfectly with
the DVCS/Bazaar model anyway)
[/quote]

// ben

Re: Switching to bzr: what Emacs developers should know?

[Stefan Monnier]

>> Can someone describe the bzr workflow for Emacs developers?
> Presumably, you've seen this, yes?:
> http://www.emacswiki.org/emacs/BzrForEmacsDevs
>> Will the switch to bzr affect the way upstream packages like Gnus
>> or Org are integrated in Emacs development?

Of course.  Gnus is the special one because it currently benefits from
a very nice setup.  For Org, I don't think it can get much worse.

>> Would there be any advantage of switching to bzr for these packages?
> Well, the wiki page currently only says:

> TODO: TBD (but frankly, this should be easy, as it fits perfectly with
> the DVCS/Bazaar model anyway)

Actually, for Gnus it seems not particularly easy, because two-way
merges like those Miles currently does don't seem to fit into the Bazaar
model nicely at all and because merging two separate package histories
into one seems to fit even worse.


        Stefan

Re: Switching to bzr: what Emacs developers should know?

[Bastien]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> Will the switch to bzr affect the way upstream packages like Gnus
>>> or Org are integrated in Emacs development?
>
> Of course.  Gnus is the special one because it currently benefits from
> a very nice setup.  For Org, I don't think it can get much worse.

That doesn't really tell how maintainance of Org will be affected by
switching to bzr.  

Also, http://www.emacswiki.org/emacs/BzrForEmacsDevs says:

  (Note: it’s not clear whether we’re going to continue the habit of
  keeping a separate ChangeLog file, now that we have a version control
  system that supports atomic multi-target commits. We should figure
  that out at some point.)

What's Emacs maintainers take on this?

FWIW, I think we shouldn't get rid of the ChangeLog because it doesn't
depend on the dVCS log format.

-- 
 Bastien

Re: Switching to bzr: what Emacs developers should know?

[Stefan Monnier]

>   (Note: it’s not clear whether we’re going to continue the habit of
>   keeping a separate ChangeLog file, now that we have a version control
>   system that supports atomic multi-target commits. We should figure
>   that out at some point.)

> What's Emacs maintainers take on this?

Check the mailing list, we've been very clear about this: we're only
going to change the underlying tool for now, so ChangeLog aren't
affected.


        Stefan

Re: Switching to bzr: what Emacs developers should know?

[Karl Fogel]

Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>   (Note: it’s not clear whether we’re going to continue the habit of
>>   keeping a separate ChangeLog file, now that we have a version control
>>   system that supports atomic multi-target commits. We should figure
>>   that out at some point.)
>
>> What's Emacs maintainers take on this?
>
> Check the mailing list, we've been very clear about this: we're only
> going to change the underlying tool for now, so ChangeLog aren't
> affected.

I've updated the wiki accordingly.

Re: Switching to bzr: what Emacs developers should know?

[Karl Fogel]

Stefan Monnier <monnier@iro.umontreal.ca> writes:
> Actually, for Gnus it seems not particularly easy, because two-way
> merges like those Miles currently does don't seem to fit into the Bazaar
> model nicely at all and because merging two separate package histories
> into one seems to fit even worse.

Well, I should get a better understanding of how Gnus is developed,
before trying to write that section of the wiki.  (Maybe my claim about
how it fits the Bzr model well is wrong...)

Is Gnus just in a separate CVS repository right now, and all the changes
get copied over in some primitive, history-destroying way when a new
Gnus is put into the Emacs distribution?  If so, the Bright Shining
Future would be to simply version Gnus within the Emacs tree, and have
the Gnus developers maintain their own long-lived branches, just as with
anything else.  They don't have to make changes outside the Gnus area
when they don't want to, after all.

Unless Gnus devs do not want to switch to Bazaar?  Can anyone here
answer, or should I be asking over in some Gnus forum?

-Karl

Re: Switching to bzr: what Emacs developers should know?

[Stefan Monnier]

> Is Gnus just in a separate CVS repository right now, and all the changes
> get copied over in some primitive, history-destroying way when a new
> Gnus is put into the Emacs distribution?

Currently Gnus has its own CVS repository and the two repositories are
sync'd two-ways by Miles via Arch mirrors.

> If so, the Bright Shining Future would be to simply version Gnus
> within the Emacs tree, and have the Gnus developers maintain their own
> long-lived branches, just as with anything else.  They don't have to
> make changes outside the Gnus area when they don't want to, after all.

For their own branches, everything is fine and solutions are easy to
find, but for the branches that are shared with Emacs, it's more
delicate: the layout between Emacs and Gnus is different, some of the
files are different, but we want changes made in either repository to
appear in the other.

This same problem appears with several other packages that are
maintained outside Emacs, tho Gnus is the only one to currently benefit
from a really nice solution.  So a good solution to this problem would
be useful for more than just Gnus.


        Stefan

Re: Switching to bzr: what Emacs developers should know?

[Stefan Monnier]

> I've been doing a bit of testing, and bzr can definitely facilitate this
> sort of thing.  In fact, you can set up a test by simply branching from
> an Emacs bzr repository and then using bzr mv and friends to put
> everything where Gnus expects it to be.  bzr will then happily keep
> things up to date even though the files are in different locations.

Yes.  But this still has 2 problems:

- as you noted, this loses Gnus's CVS history.  Maybe there's a way to
  fix it, or do something at least sufficient (e.g. keep the history
  in a completely separate branch).  All that's really needed to "do
  it right" would be for the CVS->Bzr conversion to be able to take
  "file->id" tables from one branch and apply it to the other.

- The above will happily deal with one way synchronization (i.e. future
  commits to the Emacs branch will be easy to merge into the Gnus
  branch), but I still don't know how to do the other sync (merge
  changes made on the Gnus branch onto the Emacs branch).

I've asked a similar question on the Bzr list and was mostly greeted
with silence.  I find it odd: such parallel branches are a pretty common
occurrence, in my experience, so while I understand that it might not be
easy/trivial to handle it, I'd expect at least some interest in trying
to come up with ways to support it.  Maybe there's a way to do it in
Bzr, but I haven't found it yet.  Note that most other VCS are just as
poor at it.  Arch is a bit better (mostly because you can
straightforwardly fiddle with the merge-history), and DaRCS is
positively good at it.

>> This same problem appears with several other packages that are
>> maintained outside Emacs, tho Gnus is the only one to currently
>> benefit from a really nice solution.  So a good solution to this
>> problem would be useful for more than just Gnus.
> For packages maintained outside of Emacs that don't currently have a
> solution to this problem bzr is likely to be a huge step in the right
> direction.  With the right plugin you might even be able to keep using
> git (or hg) for your own hacking and then use bzr to merge with Emacs.

I have no doubt that Bzr will make such things easier than CVS.
The two-way thingy done with Arch for Gnus is just a special treat that
requires extra manual fiddling.  Big thanks to Miles for that.


        Stefan

bzr for Gnus (was: Switching to bzr: what Emacs developers should know?)

[Ted Zlatanov]

On Tue, 11 Aug 2009 01:49:10 -0400 Karl Fogel <karl.fogel@canonical.com> wrote: 

KF> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> Actually, for Gnus it seems not particularly easy, because two-way
>> merges like those Miles currently does don't seem to fit into the Bazaar
>> model nicely at all and because merging two separate package histories
>> into one seems to fit even worse.

KF> Well, I should get a better understanding of how Gnus is developed,
KF> before trying to write that section of the wiki.  (Maybe my claim about
KF> how it fits the Bzr model well is wrong...)

KF> Is Gnus just in a separate CVS repository right now, and all the changes
KF> get copied over in some primitive, history-destroying way when a new
KF> Gnus is put into the Emacs distribution?  If so, the Bright Shining
KF> Future would be to simply version Gnus within the Emacs tree, and have
KF> the Gnus developers maintain their own long-lived branches, just as with
KF> anything else.  They don't have to make changes outside the Gnus area
KF> when they don't want to, after all.

Miles does the synchronization, and it preserves history and changes
from both sides well.

FWIW, I'd be happy switching to bzr for Gnus work.  I don't know how
well that would work for the following common needs:

- pull Gnus independently (for XEmacs users, including compatibility
  libraries)

- pull Gnus by itself (no compatibility libraries, for Emacs users)

- pull Gnus with Emacs (normal case)

In addition, there's a difference between an independent Gnus repository
synchronized with the Emacs repository and moving the entire Gnus
repository inside Emacs.  I don't know which one the Gnus developers
would prefer, or even how such a move would be decided.  My guess is
that Lars Ingebrigtsen should at least be involved in the discussion in
addition to the current Gnus contributors.

KF> Unless Gnus devs do not want to switch to Bazaar?  Can anyone here
KF> answer, or should I be asking over in some Gnus forum?

Gnus discussion is usually on the Ding list (ding@gnus.org) but most
Gnus developers read emacs-devel as well.  They may have missed your
post, though, so I'm following up to it with a new subject and copying
it to that list.

Ted

bzr for Gnus (was: Switching to bzr: what Emacs developers should know?)

[Stephen J. Turnbull]

Bcc'ing some of the people mentioned below as "concerned with XEmacs
packages" to ensure they see this post.  Sorry for any dupes.

Ted Zlatanov writes:

 > FWIW, I'd be happy switching to bzr for Gnus work.  I don't know how
 > well that would work for the following common needs:
 > 
 > - pull Gnus independently (for XEmacs users, including compatibility
 >   libraries)
 > 
 > - pull Gnus by itself (no compatibility libraries, for Emacs users)

bzr doesn't support this out of the box, yet.  Support for such
"nested trees" is planned (greatly desired by many users) but
implementation is not yet scheduled AIUI (there are some design
details to work out).  I don't speak for the bzr developers, but based
on the history of features I've followed on the bazaar@canonical list
I would be impressed if it stabilizes[1] in less than 6 months, and a
little surprised if it takes more than 12 to implement and stabilize.

bzr does support a "branch filtering" capability which could be used
to emulate this.  I'm not sure exactly how it works but it uses the
fastimport feature, so it could be done on the Gnus side basically
like maintaining a bidirectional mirror of a repo in a different VCS,
except that all the difficulties related to "impedence mismatch"
between VCSes would be avoided.

A final option for nesting trees would be to use git or Mercurial to
host the tricky parts; both have facilities (in git it's called
"submodule") for handling the inclusion of Gnus in Emacs.  git's is
limited in the sense that nesting of submodules in submodules is not
supported, but probably it would be be reasonably convenient to have a
separate XEmacs branch which contains the XEmacs compatibility code.
git would be my weapon of choice.

An alternative to a kludgy nest of "xmas" in Gnus in Emacs would be to
maintain the XEmacs compatibility code only in XEmacs package CVS, and
not have it in Gnus "upstream" at all.  Mike Kupfer has the authority
to nominate committers for the Gnus package and normally access would
be enabled within about 48 hours (an ssh key and user name are
required so there's typically a bit of back and forth needed), and if
he decides he likes the idea, that's good enough for us.  (He'll
probably consult with other directly interested parties such as Steve
Youngs and Norbert Koch, of course, but the authority is his.)  Of
course that would involve some annoyance (eg, dealing with a separate
checkout from our CVS as well as the mainline from Emacs's bzr) for
anybody working on "xmas" code, but it's an option we can consider.
Folks like Mike and Steve Youngs have done something similar to that
for a long time, so they can give accurate advice on the amount of
burden involved.

Footnotes: 
[1]  "Stabilize" in the sense of "at least as stable as a late
prerelease of Emacs".

Re: bzr for Gnus (was: Switching to bzr: what Emacs developers should know?)

[Mike Kupfer]

>>>>> "ST" == Stephen J Turnbull <stephen@xemacs.org> writes:

ST> An alternative to a kludgy nest of "xmas" in Gnus in Emacs would be
ST> to maintain the XEmacs compatibility code only in XEmacs package
ST> CVS, and not have it in Gnus "upstream" at all.  

Hmm, I'll have to think about this.  I'm reluctant to put the
XEmacs-specific code off in a separate place.  I suspect it would make
it too easy for someone to change the common code and forget to make any
corresponding changes in the XEmacs-specific code.  Using nested repos
might make that less of a concern, depending on the specifics of the
tool and how the workflow is set up.

mike

Re: bzr for Gnus

[Ted Zlatanov]

On Wed, 12 Aug 2009 06:50:36 -0700 Mike Kupfer <mike.kupfer@xemacs.org> wrote: 

>>>>>> "ST" == Stephen J Turnbull <stephen@xemacs.org> writes:
ST> An alternative to a kludgy nest of "xmas" in Gnus in Emacs would be
ST> to maintain the XEmacs compatibility code only in XEmacs package
ST> CVS, and not have it in Gnus "upstream" at all.  

MK> Hmm, I'll have to think about this.  I'm reluctant to put the
MK> XEmacs-specific code off in a separate place.  I suspect it would make
MK> it too easy for someone to change the common code and forget to make any
MK> corresponding changes in the XEmacs-specific code.  Using nested repos
MK> might make that less of a concern, depending on the specifics of the
MK> tool and how the workflow is set up.

According to http://bazaar-vcs.org/BzrForeignBranches/Subversion
it may have to wait for the 'by-reference' work to be complete.
Meanwhile we can certainly set up something manual or automatic (a
MilesBot, if you will) to synchronize the Gnus CVS repo with the Emacs
bzr repo.  As Stephen said, this may be months to 1+ years away.

Ted

Re: bzr for Gnus

[Karl Fogel]

A thought about the Bazaar switchover:

Let's not make the Gnus merge recipe a prerequisite for switching.
There are various ways to handle situations like Gnus's; I'm frankly not
sure which is the best, and I think experimentation on the part of those
closest to the code is going to be the only way to sort it out.

But the way to make that experimentation happen is to make it a
necessity.  I don't mean that callously -- it's just the way things
actually get done.  That's how it went when Emacs used CVS too.  That
is, Emacs switched to CVS (easy enough, since it had been on RCS before
that, IIRC), and externally versioned codebases that needed to integrate
with that figured out how to do so on the fly as the need arose.  That's
what's going to happen here too.

Does this sound sane?  I'm just worried about creating needlessly huge
blockers to the switchover.

-Karl

Ted Zlatanov <tzz@lifelogs.com> writes:
> On Wed, 12 Aug 2009 06:50:36 -0700 Mike Kupfer <mike.kupfer@xemacs.org> wrote: 
>
>>>>>>> "ST" == Stephen J Turnbull <stephen@xemacs.org> writes:
> ST> An alternative to a kludgy nest of "xmas" in Gnus in Emacs would be
> ST> to maintain the XEmacs compatibility code only in XEmacs package
> ST> CVS, and not have it in Gnus "upstream" at all.  
>
> MK> Hmm, I'll have to think about this.  I'm reluctant to put the
> MK> XEmacs-specific code off in a separate place.  I suspect it would make
> MK> it too easy for someone to change the common code and forget to make any
> MK> corresponding changes in the XEmacs-specific code.  Using nested repos
> MK> might make that less of a concern, depending on the specifics of the
> MK> tool and how the workflow is set up.
>
> According to http://bazaar-vcs.org/BzrForeignBranches/Subversion
> it may have to wait for the 'by-reference' work to be complete.
> Meanwhile we can certainly set up something manual or automatic (a
> MilesBot, if you will) to synchronize the Gnus CVS repo with the Emacs
> bzr repo.  As Stephen said, this may be months to 1+ years away.
>
> Ted

Re: bzr for Gnus

[Stefan Monnier]

> Let's not make the Gnus merge recipe a prerequisite for switching.

It's not.


        Stefan

Re: bzr for Gnus

[Miles Bader]

Ted Zlatanov <tzz@lifelogs.com> writes:
> Miles does the synchronization, and it preserves history and changes
> from both sides well.

BTW, I'll stop doing this when Emacs switches to bzr.

-Miles

-- 
Quotation, n. The act of repeating erroneously the words of another. The words
erroneously repeated.

Re: bzr for Gnus

[Stefan Monnier]

>> Miles does the synchronization, and it preserves history and changes
>> from both sides well.
> BTW, I'll stop doing this when Emacs switches to bzr.

I assumed so much.  Thank you again for the work you've done here, it
has been most appreciated.


        Stefan

Re: Switching to bzr: what Emacs developers should know?

[Bastien]

B Smith-Mannschott <bsmith.occs@gmail.com> writes:

> On Sat, Aug 8, 2009 at 18:24, Bastien<bastienguerry@googlemail.com> wrote:
>> Can someone describe the bzr workflow for Emacs developers?
>
> Presumably, you've seen this, yes?:
> http://www.emacswiki.org/emacs/BzrForEmacsDevs

Now I have, thanks.  Will send some comments in another message.

> Well, the wiki page currently only says:
>
> [quote]
> Merging Upstream Packages (e.g., Gnus) Into Emacs With Bazaar
>
> TODO: TBD (but frankly, this should be easy, as it fits perfectly with
> the DVCS/Bazaar model anyway)
> [/quote]

I hope someone will do this, it was the part I was particularily
interested in.

Thanks,

-- 
 Bastien

Re: Switching to bzr: what Emacs developers should know?

[Bastien]

B Smith-Mannschott <bsmith.occs@gmail.com> writes:

> Presumably, you've seen this, yes?:
> http://www.emacswiki.org/emacs/BzrForEmacsDevs

> First, initialize a repository in which to store all your branches:
> 
>       bzr init-repo --2a emacs/

(I hope we won't need this "--2a" option anytime soon.)

> Create a branch that will just be a mirror of the mainline. You’ll
> never make any changes to this branch; its job is just to reflect the
> upstream master:
> 
>       cd emacs/
>       bzr branch http://bzr.savannah.gnu.org/sources/emacs/trunk/ trunk/

Is this "trunk" branch necessary, or just a convenience?  

Why isn't the default branch enough for the trunk-tasks described 
in this page?

> And after refreshing the mirror, you’ll want to get those changes into
> your task branch, by merging them:
> 
>       cd SOME-TASKNAME/
>       bzr merge
>       bzr commit -m "Merge from mainline."

Is this merge happening between the local branch and the trunk or the
local branch and the default branch?  

> If you’re one of the Emacs maintainers, then you can just push it directly to the upstream master:
> 
>       bzr push %%bzr+ssh://bzr.savannah.gnu.org/sources/emacs/trunk/%%

Any chance to reduce this to "bzr push"?  

IIUC this sequence will do:

,----
| cd SOME-TASK/
| echo "public_branch = http://bzr.savannah.gnu.org/sources/emacs/trunk" >> .bzr/branch/config
| bzr bind http://bzr.savannah.gnu.org/sources/emacs/trunk/
| cd ..
`----

Then "bzr push" will push to the public_branch location.  Am I right?

-- 
 Bastien

Re: Switching to bzr: what Emacs developers should know?

[Óscar Fuentes]

Bastien <bastienguerry@googlemail.com> writes:

>> Presumably, you've seen this, yes?:
>> http://www.emacswiki.org/emacs/BzrForEmacsDevs
>
>> First, initialize a repository in which to store all your branches:
>> 
>>       bzr init-repo --2a emacs/
>
> (I hope we won't need this "--2a" option anytime soon.)

Why?

The 2a format speeds up some operations and will be the default format
for bzr 2.0 (to be released some weeks from now).

>> Create a branch that will just be a mirror of the mainline. You’ll
>> never make any changes to this branch; its job is just to reflect the
>> upstream master:
>> 
>>       cd emacs/
>>       bzr branch http://bzr.savannah.gnu.org/sources/emacs/trunk/ trunk/
>
> Is this "trunk" branch necessary, or just a convenience?  

A big convenience.

> Why isn't the default branch enough for the trunk-tasks described 
> in this page?

If you work with feature branches, having a local mirror saves a lot of
network traffic when creating the branches and allows branching while
off-line.

>> And after refreshing the mirror, you’ll want to get those changes into
>> your task branch, by merging them:
>> 
>>       cd SOME-TASKNAME/
>>       bzr merge
>>       bzr commit -m "Merge from mainline."
>
> Is this merge happening between the local branch and the trunk or the
> local branch and the default branch?  

By default, the merge will happen between the branch you are working on
(what you call "local branch") and your mirror branch.

>> If you’re one of the Emacs maintainers, then you can just push it
>> directly to the upstream master:
>> 
>>       bzr push %%bzr+ssh://bzr.savannah.gnu.org/sources/emacs/trunk/%%
>
> Any chance to reduce this to "bzr push"?  

bzr will remember the location you used for push, pull, etc and will use
it as the default for the next same operation.

> IIUC this sequence will do:
>
> ,----
> | cd SOME-TASK/
> | echo "public_branch = http://bzr.savannah.gnu.org/sources/emacs/trunk" >> .bzr/branch/config
> | bzr bind http://bzr.savannah.gnu.org/sources/emacs/trunk/
> | cd ..
> `----
>
> Then "bzr push" will push to the public_branch location.  Am I right?

Yes, but this is no different from what CVS does: every commit goes
straight away to the central repo.

BTW, I wouldn't recommend the echo trick quoted abobe. If in doubt, use
the --remeber option to store the location.

-- 
Óscar

Re: Switching to bzr: what Emacs developers should know?

[Bastien]

Óscar Fuentes <ofv@wanadoo.es> writes:

>> (I hope we won't need this "--2a" option anytime soon.)
>
> Why?
>
> The 2a format speeds up some operations and will be the default format
> for bzr 2.0 (to be released some weeks from now).

I meant: I hope we will be able to use the 2a format as the default
soon, without having to pass an option.

>> Why isn't the default branch enough for the trunk-tasks described 
>> in this page?
>
> If you work with feature branches, having a local mirror saves a lot of
> network traffic when creating the branches and allows branching while
> off-line.

Ok, thanks.

-- 
 Bastien

Re: Switching to bzr: what Emacs developers should know?

[CHENG Gao]

*On Sat, 8 Aug 2009 20:51:31 +0200
* Also sprach B Smith-Mannschott <bsmith.occs@gmail.com>:

> On Sat, Aug 8, 2009 at 18:24, Bastien<bastienguerry@googlemail.com> wrote:
>> Can someone describe the bzr workflow for Emacs developers?
>
> Presumably, you've seen this, yes?:
> http://www.emacswiki.org/emacs/BzrForEmacsDevs

Thank you for the info. It's great as a starter guide.

I am trying to follow it to set up using unofficial Emacs bzr repo at
http://bzr.notengoamigos.org/emacs/trunk/.

But to be honest, it's fairly painful.

On Aug. 1 I did "bzr branch", and I recorded the result as below:

,----
| $ time bzr clone http://bzr.notengoamigos.org/emacs/trunk/ emacs
| Branched 95203 revision(s).                                                                                                        
| 
| real	193m58.965s
| user	6m58.284s
| sys	1m12.954s
`----

I just did "bzr pull" and found it's fairly slow. I did not time it.

Now I am following the above mentioned BzrForEmacsDevs guide to make a
local dev branch. Very slow.

I am using Bzr 1.17 on MacOSX.


>
>> Will the switch to bzr affect the way upstream packages like Gnus
>> or Org are integrated in Emacs development?
>>
>> Would there be any advantage of switching to bzr for these packages?
>
> Well, the wiki page currently only says:
>
> [quote]
> Merging Upstream Packages (e.g., Gnus) Into Emacs With Bazaar
>
> TODO: TBD (but frankly, this should be easy, as it fits perfectly with
> the DVCS/Bazaar model anyway)
> [/quote]
>
> // ben
>
>
>

Re: Switching to bzr: what Emacs developers should know?

[Karl Fogel]

CHENG Gao <chenggao@gmail.com> writes:
> I am trying to follow it to set up using unofficial Emacs bzr repo at
> http://bzr.notengoamigos.org/emacs/trunk/.
>
> But to be honest, it's fairly painful.

Do you know how recent that branch is?  Is it in 2a format?  I've been
wondering whether Jason rolled a new one.

(I'm still pulling it...)

Re: Switching to bzr: what Emacs developers should know?

[Karl Fogel]

Jason Earl <jearl@notengoamigos.org> writes:
> Karl, I think that emacs-devel is bouncing my emails, so you might want
> to forward this.

(Whoa -- any idea why?  Are you getting bounce messages?  I think you're
right, as I don't see your mail in the list archives.)

Happy to forward when needed.  But in this case, I'll just make sure to
quote all of your mail in my followup, so all your words are here.

> I have a relatively recent 2a format version of the repository rooted at
>
> bzr://bzr.notengoamigos.org/emacs-testing
> http://bzr.notengoamigos.org/emacs-testing

Thanks.  Glad to hear it, especially as my attempt to fetch the old
repository just failed:

  $ mkdir emacs-new
  $ cd emacs-new
  $ bzr init-repo --2a .
  $ time bzr branch http://bzr.notengoamigos.org/emacs/trunk/
  bzr: ERROR: not well-formed (invalid token): line 16, column 56
  real    522m9.865s
  user    415m13.049s
  sys     1m7.948s
  $ 

That was part of a test before I responded to Cheng Gao's message of
http://lists.gnu.org/archive/html/emacs-devel/2009-08/msg00350.html, in
which he reported this:

  $ time bzr clone http://bzr.notengoamigos.org/emacs/trunk/ emacs
  Branched 95203 revision(s).                                                   

  real  193m58.965s
  user  6m58.284s
  sys   1m12.954s
  $ 

But since you have a new branch now, I'll just test with that.

> So you can get the trunk with:
>
> bzr branch bzr://bzr.notengoamigos.org/emacs-testing/trunk/ emacs
>
> I also have gzip and lzma compressed tarballs of the repository at:
>
> http://bzr.notengoamigos.org/emacs-testing.tar.gz
> http://bzr.notengoamigos.org/emacs-testing.tar.lzma

I will compare the time to 'bzr branch bzr://...' against the time to
fetch a tarball over HTTP.  However, one thing we might want to try is
offering the bzr sources via HTTP as well.  In fact, maybe you already
are?  After this current branch via bzr://, I will try it again using
http:// (note that's what Cheng Gao used -- and his use of the "clone"
command instead of "branch" makes no difference, as they are synonyms).

What kind of Net connection does bzr.notengoamigos.org have, by the way?
(I'm doing my 'bzr branch' from a server with a very wide pipe, in order
to remove my home Net connection's speed as a factor.)

> I'm sorry it took me so long to get back to you on this, but I had a bit
> of trouble with the conversion this last time around, and I have been
> under water at work.

No need to apologize -- thank you for making time for this!

Best,
-Karl

Re: Switching to bzr: what Emacs developers should know?

[Karl Fogel]

Jason Earl <jearl@notengoamigos.org> writes:
> I ran the same test as Cheng and it only to me 24m, which is more
> reasonable.  I am guessing that there might be network issues.  The
> emacs-testing branches take less time, primarily because there is a lot
> less information to move.
>
> One thing to note, the bzr: method takes longer than the http: method
> for initial checkout because the machine hosting bzr.notengoamigos.org
> is such an old piece of crap.

My results are below (this is on a machine with a pretty good pipe; I
don't remember exactly how good, but enough to be slashdotted and not
notice):

First I fetched via bzr://, like so:

   $ mkdir new-emacs
   $ cd new-emacs
   $ bzr init-repo --2a .
   Shared repository with trees (format: 2a)
   Location:
     shared repository: .
   $ time bzr branch bzr://bzr.notengoamigos.org/emacs-testing/trunk/ emacs
   Branched 96940 revision(s).
   real    47m37.010s
   user    2m33.398s
   sys     0m5.480s
   $ 

Then I fetched the tarballs:

  $ time wget http://bzr.notengoamigos.org/emacs-testing.tar.gz
  [...]
  100%[======================================>] 233,333,654 1.90M/s in 66s
  [...]
  real    1m6.515s
  user    0m0.304s
  sys     0m3.272s

  $ time wget http://bzr.notengoamigos.org/emacs-testing.tar.lzma
  [...]
  100%[======================================>] 230,333,027 3.65M/s in 77s
  [...]
  real    1m17.292s
  user    0m0.272s
  sys     0m2.836s
  $ 

Then I tried this (still in the same shared repository):

  $ time bzr branch http://bzr.notengoamigos.org/emacs-testing/trunk/ \
                    emacs-trunk-via-http
  Branched 96940 revision(s).
  real    0m21.611s
  user    0m8.117s
  sys     0m1.136s
  $ 

Good -- it used the already-fetched local revisions, as it should.  Next
I tried fetching via HTTP in a fresh (unpopulated) shared repository:

  $ cd ..
  $ mkdir emacs-new-2
  $ cd emacs-new-2
  $ bzr init-repo --2a .
  $ time bzr branch http://bzr.notengoamigos.org/emacs-testing/trunk/ \
                    emacs-trunk-via-http
  Branched 96940 revision(s).
  real    9m41.867s
  user    7m3.190s
  sys     0m5.880s
  $ 

Interesting.  I expected that to take about as long as fetching the gz
or lzma, but it took longer, though still not nearly as long as with
bzr://.  In summary, we're looking at:

  bzr branch bzr://...       ==>  47 minutes
  bzr branch http://...      ==>  10 minutes
  wget http:/.../{gz,lzma}   ==>   1 minute

> There is no question that the fastest way to get in business is to
> download the pre-made repositories and start from there.

Yup.  I was thinking we'd package up a shared repository as a .gz (or
.lzma) file.  The method would be:

  $ wget http://bzr.savanna.gnu.org/.../emacs-shared-repos.gz
  $ tar zxvf emacs-shared-repos.gz
  $ cd emacs-shared-repos            (or whatever it's called)
  $ cd trunk
  $ bzr pull                         (pull down new revs into trunk)
  $ bzr branch trunk my-branch       (start working by branching trunk)

Seem sane to you?

-Karl

Re: Switching to bzr: what Emacs developers should know?

[Karl Fogel]

Jason Earl <jearl@notengoamigos.org> writes:
> That seems remarkably similar to the advice I have on the main page at
> bzr.notengoamigos.org :).  Except I need to update it to include the
> existence of the emacs-testing repositories.

Oh, is that stuff all linked to appropriately from

  http://www.emacswiki.org/emacs-en/BzrForEmacsDevs
  http://www.emacswiki.org/emacs-en/EmacsBzrSwitchover

?  (wish I had time to check now, but don't, so posting here)

Re: Switching to bzr: what Emacs developers should know?

[CHENG Gao]

My testing result:

,----
| $ time wget http://bzr.notengoamigos.org/emacs-testing.tar.lzma
| [...]
| 2009-08-12 11:59:49 (171 KB/s) - `emacs-testing.tar.lzma' saved [230333027/230333027]
| [...]
| real	21m54.114s
| user	0m2.838s
| sys	0m14.591s
`----

Seems fairly good.

I dont know what to do then.

I uncompressed it to ~/src/emacs-testing.
There are no checkout files in trunk/.

cd trunk
bzr checkout

does not work.

cd ..
bzr branch ./trunk test

works. takes only 11.538s. I now have checkout files to use.

So I think shared repo is feasible to get users started immediately.

I cannot test how long "bzr pull" takes since the packed repo is already the latest.

Maybe Bzr switch can happen sooner.

Re: Switching to bzr: what Emacs developers should know?

[Stefan Monnier]

>   100%[======================================>] 233,333,654 1.90M/s in 66s
>   100%[======================================>] 230,333,027 3.65M/s in 77s
                                                              ^^^^^^^
LOL

>   bzr branch bzr://...       ==>  47 minutes
>   bzr branch http://...      ==>  10 minutes
>   wget http:/.../{gz,lzma}   ==>   1 minute

If you couple that with the measurement for "bzr branch" locally between
two separate repositories, maybe the Bzr guys will be interested to look
at it.  I think they already know about those performance problems, but
it's always good to remind them that they're still not solved
(especially the bulk transfer over `bzr' SHOULD be at least as fast
as over http).

>   $ wget http://bzr.savanna.gnu.org/.../emacs-shared-repos.gz
>   $ tar zxvf emacs-shared-repos.gz
>   $ cd emacs-shared-repos            (or whatever it's called)
>   $ cd trunk
>   $ bzr pull                         (pull down new revs into trunk)
>   $ bzr branch trunk my-branch       (start working by branching trunk)

> Seem sane to you?

Yes, of course.  Not just to save download time, but also because it
then automatically sets up a shared repository, which is the sanest way
to work with Bzr, in my experience.


        Stefan


PS: Seeing that there's virtually no size difference between
emacs-testing.tar.gz and emacs-testing.tar.lzma, I don't se the need to
keep them both.  And I also wonder if emacs-testing.tar wouldn't work
just as well.