Making the tarball with bzr data (was: bzr repository ready?)

Making the tarball with bzr data (was: bzr repository ready?)

[Óscar Fuentes]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> But the hard part is to integrate those 3 starting points with the
> "wget+untar" approach.

There is a very simple & safe method for creating a tarball that just
requires untarring at the other end.

First, create a bound branch of `trunk' on a shared repository [1]:

bzr init-repo emacs-repo
bzr checkout http://bzr.savannah.gnu.org/r/emacs/trunk

Now, the process of creating the tarball is:

cd emacs-repo/trunk
bzr update
cd ../..
tar the emacs-repo directory

The user just needs to download and untar to get a shared repository
containing `trunk' with read-only access to the GNU repository. A `bzr
update' is enough to get the latest changes and thus have a mirror of
the branch on the GNU repository.

If the user is an emacs hacker with write access rights, he does:

cd emacs-repo/trunk
bzr unbind
bzr bind sftp://bzr.savannah.gnu.org/r/emacs/trunk

And he is ready to start committing. [2]

If the user needs access to other branches, obtaining them with bazaar
just requires a few minutes, as only those revisions which are not
common with `trunk' will be downloaded.

If the user prefers other workflows, he has everything he needs, as he
can create branches from the mirror bound branch or unbind the mirror
branch, so this method is not restricted to the "bound work branch"
workflow.

Notes:

1. the `bzr init-repo emacs-repo' may require extra options for creating
a shared repository with the most efficient format on bazaar versions
previous to 2.0

2. something that hackers should do before the first commit is to
identify themselves with the `bzr whoami' command:

bzr whoami "Joe Hacker <joe.hacker@gnu.org>"

-- 
Óscar

Re: Making the tarball with bzr data (was: bzr repository ready?)

[Lennart Borgman]

On Tue, Nov 24, 2009 at 3:56 AM, Óscar Fuentes <ofv@wanadoo.es> wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>> But the hard part is to integrate those 3 starting points with the
>> "wget+untar" approach.
>
> There is a very simple & safe method for creating a tarball that just
> requires untarring at the other end.
>
> First, create a bound branch of `trunk' on a shared repository [1]:
>
> bzr init-repo emacs-repo
> bzr checkout http://bzr.savannah.gnu.org/r/emacs/trunk
>
> Now, the process of creating the tarball is:
>
> cd emacs-repo/trunk
> bzr update
> cd ../..
> tar the emacs-repo directory
>
> The user just needs to download and untar to get a shared repository
> containing `trunk' with read-only access to the GNU repository. A `bzr
> update' is enough to get the latest changes and thus have a mirror of
> the branch on the GNU repository.
>
> If the user is an emacs hacker with write access rights, he does:
>
> cd emacs-repo/trunk
> bzr unbind
> bzr bind sftp://bzr.savannah.gnu.org/r/emacs/trunk
>
> And he is ready to start committing. [2]
>
> If the user needs access to other branches, obtaining them with bazaar
> just requires a few minutes, as only those revisions which are not
> common with `trunk' will be downloaded.
>
> If the user prefers other workflows, he has everything he needs, as he
> can create branches from the mirror bound branch or unbind the mirror
> branch, so this method is not restricted to the "bound work branch"
> workflow.


If I already have all the Emacs files locally (possibly with some
changes) how do I do to make this a bazaar thing? (This must be the
most common situation, or?)

Re: Making the tarball with bzr data (was: bzr repository ready?)

[grischka]

> If I already have all the Emacs files locally (possibly with some
> changes) how do I do to make this a bazaar thing? (This must be the
> most common situation, or?)

In any case you need to get the official emacs bazaar repository
first.

If you have personal changes, create a branch.

Then, either:
1) copy your changed files over the ones in that branch
2) commit the resulting change as one single patch.

Or, if you have more than one change and want to have separate
commits for each, use a merge tool such as WinMerge:
1) compare your directory tree with the new development branch
2) For each feature:
    - copy the relevant chunks that affect the feature
    - commit with some message
    - repeat 2) until no differences are left.

Then you could publish your branch for others to test or improve.

HTH, grischka

Re: Making the tarball with bzr data (was: bzr repository ready?)

[Lennart Borgman]

On Mon, Nov 30, 2009 at 7:11 PM, grischka <grishka@gmx.de> wrote:
>> If I already have all the Emacs files locally (possibly with some
>> changes) how do I do to make this a bazaar thing? (This must be the
>> most common situation, or?)
>
> In any case you need to get the official emacs bazaar repository
> first.


Thanks grischka. I believe you, but this seems just crazy to me. What
are the reasons for this limitation? Are there any chances this can be
fixed?


> If you have personal changes, create a branch.
>
> Then, either:
> 1) copy your changed files over the ones in that branch
> 2) commit the resulting change as one single patch.
>
> Or, if you have more than one change and want to have separate
> commits for each, use a merge tool such as WinMerge:
> 1) compare your directory tree with the new development branch
> 2) For each feature:
>   - copy the relevant chunks that affect the feature
>   - commit with some message
>   - repeat 2) until no differences are left.
>
> Then you could publish your branch for others to test or improve.
>
> HTH, grischka
>
>

Re: Making the tarball with bzr data

[Óscar Fuentes]

Lennart Borgman <lennart.borgman@gmail.com> writes:

> On Mon, Nov 30, 2009 at 7:11 PM, grischka <grishka@gmx.de> wrote:
>>> If I already have all the Emacs files locally (possibly with some
>>> changes) how do I do to make this a bazaar thing? (This must be the
>>> most common situation, or?)
>>
>> In any case you need to get the official emacs bazaar repository
>> first.
>
>
> Thanks grischka. I believe you, but this seems just crazy to me.

Why?

> What are the reasons for this limitation?

How is that a limitation?

Let me guess. You are thinking on the CVS checkout you have with all
those patches for EmacsW32, right?

Well, a brute force solution is to setup your personal bzr Emacs branch
and then simply copy everything from your CVS checkout over the bzr
branch, or in Windows parlance, copy the contents of the folder
lennarts_CVS_checkout over lennarts_bzr_branch, overwriting
everythin. You can remove the CVS directories on the bzr branch, as they
would be useless once the switch at Savannah is complete.

It is a pity that you had no bzr at the time when EmacsW32 was started:
you could keep a clean history of the applied patches and publish the
branch on something like launchpad or from your webserver. At least, you
can start taking advantage from bzr since now: instead of having a
checkout with lots of patched files, once you copy them over the bzr
branch commit the changes:

bzr commit -m "All modifications I have so far"

Since this, for applying a new patch:

bzr commit -m "Fixes foo"

Speaking as somebody who is interested on your modifications, it would
be great if you could start from a pristine Emacs working tree and
commit one patch at a time, with a descriptive commit message, then
publish the branch :-)

I'm willing to help you (and anybody else) on the transition to bzr, but
please be more specific when describing the problems you are foreseeing.

-- 
Óscar

Re: Making the tarball with bzr data

[Lennart Borgman]

On Mon, Nov 30, 2009 at 8:35 PM, Óscar Fuentes <ofv@wanadoo.es> wrote:
> Lennart Borgman <lennart.borgman@gmail.com> writes:
>
>> On Mon, Nov 30, 2009 at 7:11 PM, grischka <grishka@gmx.de> wrote:
>>>> If I already have all the Emacs files locally (possibly with some
>>>> changes) how do I do to make this a bazaar thing? (This must be the
>>>> most common situation, or?)
>>>
>>> In any case you need to get the official emacs bazaar repository
>>> first.
>>
>>
>> Thanks grischka. I believe you, but this seems just crazy to me.
>
> Why?


Why do I have to copy the files? Why can't this setup be done with the
already existing files?


>> What are the reasons for this limitation?
>
> How is that a limitation?
>
> Let me guess. You are thinking on the CVS checkout you have with all
> those patches for EmacsW32, right?


Partly, but there is another problem. I thought I would get some help
with that since there are some helpful guys here who can teach me a
bit. Maybe I should tell you that problem...

I have put my little elisp library nXhtml at Launchpad so you can
download it with bzr. However a lot of people already have nXhtml
installed, but it was not checked out from Launchpad. It came from a
zip file or similar.

Now I believed that since bzr is a new modern system it would of
course have thought of that kind of situation. I am a bad manual
reader so I had probably just not understod how to do it... - it is of
course possible to update the files using only bzr. That was my
believe.

Maybe my believe is crazy. Maybe this should not be possible for some
reason. But I do not understand why.



> Well, a brute force solution is to setup your personal bzr Emacs branch
> and then simply copy everything from your CVS checkout over the bzr
> branch, or in Windows parlance, copy the contents of the folder
> lennarts_CVS_checkout over lennarts_bzr_branch, overwriting
> everythin. You can remove the CVS directories on the bzr branch, as they
> would be useless once the switch at Savannah is complete.


I am grateful for the explanations you and others have given for this.
I am just surprised that I have to copy the files. It certainly makes
the process slower in several ways. I have to do more. And there will
be more network traffic etc.


> It is a pity that you had no bzr at the time when EmacsW32 was started:
> you could keep a clean history of the applied patches and publish the
> branch on something like launchpad or from your webserver. At least, you
> can start taking advantage from bzr since now: instead of having a
> checkout with lots of patched files, once you copy them over the bzr
> branch commit the changes:
>
> bzr commit -m "All modifications I have so far"
>
> Since this, for applying a new patch:
>
> bzr commit -m "Fixes foo"


Thanks.


> Speaking as somebody who is interested on your modifications, it would
> be great if you could start from a pristine Emacs working tree and
> commit one patch at a time, with a descriptive commit message, then
> publish the branch :-)


I will think about it ;-)


> I'm willing to help you (and anybody else) on the transition to bzr, but
> please be more specific when describing the problems you are foreseeing.
>
> --
> Óscar

Re: Making the tarball with bzr data

[Stefan Monnier]

> Why do I have to copy the files?

Because the files are but a small part of the data that Bzr will want.

> Why can't this setup be done with the already existing files?

It can, but then you need to know which Bzr revision corresponds to the
files you have.

> Now I believed that since bzr is a new modern system it would of
> course have thought of that kind of situation. I am a bad manual
> reader so I had probably just not understod how to do it... - it is of
> course possible to update the files using only bzr. That was my
> believe.

That is the case if your zip file is a zip of the checkout Bzr tree
(probably a lightweight checkout so it isn't bloated by a complete copy
of the whole history), in which case it'll include a .bzr directory with
the necessary metadata.


        Stefan

Re: Making the tarball with bzr data

[Lennart Borgman]

On Mon, Nov 30, 2009 at 10:50 PM, Stefan Monnier
<monnier@iro.umontreal.ca> wrote:
>> Why do I have to copy the files?
>
> Because the files are but a small part of the data that Bzr will want.


Does not that depend on if I want/need the history data locally? If I
do not want to use it at all?


>> Why can't this setup be done with the already existing files?
>
> It can, but then you need to know which Bzr revision corresponds to the
> files you have.


In that case wouldn't it be possible to let bzr assume that the my
files are checked out from the current revision and then changed? (As
a separate command of course, for use in those circumstances where it
might make sense.)


>> Now I believed that since bzr is a new modern system it would of
>> course have thought of that kind of situation. I am a bad manual
>> reader so I had probably just not understod how to do it... - it is of
>> course possible to update the files using only bzr. That was my
>> believe.
>
> That is the case if your zip file is a zip of the checkout Bzr tree
> (probably a lightweight checkout so it isn't bloated by a complete copy
> of the whole history), in which case it'll include a .bzr directory with
> the necessary metadata.


Thanks.

Re: Making the tarball with bzr data

[Óscar Fuentes]

Lennart Borgman <lennart.borgman@gmail.com> writes:

> Why do I have to copy the files? Why can't this setup be done with the
> already existing files?

You can do this, but it practice, it can cause a bit of confussion if
the files you already have correspond to a different revision than the
tip of the bzr branch you download. How would bzr know that those files
are not edited but instead belongs to a past revision?

> I have put my little elisp library nXhtml at Launchpad so you can
> download it with bzr. However a lot of people already have nXhtml
> installed, but it was not checked out from Launchpad. It came from a
> zip file or similar.
>
> Now I believed that since bzr is a new modern system it would of
> course have thought of that kind of situation. I am a bad manual
> reader so I had probably just not understod how to do it... - it is of
> course possible to update the files using only bzr. That was my
> believe.

First: if all your users had the very same version of your library, you
could distribute a tarballed .bzr/ directory with the necessary
metadata. Expecting this is unrealistic, though.

Second: the amount of traffic you save by re-using the source code on
the target machine is nil: when the user downloads the metadata, he gets
a compressed history of the source code. You could use a lightweight
checkout, and this would be a very short download indeed, but then the
user would be very constrained on their Bazaar usage, as almost all
interesting features of Bazaar are not usable when everything you have
at your disposal is a lightweight branch, apart from risking on the
problem mentioned above.

Third: if you are thinking about Bazaar as a system for distributing
your software, I think that pointing potential users to a bzr branch for
obtaining a copy of your library you are raising the entry barrier quite
a bit, unless your intended audience already knows Bazaar.

> Maybe my believe is crazy. Maybe this should not be possible for some
> reason. But I do not understand why.

It is possible. It is not worthwhile.

[snip]

-- 
Óscar

Re: Making the tarball with bzr data

[Stephen J. Turnbull]

Lennart Borgman writes:

 > Why do I have to copy the files? Why can't this setup be done with
 > the already existing files?

Basically, because Bazaar is not git.  In git, each object is
summarized with an SHA1 hash, and so you can verify that you have a
clean tree (with very high probability) without actually comparing the
file content.  Bazaar uses a different, history-dependent method to
track files, assigning a globally unique, *permanent* ID to each
file.  So in order to verify that you have the right files for the
version Bazaar believes to be there, Bazaar must diff every single
file.  If you think about it, you'll see any other behavior would make
Bazaar a very unreliable tool.

There no savings in network transport (Bazaar can't send only the
content of the version you check out), and no savings in local file IO
(in the best case, every file differs in the first byte so you check
them all out anyway), by reusing existing files.

N.B. Reuse of an existing tree is a theoretical possibility only in
git.  AFAIK it has not been implemented.

In Bazaar, I suspect you can do the following:

1.  bzr branch --treeless <Emacs trunk URL> tmp
2.  mv tmp/.bzr <existing Emacs tree top directory>

But this doesn't save any network transport (you still get all the
branch data), and not much time locally since bzr will need to diff
all the files to determine which ones to commit (it will believe they
have *all* changed based on timestamp).

Probably the best procedure is

1.  bzr branch <Emacs trunk URL> bzr-emacs
2.  rsync <options> old-emacs bzr-emacs

assuming that rsync has appropriate options to ensure that it won't
change the stat data (specifically size and timestamp, IIRC) that bzr
uses to check if a file has been changed.

 > I have put my little elisp library nXhtml at Launchpad so you can
 > download it with bzr. However a lot of people already have nXhtml
 > installed, but it was not checked out from Launchpad. It came from a
 > zip file or similar.
 > 
 > Now I believed that since bzr is a new modern system it would of
 > course have thought of that kind of situation.

No, because Bazaar is a new modern system it assumes you have a new
modern network connection with which to download the Bazaar metadata
and revision history reasonably quickly.  This is a one-time cost per
project you work on if you use the workflow recommended in
BzrForEmacsDevs.

 > Maybe my believe is crazy. Maybe this should not be possible for some
 > reason. But I do not understand why.

It is possible -- with git.  Emacs isn't using git.  And there are
things that Bazaar can do (eg, tracking renames efficiently) that git
cannot.  Unfortunately, tracking renames depends exactly on the
feature I described above (tracking files in bzr vs. tracking content
in git).  (In theory you could do both and get the best of both
worlds, but this hasn't been implemented in any VCS AFAIK.)

Re: Making the tarball with bzr data

[Óscar Fuentes]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

[snip]

> Probably the best procedure is
>
> 1.  bzr branch <Emacs trunk URL> bzr-emacs
> 2.  rsync <options> old-emacs bzr-emacs
>
> assuming that rsync has appropriate options to ensure that it won't
> change the stat data (specifically size and timestamp, IIRC) that bzr
> uses to check if a file has been changed.

There is nothing wrong with changing the timestamp of a file versioned
under Bazaar. It will be not flagged as modified on `bzr status'.

OTOH, a change on size indicates a change on content, and this will be
flagged as an edited file (modulo eol settings, which bazaar supports as
system-wide options).

Lennart: if you copy files on Windows from your CVS working tree to your
bzr working tree maybe you will see files flagged as modified due to
differences on eol.

[snip]

-- 
Óscar

Re: Making the tarball with bzr data

[Stefan Monnier]

>>> Why can't this setup be done with the already existing files?
>> It can, but then you need to know which Bzr revision corresponds to the
>> files you have.
> In that case wouldn't it be possible to let bzr assume that the my
> files are checked out from the current revision and then changed? (As
> a separate command of course, for use in those circumstances where it
> might make sense.)

Yes, it'd be possible.  I know of no VCS that allows you to do that
(including CVS, which has had moe yeqars than any other to provide such
a feature), so I think it's just much too low on the list of priorities.


        Stefan

Re: Making the tarball with bzr data

[Giorgos Keramidas]

On Mon, 30 Nov 2009 22:27:00 +0100, Lennart Borgman <lennart.borgman@gmail.com> wrote:
> On Mon, Nov 30, 2009 at 8:35 PM, Óscar Fuentes <ofv@wanadoo.es> wrote:
>> Well, a brute force solution is to setup your personal bzr Emacs branch
>> and then simply copy everything from your CVS checkout over the bzr
>> branch, or in Windows parlance, copy the contents of the folder
>> lennarts_CVS_checkout over lennarts_bzr_branch, overwriting
>> everythin. You can remove the CVS directories on the bzr branch, as they
>> would be useless once the switch at Savannah is complete.
>
> I am grateful for the explanations you and others have given for this.
> I am just surprised that I have to copy the files. It certainly makes
> the process slower in several ways. I have to do more. And there will
> be more network traffic etc.

Hi Lennard,

You will have to copy a bzr-based trunk, but this is something you will
only have to do the first time.  The cost of subsequent merges with the
Emacs upstream is significantly lower than the first copy.

Once you go through the initial 'pain' of having to branch from the
Emacs trunk, there are a few things to gain too:

  * You get to merge with the Emacs upstream branch as many times as you
    want.  All the merges are shown explicitly in the history of your
    local branch, so it is easier to reason about questions like "When
    was the last time my merge worked fine?" or to move between two
    different merge points, even if one of them is arbitrarily old.

  * Your w32 branch will be visible to others too, including its full
    merge history with the Emacs upstream.  You can publish your w32
    branch either on Launchpad or on your own web server, and other
    people can look at the changes, submit their own extra changes, and
    so on.

  * Once you commit your local changes to a bzr branch, it is much safer
    to experiment with *more* changes, since the already committed
    patches will not go away even if you wipe all the files in the
    working copy.  If you accidentally modify a file and break your
    local w32 copy in some annoying way, you can just check-out a fresh
    copy from bzr.