Re: Making the tarball with bzr data (was: bzr repository ready?)

Re: Making the tarball with bzr data (was: bzr repository ready?)

[grischka]

> If I already have all the Emacs files locally (possibly with some
> changes) how do I do to make this a bazaar thing? (This must be the
> most common situation, or?)

In any case you need to get the official emacs bazaar repository
first.

If you have personal changes, create a branch.

Then, either:
1) copy your changed files over the ones in that branch
2) commit the resulting change as one single patch.

Or, if you have more than one change and want to have separate
commits for each, use a merge tool such as WinMerge:
1) compare your directory tree with the new development branch
2) For each feature:
    - copy the relevant chunks that affect the feature
    - commit with some message
    - repeat 2) until no differences are left.

Then you could publish your branch for others to test or improve.

HTH, grischka

Re: Making the tarball with bzr data (was: bzr repository ready?)

[Lennart Borgman]

On Mon, Nov 30, 2009 at 7:11 PM, grischka <grishka@gmx.de> wrote:
>> If I already have all the Emacs files locally (possibly with some
>> changes) how do I do to make this a bazaar thing? (This must be the
>> most common situation, or?)
>
> In any case you need to get the official emacs bazaar repository
> first.


Thanks grischka. I believe you, but this seems just crazy to me. What
are the reasons for this limitation? Are there any chances this can be
fixed?


> If you have personal changes, create a branch.
>
> Then, either:
> 1) copy your changed files over the ones in that branch
> 2) commit the resulting change as one single patch.
>
> Or, if you have more than one change and want to have separate
> commits for each, use a merge tool such as WinMerge:
> 1) compare your directory tree with the new development branch
> 2) For each feature:
>   - copy the relevant chunks that affect the feature
>   - commit with some message
>   - repeat 2) until no differences are left.
>
> Then you could publish your branch for others to test or improve.
>
> HTH, grischka
>
>

Re: Making the tarball with bzr data

[Óscar Fuentes]

Lennart Borgman <lennart.borgman@gmail.com> writes:

>> There is a very simple & safe method for creating a tarball that just
>> requires untarring at the other end.

[snip]

> If I already have all the Emacs files locally (possibly with some
> changes) how do I do to make this a bazaar thing? (This must be the
> most common situation, or?)

The most simple solution is to create a working bzr setup as explained
on the wiki and then copy the modified files you have on your CVS
checkout over the corresponding files on the work branch of the bzr
setup.

I'm talking here assuming that people will use the workflow documented
by Karl and Stephen.

BTW, one advantage of the workflow they recommend over the one I
documented and CVS (which are equivalent on this specific aspect) is
that, when you end your working session, you can commit your changes
locally with a comment about its state and what remains to be
done. Later, when you come back, read what you put on the last commit
and keep hacking. When the work is complete, you push to upstream (with
or without the history of your private commits).

Another advantage of their workflow is that you don't need to mix the
implementation of distint features on the same working source tree. You
can easily create your own personal local branches, one for each task
you are working on.

-- 
Óscar

Re: Making the tarball with bzr data

[Jason Earl]

Lennart Borgman <lennart.borgman@gmail.com> writes:

> On Tue, Nov 24, 2009 at 3:56 AM, Óscar Fuentes <ofv@wanadoo.es> wrote:
>> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>
>>> But the hard part is to integrate those 3 starting points with the
>>> "wget+untar" approach.
>>
>> There is a very simple & safe method for creating a tarball that just
>> requires untarring at the other end.
>>
>> First, create a bound branch of `trunk' on a shared repository [1]:
>>
>> bzr init-repo emacs-repo
>> bzr checkout http://bzr.savannah.gnu.org/r/emacs/trunk
>>
>> Now, the process of creating the tarball is:
>>
>> cd emacs-repo/trunk
>> bzr update
>> cd ../..
>> tar the emacs-repo directory
>>
>> The user just needs to download and untar to get a shared repository
>> containing `trunk' with read-only access to the GNU repository. A
>> `bzr update' is enough to get the latest changes and thus have a
>> mirror of the branch on the GNU repository.
>>
>> If the user is an emacs hacker with write access rights, he does:
>>
>> cd emacs-repo/trunk
>> bzr unbind
>> bzr bind sftp://bzr.savannah.gnu.org/r/emacs/trunk
>>
>> And he is ready to start committing. [2]
>>
>> If the user needs access to other branches, obtaining them with
>> bazaar just requires a few minutes, as only those revisions which are
>> not common with `trunk' will be downloaded.
>>
>> If the user prefers other workflows, he has everything he needs, as
>> he can create branches from the mirror bound branch or unbind the
>> mirror branch, so this method is not restricted to the "bound work
>> branch" workflow.
>
>
> If I already have all the Emacs files locally (possibly with some
> changes) how do I do to make this a bazaar thing? (This must be the
> most common situation, or?)

If you are primarily interested in tracking your own changes, then
simply doing a:

bzr init

in the root of the directory will get you started.  However, this makes
a brand new branch that is completely unrelated to the mainline Emacs
branch.  Merging your branch with the main Emacs branch would be far
more difficult than it needs to be, and so would getting new
improvements from the mainline Emacs branch.

So, that's probably not what you want to do.

What you probably want to do is to migrate the changes you have made to
Emacs into a new branch that can be used both to track changes made
upstream to Emacs and (hopefully) to allow you to develop new changes
that can be applied upstream.

The way to do that is to follow the instructions at:

http://www.emacswiki.org/emacs/BzrForEmacsDevs

to create a repository along with a "trunk" branch that tracks Emacs'
mainline development.  Then create a second branch (called, for example,
lennart).  Finally, use a tool like rsync to copy your changes into the
lennart branch.  You can then do:

bzr status

to see what files have changed and:

bzr diff

to generate a diff of what has changed.  Once you are satisfied that
your changes are the working tree for your shiny new branch.

bzr commit -m "Importing my changes into a new bzr branch"

will commit your changes to your local branch.

Hopefully this was helpful.

Jason

Re: Making the tarball with bzr data

[Óscar Fuentes]

Lennart Borgman <lennart.borgman@gmail.com> writes:

> On Mon, Nov 30, 2009 at 7:11 PM, grischka <grishka@gmx.de> wrote:
>>> If I already have all the Emacs files locally (possibly with some
>>> changes) how do I do to make this a bazaar thing? (This must be the
>>> most common situation, or?)
>>
>> In any case you need to get the official emacs bazaar repository
>> first.
>
>
> Thanks grischka. I believe you, but this seems just crazy to me.

Why?

> What are the reasons for this limitation?

How is that a limitation?

Let me guess. You are thinking on the CVS checkout you have with all
those patches for EmacsW32, right?

Well, a brute force solution is to setup your personal bzr Emacs branch
and then simply copy everything from your CVS checkout over the bzr
branch, or in Windows parlance, copy the contents of the folder
lennarts_CVS_checkout over lennarts_bzr_branch, overwriting
everythin. You can remove the CVS directories on the bzr branch, as they
would be useless once the switch at Savannah is complete.

It is a pity that you had no bzr at the time when EmacsW32 was started:
you could keep a clean history of the applied patches and publish the
branch on something like launchpad or from your webserver. At least, you
can start taking advantage from bzr since now: instead of having a
checkout with lots of patched files, once you copy them over the bzr
branch commit the changes:

bzr commit -m "All modifications I have so far"

Since this, for applying a new patch:

bzr commit -m "Fixes foo"

Speaking as somebody who is interested on your modifications, it would
be great if you could start from a pristine Emacs working tree and
commit one patch at a time, with a descriptive commit message, then
publish the branch :-)

I'm willing to help you (and anybody else) on the transition to bzr, but
please be more specific when describing the problems you are foreseeing.

-- 
Óscar

Re: Making the tarball with bzr data

[Lennart Borgman]

On Mon, Nov 30, 2009 at 8:35 PM, Óscar Fuentes <ofv@wanadoo.es> wrote:
> Lennart Borgman <lennart.borgman@gmail.com> writes:
>
>> On Mon, Nov 30, 2009 at 7:11 PM, grischka <grishka@gmx.de> wrote:
>>>> If I already have all the Emacs files locally (possibly with some
>>>> changes) how do I do to make this a bazaar thing? (This must be the
>>>> most common situation, or?)
>>>
>>> In any case you need to get the official emacs bazaar repository
>>> first.
>>
>>
>> Thanks grischka. I believe you, but this seems just crazy to me.
>
> Why?


Why do I have to copy the files? Why can't this setup be done with the
already existing files?


>> What are the reasons for this limitation?
>
> How is that a limitation?
>
> Let me guess. You are thinking on the CVS checkout you have with all
> those patches for EmacsW32, right?


Partly, but there is another problem. I thought I would get some help
with that since there are some helpful guys here who can teach me a
bit. Maybe I should tell you that problem...

I have put my little elisp library nXhtml at Launchpad so you can
download it with bzr. However a lot of people already have nXhtml
installed, but it was not checked out from Launchpad. It came from a
zip file or similar.

Now I believed that since bzr is a new modern system it would of
course have thought of that kind of situation. I am a bad manual
reader so I had probably just not understod how to do it... - it is of
course possible to update the files using only bzr. That was my
believe.

Maybe my believe is crazy. Maybe this should not be possible for some
reason. But I do not understand why.



> Well, a brute force solution is to setup your personal bzr Emacs branch
> and then simply copy everything from your CVS checkout over the bzr
> branch, or in Windows parlance, copy the contents of the folder
> lennarts_CVS_checkout over lennarts_bzr_branch, overwriting
> everythin. You can remove the CVS directories on the bzr branch, as they
> would be useless once the switch at Savannah is complete.


I am grateful for the explanations you and others have given for this.
I am just surprised that I have to copy the files. It certainly makes
the process slower in several ways. I have to do more. And there will
be more network traffic etc.


> It is a pity that you had no bzr at the time when EmacsW32 was started:
> you could keep a clean history of the applied patches and publish the
> branch on something like launchpad or from your webserver. At least, you
> can start taking advantage from bzr since now: instead of having a
> checkout with lots of patched files, once you copy them over the bzr
> branch commit the changes:
>
> bzr commit -m "All modifications I have so far"
>
> Since this, for applying a new patch:
>
> bzr commit -m "Fixes foo"


Thanks.


> Speaking as somebody who is interested on your modifications, it would
> be great if you could start from a pristine Emacs working tree and
> commit one patch at a time, with a descriptive commit message, then
> publish the branch :-)


I will think about it ;-)


> I'm willing to help you (and anybody else) on the transition to bzr, but
> please be more specific when describing the problems you are foreseeing.
>
> --
> Óscar

Re: Making the tarball with bzr data

[Stefan Monnier]

> Why do I have to copy the files?

Because the files are but a small part of the data that Bzr will want.

> Why can't this setup be done with the already existing files?

It can, but then you need to know which Bzr revision corresponds to the
files you have.

> Now I believed that since bzr is a new modern system it would of
> course have thought of that kind of situation. I am a bad manual
> reader so I had probably just not understod how to do it... - it is of
> course possible to update the files using only bzr. That was my
> believe.

That is the case if your zip file is a zip of the checkout Bzr tree
(probably a lightweight checkout so it isn't bloated by a complete copy
of the whole history), in which case it'll include a .bzr directory with
the necessary metadata.


        Stefan

Re: Making the tarball with bzr data

[Lennart Borgman]

On Mon, Nov 30, 2009 at 10:50 PM, Stefan Monnier
<monnier@iro.umontreal.ca> wrote:
>> Why do I have to copy the files?
>
> Because the files are but a small part of the data that Bzr will want.


Does not that depend on if I want/need the history data locally? If I
do not want to use it at all?


>> Why can't this setup be done with the already existing files?
>
> It can, but then you need to know which Bzr revision corresponds to the
> files you have.


In that case wouldn't it be possible to let bzr assume that the my
files are checked out from the current revision and then changed? (As
a separate command of course, for use in those circumstances where it
might make sense.)


>> Now I believed that since bzr is a new modern system it would of
>> course have thought of that kind of situation. I am a bad manual
>> reader so I had probably just not understod how to do it... - it is of
>> course possible to update the files using only bzr. That was my
>> believe.
>
> That is the case if your zip file is a zip of the checkout Bzr tree
> (probably a lightweight checkout so it isn't bloated by a complete copy
> of the whole history), in which case it'll include a .bzr directory with
> the necessary metadata.


Thanks.

Re: Making the tarball with bzr data

[Óscar Fuentes]

Lennart Borgman <lennart.borgman@gmail.com> writes:

> Why do I have to copy the files? Why can't this setup be done with the
> already existing files?

You can do this, but it practice, it can cause a bit of confussion if
the files you already have correspond to a different revision than the
tip of the bzr branch you download. How would bzr know that those files
are not edited but instead belongs to a past revision?

> I have put my little elisp library nXhtml at Launchpad so you can
> download it with bzr. However a lot of people already have nXhtml
> installed, but it was not checked out from Launchpad. It came from a
> zip file or similar.
>
> Now I believed that since bzr is a new modern system it would of
> course have thought of that kind of situation. I am a bad manual
> reader so I had probably just not understod how to do it... - it is of
> course possible to update the files using only bzr. That was my
> believe.

First: if all your users had the very same version of your library, you
could distribute a tarballed .bzr/ directory with the necessary
metadata. Expecting this is unrealistic, though.

Second: the amount of traffic you save by re-using the source code on
the target machine is nil: when the user downloads the metadata, he gets
a compressed history of the source code. You could use a lightweight
checkout, and this would be a very short download indeed, but then the
user would be very constrained on their Bazaar usage, as almost all
interesting features of Bazaar are not usable when everything you have
at your disposal is a lightweight branch, apart from risking on the
problem mentioned above.

Third: if you are thinking about Bazaar as a system for distributing
your software, I think that pointing potential users to a bzr branch for
obtaining a copy of your library you are raising the entry barrier quite
a bit, unless your intended audience already knows Bazaar.

> Maybe my believe is crazy. Maybe this should not be possible for some
> reason. But I do not understand why.

It is possible. It is not worthwhile.

[snip]

-- 
Óscar

Re: Making the tarball with bzr data

[Stephen J. Turnbull]

Lennart Borgman writes:

 > Why do I have to copy the files? Why can't this setup be done with
 > the already existing files?

Basically, because Bazaar is not git.  In git, each object is
summarized with an SHA1 hash, and so you can verify that you have a
clean tree (with very high probability) without actually comparing the
file content.  Bazaar uses a different, history-dependent method to
track files, assigning a globally unique, *permanent* ID to each
file.  So in order to verify that you have the right files for the
version Bazaar believes to be there, Bazaar must diff every single
file.  If you think about it, you'll see any other behavior would make
Bazaar a very unreliable tool.

There no savings in network transport (Bazaar can't send only the
content of the version you check out), and no savings in local file IO
(in the best case, every file differs in the first byte so you check
them all out anyway), by reusing existing files.

N.B. Reuse of an existing tree is a theoretical possibility only in
git.  AFAIK it has not been implemented.

In Bazaar, I suspect you can do the following:

1.  bzr branch --treeless <Emacs trunk URL> tmp
2.  mv tmp/.bzr <existing Emacs tree top directory>

But this doesn't save any network transport (you still get all the
branch data), and not much time locally since bzr will need to diff
all the files to determine which ones to commit (it will believe they
have *all* changed based on timestamp).

Probably the best procedure is

1.  bzr branch <Emacs trunk URL> bzr-emacs
2.  rsync <options> old-emacs bzr-emacs

assuming that rsync has appropriate options to ensure that it won't
change the stat data (specifically size and timestamp, IIRC) that bzr
uses to check if a file has been changed.

 > I have put my little elisp library nXhtml at Launchpad so you can
 > download it with bzr. However a lot of people already have nXhtml
 > installed, but it was not checked out from Launchpad. It came from a
 > zip file or similar.
 > 
 > Now I believed that since bzr is a new modern system it would of
 > course have thought of that kind of situation.

No, because Bazaar is a new modern system it assumes you have a new
modern network connection with which to download the Bazaar metadata
and revision history reasonably quickly.  This is a one-time cost per
project you work on if you use the workflow recommended in
BzrForEmacsDevs.

 > Maybe my believe is crazy. Maybe this should not be possible for some
 > reason. But I do not understand why.

It is possible -- with git.  Emacs isn't using git.  And there are
things that Bazaar can do (eg, tracking renames efficiently) that git
cannot.  Unfortunately, tracking renames depends exactly on the
feature I described above (tracking files in bzr vs. tracking content
in git).  (In theory you could do both and get the best of both
worlds, but this hasn't been implemented in any VCS AFAIK.)

Re: Making the tarball with bzr data

[Óscar Fuentes]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

[snip]

> Probably the best procedure is
>
> 1.  bzr branch <Emacs trunk URL> bzr-emacs
> 2.  rsync <options> old-emacs bzr-emacs
>
> assuming that rsync has appropriate options to ensure that it won't
> change the stat data (specifically size and timestamp, IIRC) that bzr
> uses to check if a file has been changed.

There is nothing wrong with changing the timestamp of a file versioned
under Bazaar. It will be not flagged as modified on `bzr status'.

OTOH, a change on size indicates a change on content, and this will be
flagged as an edited file (modulo eol settings, which bazaar supports as
system-wide options).

Lennart: if you copy files on Windows from your CVS working tree to your
bzr working tree maybe you will see files flagged as modified due to
differences on eol.

[snip]

-- 
Óscar

Re: Making the tarball with bzr data

[Stefan Monnier]

>>> Why can't this setup be done with the already existing files?
>> It can, but then you need to know which Bzr revision corresponds to the
>> files you have.
> In that case wouldn't it be possible to let bzr assume that the my
> files are checked out from the current revision and then changed? (As
> a separate command of course, for use in those circumstances where it
> might make sense.)

Yes, it'd be possible.  I know of no VCS that allows you to do that
(including CVS, which has had moe yeqars than any other to provide such
a feature), so I think it's just much too low on the list of priorities.


        Stefan

Re: Making the tarball with bzr data

[Giorgos Keramidas]

On Mon, 30 Nov 2009 22:27:00 +0100, Lennart Borgman <lennart.borgman@gmail.com> wrote:
> On Mon, Nov 30, 2009 at 8:35 PM, Óscar Fuentes <ofv@wanadoo.es> wrote:
>> Well, a brute force solution is to setup your personal bzr Emacs branch
>> and then simply copy everything from your CVS checkout over the bzr
>> branch, or in Windows parlance, copy the contents of the folder
>> lennarts_CVS_checkout over lennarts_bzr_branch, overwriting
>> everythin. You can remove the CVS directories on the bzr branch, as they
>> would be useless once the switch at Savannah is complete.
>
> I am grateful for the explanations you and others have given for this.
> I am just surprised that I have to copy the files. It certainly makes
> the process slower in several ways. I have to do more. And there will
> be more network traffic etc.

Hi Lennard,

You will have to copy a bzr-based trunk, but this is something you will
only have to do the first time.  The cost of subsequent merges with the
Emacs upstream is significantly lower than the first copy.

Once you go through the initial 'pain' of having to branch from the
Emacs trunk, there are a few things to gain too:

  * You get to merge with the Emacs upstream branch as many times as you
    want.  All the merges are shown explicitly in the history of your
    local branch, so it is easier to reason about questions like "When
    was the last time my merge worked fine?" or to move between two
    different merge points, even if one of them is arbitrarily old.

  * Your w32 branch will be visible to others too, including its full
    merge history with the Emacs upstream.  You can publish your w32
    branch either on Launchpad or on your own web server, and other
    people can look at the changes, submit their own extra changes, and
    so on.

  * Once you commit your local changes to a bzr branch, it is much safer
    to experiment with *more* changes, since the already committed
    patches will not go away even if you wipe all the files in the
    working copy.  If you accidentally modify a file and break your
    local w32 copy in some annoying way, you can just check-out a fresh
    copy from bzr.