BZR and the bootstrap

BZR and the bootstrap

[Angelo Graziosi]

Usually, I bootstrap Emacs [trunk] doing, in short,

cd emacs-trunk
bzr up
mkdir build
cd build
../configure...
make bootstrap

etc.

Now I have noticed, when my internet connection is on (1), that BZR is 
called during the bootstrap. This is flagged both by 'top' and the modem 
lights.

Why BZR is called? What is it doing?

What happens if I switch off the connection (2)?

It seems that if I strip the bzr tree by '.*' file (3),

$ find emacs-trunk -name ".*" -delete

and the connection is on, BZR isn't called at all!

So, which is the right way to bootstrap: (1), (2) or (3)?

Ciao,
Angelo.

Re: BZR and the bootstrap

[Eli Zaretskii]

> Date: Sat, 07 Aug 2010 18:49:54 +0200
> From: Angelo Graziosi <angelo.graziosi@alice.it>
> 
> Now I have noticed, when my internet connection is on (1), that BZR is 
> called during the bootstrap. This is flagged both by 'top' and the modem 
> lights.
> 
> Why BZR is called? What is it doing?

Probably because Emacs calls VC every time it needs to visit a file
that is in a versioned directory.

> What happens if I switch off the connection (2)?

Nothing bad should happen, because if I'm right, Emacs invokes "bzr
status" which is done entirely locally.

> It seems that if I strip the bzr tree by '.*' file (3),
> 
> $ find emacs-trunk -name ".*" -delete
> 
> and the connection is on, BZR isn't called at all!

Right, because then it's no longer a versioned directory, as far as
Emacs is concerned.

> So, which is the right way to bootstrap: (1), (2) or (3)?

All of the above, I think.

Re: BZR and the bootstrap

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> Now I have noticed, when my internet connection is on (1), that BZR is 
>> called during the bootstrap. This is flagged both by 'top' and the modem 
>> lights.
>> 
>> Why BZR is called? What is it doing?
>
> Probably because Emacs calls VC every time it needs to visit a file
> that is in a versioned directory.
>
>> What happens if I switch off the connection (2)?
>
> Nothing bad should happen, because if I'm right, Emacs invokes "bzr
> status" which is done entirely locally.

He says that the modem lights blink, indicating network activity. So it
seems that the OP is using a lightweight checkout. If that is true, I
wonder what info bzr needs from upstream. In any case, having bzr
phoning home for every visited file would be irritatingly slow and he
would already notice that. So what the OP reports is very strange.

[snip]

Re: BZR and the bootstrap

[Stephen J. Turnbull]

Óscar Fuentes writes:

 > > Nothing bad should happen, because if I'm right, Emacs invokes
 > > "bzr status" which is done entirely locally.

 > He says that the modem lights blink, indicating network
 > activity. So it seems that the OP is using a lightweight
 > checkout. If that is true, I wonder what info bzr needs from
 > upstream.

Everything.  From the help message:

Usage:   bzr checkout [BRANCH_LOCATION] [TO_LOCATION]

Options:
  --lightweight         Perform a lightweight checkout.  Lightweight checkouts
                        depend on access to the branch for every operation.

It's penny-wise and pound-foolish for anybody who is unfamiliar with
Bazaar to use lightweight checkouts.  Occasional contributors and beta
testers should invest in a full branch, bound or not depending on
their use cases.

Re: BZR and the bootstrap

[Angelo Graziosi]

Il 09/08/2010 8.57, Stephen J. Turnbull ha scritto:
> Óscar Fuentes writes:
>
>   >  >  Nothing bad should happen, because if I'm right, Emacs invokes
>   >  >  "bzr status" which is done entirely locally.
>
>   >  He says that the modem lights blink, indicating network
>   >  activity.

Right!

>   >  So it seems that the OP is using a lightweight
>   >  checkout.

Wow! Oscar, you have a crystal ball!! Really I did

$ bzr checkout --lightweight http://bzr.savannah.gnu.org/r/emacs/trunk 
emacs-trunk

the first time I used BZR!

>   >  If that is true, I wonder what info bzr needs from
>   >  upstream.

Indeed...

>
> Everything.  From the help message:
>
> Usage:   bzr checkout [BRANCH_LOCATION] [TO_LOCATION]
>
> Options:
>    --lightweight         Perform a lightweight checkout.  Lightweight checkouts
>                          depend on access to the branch for every operation.
>
> It's penny-wise and pound-foolish for anybody who is unfamiliar with
> Bazaar to use lightweight checkouts.  Occasional contributors and beta
> testers should invest in a full branch, bound or not depending on
> their use cases.

So, what is it the best way to create the development source tree of Emacs?

Is, perhaps, the following?

$ bzr checkout http://bzr.savannah.gnu.org/r/emacs/trunk emacs-trunk

I wonder if there is a way that uses 'rsync', instead, like TeXLive[*]...

Ciao,
Angelo.

---
[*] http://www.tug.org/texlive/svn :

"You can retrieve the entire development source tree via rsync:
rsync -a --delete --exclude=.svn tug.org::tldevsrc ."

Re: BZR and the bootstrap

[Angelo Graziosi]

Ok, I have done some experiment.

$ bzr checkout http://bzr.savannah.gnu.org/r/emacs/trunk emacs-trunk

$ bzr checkout --lightweight http://bzr.savannah.gnu.org/r/emacs/trunk 
emacs-light

$ rsync -av --exclude=.* --delete emacs-trunk/ emacs-stripped-01/

$ rsync -av --exclude=.* --delete emacs-light/ emacs-stripped-02/

I get

$ du -s emacs-trunk emacs-trunk-light emacs-stripped-01 emacs-stripped-02

337M    emacs-trunk
113M    emacs-trunk-light
112M    emacs-stripped-01
112M    emacs-stripped-02

So, which is the best tree for bootstrap?

BTW, 'emacs-trunk' has a '.bzr' file of 226 M!

Ciao,
Angelo.


Il 09/08/2010 10.57, Angelo Graziosi ha scritto:
> Il 09/08/2010 8.57, Stephen J. Turnbull ha scritto:
>> Óscar Fuentes writes:
>>
>> > > Nothing bad should happen, because if I'm right, Emacs invokes
>> > > "bzr status" which is done entirely locally.
>>
>> > He says that the modem lights blink, indicating network
>> > activity.
>
> Right!
>
>> > So it seems that the OP is using a lightweight
>> > checkout.
>
> Wow! Oscar, you have a crystal ball!! Really I did
>
> $ bzr checkout --lightweight http://bzr.savannah.gnu.org/r/emacs/trunk
> emacs-trunk
>
> the first time I used BZR!
>
>> > If that is true, I wonder what info bzr needs from
>> > upstream.
>
> Indeed...
>
>>
>> Everything. From the help message:
>>
>> Usage: bzr checkout [BRANCH_LOCATION] [TO_LOCATION]
>>
>> Options:
>> --lightweight Perform a lightweight checkout. Lightweight checkouts
>> depend on access to the branch for every operation.
>>
>> It's penny-wise and pound-foolish for anybody who is unfamiliar with
>> Bazaar to use lightweight checkouts. Occasional contributors and beta
>> testers should invest in a full branch, bound or not depending on
>> their use cases.
>
> So, what is it the best way to create the development source tree of Emacs?
>
> Is, perhaps, the following?
>
> $ bzr checkout http://bzr.savannah.gnu.org/r/emacs/trunk emacs-trunk
>
> I wonder if there is a way that uses 'rsync', instead, like TeXLive[*]...
>
> Ciao,
> Angelo.
>
> ---
> [*] http://www.tug.org/texlive/svn :
>
> "You can retrieve the entire development source tree via rsync:
> rsync -a --delete --exclude=.svn tug.org::tldevsrc ."

Re: BZR and the bootstrap

[Óscar Fuentes]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

>  > He says that the modem lights blink, indicating network
>  > activity. So it seems that the OP is using a lightweight
>  > checkout. If that is true, I wonder what info bzr needs from
>  > upstream.
>
> Everything.  From the help message:
>
> Usage:   bzr checkout [BRANCH_LOCATION] [TO_LOCATION]
>
> Options:
>   --lightweight         Perform a lightweight checkout.  Lightweight checkouts
>                         depend on access to the branch for every operation.

IIRC, lightweight checkouts contain information for testing if the file
is modified or missing from the file system, which provides a local `bzr
status' operation. For anything else it needs to contact upstream. It is
very similar to CVS on that aspect.

[snip]

Re: BZR and the bootstrap

[Óscar Fuentes]

Angelo Graziosi <angelo.graziosi@alice.it> writes:

[snip]

> So, which is the best tree for bootstrap?
>
> BTW, 'emacs-trunk' has a '.bzr' file of 226 M!

Can you afford those 226 MB? If yes, go with the normal checkout.

Re: BZR and the bootstrap

[Stephen J. Turnbull]

Óscar Fuentes writes:
 > Angelo Graziosi <angelo.graziosi@alice.it> writes:
 > 
 > [snip]
 > 
 > > So, which is the best tree for bootstrap?
 > >
 > > BTW, 'emacs-trunk' has a '.bzr' file of 226 M!

That's not bad, considering you can produce nearly 100,000 different
versions of Emacs from that with "bzr revert -r NNN" for NNN a
positive integer.  (It's a little more complicated than that, actually
but not much.)

 > Can you afford those 226 MB? If yes, go with the normal checkout.

+1.

You really do not want to do rsync against an active tree; there are
no consistency guarantees with rsync.  (Rsync will deliver files
correctly, I believe, but there's no guarantee that will correspond to
a version of Emacs that ever existed in any developer's workspace.)

The normal checkout is, indeed, overkill if you only ever intend to
update to latest.  However, you will often see posts saying that the
bootstrap is broken since revno 89444 or something like that, and you
can get back to something sane, *without* needing to talk to Savannah
or Launchpad, with a simple "bzr revert -r 89443".  Also, if you ever
feel the need to make a change (and I hope you will!), it's much
easier to manage those changes if you have a full local branch.

Re: BZR and the bootstrap

[Stephen J. Turnbull]

Óscar Fuentes writes:

 > IIRC, lightweight checkouts contain information for testing if the
 > file is modified or missing from the file system, which provides a
 > local `bzr status' operation. For anything else it needs to contact
 > upstream.

That may be true, but the rest of that help message strongly implies
that bzr can't even do a status in a lightweight checkout without
contacting the branch:

    Normal checkouts can perform common operations like
    diff and status without such access[...].

Re: BZR and the bootstrap

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Date: Mon, 09 Aug 2010 15:57:42 +0900
> Cc: Angelo Graziosi <angelo.graziosi@alice.it>, emacs-devel@gnu.org
> 
> It's penny-wise and pound-foolish for anybody who is unfamiliar with
> Bazaar to use lightweight checkouts.  Occasional contributors and beta
> testers should invest in a full branch, bound or not depending on
> their use cases.

If it turns out that this is how "bzr status" is supposed to work in
lightweight checkouts, perhaps VC should refrain from calling that
command when it detects a lightweight checkout.

Re: BZR and the bootstrap

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Date: Mon, 09 Aug 2010 20:57:07 +0900
> Cc: emacs-devel@gnu.org
> 
> Óscar Fuentes writes:
> 
>  > IIRC, lightweight checkouts contain information for testing if the
>  > file is modified or missing from the file system, which provides a
>  > local `bzr status' operation. For anything else it needs to contact
>  > upstream.
> 
> That may be true, but the rest of that help message strongly implies
> that bzr can't even do a status in a lightweight checkout without
> contacting the branch:
> 
>     Normal checkouts can perform common operations like
>     diff and status without such access[...].

I asked on the Bazaar list about this.

Re: BZR and the bootstrap

[Johan Bockgård]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Sat, 07 Aug 2010 18:49:54 +0200
>> From: Angelo Graziosi <angelo.graziosi@alice.it>
>> 
>> Now I have noticed, when my internet connection is on (1), that BZR is 
>> called during the bootstrap. This is flagged both by 'top' and the modem 
>> lights.
>> 
>> Why BZR is called? What is it doing?
>
> Probably because Emacs calls VC every time it needs to visit a file
> that is in a versioned directory.

But shouldn't bootstrap avoid visiting files?

find-file(-noselect) is called by

    update-directory-autoloads
    autoload-find-destination
    custom-make-dependencies
    finder-compile-keywords
    miscdic-convert
    quail-update-leim-list-file

(28 calls made during bootstrap).

Re: BZR and the bootstrap

[Angelo Graziosi]

Il 09/08/2010 19.11, Eli Zaretskii ha scritto:
>> From: "Stephen J. Turnbull"<stephen@xemacs.org>
>> Date: Mon, 09 Aug 2010 15:57:42 +0900
>> Cc: Angelo Graziosi<angelo.graziosi@alice.it>, emacs-devel@gnu.org
>>
>> It's penny-wise and pound-foolish for anybody who is unfamiliar with
>> Bazaar to use lightweight checkouts.  Occasional contributors and beta
>> testers should invest in a full branch, bound or not depending on
>> their use cases.
>
> If it turns out that this is how "bzr status" is supposed to work in
> lightweight checkouts, perhaps VC should refrain from calling that
> command when it detects a lightweight checkout.

Unless someone beats me, I would continue to do what I have always done 
after the migration to BZR system:

$ bzr checkout --lightweight http://bzr.savannah.gnu.org/r/emacs/trunk 
emacs-trunk

to be updated with 'bzr up'.

Now, if I want avoiding BZR to be called while bootstrapping, I will do:

$ rsync -av --exclude=.* --delete emacs-trunk/ emacs/

using 'emacs/' as source tree.

Omitting '--lightweight' is good too, but a file '.bzr' of 200M is 
annoying if one wants to save the bzr tree in a tar-ball archive (to be 
used on other systems, for example).

Ciao,
Angelo.

Re: BZR and the bootstrap

[Óscar Fuentes]

Angelo Graziosi <angelo.graziosi@alice.it> writes:

[snip]

> Omitting '--lightweight' is good too, but a file '.bzr' of 200M is
> annoying if one wants to save the bzr tree in a tar-ball archive (to
> be used on other systems, for example).

bzr help export

Seriously, unless you are using a wristwatch as the storage media, there
is no reason for not using a proper bzr checkout or branch.

Re: BZR and the bootstrap

[Eli Zaretskii]

> From: Johan Bockgård <bojohan@gnu.org>
> Date: Mon, 09 Aug 2010 21:42:09 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Date: Sat, 07 Aug 2010 18:49:54 +0200
> >> From: Angelo Graziosi <angelo.graziosi@alice.it>
> >> 
> >> Now I have noticed, when my internet connection is on (1), that BZR is 
> >> called during the bootstrap. This is flagged both by 'top' and the modem 
> >> lights.
> >> 
> >> Why BZR is called? What is it doing?
> >
> > Probably because Emacs calls VC every time it needs to visit a file
> > that is in a versioned directory.
> 
> But shouldn't bootstrap avoid visiting files?

Why should it?  And how can it, if it must produce loaddefs.el,
finder-inf.el, etc.?

Re: BZR and the bootstrap

[Stefan Monnier]

> But shouldn't bootstrap avoid visiting files?
> find-file(-noselect) is called by

We could try and refrain from using find-file-* during bootstrap, but
I don't think it's worth the trouble.


        Stefan

Re: BZR and the bootstrap

[Kenichi Handa]

In article <E1OiiAv-0002uy-N3@fencepost.gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:
> > > Probably because Emacs calls VC every time it needs to visit a file
> > > that is in a versioned directory.
> > 
> > But shouldn't bootstrap avoid visiting files?

> Why should it?  And how can it, if it must produce loaddefs.el,
> finder-inf.el, etc.?

I think just turning off VC-facility while building is good.
Are there any merit in having it while building?

---
Kenichi Handa
handa@m17n.org

Re: BZR and the bootstrap

[Stefan Monnier]

> I think just turning off VC-facility while building is good.
> Are there any merit in having it while building?

I don't think VC brings any benefits while building Emacs, indeed.
OTOH, if it needs to be turned off in this case, maybe it needs to be
turned off in other cases.  IOW, I'd rather try and make sure VC "never"
needs to be turned off, or only in cases where the user really has an
odd config that justifies such a measure.

E.g. we could change VC so it figures out which backend is in use
(etc...) more lazily.

        Stefan

Re: BZR and the bootstrap

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
> Date: Tue, 10 Aug 2010 11:20:57 +0200
> 
> > I think just turning off VC-facility while building is good.
> > Are there any merit in having it while building?
> 
> I don't think VC brings any benefits while building Emacs, indeed.

Nevertheless, I don't like the idea of turning it off in some
circumstances.  From the moment we made VC an inherent part of
visiting files, it should do that always, because this is now a
routine part of file ops.  If we disable it in some situations, a Lisp
programmer will be unable to tell when VC is available and when not
without consulting the code or the docs, because no one will remember
all these conditions a year from now.

> OTOH, if it needs to be turned off in this case, maybe it needs to be
> turned off in other cases.  IOW, I'd rather try and make sure VC "never"
> needs to be turned off, or only in cases where the user really has an
> odd config that justifies such a measure.

Right.

> E.g. we could change VC so it figures out which backend is in use
> (etc...) more lazily.

I think the problem here is with running "bzr status".  But we could
be smarter about doing that in a lightweight checkout.