RefTeX Merge

RefTeX Merge

[Ralf Angeli]

Hi,

for the last years RefTeX has been maintained in two separate
repositories: the one Emacs and the one of AUCTeX.  Since not much
development is happening with respect to RefTeX in the AUCTeX repository
I'd like to merge the changes from both locations now and then
discontinue the version in the AUCTeX repository.  That means the
version in the Emacs repository would then be the only place where
RefTeX is developed.

I'm now nearly finished with the task of merging the changes and I'd
like to discuss a few details before I check the changes into each
repository.

* ChangeLog information

  There is a lot of information about the changes done in each
  repository, namely the file history and the ChangeLog entries.  Is
  there a nice and easy way to preserve this information?

  I'd be especially interested in bringing the history of the standalone
  version of RefTeX to the Emacs repository, so that the changes of the
  last years can be retraced after the standalone version will be
  discontinued.  Unfortunately RefTeX does not have its own ChangeLog in
  the Emacs repository so I'm not sure where to put this stuff.  The
  commit log entry could be an option.  But I'm not particularly fond of
  it.

* Build infrastructure and standalone releases

  The standalone RefTeX has a lisp and a doc directory below the root
  directory where each location has its own Makefile for building and
  installing the program and the documentation.  There are also make
  targets for making a release.  I don't really plan to release RefTeX
  separately from Emacs after the version in the AUCTeX repositories
  will be retired but I'm a bit wary of throwing the Makefiles away.  So
  since they will not be needed in the Emacs repository I'll not move
  them over but probably do a final standalone release and have the
  whole code thereby archived on gnu.org.  Are there better ways to
  preserve such code?

* Version info in Lisp and Texinfo files

  The version and date strings for Lisp and Texinfo files of the
  standalone version is generated from the ChangeLog when make is run,
  so that it is always up-to-date, even for development versions.  The
  build process creates a reftex.el file with the version and autoload
  information and a version.texi file with the version and the date that
  is included in the main Texinfo file.  Is there something like this as
  well in Emacs?

  For `reftex-version' one could probably just use `emacs-version' but
  for the version and date info in the Texinfo manual one would need to
  pull in something like a version.texi file as well.

* Macros in Texinfo files

  The reftex.texi file of the standalone version uses macros for words
  like TeX, LaTeX, BibTeX and so on.  So there is e.g. @BibTeX{} instead
  of Bib@TeX{}.  Can those be used in the Emacs repository as well?

Once the above questions will be clarified and there are no objections I
will go ahead with the merge and the files into the AUCTeX and Emacs
repositories.

-- 
Ralf

Re: RefTeX Merge

[Paul Eggert]

On 09/23/2012 11:10 AM, Ralf Angeli wrote:
>   Unfortunately RefTeX does not have its own ChangeLog in
>   the Emacs repository so I'm not sure where to put this stuff.

One possibility is to create a new directory
lisp/textmodes/reftex, move the RefTeX source there,
and create a ChangeLog there, starting the ChangeLog with all the
old history followed by a note saying what happened.

>   Are there better ways to preserve [old Makefiles].

You could add them to the Emacs repository and then remove them,
I suppose.  Not sure it's worth the confusion, though.

> 
> a version.texi file with the version and the date that
>   is included in the main Texinfo file.  Is there something like this as
>   well in Emacs?

Not as far as I know.  Shouldn't be hard to add.

> * Macros in Texinfo files
> 
>   The reftex.texi file of the standalone version uses macros for words
>   like TeX, LaTeX, BibTeX and so on.  So there is e.g. @BibTeX{} instead
>   of Bib@TeX{}.  Can those be used in the Emacs repository as well?

Don't see why not.

Re: RefTeX Merge

[Glenn Morris]

Ralf Angeli wrote:

> * Version info in Lisp and Texinfo files
>
>   The version and date strings for Lisp and Texinfo files of the
>   standalone version is generated from the ChangeLog when make is run,
>   so that it is always up-to-date, even for development versions.  The
>   build process creates a reftex.el file with the version and autoload
>   information and a version.texi file with the version and the date that
>   is included in the main Texinfo file.  Is there something like this as
>   well in Emacs?

This is not really an option for Emacs, because the info and elc files
are pre-built in release tarfiles. So people don't even need to have
makeinfo installed to build Emacs. So you cannot generate parts of the
*.texi files at build time. This is why eg doc/emacs/emacsver.texi is
not generated by configure.

> * Macros in Texinfo files
>
>   The reftex.texi file of the standalone version uses macros for words
>   like TeX, LaTeX, BibTeX and so on.  So there is e.g. @BibTeX{} instead
>   of Bib@TeX{}.  Can those be used in the Emacs repository as well?

It depends which version of makeinfo they first appeared in. @TeX and
@LaTeX are used already elsewhere in Emacs, so are OK. I've never heard
of @BibTeX, and it is not in the texinfo 4.13 manual on my current
machine. Configure currently tests for makeinfo >= 4.7.

Re: RefTeX Merge

[Glenn Morris]

Glenn Morris wrote:

>> * Macros in Texinfo files
>>
>>   The reftex.texi file of the standalone version uses macros for words
>>   like TeX, LaTeX, BibTeX and so on.  So there is e.g. @BibTeX{} instead
>>   of Bib@TeX{}.  Can those be used in the Emacs repository as well?
>
> It depends which version of makeinfo they first appeared in.

Or maybe you meant user-defined macros? In which case I don't understand
why you even need to ask. @macro is already used in several files under
Emacs's doc/ directory. @TeX and @LaTeX are standard Texinfo commands
already though, so you should not redefine them.

Re: RefTeX Merge

[Glenn Morris]

Glenn Morris wrote:

>>   The version and date strings for Lisp and Texinfo files of the
>>   standalone version is generated from the ChangeLog when make is run,
>>   so that it is always up-to-date, even for development versions.  The
>>   build process creates a reftex.el file with the version and autoload
>>   information and a version.texi file with the version and the date that
>>   is included in the main Texinfo file.  Is there something like this as
>>   well in Emacs?
>
> This is not really an option for Emacs, because the info and elc files
> are pre-built in release tarfiles. So people don't even need to have
> makeinfo installed to build Emacs. So you cannot generate parts of the
> *.texi files at build time. This is why eg doc/emacs/emacsver.texi is
> not generated by configure.

I'm being particularly dumb today and should probably be ignored.
Of course you could have such a rule if it ran only during the building
of the info files. (But it doesn't seem worth the effort to me.)

Re: RefTeX Merge

[Glenn Morris]

Glenn Morris wrote:

> I'm being particularly dumb today and should probably be ignored.
> Of course you could have such a rule if it ran only during the building
> of the info files. (But it doesn't seem worth the effort to me.)

PS you would have to find a way to make it work under MS-Windows, which
does not seem to use "sed" at present for the build (it would simplify a
lot if it could).

Re: RefTeX Merge

[Glenn Morris]

Glenn Morris wrote:

> PS you would have to find a way to make it work under MS-Windows, which
> does not seem to use "sed" at present for the build (it would simplify a
> lot if it could).

Well, except for doc/misc/makefile.w32-in:gnus.dvi and
admin/unidata/makefile.w32-in:unidata.txt:, which are not standard
targets. And lib/makefile.w32-in:getopt_h, which has a comment about
sed.

Re: RefTeX Merge

[Chong Yidong]

Paul Eggert <eggert@cs.ucla.edu> writes:

> One possibility is to create a new directory
> lisp/textmodes/reftex, move the RefTeX source there,
> and create a ChangeLog there, starting the ChangeLog with all the
> old history followed by a note saying what happened.

No.  Since reftex.el is one file, it is not worth creating a
subdirectory just for it.

When a file is synched to the Emacs repository, the ChangeLog typically
only describes the differences relative to the previous version.  That's
usually a truncated version of the file's full ChangeLog history, but in
practice this hasn't been a big problem.  Adding a big mass of ChangeLog
entries is not very helpful for people browsing the Emacs ChangeLog.

How large are the ChangeLog entries in question?  Putting them in the
commit log may be a fine solution.  (Normally, one would use bzr join to
preserve the history, but bzr join seems to be badly broken.)

Re: RefTeX Merge

[Ralf Angeli]

* Chong Yidong (2012-09-24) writes:

> Paul Eggert <eggert@cs.ucla.edu> writes:
>
>> One possibility is to create a new directory
>> lisp/textmodes/reftex, move the RefTeX source there,
>> and create a ChangeLog there, starting the ChangeLog with all the
>> old history followed by a note saying what happened.
>
> No.  Since reftex.el is one file, it is not worth creating a
> subdirectory just for it.

RefTeX consists of eleven Lisp files.  It might not be a bad idea to put
them into their own directory.

> When a file is synched to the Emacs repository, the ChangeLog typically
> only describes the differences relative to the previous version.  That's
> usually a truncated version of the file's full ChangeLog history, but in
> practice this hasn't been a big problem.  Adding a big mass of ChangeLog
> entries is not very helpful for people browsing the Emacs ChangeLog.

That's where a separate ChangeLog file could make sense.  But I agree
that it would not be a good idea to put the whole ChangeLog from the
standalone version in there.  I'd have to check how much effort it would
be to write down the changes relative to the current version in Emacs.
It could be a bit of work. (c:

> How large are the ChangeLog entries in question?

The ChangeLog is about 26 kB.  One could shave off a bit of it by
removing the entries related to the Makefiles.  Here is a link if you
want to have a look:

http://cvs.savannah.gnu.org/viewvc/reftex/ChangeLog?root=auctex&view=markup

> Putting them in the
> commit log may be a fine solution.

If the log can take such amount of data, then I would be fine with it.
And besides that I'd prepare a shorter entry for the ChangeLog file.

> (Normally, one would use bzr join to
> preserve the history, but bzr join seems to be badly broken.)

Now that would be a nice solution.  RefTeX is currently maintained in a
CVS repository, though.  I'm not sure if this makes matters better or
worse.

-- 
Ralf

Re: RefTeX Merge

[Ralf Angeli]

* Glenn Morris (2012-09-23) writes:

> Ralf Angeli wrote:
>
>> * Version info in Lisp and Texinfo files
>
> This is not really an option for Emacs, because the info and elc files
> are pre-built in release tarfiles. So people don't even need to have
> makeinfo installed to build Emacs. So you cannot generate parts of the
> *.texi files at build time. This is why eg doc/emacs/emacsver.texi is
> not generated by configure.

Ah, okay.  But emacsver.texi is sort of what I was looking for.  Even
though it is not automatically generated, the version information in
there could be used in reftex.texi.  (As long as people are content with
RefTeX using the same version as Emacs.)

emacsver.texi lacks information about the date which can be used to
distinguish different builds during the development phase.  And I found
it nice to see how old a (released) version is.  But I guess it's not
too much of a loss if that bit of information is removed from the
manual.

>> * Macros in Texinfo files
>
> It depends which version of makeinfo they first appeared in. @TeX and
> @LaTeX are used already elsewhere in Emacs, so are OK. I've never heard
> of @BibTeX, and it is not in the texinfo 4.13 manual on my current
> machine. Configure currently tests for makeinfo >= 4.7.

As you suspected in another mail I was talking about user-defined macros
where I was not sure if they can be used.  Since they don't seem to be
problematic I'll leave them in reftex.texi.

In the standalone version there is a macro.texi file which is included
in reftex.texi but for the Emacs version I've put everything in
reftex.texi.  The same is true for install.texi with installation
instructions (which should be removed at some point in time) and
changes.texi with news related to releases.

-- 
Ralf

Re: RefTeX Merge

[Richard Stallman]

    RefTeX consists of eleven Lisp files.  It might not be a bad idea to put
    them into their own directory.

I would not make a subdirectory for just 11 source files,
because I am concerned that would make too many subdirectories.
I looked for at least 20 and preferably considerably more than 20
to make a subdirectory.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call

Re: RefTeX Merge

[Tassilo Horn]

Ralf Angeli <angeli@caeruleus.net> writes:

>> (Normally, one would use bzr join to preserve the history, but bzr
>> join seems to be badly broken.)
>
> Now that would be a nice solution.  RefTeX is currently maintained in
> a CVS repository, though.  I'm not sure if this makes matters better
> or worse.

I can import the reftex CVS repository into a bzr or git repository if
it helps to get the job done.  I've already written the needed scripts
and configurations for cvs2{bzr,git} for converting AUCTeX, and they'll
probably work well with reftex, too.

Bye,
Tassilo

Re: RefTeX Merge

[Ralf Angeli]

* Tassilo Horn (2012-09-25) writes:

> Ralf Angeli <angeli@caeruleus.net> writes:
>
>>> (Normally, one would use bzr join to preserve the history, but bzr
>>> join seems to be badly broken.)
>>
>> Now that would be a nice solution.  RefTeX is currently maintained in
>> a CVS repository, though.  I'm not sure if this makes matters better
>> or worse.
>
> I can import the reftex CVS repository into a bzr or git repository if
> it helps to get the job done.  I've already written the needed scripts
> and configurations for cvs2{bzr,git} for converting AUCTeX, and they'll
> probably work well with reftex, too.

I tried to get some information about `brz join' but there does not seem
to be much available.  And given the problems with the command discussed
on emacs-devel it looks a bit dangerous.

Also, I'm not yet sure if it would be the right tool for the job because
we want to do a merge of files existing in both repositories and the
description of `bzr join' talks about combining a tree with a subtree.

What would be the steps involved?  Commit the merged files in both
repositories, convert the RefTeX CVS repo to bzr and then join the lisp
directory of the RefTeX bzr repo into the Emacs repo?  Or would the
merge be part of the join?

-- 
Ralf

Re: RefTeX Merge

[Tassilo Horn]

Ralf Angeli <angeli@caeruleus.net> writes:

Hi Ralf,

>> I can import the reftex CVS repository into a bzr or git repository
>> if it helps to get the job done.  I've already written the needed
>> scripts and configurations for cvs2{bzr,git} for converting AUCTeX,
>> and they'll probably work well with reftex, too.
>
> I tried to get some information about `brz join' but there does not
> seem to be much available.  And given the problems with the command
> discussed on emacs-devel it looks a bit dangerous.
>
> Also, I'm not yet sure if it would be the right tool for the job
> because we want to do a merge of files existing in both repositories
> and the description of `bzr join' talks about combining a tree with a
> subtree.
>
> What would be the steps involved?  Commit the merged files in both
> repositories, convert the RefTeX CVS repo to bzr and then join the
> lisp directory of the RefTeX bzr repo into the Emacs repo?  Or would
> the merge be part of the join?

I don't have any clue, but the converted repository is here, waiting for
you to play around with it.

    bzr branch http://www.tsdh.org/reftex.bzr/trunk reftex

Bye,
Tassilo

Re: RefTeX Merge

[Stephen J. Turnbull]

Ralf Angeli writes:

 > I tried to get some information about `brz join' but there does not
 > seem to be much available.  And given the problems with the command
 > discussed on emacs-devel it looks a bit dangerous.

DVCS means never having to say you're sorry.  And I mean *never*.[1]
Remember, all dangerous operations can be conducted on a copy of the
repo.  All it costs is a small amount of disk space for a short period
of time.

 > I'm not yet sure if it would be the right tool for the job because
 > we want to do a merge of files existing in both repositories

This isn't possible in bzr, because bzr tracks "file containers", and
there is no provision for identifying "containers" from repositories
without a common ancestor.  No matter how you slice it, you're going
to get an add-add conflict unless you delete one copy or the other
first.  I can see how to do this in git, but it wouldn't be pleasant
(both ugly and painstaking, but you'd probably end up with a usable
history).

Why do you want to do that, anyway?  What are these files that exist
in both repos?  Do they really have independent lines of development
in both repos?  Or is one a slightly tweaked copy of the other?

 > and the description of `bzr join' talks about combining a tree with
 > a subtree.

Yup.  Seems to me that is what you *should* want to do.  If this falls
afoul of RMS's preference for "sufficient number of files" in a
subtree, fix that *after* the join by bzr-move'ing the files into the
appropriate place.

 > What would be the steps involved?

I can't give specific advice without answers to the questions above.

Footnotes: 
[1]  Well, until you push. ;-)

Re: RefTeX Merge

[Chong Yidong]

Ralf Angeli <angeli@caeruleus.net> writes:

>> No.  Since reftex.el is one file, it is not worth creating a
>> subdirectory just for it.
>
> RefTeX consists of eleven Lisp files.  It might not be a bad idea to put
> them into their own directory.

Sorry, my mistake.

But I have to agree with RMS; it still doesn't seem worth giving reftex
its own subdirectory.

>> Putting them in the commit log may be a fine solution.
>
> If the log can take such amount of data, then I would be fine with it.
> And besides that I'd prepare a shorter entry for the ChangeLog file.

From my estimation, once you take out entries for Makefile changes,
releases, and synchs to Emacs, the ChangeLog is about 400 lines, which
is manageable.  So I think we can put it in the commit log.

In the version to be added to the ChangeLog file, you can collate the
changes to each file, which should shorten the entry significantly.
Still, I think a ChangeLog entry of ~100 lines is acceptable.

>> (Normally, one would use bzr join to
>> preserve the history, but bzr join seems to be badly broken.)
>
> Now that would be a nice solution.  RefTeX is currently maintained in
> a CVS repository, though.  I'm not sure if this makes matters better
> or worse.

Let's not play with bzr join.  We've had bad experiences with that on
the elpa branch.

I suggest just patching the affected files manually, then committing as
though you had made the changes in one shot.  Just be sure not to
inadvertently change the headers (e.g. "This file is part of GNU Emacs")
and so forth.

Re: RefTeX Merge

[Stefan Monnier]

> I tried to get some information about `brz join' but there does not seem
> to be much available.  And given the problems with the command discussed
> on emacs-devel it looks a bit dangerous.

Here's how it could work:

   bzr co .../emacs/trunk
   cd trunk/lisp/textmodes
   bzr co .../reftex/trunk reftex
   bzr join reftex
   bzr mv reftex/foo bar
   bzr mv ...
   bzr rm reftex
   bzr commit locally
   make sure the resulting repository works right

Just make sure the reftex bzr tree hasn't gone through a "bzr-git"
translation because otherwise the resulting branch may trigger a bug in
bzr when you later checkout a new copy of that branch (at least that
seems to be the problem we bumped into in the `elpa' branch).

"bzr join" has worked reliably for me as long as I didn't try to join
a repository converted by the bzr-git plugin.  But I suspect that bzr-hg
might trigger the same problems.
   
> Also, I'm not yet sure if it would be the right tool for the job because
> we want to do a merge of files existing in both repositories and the
> description of `bzr join' talks about combining a tree with a subtree.

Indeed, the above leads to having two copies of the reftex files, so you
have to choose between the two and Bazaar won't know that they were really
the same files.  So it's not necessarily the best solution, depending on
which history you want to keep and how important it is to preserve it
directly in the Emacs repository's metadata (as opposed to just as
a ChangeLog file or a commit log text).


        Stefan

Subdirectories in `lisp'

[Stefan Monnier]

>> RefTeX consists of eleven Lisp files.  It might not be a bad idea to put
>> them into their own directory.
> But I have to agree with RMS; it still doesn't seem worth giving reftex
> its own subdirectory.

BTW, the main reason is to avoid lengthening load-path too much, so
maybe a good alternative is to have a subdirectory but not add it to
load-path.
Instead of (require 'reftex-parse) you'd do (require 'reftex/parse).


        Stefan

Re: RefTeX Merge

[Ralf Angeli]

* Chong Yidong (2012-09-26) writes:

> Ralf Angeli <angeli@caeruleus.net> writes:
>
>> If the log can take such amount of data, then I would be fine with it.
>> And besides that I'd prepare a shorter entry for the ChangeLog file.
>
> From my estimation, once you take out entries for Makefile changes,
> releases, and synchs to Emacs, the ChangeLog is about 400 lines, which
> is manageable.  So I think we can put it in the commit log.

It got about 100 lines more.  I hope this is not a problem.  At least
bzr didn't barf.

> In the version to be added to the ChangeLog file, you can collate the
> changes to each file, which should shorten the entry significantly.
> Still, I think a ChangeLog entry of ~100 lines is acceptable.

In the ChangeLog about 140 lines ended up.  Again, I hope this is okay.
If necessary I can try to shave of a few.

> Let's not play with bzr join.  We've had bad experiences with that on
> the elpa branch.
>
> I suggest just patching the affected files manually, then committing as
> though you had made the changes in one shot.  Just be sure not to
> inadvertently change the headers (e.g. "This file is part of GNU Emacs")
> and so forth.

I've made some experiments with `bzr join' and I haven't really had
success.  That's why I committed the changes as suggested above.

Thanks a lot to the people commenting on the topic for all the hints and
feedback.

-- 
Ralf

Re: RefTeX Merge

[Tassilo Horn]

Hi Ralf,

I've seen the reftex changes have landed in emacs/trunk.  As far as I
can see, it works just fine.  Now after the merge, even the TAB
completion for bib-entries at the `reftex-citation' prompt works
again. :-)

One thing which seems a bit inconvenient to me is `reftex-reference' now
always querying if one wants a \ref or \pageref.  This is made even a
bit stranger by the fact that this query pops up a selection window
immediately, whereas the next query (which labels to show) starts
without a window (it pops up only after some idle time).  IMO, either
both should pop up the selection window immediately, or both should
start with a minibuffer message and pop up the full selection window
after some idle time.

And can't the \ref or \pageref question be guessed with rather high
certainty anyway?  E.g. only if the preceeding word is "page", "Seite",
"página", etc, one wants a \pageref, else we want a \ref.

Anyway, thanks a lot for your work!  I'm pretty sure that wasn't much
fun.  Is the reftex CVS now officially obsolete, so that when I change
reftex stuff in emacs I don't need to fiddle the change in there, too?

Bye,
Tassilo

Re: RefTeX Merge

[Ralf Angeli]

* Tassilo Horn (2012-10-16) writes:

> I've seen the reftex changes have landed in emacs/trunk.  As far as I
> can see, it works just fine.  Now after the merge, even the TAB
> completion for bib-entries at the `reftex-citation' prompt works
> again. :-)

Good to know. (c:

> One thing which seems a bit inconvenient to me is `reftex-reference' now
> always querying if one wants a \ref or \pageref.

You can turn this off by setting `reftex-ref-macro-prompt' to nil.  IIRC
I made t the default value because other reference styles (see
`reftex-ref-style-alist') offer macros in addition to \ref and \pageref.
With the prompt enabled it is quite convenient to insert those.  And it
is also easier to insert \pageref with the prompt.

> This is made even a
> bit stranger by the fact that this query pops up a selection window
> immediately, whereas the next query (which labels to show) starts
> without a window (it pops up only after some idle time).  IMO, either
> both should pop up the selection window immediately, or both should
> start with a minibuffer message and pop up the full selection window
> after some idle time.

It should not be too difficult to make both behave identically because
both are implemented by calling `reftex-select-with-char'.  Personally I
don't have a problem in them behaving differently.  I dunno if one gets
so familiar with the shortcuts used for the reference styles that it is
not necessary anymore to display the selection buffer.  But people are
likely quite familiar with the shortcuts for the label types and there
it makes sense to display the selection buffer only after a short delay.

> And can't the \ref or \pageref question be guessed with rather high
> certainty anyway?  E.g. only if the preceeding word is "page", "Seite",
> "página", etc, one wants a \pageref, else we want a \ref.

That would be convenient if we were only dealing with \ref and \pageref.
But what about the macros offered by the other reference styles?  For
those you'd need the prompt nevertheless.

> Anyway, thanks a lot for your work!  I'm pretty sure that wasn't much
> fun.

And it took quite some time because I merged "by hand". (c:

> Is the reftex CVS now officially obsolete, so that when I change
> reftex stuff in emacs I don't need to fiddle the change in there, too?

Yes, I guess it is obsolete.  I still plan to do a final release based
on the state which is in CVS but I don't think it is necessary to merge
anything from the Emacs repository until then.

-- 
Ralf