Updating the GNU ELPA package of AucTeX

Updating the GNU ELPA package of AucTeX

[Stefan Monnier]

The GNU ELPA package of AucTeX is ridiculously old.

As part of updating, I'd like to "streamline" the process, which mostly
involves reducing the difference between the code in the `elpa' branch
and the code in the official auctex repository.

The differences I can see are the following:
- the `preview' sub directory (in `elpa' there's no such directory, all
  the files within it are either removed or moved up one level).
- the .dtx vs .sty files.
- the .texi vs .info files.
- the Makefile.in, configure.ac, and friends.

Ideally, I'd like the GNU ELPA package to be built directly from the
source code (currently it seems it's made from the tarball instead,
IIUC).  But contrary to the XEmacs case, the construction of the GNU
ELPA package is made by a very simplistic process (which does very
little more than create an auctex-autoloads.el file, and tar-up the
result).  I can make this process more sophisticated, but since it's run
fully automatically, I'd very much prefer not having to run code from
the package itself.

IOW, it's easy to arrange the construction so as not to include some
files (e.g. Makefile.in).  It's also possible (but I'm not very eager to
do it) to rename files during construction.  But I really would not want
to extract the sty from the dtx during construction.

So, the real problems are:

* Preview subdir:

  I see two clean and easy ways to deal with:
  - Split it (again) from AucTeX.
  - "mv preview/* ./; rmdir preview"
  Is one of those two options agreeable to you?

* The .texi files:

  The problem is not specific to auctex, so I'll have to come up with
  a solution anyway, probably building the .info files during
  construction of the ELPA package.

* The .dtx files:

  Here, building the .sty during construction of the ELPA package is
  not really an option.  So I see the following ways out:
  - Include the generated .sty file in the `elpa' branch (with the
    usual problems associated with VCS control of auto-generated files).
  - Build the .sty when *installing* the package (when package.el
    installs a package, it does various things additionally to unpacking
    the tarball, and it could be made to run tex as well).
  - Build the .sty lazily just before the user needs it.
  Does one of those options sound good to you?


-- Stefan

Re: Updating the GNU ELPA package of AucTeX

[Tassilo Horn]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

Hi Stefan,

> The GNU ELPA package of AucTeX is ridiculously old.

Yep.

> As part of updating, I'd like to "streamline" the process, which
> mostly involves reducing the difference between the code in the `elpa'
> branch and the code in the official auctex repository.

I've hoped that with the new Git ELPA, we could just checkout the
savannah auctex repository as submodule, and then add some hacks to the
makefiles to produce tarballs suitable for ELPA.  Won't work, no?

I mean, you can run auctex just fine just by

  $ ./configure
  $ make

and then setting TeX-data-directory to the auctex checkout directory.
Doesn't that mean that the ELPA build could just build auctex, package
the files meant for distribution preserving the repository directory
structure, and setting TeX-data-directory to
~/.emacs.d/elpa/auctex-<version>/?

Ah, btw, I think I've hunted down all past contributors that hadn't
signed a CA already.  So now I think there's no file left that has to be
excluded.

> So, the real problems are:
>
> * Preview subdir:
>
>   I see two clean and easy ways to deal with:
>   - Split it (again) from AucTeX.

I'd prefer to keep it in auctex.

>   - "mv preview/* ./; rmdir preview"

I could live with that.  But why is a subdirectory a problem in the
first place?

> * The .texi files:
>
>   The problem is not specific to auctex, so I'll have to come up with
>   a solution anyway, probably building the .info files during
>   construction of the ELPA package.

If the build process would just run make, they'd be built.

> * The .dtx files:
>
>   Here, building the .sty during construction of the ELPA package is
>   not really an option.

No LaTeX installed on the machine, or what's the matter?

>   So I see the following ways out:
>   - Include the generated .sty file in the `elpa' branch (with the
>     usual problems associated with VCS control of auto-generated files).
>   - Build the .sty when *installing* the package (when package.el
>     installs a package, it does various things additionally to unpacking
>     the tarball, and it could be made to run tex as well).
>   - Build the .sty lazily just before the user needs it.
>   Does one of those options sound good to you?

I don't really have a preference.  The options all sound bad compared to
just making running "make" with its LaTeX-deps possible.

Bye,
Tassilo

Re: Updating the GNU ELPA package of AucTeX

[Stefan Monnier]

> I've hoped that with the new Git ELPA, we could just checkout the
> savannah auctex repository as submodule, and then add some hacks to the
> makefiles to produce tarballs suitable for ELPA.  Won't work, no?

No, because:

   I'd very much prefer not having to run code from
   the package itself.

so I can't just run your makefile rule(s).

> Ah, btw, I think I've hunted down all past contributors that hadn't
> signed a CA already.  So now I think there's no file left that has to be
> excluded.

Great news, thank you.

>> So, the real problems are:
>> * Preview subdir:
>> I see two clean and easy ways to deal with:
>> - Split it (again) from AucTeX.
> I'd prefer to keep it in auctex.

Good.

>> - "mv preview/* ./; rmdir preview"
> I could live with that.  But why is a subdirectory a problem in the
> first place?

Because of the extra complexity in the build procedure.  Remember: it's
very simplistic.

>> * The .texi files:
>> The problem is not specific to auctex, so I'll have to come up with
>> a solution anyway, probably building the .info files during
>> construction of the ELPA package.
> If the build process would just run make, they'd be built.

As mentioned, running "make" is not an option.  But even I did run "make",
it wouldn't work: texinfo is not installed on that machine.

>> * The .dtx files:
>> Here, building the .sty during construction of the ELPA package is
>> not really an option.
> No LaTeX installed on the machine, or what's the matter?

Same as "make": automatically running TeX code from random contributors
is too risky for my taste.  But of course, TeX is not installed on the
machine either.


        Stefan

Re: Updating the GNU ELPA package of AucTeX

[Tassilo Horn]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I've hoped that with the new Git ELPA, we could just checkout the
>> savannah auctex repository as submodule, and then add some hacks to the
>> makefiles to produce tarballs suitable for ELPA.  Won't work, no?
>
> No, because:
>
>    I'd very much prefer not having to run code from
>    the package itself.
>
> so I can't just run your makefile rule(s).

Ok, I see.

>>> So, the real problems are:
>>> * Preview subdir:
>>> I see two clean and easy ways to deal with:
>>> - Split it (again) from AucTeX.
>> I'd prefer to keep it in auctex.
>
> Good.
>
>>> - "mv preview/* ./; rmdir preview"
>> I could live with that.  But why is a subdirectory a problem in the
>> first place?
>
> Because of the extra complexity in the build procedure.  Remember:
> it's very simplistic.

How about simply omitting preview from the auctex elpa package (at least
for now)?

Maybe (later) we could have a separate preview-latex elpa package more
or less by creating a symlink packages/preview-latex ->
packages/auctex/preview and running the very simplistic build procedure
in the former?

>>> * The .dtx files:
>>> Here, building the .sty during construction of the ELPA package is
>>> not really an option.
>> No LaTeX installed on the machine, or what's the matter?
>
> Same as "make": automatically running TeX code from random
> contributors is too risky for my taste.  But of course, TeX is not
> installed on the machine either.

By omitting preview, that would be solved, too.

Bye,
Tassilo

Re: Updating the GNU ELPA package of AucTeX

[David Kastrup]

Tassilo Horn <tsdh@gnu.org> writes:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>>> I've hoped that with the new Git ELPA, we could just checkout the
>>> savannah auctex repository as submodule, and then add some hacks to the
>>> makefiles to produce tarballs suitable for ELPA.  Won't work, no?
>>
>> No, because:
>>
>>    I'd very much prefer not having to run code from
>>    the package itself.
>>
>> so I can't just run your makefile rule(s).
>
> Ok, I see.

It is not clear to me why the act of importing a runnable version of
AUCTeX into ELPA should be precluded from running a Makefile rule.  It
is not like ELPA can directly access git repositories and extract
whatever it wants, so the import will always involve explicit steps.

> By omitting preview, that would be solved, too.

preview-latex is an integral part of AUCTeX by now.  That involved
moving files, dependencies, combining documentation and so on.  Ripping
it out of ELPA is not just going to be work, but is also not extremely
likely to make people happy.

-- 
David Kastrup

Re: Updating the GNU ELPA package of AucTeX

[Tassilo Horn]

David Kastrup <dak@gnu.org> writes:

>>> No, because:
>>>
>>>    I'd very much prefer not having to run code from
>>>    the package itself.
>>>
>>> so I can't just run your makefile rule(s).
>>
>> Ok, I see.
>
> It is not clear to me why the act of importing a runnable version of
> AUCTeX into ELPA should be precluded from running a Makefile rule.  It
> is not like ELPA can directly access git repositories and extract
> whatever it wants, so the import will always involve explicit steps.

The new Git ELPA could have the savannah auctex repository as a
submodule, so there wouldn't be two individual auctex repositories that
need to me synchronized manually.

>> By omitting preview, that would be solved, too.
>
> preview-latex is an integral part of AUCTeX by now.  That involved
> moving files, dependencies, combining documentation and so on.
> Ripping it out of ELPA is not just going to be work, but is also not
> extremely likely to make people happy.

Stefan said that the additional auctex/preview/ directory was a problem
for the ELPA build procedure, so I suggested to have the ELPA auctex
package not contain it (the directory would still exist in the ELPA
auctex git submodule).  Then we could possibly have a separate
preview-latex ELPA package by just symlinking packages/preview-latex to
packages/auctex/preview/.  So I actually don't propose to rip preview
out of auctex, just to split it into two ELPA packages in order to cope
with the simplistic ELPA build procedure.

That said, I'm just trying to propose ideas without knowing about
feasibility.  I'm really far from an expert in how ELPA packages are
build, and neither do I know too much about auctex's build procedure.

So David (and Ralf), it would be great if you could jump in here.  We
definitively want to have current auctex ELPA packages which requires
having a copy of the auctex code in the ELPA repository and building it
with ELPA's simplistic build procedure.  But at the same time, the
auctex savannah repository and the auctex directory in ELPA shouldn't be
completely separate (possibly even with different directory structures)
so that synchronization becomes a nightmare.

Bye,
Tassilo

Re: Updating the GNU ELPA package of AucTeX

[Stefan Monnier]

>> It is not clear to me why the act of importing a runnable version of
>> AUCTeX into ELPA should be precluded from running a Makefile rule.  It
>> is not like ELPA can directly access git repositories and extract
>> whatever it wants, so the import will always involve explicit steps.

Yes, GNU ELPA can and does "git pull" every day as part of its
automatic procedure.

> The new Git ELPA could have the savannah auctex repository as a
> submodule, so there wouldn't be two individual auctex repositories that
> need to me synchronized manually.

That's the intention, indeed (tho not technically as a Git submodule,
but morally equivalent).

>>> By omitting preview, that would be solved, too.
>> preview-latex is an integral part of AUCTeX by now.

I tend to agree with this.  I'm not sure of the potential
technical/documentation problems, but in terms of user expectation
I definitely want preview-latex to be installed when you install auctex.

> Stefan said that the additional auctex/preview/ directory was a problem
> for the ELPA build procedure, so I suggested to have the ELPA auctex
> package not contain it (the directory would still exist in the ELPA
> auctex git submodule).  Then we could possibly have a separate
> preview-latex ELPA package by just symlinking packages/preview-latex to
> packages/auctex/preview/.  So I actually don't propose to rip preview
> out of auctex, just to split it into two ELPA packages in order to cope
> with the simplistic ELPA build procedure.

I'd be OK with splitting auctex into 2 packages, but does preview-latex
work without auctex?  If not, that means we'd need to put the
dependencies in the "wrong" direction (i.e. users wouldn't automatically
get preview-latex when they install auctex).

It seems simpler to do the "mv preview/* ./; rmdir preview".


        Stefan

Re: Updating the GNU ELPA package of AucTeX

[David Kastrup]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> It is not clear to me why the act of importing a runnable version of
>>> AUCTeX into ELPA should be precluded from running a Makefile rule.
>>> It is not like ELPA can directly access git repositories and extract
>>> whatever it wants, so the import will always involve explicit steps.
>
> Yes, GNU ELPA can and does "git pull" every day as part of its
> automatic procedure.

If it expects a package to be in a finished state upon pulling, it means
that any standard GNU package (which requires ./configure && make &&
sudo make install to work) is not supported.

The AUCTeX build procedure supports creating an XEmacs package.  The
XEmacs package is built using a system agnostic configuration which
detects most things at runtime (at some cost).

Basically you are asking that we throw away all configurability of
AUCTeX and convert its repository into an installed tree with a
"neutral" configuration.

How will its Texinfo files get converted into documentation readable as
PDF or info?  How will its intro.texi get converted into README by
makeinfo?  After all, every GNU package should have a README, right?
Our current procedures create this README.

>> The new Git ELPA could have the savannah auctex repository as a
>> submodule, so there wouldn't be two individual auctex repositories
>> that need to me synchronized manually.
>
> That's the intention, indeed (tho not technically as a Git submodule,
> but morally equivalent).

"Morally equivalent"?  You make it sound like you consider the current
AUCTeX repository immoral.

At any rate, it would be easy enough to create Makefile targets that
would export a setup suitable for ELPA.  It's not really superbly
apropos

      The "source code" for a work means the preferred form of the work
    for making modifications to it.  "Object code" means any non-source
    form of a work.

    [...]

      The "Corresponding Source" for a work in object code form means
    all the source code needed to generate, install, and (for an
    executable work) run the object code and to modify the work,
    including scripts to control those activities.

but of course, once you throw away any other way of configuring the
package, what remains is by definition the "preferred form for making
modifications" since nothing else exists anymore.

> I'd be OK with splitting auctex into 2 packages, but does
> preview-latex work without auctex?

No.  It would be nice to factor out some of its quite sophisticated
functionality into something independent from AUCTeX, LaTeX and in fact
also TeX, but at the current point of time preview-latex is not
independently useful.

-- 
David Kastrup

Re: [AUCTeX-devel] Updating the GNU ELPA package of AucTeX

[Stefan Monnier]

>>>> It is not clear to me why the act of importing a runnable version of
>>>> AUCTeX into ELPA should be precluded from running a Makefile rule.
>>>> It is not like ELPA can directly access git repositories and extract
>>>> whatever it wants, so the import will always involve explicit steps.
>> Yes, GNU ELPA can and does "git pull" every day as part of its
>> automatic procedure.

> If it expects a package to be in a finished state upon pulling, it means
> that any standard GNU package (which requires ./configure && make &&
> sudo make install to work) is not supported.

The GNU guidelines only require for it to be possible to use
"./configure && make && make install".  They don't prevent parallel use
of other installation methods.

I can easily accommodate a package with extra files like "configure" and
"Makefile".

> Basically you are asking that we throw away all configurability of
> AUCTeX and convert its repository into an installed tree with a
> "neutral" configuration.

Not at all.

> How will its Texinfo files get converted into documentation readable as
> PDF or info?  How will its intro.texi get converted into README by
> makeinfo?  After all, every GNU package should have a README, right?
> Our current procedures create this README.

These are indeed the questions I asked.  I proposed 3 ways to do it,
there might be more.  None of these preclude keeping the existing
make rules.

>> That's the intention, indeed (tho not technically as a Git submodule,
>> but morally equivalent).
> "Morally equivalent"?  You make it sound like you consider the current
> AUCTeX repository immoral.

Feeling argumentative today?  Reread what I wrote: it is not talking
about AUCTeX but about the technique used to include it in `elpa'.

>> I'd be OK with splitting auctex into 2 packages, but does
>> preview-latex work without auctex?
> No.  It would be nice to factor out some of its quite sophisticated
> functionality into something independent from AUCTeX, LaTeX and in fact
> also TeX, but at the current point of time preview-latex is not
> independently useful.

I'm not sure you're answering my question, so let me rephrase it: does
it work with tex-mode.el?


        Stefan "who often uses tex-mode.el rather than AUCTeX, FWIW"

Re: Updating the GNU ELPA package of AucTeX

[David Kastrup]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>>> I'd be OK with splitting auctex into 2 packages, but does
>>> preview-latex work without auctex?
>> No.  It would be nice to factor out some of its quite sophisticated
>> functionality into something independent from AUCTeX, LaTeX and in fact
>> also TeX, but at the current point of time preview-latex is not
>> independently useful.
>
> I'm not sure you're answering my question, so let me rephrase it: does
> it work with tex-mode.el?

The answer remains "No".  "Independent from AUCTeX, LaTeX and in fact
also TeX" is a graded answer.  As preview-latex is not independently
useful without AUCTeX, it does not work with tex-mode.el.  tex-mode.el
would probably be low-hanging fruit as adapting preview-latex would
basically require finding/abstracting a common process/commandline model
for both AUCTeX and tex-mode.el.

But so far nobody hated AUCTeX enough to find that worth the trouble...

-- 
David Kastrup

Re: [AUCTeX-devel] Updating the GNU ELPA package of AucTeX

[Stefan Monnier]

>> I'm not sure you're answering my question, so let me rephrase it: does
>> it work with tex-mode.el?
> The answer remains "No".

Thanks.  Then it makes more sense to keep both auctex and preview-latex
in the same package (more specifically to do the "mv preview/* ./").


        Stefan

Re: [AUCTeX-devel] Updating the GNU ELPA package of AucTeX

[Tassilo Horn]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>>> I'm not sure you're answering my question, so let me rephrase it:
>>> does it work with tex-mode.el?
>> The answer remains "No".
>
> Thanks.  Then it makes more sense to keep both auctex and
> preview-latex in the same package (more specifically to do the "mv
> preview/* ./").

As said, that's fine with me.  But auctex/preview/ has its own autotools
build setup that needs to be merged with auctex's.  I've never used the
autotools other than from a plain user perspective, so I can't do that.

Bye,
Tassilo

Re: Updating the GNU ELPA package of AucTeX

[Stefan Monnier]

>> Thanks.  Then it makes more sense to keep both auctex and
>> preview-latex in the same package (more specifically to do the "mv
>> preview/* ./").
> As said, that's fine with me.  But auctex/preview/ has its own autotools
> build setup that needs to be merged with auctex's.  I've never used the
> autotools other than from a plain user perspective, so I can't do that.

OK, so GNU ELPA is now at 11.87.
This package was built from a Git repository
(http://git.savannah.gnu.org/gitweb/?p=emacs/elpa.git;a=shortlog;h=refs/heads/externals/auctex)
which is based on the AUCTeX Git (currently only updated up to 11.87),
but with all the preview files moved one level up, and with the
auto-generated files stored in the repository.

The auto-generated files were not built with the provided configure+make
rules, since those are broken there because of the file-moves.
Instead I added a GNUmakefile which "does the job for now".

For now, this is the best I can do on elpa's side to get closer to
AUCTeX's Git tree.  Any effort AUCTeX itself can do to get closer to
elpa's tree will be appreciated (I don't expect nor encourage AUCTeX to
include the auto-generated files, but mirroring my "mv preview/* ./"
would be great to help future syncs).

I'll try and setup some system that will send you a copy of any commit
made to elpa's repository, so you can try and incorporate it into
AUCTeX's code.


        Stefan