Call for testing: Daily snapshot builds of Emacs for Ubuntu

Call for testing: Daily snapshot builds of Emacs for Ubuntu

[Robert Park]

For the last couple of months, I have been working on enabling daily
built Ubuntu packages of Emacs' trunk bzr branch, to make it easier
for testers to install and test the latest code, hopefully narrowing
the feedback loop between testers and developers.

You may be familiar with the snapshot packaging work done by Damien
Cassou; however his method is largely to build by hand once every
couple weeks, I am using Launchpad recipes to achieve the same thing
every day, in an entirely automated fashion.

To achieve this, I have written all new packaging metadata from
scratch, discarding many years of legacy cruft from the debian build
system (where the debian packaging weighs in at 4900 lines of code, I
have nearly achieved feature parity with only 1495 lines of code).

My packages provide much of what you might expect from a debian
package of Emacs:

* Separate packages for Gtk3 and NoX builds

* Files not unique to each build grouped into a single -common package

* Uncompiled elisp files split into a separate -el package for
optional installation, saving disk space.

* postinst and prerm scripts provide support for the debian
update-alternatives system

* Can be installed alongside the existing emacs24 package that's
available in debian/ubuntu.

Known issues:

* No support for elisp packages installed by dpkg. This issue is
unfortunately more complicated than simply twiddling load-path and I'm
still investigating it. However, I consider this issue quite minor
because package.el support is *superb* in trunk and I highly recommend
it's flexibility over the elisp packages installed by dpkg.

If you want to give this a try in Ubuntu, you can run these commands:

$ sudo add-apt-repository ppa:ubuntu-elisp/ppa
$ sudo apt-get update
$ sudo apt-get install emacs-snapshot

(or emacs-snapshot-nox as you prefer)

Currently, builds are provided daily for Ubuntu Precise, Raring, and
Saucy (latest LTS, latest stable, and dev series). There is a Quantal
build available in the PPA, however I stopped making new Quantal
builds the day Raring was released ;-)

So please, take it for a spin, let me know how it's all working for
you, and let's get all bugs fixed in trunk because distropatches are
Eeeeeeevil!

Oh, and here's the recipe, for the curious:

https://code.launchpad.net/~ubuntu-elisp/+recipe/emacs-daily

Re: Call for testing: Daily snapshot builds of Emacs for Ubuntu

[W. Greenhouse]

Hi Robert,

Robert Park <r@robru.ca> writes:

> Known issues:
>
> * No support for elisp packages installed by dpkg. This issue is
> unfortunately more complicated than simply twiddling load-path and I'm
> still investigating it. However, I consider this issue quite minor
> because package.el support is *superb* in trunk and I highly recommend
> it's flexibility over the elisp packages installed by dpkg.
>
> If you want to give this a try in Ubuntu, you can run these commands:
>
> $ sudo add-apt-repository ppa:ubuntu-elisp/ppa
> $ sudo apt-get update
> $ sudo apt-get install emacs-snapshot
>
> (or emacs-snapshot-nox as you prefer)
>
> Currently, builds are provided daily for Ubuntu Precise, Raring, and
> Saucy (latest LTS, latest stable, and dev series). There is a Quantal
> build available in the PPA, however I stopped making new Quantal
> builds the day Raring was released ;-)
>
> So please, take it for a spin, let me know how it's all working for
> you, and let's get all bugs fixed in trunk because distropatches are
> Eeeeeeevil!
>
> Oh, and here's the recipe, for the curious:
>
> https://code.launchpad.net/~ubuntu-elisp/+recipe/emacs-daily

I'm quite new to Debian/dpkg packaging in general, and have no
familiarity with Launchpad recipes, but it sounds like your package
might not be loading all of the startup bits expected by
/usr/share/doc/emacsen-common/debian-emacs-policy.gz.  For starters,
does M-x load-library RET debian-startup RET from your Launchpad builds
actually point somewhere?  If so, does the `startup' library load it?
That `debian-startup' file is the entry-point for Emacs dpkg packages
setting up their autoloads and the like, and yes, unfortunately, this is
a big fat distropatch. :-)

Ubuntu's emacs-snapshot packaging may have changed this whole
infrastructure in some way that I'm not taking account of, but I would
expect that many elisp packages in Ubuntu are largely unchanged from
Debian and simply expect this to work.

I agree that it's a pretty minor complaint if the rest of the package is
working great, but dpkg packages for Emacs features are still useful in
spite of package.el, particularly if they have non-elisp dependencies,
e.g. the `auctex' package, or the `maildir-utils' package (containing
the `mu' maildir indexer which is the binary dependency for djcb's
`mu4e' Emacs mailclient).

HTH,

WGG

-- 
BOFH excuse #411:

Traffic jam on the Information Superhighway.

Re: Call for testing: Daily snapshot builds of Emacs for Ubuntu

[Robert Park]

On Sat, May 11, 2013 at 7:23 PM, W. Greenhouse <wgreenhouse@riseup.net> wrote:
> it sounds like your package
> might not be loading all of the startup bits expected by
> /usr/share/doc/emacsen-common/debian-emacs-policy.gz.

Right now, it's not doing anything ;-)

I did spend a little bit of time trying to apply this distropatch:

http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/saucy/emacs24/saucy/view/head:/debian/patches/0002-Run-debian-startup-and-set-debian-emacs-flavor.patch

But it didn't seem to be enough to work on it's own... there's some
other magic piece of the puzzle that I haven't discovered yet (some of
the other distropatches are quite mysterious to me and may interact
with this in ways that weren't obvious upon an initial skim).

Just tried enabling it again locally, and this is what I get in my
*Messages* buffer at startup:

Loading /etc/emacs/site-start.d/00debian-vars.el (source)...done
Loading /etc/emacs/site-start.d/50autoconf.el (source)...
Error while loading 50autoconf: No /usr/local/ prefixed paths in load-path
Loading /etc/emacs/site-start.d/50cmake-data.el (source)...done
Loading /etc/emacs/site-start.d/50dictionaries-common.el (source)...
Error while loading 50dictionaries-common: No /usr/local/ prefixed
paths in load-path
Loading /etc/emacs/site-start.d/50gtk-doc-tools.el (source)...done
Loading /etc/emacs/site-start.d/50lbdb.el (source)...done
Loading /etc/emacs/site-start.d/50pylint.el (source)...
Loading /usr/share/emacs/site-lisp/pylint/pylint.el (source)...done
Loading /etc/emacs/site-start.d/50pylint.el (source)...done
Loading /etc/emacs/site-start.d/50python-docutils.el (source)...done
Loading /etc/emacs/site-start.d/50python-guppy.el (source)...done
Loading /etc/emacs/site-start.d/50yaml-mode.el (source)...
Error while loading 50yaml-mode: No /usr/local/ prefixed paths in load-path
For information about GNU Emacs and the GNU system, type <f1> C-a.
Type "q" in help window to restore its previous buffer, C-M-v to scroll help.

Not sure if that's meaningful to you.

>  For starters,
> does M-x load-library RET debian-startup RET from your Launchpad builds
> actually point somewhere?

Well, I just typed in those commands and they worked...

>  If so, does the `startup' library load it?

No, because I don't currently have the above distropatch applied.

> That `debian-startup' file is the entry-point for Emacs dpkg packages
> setting up their autoloads and the like, and yes, unfortunately, this is
> a big fat distropatch. :-)

Is there any possibility of generalizing this and having it accepted
in upstream emacs? Obviously we wouldn't commit any debianisms into
emacs trunk but it seems like it might be generally useful for emacs
to have some kind of site-start.d directory for all distros to place
custom startup scripts into. I'd be willing to put a bit of effort
into this if there was some indication from upstream that it had a
chance of being accepted (ie, what are the acceptance criteria?).

> Ubuntu's emacs-snapshot packaging may have changed this whole
> infrastructure in some way that I'm not taking account of, but I would
> expect that many elisp packages in Ubuntu are largely unchanged from
> Debian and simply expect this to work.

I haven't checked closely, but yes, I expect that all the elisp debs
are imported unmodified; Emacs is not exactly a strategic package that
is subject to lots of scrutiny/modifications by Ubuntu ;-)

(most of the ubuntu distropatches that I am aware of are simply
bugfixes backported from trunk anyway).

> I agree that it's a pretty minor complaint if the rest of the package is
> working great, but dpkg packages for Emacs features are still useful in
> spite of package.el, particularly if they have non-elisp dependencies,
> e.g. the `auctex' package, or the `maildir-utils' package (containing
> the `mu' maildir indexer which is the binary dependency for djcb's
> `mu4e' Emacs mailclient).

Oh yeah, it's definitely something that's on my radar to be fixed
generally, just that I don't use any of those things you mentioned so
everything is working great for me ;-)

Re: Call for testing: Daily snapshot builds of Emacs for Ubuntu

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 1217 bytes --]

On Sun, May 12 2013, Robert Park wrote:

Hi Robert,

> You may be familiar with the snapshot packaging work done by Damien
> Cassou; however his method is largely to build by hand once every
> couple weeks, I am using Launchpad recipes to achieve the same thing
> every day, in an entirely automated fashion.

As far as I know, Damien used the packaging I maintain at:

   http://anonscm.debian.org/gitweb/?p=users/acid/emacs-snapshot.git;a=summary

> To achieve this, I have written all new packaging metadata from
> scratch, discarding many years of legacy cruft from the debian build
> system (where the debian packaging weighs in at 4900 lines of code, I
> have nearly achieved feature parity with only 1495 lines of code).

Ok, this claim makes me smile. The actual Debian packaging is not 4900
lines. Sure, the files in debian/ are around that number of lines… with
a ChangeLog files of 3325. There isn't that much code at all, even the
debian/rules file is only 400 SLOC long.

That said, what's the reason for you not to try to improve the already
existing and working code?

Cheers,
-- 
Julien Danjou
# Free Software hacker # freelance consultant
# http://julien.danjou.info

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 835 bytes --]

Re: Call for testing: Daily snapshot builds of Emacs for Ubuntu

[Robert Park]

On Mon, May 13, 2013 at 2:08 AM, Julien Danjou <julien@danjou.info> wrote:
> On Sun, May 12 2013, Robert Park wrote:
>> You may be familiar with the snapshot packaging work done by Damien
>> Cassou; however his method is largely to build by hand once every
>> couple weeks, I am using Launchpad recipes to achieve the same thing
>> every day, in an entirely automated fashion.
>
> As far as I know, Damien used the packaging I maintain at:
>
>    http://anonscm.debian.org/gitweb/?p=users/acid/emacs-snapshot.git;a=summary

What's this then? Is this a debian package dedicated to emacs
snapshots? I honestly wasn't aware of this branch at all; I'd been
referring to the packaging for emacs24 that ubuntu imports from
debian.

>> To achieve this, I have written all new packaging metadata from
>> scratch, discarding many years of legacy cruft from the debian build
>> system (where the debian packaging weighs in at 4900 lines of code, I
>> have nearly achieved feature parity with only 1495 lines of code).
>
> Ok, this claim makes me smile. The actual Debian packaging is not 4900
> lines. Sure, the files in debian/ are around that number of lines… with
> a ChangeLog files of 3325. There isn't that much code at all, even the
> debian/rules file is only 400 SLOC long.

Hmmm, nope. The debian/rules that I've been referring to is 629 lines
(compared to 166 for mine). So it seems we are talking about different
things ;-)

Admittedly I did come up with that original figure by doing a
quickndirty 'find|xargs wc -l' and it did include the changelog,
there's still a lot of cruft that I am rebelling against. Like 800
lines worth of distropatching just to rip out some GFDL stuff. I find
that kind of thing quite odious and pride myself on having a packaging
branch that has no distropatches at all ;-)

> That said, what's the reason for you not to try to improve the already
> existing and working code?

One of my major goals was Upstream Purity, so that people testing the
packages would be having an unadulterated trunk experience, hopefully
making bug reports more relevant and timely. Both the packaging branch
that you've linked and the one I've been referring to for emacs24
contain a huge load of distropatches.

Now, I'm not opposed to porting some of the work I've done to improve
the existing packages, just that on the whole I was a bit overwhelmed
by the size of the existing packages and wanted to make something
leaner. I'm quite pleased with what I've achieved and I'm using it
every day, I uninstalled emacs24 package over a month ago and haven't
looked back ;-)

Re: Call for testing: Daily snapshot builds of Emacs for Ubuntu

[Glenn Morris]

Robert Park wrote:

> there's still a lot of cruft that I am rebelling against. Like 800
> lines worth of distropatching just to rip out some GFDL stuff. I find
> that kind of thing quite odious and pride myself on having a packaging
> branch that has no distropatches at all ;-)

I only count 7 Debian patches to Emacs:
http://patch-tracker.debian.org/package/emacs24/24.3+1-1

Of course it is trivial to remove some of those if you don't need to
obide by the rules that the distribution sets (on what they consider
acceptable free software licenses in this instance) for its packages.

Of the remaining 5 patches, the rest are all small.
One is the debian-startup thing, one adds a pointer to Debian-specific
docs, one modifies the version string, one handles the fact that they
install it as "emacs24", and the last one is some flymake trivia that 
should as of just now be unnecessary.

> Is there any possibility of generalizing this and having it accepted
> in upstream emacs? Obviously we wouldn't commit any debianisms into
> emacs trunk but it seems like it might be generally useful for emacs
> to have some kind of site-start.d directory for all distros to place
> custom startup scripts into.

AFAICS, Red Hat solved this with a trivial and sensible-looking site-start.el:

;; load ".el" files in "/usr/share/emacs/site-lisp/site-start.d/" on startup
(mapc 'load
      (directory-files "/usr/share/emacs/site-lisp/site-start.d" t "\\.el\\'"))

I don't know what the debian-startup thing brings to the table beyond that.
(I'd probably ask http://lists.debian.org/debian-emacsen/ if I wanted to
know.)

Re: Call for testing: Daily snapshot builds of Emacs for Ubuntu

[Robert Park]

On Mon, May 13, 2013 at 7:48 PM, Glenn Morris <rgm@gnu.org> wrote:
> I only count 7 Debian patches to Emacs:
> http://patch-tracker.debian.org/package/emacs24/24.3+1-1

Well, there are 13 by the time it gets to ubuntu:

http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/raring/emacs24/raring/files/head:/debian/patches/

> Of course it is trivial to remove some of those if you don't need to
> obide by the rules that the distribution sets (on what they consider
> acceptable free software licenses in this instance) for its packages.

Right, this is why I made an Ubuntu package instead of a debian
package. Because I don't care about the DFSG, the GFDL is "free enough
for me", and I wanted a pure trunk experience, for better testing of
what is shipped by upstream.

>> Is there any possibility of generalizing this and having it accepted
>> in upstream emacs? Obviously we wouldn't commit any debianisms into
>> emacs trunk but it seems like it might be generally useful for emacs
>> to have some kind of site-start.d directory for all distros to place
>> custom startup scripts into.
>
> AFAICS, Red Hat solved this with a trivial and sensible-looking site-start.el:
>
> ;; load ".el" files in "/usr/share/emacs/site-lisp/site-start.d/" on startup
> (mapc 'load
>       (directory-files "/usr/share/emacs/site-lisp/site-start.d" t "\\.el\\'"))
>
> I don't know what the debian-startup thing brings to the table beyond that.
> (I'd probably ask http://lists.debian.org/debian-emacsen/ if I wanted to
> know.)

Yikes, that is trivial and sensible-looking. The site-start.el
provided by emacsen-common package in debian (and ubuntu) contains
only commented lines, so is effectively blank. Now that i see there's
such a trivial solution for this, I'm definitely not adding that
distropatch to my packaging ;-)

I'll have to do some further research to find out why it was chosen to
leave site-start.el blank and see about perhaps dropping this
particular distropatch from the official debian packaging.

Re: Call for testing: Daily snapshot builds of Emacs for Ubuntu

[Stephen J. Turnbull]

Robert Park writes:

 > I'll have to do some further research to find out why it was chosen to
 > leave site-start.el blank and see about perhaps dropping this
 > particular distropatch from the official debian packaging.

Debian admins are often role-your-own types who take "site-specific"
seriously, and get pissed off if the distro overwrites their local
changes.  Since site-start.el has historically been reserved for
site-specific initialization, Debian prefers to put its stuff in
distro-specific places.  While it's been about a decade and a half
since I discussed this with a Debian maintainer, I'd bet on no change.

Red Hat (and admins who use it) tends to be more pragmatic, preferring
simple and standardized, and willing to impose (or accept the
imposition) of (relatively) standard hardware, software, and
configurations in the interest of improving support.

Re: Call for testing: Daily snapshot builds of Emacs for Ubuntu

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 2691 bytes --]

On Mon, May 13 2013, Robert Park wrote:

> What's this then? Is this a debian package dedicated to emacs
> snapshots? I honestly wasn't aware of this branch at all; I'd been
> referring to the packaging for emacs24 that ubuntu imports from
> debian.

It's the Debian package for emacs-snapshot that has been distributed for
years in Debian, and in Ubuntu via Damien Cassou.

> Hmmm, nope. The debian/rules that I've been referring to is 629 lines
> (compared to 166 for mine). So it seems we are talking about different
> things ;-)

I didn't count the comments.

> Admittedly I did come up with that original figure by doing a
> quickndirty 'find|xargs wc -l' and it did include the changelog,
> there's still a lot of cruft that I am rebelling against. Like 800
> lines worth of distropatching just to rip out some GFDL stuff. I find
> that kind of thing quite odious and pride myself on having a packaging
> branch that has no distropatches at all ;-)

I don't think you're talking about the emacs-snapshot package here?
In emacs24 packages, that has to be since GFDL stuff aren't DFSG
compliant. But we don't do this in emacs-snapshot packages, because we
don't really care about this issue.

> One of my major goals was Upstream Purity, so that people testing the
> packages would be having an unadulterated trunk experience, hopefully
> making bug reports more relevant and timely. Both the packaging branch
> that you've linked and the one I've been referring to for emacs24
> contain a huge load of distropatches.

Did you really read the actual patches?
All patches are there to help having working default on a Debian system,
plus a few to indicate which Debian's version of the package it is.

And even if we had patches that would modify or fix upstream, a better
job would be to make sure we don't have to need them anymore.

> Now, I'm not opposed to porting some of the work I've done to improve
> the existing packages, just that on the whole I was a bit overwhelmed
> by the size of the existing packages and wanted to make something
> leaner. I'm quite pleased with what I've achieved and I'm using it
> every day, I uninstalled emacs24 package over a month ago and haven't
> looked back ;-)

No offense, but maybe you should consider that not understanding a whole
actual package isn't necessary a sign that the package is convoluted,
but may be a sign you lack some knowledge to understand everything?

Anyway, I'd be glad you'd help to improve the existing package rather
than building something new and half-baked.

-- 
Julien Danjou
;; Free Software hacker ; freelance consultant
;; http://julien.danjou.info

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 835 bytes --]

Re: Call for testing: Daily snapshot builds of Emacs for Ubuntu

[Barry Warsaw]

[-- Attachment #1: Type: text/plain, Size: 872 bytes --]

On May 11, 2013, at 05:06 PM, Robert Park wrote:

>* No support for elisp packages installed by dpkg. This issue is
>unfortunately more complicated than simply twiddling load-path and I'm
>still investigating it. However, I consider this issue quite minor
>because package.el support is *superb* in trunk and I highly recommend
>it's flexibility over the elisp packages installed by dpkg.

Still, I think this would need to be solved to be a serious contender for
replacing the Debian-inherited Emacs package for Ubuntu.  One important reason
is that many sites do not allow downloading and installing packages except
from the OS vendor repositories, so cutting that out would mean that some
users would not have access to 3rd party elisp.

(Python has a similar problem with regards to PyPI, as I'm sure is the case
with other such frameworks.)

-Barry

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Call for testing: Daily snapshot builds of Emacs for Ubuntu

[Robert Park]

On Tue, May 14, 2013 at 2:05 PM, Barry Warsaw <barry@python.org> wrote:
> On May 11, 2013, at 05:06 PM, Robert Park wrote:
>>* No support for elisp packages installed by dpkg. This issue is
>>unfortunately more complicated than simply twiddling load-path and I'm
>>still investigating it. However, I consider this issue quite minor
>>because package.el support is *superb* in trunk and I highly recommend
>>it's flexibility over the elisp packages installed by dpkg.
>
> Still, I think this would need to be solved to be a serious contender for
> replacing the Debian-inherited Emacs package for Ubuntu.

Yes, of course. I never suggested that I would want my packaging to
become the official ubuntu package without any support for elisp debs.

All I'm saying is that I have a PPA which will always contain a
version of Emacs that is less than 24 hours old. This is useful to
Ubuntu users who want to develop or test the latest version of Emacs,
because they can install it and automatically get updates every day,
without having to do any work at all. The benefits are that it's
completely automated, so even if I take a vacation, users of my PPA
continue receiving daily updates.

Julien's emacs-snapsot package does not actually get imported into
Ubuntu at all[0] (or at least not since 2009), and emacs24 seems stuck
at 24.2[1], so I am actually filling a very real void here for Ubuntu
users who wish to avoid stale versions of Emacs.

[0] https://launchpad.net/ubuntu/+source/emacs-snapshot
[1] https://launchpad.net/ubuntu/+source/emacs24