Why wasn't the 25.3 release based on the then-head of the emacs-25 branch?

Why wasn't the 25.3 release based on the then-head of the emacs-25 branch?

[Glenn Morris]

Hi,

I've just noticed that the Emacs 25.3 release seems to be literally just
the 25.2 release plus the enriched changes. Specifically, it seems to be
based on 784602b. Why wasn't it based on b638993? There were very few
changes in the emacs-25 branch post 25.2, and I don't see why they were
omitted.

In particular, it's disappointing that 94a6c96 isn't included. This
(security) issue was reported to emacs-devel three years ago.

Re: Why wasn't the 25.3 release based on the then-head of the emacs-25 branch?

[Paul Eggert]

Glenn Morris wrote:
> There were very few
> changes in the emacs-25 branch post 25.2, and I don't see why they were
> omitted.

Eli decided on a very thin 25.3 release, with just a patch for the 
newly-discovered serious bug, along with minimal other changes such as 
renumbering 25.2 to 25.3.

> In particular, it's disappointing that 94a6c96 isn't included. This
> (security) issue was reported to emacs-devel three years ago.

Although I also argued privately for releasing based on the emacs-25 branch 
instead of just a thin release, Eli worried that the other changes in emacs-25 
(including the change you mention) might have caused problems in 25.3's hasty 
release. As I recall, the security issue that you mention is not as serious as 
the one fixed in 25.3, since it occurs only in an unlikely installation nowadays 
(no gnutls library or binaries available) whereas the 25.3 issue is an easy way 
to break into anyone using Gnus to read email.

Re: Why wasn't the 25.3 release based on the then-head of the emacs-25 branch?

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Date: Thu, 14 Sep 2017 16:30:46 -0400
> 
> I've just noticed that the Emacs 25.3 release seems to be literally just
> the 25.2 release plus the enriched changes. Specifically, it seems to be
> based on 784602b. Why wasn't it based on b638993? There were very few
> changes in the emacs-25 branch post 25.2, and I don't see why they were
> omitted.

The emacs-25 branch included a couple of non-trivial code changes post
25.2, which were not widely tested.  In fact, I'm guessing that no one
actually used the branch, once 25.2 was released, more than just start
it up and maybe run the tests.  In these conditions, making 25.3 from
the branch tip would mean there would have to be a pretest or an RC,
which would have delayed the release considerably.  Doing that was
inconsistent with the sense of urgency shared by everybody of the
Emacs 25.3 release.

I thought I wrote all that here, but I see now it was off-list, with
John, Richard, and Paul.  Sorry about that, but it became hard to
track the addressees at that point among the storm of messages with
similar Subjects.

Re: Why wasn't the 25.3 release based on the then-head of the emacs-25 branch?

[Tim Cross]

[-- Attachment #1: Type: text/plain, Size: 1999 bytes --]

Hi Eli et. al.

just wanted to thank you for the background and say that as someone who
works in the information security field, I support your decision and
reasoning 100%. With a severe security issue, it is much much better to
release quickly and have a release which ONLY addresses the security issue.
In addition to potentially introducing new issues once you release other
changes, bundling more than the security fix can delay adoption as people
will tend to be more conservative due to concerns new/changed functionality
may impact existing workflows. Limiting the release to just the change
necessary to fix the vulnerability reduces concerns and risk and helps to
drive more rapid adoption. Thanks to all for their effort.

Tim


On 15 September 2017 at 16:23, Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Glenn Morris <rgm@gnu.org>
> > Date: Thu, 14 Sep 2017 16:30:46 -0400
> >
> > I've just noticed that the Emacs 25.3 release seems to be literally just
> > the 25.2 release plus the enriched changes. Specifically, it seems to be
> > based on 784602b. Why wasn't it based on b638993? There were very few
> > changes in the emacs-25 branch post 25.2, and I don't see why they were
> > omitted.
>
> The emacs-25 branch included a couple of non-trivial code changes post
> 25.2, which were not widely tested.  In fact, I'm guessing that no one
> actually used the branch, once 25.2 was released, more than just start
> it up and maybe run the tests.  In these conditions, making 25.3 from
> the branch tip would mean there would have to be a pretest or an RC,
> which would have delayed the release considerably.  Doing that was
> inconsistent with the sense of urgency shared by everybody of the
> Emacs 25.3 release.
>
> I thought I wrote all that here, but I see now it was off-list, with
> John, Richard, and Paul.  Sorry about that, but it became hard to
> track the addressees at that point among the storm of messages with
> similar Subjects.
>
>


-- 
regards,

Tim

--
Tim Cross

[-- Attachment #2: Type: text/html, Size: 2724 bytes --]

Re: Why wasn't the 25.3 release based on the then-head of the emacs-25 branch?

[Glenn Morris]

Eli Zaretskii wrote:

> The emacs-25 branch included a couple of non-trivial code changes post
> 25.2, which were not widely tested.  In fact, I'm guessing that no one
> actually used the branch, once 25.2 was released, more than just start
> it up and maybe run the tests. 

I'm not aware of any non-trival code changes.

For the record, the _only_ code change (until Paul's gcc-7 changes last
Friday, which do seem trivial) in the emacs-25 branch was the tls one I
referred to. It had been present in both emacs-25 and master since
April, as well as being distro-patched by Debian in their Emacs 25.1
package over the same time period (since they considered it a serious
issue, ie one that otherwise merits eventual removal of the affected
package; ref https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=766397 ).
Personally I consider that ample testing.

Everything else was doc fixes.

Re: Why wasn't the 25.3 release based on the then-head of the emacs-25 branch?

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Fri, 15 Sep 2017 19:39:08 -0400
> 
> Eli Zaretskii wrote:
> 
> > The emacs-25 branch included a couple of non-trivial code changes post
> > 25.2, which were not widely tested.  In fact, I'm guessing that no one
> > actually used the branch, once 25.2 was released, more than just start
> > it up and maybe run the tests. 
> 
> I'm not aware of any non-trival code changes.

Then I guess we have different notions of "trivial".

> For the record, the _only_ code change (until Paul's gcc-7 changes last
> Friday, which do seem trivial)

No, heyt aren't.  They affect the build process, so could potentially
break the build on some systems.

> in the emacs-25 branch was the tls one I referred to. It had been
> present in both emacs-25 and master since April, as well as being
> distro-patched by Debian in their Emacs 25.1 package over the same
> time period (since they considered it a serious issue, ie one that
> otherwise merits eventual removal of the affected package; ref
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=766397 ).
> Personally I consider that ample testing.

Even if the Debian distro could be considered ample testing (and I'm
not sure it could, as Debian AFAIR are generally not on the bleeding
edge with system features, library and kernel versions, etc.), there
wasn't enough time to perform the research which could convince us
that change was tested well enough for us to trust it.  In fact, there
wasn't even time to find out whether any distributions patched their
Emacsen with that change.  What little time we had was used to come up
with a reasonable patch for the problem and thoroughly test it.

Another factor that affected my decision was that when we discussed
the possible release of 25.3 back in May, no one felt this tls patch
was serious enough (to say nothing of the memory-allocation problems)
to require another bugfix release.  How come it's suddenly so
important that we were supposed to risk producing a less-than-perfect
tarball just to have it?

In general, we never released any code that was not tested via the
normal pretest/RC cycle that we are familiar with.  You are talking
about a procedure that was never used by this project, for as long as
I remember it.  I don't think it would have been prudent for us to
invent a new procedure while producing an urgent release that was
intended to plug a security vulnerability, and use it the first time
for that urgent release.  How many times did you see a new procedure
we tried here that succeeded the first time it was used?  I never saw
anything like that.

This release needed to be 100% safe.  Not 99.99%, 100%.  IME, this
calls for qualitatively different mindset and measures than the usual
process, where small risks for unlikely events are acceptable if the
gains justify them.  100% safe meant we could not tolerate _any_
risks.

Re: Why wasn't the 25.3 release based on the then-head of the emacs-25 branch?

[Eli Zaretskii]

> From: Tim Cross <theophilusx@gmail.com>
> Date: Sat, 16 Sep 2017 07:26:39 +1000
> Cc: Glenn Morris <rgm@gnu.org>, Emacs developers <emacs-devel@gnu.org>
> 
> just wanted to thank you for the background and say that as someone who works in the information security
> field, I support your decision and reasoning 100%.

Thanks.

Re: Why wasn't the 25.3 release based on the then-head of the emacs-25 branch?

[Paul Eggert]

Eli Zaretskii wrote:

> Even if the Debian distro could be considered ample testing (and I'm
> not sure it could, as Debian AFAIR are generally not on the bleeding
> edge with system features, library and kernel versions, etc.)

As a practical matter, Debian has waaayy more testing than we do. If the patch 
worked for them that's a strong signal, stronger than any information we have to 
the contrary.

> This release needed to be 100% safe.  Not 99.99%, 100%.

We would never release anything if we required 100% safety. That should not be 
the requirement. The requirement should be that the release be significantly 
better, and that it is worth the cost of making the release.

I'm not trying to relitigate 25.3's contents, and 25.3 is OK with me. This 
discussion is really more about we should do with the next emergency release, 
not the previous one. Among other things if we're just going to do single 
cherry-picked patches then there seems little point to maintaining the emacs-25 
branch any more.

Re: Why wasn't the 25.3 release based on the then-head of the emacs-25 branch?

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Sat, 16 Sep 2017 13:20:41 -0700
> 
> > This release needed to be 100% safe.  Not 99.99%, 100%.
> 
> We would never release anything if we required 100% safety.

We just did.

> The requirement should be that the release be significantly 
> better, and that it is worth the cost of making the release.

Normally, yes.  But not for an emergency release that fixes a security
vulnerability: that one must not have any issues or problems, because
people will start using it as soon as it is available.

> This discussion is really more about we should do with the next
> emergency release, not the previous one.

If it is bout that, I didn't see it.  Seriously discussing what to do
next time should not be about the details of what was done this time,
because the specific circumstances will certainly be different.

Re: Why wasn't the 25.3 release based on the then-head of the emacs-25 branch?

[Paul Eggert]

Eli Zaretskii wrote:
> But not for an emergency release that fixes a security
> vulnerability: that one must not have any issues or problems

It is impossible to achieve 100% safety in any realistic release. Even the 25.3 
release, which was simple and straightforward, had at least one minor point of 
confusion in its release announcement that could cause problems for people 
running older Emacs versions. This was because (as Glenn noted) the announcement 
gave the wrong version number for when the remote-exploit bug was introduced.

25.3 is good enough, and it's in the books now. But moving forward, we can do 
better the next time we have an emergency release. As things stand the process 
takes too much time, and requires too much manual work, and apparently only one 
person can actually make a release. These are all things that can be improved on.

Re: Why wasn't the 25.3 release based on the then-head of the emacs-25 branch?

[Eli Zaretskii]

> Cc: rgm@gnu.org, emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Sat, 16 Sep 2017 22:14:57 -0700
> 
>     But not for an emergency release that fixes a security
>     vulnerability: that one must not have any issues or problems
> 
> It is impossible to achieve 100% safety in any realistic release. Even the 25.3 release, which was simple and straightforward, had at least one minor point of confusion in its release announcement that could cause problems for people running older Emacs versions. This was because (as Glenn noted) the announcement gave the wrong version number for when the remote-exploit bug was introduced.

I think there's a misunderstanding of what "100% safe" means in this
context.  It means zero issues beyond those 25.2 had, that's all.
Yes, the glitch with the version number is unfortunate, but it doesn't
affect the code in any way, and even its impact on documentation is
insignificant at worst.  So the goal was reached, for all we know now.

> moving forward, we can do better the next time we have an emergency release. As things stand the process takes too much time, and requires too much manual work, and apparently only one person can actually make a release. These are all things that can be improved on. 

Focusing on how to do better next time is indeed a worthier investment
of our time and energy.  But doing so shouldn't involve repeated
arguments about the details of the 25.3 release -- that is a way of
preparing ourselves for the "previous war".  Next time the details of
the situation will almost certainly be different, and therefore all
the deliberations on what patch should or shouldn't have been part of
25.3 will not help us being better.

Having more people who can upload tarballs to the GNU site is one
obvious improvement.  But it's not enough, as it only accounts for
one-day delay of the 25.3 release.  Other ways to improve our process
are much more important -- and they were not even raised yet, let
alone discussed and decided.  For example:

>     Even if the Debian distro could be considered ample testing (and I'm
>     not sure it could, as Debian AFAIR are generally not on the bleeding
>     edge with system features, library and kernel versions, etc.)
> 
> As a practical matter, Debian has waaayy more testing than we do. If the patch worked for them that's a strong signal, stronger than any information we have to the contrary.

Maybe so, but Debian testing is limited to Debian systems, and Emacs
is not supposed to work well on Debian alone.  If we want to judge
whether specific changes post the last official release can be
considered safe for inclusion in emergency releases, we need to
identify the sample of GNU/Linux distributions which we could use in
similar decisions, and then have simple and reliable procedures for
finding out how many of these distros use a specific patch from the
Emacs repository, and for how long.  We then need to record these
procedures on one of the files in admin/, so that whoever is in charge
of an emergency release could quickly apply those procedures and make
the decisions.

Would someone want to come up with such a procedure, and provide the
supporting data?

> Among other things if we're just going to do single cherry-picked patches then there seems little point to maintaining the emacs-25 branch any more. 

The emacs-25 branch is not maintained since May.  It stopped being
updated when it became clear that no one is interested in releasing
Emacs 25.3 with a few bugs of 25.2 fixed.  It saw a few commits now as
fallout of this emergency release, that's all, and will fall back into
its limbo from now on.  So I'm not really sure what you are alluding
to here.

Re: Why wasn't the 25.3 release based on the then-head of the emacs-25 branch?

[Paul Eggert]

Eli Zaretskii wrote:
> Having more people who can upload tarballs to the GNU site is one
> obvious improvement.

I'll volunteer to do that, if nobody else wants to. I generally delegate this 
task to others, in the other projects I contribute to, and would prefer that 
here too. But if we're shorthanded, I'll step in.

> it only accounts for one-day delay of the 25.3 release.

25.3 was decided on by Sat, 9 Sep 2017 10:48:13 +0300 (this is the timestamp on 
private email from you). The release announcement was sent Mon, 11 Sep 2017 
22:52:00 +0200. That's a delay of about 2.5 days to turn the crank. We should do 
better next time. A reasonable goal is to have a delay of at most one hour for 
turning the crank to generate a new release. Ideally you would do it, since you 
make the decision and that would avoid email lag; but it should be done 
reasonably quickly even if someone else has to do it.

> Other ways to improve our process are much more important

Yes, as the gap between the original report (Mon, 4 Sep 2017 19:26:01 UTC) and 
the 25.3 decision was about 4.5 days, and this is bigger than 2.5 days and so 
offers more opportunity for streamlining.

As I think I mentioned, I had offers via savannah-hackers to help review Emacs 
security bug reports and proposed patches faster, and I'm inclined to take up 
these offers. I don't know of any other proposal for significantly speeding up 
this part of the process.

We should be able to do better than a one-week total turnaround for this sort of 
thing.

> Debian testing is limited to Debian systems

Sure, but the patches in question were portable and Debian testing is quite a 
strong signal that they will work elsewhere.  Obviously there is a judgment call 
here (how do we know the patches are portable? we have to read them) but that's 
OK. Going forward, we needn't come up with an elaborate bureaucracy here, nor do 
we have the resources for that.

> The emacs-25 branch is not maintained since May.  It stopped being
> updated when it became clear that no one is interested in releasing
> Emacs 25.3 with a few bugs of 25.2 fixed.

OK, I'll stop updating it. If we need another emergency patch for Emacs 25, 
should we use the emacs-25-3 branch, or create a new branch emacs-25-4? It'd be 
nice to know that now so that we avoid confusion later. I suggest that we keep 
using emacs-25-3 rather than creating a new branch. In hindsight perhaps we 
should have named this branch emacs-25-security.

Re: Why wasn't the 25.3 release based on the then-head of the emacs-25 branch?

[Eli Zaretskii]

> Cc: rgm@gnu.org, emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Sun, 17 Sep 2017 10:40:45 -0700
> 
> Eli Zaretskii wrote:
> > Having more people who can upload tarballs to the GNU site is one
> > obvious improvement.
> 
> I'll volunteer to do that, if nobody else wants to. I generally delegate this 
> task to others, in the other projects I contribute to, and would prefer that 
> here too. But if we're shorthanded, I'll step in.

Thanks.

> > it only accounts for one-day delay of the 25.3 release.
> 
> 25.3 was decided on by Sat, 9 Sep 2017 10:48:13 +0300 (this is the timestamp on 
> private email from you). The release announcement was sent Mon, 11 Sep 2017 
> 22:52:00 +0200. That's a delay of about 2.5 days to turn the crank.

That time included the time to make the tarball and test it.  The
delay between the decision and when Nicolas started to work on the
tarball was one day.

> As I think I mentioned, I had offers via savannah-hackers to help review Emacs 
> security bug reports and proposed patches faster, and I'm inclined to take up 
> these offers.

I don't think I understand what that means.  How can people outside of
the project be charged with reviewing our bugs and patches?  How will
that work in practice?  And why wouldn't those people speak up here
and work with us within our procedures?

> > Debian testing is limited to Debian systems
> 
> Sure, but the patches in question were portable and Debian testing is quite a 
> strong signal that they will work elsewhere. Obviously there is a judgment call 
> here (how do we know the patches are portable? we have to read them) but that's 
> OK. Going forward, we needn't come up with an elaborate bureaucracy here, nor do 
> we have the resources for that.

No one was arguing for additional bureaucracy.  What we need is data
and procedures to make the decision quickly and without ado.  And I
don't think Debian alone should be the basis, we need to be able to
quickly see which of our unreleased patches are backported by popular
distributions, including, but not limited to Debian.

> If we need another emergency patch for Emacs 25, 
> should we use the emacs-25-3 branch, or create a new branch
> emacs-25-4?

I don't think we can decide now, it will depend on the specific
circumstances when (if) such issues arise.

> I suggest that we keep using emacs-25-3 rather than creating a new
> branch.

That's definitely a possibility that should be one of the most
favorable ones, if not the favorable.  But it's impossible to make a
decision until the specific details of the issue are known.

Re: Why wasn't the 25.3 release based on the then-head of the emacs-25 branch?

[Paul Eggert]

Eli Zaretskii wrote:
> That time included the time to make the tarball and test it.

If making the tarball and testing it takes 1.5 days, then that was 20% of the 
overall delay last time, and there is good opportunity for speeding up the 
process. Such a process should take minutes, not hours.

> How can people outside of
> the project be charged with reviewing our bugs and patches?

These are people quite knowledgeable about security and software maintenance. 
They can be a good source for security reviews. It's another set of eyes, with 
an outside perspective, and that is helpful.

> why wouldn't those people speak up here
> and work with us within our procedures?

They're busy. Also, we haven't exactly been soliciting or welcoming their input. 
The most recent emergency release had a bit of an NIH feel about it.

> No one was arguing for additional bureaucracy.  What we need is data
> and procedures

Whatever it's called it's more work, and we lack the resources to do it. Maybe 
we can look at two disparate releases (Debian and Fedora, say). Above that there 
are diminishing returns. Outside reviewers could help here (some are Fedora 
experts).

Re: Why wasn't the 25.3 release based on the then-head of the emacs-25 branch?

[Eli Zaretskii]

> Cc: rgm@gnu.org, emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Sun, 17 Sep 2017 15:44:36 -0700
> 
>     That time included the time to make the tarball and test it.
>
> If making the tarball and testing it takes 1.5 days, then that was 20% of the overall delay last time, and there is good opportunity for speeding up the process. Such a process should take minutes, not hours.

The process of producing the tarball once you invoke make-dist is
indeed very short.  But the process of checking-out from Git, setting
the version number, committing the changes and the corresponding tag,
making the tarball itself, verifying its correctness, uploading it to
the GNU servers, then downloading it to several different systems, and
verifying the build--that process takes longer, especially if done by
several people who have their lives and live in different timezones.

Much of this involves tedious manual procedures, and is thus
error-prone.  It would be nice to automate at least some of that
(tarball content verification comes to mind), which could shorten the
time and eliminate the potential for errors.  For example, in the case
of Emacs 25.3, the original tarball included 2 minor mistakes, which
required to re-upload it after fixing them.

But whatever improvements we introduce, I think it's naïve to assume
time frames of minutes for these activities, unless we want to give up
QA.  And giving up QA would be a mistake, IMO, since emergency
releases that fix grave problems must not themselves be problematic;
if they are, that would require additional fixup releases and prolong
the time even more.

>     How can people outside of
>     the project be charged with reviewing our bugs and patches?
> 
> These are people quite knowledgeable about security and software maintenance. They can be a good source for security reviews. It's another set of eyes, with an outside perspective, and that is helpful.
> 
>     why wouldn't those people speak up here
>     and work with us within our procedures?
> 
> They're busy. Also, we haven't exactly been soliciting or welcoming their input. The most recent emergency release had a bit of an NIH feel about it.

I still don't understand what you are suggesting, in practical terms,
and how will this be different from the current procedures, where
anyone could raise a security issue and propose a solution.

>     No one was arguing for additional bureaucracy.  What we need is data
>     and procedures
> 
> Whatever it's called it's more work, and we lack the resources to do it. Maybe we can look at two disparate releases (Debian and Fedora, say). Above that there are diminishing returns. Outside reviewers could help here (some are Fedora experts). 

Are Debian and Fedora indeed enough?  What about Red Hat?  What about
Arch Linux?  I don't have the answers, but we should decide on the
representative sample of the distributions based on some principles we
agree upon, and we should do it ahead of time.

The next question is, given the sample of distributions, how does one
figure out which ones of them include a given Emacs changeset, in
which versions of Emacs, and since what time.  Asking some experts
about this is possible, but it runs the risk that those experts are
unavailable or incommunicado when we need them, so if some databases
can be searched instead, that's better.

And of course all of that should be recorded in some file in admin/,
so that no one has to remember any of this when the time comes.

Re: Why wasn't the 25.3 release based on the then-head of the emacs-25 branch?

[Paul Eggert]

Eli Zaretskii wrote:
> it's naïve to assume
> time frames of minutes for these activities, unless we want to give up
> QA.

It's routine to do the kind of QA that you mention in minutes for projects 
Emacs's size. We're not up to that now, but it's reasonable to make it a goal, 
if this sort of thing is important to us. At any rate the procedure could be 
streamlined considerably compared to what it was last time. Whether it should 
take 10 minutes or 2 hours is not something we need to decide now.

> I still don't understand what you are suggesting, in practical terms,
> and how will this be different from the current procedures, where
> anyone could raise a security issue and propose a solution.

We could try having a better relationship with Debian and one or two others, so 
that the patches they consider to be security issues cause us to consider 
issuing new versions quickly. And we could be more proactive in sending our 
potential security patches to them early in our review process.

> Are Debian and Fedora indeed enough?  What about Red Hat?

Fedora is Red Hat's early version, so we needn't worry about Red Hat separately.

> What about Arch Linux?

They wouldn't make my cut. Others of us might step up to be a liaison. openSUSE 
is also a plausible candidate for that.

> given the sample of distributions, how does one
> figure out which ones of them include a given Emacs changeset, in
> which versions of Emacs, and since what time.

This info is all public now, at least for Debian and Red Hat.

Re: Why wasn't the 25.3 release based on the then-head of the emacs-25 branch?

[Eli Zaretskii]

> Cc: rgm@gnu.org, emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Mon, 18 Sep 2017 10:10:21 -0700
> 
> Eli Zaretskii wrote:
> > it's naïve to assume
> > time frames of minutes for these activities, unless we want to give up
> > QA.
> 
> It's routine to do the kind of QA that you mention in minutes for projects 
> Emacs's size. We're not up to that now, but it's reasonable to make it a goal, 
> if this sort of thing is important to us.

With the right organization and equipment available at suitable SLA,
sure.  But we are very far from that point, IMO.

> At any rate the procedure could be streamlined considerably compared
> to what it was last time.

Indeed, working in that direction would be a good progress.  I don't
think we need to get the time down to minutes, though, as that would
probably require measures that are impractical in our conditions.

> We could try having a better relationship with Debian and one or two others, so 
> that the patches they consider to be security issues cause us to consider 
> issuing new versions quickly. And we could be more proactive in sending our 
> potential security patches to them early in our review process.

If someone could take that upon themselves, sure.

> > Are Debian and Fedora indeed enough?  What about Red Hat?
> 
> Fedora is Red Hat's early version, so we needn't worry about Red Hat separately.
> 
> > What about Arch Linux?
> 
> They wouldn't make my cut. Others of us might step up to be a liaison. openSUSE 
> is also a plausible candidate for that.

What do others think about the set of distros we should use?

> > given the sample of distributions, how does one
> > figure out which ones of them include a given Emacs changeset, in
> > which versions of Emacs, and since what time.
> 
> This info is all public now, at least for Debian and Red Hat.

OK, so it would be good to have the pointers in admin/ somewhere.

Thanks.