Move to a cadence release model?

Move to a cadence release model?

[John Yates]

[-- Attachment #1: Type: text/plain, Size: 750 bytes --]

With a new master at the helm and various changes being contemplated I
would like to see some discussion of moving to a cadence release model.

I have been impressed with open source projects that have made the change.
I am now employed at Mathworks which ships mission critical software to
very large enterprise customers on a 6 month cadence.

The biggest shift I see is away from wondering when the correct collection
of features, bug fixes, whatever have been accumulated to whether those
that have been accumulated are individually sufficiently cooked to ship.
Developers feel less urge to squeeze a not fully baked feature into the
current release when they can count on the next opportunity being only a
cadence interval in the future.

/john

[-- Attachment #2: Type: text/html, Size: 848 bytes --]

Re: Move to a cadence release model?

[John Wiegley]

>>>>> John Yates <john@yates-sheets.org> writes:

> With a new master at the helm and various changes being contemplated I would
> like to see some discussion of moving to a cadence release model.

Hi John,

I haven't worked under an Agile regime before; can you describe what you think
this would look like for Emacs development in particular?

I have used the "Git flow" model in the past to good ends. It too could have
avoided the "long pause" between 24.5 and 25.1, and the necessity of
periodically freezing master before a release.

John

Re: Move to a cadence release model?

[Eli Zaretskii]

> From: John Wiegley <jwiegley@gmail.com>
> Date: Tue, 10 Nov 2015 06:28:01 -0800
> Cc: Emacs developers <emacs-devel@gnu.org>
> 
> I have used the "Git flow" model in the past to good ends. It too could have
> avoided the "long pause" between 24.5 and 25.1, and the necessity of
> periodically freezing master before a release.

We could avoid the feature freeze altogether, if we wanted, and
instead cut the release branch and work on the release there without
freezing master.

Re: Move to a cadence release model?

[John Wiegley]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

> We could avoid the feature freeze altogether, if we wanted, and instead cut
> the release branch and work on the release there without freezing master.

That would be my preference. Let's cut release-25.1 this upcoming Friday night
at the freeze date.

It does mean a commitment by developers to use and test with this branch until
the release is out. But it allows development on 'master' to continue without
any artificial pause, or forcing everything that could have gone onto master
into feature branches.

John

Release process (was Re: Move to a cadence release model?)

[Stephen Leake]

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> Eli Zaretskii <eliz@gnu.org> writes:
>
>> We could avoid the feature freeze altogether, if we wanted, and instead cut
>> the release branch and work on the release there without freezing master.
>
> That would be my preference. Let's cut release-25.1 this upcoming Friday night
> at the freeze date.
>
> It does mean a commitment by developers to use and test with this branch until
> the release is out. 

Which is precisely why we have a feature freeze phase; it enforces this
desire.

I know I would be _very_ tempted to ignore the release branch, to keep
working on my latest Cool Feature instead.

If I know I have to wait for a release before I can merge to master
again, I'll work on the release as much as I can.

-- 
-- Stephe

Re: Release process (was Re: Move to a cadence release model?)

[John Wiegley]

>>>>> Stephen Leake <stephen_leake@stephe-leake.org> writes:

> Which is precisely why we have a feature freeze phase; it enforces this
> desire.

> I know I would be _very_ tempted to ignore the release branch, to keep
> working on my latest Cool Feature instead.

> If I know I have to wait for a release before I can merge to master again,
> I'll work on the release as much as I can.

Yeah, that's a valid management question. Either way, I can't stop anyone from
ignoring the release branch and working on their feature branch.

Also, if the release branch doesn't close its bugs and reach stability, it
will delay the next release (when your master changes reach the world), so
that is some incentive, yes?

John

Re: Release process (was Re: Move to a cadence release model?)

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Cc: john@yates-sheets.org,  emacs-devel@gnu.org
> Date: Tue, 10 Nov 2015 18:17:47 -0600
> 
> John Wiegley <jwiegley@gmail.com> writes:
> 
> >>>>>> Eli Zaretskii <eliz@gnu.org> writes:
> >
> >> We could avoid the feature freeze altogether, if we wanted, and instead cut
> >> the release branch and work on the release there without freezing master.
> >
> > That would be my preference. Let's cut release-25.1 this upcoming Friday night
> > at the freeze date.
> >
> > It does mean a commitment by developers to use and test with this branch until
> > the release is out. 
> 
> Which is precisely why we have a feature freeze phase; it enforces this
> desire.

We cannot enforce it.

> I know I would be _very_ tempted to ignore the release branch, to keep
> working on my latest Cool Feature instead.
> 
> If I know I have to wait for a release before I can merge to master
> again, I'll work on the release as much as I can.

These considerations will become valid only when we have enough
developers paying attention to bugs that are reported.  (That includes
you, Stephen, btw.)  The way things are now, almost no one does, so
the feature freeze period is just a waste of time from my POV.

Re: Release process (was Re: Move to a cadence release model?)

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stephen Leake <stephen_leake@stephe-leake.org>

>> Which is precisely why we have a feature freeze phase; it enforces this
>> desire.
>
> We cannot enforce it.

Well, the feature freeze encourages developers to work on the release
more than a non-feature freeze model does. At least, it works that way
for me.

>> I know I would be _very_ tempted to ignore the release branch, to keep
>> working on my latest Cool Feature instead.
>> 
>> If I know I have to wait for a release before I can merge to master
>> again, I'll work on the release as much as I can.
>
> These considerations will become valid only when we have enough
> developers paying attention to bugs that are reported.  (That includes
> you, Stephen, btw.)  

Yes. I don't scan the bug tool for bugs that I might be able to work on;
sometimes it seems I should. For now, I rely on someone interested in
the bug emailing me if they think I could help.

Is there a way to get an email for every new bug? I don't see anything
on the debbugs pages, but I didn't look very hard. I'm curious how much
traffic that would be.

During the last feature freeze, there were reminders on this list of the
bugs that were deemed release-critical. I looked at all of those bugs,
and decided I could not usefully contribute to fixing them.

This time around, I would argue that "fix the byte-compiler errors in
cedet/*" is a release-critical bug, and I will work on that. I have
already offered to, but I'm waiting for Eric to organize the effort. It
needs to wait until he finishes his final merge to master.

There may be other release-critical bugs that I can usefully work on.

I don't have statistics on how well the feature-freeze model works in
getting release-critical bugs fixed. Have we had other release models in
the past? did they work any better?

I do know that these same discussions were had at the start of the last
feature-freeze. So a document that records the rationale for the release
process would be useful.

-- 
-- Stephe

Re: Release process (was Re: Move to a cadence release model?)

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Cc: john@yates-sheets.org,  emacs-devel@gnu.org
> Date: Wed, 11 Nov 2015 04:45:18 -0600
> 
> >> From: Stephen Leake <address@hidden>
> 
> >> Which is precisely why we have a feature freeze phase; it enforces this
> >> desire.
> >
> > We cannot enforce it.
> 
> Well, the feature freeze encourages developers to work on the release
> more than a non-feature freeze model does. At least, it works that way
> for me.

Alternatively, someone else could simply take a break from working on
Emacs until the freeze is over.

> >> I know I would be _very_ tempted to ignore the release branch, to keep
> >> working on my latest Cool Feature instead.
> >> 
> >> If I know I have to wait for a release before I can merge to master
> >> again, I'll work on the release as much as I can.
> >
> > These considerations will become valid only when we have enough
> > developers paying attention to bugs that are reported.  (That includes
> > you, Stephen, btw.)  

(Upon re-reading, I apologize for being so blunt.  It just feels too
lonely there, at times.)

> Yes. I don't scan the bug tool for bugs that I might be able to work on;
> sometimes it seems I should. For now, I rely on someone interested in
> the bug emailing me if they think I could help.

I don't thibk this will work efficiently.

> Is there a way to get an email for every new bug?

Yes, subscribe to bug-gnu-Emacs mailing list.

> I'm curious how much traffic that would be.

You can see that in the archives.  About 30 messages per day on the
average, maybe.

> During the last feature freeze, there were reminders on this list of the
> bugs that were deemed release-critical.

That's the last resort.  There are gobs of bugs that don't block
anything, and some of them are left alone for far too long.

Even a prompt response that just says the bug was reproduced (or not),
and ideally also with results of some initial investigation and/or
request from the OP to provide more details or try something -- even
this would be progress.

And then there are small patches submitted there -- review and comment
will be appreciated.  Patches that are no-brainers, e.g., fixing a
typo or some other obvious mistake, should be pushed if there are no
comments after a week, say.

> There may be other release-critical bugs that I can usefully work on.

The problem IMO is not the release-critical bugs, it's all the rest.

> I don't have statistics on how well the feature-freeze model works in
> getting release-critical bugs fixed.

It doesn't work at all.  Release-critical bugs normally appear near
the end of the pretest.  Or at least they become important only then.

By contrast, I'm not talking about bug fixing during the pretest, I'm
talking about routine attention to the reported bugs.

Too few people taking care of bug reports, was: Re: Release process (was Re: Move to a cadence release model?)

[Dmitry Gutov]

On 11/11/2015 05:43 PM, Eli Zaretskii wrote:
>>> These considerations will become valid only when we have enough
>>> developers paying attention to bugs that are reported.  (That includes
>>> you, Stephen, btw.)
>
> (Upon re-reading, I apologize for being so blunt.  It just feels too
> lonely there, at times.)

I think a significant part of that problem is self-imposed.

Personally, I try to pay attention to bugs that are related to code that 
I at least have touched at some point, or bugs that affect me directly, 
but it seems there aren't too many of those. And I don't think it's 
reasonable to expect much more of any contributor.

Over time, we've put a lot of conditions on Emacs development. There's a 
lot of code in the core, some of which is used by only marginal 
fractions of our users and has no one personally responsible for it. Yet 
we feel obliged to keep it in Emacs, because backward compatibility and 
careful deprecation policy. Even though we're lacking in developers.

Our bug tracker is peculiar, and on its own turns many less experienced 
users away. Users that could participate in triaging bugs, at least, if 
not writing patches. Maybe trying out submitted patches, too.

Yet over several discussions that happened in the past, it was decided 
to keep it, because the ability to interact with the bug tracker via 
email (and some other advantages, though I'm not sure which ones) has 
been deemed more valuable than a functional HTML-based interface that 
allows one to manage bugs in the browser, leave comments, etc, which is 
expected by most users these days.

Re: Too few people taking care of bug reports,

[John Wiegley]

>>>>> Dmitry Gutov <dgutov@yandex.ru> writes:

> Personally, I try to pay attention to bugs that are related to code that I
> at least have touched at some point, or bugs that affect me directly, but it
> seems there aren't too many of those. And I don't think it's reasonable to
> expect much more of any contributor.

I, at least, don't expect anything more than that from contributors.

> Over time, we've put a lot of conditions on Emacs development. There's a lot
> of code in the core, some of which is used by only marginal fractions of our
> users and has no one personally responsible for it. Yet we feel obliged to
> keep it in Emacs, because backward compatibility and careful deprecation
> policy. Even though we're lacking in developers.

The upcoming clarifications on ELPA policy may well lead to reducing the
surface area of primary Emacs development. This will serve to focus the bug
load of what is "core", although we'll still be maintaining some of the ELPA
packages that don't have their own dedicated maintainers.

> Our bug tracker is peculiar, and on its own turns many less experienced
> users away. Users that could participate in triaging bugs, at least, if not
> writing patches. Maybe trying out submitted patches, too.

I very muchq agree with this. I have used many, _many_ bug trackers in the
past, but I'm finding that debbugs is inhibiting my ability to interact
conveniently with our bug database. It's hard to search for what I'm looking
for, it's hard to navigate the bug tracker from within Emacs (is it just me,
or is debbugs.el kind of terrible?), and unless I'm missing something, I can't
edit a bug after I've found it through the web interface: I have to go to my
mail reader in order to make changes to the bug via e-mail.

It could also be that I just haven't discovered all of the tricks that Eli
might know. Is there any documentation I should be reading beyond what's on
the GNU bug tracker page?

John

Re: Too few people taking care of bug reports,

[Dmitry Gutov]

On 11/11/2015 10:50 PM, John Wiegley wrote:

> The upcoming clarifications on ELPA policy may well lead to reducing the
> surface area of primary Emacs development. This will serve to focus the bug
> load of what is "core", although we'll still be maintaining some of the ELPA
> packages that don't have their own dedicated maintainers.

Still, though, the bug reports for ELPA packages that don't have any 
other upstream are supposed to go into the Emacs bug tracker anyway.

It's not like upon moving a package to ELPA, we can immediately declare 
it obsolete. Right?

> I very much agree with this. I have used many, _many_ bug trackers in the
> past, but I'm finding that debbugs is inhibiting my ability to interact
> conveniently with our bug database. It's hard to search for what I'm looking

It's also slooow.

> for, it's hard to navigate the bug tracker from within Emacs (is it just me,
> or is debbugs.el kind of terrible?),

No comments.

> and unless I'm missing something, I can't
> edit a bug after I've found it through the web interface: I have to go to my
> mail reader in order to make changes to the bug via e-mail.

It has an email interface, and a SOAP interface (sigh).

> It could also be that I just haven't discovered all of the tricks that Eli
> might know. Is there any documentation I should be reading beyond what's on
> the GNU bug tracker page?

debbugs.el has a feature like sending control messages to the bug 
tracker, which allows to close/unclose/merge bugs, etc. I've been able 
to do something useful with it once or twice.

Re: Too few people taking care of bug reports,

[John Wiegley]

>>>>> Dmitry Gutov <dgutov@yandex.ru> writes:

> Still, though, the bug reports for ELPA packages that don't have any other
> upstream are supposed to go into the Emacs bug tracker anyway.

> It's not like upon moving a package to ELPA, we can immediately declare it
> obsolete. Right?

This is true. At least it will get a tag so that we know it's in ELPA. ELPA
packages have the advantage of being able to deliver bug fixes post-release
very easily. Anything that's in core has a much longer delay, and so is more
important to fix sooner for that reason.

John

Re: Too few people taking care of bug reports,

[Eli Zaretskii]

> From: John Wiegley <jwiegley@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Stephen Leake <stephen_leake@stephe-leake.org>,  emacs-tangents@gnu.org,  john@yates-sheets.org, Richard Stallman <rms@gnu.org>
> Date: Wed, 11 Nov 2015 12:50:53 -0800
> 
> It could also be that I just haven't discovered all of the tricks that Eli
> might know. Is there any documentation I should be reading beyond what's on
> the GNU bug tracker page?

I don't have any tricks.  I'm subscribed to the bug mailing list and
read each and every bug report.  If it's something I decide to look
into, I leave the mail in my inbox and get to that when I have time.
That's all.

Taking care of a bug, for me, could be as simple as reproducing it (or
not) and posting the results, or it could be a request for the OP to
try something and report back, or it could be a full-fledged debug
session.

Either way, all I need from the tracker is the data about the bug.
Whatever debbugs does, it cannot screw that up.

Re: Too few people taking care of bug reports, was: Re: Release process (was Re: Move to a cadence release model?)

[Eli Zaretskii]

> Cc: emacs-tangents@gnu.org, john@yates-sheets.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Wed, 11 Nov 2015 22:39:25 +0200
> 
> On 11/11/2015 05:43 PM, Eli Zaretskii wrote:
> >>> These considerations will become valid only when we have enough
> >>> developers paying attention to bugs that are reported.  (That includes
> >>> you, Stephen, btw.)
> >
> > (Upon re-reading, I apologize for being so blunt.  It just feels too
> > lonely there, at times.)
> 
> I think a significant part of that problem is self-imposed.

Not sure I understand what you mean by that.  Self-imposed by me?

> Personally, I try to pay attention to bugs that are related to code that 
> I at least have touched at some point, or bugs that affect me directly, 
> but it seems there aren't too many of those. And I don't think it's 
> reasonable to expect much more of any contributor.

I disagree.  IME, frequently just looking or stepping through
unfamiliar code can reveal bugs whose reasons we can easily understand
and fix.  Just a few minutes ago I had this experience once more, see
bug#21881.  I assure you I knew nothing at all about mm-url.el, and
still don't.  Still, it took me just a few minutes to understand why
EWW barfs and see the solution that I'm sure is right.

You should try this some time.  I think everybody here should.

Worst that could happen is that you will only be able to add some
non-trivial information to the bug report, based on what you saw,
without actually finding a solution.  But even that alone could allow
someone else to suggest a solution.  That, too, have happened to me.

Bottom line is: people like you and me (and many others here) know
quite a lot about Emacs and about debugging, and can find solutions to
many bugs even in unfamiliar code.

> Our bug tracker is peculiar, and on its own turns many less experienced 
> users away. Users that could participate in triaging bugs, at least, if 
> not writing patches. Maybe trying out submitted patches, too.

Yes, triage would help as well.

As for the bug tracker, I don't see how it could have such a
detrimental effect on potential contributors.  It's just a mailing
list, not unlike this one.  In addition, we have an Emacs package that
interacts with the tracker, so one could work with bugs without ever
looking at either the Web forms or the email messages.

I'm not saying there's no place for improvements, but I cannot imagine
that the reason for such a small number of people who work on bugs is
the bug tracker.  I think it's more likely the fear of diving into
unfamiliar code.

Re: Too few people taking care of bug reports,

[John Wiegley]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

> In addition, we have an Emacs package that interacts with the tracker, so
> one could work with bugs without ever looking at either the Web forms or the
> email messages.

How do you use debbugs.el to query for the bugs you look at each day, Eli?

John

Re: Too few people taking care of bug reports,

[Eli Zaretskii]

> From: John Wiegley <jwiegley@gmail.com>
> Cc: Dmitry Gutov <dgutov@yandex.ru>,  emacs-tangents@gnu.org,  stephen_leake@stephe-leake.org,  john@yates-sheets.org
> Date: Wed, 11 Nov 2015 13:15:22 -0800
> 
> How do you use debbugs.el to query for the bugs you look at each day, Eli?

I don't use debbugs.el.  I just mentioned it for the benefit of those
who might prefer it to the email and Web interfaces.

Re: Too few people taking care of bug reports, was: Re: Release process (was Re: Move to a cadence release model?)

[Dmitry Gutov]

On 11/11/2015 11:10 PM, Eli Zaretskii wrote:

>> I think a significant part of that problem is self-imposed.
>
> Not sure I understand what you mean by that.  Self-imposed by me?

By the Emacs developers, collectively, over time. I don't know who's 
responsible for each issue in particular.

> I disagree.  IME, frequently just looking or stepping through
> unfamiliar code can reveal bugs whose reasons we can easily understand
> and fix.  Just a few minutes ago I had this experience once more, see
> bug#21881.  I assure you I knew nothing at all about mm-url.el, and
> still don't.  Still, it took me just a few minutes to understand why
> EWW barfs and see the solution that I'm sure is right.

I'm not saying I couldn't do that, but there's a limit to the number of 
areas I want to be familiar with. I also have limited of time and other 
projects, starving for attention.

I'm also currently exhausted by trying to explain the difference between 
"project" and "libraries", to some people, over a zillion email messages.

> You should try this some time.  I think everybody here should.

The above is the primary method I do get acquainted with new code.

> Bottom line is: people like you and me (and many others here) know
> quite a lot about Emacs and about debugging, and can find solutions to
> many bugs even in unfamiliar code.

Been there, done that. Especially in third-party code, where it's much 
easier to discuss the issues, explore the code, etc, everything without 
closing the browser.

> As for the bug tracker, I don't see how it could have such a
> detrimental effect on potential contributors.  It's just a mailing
> list, not unlike this one.

"Just a mailing list" is already a big barrier. First, most users are 
unaccustomed to dealing with mailing lists. Second, many that are, still 
expect a bug tracker to be something more.

> I'm not saying there's no place for improvements, but I cannot imagine
> that the reason for such a small number of people who work on bugs is
> the bug tracker.  I think it's more likely the fear of diving into
> unfamiliar code.

How do you explain the low number of users triaging the bugs?

Re: Too few people taking care of bug reports, was: Re: Release process (was Re: Move to a cadence release model?)

[Eli Zaretskii]

> Cc: emacs-tangents@gnu.org, stephen_leake@stephe-leake.org,
>  john@yates-sheets.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Wed, 11 Nov 2015 23:23:45 +0200
> 
> > I'm not saying there's no place for improvements, but I cannot imagine
> > that the reason for such a small number of people who work on bugs is
> > the bug tracker.  I think it's more likely the fear of diving into
> > unfamiliar code.
> 
> How do you explain the low number of users triaging the bugs?

I just tried to do that, above: I think it's a kind of fear.

Re: Too few people taking care of bug reports, was: Re: Release process (was Re: Move to a cadence release model?)

[Dmitry Gutov]

On 11/11/2015 11:30 PM, Eli Zaretskii wrote:

>> How do you explain the low number of users triaging the bugs?
>
> I just tried to do that, above: I think it's a kind of fear.

But you don't have to dive into the code, just to triage a bug.

Re: Too few people taking care of bug reports, was: Re: Release process (was Re: Move to a cadence release model?)

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Wed, 11 Nov 2015 23:23:45 +0200
> Cc: emacs-tangents@gnu.org, stephen_leake@stephe-leake.org,
> 	john@yates-sheets.org
> 
> > I disagree.  IME, frequently just looking or stepping through
> > unfamiliar code can reveal bugs whose reasons we can easily understand
> > and fix.  Just a few minutes ago I had this experience once more, see
> > bug#21881.  I assure you I knew nothing at all about mm-url.el, and
> > still don't.  Still, it took me just a few minutes to understand why
> > EWW barfs and see the solution that I'm sure is right.
> 
> I'm not saying I couldn't do that, but there's a limit to the number of 
> areas I want to be familiar with. I also have limited of time and other 
> projects, starving for attention.
> 
> I'm also currently exhausted by trying to explain the difference between 
> "project" and "libraries", to some people, over a zillion email messages.
> 
> > You should try this some time.  I think everybody here should.
> 
> The above is the primary method I do get acquainted with new code.
> 
> > Bottom line is: people like you and me (and many others here) know
> > quite a lot about Emacs and about debugging, and can find solutions to
> > many bugs even in unfamiliar code.
> 
> Been there, done that. Especially in third-party code, where it's much 
> easier to discuss the issues, explore the code, etc, everything without 
> closing the browser.

Then it sounds like you agree with me, and we both are doing about the
bugs whatever we can, given our free resources.  Thanks, and please
keep up.

Re: Too few people taking care of bug reports, was: Re: Release process (was Re: Move to a cadence release model?)

[Dmitry Gutov]

On 11/11/2015 11:42 PM, Eli Zaretskii wrote:

> Then it sounds like you agree with me, and we both are doing about the
> bugs whatever we can, given our free resources.  Thanks, and please
> keep up.

With many of your points, yes. And thank you too.

bug policy (was Re: Release process)

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> > These considerations will become valid only when we have enough
>> > developers paying attention to bugs that are reported.  (That includes
>> > you, Stephen, btw.)  
>
> (Upon re-reading, I apologize for being so blunt.  It just feels too
> lonely there, at times.)

I didn't see it as offensive; just a gentle reminder.

I relate the lonely; I'm inordinately pleased when someone actually
posts to the ada-mode mailing list, even if it's just to report a bug
:).

>> During the last feature freeze, there were reminders on this list of the
>> bugs that were deemed release-critical.
>
> That's the last resort.  There are gobs of bugs that don't block
> anything, and some of them are left alone for far too long.

For your definition of "too long". As far as the Emacs project is
concerned, only release-critical bugs have been left "too long".

I don't see how it can be any other way, in a volunteer-staffed free
software project; no one is paid to fix bugs.

Another motivation for working on Emacs in general is the desire to work
on a project that has a useful product. But the connection between
fixing bugs and producing a useful Emacs is very tenuous in general; it
is very direct for some bugs (the ones that occur in common or important
use cases), but not for others (obsure bugs that are hard to reproduce).
Clearly lots of people find Emacs useful despite the outstanding bugs.

The bugs in common or important use cases tend to get fixed, because the
package authors are still around to care about them.

But we might learn something from triaging the existing bugs; I'll put
that on my list of things to think about while I'm  procrastinating
everything else on the list :).

> Even a prompt response that just says the bug was reproduced (or not),
> and ideally also with results of some initial investigation and/or
> request from the OP to provide more details or try something -- even
> this would be progress.

That is the level of support I provide for ada-mode. But that started in
the context of getting paid to use Ada, so it was easy to interpret that
as getting paid to maintain ada-mode. Now that I'm retired, I find
myself much less motivated to maintain ada-mode.

The lack of such support for Emacs in general is one reason I learned to
be proficient in elisp, so I could fix any bugs that I encountered in my
Emacs use. That's not an option for every user, but it is what I
recommend to any team that wants to use Emacs; make sure there is
reasonable access to an Emacs guru for this kind of support. 

> And then there are small patches submitted there -- review and comment
> will be appreciated.  Patches that are no-brainers, e.g., fixing a
> typo or some other obvious mistake, should be pushed if there are no
> comments after a week, say.

I understand the process; the issue is the motivation. Clearly, it is
far more fun to work on the next Cool Feature, than to chase bugs in
yesterday's Boring Feature :).

>> There may be other release-critical bugs that I can usefully work on.
>
> The problem IMO is not the release-critical bugs, it's all the rest.

Exactly why are those a problem?

Clearly each bug was some sort of problem to at least one user at one
time, but why is it an important problem for the Emacs project in
general?

Do potential users get turned off by the number of bugs?

We don't lose funding by ignoring bugs; what do we lose?

-- 
-- Stephe

Re: bug policy (was Re: Release process)

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Cc: john@yates-sheets.org,  emacs-devel@gnu.org
> Date: Wed, 11 Nov 2015 17:06:20 -0600
> 
> >> During the last feature freeze, there were reminders on this list of the
> >> bugs that were deemed release-critical.
> >
> > That's the last resort.  There are gobs of bugs that don't block
> > anything, and some of them are left alone for far too long.
> 
> For your definition of "too long". As far as the Emacs project is
> concerned, only release-critical bugs have been left "too long".

I don't think I agree.  A bug can be non-critical, in the sense that
we could live with it as we did until now, but still it prevents or
interferes with someone's use case.

Or maybe we should classify more bugs as release-critical ;-)

Speaking about which: what exactly is your definition of criticality?
The one we employ now is pretty conservative; for example, a bug that
existed in previous releases is not considered release-critical.  So,
for instance, bug#18997 won't be considered critical, although it
involves an abort, something that would be triaged as "critical" in
any software maintenance out there.

> I don't see how it can be any other way, in a volunteer-staffed free
> software project; no one is paid to fix bugs.

We cannot force volunteers do anything, but we can try persuading
them.  If we don't, then what do we need project leadership for?

> Another motivation for working on Emacs in general is the desire to work
> on a project that has a useful product. But the connection between
> fixing bugs and producing a useful Emacs is very tenuous in general; it
> is very direct for some bugs (the ones that occur in common or important
> use cases), but not for others (obsure bugs that are hard to reproduce).
> Clearly lots of people find Emacs useful despite the outstanding bugs.

Many bugs in the database are neither obscure nor hard to reproduce.
Those are the ones I'm talking about.

We could also use more aggressive triage, and clearly mark
non-reproducible and obscure bugs as such.  Instead, we let them hang
in limbo, which probably gives a wrong impression to someone who takes
an impartial look at our open bugs.

> The bugs in common or important use cases tend to get fixed, because the
> package authors are still around to care about them.

I think most of our packages don't have such authors around any more.

As for the "important" part, it's many times in the eyes of the
beholder.  A bug in a feature or package you never use will not appear
important to you, and it's not easy to understand its importance even
when the OP explains that.  On balance, I find this kind of approach
not useful; a much more useful approach IME is: "do I understand the
issue, and can I fix it"?

> But we might learn something from triaging the existing bugs; I'll put
> that on my list of things to think about while I'm  procrastinating
> everything else on the list :).

Thanks, that will be useful.  It's also possible we should use
"wontfix" and "unreproducible" tags more aggressively.

> > Even a prompt response that just says the bug was reproduced (or not),
> > and ideally also with results of some initial investigation and/or
> > request from the OP to provide more details or try something -- even
> > this would be progress.
> 
> That is the level of support I provide for ada-mode. But that started in
> the context of getting paid to use Ada, so it was easy to interpret that
> as getting paid to maintain ada-mode. Now that I'm retired, I find
> myself much less motivated to maintain ada-mode.

Alas, most of Emacs is in this orphaned state.  If veteran
contributors don't step forward to help more with even the initial
analysis of the reported bugs, the job of those few who are involved
in bug fixing becomes much harder, and the result will be more bugs
left unsolved.  If the bug is specific to a platform to which I have
no easy access, or involves some software which I cannot or don't want
to install, I can only marginally help, mostly by asking questions and
suggesting ideas.  If someone who does have access to a similar system
reproduces the bug and provides its analysis, it is frequently much
easier to propose a solution and even come up with a possible patch.
Few users can help with such analysis, but most active contributors
here have enough knowledge to do so.

> The lack of such support for Emacs in general is one reason I learned to
> be proficient in elisp, so I could fix any bugs that I encountered in my
> Emacs use. That's not an option for every user, but it is what I
> recommend to any team that wants to use Emacs; make sure there is
> reasonable access to an Emacs guru for this kind of support. 

I think we have here enough of such gurus, so I'm lobbying for them to
become more involved in handling bugs reported to the tracker.  The
main motivation, IMO, is twofold: (1) solving bugs helps your fellow
users, and (2) working on a bug almost always provides an ample
opportunity to learn something new about Emacs.

> > And then there are small patches submitted there -- review and comment
> > will be appreciated.  Patches that are no-brainers, e.g., fixing a
> > typo or some other obvious mistake, should be pushed if there are no
> > comments after a week, say.
> 
> I understand the process; the issue is the motivation. Clearly, it is
> far more fun to work on the next Cool Feature, than to chase bugs in
> yesterday's Boring Feature :).

Emacs features are usually anything but boring.  Most of the code was
written by first-class experts in design and implementation (myself
excluded), so reading and understanding it is a great investment, IMO.

> >> There may be other release-critical bugs that I can usefully work on.
> >
> > The problem IMO is not the release-critical bugs, it's all the rest.
> 
> Exactly why are those a problem?
> 
> Clearly each bug was some sort of problem to at least one user at one
> time, but why is it an important problem for the Emacs project in
> general?

A project that lags on fixing bugs doesn't have a bright future, IMO,
for several reasons.

For starters, users don't like that for obvious reasons, so such a
project most probably loses users it could have retained.

A more subtle reason is that working on bugs tends to proliferate
knowledge about the project's design and internals, so not working on
bugs runs a very real risk of slowly losing that knowledge and
increasing the number of areas where no one can make non-trivial
changes.  We are almost there already: notice the bugs with font
handling and with complex script layout.  I'm afraid the X display
back end, the GTK toolkit, and the Cairo-related code will follow, now
that Jan stepped down.  These are not obscure unimportant parts of
Emacs, far from that.  We need to grow experts in those areas soon
enough, or we will start accumulating grave bugs that no one can
solve.  How else do you gain such experts if not by encouraging
motivated contributors to start working on bugs in these areas?  The
only other way of gaining experts is to wait for them to magically
materialize, but that doesn't work very well IME.

> We don't lose funding by ignoring bugs; what do we lose?

We lose our future, I'd say.

Re: bug policy (was Re: Release process)

[John Wiegley]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

> Speaking about which: what exactly is your definition of criticality?

We may always need to be somewhere flexible here: Whatever the pride of the
developers would not allow to be delivered.

If I say "crash bugs", it wouldn't mean all crash bugs, since some are very
niche. If I say "crash bugs on GNU systems", we might let a terrible bug slip
through on Mac that could have been solved easily.

I have a feeling that most engineers, looking at a bug, will have a sense of
whether it will be embarrassing to ship with that problem. These should all be
promoted to release critical, and demoted only if several other people agree
that it shouldn't hold up the release.

> We cannot force volunteers do anything, but we can try persuading them. If
> we don't, then what do we need project leadership for?

The fact is, Emacs *could* slip into a permanent maintenance state, where we
never add a single new feature, and work only on bugs very slowly. I could
well imagine myself using 24.5 for the next 20 years just fine.

But there are annoyances for some that deserve feature work to resolve, and
this is what pushes Emacs forward. That, and great ideas that are just too
good not to implement.

It's hard to motivate people when the status quo is actually pretty darn good.
We'll have to think creatively about this, since I really would like our bug
database to reach zero at some point.

> We need to grow experts in those areas soon enough, or we will start
> accumulating grave bugs that no one can solve.

I think you've just made it hard for me to fall asleep tonight.

John

Re: bug policy (was Re: Release process)

[Eli Zaretskii]

> From: John Wiegley <jwiegley@gmail.com>
> Cc: Stephen Leake <stephen_leake@stephe-leake.org>,  emacs-devel@gnu.org,  john@yates-sheets.org
> Date: Thu, 12 Nov 2015 08:39:20 -0800
> 
> > We need to grow experts in those areas soon enough, or we will start
> > accumulating grave bugs that no one can solve.
> 
> I think you've just made it hard for me to fall asleep tonight.

Sorry about that.

Re: bug policy (was Re: Release process)

[John Wiegley]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

>> > We need to grow experts in those areas soon enough, or we will start
>> > accumulating grave bugs that no one can solve.
>> 
>> I think you've just made it hard for me to fall asleep tonight.

> Sorry about that.

I should have added the smiley, it was a joke. Mostly.

Eli, this is such a good point to be made that I'll have to think very hard
about it over the coming months. Thank you; I was only being facetious.

John

Re: bug policy (was Re: Release process)

[Eli Zaretskii]

> From: John Wiegley <jwiegley@gmail.com>
> Cc: stephen_leake@stephe-leake.org,  emacs-devel@gnu.org,  john@yates-sheets.org
> Date: Thu, 12 Nov 2015 09:50:35 -0800
> 
> >>>>> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> > We need to grow experts in those areas soon enough, or we will start
> >> > accumulating grave bugs that no one can solve.
> >> 
> >> I think you've just made it hard for me to fall asleep tonight.
> 
> > Sorry about that.
> 
> I should have added the smiley, it was a joke. Mostly.

Likewise here.

Re: Release process (was Re: Move to a cadence release model?)

[John Wiegley]

>>>>> Stephen Leake <stephen_leake@stephe-leake.org> writes:

> I don't have statistics on how well the feature-freeze model works in
> getting release-critical bugs fixed. Have we had other release models in the
> past? did they work any better?

I'm not entirely sure how our freeze model worked in the past, but here's what
I intend: The 25.1 release branch isn't ready until all its critical bugs are
resolved.

This means we'll need a 25.1 tag, and we'll need to go through all current
high severity bugs to determine which should receive that tag. Then the clock
starts, until either we have closed the last one, or removed all such tags.

The motivation for working on a release branch has several aspects:

  (1) 25.1 isn't released until it's ready, so if you have work you're proud
      that you want to set free, help!

  (2) My focus will be on 25.1, and not on future features, so until we have
      25.1 out the door, our discussions may not be as fun.

  (3) It will give all of us a chance to hone our bugtracker skills, and make
      it an easier process so future releases go more smoothly. I'm open to
      making changes to reduce developer inertia here, and you can be part of
      that by telling me about your experiences.

John

Re: Release process (was Re: Move to a cadence release model?)

[Xue Fuqiao]

On Thu, Nov 12, 2015 at 12:39 AM, John Wiegley <jwiegley@gmail.com> wrote:

Hi John,

> This means we'll need a 25.1 tag, and we'll need to go through all current
> high severity bugs to determine which should receive that tag.

We already have something like that.  I've documented it in my patch, in
the "RELEASE-CRITICAL BUGS" section.  You can see it in the "Feature
freezes and Emacs 25" thread.

Re: Release process (was Re: Move to a cadence release model?)

[Xue Fuqiao]

On Tue, Nov 17, 2015 at 12:54 AM, John Wiegley <jwiegley@gmail.com> wrote:

Hi John,

> I had suggested using a release tag for 25.1 in debbugs. Is that not possible?
> At some time soon we'll need to start assessing existing bugs and applying
> that tag to them.

We don't have a "release tag", but we use the "blocking bug(s)" feature
of Debbugs for bugs need to be addressed in the next release.

See
https://github.com/emacs-mirror/emacs/blob/9a4aa0f5945a03611ae29c516025dbd353bd26ab/admin/release-process#L30-L41
for details.

Re: Release process (was Re: Move to a cadence release model?)

[Xue Fuqiao]

On Wed, Nov 11, 2015 at 6:45 PM, Stephen Leake
<stephen_leake@stephe-leake.org> wrote:

> So a document that records the rationale for the release process would
> be useful.

I'm trying to write one currently.  Please see the "Feature freezes and
Emacs 25" thread.  Comments are very welcome.

Re: Move to a cadence release model?

[Alan Mackenzie]

Hello, John.

On Tue, Nov 10, 2015 at 08:57:07AM -0500, John Yates wrote:
> With a new master at the helm and various changes being contemplated I
> would like to see some discussion of moving to a cadence release model.

I suspect that here "some" is going to be somewhat of an understatement.
;-)

> I have been impressed with open source projects that have made the change.
> I am now employed at Mathworks which ships mission critical software to
> very large enterprise customers on a 6 month cadence.

> The biggest shift I see is away from wondering when the correct collection
> of features, bug fixes, whatever have been accumulated to whether those
> that have been accumulated are individually sufficiently cooked to ship.
> Developers feel less urge to squeeze a not fully baked feature into the
> current release when they can count on the next opportunity being only a
> cadence interval in the future.

On our current release model, we typically have a pre-test phase which
is longer than 6 months.  This would have to be compressed, somehow
(assuming we retain the concept of pre-test).

> /john

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Move to a cadence release model?

[Eli Zaretskii]

> Date: Tue, 10 Nov 2015 14:32:22 +0000
> From: Alan Mackenzie <acm@muc.de>
> Cc: Emacs developers <emacs-devel@gnu.org>
> 
> On our current release model, we typically have a pre-test phase which
> is longer than 6 months.  This would have to be compressed, somehow
> (assuming we retain the concept of pre-test).

Right.  And IMO it cannot be compressed without significant changes in
the development process.  E.g., the documentation of the new features
cannot be postponed to the pretest period.

Re: Move to a cadence release model?

[Eli Zaretskii]

> Date: Tue, 10 Nov 2015 08:57:07 -0500
> From: John Yates <john@yates-sheets.org>
> 
> With a new master at the helm and various changes being contemplated I would
> like to see some discussion of moving to a cadence release model.
> 
> I have been impressed with open source projects that have made the
> change.

Which ones?

> I am now employed at Mathworks which ships mission critical software
> to very large enterprise customers on a 6 month cadence.
> 
> The biggest shift I see is away from wondering when the correct collection of
> features, bug fixes, whatever have been accumulated to whether those that have
> been accumulated are individually sufficiently cooked to ship. Developers feel
> less urge to squeeze a not fully baked feature into the current release when
> they can count on the next opportunity being only a cadence interval in the
> future.

AFAIK, cadence release model requires development that is planned in
advance, and then actively managed to make sure the features planned
for the next release are ready on time.

I don't know how things work at Mathworks, but I'm guessing there are
staff and processes in place to enable that.  There's probably someone
whose job is to collect proposals for new features, either from users,
or from developers, or from significant improvements in relevant
algorithms, etc.  Then there's a process that distills these proposals
into requirements for new features, prioritizes those requirements,
and decides on the schedule they will be implemented.  Then these
plans are given to developers, and there are procedures that manage
the development; e.g., if it turns out there's not enough manpower,
someone goes out and hires some more, or takes other managerial
actions.  This management requires, among other things, continuous
collection of status of each developer and tight control of their
advances.  Last, but not least, you have a known number of man-months
per month at your disposal, people who work day in and day out, which
allows reasonably accurate estimations of when each feature can be
ready to ship.

Is the above anywhere close to what happens at Mathworks?  If not,
perhaps you could describe the process that supports your cadence
model.  Because I don't know how it can be done without something like
that.

Cadence model can also work for FLOSS projects, provided that the
development team can count on their resources to enable a steady
stream of improvements.  E.g., there are projects where the core
developers get paid to work on the otherwise Free Software package
(GDB is one such package, but even GDB doesn't always succeed to
release according to plan).

And maybe there are other arrangements that might enable cadence.  The
key is the ability of the project leadership to ensure steady flow of
improvements and new features.  Without that, cadence model will
simply not work, because it makes very little sense to take a snapshot
of the main development branch at some arbitrary point in time and
declare it to be the future release.

Now, we don't have any of that in Emacs.  Some of what I mentioned
above might be possible, if we find the right people to do the job.
Other parts are IMO impossible even in principle: e.g., who of us
could commit to some minimal amount of hours he/she can work on Emacs
each day?  Without such a basic prerequisite, how could anyone plan
ahead Emacs development and ensure that the next release will be
meaningful to our users?

And one more thing, about the "next opportunity": this only works if
that opportunity is close.  If the next major Emacs release is 3 years
away, no one will want to wait, and it will be unreasonable for us to
ask them to wait.  It is much more reasonable for _us_ to wait for a
few weeks, and allow that feature to be finished in time for the
release.

Re: Move to a cadence release model?

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

[snip]

> And maybe there are other arrangements that might enable cadence.

The key to a successful cadence release model is QA. It works very well
for projects such as Clang/LLVM (possibly could do for GCC as well)
because their extensive test suite. Also, if you have many resources for
QA and bug-fixing, implying that most relevant problems will be
discovered and fixed on a short timeframe, cadence is good too.

As Emacs has no strong test suite (automatic testing of UI is hard, so
no cure for this on sight) neither has many resources for steady manual
testing and bug-fixing, I'm afraid that switching to the cadence model
would deteriorate the quality of releases.

A pity, because otherwise it is a great model, for the reasons given by
the OP.

[snip]

Automate Emacs UI testing? (was: Move to a cadence release model?)

[John Wiegley]

>>>>> Óscar Fuentes <ofv@wanadoo.es> writes:

> As Emacs has no strong test suite (automatic testing of UI is hard, so no
> cure for this on sight) neither has many resources for steady manual testing
> and bug-fixing, I'm afraid that switching to the cadence model would
> deteriorate the quality of releases.

Web developers have some pretty awesome tools for driving a UI and observing
the resulting behavior. I wonder what it would take to have a framework like
that for Emacs...

John

Re: Automate Emacs UI testing?

[Ashton Kemerling]

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> Óscar Fuentes <ofv@wanadoo.es> writes:
>
>> As Emacs has no strong test suite (automatic testing of UI is hard, so no
>> cure for this on sight) neither has many resources for steady manual testing
>> and bug-fixing, I'm afraid that switching to the cadence model would
>> deteriorate the quality of releases.
>
> Web developers have some pretty awesome tools for driving a UI and observing
> the resulting behavior. I wonder what it would take to have a framework like
> that for Emacs...
>
> John


And said tools are pretty notorious for their unreliability and
slowness. I've never heard of a web shop using them as anything more
than a "we literally can't isolate this bug any other way" scenario
tool.

--
Ashton

Re: Automate Emacs UI testing? (was: Move to a cadence release model?)

[Eli Zaretskii]

> From: John Wiegley <jwiegley@gmail.com>
> Date: Tue, 10 Nov 2015 09:44:11 -0800
> Cc: emacs-devel@gnu.org
> 
> >>>>> Óscar Fuentes <ofv@wanadoo.es> writes:
> 
> > As Emacs has no strong test suite (automatic testing of UI is hard, so no
> > cure for this on sight) neither has many resources for steady manual testing
> > and bug-fixing, I'm afraid that switching to the cadence model would
> > deteriorate the quality of releases.
> 
> Web developers have some pretty awesome tools for driving a UI and observing
> the resulting behavior. I wonder what it would take to have a framework like
> that for Emacs...

UI is only one issue.  The more important issue, IMO, is poor coverage
of the test suite in general.  Many packages and primitives have no
tests at all.

Re: Automate Emacs UI testing?

[joakim]

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> Óscar Fuentes <ofv@wanadoo.es> writes:
>
>> As Emacs has no strong test suite (automatic testing of UI is hard, so no
>> cure for this on sight) neither has many resources for steady manual testing
>> and bug-fixing, I'm afraid that switching to the cadence model would
>> deteriorate the quality of releases.
>
> Web developers have some pretty awesome tools for driving a UI and observing
> the resulting behavior. I wonder what it would take to have a framework like
> that for Emacs...

I work with test automation periodically.

Web testing is a special case, because there are frameworks that can
work with the browser document object model, to simulate a user, and verify
results. These tools wouldn't work with Emacs.

There is another class of tools that work at the windowing level,
including http://www.sikuli.org/. It works by simulating events, and
then analyzing the on-screen results with OpenCV, which is a computer
vision framework. I have worked with this tool, and it works, but is
tricky to set up. I have been wanting to set up my own tests with Sikuli
and Emacs, because many emacs packages I use randomly fail after
upgrade. If I could verify that everything works in a separate
environment I wouldn't need to upgrade my main emacs. 

This is the testing scenario I have worked on.

- Isolate an Emacs build and environment in a Docker container
  environment. This works well, but it's a little bit tricky getting the
  containerized Emacs to access the host screen. It is a nice strategy,
  because you can easily simulate a number of gnu/linux distributions. I
  used this strategy for my xwidget branch.
  
- Make sikuli tests that test things like Helm and Yasnippets. I haven't
  done more than verifying that the strategy works. Here is a container
  with Sikuli that I havent tested: https://hub.docker.com/r/kkochubey1/sikuli-chrome-x11vnc/

- Do these tests on each commit in a Jenkins build server that I
  have. It could be any build server with the right configuration though.






>
> John
>

-- 
Joakim Verona

Re: Automate Emacs UI testing?

[John Wiegley]

>>>>> joakim  <joakim@verona.se> writes:

> Web testing is a special case, because there are frameworks that can work
> with the browser document object model, to simulate a user, and verify
> results. These tools wouldn't work with Emacs.

People have made good points. I'd like to continue this discussion on
emacs-tangents (where I should have asked it), since it's pretty clear that we
need more core testing before we start thinking about fancy testing schemes;
and that such fanciness may not apply directly in our environment.

John

Re: Automate Emacs UI testing?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > - Isolate an Emacs build and environment in a Docker container
  >   environment.

Is Docker free software?

  > - Make sikuli tests that test things like Helm and Yasnippets.

What is a sikuli test?  Can they be done without using any nonfree
software?

  > - Do these tests on each commit in a Jenkins build server that I
  >   have. It could be any build server with the right configuration though.

It would not be good for our tests to depend on running on one particular site.
They should run on any GNU/Linux system, however it could be ok for some tests to require loading certain well-known free software packages.

  > Here is a container
  >   with Sikuli that I havent tested: https://hub.docker.com/r/kkochubey1/sikuli-chrome-x11vnc/

Maybe I could find out some of those answers by looking around at that,
but I have a feeling it would take hours to find them that way.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Automate Emacs UI testing?

[joakim]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > - Isolate an Emacs build and environment in a Docker container
>   >   environment.
>
> Is Docker free software?

Yes, it is free software. Apache v2.

>
>   > - Make sikuli tests that test things like Helm and Yasnippets.
>
> What is a sikuli test?  Can they be done without using any nonfree
> software?

Yes, Sikuli if free software, MIT license.

A Sikuli test simulates what a user would do in terms of keyboard and
mouse input. Sikuli can then verify that the correct thing happened in
the application window, by analyzing what happened in the screen. The
novelty Sikuli adds over older such test systems is that Sikuli is
better at analyzing the screen bitmaps, because it uses a computer
vision library rather than just verifying on a pixel by pixel basis.

>
>   > - Do these tests on each commit in a Jenkins build server that I
>   >   have. It could be any build server with the right configuration though.
>
> It would not be good for our tests to depend on running on one particular site.
> They should run on any GNU/Linux system, however it could be ok for some tests to require loading certain well-known free software packages.

The tests should run on any environment that support Docker, which is
only free operating system AFAIK. (You can run parts of Docker on
non-free operating systems, but that is of no real concern for us I
would say)

An important thing Docker helps with is packaging software so that it is
easy to build and run the software. Dependencies can be clearly declared
so building a container is portable. One way to look at it is that
Docker is like a standardized chroot build and run environment. 


>
>   > Here is a container
>   >   with Sikuli that I havent tested: https://hub.docker.com/r/kkochubey1/sikuli-chrome-x11vnc/
>
> Maybe I could find out some of those answers by looking around at that,
> but I have a feeling it would take hours to find them that way.

I hope this helps.


-- 
Joakim Verona

Re: Automate Emacs UI testing?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

It appears that Docker and Sikuli are ok for us to use,
if we want to.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

official Emacs Docker image (was: Automate Emacs UI testing?)

[Ted Zlatanov]

On Wed, 11 Nov 2015 18:27:27 -0500 Richard Stallman <rms@gnu.org> wrote: 

RS> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
RS> [[[ whether defending the US Constitution against all enemies,     ]]]
RS> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]

RS> It appears that Docker and Sikuli are ok for us to use,
RS> if we want to.

There is no official Docker image for Emacs on the Docker Hub:

https://hub.docker.com/search/?q=emacs&page=1&isAutomated=0&isOfficial=0&starCount=0&pullCount=0

I think it would be very useful to make an official image and link it to
Git so it builds automatically. It would make it easy to automate some
types of tests, but it would also be a convenient way for users to run a
very recent Emacs without downloading all the required libraries and
building it locally. It would be good to make images with several
platforms for testing, but for users I imagine a simple Debian image
would fit most needs.

Another interesting use case is running the daemon in a disposable and
theoretically isolated container.

John and Richard, the only issue I can see there is that the GNU Project
may wish to maintain their own Docker image repository. Because there
are already some official images from GNU packages on the Docker Hub
such as https://hub.docker.com/_/gcc/ though, I assume this is OK. You
can see all the official images at
https://github.com/docker-library/official-images/tree/master/library

If you decide to pursue the official repo path, you can see the
instructions at https://docs.docker.com/docker-hub/official_repos/
require a single author. In the Emacs case, I didn't want to start the
process myself because it's probably better to create a shared
authorship, but am happy to contribute a Dockerfile and anything else
needed.

Ted

Re: official Emacs Docker image

[Nic Ferrier]

Ted Zlatanov <tzz@lifelogs.com> writes:

> There is no official Docker image for Emacs on the Docker Hub:
>
> https://hub.docker.com/search/?q=emacs&page=1&isAutomated=0&isOfficial=0&starCount=0&pullCount=0

I made one though. I use it to host marmalade.


Nic

Re: official Emacs Docker image

[Ricardo Wurmus]

Nic Ferrier <nferrier@ferrier.me.uk> writes:

> Ted Zlatanov <tzz@lifelogs.com> writes:
>
>> There is no official Docker image for Emacs on the Docker Hub:
>>
>> https://hub.docker.com/search/?q=emacs&page=1&isAutomated=0&isOfficial=0&starCount=0&pullCount=0
>
> I made one though. I use it to host marmalade.

With GNU Guix you can run Emacs in a container without having to use
Docker:

    guix environment --container --ad-hoc emacs -- emacs

This creates an isolated ad-hoc environment containing only Emacs and
then runs “emacs”; all without Docker and the need for a disk image.

~~ Ricardo

Re: official Emacs Docker image (was: Automate Emacs UI testing?)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > John and Richard, the only issue I can see there is that the GNU Project
  > may wish to maintain their own Docker image repository. Because there
  > are already some official images from GNU packages on the Docker Hub
  > such as https://hub.docker.com/_/gcc/ though, I assume this is OK.

Anyone is free, and welcome, to use our packages this way.  The one
concern I have about it is that using Docker (the server) for this is SaaSS.

If we want to do testing using Docker (the software), we should write something
that quickly and easily configures Docker (the software) on any machine,
rather than recommending people use Docker (the server).

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: official Emacs Docker image

[joakim]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > John and Richard, the only issue I can see there is that the GNU Project
>   > may wish to maintain their own Docker image repository. Because there
>   > are already some official images from GNU packages on the Docker Hub
>   > such as https://hub.docker.com/_/gcc/ though, I assume this is OK.
>
> Anyone is free, and welcome, to use our packages this way.  The one
> concern I have about it is that using Docker (the server) for this is SaaSS.
>
> If we want to do testing using Docker (the software), we should write something
> that quickly and easily configures Docker (the software) on any machine,
> rather than recommending people use Docker (the server).

I think you mean the docker registry, which is the server that provides
images for the docker client. It is free software, Apache V2.

It is pretty easy to set up the entire Docker stack on a free system,
and all of it is free software as far as I can determine anyway.

That said, there might be reasons why GNU would like to host it's own
docker registry. 

-- 
Joakim Verona

Re: official Emacs Docker image

[Ted Zlatanov]

On Thu, 12 Nov 2015 17:31:58 -0500 Richard Stallman <rms@gnu.org> wrote: 

>> John and Richard, the only issue I can see there is that the GNU Project
>> may wish to maintain their own Docker image repository. Because there
>> are already some official images from GNU packages on the Docker Hub
>> such as https://hub.docker.com/_/gcc/ though, I assume this is OK.

RS> Anyone is free, and welcome, to use our packages this way.  The one
RS> concern I have about it is that using Docker (the server) for this is SaaSS.

Understood.

RS> If we want to do testing using Docker (the software), we should write something
RS> that quickly and easily configures Docker (the software) on any machine,
RS> rather than recommending people use Docker (the server).

Yup. That's what I'd like to do. Setting up an Emacs image is easy in
itself, you write a Dockerfile which is sort of a shell script for
building Docker images. The output is a binary artifact.

I would probably set up multiple builds (which the Docker Hub provides
as a free service) for several platforms, so pulling
emacs-official:debian would get an image of Emacs running in Debian, for
instance. That's the part where it's nice to use the Docker Hub service;
it doesn't provide anything that can't be done locally but it simplifies
automated builds, and provides convenient central searching and storage
for many people.

The key thing is to make the image "official" so it's trusted by Docker
Hub users and has a top-level namespace. That will increase the
popularity of the image. I think this is important because there are
other GNU packages, such as GCC linked above, that are "official" and
only seem to be maintained by Docker Inc. staff (see
https://github.com/docker-library/gcc for the history of the GCC
official image).

I recommend creating a FSF/GNU organization on Docker Hub, which can
then be joined by interested contributors and can streamline this work.
Contributing individually won't scale.

On Thu, 12 Nov 2015 23:49:52 +0800 Nic Ferrier <nferrier@ferrier.me.uk> wrote: 

NF> I made one though. I use it to host marmalade.

Could you point me to it? I looked and found nothing.

Thanks!
Ted

Re: official Emacs Docker image

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I think this is important because there are
  > other GNU packages, such as GCC linked above, that are "official"

Could you explain what you mean by "official"?  In what sense are
those "official"?

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: official Emacs Docker image

[Ted Zlatanov]

On Thu, 07 Jul 2016 17:56:35 -0400 Richard Stallman <rms@gnu.org> wrote: 

>> I think this is important because there are
>> other GNU packages, such as GCC linked above, that are "official"

RS> Could you explain what you mean by "official"?  In what sense are
RS> those "official"?

The Docker Hub web site marks them so, and users trust them more. They
tend to be up to date.

Docker Inc. provides some maintenance and review for such images, so
they are in many ways an investment for the company. I don't know more,
I've never made or maintained an official image. But GCC is up there, as
I said, so there's a precedent.

I think FSF/GNU software can be part of that ecosystem, but if not then
we should look at our own FSF/GNU Docker Hub repository. Either way,
users will benefit. Right now there's nothing official.

Ted

Re: official Emacs Docker image

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > RS> Could you explain what you mean by "official"?  In what sense are
  > RS> those "official"?

  > The Docker Hub web site marks them so, and users trust them more. They
  > tend to be up to date.

Who said these are "official"?

  > Docker Inc. provides some maintenance and review for such images, so
  > they are in many ways an investment for the company. I don't know more,
  > I've never made or maintained an official image. But GCC is up there, as
  > I said, so there's a precedent.

Whether we should recommend Docker containers is a ethical question
and a political question.  If the answer is yes, whether we should
post them on the Docker site is another ethical question and another
political question.

Precedent is not the way to reach conclusions about such questions.
In other words, the fact that someone is doing something does not
mean it is a good thing to do.

The GNU Project has no position on those questions as yet.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: official Emacs Docker image

[Ted Zlatanov]

On Sat, 09 Jul 2016 12:54:33 -0400 Richard Stallman <rms@gnu.org> wrote: 

RS> Could you explain what you mean by "official"?  In what sense are
RS> those "official"?

>> The Docker Hub web site marks them so, and users trust them more. They
>> tend to be up to date.

RS> Who said these are "official"?

Docker Inc., who runs the Docker Hub. They essentially own the top
namespace, so pulling "ubuntu" gets the official Ubuntu image, while
"rms/ubuntu" is under the "rms" (usually a user name) namespace.

>> Docker Inc. provides some maintenance and review for such images, so
>> they are in many ways an investment for the company. I don't know more,
>> I've never made or maintained an official image. But GCC is up there, as
>> I said, so there's a precedent.

RS> Whether we should recommend Docker containers is a ethical question
RS> and a political question.  If the answer is yes, whether we should
RS> post them on the Docker site is another ethical question and another
RS> political question.

RS> Precedent is not the way to reach conclusions about such questions.
RS> In other words, the fact that someone is doing something does not
RS> mean it is a good thing to do.

RS> The GNU Project has no position on those questions as yet.

Yes, right, and that's why I am trying to start the discussion instead
of just putting up Emacs images on the Docker Hub :)

What's the plan for coming to a position? Is there some timeframe, or
some milestones, that have to be reached? Will the discussions be
private and then an announcement made?

Containers are in many ways hiding free software from the users, who are
not aware of all the GNU or other software at every layer, and usually
just use the top layer as a service (wrapping all the others). Docker in
particular does a great job at simplifying the process, so users don't
have to pay attention to licenses or provenance. So I think establishing
a FSF/GNU presence and even branding in the Docker community now is
important. I hope you'll consider that.

Thanks
Ted

Re: official Emacs Docker image

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Docker Inc., who runs the Docker Hub. They essentially own the top
  > namespace, so pulling "ubuntu" gets the official Ubuntu image, while
  > "rms/ubuntu" is under the "rms" (usually a user name) namespace.

Needless to say, Docker's label of "official" carries no weight in the
GNU Project.  I don't know what Docker's criteria for an "official"
distribution are, so I am not ready to reach any conclusion about what
stance we should take towards that.

  > What's the plan for coming to a position? Is there some timeframe, or
  > some milestones, that have to be reached? Will the discussions be
  > private and then an announcement made?

I don't have a formal plan like that for studying a question.

We discussed docker containers and reached the conclusion
that there is nothing inherently wrong with them.  So it is ok
for GNU packages to make and distribute them.

I will ask some advisors to look at the Docker site and see what is
good or bad about it.  Some of the criteria for source repositories
(http://gnu.org/software/repo-criteria.html will be relevant, though
others will not be.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: official Emacs Docker image

[John Wiegley]

[-- Attachment #1: Type: text/plain, Size: 1177 bytes --]

>>>>> "TZ" == Ted Zlatanov <tzz@lifelogs.com> writes:

TZ> Docker Inc., who runs the Docker Hub. They essentially own the top
TZ> namespace, so pulling "ubuntu" gets the official Ubuntu image, while
TZ> "rms/ubuntu" is under the "rms" (usually a user name) namespace.

Wouldn't the "official" Docker image for Emacs be some "official" base OS,
such as debian:wheezy, and then the current Emacs release tarball built with
default options upon that base?

I say this because there are already "official" versions of Ubuntu and Emacs,
released on their respective websites, and Docker is just combining them into
another delivery form. Since they don't add or change anything about the code
being deployed, the officialness of Emacs running in a container should be
transitively determined by where its sources came from.

In all of this the FSF needn't be involved, except to be sure that whoever is
packaging these Emacs containers isn't adding or changing anything from what
is released on ftp.gnu.org.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 629 bytes --]

Re: official Emacs Docker image

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Wouldn't the "official" Docker image for Emacs be some "official" base OS,
  > such as debian:wheezy,

I don't understand concretely what the scenario is.
Who is doing what?  Who is deciding what?

The FSF won't officially base anything on Debian, because we don't
endorse Debian.  See http://gnu.org/distros/common-distros.html
for the reason we don't.

  >  the officialness of Emacs running in a container should be
  > transitively determined by where its sources came from.

Not solely from that!  The container would contain other programs
besides Emacs, and our ethical concerns apply to them too.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: official Emacs Docker image

[John Wiegley]

>>>>> Richard Stallman <rms@gnu.org> writes:

> I don't understand concretely what the scenario is. Who is doing what? Who
> is deciding what?

Ok, here is the scenario:

You create a Docker build recipe by picking a "base" GNU/Linux distribution
(some variant of GNU/Linux that has an absolute minimum of included programs),
a recipe to install build tools, and then a recipes to build the target
program. There are many ways to optimize this so that the result is as small
as possible, but that's the basics in a nutshell.

If the FSF doesn't endorse Debian, then an alternate GNU/Linux base can be
chosen, and the build step crafted to suit it.

The final "image" as they call it will be an Emacs binary, built on that base,
that users can download and directly run from any of the major operating
systems (Mac OS X, Windows, any flavor of GNU/Linux). It does this for non-
GNU/Linux operating systems by either running the image in a virtual machine,
or by using hyper-visor technology.

So: base + tools + Emacs source -> build -> Docker image. This makes it
possible to know *exactly* what is being contained in the image, and how it
was produced.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: official Emacs Docker image

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Please forgive me for taking 6 months to respond to this.

  > You create a Docker build recipe by picking a "base" GNU/Linux distribution
  > (some variant of GNU/Linux that has an absolute minimum of included programs),
  > a recipe to install build tools, and then a recipes to build the target
  > program. There are many ways to optimize this so that the result is as small
  > as possible, but that's the basics in a nutshell.

  > If the FSF doesn't endorse Debian, then an alternate GNU/Linux base can be
  > chosen, and the build step crafted to suit it.

I think I follow this part.

  > The final "image" as they call it will be an Emacs binary, built on that base,
  > that users can download and directly run from any of the major operating
  > systems (Mac OS X, Windows, any flavor of GNU/Linux).

At this point, I am confused, because the statements seem to conflict.

Would the "Docker image" of Emacs _include_ the base system?  Or would
it be an executable Emacs package that could be installed straight _on
top of_ that base system?

If it is the latter, I don't see how it could run on any other
GNU/Linux system version aside from the one chosen as the base,
except using a virtual machine containing the chosen base system.

If it is the latter, then I see how it could run on another GNU/Linux system
in a virtual machine, but using the virtual machine seems like a drawback.

Can you please explain?


I do see how running on other systems by running GNU/Linux in a
virtual machine could make maintenance easier for us -- we could
perhaps delete some of our code used to support those other systems.


-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: official Emacs Docker image

[John Wiegley]

>>>>> Richard Stallman <rms@gnu.org> writes:

>> The final "image" as they call it will be an Emacs binary, built on that
>> base, that users can download and directly run from any of the major
>> operating systems (Mac OS X, Windows, any flavor of GNU/Linux).

> At this point, I am confused, because the statements seem to conflict.

> Would the "Docker image" of Emacs _include_ the base system? Or would it be
> an executable Emacs package that could be installed straight _on top of_
> that base system?

I see now that I was unclear: A Docker image is a self-contained tarball
containing a GNU/Linux kernel, necessary system software, and the final Emacs
executable that was built by the image recipe.

The Docker image contents, thus, can be entirely free software. Executing the
image on some platforms (such as Windows) may use proprietary software to
perform the execution (for example, VM provisioning software).

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: official Emacs Docker image

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I see now that I was unclear: A Docker image is a self-contained tarball
  > containing a GNU/Linux kernel, necessary system software, and the final Emacs
  > executable that was built by the image recipe.

Now it is coherent.

  > The Docker image contents, thus, can be entirely free software. Executing the
  > image on some platforms (such as Windows) may use proprietary software to
  > perform the execution (for example, VM provisioning software).

I see how it makes sense to use this on Windows.  But it seems absurd
to use this on GNU/Linux.  Why does anyone do that?

How big would such a docker image be?  I know that disks are getting
bigger, but how many such applications could fit on a typical laptop?

However, I don't see any ethical issue about making and distributing
Docker images of Emacs as long as we get the details right: for
instance, use an endorsed free GNU/Linux distro.

Do you see any specific issues we need to consider?

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: official Emacs Docker image

[joakim]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > I see now that I was unclear: A Docker image is a self-contained tarball
>   > containing a GNU/Linux kernel, necessary system software, and the final Emacs
>   > executable that was built by the image recipe.
>
> Now it is coherent.
>
>   > The Docker image contents, thus, can be entirely free software. Executing the
>   > image on some platforms (such as Windows) may use proprietary software to
>   > perform the execution (for example, VM provisioning software).
>
> I see how it makes sense to use this on Windows.  But it seems absurd
> to use this on GNU/Linux.  Why does anyone do that?

It is quite useful in order to handle dependencies.

When developing an emacs feature I found it useful to test the feature
by having many different emacs docker containers, each one containing
different gnu/linux distributions and different emacs compile flags.
It was very convenient in order to test all the different configurations
on a single machine.

A container is a bit like a virtual machine but also a bit like a
traditional changeroot.

Another thing to consider is that Docker isn't fantastic for desktop
applications, at least when I used Docker for this purpose. It is
doable, by manipulating X sockets and whatnot.

For this reason alternatives are emerging with focus on desktop
applications, such as Flatpak. Gnome applications can be accessed with
Flatpak for instance.

Another method is using Gnu Guix.

All methods have their pros and cons and different usage scenarios.

>
> How big would such a docker image be?  I know that disks are getting
> bigger, but how many such applications could fit on a typical laptop?

Docker uses layered copy on write file systems in order to save space.
Basically, each container can re-use the filesystem usage of other
containers. I can't provide an exact figure of how many docker packaged
applications can fit on a laptop, but "many" is a reasonable assumption.

> However, I don't see any ethical issue about making and distributing
> Docker images of Emacs as long as we get the details right: for
> instance, use an endorsed free GNU/Linux distro.
>
> Do you see any specific issues we need to consider?
-- 
Joakim Verona
joakim@verona.se
+46705459454

Re: official Emacs Docker image

[John Wiegley]

>>>>> Richard Stallman <rms@gnu.org> writes:

> I see how it makes sense to use this on Windows. But it seems absurd to use
> this on GNU/Linux. Why does anyone do that?

On GNU/Linux systems, it runs the image in a low-cost container, providing
better process separation than chroot, while also allowing you to use
alternative kernel versions, or a different distro (for example, running
something built for CentOS on a Debian box).

> How big would such a docker image be? I know that disks are getting bigger,
> but how many such applications could fit on a typical laptop?

Probably <200M, if constructed well, and with all sources and documentation
made available within the image.

> However, I don't see any ethical issue about making and distributing Docker
> images of Emacs as long as we get the details right: for instance, use an
> endorsed free GNU/Linux distro.
>
> Do you see any specific issues we need to consider?

No, I think we can provide a 100% free image, that others could run on any
system supporting Docker-engine (Windows, FreeBSD, Mac, GNU/Linux). This will
allow them to experience identical behavior on all those systems. It would
also be able to edit host files, if the user passes -v to map one or more host
directories into the container.

As to *why* anyone would want to do this: The biggest advantage to Docker is
not packaging a single application, since Emacs can already run on all these
systems. It's having the ability to test a perfectly self-contained Emacs,
that runs within a virgin environment every time (any changes made to the
container are lost as soon as the container is stopped). One could then have
multiple Emacsen available, and be able to run tests on each version, without
disturbing their host machine's installation.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: official Emacs Docker image

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > I see how it makes sense to use this on Windows. But it seems absurd to use
  > > this on GNU/Linux. Why does anyone do that?

  > On GNU/Linux systems, it runs the image in a low-cost container, providing
  > better process separation than chroot, while also allowing you to use
  > alternative kernel versions, or a different distro (for example, running
  > something built for CentOS on a Debian box).

I guess that makes sense for special circumstances.  Anyway, it does
no wrong to anyone.  So please support it if you want to.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: official Emacs Docker image

[Ted Zlatanov]

On Sat, 31 Dec 2016 13:25:32 -0500 Richard Stallman <rms@gnu.org> wrote: 

RS> I guess that makes sense for special circumstances.  Anyway, it does
RS> no wrong to anyone.  So please support it if you want to.

After talking to RMS, I created https://hub.docker.com/u/fsfemacs and am
currently the owner of the organization. The image, when ready, will be
"fsfemacs/emacs". I named it that way to emphasize the FSF organization,
rather than "gnuemacs" which would emphasize the GNU project.

If you're interested in collaborating on a Dockerfile that builds 24.x
and 25.x/master, let me know here or in e-mail. If you have one already,
even better.

Thanks
Ted

Re: official Emacs Docker image

[Filipe Silva]

[-- Attachment #1: Type: text/plain, Size: 1140 bytes --]

Ted, I think that before the portable dumper branch get's merged or the big
elc file branch gets merged, you are going to have a really hard time
writing a docker file for that because of:
https://github.com/docker/docker/issues/22801


You'll probably will have to rely on "docker commiting" your image. That's
how I do it for now. I built emacs from git head with it.


On Fri, Jan 27, 2017 at 12:56 PM, Ted Zlatanov <tzz@lifelogs.com> wrote:

> On Sat, 31 Dec 2016 13:25:32 -0500 Richard Stallman <rms@gnu.org> wrote:
>
> RS> I guess that makes sense for special circumstances.  Anyway, it does
> RS> no wrong to anyone.  So please support it if you want to.
>
> After talking to RMS, I created https://hub.docker.com/u/fsfemacs and am
> currently the owner of the organization. The image, when ready, will be
> "fsfemacs/emacs". I named it that way to emphasize the FSF organization,
> rather than "gnuemacs" which would emphasize the GNU project.
>
> If you're interested in collaborating on a Dockerfile that builds 24.x
> and 25.x/master, let me know here or in e-mail. If you have one already,
> even better.
>
> Thanks
> Ted
>
>
>

[-- Attachment #2: Type: text/html, Size: 2231 bytes --]

Re: official Emacs Docker image

[Ted Zlatanov]

On Fri, 27 Jan 2017 17:45:16 -0200 Filipe Silva <filipe.silva@gmail.com> wrote: 

FS> Ted, I think that before the portable dumper branch get's merged or the big
FS> elc file branch gets merged, you are going to have a really hard time
FS> writing a docker file for that because of:
FS> https://github.com/docker/docker/issues/22801

FS> You'll probably will have to rely on "docker commiting" your image. That's
FS> how I do it for now. I built emacs from git head with it.

One of the goals is to let every user `docker build' it locally. I can
wait for the portable dumper, and meanwhile I can push a manual image if
you show me how. Or I can make you one of the owners and you can push
it yourself.

On Fri, 27 Jan 2017 21:18:17 -0500 Richard Stallman <rms@gnu.org> wrote: 

RS> For this activity it is more correct to emphasize GNU rather than the FSF.

OK, it's "gnuemacs" on Docker Hub now. As before, e-mail me if you think
you can help develop the "gnuemacs/emacs" image for our users and I'll
add you.

Thanks
Ted

Re: official Emacs Docker image

[Ricardo Wurmus]

Filipe Silva <filipe.silva@gmail.com> writes:

> Ted, I think that before the portable dumper branch get's merged or the big
> elc file branch gets merged, you are going to have a really hard time
> writing a docker file for that because of:
> https://github.com/docker/docker/issues/22801

I would like to state again that we (i.e. the GNU project) already have
a way to build valid Docker images for Emacs using GNU Guix.  It does
not even involve the use of Docker, nor does it require a third-party
“base image” of a GNU+Linux system.

For the lack of up-to-date HTML documentation see the Texinfo source
file of the manual here:

    http://git.savannah.gnu.org/cgit/guix.git/tree/doc/guix.texi#n2480

Since Guix abides by the FSDG we can also be certain that the generated
image contains nothing but software libre.

Would it be helpful if the Guix project provided a Docker image for the
latest release for download?  To me it seems only natural for GNU Emacs
and GNU Guix to cooperate; it’s all GNU.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: official Emacs Docker image

[Ted Zlatanov]

On Mon, 30 Jan 2017 18:10:53 +0100 Ricardo Wurmus <rekado@elephly.net> wrote: 

RW> Filipe Silva <filipe.silva@gmail.com> writes:

>> Ted, I think that before the portable dumper branch get's merged or the big
>> elc file branch gets merged, you are going to have a really hard time
>> writing a docker file for that because of:
>> https://github.com/docker/docker/issues/22801

RW> I would like to state again that we (i.e. the GNU project) already have
RW> a way to build valid Docker images for Emacs using GNU Guix.  It does
RW> not even involve the use of Docker, nor does it require a third-party
RW> “base image” of a GNU+Linux system.

I don't see a problem providing both as "gnuemacs/guix-emacs" and
"gnuemacs/docker-emacs" or something like that. Or as tags of
"gnuemacs/emacs". I don't think they are equivalent, though, so the need
for a `docker build' solution is still there.

RW> Would it be helpful if the Guix project provided a Docker image for the
RW> latest release for download?  To me it seems only natural for GNU Emacs
RW> and GNU Guix to cooperate; it’s all GNU.

Sure. But there's more that Docker Hub offers: a global namespace; a
distributed download service; automated builds. You can upload the Guix
image to Docker Hub and get all of those benefits except the automated
builds: see https://docs.docker.com/docker-hub/repos/

If you can pack an ARM build into the image so it's multiarch, that's
great too. I know there's a way to do it with the Docker tools, so the
image format supports it. But it's definitely not a requirement.

I can add you to the Docker Hub account so you can do at least the first
uploads. Later we can automate them through Hydra or some other CI tool.

Ted

Re: official Emacs Docker image

[Filipe Silva]

[-- Attachment #1: Type: text/plain, Size: 2014 bytes --]

Ted,

https://hub.docker.com/r/gnuemacs/emacs/ is giving me http 404. Is that the
correct address?

On Mon, Jan 30, 2017 at 3:44 PM, Ted Zlatanov <tzz@lifelogs.com> wrote:

> On Mon, 30 Jan 2017 18:10:53 +0100 Ricardo Wurmus <rekado@elephly.net>
> wrote:
>
> RW> Filipe Silva <filipe.silva@gmail.com> writes:
>
> >> Ted, I think that before the portable dumper branch get's merged or the
> big
> >> elc file branch gets merged, you are going to have a really hard time
> >> writing a docker file for that because of:
> >> https://github.com/docker/docker/issues/22801
>
> RW> I would like to state again that we (i.e. the GNU project) already have
> RW> a way to build valid Docker images for Emacs using GNU Guix.  It does
> RW> not even involve the use of Docker, nor does it require a third-party
> RW> “base image” of a GNU+Linux system.
>
> I don't see a problem providing both as "gnuemacs/guix-emacs" and
> "gnuemacs/docker-emacs" or something like that. Or as tags of
> "gnuemacs/emacs". I don't think they are equivalent, though, so the need
> for a `docker build' solution is still there.
>
> RW> Would it be helpful if the Guix project provided a Docker image for the
> RW> latest release for download?  To me it seems only natural for GNU Emacs
> RW> and GNU Guix to cooperate; it’s all GNU.
>
> Sure. But there's more that Docker Hub offers: a global namespace; a
> distributed download service; automated builds. You can upload the Guix
> image to Docker Hub and get all of those benefits except the automated
> builds: see https://docs.docker.com/docker-hub/repos/
>
> If you can pack an ARM build into the image so it's multiarch, that's
> great too. I know there's a way to do it with the Docker tools, so the
> image format supports it. But it's definitely not a requirement.
>
> I can add you to the Docker Hub account so you can do at least the first
> uploads. Later we can automate them through Hydra or some other CI tool.
>
> Ted
>
>
>

[-- Attachment #2: Type: text/html, Size: 3172 bytes --]

Re: official Emacs Docker image

[Ted Zlatanov]

On Mon, 30 Jan 2017 22:14:46 -0200 Filipe Silva <filipe.silva@gmail.com> wrote: 

FS> https://hub.docker.com/r/gnuemacs/emacs/ is giving me http 404. Is that the
FS> correct address?

Create a Docker Hub account and let me know what it is. I'll add you to
the owners so you can upload the image.

Ted

Re: official Emacs Docker image

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Create a Docker Hub account and let me know what it is. I'll add you to
  > the owners so you can upload the image.

Here we seem to have returned to the use of the accounts that
require nonfree Javascript code to create.

Please DO NOT post anything using a Docker Hub account
if making or using such an account requires nonfree software.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: official Emacs Docker image

[Ted Zlatanov]

On Tue, 31 Jan 2017 22:03:57 -0500 Richard Stallman <rms@gnu.org> wrote: 

>> Create a Docker Hub account and let me know what it is. I'll add you to
>> the owners so you can upload the image.

RS> Here we seem to have returned to the use of the accounts that
RS> require nonfree Javascript code to create.

RS> Please DO NOT post anything using a Docker Hub account
RS> if making or using such an account requires nonfree software.

On Tue, 31 Jan 2017 22:03:07 -0500 Richard Stallman <rms@gnu.org> wrote: 

RS> A concrete practical question: does it work to prepare and upload
RS> images without running nonfree JS code?

RS> If so, we can go ahead and upload images.

>> Yes, correct. Just think of it as a package repository with automated
>> builds. You can upload your own package, built locally. That's what we
>> may do for the Guix images, if the Guix developers are interested.

RS> That's good.

RS> What exactly is it that requires nonfree JS code to use?
RS> And what job does it do?

I went to Docker Hub, turned off JS, and was able to use the settings
dialogs necessary to maintain the "gnuemacs" account going forward.
Account creation also seemed to work.

This is completely separate from preparing, signing, and uploading
images to the Docker Hub, which is like pushing packages to a
repository and does not involve web browsing.

Can we move on?

Thanks
Ted

Re: official Emacs Docker image

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I went to Docker Hub, turned off JS, and was able to use the settings
  > dialogs necessary to maintain the "gnuemacs" account going forward.
  > Account creation also seemed to work.

That's sounds favorable.  But could you please compare notes with Mike
Gurwitz about what does or doesn't require nonfree JS code?  You and
he are getting different results; the anomaly is troubling.

Please don't try to push us to "move on" without establishing
the facts of the situation clearly.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: official Emacs Docker image

[Mike Gerwitz]

[-- Attachment #1: Type: text/plain, Size: 1916 bytes --]

On Wed, Feb 01, 2017 at 21:53:27 -0500, Richard Stallman wrote:
>   > I went to Docker Hub, turned off JS, and was able to use the settings
>   > dialogs necessary to maintain the "gnuemacs" account going forward.
>   > Account creation also seemed to work.
>
> That's sounds favorable.  But could you please compare notes with Mike
> Gurwitz about what does or doesn't require nonfree JS code?  You and
> he are getting different results; the anomaly is troubling.

Account registration on the homepage of hub.docker.com does not work
without JS.  Is there somewhere else you can register?

Since the page uses React, the form itself is not a standard HTML form:
it form contains no action or URL to post to, so clicking the "Sign Up"
button simply reloads the page (GET /):

  <form class="row" data-reactid=".1xy13jx9bls.0.0.1.0.1.0.1">

(With JS enabled, it posts to /v2/users/signup/)

AFAIK, pushing/pulling images and such can be done using the
command-line utility, which requires no proprietary software.  It's
getting to the point where one _can push_ images on Docker Hub (which
requires an account and some upfront work on Docker Hub) that is a
problem.

While I have an account for work, I haven't done anything with the
images, so I don't know if e.g. pushing an image that doesn't exist will
create it.  If not, it doesn't appear to be possible to create new
e.g. repositories without JS.

There's no harm done to users looking to download the image.  The harm
is done to users registering accounts to administer them or otherwise
participate on Docker Hub.

I did inquire about this here:

  https://github.com/docker/hub-feedback/issues/946

-- 
Mike Gerwitz
Free Software Hacker+Activist | GNU Maintainer & Volunteer
GPG: D6E9 B930 028A 6C38 F43B  2388 FEF6 3574 5E6F 6D05
Old: 2217 5B02 E626 BC98 D7C0  C2E5 F22B B815 8EE3 0EAB
https://mikegerwitz.com

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]

Re: official Emacs Docker image

[Ted Zlatanov]

On Wed, 01 Feb 2017 21:53:27 -0500 Richard Stallman <rms@gnu.org> wrote: 

>> I went to Docker Hub, turned off JS, and was able to use the settings
>> dialogs necessary to maintain the "gnuemacs" account going forward.
>> Account creation also seemed to work.

RS> That's sounds favorable.  But could you please compare notes with Mike
RS> Gurwitz about what does or doesn't require nonfree JS code?  You and
RS> he are getting different results; the anomaly is troubling.

Unfortunately I was rushed and meant to write "organization account
creation" within an existing Docker Hub user account (you need one to
create an organization). I did not test creating the user account. But I
don't think any of that will make a difference.

At this point I've lost interest and will not continue this work. I
don't see a way to reconcile your positions with my reality, and can't
change either.

RS> Please don't try to push us to "move on" without establishing
RS> the facts of the situation clearly.

I think we can move on with other things now.

Thanks
Ted

Re: official Emacs Docker image

[Stefan Reichör]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Wed, 01 Feb 2017 21:53:27 -0500 Richard Stallman <rms@gnu.org> wrote: 
>
>>> I went to Docker Hub, turned off JS, and was able to use the settings
>>> dialogs necessary to maintain the "gnuemacs" account going forward.
>>> Account creation also seemed to work.
>
> RS> That's sounds favorable.  But could you please compare notes with Mike
> RS> Gurwitz about what does or doesn't require nonfree JS code?  You and
> RS> he are getting different results; the anomaly is troubling.
>
> Unfortunately I was rushed and meant to write "organization account
> creation" within an existing Docker Hub user account (you need one to
> create an organization). I did not test creating the user account. But I
> don't think any of that will make a difference.
>
> At this point I've lost interest and will not continue this work. I
> don't see a way to reconcile your positions with my reality, and can't
> change either.
>
> RS> Please don't try to push us to "move on" without establishing
> RS> the facts of the situation clearly.
>
> I think we can move on with other things now.
>

As a happy emacs user I am sad to see another good idea that helps to
increase the visibility of emacs is shot down by this oh so important
free software argument.

For me it is the same thing that happened to the proposed improvements
in intellisense behaviour. Richard wanted to talk to a lawyer about
this. And nothing happened. The person that wanted to implement this
stuff just resigned.

Stefan.

Re: official Emacs Docker image

[Filipe Silva]

[-- Attachment #1: Type: text/plain, Size: 2583 bytes --]

Stefan, GNU/FSF is first a political/philosophical organisation before
being a technical/software organization. The fact that they've managed to
construct such an amazing piece of software, Emacs, is a miracle, but
secondary to that primary characteristics.

And don't be mistaken, that comes with advantages too. As a non-profit
organisation, driven by philosophy, you can rest assured that emacs will be
here essently forever while sublime text, visual studio code, and others
will come and go.

Mike Gerwitz, back to the topic, you've posted an issue on the github site,
which is run with non-free software, and I'm pretty sure that it runs
non-free javascript. The fact that you will be running "non-free"
javascript seems a bit unescapeable, doesn't it?

(I'm not trying to pick an argument, just trying to understand the
philosophy)



On Fri, Feb 3, 2017 at 5:00 AM, Stefan Reichör <stefan@xsteve.at> wrote:

> Ted Zlatanov <tzz@lifelogs.com> writes:
>
> > On Wed, 01 Feb 2017 21:53:27 -0500 Richard Stallman <rms@gnu.org> wrote:
> >
> >>> I went to Docker Hub, turned off JS, and was able to use the settings
> >>> dialogs necessary to maintain the "gnuemacs" account going forward.
> >>> Account creation also seemed to work.
> >
> > RS> That's sounds favorable.  But could you please compare notes with
> Mike
> > RS> Gurwitz about what does or doesn't require nonfree JS code?  You and
> > RS> he are getting different results; the anomaly is troubling.
> >
> > Unfortunately I was rushed and meant to write "organization account
> > creation" within an existing Docker Hub user account (you need one to
> > create an organization). I did not test creating the user account. But I
> > don't think any of that will make a difference.
> >
> > At this point I've lost interest and will not continue this work. I
> > don't see a way to reconcile your positions with my reality, and can't
> > change either.
> >
> > RS> Please don't try to push us to "move on" without establishing
> > RS> the facts of the situation clearly.
> >
> > I think we can move on with other things now.
> >
>
> As a happy emacs user I am sad to see another good idea that helps to
> increase the visibility of emacs is shot down by this oh so important
> free software argument.
>
> For me it is the same thing that happened to the proposed improvements
> in intellisense behaviour. Richard wanted to talk to a lawyer about
> this. And nothing happened. The person that wanted to implement this
> stuff just resigned.
>
> Stefan.
>
>
>

[-- Attachment #2: Type: text/html, Size: 3962 bytes --]

Re: official Emacs Docker image

[Mike Gerwitz]

[-- Attachment #1: Type: text/plain, Size: 3462 bytes --]

Hey, Filipe:

On Fri, Feb 03, 2017 at 09:18:59 -0200, Filipe Silva wrote:
> Mike Gerwitz, back to the topic, you've posted an issue on the github site,
> which is run with non-free software, and I'm pretty sure that it runs
> non-free javascript. The fact that you will be running "non-free"
> javascript seems a bit unescapeable, doesn't it?

I was expecting this point to come up. ;)

I first want to state that I do not recommend that anyone make use of
GitHub for their own projects.  I have contacted them in the past asking
them if they're liberate their JavaScript, and they stated that they
have no interest in doing so:

  https://mikegerwitz.com/about/githubbub

Further, GitHub does not meet the GNU Project's ethical repository
criteria for those reasons and more:

  https://www.gnu.org/software/repo-criteria.html

GitHub does serve and often relies on non-free JavaScript.  Fortunately,
many things happen to work without it, or work well enough.  I do not
run JavaScript in my browser even for sites with free JS, with very few
exceptions.

I'm able to log in to GitHub, open issues, and comment on issues without
running any JavaScript, among other things.  I don't use GitHub
personally anymore aside from a mirror for certain projects (and my
employer uses it, so I'll push liberated code here on their behalf), but
in the past, there were certain things I'd have simply Greasemonkey
scripts for.  For example, you couldn't (and probably still can't)
change the repository description without running non-free JS.  So I
briefly studied what I did and wrote a free replacement for that
functionality (which was trivial).

Now, as a direct parallel to this Docker Hub discussion:

It isn't reasonable to register an account on GitHub without running
non-free JS.[*]  It happens that I registered an account a number of
years back (2009) when I was still relatively new to the free software
movement and didn't consider JS as a possible issue.  Since I have an
account and can use it without running non-free JS, I will use it when
pressed.  But I will never recommend that someone else create an account
or otherwise use GitHub.

Hopefully that answers you question.  Just because a site serves
non-free JS doesn't mean that it needs it.  Websites would be wise and
respectful to consider a progressive enhancement methodology.[0]



[*] I was able to register a new account (which successfully logs in):
The first registration page required no JS, and the second page didn't
want to move forward because of HTML5 validations on hidden (Credit
Card) fields.  I removed those fields from the DOM and was able to
continue successfully.  This also worked over Tor and served no CAPTCHA.
So I could trivially write a script/bookmarklet that a user could use to
register, but I wouldn't want to do that, since they still rely on
proprietary JS for so many other things.  If I did _not_ have a
GitHub account, and creating one were the only way I could report issues
like this Docker Hub one for reasons of activism, I would create an
account using this method.  I have no such workaround for Docker Hub.

[0]: https://en.wikipedia.org/wiki/Progressive_enhancement

-- 
Mike Gerwitz
Free Software Hacker+Activist | GNU Maintainer & Volunteer
GPG: D6E9 B930 028A 6C38 F43B  2388 FEF6 3574 5E6F 6D05
Old: 2217 5B02 E626 BC98 D7C0  C2E5 F22B B815 8EE3 0EAB
https://mikegerwitz.com

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]

Re: official Emacs Docker image

[Ted Zlatanov]

On Fri, 3 Feb 2017 09:18:59 -0200 Filipe Silva <filipe.silva@gmail.com> wrote: 

FS> Stefan, GNU/FSF is first a political/philosophical organisation before
FS> being a technical/software organization. The fact that they've managed to
FS> construct such an amazing piece of software, Emacs, is a miracle, but
FS> secondary to that primary characteristics.
...
FS> Mike Gerwitz, back to the topic, you've posted an issue on the github site,
FS> which is run with non-free software, and I'm pretty sure that it runs
FS> non-free javascript. The fact that you will be running "non-free"
FS> javascript seems a bit unescapeable, doesn't it?

This discrepancy between principles and reality is exactly what I was
referring to. It's not Mike Gerwitz specifically, but the reality is
that there is a vast majority of free software advocates that uses
websites without concern for the Javascript licenses. So the *FSF
principles* that are in reality not observed are blocking a vitally
important way for the *GNU Emacs project* (note they are not the same
organization, so Filipe's merging them somewhat inaccurately) to reach
users, because of a setup step invisible to those users.

Look, principled opposition taken to an extreme is self-defeating.
Imagine if vegans refused to *visit* vegetarians' houses and breathe
their air because they are not pure enough. Meanwhile the
slaughterhouses are running next door.

Ted

Re: official Emacs Docker image

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > It's not Mike Gerwitz specifically, but the reality is
  > that there is a vast majority of free software advocates that uses
  > websites without concern for the Javascript licenses.

It is true that most of them do this.

That is why it is vital for the FSF to spread the message that this
is a bad thing to do.  In order to do that without being hypocritical,
we have to set an example of doing what we say is right.

  >  So the *FSF
  > principles* that are in reality not observed are blocking a vitally
  > important way for the *GNU Emacs project*

Distributing docker images is hardly a vital issue.
But this is not about whether to distribute docker images;
we can certainly do that one way or another.  This is just
about _how_ to distribute them.


Developing GNU Emacs is a part of the GNU Project.
Winning freedom, defeating proprietary injustice, is its purpose.
That's why I wrote GNU Emacs in the first place.

You don't seem to value this purpose very much, so you resent that it
is the basis for our decisions.  But that has to be the basis.

  > Look, principled opposition taken to an extreme is self-defeating.
  > Imagine if vegans refused to *visit* vegetarians' houses and breathe
  > their air because they are not pure enough.

Aside from the visible exaggeration, it's a bad analogy.  For a better
analogy, imagine that the habitants of those houses demanded that
every visitor eat some meat at the door in order to be allowed in.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: official Emacs Docker image

[Ted Zlatanov]

On Fri, 03 Feb 2017 16:59:25 -0500 Richard Stallman <rms@gnu.org> wrote: 

>> the reality is that there is a vast majority of free software
>> advocates that uses websites without concern for the Javascript
>> licenses.

RS> That is why it is vital for the FSF to spread the message that this
RS> is a bad thing to do.  In order to do that without being hypocritical,
RS> we have to set an example of doing what we say is right.
...
RS> Developing GNU Emacs is a part of the GNU Project.
RS> Winning freedom, defeating proprietary injustice, is its purpose.
RS> That's why I wrote GNU Emacs in the first place.

RS> You don't seem to value this purpose very much, so you resent that it
RS> is the basis for our decisions.  But that has to be the basis.

Your implicit assumption here is that if any part of a service involves
nonfree software (in this case, Docker Hub account registration and
maintenance), then using any other part of the service (in this case,
Docker Hub as an image repository) is against this purpose.

That specific assumption, in my opinion, is overreaching and does not
serve the original purpose you quoted, yet it has become an important
argument for the FSF.

Ted

Re: official Emacs Docker image

[Jean-Christophe Helary]

> 2017/02/04 8:38、Ted Zlatanov <tzz@lifelogs.com>のメール:

> Your implicit assumption here is that if any part of a service involves
> nonfree software (in this case, Docker Hub account registration and
> maintenance), then using any other part of the service (in this case,
> Docker Hub as an image repository) is against this purpose.
> 
> That specific assumption, in my opinion, is overreaching and does not
> serve the original purpose you quoted, yet it has become an important
> argument for the FSF.

Instead of arguing here, why don't you ask Docker Hub why they require non-free software for such a trivial task and try to convince them to not do that?
*That* would better promote free software than having Emacs hosted there while ignoring the non-free aspects of the system.
A much better way to spend your energy in my opinion.

Jean-Christophe Helary 

Re: official Emacs Docker image

[Filipe Silva]

[-- Attachment #1: Type: text/plain, Size: 4161 bytes --]

Sometimes people minify js to make better usage of the network, not because
they want to hide from you the js source code to gain more power over you
as a user. There's no practical way to run your specific modified version
of js in a browser to customize your own site experience because the js
code is just a tailor made client for the site's server code.

Look this way: They could have rendered the whole html that you see in the
server side, and still there are tons of websites that do that. and in that
case you would have just the bare html and css in your hands, which doesn't
change the fact that you are using non-free software that generates that
html; still you wouldn't have a problem with that because "you're not
running non-free js code on your browser".

but instead they chose to split the software: they hand it to your browser
a piece of the software in js so that you can have a more dynamic
experience. As a plus your browser doesn't have to call the server all the
time to generate an updated DOM. Now UI logic can be very hard and complex
so instead of handing you a plain js file which can be many KBs long, they
minify it, and maybe gzip it so they can better use the network resources,
potentially saving energy for the benefit off all mankind.

I see this way: It's just a website, but now some of the server's code is
offloaded to the browser by means of js. But it is still a website.

I don't think it is reasonable asking every website to stop minifying the
js and to provide a way to submit a custom modified form of js client code
that interacts with the website in a customized way. That would require to
also release the server code, which would mean that to be free all websites
that deliver js client code to web browsers would have to release their
server code.

Can you all see the problem here? js client code is not the same thing as
regular installed on my pc software.

I have to trust my libre browser, which is in fact installed on my local
computer and which I can modify and inspect the source code. I trust that
it is written in a way that stops client js code from doing harm to my
privacy and freedom.

We have different classes of software here. Software fully installed
locally on my machine is one class of software. Server software, which is
not running on my machine is a different kind of software. And software
that is split between server software and client software *running in a
libre sandboxed environment installed on my machine* is another class of
software entirely.

I would assert that each class of software described above brings about
different philosophical and ethical questions and deserve to be differently
treated.

For starters, I would argue that a js client code, *running on my libre
sandboxed browser environment*, which is really just a part of a much
larger software, which is a website, does not harm my freedom and my
privacy as long as the libre browser is properly constructed to properly
handle js client code.

Doesn't this proposition seem reasonable to you gentleman?

Filipe


On Fri, Feb 3, 2017 at 9:54 PM, Jean-Christophe Helary <
jean.christophe.helary@gmail.com> wrote:

> > 2017/02/04 8:38、Ted Zlatanov <tzz@lifelogs.com>のメール:
>
> > Your implicit assumption here is that if any part of a service involves
> > nonfree software (in this case, Docker Hub account registration and
> > maintenance), then using any other part of the service (in this case,
> > Docker Hub as an image repository) is against this purpose.
> >
> > That specific assumption, in my opinion, is overreaching and does not
> > serve the original purpose you quoted, yet it has become an important
> > argument for the FSF.
>
> Instead of arguing here, why don't you ask Docker Hub why they require
> non-free software for such a trivial task and try to convince them to not
> do that?
> *That* would better promote free software than having Emacs hosted there
> while ignoring the non-free aspects of the system.
> A much better way to spend your energy in my opinion.
>
> Jean-Christophe Helary
>

[-- Attachment #2: Type: text/html, Size: 6465 bytes --]

Re: official Emacs Docker image

[Jean-Christophe Helary]

> 2017/02/04 9:37、Filipe Silva <filipe.silva@gmail.com>のメール:
> 
> Sometimes people minify js to make better usage of the network,

That's totally unrelated to the issue. Free software is not "non minified" software. It is software available under a specific licence.
Any "minified" js code can be free software if available under such a licence.

Jean-Christophe Helary

Re: official Emacs Docker image

[Filipe Silva]

[-- Attachment #1: Type: text/plain, Size: 556 bytes --]

that's one nitpick over my many points.

On Fri, Feb 3, 2017 at 11:12 PM, Jean-Christophe Helary <
jean.christophe.helary@gmail.com> wrote:

>
> > 2017/02/04 9:37、Filipe Silva <filipe.silva@gmail.com>のメール:
> >
> > Sometimes people minify js to make better usage of the network,
>
> That's totally unrelated to the issue. Free software is not "non minified"
> software. It is software available under a specific licence.
> Any "minified" js code can be free software if available under such a
> licence.
>
> Jean-Christophe Helary

[-- Attachment #2: Type: text/html, Size: 1107 bytes --]

Re: official Emacs Docker image

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > That's totally unrelated to the issue. Free software is not "non minified" software. It is software available under a specific licence.
  > Any "minified" js code can be free software if available under such a licence.

Actually, _both_ of these criteria are necessary.  To be free, a
Javascript program must be (1) covered by some free software license
(see https://gnu.org/philosophy/free-sw.html and
https://gnu.org/licenses/license-list.html) and (2) available in souce
code form.

Source code means the preferred form of the code for editing it.
Obfuscated code (including minified Javascript code) is not source
code.  It is a kind of compiled code.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: official Emacs Docker image

[Jean-Christophe Helary]

> 2017/02/05 8:52、Richard Stallman <rms@gnu.org>のメール:
> 
> Source code means the preferred form of the code for editing it.
> Obfuscated code (including minified Javascript code) is not source
> code.  It is a kind of compiled code.

Apologies if my wording was unclear. Since a Free licence implies access to editable source code, the following:

> Any "minified" js code can be free software if available under such a licence.

meant for me that we have access to a readable version of the source code, which exclude obfuscated code.

Jean-Christophe Helary

Re: official Emacs Docker image

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Look this way: They could have rendered the whole html that you see in the
  > server side, and still there are tons of websites that do that. and in that
  > case you would have just the bare html and css in your hands,

That's the way they SHOULD do it.

								  which doesn't
  > change the fact that you are using non-free software that generates that
  > html;

No you aren't.  That code isn't working for you.  It's working for the
owner of the web server, and that's who should have control over it.

  > but instead they chose to split the software: they hand it to your browser
  > a piece of the software in js so that you can have a more dynamic
  > experience.

That's exactly what's wrong: trying to run their code on my computer,
without giving me control over it.

I don't expect or ask to have control over the code that runs on their
web server.  That code is doing their computing(*), so I think they
ought to have control over it.  I don't have a copy of it in any form
-- neither source nor binary.

However, I insist on having control over the code that runs on my
computer.  It has to be free, and I should be able to install a
different version and change it.

  > I trust that
  > it is written in a way that stops client js code from doing harm to my
  > privacy and freedom.

If you're running it and you don't have control over it, that is
denying you freedom.

See https://gnu.org/philosophy/javascript-trap.html.


* That's nornally the case, but there are exceptions.  If that code is
doing my computing, then it is SaaSS, which is wrong.  See
https://gnu.org/philosophy/who-does-that-server-really-serve.html.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: official Emacs Docker image

[Mike Gerwitz]

[-- Attachment #1: Type: text/plain, Size: 1580 bytes --]

On Fri, Feb 03, 2017 at 18:38:41 -0500, Ted Zlatanov wrote:
> Your implicit assumption here is that if any part of a service involves
> nonfree software (in this case, Docker Hub account registration and
> maintenance), then using any other part of the service (in this case,
> Docker Hub as an image repository) is against this purpose.

If using X involves temporarily setting aside freedom, then that harms
your freedom, but that's your choice.  But if you then ask others to use
X, and therefore temporarily set aside their freedoms, that is a
different situation entirely.  Further, complacency in that action from
a large enough group will cement it; we need people to speak out against
those types of things so that they stop happening.

If there is an alternative means to register with Docker Hub, then
perhaps it wouldn't be a barrier, because we wouldn't have to ask others
to temporarily surrender their freedoms.  For example, if Docker Hub
offered a register@hub.docker.com e-mail address where someone manually
registered an account on your behalf to circumvent the issue, then
it wouldn't be a problem for as long as that e-mail service remained
available to others.  If someone developed a free replacement for their
proprietary JS that allowed for registration, it wouldn't be an issue
for as long as that script works.

-- 
Mike Gerwitz
Free Software Hacker+Activist | GNU Maintainer & Volunteer
GPG: D6E9 B930 028A 6C38 F43B  2388 FEF6 3574 5E6F 6D05
Old: 2217 5B02 E626 BC98 D7C0  C2E5 F22B B815 8EE3 0EAB
https://mikegerwitz.com

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]

Re: official Emacs Docker image

[Ted Zlatanov]

On Fri, 03 Feb 2017 22:11:31 -0500 Mike Gerwitz <mtg@gnu.org> wrote: 

MG> On Fri, Feb 03, 2017 at 18:38:41 -0500, Ted Zlatanov wrote:
>> Your implicit assumption here is that if any part of a service involves
>> nonfree software (in this case, Docker Hub account registration and
>> maintenance), then using any other part of the service (in this case,
>> Docker Hub as an image repository) is against this purpose.

MG> If using X involves temporarily setting aside freedom, then that harms
MG> your freedom, but that's your choice.  But if you then ask others to use
MG> X, and therefore temporarily set aside their freedoms, that is a
MG> different situation entirely.

Mike, I understand all of that. The assumption RMS and others are making
is that we're talking about the same X, as I said in the text you
quoted. That's what I mean by "overreaching": the assumption is that any
non-free software in a service (in this case, a web site) contaminates
any other part of the service (in this case, an image repository).

Coming back to the vegan/vegetarian analogy, if I eat non-vegan foods in
my house but not while you visit, does that mean you can never be a
guest in my house? This is not an extreme analogy: we're literally
excluding tools, people, and communities because of things they choose
to do outside of our domain.

MG> If there is an alternative means to register with Docker Hub, then
MG> perhaps it wouldn't be a barrier, because we wouldn't have to ask others
MG> to temporarily surrender their freedoms.

That would legitimize the assumption I'm questioning, so I would rather
not divert the discussion down that path.

Ted

Re: official Emacs Docker image

[Mike Gerwitz]

[-- Attachment #1: Type: text/plain, Size: 1954 bytes --]

On Fri, Feb 03, 2017 at 23:13:58 -0500, Ted Zlatanov wrote:
> Mike, I understand all of that. The assumption RMS and others are making
> is that we're talking about the same X, as I said in the text you
> quoted. That's what I mean by "overreaching": the assumption is that any
> non-free software in a service (in this case, a web site) contaminates
> any other part of the service (in this case, an image repository).

The use of the underlying repository ("registry") is fine: it doesn't
require any non-free software.  I can use any image in the Docker Hub
repository without any ethical concerns.  If there's an Emacs docker
image, then I as a user of that image lose nothing freedom-wise.

The problem is that to upload an image you need an account on Docker
Hub.  And to register that account, you need to go to hub.docker.com,
which requires that you run a non-free JavaScript program.

So it's only the JavaScript program on hub.docker.com that's
bad.  Unfortunately, it's a necessary step to get the Emacs image into
the repository to begin with.

So I think we might be in agreement: there's nothing wrong with the
registry.

> Coming back to the vegan/vegetarian analogy, if I eat non-vegan foods in
> my house but not while you visit, does that mean you can never be a
> guest in my house? This is not an extreme analogy: we're literally
> excluding tools, people, and communities because of things they choose
> to do outside of our domain.

In rms' analogy, your house would be a place to share vegan meals, but
to bring a meal to share with others, you must first try a dish
containing meat.  But if you're not bringing a dish to share, you can
simply enter and enjoy the festivities.

-- 
Mike Gerwitz
Free Software Hacker+Activist | GNU Maintainer & Volunteer
GPG: D6E9 B930 028A 6C38 F43B  2388 FEF6 3574 5E6F 6D05
Old: 2217 5B02 E626 BC98 D7C0  C2E5 F22B B815 8EE3 0EAB
https://mikegerwitz.com

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]

Re: official Emacs Docker image

[Giuseppe Scrivano]

Mike Gerwitz <mtg@gnu.org> writes:

> The problem is that to upload an image you need an account on Docker
> Hub.  And to register that account, you need to go to hub.docker.com,
> which requires that you run a non-free JavaScript program.

While the registration page doesn't work without JS, from a quick view I
can see only some trivial JS code there.  I thought it was fine to not
consider such code as a program.  Or is there anything more than that?

Regards,
Giuseppe

Re: official Emacs Docker image

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Please forgive me for taking a week to answer.

  > While the registration page doesn't work without JS, from a quick view I
  > can see only some trivial JS code there.  I thought it was fine to not
  > consider such code as a program.  Or is there anything more than that?

Maybe.  Could you try accessing that using LibreJS and see if it says
that code is trivial?

Can you show us the complete JS code involved?

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: official Emacs Docker image

[Giuseppe Scrivano]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> Please forgive me for taking a week to answer.
>
>   > While the registration page doesn't work without JS, from a quick view I
>   > can see only some trivial JS code there.  I thought it was fine to not
>   > consider such code as a program.  Or is there anything more than that?
>
> Maybe.  Could you try accessing that using LibreJS and see if it says
> that code is trivial?
>
> Can you show us the complete JS code involved?

LibreJS reports these three chunks of code:

(
        function(w,d,s,l,i){
          w[l]=w[l]||[];
          w[l].push({'gtm.start': new Date().getTime(),event:'gtm.js'});
          var f=d.getElementsByTagName(s)[0],
              j=d.createElement(s),
              dl=l!='dataLayer'?'&l='+l:'';
          j.async=true;
          j.src='//www.googletagmanager.com/gtm.js?id='+i+dl;
          f.parentNode.insertBefore(j,f);
        }
      )(window,document,'script','dataLayer','GTM-KB4JTX');

/////////////////////////////////////////////////////////////////////////

(
      function(){
        window._pxAppId ='PXPmP8ILuI';
        window._pxPubHost = 'collector.a';
        var p = document.getElementsByTagName('script')[0],
        s = document.createElement('script');
        s.async = 1;
        s.src = '//client.a.pxi.pub/PXPmP8ILuI/main.min.js';
        p.parentNode.insertBefore(s,p);
      }());

/////////////////////////////////////////////////////////////////////////

window.App={"context":{"dispatcher":{"stores":{"SSOStore":{"isSSOEnabled":true},"ApplicationStore":{"route":{"routes":[{"name":"app","component":function StoreConnector(props, context) {
            React.Component.apply(this, arguments);
            this.state = this.getStateFromStores();
            this._onStoreChange = null;
            this._isMounted = false;
        },"childRoutes":[{"name":"login","path":"\u002Flogin\u002F","component":function (props, context, updater) {
      // This constructor is overridden by mocks. The argument is used
      // by mocks to assert on what gets mounted.
 
      if (process.env.NODE_ENV !== 'production') {
        process.env.NODE_ENV !== 'production' ? warning(this instanceof Constructor, 'Something is calling a React component directly. Use a factory or ' + 'JSX instead. See: https://fb.me/react-legacyfactory') : undefined;
      }
 
      // Wire up auto-binding
      if (this.__reactAutoBindMap) {
        bindAutoBindMethods(this);
      }
 
      this.pr…

Regards,
Giuseppe

Re: official Emacs Docker image

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > (
  >         function(w,d,s,l,i){
  >           w[l]=w[l]||[];
  >           w[l].push({'gtm.start': new Date().getTime(),event:'gtm.js'});
  >           var f=d.getElementsByTagName(s)[0],
  >               j=d.createElement(s),
  >               dl=l!='dataLayer'?'&l='+l:'';
  >           j.async=true;
  >           j.src='//www.googletagmanager.com/gtm.js?id='+i+dl;
  >           f.parentNode.insertBefore(j,f);
  >         }
  >       )(window,document,'script','dataLayer','GTM-KB4JTX');


That appears to be spywaqre, and there should be no problem if it
does not execute.

  > (
  >       function(){
  >         window._pxAppId ='PXPmP8ILuI';
  >         window._pxPubHost = 'collector.a';
  >         var p = document.getElementsByTagName('script')[0],
  >         s = document.createElement('script');
  >         s.async = 1;
  >         s.src = '//client.a.pxi.pub/PXPmP8ILuI/main.min.js';
  >         p.parentNode.insertBefore(s,p);
  >       }());

I'd say that is trivial.  What doesw LibreJS say about it?

  > window.App={"context":{"dispatcher":{"stores":{"SSOStore":{"isSSOEnabled":true},"ApplicationStore":{"route":{"routes":[{"name":"app","component":function StoreConnector(props, context) {
  >             React.Component.apply(this, arguments);
  >             this.state = this.getStateFromStores();
  >             this._onStoreChange = null;
  >             this._isMounted = false;
  >         },"childRoutes":[{"name":"login","path":"\u002Flogin\u002F","component":function (props, context, updater) {
  >       // This constructor is overridden by mocks. The argument is used
  >       // by mocks to assert on what gets mounted.
 
  >       if (process.env.NODE_ENV !== 'production') {
  >         process.env.NODE_ENV !== 'production' ? warning(this instanceof Constructor, 'Something is calling a React component directly. Use a factory or ' + 'JSX instead. See: https://fb.me/react-legacyfactory') : undefined;
  >       }
 
  >       // Wire up auto-binding
  >       if (this.__reactAutoBindMap) {
  >         bindAutoBindMethods(this);
  >       }
 
  >       this.pr…

This seems to break off in the middle -- 
what's the rest?

I think it is nontrivial.  But I can't imagine why they would
object to putting a free license on it.  Can someone please ask them?

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: official Emacs Docker image

[Mike Gerwitz]

[-- Attachment #1: Type: text/plain, Size: 694 bytes --]

On Mon, Feb 13, 2017 at 19:48:41 -0500, Richard Stallman wrote:
> I think it is nontrivial.  But I can't imagine why they would
> object to putting a free license on it.  Can someone please ask them?

(See my previous message---there's a lot of potentially non-free code.)

I asked them on the 1st of this month.  Not even an acknowledgment:

  https://github.com/docker/hub-feedback/issues/946

Surely there is someone on this list who knows a Docker developer.

-- 
Mike Gerwitz
Free Software Hacker+Activist | GNU Maintainer & Volunteer
GPG: D6E9 B930 028A 6C38 F43B  2388 FEF6 3574 5E6F 6D05
Old: 2217 5B02 E626 BC98 D7C0  C2E5 F22B B815 8EE3 0EAB
https://mikegerwitz.com

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]

Re: official Emacs Docker image

[Mike Gerwitz]

[-- Attachment #1: Type: text/plain, Size: 1381 bytes --]

Giuseppe:

On Mon, Feb 06, 2017 at 11:49:03 +0100, Giuseppe Scrivano wrote:
> While the registration page doesn't work without JS, from a quick view I
> can see only some trivial JS code there.  I thought it was fine to not
> consider such code as a program.  Or is there anything more than that?

Sorry, I had been meaning to reply to this for a while; I've been a bit
behind.

hub.docker.com actually loads a great deal of proprietary/unlicensed
JavaScript:

The hub.docker.com index alone has ~45K of JS:

  $ curl -s https://hub.docker.com \
    | tr -d '\n' \
    | grep -oP '<script>.*?</script>' \
    | wc -c
  45039

It loads a 3.97MB /public/js/client.*.js ("*" being a hash), which
is a minified and concatenated file containing a huge number of
individual scripts, which can be seen by loading it in your browser's
debugger (so long as it has source map support).  Some of them are
shared (and free) libraries, many are Docker Hub code.

There is JS loaded on their account site as well, which the user is
redirected to on login.  I don't have my work account password atm, so
I'm not going to dig into that right now.

-- 
Mike Gerwitz
Free Software Hacker+Activist | GNU Maintainer & Volunteer
GPG: D6E9 B930 028A 6C38 F43B  2388 FEF6 3574 5E6F 6D05
Old: 2217 5B02 E626 BC98 D7C0  C2E5 F22B B815 8EE3 0EAB
https://mikegerwitz.com

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]

Re: official Emacs Docker image

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Mike, I understand all of that. The assumption RMS and others are making
  > is that we're talking about the same X, as I said in the text you
  > quoted.

I don't KNOW which entities are involved and what each one does.
I don't know which of them require nonfree javascropt.

I've asked repeatedly for clarification about this,
but I haven't seen a clear self-contained complete description
of what these various Docker entities are and what job each one does.
I've seen a number of partial descriptions, but I can't fit them together.

That's why I keep making statements with if conditions.
Those conditions are about the relationships I don't know.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: official Emacs Docker image

[Ted Zlatanov]

On Sat, 04 Feb 2017 18:48:55 -0500 Richard Stallman <rms@gnu.org> wrote: 

>> At this point I've lost interest and will not continue this work. I
>> don't see a way to reconcile your positions with my reality, 

RS> We share the same reality.  The clash is between your values and the
RS> GNU Project values.

I'm sorry you feel that way and sincerely hope we are done with this topic.

Ted

Re: official Emacs Docker image

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > At this point I've lost interest and will not continue this work. I
  > don't see a way to reconcile your positions with my reality, 

We share the same reality.  The clash is between your values and the
GNU Project values.


-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: official Emacs Docker image

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > After talking to RMS, I created https://hub.docker.com/u/fsfemacs and am
  > currently the owner of the organization. The image, when ready, will be
  > "fsfemacs/emacs". I named it that way to emphasize the FSF organization,
  > rather than "gnuemacs" which would emphasize the GNU project.

For this activity it is more correct to emphasize GNU rather than the FSF.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: official Emacs Docker image

[Philippe Vaucher]

[-- Attachment #1: Type: text/plain, Size: 423 bytes --]

>
> If you're interested in collaborating on a Dockerfile that builds 24.x
> and 25.x/master, let me know here or in e-mail. If you have one already,
> even better.
>

Here is the one I used when emacs did still build in a docker
https://github.com/Silex/docker-emacs/blob/master/24.5/Dockerfile

It was meant to be used like "docker run -it --rm -e DISPLAY=$DISPLAY -v
/tmp/.X11-unix:/tmp/.X11-unix silex/emacs"

Philippe

[-- Attachment #2: Type: text/html, Size: 812 bytes --]

Re: official Emacs Docker image

[Ted Zlatanov]

On Tue, 21 Feb 2017 16:01:22 +0100 Philippe Vaucher <philippe.vaucher@gmail.com> wrote: 

>> If you're interested in collaborating on a Dockerfile that builds 24.x
>> and 25.x/master, let me know here or in e-mail. If you have one already,
>> even better.

PV> Here is the one I used when emacs did still build in a docker
PV> https://github.com/Silex/docker-emacs/blob/master/24.5/Dockerfile

PV> It was meant to be used like "docker run -it --rm -e DISPLAY=$DISPLAY -v
PV> /tmp/.X11-unix:/tmp/.X11-unix silex/emacs"

Thank you, Philippe. I'll be glad to maintain and publish an *unofficial*
Emacs Docker image with your help, once 25.x can build inside Docker.
For now I guess it's not possible?

Ted

Re: official Emacs Docker image

[Philippe Vaucher]

[-- Attachment #1: Type: text/plain, Size: 718 bytes --]

>
> Thank you, Philippe. I'll be glad to maintain and publish an *unofficial*
> Emacs Docker image with your help, once 25.x can build inside Docker.
> For now I guess it's not possible?
>

For now it's only possible in two ways:

- "manually": by building docker on the host and copying the binaries
inside the image, then pushing to the docker hub.
- By using my Dockerfile but disabling /proc/sys/kernel/randomize_va_space,
and then pushing the image.

Both of these solutions are not good in that they don't use the automated
build feature of the docker hub.

Ideally, you could simply push the Dockerfile inside the emacs repo and the
docker hub would build emacs for each new tag/branch automatically.

Philippe

[-- Attachment #2: Type: text/html, Size: 1096 bytes --]

Re: official Emacs Docker image

[Philippe Vaucher]

[-- Attachment #1: Type: text/plain, Size: 427 bytes --]

>
> - By using my Dockerfile but disabling /proc/sys/kernel/randomize_va_space,
> and then pushing the image.
>

Well, I just had some kind of epiphany when explaining that to you I then
asked myself: "why don't I do that?"

I think I'll resume my project, something that monitors the emacs
repository and builds-push docker images to the docker hub when necessary.

It'll contain images for 24.3+ and latest master.

Philippe

[-- Attachment #2: Type: text/html, Size: 835 bytes --]

Re: official Emacs Docker image

[Philippe Vaucher]

[-- Attachment #1: Type: text/plain, Size: 411 bytes --]

>
> I think I'll resume my project, something that monitors the emacs
> repository and builds-push docker images to the docker hub when necessary.
> It'll contain images for 24.3+ and latest master.
>

For information, https://hub.docker.com/r/silex/emacs/ is up & running.

Offered images are 24.3, 24.4, 24.5, 25.1 and master branch.

I plan on adding `*-rc1` images, allowing people to test early.

Philippe

[-- Attachment #2: Type: text/html, Size: 892 bytes --]

Re: official Emacs Docker image

[Philippe Vaucher]

[-- Attachment #1: Type: text/plain, Size: 142 bytes --]

>
> Offered images are 24.3, 24.4, 24.5, 25.1 and master branch.
>

Added `25.2-rc2` image at https://hub.docker.com/r/silex/emacs.

Philippe

[-- Attachment #2: Type: text/html, Size: 615 bytes --]

Re: official Emacs Docker image

[Philippe Vaucher]

[-- Attachment #1: Type: text/plain, Size: 404 bytes --]

>
> For information, https://hub.docker.com/r/silex/emacs is up & running.
>>
>
Hello,

I thought it might interest some that quite some work was done recently,
there is now images from 23.4 and upward (including master and 26.0.91),
`-alpine` variants, `-dev` variants that contains the Emacs source, the dev
tools & Cask so they can be easily used to build other variants, etc.

Kind regards,
Philippe

[-- Attachment #2: Type: text/html, Size: 1138 bytes --]

Re: official Emacs Docker image

[Ricardo Wurmus]

Richard Stallman <rms@gnu.org> writes:

> However, I don't see any ethical issue about making and distributing
> Docker images of Emacs as long as we get the details right: for
> instance, use an endorsed free GNU/Linux distro.

We could use Guix to generate the image contents from scratch, i.e. we
wouldn’t even have to use a “base image” (usually a minimal GNU/Linux
system).  The idea is to export the package closure for the “emacs”
package from the Guix store (this includes all runtime dependencies) and
wrap up the files in a plain docker image.

200MB is quite optimistic, though.  A fully functional GTK+ installation
with all dependencies is already larger than that.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
http://elephly.net

Re: official Emacs Docker image

[Elias Mårtenson]

[-- Attachment #1: Type: text/plain, Size: 779 bytes --]

On 1 January 2017 at 06:04, Ricardo Wurmus <rekado@elephly.net> wrote:

>
> Richard Stallman <rms@gnu.org> writes:
>
> > However, I don't see any ethical issue about making and distributing
> > Docker images of Emacs as long as we get the details right: for
> > instance, use an endorsed free GNU/Linux distro.
>
> We could use Guix to generate the image contents from scratch, i.e. we
> wouldn’t even have to use a “base image” (usually a minimal GNU/Linux
> system).  The idea is to export the package closure for the “emacs”
> package from the Guix store (this includes all runtime dependencies) and
> wrap up the files in a plain docker image.
>

Arguably there should be a fully free distribution that could be the base
for more than just Emacs.

[-- Attachment #2: Type: text/html, Size: 1182 bytes --]

Re: official Emacs Docker image

[Ricardo Wurmus]

Elias Mårtenson <lokedhs@gmail.com> writes:

> On 1 January 2017 at 06:04, Ricardo Wurmus <rekado@elephly.net> wrote:
>
>>
>> Richard Stallman <rms@gnu.org> writes:
>>
>> > However, I don't see any ethical issue about making and distributing
>> > Docker images of Emacs as long as we get the details right: for
>> > instance, use an endorsed free GNU/Linux distro.
>>
>> We could use Guix to generate the image contents from scratch, i.e. we
>> wouldn’t even have to use a “base image” (usually a minimal GNU/Linux
>> system).  The idea is to export the package closure for the “emacs”
>> package from the Guix store (this includes all runtime dependencies) and
>> wrap up the files in a plain docker image.
>>
>
> Arguably there should be a fully free distribution that could be the base
> for more than just Emacs.

GuixSD is a fully free GNU distribution and it comes with the tools that
are needed to determine runtime dependencies that can then turned into a
Docker image.  Since Guix is a functional package manager and thus can
account for the full closure of a package it doesn’t require a base
image and can be the base itself.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
http://elephly.net

Re: official Emacs Docker image

[Elias Mårtenson]

[-- Attachment #1: Type: text/plain, Size: 825 bytes --]

On 1 January 2017 at 17:03, Ricardo Wurmus <rekado@elephly.net> wrote:

>
> Elias Mårtenson <lokedhs@gmail.com> writes:
>
> > Arguably there should be a fully free distribution that could be the base
> > for more than just Emacs.
>
> GuixSD is a fully free GNU distribution and it comes with the tools that
> are needed to determine runtime dependencies that can then turned into a
> Docker image.  Since Guix is a functional package manager and thus can
> account for the full closure of a package it doesn’t require a base
> image and can be the base itself.


I have to admit that I don't fully know what Guix is, but based on what you
say, it would still make sense to have a base (Guix, in this case) that
other containers base themselves on. That way the storage can be shared
between containers.

[-- Attachment #2: Type: text/html, Size: 1248 bytes --]

Re: official Emacs Docker image

[Ricardo Wurmus]

Elias Mårtenson <lokedhs@gmail.com> writes:

> On 1 January 2017 at 17:03, Ricardo Wurmus <rekado@elephly.net> wrote:
>
>>
>> Elias Mårtenson <lokedhs@gmail.com> writes:
>>
>> > Arguably there should be a fully free distribution that could be the base
>> > for more than just Emacs.
>>
>> GuixSD is a fully free GNU distribution and it comes with the tools that
>> are needed to determine runtime dependencies that can then turned into a
>> Docker image.  Since Guix is a functional package manager and thus can
>> account for the full closure of a package it doesn’t require a base
>> image and can be the base itself.
>
>
> I have to admit that I don't fully know what Guix is, but based on what you
> say, it would still make sense to have a base (Guix, in this case) that
> other containers base themselves on. That way the storage can be shared
> between containers.

Guix is a functional package manager.  Given a package it can extract
the “closure” of the package, i.e. the package and all its dependencies,
recursively.

Since commit 03476a23f Guix can build Docker images without requiring
Docker, simply by recursively exporting a package along with its
dependencies, all the way down to things like the glibc.

Here’s how to build a Docker image of Emacs (along with coreutils and
bash, so that M-x shell and a couple more things work out of the box):

    guix environment --ad-hoc emacs-no-x-toolkit coreutils bash
    guix archive --export -f docker $GUIX_ENVIRONMENT

The generated archive can the be loaded with “docker load”.  A user can
then run Emacs with the usual “docker run … $imageid /bin/emacs”
invocation.

For simpler packages it would be sufficient to do something like this:

    guix archive --export -f docker $(guix build the-package)

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
http://elephly.net

Re: official Emacs Docker image

[Philippe Vaucher]

[-- Attachment #1: Type: text/plain, Size: 271 bytes --]

>
> Since commit 03476a23f Guix can build Docker images without requiring
> Docker, simply by recursively exporting a package along with its
> dependencies, all the way down to things like the glibc.
>

That is very interesting! Thank you for mentionning that.

Philippe

[-- Attachment #2: Type: text/html, Size: 541 bytes --]

Re: official Emacs Docker image

[Yann Hodique]

>>>>> "John" == John Wiegley <jwiegley@gmail.com> writes:

>>>>> Richard Stallman <rms@gnu.org> writes:
>>> The final "image" as they call it will be an Emacs binary, built on that
>>> base, that users can download and directly run from any of the major
>>> operating systems (Mac OS X, Windows, any flavor of GNU/Linux).

>> At this point, I am confused, because the statements seem to conflict.

>> Would the "Docker image" of Emacs _include_ the base system? Or would it be
>> an executable Emacs package that could be installed straight _on top of_
>> that base system?

> I see now that I was unclear: A Docker image is a self-contained tarball
> containing a GNU/Linux kernel, necessary system software, and the final Emacs
> executable that was built by the image recipe.

That is not accurate: the Linux kernel is intentionally *not* part of
the Docker image, since it's meant to be shared between all container
instances and provide the runtime constructs (cgroups, namespaces, ...)
that Docker relies on.

Instead it is provided by the system running the Docker platform: your
regular GNU/Linux distribution, or for other systems typically a virtual
machine running a minimal system (with at least a Linux kernel, and
often not much more) dedicated to running only Docker.

Thanks

Yann.

-- 
Speak the truth.  That is always much easier,
and is often the most powerful argument.

  -- Bene Gesserit Axiom

Re: official Emacs Docker image

[John Wiegley]

>>>>> "YH" == Yann Hodique <yann.hodique@gmail.com> writes:

YH> Instead it is provided by the system running the Docker platform: your
YH> regular GNU/Linux distribution, or for other systems typically a virtual
YH> machine running a minimal system (with at least a Linux kernel, and often
YH> not much more) dedicated to running only Docker.

Thank you for the clarification, you are quite right:

    http://superuser.com/questions/889472/docker-containers-have-their-own-kernel-or-not

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: official Emacs Docker image

[Alex Bennée]

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> Richard Stallman <rms@gnu.org> writes:
>
>>> The final "image" as they call it will be an Emacs binary, built on that
>>> base, that users can download and directly run from any of the major
>>> operating systems (Mac OS X, Windows, any flavor of GNU/Linux).
>
>> At this point, I am confused, because the statements seem to conflict.
>
>> Would the "Docker image" of Emacs _include_ the base system? Or would it be
>> an executable Emacs package that could be installed straight _on top of_
>> that base system?
>
> I see now that I was unclear: A Docker image is a self-contained tarball
> containing a GNU/Linux kernel, necessary system software, and the final Emacs
> executable that was built by the image recipe.

Just a minor correction, Docker images do not contain kernels. They are
purely a user-space construct. They are built to use the Linux kernel
ABI and are virtualised to the extent they have their own
pid/network/filesystem namespaces.

> The Docker image contents, thus, can be entirely free software. Executing the
> image on some platforms (such as Windows) may use proprietary software to
> perform the execution (for example, VM provisioning software).

--
Alex Bennée

Re: official Emacs Docker image

[Philippe Vaucher]

[-- Attachment #1: Type: text/plain, Size: 767 bytes --]

>
>   > You create a Docker build recipe by picking a "base" GNU/Linux
> distribution
>   > (some variant of GNU/Linux that has an absolute minimum of included
> programs),
>   > a recipe to install build tools, and then a recipes to build the target
>   > program. There are many ways to optimize this so that the result is as
> small
>   > as possible, but that's the basics in a nutshell.
>

I missed the start of this thread, but I just wanted to mention that Emacs
cannot build inside a docker container at the moment because of the
personality syscall.

See https://debbugs.gnu.org/cgi/bugreport.cgi?bug=23529

The result of the "portable dumper" thread is likely to yield a solution to
this problem (wether we end up using a portable dumper or not).

Philippe

[-- Attachment #2: Type: text/html, Size: 1218 bytes --]

Re: official Emacs Docker image

[Ted Zlatanov]

On Tue, 12 Jul 2016 15:45:57 -0700 John Wiegley <jwiegley@gmail.com> wrote: 

>>>>>> "TZ" == Ted Zlatanov <tzz@lifelogs.com> writes:
TZ> Docker Inc., who runs the Docker Hub. They essentially own the top
TZ> namespace, so pulling "ubuntu" gets the official Ubuntu image, while
TZ> "rms/ubuntu" is under the "rms" (usually a user name) namespace.

JW> Wouldn't the "official" Docker image for Emacs be some "official" base OS,
JW> such as debian:wheezy, and then the current Emacs release tarball built with
JW> default options upon that base?

It's a package from the user's viewpoint. Package maintenance is more
involved than just installing a tarball. Docker Inc. provides security
scans, automated builds, testing, and some user support for official
images.

JW> I say this because there are already "official" versions of Ubuntu and Emacs,
JW> released on their respective websites, and Docker is just combining them into
JW> another delivery form. Since they don't add or change anything about the code
JW> being deployed, the officialness of Emacs running in a container should be
JW> transitively determined by where its sources came from.

JW> In all of this the FSF needn't be involved, except to be sure that whoever is
JW> packaging these Emacs containers isn't adding or changing anything from what
JW> is released on ftp.gnu.org.

When you get the official "nginx" Docker image, you are usually not
aware (unless you dig) that it has other GNU or generally free/non-free
software. The software you run is hidden. Often there are many people
packaging the same software in different ways for different purposes.
That's why I suggested official Docker images with FSF/GNU branding
might be good for Docker users, if only to present a consistently free
package.

Ted

Re: Automate Emacs UI testing?

[Mathieu Lirzin]

Hi,

joakim@verona.se writes:

> This is the testing scenario I have worked on.
>
> - Isolate an Emacs build and environment in a Docker container
>   environment. This works well, but it's a little bit tricky getting the
>   containerized Emacs to access the host screen. It is a nice strategy,
>   because you can easily simulate a number of gnu/linux distributions. I
>   used this strategy for my xwidget branch.

I don't think this is the right way to “easily” simulate different
GNU/linux distributions, because IIUC having one Docker container for
each distro implies maintaining them individually.

But since GNU/linux distributions differ only by having different
combinaisons of packages or versions, So it is more lightweight and
maintainable to use a package manager like GNU Guix which handles the
installation of multiple versions of the same package and provides tools
to create isolated environments for each combinaison:

  https://www.gnu.org/software/guix/manual/guix.html#Invoking-guix-environment

WDYT?

--
Mathieu Lirzin

Re: Automate Emacs UI testing?

[joakim]

Mathieu Lirzin <mthl@gnu.org> writes:

> Hi,
>
> joakim@verona.se writes:
>
>> This is the testing scenario I have worked on.
>>
>> - Isolate an Emacs build and environment in a Docker container
>>   environment. This works well, but it's a little bit tricky getting the
>>   containerized Emacs to access the host screen. It is a nice strategy,
>>   because you can easily simulate a number of gnu/linux distributions. I
>>   used this strategy for my xwidget branch.
>
> I don't think this is the right way to “easily” simulate different
> GNU/linux distributions, because IIUC having one Docker container for
> each distro implies maintaining them individually.
>
> But since GNU/linux distributions differ only by having different
> combinaisons of packages or versions, So it is more lightweight and
> maintainable to use a package manager like GNU Guix which handles the
> installation of multiple versions of the same package and provides tools
> to create isolated environments for each combinaison:
>
>   https://www.gnu.org/software/guix/manual/guix.html#Invoking-guix-environment
>
> WDYT?

Sure, but in my case I wanted to explicitly test different
distributions, and also that the distribution packaging worked. I wanted
this for my xwidget branch. I'm not saying Docker is the correct
solution in all scenarios.

>
> --
> Mathieu Lirzin

-- 
Joakim Verona

Re: Move to a cadence release model?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I have been impressed with open source projects that have made the change.
  > I am now employed at Mathworks which ships mission critical software to
  > very large enterprise customers on a 6 month cadence.

We don't consider Emacs an "open source" project, but that doesn't
mean we can't think about this proposal.  However, you haven't said
what it means.  Can you please spell out in concrete terms what you
have in mind?

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Move to a cadence release model?

[Xue Fuqiao]

On Wed, Nov 11, 2015 at 2:20 AM, Richard Stallman <rms@gnu.org> wrote:

Hi Richard,

>   > I have been impressed with open source projects that have made the change.
>   > I am now employed at Mathworks which ships mission critical software to
>   > very large enterprise customers on a 6 month cadence.
>
> We don't consider Emacs an "open source" project, but that doesn't
> mean we can't think about this proposal.  However, you haven't said
> what it means.  Can you please spell out in concrete terms what you
> have in mind?

I think what John had in mind was to make Emacs release earlier and more
frequently (i.e., be time-based instead of feature-based), thus creating
a tight feedback loop between developers and users and eliminating the
risk of creating a new release that no one will use.  This means that
users will see smaller changes more frequently.

Some examples of this model are Linux (a new release every few months,
although there is a separate set of "stable" branches), Firefox (a new
release every six weeks), Chromium (roughly the same as Firefox), and
LibreOffice (six monthly releases).

Re: Move to a cadence release model?

[John Wiegley]

>>>>> Xue Fuqiao <xfq.free@gmail.com> writes:

> I think what John had in mind was to make Emacs release earlier and more
> frequently (i.e., be time-based instead of feature-based), thus creating a
> tight feedback loop between developers and users and eliminating the risk of
> creating a new release that no one will use. This means that users will see
> smaller changes more frequently.

I would indeed like to see several .x releases within a cycle, fairly evenly
spaced. x.y releases would wait until we have a larger set of features ready.

Doing this properly may mean dividing how we develop Emacs, though, with
active development on both the "current release branch (25.x)", and the "next
major release (26.1)". Merges would proceed daily from the former to the
latter, but rarely in the other direction (and only by cherry-picking).

We want an active focus on bugs -- toward a stabler .x -- but we don't want to
inhibit integration of feature branches toward the next x.y as they become
ready.

What we're doing now is working, so this isn't a change I propose making right
now. While we work on getting 25.1 out the door, we can keep thinking about
our development and release model.

John

Re: Move to a cadence release model?

[John Yates]

[-- Attachment #1: Type: text/plain, Size: 4649 bytes --]

Mea culpa.  I let my enthusiasm get the better of me is when I wrote

> I have been impressed with open source projects that have made the change.

I have gone back and tried to identify smaller projects which have made
such a change.  To my shame I am unable to find any.  Not say that they do
not exist (I am sure that they do) but I am unable to formulate a google
query that turns any up.  (It does not help that there is a prominent
software company named Cadence Design Systems :-).  That said I would add
to Xue Fugiao's list OpenStack (6 month cadence).

Re Eli's points: In my short time at Mathworks it is clear to me that the
model is very different from what he imagines.  At Mathworks a culture of
cadence-base release starts with an absolute awareness that no one knows
what will be in the next release.  Concretely that translates into _never_
making such commitments to customers.  I am told that what corresponds to
"roadmap presentations" come down to describing areas that the company
views as strategic and hence where it is investing development resources.
I have not attended such a presentation (I am, after all, a lowly
engineer).  Still I have been lead to believe that such presentations are
quite honest about relative levels of investment and priorities attached to
various development activities.  The customers in turn
are knowledgeable enough to understand that - to the extant that
development resources are fungible - under some circumstance resources may
be pulled from lower priority work to accelerate delivery of higher
priority work.  Eli is correct that there is some attempt at the start of a
development cycle to synchronize and coordinate activities that involve
cross-group dependencies.  But what has struck me most strongly, based on
more than 40 years prior experience in commercial software development, is
how readily features that had been on a glide path toward shipment in a
given release can get cut.  The clear message is that shipping a
high-quality product on a predictable schedule is more important than
delaying the release or shipping something buggy.  If a feature was nearly
ready for this release then surely it will be in truly great shape for the
release 6 months later.  So why take a chance?  The other part of taking
care of customers is quick publication of descriptions of any bugs that do
make it into the field, including either publication of work-arounds where
appropriate and when necessary rapid availability of bug fix releases
containing _nothing_ but high priority bug fixes.

I have been thinking about the difference between Emacs' development
culture and that of project with a strong commitment to maintaining a
cadence.  My conclusion is that it comes down to the effort put into
keeping the master branch so healthy that a release branch could be cut at
nearly any moment.  Inevitably this comes with a strong emphasis on code
review, gatekeeping and usually continuous testing.  The Linux kernel is
somewhat atypical in having its gatekeeping function  entrusted entirely to
humans (Linus' lieutenants and their underlings).  (I am unfamiliar with
Firefox.  I did read
https://www.mozilla.org/en-US/about/governance/policies/commit/access-policy/
but failed to get a good sense of how things work there.)  All of the
remaining projects (Chromium, LibreOffice and OpenStack) use gerrit to
enforce a review processes (and to support continuous integration
testing).  The important part of all of these review cultures is that work
_must_ be presented in reviewable units.  No matter how good the work and
no matter how great the reputation of the developer if what is offered for
review is too large or amorphous then it is entirely expected that it will
be reject with a request to break it up into more digestible units.
Keeping the master branch ready for release also means not allowing
incomplete work to get promoted.  Hence none of these projects would
deem acceptable code without supporting documentation or unit tests.
Absence of either would be grounds for a failed review.

Re current pattern of 6 month code freeze: This seems to be a manifestation
of that fact that once a sufficient collection of new features have been
accumulated we recognize in our heart of hearts that our code is not ready
for release.  At a minimum it has not necessarily been built on all of the
platforms we claim to support, various feature have received waivers (hence
are incomplete) and lots of documentation remains to be written.  Moving
more rapidly from cutting a release branch to asserting a release-ready
tarball requires addressing those cultural patterns.

/john

[-- Attachment #2: Type: text/html, Size: 5414 bytes --]

Re: Move to a cadence release model?

[Xue Fuqiao]

On Wed, Nov 11, 2015 at 9:55 AM, John Yates <john@yates-sheets.org> wrote:

Hi John,

> I am unfamiliar with Firefox.  I did read
> https://www.mozilla.org/en-US/about/governance/policies/commit/access-policy/
> but failed to get a good sense of how things work there.

I've contributed to several Mozilla projects, and I know a bit about
Firefox.  I promise that I won't talk too much about this.  If you're
interested, we can discuss on the emacs-tangents list.

Mozilla operates under a module ownership governance system[1].  The
Firefox module owner and peers[2] review the code for Firefox.  The "r+"
flag on Bugzilla means that the code is accepted.  Well, they do require
that the patches should be small, functionally divided, including tests
and with descriptive commit messages.

Most Mozilla projects have two levels of review, known as "review" and
"super-review".  Reviewers look at the actual fix/feature/design and its
maintainability/security/compatibility/localization/coding-style;
super-reviewers look how a patch fits into the broader Mozilla codebase.
Similiar to Chromium/LibreOffice/OpenStack, Mozilla also uses a code
review platforms, such as MozReview[3].

For continuous integration testing, Mozilla uses/develops an extensive
set of tools like Buildbot/TaskCluster and Treeherder.  See
https://developer.mozilla.org/en-US/docs/Continuous_Integration for details.

[1] https://www.mozilla.org/en-US/about/governance/policies/module-ownership/
[2] https://wiki.mozilla.org/Modules/Firefox
[3] https://mozilla-version-control-tools.readthedocs.org/en/latest/mozreview.html

Re: Move to a cadence release model?

[Eli Zaretskii]

> Date: Tue, 10 Nov 2015 20:55:59 -0500
> From: John Yates <john@yates-sheets.org>
> 
> Re Eli's points: In my short time at Mathworks it is clear to me that the model is very different from what he imagines.  At Mathworks a culture of cadence-base release starts with an absolute awareness that no one knows what will be in the next release.

I sincerely doubt that.  Someone in the company's management must
know.  Moreover, someone must actively _control_ that.  You cannot run
a successful business any other way.

> Concretely that translates into _never_ making such commitments to customers.

I never said it should be known to customers.  It should, however, be
known internally.  Otherwise, Mathworks is just another chaotic firm
which we have nothing to learn from (and can actually teach them some ;-).

> I am told that what corresponds to "roadmap presentations" come down to describing areas that the company views as strategic and hence where it is investing development resources.

There you go: we don't have any such roadmap in Emacs.  And I actually
very much doubt if we can practically have one.  At least all past
attempts to start something like that failed spectacularly.  Countless
discussions about adding WYSIWYG word-processor like editing features,
developing an Emacs IDE, etc. -- these all are attempts to set just
the first milestone on that roadmap.  They all eat dust every single
time the issues are raised here.  Doesn't that say something about our
culture and preferences?

> The clear message is that shipping a high-quality product on a predictable schedule is more important than delaying the release or shipping something buggy.

Emacs doesn't "ship buggy", so that's not the issue here.

As for predictable release schedule, whether this is important should
be based on user surveys.  Did someone make such surveys for Emacs?  I
don't think so.  FWIW, it makes little sense to me to install a new
non-bugfix release of a major package such as Emacs without getting
significant new features, but I'm just one user, and most probably not
the typical one.

In any case, if we want to focus on frequent high-quality releases, we
should work on our QA.  That's the point Óscar was making: our QA is
abysmally inadequate.

> rapid availability of bug fix releases containing _nothing_ but high priority bug fixes.

That is relatively much easier, and we already tried that on the
Emacs-24 branch: the last bugfix release was out the door in about 10
days since the first pretest (which we declared an RC).  But this is
only possible on the release branch, and only if we allow absolutely
nothing on that branch except relatively safe bug fixes.

> I have been thinking about the difference between Emacs' development culture and that of project with a strong commitment to maintaining a cadence.  My conclusion is that it comes down to the effort put into keeping the master branch so healthy that a release branch could be cut at nearly any moment.  Inevitably this comes with a strong emphasis on code review, gatekeeping and usually continuous testing.  The Linux kernel is somewhat atypical in having its gatekeeping function  entrusted entirely to humans (Linus' lieutenants and their underlings).  (I am unfamiliar with Firefox.  I did read https://www.mozilla.org/en-US/about/governance/policies/commit/access-policy/ but failed to get a good sense of how things work there.)  All of the remaining projects (Chromium, LibreOffice and OpenStack) use gerrit to enforce a review processes (and to support continuous integration testing).  The important part of all of these review cultures is that work _must_ be presented in reviewable uni

So we essentially agree: switching to any cadence model requires deep
changes in how Emacs development works, which in turn requires a much
larger core development team and a very different organization of that
team.  E.g., a gatekeeper model won't work without a gatekeeper.  We
can try working towards such new models, if we believe we can achieve
them.  But until we get close to that goal, discussing such radically
different models will remain a pipe dream at best.

> Re current pattern of 6 month code freeze: This seems to be a manifestation of that fact that once a sufficient collection of new features have been accumulated we recognize in our heart of hearts that our code is not ready for release.  At a minimum it has not necessarily been built on all of the platforms we claim to support, various feature have received waivers (hence are incomplete) and lots of documentation remains to be written.  Moving more rapidly from cutting a release branch to asserting a release-ready tarball requires addressing those cultural patterns.

There is no 6-month code freeze.  There are long pretest periods for
major version releases.  Emacs 21.1, which included a complete rewrite
of the display engine and many other significant changes, took 11
months since the first pretest till the release.  Emacs 22.1 took 7.5
months, Emacs 23.1 5 months, Emacs 24.1 8.5 months.  A large portion
of that time is spent updating the documentation to match the changes,
especially documenting the new features, and then proof-reading the
manuals.  (I'm guessing at Mathworks you don't have developers writing
and proof-reading the documentation, you have special staff for that.
We don't have such luxury.)  Slashing those months-long pretest
periods means requesting that every user-visible change be accompanied
by the corresponding documentation changes -- are we prepared to do
that?  Do we have enough people in place to review and comment on
those documentation changes if and when they do come?  Don't forget
that some of us cannot write good English documentation and/or don't
command written technical English enough to produce something decent.
And I didn't even start talking about how many of us _want_ to deal
with documentation even if their English is fine.

The other reason for the long pretests is indeed the need to test the
upcoming release on a wide array of different platforms and system
configurations.  We used to have a group of pretesters who represented
the supported platforms, and who were instrumental in that process.
We no longer seem to have such a group, and the platforms we care
about changed anyway.  As result, our coverage of the possible
configurations is mostly random, and we can no longer be sure that a
long enough pretest period will shake out enough bugs.  Note that it
is not enough to just build on a platform, you must actually use Emacs
there to find bugs.  And since a typical user uses only a small
fraction of Emacs features, a single tester per platform/configuration
is not enough for thorough testing.  Some work in that area is needed,
e.g. to identify the features which might be platform-dependent
(example: file notifications), and then compile a list of platforms
that share the same implementation of a given feature.  Then we need a
few testers for each such group, and we need to inform them about each
pretest (there used to be a mailing list for that) and be sure to
collect their reports.

Lots of job opportunities there, and too few volunteers...

Re: Move to a cadence release model?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  >   The important part of all of these review cultures is that work
  > _must_ be presented in reviewable units.

I am not quite sure what that would mean, concretely, for Emacs.

Currently, we install a new feature as a whole.  That can be large;
perhaps not a "reviewable unit".  If so, what would we do instead?
Install small parts of the code one by one?

The problem is, they won't actually do anything by themselves.
And we won't be able to see their interactions with the rest of Emacs features
and usage patterns, until the whole thing is installed.

  > Re current pattern of 6 month code freeze: This seems to be a manifestation
  > of that fact that once a sufficient collection of new features have been
  > accumulated we recognize in our heart of hearts that our code is not ready
  > for release.

And some of the new features or changes won't work in combination with
various other existing features.

There are so many features in Emacs, so many use cases and
interactions, that we don't know how to test that a new feature works
the way users would like in all cases.  It may not even be clear what
behavior users would like in all of them.

It comes down to a question of what "reviewable unit" means.  That we
can study the code of the feature for bugs?  That's what we do now for
the whole new feature.  That we can determine whether it interacts
badly with anything else?  I don't see how we can do that, no small the
parts are.

Have I missed the point somehow?


-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Move to a cadence release model?

[Eli Zaretskii]

> From: John Wiegley <jwiegley@gmail.com>
> Date: Tue, 10 Nov 2015 15:58:20 -0800
> Cc: John Yates <john@yates-sheets.org>, rms@gnu.org,
> 	Emacs-devel <emacs-devel@gnu.org>
> 
> I would indeed like to see several .x releases within a cycle, fairly evenly
> spaced. x.y releases would wait until we have a larger set of features ready.

We more or less do this already, so no significant change there.
Releases from emacs-24 branch were done approximately every 6 months.
The intervals could be made shorter if we don't allow new features on
the release branch.

> Doing this properly may mean dividing how we develop Emacs, though, with
> active development on both the "current release branch (25.x)", and the "next
> major release (26.1)". Merges would proceed daily from the former to the
> latter, but rarely in the other direction (and only by cherry-picking).

Careful here: you are talking about maintaining more than 2 active
branches.  We still have veteran developers who regularly shoot
themselves in the foot with even 2 branches, and others who evidently
cannot safely work on a feature branch without messing up their DAG.
Last, but not least, gitmerge.el is still very much in diapers, and
doesn't support more than 2 branches.

Most places I know about have separate teams working on each branch.
Do we have enough people to do that?

> We want an active focus on bugs -- toward a stabler .x -- but we don't want to
> inhibit integration of feature branches toward the next x.y as they become
> ready.

As I wrote earlier, "active focus on bugs" is a pipe dream as long as
the number of people who care about bugs enough to pick them up,
investigate, and work with the submitters or by themselves to debug
them is some prime number less than 3.  We will never succeed in
moving forward to some more elaborate development model if we don't
have this issue covered much better than we do now.

Re: Move to a cadence release model?

[Stelian Iancu]

On 11/11/15 16:49, Eli Zaretskii wrote:
>
> As I wrote earlier, "active focus on bugs" is a pipe dream as long as
> the number of people who care about bugs enough to pick them up,
> investigate, and work with the submitters or by themselves to debug
> them is some prime number less than 3.  We will never succeed in
> moving forward to some more elaborate development model if we don't
> have this issue covered much better than we do now.

Is the bug handling process documented somehwere?

Re: Move to a cadence release model?

[Eli Zaretskii]

> From: Stelian Iancu <stelian@iancu.ch>
> Date: Thu, 12 Nov 2015 12:27:24 +0100
> 
> Is the bug handling process documented somewhere?

Not sure what you mean by that.  When someone decides they want to
work on a bug, they just do it: reproduce it on their system, use the
debugger, ask the OP questions if needed, discuss their findings on
the mailing list with other developers, etc.  Once the bug is
identified and fixed, the bug report is closed by sending an email to
special address @debbugs.gnu.org; if it is decided not to fix the bug,
the bug is tagged accordingly by sending a similar message.  The last
part is the only thing I could call "bug handling process", and this
_is_ documented in admin/notes/bugtracker.  All the rest is just
investigation and debugging.

Does that answer your question?

Re: Move to a cadence release model?

[Stelian Iancu]

On 12/11/15 17:22, Eli Zaretskii wrote:
>> From: Stelian Iancu <stelian@iancu.ch>
>> Date: Thu, 12 Nov 2015 12:27:24 +0100
>>
>> Is the bug handling process documented somewhere?
>
> Not sure what you mean by that.  When someone decides they want to
> work on a bug, they just do it: reproduce it on their system, use the
> debugger, ask the OP questions if needed, discuss their findings on
> the mailing list with other developers, etc.  Once the bug is
> identified and fixed, the bug report is closed by sending an email to
> special address @debbugs.gnu.org; if it is decided not to fix the bug,
> the bug is tagged accordingly by sending a similar message.  The last
> part is the only thing I could call "bug handling process", and this
> _is_ documented in admin/notes/bugtracker.  All the rest is just
> investigation and debugging.
>
> Does that answer your question?
>
>

It does, to a certain degree. I'm only used to, let's say, more formal 
bug handling, where there is a certain process that is followed by both 
the reporter of the bug as well as the developer that is working on the 
bug. For example, in the places I've worked, there were different states 
that the bug was on, like "new", "confirmed", "reproducible", etc. Is 
there such a thing for Emacs bugs? If yes, how does one go about 
changing it? And where?

But I kind of get the workflow from your answer.

I've had a look at the web interface for the bug system and it's not 
very user friendly. Let's say I'd like to have a look at the existing 
bugs, trying to at least reproduce them and comment in the bug reports. 
How would I practically do that?

I'm following the bugs mailing list on gmane for the past week or so, btw.

Re: Move to a cadence release model?

[Eli Zaretskii]

> From: Stelian Iancu <si@siancu.net>
> Cc: emacs-devel@gnu.org
> Date: Thu, 12 Nov 2015 17:44:24 +0100
> 
> It does, to a certain degree. I'm only used to, let's say, more formal 
> bug handling, where there is a certain process that is followed by both 
> the reporter of the bug as well as the developer that is working on the 
> bug. For example, in the places I've worked, there were different states 
> that the bug was on, like "new", "confirmed", "reproducible", etc. Is 
> there such a thing for Emacs bugs?

See admin/notes/bugtracker.  There are some tags you can tag the bug,
and there is "severity".  There's no formal process to set these,
though.

> Let's say I'd like to have a look at the existing bugs, trying to at
> least reproduce them and comment in the bug reports.  How would I
> practically do that?

The Web page provides filtering that you could use.  Alternatively,
install debbugs.el from ELPA and use that.

> I'm following the bugs mailing list on gmane for the past week or so, btw.

Thanks.

Re: Move to a cadence release model?

[Eli Zaretskii]

> Date: Wed, 11 Nov 2015 07:50:14 +0800
> From: Xue Fuqiao <xfq.free@gmail.com>
> Cc: Emacs-devel <emacs-devel@gnu.org>, John Yates <john@yates-sheets.org>
> 
> Some examples of this model are Linux (a new release every few months,
> although there is a separate set of "stable" branches), Firefox (a new
> release every six weeks), Chromium (roughly the same as Firefox), and
> LibreOffice (six monthly releases).

I think we can only have useful discussions of those other models if
they are not just mentioned, but described in some detail.  Relevant
details IMO include the number of active developers, the number of
gatekeepers and/or people actively involved in the patch review
process, some statistics about the commit rate, etc.  Only armed with
those details can we reason whether any of those models are applicable
to Emacs, and what would be the prerequisites of each model.  E.g.,
people talk about reviewing patches, pull requests, gerrit, etc., but
we don't even _have_ a patch review process per se.

Re: Move to a cadence release model?

[Xue Fuqiao]

On Wed, Nov 11, 2015 at 11:48 PM, Eli Zaretskii <eliz@gnu.org> wrote:

Hi Eli,

>> Date: Wed, 11 Nov 2015 07:50:14 +0800
>> From: Xue Fuqiao <xfq.free@gmail.com>
>> Cc: Emacs-devel <emacs-devel@gnu.org>, John Yates <john@yates-sheets.org>
>>
>> Some examples of this model are Linux (a new release every few months,
>> although there is a separate set of "stable" branches), Firefox (a new
>> release every six weeks), Chromium (roughly the same as Firefox), and
>> LibreOffice (six monthly releases).
>
> I think we can only have useful discussions of those other models if
> they are not just mentioned, but described in some detail.  Relevant
> details IMO include the number of active developers, the number of
> gatekeepers and/or people actively involved in the patch review
> process, some statistics about the commit rate, etc.  Only armed with
> those details can we reason whether any of those models are applicable
> to Emacs, and what would be the prerequisites of each model.  E.g.,
> people talk about reviewing patches, pull requests, gerrit, etc., but
> we don't even _have_ a patch review process per se.

OK.  Here are some relevant details I found.  According to Black Duck
Open Hub[1]:

Linux has
* 705 contributors and 3926 commits in the last 30 days
* 3826 contributors and 68190 commits in the last 12 months
* 1000+ maintainers[2]

Firefox has
* 396 contributors and 4558 commits in the last 30 days
* 1180 contributors and 56950 commits in the last 12 months
* About 30 gatekeepers[3]

Chromium has
* 909 contributors and 11410 commits in the last 30 days
* 2457 contributors and 61454 commits in the last 12 months
* (I have not yet counted the number of reviewers in Chromium.  They are
  listed in the dir/OWNERS files of the source code.)

OpenStack has
* 321 contributors and 2312 commits in the last 30 days
* 1757 contributors and 40212 commits in the last 12 months
* (I don't know the number of people actively involved in the patch
  review process in OpenStack.)

Finally, Emacs has
* 40 contributors and 319 commits in the last 30 days
* 200 contributors and 4244 commits in the last 12 months
* Less than 10 people actively involved in code review[4]

Footnotes:

[1] https://www.openhub.net/
[2] https://www.kernel.org/doc/linux/MAINTAINERS
[3] https://wiki.mozilla.org/Modules/Firefox
[4] My impression: Dmitry Gutov, Glenn Morris, Michael Albinus, Stefan
    Monnier, Artur Malabarba, and you.

Re: Move to a cadence release model?

[Xue Fuqiao]

On Thu, Nov 12, 2015 at 3:23 PM, Xue Fuqiao <xfq.free@gmail.com> wrote:
> On Wed, Nov 11, 2015 at 11:48 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>
> Hi Eli,
>
>>> Date: Wed, 11 Nov 2015 07:50:14 +0800
>>> From: Xue Fuqiao <xfq.free@gmail.com>
>>> Cc: Emacs-devel <emacs-devel@gnu.org>, John Yates <john@yates-sheets.org>
>>>
>>> Some examples of this model are Linux (a new release every few months,
>>> although there is a separate set of "stable" branches), Firefox (a new
>>> release every six weeks), Chromium (roughly the same as Firefox), and
>>> LibreOffice (six monthly releases).
>>
>> I think we can only have useful discussions of those other models if
>> they are not just mentioned, but described in some detail.  Relevant
>> details IMO include the number of active developers, the number of
>> gatekeepers and/or people actively involved in the patch review
>> process, some statistics about the commit rate, etc.  Only armed with
>> those details can we reason whether any of those models are applicable
>> to Emacs, and what would be the prerequisites of each model.  E.g.,
>> people talk about reviewing patches, pull requests, gerrit, etc., but
>> we don't even _have_ a patch review process per se.
>
> OK.  Here are some relevant details I found.  According to Black Duck
> Open Hub[1]:
>
> Linux has
> * 705 contributors and 3926 commits in the last 30 days
> * 3826 contributors and 68190 commits in the last 12 months
> * 1000+ maintainers[2]
>
> Firefox has
> * 396 contributors and 4558 commits in the last 30 days
> * 1180 contributors and 56950 commits in the last 12 months
> * About 30 gatekeepers[3]
>
> Chromium has
> * 909 contributors and 11410 commits in the last 30 days
> * 2457 contributors and 61454 commits in the last 12 months
> * (I have not yet counted the number of reviewers in Chromium.  They are
>   listed in the dir/OWNERS files of the source code.)
>
> OpenStack has
> * 321 contributors and 2312 commits in the last 30 days
> * 1757 contributors and 40212 commits in the last 12 months
> * (I don't know the number of people actively involved in the patch
>   review process in OpenStack.)
>
> Finally, Emacs has
> * 40 contributors and 319 commits in the last 30 days
> * 200 contributors and 4244 commits in the last 12 months
> * Less than 10 people actively involved in code review[4]

Sorry, I inadvertently omitted LibreOffice.

LibreOffice has
* 81 contributors and 1871 commits in the last 30 days
* 277 contributors and 19617 commits in the last 12 months
* (I don't know the number of people actively involved in the patch
  review process in LibreOffice.)

Re: Move to a cadence release model?

[Eli Zaretskii]

> Date: Thu, 12 Nov 2015 15:23:30 +0800
> From: Xue Fuqiao <xfq.free@gmail.com>
> Cc: rms@gnu.org, Emacs-devel <emacs-devel@gnu.org>, john@yates-sheets.org
> 
> OK.  Here are some relevant details I found.  According to Black Duck
> Open Hub[1]:

Thanks.  (It's good to know about that site.)  Clearly, we have some
serious catching up to do before we can think about adopting those
advanced models.

Re: Move to a cadence release model?

[Stefan Monnier]

> With a new master at the helm and various changes being contemplated I
> would like to see some discussion of moving to a cadence release model.

FWIW, I've tried to follow a "one release per year" cadence ;-)


        Stefan