Emacs 23.3 released

Emacs 23.3 released

[Chong Yidong]

GNU Emacs 23.3 has been released, and is now available at
ftp.gnu.org/gnu/emacs/ and the GNU FTP mirrors (see
http://www.gnu.org/order/ftp.html).

The SHA1 checksums are:

 de2c924655f0f4e012cc119840cd341a1d40e32c  emacs-23.3.tar.bz2
 53bdfe584f98c13265caa3eea3cfdf79ca0f7584  emacs-23.3.tar.gz

Please send bug reports to <bug-gnu-emacs@gnu.org>.  You can use the
command M-x report-emacs-bug to do this.

Emacs 23.3 is primarily a bugfix release.  Also included are a number of
enhancements to VC and Rmail, and a new indentation library, smie.el.
For details, see the file etc/NEWS.

Re: Emacs 23.3 released

[Bastien]

Chong Yidong <cyd@stupidchicken.com> writes:

> GNU Emacs 23.3 has been released, and is now available at
> ftp.gnu.org/gnu/emacs/ and the GNU FTP mirrors (see
> http://www.gnu.org/order/ftp.html).

Thanks to all who worked hard for this release!

And congrats to the maintainers for keeping with such a pace.

:)

-- 
 Bastien

Re: Emacs 23.3 released

[Stefan Monnier]

A big thanks to everybody for the work they put into this release.

I hope we will be able to jump right to 24.1 without an intermediate
23.4, but if you find bugs that are not specific to trunk, please still
commit them to emacs-23.

Talking about 24.1, I now consider the lexbind-new branch as "fully
functional": there are still a few changes I want to do there before
merging, but they should only impact performance and code quality, so
I encourage people to try it out and report any regression so they can
be addressed before we merge it into trunk.


        Stefan


>>>>> "Chong" == Chong Yidong <cyd@stupidchicken.com> writes:

> GNU Emacs 23.3 has been released, and is now available at
> ftp.gnu.org/gnu/emacs/ and the GNU FTP mirrors (see
> http://www.gnu.org/order/ftp.html).

> The SHA1 checksums are:

>  de2c924655f0f4e012cc119840cd341a1d40e32c  emacs-23.3.tar.bz2
>  53bdfe584f98c13265caa3eea3cfdf79ca0f7584  emacs-23.3.tar.gz

> Please send bug reports to <bug-gnu-emacs@gnu.org>.  You can use the
> command M-x report-emacs-bug to do this.

> Emacs 23.3 is primarily a bugfix release.  Also included are a number of
> enhancements to VC and Rmail, and a new indentation library, smie.el.
> For details, see the file etc/NEWS.

Re: Emacs 23.3 released

[David Kastrup]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> A big thanks to everybody for the work they put into this release.
>
> I hope we will be able to jump right to 24.1 without an intermediate
> 23.4, but if you find bugs that are not specific to trunk, please still
> commit them to emacs-23.
>
> Talking about 24.1, I now consider the lexbind-new branch as "fully
> functional": there are still a few changes I want to do there before
> merging, but they should only impact performance and code quality, so
> I encourage people to try it out and report any regression so they can
> be addressed before we merge it into trunk.

I think it would send a bad message if 24.1 would be released without
the source code (*.wy) for the various Semantic parsers (*-wy.el IIRC),
as well as the parser generators themselves and the corresponding
documentation.

Including non-serviceable components without source code just does not
seem to jibe with the spirit of the GPL, whether or not we are the
copyright holders and consequently allowed to behave like downstream
never could.

-- 
David Kastrup

Re: Emacs 23.3 released

[Chong Yidong]

David Kastrup <dak@gnu.org> writes:

> I think it would send a bad message if 24.1 would be released without
> the source code (*.wy) for the various Semantic parsers (*-wy.el IIRC),
> as well as the parser generators themselves and the corresponding
> documentation.

It's on the todo list.  If you have the time to help, it would be
welcome.

Re: Emacs 23.3 released

[David Kastrup]

Chong Yidong <cyd@stupidchicken.com> writes:

> David Kastrup <dak@gnu.org> writes:
>
>> I think it would send a bad message if 24.1 would be released without
>> the source code (*.wy) for the various Semantic parsers (*-wy.el IIRC),
>> as well as the parser generators themselves and the corresponding
>> documentation.
>
> It's on the todo list.  If you have the time to help, it would be
> welcome.

As far as I remember, the idea/rationale was to merge a more complete
version once Cedet had moved to Bazaar.

I don't see that I can do anything to speed up _that_ process (which
likely has not even started).  It does not seem like a one-time merge of
Cedet in its current state and version control system would make people
happy or things easier for them.

-- 
David Kastrup

Re: Emacs 23.3 released

[Lennart Borgman]

On Thu, Mar 10, 2011 at 6:54 PM, David Kastrup <dak@gnu.org> wrote:
> Chong Yidong <cyd@stupidchicken.com> writes:
>
>> David Kastrup <dak@gnu.org> writes:
>>
>>> I think it would send a bad message if 24.1 would be released without
>>> the source code (*.wy) for the various Semantic parsers (*-wy.el IIRC),
>>> as well as the parser generators themselves and the corresponding
>>> documentation.
>>
>> It's on the todo list.  If you have the time to help, it would be
>> welcome.
>
> As far as I remember, the idea/rationale was to merge a more complete
> version once Cedet had moved to Bazaar.

Has not Cedet switched to Bazaar already?

> I don't see that I can do anything to speed up _that_ process (which
> likely has not even started).  It does not seem like a one-time merge of
> Cedet in its current state and version control system would make people
> happy or things easier for them.
>
> --
> David Kastrup
>
>
>

State of the CEDET merge (was: Emacs 23.3 released)

[David Engster]

Lennart Borgman writes:
> On Thu, Mar 10, 2011 at 6:54 PM, David Kastrup <dak@gnu.org> wrote:
>> Chong Yidong <cyd@stupidchicken.com> writes:
>>
>>> David Kastrup <dak@gnu.org> writes:
>>>
>>>> I think it would send a bad message if 24.1 would be released without
>>>> the source code (*.wy) for the various Semantic parsers (*-wy.el IIRC),
>>>> as well as the parser generators themselves and the corresponding
>>>> documentation.
>>>
>>> It's on the todo list.  If you have the time to help, it would be
>>> welcome.
>>
>> As far as I remember, the idea/rationale was to merge a more complete
>> version once Cedet had moved to Bazaar.
>
> Has not Cedet switched to Bazaar already?

Yes, months ago. Lluís has invested much time in that and he has also
begun merging with the Emacs version. He's working on that a separate
branch 'file-rename'. I'm not sure how much work is left; I added him to
the CC of this thread so he might add his opinion.

-David

Re: State of the CEDET merge

[Lluís]

David Engster writes:

> Lennart Borgman writes:
>> Has not Cedet switched to Bazaar already?

> Yes, months ago. Lluís has invested much time in that and he has also
> begun merging with the Emacs version. He's working on that a separate
> branch 'file-rename'. I'm not sure how much work is left.

Yup, such a branch exists and is partially ported. But lots of changes
have been inserted in both sides (Emacs and Cedet), which forced me to
do a manual merge reviewing code line-by-line, instead of just renaming
files.

My current approach is (it's the best I could find that provided some
semi-automatic merging):

   - look for the date (D) where a file was added into the emacs
     repository
   - build a patch (P) of the changes in Emacs since date D to present
     day
   - apply patch P into the Cedet repository (file-rename branch)

So the currently done work is:

   - rename most (all?) the grammar files in cedet
   - fix these to build with their new naming scheme
   - finished merging eieio
   - finished merging speedbar
   - added a list of which files are not present on both repositories
   - started merging common cedet code

This is an extremely time consuming process (specially because I'm not
much knowledgeable of the cedet codebase, nor an elisp ninja), and I've
not been able to devote much to it since long ago.


Lluis

--
 "And it's much the same thing with knowledge, for whenever you learn
 something new, the whole world becomes that much richer."
 -- The Princess of Pure Reason, as told by Norton Juster in The Phantom
 Tollbooth

Re: State of the CEDET merge

[David Kastrup]

Lluís <xscript@gmx.net> writes:

> Yup, such a branch exists and is partially ported. But lots of changes
> have been inserted in both sides (Emacs and Cedet), which forced me to
> do a manual merge reviewing code line-by-line, instead of just renaming
> files.
>
> My current approach is (it's the best I could find that provided some
> semi-automatic merging):
>
>    - look for the date (D) where a file was added into the emacs
>      repository
>    - build a patch (P) of the changes in Emacs since date D to present
>      day
>    - apply patch P into the Cedet repository (file-rename branch)
>
> So the currently done work is:
>
>    - rename most (all?) the grammar files in cedet
>    - fix these to build with their new naming scheme
>    - finished merging eieio
>    - finished merging speedbar
>    - added a list of which files are not present on both repositories
>    - started merging common cedet code
>
> This is an extremely time consuming process (specially because I'm not
> much knowledgeable of the cedet codebase, nor an elisp ninja), and I've
> not been able to devote much to it since long ago.

Sounds like having to track two separately moving targets.  Any idea how
to make the respective developers aware of the problem and move in a
more synchronized fashion, so as to decrease the speed with which the
task you have focused on grows?

-- 
David Kastrup

Re: State of the CEDET merge

[Lluís]

David Kastrup writes:
> Sounds like having to track two separately moving targets.

In fact, I'm only tracking changes introduced in Emacs. But not those
introduced after my last merge for a specific file (these will have to
be manually re-checked).

Changes in the cedet trunk should be automatically merged when I merge
the branch.


> Any idea how to make the respective developers aware of the problem
> and move in a more synchronized fashion, so as to decrease the speed
> with which the task you have focused on grows?

There's no easy solution.

On one hand, files in Emacs where introduced with modification wrt the
cedet CVS, so some of them are hard to track.

On the other hand, people won't be able to contribute all the fixes into
cedet instead of emacs and expect emacs tu pull from cedet; not until I
finish the file-rename branch.

All this, added with my lack of elisp skill, knowledge on cedet
internals and knowledge on what has been changed and why, make the task
a tough one.


Lluis

--
 "And it's much the same thing with knowledge, for whenever you learn
 something new, the whole world becomes that much richer."
 -- The Princess of Pure Reason, as told by Norton Juster in The Phantom
 Tollbooth

Re: State of the CEDET merge

[David Kastrup]

Lluís <xscript@gmx.net> writes:

> David Kastrup writes:
>> Sounds like having to track two separately moving targets.
>
> In fact, I'm only tracking changes introduced in Emacs. But not those
> introduced after my last merge for a specific file (these will have to
> be manually re-checked).
>
> Changes in the cedet trunk should be automatically merged when I merge
> the branch.
>
>
>> Any idea how to make the respective developers aware of the problem
>> and move in a more synchronized fashion, so as to decrease the speed
>> with which the task you have focused on grows?
>
> There's no easy solution.
>
> On one hand, files in Emacs where introduced with modification wrt the
> cedet CVS, so some of them are hard to track.
>
> On the other hand, people won't be able to contribute all the fixes into
> cedet instead of emacs and expect emacs tu pull from cedet; not until I
> finish the file-rename branch.
>
> All this, added with my lack of elisp skill, knowledge on cedet
> internals and knowledge on what has been changed and why, make the task
> a tough one.

Let's assume that you get the task completed in the manner you envision
and you are working on right now, and the merge and synch happens as
planned.

Will that leave Cedet and Emacs in a state where future synchronizations
of Cedet to Emacs will be possible in a semi-automatic manner, like Gnus
is synchronized frequently right now?

Or will it mean that every future synchronization will require just as
much effort as your current work?

In short: if you manage to catch up with your target, will it be
reasonably easy to keep it from running off again?

-- 
David Kastrup

Re: State of the CEDET merge

[Eric M. Ludlam]

On 03/12/2011 04:42 AM, David Kastrup wrote:
> Lluís<xscript@gmx.net>  writes:
>
>> David Kastrup writes:
>>> Sounds like having to track two separately moving targets.
>>
>> In fact, I'm only tracking changes introduced in Emacs. But not those
>> introduced after my last merge for a specific file (these will have to
>> be manually re-checked).
>>
>> Changes in the cedet trunk should be automatically merged when I merge
>> the branch.
>>
>>
>>> Any idea how to make the respective developers aware of the problem
>>> and move in a more synchronized fashion, so as to decrease the speed
>>> with which the task you have focused on grows?
>>
>> There's no easy solution.
>>
>> On one hand, files in Emacs where introduced with modification wrt the
>> cedet CVS, so some of them are hard to track.
>>
>> On the other hand, people won't be able to contribute all the fixes into
>> cedet instead of emacs and expect emacs tu pull from cedet; not until I
>> finish the file-rename branch.
>>
>> All this, added with my lack of elisp skill, knowledge on cedet
>> internals and knowledge on what has been changed and why, make the task
>> a tough one.
>
> Let's assume that you get the task completed in the manner you envision
> and you are working on right now, and the merge and synch happens as
> planned.
>
> Will that leave Cedet and Emacs in a state where future synchronizations
> of Cedet to Emacs will be possible in a semi-automatic manner, like Gnus
> is synchronized frequently right now?
>
> Or will it mean that every future synchronization will require just as
> much effort as your current work?
>
> In short: if you manage to catch up with your target, will it be
> reasonably easy to keep it from running off again?
>

I'm not that familiar with bzr, but the idea is that once this 
conversion is done, merges could be automated on some way, except for 
conflicts.  My dream is some sort of cron job that merges from Emacs 
back into a CEDET integration branch, and also from the CEDET trunk into 
the integration branch, and warns when a merge fails. When Emacs wants 
to sync back, the merge would be quite simple from the integration 
branch at any time.  Periodically, someone could merge the Emacs 
integration branch into CEDET trunk.

The challenge is just finding someone with the right skills and time to 
help out.

Eric

Re: State of the CEDET merge

[Stefan Monnier]

> The challenge is just finding someone with the right skills and time to
> help out.

The Gnus guys do it, so I strongly recommend you ask them how they
do it.  It would be good to document it somewhere (a savannah page
about how to keep different repositories in sync).


        Stefan

Re: State of the CEDET merge

[Ted Zlatanov]

On Sat, 12 Mar 2011 15:49:40 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> The challenge is just finding someone with the right skills and time to
>> help out.

SM> The Gnus guys do it, so I strongly recommend you ask them how they
SM> do it.  It would be good to document it somewhere (a savannah page
SM> about how to keep different repositories in sync).

Katsumi Yamaoka, who kindly handles all this tedious work, posted some
explanation and links here:

http://thread.gmane.org/gmane.emacs.gnus.general/69770/focus=69777

A more general approach would be nice but seems hard to achieve.

Ted

Re: State of the CEDET merge

[Lluís]

Ted Zlatanov writes:

> On Sat, 12 Mar 2011 15:49:40 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>>> The challenge is just finding someone with the right skills and time to
>>> help out.

SM> The Gnus guys do it, so I strongly recommend you ask them how they
SM> do it.  It would be good to document it somewhere (a savannah page
SM> about how to keep different repositories in sync).

> Katsumi Yamaoka, who kindly handles all this tedious work, posted some
> explanation and links here:

> http://thread.gmane.org/gmane.emacs.gnus.general/69770/focus=69777

> A more general approach would be nice but seems hard to achieve.

Thanks, I'll have a look at it, but I think the discussion we had here
settled down to merging only in the cedet to emacs direction (using a
branch in cedet that were only meant to merge into
emacs).

Unfortunately, this would force changes from the emacs community to be
applied into the cedet repository.


Lluis

--
 "And it's much the same thing with knowledge, for whenever you learn
 something new, the whole world becomes that much richer."
 -- The Princess of Pure Reason, as told by Norton Juster in The Phantom
 Tollbooth

Re: State of the CEDET merge

[Lennart Borgman]

On Mon, Mar 14, 2011 at 8:55 PM, Lluís <xscript@gmx.net> wrote:
>
> Thanks, I'll have a look at it, but I think the discussion we had here
> settled down to merging only in the cedet to emacs direction (using a
> branch in cedet that were only meant to merge into
> emacs).
>
> Unfortunately, this would force changes from the emacs community to be
> applied into the cedet repository.

Is that decision (and an explanation) written down somewhere? (And
does bzr contain any tool to protect it?)

Re: State of the CEDET merge

[Lluís]

Lennart Borgman writes:

> On Mon, Mar 14, 2011 at 8:55 PM, Lluís <xscript@gmx.net> wrote:
>>
>> Thanks, I'll have a look at it, but I think the discussion we had here
>> settled down to merging only in the cedet to emacs direction (using a
>> branch in cedet that were only meant to merge into
>> emacs).
>>
>> Unfortunately, this would force changes from the emacs community to be
>> applied into the cedet repository.

> Is that decision (and an explanation) written down somewhere? (And
> does bzr contain any tool to protect it?)

Sorry, I cannot find it right now. But I do remember someone (Stefan?)
proposing to use a specific branch for single-way cedet-to-emacs merges,
which seemed the easiest solution that could be automated.

Even better, I could probably write a simple script to maintain the
original history logs in the cases of file content changes (file
additions, removals and renamings cannot be handled so easily using only
diffs), just by importing the changes one by one and extracting their
log messages at the same time.

But supporting two-way merges is much more complex.

I also asked the bazaar people about current tools for inter-project
two-way sub-directory merges, but AFAIR, this needed to wait for the
"nested trees" [1] feature to be fully implemented.

[1] https://blueprints.launchpad.net/bzr/+spec/nested-tree-support

If this were supported, emacs would just have a local CEDET branch
nested inside an emacs branch, so that all CEDET branches can be easily
merged back and forth.


Lluis

--
 "And it's much the same thing with knowledge, for whenever you learn
 something new, the whole world becomes that much richer."
 -- The Princess of Pure Reason, as told by Norton Juster in The Phantom
 Tollbooth

Re: State of the CEDET merge

[David Engster]

Lluís writes:
> In fact, I'm only tracking changes introduced in Emacs. But not those
> introduced after my last merge for a specific file (these will have to
> be manually re-checked).
>
> Changes in the cedet trunk should be automatically merged when I merge
> the branch.

I must admit I don't fully understand the 'file-rename' branch,
especially at what point you would call this branch "finished". Could
you (very roughly) outline how you plan to merge the different versions
in the end? Since both versions differ substantially, I guess you would
need some kind of diff-file which contains the (wanted) differences
between the two versions, like the Gnus people use?

> All this, added with my lack of elisp skill, knowledge on cedet
> internals and knowledge on what has been changed and why, make the task
> a tough one.

It would help if you could name some concrete tasks which have to be
done next. I've seen the 'open_issues' file, but I don't really
understand those items listed there. Please don't take these questions
the wrong way; I very much appreciate your work you've already done on
the merge and the bzr transition.

-David

Re: State of the CEDET merge

[Lluís]

David Engster writes:

> Lluís writes:
>> In fact, I'm only tracking changes introduced in Emacs. But not those
>> introduced after my last merge for a specific file (these will have to
>> be manually re-checked).
>>
>> Changes in the cedet trunk should be automatically merged when I merge
>> the branch.

> I must admit I don't fully understand the 'file-rename' branch,
> especially at what point you would call this branch "finished". Could
> you (very roughly) outline how you plan to merge the different versions
> in the end?

The initial process (the one is being used right now) was described on
previous mails (see script crude-update in the file-rename branch).

Once that's finished, according to previous discussions in this list:

* freeze cedet-related changes in the emacs repository
* see if new changes from emacs must be transplanted into cedet (these
  should be just a few)
* merge file-rename into cedet's trunk
* create a branch (to-emacs) in cedet for changes meant to go into cedet
* any change in cedet that is ready for upstream emacs should be merged
  into the 'to-emacs' branch
* _somehow_ move changes from 'to-emacs' into emacs proper (emacs can
  just have a file indicating the last revision in 'to-emacs' that was
  merged into it, and then applying a patch with the differences since
  that revision)

This is all supposing that bazaar has no support for
externals/sub-modules (as is the case right now AFAIK).


[...]
> It would help if you could name some concrete tasks which have to be
> done next. I've seen the 'open_issues' file, but I don't really
> understand those items listed there.

There are two files to look into:

* not_in_emacs
  Files that are present in the cedet repository but not in emacs. Some
  of these must be considered as "contrib" files (and thus stay out of
  emacs), while some of these may need to be added into emacs proper.

* open_issues
  Things that must be revised before considering the file-rename done

  Some of these refer to functions that were introduced in cedet to
  support other emacs/xemacs versions.

  In the case of emacs, for example, they should probably be dropped, as
  emacs will already ship on each version a fully-compatible cedet from
  now on. If you want to use a newer cedet version, then you should
  probably use a new emacs version.


> Please don't take these questions the wrong way; I very much
> appreciate your work you've already done on the merge and the bzr
> transition.

If you're thinking that I might get angry about your comments, then I'm
not taking these the wrong way :)


Lluis

--
 "And it's much the same thing with knowledge, for whenever you learn
 something new, the whole world becomes that much richer."
 -- The Princess of Pure Reason, as told by Norton Juster in The Phantom
 Tollbooth

Re: State of the CEDET merge

[Stefan Monnier]

> * freeze cedet-related changes in the emacs repository

FWIW, strictly speaking, this is not an option.
E.g. I had to make changes to CEDET in the lexbind branch for it to
compile without errors.


        Stefan

Re: State of the CEDET merge

[Lluís]

Stefan Monnier writes:

>> * freeze cedet-related changes in the emacs repository
> FWIW, strictly speaking, this is not an option.
> E.g. I had to make changes to CEDET in the lexbind branch for it to
> compile without errors.

Ok. What I meant is that changes should not be introduced *directly*
into the emacs repositories.

Instead, the emacs community should have a branch of the cedet
repository for each emacs branch.

Then, emacs devs commit into these emacs-controlled cedet branches,
which can be merged back and forth with any other cedet branch.

In order to make changes in the emacs-controlled cedet branches visible
to emacs releases, the changes must be imported somehow into the emacs
repository.

How? I have to read how the gnus people do it, because thinking about it
for a while raised a lot of non-trivial corner cases.

My question then is whether emacs is predisposed to ship a repository
that is not complete, which to me seems like the most robust and easy
way. What I mean is that instead of emacs shipping the cedet files in
its repo tree, it can contain symlinks all pointing to a directory
containing a cedet checkout (from one of the emacs-controlled branches).

Like this:

  emacs
  |- cedet-repo
  |- lisp/cedet -> ../cedet-repo/lisp/cedet
  ...

Now the question is how do you check out a copy of emacs together with
the corresponding cedet branch.

1) You can use this plugin:

      https://launchpad.net/bzr-externals

   The con is that it's not shipped in vanilla bazaar, but I don't know
   how much is to ask users to have it installed prior to checking out
   an emacs branch.

2) Let the build scripts / Makefiles perform the checkout of the cedet
   branch for you.

   This would work without any external tools, and in fact can be put
   together with the use of the bzr-externals plugin, as a backup plan.

Of course, tarballs should be shipped with the contents of both, so that
they are complete.

Thoughts?


Lluis

--
 "And it's much the same thing with knowledge, for whenever you learn
 something new, the whole world becomes that much richer."
 -- The Princess of Pure Reason, as told by Norton Juster in The Phantom
 Tollbooth

Re: State of the CEDET merge

[Stefan Monnier]

> How? I have to read how the gnus people do it, because thinking about it
> for a while raised a lot of non-trivial corner cases.

I believe once this is understood, the rest will fall into place.


        Stefan

Re: State of the CEDET merge

[David Kastrup]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> How? I have to read how the gnus people do it, because thinking about
>> it for a while raised a lot of non-trivial corner cases.
>
> I believe once this is understood, the rest will fall into place.

With all due respect, that awfully sounds like a particle physician on
the hunt for funds.

-- 
David Kastrup

Re: State of the CEDET merge

[Lluís]

David Kastrup writes:
> With all due respect, that awfully sounds like a particle physician on
> the hunt for funds.

So, do you think that the diff boson will be finally observed?


-- 
 "And it's much the same thing with knowledge, for whenever you learn
 something new, the whole world becomes that much richer."
 -- The Princess of Pure Reason, as told by Norton Juster in The Phantom
 Tollbooth

Re: State of the CEDET merge

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Wed, 16 Mar 2011 11:24:14 -0400
> Cc: emacs-devel@gnu.org
> 
> > How? I have to read how the gnus people do it, because thinking about it
> > for a while raised a lot of non-trivial corner cases.
> 
> I believe once this is understood, the rest will fall into place.
> 
> 
>         Stefan
> 
> 

Re: State of the CEDET merge

[Eli Zaretskii]

> Date: Fri, 18 Mar 2011 13:43:14 -0400
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: Lluís <xscript@gmx.net>, emacs-devel@gnu.org
> 
> > From: Stefan Monnier <monnier@iro.umontreal.ca>
> > Date: Wed, 16 Mar 2011 11:24:14 -0400
> > Cc: emacs-devel@gnu.org
> > 
> > > How? I have to read how the gnus people do it, because thinking about it
> > > for a while raised a lot of non-trivial corner cases.
> > 
> > I believe once this is understood, the rest will fall into place.
> > 
> > 
> >         Stefan
> > 
> > 

Sorry, please ignore this.

Re: State of the CEDET merge

[David Kastrup]

David Engster <deng@randomsample.de> writes:

> Lluís writes:
>> In fact, I'm only tracking changes introduced in Emacs. But not those
>> introduced after my last merge for a specific file (these will have to
>> be manually re-checked).
>>
>> Changes in the cedet trunk should be automatically merged when I merge
>> the branch.
>
> I must admit I don't fully understand the 'file-rename' branch,
> especially at what point you would call this branch "finished". Could
> you (very roughly) outline how you plan to merge the different versions
> in the end? Since both versions differ substantially, I guess you would
> need some kind of diff-file which contains the (wanted) differences
> between the two versions, like the Gnus people use?
>
>> All this, added with my lack of elisp skill, knowledge on cedet
>> internals and knowledge on what has been changed and why, make the task
>> a tough one.
>
> It would help if you could name some concrete tasks which have to be
> done next. I've seen the 'open_issues' file, but I don't really
> understand those items listed there. Please don't take these questions
> the wrong way; I very much appreciate your work you've already done on
> the merge and the bzr transition.

So, a number of months have passed, and feature freezes are around the
corner or done.  Does it mean that we are finally planning to ship
Emacs-24 with unmaintainable binary content without source code, the
preferred form of modification?  Because that is what the compiled CEDET
parsers are.  The grammars are notably absent, and so are the parser
generators (bovinator, wisent and something else) and their
documentation.

-- 
David Kastrup

Re: State of the CEDET merge

[David Engster]

David Kastrup writes:
> David Engster <deng@randomsample.de> writes:
>> Lluís writes:
>>> In fact, I'm only tracking changes introduced in Emacs. But not those
>>> introduced after my last merge for a specific file (these will have to
>>> be manually re-checked).
>>>
>>> Changes in the cedet trunk should be automatically merged when I merge
>>> the branch.
>>
>> I must admit I don't fully understand the 'file-rename' branch,
>> especially at what point you would call this branch "finished". Could
>> you (very roughly) outline how you plan to merge the different versions
>> in the end? Since both versions differ substantially, I guess you would
>> need some kind of diff-file which contains the (wanted) differences
>> between the two versions, like the Gnus people use?
>>
>>> All this, added with my lack of elisp skill, knowledge on cedet
>>> internals and knowledge on what has been changed and why, make the task
>>> a tough one.
>>
>> It would help if you could name some concrete tasks which have to be
>> done next. I've seen the 'open_issues' file, but I don't really
>> understand those items listed there. Please don't take these questions
>> the wrong way; I very much appreciate your work you've already done on
>> the merge and the bzr transition.
>
> So, a number of months have passed, and feature freezes are around the
> corner or done.  Does it mean that we are finally planning to ship
> Emacs-24 with unmaintainable binary content without source code, the
> preferred form of modification?  Because that is what the compiled CEDET
> parsers are.  The grammars are notably absent, and so are the parser
> generators (bovinator, wisent and something else) and their
> documentation.

We are working on it in the mentioned CEDET file-rename branch. We
didn't make the Emacs24 freeze, but thanks for reminding us.

-David

Re: State of the CEDET merge

[David Kastrup]

David Engster <deng@randomsample.de> writes:

> David Kastrup writes:
>
>> So, a number of months have passed, and feature freezes are around
>> the corner or done.  Does it mean that we are finally planning to
>> ship Emacs-24 with unmaintainable binary content without source code,
>> the preferred form of modification?  Because that is what the
>> compiled CEDET parsers are.  The grammars are notably absent, and so
>> are the parser generators (bovinator, wisent and something else) and
>> their documentation.
>
> We are working on it in the mentioned CEDET file-rename branch. We
> didn't make the Emacs24 freeze, but thanks for reminding us.

I have been postponing working on a proper Lilypond mode for several
years now, since it would not make sense to do so without evaluating
CEDET, CEDET-based sources would be useless to most people, and
distributing the compiled mode alone would be problematic for
maintenance and too far from the spirit of the GPL according my personal
taste to make me shift my priorities enough to commit to work which
requires a non-standard setup and an uncertain roadmap.

I suspect that this situation is not unique to me and Lilypond, and that
takeup of CEDET would grow considerably once it became part of the main
stream.

-- 
David Kastrup

Re: State of the CEDET merge

[Chong Yidong]

David Engster <deng@randomsample.de> writes:

> We are working on it in the mentioned CEDET file-rename branch. We
> didn't make the Emacs24 freeze, but thanks for reminding us.

I too regret that this could not be done for the 24.1 freeze, but we
should definitely get plan to bring Emacs CEDET back in synch with
upstream for Emacs 24.2.  (Since CEDET was originally merged in 23.2,
there is at least some symmetry in this).

Re: State of the CEDET merge

[David Kastrup]

Chong Yidong <cyd@stupidchicken.com> writes:

> David Engster <deng@randomsample.de> writes:
>
>> We are working on it in the mentioned CEDET file-rename branch. We
>> didn't make the Emacs24 freeze, but thanks for reminding us.
>
> I too regret that this could not be done for the 24.1 freeze, but we
> should definitely get plan to bring Emacs CEDET back in synch with
> upstream for Emacs 24.2.  (Since CEDET was originally merged in 23.2,
> there is at least some symmetry in this).

The source for the CEDET parser generators and grammars has not ever
been merged.  "Getting back in synch" sounds like getting the newest
generated parsers into Emacs distribution, something which I have no
particular opinion about.  But whatever version gets distributed, I find
it worrying that we don't distribute the corresponding sources and code
generators along with the generated code, so that a user may neither fix
nor extend the existing functionality.

Maintenance is impossible since you won't be able to get the parser
sources corresponding to the particular version of Emacs you are using.

We are, in essence, distributing binary blobs with Emacs, and I find it
surprising that this does not seem to disturb anybody much except
myself.

-- 
David Kastrup

Re: State of the CEDET merge

[David Engster]

David Kastrup writes:
> Chong Yidong <cyd@stupidchicken.com> writes:
>
>> David Engster <deng@randomsample.de> writes:
>>
>>> We are working on it in the mentioned CEDET file-rename branch. We
>>> didn't make the Emacs24 freeze, but thanks for reminding us.
>>
>> I too regret that this could not be done for the 24.1 freeze, but we
>> should definitely get plan to bring Emacs CEDET back in synch with
>> upstream for Emacs 24.2.  (Since CEDET was originally merged in 23.2,
>> there is at least some symmetry in this).
>
> The source for the CEDET parser generators and grammars has not ever
> been merged.  "Getting back in synch" sounds like getting the newest
> generated parsers into Emacs distribution, something which I have no
> particular opinion about. 
>
> But whatever version gets distributed, I find it worrying that we
> don't distribute the corresponding sources and code generators along
> with the generated code, so that a user may neither fix nor extend the
> existing functionality.
>
> Maintenance is impossible since you won't be able to get the parser
> sources corresponding to the particular version of Emacs you are using.
>
> We are, in essence, distributing binary blobs with Emacs, and I find it
> surprising that this does not seem to disturb anybody much except
> myself.

The parser generators will be merged. Please take a look at the
mentioned file-rename branch[1] before speculating further on what the
merge might not include.

-David

[1] http://cedet.bzr.sourceforge.net/bzr/cedet/code/file-rename/files

Re: State of the CEDET merge

[David Kastrup]

David Engster <deng@randomsample.de> writes:

> David Kastrup writes:
>> Chong Yidong <cyd@stupidchicken.com> writes:
>>
>>> David Engster <deng@randomsample.de> writes:
>>>
>>>> We are working on it in the mentioned CEDET file-rename branch. We
>>>> didn't make the Emacs24 freeze, but thanks for reminding us.
>>>
>>> I too regret that this could not be done for the 24.1 freeze, but we
>>> should definitely get plan to bring Emacs CEDET back in synch with
>>> upstream for Emacs 24.2.  (Since CEDET was originally merged in
>>> 23.2, there is at least some symmetry in this).
>>
>> We are, in essence, distributing binary blobs with Emacs, and I find it
>> surprising that this does not seem to disturb anybody much except
>> myself.
>
> The parser generators will be merged. Please take a look at the
> mentioned file-rename branch[1] before speculating further on what the
> merge might not include.

I am not speculating on what the merge might or might not include once
it happens: I have no reason to assume that it will be incomplete with
regard to fulfilling the source requirements.

Most of us have signed a copyright assignment contract to the FSF.  This
contract states, among other things:

    The Foundation promises that any program "based on the Work" offered
    to the public by the Foundation or its assignees shall be offered in
    the form of machine-readable source code, in addition to any other
    forms of the Foundation's choosing. However, the Foundation is free
    to choose at its convenience the media of distribution for
    machine-readable source code.

The definition of source code, according to the GPL, is

      1. Source Code.

      The "source code" for a work means the preferred form of the work
    for making modifications to it.  "Object code" means any non-source
    form of a work.

[...]

      The "Corresponding Source" for a work in object code form means all
    the source code needed to generate, install, and (for an executable
    work) run the object code and to modify the work, including scripts to
    control those activities.  However, it does not include the work's
    System Libraries, or general-purpose tools or generally available free
    programs which are used unmodified in performing those activities but
    which are not part of the work.  For example, Corresponding Source
    includes interface definition files associated with source files for
    the work, and the source code for shared libraries and dynamically
    linked subprograms that the work is specifically designed to require,
    such as by intimate data communication or control flow between those
    subprograms and other parts of the work.

      The Corresponding Source need not include anything that users
    can regenerate automatically from other parts of the Corresponding
    Source.

      The Corresponding Source for a work in source code form is that
    same work.


When we release an Emacs version with a tarball and a new major version
number, nobody will seriously claim that this does not constitute a
release into the public.

So the plan is to openly breach the contract that the FSF has made with
contributors.

Personally, I'd consider that a show-stopper.  And I don't understand
that nobody cares or considers this much of a priority.

-- 
David Kastrup

Re: State of the CEDET merge

[Andreas Röhler]

> So the plan is to openly breach the contract that the FSF has made with
> contributors.
>


Seems you happily reached one of the often criticized aspects of the 
GPL. It's GPL's partly insanity, not the fault of the developers.

Cheers,

Andreas

Re: State of the CEDET merge

[David Kastrup]

Andreas Röhler <andreas.roehler@online.de> writes:

>> So the plan is to openly breach the contract that the FSF has made with
>> contributors.
>>
>
>
> Seems you happily reached one of the often criticized aspects of the
> GPL. It's GPL's partly insanity, not the fault of the developers.

You are not just far from the mark but on the wrong shooting range.

-- 
David Kastrup

Compiled files without sources????

[Richard Stallman]

    >> We are, in essence, distributing binary blobs with Emacs, and I find it
    >> surprising that this does not seem to disturb anybody much except
    >> myself.

If true, that is a very grave problem-- not just legally but ethically.

Could you be so kind as to tell me, concisely, what the situation is?
Which files have been included, which sources are missing, and what
version are they included in?

(I have not followed this thread until now, since I thought it was
just a technical issue.)

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use free telephony http://directory.fsf.org/category/tel/

Re: Compiled files without sources????

[David Kastrup]

Richard Stallman <rms@gnu.org> writes:

>     >> We are, in essence, distributing binary blobs with Emacs, and I
>     >> find it surprising that this does not seem to disturb anybody
>     >> much except myself.
>
> If true, that is a very grave problem-- not just legally but ethically.
>
> Could you be so kind as to tell me, concisely, what the situation is?
> Which files have been included, which sources are missing, and what
> version are they included in?
>
> (I have not followed this thread until now, since I thought it was
> just a technical issue.)

CEDET uses parsers for figuring out syntax of languages.  It has parser
generators that convert bison grammars to Elisp.  The parser generators
are not in Emacs (inconvenient), and the bison grammars from which the
parsers are generated are not in Emacs (bad).

Since the documentation of the parser generators is also omitted, it is
not quite easy for me to tell what the original files for the parser
were (there are several parser generators in CEDET).

At least the following files, according to the comment section in them,
are autogenerated from (nondistributed) grammar files with extension wy
and by.

./lisp/cedet/srecode/srt-wy.el
./lisp/cedet/semantic/wisent/js-wy.el
./lisp/cedet/semantic/wisent/javat-wy.el
./lisp/cedet/semantic/wisent/python-wy.el
./lisp/cedet/semantic/grammar-wy.el
./lisp/cedet/semantic/bovine/scm-by.el
./lisp/cedet/semantic/bovine/make-by.el
./lisp/cedet/semantic/bovine/c-by.el

It is not clear to me what other files may be affected by stripping
CEDET to a runtime environment and matching it to the Emacs tree.

-- 
David Kastrup

Re: Compiled files without sources????

[Richard Stallman]

    CEDET uses parsers for figuring out syntax of languages.  It has parser
    generators that convert bison grammars to Elisp.  The parser generators
    are not in Emacs (inconvenient), and the bison grammars from which the
    parsers are generated are not in Emacs (bad).

The bison grammars MUST be included in Emacs if the parsers are.  The
bison grammars are the source code.  If the Emacs release contains a
compiled program and not its source code, anyone redistributing that
release by itself would violate the GPL.

We must not lead people to violate the GPL!

The parser generators don't have to be included in Emacs as long
as they are free software and we say where to find them.

    It is not clear to me what other files may be affected by stripping
    CEDET to a runtime environment and matching it to the Emacs tree.

I do not understand what that means, or how it relates to the issue.
Could you explain?

    At least the following files, according to the comment section in them,
    are autogenerated from (nondistributed) grammar files with extension wy
    and by.

Thanks for the explanation.  This is a serious issue.

To understand the situation, I need to know one more thing.
Which versions of Emacs are these non-source files included in?

Just some Bzr branches?  Some pretests?  Some releases?
If they are in some releases, which ones?

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use free telephony http://directory.fsf.org/category/tel/

Re: Compiled files without sources????

[Paul Eggert]

On 07/27/11 22:53, Richard Stallman wrote:

> Which versions of Emacs are these non-source files included in?
> 
> Just some Bzr branches?  Some pretests?  Some releases?
> If they are in some releases, which ones?

They are in Emacs releases 23.2 and 23.3.  They have been in all
pretests since emacs-23.1.90, dated 2009-12-09.  They were merged
into the Emacs trunk in bzr 97804, dated 2009-09-28; here's a
URL for that:

http://bzr.savannah.gnu.org/lh/emacs/trunk/revision/97804

Re: Compiled files without sources????

[Richard Stallman]

    They are in Emacs releases 23.2 and 23.3.  They have been in all
    pretests since emacs-23.1.90, dated 2009-12-09.  They were merged
    into the Emacs trunk in bzr 97804, dated 2009-09-28; here's a
    URL for that:

We have made a very bad mistake.  Anyone redistributing those versions
is violating the GPL, through no fault of his own.

We need to fix those releases retroactively (or else delete them), and
we need to do it right away.

I see two quick ways to fix them: to delete the compiled files, or to
add the sources they are made from.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use free telephony http://directory.fsf.org/category/tel/

Re: Compiled files without sources????

[Paul Eggert]

On 07/28/11 16:00, Richard Stallman wrote:
> I see two quick ways to fix them: to delete the compiled files, or to
> add the sources they are made from.

Presumably the latter would be less intrusive, technically.
I don't know where the sources are, though; I don't even
know what CEDET is, other than the name.

Chong merged them into the trunk; perhaps he can say where
the sources are, add them to the trunk, and regenerate the
23.2 and 23.3 tarballs?  I'll CC: this email to him.  (Chong, this thread starts at
<http://lists.gnu.org/archive/html/emacs-devel/2011-07/msg01085.html>.)

Re: Compiled files without sources????

[Milan]

Will it help to look at the upstream cedet?
For the record:
http://cedet.bzr.sourceforge.net/bzr/cedet/code/trunk/files/head%3A/semantic/wisent/

Re: Compiled files without sources????

[Chong Yidong]

Paul Eggert <eggert@cs.ucla.edu> writes:

> On 07/28/11 16:00, Richard Stallman wrote:
>> I see two quick ways to fix them: to delete the compiled files, or to
>> add the sources they are made from.
>
> Presumably the latter would be less intrusive, technically.
> I don't know where the sources are, though; I don't even
> know what CEDET is, other than the name.
>
> Chong merged them into the trunk; perhaps he can say where
> the sources are, add them to the trunk, and regenerate the
> 23.2 and 23.3 tarballs?  I'll CC: this email to him.  (Chong, this
> thread starts at
> <http://lists.gnu.org/archive/html/emacs-devel/2011-07/msg01085.html>.)

Yes, I was the one who handled the big CEDET merge; since the parsers
are Emacs Lisp source code, I didn't think there would be an issue with
the GPL.  My apologies for the screw-up.

If we must include the original grammar files, they are these six files
from the CEDET tarball:

c.by
make.by
scheme.by
javascript-jv.wy
java-tags.wy

There are two options: I can either put these six files in the etc/
directory and regenerate the tarballs, or put them in a separate tarball
in the FTP directory as the Emacs tarballs.  Since the .wy and .by files
are not currently integrated into the Emacs make process (that is one of
the things planned for 24.2), but have to be generated by hand to obtain
the files found in the lisp/ tree, these two options are functionally
equivalent AFAICT.

Re: Compiled files without sources????

[Chong Yidong]

As a first pass, I have added a etc/grammars directory containing the
grammar files in trunk.  The instructions for compiling the grammars is
in etc/grammars/README.  Note that the resulting output differs from the
actual grammar data in lisp/cedet/semantic/[bovine|wisent], because
those were manually modified subsequently.  The README also states that
the contents of these directory should not be depended on existing in
future Emacs version, as they may be changed in the CEDET merge planned
in 24.2.

Re: Compiled files without sources????

[Richard Stallman]

    As a first pass, I have added a etc/grammars directory containing the
    grammar files in trunk.  The instructions for compiling the grammars is
    in etc/grammars/README.  Note that the resulting output differs from the
    actual grammar data in lisp/cedet/semantic/[bovine|wisent], because
    those were manually modified subsequently.

This difference is a problem -- it is like distributing a patched binary.
It doesn't correspond to the source code, and doesn't comply with the GPL.

If you add a script that can patch the generated files in the same
way you edited them, that would do the job.


-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use free telephony http://directory.fsf.org/category/tel/

Re: Compiled files without sources????

[Chong Yidong]

Richard Stallman <rms@gnu.org> writes:

>     As a first pass, I have added a etc/grammars directory containing the
>     grammar files in trunk.  The instructions for compiling the grammars is
>     in etc/grammars/README.  Note that the resulting output differs from the
>     actual grammar data in lisp/cedet/semantic/[bovine|wisent], because
>     those were manually modified subsequently.
>
> This difference is a problem -- it is like distributing a patched
> binary.  It doesn't correspond to the source code, and doesn't comply
> with the GPL.

If developers are forbidden from manually altering the output of a
compiled grammar, that is a usage restriction.

Re: Compiled files without sources????

[Thien-Thi Nguyen]

() Chong Yidong <cyd@stupidchicken.com>
() Sat, 30 Jul 2011 16:33:16 -0400

   Richard Stallman <rms@gnu.org> writes:

   >     As a first pass, I have added a etc/grammars directory containing the
   >     grammar files in trunk.  The instructions for compiling the grammars
   >     is in etc/grammars/README.  Note that the resulting output differs
   >     from the actual grammar data in lisp/cedet/semantic/[bovine|wisent],
   >     because those were manually modified subsequently.
   >
   > This difference is a problem -- it is like distributing a patched
   > binary.  It doesn't correspond to the source code, and doesn't comply
   > with the GPL.

   If developers are forbidden from manually altering the output of a
   compiled grammar, that is a usage restriction.

The problem is that the differences aren't accountable, not that there are
differences.  To be accountable, the path from original to current must be
transparent.  If that path involves manual editing, a commit (+ ChangeLog)
should be generated.  If that path is machine manifested, the program that
does the change must be distributed (+ commit + ChangeLog, of course).  In
the case of distribution, GPL comes into play (recursively).

Re: Compiled files without sources????

[Tim Cross]

On Sun, Jul 31, 2011 at 9:20 AM, Thien-Thi Nguyen <ttn@gnuvola.org> wrote:
> () Chong Yidong <cyd@stupidchicken.com>
> () Sat, 30 Jul 2011 16:33:16 -0400
>
>   Richard Stallman <rms@gnu.org> writes:
>
>   >     As a first pass, I have added a etc/grammars directory containing the
>   >     grammar files in trunk.  The instructions for compiling the grammars
>   >     is in etc/grammars/README.  Note that the resulting output differs
>   >     from the actual grammar data in lisp/cedet/semantic/[bovine|wisent],
>   >     because those were manually modified subsequently.
>   >
>   > This difference is a problem -- it is like distributing a patched
>   > binary.  It doesn't correspond to the source code, and doesn't comply
>   > with the GPL.
>
>   If developers are forbidden from manually altering the output of a
>   compiled grammar, that is a usage restriction.
>
> The problem is that the differences aren't accountable, not that there are
> differences.  To be accountable, the path from original to current must be
> transparent.  If that path involves manual editing, a commit (+ ChangeLog)
> should be generated.  If that path is machine manifested, the program that
> does the change must be distributed (+ commit + ChangeLog, of course).  In
> the case of distribution, GPL comes into play (recursively).
>
>

FYI, for anyone who perhaps may not recognize how important this is,
this issue has been reported in the latest issue of LWN.

Tim

Re: Compiled files without sources????

[David Kastrup]

Tim Cross <theophilusx@gmail.com> writes:

> On Sun, Jul 31, 2011 at 9:20 AM, Thien-Thi Nguyen <ttn@gnuvola.org> wrote:
>
>> The problem is that the differences aren't accountable, not that
>> there are differences.  To be accountable, the path from original to
>> current must be transparent.  If that path involves manual editing, a
>> commit (+ ChangeLog) should be generated.  If that path is machine
>> manifested, the program that does the change must be distributed (+
>> commit + ChangeLog, of course).  In the case of distribution, GPL
>> comes into play (recursively).
>
> FYI, for anyone who perhaps may not recognize how important this is,
> this issue has been reported in the latest issue of LWN.

Both article and comments are quite off-color.  For one thing, they
assume that the parser generator has been Bison and the respective code
C.  That does not really make all that much of a difference with regard
to the implications except that it might have been easier to hand-edit
the files.

Then they assume that the FSF is in violation of the GPL.  Since Emacs
is (c) FSF, the FSF can't be in violation of its own license.  The
issues are different, but still need fixing.

Anyway, I find it bad taste to drag an internal discussion from the list
into the limelight like that.  Even if the original reporter would have
gotten his facts right, the conclusions of the average reader would have
likely been off-color.  The mud-slinging is detrimental to making a
solid and timely fix.

The article in LWN is at <URL:http://lwn.net/Articles/453374/>.  The
front page at <URL:http://lwn.net> has the lead-in

    Emacs distributions are not GPL-compliant
    [Development] Posted Jul 29, 2011 12:10 UTC (Fri) by corbet

    It turns out that the Emacs 23.2 and 23.3 releases contain a number
    of parsers created with Bison, but the source for those parsers was
    not included. That has led Richard Stallman to say: "We have made a
    very bad mistake. Anyone redistributing those versions is violating
    the GPL, through no fault of his own. We need to fix those releases
    retroactively (or else delete them), and we need to do it right
    away." This state of affairs is clearly just a slip which will be
    fixed quickly, but it shows that anybody can make mistakes.

I can't seem to find a good permanent link for the article that would
include this lead-in.

Anyway, even though the publication seems to me to be in bad taste and
ill-advised, I would not want to have mailing list or archives limited
to a closed circle.  It's a risk we have to live with.

-- 
David Kastrup

Re: Compiled files without sources????

[Tim Cross]

On Sun, Jul 31, 2011 at 9:03 PM, David Kastrup <dak@gnu.org> wrote:
> Tim Cross <theophilusx@gmail.com> writes:
>
>> On Sun, Jul 31, 2011 at 9:20 AM, Thien-Thi Nguyen <ttn@gnuvola.org> wrote:
>>
>>> The problem is that the differences aren't accountable, not that
>>> there are differences.  To be accountable, the path from original to
>>> current must be transparent.  If that path involves manual editing, a
>>> commit (+ ChangeLog) should be generated.  If that path is machine
>>> manifested, the program that does the change must be distributed (+
>>> commit + ChangeLog, of course).  In the case of distribution, GPL
>>> comes into play (recursively).
>>
>> FYI, for anyone who perhaps may not recognize how important this is,
>> this issue has been reported in the latest issue of LWN.
>
> Both article and comments are quite off-color.  For one thing, they
> assume that the parser generator has been Bison and the respective code
> C.  That does not really make all that much of a difference with regard
> to the implications except that it might have been easier to hand-edit
> the files.
>
> Then they assume that the FSF is in violation of the GPL.  Since Emacs
> is (c) FSF, the FSF can't be in violation of its own license.  The
> issues are different, but still need fixing.
>
> Anyway, I find it bad taste to drag an internal discussion from the list
> into the limelight like that.  Even if the original reporter would have
> gotten his facts right, the conclusions of the average reader would have
> likely been off-color.  The mud-slinging is detrimental to making a
> solid and timely fix.
>
> The article in LWN is at <URL:http://lwn.net/Articles/453374/>.  The
> front page at <URL:http://lwn.net> has the lead-in
>
>    Emacs distributions are not GPL-compliant
>    [Development] Posted Jul 29, 2011 12:10 UTC (Fri) by corbet
>
>    It turns out that the Emacs 23.2 and 23.3 releases contain a number
>    of parsers created with Bison, but the source for those parsers was
>    not included. That has led Richard Stallman to say: "We have made a
>    very bad mistake. Anyone redistributing those versions is violating
>    the GPL, through no fault of his own. We need to fix those releases
>    retroactively (or else delete them), and we need to do it right
>    away." This state of affairs is clearly just a slip which will be
>    fixed quickly, but it shows that anybody can make mistakes.
>
> I can't seem to find a good permanent link for the article that would
> include this lead-in.
>
> Anyway, even though the publication seems to me to be in bad taste and
> ill-advised, I would not want to have mailing list or archives limited
> to a closed circle.  It's a risk we have to live with.
>

Personally, I was surprised when I read the lead-in. I also thought it
was an 'internal' matter being dealt with, but thought it worth
pointing out, regardless of its accuracy. I also suspect it is really
only getting such attention because of emacs being a well known FSF
package  and Richard's direct involvement.

I guess it is a positive that the issue was one identified within the
emacs dev community and one being dealt with appropriately and quickly
once identified. It does highlights the need for care. In some ways
emacs' and other key FSF projects need to set the standards, which I
think was one of Richard's main points.

Tim

Re: Compiled files without sources????

[Andreas Röhler]

Am 31.07.2011 15:29, schrieb Tim Cross:
> On Sun, Jul 31, 2011 at 9:03 PM, David Kastrup<dak@gnu.org>  wrote:
>> Tim Cross<theophilusx@gmail.com>  writes:
>>
>>> On Sun, Jul 31, 2011 at 9:20 AM, Thien-Thi Nguyen<ttn@gnuvola.org>  wrote:
>>>
>>>> The problem is that the differences aren't accountable, not that
>>>> there are differences.  To be accountable, the path from original to
>>>> current must be transparent.  If that path involves manual editing, a
>>>> commit (+ ChangeLog) should be generated.  If that path is machine
>>>> manifested, the program that does the change must be distributed (+
>>>> commit + ChangeLog, of course).  In the case of distribution, GPL
>>>> comes into play (recursively).
>>>
>>> FYI, for anyone who perhaps may not recognize how important this is,
>>> this issue has been reported in the latest issue of LWN.
>>
>> Both article and comments are quite off-color.  For one thing, they
>> assume that the parser generator has been Bison and the respective code
>> C.  That does not really make all that much of a difference with regard
>> to the implications except that it might have been easier to hand-edit
>> the files.
>>
>> Then they assume that the FSF is in violation of the GPL.  Since Emacs
>> is (c) FSF, the FSF can't be in violation of its own license.  The
>> issues are different, but still need fixing.
>>
>> Anyway, I find it bad taste to drag an internal discussion from the list
>> into the limelight like that.  Even if the original reporter would have
>> gotten his facts right, the conclusions of the average reader would have
>> likely been off-color.  The mud-slinging is detrimental to making a
>> solid and timely fix.
>>
>> The article in LWN is at<URL:http://lwn.net/Articles/453374/>.  The
>> front page at<URL:http://lwn.net>  has the lead-in
>>
>>     Emacs distributions are not GPL-compliant
>>     [Development] Posted Jul 29, 2011 12:10 UTC (Fri) by corbet
>>
>>     It turns out that the Emacs 23.2 and 23.3 releases contain a number
>>     of parsers created with Bison, but the source for those parsers was
>>     not included. That has led Richard Stallman to say: "We have made a
>>     very bad mistake. Anyone redistributing those versions is violating
>>     the GPL, through no fault of his own. We need to fix those releases
>>     retroactively (or else delete them), and we need to do it right
>>     away." This state of affairs is clearly just a slip which will be
>>     fixed quickly, but it shows that anybody can make mistakes.
>>
>> I can't seem to find a good permanent link for the article that would
>> include this lead-in.
>>
>> Anyway, even though the publication seems to me to be in bad taste and
>> ill-advised, I would not want to have mailing list or archives limited
>> to a closed circle.  It's a risk we have to live with.
>>
>
> Personally, I was surprised when I read the lead-in. I also thought it
> was an 'internal' matter being dealt with, but thought it worth
> pointing out, regardless of its accuracy. I also suspect it is really
> only getting such attention because of emacs being a well known FSF
> package  and Richard's direct involvement.
>
> I guess it is a positive that the issue was one identified within the
> emacs dev community and one being dealt with appropriately and quickly
> once identified. It does highlights the need for care. In some ways
> emacs' and other key FSF projects need to set the standards, which I
> think was one of Richard's main points.
>
> Tim
>
>

Hi,

IMHO exists a certain danger GPL might be used against contributors and 
vendors of GPL'ed stuff.

GPL harbors several misconceptions, starting with the famous copyleft.
Might turn out as sword of damocles re-invented.

Cheers,

Andreas

--
https://launchpad.net/python-mode
https://launchpad.net/s-x-emacs-werkstatt/

Re: Compiled files without sources????

[Stephen J. Turnbull]

 > On Sun, Jul 31, 2011 at 9:03 PM, David Kastrup <dak@gnu.org> wrote:

 > > Anyway, I find it bad taste to drag an internal discussion from
 > > the list into the limelight like that.

It's not an "internal discussion".  It is infringement of copyright
for anyone but the FSF (technically, including members of the Emacs
project publishing private branches!) to redistribute a complete Emacs
of those versions verbatim.  There are probably millions of copies in
the wild, and in theory a for-profit distributor could be sued for
statutory damages and subjected to criminal prosecution[1], and *that*
is *no joke*.  The public deserves to be told about this, just as they
are told about other major defects such as security holes in
webservers.

Since this is the FSF's mistake, I can't really see it doing more than
informing people that their distributions are infringing and politely
requesting them to cease and desist until upstream is fixed, of
course.

 > > The mud-slinging is detrimental to making a solid and timely fix.

I don't see why.  Making the fix *is* a purely internal matter.  What
LWN says should (and I expect will) have no effect on the fix (which
is already in progress).

True, the fact that even the FSF can mess up this way is going to make
a few people wonder whether the downsides of copyleft are more serious
than had previously been apparent.  So make a solid and timely fix and
those who have open minds will dismiss the whole thing as a rather
large typo.

Tim Cross writes:

 > Personally, I was surprised when I read the lead-in. I also thought it
 > was an 'internal' matter being dealt with, but thought it worth
 > pointing out, regardless of its accuracy. I also suspect it is really
 > only getting such attention because of emacs being a well known FSF
 > package  and Richard's direct involvement.

Of course.  This is *news*.  Prurient and unimportant, but that's what
sells newspapers, unfortunately.

 > In some ways emacs' and other key FSF projects need to set the
 > standards, which I think was one of Richard's main points.

This isn't really a question of "setting (high) standards."  It's a
matter of (minimal) due diligence (which is required by the license of
licensees), so that the licensees can legally exercise all rights the
license purports to grant.  It's unfortunate that in this case the due
diligence is cleaning up, rather than preventing, a mistake.


Footnotes: 
[1]  Conviction is highly unlikely in the circumstances, of course.
But the law would likely be on the FSF's side as far as forcing the
distributor to try to recover copies, etc., even though the
distribution was 100% in good faith.  That could be rather expensive
for the likes of Red Hat or Canonical!

Re: Compiled files without sources????

[Lars Ingebrigtsen]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> There are probably millions of copies in the wild, and in theory a
> for-profit distributor could be sued for statutory damages and
> subjected to criminal prosecution[1], and *that* is *no joke*.

I think this is, though.

Yes, "in theory" this could happen.  Many things can happen "in theory".
The likelihood of this happening is...  how should I put this...  not
great.

It was a minor boo-boo, and it's been fixed, so let's move on.  I'm sure
you have a ton of bug fixes ready to be applied to the Emacs sources,
right?

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Compiled files without sources????

[Stephen J. Turnbull]

Lars Ingebrigtsen writes:

 > It was a minor boo-boo, and it's been fixed,

It wasn't a "boo-boo", it was a deliberate omission for the
convenience of the Emacs and CEDET developers who considered
integration of CEDET functionality more important than the principle
of software freedom.  The fact that you consider it a "minor boo-boo"
suggests that the underlying problem is nowhere near to being fixed.

And if I were a redistributor of Emacs, I wouldn't consider it
technically fixed until the FSF explicitly licenses my past
distributions which, as it turns out, technically infringe FSF
copyrights since they don't conform to the conditions of the GPL.

Re: Compiled files without sources????

[David Engster]

Stephen J. Turnbull writes:
> Lars Ingebrigtsen writes:
>
>  > It was a minor boo-boo, and it's been fixed,
>
> It wasn't a "boo-boo", it was a deliberate omission for the
> convenience of the Emacs and CEDET developers who considered
> integration of CEDET functionality more important than the principle
> of software freedom.

Since you start with the finger pointing: This merge didn't happen
behind closed doors, but was openly discussed on this very list. It was
actually me who suggested the current setup, and it was in this thread:

http://thread.gmane.org/gmane.emacs.devel/115053

The compromise was to let grammar development happen in CEDET
upstream. I did suggest this in good faith; in fact, it would have taken
almost no work at all to commit the plain grammar files to Emacs
trunk. But without Bovine and Wisent, the CEDET parser generators, those
grammar files are practically useless; contrary to what was written on
LWN/Slashdot/wherever, the grammars are not Bison compatible. To do any
actual work with them, you need to get CEDET, and then you will get all
the grammars anyway. *This* was the reason I suggested to not also
include the grammars in Emacs: I simply thought including them without
the proper parser generators would make no sense. I know now that they
are "The Source" and hence must be included; still, to say that I
suggested leaving them in CEDET upstream because I don't care about
software freedom is simply untrue.

-David

Re: Compiled files without sources????

[David Kastrup]

David Engster <deng@randomsample.de> writes:

> Stephen J. Turnbull writes:
>> Lars Ingebrigtsen writes:
>>
>>  > It was a minor boo-boo, and it's been fixed,
>>
>> It wasn't a "boo-boo", it was a deliberate omission for the
>> convenience of the Emacs and CEDET developers who considered
>> integration of CEDET functionality more important than the principle
>> of software freedom.
>
> Since you start with the finger pointing: This merge didn't happen
> behind closed doors, but was openly discussed on this very list. It was
> actually me who suggested the current setup, and it was in this thread:
>
> http://thread.gmane.org/gmane.emacs.devel/115053
>
> The compromise was to let grammar development happen in CEDET
> upstream. I did suggest this in good faith; in fact, it would have taken
> almost no work at all to commit the plain grammar files to Emacs
> trunk. But without Bovine and Wisent, the CEDET parser generators, those
> grammar files are practically useless; contrary to what was written on
> LWN/Slashdot/wherever, the grammars are not Bison compatible. To do any
> actual work with them, you need to get CEDET, and then you will get all
> the grammars anyway.

Not the corresponding versions to what is included in Emacs.

Now Chong stated that he had to hand-edit the resulting grammars to make
them fit into what is included in Emacs.  That seems like a bad idea
with regard to meeting the source obligations and maintainability.  So
what would be entailed to let upstream CEDET generate compiled grammars
directly usable in Emacs?  Why don't they work out of the box when the
parser runtime is ostensibly what is present in upstream CEDET?  Can the
differences be factored out into variables?

-- 
David Kastrup

Re: Compiled files without sources????

[David Engster]

David Kastrup writes:
> David Engster <deng@randomsample.de> writes:
>> The compromise was to let grammar development happen in CEDET
>> upstream. I did suggest this in good faith; in fact, it would have taken
>> almost no work at all to commit the plain grammar files to Emacs
>> trunk. But without Bovine and Wisent, the CEDET parser generators, those
>> grammar files are practically useless; contrary to what was written on
>> LWN/Slashdot/wherever, the grammars are not Bison compatible. To do any
>> actual work with them, you need to get CEDET, and then you will get all
>> the grammars anyway.
>
> Not the corresponding versions to what is included in Emacs.

True. I was fixed on development, and so you would always work with the
newest grammars form bzr (there are very few changes there anyway). The
older ones are of course not lost, but we should have included them. I
understand that. AFAIK the grammars used in Emacs are those from the
CEDET 1.0 release.

> Now Chong stated that he had to hand-edit the resulting grammars to make
> them fit into what is included in Emacs.  That seems like a bad idea
> with regard to meeting the source obligations and maintainability.

Yes. As you know, this was meant as a transitory period. I know this
sounds silly after two years, but it's already much work just keeping
CEDET working with Emacs24 development and fixing bugs which pop up on
the mailing lists. And sometimes I also like to actually code something
new.

> So what would be entailed to let upstream CEDET generate compiled
> grammars directly usable in Emacs?  Why don't they work out of the box
> when the parser runtime is ostensibly what is present in upstream
> CEDET?  Can the differences be factored out into variables?

The most important change is the 8+3 file renaming; this is replicated
in the file-rename branch. Also, the CEDET compilation process was
complicated and dependant on autoloads from EIEIO classes and methods,
which Emacs doesn't support and which therefore was also removed; Lluís
has written a completely new Makefile for that. I haven't tested yet if
the grammars generated in the new branch work right away, but we should
be close. I'm pretty sure there aren't any differences regarding the
actual grammar rules.

-David

Re: Compiled files without sources????

[Chong Yidong]

David Engster <deng@randomsample.de> writes:

>> So what would be entailed to let upstream CEDET generate compiled
>> grammars directly usable in Emacs?  Why don't they work out of the box
>> when the parser runtime is ostensibly what is present in upstream
>> CEDET?  Can the differences be factored out into variables?
>
> The most important change is the 8+3 file renaming; this is replicated
> in the file-rename branch. Also, the CEDET compilation process was
> complicated and dependant on autoloads from EIEIO classes and methods,
> which Emacs doesn't support and which therefore was also removed; Lluís
> has written a completely new Makefile for that.

It's actually not as complicated as that.  The major changes between the
compiled grammars and the files actually included in Emacs are (i)
renaming of the files, (ii) fixed-up copyright headers, (iii) changes in
the `feature' keyword of the file plus the filename, and (iv)
re-ordering of some code chunks in the file to avoid some byte
compilation warnings (IIRC).  These changes are not really practical to
do except by hand.

Here is an example, comparing the compiled from etc/grammars/make.by to
lisp/cedet/semantic/bovine/make-by.el on the trunk.  (The bovine parser
generator is currently broken on the trunk, because of a bug related to
the "backquote not followed by a space" change, so generation was
actually done with Emacs 23.)


*** trunk/etc/grammars/semantic-make-by.el	2011-07-31 19:58:28.774752136 -0400
--- trunk/lisp/cedet/semantic/bovine/make-by.el	2011-07-30 20:08:50.987784021 -0400
***************
*** 1,36 ****
! ;;; semantic-make-by.el --- Generated parser support file
  
! ;; Copyright (C) 1999-2011 Free Software Foundation, Inc.
  
! ;; Author: Chong Yidong <cyd@furball>
! ;; Created: 2011-07-31 19:58:28-0400
! ;; Keywords: syntax
! ;; X-RCS: $Id$
  
! ;; This file is not part of GNU Emacs.
  
! ;; This program is free software; you can redistribute it and/or
! ;; modify it under the terms of the GNU General Public License as
! ;; published by the Free Software Foundation, either version 3 of
! ;; the License, or (at your option) any later version.
! 
! ;; This software is distributed in the hope that it will be useful,
  ;; but WITHOUT ANY WARRANTY; without even the implied warranty of
! ;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
! ;; General Public License for more details.
! ;;
  ;; You should have received a copy of the GNU General Public License
! ;; along with this program.  If not, see <http://www.gnu.org/licenses/>.
  
  ;;; Commentary:
  ;;
! ;; PLEASE DO NOT MANUALLY EDIT THIS FILE!  It is automatically
! ;; generated from the grammar file make.by.
! 
! ;;; History:
! ;;
  
  ;;; Code:
  \f
  ;;; Prologue
  ;;
--- 1,31 ----
! ;;; semantic/bovine/make-by.el --- Generated parser support file
  
! ;; Copyright (C) 1999-2004, 2008-2011  Free Software Foundation, Inc.
  
! ;; This file is part of GNU Emacs.
  
! ;; GNU Emacs is free software: you can redistribute it and/or modify
! ;; it under the terms of the GNU General Public License as published by
! ;; the Free Software Foundation, either version 3 of the License, or
! ;; (at your option) any later version.
  
! ;; GNU Emacs is distributed in the hope that it will be useful,
  ;; but WITHOUT ANY WARRANTY; without even the implied warranty of
! ;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
! ;; GNU General Public License for more details.
! 
  ;; You should have received a copy of the GNU General Public License
! ;; along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.
  
  ;;; Commentary:
  ;;
! ;; This file was generated from etc/grammars/make.by.
  
  ;;; Code:
+ 
+ (require 'semantic/lex)
+ (eval-when-compile (require 'semantic/bovine))
+ 
  \f
  ;;; Prologue
  ;;
***************
*** 70,76 ****
  
  (defconst semantic-make-by--parse-table
    `(
!     (bovine-toplevel 
       (Makefile)
       ) ;; end bovine-toplevel
  
--- 65,71 ----
  
  (defconst semantic-make-by--parse-table
    `(
!     (bovine-toplevel
       (Makefile)
       ) ;; end bovine-toplevel
  
***************
*** 385,399 ****
  	semantic-flex-keywords-obarray semantic-make-by--keyword-table
  	))
  
! \f
! ;;; Analyzers
! ;;
! (require 'semantic-lex)
! 
! \f
! ;;; Epilogue
! ;;
! 
! (provide 'semantic-make-by)
  
! ;;; semantic-make-by.el ends here
--- 380,385 ----
  	semantic-flex-keywords-obarray semantic-make-by--keyword-table
  	))
  
! (provide 'semantic/bovine/make-by)
  
! ;;; semantic/bovine/make-by.el ends here

Re: Compiled files without sources????

[Tim Cross]

On Mon, Aug 1, 2011 at 6:06 AM, Stephen J. Turnbull <stephen@xemacs.org> wrote:
> Lars Ingebrigtsen writes:
>
>  > It was a minor boo-boo, and it's been fixed,
>
> It wasn't a "boo-boo", it was a deliberate omission for the
> convenience of the Emacs and CEDET developers who considered
> integration of CEDET functionality more important than the principle
> of software freedom.  The fact that you consider it a "minor boo-boo"
> suggests that the underlying problem is nowhere near to being fixed.
>

I think this is a completely unjustified statement. There has been
nothing in this thread that suggests these files were not included
simply for the convenience of developers or because functionality was
prioritized over licensing. The fact the files were able to be added
without any significant problems of integration does not support your
argument as including or not including has little convenience value.
The decision to not include them may have been deliberate, but there
is simply insufficient information to judge the reasons why they were
omitted. Licensing and legal matters are complex, often surprising and
frequently open to multiple interpretations. I think it is far more
likely the error was due to an honest error in
understanding/interpretation of the licensing requirements rather than
the somewhat malicious and deliberate interpretation you have
suggested.

Tim

Re: Compiled files without sources????

[Stephen J. Turnbull]

Tim Cross writes:

 > I think it is far more likely the error was due to an honest error
 > in understanding/interpretation of the licensing requirements
 > rather than the somewhat malicious and deliberate interpretation
 > you have suggested.

No malice at all is implied, just a lack of due diligence.  Obviously
there was an error in understanding the implications of the licensing
requirements for what Emacs *should* do (though Emacs is in no way
legally bound, of course).  More important in my opinion is the fact
that portions of the build process and sources were removed to make
maintenance easier, leaving "unmaintainable binary content" as David
Kastrup describes it.  Evidently the maintainers did not consider at
all that the users' rights to study, modify, and redistribute the code
would thereby be impaired.

Deliberate, yes.  In http://article.gmane.org/gmane.emacs.devel/115057
Chong Yidong writes: "I think it's better for us to merge just the
generated Lisp grammar files, leaving the grammar development for
upstream.  It's an awful lot of infrastructure to pull in...."  IMO,
"just the generated files" should be a red flag for any maintainer
working with copyleft licensing.

Re: Compiled files without sources????

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> Tim Cross writes:
>
>  > I think it is far more likely the error was due to an honest error
>  > in understanding/interpretation of the licensing requirements
>  > rather than the somewhat malicious and deliberate interpretation
>  > you have suggested.
>
> No malice at all is implied, just a lack of due diligence.  Obviously
> there was an error in understanding the implications of the licensing
> requirements for what Emacs *should* do (though Emacs is in no way
> legally bound, of course).

The FSF, being the principal copyright holder of Emacs, is not bound to
the conditions of the GPL in a legal sense, but setting a bad example
like that is not going to increase the respect from others.

The FSF is bound to the conditions of the copyright assignment contracts
from various contributors, however.  While they are free to pick a
number of free licensing arrangements for Emacs code not restricted to
the GPL, they are not free to distribute generated code without source.

People refusing to license under a GPLv$$+ scheme or assign their
copyright to the FSF because either would be tantamount to a blank
cheque and the FSF could turn bad, tend to forget that the FSF has
entered into legally binding agreements as well.  Even if the FSF became
a subsidiary of the Gates foundation tomorrow, nothing really bad could
happen to the software for which the FSF signed contracts.

So "Emacs" (which is not a sentient entity yet) is certainly not just
morally, but also legally bound to providing corresponding source code
to its binaries.

> Deliberate, yes.  In http://article.gmane.org/gmane.emacs.devel/115057
> Chong Yidong writes: "I think it's better for us to merge just the
> generated Lisp grammar files, leaving the grammar development for
> upstream.  It's an awful lot of infrastructure to pull in...."  IMO,
> "just the generated files" should be a red flag for any maintainer
> working with copyleft licensing.

Well, you have the advantage of long years of training looking for such
pitfalls as project/release manager of XEmacs.  Emacs has had less
reason or opportunity to try staying under the radar and/or out of reach
of Richard.  Perhaps you should offer courses.

-- 
David Kastrup

Re: Compiled files without sources????

[Stephen J. Turnbull]

David Kastrup writes:

 > The FSF is bound to the conditions of the copyright assignment contracts
 > from various contributors, however.  While they are free to pick a
 > number of free licensing arrangements for Emacs code not restricted to
 > the GPL, they are not free to distribute generated code without
 > source.

I stand corrected.

 > Well, you have the advantage of long years of training looking for such
 > pitfalls as project/release manager of XEmacs.

Actually, that has very little to do with it.  We basically have a
philosophy that our product is source code and build/installation
framework, and any prebuilt stuff is purely for installation
convenience.  That makes these decisions (mostly) easy.  The reason I
know anything about this legal stuff is that I take an interest in the
philosophy of software freedom.  Most XEmacs workers, including
previous managers, didn't know or care as much, and didn't have to.

I would think that taking a philosophical interest would be a
prerequisite for maintainership of Emacs, though.

 > Perhaps you should offer courses.

Nah, philosophers are born, not made.

Re: Compiled files without sources????

[Thien-Thi Nguyen]

() "Stephen J. Turnbull" <stephen@xemacs.org>
() Mon, 01 Aug 2011 23:37:57 +0900

   Nah, philosophers are born, not made.

Well, there is marriage... :-D

Re: Compiled files without sources????

[Richard Stallman]

I don't think that we violated the assignment contract.  We do release
the source files (the grammar files), which is what the contract
requires.  Our error was to release them in a different package, and
not in the same package with the non-source form of the same code.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use free telephony http://directory.fsf.org/category/tel/

Re: Compiled files without sources????

[David Kastrup]

Richard Stallman <rms@gnu.org> writes:

> I don't think that we violated the assignment contract.  We do release
> the source files (the grammar files), which is what the contract
> requires.

Who is "we"?  CEDET is not a GNU project.  It is hosted on SourceForge.
It is not nor has ever been released by the FSF short of the parts
distributed with Emacs, and is not to be found on GNU servers otherwise.

> Our error was to release them in a different package, and not in the
> same package with the non-source form of the same code.

Could you give a pointer to that package?

-- 
David Kastrup

Re: Compiled files without sources????

[Richard Stallman]

    Who is "we"?  CEDET is not a GNU project.  It is hosted on SourceForge.
    It is not nor has ever been released by the FSF short of the parts
    distributed with Emacs, and is not to be found on GNU servers otherwise.

The assignment contract that is relevant is the assignment for CEDET
itself.  I think our release of patched generated files without
corresponding source contradicts our commitments in that assignment.
Yet another reason this must never be done.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use free telephony http://directory.fsf.org/category/tel/

Letting the users off the hook

[Richard Stallman]

The FSF hereby grants permission to distribute the old Emacs 23.1 and
23.2 releases, notwithstanding the absence therein of the real source
files for the CEDET parsers (the grammar files).  This permission
extends to versions with unrelated modifications, provided those
versions comply with GNU GPL version 3 in all other respects.

Thanks to improvements in GPL version 3, we don't need to explicitly
forgive past violations caused by redistributing Emacs 23.1 and 23.2,
because those violations would only become legally significant if the
FSF were to complain about them to the redistributors within 60 days.
The FSF, having granted the above permission to distribute those
releases, could not complain about it.

The FSF urges all redistributors to upgrade to the revised Emacs 23.1
and 23.2 releases, which do not have this problem.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use free telephony http://directory.fsf.org/category/tel/

Letting the users off the hook

[Richard Stallman]

This time with the right version numbers.


The FSF hereby grants permission to distribute the old Emacs 23.2 and
23.3 releases, notwithstanding the absence therein of the real source
files for the CEDET parsers (the grammar files).  This permission
extends to versions with unrelated modifications, provided those
versions comply with GNU GPL version 3 in all other respects.

Thanks to improvements in GPL version 3, we don't need to explicitly
forgive past violations caused by redistributing Emacs 23.1 and 23.2,
because those violations would only become legally significant if the
FSF were to complain about them to the redistributors within 60 days.
The FSF, having granted the above permission to distribute those
releases, could not complain about it.

The FSF urges all redistributors to upgrade to the revised Emacs 23.1
and 23.2 releases, which do not have this problem.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use free telephony http://directory.fsf.org/category/tel/

Letting the users off the hook

[Stephen J. Turnbull]

Richard Stallman writes:

 > This time with the right version numbers.
 > 
 > 
 > The FSF hereby grants permission to distribute the old Emacs 23.2 and
 > 23.3 releases, notwithstanding the absence therein of the real source
 > files for the CEDET parsers (the grammar files).

Thank you!  I'm very glad it was this easy (the technical part; I
suppose that behind the scenes the FSF legal staff had to scratch
their collective head for a few hours; many thanks to them too!)

Steve

Re: Letting the users off the hook

[Richard Stallman]

Thanks for upholding the importance of this.


-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use free telephony http://directory.fsf.org/category/tel/

Re: Compiled files without sources????

[Richard Stallman]

    It wasn't a "boo-boo", it was a deliberate omission for the
    convenience of the Emacs and CEDET developers who considered
    integration of CEDET functionality more important than the principle
    of software freedom.  The fact that you consider it a "minor boo-boo"
    suggests that the underlying problem is nowhere near to being fixed.

Hear, hear!  Part of the responsibility of GNU maintainers is
to give the principle of software freedom highest priority.

    And if I were a redistributor of Emacs, I wouldn't consider it
    technically fixed until the FSF explicitly licenses my past
    distributions which, as it turns out, technically infringe FSF
    copyrights since they don't conform to the conditions of the GPL.

That is a good point.  Of course, we should do this.  But first we
should give them corrected tarballs as replacements to switch to.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use free telephony http://directory.fsf.org/category/tel/

Re: Compiled files without sources????

[Richard Stallman]

    > This difference is a problem -- it is like distributing a patched
    > binary.  It doesn't correspond to the source code, and doesn't comply
    > with the GPL.

    If developers are forbidden from manually altering the output of a
    compiled grammar, that is a usage restriction.

I'm not sure who you mean by "developers", and I can't understand the
rest of the sentence concretely ether, but I think you've changed the
subject.  The issue here is what we must do.

For any generated files we distribute, we must provide the
corresponding source code.  Somewhat related source code is not
enough.

If you write a script that does the editing, that will fill the gap.
The grammar file, plus the script, will add up to the source code
to produce the needed generated file.  Can you please add that
to the repository and the release tarballs?


-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use free telephony http://directory.fsf.org/category/tel/

Re: Compiled files without sources????

[Chong Yidong]

Richard Stallman <rms@gnu.org> writes:

> I'm not sure who you mean by "developers", and I can't understand the
> rest of the sentence concretely ether, but I think you've changed the
> subject.  For any generated files we distribute, we must provide the
> corresponding source code.  Somewhat related source code is not
> enough.

The point is that the generated code was subsequently modified as though
it was source code, not as though it was object code.  If this activity
is forbidden, that seems like an unreasonable restriction.

> If you write a script that does the editing, that will fill the gap.
> The grammar file, plus the script, will add up to the source code to
> produce the needed generated file.  Can you please add that to the
> repository and the release tarballs?

A "script" is just a diff of the old and new source files, which anybody
can trivially do.

Re: Compiled files without sources????

[Richard Stallman]

    > If you write a script that does the editing, that will fill the gap.
    > The grammar file, plus the script, will add up to the source code to
    > produce the needed generated file.  Can you please add that to the
    > repository and the release tarballs?

    A "script" is just a diff of the old and new source files, which anybody
    can trivially do.

A script might include that diff, plus a command to run patch on the
file.  Since you say it is trivial, please do it.

However, a better script would be more flexible.  It would search for
certain things, delete them from where they are, and insert them in
the right place.  Etc.  That way, it would work right even if the
source files are edited to some extent.

That won't be quite trivial, but it shouldn't take more than an hour.

Making our releases follow the GPL is the wrong place to cut corners.
Please don't argue that 99% is enough.  We need to do this 100%,
and that is what we will do -- one way or another.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use free telephony http://directory.fsf.org/category/tel/

Re: Compiled files without sources????

[Chong Yidong]

Richard Stallman <rms@gnu.org> writes:

> A script might include that diff, plus a command to run patch on the
> file.  Since you say it is trivial, please do it.

OK, I will add a diff to etc/grammars and instructions on how to apply
it, and upload the corrected 23.2 and 23.3 tarballs tonight.

Replacing tarballs on-the-fly (was: Re: Compiled files without sources????)

[Ulrich Mueller]

>>>>> On Sun, 31 Jul 2011, Chong Yidong wrote:

> OK, I will add a diff to etc/grammars and instructions on how to
> apply it, and upload the corrected 23.2 and 23.3 tarballs tonight.

Is there a particular reason why emacs-23.2.tar.bz2 on
http://ftp.gnu.org/gnu/emacs/ hasn't been updated?

Also, it would have been much cleaner if the updated tarballs hadn't
been replaced on-the-fly, but been released under a new name, like
emacs-23.3a.tar.bz2, in order to avoid confusion.

(For example, Gentoo as a source-based distro does file integrity
checks based on SHA256 checksums. I've now updated the checksums for
the Emacs tarballs, but of course the check will now fail if the user
downloads the old version of the tarball from a mirror. There's no
good way to distinguish old and new before downloading the file.)

Ulrich

Re: Replacing tarballs on-the-fly

[Chong Yidong]

Ulrich Mueller <ulm@gentoo.org> writes:

> Is there a particular reason why emacs-23.2.tar.bz2 on
> http://ftp.gnu.org/gnu/emacs/ hasn't been updated?

Operator error in the ftp upload.  The replaced files are now all on the
FTP server, and named emacs-23.2a.tar.gz etc.

Re: Replacing tarballs on-the-fly

[Ulrich Mueller]

>>>>> On Tue, 02 Aug 2011, Chong Yidong wrote:

> Operator error in the ftp upload. The replaced files are now all on
> the FTP server, and named emacs-23.2a.tar.gz etc.

Thank you.

Another minor point: There's still an emacs-23.1-23.2.xdelta with
a date from 2010. Probably that file should be deleted too?

Ulrich

Re: Replacing tarballs on-the-fly

[Richard Stallman]

    Another minor point: There's still an emacs-23.1-23.2.xdelta with
    a date from 2010. Probably that file should be deleted too?

Yes, it should be.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use free telephony http://directory.fsf.org/category/tel/

Re: Replacing tarballs on-the-fly

[Chong Yidong]

Richard Stallman <rms@gnu.org> writes:

>     Another minor point: There's still an emacs-23.1-23.2.xdelta with
>     a date from 2010. Probably that file should be deleted too?
>
> Yes, it should be.

Deleted.

I didn't bother generating a new xdelta; the fact that no one has ever
complained about a lack of 23.2-23.3 xdelta indicates that nobody is
using it.

Re: Compiled files without sources????

[Richard Stallman]

    OK, I will add a diff to etc/grammars and instructions on how to apply
    it, and upload the corrected 23.2 and 23.3 tarballs tonight.

Thank you.  That is a real fix.

A more intelligent and general correction script might be easy to do.
If you can spare an hour, that would be a significant improvement.

The new tarballs should have different names from the old, so people
can see which one they have got.  Perhaps emacs-23.2a.tar, etc.  The
Emacs versions per se should be unchanged, and the directory it
unpacks to should unchanged -- only the tar file name should change.

The old tar files should be deleted.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use free telephony http://directory.fsf.org/category/tel/

Re: Compiled files without sources????

[Chong Yidong]

Chong Yidong <cyd@stupidchicken.com> writes:

> OK, I will add a diff to etc/grammars and instructions on how to apply
> it, and upload the corrected 23.2 and 23.3 tarballs tonight.

The corrected files can now be found at the following locations:

ftp://ftp.gnu.org/gnu/emacs/emacs-23.2a.tar.bz2
ftp://ftp.gnu.org/gnu/emacs/emacs-23.2a.tar.gz
ftp://ftp.gnu.org/gnu/emacs/emacs-23.3a.tar.bz2
ftp://ftp.gnu.org/gnu/emacs/emacs-23.3a.tar.gz

Re: Compiled files without sources????

[Richard Stallman]

I think the only thing we still need to do is make sure that all the
branches that contain the generated files also contain the sources.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use free telephony http://directory.fsf.org/category/tel/

Re: Compiled files without sources????

[Richard Stallman]

    There are two options: I can either put these six files in the etc/
    directory and regenerate the tarballs, or put them in a separate tarball
    in the FTP directory as the Emacs tarballs.

Please include them in the Emacs tarballs.  We want redistribution of
those tarballs in the most natural way to comply with the GPL.  To
make that true, we need to fix those tarballs.

Putting the grammar files in a separate tar file would not achieve
this goal.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use free telephony http://directory.fsf.org/category/tel/

Re: Compiled files without sources????

[Evgeny M. Zubok]

Richard Stallman <rms@gnu.org> writes:

> Could you be so kind as to tell me, concisely, what the situation is?
> Which files have been included, which sources are missing, and what
> version are they included in?

Second this question. Some packages in ELPA provide only compiled
documentation (info, PostScript, PDF). The sources of documentation
exist only on upstream. Should the packages always provide the sources
of documentation in ELPA tree? Is it acceptable that the sources exist
somewhere on project's hosts?

Re: Compiled files without sources????

[David Kastrup]

"Evgeny M. Zubok" <evgeny.zubok@tochka.ru> writes:

> Richard Stallman <rms@gnu.org> writes:
>
>> Could you be so kind as to tell me, concisely, what the situation is?
>> Which files have been included, which sources are missing, and what
>> version are they included in?
>
> Second this question. Some packages in ELPA provide only compiled
> documentation (info, PostScript, PDF). The sources of documentation
> exist only on upstream. Should the packages always provide the sources
> of documentation in ELPA tree? Is it acceptable that the sources exist
> somewhere on project's hosts?

It might be worth taking a look at the structure of an XEmacs package.
While the source of them in the XEmacs package tree has a different
layout and contains additional structural information files relevant for
generating the package, the compiled XEmacs package contains all
relevant sources for the included material, in particular the Texinfo
sources.

-- 
David Kastrup

Re: Compiled files without sources????

[Richard Stallman]

    Second this question. Some packages in ELPA provide only compiled
    documentation (info, PostScript, PDF). The sources of documentation
    exist only on upstream.

That is not acceptable.

			    Should the packages always provide the sources
    of documentation in ELPA tree?

Yes, they must.  That requirement applies to _everything_ we distribute.

We can distribute non-source files too, together with the sources or
separately, but the _source_ package must include source for
_everything in it_!

There are multiple archives called "ELPA" and I am not sure which one
you mean.  If it is the GNU one, we definitely must add the
documentation source.  If it is another, we cannot tell those people
what to do with code that isn't copyright FSF.  But we won't refer the
users to any archive that fails to follow thus rule.


-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use free telephony http://directory.fsf.org/category/tel/

Re: Compiled files without sources????

[Evgeny M. Zubok]

Richard Stallman <rms@gnu.org> writes:

> There are multiple archives called "ELPA" and I am not sure which one
> you mean.  If it is the GNU one, we definitely must add the
> documentation source.  If it is another, we cannot tell those people
> what to do with code that isn't copyright FSF.  But we won't refer the
> users to any archive that fails to follow thus rule.

Yes, I meant the GNU ELPA and the packages `muse' and `auctex'.

Re: Compiled files without sources????

[David Kastrup]

"Evgeny M. Zubok" <evgeny.zubok@tochka.ru> writes:

> Richard Stallman <rms@gnu.org> writes:
>
>> There are multiple archives called "ELPA" and I am not sure which one
>> you mean.  If it is the GNU one, we definitely must add the
>> documentation source.  If it is another, we cannot tell those people
>> what to do with code that isn't copyright FSF.  But we won't refer the
>> users to any archive that fails to follow thus rule.
>
> Yes, I meant the GNU ELPA and the packages `muse' and `auctex'.

I am somewhat surprised since the upstream AUCTeX source includes an
XEmacs package in the Makefile that does come with the documentation
sources.

Their package format has been designed with one goal being not to have
the GPL compliance police breathe down the XEmacs developers' necks, so
it would seem fair that we ourselves try to meet the standards others
can reasonably expect to be subjected to by us.

-- 
David Kastrup

Re: Compiled files without sources????

[Chong Yidong]

"Evgeny M. Zubok" <evgeny.zubok@tochka.ru> writes:

> Richard Stallman <rms@gnu.org> writes:
>
>> There are multiple archives called "ELPA" and I am not sure which one
>> you mean.  If it is the GNU one, we definitely must add the
>> documentation source.  If it is another, we cannot tell those people
>> what to do with code that isn't copyright FSF.  But we won't refer the
>> users to any archive that fails to follow thus rule.
>
> Yes, I meant the GNU ELPA and the packages `muse' and `auctex'.

Thanks for the heads up, I've added the Texinfo files to these packages
in the respository.

Re: Compiled files without sources????

[Richard Stallman]

    Thanks for the heads up, I've added the Texinfo files to these packages
    in the respository.

Thanks for getting that fixed.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use free telephony http://directory.fsf.org/category/tel/

Re: Emacs 23.3 released

[Juanma Barranquero]

On Thu, Mar 10, 2011 at 17:07, Stefan Monnier <monnier@iro.umontreal.ca> wrote:

> Talking about 24.1, I now consider the lexbind-new branch as "fully
> functional": there are still a few changes I want to do there before
> merging, but they should only impact performance and code quality, so
> I encourage people to try it out and report any regression so they can
> be addressed before we merge it into trunk.

Trying to bootstrap it on Windows I get 15 errors of the kind

  Debugger entered--Lisp error: (cyclic-function-indirection vc-update)

followed by debugger backtraces (they're quite longish, I can send you
a gzipped log if needed).

Finally, the bootstrap hangs after

  Wrote c:/emacs/lexbind-new/lisp/vc/vc-svn.elc

    Juanma

Re: Emacs 23.3 released

[Stefan Monnier]

> Trying to bootstrap it on Windows I get 15 errors of the kind
>   Debugger entered--Lisp error: (cyclic-function-indirection vc-update)

Hmm... I don't remember seeing such errors and I can't think of what
change in lexbind could get us there.  Any chance you didn't start from
a clean tree?

> followed by debugger backtraces (they're quite longish, I can send you
> a gzipped log if needed).

The backtraces are very verbose because I changed debug.el's
print-length and/or print-depth so *Backtrace* has more data, but indeed
it makes the backtraces on stdout unusable, so you may want to check
lisp/emacs-lisp/debug.el and undo the change there.

> Finally, the bootstrap hangs after
>   Wrote c:/emacs/lexbind-new/lisp/vc/vc-svn.elc

Again, I don't have much idea how/why that could happen :-(


        Stefan

Re: Emacs 23.3 released

[Juanma Barranquero]

On Fri, Mar 11, 2011 at 05:26, Stefan Monnier <monnier@iro.umontreal.ca> wrote:

> Hmm... I don't remember seeing such errors and I can't think of what
> change in lexbind could get us there.  Any chance you didn't start from
> a clean tree?

No, I started with "bzr clean-tree --unknown --ignored --detritus".

> The backtraces are very verbose because I changed debug.el's
> print-length and/or print-depth so *Backtrace* has more data, but indeed
> it makes the backtraces on stdout unusable, so you may want to check
> lisp/emacs-lisp/debug.el and undo the change there.

OK, I'll try that.

    Juanma

Re: Emacs 23.3 released

[Andreas Schwab]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Trying to bootstrap it on Windows I get 15 errors of the kind
>>   Debugger entered--Lisp error: (cyclic-function-indirection vc-update)
>
> Hmm... I don't remember seeing such errors and I can't think of what
> change in lexbind could get us there.  Any chance you didn't start from
> a clean tree?

Did you keep ldefs-boot.el up-to-date?

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Emacs 23.3 released

[Stefan Monnier]

>>> Trying to bootstrap it on Windows I get 15 errors of the kind
>>> Debugger entered--Lisp error: (cyclic-function-indirection vc-update)
>> Hmm... I don't remember seeing such errors and I can't think of what
>> change in lexbind could get us there.  Any chance you didn't start from
>> a clean tree?
> Did you keep ldefs-boot.el up-to-date?

The new branch introduces little if any changes w.r.t autoloads.
Most/all new functions are in cconv.el which is `require'd when needed
(tho it's also autoloaded).
So, no I haven't touched ldefs-boot.el and am happy for it because
I avoid merge conflicts this way ;-)


        Stefan

Re: Emacs 23.3 released

[Andreas Schwab]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

> So, no I haven't touched ldefs-boot.el

That's your problem.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Emacs 23.3 released

[Juanma Barranquero]

On Sat, Mar 12, 2011 at 00:03, Andreas Schwab <schwab@linux-m68k.org> wrote:
> Stefan Monnier <monnier@IRO.UMontreal.CA> writes:
>
>> So, no I haven't touched ldefs-boot.el
>
> That's your problem.

Certainly, copying the trunk's ldefs-boot.el to lexbind-new/lisp
helps; with it, bootstrap does not emit weird messages and does not
hang.

Bootstrapping still fails, though, with this:

make[1]: Leaving directory `C:/emacs/lexbind-new/lib-src'
make[1]: *** No rule to make target `../lisp/files.elc', needed by `DOC'.  Stop.
make: *** [all-other-dirs-gmake] Error 2

because files.el has not been compiled:

 Lisp nesting exceeds `max-lisp-eval-depth'

    Juanma

Re: Emacs 23.3 released

[Stefan Monnier]

> because files.el has not been compiled:
>  Lisp nesting exceeds `max-lisp-eval-depth'

That's easier to fix: check lisp/Makefile.in where I pass higher values
of max-lisp-eval-depth.  Hmm... I guess I should have expected it, the
w32 doesn't use Makefile.in.

Try the 100% guaranteed untested patch below, which should more or less
mimick what we use on POSIX platforms.


        Stefan


=== modified file 'lisp/makefile.w32-in'
--- lisp/makefile.w32-in	2011-02-14 00:23:11 +0000
+++ lisp/makefile.w32-in	2011-03-12 01:37:28 +0000
@@ -66,6 +66,15 @@
 	$(lisp)/cedet/semantic/loaddefs.el $(lisp)/cedet/ede/loaddefs.el \
 	$(lisp)/cedet/srecode/loaddefs.el
 
+# Value of max-lisp-eval-depth when compiling initially.
+# During bootstrapping the byte-compiler is run interpreted when compiling
+# itself, and uses more stack than usual.
+#
+BIG_STACK_DEPTH = 1200
+BIG_STACK_OPTS = --eval "(setq max-lisp-eval-depth $(BIG_STACK_DEPTH))"
+
+BYTE_COMPILE_FLAGS = $(BIG_STACK_OPTS) $(BYTE_COMPILE_EXTRA_FLAGS)
+
 # Files to compile before others during a bootstrap.  This is done to
 # speed up the bootstrap process.  The CC files are compiled first
 # because CC mode tweaks the compilation process, and requiring
@@ -75,6 +84,9 @@
 COMPILE_FIRST = \
 	$(lisp)/emacs-lisp/byte-opt.el \
 	$(lisp)/emacs-lisp/bytecomp.el \
+	$(lisp)/emacs-lisp/pcase.elc \
+	$(lisp)/emacs-lisp/macroexp.elc \
+	$(lisp)/emacs-lisp/cconv.elc \
 	$(lisp)/subr.el \
 	$(lisp)/progmodes/cc-mode.el \
 	$(lisp)/progmodes/cc-vars.el
@@ -287,7 +299,7 @@
 .SUFFIXES: .elc .el
 
 .el.elc:
-	-$(emacs) $(BYTE_COMPILE_EXTRA_FLAGS) -f batch-byte-compile $<
+	-$(emacs) $(BYTE_COMPILE_FLAGS) -f batch-byte-compile $<
 
 # Compile all Lisp files, but don't recompile those that are up to
 # date.  Some files don't actually get compiled because they set the
@@ -307,22 +319,22 @@
 compile-CMD:
 #	-for %%f in ($(lisp) $(WINS)) do for %%g in (%%f\*.elc) do @attrib -r %%g
 	for %%f in ($(COMPILE_FIRST)) do \
-	  $(emacs) -l loaddefs $(BYTE_COMPILE_EXTRA_FLAGS) -f batch-byte-compile-if-not-done %%f
+	  $(emacs) -l loaddefs $(BYTE_COMPILE_FLAGS) -f batch-byte-compile-if-not-done %%f
 	for %%f in (. $(WINS)) do for %%g in (%%f/*.el) do \
-	  $(emacs) -l loaddefs $(BYTE_COMPILE_EXTRA_FLAGS) -f batch-byte-compile-if-not-done %%f/%%g
+	  $(emacs) -l loaddefs $(BYTE_COMPILE_FLAGS) -f batch-byte-compile-if-not-done %%f/%%g
 
 compile-SH:
 #	for elc in $(lisp)/*.elc $(lisp)/*/*.elc; do attrib -r $$elc; done
 	for el in $(COMPILE_FIRST); do \
 	  echo Compiling $$el; \
-	  $(emacs) -l loaddefs $(BYTE_COMPILE_EXTRA_FLAGS) -f batch-byte-compile-if-not-done $$el; \
+	  $(emacs) -l loaddefs $(BYTE_COMPILE_FLAGS) -f batch-byte-compile-if-not-done $$el; \
 	done
 	for dir in $(lisp) $(WINS); do \
 	  for el in $$dir/*.el; do \
 	    if test -f $$el; \
 	    then \
 	      echo Compiling $$el; \
-	      $(emacs) -l loaddefs $(BYTE_COMPILE_EXTRA_FLAGS) -f batch-byte-compile-if-not-done $$el; \
+	      $(emacs) -l loaddefs $(BYTE_COMPILE_FLAGS) -f batch-byte-compile-if-not-done $$el; \
 	    fi \
 	  done; \
 	done
@@ -335,31 +347,31 @@
 
 compile-always-CMD:
 #	-for %%f in ($(lisp) $(WINS)) do for %%g in (%%f\*.elc) do @attrib -r %%g
-	for %%f in ($(COMPILE_FIRST)) do $(emacs) $(BYTE_COMPILE_EXTRA_FLAGS) -f batch-byte-compile %%f
-	for %%f in (. $(WINS)) do for %%g in (%%f/*.el) do $(emacs) $(BYTE_COMPILE_EXTRA_FLAGS) -f batch-byte-compile %%f/%%g
+	for %%f in ($(COMPILE_FIRST)) do $(emacs) $(BYTE_COMPILE_FLAGS) -f batch-byte-compile %%f
+	for %%f in (. $(WINS)) do for %%g in (%%f/*.el) do $(emacs) $(BYTE_COMPILE_FLAGS) -f batch-byte-compile %%f/%%g
 
 compile-always-SH:
 #	for elc in $(lisp)/*.elc $(lisp)/*/*.elc; do attrib -r $$elc; done
 	for el in $(COMPILE_FIRST); do \
 	  echo Compiling $$el; \
-	  $(emacs) $(BYTE_COMPILE_EXTRA_FLAGS) -f batch-byte-compile $$el || exit 1; \
+	  $(emacs) $(BYTE_COMPILE_FLAGS) -f batch-byte-compile $$el || exit 1; \
 	done
 	for dir in $(lisp) $(WINS); do \
 	  for el in $$dir/*.el; do \
 	    echo Compiling $$el; \
-	    $(emacs) $(BYTE_COMPILE_EXTRA_FLAGS) -f batch-byte-compile $$el || exit 1; \
+	    $(emacs) $(BYTE_COMPILE_FLAGS) -f batch-byte-compile $$el || exit 1; \
 	  done; \
 	done
 
 compile-calc: compile-calc-$(SHELLTYPE)
 
 compile-calc-CMD:
-	for %%f in ($(lisp)/calc/*.el) do $(emacs) $(BYTE_COMPILE_EXTRA_FLAGS) -f batch-byte-compile %%f
+	for %%f in ($(lisp)/calc/*.el) do $(emacs) $(BYTE_COMPILE_FLAGS) -f batch-byte-compile %%f
 
 compile-calc-SH:
 	for el in $(lisp)/calc/*.el; do \
 	  echo Compiling $$el; \
-	  $(emacs) $(BYTE_COMPILE_EXTRA_FLAGS) -f batch-byte-compile $$el || exit 1; \
+	  $(emacs) $(BYTE_COMPILE_FLAGS) -f batch-byte-compile $$el || exit 1; \
 	done
 
 # Backup compiled Lisp files in elc.tar.gz.  If that file already

Re: Emacs 23.3 released

[Thierry Volpiatto]

Hi Stefan,

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> A big thanks to everybody for the work they put into this release.
>
> I hope we will be able to jump right to 24.1 without an intermediate
> 23.4, but if you find bugs that are not specific to trunk, please still
> commit them to emacs-23.
>
> Talking about 24.1, I now consider the lexbind-new branch as "fully
> functional": there are still a few changes I want to do there before
> merging, but they should only impact performance and code quality, so
> I encourage people to try it out and report any regression so they can
> be addressed before we merge it into trunk.

A quick try to lexbind give me this error on startup:

--8<---------------cut here---------------start------------->8---
Debugger entered--Lisp error: (wrong-type-argument listp 0)
  ad-parse-arglist(0)
  ad-map-arglists(0 0)
  ad-make-mapped-call(0 0 ad-Orig-dired-find-file-other-window)
  ad-make-advised-definition(dired-find-file-other-window)
  ad-activate-advised-definition(dired-find-file-other-window nil)
  ad-activate(dired-find-file-other-window nil)
  byte-code(
[...]

 [current-load-list elscreen-dired-version (lambda (defconst-tmp-var) (defconst elscreen-dired-version defconst-tmp-var)) "0.1.0 (November 6, 2005)" provide elscreen-dired require elscreen ad-add-advice dired-find-file-other-window (elscreen-dired-find-file-other-window nil t (advice lambda nil (let ((window-configuration (current-window-configuration)) (buffer nil)) ad-do-it (unless (eq major-mode 'dired-mode) (setq buffer (current-buffer)) (set-window-configuration window-configuration) (elscreen-find-and-goto-by-buffer buffer t))))) around nil ad-activate] 5)
  require(elscreen-dired)
  eval-buffer(#<buffer  *load*> nil "/home/thierry/.emacs.el" nil t)  ; Reading at buffer position 45697
  load-with-code-conversion("/home/thierry/.emacs.el" "/home/thierry/.emacs.el" t t)
  load("~/.emacs" t t)

[...]

 [init-file-user system-type user-init-file-1 user-init-file otherfile source ms-dos "~" "/_emacs" windows-nt directory-files nil "^\\.emacs\\(\\.elc?\\)?$" "~/.emacs" "^_emacs\\(\\.elc?\\)?$" "~/_emacs" "/.emacs" t load expand-file-name "init" file-name-as-directory "/.emacs.d" file-name-extension "elc" file-name-sans-extension ".el" file-exists-p file-newer-than-file-p message "Warning: %s is newer than %s" sit-for 1 "default" alt inhibit-default-init inhibit-startup-screen] 7]()
  command-line()
  normal-top-level()
--8<---------------cut here---------------end--------------->8---

No error with emacs -Q.

-- 
A+ Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997 

Re: Emacs 23.3 released

[Stefan Monnier]

> A quick try to lexbind give me this error on startup:

> --8<---------------cut here---------------start------------->8---
> Debugger entered--Lisp error: (wrong-type-argument listp 0)
>   ad-parse-arglist(0)

Does the patch below fix it for you?


        Stefan


=== modified file 'lisp/emacs-lisp/advice.el'
--- lisp/emacs-lisp/advice.el	2011-01-28 21:42:11 +0000
+++ lisp/emacs-lisp/advice.el	2011-03-11 04:16:59 +0000
@@ -2535,17 +2535,11 @@
   "Return the argument list of DEFINITION.
 If DEFINITION could be from a subr then its NAME should be
 supplied to make subr arglist lookup more efficient."
-  (cond ((ad-compiled-p definition)
-	 (aref (ad-compiled-code definition) 0))
-	((consp definition)
-	 (car (cdr (ad-lambda-expression definition))))
-	((ad-subr-p definition)
-	 (if name
-	     (ad-subr-arglist name)
-	   ;; otherwise get it from its printed representation:
-	   (setq name (format "%s" definition))
-	   (string-match "^#<subr \\([^>]+\\)>$" name)
-	   (ad-subr-arglist (intern (match-string 1 name)))))))
+  (require 'help-fns)
+  (cond
+   ((or (ad-macro-p definition) (ad-advice-p))
+    (help-function-arglist (cdr definition)))
+   (t (help-function-arglist definition))))
 
 ;; Store subr-args as `((arg1 arg2 ...))' so I can distinguish
 ;; a defined empty arglist `(nil)' from an undefined arglist:

=== modified file 'lisp/help-fns.el'
--- lisp/help-fns.el	2011-03-06 21:22:16 +0000
+++ lisp/help-fns.el	2011-03-11 04:15:05 +0000
@@ -124,6 +124,21 @@
       (nreverse arglist)))
    ((byte-code-function-p def) (aref def 0))
    ((eq (car-safe def) 'lambda) (nth 1 def))
+   ((subrp def)
+    (let ((arity (subr-arity def))
+          (arglist ()))
+      (dotimes (i (car arity))
+        (push (intern (concat "arg" (number-to-string (1+ i)))) arglist))
+      (if (not (numberp (cdr arglist)))
+          (progn
+            (push '&rest arglist)
+            (push 'rest arglist))
+        (push '&optional arglist)
+        (dotimes (i (- (cdr arity) (car arity)))
+          (push (intern (concat "arg" (number-to-string
+                                       (1+ i (car arity)))))
+                arglist)))
+      (nreverse arglist)))
    ((and (eq (car-safe def) 'autoload) (not (eq (nth 4 def) 'keymap)))
     "[Arg list not available until function definition is loaded.]")
    (t t)))

Re: Emacs 23.3 released

[Thierry Volpiatto]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Does the patch below fix it for you?

The patch apply fine but now i have an error when compiling:

--8<---------------cut here---------------start------------->8---
byte-compile-fix-header old-style-backquotes form] 4]()
  byte-compile-from-buffer(#<buffer  *Compiler Input*> "/home/thierry/tmp/emacs-lexbind/lisp/ses.el")
  byte-compile-file("ses.el")
  batch-byte-compile-file("ses.el")
  batch-byte-compile()
  command-line-1(("--eval" "(setq max-lisp-eval-depth 1200)" "-f" "batch-byte-compile" "ses.el"))
  command-line()
  normal-top-level()

make[2]: *** [ses.elc] Erreur 255
make[2]: quittant le répertoire « /home/thierry/tmp/emacs-lexbind/lisp »
make[1]: *** [compile-main] Erreur 2
make[1]: quittant le répertoire « /home/thierry/tmp/emacs-lexbind/lisp »
make: *** [lisp] Erreur 2
--8<---------------cut here---------------end--------------->8---

-- 
A+ Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997 

Re: Emacs 23.3 released

[Stefan Monnier]

>> Does the patch below fix it for you?
> The patch apply fine but now i have an error when compiling:

The backtrace you showed seemed incomplete.  But the patch I sent had
a gross error.  I've installed a better version on the lexbind branch,
so please try again.


        Stefan

Re: Emacs 23.3 released

[Juanma Barranquero]

I'm now bootstrapping lexbind-new straight from the repo, with just two changes:

- Using loaddefs.el from a previous bootstrap as ldefs-boot.el.
- Reverting the change to lisp/emacs-lisp/debug.el so backtraces are manageable.

On make'ing the target compile-CMD, byecompiling the COMPILE_FIRST
files throws errors for pcase.el, macroexp.el and cconv.el, like
these:

  Debugger entered--Lisp error: (file-error "Opening input file" "no
such file or directory"
"C:/emacs/lexbind-new/lisp/emacs-lisp/pcase.elc")
    insert-file-contents("C:/emacs/lexbind-new/lisp/emacs-lisp/pcase.elc")
    byte-compile-file("C:/emacs/lexbind-new/lisp/emacs-lisp/pcase.elc")
    batch-byte-compile-file("C:/emacs/lexbind-new/lisp/emacs-lisp/pcase.elc")
    batch-byte-compile(t)
    batch-byte-compile-if-not-done()
    funcall(batch-byte-compile-if-not-done)
    [...etc...]

Bootstrapping then continues normally and produces a working
executable (pcase.el, macroexp.el and cconv.el get correctly compiled
along with everything else).

    Juanma

Re: Emacs 23.3 released

[Stefan Monnier]

> I'm now bootstrapping lexbind-new straight from the repo, with just
> two changes:

Great.  So my changes to makefile.w32-in work?  The mind boggles!

> - Using loaddefs.el from a previous bootstrap as ldefs-boot.el.

Hmm... I'm really curious why that's needed.  I rarely bootstrap my
usual Emacs, but I bootstrap the lexbind branch pretty often (using
some underused old 16-core Athlon X2 machine, it's barely 3mins to
bootstrap, less than a minute and a half to recompile all the files) for
benchmarking purposes.  So under GNU/Linux it bootstraps fine, and the
loaddefs.el should not be affected much by platform differences, AFAICT.

I just noticed that, contrary to my belief, the lexbind branch does have
changes in ldefs-boot (so I'll get conflicts at the next merge that
changes ldefs-boot).  Could you try

   bzr revert -r ancestor:../trunk lisp/ldefs-boot.el

or copy the trunk's ldefs-boot.el to see if that fixes the problem?
(both the trunk's and the current lexbind version work fine for me under
GNU/Linux).

> On make'ing the target compile-CMD, bytecompiling the COMPILE_FIRST
> files throws errors for pcase.el, macroexp.el and cconv.el, like
> these:

I guess the patch below will fix those problem.


        Stefan


=== modified file 'lisp/makefile.w32-in'
--- lisp/makefile.w32-in	2011-03-12 03:32:43 +0000
+++ lisp/makefile.w32-in	2011-03-14 13:49:12 +0000
@@ -84,9 +84,9 @@
 COMPILE_FIRST = \
 	$(lisp)/emacs-lisp/byte-opt.el \
 	$(lisp)/emacs-lisp/bytecomp.el \
-	$(lisp)/emacs-lisp/pcase.elc \
-	$(lisp)/emacs-lisp/macroexp.elc \
-	$(lisp)/emacs-lisp/cconv.elc \
+	$(lisp)/emacs-lisp/pcase.el \
+	$(lisp)/emacs-lisp/macroexp.el \
+	$(lisp)/emacs-lisp/cconv.el \
 	$(lisp)/subr.el \
 	$(lisp)/progmodes/cc-mode.el \
 	$(lisp)/progmodes/cc-vars.el

Re: Emacs 23.3 released

[Juanma Barranquero]

On Mon, Mar 14, 2011 at 15:10, Stefan Monnier <monnier@iro.umontreal.ca> wrote:

>   bzr revert -r ancestor:../trunk lisp/ldefs-boot.el
[...]
> I guess the patch below will fix those problem.

Yeah, lexbind-new bootstraps now fine on Windows.

    Juanma

RE: Lexbind (was: Emacs 23.3 released)

[Stefan Monnier]

>>   bzr revert -r ancestor:../trunk lisp/ldefs-boot.el
>> I guess the patch below will fix those problem.
> Yeah, lexbind-new bootstraps now fine on Windows.

Great, thanks.  Now on to actual run-time errors.


        Stefan

Re: Lexbind (was: Emacs 23.3 released)

[Juanma Barranquero]

> Now on to actual run-time errors.

I've had several hangups while reading from the minibuffer. No CPU
usage, but no response whatsoever so I had to kill Emacs.

I just had one while running under GDB, so I Ctrl-C and I got the
attached backtrace.

Note that I'm not running emacs -Q, but with my normal setup
(icomplete, ido, etc.)

    Juanma



(gdb) bt
#0  0x0112c987 in next_element_from_buffer (it=0x88c250) at xdisp.c:6744
#1  0x0112171f in get_next_display_element (it=0x88c250) at xdisp.c:5630
#2  0x0112292b in move_it_in_display_line_to (it=<value optimized
out>, to_charpos=-1, to_x=-1, op=0) at xdisp.c:6956
#3  0x0112620f in move_it_to (it=0x88c250, to_charpos=-1, to_x=-1,
to_y=-1, to_vpos=1, op=4) at xdisp.c:7333
#4  0x0112e87f in move_it_by_lines (it=0x88c250, dvpos=1, need_y_p=1)
at xdisp.c:7757
#5  0x01136a6e in try_scrolling (window=<value optimized out>,
just_this_one_p=0, arg_scroll_conservatively=143165576, scroll_step=0,
    temp_scroll_step=0, last_line_misfit=0) at xdisp.c:13128
#6  0x0114e0cd in redisplay_window (window=<value optimized out>,
just_this_one_p=0) at xdisp.c:14168
#7  0x011502b3 in redisplay_window_0 (window=56712709) at xdisp.c:12362
#8  0x01012961 in internal_condition_case_1 (bfun=0x1150290
<redisplay_window_0>, arg=56712709, handlers=53689030,
    hfun=0x1109770 <redisplay_window_error>) at eval.c:1536
#9  0x0110f137 in redisplay_windows (window=<value optimized out>) at
xdisp.c:12342
#10 0x011310f6 in redisplay_internal (preserve_echo_area=<value
optimized out>) at xdisp.c:11919
#11 0x01132a65 in redisplay_preserve_echo_area (from_where=8) at xdisp.c:12174
#12 0x0103384a in detect_input_pending_run_timers (do_display=1) at
keyboard.c:10365
#13 0x0101e7e1 in wait_reading_process_output (time_limit=450,
microsecs=0, read_kbd=-1, do_display=1, wait_for_cell=53704730,
wait_proc=0x0,
    just_wait_proc=0) at process.c:4856
#14 0x01081f63 in sit_for (timeout=1800, reading=1, do_display=1) at
dispnew.c:6017
#15 0x01036e95 in read_char (commandflag=1, nmaps=4, maps=0x88f190,
prev_event=53704730, used_mouse_menu=0x88f288, end_time=0x0) at
keyboard.c:2601
#16 0x0103853e in read_key_sequence (keybuf=0x88f2f8, prompt=53704730,
dont_downcase_last=0, can_return_switch_frame=1, fix_current_buffer=1,
    bufsize=30) at keyboard.c:9199
#17 0x0103b07a in command_loop_1 () at keyboard.c:1409
#18 0x0101288b in internal_condition_case (bfun=0x103ae30
<command_loop_1>, handlers=53758362, hfun=0x102ae50 <cmd_error>) at
eval.c:1491
#19 0x0102a51d in command_loop_2 (ignore=53704730) at keyboard.c:1129
#20 0x010127c9 in internal_catch (tag=53860642, func=0x102a4f0
<command_loop_2>, arg=53704730) at eval.c:1235
#21 0x0102afd7 in command_loop () at keyboard.c:1094
#22 0x0102b0cb in recursive_edit_1 () at keyboard.c:731
#23 0x010accc3 in read_minibuf (map=53693814, initial=53704730,
prompt=<value optimized out>, backup_n=<value optimized out>,
expflag=0,
    histvar=58957570, histpos=0, defalt=82462646, allow_props=0,
inherit_input_method=1) at minibuf.c:661
#24 0x010148d7 in Ffuncall (nargs=8, args=0x88f630) at eval.c:3012
#25 0x010c2e07 in exec_byte_code (bytestr=<value optimized out>,
vector=20030189, maxdepth=36, args_template=53704730, nargs=0,
    args=<value optimized out>) at bytecode.c:766
#26 0x01014231 in funcall_lambda (fun=20030013, nargs=2,
arg_vector=0x88f7e4) at eval.c:3218
#27 0x010146d0 in Ffuncall (nargs=3, args=0x88f7e0) at eval.c:3035
#28 0x010c2e07 in exec_byte_code (bytestr=<value optimized out>,
vector=20035461, maxdepth=28, args_template=53704730, nargs=0,
    args=<value optimized out>) at bytecode.c:766
#29 0x01014231 in funcall_lambda (fun=20035341, nargs=0,
arg_vector=0x88f900) at eval.c:3218
#30 0x010131ee in apply_lambda (fun=20035341, args=<value optimized
out>) at eval.c:3096
#31 0x01013507 in eval_sub (form=20035950) at eval.c:2406
#32 0x01017312 in Feval (form=20035950, lexical=53704730) at eval.c:2212
#33 0x010c3da7 in Fcall_interactively (function=54514858,
record_flag=53704730, keys=53725957) at callint.c:343
#34 0x010149d9 in Ffuncall (nargs=4, args=0x88fbb0) at eval.c:2993
#35 0x01014dc1 in call3 (fn=53872642, arg1=54514858, arg2=53704730,
arg3=53704730) at eval.c:2785
#36 0x0103b387 in command_loop_1 () at keyboard.c:1528
#37 0x0101288b in internal_condition_case (bfun=0x103ae30
<command_loop_1>, handlers=53758362, hfun=0x102ae50 <cmd_error>) at
eval.c:1491
#38 0x0102a51d in command_loop_2 (ignore=53704730) at keyboard.c:1129
#39 0x010127c9 in internal_catch (tag=53756458, func=0x102a4f0
<command_loop_2>, arg=53704730) at eval.c:1235
#40 0x0102b021 in command_loop () at keyboard.c:1108
#41 0x0102b0cb in recursive_edit_1 () at keyboard.c:731
#42 0x0102b265 in Frecursive_edit () at keyboard.c:793
#43 0x01002e5f in main (argc=<value optimized out>, argv=0xa22d08) at
emacs.c:1684

Lisp Backtrace:
"read-from-minibuffer" (0x88f634)
"read-regexp" (0x88f7e4)
"occur-read-primary-args" (0x88f900)
"call-interactively" (0x88fbb4)
(gdb)

Re: Lexbind (was: Emacs 23.3 released)

[Eli Zaretskii]

> From: Juanma Barranquero <lekktu@gmail.com>
> Date: Thu, 17 Mar 2011 02:59:38 +0100
> Cc: emacs-devel@gnu.org
> 
> I've had several hangups while reading from the minibuffer. No CPU
> usage, but no response whatsoever so I had to kill Emacs.
> 
> I just had one while running under GDB, so I Ctrl-C and I got the
> attached backtrace.
> [...]
> (gdb) bt
> #0  0x0112c987 in next_element_from_buffer (it=0x88c250) at xdisp.c:6744

Is this with today's sources from the lexbind-new branch?  Because
line 6744 in xdisp.c is the closing brace of next_element_from_buffer
with today's bzr repo.  How can Emacs get stuck in such a line?

However, since this seems to be an optimized build, the backtrace is
not guaranteed to make sense.  I suggest to recompile without
optimizations and post a backtrace if and when that hangs.

> #1  0x0112171f in get_next_display_element (it=0x88c250) at xdisp.c:5630
> #2  0x0112292b in move_it_in_display_line_to (it=<value optimized
> out>, to_charpos=-1, to_x=-1, op=0) at xdisp.c:6956
> #3  0x0112620f in move_it_to (it=0x88c250, to_charpos=-1, to_x=-1,
> to_y=-1, to_vpos=1, op=4) at xdisp.c:7333
> #4  0x0112e87f in move_it_by_lines (it=0x88c250, dvpos=1, need_y_p=1)
> at xdisp.c:7757
> #5  0x01136a6e in try_scrolling (window=<value optimized out>,
> just_this_one_p=0, arg_scroll_conservatively=143165576, scroll_step=0,
>     temp_scroll_step=0, last_line_misfit=0) at xdisp.c:13128
> #6  0x0114e0cd in redisplay_window (window=<value optimized out>,
> just_this_one_p=0) at xdisp.c:14168
> #7  0x011502b3 in redisplay_window_0 (window=56712709) at xdisp.c:12362
> #8  0x01012961 in internal_condition_case_1 (bfun=0x1150290
> <redisplay_window_0>, arg=56712709, handlers=53689030,
>     hfun=0x1109770 <redisplay_window_error>) at eval.c:1536
> #9  0x0110f137 in redisplay_windows (window=<value optimized out>) at
> xdisp.c:12342

This all looks like a normal redisplay cycle to me, and I don't see
how it could possibly hang, especially without using CPU.  If you can
reproduce this in a non-optimized build, try stepping after
interrupting Emacs, and see where it loops or in what function call it
gets stuck.

Also, perhaps try bt in the other threads, maybe they are stuck, not
the Lisp thread.

Re: Lexbind (was: Emacs 23.3 released)

[Juanma Barranquero]

On Thu, Mar 17, 2011 at 11:28, Eli Zaretskii <eliz@gnu.org> wrote:

> Is this with today's sources from the lexbind-new branch?

The ones from yesterday, though today's changes haven't touched
xdisp.c so far as I can see.

> Because
> line 6744 in xdisp.c is the closing brace of next_element_from_buffer
> with today's bzr repo.  How can Emacs get stuck in such a line?

No idea.

> However, since this seems to be an optimized build, the backtrace is
> not guaranteed to make sense.  I suggest to recompile without
> optimizations and post a backtrace if and when that hangs.

It wasn't optimized, theoretically speaking, but yesterday I spent
some time reworking my build scripts and it's not entirely unlikely
that I screwed up something. I'll check.

> If you can
> reproduce this in a non-optimized build, try stepping after
> interrupting Emacs, and see where it loops or in what function call it
> gets stuck.
>
> Also, perhaps try bt in the other threads, maybe they are stuck, not
> the Lisp thread.

Will do.

Thanks,

    Juanma

Re: Lexbind (was: Emacs 23.3 released)

[Juanma Barranquero]

OK, please forget my crappy previous report.

I just had it hang up on electric-describe-variable. It *does* use CPU
(around 30% on my Core2 Quad G9300).

The trace is after C-c.

I have the debugger session open, so if you want to examine some
value, just ask.


[Switching to Thread 3584.0x1008]
0x76ea01c4 in ntdll!LdrFindResource_U () from C:\Windows\system32\ntdll.dll
Quit (expect signal SIGINT when the program is resumed)
(gdb) bt
#0  0x76ea01c4 in ntdll!LdrFindResource_U () from C:\Windows\system32\ntdll.dll

Lisp Backtrace:
"completing-read" (0x88ecf0)
"setq" (0x88eee0)
"let" (0x88f0d0)
"call-interactively" (0x88f394)
"electric-helpify" (0x88f614)
"electric-describe-variable" (0x88f8c4)
"call-interactively" (0x88fb34)
(gdb) thread 1
[Switching to thread 1 (Thread 3584.0x10b0)]#0  0x7568a048 in
msvcrt!memmove () from C:\Windows\syswow64\msvcrt.dll
(gdb) bt
#0  0x7568a048 in msvcrt!memmove () from C:\Windows\syswow64\msvcrt.dll
#1  0x7568a00b in msvcrt!memmove () from C:\Windows\syswow64\msvcrt.dll
#2  0x011a60db in try_scrolling (window=55221765, just_this_one_p=0,
arg_scroll_conservatively=143165576, scroll_step=0,
temp_scroll_step=0,
    last_line_misfit=0) at xdisp.c:13129
#3  0x011aac08 in redisplay_window (window=55221765,
just_this_one_p=0) at xdisp.c:14168
#4  0x011a3203 in redisplay_window_0 (window=55221765) at xdisp.c:12362
#5  0x01037fe6 in internal_condition_case_1 (bfun=0x11a31d1
<redisplay_window_0>, arg=55221765, handlers=52234950,
    hfun=0x11a31b0 <redisplay_window_error>) at eval.c:1536
#6  0x011a31a0 in redisplay_windows (window=55221765) at xdisp.c:12342
#7  0x011a0730 in redisplay_internal (preserve_echo_area=1) at xdisp.c:11919
#8  0x011a234c in redisplay_preserve_echo_area (from_where=8) at xdisp.c:12174
#9  0x01023627 in detect_input_pending_run_timers (do_display=1) at
keyboard.c:10365
#10 0x010547ac in wait_reading_process_output (time_limit=750,
microsecs=0, read_kbd=-1, do_display=1, wait_for_cell=52250650,
wait_proc=0x0,
    just_wait_proc=0) at process.c:4856
#11 0x010a696a in sit_for (timeout=3000, reading=1, do_display=1) at
dispnew.c:6017
#12 0x01009665 in read_char (commandflag=1, nmaps=4, maps=0x88e620,
prev_event=52250650, used_mouse_menu=0x88e7bc, end_time=0x0) at
keyboard.c:2601
#13 0x0101dbfd in read_key_sequence (keybuf=0x88e8b0, bufsize=30,
prompt=52250650, dont_downcase_last=0, can_return_switch_frame=1,
    fix_current_buffer=1) at keyboard.c:9199
#14 0x01005ffb in command_loop_1 () at keyboard.c:1409
#15 0x01037ed6 in internal_condition_case (bfun=0x1005755
<command_loop_1>, handlers=52304282, hfun=0x1004e81 <cmd_error>) at
eval.c:1491
#16 0x01005357 in command_loop_2 (ignore=52250650) at keyboard.c:1129
#17 0x010378f5 in internal_catch (tag=52406562, func=0x1005334
<command_loop_2>, arg=52250650) at eval.c:1235
#18 0x010052be in command_loop () at keyboard.c:1094
#19 0x01004573 in recursive_edit_1 () at keyboard.c:731
#20 0x011210c4 in read_minibuf (map=52239766, initial=52250650,
prompt=83802241, backup_n=0, expflag=0, histvar=52436258, histpos=0,
    defalt=52250650, allow_props=0, inherit_input_method=0) at minibuf.c:661
#21 0x011247b7 in Fcompleting_read (prompt=83802241,
collection=52252677, predicate=53020518, require_match=52250674,
initial_input=52250650,
    hist=52250650, def=52250650, inherit_input_method=52250650) at
minibuf.c:1729
#22 0x0103a9b7 in eval_sub (form=53019750) at eval.c:2391
#23 0x01035459 in Fsetq (args=53019718) at eval.c:442
#24 0x01039f39 in eval_sub (form=53019710) at eval.c:2308
#25 0x01035211 in Fprogn (args=53022686) at eval.c:345
#26 0x01037427 in Flet (args=53019702) at eval.c:1083
#27 0x01039f39 in eval_sub (form=53050870) at eval.c:2308
#28 0x01039951 in Feval (form=53050870, lexical=52250650) at eval.c:2212
#29 0x0113ed56 in Fcall_interactively (function=53457858,
record_flag=52250650, keys=52271877) at callint.c:343
#30 0x0103c954 in Ffuncall (nargs=2, args=0x88f390) at eval.c:2993
#31 0x01141d56 in exec_byte_code (bytestr=83801473, vector=83604869,
maxdepth=20, args_template=52250650, nargs=0, args=0x0) at
bytecode.c:766
#32 0x0103d91b in funcall_lambda (fun=55663685, nargs=1,
arg_vector=0x88f614) at eval.c:3218
#33 0x0103cd9b in Ffuncall (nargs=2, args=0x88f610) at eval.c:3035
#34 0x01141d56 in exec_byte_code (bytestr=52378161, vector=54339765,
maxdepth=8, args_template=52250650, nargs=0, args=0x0) at
bytecode.c:766
#35 0x0103d91b in funcall_lambda (fun=55662373, nargs=0,
arg_vector=0x88f8c4) at eval.c:3218
#36 0x0103cd9b in Ffuncall (nargs=1, args=0x88f8c0) at eval.c:3035
#37 0x0103b974 in apply1 (fn=83991578, arg=52250650) at eval.c:2721
#38 0x0113eecc in Fcall_interactively (function=83991578,
record_flag=52250650, keys=52271877) at callint.c:376
#39 0x0103c954 in Ffuncall (nargs=4, args=0x88fb30) at eval.c:2993
#40 0x0103ba80 in call3 (fn=52418562, arg1=83991578, arg2=52250650,
arg3=52250650) at eval.c:2785
#41 0x01022d8d in Fcommand_execute (cmd=83991578,
record_flag=52250650, keys=52250650, special=52250650) at
keyboard.c:10186
#42 0x01006d3a in command_loop_1 () at keyboard.c:1528
#43 0x01037ed6 in internal_condition_case (bfun=0x1005755
<command_loop_1>, handlers=52304282, hfun=0x1004e81 <cmd_error>) at
eval.c:1491
#44 0x01005357 in command_loop_2 (ignore=52250650) at keyboard.c:1129
#45 0x010378f5 in internal_catch (tag=52302378, func=0x1005334
<command_loop_2>, arg=52250650) at eval.c:1235
#46 0x0100530f in command_loop () at keyboard.c:1108
#47 0x01004573 in recursive_edit_1 () at keyboard.c:731
#48 0x01004a97 in Frecursive_edit () at keyboard.c:793
#49 0x01002797 in main (argc=2, argv=0xe82d00) at emacs.c:1684

Lisp Backtrace:
"completing-read" (0x88ecf0)
"setq" (0x88eee0)
"let" (0x88f0d0)
"call-interactively" (0x88f394)
"electric-helpify" (0x88f614)
"electric-describe-variable" (0x88f8c4)
"call-interactively" (0x88fb34)
(gdb) thread 2
[Switching to thread 2 (Thread 3584.0x12d8)]#0  0x76eb014d in
ntdll!RtlEnableEarlyCriticalSectionEventCreation () from
C:\Windows\system32\ntdll.dll
(gdb) bt
#0  0x76eb014d in ntdll!RtlEnableEarlyCriticalSectionEventCreation ()
from C:\Windows\system32\ntdll.dll
#1  0x76eb014d in ntdll!RtlEnableEarlyCriticalSectionEventCreation ()
from C:\Windows\system32\ntdll.dll
#2  0x76ed431f in ntdll!TpCallbackMayRunLong () from
C:\Windows\system32\ntdll.dll
#3  0x00000003 in ?? ()
#4  0x00bdfdd8 in ?? ()
#5  0x75c233ca in KERNEL32!BaseCleanupAppcompatCacheSupport () from
C:\Windows\syswow64\kernel32.dll
#6  0x00000000 in ?? ()

Lisp Backtrace:
"completing-read" (0x88ecf0)
"setq" (0x88eee0)
"let" (0x88f0d0)
"call-interactively" (0x88f394)
"electric-helpify" (0x88f614)
"electric-describe-variable" (0x88f8c4)
"call-interactively" (0x88fb34)
(gdb) thread 3
[Switching to thread 3 (Thread 3584.0x136c)]#0  0x74a678d7 in
USER32!IsDialogMessage () from C:\Windows\syswow64\user32.dll
(gdb) bt
#0  0x74a678d7 in USER32!IsDialogMessage () from C:\Windows\syswow64\user32.dll
#1  0x74a678d7 in USER32!IsDialogMessage () from C:\Windows\syswow64\user32.dll
#2  0x74a67c1d in USER32!GetClientRect () from C:\Windows\syswow64\user32.dll
#3  0x2167fed0 in ?? ()
#4  0x012647ea in w32_msg_pump (msg_buf=0x2167ff20) at w32fns.c:2254
#5  0x01264d4a in w32_msg_worker (arg=0x0) at w32fns.c:2473
#6  0x75c233ca in KERNEL32!BaseCleanupAppcompatCacheSupport () from
C:\Windows\syswow64\kernel32.dll
#7  0x00000000 in ?? ()

Lisp Backtrace:
"completing-read" (0x88ecf0)
"setq" (0x88eee0)
"let" (0x88f0d0)
"call-interactively" (0x88f394)
"electric-helpify" (0x88f614)
"electric-describe-variable" (0x88f8c4)
"call-interactively" (0x88fb34)
gdb) thread 1
[Switching to thread 1 (Thread 3584.0x10b0)]#0  0x7568a048 in
msvcrt!memmove () from C:\Windows\syswow64\msvcrt.dll
(gdb) next
Single stepping until exit from function msvcrt!memmove,
which has no line number information.
try_scrolling (window=55221765, just_this_one_p=0,
arg_scroll_conservatively=143165576, scroll_step=0,
temp_scroll_step=0, last_line_misfit=0)
    at xdisp.c:13130
13130             } while (line_bottom_y (&it1) - start_y < amount_to_scroll);

Re: Lexbind (was: Emacs 23.3 released)

[Eli Zaretskii]

> From: Juanma Barranquero <lekktu@gmail.com>
> Date: Thu, 17 Mar 2011 16:13:44 +0100
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> I just had it hang up on electric-describe-variable. It *does* use CPU
> (around 30% on my Core2 Quad G9300).

That could indicate that it is looping in redisplay.

> (gdb) next
> Single stepping until exit from function msvcrt!memmove,
> which has no line number information.
> try_scrolling (window=55221765, just_this_one_p=0,
> arg_scroll_conservatively=143165576, scroll_step=0,
> temp_scroll_step=0, last_line_misfit=0)
>     at xdisp.c:13130
> 13130             } while (line_bottom_y (&it1) - start_y < amount_to_scroll);

If you continue stepping, do you ever get out of redisplay?  That is,
does redisplay_internal return?  You can find this out by repeatedly
typing "finish", until it does not return to a higher frame.

If it never gets back to wait_reading_process_output (frame #10 in
your backtrace), that would explain why Emacs is not responding to
your input.  The next step then would be to understand why it gets
stuck in redisplay.

Re: Lexbind (was: Emacs 23.3 released)

[Juanma Barranquero]

On Thu, Mar 17, 2011 at 17:07, Eli Zaretskii <eliz@gnu.org> wrote:

> That could indicate that it is looping in redisplay.

I suppose you've not forgotten, but this is the lexical-new branch. I
have no trouble on trunk. Also, I'm using my setup, which includes

  (setq-default bidi-display-reordering (boundp 'bidi-display-reordering))

After I type finish once, I get

(gdb) finish
Run till exit from #0  try_scrolling (window=55221765,
just_this_one_p=0, arg_scroll_conservatively=143165576, scroll_step=0,
temp_scroll_step=0,
    last_line_misfit=0) at xdisp.c:13128

and it's stuck again with ~28% CPU. A further C-c (or four, to be
precise) bring this:

[New Thread 3584.0xdcc]
[New Thread 3584.0x850]
[New Thread 3584.0x1238]
[New Thread 3584.0x11fc]
[Switching to Thread 3584.0x11fc]
0x76ea01c4 in ntdll!LdrFindResource_U () from C:\Windows\system32\ntdll.dll
Quit (expect signal SIGINT when the program is resumed)
(gdb)
"finish" not meaningful in the outermost frame.
(gdb) bt
#0  0x76ea01c4 in ntdll!LdrFindResource_U () from C:\Windows\system32\ntdll.dll

Lisp Backtrace:
"completing-read" (0x88ecf0)
"setq" (0x88eee0)
"let" (0x88f0d0)
"call-interactively" (0x88f394)
"electric-helpify" (0x88f614)
"electric-describe-variable" (0x88f8c4)
"call-interactively" (0x88fb34)
(gdb)

Re: Lexbind (was: Emacs 23.3 released)

[Eli Zaretskii]

> From: Juanma Barranquero <lekktu@gmail.com>
> Date: Thu, 17 Mar 2011 19:06:48 +0100
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> On Thu, Mar 17, 2011 at 17:07, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > That could indicate that it is looping in redisplay.
> 
> I suppose you've not forgotten, but this is the lexical-new branch. I
> have no trouble on trunk.

Yes, but the lexical-new branch still has redisplay, doesn't it? ;-)

> Also, I'm using my setup, which includes
> 
>   (setq-default bidi-display-reordering (boundp 'bidi-display-reordering))

It would make sense to turn off bidi reordering and see if the hangs
still happen.

> After I type finish once, I get
> 
> (gdb) finish
> Run till exit from #0  try_scrolling (window=55221765,
> just_this_one_p=0, arg_scroll_conservatively=143165576, scroll_step=0,
> temp_scroll_step=0,
>     last_line_misfit=0) at xdisp.c:13128
> 
> and it's stuck again with ~28% CPU.

Which means it loops inside try_scrolling.  Next step is to see where
it loops there, and why it doesn't return.

Re: Lexbind (was: Emacs 23.3 released)

[Juanma Barranquero]

On Thu, Mar 17, 2011 at 20:24, Eli Zaretskii <eliz@gnu.org> wrote:

> Yes, but the lexical-new branch still has redisplay, doesn't it? ;-)

Just wanted to be sure you knew and I wasn't sending you in a wild goose chase.

> It would make sense to turn off bidi reordering and see if the hangs
> still happen.

OK, I'll do.

> Which means it loops inside try_scrolling.  Next step is to see where
> it loops there, and why it doesn't return.

Now that you say try_scrolling, perhaps is a good time to mention that
I'm a line-by-line-scrolling zealot, with things like this in my
.emacs:

(setq scroll-preserve-screen-position :always
      scroll-conservatively           most-positive-fixnum
      scroll-step                     0)

    Juanma

Re: Lexbind (was: Emacs 23.3 released)

[Juanma Barranquero]

On Thu, Mar 17, 2011 at 20:24, Eli Zaretskii <eliz@gnu.org> wrote:

> It would make sense to turn off bidi reordering and see if the hangs
> still happen.

Yes, it hangs with bidi turned off.

> Which means it loops inside try_scrolling.  Next step is to see where
> it loops there, and why it doesn't return.

It's looping here (in the else part, obviously):

      start_display (&it, w, startp);
      if (scroll_max < INT_MAX)
	move_it_vertically (&it, amount_to_scroll);
      else
	{
	  /* Extra precision for users who set scroll-conservatively
	     to most-positive-fixnum: make sure the amount we scroll
	     the window start is never less than amount_to_scroll,
	     which was computed as distance from window bottom to
	     point.  This matters when lines at window top and lines
	     below window bottom have different height.  */
	  struct it it1 = it;
	  /* We use a temporary it1 because line_bottom_y can modify
	     its argument, if it moves one line down; see there.  */
	  int start_y = line_bottom_y (&it1);

	  do {
	    move_it_by_lines (&it, 1, 1);
	    it1 = it;
	  } while (line_bottom_y (&it1) - start_y < amount_to_scroll);
	}

In my current session, line_bottom_y (&it1) == 15, start_y == 15,
amount_to_scroll == 15, forever and ever and ever.

    Juanma



(gdb) p it
$2 = {
  window = 55221765,
  w = 0x34a9e00,
  f = 0x343c800,
  method = GET_FROM_BUFFER,
  stop_charpos = 53,
  prev_stop = 0,
  base_level_stop = 0,
  end_charpos = 53,
  s = 0x0,
  string_nchars = 0,
  region_beg_charpos = -1,
  region_end_charpos = -1,
  redisplay_end_trigger_charpos = 0,
  multibyte_p = 1,
  header_line_p = 0,
  string_from_display_prop_p = 0,
  ellipsis_p = 0,
  avoid_cursor_p = 0,
  dp = 0x343cc00,
  dpvec = 0x0,
  dpend = 0x0,
  dpvec_char_len = 0,
  dpvec_face_id = 0,
  saved_face_id = 12,
  ctl_chars = {0 <repeats 16 times>},
  start = {
    pos = {
      charpos = 1,
      bytepos = 1
    },
    overlay_string_index = -1,
    string_pos = {
      charpos = -1,
      bytepos = -1
    },
    dpvec_index = -1
  },
  current = {
    pos = {
      charpos = 53,
      bytepos = 53
    },
    overlay_string_index = -1,
    string_pos = {
      charpos = -1,
      bytepos = -1
    },
    dpvec_index = -1
  },
  n_overlay_strings = 0,
  overlay_strings_charpos = 53,
  overlay_strings = {84671601, 0 <repeats 15 times>},
  string_overlays = {81053227, 0 <repeats 15 times>},
  string = 52250650,
  from_overlay = 0,
  stack = {{
      string = 52250650,
      string_nchars = 0,
      end_charpos = 53,
      stop_charpos = 2,
      prev_stop = 0,
      base_level_stop = 0,
      cmp_it = {
        stop_pos = 50,
        id = -1,
        ch = -2,
        rule_idx = 0,
        lookback = 0,
        nglyphs = 0,
        reversed_p = 0,
        charpos = 0,
        nchars = 0,
        nbytes = 0,
        from = 0,
        to = 0,
        width = 0
      },
      face_id = 12,
      u = {
        image = {
          object = 0,
          slice = {
            x = 0,
            y = 0,
            width = 0,
            height = 0
          },
          image_id = 0
        },
        comp = {
          object = 0
        },
        stretch = {
          object = 0
        }
      },
      position = {
        charpos = 1,
        bytepos = 1
      },
      current = {
        pos = {
          charpos = 1,
          bytepos = 1
        },
        overlay_string_index = 0,
        string_pos = {
          charpos = -1,
          bytepos = -1
        },
        dpvec_index = -1
      },
      from_overlay = 0,
      area = TEXT_AREA,
      method = GET_FROM_BUFFER,
      multibyte_p = 1,
      string_from_display_prop_p = 0,
      display_ellipsis_p = 0,
      avoid_cursor_p = 0,
      line_wrap = WINDOW_WRAP,
      voffset = 0,
      space_width = 52250650,
      font_height = 52250650
    }, {
      string = 0,
      string_nchars = 0,
      end_charpos = 0,
      stop_charpos = 0,
      prev_stop = 0,
      base_level_stop = 0,
      cmp_it = {
        stop_pos = 0,
        id = 0,
        ch = 0,
        rule_idx = 0,
        lookback = 0,
        nglyphs = 0,
        reversed_p = 0,
        charpos = 0,
        nchars = 0,
        nbytes = 0,
        from = 0,
        to = 0,
        width = 0
      },
      face_id = 0,
      u = {
        image = {
          object = 0,
          slice = {
            x = 0,
            y = 0,
            width = 0,
            height = 0
          },
          image_id = 0
        },
        comp = {
          object = 0
        },
        stretch = {
          object = 0
        }
      },
      position = {
        charpos = 0,
        bytepos = 0
      },
      current = {
        pos = {
          charpos = 0,
          bytepos = 0
        },
        overlay_string_index = 0,
        string_pos = {
          charpos = 0,
          bytepos = 0
        },
        dpvec_index = 0
      },
      from_overlay = 0,
      area = LEFT_MARGIN_AREA,
      method = GET_FROM_BUFFER,
      multibyte_p = 0,
      string_from_display_prop_p = 0,
      display_ellipsis_p = 0,
      avoid_cursor_p = 0,
      line_wrap = TRUNCATE,
      voffset = 0,
      space_width = 0,
      font_height = 0
    }, {
      string = 0,
      string_nchars = 0,
      end_charpos = 0,
      stop_charpos = 0,
      prev_stop = 0,
      base_level_stop = 0,
      cmp_it = {
        stop_pos = 0,
        id = 0,
        ch = 0,
        rule_idx = 0,
        lookback = 0,
        nglyphs = 0,
        reversed_p = 0,
        charpos = 0,
        nchars = 0,
        nbytes = 0,
        from = 0,
        to = 0,
        width = 0
      },
      face_id = 0,
      u = {
        image = {
          object = 0,
          slice = {
            x = 0,
            y = 0,
            width = 0,
            height = 0
          },
          image_id = 0
        },
        comp = {
          object = 0
        },
        stretch = {
          object = 0
        }
      },
      position = {
        charpos = 0,
        bytepos = 0
      },
      current = {
        pos = {
          charpos = 0,
          bytepos = 0
        },
        overlay_string_index = 0,
        string_pos = {
          charpos = 0,
          bytepos = 0
        },
        dpvec_index = 0
      },
      from_overlay = 0,
      area = LEFT_MARGIN_AREA,
      method = GET_FROM_BUFFER,
      multibyte_p = 0,
      string_from_display_prop_p = 0,
      display_ellipsis_p = 0,
      avoid_cursor_p = 0,
      line_wrap = TRUNCATE,
      voffset = 0,
      space_width = 0,
      font_height = 0
    }, {
      string = 0,
      string_nchars = 0,
      end_charpos = 0,
      stop_charpos = 0,
      prev_stop = 0,
      base_level_stop = 0,
      cmp_it = {
        stop_pos = 0,
        id = 0,
        ch = 0,
        rule_idx = 0,
        lookback = 0,
        nglyphs = 0,
        reversed_p = 0,
        charpos = 0,
        nchars = 0,
        nbytes = 0,
        from = 0,
        to = 0,
        width = 0
      },
      face_id = 0,
      u = {
        image = {
          object = 0,
          slice = {
            x = 0,
            y = 0,
            width = 0,
            height = 0
          },
          image_id = 0
        },
        comp = {
          object = 0
        },
        stretch = {
          object = 0
        }
      },
      position = {
        charpos = 0,
        bytepos = 0
      },
      current = {
        pos = {
          charpos = 0,
          bytepos = 0
        },
        overlay_string_index = 0,
        string_pos = {
          charpos = 0,
          bytepos = 0
        },
        dpvec_index = 0
      },
      from_overlay = 0,
      area = LEFT_MARGIN_AREA,
      method = GET_FROM_BUFFER,
      multibyte_p = 0,
      string_from_display_prop_p = 0,
      display_ellipsis_p = 0,
      avoid_cursor_p = 0,
      line_wrap = TRUNCATE,
      voffset = 0,
      space_width = 0,
      font_height = 0
    }, {
      string = 0,
      string_nchars = 0,
      end_charpos = 0,
      stop_charpos = 0,
      prev_stop = 0,
      base_level_stop = 0,
      cmp_it = {
        stop_pos = 0,
        id = 0,
        ch = 0,
        rule_idx = 0,
        lookback = 0,
        nglyphs = 0,
        reversed_p = 0,
        charpos = 0,
        nchars = 0,
        nbytes = 0,
        from = 0,
        to = 0,
        width = 0
      },
      face_id = 0,
      u = {
        image = {
          object = 0,
          slice = {
            x = 0,
            y = 0,
            width = 0,
            height = 0
          },
          image_id = 0
        },
        comp = {
          object = 0
        },
        stretch = {
          object = 0
        }
      },
      position = {
        charpos = 0,
        bytepos = 0
      },
      current = {
        pos = {
          charpos = 0,
          bytepos = 0
        },
        overlay_string_index = 0,
        string_pos = {
          charpos = 0,
          bytepos = 0
        },
        dpvec_index = 0
      },
      from_overlay = 0,
      area = LEFT_MARGIN_AREA,
      method = GET_FROM_BUFFER,
      multibyte_p = 0,
      string_from_display_prop_p = 0,
      display_ellipsis_p = 0,
      avoid_cursor_p = 0,
      line_wrap = TRUNCATE,
      voffset = 0,
      space_width = 0,
      font_height = 0
    }},
  sp = 0,
  selective = 0,
  what = IT_EOB,
  face_id = 0,
  selective_display_ellipsis_p = 1,
  ctl_arrow_p = 1,
  face_box_p = 0,
  start_of_box_run_p = 0,
  end_of_box_run_p = 0,
  overlay_strings_at_end_processed_p = 1,
  ignore_overlay_strings_at_pos_p = 0,
  glyph_not_available_p = 0,
  starts_in_middle_of_char_p = 0,
  face_before_selective_p = 0,
  constrain_row_ascent_descent_p = 0,
  line_wrap = WINDOW_WRAP,
  base_face_id = 0,
  c = 101,
  len = 1,
  cmp_it = {
    stop_pos = 50,
    id = -1,
    ch = -2,
    rule_idx = 0,
    lookback = 0,
    nglyphs = 0,
    reversed_p = 0,
    charpos = 0,
    nchars = 0,
    nbytes = 0,
    from = 0,
    to = 0,
    width = 0
  },
  char_to_display = 101,
  glyphless_method = GLYPHLESS_DISPLAY_THIN_SPACE,
  image_id = 0,
  slice = {
    x = 52250650,
    y = 52250650,
    width = 52250650,
    height = 52250650
  },
  space_width = 52250650,
  voffset = 0,
  tab_width = 8,
  font_height = 52250650,
  object = 80814597,
  position = {
    charpos = 53,
    bytepos = 53
  },
  truncation_pixel_width = 0,
  continuation_pixel_width = 0,
  first_visible_x = 0,
  last_visible_x = 928,
  last_visible_y = 15,
  extra_line_spacing = 0,
  max_extra_line_spacing = 0,
  override_ascent = -1,
  override_descent = 0,
  override_boff = 0,
  glyph_row = 0x4ce0800,
  area = TEXT_AREA,
  nglyphs = 1,
  pixel_width = 8,
  ascent = 12,
  descent = 3,
  max_ascent = 12,
  max_descent = 3,
  phys_ascent = 12,
  phys_descent = 3,
  max_phys_ascent = 12,
  max_phys_descent = 3,
  current_x = 432,
  continuation_lines_width = 0,
  eol_pos = {
    charpos = 0,
    bytepos = 0
  },
  current_y = 0,
  first_vpos = 0,
  vpos = 0,
  hpos = 54,
  left_user_fringe_bitmap = 0,
  right_user_fringe_bitmap = 0,
  left_user_fringe_face_id = 0,
  right_user_fringe_face_id = 0,
  bidi_p = 0,
  bidi_it = {
    bytepos = 0,
    charpos = 0,
    ch = 0,
    ch_len = 0,
    type = UNKNOWN_BT,
    type_after_w1 = UNKNOWN_BT,
    orig_type = UNKNOWN_BT,
    resolved_level = 0,
    invalid_levels = 0,
    invalid_rl_levels = 0,
    prev_was_pdf = 0,
    prev = {
      bytepos = 0,
      charpos = 0,
      type = UNKNOWN_BT,
      type_after_w1 = UNKNOWN_BT,
      orig_type = UNKNOWN_BT
    },
    last_strong = {
      bytepos = 0,
      charpos = 0,
      type = UNKNOWN_BT,
      type_after_w1 = UNKNOWN_BT,
      orig_type = UNKNOWN_BT
    },
    next_for_neutral = {
      bytepos = 0,
      charpos = 0,
      type = UNKNOWN_BT,
      type_after_w1 = UNKNOWN_BT,
      orig_type = UNKNOWN_BT
    },
    prev_for_neutral = {
      bytepos = 0,
      charpos = 0,
      type = UNKNOWN_BT,
      type_after_w1 = UNKNOWN_BT,
      orig_type = UNKNOWN_BT
    },
    next_for_ws = {
      bytepos = 0,
      charpos = 0,
      type = UNKNOWN_BT,
      type_after_w1 = UNKNOWN_BT,
      orig_type = UNKNOWN_BT
    },
    next_en_pos = 0,
    ignore_bn_limit = 0,
    sor = NEUTRAL_DIR,
    scan_dir = 0,
    stack_idx = 0,
    level_stack = {{
        level = 0,
        override = NEUTRAL_DIR
      } <repeats 64 times>},
    first_elt = 0,
    paragraph_dir = NEUTRAL_DIR,
    new_paragraph = 0,
    separator_limit = 0
  },
  paragraph_embedding = NEUTRAL_DIR
}
(gdb) n
13129               it1 = it;
(gdb) n
13130             } while (line_bottom_y (&it1) - start_y < amount_to_scroll);
(gdb) p start_y
$3 = 15
(gdb) p amount_to_scroll
$4 = 15
(gdb) p line_bottom_y (&it1)
$5 = 15
(gdb) n
13128               move_it_by_lines (&it, 1, 1);
(gdb)
13129               it1 = it;
(gdb)
13130             } while (line_bottom_y (&it1) - start_y < amount_to_scroll);
(gdb)
13128               move_it_by_lines (&it, 1, 1);
(gdb)
13129               it1 = it;
(gdb)
13130             } while (line_bottom_y (&it1) - start_y < amount_to_scroll);
(gdb) p line_bottom_y (&it1)
$6 = 15
(gdb)

Re: Lexbind (was: Emacs 23.3 released)

[Eli Zaretskii]

> From: Juanma Barranquero <lekktu@gmail.com>
> Date: Sat, 19 Mar 2011 01:54:43 +0100
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> It's looping here (in the else part, obviously):
> 
>       start_display (&it, w, startp);
>       if (scroll_max < INT_MAX)
> 	move_it_vertically (&it, amount_to_scroll);
>       else
> 	{
> 	  /* Extra precision for users who set scroll-conservatively
> 	     to most-positive-fixnum: make sure the amount we scroll
> 	     the window start is never less than amount_to_scroll,
> 	     which was computed as distance from window bottom to
> 	     point.  This matters when lines at window top and lines
> 	     below window bottom have different height.  */
> 	  struct it it1 = it;
> 	  /* We use a temporary it1 because line_bottom_y can modify
> 	     its argument, if it moves one line down; see there.  */
> 	  int start_y = line_bottom_y (&it1);
> 
> 	  do {
> 	    move_it_by_lines (&it, 1, 1);
> 	    it1 = it;
> 	  } while (line_bottom_y (&it1) - start_y < amount_to_scroll);
> 	}
> 
> In my current session, line_bottom_y (&it1) == 15, start_y == 15,
> amount_to_scroll == 15, forever and ever and ever.

So you are saying that the do-while loop is never exited, is that it?
If so, could you step inside move_it_by_lines and see why it doesn't
(move, that is)?  It is supposed to move one line down.

Also, can you show the text through which this loop is supposed to
move down?  If that text is displayed on the screen, are there some
unusual features it uses, like invisible text or continuation lines?

Re: Lexbind (was: Emacs 23.3 released)

[Juanma Barranquero]

On Sat, Mar 19, 2011 at 09:35, Eli Zaretskii <eliz@gnu.org> wrote:

> So you are saying that the do-while loop is never exited, is that it?

Yes.

> If so, could you step inside move_it_by_lines and see why it doesn't
> (move, that is)?  It is supposed to move one line down.

Unfortunately, I closed the gdb session by accident. I'll try that
once it hangs again.

> Also, can you show the text through which this loop is supposed to
> move down?  If that text is displayed on the screen, are there some
> unusual features it uses, like invisible text or continuation lines?

There was just one window, and was showing an .ini file (major mode
was Ini-Generic). I think I had nothing active that would use
invisible text, nor continuation lines.

    Juanma

Re: Lexbind

[Stefan Monnier]

> I just had one while running under GDB, so I Ctrl-C and I got the
> attached backtrace.

Hmm... the lexbind branch does not touch the redisplay code at all, so
this is perplexing.  My best guess so far is that some part of the
changed C code corrupts some part of the memory, leading to the inf-loop.


        Stefan

Re: Lexbind

[Wojciech Meyer]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>> I just had one while running under GDB, so I Ctrl-C and I got the
>> attached backtrace.
>
> Hmm... the lexbind branch does not touch the redisplay code at all, so
> this is perplexing.  My best guess so far is that some part of the
> changed C code corrupts some part of the memory, leading to the inf-loop.
>
>
>         Stefan

I noticed some hang ups few days ago with a fresh update. Not only on
`lexbind' branch but on `trunk' as well. I didn't try to debug it. Now I
rebuilt it, with complete cleanup of repository, and it *seems* to be
better. The hangup was happening usually after M-x and then Emacs was
not responsive and taking significant CPU time. `pkill emacs' had seemed to
interrupt the hang, but later it was happening again and `pkill emacs'
was killing Emacs completely. I don't know if it is relevant to the
discussion. I do have similar configuration that Juanma cited.

Wojciech

Re: Lexbind

[Juanma Barranquero]

On Thu, Mar 17, 2011 at 21:01, Stefan Monnier <monnier@iro.umontreal.ca> wrote:

> Hmm... the lexbind branch does not touch the redisplay code at all, so
> this is perplexing.

Perhaps it is just bad luck that I didn't get lockups running the
trunk, but I don't think so. It's happened at least five or six times
with lexbind-new, and I don't remember the last time I had trouble
with the trunk.

> My best guess so far is that some part of the
> changed C code corrupts some part of the memory, leading to the inf-loop.

So it's going to be difficult to catch :-(

    Juanma

Re: Lexbind

[joakim]

Juanma Barranquero <lekktu@gmail.com> writes:

> On Thu, Mar 17, 2011 at 21:01, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>
>> Hmm... the lexbind branch does not touch the redisplay code at all, so
>> this is perplexing.
>
> Perhaps it is just bad luck that I didn't get lockups running the
> trunk, but I don't think so. It's happened at least five or six times
> with lexbind-new, and I don't remember the last time I had trouble
> with the trunk.
>
>> My best guess so far is that some part of the
>> changed C code corrupts some part of the memory, leading to the inf-loop.
>
> So it's going to be difficult to catch :-(

What's the state of running Emacs under Valgrind these days? I have a
vague memory that it might be possible with an undumped Emacs.

>     Juanma
>

-- 
Joakim Verona

Re: Emacs 23.3 released

[Mike Mattie]

[-- Attachment #1: Type: text/plain, Size: 1482 bytes --]

On Thu, Mar 10, 2011 at 11:07:37AM -0500, Stefan Monnier wrote:
> A big thanks to everybody for the work they put into this release.
> 
> I hope we will be able to jump right to 24.1 without an intermediate
> 23.4, but if you find bugs that are not specific to trunk, please still
> commit them to emacs-23.
> 
> Talking about 24.1, I now consider the lexbind-new branch as "fully
> functional": there are still a few changes I want to do there before
> merging, but they should only impact performance and code quality, so
> I encourage people to try it out and report any regression so they can
> be addressed before we merge it into trunk.
> 
> 
>         Stefan
> 

This is excellent work ! My hat off to the Emacs developers !

> 
> >>>>> "Chong" == Chong Yidong <cyd@stupidchicken.com> writes:
> 
> > GNU Emacs 23.3 has been released, and is now available at
> > ftp.gnu.org/gnu/emacs/ and the GNU FTP mirrors (see
> > http://www.gnu.org/order/ftp.html).
> 
> > The SHA1 checksums are:
> 
> >  de2c924655f0f4e012cc119840cd341a1d40e32c  emacs-23.3.tar.bz2
> >  53bdfe584f98c13265caa3eea3cfdf79ca0f7584  emacs-23.3.tar.gz
> 
> > Please send bug reports to <bug-gnu-emacs@gnu.org>.  You can use the
> > command M-x report-emacs-bug to do this.
> 
> > Emacs 23.3 is primarily a bugfix release.  Also included are a number of
> > enhancements to VC and Rmail, and a new indentation library, smie.el.
> > For details, see the file etc/NEWS.
> 

[-- Attachment #2: Type: application/pgp-signature, Size: 0 bytes --]

Re: Emacs 23.3 released

[Glenn Morris]

Stefan Monnier wrote:

> but if you find bugs that are not specific to trunk, please still
> commit them to emacs-23.

I'm pretty sure many of these have been committed to just trunk already,
so if there is a 23.4, Someone (TM) will have to go searching for such
changes and backport them. Eg there are 20 bugs marked as found in 23.x,
fixed in 24.1, and I'm sure there are more that just didn't have the
fixed version specified when they were closed.

I do encourage people to start marking bugs with the version number when
they close them. Eg:

close 123 24.1

or:

To: 123-done@debbugs

-- body starts below:
Version: 24.1

Re: Emacs 23.3 released

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Thu, 10 Mar 2011 11:07:37 -0500
> Cc: Chong Yidong <cyd@stupidchicken.com>
> 
> I hope we will be able to jump right to 24.1 without an intermediate
> 23.4, but if you find bugs that are not specific to trunk, please still
> commit them to emacs-23.

Will the emacs-23 branch still be merged to the trunk once in a short
while?  Or are the committers there expected to do that themselves?

Re: Emacs 23.3 released

[Chong Yidong]

Eli Zaretskii <eliz@gnu.org> writes:

> Will the emacs-23 branch still be merged to the trunk once in a short
> while?

Yes.

Re: Emacs 23.3 released

[Stefan Monnier]

>> I hope we will be able to jump right to 24.1 without an intermediate
>> 23.4, but if you find bugs that are not specific to trunk, please still
>> commit them to emacs-23.
> Will the emacs-23 branch still be merged to the trunk once in a short
> while?

Yes.  Tho how "short" depends on happenstance, so if you don't feel like
merging yourself and it's taking longer than you like, please ask here.


        Stefan

Re: Emacs 23.3 released

[Chong Yidong]

Note, by the way, that the manuals on the webpage are not yet updated
yet.  I will do that over the next few days.

updating web manuals [Re: Emacs 23.3 released

[Jim Meyering]

Chong Yidong wrote:
> Note, by the way, that the manuals on the webpage are not yet updated
> yet.  I will do that over the next few days.

That can be a tediously er,... manual task, so for packages like
coreutils, diffutils, grep, etc., I automated it with this script:

    http://git.sv.gnu.org/cgit/gnulib.git/tree/build-aux/gnu-web-doc-update

Now, I run "build-aux/gnu-web-doc-update" after each release,
and it does everything.  Slightly hard to see is that it uses
the web-manual rule from gnulib's maint.mk:

.PHONY: web-manual
web-manual:
        @test -z "$(manual_title)" \
          && { echo define manual_title in cfg.mk 1>&2; exit 1; } || :
        @cd '$(srcdir)/doc'; \
          $(SHELL) ../build-aux/gendocs.sh $(gendocs_options_) \
             -o '$(abs_builddir)/doc/manual' \
             --email $(PACKAGE_BUGREPORT) $(PACKAGE) \
            "$(PACKAGE_NAME) - $(manual_title)"
        @echo " *** Upload the doc/manual directory to web-cvs."

It is obviously tailored to the use of git and other aspects of those
projects, but may be useful if you want to do something similar for emacs.