Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[Dmitry Gutov]

Hi Stefan,

Stefan Monnier <monnier@iro.umontreal.ca> writes:
> -			 ;; Simple sanity check.
> +			 ;; Simple sanity checks.
> +                         (< (point) pos) ; backward-paragraph can fail here.

Why `<' and not `<='?

In mmm-mode, we call the default font-lock-fontify-region function with
bounds of each subregion as arguments, and we also try to limit
`syntax-ppss' to these regions, so the `syntax-begin-function' brings
point to the beginning of the current subregion.

So `mmm-beginning-of-syntax' won't move point at all if it's at the
beginning of a subregion. And I think it's okay, and `syntax-ppss'
should use the position obtained this way.

Blast from the past over,
Dmitry.

Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[Stefan Monnier]

>> -			 ;; Simple sanity check.
>> +			 ;; Simple sanity checks.
>> +                         (< (point) pos) ; backward-paragraph can fail here.

> Why `<' and not `<='?

IIRC the reason was so that (syntax-ppss) called from a place like
a "column-0 just before an open paren" can correctly indicate if this
open paren is within a string/comment while at the same time parsing the
subsequent text assuming that this open paren was outside of any
string/comment.

Maybe there is another reason, but that's the one I remember.
Not a terribly strong reason, admittedly.

> In mmm-mode, we call the default font-lock-fontify-region function with
> bounds of each subregion as arguments, and we also try to limit
> `syntax-ppss' to these regions, so the `syntax-begin-function' brings
> point to the beginning of the current subregion.

But there's no guarantee syntax-begin-function will be used at
all, so I'm not sure how important changing this test is.
Is it really sufficient to make syntax-ppss understand mmm's sub-modes?


        Stefan

Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[Dmitry Gutov]

On 10.02.2014 18:18, Stefan Monnier wrote:
> IIRC the reason was so that (syntax-ppss) called from a place like
> a "column-0 just before an open paren" can correctly indicate if this
> open paren is within a string/comment while at the same time parsing the
> subsequent text assuming that this open paren was outside of any
> string/comment.

Why? That sounds confusing.

>> In mmm-mode, we call the default font-lock-fontify-region function with
>> bounds of each subregion as arguments, and we also try to limit
>> `syntax-ppss' to these regions, so the `syntax-begin-function' brings
>> point to the beginning of the current subregion.
>
> But there's no guarantee syntax-begin-function will be used at
> all, so I'm not sure how important changing this test is.
> Is it really sufficient to make syntax-ppss understand mmm's sub-modes?

Not exactly. But if it worked the way I expected, it would've been 
easier for me to fix fontification in subregions when it's affected by a 
character in the primary region above it when it's "string 
opener/closer" in the submode's syntax table, but punctuation in the 
primary mode syntax: https://github.com/purcell/mmm-mode/issues/37

I would've bound `syntax-ppss-last` and `syntax-ppss-cache` to nil 
around the region and let it be forced to call the syntax-begin-function.

Since that didn't work, I now seem to be successfully fooling 
syntax-ppss by let-binding `syntax-ppss-last' to (beg-of-subregion . 
nil), and fontification is as expected. Maybe that'll even work for 
indentation, too.

But it seems like `mmm-beginning-of-syntax' is entirely useless now 
(since its result is discarded most of the time, and when not, it could 
only bring inconsistency). Because it dates back from 2000, about 10 
years prior to the aforementioned revision, maybe it should take 
priority. Or not, since nobody complained in the 3.5 years since.

Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[Stefan Monnier]

>> IIRC the reason was so that (syntax-ppss) called from a place like
>> a "column-0 just before an open paren" can correctly indicate if this
>> open paren is within a string/comment while at the same time parsing the
>> subsequent text assuming that this open paren was outside of any
>> string/comment.
> Why? That sounds confusing.

That's the point: it makes it possible to find those sources of
confusion and highlight them.

E.g. I had some font-lock code which would highlight an
open-paren-in-column-0-in-string/comment with the `warning' face.
So such an "incorrect" open paren would still cause incorrect
highlighting, but the `warning' face on it would provide the clue as to
what was the source of the problem.

> I would've bound `syntax-ppss-last` and `syntax-ppss-cache` to nil around
> the region and let it be forced to call the syntax-begin-function.

Right, but that largely defeats the purpose of syntax-ppss (which is to
use caching to speed up (parse-partial-sexp (point-min) (point))).

To give you some background: I think syntax-begin-function is basically
useless.  It's used in very few places (it used to be used in lisp-mode,
but that was disabled recently, it's still used in js-mode, but it
should probably be disabled there as well, and apparently mmm-mode also
uses it, but these are the only cases I know) and is more trouble than
it's worth.  It was meant and is designed as an optimization, but it is
vanishingly often useful.

For your use case, you don't want to provide an optimization.  We should
instead provide a new hook whose use would be mandatory (i.e. which
syntax.el has to obey).

I'm not completely sure what it could/should look like, tho.

One option is to have a hook that takes a (POS . PPSS) pair, which
syntax-ppss intends to use as a starting point for parsing, and return
a new such pair to use instead, where the returned position should
always be >= POS.

This way, syntax-ppss could make full use of its cache, but mmm-mode
could tell it about chunk boundaries (and decide what state to use at
the beginning of a boundary).

The main problem I see with this approach is that this hook would be
called maybe too many times, so we'd want to improve the "fast path"
(i.e. the first branch in syntax-ppss which tries to use
syntax-ppss-last) so it can know when calling this new hook is unneeded.

Maybe for that, the new hook should return not just a new (POS . PPSS)
but also a "next-boundary" so we know we don't need to call this hook
again as long as we're within POS...NEXT-BOUNDARY.


        Stefan

Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[Dmitry Gutov]

On 12.02.2014 03:30, Stefan Monnier wrote:
> E.g. I had some font-lock code which would highlight an
> open-paren-in-column-0-in-string/comment with the `warning' face.
> So such an "incorrect" open paren would still cause incorrect
> highlighting, but the `warning' face on it would provide the clue as to
> what was the source of the problem.

I don't fully understand the explanation, but the logic "if 
syntax-beginning equals point, go to previous syntax-beginning" could've 
been handled in the specific syntax-beginning-function instead.

> Right, but that largely defeats the purpose of syntax-ppss (which is to
> use caching to speed up (parse-partial-sexp (point-min) (point))).

The optimization is still used if `syntax-ppss' is called several times 
during the syntax-propertization or fontification of one region.

Same with indentation, if we did that.

> To give you some background: I think syntax-begin-function is basically
> useless.  It's used in very few places (it used to be used in lisp-mode,
> but that was disabled recently, it's still used in js-mode, but it
> should probably be disabled there as well, and apparently mmm-mode also
> uses it, but these are the only cases I know) and is more trouble than
> it's worth.  It was meant and is designed as an optimization, but it is
> vanishingly often useful.

Okay, I can understand that.

> One option is to have a hook that takes a (POS . PPSS) pair, which
> syntax-ppss intends to use as a starting point for parsing, and return
> a new such pair to use instead, where the returned position should
> always be >= POS.

Sounds fine to me. As long as the hook is called at the same point 
`syntax-ppss' is called at, we can check whether POS is in the same 
region, look for nested submode regions between POS and point, and 
either discard the passed PPSS if the current subregion begins after 
POS, or manually `parse-partial-sexp' each piece of the current 
subregion (of the primary mode region, if we're there) between POS and 
some position closer to point.

We could parse the buffer till point itself, though. It wouldn't be 
harder coding-wise (we'll do `parse-partial-sexp's anyway), and that way 
the hook could be more flexible. Then the meaning of the hook would be 
"here's the last saved position and value, what will be the value at 
point?".

> This way, syntax-ppss could make full use of its cache, but mmm-mode
> could tell it about chunk boundaries (and decide what state to use at
> the beginning of a boundary).
>
> The main problem I see with this approach is that this hook would be
> called maybe too many times, so we'd want to improve the "fast path"
> (i.e. the first branch in syntax-ppss which tries to use
> syntax-ppss-last) so it can know when calling this new hook is unneeded.

Maybe we want that, but scanning the buffer for overlays should still be 
a) proportional to the distance between bounds, b) faster than 
`parse-partial-sexp', so at worst in mmm-mode the new scheme will just 
be slower than plain `syntax-ppss' by some constant ratio, on average.

> Maybe for that, the new hook should return not just a new (POS . PPSS)
> but also a "next-boundary" so we know we don't need to call this hook
> again as long as we're within POS...NEXT-BOUNDARY.

Not sure if it'll work. Suppose we're in some region, which spans 400 
chars after point, and then it's another region.

We call `syntax-ppss', happily report to it that the value at point (or 
some position near it) can be used until point + 400. Then move a few 
chars lower and delete the rest of the given region. NEXT-BOUNDARY 
becomes stale, and calling `syntax-ppss' from the region below can 
return a wrong value.

Using markers should work better, but maybe some problems are lurking 
there as well.

Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[Dmitry Gutov]

> On 12.02.2014 03:30, Stefan Monnier wrote:
>> One option is to have a hook that takes a (POS . PPSS) pair, which
>> syntax-ppss intends to use as a starting point for parsing, and return
>> a new such pair to use instead, where the returned position should
>> always be >= POS.

A problem: suppose point is in a primary mode region, and POS is in a 
submode region above it. Then we can't use the value of PPSS at all.
Does the hook re-scan the buffer from the beginning?

On 12.02.2014 04:49, Dmitry Gutov wrote:
> We call `syntax-ppss', happily report to it that the value at point (or
> some position near it) can be used until point + 400. Then move a few
> chars lower and delete the rest of the given region. NEXT-BOUNDARY
> becomes stale, and calling `syntax-ppss' from the region below can
> return a wrong value.

I suppose this could be handled if `syntax-ppss-flush-cache' removed or 
modified the cache entries where NEXT-BOUNDARY is after BEG. And by 
modified, I mean set NEXT-BOUNDARY to BEG.

Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[Stefan Monnier]

> I don't fully understand the explanation, but the logic "if syntax-beginning
> equals point, go to previous syntax-beginning" could've been handled in the
> specific syntax-beginning-function instead.

Could be.

>> Right, but that largely defeats the purpose of syntax-ppss (which is to
>> use caching to speed up (parse-partial-sexp (point-min) (point))).
> The optimization is still used if `syntax-ppss' is called several times
> during the syntax-propertization or fontification of one region.

In 99% of the cases, syntax-ppss is only called once during font-lock.
So, while in theory, yes, we might still get some benefits, in practice
we don't.

>> One option is to have a hook that takes a (POS . PPSS) pair, which
>> syntax-ppss intends to use as a starting point for parsing, and return
>> a new such pair to use instead, where the returned position should
>> always be >= POS.
> Sounds fine to me.  As long as the hook is called at the same point
> `syntax-ppss' is called at,

Yes, the intention would be to call this hook just before calling
(parse-partial-sexp POS (point) nil nil PPSS).

> we can check whether POS is in the same region,
> look for nested submode regions between POS and point, and either discard
> the passed PPSS if the current subregion begins after POS, or manually
> `parse-partial-sexp' each piece of the current subregion (of the primary
> mode region, if we're there) between POS and some position closer to point.

Right, the hook could find the nearest boundary and if it's before POS,
return that boundary together with the PPSS to use for it.

> We could parse the buffer till point itself, though. It wouldn't be harder
> coding-wise (we'll do `parse-partial-sexp's anyway), and that way the hook
> could be more flexible. Then the meaning of the hook would be "here's the
> last saved position and value, what will be the value at point?".

I think I'd still prefer syntax-ppss to do the call to
parse-partial-sexp (e.g. if the region to parse is very large,
syntax-ppss parses it in several chunks, storing the intermediate state
in the cache).  Also syntax-ppss would probably want to add the
(BOUNDARY . PPSS) returned by the hook to its cache.

Of course, another issue is "which syntax-table and syntax-propertize
function to use".  How does mmm-mode handle that currently?

>> This way, syntax-ppss could make full use of its cache, but mmm-mode
>> could tell it about chunk boundaries (and decide what state to use at
>> the beginning of a boundary).
>> 
>> The main problem I see with this approach is that this hook would be
>> called maybe too many times, so we'd want to improve the "fast path"
>> (i.e. the first branch in syntax-ppss which tries to use
>> syntax-ppss-last) so it can know when calling this new hook is unneeded.
> Maybe we want that, but scanning the buffer for overlays should still be a)
> proportional to the distance between bounds, b) faster than
> `parse-partial-sexp', so at worst in mmm-mode the new scheme will just be
> slower than plain `syntax-ppss' by some constant ratio, on average.

In the fast path, parse-partial-sexp may often end up parsing just
a handful (or even 0) of characters, which can be much less than the
distance to the nearest boundary.  It's important for this case to be
fast so that we can use syntax-ppss liberally, knowing it's very cheap
because we just called it nearby recently.

But maybe you're right that the performance impact might not be
that important.

> We call `syntax-ppss', happily report to it that the value at point (or some
> position near it) can be used until point + 400. Then move a few chars lower
> and delete the rest of the given region. NEXT-BOUNDARY becomes stale, and
> calling `syntax-ppss' from the region below can return a wrong value.

That's OK.  syntax-ppss already has an after-change-function to
flush its various caches.


        Stefan

Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[Stefan Monnier]

>>> One option is to have a hook that takes a (POS . PPSS) pair, which
>>> syntax-ppss intends to use as a starting point for parsing, and return
>>> a new such pair to use instead, where the returned position should
>>> always be >= POS.
> A problem: suppose point is in a primary mode region, and POS is
> in a submode region above it. Then we can't use the value of PPSS at all.

I don't understand the problem.  Of course your hook would return
a completely different PPSS to use along with the new buffer position
from where to start parsing.

> Does the hook re-scan the buffer from the beginning?

I don't understand the question.

> I suppose this could be handled if `syntax-ppss-flush-cache' removed or
> modified the cache entries where NEXT-BOUNDARY is after BEG.

Exactly.

> And by modified, I mean set NEXT-BOUNDARY to BEG.

Yes, something like that.


        Stefan

Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[Dmitry Gutov]

On 12.02.2014 16:26, Stefan Monnier wrote:
>>>> One option is to have a hook that takes a (POS . PPSS) pair, which
>>>> syntax-ppss intends to use as a starting point for parsing, and return
>>>> a new such pair to use instead, where the returned position should
>>>> always be >= POS.
>> A problem: suppose point is in a primary mode region, and POS is
>> in a submode region above it. Then we can't use the value of PPSS at all.
>
> I don't understand the problem.  Of course your hook would return
> a completely different PPSS to use along with the new buffer position
> from where to start parsing.

How would it calculate the returned PPSS? If the PPSS passed in is in a 
submode region, the primary mode won't be able to use its value.

And the hook won't have access to earlier cached values. Or would it?

Or course, if the cache includes NEXT-BOUNDARY information, this 
situation should never happen. This'll have to be handled in the new 
`syntax-ppss' cache lookup logic.

>> Does the hook re-scan the buffer from the beginning?
>
> I don't understand the question.

If we don't have a cached value we can use, we have to 
`parse-partial-sexp' from the beginning of the buffer.

Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[Stefan Monnier]

>> I don't understand the problem.  Of course your hook would return
>> a completely different PPSS to use along with the new buffer position
>> from where to start parsing.
> How would it calculate the returned PPSS?

In many cases I'd expect the returned PPSS would be nil.

> And the hook won't have access to earlier cached values.

I don't see any reason why the hook wouldn't be able to call
syntax-ppss recursively.


        Stefan

Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[Dmitry Gutov]

On 13.02.2014 02:13, Stefan Monnier wrote:
> In many cases I'd expect the returned PPSS would be nil.

What would the nil value mean? Beginning of a new subregion?

>> And the hook won't have access to earlier cached values.
>
> I don't see any reason why the hook wouldn't be able to call
> syntax-ppss recursively.

Hm yes, that should work.

Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[Dmitry Gutov]

On 12.02.2014 16:23, Stefan Monnier wrote:
> In 99% of the cases, syntax-ppss is only called once during font-lock.
> So, while in theory, yes, we might still get some benefits, in practice
> we don't.

`syntax-propertize' can call it multiple times, though (at least in the 
case of ruby-mode), and that's called as often as font-lock.

> Of course, another issue is "which syntax-table and syntax-propertize
> function to use".  How does mmm-mode handle that currently?

It has a function called `mmm-update-submode-region', which restores the 
saved buffer-local values, syntax table, etc, for the mode at point. 
It's called:

* From post-command-hook.
* From `mmm-syntax-propertize-function', which iterates over regions, 
calls the aforementioned function and then the submode's 
syntax-propertize-function.
* From `mmm-fontify-region', which is similar, but for font-lock.
* From `mmm-indent-line' and the specialized `mmm-indent-line', because 
we usually want to indent by the rules of the submode at indentation, 
and it might be different in the middle of the line.

IOW, various stuff works, as long as mmm-mode is aware of the respective 
facility. Each has to be supported explicitly, though.

By the way, AFAICT, the juggling of local variables between modes (save, 
restore, rinse, repeat) is one of the major factors in mmm-mode performance.

There was an old `multi-mode' by Dave Love that used indirect buffers to 
keep the buffer-local values around, and switched between them in 
post-command-hook (which might or might not have been a good idea) and 
in `multi-fontify-region', taking advantage of the fact that 
fontification in an indirect buffer translates into the base buffer.

Now I hear that we shouldn't use indirect buffers. Would there be a fast 
way to save and restore all local variables in a buffer? This would cut 
down on both mmm-mode overhead and complexity.

Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[Stefan Monnier]

>> In many cases I'd expect the returned PPSS would be nil.
> What would the nil value mean? Beginning of a new subregion?

Yes, nil is the PPS state "outside of everything", e.g. at BoB.

>>> And the hook won't have access to earlier cached values.
>> I don't see any reason why the hook wouldn't be able to call
>> syntax-ppss recursively.
> Hm yes, that should work.

Of course, only on positions before the current one, to avoid
inf-looping.


        Stefan

Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[Stefan Monnier]

>> In 99% of the cases, syntax-ppss is only called once during font-lock.
>> So, while in theory, yes, we might still get some benefits, in practice
>> we don't.
> `syntax-propertize' can call it multiple times, though (at least in the case
> of ruby-mode), and that's called as often as font-lock.

Yes, that's part of the remaining 1%.

> By the way, AFAICT, the juggling of local variables between modes (save,
> restore, rinse, repeat) is one of the major factors in mmm-mode performance.

I can believe that.

> There was an old `multi-mode' by Dave Love that used indirect buffers to
> keep the buffer-local values around, and switched between them in
> post-command-hook (which might or might not have been a good idea) and in
> `multi-fontify-region', taking advantage of the fact that fontification in
> an indirect buffer translates into the base buffer.
> Now I hear that we shouldn't use indirect buffers.

Indeed, I don't think using an indirect buffer is a good solution.
I'm not even sure it's faster than mmm's "manual" variable switch.

> Would there be a fast way to save and restore all local variables in
> a buffer? This would cut down on both mmm-mode overhead
> and complexity.

Not sure how best to speed it up.  We could of course reduce the
interpretation overhead by providing a kind of "context-switch"
primitive written in C.  This would probably speed it up significantly.

But making it do really less work is difficult: there's the
"local-variables" alist in the "struct buffer" which we could switch in
one step, but:
- it doesn't account for all local variables (doesn't account for those
  local-vars held directly in the "struct buffer" such as tab-width).
- it would switch some vars which should not be switched.


        Stefan

Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[David Kastrup]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> In 99% of the cases, syntax-ppss is only called once during font-lock.
>>> So, while in theory, yes, we might still get some benefits, in practice
>>> we don't.
>> `syntax-propertize' can call it multiple times, though (at least in the case
>> of ruby-mode), and that's called as often as font-lock.
>
> Yes, that's part of the remaining 1%.
>
>> By the way, AFAICT, the juggling of local variables between modes (save,
>> restore, rinse, repeat) is one of the major factors in mmm-mode performance.
>
> I can believe that.
>
>> There was an old `multi-mode' by Dave Love that used indirect buffers to
>> keep the buffer-local values around, and switched between them in
>> post-command-hook (which might or might not have been a good idea) and in
>> `multi-fontify-region', taking advantage of the fact that fontification in
>> an indirect buffer translates into the base buffer.
>> Now I hear that we shouldn't use indirect buffers.
>
> Indeed, I don't think using an indirect buffer is a good solution.
> I'm not even sure it's faster than mmm's "manual" variable switch.

Using different modes on a buffer would seem like a rather canonical use
case for indirect buffers.  If they don't work well for that, one needs
to think about _what_ would work better.  That may mean improving
indirect buffers or creating something different.  In which case one
should recheck whether indirect buffers work well for _anything_.

-- 
David Kastrup

Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[Dmitry Gutov]

On 13.02.2014 15:28, Stefan Monnier wrote:
> Indeed, I don't think using an indirect buffer is a good solution.
> I'm not even sure it's faster than mmm's "manual" variable switch.

Why not? `switch-buffer' has to be faster than "iterate these lists, 
save values of symbols, set them to values from those alists".

Or do you mean that indirect buffers add their own overhead, just by 
having (and somehow coordinating, I guess) the same contents?

> Not sure how best to speed it up.  We could of course reduce the
> interpretation overhead by providing a kind of "context-switch"
> primitive written in C.  This would probably speed it up significantly.
>
> But making it do really less work is difficult: there's the
> "local-variables" alist in the "struct buffer" which we could switch in
> one step, but:
> - it doesn't account for all local variables (doesn't account for those
>    local-vars held directly in the "struct buffer" such as tab-width).
> - it would switch some vars which should not be switched.

This, together with a blacklist and a small "additional variables" list 
would still probably be a lot neater than mmm-vars.el:


https://github.com/purcell/mmm-mode/blob/master/mmm-vars.el#L111-L302

Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[Stefan Monnier]

> Using different modes on a buffer would seem like a rather canonical use
> case for indirect buffers.  If they don't work well for that, one needs
> to think about _what_ would work better.

While it can be "good enough" for some particular uses, overall they
don't work well for that use case.

> That may mean improving indirect buffers or creating something
> different.  In which case one should recheck whether indirect buffers
> work well for _anything_.

AFAIC they don't work well for anything.  As for creating something
different, well that's what mmm-mode is doing.


        Stefan

Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Thu, 13 Feb 2014 14:46:11 +0100
> 
> Using different modes on a buffer would seem like a rather canonical use
> case for indirect buffers.  If they don't work well for that, one needs
> to think about _what_ would work better.  That may mean improving
> indirect buffers or creating something different.  In which case one
> should recheck whether indirect buffers work well for _anything_.

The problem, AFAIU, is that different use cases expect different
things from indirect buffers.  E.g., some expect them to share text
properties, while others want them not to.  Since indirect buffers
share text with their base buffer, such conflicting requirements are
impossible to implement.

IOW, this is business as usual: a feature is invented having a certain
use case in mind, but then it gets used for entirely different use
cases, and everybody expects the new toy to do exactly what they want.

Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[Eli Zaretskii]

> Date: Thu, 13 Feb 2014 16:12:04 +0200
> From: Dmitry Gutov <dgutov@yandex.ru>
> Cc: emacs-devel@gnu.org
> 
> Or do you mean that indirect buffers add their own overhead, just by 
> having (and somehow coordinating, I guess) the same contents?

There's no coordination.  Quite simply, the pointer to buffer text in
an indirect buffer points to the text of its base buffer.

Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[Stefan Monnier]

>> Indeed, I don't think using an indirect buffer is a good solution.
>> I'm not even sure it's faster than mmm's "manual" variable switch.
> Why not? `switch-buffer' has to be faster than "iterate these lists, save
> values of symbols, set them to values from those alists".

Of course, if the code uses switch-to-buffer, then it gets much slower
because of the redisplays.

Just `set-buffer' might be faster, indeed, but mostly because it's
written in C (e.g. it also has to iterate through the local variables to
save the old value and reload the new value for some of them).

But then you have to add things like the need to handle variables that
are buffer-local but should be shared among the indirect buffers, and
I doubt the end result will be faster.  Plus handle all the bugs that
indirect buffers introduce (e.g. a modification in one of the indirect
buffers only runs its own after-change-functions, not the one of the
base buffer or any of the other indirect buffers; overlays added by
a major/minor mode in its indirect buffer won't affect the display of
the base buffer; ...).

> Or do you mean that indirect buffers add their own overhead, just by having
> (and somehow coordinating, I guess) the same contents?

I wasn't thinking of that, no, although it is true that the presence of
an indirect buffer can slow down some operations.

>> But making it do really less work is difficult: there's the
>> "local-variables" alist in the "struct buffer" which we could switch in
>> one step, but:
>> - it doesn't account for all local variables (doesn't account for those
>> local-vars held directly in the "struct buffer" such as tab-width).
>> - it would switch some vars which should not be switched.
> This, together with a blacklist and a small "additional variables" list
> would still probably be a lot neater than mmm-vars.el:

Hmm... I wonder why mmm-save-local-variables is defined the way it is.
It doesn't seem to make use of `buffer-local-variables', even though it's
the most natural starting point.  E.g. there shouldn't be a need to list
explicitly all the c-* variables.  Maybe the author didn't know about
buffer-local-variables (or it didn't exist back then)?

IOW I'd expect to start with a loop through the output of
(buffer-local-variables), skipping the permanent-local vars.  Then add
special cases (e.g. current-local-map, syntax-table which aren't
variables or syntax-ppss-cache which should probably be treated like
a permanent-local).

Of course, maybe it wouldn't be faster: (buffer-local-variables) returns
a fairly long list (like 100 more than elements), and at every
mode-switch we'd have to go through 2 such lists (the "from" and the
"to").  If we can make this clean enough, we could implement it in C,
which might speed it up significantly.


        Stefan

Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[Dmitry Gutov]

On 13.02.2014 18:42, Stefan Monnier wrote:
> Just `set-buffer' might be faster, indeed, but mostly because it's
> written in C (e.g. it also has to iterate through the local variables to
> save the old value and reload the new value for some of them).

Not the 100+ of them, though?

> But then you have to add things like the need to handle variables that
> are buffer-local but should be shared among the indirect buffers, and
> I doubt the end result will be faster.  Plus handle all the bugs that
> indirect buffers introduce (e.g. a modification in one of the indirect
> buffers only runs its own after-change-functions, not the one of the
> base buffer or any of the other indirect buffers; overlays added by
> a major/minor mode in its indirect buffer won't affect the display of
> the base buffer; ...).

This clearly calls for introduction of base-buffer-local variables!

Half-kidding aside, yeah, it sounds like indirect buffers might need 
some bugfixing to be usable as a building block for mmm-mode.

> Hmm... I wonder why mmm-save-local-variables is defined the way it is.
> It doesn't seem to make use of `buffer-local-variables', even though it's
> the most natural starting point.  E.g. there shouldn't be a need to list
> explicitly all the c-* variables.  Maybe the author didn't know about
> buffer-local-variables (or it didn't exist back then)?

I'm not quite sure, but this way we only save the variables that are 
known to be safe in the multiple-mode context. And also don't touch 
anything mmm-mode itself introduced. For non-cc-engine modes, the lists 
of variables to save are quite short, on the order of ~30 items.

Apparently I was even wrong about saving and restoring being the slow 
part: `mmm-update-submode-region', which includes the calls to 
`mmm-save-changed-local-variables' and `mmm-set-local-variables', is 
faster than simply calling `buffer-local-variables'.

In a mixed-mode test file:

(js2-time (dotimes (_ 1000) (buffer-local-variables))) => 0.098

(js2-time (dotimes (_ 1000) (mmm-update-submode-region))) => 0.055

^- the latter was without the check "if mode unchanged, do nothing".

Now that I've checked, parsing the buffer into regions is definitely the 
slowest part (followed by fontification, which takes about 40% of the 
time in the current test example):

(js2-time (dotimes (_ 1000) (mmm-parse-buffer))) => 7.97

^- includes both `mmm-apply-classes' and `mmm-refontify-maybe'.

And it's a 30-line file.

Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[Eli Zaretskii]

> Date: Fri, 14 Feb 2014 06:44:13 +0200
> From: Dmitry Gutov <dgutov@yandex.ru>
> Cc: emacs-devel@gnu.org
> 
> Half-kidding aside, yeah, it sounds like indirect buffers might need 
> some bugfixing to be usable as a building block for mmm-mode.

I think features that want mode-specific behavior in some region of a
file need new infrastructure (that needs to be discussed and designed
first).  Doing this mmm-style is IMO a terrible kludge that cannot
possibly work well.  Serious functionality like that should stop
piggy-backing unrelated features in Emacs, and request infrastructure
that will serve them.  Straining Emacs Lisp-related extensibility for
that is simply wrong, as it puts gratuitous pressure on existing
infrastructure, and stands in the way of future development by
imposing impossible backward compatibility requirements.

Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Fri, 14 Feb 2014 06:44:13 +0200
>> From: Dmitry Gutov <dgutov@yandex.ru>
>> Cc: emacs-devel@gnu.org
>> 
>> Half-kidding aside, yeah, it sounds like indirect buffers might need 
>> some bugfixing to be usable as a building block for mmm-mode.
>
> I think features that want mode-specific behavior in some region of a
> file need new infrastructure (that needs to be discussed and designed
> first).  Doing this mmm-style is IMO a terrible kludge that cannot
> possibly work well.  Serious functionality like that should stop
> piggy-backing unrelated features in Emacs, and request infrastructure
> that will serve them.  Straining Emacs Lisp-related extensibility for
> that is simply wrong, as it puts gratuitous pressure on existing
> infrastructure, and stands in the way of future development by
> imposing impossible backward compatibility requirements.

It's not like there isn't use for it: flex and bison files
(pattern/action) are multiple mode on a large scale, and even
address/action languages like sed and awk have facets of multiple modes.

LilyPond uses @lilypond tags in Texinfo to have examples written in
LilyPond code (and embedded as images in the info files).

Emacs can use text properties to switch syntax tables or categories or
even keymaps in mid-buffer.  Maybe switching the whole mode would be
feasible in a similar manner.

-- 
David Kastrup

Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Fri, 14 Feb 2014 10:54:49 +0100
> 
> It's not like there isn't use for it: flex and bison files
> (pattern/action) are multiple mode on a large scale, and even
> address/action languages like sed and awk have facets of multiple modes.

I agree that being able to support portions of a buffer that specify
their own major mode is an important feature.  I'm just saying that we
need a suitable infrastructure for that, instead of trying to exploit
unrelated features, which seem like "a good idea".

> Emacs can use text properties to switch syntax tables or categories or
> even keymaps in mid-buffer.  Maybe switching the whole mode would be
> feasible in a similar manner.

Yes, something like that.  Basically, somehow present to major-mode
features only a portion of the buffer (e.g., by narrowing internally).

Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[Dmitry Gutov]

Eli Zaretskii <eliz@gnu.org> writes:

> I agree that being able to support portions of a buffer that specify
> their own major mode is an important feature.  I'm just saying that we
> need a suitable infrastructure for that, instead of trying to exploit
> unrelated features, which seem like "a good idea".

The first hint of a "suitable infrastructure" would be the adaptation of
`syntax-ppss'. Where the information about region boundaries would come
from, if not from mmm-mode? Text properties? Overlay properties? Where
would they come from, if not from mmm-mode or similar package?

I guess adapting the default font-lock-fontify-region function and
syntax-propertize to be aware of them would be good, but that could be
done later.

> Yes, something like that.  Basically, somehow present to major-mode
> features only a portion of the buffer (e.g., by narrowing internally).

This doest address primary mode regions, which a) need to see the
previous chunks of the same mode, b) (for most functions) ignore the
chunks of the submodes. But not for indentation: in some (!) mode
combinations, the contents of submode regions influence indentation in
the primary regions, namely in JSP, ERB and similar templates.

Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[Stefan Monnier]

> I think features that want mode-specific behavior in some region of a
> file need new infrastructure (that needs to be discussed and designed
> first).

I've been following those multiple major mode thingies for quite a few
years now.  I don't think we really need a significant new
infrastructure at the C level.  What we need instead is some conventions
that major modes need to follow to play well in things like mmm-mode
or mumamo.

E.g. I don't think "pseudo-narrowing" to magically pretend that only the
current "code chunk" exists is going to fly: it will break major modes
that don't follow pretty much the same conventions.

As for whether multiple major modes are frequent: I consider comments
and strings as being "text-mode" chunks.  Not sure if we'll ever get to
the point of integrating the handling of strings/comments with the
handling of other forms of multi-major-modes, tho.


        Stefan

Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[Stefan Monnier]

>> Just `set-buffer' might be faster, indeed, but mostly because it's
>> written in C (e.g. it also has to iterate through the local variables to
>> save the old value and reload the new value for some of them).
> Not the 100+ of them, though?

It does iterate over the whole list, yes.  It ignores most elements, tho.

>> Hmm... I wonder why mmm-save-local-variables is defined the way it is.
>> It doesn't seem to make use of `buffer-local-variables', even though it's
>> the most natural starting point.  E.g. there shouldn't be a need to list
>> explicitly all the c-* variables.  Maybe the author didn't know about
>> buffer-local-variables (or it didn't exist back then)?
> I'm not quite sure, but this way we only save the variables that are known
> to be safe in the multiple-mode context.

The way I see it, it's rather "this way we can only work for those major
modes we've been taught to handle".

I'm really interested in adding something like mmm-mode (or mumamo or
web-mode) into Emacs.  It'll have to start in GNU ELPA, I think, but in
any case before it can make it into Emacs it will have to be structured
such that new major modes can be handled without having to modify
mmm-mode.

> And also don't touch anything mmm-mode itself introduced.

This is the list that we can trivially know, so it's OK if we have to
handle these specially.

BTW, I really don't know yet what will work best.  Maybe the
"save&restore only those vars we've been told" as is currently done is
indeed the best option.  But in that case it shouldn't be done via
a variable listing those symbols, but e.g. by adding a special
property to those symbols.

> Now that I've checked, parsing the buffer into regions is definitely the
> slowest part (followed by fontification, which takes about 40% of the time
> in the current test example):

Does mmm-mode parse the buffer lazily?
Is fontification done "the jit-lock way"?


        Stefan

Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Cc: David Kastrup <dak@gnu.org>,  emacs-devel@gnu.org
> Date: Fri, 14 Feb 2014 16:08:42 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I agree that being able to support portions of a buffer that specify
> > their own major mode is an important feature.  I'm just saying that we
> > need a suitable infrastructure for that, instead of trying to exploit
> > unrelated features, which seem like "a good idea".
> 
> The first hint of a "suitable infrastructure" would be the adaptation of
> `syntax-ppss'. Where the information about region boundaries would come
> from, if not from mmm-mode? Text properties? Overlay properties? Where
> would they come from, if not from mmm-mode or similar package?

Sorry, I'm not sure I understand the questions.  What do you mean by
"adaptation of `syntax-ppss'"?

Information of region boundaries should surely come from the
application level, but I wasn't talking about that.  I was talking
about how this information is _used_.

> I guess adapting the default font-lock-fontify-region function and
> syntax-propertize to be aware of them would be good

To be aware of what?

> but that could be done later.

No, not "later", "sooner".  We need to design first and implement
later, not the other way around.

We need to define the list of mode-specific features (such as
indentation and fontification), and then design how these features
will be applied only to a region of the text.  One possibility would
be a special text property, whose value is the major mode in effect
for the text covered by that property.  (I'm not saying this is the
best idea, I'm just trying to explain what kind of infrastructure
would be needed.)

> > Yes, something like that.  Basically, somehow present to major-mode
> > features only a portion of the buffer (e.g., by narrowing internally).
> 
> This doest address primary mode regions, which a) need to see the
> previous chunks of the same mode, b) (for most functions) ignore the
> chunks of the submodes. But not for indentation: in some (!) mode
> combinations, the contents of submode regions influence indentation in
> the primary regions, namely in JSP, ERB and similar templates.

I never said the region(s) must be contiguous.  (Btw, explanation of
why a given chunk needs to be aware of the previous chunk would be
appreciated.)

But yes, all of what you mention, and more, need to be considered, and
the design should address these requirements.

My point is that we should come up with a list of the requirements,
and then design and implement the infrastructure which will support
them, instead of implementing multi-mode buffers by piggybacking
existing infrastructure, which was never designed to support such
features.

Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[Dmitry Gutov]

On 14.02.2014 16:28, Eli Zaretskii wrote:
> Sorry, I'm not sure I understand the questions.  What do you mean by
> "adaptation of `syntax-ppss'"?

The introduction of a new hook Stefan suggested earlier.

> Information of region boundaries should surely come from the
> application level, but I wasn't talking about that.  I was talking
> about how this information is _used_.
>
>> I guess adapting the default font-lock-fontify-region function and
>> syntax-propertize to be aware of them would be good
>
> To be aware of what?

Of chunk boundaries.

> I never said the region(s) must be contiguous.  (Btw, explanation of
> why a given chunk needs to be aware of the previous chunk would be
> appreciated.)

Consider this ERB snippet:

<div class="<% @div_class %>-42">foofoofoo</div>

The class value string has to continue after the ruby chunk.

> My point is that we should come up with a list of the requirements,
> and then design and implement the infrastructure which will support
> them, instead of implementing multi-mode buffers by piggybacking
> existing infrastructure, which was never designed to support such
> features.

The need to "come up, design and implement" these things has been there 
for many years now. Unless things start moving in that direction 
step-by-step, I'm not confident anything "proper" will happen at all.

Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Dmitry Gutov <dgutov@yandex.ru>,  emacs-devel@gnu.org
> Date: Fri, 14 Feb 2014 09:10:47 -0500
> 
> I've been following those multiple major mode thingies for quite a few
> years now.  I don't think we really need a significant new
> infrastructure at the C level.

Please explain why you think so.

My line of reasoning was that since fontification happens on the
display engine level, we need to have a way to call a different
fontification function for different portions of text.  And similarly
with indentation and other mode-specific behavior.

> What we need instead is some conventions that major modes need to
> follow to play well in things like mmm-mode or mumamo.

Like what?

> E.g. I don't think "pseudo-narrowing" to magically pretend that only the
> current "code chunk" exists is going to fly: it will break major modes
> that don't follow pretty much the same conventions.

Examples, please: which conventions are those?  There's almost no code
in Emacs that ignores the buffer restrictions (everything uses BEGV
and ZV).  Even the display engine itself does that.  The text outside
these restrictions simply does not exist, as far as Emacs is
concerned.

> As for whether multiple major modes are frequent: I consider comments
> and strings as being "text-mode" chunks.  Not sure if we'll ever get to
> the point of integrating the handling of strings/comments with the
> handling of other forms of multi-major-modes, tho.

Literate programming is impossible without multi-mode support.
Org-style notes and documents are another frequent use case.  And I'm
sure people will come with more.

So I think these are quite important, and we should support them
reasonably well, not through kludges.

Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[Eli Zaretskii]

> Date: Fri, 14 Feb 2014 16:34:45 +0200
> From: Dmitry Gutov <dgutov@yandex.ru>
> CC: dak@gnu.org, emacs-devel@gnu.org
> 
> > My point is that we should come up with a list of the requirements,
> > and then design and implement the infrastructure which will support
> > them, instead of implementing multi-mode buffers by piggybacking
> > existing infrastructure, which was never designed to support such
> > features.
> 
> The need to "come up, design and implement" these things has been there 
> for many years now. Unless things start moving in that direction 
> step-by-step, I'm not confident anything "proper" will happen at all.

Nothing will start moving unless Someone(TM) will move things.  I hope
volunteers will eventually come and do that, or else Emacs will
continue to be what it is today.  My chiming into this thread was in
the hope that what I say will inspire someone.

Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[Dmitry Gutov]

On 14.02.2014 16:23, Stefan Monnier wrote:
> It does iterate over the whole list, yes.  It ignores most elements, tho.

Hmm. Not looking at the C code, I figured access to buffer-local 
variables happens through the buffer structure, each time they're looked 
up or modified. Maybe this way is faster.

>>> Hmm... I wonder why mmm-save-local-variables is defined the way it is.
>>> It doesn't seem to make use of `buffer-local-variables', even though it's
>>> the most natural starting point.  E.g. there shouldn't be a need to list
>>> explicitly all the c-* variables.  Maybe the author didn't know about
>>> buffer-local-variables (or it didn't exist back then)?
>> I'm not quite sure, but this way we only save the variables that are known
>> to be safe in the multiple-mode context.
>
> The way I see it, it's rather "this way we can only work for those major
> modes we've been taught to handle".

Yes and no. An mmm-mode aware package can modify 
`mmm-save-local-variables', too.

> I'm really interested in adding something like mmm-mode (or mumamo or
> web-mode) into Emacs.

Last time I checked, web-mode wasn't in this category: it's a major mode 
with explicit support for some templating languages.

mumamo is an unsupported mess (sorry, Lennart).

mmm-mode would require a hunt for CA signatures, but it's not outside 
the realm of possibility.

multi-mode would be the most likely option, I believe (it's already 
copyright-assigned), but it'll either require major changes, or indirect 
buffers would have to be "fixed", which is not likely, judging by this 
conversation.

> It'll have to start in GNU ELPA, I think, but in
> any case before it can make it into Emacs it will have to be structured
> such that new major modes can be handled without having to modify
> mmm-mode.

I think you mean "can be handled without them being explicitly aware of 
mmm-mode", because modifying `mmm-save-local-variables' works from 
outside, too.

>> And also don't touch anything mmm-mode itself introduced.
>
> This is the list that we can trivially know, so it's OK if we have to
> handle these specially.

I haven't measured it, but wouldn't adding an (unless (memq var (list 
with 20 elements))) around processing each variable be, like, slow?

> BTW, I really don't know yet what will work best.  Maybe the
> "save&restore only those vars we've been told" as is currently done is
> indeed the best option.  But in that case it shouldn't be done via
> a variable listing those symbols, but e.g. by adding a special
> property to those symbols.

A special property would be neater, I guess. But then we'll really have 
to go though `buffer-local-variables'.

>> Now that I've checked, parsing the buffer into regions is definitely the
>> slowest part (followed by fontification, which takes about 40% of the time
>> in the current test example):
>
> Does mmm-mode parse the buffer lazily?

Nope (unlike mumamo and, sorta, multi-mode). I guess it would be the 
next major feature to implement there.

Originally the parsing was triggered only manually. I added an idle 
timer, but it's still a kludge.

> Is fontification done "the jit-lock way"?

Most of the time yes, but "reparsing" the buffer triggers the full 
refontification. There are no hard reasons why it can't be fixed, though.

Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[Dmitry Gutov]

Eli Zaretskii <eliz@gnu.org> writes:

>> The need to "come up, design and implement" these things has been there 
>> for many years now. Unless things start moving in that direction 
>> step-by-step, I'm not confident anything "proper" will happen at all.
>
> Nothing will start moving unless Someone(TM) will move things.

I'd expect each participant in the discussion to be willing to implement
things they discuss that are within their area of expertise. E.g. Stefan
to at least add that hook and modify the syntax-ppss caching behavior.

> I hope
> volunteers will eventually come and do that, or else Emacs will
> continue to be what it is today.  My chiming into this thread was in
> the hope that what I say will inspire someone.

I really don't do C, and learning it just to contribute to Emacs seems
like a major undertaking.

Likewise, I'm seeing a surge of users interested in writing Emacs Lisp.
Less of that in the Emacs core, but hopefully the transition will come,
too.

Why there aren't many new contributors capable and interested in
implementing new features at the C level, your guess is as good as mine
(probably better).

Re: /srv/bzr/emacs/trunk r101338:*lisp/emacs-lisp/syntax.el(syntax-ppss): More sanity check to catchtch

[E Sabof]

Dmitry Gutov <dgutov@yandex.ru> writes:

> The first hint of a "suitable infrastructure" would be the adaptation of
> `syntax-ppss'. Where the information about region boundaries would come
> from, if not from mmm-mode? Text properties? Overlay properties? Where
> would they come from, if not from mmm-mode or similar package?

I'm not familiar with mmm-mode, but somehow the last sentance strikes me
as wrong. I don't want to use (or be aware of as a user in any case) of
an mmm (or similar) mode. I want to use org-mode with snippets from
other languages. Or maybe php-mode with html/css/js snippets.
Or html-mode with js/css. (Recursively) embeddable (normal) modes seem
to fit better conceptually. I don't know if it would make a difference
implementationally.

Evgeni

Re: /srv/bzr/emacs/trunk r101338:*lisp/emacs-lisp/syntax.el(syntax-ppss): More sanity check to catchtch

[Dmitry Gutov]

E Sabof <esabof@gmail.com> writes:

> I'm not familiar with mmm-mode, but somehow the last sentance strikes me
> as wrong. I don't want to use (or be aware of as a user in any case) of
> an mmm (or similar) mode. I want to use org-mode with snippets from
> other languages.

For org-mode, org-babel would be the proper place to handle the chunk
boundaries, I believe.

> Or maybe php-mode with html/css/js snippets.
> Or html-mode with js/css.

How would html-mode know about js and css? Or, more generally, how would
Emacs know about *new* modes and their delimiters that need to be handled?

Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[David Kastrup]

Dmitry Gutov <dgutov@yandex.ru> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> The need to "come up, design and implement" these things has been
>>> there for many years now. Unless things start moving in that
>>> direction step-by-step, I'm not confident anything "proper" will
>>> happen at all.
>>
>> Nothing will start moving unless Someone(TM) will move things.
>
> I'd expect each participant in the discussion to be willing to
> implement things they discuss that are within their area of
> expertise. E.g. Stefan to at least add that hook and modify the
> syntax-ppss caching behavior.

That's a popular misconception: the dictatoriate of the incompetent.
Just because somebody went to the pains to learn an art does not mean
that he is bound to do your bidding.  Gifts are foremost gifts, not
punishment.

> I really don't do C, and learning it just to contribute to Emacs seems
> like a major undertaking.

Oh, you could help a lot of people and projects in your newly gained
area of expertise then.

> Why there aren't many new contributors capable and interested in
> implementing new features at the C level, your guess is as good as
> mine (probably better).

They probably don't like the obligations coming with it.

-- 
David Kastrup

Re: /srv/bzr/emacs/trunk r101338:*lisp/emacs-lisp/syntax.el(syntax-ppss): More sanity check to catchtch

[E Sabof]

Dmitry Gutov <dgutov@yandex.ru> writes:

> For org-mode, org-babel would be the proper place to handle the chunk
> boundaries, I believe.

Not sure how they handle it. But fontification is the only thing that is supported.

>> Or maybe php-mode with html/css/js snippets.
>> Or html-mode with js/css.
>
> How would html-mode know about js and css? Or, more generally, how would
> Emacs know about *new* modes and their delimiters that need to be handled?

The "parent" mode gives emacs a "child" region, and tells it which mode to use in it. Fontification could be handled this way. Additonal data might be needed for indentation. Potentally the parent mode might tell emacs to treat a number of regions "as one". _Maybe_, this way flymake could still work on embedded modes.

Evgeni

Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[Dmitry Gutov]

On 14.02.2014 17:30, David Kastrup wrote:
>> I'd expect each participant in the discussion to be willing to
>> implement things they discuss that are within their area of
>> expertise. E.g. Stefan to at least add that hook and modify the
>> syntax-ppss caching behavior.
>
> That's a popular misconception: the dictatoriate of the incompetent.
> Just because somebody went to the pains to learn an art does not mean
> that he is bound to do your bidding.  Gifts are foremost gifts, not
> punishment.

Replace "they discuss" in my quote above with "they suggest". It would 
be their own bidding, not mine. Or if the person is not interested in 
implementing said things, an explicit warning would be nice.

>> I really don't do C, and learning it just to contribute to Emacs seems
>> like a major undertaking.
>
> Oh, you could help a lot of people and projects in your newly gained
> area of expertise then.

Yeah, how much effort other people would have to spend correcting my 
mistakes before I gain said expertise? Not to mention the expenditure of 
time on my own part.

>> Why there aren't many new contributors capable and interested in
>> implementing new features at the C level, your guess is as good as
>> mine (probably better).
>
> They probably don't like the obligations coming with it.

I don't believe most even think that far.

Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[David Kastrup]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 14.02.2014 17:30, David Kastrup wrote:
>>> I'd expect each participant in the discussion to be willing to
>>> implement things they discuss that are within their area of
>>> expertise. E.g. Stefan to at least add that hook and modify the
>>> syntax-ppss caching behavior.
>>
>> That's a popular misconception: the dictatoriate of the incompetent.
>> Just because somebody went to the pains to learn an art does not mean
>> that he is bound to do your bidding.  Gifts are foremost gifts, not
>> punishment.
>
> Replace "they discuss" in my quote above with "they suggest".

Doesn't change a thing.

> It would be their own bidding, not mine. Or if the person is not
> interested in implementing said things, an explicit warning would be
> nice.

So let me warn you explicitly: just because I am willing to offer
suggestions and/or expertise does not and never imply that I offer
anything else.  And I consider it absurd to assume differently.  Not
just from me, from anyone.

>>> I really don't do C, and learning it just to contribute to Emacs
>>> seems like a major undertaking.
>>
>> Oh, you could help a lot of people and projects in your newly gained
>> area of expertise then.
>
> Yeah, how much effort other people would have to spend correcting my
> mistakes before I gain said expertise?

But then you would be able to help with and carry on their work.

> Not to mention the expenditure of time on my own part.

No art carries on without gaining new practitioners.

>>> Why there aren't many new contributors capable and interested in
>>> implementing new features at the C level, your guess is as good as
>>> mine (probably better).
>>
>> They probably don't like the obligations coming with it.
>
> I don't believe most even think that far.

Don't underestimate people.  "I don't know how to do it" for many people
is more a weapon rather than a shortcoming.

It's very popular in the context of Free Software, and many developers
have heard it more often than they really care for.  So it tends to be a
bad move.

-- 
David Kastrup

Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[Stefan Monnier]

> My line of reasoning was that since fontification happens on the
> display engine level, we need to have a way to call a different
> fontification function for different portions of text.  And similarly
> with indentation and other mode-specific behavior.

And all of those are done in Elisp, so there's no need for C-level changes.

>> What we need instead is some conventions that major modes need to
>> follow to play well in things like mmm-mode or mumamo.
> Like what?

Like "font-lock-keywords can only look before/after the provided region
boundaries by calling special functions".

> Examples, please: which conventions are those?  There's almost no code
> in Emacs that ignores the buffer restrictions (everything uses BEGV
> and ZV).

That's not the issue.  The issue is that code can misbehave if it is
prevented from seeing some of the "outer" text.  Narrowing is used
sufficiently rarely that we don't see it too often, but such problems do
occur already with narrowing.

> So I think these are quite important, and we should support them
> reasonably well, not through kludges.

We violently agree.


        Stefan

Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[Stefan Monnier]

>> The way I see it, it's rather "this way we can only work for those major
>> modes we've been taught to handle".
> Yes and no.  An mmm-mode aware package can modify
> mmm-save-local-variables', too.

Right.  In my experience, this doesn't work nearly as well as adding
symbol properties.

> Last time I checked, web-mode wasn't in this category: it's a major mode
> with explicit support for some templating languages.

That's right, it doesn't try to handle major modes generically, but it
does have to solve similar problems, just with the benefit of being able
to custom-make its major modes.

> mmm-mode would require a hunt for CA signatures, but it's not outside the
> realm of possibility.

The way I see it rather is to start over from scratch, based on the
experience gained with mmm-mode, but with a different view: instead of
trying to make it work without changing existing modes, assume that
the major modes will be changed to fit the framework.

The result should be much simpler/cleaner/shorter.

>>> And also don't touch anything mmm-mode itself introduced.
>> This is the list that we can trivially know, so it's OK if we have to
>> handle these specially.
> I haven't measured it, but wouldn't adding an (unless (memq var (list with
> 20 elements))) around processing each variable be, like, slow?

Could be.  (unless (get var 'mmm-foo) may be faster (mostly because the
plist is usually less than 20 elements), if needed.

I tend to assume that mode switches shouldn't be that super frequent to
be a real performance problem, and that if they need to be sped up,
there are ways to do that, e.g. by providing a few "subr", so I'm not
too worried about it.


        Stefan

RE: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[Drew Adams]

eli>> Emacs can use text properties to switch syntax tables or
eli>> categories or even keymaps in mid-buffer.  Maybe switching the
eli>> whole mode would be feasible in a similar manner.
sm> 
sm> Yes, something like that.  Basically, somehow present to major-mode
sm> features only a portion of the buffer (e.g., by narrowing
sm> internally).

and

eli> One possibility would be a special text property, whose value is
eli> the major mode in effect for the text covered by that property.
eli> (I'm not saying this is the best idea, I'm just trying to explain
eli> what kind of infrastructure would be needed.)

FWIW -

Lennart Borgman's MuMaMo multiple-major-mode code uses/used a text
property, `mumamo-major-mode', to distinguish zones mapped to particular
major modes.

I know very little about MuMaMo.  The reason I know about this property
is that my library `isearch-prop.el' takes this into account, to allow
isearching (only) the zones ("chunks") that are mapped to a particular
major mode.

The value of property `mumamo-major-mode' is handled as a special case
by `isearch-prop.el'.  Properties `face' and `font-lock-face' are also
handled specially by default, and customization allows special handling
of others. 

Normally, searching uses `equal' to check whether a property value
matches a value specified by the user.  For `face', `font-lock-face',
and `mumamo-major-mode', the matching check is not `equal'.

For the first two, text is searched that has a face property value
that includes any of the faces specified by the user.  For
`mumamo-major-mode', the user-specified major-mode name is matched
against the car of the `mumamo-major-mode' property value.

IOW, this is the default matching function:

(defun isearchp-property-default-match-fn (property)
  "Return the default match function for text or overlay PROPERTY.
Properties `face', `font-lock-face', and `mumamo-major-mode' are
handled specially.  For other properties the values are matched
using `equal'."
  (case property
    ((face font-lock-face)
     (lambda (val rprop)
       (if (consp rprop)
           (condition-case nil    ; Allow for dotted cons.
               (member val rprop)
             (error nil))
         (eq val rprop))))
    ((mumamo-major-mode)
     (lambda (val rprop) (equal val (car rprop))))
    (t
     #'equal)))

Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: dgutov@yandex.ru,  emacs-devel@gnu.org
> Date: Fri, 14 Feb 2014 11:53:48 -0500
> 
> > My line of reasoning was that since fontification happens on the
> > display engine level, we need to have a way to call a different
> > fontification function for different portions of text.  And similarly
> > with indentation and other mode-specific behavior.
> 
> And all of those are done in Elisp, so there's no need for C-level changes.

Maybe I'm missing something, but how do you define fontification
functions that are only in effect in a portion of a buffer?

> >> What we need instead is some conventions that major modes need to
> >> follow to play well in things like mmm-mode or mumamo.
> > Like what?
> 
> Like "font-lock-keywords can only look before/after the provided region
> boundaries by calling special functions".

And why would this not play well?

> > Examples, please: which conventions are those?  There's almost no code
> > in Emacs that ignores the buffer restrictions (everything uses BEGV
> > and ZV).
> 
> That's not the issue.  The issue is that code can misbehave if it is
> prevented from seeing some of the "outer" text.  Narrowing is used
> sufficiently rarely that we don't see it too often, but such problems do
> occur already with narrowing.

We are not talking about narrowing to an arbitrary portion of text.
We are talking about narrowing to a portion that presents a
syntactically complete snippet of code.

Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Cc: dak@gnu.org,  emacs-devel@gnu.org
> Date: Fri, 14 Feb 2014 17:15:33 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> The need to "come up, design and implement" these things has been there 
> >> for many years now. Unless things start moving in that direction 
> >> step-by-step, I'm not confident anything "proper" will happen at all.
> >
> > Nothing will start moving unless Someone(TM) will move things.
> 
> I'd expect each participant in the discussion to be willing to implement
> things they discuss that are within their area of expertise. E.g. Stefan
> to at least add that hook and modify the syntax-ppss caching behavior.

I think your expectations are a bit exaggerated.  You cannot possibly
expect of anyone who offers some idea to implement it.  People
implement ideas because they have an itch to scratch, not because they
just had the idea.

> > I hope
> > volunteers will eventually come and do that, or else Emacs will
> > continue to be what it is today.  My chiming into this thread was in
> > the hope that what I say will inspire someone.
> 
> I really don't do C, and learning it just to contribute to Emacs seems
> like a major undertaking.

First, C is a really simple language, and compilers nowadays are good
at diagnosing mistakes.

But even if you decide not to go that way, there are a lot of place to
contribute to the design and the Lisp portions of the implementation.
Even just formulating the requirements is a huge step forward, IMO.

> Likewise, I'm seeing a surge of users interested in writing Emacs Lisp.
> Less of that in the Emacs core, but hopefully the transition will come,
> too.
> 
> Why there aren't many new contributors capable and interested in
> implementing new features at the C level, your guess is as good as mine
> (probably better).

Actually, I don't see many new contributors that do it in Lisp,
either.  Sure, there's a lot of code being committed every day, but
awfully few features that really advance forward Emacs as the
programming environment.  E.g., witness the lack of any significant
progress in adding IDE features.

Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[Eli Zaretskii]

> Date: Fri, 14 Feb 2014 17:38:45 +0200
> From: Dmitry Gutov <dgutov@yandex.ru>
> CC: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> 
> >> I really don't do C, and learning it just to contribute to Emacs seems
> >> like a major undertaking.
> >
> > Oh, you could help a lot of people and projects in your newly gained
> > area of expertise then.
> 
> Yeah, how much effort other people would have to spend correcting my 
> mistakes before I gain said expertise?

Ideally, none: you will correct them yourself, when they are
discovered and reported.

> Not to mention the expenditure of time on my own part.

That, you must commit to in advance, of course.  But why would anyone
expect to make any significant progress in any endeavor without
investing time and energy?

Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[Stefan Monnier]

> Maybe I'm missing something, but how do you define fontification
> functions that are only in effect in a portion of a buffer?

Here's one way:
From the C code's point of view, fontification-functions will always
point to the one and only jit-lock-function, and then jit-lock will call
font-lock which will go through font-lock-fontify-region-function will
be hooked by mmm-mode to dispatch to use the proper font-lock-keywords
depending on the major mode in that region.

We could also provide more direct support either directly in font-lock
or directly in jit-lock, but the existing hooks might be sufficient.

>> >> What we need instead is some conventions that major modes need to
>> >> follow to play well in things like mmm-mode or mumamo.
>> > Like what?
>> Like "font-lock-keywords can only look before/after the provided region
>> boundaries by calling special functions".
> And why would this not play well?

Not sure I understand the question.  If the major modes don't obey this
rule, they may end up looking at code which is written in another
language and hence go them all confused.

> We are not talking about narrowing to an arbitrary portion of text.
> We are talking about narrowing to a portion that presents a
> syntactically complete snippet of code.

Not necessarily.  As mentioned earlier, it may have to start indentation
at a non-0 column, for example.

Or, if you think of a comment as a "chunk" in another major mode, then
you'll want to be able to jump from a brace to the matching other end,
even if there's a comment between the two, so you'll have to jump over
that "chunk" somehow, but "narrowing" won't do it, since it would have
to "hide" something inside rather than something outside.

Anyway, if you can hash out a design and implement it, go for it.


        Stefan

Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[Michael Welsh Duggan]

Eli Zaretskii <eliz@gnu.org> writes:

> My point is that we should come up with a list of the requirements,
> and then design and implement the infrastructure which will support
> them, instead of implementing multi-mode buffers by piggybacking
> existing infrastructure, which was never designed to support such
> features.

Although I might be setting the cat among the pigeons, I thought I'd
throw out an example of a possible infrastructure change along these
lines.

What if we could have a display property (or something like a display
property) that, rather than displaying a string or an image, displays
(a portion of?) another buffer.  Moreover, when the cursor is in this
region, all events go to the subsidiary buffer rather than the displayed
one.  Changes are actually happening in another buffer, with full mode
and font-lock support, but those changes are visibly mirrored in the
"display" buffer.  Write hooks (and copy hooks) can replace the dislpay
regions with the actual contents of the subsidiary buffers.

Now, I am certain there are a lot of potential problems with this idea
(I can think of several off the top of my head), but this is, once
again, an example of a proposed feature that might be able to solve this
type of problem.

-- 
Michael Welsh Duggan
(md5i@md5i.com)

Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[Eli Zaretskii]

> From: Michael Welsh Duggan <mwd@md5i.com>
> Date: Sat, 15 Feb 2014 00:52:58 -0500
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > My point is that we should come up with a list of the requirements,
> > and then design and implement the infrastructure which will support
> > them, instead of implementing multi-mode buffers by piggybacking
> > existing infrastructure, which was never designed to support such
> > features.
> 
> Although I might be setting the cat among the pigeons, I thought I'd
> throw out an example of a possible infrastructure change along these
> lines.
> 
> What if we could have a display property (or something like a display
> property) that, rather than displaying a string or an image, displays
> (a portion of?) another buffer.

That's implementation, not requirements (I'm sure you know that).
While it is possible that it will allow what is needed, specifying
implementation first does not allow to come up with alternative
implementation ideas that satisfy the same requirements.  E.g., what
you suggest would require a serious surgery on the design of the
display engine, and the question remains whether such a surgery is
indeed the best or the only way to go.  (Personally, I hope not.)

So I really think we should define the requirements first.

Emacs contributions, C and Lisp (was: Re: /srv/bzr/emacs/trunk r101338: ...)

[Dmitry Gutov]

On 14.02.2014 20:30, Eli Zaretskii wrote:
> I think your expectations are a bit exaggerated.  You cannot possibly
> expect of anyone who offers some idea to implement it.  People
> implement ideas because they have an itch to scratch, not because they
> just had the idea.

Okay, never mind. Guess my phrasing was too broad.

> First, C is a really simple language, and compilers nowadays are good
> at diagnosing mistakes.

People like to say that, but C being simple and having a weak typing 
system just means one has to learn more things beyond the language 
syntax itself.

> But even if you decide not to go that way, there are a lot of place to
> contribute to the design and the Lisp portions of the implementation.
> Even just formulating the requirements is a huge step forward, IMO.

Most of the requirements for multiple mode support have been discussed 
over the years, and the parent discussion has some details that make the 
picture clearer, at least for me. If you think that's insufficient, 
please feel free to ask questions, or suggest another format for 
recording the requirements.

> Actually, I don't see many new contributors that do it in Lisp,
> either.  Sure, there's a lot of code being committed every day, but
> awfully few features that really advance forward Emacs as the
> programming environment.

You may want to look at the Emacs community at large, not limited to the 
core. There are a lot of packages created and added to MELPA, every week.

> E.g., witness the lack of any significant
> progress in adding IDE features.

I guess that depends on what features you expect.

It's true that CEDET hasn't seen a lot of progress feature-wise lately, 
but that's not very surprising: it's complex, it's relatively hard to 
set up for a novice, it has a weird separation of extra features between 
the versions in Emacs and its own trunk, and it's not really suitable 
for dynamic languages, or languages with type inference. AFAIK, that is.

As far as code completion interface goes, Company development continues.

Despite certain expectations that everything is better when written in 
Emacs Lisp, context-dependent code completion and documentation display 
support is usually delegated to an external process, and there are 
relatively new packages that use editor-agnostic services: Jedi for 
Python, nREPL for Clojure, Tern for JavaScript, OmniSharp for C#.

There are even several packages that provide support for C/C++ code 
completion using Clang or libclang. They are older, though, than the 
ones mentioned above.

One feature that has seen less attention is refactoring, but there 
already are a few packages providing simplistic (and some, even more 
complex) refactorings: js2-refactor, ruby-refactor, clj-refactor, traad 
and others.

What other features are missing? Class diagrams?

Re: /srv/bzr/emacs/trunk r101338:*lisp/emacs-lisp/syntax.el(syntax-ppss): More sanity check to catchtch

[Dmitry Gutov]

On 14.02.2014 17:38, E Sabof wrote:
>> For org-mode, org-babel would be the proper place to handle the chunk
>> boundaries, I believe.
>
> Not sure how they handle it. But fontification is the only thing that is supported.

They probably use a similar approach to mmm-mode and haml-mode: define a 
fontify-region-function that calls each subregion function's in turn.

>> How would html-mode know about js and css? Or, more generally, how would
>> Emacs know about *new* modes and their delimiters that need to be handled?
>
> The "parent" mode gives emacs a "child" region, and tells it which mode to use in it.

In other words, the parent mode suddenly has to become multiple-mode 
aware. So html-mode, for example, wouldn't be suitable anymore.

And if we want to make the multiple-mode functionality somehow reusable, 
extracting it to some common framework might be in order. You see where 
this is going.

Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[Dmitry Gutov]

On 14.02.2014 19:08, Stefan Monnier wrote:
>> Last time I checked, web-mode wasn't in this category: it's a major mode
>> with explicit support for some templating languages.
>
> That's right, it doesn't try to handle major modes generically, but it
> does have to solve similar problems, just with the benefit of being able
> to custom-make its major modes.

I'm not sure what you mean. web-mode has only one major mode, and its 
support of multiple template engines looks like this in the code:

(defun web-mode-scan-blocks (reg-beg reg-end)
...
         (cond
          ((string= web-mode-engine "php")
...
          ((string= web-mode-engine "django")
...
          ((string= web-mode-engine "erb")
...

and so on.

>> mmm-mode would require a hunt for CA signatures, but it's not outside the
>> realm of possibility.
>
> The way I see it rather is to start over from scratch, based on the
> experience gained with mmm-mode, but with a different view: instead of
> trying to make it work without changing existing modes, assume that
> the major modes will be changed to fit the framework.
>
> The result should be much simpler/cleaner/shorter.

The implementation of region classes definitions and parsing of the 
buffer could still be reused with this approach. Although, if the goal 
is to parse the buffer lazily, maybe adapting the relevant multi-mode 
code instead would be easier.

> I tend to assume that mode switches shouldn't be that super frequent to
> be a real performance problem, and that if they need to be sped up,
> there are ways to do that, e.g. by providing a few "subr", so I'm not
> too worried about it.

Ok.

Re: /srv/bzr/emacs/trunk r101338:*lisp/emacs-lisp/syntax.el(syntax-ppss): More sanity check to catchtch

[E Sabof]

Dmitry Gutov <dgutov@yandex.ru> writes:

> In other words, the parent mode suddenly has to become multiple-mode
> aware. So html-mode, for example, wouldn't be suitable anymore.

html-mode is part of Emacs. Extending it shouldn't be (socially) hard.

> And if we want to make the multiple-mode functionality somehow reusable,
> extracting it to some common framework might be in order. You see where
> this is going.

I had in mind something along the lines of (set-region-mode MODE BEGINNING END), provided by emacs infrastructure. Does it somehow go wrong from there? That's not unlikely, but I can't think of something specific.

Evgeni

Re: /srv/bzr/emacs/trunk r101338:*lisp/emacs-lisp/syntax.el(syntax-ppss): More sanity check to catchtch

[Dmitry Gutov]

E Sabof <esabof@gmail.com> writes:

>> In other words, the parent mode suddenly has to become multiple-mode
>> aware. So html-mode, for example, wouldn't be suitable anymore.
>
> html-mode is part of Emacs. Extending it shouldn't be (socially) hard.

Well, suppose Emacs just provides a derived mode for every templating
language we want to support (some deriving from html-mode others not).

> I had in mind something along the lines of (set-region-mode MODE
> BEGINNING END), provided by emacs infrastructure. Does it somehow go
> wrong from there? That's not unlikely, but I can't think of something
> specific.

Until you define what such a function would do, it's hard for things to
go wrong.

But let's just look at things in motion. What would make a major mode
call this function? It would have to know about a set of template
delimiters, right? And somehow translate them into `set-region-mode'
calls?

Are BEGINNING and END buffer positions? What happens if the user inserts
some text in the buffer? Will all regions below suddenly become invalid?

Okay, let's make them markers. What happens when the user removes a
region? Or just one delimiter? Or adds a new delimiter, or region?

Re: Emacs contributions, C and Lisp (was: Re: /srv/bzr/emacs/trunk r101338: ...)

[Eli Zaretskii]

> Date: Sun, 16 Feb 2014 03:47:06 +0200
> From: Dmitry Gutov <dgutov@yandex.ru>
> CC: dak@gnu.org, emacs-devel@gnu.org
> 
> > First, C is a really simple language, and compilers nowadays are good
> > at diagnosing mistakes.
> 
> People like to say that, but C being simple and having a weak typing 
> system just means one has to learn more things beyond the language 
> syntax itself.

Not sure what you mean.  Also, how is this different from Lisp?

> > But even if you decide not to go that way, there are a lot of place to
> > contribute to the design and the Lisp portions of the implementation.
> > Even just formulating the requirements is a huge step forward, IMO.
> 
> Most of the requirements for multiple mode support have been discussed 
> over the years, and the parent discussion has some details that make the 
> picture clearer, at least for me. If you think that's insufficient, 
> please feel free to ask questions, or suggest another format for 
> recording the requirements.

What I would suggest is to collect the requirements in a single list,
and file a wishlist bug report with them.  I wouldn't expect
interested people, if there are any, to glean the requirements from
that longish discussion.

> > Actually, I don't see many new contributors that do it in Lisp,
> > either.  Sure, there's a lot of code being committed every day, but
> > awfully few features that really advance forward Emacs as the
> > programming environment.
> 
> You may want to look at the Emacs community at large, not limited to the 
> core. There are a lot of packages created and added to MELPA, every week.

Yes, everybody plays with their favorite toys.  But how does this
advance Emacs, and do we even have a roadmap for where we want it to
go?

> > E.g., witness the lack of any significant
> > progress in adding IDE features.
> 
> I guess that depends on what features you expect.

There was a discussion about this some time ago, although I cannot
find it now.  But how about starting with those GUI code folding
decorations like every IDE nowadays has, while Emacs doesn't?  (ECB
comes close, but why shouldn't an Emacs user have that out of the box,
especially when Speedbar does something very similar for ages?)

> It's true that CEDET hasn't seen a lot of progress feature-wise lately, 
> but that's not very surprising: it's complex, it's relatively hard to 
> set up for a novice, it has a weird separation of extra features between 
> the versions in Emacs and its own trunk, and it's not really suitable 
> for dynamic languages, or languages with type inference. AFAIK, that is.

Then the immediate task would be to make it simpler to set up and
suitable for those languages.  _That_ would really move Emacs forward,
towards a point where, hopefully, it will once again be relevant for
modern development paradigm and provide what programmers expect.

> As far as code completion interface goes, Company development continues.

I hope this will some day be bundled with Emacs.  (I also hope we
rename it to something more palatable before that happens, because
having newbies learn that completion-related features are called
"company-SOMETHING" is voluntarily falling into the same trap as with
"kill/yank" vs "cut/paste" and "frame" vs "window", except that this
time we have no excuses.)

> Despite certain expectations that everything is better when written in 
> Emacs Lisp, context-dependent code completion and documentation display 
> support is usually delegated to an external process, and there are 
> relatively new packages that use editor-agnostic services: Jedi for 
> Python, nREPL for Clojure, Tern for JavaScript, OmniSharp for C#.

Where's their integration with Emacs, though?

> There are even several packages that provide support for C/C++ code 
> completion using Clang or libclang. They are older, though, than the 
> ones mentioned above.

And where's _their_ integration with Emacs?

> One feature that has seen less attention is refactoring, but there 
> already are a few packages providing simplistic (and some, even more 
> complex) refactorings: js2-refactor, ruby-refactor, clj-refactor, traad 
> and others.

Yes, refactoring is very important, and we should have it before we
can call ourselves IDE.

> What other features are missing? Class diagrams?

Yes, that too.

Re: Emacs contributions, C and Lisp (was: Re: /srv/bzr/emacs/trunk r101338: ...)

[Jorgen Schaefer]

On Sun, 16 Feb 2014 18:45:36 +0200
Eli Zaretskii <eliz@gnu.org> wrote:

> But how about starting with those GUI code folding
> decorations like every IDE nowadays has, while Emacs doesn't?  (ECB
> comes close, but why shouldn't an Emacs user have that out of the box,
> especially when Speedbar does something very similar for ages?)

Assuming you mean something like this:

http://pythonthusiast.pythonblogs.com/gallery/230/pycharm2.PNG

Yes please.

The whole code folding infrastructure in Emacs needs work, really. The
problem, as often with Emacs, is not that it wouldn't exist at all, but
that there are two dozen solutions that all have different
shortcomings.
 
> > Despite certain expectations that everything is better when written
> > in Emacs Lisp, context-dependent code completion and documentation
> > display support is usually delegated to an external process, and
> > there are relatively new packages that use editor-agnostic
> > services: Jedi for Python, nREPL for Clojure, Tern for JavaScript,
> > OmniSharp for C#.
> 
> Where's their integration with Emacs, though?

Jedi: https://github.com/tkf/emacs-jedi
Jedi and Rope: https://github.com/jorgenschaefer/elpy
nrepl: https://github.com/technomancy/nrepl.el (and others)
tern: https://github.com/marijnh/tern/tree/master/emacs (and others)
OmniSharp: https://github.com/sp3ctum/omnisharp-emacs (apparently)

Some of them have integration with company, too.

As I said in the thread you mentioned, the problem is not that these
extensions wouldn't exist for Emacs, it's that each and every one of
them reinvents most of the wheel to get similar features to Emacs.

> > There are even several packages that provide support for C/C++ code 
> > completion using Clang or libclang. They are older, though, than
> > the ones mentioned above.
> 
> And where's _their_ integration with Emacs?

The integration was declined on this list because Emacs should prefer
gcc over clang to protect user's freedoms, and gcc does not provide such
features.

> > One feature that has seen less attention is refactoring, but there 
> > already are a few packages providing simplistic (and some, even
> > more complex) refactorings: js2-refactor, ruby-refactor,
> > clj-refactor, traad and others.
> 
> Yes, refactoring is very important, and we should have it before we
> can call ourselves IDE.

Various Rope-based extensions for Emacs (ropemacs, elpy) offer access to
Rope's Python refactoring tools. It's still short of "real"
integration, though, but the first steps are there.

Regards,
Jorgen

Re: Emacs contributions, C and Lisp

[Dmitry Gutov]

On 16.02.2014 18:45, Eli Zaretskii wrote:
>> People like to say that, but C being simple and having a weak typing
>> system just means one has to learn more things beyond the language
>> syntax itself.
>
> Not sure what you mean.  Also, how is this different from Lisp?

Manual memory management? And when I'm writing Lisp, I usually don't 
have to worry about standard library including both modern and obsolete 
(and unsecure!) functions doing the same thing, working with baroque 
build system and not breaking things on different (often old) compilers 
and platforms I can't test.

> What I would suggest is to collect the requirements in a single list,
> and file a wishlist bug report with them.  I wouldn't expect
> interested people, if there are any, to glean the requirements from
> that longish discussion.

Ok, I'll try to get around to it in the near future.

>> You may want to look at the Emacs community at large, not limited to the
>> core. There are a lot of packages created and added to MELPA, every week.
>
> Yes, everybody plays with their favorite toys.  But how does this
> advance Emacs

They also share their toys. And many people rely on certain "toys" for 
their work.

 > and do we even have a roadmap for where we want it to go?

That's a rhetorical question, isn't it?

> There was a discussion about this some time ago, although I cannot
> find it now.  But how about starting with those GUI code folding
> decorations like every IDE nowadays has, while Emacs doesn't?

hs-minor-mode? outline-minor-mode? I think CEDET also includes something 
similar.

I dislike code folding, so this is not an area I examined thoroughly.

> (ECB
> comes close, but why shouldn't an Emacs user have that out of the box,
> especially when Speedbar does something very similar for ages?)

Speedbar is a part of Emacs, isn't it?

>> It's true that CEDET hasn't seen a lot of progress feature-wise lately,
>> but that's not very surprising: it's complex, it's relatively hard to
>> set up for a novice, it has a weird separation of extra features between
>> the versions in Emacs and its own trunk, and it's not really suitable
>> for dynamic languages, or languages with type inference. AFAIK, that is.
>
> Then the immediate task would be to make it simpler to set up and

Maybe someone who programs in languages it supports well (such as 
yourself?) can write up a short "quickstart" text, or at least formulate 
the requirements for one.

http://cedet.sourceforge.net/setup.shtml is vastly outdated.

http://alexott.net/en/writings/emacs-devenv/EmacsCedet.html seems 
comprehensive, but it's quite long.

> suitable for those languages.

I don't really know, but this may be nearly impossible given the current 
CEDET architecture.

>> As far as code completion interface goes, Company development continues.
>
> I hope this will some day be bundled with Emacs.

Maybe it will. But not everything has to be bundled with Emacs.

It's in GNU ELPA already, and as long as major modes in Emacs provide 
the relevant information, it may be good enough.

I'd much rather have a lean Emacs distribution and an easily accessible 
repository of packages each user can pick and install for themselves.

For most of Emacs enthusiasts I know, Emacs is a DIY kind of tool.

> (I also hope we
> rename it to something more palatable before that happens, because
> having newbies learn that completion-related features are called
> "company-SOMETHING" is voluntarily falling into the same trap as with
> "kill/yank" vs "cut/paste" and "frame" vs "window", except that this
> time we have no excuses.)

Yeah, we probably will.

>> Despite certain expectations that everything is better when written in
>> Emacs Lisp, context-dependent code completion and documentation display
>> support is usually delegated to an external process, and there are
>> relatively new packages that use editor-agnostic services: Jedi for
>> Python, nREPL for Clojure, Tern for JavaScript, OmniSharp for C#.
>
> Where's their integration with Emacs, though?

In addition to Jorgen's list:

https://github.com/proofit404/company-jedi/
https://github.com/clojure-emacs/company-cider
https://github.com/proofit404/company-tern

>> There are even several packages that provide support for C/C++ code
>> completion using Clang or libclang. They are older, though, than the
>> ones mentioned above.
>
> And where's _their_ integration with Emacs?

When I say "packages" here, I mean Emacs packages.

Take a look at company-clang in GNU ELPA, for example. Apparently, 
that's as close as Clang is ever going to get to Emacs.

> Yes, refactoring is very important, and we should have it before we
> can call ourselves IDE.

Here, some common infrastructure might be useful indeed.

Although I'd start with a rename/replace-in-project feature that doesn't 
involve grep, find and dired. Or at least doesn't involve them explicitly.

And a similar feature for mass-rename in filenames across the project.

(Replace "project" with "directory" if you will, the directory itself 
could be prompted for. Explicit support for projects is less important.)

>> What other features are missing? Class diagrams?
>
> Yes, that too.

Again, this is something left up to external tools to implement. I'm not 
sure if Jedi or OmniSharp implement an editor-agnostic interfaces for 
this feature, but probably not yet.

Re: Emacs contributions, C and Lisp (was: Re: /srv/bzr/emacs/trunk r101338: ...)

[Eli Zaretskii]

[-- Attachment #1: Type: text/plain, Size: 621 bytes --]

> Date: Sun, 16 Feb 2014 18:07:12 +0100
> From: Jorgen Schaefer <forcer@forcix.cx>
> Cc: Dmitry Gutov <dgutov@yandex.ru>, emacs-devel@gnu.org
> 
> On Sun, 16 Feb 2014 18:45:36 +0200
> Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > But how about starting with those GUI code folding
> > decorations like every IDE nowadays has, while Emacs doesn't?  (ECB
> > comes close, but why shouldn't an Emacs user have that out of the box,
> > especially when Speedbar does something very similar for ages?)
> 
> Assuming you mean something like this:
> 
> http://pythonthusiast.pythonblogs.com/gallery/230/pycharm2.PNG

Or like this:


[-- Attachment #2: VS_Outlining.png --]
[-- Type: image/png, Size: 19147 bytes --]

[-- Attachment #3: Type: text/plain, Size: 2524 bytes --]


> The whole code folding infrastructure in Emacs needs work, really. The
> problem, as often with Emacs, is not that it wouldn't exist at all, but
> that there are two dozen solutions that all have different
> shortcomings.

Then here's a clear bunch of todo items, right?  I hope someone will
volunteer.

> > > Despite certain expectations that everything is better when written
> > > in Emacs Lisp, context-dependent code completion and documentation
> > > display support is usually delegated to an external process, and
> > > there are relatively new packages that use editor-agnostic
> > > services: Jedi for Python, nREPL for Clojure, Tern for JavaScript,
> > > OmniSharp for C#.
> > 
> > Where's their integration with Emacs, though?
> 
> Jedi: https://github.com/tkf/emacs-jedi
> Jedi and Rope: https://github.com/jorgenschaefer/elpy
> nrepl: https://github.com/technomancy/nrepl.el (and others)
> tern: https://github.com/marijnh/tern/tree/master/emacs (and others)
> OmniSharp: https://github.com/sp3ctum/omnisharp-emacs (apparently)

Why aren't they part of Emacs?

> As I said in the thread you mentioned, the problem is not that these
> extensions wouldn't exist for Emacs, it's that each and every one of
> them reinvents most of the wheel to get similar features to Emacs.

Bringing them into Emacs bundle is a significant first step towards
solving the deficiencies, since Emacs maintainers will work with the
authors to fix them, and will continue maintaining them through the
years.  If they are left outside, they will continue being on the
fringe.

> > > There are even several packages that provide support for C/C++ code 
> > > completion using Clang or libclang. They are older, though, than
> > > the ones mentioned above.
> > 
> > And where's _their_ integration with Emacs?
> 
> The integration was declined on this list because Emacs should prefer
> gcc over clang to protect user's freedoms, and gcc does not provide such
> features.

GCC supports plugins, so it should be possible to do the same.

Or start with CEDET and Semantic, as was already suggested several
times.

In any case, clinging with clang clearly makes no sense, if we want
these features in Emacs.

> > Yes, refactoring is very important, and we should have it before we
> > can call ourselves IDE.
> 
> Various Rope-based extensions for Emacs (ropemacs, elpy) offer access to
> Rope's Python refactoring tools. It's still short of "real"
> integration, though, but the first steps are there.

Bring those into Emacs, please.

Re: /srv/bzr/emacs/trunk r101338:*lisp/emacs-lisp/syntax.el(syntax-ppss): More sanity check to catchtch

[E Sabof]

Dmitry Gutov <dgutov@yandex.ru> writes:

> Until you define what such a function would do, it's hard for things to
> go wrong.

I should probably get re-acquainted with mmm. It might contain useful functionality that I haven't considered.

From personal experience of dealing with multi-lingual files, I'd say I'd like fontification, language-sensitive navigation and indentation. I suppose some keys might change as well, but I'not convinced that the benefits outweigh the additional complexity.

It might be possible to achieve consistent navigation by making functions such as beginning-of-defun consult a region property before consulting the buffer-local beginning-of-defun-function.

I might also like error-checking, but it's conceivable to move this responsibility to flymake.

>
> But let's just look at things in motion. What would make a major mode
> call this function? It would have to know about a set of template
> delimiters, right? And somehow translate them into `set-region-mode'
> calls?
>
> Are BEGINNING and END buffer positions? What happens if the user inserts
> some text in the buffer? Will all regions below suddenly become invalid?
>
> Okay, let's make them markers. What happens when the user removes a
> region? Or just one delimiter? Or adds a new delimiter, or region?

This requires a parsing loop, which I'd guess is how mmm handles it.

Evgeni

Re: Emacs contributions, C and Lisp

[Eli Zaretskii]

> Date: Sun, 16 Feb 2014 19:27:56 +0200
> From: Dmitry Gutov <dgutov@yandex.ru>
> CC: dak@gnu.org, emacs-devel@gnu.org
> 
> On 16.02.2014 18:45, Eli Zaretskii wrote:
> >> People like to say that, but C being simple and having a weak typing
> >> system just means one has to learn more things beyond the language
> >> syntax itself.
> >
> > Not sure what you mean.  Also, how is this different from Lisp?
> 
> Manual memory management?

Mostly a red herring in Emacs's C layers, unless you are inventing a
new Lisp data type or mess with GC.

> And when I'm writing Lisp, I usually don't 
> have to worry about standard library including both modern and obsolete 
> (and unsecure!) functions doing the same thing, working with baroque 
> build system and not breaking things on different (often old) compilers 
> and platforms I can't test.

These problems exist in Lisp as they do (or don't) in C, as far as
Emacs is concerned.  E.g., see the latest discussion of crypto
features.

> > What I would suggest is to collect the requirements in a single list,
> > and file a wishlist bug report with them.  I wouldn't expect
> > interested people, if there are any, to glean the requirements from
> > that longish discussion.
> 
> Ok, I'll try to get around to it in the near future.

Thanks!

> >> You may want to look at the Emacs community at large, not limited to the
> >> core. There are a lot of packages created and added to MELPA, every week.
> >
> > Yes, everybody plays with their favorite toys.  But how does this
> > advance Emacs
> 
> They also share their toys. And many people rely on certain "toys" for 
> their work.

Of course.  And that's OK.  I'm just saying that advancing Emacs needs
more than that.

>  > and do we even have a roadmap for where we want it to go?
> 
> That's a rhetorical question, isn't it?

I guess so.

> > There was a discussion about this some time ago, although I cannot
> > find it now.  But how about starting with those GUI code folding
> > decorations like every IDE nowadays has, while Emacs doesn't?
> 
> hs-minor-mode? outline-minor-mode? I think CEDET also includes something 
> similar.

See the other mail for what I (and Jorgen) had in mind.  I don't
think we have anything like that in Emacs, even with CEDET.  ECB
(which isn't bundled) is the only thing that comes close, but IMO we
should have had this in Emacs core for a long time.

> I dislike code folding, so this is not an area I examined thoroughly.

I don't use it much myself, but it can be invaluable when studying the
structure of a large and complex source file.

> > (ECB
> > comes close, but why shouldn't an Emacs user have that out of the box,
> > especially when Speedbar does something very similar for ages?)
> 
> Speedbar is a part of Emacs, isn't it?

Yes, it is.  But it only does this with lists of files and tags, not
with source buffers.

> >> It's true that CEDET hasn't seen a lot of progress feature-wise lately,
> >> but that's not very surprising: it's complex, it's relatively hard to
> >> set up for a novice, it has a weird separation of extra features between
> >> the versions in Emacs and its own trunk, and it's not really suitable
> >> for dynamic languages, or languages with type inference. AFAIK, that is.
> >
> > Then the immediate task would be to make it simpler to set up and
> 
> Maybe someone who programs in languages it supports well (such as 
> yourself?) can write up a short "quickstart" text, or at least formulate 
> the requirements for one.

I certainly hope so.

> > suitable for those languages.
> 
> I don't really know, but this may be nearly impossible given the current 
> CEDET architecture.

I certainly hope not, but I don't know enough about CEDET.

> >> As far as code completion interface goes, Company development continues.
> >
> > I hope this will some day be bundled with Emacs.
> 
> Maybe it will. But not everything has to be bundled with Emacs.

Things that we consider central and important should be.

> >> What other features are missing? Class diagrams?
> >
> > Yes, that too.
> 
> Again, this is something left up to external tools to implement. I'm not 
> sure if Jedi or OmniSharp implement an editor-agnostic interfaces for 
> this feature, but probably not yet.

CEDET has COGRE that supports this already, at least according to
http://cedet.sourceforge.net/ (search for "UML").  Somewhat clunky and
based on "ASCII art", though.

Re: Emacs contributions, C and Lisp

[Dmitry Gutov]

On 16.02.2014 19:50, Eli Zaretskii wrote:
> These problems exist in Lisp as they do (or don't) in C, as far as
> Emacs is concerned.  E.g., see the latest discussion of crypto
> features.

I'd say in Lisp they exist to a considerably lesser degree. And if 
someone is going to implement crypto support, I'd expect them to be 
familiar with C anyway.

> See the other mail for what I (and Jorgen) had in mind.  I don't
> think we have anything like that in Emacs, even with CEDET.  ECB
> (which isn't bundled) is the only thing that comes close, but IMO we
> should have had this in Emacs core for a long time.

This looks like something that might require fancier rendering on the 
fringe or a nearby area.

Layered rendering on the fringe is something we could use in diff-hl as 
well: https://github.com/dgutov/diff-hl/issues/16

>> Maybe someone who programs in languages it supports well (such as
>> yourself?) can write up a short "quickstart" text, or at least formulate
>> the requirements for one.
>
> I certainly hope so.

*chuckle*

>> not everything has to be bundled with Emacs.
>
> Things that we consider central and important should be.

Guess I agree that company-the-framework would work better bundled.

Support for different languages might come separately, though.

> CEDET has COGRE that supports this already, at least according to
> http://cedet.sourceforge.net/ (search for "UML").  Somewhat clunky and
> based on "ASCII art", though.

Yes, COGRE could be the instrument, but we still need information to use 
it on, and language-specific tools are most likely the way to go.

Re: /srv/bzr/emacs/trunk r101338:*lisp/emacs-lisp/syntax.el(syntax-ppss): More sanity check to catchtch

[Dmitry Gutov]

On 16.02.2014 19:48, E Sabof wrote:
> I suppose some keys might change as well, but I'not convinced that the benefits outweigh the additional complexity.

Changing local maps based on regions is fairly easy.

> It might be possible to achieve consistent navigation by making functions such as beginning-of-defun consult a region property before consulting the buffer-local beginning-of-defun-function.

Sure.

> This requires a parsing loop, which I'd guess is how mmm handles it.

Like I mentioned in another email, not really. Although it does have an 
idle parsing feature tackled on.

This an obvious area for improvement.

Re: Emacs contributions, C and Lisp

[Dmitry Gutov]

On 16.02.2014 19:36, Eli Zaretskii wrote:
>> Jedi: https://github.com/tkf/emacs-jedi
>> Jedi and Rope: https://github.com/jorgenschaefer/elpy
>> nrepl: https://github.com/technomancy/nrepl.el (and others)
>> tern: https://github.com/marijnh/tern/tree/master/emacs (and others)
>> OmniSharp: https://github.com/sp3ctum/omnisharp-emacs (apparently)
>
> Why aren't they part of Emacs?

Here's an obvious answer: CA.

They are available through package archives, though.

>> As I said in the thread you mentioned, the problem is not that these
>> extensions wouldn't exist for Emacs, it's that each and every one of
>> them reinvents most of the wheel to get similar features to Emacs.

I'd like the see the wheels enumerated.

> Bringing them into Emacs bundle is a significant first step towards
> solving the deficiencies, since Emacs maintainers will work with the
> authors to fix them, and will continue maintaining them through the
> years.

Hopefully, this is something that would accompany bringing a package 
into GNU ELPA, too.

 > If they are left outside, they will continue being on the fringe.

I don't know about being on the fringe. The ones above are fairly popular.

>> The integration was declined on this list because Emacs should prefer
>> gcc over clang to protect user's freedoms, and gcc does not provide such
>> features.
>
> GCC supports plugins, so it should be possible to do the same.

And this brings us back to contributors interested in writing C code. 
Moreover, in this case, to duplicate functionality that's already 
available for most users anyway.

> In any case, clinging with clang clearly makes no sense, if we want
> these features in Emacs.

Do we? Until there are no big problems to solve in Emacs left, I'd be 
pretty comfortable with recommending users who want better completion in 
C/C++/ObjC to install a third-party package.

Re: Emacs contributions, C and Lisp

[Eli Zaretskii]

> Date: Sun, 16 Feb 2014 20:27:00 +0200
> From: Dmitry Gutov <dgutov@yandex.ru>
> CC: dak@gnu.org, emacs-devel@gnu.org
> 
> On 16.02.2014 19:50, Eli Zaretskii wrote:
> > These problems exist in Lisp as they do (or don't) in C, as far as
> > Emacs is concerned.  E.g., see the latest discussion of crypto
> > features.
> 
> I'd say in Lisp they exist to a considerably lesser degree.

I don't think so: after all, we keep unencrypted text in the clear in
memory, within Lisp data structures.

> > See the other mail for what I (and Jorgen) had in mind.  I don't
> > think we have anything like that in Emacs, even with CEDET.  ECB
> > (which isn't bundled) is the only thing that comes close, but IMO we
> > should have had this in Emacs core for a long time.
> 
> This looks like something that might require fancier rendering on the 
> fringe or a nearby area.

Fringe is one possibility, but IMO it's not the best or easiest one.
I think it will be easier to simply display small images as part of
the buffer.  Like Speedbar already does.

Re: Emacs contributions, C and Lisp

[Dmitry Gutov]

On 16.02.2014 20:43, Eli Zaretskii wrote:
> Fringe is one possibility, but IMO it's not the best or easiest one.
> I think it will be easier to simply display small images as part of
> the buffer.  Like Speedbar already does.

Speedbar contents are not directly modifiable, though.

Re: Emacs contributions, C and Lisp (was: Re: /srv/bzr/emacs/trunk r101338: ...))

[E Sabof]

Eli Zaretskii <eliz@gnu.org> writes:

> Then the immediate task would be to make it simpler to set up and
> suitable for those languages.  _That_ would really move Emacs forward,
> towards a point where, hopefully, it will once again be relevant for
> modern development paradigm and provide what programmers expect.
>

I couldn't agree more. Trying to find a decent snippet/package, in a
complex and unfamiliar enviroment, is far from a pleasant experience.
And if the no choice is pre-made, that's what every newcomer has to do.

And if the choice turns out to be wrong, changing "bash" to "ksh", is
not any harder than changing nil to "ksh".

Evgeni

Re: Emacs contributions, C and Lisp

[Eli Zaretskii]

> Date: Sun, 16 Feb 2014 20:49:32 +0200
> From: Dmitry Gutov <dgutov@yandex.ru>
> CC: dak@gnu.org, emacs-devel@gnu.org
> 
> On 16.02.2014 20:43, Eli Zaretskii wrote:
> > Fringe is one possibility, but IMO it's not the best or easiest one.
> > I think it will be easier to simply display small images as part of
> > the buffer.  Like Speedbar already does.
> 
> Speedbar contents are not directly modifiable, though.

So?  It is still an Emacs buffer that displays text, like almost
everything else in Emacs.  Display of small images alongside that text
is a no-brainer.

Re: /srv/bzr/emacs/trunk r101338:*lisp/emacs-lisp/syntax.el(syntax-ppss): More sanity check to catchtch

[E Sabof]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 16.02.2014 19:48, E Sabof wrote:
>> I suppose some keys might change as well, but I'not convinced that the benefits outweigh the additional complexity.
>
> Changing local maps based on regions is fairly easy.

I know. I meant complexity of use.

Evgeni

Re: Emacs contributions, C and Lisp

[Dmitry Gutov]

On 16.02.2014 20:55, Eli Zaretskii wrote:
> So?  It is still an Emacs buffer that displays text, like almost
> everything else in Emacs.  Display of small images alongside that text
> is a no-brainer.

I was thinking it might be difficult to update the images and their 
positions while the buffer is being edited. But suppose that's solved.

If the function definition you'd like to fold starts at column 0, where 
would the image be rendered?

Re: Emacs contributions, C and Lisp

[Eli Zaretskii]

> Date: Sun, 16 Feb 2014 21:06:02 +0200
> From: Dmitry Gutov <dgutov@yandex.ru>
> CC: dak@gnu.org, emacs-devel@gnu.org
> 
> On 16.02.2014 20:55, Eli Zaretskii wrote:
> > So?  It is still an Emacs buffer that displays text, like almost
> > everything else in Emacs.  Display of small images alongside that text
> > is a no-brainer.
> 
> I was thinking it might be difficult to update the images and their 
> positions while the buffer is being edited. But suppose that's solved.

They are just 'display' text properties or overlays, so they can
easily move (or not move, as the case may be) with the text.

> If the function definition you'd like to fold starts at column 0, where 
> would the image be rendered?

One obvious possibility is in the margins.

Or you could put a before-string on each line, which would then shift
the source to the right.

Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[Stefan Monnier]

> E.g., what you suggest would require a serious surgery on the design
> of the display engine,

Not just the display engine: you'd also have to figure out how to handle
the load&save of this, how to get things like next-line or
rectangle-mark-mode to handle it properly, ...

It might be an interesting idea that could be occasionally attractive
(maybe for Gnus's rendering of attachments?), but for mmm-mode, it
sounds like adding problems rather than providing a solution.

> So I really think we should define the requirements first.

To me, the requirements look like:
- font-lock and syntax-propertize, each sub-part according to its own
  mode and according to its context which can depend on the immediately
  preceding chunk (even tho it's in a different mode) as well as on some
  earlier chunk in the same mode.
- indentation is done according to the mode to which BOL belongs, I think.
- there can be nesting relationships between chunks, e.g. with two major
  modes A and B, a sequence of chunks of "A1, B1, A2, B2, A3" could
  represent "(A1 (B1) A2 (B2) A3)" or "(A1 (B1 (A2) B2) A3)", where
  indentation of A3 could depend on A1 in the second case but on A2 in
  the first.
- some minor modes might be specific to "chunks in major mode A", while
  others are "global to the whole buffer".
- key-bindings should probably be looked up according to the "mode at
  point".
- navigation should ideally understand the overall structure of the
  buffer, so forward-sexp knows how to jump over a "sub-chunk" written
  in another mode.
- M-; and many other such commands should use the "mode at point".
- not sure whether imenu should combine the list of definitions of all
  chunks in the buffer.
- commands which are mode-agnostic (such as next-line or
  rectangle-mark-mode) should be unaffected.
Not sure if it's the kind of requirements you were thinking about.


        Stefan

Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[Stefan Monnier]

> The implementation of region classes definitions and parsing of the buffer
> could still be reused with this approach.

I'm sure some part could be reused, yes.

> Although, if the goal is to parse the buffer lazily,

Yes, that's indispensable.

> maybe adapting the relevant multi-mode code instead would be easier.

Even better, since we already have paperwork for that one, IIUC.


        Stefan

Re: /srv/bzr/emacs/trunk r101338:*lisp/emacs-lisp/syntax.el(syntax-ppss): More sanity check to catchtch

[Stefan Monnier]

>> Changing local maps based on regions is fairly easy.
> I know.  I meant complexity of use.

Hmm... I guess you're right.  There's an argument to be made for keeping
the keymap "buffer-global" (i.e. not switching it based on the mode at
point).


        Stefan

Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[Dmitry Gutov]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> To me, the requirements look like:

Looks fairly comprehensive to me.

> - indentation is done according to the mode to which BOL belongs, I think.

Yep.

> - there can be nesting relationships between chunks, e.g. with two major
>   modes A and B, a sequence of chunks of "A1, B1, A2, B2, A3" could
>   represent "(A1 (B1) A2 (B2) A3)" or "(A1 (B1 (A2) B2) A3)", where
>   indentation of A3 could depend on A1 in the second case but on A2 in
>   the first.

Indentation can depend on preceding nested chunks, too. See control
structures in JSP, ERB, PHP, etc.

> - navigation should ideally understand the overall structure of the
>   buffer, so forward-sexp knows how to jump over a "sub-chunk" written
>   in another mode.

Do you think a mmm-mode-aware forward-sexp-function is the best way to
support this?

Re: Emacs contributions, C and Lisp

[Stefan Monnier]

> The integration was declined on this list because Emacs should prefer

Declined by Richard.  I'm not opposed it.


        Stefan

Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[Juanma Barranquero]

On Mon, Feb 17, 2014 at 3:40 AM, Stefan Monnier
<monnier@iro.umontreal.ca> wrote:

> - font-lock and syntax-propertize, each sub-part according to its own
>   mode and according to its context which can depend on the immediately
>   preceding chunk (even tho it's in a different mode) as well as on some
>   earlier chunk in the same mode.
> - indentation is done according to the mode to which BOL belongs, I think.

For multi-modes representing dynamically generated content (PHP +
HTML, etc.) there's going to be fragments that will be hard or
impossible to indent meaningfully. Can't be avoided, I think.

> Not sure if it's the kind of requirements you were thinking about.

What about code that deals with the buffer's major-mode? In a PHP +
HTML buffer, or a Literate Haskell source (assuming it's a multi-mode
buffer with text and Haskell submodes), what will (with-current-buffer
"my-multi-buffer" major-mode) return? Does it depend on `point'? If
you switch the buffer's major-mode, does it change for all fragments?

   J

Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Michael Welsh Duggan <mwd@md5i.com>,  emacs-devel@gnu.org
> Date: Sun, 16 Feb 2014 21:40:18 -0500
> 
> To me, the requirements look like:
> [...]
> Not sure if it's the kind of requirements you were thinking about.

Something like that, yes, but perhaps with a bit more details (e.g.,
the "nesting relationships" part could use some more explanations of
the situation and the intent).

I suggest to record all these in a bug report.

Thanks.

Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[Stefan Monnier]

> For multi-modes representing dynamically generated content (PHP +
> HTML, etc.) there's going to be fragments that will be hard or
> impossible to indent meaningfully. Can't be avoided, I think.

Indeed.  Just as is the case with macros and preprocessors.  If/when
this problems turns out to be significant, we'll see if there's
something we can try to do, but for now, this seems like a fairly
remote problem.

> What about code that deals with the buffer's major-mode? In a PHP +
> HTML buffer, or a Literate Haskell source (assuming it's a multi-mode
> buffer with text and Haskell submodes), what will (with-current-buffer
> "my-multi-buffer" major-mode) return? Does it depend on `point'? If
> you switch the buffer's major-mode, does it change for all fragments?

I'm not sure.  Maybe it will depend.  I'm leaning towards having
a "main" major mode and leaving `major-mode' pointing to that.
As mentioned elsewhere, it might also be a better choice to let this
main major mode control the keymap.


        Stefan

Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[Stefan Monnier]

> Do you think a mmm-mode-aware forward-sexp-function is the best way to
> support this?

No, probably not the best.  But we'll probably start with that, yes.


        Stefan

Re: /srv/bzr/emacs/trunkr101338:*lisp/emacs-lisp/syntax.el(syntax-ppss): More sanitycheck to catchcatch

[E Sabof]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> - there can be nesting relationships between chunks, e.g. with two major
>   modes A and B, a sequence of chunks of "A1, B1, A2, B2, A3" could
>   represent "(A1 (B1) A2 (B2) A3)" or "(A1 (B1 (A2) B2) A3)", where
>   indentation of A3 could depend on A1 in the second case but on A2 in
>   the first.

Might the effect be the same, if each chunk increased or decreased the depth?

Evgeni

Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[Dmitry Gutov]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> I'm not sure.  Maybe it will depend.  I'm leaning towards having
> a "main" major mode and leaving `major-mode' pointing to that.

It could be best. Any code that depends on a major mode will likely
break in multi-mode environment if it's not aware of it.

Maybe have a secondary local variable that would point to the current
submode (like mmm-current-mode).

> As mentioned elsewhere, it might also be a better choice to let this
> main major mode control the keymap.

I'd rather not do that. Some modes override common commands via the
keymap, rather than some `-function[s]' hook.

What about submode-local minor modes and their keymaps? Do we prohibit
those?

Re: /srv/bzr/emacs/trunkr101338:*lisp/emacs-lisp/syntax.el(syntax-ppss): More sanitycheck to catchcatch

[Dmitry Gutov]

E Sabof <esabof@gmail.com> writes:

> Might the effect be the same, if each chunk increased or decreased the depth?

Some chunks create a "temporary" outdent:

<% if foo %>
  <p>woo</p>
<% else %>
  <p>wah wah</p>
<% end %>

Re: Emacs contributions, C and Lisp (was: Re: /srv/bzr/emacs/trunk r101338: ...)

[Jorgen Schaefer]

On Sun, 16 Feb 2014 19:36:30 +0200
Eli Zaretskii <eliz@gnu.org> wrote:

> > The whole code folding infrastructure in Emacs needs work, really.
> > The problem, as often with Emacs, is not that it wouldn't exist at
> > all, but that there are two dozen solutions that all have different
> > shortcomings.  
> 
> Then here's a clear bunch of todo items, right?  I hope someone will
> volunteer.

To write yet another folding infrastructure that is deficient in some
other way? :-)

(Only partly joking; I was thinking about doing this, but then I
realized I don't really know the idiosyncrasies of current C dialects
well enough to make a good choice there, and then I'd write something
that works for Python but not for other things. I'll mull it over in my
head and maybe come to a point where I feel comfortable writing yet
another infrastructure, but see below.)

> > > > Despite certain expectations that everything is better when
> > > > written in Emacs Lisp, context-dependent code completion and
> > > > documentation display support is usually delegated to an
> > > > external process, and there are relatively new packages that
> > > > use editor-agnostic services: Jedi for Python, nREPL for
> > > > Clojure, Tern for JavaScript, OmniSharp for C#.  
> > > 
> > > Where's their integration with Emacs, though?  
> > 
> > Jedi: https://github.com/tkf/emacs-jedi
> > Jedi and Rope: https://github.com/jorgenschaefer/elpy
> > nrepl: https://github.com/technomancy/nrepl.el (and others)
> > tern: https://github.com/marijnh/tern/tree/master/emacs (and others)
> > OmniSharp: https://github.com/sp3ctum/omnisharp-emacs (apparently)  
> 
> Why aren't they part of Emacs?

I can not answer for the others, but elpy is not part of Emacs for a
number of reasons. The most important for me is the slow release cycle
of Emacs. The library is still evolving more or less quickly. I do not
feel like "freezing" it in the current state for the time it takes for
another Emacs release to happen.

If with "part of Emacs" you mean "in GNU ELPA", then this would be
because a number of the dependencies I'd like to bring in are not in
GNU ELPA. This will improve over time as I either replace the
dependencies with my own libraries or implementations, or making them
optional enough to be acceptable for GNU ELPA. I guess suggesting
adding external package archives to be able to enable certain
configuration options is politically tricky.

There are a few other minor problems for me. For example, my last foray
in adding a patch to Emacs was so scary regarding the amount of red
tape involved in the whole process that I am somewhat reluctant to
commit to doing that regularly. But if the other problems weren't
there, I guess I'd get over this quite ok.

(Incidentally, copyright assignment is not an issue for me at all.
Keeping track of contributors can be, but so far, it's not a problem.)

> > As I said in the thread you mentioned, the problem is not that these
> > extensions wouldn't exist for Emacs, it's that each and every one of
> > them reinvents most of the wheel to get similar features to Emacs.  
> 
> Bringing them into Emacs bundle is a significant first step towards
> solving the deficiencies, since Emacs maintainers will work with the
> authors to fix them, and will continue maintaining them through the
> years.  If they are left outside, they will continue being on the
> fringe.

I'm not sure this is true. There are plenty of things already in
Emacs that have the same problem.

Look at code folding as an example. Emacs already includes
outline-minor-mode, hs-minor-mode, and allout-mode, all of which are
there for code folding. What deficiencies will be solved by adding
*more* minor modes that allow folding inside of code?

What are authors of language major modes supposed to do to enable code
folding? python.el supports *both* hs-minor-mode and
outline-minor-mode, but not allout-mode. Are all extensions meant to
support both? Which one are users supposed to learn when they want to
use code folding in Emacs? If I want to add the icons in the fringe we
talked about, which mode would I extend? All three?

How about communicating with a subprocess for completions? python.el
already does that. It just uses the inferior Python process, which
has a few issues. Has adding this to Emacs done anything to make others
not reinvent the wheels all the time? I don't see that.

CEDET is in Emacs and provides a lot of infrastructure for IDE
development. How many of the libraries above use it? None.

Looking at the available data, I have a hard time believing that the
main problem here is that these libraries with their half a dozen
slightly different reinvented wheels haven't made it into Emacs yet.

The problem as far as I can see is that these libraries all exist to
scratch someone's particular itch (e.g. I wrote elpy because I needed
a better programming interface to Python, no other reason), so no one
is going to put in a lot of effort of trying to bring all of those
scratchings together and find a common abstraction.

And even if someone did, it does not seem to me as if there is any kind
of interest in Emacs to decide for one particular interface and declare
it as "the way to go". But unless someone sits down and decides for
this, wheels will keep getting reinvented.

> In any case, clinging with clang clearly makes no sense, if we want
> these features in Emacs.

You asked why they were not in Emacs. I gave you the reason. I am not
arguing whether the decision is good or bad, but after having a few
multi-page threads about why clang-based extensions aren't allowed in
Emacs, it does not make sense to ask a week later "why aren't they in
Emacs?"


Regards,
Jorgen

Re: Emacs contributions, C and Lisp (was: Re: /srv/bzr/emacs/trunk r101338: ...)

[Eli Zaretskii]

> Date: Mon, 17 Feb 2014 20:31:45 +0100
> From: Jorgen Schaefer <forcer@forcix.cx>
> Cc: emacs-devel@gnu.org, dgutov@yandex.ru
> 
> On Sun, 16 Feb 2014 19:36:30 +0200
> Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > > The whole code folding infrastructure in Emacs needs work, really.
> > > The problem, as often with Emacs, is not that it wouldn't exist at
> > > all, but that there are two dozen solutions that all have different
> > > shortcomings.  
> > 
> > Then here's a clear bunch of todo items, right?  I hope someone will
> > volunteer.
> 
> To write yet another folding infrastructure that is deficient in some
> other way? :-)

No, improve one of the existing ones (outline-mode or hs-mode, for
example).

> (Only partly joking; I was thinking about doing this, but then I
> realized I don't really know the idiosyncrasies of current C dialects
> well enough to make a good choice there, and then I'd write something
> that works for Python but not for other things. I'll mull it over in my
> head and maybe come to a point where I feel comfortable writing yet
> another infrastructure, but see below.)

I would suggest starting with just the visual indicators, and leaving
the automatic detection of code blocks for a later changeset.

> > > Jedi: https://github.com/tkf/emacs-jedi
> > > Jedi and Rope: https://github.com/jorgenschaefer/elpy
> > > nrepl: https://github.com/technomancy/nrepl.el (and others)
> > > tern: https://github.com/marijnh/tern/tree/master/emacs (and others)
> > > OmniSharp: https://github.com/sp3ctum/omnisharp-emacs (apparently)  
> > 
> > Why aren't they part of Emacs?
> 
> I can not answer for the others, but elpy is not part of Emacs for a
> number of reasons. The most important for me is the slow release cycle
> of Emacs.

Since many people and GNU/Linux distros use the development snapshots,
I don't see why this would be a problem.

> There are a few other minor problems for me. For example, my last foray
> in adding a patch to Emacs was so scary regarding the amount of red
> tape involved in the whole process that I am somewhat reluctant to
> commit to doing that regularly.

What red tape?  Emacs is about the most red-tape-less project as you
can find, as far as the procedure of admitting a patch is considered.

> > > As I said in the thread you mentioned, the problem is not that these
> > > extensions wouldn't exist for Emacs, it's that each and every one of
> > > them reinvents most of the wheel to get similar features to Emacs.  
> > 
> > Bringing them into Emacs bundle is a significant first step towards
> > solving the deficiencies, since Emacs maintainers will work with the
> > authors to fix them, and will continue maintaining them through the
> > years.  If they are left outside, they will continue being on the
> > fringe.
> 
> I'm not sure this is true.

I am.

> There are plenty of things already in Emacs that have the same
> problem.

Please point them out.  I'm quite sure they were in much worse shape
before.

> Look at code folding as an example. Emacs already includes
> outline-minor-mode, hs-minor-mode, and allout-mode, all of which are
> there for code folding. What deficiencies will be solved by adding
> *more* minor modes that allow folding inside of code?

I didn't suggest adding one more mode, I suggested to pick one and add
the features discussed to it.

> What are authors of language major modes supposed to do to enable code
> folding? python.el supports *both* hs-minor-mode and
> outline-minor-mode, but not allout-mode. Are all extensions meant to
> support both? Which one are users supposed to learn when they want to
> use code folding in Emacs? If I want to add the icons in the fringe we
> talked about, which mode would I extend? All three?

Only one, IMO, certainly for starters.

> CEDET is in Emacs and provides a lot of infrastructure for IDE
> development. How many of the libraries above use it? None.

I'm still waiting to see someone (you?) to put their money where their
mouth is, and actually start working on that.

> Looking at the available data, I have a hard time believing that the
> main problem here is that these libraries with their half a dozen
> slightly different reinvented wheels haven't made it into Emacs yet.

The problem as I see it is that no one steps forward to advance all
the ideas to the state where others could actually use these features.
The moment that happens, these problems will be over, and people will
quickly switch to the package which offers them and drop the
alternatives.

> The problem as far as I can see is that these libraries all exist to
> scratch someone's particular itch (e.g. I wrote elpy because I needed
> a better programming interface to Python, no other reason), so no one
> is going to put in a lot of effort of trying to bring all of those
> scratchings together and find a common abstraction.
> 
> And even if someone did, it does not seem to me as if there is any kind
> of interest in Emacs to decide for one particular interface and declare
> it as "the way to go". But unless someone sits down and decides for
> this, wheels will keep getting reinvented.

If you mean that we also need more leadership in these matters, then I
agree.

> > In any case, clinging with clang clearly makes no sense, if we want
> > these features in Emacs.
> 
> You asked why they were not in Emacs. I gave you the reason. I am not
> arguing whether the decision is good or bad, but after having a few
> multi-page threads about why clang-based extensions aren't allowed in
> Emacs, it does not make sense to ask a week later "why aren't they in
> Emacs?"

I didn't know they were based on clang when I asked, that's all.

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    > The integration was declined on this list because Emacs should prefer

    Declined by Richard.  I'm not opposed it.

This is a GNU Project policy decision.
Emacs will give firm support to promote use of GCC.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[Stefan Monnier]

> Maybe have a secondary local variable that would point to the current
> submode (like mmm-current-mode).

I'd rather it be a function, but yes that's probably going to be the
better option.

>> As mentioned elsewhere, it might also be a better choice to let this
>> main major mode control the keymap.
> I'd rather not do that.  Some modes override common commands via the
> keymap, rather than some `-function[s]' hook.

There are advantages on each side.  We'll have to look at the actual
cases to have a better idea of what needs to be done when.


        Stefan

Re: Emacs contributions, C and Lisp

[Stefan Monnier]

> And even if someone did, it does not seem to me as if there is any kind
> of interest in Emacs to decide for one particular interface and declare
> it as "the way to go". But unless someone sits down and decides for
> this, wheels will keep getting reinvented.

Actually, there is such an interest, in the abstract.  But concretely
it's a lot of work and I don't know of anyone how has the energy and
knowledge to do it.

Consolidating hs-minor-mode and outline-minor-mode would be a good
start, tho (AFAIK allout-mode is a different beast altogether).


        Stefan "who'd be happy to hear more about the hurdles to
                integration of elpy into GNU ELPA"

Re: Emacs contributions, C and Lisp

[Stefan Monnier]

> Emacs will give firm support to promote use of GCC.

Very much so, as well as support for any other Free Software package,


        Stefan

Re: /srv/bzr/emacs/trunkr101338:*lisp/emacs-lisp/syntax.el(syntax-ppss): More sanitycheck to catchcatch

[E Sabof]

Dmitry Gutov <dgutov@yandex.ru> writes:

> E Sabof <esabof@gmail.com> writes:
>
>> Might the effect be the same, if each chunk increased or decreased the depth?
>
> Some chunks create a "temporary" outdent:
>
> <% if foo %>
>   <p>woo</p>
> <% else %>
>   <p>wah wah</p>
> <% end %>

The depth is meant to be treated as a "baseline", not a rule. Closing "things", are usually indented at

(1- (nth 0 (syntax-ppss (line-beginning-position))))

That this is the "correct" way to indent might be subject to dispute, but here is an example where the depth is ignored altogether.

<!DOCTYPE html>
<html lang="en-gb" > (incf depth)
  <head> (incf depth)
    <style type="text/css"> (incf depth)
(setq depth-store depth
      depth 0)
body {
  color: green;
}
(setq depth depth-store)
    </style> (decf depth)
  </head> (decf depth)
</html> (decf depth)

Evgeni

Re: Emacs contributions, C and Lisp

[Phillip Lord]

Jorgen Schaefer <forcer@forcix.cx> writes:
>> Bringing them into Emacs bundle is a significant first step towards
>> solving the deficiencies, since Emacs maintainers will work with the
>> authors to fix them, and will continue maintaining them through the
>> years.  If they are left outside, they will continue being on the
>> fringe.
>
> I'm not sure this is true. There are plenty of things already in
> Emacs that have the same problem.
>
> Look at code folding as an example. Emacs already includes
> outline-minor-mode, hs-minor-mode, and allout-mode, all of which are
> there for code folding. What deficiencies will be solved by adding
> *more* minor modes that allow folding inside of code?
>
> What are authors of language major modes supposed to do to enable code
> folding? python.el supports *both* hs-minor-mode and
> outline-minor-mode, but not allout-mode. Are all extensions meant to
> support both? Which one are users supposed to learn when they want to
> use code folding in Emacs? If I want to add the icons in the fringe we
> talked about, which mode would I extend? All three?

There's no reason that icons in the fringe should not happen at a
lowever level than this. If they operated as an option on invisible
overlays, this would come for free.


> How about communicating with a subprocess for completions? python.el
> already does that. It just uses the inferior Python process, which
> has a few issues. Has adding this to Emacs done anything to make others
> not reinvent the wheels all the time? I don't see that.

Perhaps part of the problem is that there are not enough Emacs libraries
aimed at the programmer. The subprocess support in Emacs provides you
with the primitives that you need, but only that. Getting libraries in
to make life easier for Emacs developers would be a good thing.

A good example of this is dash.el. It's a nice, rich library. Having
started to use it, the only thing that surprises me is that it took so
long for someone to think to write it.


> CEDET is in Emacs and provides a lot of infrastructure for IDE
> development. How many of the libraries above use it? None.

CEDET is, of course, the counter example to my theory that more
libraries for programmers would be good. I think you are right, that it
is not as widely used within Emacs as it should be.


> The problem as far as I can see is that these libraries all exist to
> scratch someone's particular itch (e.g. I wrote elpy because I needed
> a better programming interface to Python, no other reason), so no one
> is going to put in a lot of effort of trying to bring all of those
> scratchings together and find a common abstraction.

I think this is not necessarily true. People itch in a lot of strange
ways. 

Phil

Re: Emacs contributions, C and Lisp

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> There are a few other minor problems for me. For example, my last foray
>> in adding a patch to Emacs was so scary regarding the amount of red
>> tape involved in the whole process that I am somewhat reluctant to
>> commit to doing that regularly.
>
> What red tape?  Emacs is about the most red-tape-less project as you
> can find, as far as the procedure of admitting a patch is considered.


This is an interesting difference of opinion and is probably worth
exploring; it's worth knowing what the difficulties where.

It might also be writing this up -- is there a description anywhere of
"how to submit a change to Emacs", and should it not be in the elisp
manual?

Incidentally, I say "change" and not "patch". Submitting a change these
days generally means "clone, branch, pull request".

Phil

Re: Emacs contributions, C and Lisp

[Eli Zaretskii]

> From: phillip.lord@newcastle.ac.uk (Phillip Lord)
> Cc: Jorgen Schaefer <forcer@forcix.cx>, <dgutov@yandex.ru>,
>         <emacs-devel@gnu.org>
> Date: Tue, 18 Feb 2014 10:00:25 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> There are a few other minor problems for me. For example, my last foray
> >> in adding a patch to Emacs was so scary regarding the amount of red
> >> tape involved in the whole process that I am somewhat reluctant to
> >> commit to doing that regularly.
> >
> > What red tape?  Emacs is about the most red-tape-less project as you
> > can find, as far as the procedure of admitting a patch is considered.
> 
> 
> This is an interesting difference of opinion and is probably worth
> exploring; it's worth knowing what the difficulties where.

That's why I asked what kind of red tape was being alluded to here.

> It might also be writing this up -- is there a description anywhere of
> "how to submit a change to Emacs", and should it not be in the elisp
> manual?

There's nothing to describe, IMO.  If you don't have write access,
just post the changes and ask for review or approval; if you do have
write access, you just commit.

> Incidentally, I say "change" and not "patch". Submitting a change these
> days generally means "clone, branch, pull request".

Sorry, I don't understand the subtlety.

Re: Emacs contributions, C and Lisp

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: phillip.lord@newcastle.ac.uk (Phillip Lord)
>> Cc: Jorgen Schaefer <forcer@forcix.cx>, <dgutov@yandex.ru>,
>>         <emacs-devel@gnu.org>
>> Date: Tue, 18 Feb 2014 10:00:25 +0000
>
>> Incidentally, I say "change" and not "patch". Submitting a change these
>> days generally means "clone, branch, pull request".
>
> Sorry, I don't understand the subtlety.

I think it's a paraphrase of "real developers use GitHub", a service
running on proprietary software and having a variety of commercial
offerings, including a rather popular "take it or leave it" zero
pricetag offering.

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[Glenn Morris]

Phillip Lord wrote:

> It might also be writing this up -- is there a description anywhere of
> "how to submit a change to Emacs", and should it not be in the elisp
> manual?

etc/CONTRIBUTE
Info node `(emacs)Contributing'

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    > Emacs will give firm support to promote use of GCC.

    Very much so, as well as support for any other Free Software package,

My decision is that Emacs will support GCC by NOT specially promoting
the use of LLVM in any way.  Whatever features we put in Emacs for
interacting with a compiler in any fashion should work with GCC.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[chad]

On 18 Feb 2014, at 07:31, David Kastrup <dak@gnu.org> wrote:

> Eli Zaretskii <eliz@gnu.org> writes:
> 
>>> Incidentally, I say "change" and not "patch". Submitting a change these
>>> days generally means "clone, branch, pull request".
>> 
>> Sorry, I don't understand the subtlety.
> 
> I think it's a paraphrase of "real developers use GitHub", a service
> running on proprietary software and having a variety of commercial
> offerings, including a rather popular "take it or leave it" zero
> pricetag offering.

I think it’s a paraphrase of “lots of developers are used to a model where they submit changes that are taken, with or without modification, rather than a process whereby they suggest a change and then begin a dialog”. Github is definitely one of the promulgators of this model, but it’s certainly not the only one.

Without digging into real data (sorry), I’d say that around 33-50% of the changes submitted to emacs-devel from new people don’t land, either because of explicit rejection or because the effective hurdle is high enough that people don’t get over it. I’d further say that at least half of those failures to land result from platform-specific issues.  There are good reasons for the policies that create these hurdles, but they surely make it much harder for people to start working on emacs, even compared to other free/libre software projects.

I hope that helps,
~Chad

Re: Emacs contributions, C and Lisp

[Dmitry Gutov]

Richard Stallman <rms@gnu.org> writes:

> Whatever features we put in Emacs for
> interacting with a compiler in any fashion should work with GCC.

Richard, have you tried discussing this with GCC developers recently?
Are any of them interested in providing code completion interface?

Considering your distaste for Clang, this page should look pretty
embarrassing: http://gcc.gnu.org/wiki/CodeCompletion

Re: Emacs contributions, C and Lisp

[Eli Zaretskii]

> From: chad <chadpbrown@gmail.com>
> Date: Tue, 18 Feb 2014 16:43:03 -0800
> 
> 
> On 18 Feb 2014, at 07:31, David Kastrup <dak@gnu.org> wrote:
> 
> > Eli Zaretskii <eliz@gnu.org> writes:
> > 
> >>> Incidentally, I say "change" and not "patch". Submitting a change these
> >>> days generally means "clone, branch, pull request".
> >> 
> >> Sorry, I don't understand the subtlety.
> > 
> > I think it's a paraphrase of "real developers use GitHub", a service
> > running on proprietary software and having a variety of commercial
> > offerings, including a rather popular "take it or leave it" zero
> > pricetag offering.
> 
> I think it’s a paraphrase of “lots of developers are used to a model where they submit changes that are taken, with or without modification, rather than a process whereby they suggest a change and then begin a dialog”. Github is definitely one of the promulgators of this model, but it’s certainly not the only one.
> 
> Without digging into real data (sorry), I’d say that around 33-50% of the changes submitted to emacs-devel from new people don’t land, either because of explicit rejection or because the effective hurdle is high enough that people don’t get over it. I’d further say that at least half of those failures to land result from platform-specific issues.  There are good reasons for the policies that create these hurdles, but they surely make it much harder for people to start working on emacs, even compared to other free/libre software projects.

So you are saying that patch review process is that "red tape" that
was mentioned earlier?  If so, all the projects I'm involved with
insist on the "dialog", i.e. that the original contributors improve
and fix their contributions until they are acceptable.  Which projects
don't?

Of course, volunteers are welcome to help with that, i.e. pick up such
patches and rework them into acceptable state.  Because so far the
number of people who do that is countable on a single hand.  Given
that, you cannot expect those people to also do the reworking, even if
they thought this was the way to go.

Re: Emacs contributions, C and Lisp

[Stefan Monnier]

> Whatever features we put in Emacs for interacting with a compiler in
> any fashion should work with GCC.

Only to the extent that GCC supports that feature.


        Stefan

Re: Emacs contributions, C and Lisp

[chad]

[-- Attachment #1: Type: text/plain, Size: 2169 bytes --]

On 18 Feb 2014, at 19:55, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> So you are saying that patch review process is that "red tape" that
> was mentioned earlier?  If so, all the projects I'm involved with
> insist on the "dialog", i.e. that the original contributors improve
> and fix their contributions until they are acceptable. 

To some degree, but (my impression is that) Emacs stops a lot of
changes at the need for support on more platforms than the submitter
can easily develop on.  Note that Im not saying that these are
unreasonable - just that theyre not based on github.

> Which projects don’t?

Most software projects today are much smaller, more nimble, and
piecemeal-clockwork than emacs. Emacs was built in a heavily
monolithic era, and continues with most of that lineage. Far more
software projects today are assembled from many interlocking pieces
of libraries, middleware, and plugin code. Again, there are good
reasons for this, but it has a cost, and much of that cost is
expressed as a high barrier to entry.

Compare this to typical java, python, or ruby projects; they usually
use a variety of smaller, self-contained projects, each of which
is a viable entry point for a new contributor. To the extent that
Emacs uses libraries, its often more hindrance than help, as the
new contributor is faced with X, GTK, ns/OpenStep, and w32 libraries,
or learning the daunting Emacs internal wrappers for each.

Another thing that hits Emacs: people working on changing or extending
Eclipse do it IN java, which is the same language they use Eclipse
FOR. While there are surely Emacs developers using Emacs to develop
Emacs Lisp, theyre doing so mostly just for Emacs, not for anything
else; this internal focus shows in the results. There was a time
when Emacs was a great way to write C code, but its really fallen
behind in some key areas, and Emacs own C code is highly idiosyncratic.

Again, Im aware that there are good reasons for all of these things
to be true, but each of them makes it harder for people to make
significant contributions to Emacs, which is the topic at hand.

I hope that helps,
~Chad

[-- Attachment #2: Type: text/html, Size: 3233 bytes --]

Re: Emacs contributions, C and Lisp (was: Re: /srv/bzr/emacs/trunk r101338: ...)

[Jorgen Schaefer]

On Mon, 17 Feb 2014 22:29:47 +0200
Eli Zaretskii <eliz@gnu.org> wrote:

> > There are a few other minor problems for me. For example, my last
> > foray in adding a patch to Emacs was so scary regarding the amount
> > of red tape involved in the whole process that I am somewhat
> > reluctant to commit to doing that regularly.
> 
> What red tape?  Emacs is about the most red-tape-less project as you
> can find, as far as the procedure of admitting a patch is considered.

If I want to contribute to Emacs, and I want to be good contributor, I
have the following things to keep in mind:

- Make sure I have done my copyright assignment, and for larger work,
  do the same for all contributors.
- Read etc/CONTRIBUTE and follow all the steps
- Which asks me to read the GNU Coding Standards
- Oh, and "Tips and Conventions" in the emacs Lisp Reference
  appendix
- Learn a new tool (bzr), for which I have to read
  http://www.emacswiki.org/emacs/BzrForEmacsDevs
  (and re-read every time I make a contribution, because bzr is close
  but suitably different to my usual tools)
- Make sure I edit the right ChangeLog in the right format
- Remember that the commit message uses the same format as the
  ChangeLog, and not the usual, lighter, commit message format used by
  other (git-based) projects
- Possibly remember the NEWS entry and the manual update
- Remember to add the right "fixes" tag to the bzr commit if I
  fixed a bug
- Have the nagging feeling in the back of my head that if I submit a
  patch for review, and get told "please install", I have to fight bzr
  again, or face the real possibility of being berated and attacked
  publicly on the mailing list, which I have seen happen to at least
  one other contributor. (Though I should say that the last time I made
  some contribution mistake, I got a very friendly mail from Stefan
  asking me to remember the things I forgot next time, a much nicer
  tone than I am used to from some other projects.)

I'd not be surprised if I forgot something.

And if I am thinking about contributing a full package, I have to
commit to following all of that for the foreseeable future, too.

Now, do not get me wrong. I am not complaining about these requirements
(so, re-reading the Wikipedia entry on "red tape" I guess the term was
badly chosen, sorry, not a native speaker; what's a good term for
"*lots* of regulation and rigid conformity to formal rules", as opposed
to "*excessive*"?), but I do think it's important to keep in mind that
these procedures exist. They do exist for various reasons, usually good
ones, but they do reduce the appeal of contributing.

Emacs just thinks it's more important to have those procedures than to
have more contributors. Which is a perfectly valid decision to make.


Regards,
Jorgen

Re: Emacs contributions, C and Lisp

[Jorgen Schaefer]

On Mon, 17 Feb 2014 18:58:21 -0500
Stefan Monnier <monnier@iro.umontreal.ca> wrote:

>         Stefan "who'd be happy to hear more about the hurdles to
>                 integration of elpy into GNU ELPA"

Well, one hurdle is that I haven't found a document that explains what
requirements there are to get a package into GNU ELPA. I know from
reading this list that copyright assignment is required (which is a bit
annoying for new contributors) and that it somehow involves pushing to
a git repo.

But the main reason I haven't even looked at it any further is that
elpy currently depends on a number of packages that are not in GNU
ELPA. Specifically:

- auto-complete
- find-file-in-project
- highlight-indentation
- iedit
- nose

Additionally, there has been some pressure from users to replace
flymake with flycheck, which is also not in Emacs nor GNU ELPA.

(Well, and until recently there was virtualenv, but I replaced that with
my own pyvenv which I could contribute to GNU ELPA quite easily.)

From reading this list, I got the impression that GNU packages shouldn't
suggest using third-party package archives because they usually do not
care about promoting free software, so I can not simply ask users to
add e.g. Marmalade and get the packages from there.

I plan to add some kind of extension handling mechanism in elpy to
enable/disable optional dependencies, but haven't progressed much in
figuring out how to do this well while keeping the general idea of elpy
to "install it and it just works". And of course that requires some
time to actually implement it.

Regards,
Jorgen

Re: Emacs contributions, C and Lisp

[Thien-Thi Nguyen]

[-- Attachment #1: Type: text/plain, Size: 916 bytes --]

() Jorgen Schaefer <forcer@forcix.cx>
() Wed, 19 Feb 2014 08:05:24 +0100

   (so, re-reading the Wikipedia entry on "red tape" I guess
   the term was badly chosen, sorry, not a native speaker;
   what's a good term for "*lots* of regulation and rigid
   conformity to formal rules", as opposed to "*excessive*"?)

Coordinated motion can be "dance" if all participants are
harmonious, "grind" if duration makes for tiredness, "combat"
if there is adversarial nature, and of course, any combination
of these independent qualities (or lack, or partial).  I think
everyone has different ideas on what is the ideal combination.
(IMHO, the ideal is non-grinding dancing combat.)

-- 
Thien-Thi Nguyen
   GPG key: 4C807502
   (if you're human and you know it)
      read my lisp: (responsep (questions 'technical)
                               (not (via 'mailing-list)))
                     => nil

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: Emacs contributions, C and Lisp

[David Kastrup]

Jorgen Schaefer <forcer@forcix.cx> writes:

> On Mon, 17 Feb 2014 22:29:47 +0200
> Eli Zaretskii <eliz@gnu.org> wrote:
>
>> > There are a few other minor problems for me. For example, my last
>> > foray in adding a patch to Emacs was so scary regarding the amount
>> > of red tape involved in the whole process that I am somewhat
>> > reluctant to commit to doing that regularly.
>> 
>> What red tape?  Emacs is about the most red-tape-less project as you
>> can find, as far as the procedure of admitting a patch is considered.
>
> If I want to contribute to Emacs, and I want to be good contributor, I
> have the following things to keep in mind:
>
> - Make sure I have done my copyright assignment, and for larger work,
>   do the same for all contributors.
> - Read etc/CONTRIBUTE and follow all the steps
> - Which asks me to read the GNU Coding Standards
> - Oh, and "Tips and Conventions" in the emacs Lisp Reference
>   appendix
> - Learn a new tool (bzr), for which I have to read

[... Lots of bullet points centered around bzr]

I've never learnt working with Bzr and never checked out the Emacs
repository with it.

That did not preclude me from sending the occasional patch to the list,
based on the Git mirror of the Emacs repository.

You are confusing "contributing to Emacs" with "being a core developer
of Emacs", one who is able and willing to make the decisions about what
to push where and when to ask back all on his own.

> Emacs just thinks it's more important to have those procedures than to
> have more contributors. Which is a perfectly valid decision to make.

That's a bad representation of the issue.  We are not talking about
"more important" but rather "prerequisites".  With regard to coding
styles, it's fine to submit a patch without heeding them but it does
cause more work for everybody involved than if you take a look at the
respective perfectly available information yourself at your own pace and
initiative.

With regard to copyright assignments, you pretend that it is some magic
ritual of initiation.  There is nobody who'd be more glad than the FSF
if this kind of paperwork was without merit and unneeded.  Thank your
law makers for that.  Or better, tell them that you want them to work on
getting back the situation before the Berne convention where copyright
is only asserted when a copyright notice is placed in a document.

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[Bastien]

Hi Jorgen,

Jorgen Schaefer <forcer@forcix.cx> writes:

> Emacs just thinks it's more important to have those procedures than to
> have more contributors. Which is a perfectly valid decision to make.

Emacs does not "think" anything, but maintainers, contributors and
users do.

The current behavior of maintainers is to apply patches that are not
problematic and to discuss those which are, I don't really see what
can be possibly wrong here.

Anyway, feel free.

-- 
 Bastien

Re: Emacs contributions, C and Lisp

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:
>> > What red tape?  Emacs is about the most red-tape-less project as you
>> > can find, as far as the procedure of admitting a patch is considered.
>> 
>> 
>> This is an interesting difference of opinion and is probably worth
>> exploring; it's worth knowing what the difficulties where.
>
> That's why I asked what kind of red tape was being alluded to here.

My apologies I had taken it as a rhetorical question.


>> It might also be writing this up -- is there a description anywhere of
>> "how to submit a change to Emacs", and should it not be in the elisp
>> manual?
>
> There's nothing to describe, IMO.  If you don't have write access,
> just post the changes and ask for review or approval; if you do have
> write access, you just commit.


Well, you just did just describe it, and it's a very nice, short and
succient description.


>> Incidentally, I say "change" and not "patch". Submitting a change these
>> days generally means "clone, branch, pull request".
>
> Sorry, I don't understand the subtlety.

Sending patches is quite slow and generally painful. This can be quite
an impediment to "drive-by" fixes. Consider (my sole) contribution to
biopython:

commit a85cb7b546a7a3928a578ed05170581c0365c438
Author: Phillip Lord <phillip.lord@newcastle.ac.uk>
Date:   Mon Sep 30 17:00:13 2013 +0100

    Fix to the English of a section title.

	Modified   Doc/Tutorial.tex
diff --git a/Doc/Tutorial.tex b/Doc/Tutorial.tex
index dc1d0a3..f46007d 100644
--- a/Doc/Tutorial.tex
+++ b/Doc/Tutorial.tex
@@ -1380,7 +1380,7 @@ used in sequencing work hold quality scores but they \emph{never} contain a
 sequence -- instead there is a partner FASTA file which \emph{does} have the
 sequence.
 
-\section{Working with directly strings}
+\section{Working with strings directly}
 \label{sec:seq-module-functions}
 To close this chapter, for those you who \emph{really} don't want to use the sequence
 objects (or who prefer a functional programming style to an object orientated one),


Only doable when the effort is very small.

Phil

Re: Emacs contributions, C and Lisp

[Eli Zaretskii]

> From: chad <chadpbrown@gmail.com>
> Date: Tue, 18 Feb 2014 21:33:26 -0800
> Cc: emacs <emacs-devel@gnu.org>
> 
> To some degree, but (my impression is that) Emacs stops a lot of
> changes at the need for support on more platforms than the submitter
> can easily develop on.

Can you give a couple of examples?

In any case, this is mainly relevant to the C level; on the Lisp
level, the code is mostly platform independent.

> > Which projects don’t?
> 
> Most software projects today are much smaller, more nimble, and
> piecemeal-clockwork than emacs. Emacs was built in a heavily
> monolithic era, and continues with most of that lineage. Far more
> software projects today are assembled from many interlocking pieces
> of libraries, middleware, and plugin code. Again, there are good
> reasons for this, but it has a cost, and much of that cost is
> expressed as a high barrier to entry.

But then it's not very useful to talk about this, because obviously
Emacs will never become as small as those projects, or somehow will be
divided into many almost unrelated sub-projects.  There's nothing we
can do about Emacs being a large and complex package.

> Another thing that hits Emacs: people working on changing or extending
> Eclipse do it IN java, which is the same language they use Eclipse
> FOR. While there are surely Emacs developers using Emacs to develop
> Emacs Lisp, theyre doing so mostly just for Emacs, not for anything
> else; this internal focus shows in the results.

Not sure what you want to say here.  Is it that most people don't know
Lisp and need to learn it?

> Emacs own C code is highly idiosyncratic.

??? When did you last look at the C sources of a comparable project,
like GDB or GCC?  Or Guile, for that matter?

> Again, Im aware that there are good reasons for all of these things
> to be true, but each of them makes it harder for people to make
> significant contributions to Emacs, which is the topic at hand.

It's not useful, IMO, to discuss circumstances we cannot possibly
change.  We need to make contributing as simple as possible, but no
simpler.  Finding that golden path is the challenge that is worth
discussing.

Re: Emacs contributions, C and Lisp

[Phillip Lord]

David Kastrup <dak@gnu.org> writes:
>>> Incidentally, I say "change" and not "patch". Submitting a change these
>>> days generally means "clone, branch, pull request".
>>
>> Sorry, I don't understand the subtlety.
>
> I think it's a paraphrase of "real developers use GitHub", a service
> running on proprietary software and having a variety of commercial
> offerings, including a rather popular "take it or leave it" zero
> pricetag offering.

No, it really wasn't. Github is useful but also problematic for the
reasons that you give. Even aside from the ethical issues, the zero
pricetag option could disappear anytime.

But pull requests are not a feature of github only. They are just a
workflow and an easy to use one.

I'm not a real developer anyway. Check out my code, I'm sure you'd
agree.

Phil

Re: Emacs contributions, C and Lisp

[Phillip Lord]

Glenn Morris <rgm@gnu.org> writes:

> Phillip Lord wrote:
>
>> It might also be writing this up -- is there a description anywhere of
>> "how to submit a change to Emacs", and should it not be in the elisp
>> manual?
>
> etc/CONTRIBUTE
> Info node `(emacs)Contributing'

Ah, thanks. Was looking in the elisp manual instead, but it is good.

Once I had found etc/CONTRIBUTE, I read it. It is quite long though. And
it has statements like:

It is important to write your patch based on the latest version. If you
start from an older version, your patch may be outdated (so that
maintainers will have a hard time applying it), or changes in Emacs may
have made your patch unnecessary.

Every patch must have several pieces of information before we can
properly evaluate it. When you have all these pieces, bundle them up in
a mail message and send it to the developers.

Please use "Context Diff" format.



It does seem rather heavy-weight to me.

Phil

Re: Emacs contributions, C and Lisp (was: Re: /srv/bzr/emacs/trunk r101338: ...)

[Eli Zaretskii]

> Date: Wed, 19 Feb 2014 08:05:24 +0100
> From: Jorgen Schaefer <forcer@forcix.cx>
> Cc: emacs-devel@gnu.org
> 
> On Mon, 17 Feb 2014 22:29:47 +0200
> Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > > There are a few other minor problems for me. For example, my last
> > > foray in adding a patch to Emacs was so scary regarding the amount
> > > of red tape involved in the whole process that I am somewhat
> > > reluctant to commit to doing that regularly.
> > 
> > What red tape?  Emacs is about the most red-tape-less project as you
> > can find, as far as the procedure of admitting a patch is considered.
> 
> If I want to contribute to Emacs, and I want to be good contributor, I
> have the following things to keep in mind:
> [...]

Are you saying that other comparable projects don't have similar
requirements?  Because that'd be definitely not true for the projects
I'm familiar with.

Here are the requirements for GDB contributors:

  https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=blob;f=gdb/CONTRIBUTE;hb=HEAD

Here are the GCC requirements:

  http://gcc.gnu.org/contribute.html

Here's LLVM's (which cover clang):

  http://llvm.org/docs/DeveloperPolicy.html

As you see, Emacs is the norm here, not the odd one out.  In fact, the
Emacs requirements are simpler and fewer than those of other projects.
E.g., GDB requires that every patch be accompanied by a test (which
requires you to install Expect and DejaGnu, and learn how to write
tests), and any user-visible change must come with suitable changes to
the user manual.  You can browse the discussions on gdb-patches list,
where you will see that some changesets are submitted quite a few
times before they are finally approved, something that you will almost
never see here.  As another example, I was recently asked to submit my
changes to Guile in "git format-patch" format, whereas Emacs never
required any bzr-specific procedures for patch submission (as David
points out).

So I really have no idea what is this gripe all about; we just do what
every other project out there does.  I can see how it would be easier
for contributors to just use fire-and-forget methods, but that's not
how projects such as Emacs work.

> Now, do not get me wrong. I am not complaining about these requirements
> (so, re-reading the Wikipedia entry on "red tape" I guess the term was
> badly chosen, sorry, not a native speaker; what's a good term for
> "*lots* of regulation and rigid conformity to formal rules", as opposed
> to "*excessive*"?), but I do think it's important to keep in mind that
> these procedures exist. They do exist for various reasons, usually good
> ones, but they do reduce the appeal of contributing.

If there are better alternatives that are practical, please describe
them.  Since many other projects of comparable size work like we do, I
rather doubt there is a good alternative.  And if so, why do we need
to waste time discussing something that cannot be changed?

> Emacs just thinks it's more important to have those procedures than to
> have more contributors. Which is a perfectly valid decision to make.

Please show some comparable project that made some other decision in
this situation.

Re: Emacs contributions, C and Lisp

[Eli Zaretskii]

> From: phillip.lord@newcastle.ac.uk (Phillip Lord)
> Cc: <forcer@forcix.cx>, <dgutov@yandex.ru>, <emacs-devel@gnu.org>
> Date: Wed, 19 Feb 2014 16:43:54 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> >> > What red tape?  Emacs is about the most red-tape-less project as you
> >> > can find, as far as the procedure of admitting a patch is considered.
> >> 
> >> 
> >> This is an interesting difference of opinion and is probably worth
> >> exploring; it's worth knowing what the difficulties where.
> >
> > That's why I asked what kind of red tape was being alluded to here.
> 
> My apologies I had taken it as a rhetorical question.

It wasn't.  "Red tape" means bureaucracy, as I'm sure you know, and I
wondered where's that in Emacs.

> Sending patches is quite slow and generally painful. This can be quite
> an impediment to "drive-by" fixes. Consider (my sole) contribution to
> biopython:

See my other mail in this thread.  In a nutshell, this is how things
are done in every project I was ever involved with, so Emacs is not an
exception here.

Re: Emacs contributions, C and Lisp

[Phillip Lord]

David Kastrup <dak@gnu.org> writes:
> I've never learnt working with Bzr and never checked out the Emacs
> repository with it.
>
> That did not preclude me from sending the occasional patch to the list,
> based on the Git mirror of the Emacs repository.

etc/CONTRIBUTE doesn't mention this.

> With regard to copyright assignments, you pretend that it is some magic
> ritual of initiation.  There is nobody who'd be more glad than the FSF
> if this kind of paperwork was without merit and unneeded.  Thank your
> law makers for that.  Or better, tell them that you want them to work on
> getting back the situation before the Berne convention where copyright
> is only asserted when a copyright notice is placed in a document.

Copyright assignment is a significant barrier to entry, I think, when
projects have started *outside* emacs and then come in. Getting your own
papers done is enough without having to encouraging others. Not the FSF
fault, I agree.

Perhaps the FSF should generate a nice icon, or have a public list (for
those that want it) of those who have already done this assignment. It
would be a small badge of honour, but would also make it easier for
people developing outside whether a potential contributor has already
had this done. A FSF-ready icon would be good also, that people could
put in their README.

Just a thought.

Phil

Re: Emacs contributions, C and Lisp

[David Kastrup]

phillip.lord@newcastle.ac.uk (Phillip Lord) writes:

> David Kastrup <dak@gnu.org> writes:
>> I've never learnt working with Bzr and never checked out the Emacs
>> repository with it.
>>
>> That did not preclude me from sending the occasional patch to the list,
>> based on the Git mirror of the Emacs repository.
>
> etc/CONTRIBUTE doesn't mention this.

It states for the source code:

    * Getting the Source Code

    The latest version of Emacs can be downloaded using Bazaar from the
    Savannah web site.  It is important to write your patch based on the
    latest version.  If you start from an older version, your patch may be
    outdated (so that maintainers will have a hard time applying it), or
    changes in Emacs may have made your patch unnecessary.

    After you have downloaded the repository source, you should read the file
    INSTALL.REPO for build instructions (they differ to some extent from a
    normal build).

    Ref: http://savannah.gnu.org/projects/emacs

and that reference is equally valid for Bzr and Git.  For the patch, you
get

    ** The patch itself.

    If you are accessing the Bazaar repository, make sure your copy is
    up-to-date (e.g. with `bzr pull'), then use
            bzr diff --no-aliases --diff-options=-cp
    Else, use
            diff -cp OLD NEW

As you see there is an _explicit_ recipe for creating a patch without
using Bzr.  With regard to git, one can use C-u C-x v = to create a
patch, and if one wants to be really RMS-compatible, C-u C-c C-u in the
*vc-diff* buffer will convert it into a context diff.

It turns out that most Emacs developers nowadays are fine with unified
diffs, however.

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[David Engster]

Eli Zaretskii writes:
> As you see, Emacs is the norm here, not the odd one out.  In fact, the
> Emacs requirements are simpler and fewer than those of other projects.

One should also add that Stefan isn't shy of giving access to the
repository, which - at least from my experience - is rather unusual.

-David

Re: Emacs contributions, C and Lisp

[Glenn Morris]

> It is important to write your patch based on the latest version. If you
> start from an older version, your patch may be outdated (so that
> maintainers will have a hard time applying it), or changes in Emacs may
> have made your patch unnecessary.

From experience, this is a necessary statement.

For example, you are proving that here by commenting on a CONTRIBUTE
version that I can tell is not the version in trunk. :)
(Probably there are no major differences.)


I think the only really important stuff is the first sentence.
Everything else boils down to "send a patch".

Re: Emacs contributions, C and Lisp

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:
> So I really have no idea what is this gripe all about; we just do what
> every other project out there does.  I can see how it would be easier
> for contributors to just use fire-and-forget methods, but that's not
> how projects such as Emacs work.
>
>> Now, do not get me wrong. I am not complaining about these requirements
>> (so, re-reading the Wikipedia entry on "red tape" I guess the term was
>> badly chosen, sorry, not a native speaker; what's a good term for
>> "*lots* of regulation and rigid conformity to formal rules", as opposed
>> to "*excessive*"?), but I do think it's important to keep in mind that
>> these procedures exist. They do exist for various reasons, usually good
>> ones, but they do reduce the appeal of contributing.
>
> If there are better alternatives that are practical, please describe
> them.  Since many other projects of comparable size work like we do, I
> rather doubt there is a good alternative.  And if so, why do we need
> to waste time discussing something that cannot be changed?


The difference between Emacs and GDB is, I think, that GDB is all quite
low-level. Emacs covers a lot more ground from the C guts, to the user
facing code and documentation.

If I ever submit a patch to the C code base, I'd probably expect a lot
of discussion. But, for some parts of Emacs, fire-and-forget is probably
much more reasonable.

So, here are my two alternatives: 

1) Rewrite CONTRIBUTE so that it's info (and thus also webable). Random
   files in etc are only easy to find iff you have a source tarball.

2) Section out CONTRIBUTE, so those wanting to contribute to the easier
   parts don't have to read everything. I think this might help to
   decrease the wall of text experience that it is at the moment.


I'll get to it once I've redone the tutorial which I offered to do a
while back!

Phil

Re: Emacs contributions, C and Lisp

[Eli Zaretskii]

> From: phillip.lord@newcastle.ac.uk (Phillip Lord)
> Cc: Jorgen Schaefer <forcer@forcix.cx>, <emacs-devel@gnu.org>
> Date: Wed, 19 Feb 2014 18:06:16 +0000
> 
> The difference between Emacs and GDB is, I think, that GDB is all quite
> low-level. Emacs covers a lot more ground from the C guts, to the user
> facing code and documentation.

The same is true for GDB: it is a complete program, and includes both
low-level stuff, and user interaction and documentation.

Re: Emacs contributions, C and Lisp

[Glenn Morris]

Phillip Lord wrote:

> 1) Rewrite CONTRIBUTE so that it's info (and thus also webable).

I forgot to mention that it's linked from our homepage.

http://www.gnu.org/software/emacs/CONTRIBUTE

> Random files in etc are only easy to find iff you have a source
> tarball.

Or read the manual, or do a web-search.

Oh, it's also linked from the "About Emacs" screen.

Re: Emacs contributions, C and Lisp

[Stefan Monnier]

> Perhaps the FSF should generate a nice icon, or have a public list (for
> those that want it) of those who have already done this assignment.

Some way to encourage people to do that and promote it as something
desirable would be a good idea, indeed.


        Stefan

Re: Emacs contributions, C and Lisp (was: Re: /srv/bzr/emacs/trunk r101338: ...)

[Jorgen Schaefer]

On Wed, 19 Feb 2014 19:11:57 +0200
Eli Zaretskii <eliz@gnu.org> wrote:

> So I really have no idea what is this gripe all about

My only gripe is about you continuously ignoring what I am saying and
respond to something entirely different.

You asked: Why is Elpy not in Emacs?

I gave you a list of reasons. Among them, I mentioned, as a *minor*
point, that *I* am currently not prepared to put in the commitment to
follow all the requirements for a proper contribution to Emacs.

I even said *explicitly* that I am *not complaining* about these
requirements.

Nevertheless, you managed to completely ignore what I said and respond
as if I said something entirely different.

And you managed to do this repeatedly.

I say "there are plenty of things (packages) already in Emacs that
have the same problem", and followed that statement with a list of
examples. You manage to quote that first sentence and go "please
point them out."

You say "get these packages into Emacs and the problems will become
less". I give CEDET as an example of a package that *is* in Emacs,
which hasn't solved the problems at all. You manage to respond to this
by asking me to "put my money where my mouth is".

Not to mention that you managed to respond to a statement about
"clang-based extensions", have a multi-email discussion about where you
quoted this phrase, and in the end managed to drop the quote and go
"oh, I didn't know they're clang-based".

I have no idea how I can have a constructive discussion with someone
who is so obviously intent on reading what they want to read instead of
trying to read what the other party is saying, so I will now cease to
try.

Have a good day.

Jorgen

Re: Emacs contributions, C and Lisp (was: Re: /srv/bzr/emacs/trunk r101338: ...)

[Eli Zaretskii]

> Date: Wed, 19 Feb 2014 20:05:06 +0100
> From: Jorgen Schaefer <forcer@forcix.cx>
> Cc: emacs-devel@gnu.org
> 
> On Wed, 19 Feb 2014 19:11:57 +0200
> Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > So I really have no idea what is this gripe all about
> 
> My only gripe is about you continuously ignoring what I am saying and
> respond to something entirely different.

I don't think I did:

> > > There are a few other minor problems for me. For example, my last
> > > foray in adding a patch to Emacs was so scary regarding the amount
> > > of red tape involved in the whole process that I am somewhat
> > > reluctant to commit to doing that regularly.
> > 
> > What red tape?  Emacs is about the most red-tape-less project as you
> > can find, as far as the procedure of admitting a patch is considered.
> 
> If I want to contribute to Emacs, and I want to be good contributor, I
> have the following things to keep in mind:

You mentioned "red tape", I asked what was that red tape, you
described it, I replied that you will see the same in any other
project.  That is all.

> I have no idea how I can have a constructive discussion with someone
> who is so obviously intent on reading what they want to read instead of
> trying to read what the other party is saying, so I will now cease to
> try.

Sorry.

Re: Emacs contributions, C and Lisp

[David Kastrup]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Perhaps the FSF should generate a nice icon, or have a public list (for
>> those that want it) of those who have already done this assignment.
>
> Some way to encourage people to do that and promote it as something
> desirable would be a good idea, indeed.

Assignments are not per-person, but rather per-person-and-project.
I think there is a single person who managed to get a general assignment
for _all_ GNU projects because he insisted.  But that's a rather big
exception.

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[Stephen J. Turnbull]

David Kastrup writes:

 > Assignments are not per-person, but rather per-person-and-project.

Indeed, that's a practical issue.  I don't think it would be all that
hard to get an exception if you really want it -- but it could make
you unemployable for many companies.

However, even given the practical problem, if a person has signed for
*one* project, it's relatively likely that they'll sign for others.
So it's a worthwhile idea.

Re: Emacs contributions, C and Lisp

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: phillip.lord@newcastle.ac.uk (Phillip Lord)
>> Cc: Jorgen Schaefer <forcer@forcix.cx>, <emacs-devel@gnu.org>
>> Date: Wed, 19 Feb 2014 18:06:16 +0000
>> 
>> The difference between Emacs and GDB is, I think, that GDB is all quite
>> low-level. Emacs covers a lot more ground from the C guts, to the user
>> facing code and documentation.
>
> The same is true for GDB: it is a complete program, and includes both
> low-level stuff, and user interaction and documentation.


GDB has an interface for interacting with programmers. Emacs is a text
editor; I think that there are some people using it just because of
org-mode. So, different things, I think.

Phil

Re: Emacs contributions, C and Lisp

[Phillip Lord]

Glenn Morris <rgm@gnu.org> writes:
>>From experience, this is a necessary statement.
>
> For example, you are proving that here by commenting on a CONTRIBUTE
> version that I can tell is not the version in trunk. :)
> (Probably there are no major differences.)
>
>
> I think the only really important stuff is the first sentence.
> Everything else boils down to "send a patch".


If this is true, the main entry point should be a page which says "send
a patch" and little else.

I think the issue here is what barriers are there to people who have not
contributed from doing so. Not, why do people on emacs-devel find no
barriers to contribution.

Phil

Re: Emacs contributions, C and Lisp

[Phillip Lord]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> David Kastrup writes:
>
>  > Assignments are not per-person, but rather per-person-and-project.
>
> Indeed, that's a practical issue.  I don't think it would be all that
> hard to get an exception if you really want it -- but it could make
> you unemployable for many companies.
>
> However, even given the practical problem, if a person has signed for
> *one* project, it's relatively likely that they'll sign for others.
> So it's a worthwhile idea.

Especially if the icon is clickable and jumps to page giving a personal
profile.

Phil

Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[Davis Herring]

> BTW, I really don't know yet what will work best.  Maybe the
> "save&restore only those vars we've been told" as is currently done is
> indeed the best option.  But in that case it shouldn't be done via
> a variable listing those symbols, but e.g. by adding a special
> property to those symbols.

A C programmer wouldn't save and restore many variables, but rather
would swap pointers to structs (or, in the dynamic case, lists).  This
suggests that there could be a "mode-local" object: either a well-known
map in ELisp (which would require explicit lookups, as in the C case) or
a parallel to buffer-local variables in C.  One could even obtain it
from text properties at point.

Davis

-- 
This product is sold by volume, not by mass.  If it appears too dense or
too sparse, it is because mass-energy conversion has occurred during
shipping.

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    > Whatever features we put in Emacs for interacting with a compiler in
    > any fashion should work with GCC.

    Only to the extent that GCC supports that feature.

This decision applies also to features that GCC doesn't yet support.
The outcome we want is that GCC supports them and Emacs uses GCC.

Emacs should not aim to advance by helping LLVM push GCC down.

If you need specific features added to GCC, please tell me.  I will
push on the GCC maintainers to prioritize them, and Emacs developers
can help by offering patches to implement parts of them.


-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    > Whatever features we put in Emacs for
    > interacting with a compiler in any fashion should work with GCC.

    Richard, have you tried discussing this with GCC developers recently?

I don't follow you.  What exactly is it you suggest I discuss with
them?  I can discuss things with them, but to do so usefully I need
to know what issue to raise.

    Considering your distaste for Clang, this page should look pretty
    embarrassing: http://gcc.gnu.org/wiki/CodeCompletion

I looked at it and saw no obvious point.  However, I did not follow
the links because that would have taken me another day.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[Dmitry Gutov]

On 20.02.2014 20:13, Richard Stallman wrote:
>      Richard, have you tried discussing this with GCC developers recently?
>
> I don't follow you.  What exactly is it you suggest I discuss with
> them?  I can discuss things with them, but to do so usefully I need
> to know what issue to raise.

You could discuss adding code completion, as a core GCC feature, or as 
an actively maintained plugin.

Basically, you'd be able to call GCC with a certain argument, pass it 
the path to a file, line and column within it, and GCC would output a 
list of possible completions at that point (with some type information: 
e.g. if a completion is a function, it would include the arglist and the 
return type).

It should also support receiving the contents of the file from stdin, 
because this way the buffer doesn't have to be saved, for code 
completion to work. Clang supports that.

I couldn't find any real documentation for the format of the Clang 
completion output, but you can see some examples at the beginning of 
this page of this random forum thread: 
http://forums.codeblocks.org/index.php?topic=13559.15

>      Considering your distaste for Clang, this page should look pretty
>      embarrassing: http://gcc.gnu.org/wiki/CodeCompletion
>
> I looked at it and saw no obvious point.  However, I did not follow
> the links because that would have taken me another day.

No need. My point is, the wiki of the official GCC website "promotes" 
Clang because GCC itself lacks the respective feature.

Re: Emacs contributions, C and Lisp

[David Kastrup]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 20.02.2014 20:13, Richard Stallman wrote:
>>      Richard, have you tried discussing this with GCC developers recently?
>>
>> I don't follow you.  What exactly is it you suggest I discuss with
>> them?  I can discuss things with them, but to do so usefully I need
>> to know what issue to raise.
>
> You could discuss adding code completion, as a core GCC feature, or as
> an actively maintained plugin.
>
> Basically, you'd be able to call GCC with a certain argument, pass it
> the path to a file, line and column within it, and GCC would output a
> list of possible completions at that point (with some type
> information: e.g. if a completion is a function, it would include the
> arglist and the return type).

I think it would make sense to pass that info via pipe or socket and
receive the output similarly.  That way each file only needs to get
parsed once, not repeatedly for every completion.

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[Dmitry Gutov]

David Kastrup <dak@gnu.org> writes:

> I think it would make sense to pass that info via pipe or socket and
> receive the output similarly.  That way each file only needs to get
> parsed once, not repeatedly for every completion.

Isn't this what I wrote in the next paragraph, after the one you quoted?

I don't see how you would parse the file only once, though. The user
types a new word -> you have to parse it again, no?

Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[Andreas Röhler]

Am 12.02.2014 02:30, schrieb Stefan Monnier:
>>> IIRC the reason was so that (syntax-ppss) called from a place like
>>> a "column-0 just before an open paren" can correctly indicate if this
>>> open paren is within a string/comment while at the same time parsing the
>>> subsequent text assuming that this open paren was outside of any
>>> string/comment.
>> Why? That sounds confusing.
>
> That's the point: it makes it possible to find those sources of
> confusion and highlight them.
>
> E.g. I had some font-lock code which would highlight an
> open-paren-in-column-0-in-string/comment with the `warning' face.
> So such an "incorrect" open paren would still cause incorrect
> highlighting, but the `warning' face on it would provide the clue as to
> what was the source of the problem.
>
>> I would've bound `syntax-ppss-last` and `syntax-ppss-cache` to nil around
>> the region and let it be forced to call the syntax-begin-function.
>
> Right, but that largely defeats the purpose of syntax-ppss (which is to
> use caching to speed up (parse-partial-sexp (point-min) (point))).
>
>
[ ... ]

Wouldn't it be cleaner to forget about syntax-ppss here and resort to (parse-partial-sexp BEGINNING-of-Subsection point) ?

BTW org-babel deals with a lot of modes nicely. Why not port the code from there?

Re: Emacs contributions, C and Lisp

[John Yates]

[-- Attachment #1: Type: text/plain, Size: 1863 bytes --]

I would like to hear the gcc developers weigh in on whether such a tool
(namely a variant invocation of gcc running through all of its lexing,
parsing and semantic analysis) could ever be made to operate at speeds that
would make gcc-based completion tolerable.  And if not then in what way
could the gcc codebase be leverage to support realistic completion?

This goes to a crucial part of the discussion that I feel RMS fails to
acknowlege.  Namely that clang is not a compiler frontend per se but a set
of performance-focused components targeted at building C++-aware (often
interactive) tools.  clang-based completion is not an invocation of a
compiler built by grafting clang onto llvm but rather communication with a
caching server built using clang components.

We have existence proof that one can use clang libraries to build a C++
compiler competing with gcc.  I wonder how realistic it is to expect gcc to
be turned into components enabling construction of tools similar to those
using clang:

http://clang.llvm.org/doxygen/classclang_1_1tooling_1_1RefactoringTool.html
http://clang-analyzer.llvm.org/
https://code.google.com/p/include-what-you-use/

To what extent does the "do not offer interfaces that might enable
repurposing of gcc code for non-free projects" make it unlikely that such
components might ever emerge?

/john



On Thu, Feb 20, 2014 at 3:51 PM, Dmitry Gutov <dgutov@yandex.ru> wrote:

> David Kastrup <dak@gnu.org> writes:
>
> > I think it would make sense to pass that info via pipe or socket and
> > receive the output similarly.  That way each file only needs to get
> > parsed once, not repeatedly for every completion.
>
> Isn't this what I wrote in the next paragraph, after the one you quoted?
>
> I don't see how you would parse the file only once, though. The user
> types a new word -> you have to parse it again, no?
>
>

[-- Attachment #2: Type: text/html, Size: 2661 bytes --]

Re: Emacs contributions, C and Lisp

[David Kastrup]

John Yates <john@yates-sheets.org> writes:

> I would like to hear the gcc developers weigh in on whether such a tool
> (namely a variant invocation of gcc running through all of its lexing,
> parsing and semantic analysis) could ever be made to operate at speeds that
> would make gcc-based completion tolerable.

My guess is that switching off optimization, you'd not be all that bad.

> This goes to a crucial part of the discussion that I feel RMS fails to
> acknowlege.

"Fails to acknowledge" is not the same as "is not going to accept".

> Namely that clang is not a compiler frontend per se but a set of
> performance-focused components targeted at building C++-aware (often
> interactive) tools.  clang-based completion is not an invocation of a
> compiler built by grafting clang onto llvm but rather communication
> with a caching server built using clang components.
>
> We have existence proof that one can use clang libraries to build a
> C++ compiler competing with gcc.

Well, given the right article author.  If we are talking about the
conclusions of Phoronix, see
<URL:http://draketo.de/light/english/free-software/phoronix-distort-results-gcc-llvm-clang-amd-vishera>.

> I wonder how realistic it is to expect gcc to be turned into
> components enabling construction of tools similar to those using
> clang:
>
> http://clang.llvm.org/doxygen/classclang_1_1tooling_1_1RefactoringTool.html
> http://clang-analyzer.llvm.org/
> https://code.google.com/p/include-what-you-use/
>
> To what extent does the "do not offer interfaces that might enable
> repurposing of gcc code for non-free projects" make it unlikely that
> such components might ever emerge?

Well, GNU offers awk, sed, ed, bash, cat, cut, paste, sort, tac, ptx,
bison, gcc itself and a number of other tools that can be used to build
and/or plug together proprietary applications from free components.

There are also specialized tools like idutils
<URL:https://www.gnu.org/software/idutils/manual/html_node/index.html>
that are explicitly intended to extract and provide program data.

And while it would be cumbersome to extract, it would be possible to
compile with -O0 -ggdb and get a lot of type info in the object file.

So there is quite a bit of precedence: this question is not a
black&white one but needs to get answered by weighing advantages and
disadvantages.  Splitting parser and code generator into separately
usable components that can be considered independent would allow using
GNU frontends with proprietary backends for selected
processors/coprocessors without causing the "software as a whole" clause
to trigger.

Or proprietary frontends (like the Objective C compiler once was) with
the code generator of GCC.

Allowing for either of these use cases is giving up a lot of control
over derivatives that is the principal motivation for the GPL.  The
disadvantage of that has to be weighed against the advantages.

Even when that has been done, one still has to do an actual
implementation and _then_ have to see whether under the constraints of
the current implementation the hoped-for advantages are actually as
envisioned, and the accepted drawbacks don't turn out larger than
expected.  And there is not much of an option to turn back the clock,
either.

In practice, there is not all that much willingness to do an open-minded
and open-ended investigation and accept the conclusions either way.
I don't exactly blame Richard for not being eager to open a can of worms
that is causing enough of a nuisance already while the lid is on.

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[John Yates]

[-- Attachment #1: Type: text/plain, Size: 2593 bytes --]

On Thu, Feb 20, 2014 at 6:53 PM, David Kastrup <dak@gnu.org> wrote:
>
> My guess is that switching off optimization, you'd not be all that bad.
>

I already assumed that a magic "give me completions" flag would turn off
optimization and code generation.  I am concerned about process startup,
I/O (even if the required headers are already in the block cache), lexing,
parsing and semantic analysis.  Remember, we are talking C++ here, not C.
 Trivial programs pull in mega headers (stl, boost, etc) with increasing
use of such wonderful language features as "Turing-complete template
meta-programming".

"Fails to acknowledge" is not the same as "is not going to accept".
>

Let me state that more clearly.  Correct me if I am wrong but I believe
that RMS has never done any C++ development.  As such he has little
appreciation for the value C++ developers may attach to tools with deep
knowledge of C++ semantics.

Well, given the right article author.  If we are talking about the
> conclusions of Phoronix, see
> <URL:
> http://draketo.de/light/english/free-software/phoronix-distort-results-gcc-llvm-clang-amd-vishera
> >.
>

I used the word "competing" to mean "offering similar service or
functionality".  I did not intend to bring in any notion of relative
performance of generated code.

Splitting parser and code generator into separately
> usable components...


That is not the issue.  Given that the gcc backend can be linked with a
number of different frontends I assume that stubbing off a few functions
would provide a free-standing frontend.  OTOH if gcc wants to "compete"
with clang in the tools space it will have to provide interfaces for
querying and manipulating its symbol table and intermediate representation.
 That is some rather heavy lifting for a codebase not designed from the
outset with that goal in mind


> Even when that has been done, one still has to do an actual
> implementation and _then_ have to see whether under the constraints of
> the current implementation the hoped-for advantages are actually as
> envisioned, and the accepted drawbacks don't turn out larger than
> expected.  And there is not much of an option to turn back the clock,
> either.
>

Do I understand you correctly here?  Did you just conceded my original
point, namely that while RMS does not want us to dissipate the pent up
demand that could drive gcc into the tool space, that transformation most
likely will never happen?  IOW emacs-based C++ developers are denied
valuable tools that are becoming ever more common in other development
environments based on a chimera.

/john

[-- Attachment #2: Type: text/html, Size: 4119 bytes --]

Re: Emacs contributions, C and Lisp

[David Kastrup]

John Yates <john@yates-sheets.org> writes:

>> Even when that has been done, one still has to do an actual
>> implementation and _then_ have to see whether under the constraints
>> of the current implementation the hoped-for advantages are actually
>> as envisioned, and the accepted drawbacks don't turn out larger than
>> expected.  And there is not much of an option to turn back the clock,
>> either.
>
> Do I understand you correctly here?  Did you just conceded my original
> point, namely that while RMS does not want us to dissipate the pent up
> demand that could drive gcc into the tool space, that transformation
> most likely will never happen?

It feels like talking to a wall with you guys.  I am not even mentioning
Richard here.  I am explaining the situation one needs to take into
account.  If you refuse thinking about the situation, you won't ever
reach an agreement.

> IOW emacs-based C++ developers are denied valuable tools that are
> becoming ever more common in other development environments based on a
> chimera.

Stomping your feet and pouting is all very nice but I suggest that you
bother with actually putting yourself in the shoes of Richard with
regard to the conflicting goals he is trying to manage.  Without
bothering to see his side, you will have a hard time changing his views
and what results they lead to.  In particular, since you're one of those
who is going to oversee the technical execution.  If you don't appear to
understand what this is even about, whether or not you share the exact
priorities, you'll not be able to stay on a path that is going to be
quite trickier than a plain "we won't go there".

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    You could discuss adding code completion, as a core GCC feature, or as 
    an actively maintained plugin.

I will do that.  Thanks.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[Stephen J. Turnbull]

Richard Stallman writes:
 > [[[ To any NSA and FBI agents reading my email: please consider    ]]]
 > [[[ whether defending the US Constitution against all enemies,     ]]]
 > [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
 > 
 >     > Whatever features we put in Emacs for interacting with a compiler in
 >     > any fashion should work with GCC.
 > 
 >     Only to the extent that GCC supports that feature.
 > 
 > This decision applies also to features that GCC doesn't yet support.
 > The outcome we want is that GCC supports them and Emacs uses GCC.
 > 
 > Emacs should not aim to advance by helping LLVM push GCC down.
 > 
 > If you need specific features added to GCC, please tell me.

In this particular case, you've been told that outputting fully
annotated ASTs is a desirable feature, and you decided it's a bad idea
(for software freedom) to support it because it allows nonfree
frontends and backends to be written for GCC backends and frontends
respectively.  So you've opposed GCC support for that feature.

In this case you need to choose between allowing any LLVM feature to
be supported because it's free software, or saying LLVM may be free
software but the feature it supports that GCC doesn't is freedom-
denying, and therefore no matter how useful it may be, Emacs is not
going to support that feature.

Re: Emacs contributions, C and Lisp

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> Richard Stallman writes:
>  > [[[ To any NSA and FBI agents reading my email: please consider    ]]]
>  > [[[ whether defending the US Constitution against all enemies,     ]]]
>  > [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>  > 
>  >     > Whatever features we put in Emacs for interacting with a compiler in
>  >     > any fashion should work with GCC.
>  > 
>  >     Only to the extent that GCC supports that feature.
>  > 
>  > This decision applies also to features that GCC doesn't yet support.
>  > The outcome we want is that GCC supports them and Emacs uses GCC.
>  > 
>  > Emacs should not aim to advance by helping LLVM push GCC down.
>  > 
>  > If you need specific features added to GCC, please tell me.
>
> In this particular case, you've been told that outputting fully
> annotated ASTs is a desirable feature, and you decided it's a bad idea
> (for software freedom) to support it because it allows nonfree
> frontends and backends to be written for GCC backends and frontends
> respectively.  So you've opposed GCC support for that feature.

I think it makes some sense to reevaluate this particular choice.  If we
take a look those parties most likely to exploit it, they have moved on
to LLVM mostly.  _And_ they tend to consider the patent clauses of the
GPLv3 a poison pill for any use anyway.

I don't see that the information we are talking about will make it
reasonably straightforward to substitute backends and arrive at a
competitive compiler, though compilation speed is of course secondary to
runtime speed.

So I think that the existence of both LLVM and GPLv3 lessens the
positive impact from the decision while not really changing the negative
consequences.  There is a valid point about not giving people ideas
about how to outnavigate what the GNU project can achieve with the GPL
by introducing projects like LLVM.  But GCC seems rather too important
to cede a tactical loss like that here.

> In this case you need to choose between allowing any LLVM feature to
> be supported because it's free software, or saying LLVM may be free
> software but the feature it supports that GCC doesn't is freedom-
> denying, and therefore no matter how useful it may be, Emacs is not
> going to support that feature.

Well, the last point is where we at nowadays.  I think it makes sense
reevaluating the previous choice and thus making this one moot.  But
yes, that's Richard's call to make, and it would take quite a lot of
time and work to follow up on the consequences of changing that bit.

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[Tom]

Stephen J. Turnbull <stephen <at> xemacs.org> writes:
> 
> In this case you need to choose between allowing any LLVM feature to
> be supported because it's free software, or saying LLVM may be free
> software but the feature it supports that GCC doesn't is freedom-
> denying, and therefore no matter how useful it may be, Emacs is not
> going to support that feature.
> 

Emacs will support that feature, because it's an open system and there
are plenty of clang-based C++ completion packages already. The default
installation may not support it, but it hardly matters considering
most people install external packages anyway to improve on the default
emacs experience. That's what emacs is about after all. Extending and
customizing.

The cat is already out of the bag with clang, so not supporting similar
features in gcc will only push more people towards clang based emacs
packages, considering they are mentioned in the top google results
when you search for "emacs c++ completion".

Re: Emacs contributions, C and Lisp

[David Kastrup]

Tom <adatgyujto@gmail.com> writes:

> The cat is already out of the bag with clang, so not supporting
> similar features in gcc will only push more people towards clang based
> emacs packages, considering they are mentioned in the top google
> results when you search for "emacs c++ completion".

The "cat is out of the bag" for a host of proprietary solutions
(including zero-cost ones) too, so that is a red herring.

The default installations and offerings are something which people
assume to be "politically correct" by default.  That's what they will
feel fine using, contributing to, and recommending to others without the
expectation of requiring second thoughts even if they are fully in
support of the GNU project and the FSF's goals.

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    Namely that clang is not a compiler frontend per se but a set
    of performance-focused components targeted at building C++-aware (often
    interactive) tools.


Remember that the main purpose of the GNU system -- including GNU
Emacs -- is freedom.  Technical progress and convenience are
_secondary_ goals.

Copyleft is needed to defend freedom, which is why Clang is so harmful
to our freedom.  There are already nonfree versions of Clang that do
tremendous harm to our movement.

Allowing nonfree versions of GCC would not help us "win" anything that
matters -- it would only mean surrender.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    In this particular case, you've been told that outputting fully
    annotated ASTs is a desirable feature, and you decided it's a bad idea
    (for software freedom) to support it because it allows nonfree
    frontends and backends to be written for GCC backends and frontends
    respectively.  So you've opposed GCC support for that feature.

Right.  And you understand why.

However, a feature make a background process that responds to
inquiries such as "What's the type of symbol FOO on line L, char C" is
ok.  And that's exactly what we would want for this.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[David Kastrup]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>     Namely that clang is not a compiler frontend per se but a set
>     of performance-focused components targeted at building C++-aware (often
>     interactive) tools.
>
>
> Remember that the main purpose of the GNU system -- including GNU
> Emacs -- is freedom.  Technical progress and convenience are
> _secondary_ goals.
>
> Copyleft is needed to defend freedom, which is why Clang is so harmful
> to our freedom.  There are already nonfree versions of Clang that do
> tremendous harm to our movement.

Quite so.  And there is no point in foregoing potential benefits in
order to protect assets that we no longer have exactly because Clang's
progress has demolished them.

> Allowing nonfree versions of GCC would not help us "win" anything that
> matters -- it would only mean surrender.

Sure, but nobody was talking about "allowing nonfree versions of GCC".
The problem in this discussion is that everybody is talking about
different endeavors, lumping everybody else into the camp of being
opposed to all the goals that he was proposing a project for.

In the context of context-sensitive completion, I see a variety of
actual rather than strawman proposals here:

a) let GCC output the whole AST representation of the input (whatever
that may be) on demand
b) split GCC into separate components with well-defined interfaces
c) create a plugin interface into GCC that is generic enough to provide
the completion stuff
d) extend GCC with specific code exclusively for the intent of providing
completion

None of that would "allow a nonfree version of GCC", but would to
different degrees allow using GCC as a component in a solution that is
not, in itself, free.  Of course, while at the same time allowing using
GCC as a component in a solution that is free.

What to do here depends on how one estimates the respective
probabilities of

a) liberally licensed solutions built on that base
b) GPLed licensed solutions built on that base
c) proprietary solutions build on that base

a) of course enables c) to a certain degree.  Since both a) and c) can
already build on LLVM and since the enthusiastic a) camp will forego any
GPLed solution (which galls them) as much as possible and the c) camp
will try avoiding getting the patent poison pill of GPLv3 in any form
(no solution containing GCC in any form is going to end up in the iOS
Appshop either way, for example), we don't need to bend over backwards
guarding the bag which the cat basically left already.

Which gives us leeway to consider the respective benefits for the task
at hand here, namely completion in Emacs, when looking at any of the
previously mentioned

a) let GCC output the whole AST representation of the input (whatever
that may be) on demand
b) split GCC into separate components with well-defined interfaces
c) create a plugin interface into GCC that is generic enough to provide
the completion stuff
d) extend GCC with specific code exclusively for the intent of providing
completion

I think that Richard already contacted GCC people about option d) so
we'll get this particular angle addressed.

Tightly purposed extensions within the GNU project will likely always be
manageable, and they should address the current problem of completion
fine.

However, they require dedicated extensions in upstream GCC and thus they
require project coordination and cause time lag in the order of years
before downstream can reliably expect to provide its users with working
functionality.

More generic interfaces to extension decouple work on GCC from projects
employing it and thus gives more freedom to create new solutions in a
timely manner.

This decoupling also decouples the control over the direction individual
projects are taking, with aim and licensing.

As I stated: in the current landscape, GPL and particularly GPLv3 with
its patent indemnification clause appear to me as enough of a nuisance
for most prospective code adopters that we stand little risk for a
proliferation of proprietary solutions containing GCC as one separate
component as long as Clang is available in its current state.

The worst case scenario basically is that Clang will end up a footnote
in history and we opened the possibilities of using GCC as a component
in a liberally or proprietarily licensed solution imprudently and cannot
get the lid back on that can of worms.

However, I consider it very unlikely that Clang will end up a footnote
in history all by itself.  Not by now.

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Your argument seems to be based on hostility: you're telling the GNU
Project to give up because defeat is inevitable.

I don't take advice from people who don't seem to support
our goals.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    > Copyleft is needed to defend freedom, which is why Clang is so harmful
    > to our freedom.  There are already nonfree versions of Clang that do
    > tremendous harm to our movement.

    Quite so.  And there is no point in foregoing potential benefits in
    order to protect assets that we no longer have exactly because Clang's
    progress has demolished them.

As a general statement, that is valid -- but I think you're
overestimating Clang's effects on GCC.

    > Allowing nonfree versions of GCC would not help us "win" anything that
    > matters -- it would only mean surrender.

    Sure, but nobody was talking about "allowing nonfree versions of GCC".

Actually yes they were (though not with those words).  Someone cited
my decision against having GCC write a complete syntax tree.  That output
would make it easy to use GCC as a front end for nonfree back-ends.
That would be tantamount to making nonfree versions of GCC.

Splitting up GCC would have the same effect.

The lookup and completion features that people want can be implemented
by making GCC answer questions sent to it, as Aspell does for M-$.
That change would be welcome.  I think it could be implemented using
GCC's existing plug-in mechanism, but it would be better to put
the code into GCC itself.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[David Kastrup]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>     > Copyleft is needed to defend freedom, which is why Clang is so harmful
>     > to our freedom.  There are already nonfree versions of Clang that do
>     > tremendous harm to our movement.
>
>     Quite so.  And there is no point in foregoing potential benefits in
>     order to protect assets that we no longer have exactly because Clang's
>     progress has demolished them.
>
> As a general statement, that is valid -- but I think you're
> overestimating Clang's effects on GCC.

"tremendous harm to our movement" were your words.  Of course, Clang's
effects on GCC are not more or less than which changes we do and permit
doing to GCC.  But the impact of changes to GCC on the rest of the world
is no longer the same due to Clang.

Since we now have our house back to ourselves, we can rethink the rules
we had to make to keep the worst guests from misbehaving.  They've made
themselves at home with Clang now (and I think that the GPLv3 was pretty
effective with that) and it's just a question of time until the brawls
start up there and everybody erupts in proprietary variants and patent
fights.

>     > Allowing nonfree versions of GCC would not help us "win"
>     > anything that matters -- it would only mean surrender.
>
>     Sure, but nobody was talking about "allowing nonfree versions of
>     GCC".
>
> Actually yes they were (though not with those words).  Someone cited
> my decision against having GCC write a complete syntax tree.  That
> output would make it easy to use GCC as a front end for nonfree
> back-ends.  That would be tantamount to making nonfree versions of
> GCC.

I disagree.  M4 output can be used as input for nonfree tools, but if
somebody ties GNU M4 into some task with non-free programs, that is not
tantamount to making a nonfree version of autoconf.

And it's not like a "complete syntax tree" representing all information
passed between GCC front- and backend could hope to retain stability
between versions.  So I think the "complete syntax tree" angle is mostly
hypothetical.

What's definitely less hypothetical is getting at selected subsets of
information.

> Splitting up GCC would have the same effect.

I'm not sure about that.  If a GPLv3 licensed subpackage does a smaller
job than the whole of GCC, I think that also the willingness to swallow
the accompanying GPLv3 poison pill in return for employing that package
would become correspondingly smaller.

So I am less than convinced that more modularity would lead to a
proportionally larger uptake by proprietary vendors than by the Free
Software community itself.

> The lookup and completion features that people want can be implemented
> by making GCC answer questions sent to it, as Aspell does for M-$.

Yes, most definitely.  And a special-cased approach like that would more
likely than not show better performance than one built from modular
building blocks.

But the organizational cost is high.  You can't hack up and play around
and have a working prototype of an Emacs package in weeks.  It takes
months to arrive at a prototype accessible to people compiling their own
GCC, and years for those who install stock versions of GCC.

Much of the UNIX philosophy revolves around having a versatile toolbox
allowing for rapid prototyping: the most important utility for a good
programmer, scientist, or writer is the wastebasket.

Being able to do experiments cheaply is one of the tenets of hacking.

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[Dmitry Gutov]

On 24.02.2014 19:33, Richard Stallman wrote:
> The lookup and completion features that people want can be implemented
> by making GCC answer questions sent to it, as Aspell does for M-$.
> That change would be welcome.  I think it could be implemented using
> GCC's existing plug-in mechanism, but it would be better to put
> the code into GCC itself.

Could it support a "daemon" mode?

The current situation in Clang is, its executable only supports the 
simplest mode of operation: one process call per completion. This way, 
it has to parse all relevant files (headers, etc) each time completion 
is requested.

But the completion information is available via libclang, and there are 
a few projects that build on it to spin up a persistent server, which 
only processes different files anew when they've changed since the last 
time (or at least that's the intention). These projects also support 
different build systems and extract the necessary dependency 
information, include paths, etc.

I don't really see GCC itself doing that (at least, the last part).
It would require quite a bit of work that might be better left to 
third-party developers.

Re: Emacs contributions, C and Lisp

[Andreas Röhler]

Am 19.02.2014 18:11, schrieb Eli Zaretskii:
>> Date: Wed, 19 Feb 2014 08:05:24 +0100
>> From: Jorgen Schaefer <forcer@forcix.cx>
>> Cc: emacs-devel@gnu.org
>>
>> On Mon, 17 Feb 2014 22:29:47 +0200
>> Eli Zaretskii <eliz@gnu.org> wrote:
>>
>>>> There are a few other minor problems for me. For example, my last
>>>> foray in adding a patch to Emacs was so scary regarding the amount
>>>> of red tape involved in the whole process that I am somewhat
>>>> reluctant to commit to doing that regularly.
>>>
>>> What red tape?  Emacs is about the most red-tape-less project as you
>>> can find, as far as the procedure of admitting a patch is considered.
>>
>> If I want to contribute to Emacs, and I want to be good contributor, I
>> have the following things to keep in mind:
>> [...]
>
> Are you saying that other comparable projects don't have similar
> requirements?  Because that'd be definitely not true for the projects
> I'm familiar with.
>
> Here are the requirements for GDB contributors:
>
>    https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=blob;f=gdb/CONTRIBUTE;hb=HEAD
>
> Here are the GCC requirements:
>
>    http://gcc.gnu.org/contribute.html
>
> Here's LLVM's (which cover clang):
>
>    http://llvm.org/docs/DeveloperPolicy.html
>
> As you see, Emacs is the norm here, not the odd one out.


The LLVM project does not require copyright assignments, which means that the copyright for the code in the project is held by its respective contributors who have each 
agreed to release their contributed code under the terms of the LLVM License.

Re: Emacs contributions, C and Lisp

[David Kastrup]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 24.02.2014 19:33, Richard Stallman wrote:
>> The lookup and completion features that people want can be implemented
>> by making GCC answer questions sent to it, as Aspell does for M-$.
>> That change would be welcome.  I think it could be implemented using
>> GCC's existing plug-in mechanism, but it would be better to put
>> the code into GCC itself.
>
> Could it support a "daemon" mode?
>
> The current situation in Clang is, its executable only supports the
> simplest mode of operation: one process call per completion. This way,
> it has to parse all relevant files (headers, etc) each time completion
> is requested.

That leaves something to be desired.

> But the completion information is available via libclang, and there
> are a few projects that build on it to spin up a persistent server,
> which only processes different files anew when they've changed since
> the last time (or at least that's the intention). These projects also
> support different build systems and extract the necessary dependency
> information, include paths, etc.
>
> I don't really see GCC itself doing that (at least, the last part).
> It would require quite a bit of work that might be better left to
> third-party developers.

The one thing that the basic approach "output the annotated parse tree"
or whatever other cooked summary output via plugin or whatever else has
over "stay resident after compilation and answer queries" is that a
daemon in a separate process that will first read the respective data
and then answer queries about it allows the actual parser process to
terminate and shed all the memory and temporary data and parsing tables
it needed for arriving at the parse tree.

That means that the part staying resident and answering queries will not
have its memory fragmented by lots of stuff that is no longer needed.

The "cooked summary" could include a list of processed source files so
that the daemon can reexec the parser and possibly itself whenever it
finds that a file changed between queries.

So a process and program boundary seems to make a lot of sense.  Whether
the data between processes/programs is passed via file/pipe/shared
memory is a different consideration.

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[Glenn Morris]

John Yates wrote:

> I would like to hear the gcc developers weigh in [...]

They probably would do, if this discussion were happening on a gcc
mailing list.

Re: Emacs contributions, C and Lisp

[David Kastrup]

Glenn Morris <rgm@gnu.org> writes:

> John Yates wrote:
>
>> I would like to hear the gcc developers weigh in [...]
>
> They probably would do, if this discussion were happening on a gcc
> mailing list.

Good thing that it isn't, then.  A discussion as opposed to a fist fight
is something where there is a chance of people leaving with a different
opinion than what they started with.

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[David Kastrup]

David Kastrup <dak@gnu.org> writes:

> Glenn Morris <rgm@gnu.org> writes:
>
>> John Yates wrote:
>>
>>> I would like to hear the gcc developers weigh in [...]
>>
>> They probably would do, if this discussion were happening on a gcc
>> mailing list.
>
> Good thing that it isn't, then.  A discussion as opposed to a fist
> fight is something where there is a chance of people leaving with a
> different opinion than what they started with.

To followup on myself: smart completion is a concrete actual application
for Emacs within the GNU universe so this discussion is less likely to
get lost in an abstract showdown.

It's definitely an issue that can be solved using special-casing.  It's
non-trivial enough to discuss its implications concretely.

In this particular case, the "annotated syntax tree" question in
particular is mostly uninteresting since we are talking about
characterizing identifiers.  It is "mostly" uninteresting since the
resolution of an identifier depends on scopes, so the basic question
that likely needs solving is "given the following source location and
the following identifier, what data structures and definitions does it
refer to".  Resolving identifiers based on source location efficiently
will require suitable data structures, and any daemon answering
questions accordingly will have to get raw data for building them.

Accessing that raw data and preparing a dump suitable for turning into a
data structure for such a daemon would seem like an obvious case for a
GCC plugin.  So we are getting more at something like "source location
dependent data structure dump" here.

The "annotated syntax tree" question would become more relevant for
things like sourcecode highlighting.  But in the interest of usefully
fast feedback when editing, it would likely make more sense to let Emacs
do the highlighting with local rules on its own and only converse with
GCC when it becomes necessary to resolve ambiguities (like
declaration/expression distinctions): GCC can only make helpful
suggestions regarding the last time the source code was syntactically
correct, so most of the time Emacs will need to go ahead with _some_
idea anyway.

At any rate, figuring out from the angle of Emacs and completion what
would be required from GCC will provide us with a better idea about the
balance of utility for us and possible proprietary systems trying to
make use of the same code or data.

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[Stephen Leake]

David Kastrup <dak@gnu.org> writes:

> In this particular case, the "annotated syntax tree" question in
> particular is mostly uninteresting since we are talking about
> characterizing identifiers.  It is "mostly" uninteresting since the
> resolution of an identifier depends on scopes, so the basic question
> that likely needs solving is "given the following source location and
> the following identifier, what data structures and definitions does it
> refer to".  Resolving identifiers based on source location efficiently
> will require suitable data structures, and any daemon answering
> questions accordingly will have to get raw data for building them.
>
> Accessing that raw data and preparing a dump suitable for turning into a
> data structure for such a daemon would seem like an obvious case for a
> GCC plugin.  So we are getting more at something like "source location
> dependent data structure dump" here.

That is what 'gcc -fdump-xref' does now.

AdaCore provides a tool 'gnatinspect' that reads that data and answers
queries about it: http://libre.adacore.com/, gnatcoll package.

Emacs Ada mode 5.0.1 (in Gnu ELPA) has experimental code to start
gnatinspect in a process and feed it queries, for Ada, C, C++.

-- 
-- Stephe

Re: Emacs contributions, C and Lisp

[David Kastrup]

Stephen Leake <stephen_leake@stephe-leake.org> writes:

> David Kastrup <dak@gnu.org> writes:
>
>> In this particular case, the "annotated syntax tree" question in
>> particular is mostly uninteresting since we are talking about
>> characterizing identifiers.  It is "mostly" uninteresting since the
>> resolution of an identifier depends on scopes, so the basic question
>> that likely needs solving is "given the following source location and
>> the following identifier, what data structures and definitions does it
>> refer to".  Resolving identifiers based on source location efficiently
>> will require suitable data structures, and any daemon answering
>> questions accordingly will have to get raw data for building them.
>>
>> Accessing that raw data and preparing a dump suitable for turning into a
>> data structure for such a daemon would seem like an obvious case for a
>> GCC plugin.  So we are getting more at something like "source location
>> dependent data structure dump" here.
>
> That is what 'gcc -fdump-xref' does now.

Can't find it in the GCC 4.8 docs.

> AdaCore provides a tool 'gnatinspect' that reads that data and answers
> queries about it: http://libre.adacore.com/, gnatcoll package.
>
> Emacs Ada mode 5.0.1 (in Gnu ELPA) has experimental code to start
> gnatinspect in a process and feed it queries, for Ada, C, C++.

Well, as I explicated previously, I consider it a good idea that any
semi-persistent program would be a different executable from the
original GCC process in order not to have all the parsing memory garbage
staying around indefinitely.

gcc -fdump-xref would likely use an external file rather than a pipe but
I think the preprocessing stages likely use external files anyway so
avoiding temporary disk usage in the setup space is not likely feasible
anyway.

Other than that, this seems like it would likely fit the bill.  Pity
that does not seem to be in GCC 4.8:

dak@lola:/tmp$ gcc -fdump-xref test.c
cc1: error: unrecognized command line option ‘-fdump-xref’
dak@lola:/tmp$ gcc --version
gcc (Ubuntu/Linaro 4.8.1-10ubuntu9) 4.8.1
Copyright (C) 2013 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

dak@lola:/tmp$

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Tue, 25 Feb 2014 10:41:42 +0100
> 
> The "annotated syntax tree" question would become more relevant for
> things like sourcecode highlighting.  But in the interest of usefully
> fast feedback when editing, it would likely make more sense to let Emacs
> do the highlighting with local rules on its own and only converse with
> GCC when it becomes necessary to resolve ambiguities (like
> declaration/expression distinctions): GCC can only make helpful
> suggestions regarding the last time the source code was syntactically
> correct, so most of the time Emacs will need to go ahead with _some_
> idea anyway.

Is it certain that we actually need a compiler for that?  Did someone
investigate whether CEDET infrastructure is capable of doing something
like that?  Since, as you point out, a compiler will probably choke on
syntactically incorrect input, shouldn't we try to look elsewhere?
After all, we don't need to parse the source completely, only as much
as needed for completion.

Re: Emacs contributions, C and Lisp

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Kastrup <dak@gnu.org>
>> Date: Tue, 25 Feb 2014 10:41:42 +0100
>> 
>> The "annotated syntax tree" question would become more relevant for
>> things like sourcecode highlighting.  But in the interest of usefully
>> fast feedback when editing, it would likely make more sense to let Emacs
>> do the highlighting with local rules on its own and only converse with
>> GCC when it becomes necessary to resolve ambiguities (like
>> declaration/expression distinctions): GCC can only make helpful
>> suggestions regarding the last time the source code was syntactically
>> correct, so most of the time Emacs will need to go ahead with _some_
>> idea anyway.
>
> Is it certain that we actually need a compiler for that?  Did someone
> investigate whether CEDET infrastructure is capable of doing something
> like that?  Since, as you point out, a compiler will probably choke on
> syntactically incorrect input, shouldn't we try to look elsewhere?
> After all, we don't need to parse the source completely, only as much
> as needed for completion.

Nobody can parse C++ reliably.  GCC has given up on trying to teach
Bison (aka LALR(1) and then some) how to parse C++ and has implemented
its own hand-written parser.  Existing C++ compilers went through years
of bugfixing until they got things standard-compliant most of the time.
Since C++ has overloaded functions and operators and a rather complex
type conversion lattice, figuring out just _which_ fully qualified
function or operator will get called in a large composite expression is
quite a hard feat.

If you want to be able to reliably follow references to their
_corresponding_ definition, it's almost inavoidable to ask some compiler
for its opinion just what fully qualified function/operator is to be
considered to correspond to the source.

Things like indentation and syntax highlighting are somewhat more
amicable since overloading resolution does not influence things like
expression priorities.

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    > The lookup and completion features that people want can be implemented
    > by making GCC answer questions sent to it, as Aspell does for M-$.
    > That change would be welcome.  I think it could be implemented using
    > GCC's existing plug-in mechanism, but it would be better to put
    > the code into GCC itself.

    Could it support a "daemon" mode?

Yes.  (Though I'm not sure how that differs, practically speaking,
from accepting commands on stdin and answering them on stdout.)

    I don't really see GCC itself doing that (at least, the last part).
    It would require quite a bit of work that might be better left to 
    third-party developers.

I hope some of the people on this list will help implement it.
I will try to ask GCC developers to develop it too.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    The LLVM project does not require copyright assignments, which
    means that the copyright for the code in the project is held by
    its respective contributors who have each agreed to release their
    contributed code under the terms of the LLVM License.

This is because LLVM does not try to defend users' freedom at all.
Its developers willingly permit nonfree modified versions, which is
the reason that LLVM has done harm.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    > As a general statement, that is valid -- but I think you're
    > overestimating Clang's effects on GCC.

    "tremendous harm to our movement" were your words.

That's right, but we are miscommunicating.

The harm is tremendous, but at the same time it is limited in range:
it affects C and C++.  It doesn't affect the other languages that they
don't intend to support.

If we were to take this as a reason to give up on resisting the use of
parts of GCC as part of a nonfree compiler (*), we would extend the
problems to the other languages.

    > Actually yes they were (though not with those words).  Someone cited
    > my decision against having GCC write a complete syntax tree.  That
    > output would make it easy to use GCC as a front end for nonfree
    > back-ends.  That would be tantamount to making nonfree versions of
    > GCC.

    I disagree.

Disagree if you like, but I think it is true in this case.
The case of M4 and Autoconf might not be comparable.


* When I say "nonfree versions of GCC", what I mean is the use of
parts of GCC as part of a nonfree compiler.  There are various ways
that could be implemented, but the harm is the same.

I won't always remember to state it so carefully.  Please don't
quibble when I don't.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Tue, 25 Feb 2014 17:37:14 +0100
> 
> > Is it certain that we actually need a compiler for that?  Did someone
> > investigate whether CEDET infrastructure is capable of doing something
> > like that?  Since, as you point out, a compiler will probably choke on
> > syntactically incorrect input, shouldn't we try to look elsewhere?
> > After all, we don't need to parse the source completely, only as much
> > as needed for completion.
> 
> Nobody can parse C++ reliably.  GCC has given up on trying to teach
> Bison (aka LALR(1) and then some) how to parse C++ and has implemented
> its own hand-written parser.

I understand the potential difficulties, but since we only need a
relatively small part of parsing, perhaps it's worth trying first?  If
push comes to shove, what was implemented in GCC can be reimplemented
in Emacs Lisp, no?  Though I have hard time believing that we will
need to go that far.

> If you want to be able to reliably follow references to their
> _corresponding_ definition, it's almost inavoidable to ask some compiler
> for its opinion just what fully qualified function/operator is to be
> considered to correspond to the source.

I don't understand why, sorry.  Are we still talking about smart
completion, i.e. displaying candidate expansions of text at or near
point?  Or are we talking about something else?

Re: Emacs contributions, C and Lisp

[Dmitry Gutov]

On 25.02.2014 19:14, Richard Stallman wrote:
>      Could it support a "daemon" mode?
>
> Yes.  (Though I'm not sure how that differs, practically speaking,
> from accepting commands on stdin and answering them on stdout.)

I don't understand the question. The way it receives commands (from 
stdout, via a socket, or a network port) is not too dependent on the 
mode of operation. I'd expect a completion server to receive commands 
using some http-like interface, but maybe it's just me.

>      I don't really see GCC itself doing that (at least, the last part).
>      It would require quite a bit of work that might be better left to
>      third-party developers.
>
> I hope some of the people on this list will help implement it.
> I will try to ask GCC developers to develop it too.

Will GCC have to support every build system under the sun? Or those that 
are free, at least?

And have to update the code whenever a new build system comes out, or an 
existing build system adds a relevant feature?

Re: Emacs contributions, C and Lisp

[David Kastrup]

Richard Stallman <rms@gnu.org> writes:

>     > Actually yes they were (though not with those words).  Someone
>     > cited my decision against having GCC write a complete syntax
>     > tree.  That output would make it easy to use GCC as a front end
>     > for nonfree back-ends.  That would be tantamount to making
>     > nonfree versions of GCC.
>
>     I disagree.
>
> Disagree if you like, but I think it is true in this case.

Well, I think "complete syntax tree" is probably mostly a red herring
not worth the fight we have been seeing over it.  It is most likely
rather version- and language-specific and likely quite bound into GCC's
internals whenever one wants to be serious about "complete".

It would appear (from a separate reply) that the information written by
-fdump-xref is basically the info that any completion server/daemon
would have to provide access to (if one does not just parse it in Elisp
itself), and that in the context of the Ada backend there has already
been written a tool that reads it and provides the respective
information for Ada, C, and C++.

So it would appear that these eggs have not just been laid but already
hatched and most of the emotions invested into this discussion are moot
anyway.

> * When I say "nonfree versions of GCC", what I mean is the use of
> parts of GCC as part of a nonfree compiler.  There are various ways
> that could be implemented, but the harm is the same.

Well, GCC consists of multiple executables implementing multiple passes.
One could always replace the second executable, but I suppose that this
would be enough of a mess that calling the combination a "mere
aggregation" would, in spite of separate address spaces, be ridiculous
enough that nobody would consider that a valid defense.

I think similar restraint would apply for most operations resulting in
an actual full compiler.

Things are different with plugging GCC into a fully proprietary IDE.  I
don't really see that any interfaces we concoct in order to use GCC as a
more-than-just compiler component of an Emacs-based workflow could be
kept from getting used for integrating GCC for the same purpose into a
proprietary IDE without causing a combined work to be covered by the
"software as a whole rather than mere aggregation" clause.

So I'm not sure that much to really fight over remains once we are
delving into concrete applications we want to implement based on GCC in
an environment such as Emacs.  Either everybody gets it or nobody.

What _is_ satisfying, however, is that once big companies with a patent
portfolio use and distribute GCC for getting more value out of
proprietary offerings from them, we more or less get patent
indemnification for doing the same feature inside of Emacs as an IDE:
that's a nice benefit of GPLv3.

Which makes me suspect that we won't see large-scale misuse of GCC.
None of the big players want to risk effectively foregoing any of their
patents all of which, no matter how ridiculous, are potentially worth
millions or even billions of dollars.

GPLv3 was accompanied by enough of a predictable backlash that it seems
wasteful not to amend our strategies where we consciously paid the price
for changing the battlefield to our advantage.

We lost ground with LLVM, and we gained ground with GPLv3.  We should
take both into account.

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> Nobody can parse C++ reliably.  GCC has given up on trying to teach
>> Bison (aka LALR(1) and then some) how to parse C++ and has implemented
>> its own hand-written parser.
>
> I understand the potential difficulties, but since we only need a
> relatively small part of parsing,

Why do we need a small part of parsing? For implementing C++ smart
completion on a reliable way, you need semantic analysis. Furthermore,
people here are talking too much about completion, but there are other
features that require whole-program semantic analysis and hence are out
of reach of the approaches mentioned here based on gcc spitting
not-quite-comprehensive information.

> perhaps it's worth trying first?  If
> push comes to shove, what was implemented in GCC can be reimplemented
> in Emacs Lisp, no?

Right now the available systems for smart code completion are annoyingly
slow. They are implemented on C/C++. It is reasonable to expect from a
Elisp-based solution to be unbearably slow, not to mention the
complexity.

Why reinvent the wheel?

> Though I have hard time believing that we will
> need to go that far.
>
>> If you want to be able to reliably follow references to their
>> _corresponding_ definition, it's almost inavoidable to ask some compiler
>> for its opinion just what fully qualified function/operator is to be
>> considered to correspond to the source.
>
> I don't understand why, sorry.  Are we still talking about smart
> completion, i.e. displaying candidate expansions of text at or near
> point?  Or are we talking about something else?

Dunno about David, but I say "yes", for smart completion you need way
more than quick & dirty parsing.

Re: Emacs contributions, C and Lisp

[Stephen Leake]

David Kastrup <dak@gnu.org> writes:

> Stephen Leake <stephen_leake@stephe-leake.org> writes:
>
>> David Kastrup <dak@gnu.org> writes:
>>
>> That is what 'gcc -fdump-xref' does now.
>
> Can't find it in the GCC 4.8 docs.

Ah. That's a problem.

> Other than that, this seems like it would likely fit the bill.  Pity
> that does not seem to be in GCC 4.8:

I've only actually used this with compilers provided by AdaCore, so apparently
-fdump-xref is an AdaCore extension.

However, it is GPL-3 with FSF assignment (as is all AdaCore code related
to gcc), and could be merged into mainstream gcc. I assume they plan to
do that at some point.

-- 
-- Stephe

Re: Emacs contributions, C and Lisp

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Tue, 25 Feb 2014 20:50:11 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Nobody can parse C++ reliably.  GCC has given up on trying to teach
> >> Bison (aka LALR(1) and then some) how to parse C++ and has implemented
> >> its own hand-written parser.
> >
> > I understand the potential difficulties, but since we only need a
> > relatively small part of parsing,
> 
> Why do we need a small part of parsing? For implementing C++ smart
> completion on a reliable way, you need semantic analysis.

Can you explain why?  I'm probably missing something.

> Furthermore, people here are talking too much about completion

Well, that's the only thing that clang has that prompted this thread,
right?

> but there are other features that require whole-program semantic
> analysis and hence are out of reach of the approaches mentioned here
> based on gcc spitting not-quite-comprehensive information.

Can you list those features?

> > perhaps it's worth trying first?  If push comes to shove, what was
> > implemented in GCC can be reimplemented in Emacs Lisp, no?
> 
> Right now the available systems for smart code completion are annoyingly
> slow. They are implemented on C/C++. It is reasonable to expect from a
> Elisp-based solution to be unbearably slow, not to mention the
> complexity.

We can always prototype in Lisp, then reimplement the slow parts in C
if needed.

> Why reinvent the wheel?

Because we cannot get the one that's already invented?

Re: Emacs contributions, C and Lisp

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Kastrup <dak@gnu.org>
>> Date: Tue, 25 Feb 2014 10:41:42 +0100
>> 
>> The "annotated syntax tree" question would become more relevant for
>> things like sourcecode highlighting.  But in the interest of usefully
>> fast feedback when editing, it would likely make more sense to let Emacs
>> do the highlighting with local rules on its own and only converse with
>> GCC when it becomes necessary to resolve ambiguities (like
>> declaration/expression distinctions): GCC can only make helpful
>> suggestions regarding the last time the source code was syntactically
>> correct, so most of the time Emacs will need to go ahead with _some_
>> idea anyway.
>
> Is it certain that we actually need a compiler for that?  Did someone
> investigate whether CEDET infrastructure is capable of doing something
> like that?  

I recently rewrote the Ada indentation engine, and tried to use the CEDET
infrastructure. It was not adequate, so I ended up writing a full parser
(it is similar to but more powerful than the CEDET lexer/parser).

Even so, I don't use that for completion, especially when looking for
the parameters of functions. As I pointed out in another email, AdaCore
provides 'gcc -fdump-xref', which dumps all the information needed for
that, and a tool 'gnatinspect' that interfaces nicely with Emacs to
retrieve it. I have not fully implemented completion in Emacs using
that, but AdaCore has in their editor GPS, so I know it's possible :).

> Since, as you point out, a compiler will probably choke on
> syntactically incorrect input, shouldn't we try to look elsewhere?

The current hippie-expand mechanisms offer completion on identifiers in
current buffers; beyond that, I don't see any solution other than to
rely on compiled code for completion, especially of parameters of functions.

> After all, we don't need to parse the source completely, only as much
> as needed for completion.

That's the same thing, as far as I can tell. You need even less info for
indentation, but I ended up needing a full file parse for that.

Actually, only a full "compilation unit" parse; in Ada that is normally
a full file. In C++ it would be a class or function definition. Still a
complete language parser.

-- 
-- Stephe

Re: Emacs contributions, C and Lisp

[Stephen Leake]

Óscar Fuentes <ofv@wanadoo.es> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> Nobody can parse C++ reliably.  GCC has given up on trying to teach
>>> Bison (aka LALR(1) and then some) how to parse C++ and has implemented
>>> its own hand-written parser.
>>
>> I understand the potential difficulties, but since we only need a
>> relatively small part of parsing,
>
> Why do we need a small part of parsing? For implementing C++ smart
> completion on a reliable way, you need semantic analysis. Furthermore,
> people here are talking too much about completion, but there are other
> features that require whole-program semantic analysis and hence are out
> of reach of the approaches mentioned here based on gcc spitting
> not-quite-comprehensive information.

+1

Refactoring tools, for example; rename this class member function, and
all references to it, but _not_ references to the same name in _other_
classes, or the same name declared as a local variable. Do rename in
classes that derive from this class.

AdaCore GPS does this, for Ada and C++.

>> I don't understand why, sorry.  Are we still talking about smart
>> completion, i.e. displaying candidate expansions of text at or near
>> point?  Or are we talking about something else?
>
> Dunno about David, but I say "yes", for smart completion you need way
> more than quick & dirty parsing.

Especially when one of the possible completions is a list of parameters
for a function call.

-- 
-- Stephe

Re: Emacs contributions, C and Lisp

[Stephen Leake]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>     > The lookup and completion features that people want can be implemented
>     > by making GCC answer questions sent to it, as Aspell does for M-$.
>     > That change would be welcome.  I think it could be implemented using
>     > GCC's existing plug-in mechanism, but it would be better to put
>     > the code into GCC itself.
>
>     Could it support a "daemon" mode?
>
> Yes.  (Though I'm not sure how that differs, practically speaking,
> from accepting commands on stdin and answering them on stdout.)
>
>     I don't really see GCC itself doing that (at least, the last part).
>     It would require quite a bit of work that might be better left to 
>     third-party developers.
>
> I hope some of the people on this list will help implement it.
> I will try to ask GCC developers to develop it too.

AdaCore has already done it, via the -fdump-xref option to gcc.

-- 
-- Stephe

Re: Emacs contributions, C and Lisp

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> > I understand the potential difficulties, but since we only need a
>> > relatively small part of parsing,
>> 
>> Why do we need a small part of parsing? For implementing C++ smart
>> completion on a reliable way, you need semantic analysis.
>
> Can you explain why?  I'm probably missing something.

See this simple example:

struct A {
  int foo();
};

struct B {
  int bar();
};

A baz(int);
B baz(A);

int quux(char c) {
  return baz(c). ???????????
}

For knowing the correct completion candidates at the point indicated by
the question marks you need to know which `baz' overload to choose,
which in turn requires to know that `c' is a `char' and then knowing
that a `char' is acceptable by the `baz' overload that takes an `int'.
Therefore you know that the correct `baz' overload returns an `A' and
therefore the list of completion candidates is `foo'.

Even this very simple case requires knowing the semantics of C++. Even
if this looks approachable by a parser armed with some ad-hoc
heuristics, those will surely fail by just adding/modifying a couple of
lines on the test case above, unless those heuristics implement semantic
analysis under cover.

>> Furthermore, people here are talking too much about completion
>
> Well, that's the only thing that clang has that prompted this thread,
> right?

Clang provides code completion as a sample of its capabilities.
Clang/LLVM in fact is a set of libraries for dealing with C/C++ code.
You can use those libraries for code completion and for any other
feature that requires accessing/processing information contained on
source code: extracting declarations, sanitizing, instrumenting,
optimizing, generating object code...

>> but there are other features that require whole-program semantic
>> analysis and hence are out of reach of the approaches mentioned here
>> based on gcc spitting not-quite-comprehensive information.
>
> Can you list those features?

One was already mentioned by Stephen Leake: refactoring. Actually,
anything that requires semantic knowledge of the source code you are
working on. You could ask for a listing of places dependent of word
size, for instance, or highlight the places where certain idiom is used.

>> > perhaps it's worth trying first?  If push comes to shove, what was
>> > implemented in GCC can be reimplemented in Emacs Lisp, no?
>> 
>> Right now the available systems for smart code completion are annoyingly
>> slow. They are implemented on C/C++. It is reasonable to expect from a
>> Elisp-based solution to be unbearably slow, not to mention the
>> complexity.
>
> We can always prototype in Lisp, then reimplement the slow parts in C
> if needed.

IIRC I already told you this a few weeks ago, but I'll repeat: a C++
front-end (even without code generation capabilities) requires an
immense amount of work from highly specialized people, and then needs
improvements as new standards are published.

>> Why reinvent the wheel?
>
> Because we cannot get the one that's already invented?

"we cannot" isn't the right expression. "we are not allowed" is the
correct description.

As others pointed out, the cat is out of the bag. Emacs will have
Clang-based features provided as external packages. Sincerely, I don't
care much if the Emacs project does not allow those features into the
core distribution or ELPA. People will install them from alternative
repositories, that's all.

Re: Emacs contributions, C and Lisp

[Stephen J. Turnbull]

David Kastrup writes:

 > In this particular case, the "annotated syntax tree" question in
 > particular is mostly uninteresting since we are talking about
 > characterizing identifiers.  It is "mostly" uninteresting since the
 > resolution of an identifier depends on scopes,

Nice try, but I don't think you can deprecate the value of the
information GCC is not allowed to emit that easily.

Emacs's treatment should depend on types and usage (for example, a
keyword used as an identifier in a context where that is a syntax
error should not be included in a completion candidate list at that
point).  Even if you restrict consideration to the simple context of
completion (fontification would be another application where the
additional information would be of great use), the annotation
information would improve accuracy in generating candidate lists.

And of course there are areas where Emacs lags badly (refactoring
tools) where complete information about the syntax tree would be very
useful.

Re: Emacs contributions, C and Lisp

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> David Kastrup writes:
>
>  > In this particular case, the "annotated syntax tree" question in
>  > particular is mostly uninteresting since we are talking about
>  > characterizing identifiers.  It is "mostly" uninteresting since the
>  > resolution of an identifier depends on scopes,
>
> Nice try, but I don't think you can deprecate the value of the
> information GCC is not allowed to emit that easily.
>
> Emacs's treatment should depend on types and usage [...]

Most of your reply has already been addressed in the parts you did not
quote.  If the discussion is supposed to lead somewhere rather than run
in circles, it would make sense not to cherry-pick one's controversy.

At the current point of time, you are trying to retract the discussion
back into discussing general principles.  We had hit a wall with that
approach already, so unless we particularly like that wall, I suggest
that we go back to the specific problem of completion, figure out the
respective approaches leading to solutions there, and try figuring out
lessons for other applications afterwards.

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[Stephen J. Turnbull]

Richard Stallman writes:

 > This is because LLVM does not try to defend users' freedom at all.

Of course it does.  It gives them the choice of using excellent free
software.  Apple (inter alia) may provide proprietary versions of LLVM
or its components, but many users will choose to use the free ones for
various reasons.

 > Its developers willingly permit nonfree modified versions, which is
 > the reason that LLVM has done harm.

Anyone using a non-strong-copyleft license does the same kind of harm.
That includes the LGPL, which has an even more harmful effect than
that of the proposed GCC which emits AST information, since you can
include LGPLed code in the same process with proprietary code -- in
the completion application in question, that usually minor efficiency
might actually be useful.

So, precisely what is the difference between the harm (of quite
limited scope, as DAK points out) done by LLVM, and the much broader
harm[1] done by Debian, which distributes a very large proportion of
its software under non-strong-copyleft licenses?  The point is that
with the exception of the deliberately perverse licenses like SSLeay,
almost all are GPL-compatible, and Debian *could* distribute them as a
collective work under the GPL (with exceptions for the perverse
permissive licenses -- n.b. GPL-incompatible strong copyleft licenses
don't cause this kind of harm), which would have a useful psychological
effect, even if in practice very little legal effect?

And why should Emacs users suffer for that difference?


Footnotes: 
[1]  Many proprietary webservers and databases that can be used in
"LAMP stacks" happily run on Debian, as well as Ubuntu, Red Hat, and
the rest of the commercial distros.  The webservers at least are often
based on free software, ie, Apache.

Re: Emacs contributions, C and Lisp

[Stephen J. Turnbull]

David Kastrup writes:
 > "Stephen J. Turnbull" <stephen@xemacs.org> writes:
 > > David Kastrup writes:

 > >  > In this particular case, the "annotated syntax tree" question in
 > >  > particular is mostly uninteresting since we are talking about
 > >  > characterizing identifiers.  It is "mostly" uninteresting since the
 > >  > resolution of an identifier depends on scopes,

 > > Nice try, but I don't think you can deprecate the value of the
 > > information GCC is not allowed to emit that easily.

 > > Emacs's treatment should depend on types and usage [...]

 > Most of your reply has already been addressed in the parts you did not
 > quote.

OK.  I guess what you are arguing is that you can throw away the tree
and keep only the local annotations.  All I'm saying is that you need
those local annotations, and you need "position in program"
information.  However, a conventional xref table is not going to be
enough because you need the scope (which is required to restrict the
list of completions).  Even if your xref table provides enough
information to determine scopes properly, you'll end up rebuilding
something very close to a syntax tree to associate those scopes with
each potential use of an identifier (and thus the most accurate list
of potential completions at each point in the buffer).

IOW, my point is not really that you can't throw away the annotations;
it's that you don't want to throw away either the syntax tree or the
annotations.

OTOH, it will be interesting to see if the Ada -fdump-xref feature is
actually good enough to handle the insanity of C++ without a full
syntax-tree.  Óscar's argument that you really need the whole syntax
tree to achieve accuracy in C++ seems pretty plausible to me (although
I don't think completion is a compelling application -- we already do
pretty well).

Re: Emacs contributions, C and Lisp

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> David Kastrup writes:
>  > "Stephen J. Turnbull" <stephen@xemacs.org> writes:
>  > > David Kastrup writes:
>
>  > >  > In this particular case, the "annotated syntax tree" question in
>  > >  > particular is mostly uninteresting since we are talking about
>  > >  > characterizing identifiers.  It is "mostly" uninteresting since the
>  > >  > resolution of an identifier depends on scopes,
>
>  > > Nice try, but I don't think you can deprecate the value of the
>  > > information GCC is not allowed to emit that easily.
>
>  > > Emacs's treatment should depend on types and usage [...]
>
>  > Most of your reply has already been addressed in the parts you did not
>  > quote.
>
> OK.  I guess what you are arguing is that you can throw away the tree
> and keep only the local annotations.  All I'm saying is that you need
> those local annotations, and you need "position in program"
> information.

Really, I addressed this _exhaustively_ in
<URL:https://lists.gnu.org/archive/html/emacs-devel/2014-02/msg00546.html>.
I could basically quote the entire message in reply to your "guess what
I am arguing" but it seems to make awfully little sense.

So perhaps try answering the whole message rather than picking a
half-paragraph and blocking the rest out.

> However, a conventional xref table is not going to be enough because
> you need the scope (which is required to restrict the list of
> completions).

See above.

> Even if your xref table provides enough information to determine
> scopes properly, you'll end up rebuilding something very close to a
> syntax tree to associate those scopes with each potential use of an
> identifier (and thus the most accurate list of potential completions
> at each point in the buffer).

See above.

> IOW, my point is not really that you can't throw away the annotations;
> it's that you don't want to throw away either the syntax tree or the
> annotations.

We already saw where the "I want it all, and now" leads us.  In
particular since "all" is rather ill-defined and very likely tied into a
particular internal representation/focus of GCC.

I repeat (now indeed quoting from the mail you "answered" previously):

    To followup on myself: smart completion is a concrete actual
    application for Emacs within the GNU universe so this discussion is
    less likely to get lost in an abstract showdown.

    It's definitely an issue that can be solved using special-casing.
    It's non-trivial enough to discuss its implications concretely.

> OTOH, it will be interesting to see if the Ada -fdump-xref feature is
> actually good enough to handle the insanity of C++ without a full
> syntax-tree.

I have no idea.  But a full syntax tree is not required.  I repeat:

    [...] so the basic question that likely needs solving is "given the
    following source location and the following identifier, what data
    structures and definitions does it refer to".  Resolving identifiers
    based on source location efficiently will require suitable data
    structures, and any daemon answering questions accordingly will have
    to get raw data for building them.

An efficient and compact representation might make use of typical
scoping structures, like Git's archives rely on "delta chain
compression" designed to make use of typical history dependencies
without actually delivering any actually useful relation information.
You want the data structure for answering the "meaning of x at source
location y" question make good use of the typical correlation of
identifier sets between source location y and source location y+dy.

But that does not mean that digging through the whole syntax tree is
going to be what you want.  That's more likely than not too inefficient
to be fun.

At any rate, this is hypothetical, while it would appear that
-fdump-xref isn't.  There seems to be little point in fighting a lot
over it without actually looking what -fdump-xref actually does.

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[David Engster]

Óscar Fuentes writes:
> See this simple example:
>
> struct A {
>   int foo();
> };
>
> struct B {
>   int bar();
> };
>
> A baz(int);
> B baz(A);
>
> int quux(char c) {
>   return baz(c). ???????????
> }
>
> For knowing the correct completion candidates at the point indicated by
> the question marks you need to know which `baz' overload to choose,
> which in turn requires to know that `c' is a `char' and then knowing
> that a `char' is acceptable by the `baz' overload that takes an `int'.
> Therefore you know that the correct `baz' overload returns an `A' and
> therefore the list of completion candidates is `foo'.

With latest Emacs from trunk:

- Put the above in file test.c
- Load it
- M-x semantic-mode
- Put point behind "baz(c)."
- C-c , l

Also, run "M-x bovinate" to see the tags the parser generates. In the
function, run "M-x semantic-calculate-scope" to see what it knows there.

> Even this very simple case requires knowing the semantics of C++. Even
> if this looks approachable by a parser armed with some ad-hoc
> heuristics, those will surely fail by just adding/modifying a couple of
> lines on the test case above, unless those heuristics implement semantic
> analysis under cover.

Yes. You will notice that if you change 'quux' to accept a struct A, it
will still say 'foo'. But this is actually not a parser problem, but a
missing feature in the semantic database, which currently cannot deal
with overloads, so it just takes the first one it sees. That should not
be very hard to add, but - as usual - someone has to do it.

> One was already mentioned by Stephen Leake: refactoring. Actually,
> anything that requires semantic knowledge of the source code you are
> working on. You could ask for a listing of places dependent of word
> size, for instance, or highlight the places where certain idiom is used.

CEDET will most probably never be able to refactor C++ code, aside from
very simple cases. There are very few IDEs out there which even try to
do that; from my experience, none of them do it 100% reliably (just
bring some meta template programming into the game and see what
happens). IMHO, "Refactoring C++" should not be in the job description.

-David

Re: Emacs contributions, C and Lisp

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Tue, 25 Feb 2014 23:36:33 +0100
> 
> struct A {
>   int foo();
> };
> 
> struct B {
>   int bar();
> };
> 
> A baz(int);
> B baz(A);
> 
> int quux(char c) {
>   return baz(c). ???????????
> }
> 
> For knowing the correct completion candidates at the point indicated by
> the question marks you need to know which `baz' overload to choose,
> which in turn requires to know that `c' is a `char' and then knowing
> that a `char' is acceptable by the `baz' overload that takes an `int'.
> Therefore you know that the correct `baz' overload returns an `A' and
> therefore the list of completion candidates is `foo'.

Would it be so bad if it showed 2 candidates instead of one?

And what happens if you add

  B baz(char);

to the above -- will it show 2 candidates or just one?

> Even this very simple case requires knowing the semantics of C++. Even
> if this looks approachable by a parser armed with some ad-hoc
> heuristics, those will surely fail by just adding/modifying a couple of
> lines on the test case above, unless those heuristics implement semantic
> analysis under cover.

I don't know why you are saying this.  E.g., a "parser armed with some
ad-hoc heuristics" that we have in ebrowse seems to be able to catch a
significant subset of C++, certainly of C++ that existed when ebrowse
was written.  Why couldn't we base this feature on an extended ebrowse
engine?

ECB also supports mart completion.  4 years ago people complained that
it's slow, but machines became faster since then, so perhaps things
are not so bad now, and we could use Semantic for this purpose.

> >> Furthermore, people here are talking too much about completion
> >
> > Well, that's the only thing that clang has that prompted this thread,
> > right?
> 
> Clang provides code completion as a sample of its capabilities.
> Clang/LLVM in fact is a set of libraries for dealing with C/C++ code.
> You can use those libraries for code completion and for any other
> feature that requires accessing/processing information contained on
> source code: extracting declarations, sanitizing, instrumenting,
> optimizing, generating object code...

Are there any Emacs packages that support those features?

> One was already mentioned by Stephen Leake: refactoring.

The only Emacs package for this that I could find is proprietary
(Xrefactory).  Do you happen to know about any free ones?

> Actually,
> anything that requires semantic knowledge of the source code you are
> working on. You could ask for a listing of places dependent of word
> size, for instance, or highlight the places where certain idiom is used.

Doesn't ECB already support such features?

> > We can always prototype in Lisp, then reimplement the slow parts in C
> > if needed.
> 
> IIRC I already told you this a few weeks ago, but I'll repeat: a C++
> front-end (even without code generation capabilities) requires an
> immense amount of work from highly specialized people, and then needs
> improvements as new standards are published.

Only if you need to be 110% accurate, which is certainly a requirement
for a compiler.  But we don't need such strict requirements for the
features we are discussing, I think.

> >> Why reinvent the wheel?
> >
> > Because we cannot get the one that's already invented?
> 
> "we cannot" isn't the right expression. "we are not allowed" is the
> correct description.

I'm trying to keep this part of the thread out of politics and into
something that could hopefully lead to a working implementation.

Re: Emacs contributions, C and Lisp

[Josh]

[-- Attachment #1: Type: text/plain, Size: 1389 bytes --]

On Wed, Feb 26, 2014 at 8:28 AM, David Engster <deng@randomsample.de> wrote:

> > One was already mentioned by Stephen Leake: refactoring. Actually,
> > anything that requires semantic knowledge of the source code you are
> > working on. You could ask for a listing of places dependent of word
> > size, for instance, or highlight the places where certain idiom is used.
>
> CEDET will most probably never be able to refactor C++ code, aside from
> very simple cases. There are very few IDEs out there which even try to
> do that; from my experience, none of them do it 100% reliably (just
> bring some meta template programming into the game and see what
> happens). IMHO, "Refactoring C++" should not be in the job description.


Though I haven't used it myself, presumably one of these IDEs is
Eclipse CDT[0].  According to this article[1], CDT includes a full
C/C++ parser that's under active development (item 4) as well as
extensive support for static analysis (item 5) and refactoring
(item 6).  Perhaps there are technical or licensing-related issues
(see also item 9 regarding LLVM and Clang) that would make it a bad
fit for integrating with Emacs in some way, but otherwise it might
be worth a look.  The wiki is here[2].

[0] https://www.eclipse.org/cdt/
[1]
http://www.eclipse.org/community/eclipse_newsletter/2013/october/article1.php
[2] https://wiki.eclipse.org/CDT/

Josh

[-- Attachment #2: Type: text/html, Size: 2579 bytes --]

Re: Emacs contributions, C and Lisp

[Stephen J. Turnbull]

David Kastrup writes:

 > Really, I addressed this _exhaustively_ in

Hardly.  You have a hypothetical data structure containing an
unspecified set of information much of which is known to be available
in the AST, and unsubstantiated claims that "most" of the AST
information isn't needed.  For example:

 > But a full syntax tree is not required.  I repeat:
 > 
 >     [...] so the basic question that likely needs solving is "given the
 >     following source location and the following identifier, what data
 >     structures and definitions does it refer to".  Resolving identifiers
 >     based on source location efficiently will require suitable data
 >     structures, and any daemon answering questions accordingly will have
 >     to get raw data for building them.

"Likely."  "Suitable."  "Get raw data."  In other words, you have no
idea whether a full syntax tree is required or not.  All you know is
that if and when you get a spec for the data and a design for the
daemon, the rest will be a SMOP.  Thanks!  But, well, I already knew
that, that's just how programming works.

 > At any rate, this is hypothetical, while it would appear that
 > -fdump-xref isn't.  There seems to be little point in fighting a lot
 > over it without actually looking what -fdump-xref actually does.

OK.

Re: Emacs contributions, C and Lisp

[David Engster]

'josh' writes:
> Though I haven't used it myself, presumably one of these IDEs is
> Eclipse CDT[0].  According to this article[1], CDT includes a full
> C/C++ parser that's under active development (item 4) as well as
> extensive support for static analysis (item 5) and refactoring
> (item 6).  Perhaps there are technical or licensing-related issues
> (see also item 9 regarding LLVM and Clang) that would make it a bad
> fit for integrating with Emacs in some way, but otherwise it might
> be worth a look.  The wiki is here[2].

I have looked at pretty much any C++ parser out there. That includes
projects like CDT, cppcheck, KDevelop, and OpenC++. I have also looked
at gcc's AST dump (which contrary to popular opinion does exist; it
seems people don't search the web anymore). All of these are not
straightforward to use from Emacs, for various reasons, so I decided to
keep working on Semantic. I do plan to use gcc for helping with
preprocessor work, though.

-David

Re: Emacs contributions, C and Lisp

[Óscar Fuentes]

David Engster <deng@randomsample.de> writes:

> With latest Emacs from trunk:
>
> - Put the above in file test.c
> - Load it
> - M-x semantic-mode
> - Put point behind "baz(c)."
> - C-c , l

Impressive, but...

[snip]

>> Even this very simple case requires knowing the semantics of C++. Even
>> if this looks approachable by a parser armed with some ad-hoc
>> heuristics, those will surely fail by just adding/modifying a couple of
>> lines on the test case above, unless those heuristics implement semantic
>> analysis under cover.
>
> Yes. You will notice that if you change 'quux' to accept a struct A, it
> will still say 'foo'. But this is actually not a parser problem, but a
> missing feature in the semantic database, which currently cannot deal
> with overloads, so it just takes the first one it sees.

... it works just by chance.

> That should not
> be very hard to add, but - as usual - someone has to do it.

Overload resolution is one of the most difficult parts of "C++ with
classes" (the language without templates). Another very tricky part is
namespace resolution.

> CEDET will most probably never be able to refactor C++ code, aside from
> very simple cases. There are very few IDEs out there which even try to
> do that; from my experience, none of them do it 100% reliably (just
> bring some meta template programming into the game and see what
> happens). IMHO, "Refactoring C++" should not be in the job description.

Just because current solutions are no 100% accurate doesn't mean that
the feature should be ignored. Until recently, those solutions falled
into two classes: "heroic" parsers (i.e. Eclipse's) and compilers acting
as analyzers (MS Visual Studio, which uses a propietary compiler
front-end (not their own)) inspecting its internal state. The former is
absolutely insufficient and the later is like using an square peg in a
round hole, because the compiler was never intended to do that (same
could be said of GCC.) Clang is changing all that because it is designed
from the start for making available and manageable the required
information.

Re: Emacs contributions, C and Lisp

[David Engster]

Óscar Fuentes writes:
> ... it works just by chance.

You are really fast at dismissing things.

>> That should not
>> be very hard to add, but - as usual - someone has to do it.
>
> Overload resolution is one of the most difficult parts of "C++ with
> classes" (the language without templates). Another very tricky part is
> namespace resolution.

Thanks for the lecture.

> Just because current solutions are no 100% accurate doesn't mean that
> the feature should be ignored. Until recently, those solutions falled
> into two classes: "heroic" parsers (i.e. Eclipse's) and compilers acting
> as analyzers (MS Visual Studio, which uses a propietary compiler
> front-end (not their own)) inspecting its internal state. The former is
> absolutely insufficient and the later is like using an square peg in a
> round hole, because the compiler was never intended to do that (same
> could be said of GCC.) Clang is changing all that because it is designed
> from the start for making available and manageable the required
> information.

Then by all means go ahead with whatever you are proposing.

-David

Re: Emacs contributions, C and Lisp

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

> Would it be so bad if it showed 2 candidates instead of one?

IMO, yes. The "smart" part of "smart completion" is that only suitable
candidates are shown. Something else just causes confusion and
compile-edit cycles. Also, that example used a method for each class.
Real-world cases are not limited to that.

> And what happens if you add
>
>   B baz(char);
>
> to the above -- will it show 2 candidates or just one?

Dunno. As David Engster explained, CEDET does the simplest thing (take
one of the overloads and ignore the rest.)

> I don't know why you are saying this.  E.g., a "parser armed with some
> ad-hoc heuristics" that we have in ebrowse seems to be able to catch a
> significant subset of C++, certainly of C++ that existed when ebrowse
> was written.  Why couldn't we base this feature on an extended ebrowse
> engine?

Today's C++ common usage would look totally alien to a programmer coming
from the year 2000. Ebrowse is for "C with classes" more than C++. AFAIK
Ebrowse will choke at the first occurrence of a template, for example.
Also its intended usage is quite more modest than the functionality
required for smart completion, not to mention refactoring.

> ECB also supports mart completion.

For C++? Not really. It only works as long as your code is simple
enough, but that's because it implements the simplest cases. I tried it
several times on my code base and the time saved on the cases where it
worked was not enough to compensate for the cases where it didn't,
leaving aside the frustration of the continous wtf!! moments.

>  4 years ago people complained that
> it's slow, but machines became faster since then, so perhaps things
> are not so bad now, and we could use Semantic for this purpose.

If Semantic is slow supporting the easiest, simplest parts of the C++
source code analysis... think about how slow would it be if it
implemented the missing parts, those that make C++ compilers really slow
compared to their C counterparts.

>> Clang provides code completion as a sample of its capabilities.
>> Clang/LLVM in fact is a set of libraries for dealing with C/C++ code.
>> You can use those libraries for code completion and for any other
>> feature that requires accessing/processing information contained on
>> source code: extracting declarations, sanitizing, instrumenting,
>> optimizing, generating object code...
>
> Are there any Emacs packages that support those features?

Dunno. But the point is that now that a tool exists for doing the heavy
lifting, creating an Emacs package for exploiting C++ semantic
information is feasible.

>> One was already mentioned by Stephen Leake: refactoring.
>
> The only Emacs package for this that I could find is proprietary
> (Xrefactory).  Do you happen to know about any free ones?

No. My knowledge is far from exhaustive, though.

>> Actually,
>> anything that requires semantic knowledge of the source code you are
>> working on. You could ask for a listing of places dependent of word
>> size, for instance, or highlight the places where certain idiom is used.
>
> Doesn't ECB already support such features?

I don't think so. It can't do that since its understanding of C++ is
quite limited.

>> IIRC I already told you this a few weeks ago, but I'll repeat: a C++
>> front-end (even without code generation capabilities) requires an
>> immense amount of work from highly specialized people, and then needs
>> improvements as new standards are published.
>
> Only if you need to be 110% accurate,

Since 100% is perfect, why should I wish more? ;-)

> which is certainly a requirement
> for a compiler.  But we don't need such strict requirements for the
> features we are discussing, I think.

A defective refactoring tool can easily cause more work than it saves.
It can introduce subtle bugs, too.

>> >> Why reinvent the wheel?
>> >
>> > Because we cannot get the one that's already invented?
>> 
>> "we cannot" isn't the right expression. "we are not allowed" is the
>> correct description.
>
> I'm trying to keep this part of the thread out of politics and into
> something that could hopefully lead to a working implementation.

I'm not interested on politics either. I just wanted to be accurate :-)

Re: Emacs contributions, C and Lisp

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Wed, 26 Feb 2014 21:17:07 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Would it be so bad if it showed 2 candidates instead of one?
> 
> IMO, yes. The "smart" part of "smart completion" is that only suitable
> candidates are shown. Something else just causes confusion and
> compile-edit cycles. Also, that example used a method for each class.
> Real-world cases are not limited to that.
> 
> > And what happens if you add
> >
> >   B baz(char);
> >
> > to the above -- will it show 2 candidates or just one?
> 
> Dunno.

I suggest to try (I did).  Maybe then you will be less radical in your
judgment of having N+1 candidates when only N are strictly needed.

> As David Engster explained, CEDET does the simplest thing (take
> one of the overloads and ignore the rest.)

I see nothing wrong with doing the simplest thing, if it gives
reasonable results.  This is an engineering discipline, not an exact
science; compromises are our bread and butter.  I'm sure you are well
aware of that.

> Today's C++ common usage would look totally alien to a programmer coming
> from the year 2000. Ebrowse is for "C with classes" more than C++. AFAIK
> Ebrowse will choke at the first occurrence of a template, for example.
> Also its intended usage is quite more modest than the functionality
> required for smart completion, not to mention refactoring.

There's a thing called "extensions", you know.  The Emacs 23 display
engine was abysmally inadequate for bidirectional scripts, and yet
relatively minor changes extended it to be adequate.  I say "minor"
because the basic design of the display engine remains intact.

So I don't quite understand why you decided (without trying) that none
of the existing solutions can be extended to fit the bill.  Are you
seriously claiming that clang is the _only_ way to go?  I hope not.

> > ECB also supports mart completion.
> 
> For C++? Not really.

How do you know?  When did you last try?  If not recently, perhaps it
got better since then?  Did you attempt to analyze what is missing and
how hard would it be to add that?

> >  4 years ago people complained that
> > it's slow, but machines became faster since then, so perhaps things
> > are not so bad now, and we could use Semantic for this purpose.
> 
> If Semantic is slow supporting the easiest, simplest parts of the C++
> source code analysis... think about how slow would it be if it
> implemented the missing parts, those that make C++ compilers really slow
> compared to their C counterparts.

Who said it was slow _today_?  I found complaints from 2009, are you
really going to claim they are still relevant, without checking out?

> >> Clang provides code completion as a sample of its capabilities.
> >> Clang/LLVM in fact is a set of libraries for dealing with C/C++ code.
> >> You can use those libraries for code completion and for any other
> >> feature that requires accessing/processing information contained on
> >> source code: extracting declarations, sanitizing, instrumenting,
> >> optimizing, generating object code...
> >
> > Are there any Emacs packages that support those features?
> 
> Dunno. But the point is that now that a tool exists for doing the heavy
> lifting, creating an Emacs package for exploiting C++ semantic
> information is feasible.
> 
> >> One was already mentioned by Stephen Leake: refactoring.
> >
> > The only Emacs package for this that I could find is proprietary
> > (Xrefactory).  Do you happen to know about any free ones?
> 
> No. My knowledge is far from exhaustive, though.

Then perhaps the assertiveness of your opinions should be on par with
how much you know.

> >> Actually,
> >> anything that requires semantic knowledge of the source code you are
> >> working on. You could ask for a listing of places dependent of word
> >> size, for instance, or highlight the places where certain idiom is used.
> >
> > Doesn't ECB already support such features?
> 
> I don't think so. It can't do that since its understanding of C++ is
> quite limited.

Statistics doesn't understand anything about the underlying phenomena,
and yet it is able to produce very useful results.  IOW, we don't need
to understand C++, we just need to be able to do certain jobs.
Understanding (parts of) it is the means to an end, and that's all.

> >> IIRC I already told you this a few weeks ago, but I'll repeat: a C++
> >> front-end (even without code generation capabilities) requires an
> >> immense amount of work from highly specialized people, and then needs
> >> improvements as new standards are published.
> >
> > Only if you need to be 110% accurate,
> 
> Since 100% is perfect, why should I wish more? ;-)

I don't know, you tell me.

> > which is certainly a requirement
> > for a compiler.  But we don't need such strict requirements for the
> > features we are discussing, I think.
> 
> A defective refactoring tool can easily cause more work than it saves.
> It can introduce subtle bugs, too.

"Defective" is a far cry from "non-strict requirements", don't you
think?

> >> >> Why reinvent the wheel?
> >> >
> >> > Because we cannot get the one that's already invented?
> >> 
> >> "we cannot" isn't the right expression. "we are not allowed" is the
> >> correct description.
> >
> > I'm trying to keep this part of the thread out of politics and into
> > something that could hopefully lead to a working implementation.
> 
> I'm not interested on politics either. I just wanted to be accurate :-)

To what end?

Re: Emacs contributions, C and Lisp

[Óscar Fuentes]

David Engster <deng@randomsample.de> writes:

> You are really fast at dismissing things.

[snip]

> Thanks for the lecture.

Sorry, David. I can see how my position could be interpreted as
the one who belittles yours and Eric's work. I have a lot of respect
for what you achieved. Actually, I think it is possibly the most
impressive piece of work on the Emacs community.

At the same time, I have a bit of knowledge about the work required for
putting together a modern C++ front-end. I'm no expert, but at some
point on the distant past I could qualify as a C++ language lawyer and
had the privilege of observing and interacting with the people who
worked on C++ front-ends. Just making sense of the Standard is something
that requires plenty of work and communication with your peers (the
Standard is too ambiguous on some parts for the level of precission that
an implementer needs, so things are agreed among the handful of people
who write the front-ends and later the agreement is incorporated on the
next Standard, if possible.) And then comes the hardest part: figuring
out how to implement the feature.

Creating a C++ front-end requires years of full-time work from teams of
specialized, experienced, extremely brilliant compiler writers. Please
understand that I would be delusional if I expected that you, on your
free time and on a short enough timeframe to not lag more and more
behind the current language standard, could implement the required
analysis features for accurate and efficient smart code completion.

Sorry again for appearing dismissive towards your work.

[snip]

Re: Emacs contributions, C and Lisp

[John Yates]

[-- Attachment #1: Type: text/plain, Size: 958 bytes --]

On Wed, Feb 26, 2014 at 11:28 AM, David Engster <deng@randomsample.de>
wrote:
>
> IMHO, "Refactoring C++" should not be in the job description.

Yet Google is happily applying clang to just that problem:

Refactoring C++ with Clang:
  http://www.youtube.com/watch?v=U98rhV6wONo

Clang MapReduce -- Automatic C++ Refactoring at Google Scale:
  http://www.youtube.com/watch?v=mVbDzTM21BQ

To me it is the publicizing of that kind of work that undercuts gcc, not
whether DragonEgg links clang with the gcc backend or whether somebody
might offer a proprietary product involving some gcc code.  (If the product
meets with minimal commercial success do we really care that deeply?)

Gcc may still produce better code than clang+llvm but clang is enabling
exciting computer science and engineering for which gcc is inappropriate.
 Not because the source is not available.  But because the gcc codebase
does not expose the right interfaces and abstractions.

/john

[-- Attachment #2: Type: text/html, Size: 1272 bytes --]

Re: Emacs contributions, C and Lisp

[David Kastrup]

Óscar Fuentes <ofv@wanadoo.es> writes:

> Just making sense of the Standard is something that requires plenty of
> work and communication with your peers (the Standard is too ambiguous
> on some parts for the level of precission that an implementer needs,

[...]

> Creating a C++ front-end requires years of full-time work from teams
> of specialized, experienced, extremely brilliant compiler writers.

I seem to remember that the old Germanic Thing gatherings required
making and confirming decisions in two passes, once drunk, once sober.
I have the suspicion that the processes of C++ language design and
implementation are split in a similar manner.

There was a time when C programmers shook their head at the complexity
of Ada.

Now they have C++.

> Please understand that I would be delusional if I expected that you,
> on your free time and on a short enough timeframe to not lag more and
> more behind the current language standard, could implement the
> required analysis features for accurate and efficient smart code
> completion.

Time to go back to Scheme.  At least it isn't hard to ask the "compiler"
about identifiers.  No need to even think about plugins and similar.

Sorry for venting.  In the mean time, we still have to deal with the
thing that is C++.

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[David Engster]

John Yates writes:
> On Wed, Feb 26, 2014 at 11:28 AM, David Engster <deng@randomsample.de> wrote:
>>
>> IMHO, "Refactoring C++" should not be in the job description.
>
> Yet Google is happily applying clang to just that problem:

Where did I say that clang cannot do that?

I did say that if you will accept nothing less than perfection, then by
all means implement your clang-based silver bullet. Stefan already said
that he is not opposed to it.

-David

Re: Emacs contributions, C and Lisp

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> > And what happens if you add
>> >
>> >   B baz(char);
>> >
>> > to the above -- will it show 2 candidates or just one?
>> 
>> Dunno.
>
> I suggest to try (I did).  Maybe then you will be less radical in your
> judgment of having N+1 candidates when only N are strictly needed.

Adding an overload to just make an specific case work on certain
completion package is unacceptable, to say it mildly.

>> As David Engster explained, CEDET does the simplest thing (take
>> one of the overloads and ignore the rest.)
>
> I see nothing wrong with doing the simplest thing, if it gives
> reasonable results.  This is an engineering discipline, not an exact
> science; compromises are our bread and butter.  I'm sure you are well
> aware of that.

Yes, I'm aware of that. That's the reason why when I re-try CEDET every
year, I use it on the most simple parts of my code and then decide that
it is not mature enough for even that usage.

[Lurker: CEDET can be productive when your code is simple enough (or if
you have a high tolerance to completion failures.) There is plenty of
C++ code like that out there, so don't be discouraged by my experience
and try it yourself.]

> So I don't quite understand why you decided (without trying) that none
> of the existing solutions can be extended to fit the bill.

How do you know that I didn't tried?

> Are you seriously claiming that clang is the _only_ way to go? I hope
> not.

On terms of reduced effort, it is the easiest way by far.

>> > ECB also supports mart completion.
>> 
>> For C++? Not really.
>
> How do you know?  When did you last try?

A few hours ago, as described on this very same sub-thread. See above,
you can see a test case where it fails and you discussed it. Apart from
that, I try it every year or so. What makes you think that I'm talking
about CEDET without having experience with it?

> If not recently, perhaps it got better since then?

Surely it got better, but not enough, as demonstrated two messages ago.

> Did you attempt to analyze what is missing and
> how hard would it be to add that?

I'm no compiler expert, but as stated multiple times by now, for
expecting CEDET to work on modern C++ code bases the required effort is
*huge*. And that's suppossing you are a compiler writer with experience
implementing C++ front-ends.

> Who said it was slow _today_?  I found complaints from 2009, are you
> really going to claim they are still relevant, without checking out?

Again, why do you assume that I didn't tried?

IIRC last time I seriously tried CEDET (a year ago) it was fast enough
(although it missed/confused most symbols) on my projects, which are on
the tens of thousands of lines (whithout external libraries). There was
a perceivable lag on each completion request while working on a
destktop-class machine. Other C++ projects which I tinker with are two
orders of magnitude larger than that.

But the important point here is that the most time-consuming analysis
features seem missing from CEDET.

>> > The only Emacs package for this that I could find is proprietary
>> > (Xrefactory).  Do you happen to know about any free ones?
>> 
>> No. My knowledge is far from exhaustive, though.
>
> Then perhaps the assertiveness of your opinions should be on par with
> how much you know.

What are you talking about? What relevance has on this discussion my
knowledge of available tools for Emacs?

> Statistics doesn't understand anything about the underlying phenomena,
> and yet it is able to produce very useful results.  IOW, we don't need
> to understand C++, we just need to be able to do certain jobs.
> Understanding (parts of) it is the means to an end, and that's all.

A C++ source code analysis tool has no need to understand C++?

>> >> IIRC I already told you this a few weeks ago, but I'll repeat: a C++
>> >> front-end (even without code generation capabilities) requires an
>> >> immense amount of work from highly specialized people, and then needs
>> >> improvements as new standards are published.
>> >
>> > Only if you need to be 110% accurate,
>> 
>> Since 100% is perfect, why should I wish more? ;-)
>
> I don't know, you tell me.

I detect a tendency to hyperbole and all-or-nothing argumentation on
your messages. In this case, my emphasis on accurate results is
represented by you as a 110% requirement. This is not constructive.

>> > which is certainly a requirement
>> > for a compiler.  But we don't need such strict requirements for the
>> > features we are discussing, I think.
>> 
>> A defective refactoring tool can easily cause more work than it saves.
>> It can introduce subtle bugs, too.
>
> "Defective" is a far cry from "non-strict requirements", don't you
> think?

A tool that fails on some cases is defective, unless you described its
shortcomings and advertised that it doesn't work on C++, but on a subset
of the language.

It is true that it is unreasonable to expect correct behavior on
concocted cases or even the rare ones, but anything less than that is a
defect.

>> >> "we cannot" isn't the right expression. "we are not allowed" is the
>> >> correct description.
>> >
>> > I'm trying to keep this part of the thread out of politics and into
>> > something that could hopefully lead to a working implementation.
>> 
>> I'm not interested on politics either. I just wanted to be accurate :-)
>
> To what end?

"cannot" is fundamentally different from "not allowed", if you are
looking for the less-resistance path.

Re: Emacs contributions, C and Lisp

[John Yates]

[-- Attachment #1: Type: text/plain, Size: 1188 bytes --]

Daivd,

On Wed, Feb 26, 2014 at 4:49 PM, David Engster <deng@randomsample.de> wrote:

> Where did I say that clang cannot do that [refactor C++]?
>

Please forgive me if I mis-interpreted what you wrote.  Just like Oscar I
have tremendous respect for your contributions to Emacs and CEDET.

I was responding to this paragraph:

> CEDET will most probably never be able to refactor C++ code, aside from
> very simple cases. There are very few IDEs out there which even try to
> do that; from my experience, none of them do it 100% reliably (just
> bring some meta template programming into the game and see what
> happens). IMHO, "Refactoring C++" should not be in the job description.

To me that sounded like you were dismissing all attempts across all IDEs to
implement "Refactoring C++".  Perhaps on re-reading what you meant was that
100% reliable refactoring of C++ should not be a CEDET goal.  Was that what
you meant?  Or was it still something else?

I did say that if you will accept nothing less than perfection, then by
> all means implement your clang-based silver bullet.


Do you foresee a future in which Emacs + ELPA will offer "lead bullet"
level C++ refactoring?

/john

[-- Attachment #2: Type: text/html, Size: 2670 bytes --]

Re: Emacs contributions, C and Lisp

[Stefan Monnier]

> Time to go back to Scheme.  At least it isn't hard to ask the "compiler"
> about identifiers.  No need to even think about plugins and similar.

BTW, one of the points mentioned in this thread is the difficulty of
correct refactoring in the face of metaprogramming, and indeed for
Scheme, the same problem can appear with macros (tho hygiene helps
a bit).


        Stefan

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

     > This is because LLVM does not try to defend users' freedom at all.

    Of course it does.  It gives them the choice of using excellent free
    software.

You have misinterpreted my statement.  LLVM is free software, but it
does not defend users' freedom because only copyleft does that.

Remember that my statement was made in a specific context: LLVM's
practices regarding copyright assignment.  Since they do not maintain
a copyleft, they also don't take the steps necessary to uphold the
copyleft.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Wed, 26 Feb 2014 23:34:53 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> > And what happens if you add
> >> >
> >> >   B baz(char);
> >> >
> >> > to the above -- will it show 2 candidates or just one?
> >> 
> >> Dunno.
> >
> > I suggest to try (I did).  Maybe then you will be less radical in your
> > judgment of having N+1 candidates when only N are strictly needed.
> 
> Adding an overload to just make an specific case work on certain
> completion package is unacceptable, to say it mildly.

I don't know what exactly you tried, and why did you decide I was
doing unacceptable things, but what I actually tried was clang.  After
adding the above line, it only shows one candidate, whereas I think it
should have shown 2.  IMO, showing more candidates than strictly
necessary is a lesser evil than showing less than necessary.

> > So I don't quite understand why you decided (without trying) that none
> > of the existing solutions can be extended to fit the bill.
> 
> How do you know that I didn't tried?

Because you continue saying that nothing but clang is, or can be, good
enough.

> > Are you seriously claiming that clang is the _only_ way to go? I hope
> > not.
> 
> On terms of reduced effort, it is the easiest way by far.

What basis do you have for this assertion?  How many Emacs completion
packages that use clang did you try?  There's only one that I could
find, and it has the following disclosure, which speaks for itself:

  Note that this minor mode isn't meant for serious use: it is meant
  to help experiment with code completion based on Clang.

Given such a small sample, I don't see how can anyone publish claims
such as yours.

> >> > ECB also supports mart completion.
> >> 
> >> For C++? Not really.
> >
> > How do you know?  When did you last try?
> 
> A few hours ago, as described on this very same sub-thread. See above,
> you can see a test case where it fails and you discussed it. Apart from
> that, I try it every year or so. What makes you think that I'm talking
> about CEDET without having experience with it?

I didn't think that, I just asked a question when did you try last.
This kind of stuff needs to be checked frequently, otherwise the
information becomes stale and thus misleading.

> > If not recently, perhaps it got better since then?
> 
> Surely it got better, but not enough, as demonstrated two messages ago.

Nothing of the kind was demonstrated 2 messages ago.

> > Did you attempt to analyze what is missing and
> > how hard would it be to add that?
> 
> I'm no compiler expert, but as stated multiple times by now, for
> expecting CEDET to work on modern C++ code bases the required effort is
> *huge*. And that's suppossing you are a compiler writer with experience
> implementing C++ front-ends.

I think you have a different application in mind.  In any case,
reiterating your opinion doesn't make it more credible, unless you back
that up by some specific evidence or data.

> > Who said it was slow _today_?  I found complaints from 2009, are you
> > really going to claim they are still relevant, without checking out?
> 
> Again, why do you assume that I didn't tried?

How about publishing your data, then?  Perhaps CEDET developers can do
something with your bug reports, and then you will find a year from
now that things did change.

> IIRC last time I seriously tried CEDET (a year ago) it was fast enough
> (although it missed/confused most symbols) on my projects, which are on
> the tens of thousands of lines (whithout external libraries). There was
> a perceivable lag on each completion request while working on a
> destktop-class machine. Other C++ projects which I tinker with are two
> orders of magnitude larger than that.

A year ago my machine was 4 times slower than what I have on my
desktop now.

> >> > The only Emacs package for this that I could find is proprietary
> >> > (Xrefactory).  Do you happen to know about any free ones?
> >> 
> >> No. My knowledge is far from exhaustive, though.
> >
> > Then perhaps the assertiveness of your opinions should be on par with
> > how much you know.
> 
> What are you talking about? What relevance has on this discussion my
> knowledge of available tools for Emacs?

This discussion _is_ about Emacs!  It is not about parsing C++ for any
other purpose, or refactoring by other tools.  Of course, knowledge of
available tools for Emacs is extremely relevant.

> > Statistics doesn't understand anything about the underlying phenomena,
> > and yet it is able to produce very useful results.  IOW, we don't need
> > to understand C++, we just need to be able to do certain jobs.
> > Understanding (parts of) it is the means to an end, and that's all.
> 
> A C++ source code analysis tool has no need to understand C++?

Just enough to do its job, but no more.

As another example, look at etags: it doesn't understand any language
it supports, and yet it generally does a very good job in finding the
identifiers.

> >> > Only if you need to be 110% accurate,
> >> 
> >> Since 100% is perfect, why should I wish more? ;-)
> >
> > I don't know, you tell me.
> 
> I detect a tendency to hyperbole and all-or-nothing argumentation on
> your messages. In this case, my emphasis on accurate results is
> represented by you as a 110% requirement. This is not constructive.

I agree it is not constructive to raise the bar too high.  But isn't
that what you are doing in this thread?  Why not start with some
moderate requirements, and then move forward?  Not everyone works on
projects in hundreds of thousands of lines of code.

> >> A defective refactoring tool can easily cause more work than it saves.
> >> It can introduce subtle bugs, too.
> >
> > "Defective" is a far cry from "non-strict requirements", don't you
> > think?
> 
> A tool that fails on some cases is defective

Then you will surely call Emacs "defective", because most if its
heuristics are not perfect.

> "cannot" is fundamentally different from "not allowed", if you are
> looking for the less-resistance path.

Why would we be looking for the less-resistance path?  It's not like
clang-using packages are queuing up for inclusion, it actually sounds
like they don't even exist yet.  A perfect time for looking elsewhere
and finding a solution that will do its job reasonably well.

Re: Emacs contributions, C and Lisp

[Dmitry Gutov]

Eli Zaretskii <eliz@gnu.org> writes:

>> > I suggest to try (I did).  Maybe then you will be less radical in your
>> > judgment of having N+1 candidates when only N are strictly needed.
>> 
>> Adding an overload to just make an specific case work on certain
>> completion package is unacceptable, to say it mildly.
>
> I don't know what exactly you tried, and why did you decide I was
> doing unacceptable things, but what I actually tried was clang.  After
> adding the above line, it only shows one candidate, whereas I think it
> should have shown 2.  IMO, showing more candidates than strictly
> necessary is a lesser evil than showing less than necessary.

It shows only 'bar' to me (and another, broken option, which probably
means the package I tried - company-clang - doesn't handle the output
well enough).

I believe it's the correct result, though. 'char' is the more specific
overload, so the compiler should choose it.

> How many Emacs completion packages that use clang did you try? There's
> only one that I could find

You certainly haven't tried hard. I personally gave you links to several
in this very thread.

>> >> A defective refactoring tool can easily cause more work than it saves.
>> >> It can introduce subtle bugs, too.
>> >
>> > "Defective" is a far cry from "non-strict requirements", don't you
>> > think?
>> 
>> A tool that fails on some cases is defective
>
> Then you will surely call Emacs "defective", because most if its
> heuristics are not perfect.

You both might want to step back from generailzations.

The statement about a "defective refactoring tool" is one most
developers would agree.

A good refactoring tool is one you can apply to a large codebase and
then calmly sleep at night without hand-checking every transformation
that it did.

> Why would we be looking for the less-resistance path?  It's not like
> clang-using packages are queuing up for inclusion, it actually sounds
> like they don't even exist yet.

Ahem.

Re: Emacs contributions, C and Lisp

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> >> > And what happens if you add
>> >> >
>> >> >   B baz(char);
>> >> >
>> >> > to the above -- will it show 2 candidates or just one?
>> >> 
>> >> Dunno.
>> >
>> > I suggest to try (I did).  Maybe then you will be less radical in your
>> > judgment of having N+1 candidates when only N are strictly needed.
>> 
>> Adding an overload to just make an specific case work on certain
>> completion package is unacceptable, to say it mildly.
>
> I don't know what exactly you tried, and why did you decide I was
> doing unacceptable things, but what I actually tried was clang.  After
> adding the above line, it only shows one candidate, whereas I think it
> should have shown 2.

No. After adding your overload, there is just one candidate, namely,
the one that exactly matches its argument type: char.

> IMO, showing more candidates than strictly
> necessary is a lesser evil than showing less than necessary.

If this was an attempt of pointing to a defect on Clang, you failed, and
it is not surprising, because Clang uses the same machinery for deciding
the correct overload when it is acting as a compiler as when it is asked
for completion candidates. Surely, if Clang failed to give the correct
answer for the completion case, it would be a very broken C++ compiler
too.

>> > So I don't quite understand why you decided (without trying) that none
>> > of the existing solutions can be extended to fit the bill.
>> 
>> How do you know that I didn't tried?
>
> Because you continue saying that nothing but clang is, or can be, good
> enough.

I'm saying that a compiler infraestructure, with the correct interfaces,
makes any other approach look as a hack, for the reason stated on my
previous paragraph. Right now only Clang fits the bill (possibly GCC
too, modulo API convenience) but more compilers could qualify on the
future.

>> > Are you seriously claiming that clang is the _only_ way to go? I hope
>> > not.
>> 
>> On terms of reduced effort, it is the easiest way by far.
>
> What basis do you have for this assertion?  How many Emacs completion
> packages that use clang did you try?  There's only one that I could
> find, and it has the following disclosure, which speaks for itself:
>
>   Note that this minor mode isn't meant for serious use: it is meant
>   to help experiment with code completion based on Clang.
>
> Given such a small sample, I don't see how can anyone publish claims
> such as yours.

What are you trying to say with this? Does Clang's usefulness depend on
whether or not an Emacs package exists?

[snip]

>> > If not recently, perhaps it got better since then?
>> 
>> Surely it got better, but not enough, as demonstrated two messages ago.
>
> Nothing of the kind was demonstrated 2 messages ago.

Are you serious? It failed on a case that could come from the first day
of a course on C++ fundamentals. No complex expression type deduction,
no template metaprogramming, no namespace resolution... it's C++ coming
from the 1980s.

Eli, are you a C++ programmer? Do you code in C++ on a regular basis?

If you answer "no" to any of those questions, we'll better stop this
discussion here and now.

>> > Did you attempt to analyze what is missing and
>> > how hard would it be to add that?
>> 
>> I'm no compiler expert, but as stated multiple times by now, for
>> expecting CEDET to work on modern C++ code bases the required effort is
>> *huge*. And that's suppossing you are a compiler writer with experience
>> implementing C++ front-ends.
>
> I think you have a different application in mind.  In any case,
> reiterating your opinion doesn't make it more credible, unless you back
> that up by some specific evidence or data.
>
>> > Who said it was slow _today_?  I found complaints from 2009, are you
>> > really going to claim they are still relevant, without checking out?
>> 
>> Again, why do you assume that I didn't tried?
>
> How about publishing your data, then?

Which data? That CEDET doesn't know about overloads? They are very aware
of that, as one of the maintainers acknowledged on this thread.

> Perhaps CEDET developers can do
> something with your bug reports, and then you will find a year from
> now that things did change.

Sorry, but this looks more like handwaving than real argumentation. I'm
not the one who decided to work on a Elisp-based C++ code analysis tool.
I'm the one who thinks that that enterprise has a limited range of
applicability and there are other approaches with are much more
promising. Why should I invest my time on CEDET?

[snip]

> This discussion _is_ about Emacs!  It is not about parsing C++ for any
> other purpose, or refactoring by other tools.  Of course, knowledge of
> available tools for Emacs is extremely relevant.

If the tools were available, this discussion wouldn't arise.

>> > Statistics doesn't understand anything about the underlying phenomena,
>> > and yet it is able to produce very useful results.  IOW, we don't need
>> > to understand C++, we just need to be able to do certain jobs.
>> > Understanding (parts of) it is the means to an end, and that's all.
>> 
>> A C++ source code analysis tool has no need to understand C++?
>
> Just enough to do its job, but no more.
>
> As another example, look at etags: it doesn't understand any language
> it supports, and yet it generally does a very good job in finding the
> identifiers.

It is obvious that you don't understand the problem (well, it was
obvious since the first time you suggested to implement the required
functionality on Elisp all the way down). What Etags does is absolutely
trivial compared to the required analysis for smart code completion, not
to mention refactoring.

[snip]

>> A tool that fails on some cases is defective
>
> Then you will surely call Emacs "defective", because most if its
> heuristics are not perfect.

Here you go with all-or-nothing again. I can use Emacs for hours without
facing any glitch. My experience with CEDET C++ code completion is that
failure is the most likely outcome. Worse: while using it I need to
direct my attention to checking that what CEDET suggests makes sense
(when it manages to suggest something) and that makes me less productive
even on the cases where CEDET gives the right answer.

>> "cannot" is fundamentally different from "not allowed", if you are
>> looking for the less-resistance path.
>
> Why would we be looking for the less-resistance path?  It's not like
> clang-using packages are queuing up for inclusion,

Surely they will not queue for inclusion, because they all were rejected
in advance.

> it actually sounds
> like they don't even exist yet.  A perfect time for looking elsewhere
> and finding a solution that will do its job reasonably well.

No, a solution that fits RMS decision. Not to be confused with "doing
the job well".

Re: Emacs contributions, C and Lisp

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Cc: Óscar Fuentes <ofv@wanadoo.es>,  emacs-devel@gnu.org
> Date: Thu, 27 Feb 2014 21:06:32 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> > I suggest to try (I did).  Maybe then you will be less radical in your
> >> > judgment of having N+1 candidates when only N are strictly needed.
> >> 
> >> Adding an overload to just make an specific case work on certain
> >> completion package is unacceptable, to say it mildly.
> >
> > I don't know what exactly you tried, and why did you decide I was
> > doing unacceptable things, but what I actually tried was clang.  After
> > adding the above line, it only shows one candidate, whereas I think it
> > should have shown 2.  IMO, showing more candidates than strictly
> > necessary is a lesser evil than showing less than necessary.
> 
> It shows only 'bar' to me (and another, broken option, which probably
> means the package I tried - company-clang - doesn't handle the output
> well enough).

That's what I saw, yes.

> I believe it's the correct result, though. 'char' is the more specific
> overload, so the compiler should choose it.

??? Are you saying that the one shown as completion without that
additional line was incorrect?  If it was correct then, it is still
correct after the addition, surely.

> > How many Emacs completion packages that use clang did you try? There's
> > only one that I could find
> 
> You certainly haven't tried hard. I personally gave you links to several
> in this very thread.

Not for C++, AFAICT.

> The statement about a "defective refactoring tool" is one most
> developers would agree.

"Most" as in 2?

> A good refactoring tool is one you can apply to a large codebase and
> then calmly sleep at night without hand-checking every transformation
> that it did.

Where did you find good refactoring tools for C++?

> > Why would we be looking for the less-resistance path?  It's not like
> > clang-using packages are queuing up for inclusion, it actually sounds
> > like they don't even exist yet.
> 
> Ahem.

Yeah, right.

Re: Emacs contributions, C and Lisp

[David Engster]

John Yates writes:
> I was responding to this paragraph:
>
>> CEDET will most probably never be able to refactor C++ code, aside from
>> very simple cases. There are very few IDEs out there which even try to
>> do that; from my experience, none of them do it 100% reliably (just
>> bring some meta template programming into the game and see what
>> happens). IMHO, "Refactoring C++" should not be in the job description.
>
> To me that sounded like you were dismissing all attempts across all IDEs to
> implement "Refactoring C++".  Perhaps on re-reading what you meant was that
> 100% reliable refactoring of C++ should not be a CEDET goal.  Was that what you
> meant?

It is not *my* goal, and it is used here to steer people away from
working on Semantic's C++ parser, saying it is pointless if it can never
do refactoring, which is a *much* harder problem than providing
completions. I don't see the point in discouraging people like that.

Anyway, Stefan gave is OK on libclang usage, so the initial problem that
started this discussion is solved, and I suggest you start hacking.

>     I did say that if you will accept nothing less than perfection,
>     then by all means implement your clang-based silver bullet.
>
> Do you foresee a future in which Emacs + ELPA will offer "lead bullet" level
> C++ refactoring?

As always: unless someone starts working on these things, nothing will
happen, and surely nothing comes out of those centi-threads we seem to
have every month nowadays.

-David

Re: Emacs contributions, C and Lisp

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Thu, 27 Feb 2014 20:08:30 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> >> > And what happens if you add
> >> >> >
> >> >> >   B baz(char);
> >> >> >
> >> >> > to the above -- will it show 2 candidates or just one?
> >> >> 
> >> >> Dunno.
> >> >
> >> > I suggest to try (I did).  Maybe then you will be less radical in your
> >> > judgment of having N+1 candidates when only N are strictly needed.
> >> 
> >> Adding an overload to just make an specific case work on certain
> >> completion package is unacceptable, to say it mildly.
> >
> > I don't know what exactly you tried, and why did you decide I was
> > doing unacceptable things, but what I actually tried was clang.  After
> > adding the above line, it only shows one candidate, whereas I think it
> > should have shown 2.
> 
> No. After adding your overload, there is just one candidate, namely,
> the one that exactly matches its argument type: char.

What happened to the other one?  It is still in the source.

> > IMO, showing more candidates than strictly
> > necessary is a lesser evil than showing less than necessary.
> 
> If this was an attempt of pointing to a defect on Clang, you failed, and
> it is not surprising, because Clang uses the same machinery for deciding
> the correct overload when it is acting as a compiler as when it is asked
> for completion candidates. Surely, if Clang failed to give the correct
> answer for the completion case, it would be a very broken C++ compiler
> too.

So?

> I'm saying that a compiler infraestructure, with the correct interfaces,
> makes any other approach look as a hack, for the reason stated on my
> previous paragraph.

Well, I disagree, and I guess we will have to leave this at that.

> Right now only Clang fits the bill

No, it doesn't, not as long as there are no decent production-quality
Emacs packages that use it.

> Are you serious?

Yes.

> Eli, are you a C++ programmer? Do you code in C++ on a regular basis?

Yes!

> >> > Who said it was slow _today_?  I found complaints from 2009, are you
> >> > really going to claim they are still relevant, without checking out?
> >> 
> >> Again, why do you assume that I didn't tried?
> >
> > How about publishing your data, then?
> 
> Which data?

See above: that it is slow.

> Sorry, but this looks more like handwaving than real argumentation.

And you didn't?

> I'm not the one who decided to work on a Elisp-based C++ code
> analysis tool.  I'm the one who thinks that that enterprise has a
> limited range of applicability and there are other approaches with
> are much more promising. Why should I invest my time on CEDET?

I hope because you want Emacs to become a better programming editor.

> > This discussion _is_ about Emacs!  It is not about parsing C++ for any
> > other purpose, or refactoring by other tools.  Of course, knowledge of
> > available tools for Emacs is extremely relevant.
> 
> If the tools were available, this discussion wouldn't arise.

But since they aren't available, we have nothing to compare to, and so
saying that CEDET or ECB are inadequate without that feature doesn't
have any real basis.  It's just propaganda.

> >> > Statistics doesn't understand anything about the underlying phenomena,
> >> > and yet it is able to produce very useful results.  IOW, we don't need
> >> > to understand C++, we just need to be able to do certain jobs.
> >> > Understanding (parts of) it is the means to an end, and that's all.
> >> 
> >> A C++ source code analysis tool has no need to understand C++?
> >
> > Just enough to do its job, but no more.
> >
> > As another example, look at etags: it doesn't understand any language
> > it supports, and yet it generally does a very good job in finding the
> > identifiers.
> 
> It is obvious that you don't understand the problem

And you evidently don't understand the concept of analogy.

> Surely they will not queue for inclusion, because they all were rejected
> in advance.

They don't exist, first and foremost.  There's nothing to reject.

Re: Emacs contributions, C and Lisp

[Eli Zaretskii]

> From: David Engster <deng@randomsample.de>
> Date: Thu, 27 Feb 2014 21:31:44 +0100
> Cc: Óscar Fuentes <ofv@wanadoo.es>,
> 	Emacs developers <emacs-devel@gnu.org>
> 
> It is not *my* goal, and it is used here to steer people away from
> working on Semantic's C++ parser, saying it is pointless if it can never
> do refactoring, which is a *much* harder problem than providing
> completions. I don't see the point in discouraging people like that.

Right.

> As always: unless someone starts working on these things, nothing will
> happen, and surely nothing comes out of those centi-threads we seem to
> have every month nowadays.

Indeed, I hope someone will put their money where their mouth is, and
start coding.

Re: Emacs contributions, C and Lisp

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

> What happened to the other one?  It is still in the source.

Smart completion is all about showing the appropriate completions for
the context where the point is. Once you know the number and type of
arguments, if you have zero or more than one acceptable overloads, the
code is malformed. Barred that, there is one and only one correct
overload, which is the one the compiler will use. A correct smart code
completion system will show precisely that overload.

That's what a C++ programmer expects, and then I'm perplexed when I read
this:

[snip]

>> Eli, are you a C++ programmer? Do you code in C++ on a regular basis?
>
> Yes!

It is obvious then that we have here a complete miscomunication, so I'll
stop the discussion here.

[snip]

Re: Emacs contributions, C and Lisp

[Dmitry Gutov]

Eli Zaretskii <eliz@gnu.org> writes:

>> I believe it's the correct result, though. 'char' is the more specific
>> overload, so the compiler should choose it.
>
> ??? Are you saying that the one shown as completion without that
> additional line was incorrect?  If it was correct then, it is still
> correct after the addition, surely.

Not really. Adding a new function might make the code invalid (say, if
it has the same name and the types of arguments as an existing one), or
make it behave in a different way.

http://en.cppreference.com/w/cpp/language/overload_resolution

>> > How many Emacs completion packages that use clang did you try? There's
>> > only one that I could find
>> 
>> You certainly haven't tried hard. I personally gave you links to several
>> in this very thread.
>
> Not for C++, AFAICT.

Those are the same packages that offer C completion. Thank the magical
LLVM sprinkles.

Although it seems I haven't mentioned any specific ones here, after all,
only all of them in general.

Here are the bigger ones:

https://github.com/Sarcasm/irony-mode
https://github.com/Golevka/emacs-clang-complete-async/

And there's also company-clang in
http://elpa.gnu.org/packages/company.html, which I'm personally insulted
you've ignored.

It's much simpler, but it also has its users. Unlike those above, it has
no persistent server, but it additionally offers Objective-C support.

Re: Emacs contributions, C and Lisp

[Stephen J. Turnbull]

Richard Stallman writes:
 > [[[ To any NSA and FBI agents reading my email: please consider    ]]]
 > [[[ whether defending the US Constitution against all enemies,     ]]]
 > [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
 > 
 >      > This is because LLVM does not try to defend users' freedom at all.
 > 
 >     Of course it does.  It gives them the choice of using excellent free
 >     software.
 > 
 > You have misinterpreted my statement.  LLVM is free software, but it
 > does not defend users' freedom because only copyleft does that.

That's a nice soundbite, but I'll resist the temptation to disagree
verbosely.

However, this situation is easily enough changed.  The useful programs
from the LLVM project can be forked (AFAIK their license is not
perversely incompatible with the GPL) as GNU projects under the "GPL
v3 or later" permission schema.

Would you object to that?

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    Anyway, Stefan gave is OK on libclang usage, 

That statement seems to be a misunderstanding.  My decision, as head
of the GNU Project, is that we will not install anything in Emacs or
ELPA that uses clang or LLVM.

You can use CEDET, you can use GCC, you can use both, or you
can use something else.  But not clang or LLVM.

This decision is necessary for achieving more the goal
of the GNU Project, which is to give computer users' freedom
in their computing in general -- not just in their text editing.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[Dmitry Gutov]

Richard Stallman <rms@gnu.org> writes:

> or ELPA that uses clang or LLVM.
  ^^^^^^^

This ship has sailed.

> You can use CEDET, you can use GCC, you can use both, or you
> can use something else.  But not clang or LLVM.

I think this conflicts with your previous statement that using Clang
for some features would be fine as long as Emacs supports using GCC for
the same features.

Re: Emacs contributions, C and Lisp

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Thu, 27 Feb 2014 22:15:37 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > What happened to the other one?  It is still in the source.
> 
> Smart completion is all about showing the appropriate completions for
> the context where the point is. Once you know the number and type of
> arguments, if you have zero or more than one acceptable overloads, the
> code is malformed. Barred that, there is one and only one correct
> overload, which is the one the compiler will use. A correct smart code
> completion system will show precisely that overload.
> 
> That's what a C++ programmer expects

Not this one.

> >> Eli, are you a C++ programmer? Do you code in C++ on a regular basis?
> >
> > Yes!
> 
> It is obvious then that we have here a complete miscomunication, so I'll
> stop the discussion here.

It is indeed time to stop this, as it doesn't seem to lead anywhere.

All I can say is that it is sad to see veteran Emacs contributors
taking a stance that discourages newcomers from working on important
features, just to make some point they personally see as important.

And to others I say: don't be discouraged by skeptics.  Don't believe
those who make unsubstantiated claims that advancing Emacs in this
area needs many man-years of work by people who must be world-class
experts in C++.  You have several starting points already, they were
mentioned in this thread.  Pick up the one that best suits your needs
and talents, or the one you just happen to like the most, and start
coding.  Heck, even comparing the available solutions (including
clang-based) applied to a non-toy package could be a very good start:
it could lead to useful discussions about the best routes to advance
and how best to attack the existing weaknesses.  At the very least we
will know where we stand, and it will be harder to hand-wave after
that.

Happy hacking!

Re: Emacs contributions, C and Lisp

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Cc: ofv@wanadoo.es,  emacs-devel@gnu.org
> Date: Fri, 28 Feb 2014 03:40:41 +0200
> 
> https://github.com/Sarcasm/irony-mode
> https://github.com/Golevka/emacs-clang-complete-async/

Thanks.  The second one seems not to be actively maintained anymore.

> And there's also company-clang in
> http://elpa.gnu.org/packages/company.html, which I'm personally insulted
> you've ignored.
> 
> It's much simpler, but it also has its users. Unlike those above, it has
> no persistent server, but it additionally offers Objective-C support.

How about teaching Company mode to work with Semantic?

Re: Emacs contributions, C and Lisp

[Dmitry Gutov]

On 28.02.2014 08:53, Eli Zaretskii wrote:
> Thanks.  The second one seems not to be actively maintained anymore.

Possibly. It still has users, though.

> How about teaching Company mode to work with Semantic?

It already does.

Re: Emacs contributions, C and Lisp

[Eli Zaretskii]

> Date: Fri, 28 Feb 2014 08:58:53 +0200
> From: Dmitry Gutov <dgutov@yandex.ru>
> CC: ofv@wanadoo.es, emacs-devel@gnu.org
> 
> On 28.02.2014 08:53, Eli Zaretskii wrote:
> > Thanks.  The second one seems not to be actively maintained anymore.
> 
> Possibly. It still has users, though.
> 
> > How about teaching Company mode to work with Semantic?
> 
> It already does.

Good.

Re: Emacs contributions, C and Lisp

[David Kastrup]

Óscar Fuentes <ofv@wanadoo.es> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> What happened to the other one?  It is still in the source.
>
> Smart completion is all about showing the appropriate completions for
> the context where the point is. Once you know the number and type of
> arguments, if you have zero or more than one acceptable overloads, the
> code is malformed. Barred that, there is one and only one correct
> overload, which is the one the compiler will use. A correct smart code
> completion system will show precisely that overload.
>
> That's what a C++ programmer expects, and then I'm perplexed when I read
> this:
>
> [snip]
>
>>> Eli, are you a C++ programmer? Do you code in C++ on a regular basis?
>>
>> Yes!
>
> It is obvious then that we have here a complete miscomunication, so I'll
> stop the discussion here.

The miscommunication appears to be that you apparently are convinced
that coding C++ on a regular basis is impossible without using crutches.
It is my opinion that writing a code style that can neither be written
nor understood without reverting to computer help is a bad idea.

Writing a code style that can neither be written nor understood without
letting a full-blown compiler take a look at it is an even worse idea.
If you consider that a _regular_ part of your coding practices, you are
writing code that can only be maintained by throwing it away and
rewriting from scratch.  This step will become necessary in particular
whenever the C++ standards change in some manner.

That may be putting it overbluntly, but I am absolutely unsurprised that
someone coding C++ on a regular basis does _not_ find himself on the
edge of ambiguities so often that he requires constant handholding by
his tools in order to produce working code.

When one declaration changes the meaning and syntax of a program all
over one file (and yes, this sort of thing _can_ happen with C++),
getting things right might require a full-file parse.  When presented
with a preexisting C++ file, being able to get the actual meaning out by
the use of exhaustive tools is nice.  When _writing_ a C++ program, it's
preferable to stay away from those edges and thus get along with more
simplistic tools.  Or even none at all.

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 704 bytes --]

On 02/28/2014 01:13 AM, David Kastrup wrote:
> When one declaration changes the meaning and syntax of a program all
> over one file (and yes, this sort of thing _can_ happen with C++),
> getting things right might require a full-file parse.  When presented
> with a preexisting C++ file, being able to get the actual meaning out by
> the use of exhaustive tools is nice.  When _writing_ a C++ program, it's
> preferable to stay away from those edges and thus get along with more
> simplistic tools.  Or even none at all.

You might believe that --- and you may even be right --- but your
personal prescriptions for software development shouldn't affect the
feature-set of a generic editor.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 901 bytes --]

Re: Emacs contributions, C and Lisp

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> Richard Stallman writes:
>  > [[[ To any NSA and FBI agents reading my email: please consider    ]]]
>  > [[[ whether defending the US Constitution against all enemies,     ]]]
>  > [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>  > 
>  >      > This is because LLVM does not try to defend users' freedom at all.
>  > 
>  >     Of course it does.  It gives them the choice of using excellent free
>  >     software.
>  > 
>  > You have misinterpreted my statement.  LLVM is free software, but it
>  > does not defend users' freedom because only copyleft does that.
>
> That's a nice soundbite, but I'll resist the temptation to disagree
> verbosely.
>
> However, this situation is easily enough changed.  The useful programs
> from the LLVM project can be forked (AFAIK their license is not
> perversely incompatible with the GPL) as GNU projects under the "GPL
> v3 or later" permission schema.
>
> Would you object to that?

I'd object, for basically practical reasons.  You can fork code, but you
cannot fork a community.  A fork of the LLVM codebase under the GPLv3
makes only sense if you actually add nontrivial nonseparable components
under the GPLv3 or the code base can be just swapped out.  If you have
nontrivial nonseparable components, but an actively developed important
upstream, you need to constantly reintegrate the upstream work in order
to keep the edge.

Now if the upstream is a weak license like X11, at some point of time
the developers might say "that fork is annoying, let's relicense what we
have now under BSD with advertising clause and cut off those others".
They'll likely retain the majority of their development community but
hang out the fork under GPLv3 to dry.  That will always be a risk you
work with, and the source of that risk is that the principal work is
made available under a non-copyleft license.

So basically, with an already determined development community, this
sort of licensed fork is a non-starter because of not carrying a
community with it.  Skipping over the problem of _starting_ a new
community, we can take a look at the psychological effects of this kind
of setup by looking at the OpenOffice/LibreOffice fork.  Here we can
glance over the non-starter aspect since the fork happened on an
existing code base, and the permissively licensed "upstream" (with
regard to licensing and thus automatically permitted code flow) got its
licensing change considerable time _after_ the fork.

The relations between both sides of the fork are strained, and a
considerable part of that strain is the asymmetric licensing situation.
This strain clearly keeps the communities from intermingling, meaning
that many constributors identify with one side of the fork.

So an LLVM fork would be a move predictably causing lots of bad blood.
It's not like we haven't paid that sort of price quite a few times.  But
it would also predictably be a move that would lead nowhere.  And that's
self-defeating.

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[David Kastrup]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>     Anyway, Stefan gave is OK on libclang usage, 
>
> That statement seems to be a misunderstanding.  My decision, as head
> of the GNU Project, is that we will not install anything in Emacs or
> ELPA that uses clang or LLVM.

I understood it so far as "that does not specifically use clang or
LLVM".  Like with supporting Windows, I would have guessed that we are
not going to introduce functionality that _requires_ the use of clang or
LLVM but would not object to functionality supporting several compilers
in a way that does not lead people to _prefer_ clang/LLVM over GCC.

I think any state where there would be no obvious incentive to install
clang/LLVM if your end goal was to compile software using GCC would be
fine.

Now the case of Windows is obviously different (as Windows is
proprietary and is more of a platform rather than a tool), but I would
guess that it would be reasonable to apply the same logic here.

Is that a correct interpretation?

> You can use CEDET, you can use GCC, you can use both, or you
> can use something else.  But not clang or LLVM.
>
> This decision is necessary for achieving more the goal
> of the GNU Project, which is to give computer users' freedom
> in their computing in general -- not just in their text editing.

Well, our usual stance was more or less that there is nothing wrong with
bringing free systems into the reach of people using some proprietary
software as long as we are not responsible for making the proprietary
software more desirable than its free counterpart.

While supporting non-copyleft free software is basically implicitly
supporting proprietary forks of it, I don't see that we need to apply
stricter rules to the non-copyleft free software itself than to the
proprietary versions.

So far, I did not see that we did.  And I consider it important that we
get this point cleared up as it will have quite a bit of impact on the
motivation of those working on GCC-based support of completion in Emacs
as well if they know in advance whether this feature will have to be
GCC-only or can be extended to include other compilers once the
GCC-based support is solid and convincing.

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[David Kastrup]

Dmitry Gutov <dgutov@yandex.ru> writes:

> Richard Stallman <rms@gnu.org> writes:
>
>> or ELPA that uses clang or LLVM.
>   ^^^^^^^
>
> This ship has sailed.

Any ship sailing under our flag can be called back to port in case of
necessity.  That's not a criterion.

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[David Kastrup]

Daniel Colascione <dancol@dancol.org> writes:

> On 02/28/2014 01:13 AM, David Kastrup wrote:
>> When one declaration changes the meaning and syntax of a program all
>> over one file (and yes, this sort of thing _can_ happen with C++),
>> getting things right might require a full-file parse.  When presented
>> with a preexisting C++ file, being able to get the actual meaning out
>> by the use of exhaustive tools is nice.  When _writing_ a C++
>> program, it's preferable to stay away from those edges and thus get
>> along with more simplistic tools.  Or even none at all.
>
> You might believe that --- and you may even be right --- but your
> personal prescriptions for software development shouldn't affect the
> feature-set of a generic editor.

Ultimately, reality will affect the feature set of a generic editor.
Any feature that requires per-keystroke reparsing of the entire
compilation unit to work is not feasible in an editing workflow.  That
kind of thing is ok for code browsing, not for writing.

At any rate, it was Óscar's claim that it is so utterly absurd to state
being a regular C++ programmer when one does not rely on code-explaining
support tools that he basically called Eli a fraud.

That's a bit stronger than "personal prescriptions for software
development".

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1199 bytes --]

On 02/28/2014 02:07 AM, David Kastrup wrote:
> Daniel Colascione <dancol@dancol.org> writes:
> 
>> On 02/28/2014 01:13 AM, David Kastrup wrote:
>>> When one declaration changes the meaning and syntax of a program all
>>> over one file (and yes, this sort of thing _can_ happen with C++),
>>> getting things right might require a full-file parse.  When presented
>>> with a preexisting C++ file, being able to get the actual meaning out
>>> by the use of exhaustive tools is nice.  When _writing_ a C++
>>> program, it's preferable to stay away from those edges and thus get
>>> along with more simplistic tools.  Or even none at all.
>>
>> You might believe that --- and you may even be right --- but your
>> personal prescriptions for software development shouldn't affect the
>> feature-set of a generic editor.
> 
> Ultimately, reality will affect the feature set of a generic editor.
> Any feature that requires per-keystroke reparsing of the entire
> compilation unit to work is not feasible in an editing workflow.  That
> kind of thing is ok for code browsing, not for writing.

http://pdf.aminer.org/000/542/897/incremental_analysis_of_real_programming_languages.pdf


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 901 bytes --]

Re: Emacs contributions, C and Lisp

[Eli Zaretskii]

> Date: Fri, 28 Feb 2014 01:20:34 -0800
> From: Daniel Colascione <dancol@dancol.org>
> 
> your personal prescriptions for software development shouldn't
> affect the feature-set of a generic editor.

They should _allow_ them, though.  They could even be the default
behavior, see the way Emacs indents C code by default.

Re: Emacs contributions, C and Lisp

[Óscar Fuentes]

David Kastrup <dak@gnu.org> writes:

[snip]

> At any rate, it was Óscar's claim that it is so utterly absurd to state
> being a regular C++ programmer when one does not rely on code-explaining
> support tools that he basically called Eli a fraud.

This is beyond inflammatory. Almost libelous, I'll say.

Please stop.

[snip]

Re: Emacs contributions, C and Lisp

[Óscar Fuentes]

David Kastrup <dak@gnu.org> writes:

[snip]

>> It is obvious then that we have here a complete miscomunication, so I'll
>> stop the discussion here.
>
> The miscommunication appears to be that you apparently are convinced
> that coding C++ on a regular basis is impossible without using
> crutches.

You are completely lost at this.

> It is my opinion that writing a code style that can neither be written
> nor understood without reverting to computer help is a bad idea.

FYI: I've coded C++ for almost 20 years. Never depended on that sort of
tool.

[more nonsense snipped]

Re: Emacs contributions, C and Lisp

[David Kastrup]

Óscar Fuentes <ofv@wanadoo.es> writes:

> David Kastrup <dak@gnu.org> writes:
>
> [snip]
>
>> At any rate, it was Óscar's claim that it is so utterly absurd to state
>> being a regular C++ programmer when one does not rely on code-explaining
>> support tools that he basically called Eli a fraud.
>
> This is beyond inflammatory. Almost libelous, I'll say.
>
> Please stop.
>
> [snip]

Let's quote your statement in full (every [snip] in here was done by
yourself):

    Eli Zaretskii <eliz@gnu.org> writes:

    > What happened to the other one?  It is still in the source.

    Smart completion is all about showing the appropriate completions for
    the context where the point is. Once you know the number and type of
    arguments, if you have zero or more than one acceptable overloads, the
    code is malformed. Barred that, there is one and only one correct
    overload, which is the one the compiler will use. A correct smart code
    completion system will show precisely that overload.

    That's what a C++ programmer expects, and then I'm perplexed when I read
    this:

    [snip]

    >> Eli, are you a C++ programmer? Do you code in C++ on a regular basis?
    >
    > Yes!

    It is obvious then that we have here a complete miscomunication, so I'll
    stop the discussion here.

    [snip]

You make an ex cathedra statement "That's what a C++ programmer
expects", state that you are "perplexed" when somebody who disagrees
purports to be a C++ programmer and take it as a reason to abort the
discussion because there is no common basis for communication.

Correct, or not?

Yes, that's inflammatory and almost libelous.  Because it is a summary
of something that can hardly be read in any way that is _not_
inflammatory and almost libelous.  Feel free to point out any other
valid reading of it, even though it could lead to a continuation of a
discussion you want to stop.

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[Óscar Fuentes]

David Kastrup <dak@gnu.org> writes:

[snip]

> You make an ex cathedra statement "That's what a C++ programmer
> expects", state that you are "perplexed" when somebody who disagrees
> purports to be a C++ programmer and take it as a reason to abort the
> discussion because there is no common basis for communication.
>
> Correct, or not?

Indeed there was no common basis for communication, but that is not the
same as dismissing the interlocutor's POV as worthless. Much less to
accuse him of being a fraud.

I'm genuinely confused by Eli's stance. At the point you quoted I was
already suspecting that the case he was talking about was not the same
as mine and clarifying the issue seemed of little value because, for me,
the main motivation of the discussion did not depend on our opinions
about an specific case.

Also, there are lots of programmers who are proficient on a very small
subset of the language and claim to be C++ programmers. If I were one of
those, I wouldn't make that claim, but truth is that they do useful,
productive work and in that sense they are C++ programmers. So I
acknowledge Eli's right to say he is a C++ programmer even *if* he is
unacquainted with such basic concepts as function overloading.

> Yes, that's inflammatory and almost libelous.  Because it is a summary
> of something that can hardly be read in any way that is _not_
> inflammatory and almost libelous.  Feel free to point out any other
> valid reading of it, even though it could lead to a continuation of a
> discussion you want to stop.

Eli took no offense from my response. That should be a strong hint for
you.

Re: Emacs contributions, C and Lisp

[John Yates]

[-- Attachment #1: Type: text/plain, Size: 291 bytes --]

On Thu, Feb 27, 2014 at 8:40 PM, Dmitry Gutov <dgutov@yandex.ru> wrote:

> Here are the bigger ones:
>
> https://github.com/Sarcasm/irony-mode
> https://github.com/Golevka/emacs-clang-complete-async/


For completeness and actively maintained::

https://github.com/Andersbakken/rtags

/john

[-- Attachment #2: Type: text/html, Size: 938 bytes --]

Re: Emacs contributions, C and Lisp

[David Kastrup]

Óscar Fuentes <ofv@wanadoo.es> writes:

> David Kastrup <dak@gnu.org> writes:
>
>> Yes, that's inflammatory and almost libelous.  Because it is a summary
>> of something that can hardly be read in any way that is _not_
>> inflammatory and almost libelous.  Feel free to point out any other
>> valid reading of it, even though it could lead to a continuation of a
>> discussion you want to stop.
>
> Eli took no offense from my response. That should be a strong hint for
> you.

Eli did not reply after you unilaterally declared the discussion closed.
Declaring this to mean that he took no offense is quite a leap.

You did not bother explaining how any of the post I quoted from you
could be interpreted in any other way than I did.  In fact, you
completely cut out again all of that quote from your reply.

I don't see that it makes sense to criticize my interpretation of your
words without viewing what you wrote.

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[Óscar Fuentes]

David Kastrup <dak@gnu.org> writes:

>> Eli took no offense from my response. That should be a strong hint for
>> you.
>
> Eli did not reply after you unilaterally declared the discussion
> closed.

Yes, he did. Look harder.

> Declaring this to mean that he took no offense is quite a leap.
>
> You did not bother explaining

I already took the time to answer your unsolicited intervention in Eli's
defense, but it seems you will not be happy until I acknowledge that
your over-the-top intervention was justified.

No.

[snip]

Re: Emacs contributions, C and Lisp

[David Kastrup]

Óscar Fuentes <ofv@wanadoo.es> writes:

> David Kastrup <dak@gnu.org> writes:
>
>>> Eli took no offense from my response. That should be a strong hint for
>>> you.
>>
>> Eli did not reply after you unilaterally declared the discussion
>> closed.
>
> Yes, he did. Look harder.
>
>> Declaring this to mean that he took no offense is quite a leap.
>>
>> You did not bother explaining
>
> I already took the time to answer your unsolicited intervention in Eli's
> defense, but it seems you will not be happy until I acknowledge that
> your over-the-top intervention was justified.
>
> No.

You make it a habit of inventing your own histories.  I did not
intervene at all in Eli's defense.  Again quoting _in_ _full_ what you
are alluding to:

    Daniel Colascione <dancol@dancol.org> writes:

    > On 02/28/2014 01:13 AM, David Kastrup wrote:
    >> When one declaration changes the meaning and syntax of a program all
    >> over one file (and yes, this sort of thing _can_ happen with C++),
    >> getting things right might require a full-file parse.  When presented
    >> with a preexisting C++ file, being able to get the actual meaning out
    >> by the use of exhaustive tools is nice.  When _writing_ a C++
    >> program, it's preferable to stay away from those edges and thus get
    >> along with more simplistic tools.  Or even none at all.
    >
    > You might believe that --- and you may even be right --- but your
    > personal prescriptions for software development shouldn't affect the
    > feature-set of a generic editor.

    Ultimately, reality will affect the feature set of a generic editor.
    Any feature that requires per-keystroke reparsing of the entire
    compilation unit to work is not feasible in an editing workflow.  That
    kind of thing is ok for code browsing, not for writing.

    At any rate, it was Óscar's claim that it is so utterly absurd to state
    being a regular C++ programmer when one does not rely on code-explaining
    support tools that he basically called Eli a fraud.

    That's a bit stronger than "personal prescriptions for software
    development".

That's not an "intervention in Eli's defense".  It's just pointing out
that I happen to share some of those of Eli's stances on programming
that are portrayed as being so outlandish and/or not worth of
consideration as to render any further discussion moot.

At any rate, if you want to comment on any part of the discussion,
I _strongly_ suggest that you _quote_ what you are talking about instead
of postulating your interpretation.

Since we already established that we have a wide variety of how readers
of this list interpret any given choice of words, quoting what you refer
to avoids the interpretations taking on a life of their own like when
playing "Broken Telephone".

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Fri, 28 Feb 2014 13:14:04 +0100
> 
> even *if* [Eli] is unacquainted with such basic concepts as function
> overloading.

Of course, I am.

> Eli took no offense from my response.

I don't take offense from responses based on misunderstandings.

Re: Emacs contributions, C and Lisp

[Dmitry Gutov]

On 28.02.2014 14:53, John Yates wrote:
> For completeness and actively maintained::
>
> https://github.com/Andersbakken/rtags

Thanks for the link. It's a curious project.

It has functions like "go to definition" (a priority that I can agree 
with), but apparently no functional code completion.

Re: Emacs contributions, C and Lisp

[Eli Zaretskii]

> Date: Fri, 28 Feb 2014 02:10:50 -0800
> From: Daniel Colascione <dancol@dancol.org>
> Cc: emacs-devel@gnu.org
> 
> > Any feature that requires per-keystroke reparsing of the entire
> > compilation unit to work is not feasible in an editing workflow.  That
> > kind of thing is ok for code browsing, not for writing.
> 
> http://pdf.aminer.org/000/542/897/incremental_analysis_of_real_programming_languages.pdf

Not sure what is the point you wanted to make with this (old)
article.  Can you clarify?

Re: Emacs contributions, C and Lisp

[David Engster]

Richard Stallman <rms@gnu.org> writes:
>     Anyway, Stefan gave is OK on libclang usage, 
>
> That statement seems to be a misunderstanding.

I don't think so. You and Stefan simply have differing opinions.

> My decision, as head of the GNU Project, is that we will not install
> anything in Emacs or ELPA that uses clang or LLVM.
>
> You can use CEDET, you can use GCC, you can use both, or you
> can use something else.  But not clang or LLVM.
>
> This decision is necessary for achieving more the goal
> of the GNU Project, which is to give computer users' freedom
> in their computing in general -- not just in their text editing.

This decision is a mistake. Please reconsider.

If I understand you correctly, you are saying we should not support any
feature in clang/LLVM unless we can do the same with gcc, because
although clang/LLVM is free software, it is non-copyleft. At least
that's what I gathered from your other posts in this thread.

But by the same reasoning you are applying to clang/LLVM, Emacs' must
not support any feature from Subversion (which is Apache-licensed)
unless the same feature is present in git, hg, or bzr, or we must
implement it there first. Likewise, we must not support any feature in
CMake (which is a BSD-licensed build tool), unless the same feature is
present in GNU Make, or we must first implement it there as well.

Is that correct? Because if so, you are drawing the line now between
copyleft and non-copyleft, whereas it used to be between free and
non-free.

-David

Re: Emacs contributions, C and Lisp

[David Kastrup]

David Engster <deng@randomsample.de> writes:

> Richard Stallman <rms@gnu.org> writes:
>>     Anyway, Stefan gave is OK on libclang usage, 
>>
>> That statement seems to be a misunderstanding.
>
> I don't think so. You and Stefan simply have differing opinions.

The misunderstanding is that outsiders are free to heed just those of
the "differing opinions" they like.  Emacs is copyrighted by the FSF of
which Richard is president.

It's unfortunate that what amounts to an internal disagreement was
carried to this list.  But there is no question about the consequences
while there is no agreement on record.

>> My decision, as head of the GNU Project, is that we will not install
>> anything in Emacs or ELPA that uses clang or LLVM.
>>
>> You can use CEDET, you can use GCC, you can use both, or you can use
>> something else.  But not clang or LLVM.
>>
>> This decision is necessary for achieving more the goal of the GNU
>> Project, which is to give computer users' freedom in their computing
>> in general -- not just in their text editing.
>
> This decision is a mistake. Please reconsider.

That opinion is redundant.  We have already established that this is not
a voting process or shouting match, so do everyone a favor, read up in
the discussion, and contribute something to it once you have figured out
something that has not already been said and that actually applies to a
policy decision.

> If I understand you correctly, you are saying we should not support
> any feature in clang/LLVM unless we can do the same with gcc, because
> although clang/LLVM is free software, it is non-copyleft. At least
> that's what I gathered from your other posts in this thread.
>
> But by the same reasoning you are applying to clang/LLVM, Emacs' must
> not support any feature from Subversion (which is Apache-licensed)
> unless the same feature is present in git, hg, or bzr, or we must
> implement it there first. Likewise, we must not support any feature in
> CMake (which is a BSD-licensed build tool), unless the same feature is
> present in GNU Make, or we must first implement it there as well.

See, you are trying to apply logic here, something like reductio ad
absurdum.  That's fine for, say, speculating about the legal
consequences of license term when viewed by a court trying to follow the
letter of some written law.

But this does not apply to a policy decision.  The whole point of making
an actual decision is that it happens with _deliberation_, not by
applying a fixed set of rules.

If you want to have Richard change a policy decision (which, by the way,
is a rather ambitious undertaking even outside of a mailing list
discussion), you need to convince him that his evaluation of the
perceived benefits and downsides of a decision is skewed to a degree
where the cost of changing the decision is less than that of staying
with it.

> Is that correct? Because if so, you are drawing the line now between
> copyleft and non-copyleft, whereas it used to be between free and
> non-free.

In this case, the line is between using Emacs' leverage for preserving
GCC's importance over betting the horse on a product with an ultimate
fate out of the GNU project's control, and out of its original authors'
and current community's control.

In a similar vein, it was a political decision to move Emacs from CVS to
Bzr as its dedicated version control system, even though there was a set
of other free contenders to choose from.

For GNU projects in general, the GNU maintainer guidelines provide
guidance about how to make decisions and usually do a pretty good job.
In the case of Emacs and GCC (both are copyright-assigned to the FSF
because of their relative importance to the GNU project), Richard can
and does exercise more direct control over the decisions.

So if you want to convince him otherwise (and this discussion has gone
on for too long to have a good chance of finding him attentive still),
you have to convince him of consequences worse than those he has been
able to fathom.  And Richard is not renowned for his optimism.

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[John Yates]

[-- Attachment #1: Type: text/plain, Size: 320 bytes --]

On Fri, Feb 28, 2014 at 9:24 AM, Dmitry Gutov <dgutov@yandex.ru> wrote:

> It has functions like "go to definition" (a priority that I can agree
> with), but apparently no functional code completion.
>

Are you saying that based purely on the project's home page?  Take a look
at src/rtags.el in the source tree.

/john

[-- Attachment #2: Type: text/html, Size: 673 bytes --]

Re: Emacs contributions, C and Lisp

[Dmitry Gutov]

On 28.02.2014 19:05, John Yates wrote:
> Are you saying that based purely on the project's home page?

Based on the comments here:

https://github.com/Andersbakken/rtags/issues/127

Re: Emacs contributions, C and Lisp

[Juanma Barranquero]

On Fri, Feb 28, 2014 at 5:57 PM, David Kastrup <dak@gnu.org> wrote:

> In this case, the line is between using Emacs' leverage for preserving
> GCC's importance over betting the horse on a product with an ultimate
> fate out of the GNU project's control, and out of its original authors'
> and current community's control.

It would be interesting to measure (somehow ;-) whether that "Emacs
leverage" is enough to preserve GCC's importance. I mean, if using the
clang libraries allows much powerful, developer-friendly environments
for C/C++ programmers, do we think that a significant number of them
will still choose GCC because of Emacs? If the answer is "yes",
Richard's position strenghtens GCC without prejudice for Emacs. If the
answer is "not", that position hurts Emacs without helping GCC.

Re: Emacs contributions, C and Lisp

[David Kastrup]

Juanma Barranquero <lekktu@gmail.com> writes:

> On Fri, Feb 28, 2014 at 5:57 PM, David Kastrup <dak@gnu.org> wrote:
>
>> In this case, the line is between using Emacs' leverage for preserving
>> GCC's importance over betting the horse on a product with an ultimate
>> fate out of the GNU project's control, and out of its original authors'
>> and current community's control.
>
> It would be interesting to measure (somehow ;-) whether that "Emacs
> leverage" is enough to preserve GCC's importance.

Shrug.  GNU Emacs is the application for which the GPL was written and
which started the GNU project.  Sometimes you just have to do your best
and be prepared for success as well as for failure.

> I mean, if using the clang libraries allows much powerful,
> developer-friendly environments for C/C++ programmers, do we think
> that a significant number of them will still choose GCC because of
> Emacs?

That's not the interesting question.  Do we think that a relevant number
of people will choose to work on GCC until it works for that purpose in
stock Emacs rather than just install another compiler?

The history of several Windows-only features that were only admitted
into Emacs proper once they were supported reasonably well under other
platforms would suggest that the answer to this kind of question may
very well be "yes".

> If the answer is "yes", Richard's position strenghtens GCC without
> prejudice for Emacs. If the answer is "not", that position hurts Emacs
> without helping GCC.

You are trying to view Emacs and GCC in isolation.  Both are core parts
of the GNU system and we don't want the GNU system to become one where
working with one GNU program would be a reason to prefer using a non-GNU
program over another GNU program.

Emacs and GCC are the oldest central components and cornerstones of the
GNU project.  If any two applications have reason to stick up for each
other, it are those.

We don't want Emacs to become more useful to the detriment of GCC.  Of
course, this is the Emacs developer list so it is to be expected that
some list members are less than enthused about the principle underlying
this kind of decision.  But not taking the underlying principle into
account when dissenting means that the dissent is only relevant to a
part of the decisionmaking, and the decisionmaking is exactly about
finding a _balance_.

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[Juanma Barranquero]

On Fri, Feb 28, 2014 at 6:53 PM, David Kastrup <dak@gnu.org> wrote:

> GNU Emacs is the application for which the GPL was written and
> which started the GNU project.  Sometimes you just have to do your best
> and be prepared for success as well as for failure.

Fair enough.

> That's not the interesting question.

Well, I think is *an* interesting question, though whether is *the*
interesting question is debatable. Reasons below.

> Do we think that a relevant number
> of people will choose to work on GCC until it works for that purpose in
> stock Emacs rather than just install another compiler?

...Talking about leveraging Emacs to preserve GCC's importance is not
just about the programmers that could/would work on GCC (assuming that
by "work on GCC" you mean "working on extending GCC"; if not, my
mistake). Most C/C++ programmers that use Emacs and GCC have neither
the interest, nor the knowledge, ability or disposition to work on
GCC. If Emacs/GCC is insufficient for them and they switch to a
non-free, or less free, toolset, this is a net loss for the GNU
project, I think.

> The history of several Windows-only features that were only admitted
> into Emacs proper once they were supported reasonably well under other
> platforms would suggest that the answer to this kind of question may
> very well be "yes".

Which features? (Not doubting you, just curious.)

> You are trying to view Emacs and GCC in isolation.

I'm not *trying* to view them in isolation, though perhaps I'm doing
it out of naïvete.

> Both are core parts
> of the GNU system and we don't want the GNU system to become one where
> working with one GNU program would be a reason to prefer using a non-GNU
> program over another GNU program.

Agreed. But at some point, all these kind of reasons interact in
extremely non-obvious ways. If Emacs were to work better with clang,
would that be a reason for people to prefer clang over GCC, or a
reason for people to want to extend GCC to match clang's capabilities?
I.e., I question the strict cause-effect relationship, not because I
think is wrong, but because I don't think anyone can really say
whether it is right.

> Emacs and GCC are the oldest central components and cornerstones of the
> GNU project.  If any two applications have reason to stick up for each
> other, it are those.

Yes, if that means that both evolve to better match the needs of their
users. Not so clear, if that means both stagnate, because, to be
relevant for the user's freedoms, they have to be used.

> We don't want Emacs to become more useful to the detriment of GCC.  Of
> course, this is the Emacs developer list so it is to be expected that
> some list members are less than enthused about the principle underlying
> this kind of decision.  But not taking the underlying principle into
> account when dissenting means that the dissent is only relevant to a
> part of the decisionmaking, and the decisionmaking is exactly about
> finding a _balance_.

FWIW, I'm not arguing against your position. I just happen to think
that, in this matter, decision making is hardly fact-based, because
facts are scarce. It's much more about how one side and the other see
the consequences of these decisions. I happen to be in the less
pessimist side, but that's just a gut feeling, entirely fact-free.

Re: Emacs contributions, C and Lisp

[David Engster]

David Kastrup writes:
> That opinion is redundant.  We have already established that this is not
> a voting process or shouting match, so do everyone a favor, read up in
> the discussion, and contribute something to it once you have figured out
> something that has not already been said and that actually applies to a
> policy decision.

Fair enough. I will no longer pollute this thread with my redundancy as
to not divert attention from your lucid postings. Maybe we can set up
some kind of moderation queue, so that you can review my mails before
they reach the list? That would surely be most helpful.

L a f p RET'ly yours,
David

Re: Emacs contributions, C and Lisp

[David Kastrup]

Juanma Barranquero <lekktu@gmail.com> writes:

> On Fri, Feb 28, 2014 at 6:53 PM, David Kastrup <dak@gnu.org> wrote:
>
>> We don't want Emacs to become more useful to the detriment of GCC.
>> Of course, this is the Emacs developer list so it is to be expected
>> that some list members are less than enthused about the principle
>> underlying this kind of decision.  But not taking the underlying
>> principle into account when dissenting means that the dissent is only
>> relevant to a part of the decisionmaking, and the decisionmaking is
>> exactly about finding a _balance_.
>
> FWIW, I'm not arguing against your position. I just happen to think
> that, in this matter, decision making is hardly fact-based, because
> facts are scarce. It's much more about how one side and the other see
> the consequences of these decisions. I happen to be in the less
> pessimist side, but that's just a gut feeling, entirely fact-free.

I am not necessarily describing my own position here.  I'm feeling
somewhat ambiguous about the best course forward, but I agree with the
position of Richard in as far that I would consider it a shame if Emacs
had to rely on LLVM to provide a useful environment for writing code to
be compiled with GCC.  Even taking the question of policy aside, one
wants things like completion to work properly with all GNU extensions to
the various languages GCC supports.  GNU extensions are basically dead
in the water if Emacs refuses dealing with them because its modes are
based on working with LLVM.

Richard's position as far as I understand is not "we don't want
compiler-supported completion in Emacs" but rather "we don't want
completion in Emacs that is only supported by using LLVM".  Whether or
not one wants to consider relying on LLVM bad politically, it means that
Emacs will be constrained to support only languages and language
dialects at the choice of LLVM developers.

At any rate, given the nature of this decision, any attempt to change it
would have to be based on bringing previously unconsidered facts to the
attention of Richard.  "I come to a different conclusion based on the
same information" is not going to help.

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[Juanma Barranquero]

On Fri, Feb 28, 2014 at 9:53 PM, David Kastrup <dak@gnu.org> wrote:

> I am not necessarily describing my own position here.

OK, you're right. Sorry if I misrepresented your position.

> I agree with the
> position of Richard in as far that I would consider it a shame if Emacs
> had to rely on LLVM to provide a useful environment for writing code to
> be compiled with GCC.

But this is not a binary, now-or-never issue. Even if, in some cases,
Emacs had to rely on LLVM, that would change as soon as GCC provided
equivalent support.

> Richard's position as far as I understand is not "we don't want
> compiler-supported completion in Emacs" but rather "we don't want
> completion in Emacs that is only supported by using LLVM".

That can be fixed (for some values of "fixed") by adding compiler
support for completion to GCC. The fact that LLVM has it already, and
that some Emacs modes could be written to support it, does not
preclude extending GCC or creating packages to interact with it (or
extending the LLVM-related ones to support both).

> Whether or
> not one wants to consider relying on LLVM bad politically, it means that
> Emacs will be constrained to support only languages and language
> dialects at the choice of LLVM developers.

I don't see it, really. That's a bit like saying that, because VC was
written to support RCS and CVS, it will forever be constrained by the
design decissions of CVS developers. And yet it has been extended to
grok Subversion, arch, Bazaar, git and Mercurial. I fail to see how "a
mode supports LLVM" turns into "depends on LLVM and nothing more,
forever". The only thing that would preclude these new modes to also
be able to use GCC is if GCC is never extended to provide that
facilities, and in that case, we are in deep trouble, because C++
developers will eventually turn to the tools that make their lives
easier.

> At any rate, given the nature of this decision, any attempt to change it
> would have to be based on bringing previously unconsidered facts to the
> attention of Richard.  "I come to a different conclusion based on the
> same information" is not going to help.

Yes, sure. I'm not trying to change anyone's mind, just trying to
understand the differente issues.

   J

Re: Emacs contributions, C and Lisp

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 992 bytes --]

On 02/28/2014 07:08 AM, Eli Zaretskii wrote:
>> Date: Fri, 28 Feb 2014 02:10:50 -0800
>> From: Daniel Colascione <dancol@dancol.org>
>> Cc: emacs-devel@gnu.org
>>
>>> Any feature that requires per-keystroke reparsing of the entire
>>> compilation unit to work is not feasible in an editing workflow.  That
>>> kind of thing is ok for code browsing, not for writing.
>>
>> http://pdf.aminer.org/000/542/897/incremental_analysis_of_real_programming_languages.pdf
> 
> Not sure what is the point you wanted to make with this (old)
> article.  Can you clarify?

I'm trying to debunk the argument that the features we're talking about
require per-keystroke reparsing of the entire compilation unit. There's
a pretty decent literature in incremental parsing algorithms that might
be efficient enough to make online updating of syntax trees viable.

There's also an existence proof called "Visual Studio" that shows that
it's possible to efficiently implement these features.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 901 bytes --]

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    I'd object, for basically practical reasons.  You can fork code, but you
    cannot fork a community.  A fork of the LLVM codebase under the GPLv3
    makes only sense if you actually add nontrivial nonseparable components
    under the GPLv3 or the code base can be just swapped out.  If you have
    nontrivial nonseparable components, but an actively developed important
    upstream, you need to constantly reintegrate the upstream work in order
    to keep the edge.

That is true.  Legally, we would be allowed to make a copylefted
branch of LLVM.  In practice, it is not a substantive option.  Unless
we were to add important facilities, we would only be pretending to
have changed the situation.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    > or ELPA that uses clang or LLVM.
      ^^^^^^^

    This ship has sailed.

Which packages in ELPA have code to work specifically with Clang or LLVM?

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    > You can use CEDET, you can use GCC, you can use both, or you
    > can use something else.  But not clang or LLVM.

    I think this conflicts with your previous statement that using Clang
    for some features would be fine as long as Emacs supports using GCC for
    the same features.

Thanks for calling my attention to this wrinkle of the issue.

Features that only want to do C (or C++) compilation may as well be
implemented to work with any and all compilers.

For features that interact with the compiler beyond just using it to
compile a program, and therefore can't easily be implemented to work
with any and all compilers, there are two sensible approaches:

* Support only GCC.

* Support GCC first; once GCC is supported, support for other compilers
  is acceptable too.

Rather than making a general decision a priori, I'd rather look at
these cases individually when they arise.  After deciding a few such
cases, I hope I will see a pattern that would suggest a general policy.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    That can be fixed (for some values of "fixed") by adding compiler
    support for completion to GCC. The fact that LLVM has it already, and
    that some Emacs modes could be written to support it, does not
    preclude extending GCC or creating packages to interact with it (or
    extending the LLVM-related ones to support both).

The point is to give people a strong motivation to implement the
necessary support with GCC.  What you have in mind would give people
no motivation to do that, so it would not help GCC.  It is more
of an excuse for giving up than a resolve to resist.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[Dmitry Gutov]

On 01.03.2014 05:37, Richard Stallman wrote:
> Which packages in ELPA have code to work specifically with Clang or LLVM?

The one I'm maintaining, and which has been mentioned a few times in 
this and other threads: http://elpa.gnu.org/packages/company.html

Specifically, it includes a backend `company-clang', see the eponymous 
file in the archive.

Re: Emacs contributions, C and Lisp

[David Kastrup]

Juanma Barranquero <lekktu@gmail.com> writes:

> On Fri, Feb 28, 2014 at 9:53 PM, David Kastrup <dak@gnu.org> wrote:
>
>> I am not necessarily describing my own position here.
>
> OK, you're right. Sorry if I misrepresented your position.
>
>> I agree with the
>> position of Richard in as far that I would consider it a shame if Emacs
>> had to rely on LLVM to provide a useful environment for writing code to
>> be compiled with GCC.
>
> But this is not a binary, now-or-never issue. Even if, in some cases,
> Emacs had to rely on LLVM, that would change as soon as GCC provided
> equivalent support.

Well, a frequently cited motivation of Free Software authors is that
they are "scratching an itch".  It helps if there is an actual itch.
Now this thread shows ample demonstration that Emacs using LLVM for
completion is not an itch to a lot of people, whereas not having
compiler-based completion at all in Emacs is.  Coupling those itches
seems to make it more likely somebody will work on the former.

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[Eli Zaretskii]

> From: Juanma Barranquero <lekktu@gmail.com>
> Date: Fri, 28 Feb 2014 18:31:53 +0100
> Cc: Emacs developers <emacs-devel@gnu.org>
> 
> It would be interesting to measure (somehow ;-) whether that "Emacs
> leverage" is enough to preserve GCC's importance. I mean, if using the
> clang libraries allows much powerful, developer-friendly environments
> for C/C++ programmers, do we think that a significant number of them
> will still choose GCC because of Emacs? If the answer is "yes",
> Richard's position strenghtens GCC without prejudice for Emacs. If the
> answer is "not", that position hurts Emacs without helping GCC.

I actually don't understand why are we talking about compilers as the
only viable infrastructure for the features being discussed.  IMNSHO,
it is bad design to ask users to install a particular compiler, be it
GCC or clang, just to be able to have decent editing capabilities for
a program source.  What next? shall we require LibreOffice to be able
to edit text files conveniently?

We already have elaborate infrastructure for this kind of features:
CEDET and in particular Semantic.  Moreover, we included CEDET in
Emacs several releases ago precisely _because_ we wanted to move in
the direction of making Emacs a better IDE.  If we need more of what
is in ECB and the upstream CEDET, we can bring that in.  This is IMO
the preferred direction: make Emacs features depend on nothing but
Emacs packages.  Then all this prolonged discussion about GCC vs clang
is a moot point, and we also gain independence of other projects'
priorities and policies as a nice bonus.

If we need policy decisions in this matter, _that_ is the only one
that should be made.  To me, it is almost a no-brainer; I wonder why
it is not clear-cut to others here.

Re: Emacs contributions, C and Lisp

[Eli Zaretskii]

> From: Juanma Barranquero <lekktu@gmail.com>
> Date: Fri, 28 Feb 2014 19:23:03 +0100
> Cc: Emacs developers <emacs-devel@gnu.org>
> 
> ...Talking about leveraging Emacs to preserve GCC's importance is not
> just about the programmers that could/would work on GCC (assuming that
> by "work on GCC" you mean "working on extending GCC"; if not, my
> mistake). Most C/C++ programmers that use Emacs and GCC have neither
> the interest, nor the knowledge, ability or disposition to work on
> GCC. If Emacs/GCC is insufficient for them and they switch to a
> non-free, or less free, toolset, this is a net loss for the GNU
> project, I think.

Another huge advantage of my proposal to use CEDET is that CEDET and
Semantic are Lisp packages, so we should need neither the knowledge
nor the proficiency of GCC hackers to push in the direction we want.
The largest pool of Lisp coding talents are right here.

Re: Emacs contributions, C and Lisp

[Eli Zaretskii]

> Date: Fri, 28 Feb 2014 13:51:21 -0800
> From: Daniel Colascione <dancol@dancol.org>
> CC: dak@gnu.org, emacs-devel@gnu.org
> 
> >> http://pdf.aminer.org/000/542/897/incremental_analysis_of_real_programming_languages.pdf
> > 
> > Not sure what is the point you wanted to make with this (old)
> > article.  Can you clarify?
> 
> I'm trying to debunk the argument that the features we're talking about
> require per-keystroke reparsing of the entire compilation unit. There's
> a pretty decent literature in incremental parsing algorithms that might
> be efficient enough to make online updating of syntax trees viable.

Are there any implementations of these R&D papers in popular compilers
(apart of Visual Studio, which is obviously not interesting in this
context)?

Isn't it true that all the packages that use clang for completion save
the buffer whenever completion is required?  At least the one I tried
did just that, which sounds an awful design decision to me.  Since
when do we let a feature decide for the user when to save the buffer
she is editing?

Re: Emacs contributions, C and Lisp

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Sat, 01 Mar 2014 07:26:02 +0100
> 
> Now this thread shows ample demonstration that Emacs using LLVM for
> completion is not an itch to a lot of people, whereas not having
> compiler-based completion at all in Emacs is.

Perhaps because in C such completion is trivially available, while C++
is a dying language.

(On my daytime job, customers started frowning on projects implemented
in C++ several years ago.  They want Java, C#, Python, and Ruby
instead.  Newbie programmers we hire don't know C++ at all, and don't
want to learn it, because they say it is not required on the market.)

Re: Emacs contributions, C and Lisp

[Paul Eggert]

Eli Zaretskii wrote:
> On my daytime job, customers started frowning on projects implemented in C++ several years ago.

Reliable statistics in this area are hard to come by, but for what it's 
worth the TIOBE index for February 2014 says:

  18.334% C
  17.316% Java
  11.341% Objective C
   6.892% C++
   6.450% C#
   4.219% PHP
   2.759% Visual Basic
   2.157% Python
   1.929% JavaScript
   1.798% Visual Basic .NET
   ...
   0.236% Emacs Lisp
   ...

A decade ago, C++'s popularity was in the 15-20% range.  The biggest 
up-and-comer on the above list is Objective C.

http://www.tiobe.com/index.php/content/paperinfo/tpci/index.html

Re: Emacs contributions, C and Lisp

[David Engster]

Eli Zaretskii writes:
> Isn't it true that all the packages that use clang for completion save
> the buffer whenever completion is required?  At least the one I tried
> did just that, which sounds an awful design decision to me.

Weirdly enough, while clang compiles from stdin without problems, it
needs a file to perform completions (last I checked). So you either save
the current file, which I agree is a no-no, or you save the current
buffer into a temporary file (which is what semantic-clang does, which
only exists in CEDET upstream).

-David

Re: Emacs contributions, C and Lisp

[Eli Zaretskii]

> Date: Sat, 01 Mar 2014 00:37:51 -0800
> From: Paul Eggert <eggert@cs.ucla.edu>
> 
> Eli Zaretskii wrote:
> > On my daytime job, customers started frowning on projects implemented in C++ several years ago.
> 
> Reliable statistics in this area are hard to come by, but for what it's 
> worth the TIOBE index for February 2014 says:
> 
>   18.334% C
>   17.316% Java
>   11.341% Objective C
>    6.892% C++
>    6.450% C#
>    4.219% PHP
>    2.759% Visual Basic
>    2.157% Python
>    1.929% JavaScript
>    1.798% Visual Basic .NET
>    ...
>    0.236% Emacs Lisp
>    ...
> 
> A decade ago, C++'s popularity was in the 15-20% range.  The biggest 
> up-and-comer on the above list is Objective C.

Thanks.

This basically tells the same story as what I had in my mind.  Of
course, the statistics also varies according to the application's
domain and the target OS (in my case, it's either Windows or
GNU/Linux, so Objective C is never required).

Re: Emacs contributions, C and Lisp

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Juanma Barranquero <lekktu@gmail.com>
>> Date: Fri, 28 Feb 2014 18:31:53 +0100
>> Cc: Emacs developers <emacs-devel@gnu.org>
>> 
>> It would be interesting to measure (somehow ;-) whether that "Emacs
>> leverage" is enough to preserve GCC's importance. I mean, if using
>> the clang libraries allows much powerful, developer-friendly
>> environments for C/C++ programmers, do we think that a significant
>> number of them will still choose GCC because of Emacs? If the answer
>> is "yes", Richard's position strenghtens GCC without prejudice for
>> Emacs. If the answer is "not", that position hurts Emacs without
>> helping GCC.
>
> I actually don't understand why are we talking about compilers as the
> only viable infrastructure for the features being discussed.

It's basically the UNIX philosophy to employ dedicated tools for the job
they have been written for.  Parsing is a nontrivial task, so using an
existing parser makes sense.  Particularly when we are talking about
languages with complex namespace and type rules and overloading and
ambiguous grammar and three aspects of compiler technology
(preprocessing, normal execution, template instantiation) working
largely independently with partly surprising consequences.

Now we only want to have parsing and no code generation.  That's a kind
of task that GCC does not currently offer separately, and part of the
reason for it is that others might find it useful for creating non-free
applications around such functionality.

It's my opinion that this is a cost we hardly can avoid paying without
also creating serious road blocks for the creation of free tools.  In
the context of possibilities to refactor GCC to make parts of it
separately useful and/or allow other ways of plugging things together,
Richard tends to be conservative, and indeed it's a cat that would be
pretty hard to stuff back in the bag if one eventually finds the
consequences different than what one hoped for.

In the current situation, the GPLv3 with its patent protection clauses
provides enough of a disincentive in my book that one would have to be
overly afraid of creating proprietary spinoffs in the compiler
technology area.  However, there _are_ hopeful signs that the current
patent madness will eventually be fixed by legislational means, and in
that case the patent clauses of the GPLv3 would be quite less of an
influence on a big proprietary business' bottomline.

But should patents lose much of their sting, it would also mean a hit in
the general attractiveness of LLVM over GCC: the GPLv3 is more or less
cited as the main reason several proprietary vendors stopped working
with GCC.  So _iff_ the poison pill of the GPLv3 would become inert, we
have a situation where we'd want to be able to play to our strengths for
making proprietary vendors (and others) pick the GCC and thus provide
their users with some degree of freedom.

And that will basically happen on technical merit.

> We already have elaborate infrastructure for this kind of features:
> CEDET and in particular Semantic.  Moreover, we included CEDET in
> Emacs several releases ago precisely _because_ we wanted to move in
> the direction of making Emacs a better IDE.  If we need more of what
> is in ECB and the upstream CEDET, we can bring that in.  This is IMO
> the preferred direction: make Emacs features depend on nothing but
> Emacs packages.  Then all this prolonged discussion about GCC vs clang
> is a moot point, and we also gain independence of other projects'
> priorities and policies as a nice bonus.

Well, if we get to stage where we can say "get a language supported by
GCC, and Emacs will provide good highlighting, semantic completion, nice
formatting", then that would be a very strong selling point for GCC as
well as Emacs.  In particular if the output enabling those kinds of
feature can be used by other IDEs than Emacs.

And that's exactly because it would avoid duplication of work.  However,
it would also require incremental parsing to be really good.  Could an
incremental and/or differential parser (reparse just part of a file and
update all info about the whole compilation unit) be fit in the GCC
framework?  An interesting starting point would be, of course, trying to
make Bison skeletons that work for that purpose.  Unfortunately, the
highest-profile languages by GCC don't use Bison IIRC.

An any rate, it is a much more realistic goal to get 60% of all computer
users to use GCC than it is to get 60% of all computer users to use
Emacs.  And compilers are really important.  We should really think
about how we can leverage GCC to make, say, manufacturers of proprietary
hardware open up their hardware in order to use GCC rather than just go
somewhere else.  And that's a somewhat pressing problem.

> If we need policy decisions in this matter, _that_ is the only one
> that should be made.  To me, it is almost a no-brainer; I wonder why
> it is not clear-cut to others here.

The main problem I see here is that our policy decisions regarding Emacs
are, to a large degree, trickle-down decisions because of the role GCC
can or should play for computing.

In the concrete situation we have here, basically company-mode offering
semantic assistance based on Clang/LLVM, the solution I'd have preferred
to see is, "Oh, that's a shortcoming, but easily fixed.  Let's just hack
up something that works with current GCC, and then people have a
choice."

The current situation with GCC is such that "let's just hack up
something that works with current GCC" seems to be out of reach.  There
are two ways to address that: one is hacking something up in GCC that
works for that particular purpose.

We have the statement here (and far too little attention has been paid
to it) that -fdump-xref and associated programs of the GNAT project
would do the trick for the editing task currently at issue.  So there is
a prototype people could be testing and experimenting with instead of
doing flame wars.

It means using a specific version of GCC, but it would be easy enough to
turn that into a default version if we consider that a priority.  And I
think we should.  Really, really, really.

That could be a feasible approach to address the "batch data collection
for semantic completion" angle.

Now the Ada thing is a stroke of luck in its timing, and we should not
waste it.

But that still gives the question: "how about the longterm viability to
use GCC and its parsers to hack up something fast that does not require
changes to GCC"?

Because this would avoid all the discussion and friction and give people
the freedom to hack up their own solutions centered around GCC.
Including solutions we don't want to see.  I think that in this
particular case, the cost of tight control beyond that offered by the
GPLv3 (which will likely change along with the patent landscape) for the
underlying code base) is likely to be a cause for repeated roadblocks of
the kind we are seeing here with Emacs.

This particular Emacs flame war has been less than pretty.  But it is
more or less a downstream flamewar.  The current code and also partly
policy situation with GCC are what are at issue here.

While this particular situation might be curable with -fdump-xref
eventually, the respective decision-making in the context of Emacs
appears heavy-handed, and there are few other applications where a heavy
hand could even reach.

So I think that a long-term tenable solution of the problem will require
work and/or policy changes on GCC that would have made this a non-issue
or moderate-effort issue for Emacs.

To get back to what I replied to:

> This is IMO
> the preferred direction: make Emacs features depend on nothing but
> Emacs packages. [...]
> If we need policy decisions in this matter, _that_ is the only one
> that should be made.  To me, it is almost a no-brainer; I wonder why
> it is not clear-cut to others here.

Well, that is not really much in need of a policy since Emacs can
support multiple feature sets.  I would prefer if we needed no policy
decisions at all in this matter.  We don't get that, obviously.  But my
next preference is to have GCC move/develop in a manner where we would
not have needed a policy decision.  Because Emacs is, all in all, not a
particularly effective venue for making policy decisions matter in the
great play.  If we can get to a stage where our preferred action would
also be in accord with running policies, that means that the preferred
action in non-policed applications would be the same.

In other words: things are going wrong with GCC policies upstream
regarding reusability in Free Software applications that Emacs, as one
compiler user, is too small too fix.

If this is not approached in a more general manner in GCC, we'll have
this problem reoccuring in various iterations on the Emacs list, and it
will be reoccuring for the maintainers other Free Software which will
then choose to pick non-copyleft compilers entirely instead of playing
mix-and-match games or writing their own parsers.

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[Thien-Thi Nguyen]

[-- Attachment #1: Type: text/plain, Size: 470 bytes --]

() Eli Zaretskii <eliz@gnu.org>
() Sat, 01 Mar 2014 10:15:51 +0200

   Perhaps because in C such completion is trivially available,
   while C++ is a dying language.

Yeah, they tell me C++ programmers buy a lot of hair colorant.  :-D

-- 
Thien-Thi Nguyen
   GPG key: 4C807502
   (if you're human and you know it)
      read my lisp: (responsep (questions 'technical)
                               (not (via 'mailing-list)))
                     => nil

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: Emacs contributions, C and Lisp

[David Kastrup]

Thien-Thi Nguyen <ttn@gnu.org> writes:

> () Eli Zaretskii <eliz@gnu.org>
> () Sat, 01 Mar 2014 10:15:51 +0200
>
>    Perhaps because in C such completion is trivially available,
>    while C++ is a dying language.
>
> Yeah, they tell me C++ programmers buy a lot of hair colorant.  :-D

I shudder to think what for.  I'd have expected them to have torn out
all of their hair long ago.

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[Óscar Fuentes]

David Kastrup <dak@gnu.org> writes:

[snip]

>> We already have elaborate infrastructure for this kind of features:
>> CEDET and in particular Semantic.  Moreover, we included CEDET in
>> Emacs several releases ago precisely _because_ we wanted to move in
>> the direction of making Emacs a better IDE.  If we need more of what
>> is in ECB and the upstream CEDET, we can bring that in.  This is IMO
>> the preferred direction: make Emacs features depend on nothing but
>> Emacs packages.

I know that you don't trust me when I say that this is foolish, so I'll
recommend to you to ask on gcc-devel: "how much work it is to implement
the necessary features from scratch (say on Elisp) for providing C++
smart code completion as Clang does?"

You can also take a look at GCC or Clang sources to get an idea of the
involved complexity (not that the sources reflects all the complexity.)

[snip]

> We have the statement here (and far too little attention has been paid
> to it) that -fdump-xref and associated programs of the GNAT project
> would do the trick for the editing task currently at issue.  So there is
> a prototype people could be testing and experimenting with instead of
> doing flame wars.

You can see -fdump-xref as adequate only if you look at it with the
Etags mindset. -fdump-xref does not solve this most common pattern where
you type this two lines in sequence:

auto x = <some-expression>;
x. <- request code completion here

You need to figure out the type of `x' for providing the completion
candidates on the second line. For that, you'll need to start by
rebuilding and reading the output of -fdump-xref for the current compile
unit, which is very resource-intensive. Also, -fdump-xref would need to
work well enough on malformed source code.

[snip]

Re: Emacs contributions, C and Lisp

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Sat, 01 Mar 2014 12:04:04 +0100
> 
> > I actually don't understand why are we talking about compilers as the
> > only viable infrastructure for the features being discussed.
> 
> It's basically the UNIX philosophy to employ dedicated tools for the job
> they have been written for.

I'm not at all sure this philosophy is applicable to the issue at
hand, or to complex interactive environments the kind of Emacs in
general.  We are a long way from the classic batch-mode Unix pipeline,
where (AFAIK) this philosophy originated.

This is a topic worthy of a separate non-trivial discussion, but just
to hint on the potential problems, you may wish to reflect on Emacs
features that do try to use this approach: Dired, GUD, and spellers.
My take from that is that the success is limited at best, and even
then only when those "dedicated tools" go out of their way to support
Emacs, i.e. became "dedicated" to Emacs.  That's already a far cry
from the Unix philosophy mentioned above, which assumes tools
dedicated to their respective tasks, not to their consumers, certainly
not to one particular consumer.

But even if one accepts the philosophy, all we are talking about is
whether to prefer one parser to another.  That one of them is written
in Emacs Lisp is an implementation detail; it is still a "dedicated
tool", just like GCC.  (GCC can, and usually does, do much more than
just parsing and semantic analysis, but that is not important for the
issue at hand.)

OTOH, using an ELisp-based package brings a tremendous advantage,
starting with the fact that such a parser has no problems accessing
Emacs buffers, something external programs can only dream about.  This
alone tells me that CEDET based solutions could be integrated with
interactive editing much more easily than any other solution, and in
particular, several problems simply don't exist with CEDET.

> Well, if we get to stage where we can say "get a language supported by
> GCC, and Emacs will provide good highlighting, semantic completion, nice
> formatting", then that would be a very strong selling point for GCC as
> well as Emacs.

It cannot be Emacs's task to provide selling points for GCC, exactly
as GCC doesn't provide selling points for Emacs.  If GCC acquires
these features, it will make sense to add support for them in Emacs;
but as long as it doesn't, no one can or should ask Emacs to sell GCC
by refraining from using a similar tool that already exists in Emacs.
No project can be requested to be that altruistic.

> And that's exactly because it would avoid duplication of work.  However,
> it would also require incremental parsing to be really good.  Could an
> incremental and/or differential parser (reparse just part of a file and
> update all info about the whole compilation unit) be fit in the GCC
> framework?  An interesting starting point would be, of course, trying to
> make Bison skeletons that work for that purpose.  Unfortunately, the
> highest-profile languages by GCC don't use Bison IIRC.

It is IMO impractical to expect that this particular development will
be advanced by Emacs.  Emacs is the main interested party in this
development, but our knowhow is nowhere near what is needed for any
serious hacking of GCC internals, especially on the level of GIMPLE
representation.

By contrast, improving infrastructure for this written in Emacs Lisp
is a much more practical solution, especially since it already exists
and does its job.  It is IMO silly not to exploit the potential we
have there.

> An any rate, it is a much more realistic goal to get 60% of all computer
> users to use GCC than it is to get 60% of all computer users to use
> Emacs.

That's not the correct comparison, IMO.  We are talking about a user
who already has Emacs and uses it, and who will be using it even if
those advanced features will not be available.  When that user doesn't
have, or cannot use GCC, asking her to install GCC is not practical at
best.

> We have the statement here (and far too little attention has been paid
> to it) that -fdump-xref and associated programs of the GNAT project
> would do the trick for the editing task currently at issue.  So there is
> a prototype people could be testing and experimenting with instead of
> doing flame wars.

I agree that prototyping and comparative testing would be a much
better use of time and efforts invested in this discussion (and many
others similar to it).  But the question that bothers me is what would
be the next step?  It is almost certain that trying to use that option
will discover shortcomings -- who will fix them when they are found?
Unless GCC as a project supports that option and is willing to
maintain and improve it, we are again at a dead end.

> > This is IMO
> > the preferred direction: make Emacs features depend on nothing but
> > Emacs packages. [...]
> > If we need policy decisions in this matter, _that_ is the only one
> > that should be made.  To me, it is almost a no-brainer; I wonder why
> > it is not clear-cut to others here.
> 
> Well, that is not really much in need of a policy since Emacs can
> support multiple feature sets.

Emacs can only support features that exist for which the corresponding
front ends were written in Emacs Lisp.  I don't quite see those
queuing up for providing the features we are discussing.  So there are
no multiple feature sets to support, as yet.

When several potential ways exist for implementing a feature, it is
the task of the project leadership to define the preferences, and then
encourage volunteers to go in those directions that are preferred.
These are the policy decisions I alluded to.  They should remove the
confusion over what we want as a project, and let volunteers work in
those directions without risking that their work will not be used.

> I would prefer if we needed no policy decisions at all in this
> matter.  We don't get that, obviously.  But my next preference is to
> have GCC move/develop in a manner where we would not have needed a
> policy decision.

I think it is impractical to base our development on something
extremely non-trivial that another project needs to do.  Read the GCC
mailing lists, and you will see that the current focus of the project
is very different.  Why should we build our plans on the slim hope
that this focus will change?  It makes very little sense to me.

OTOH, we have a potentially viable solution at our fingertips, and we
have it _now_.  Why trade that for some unsubstantiated hope?

> In other words: things are going wrong with GCC policies upstream
> regarding reusability in Free Software applications that Emacs, as one
> compiler user, is too small too fix.

And we shouldn't.  If GCC acquires these features, we should support
them then, but no earlier.

> If this is not approached in a more general manner in GCC, we'll have
> this problem reoccuring in various iterations on the Emacs list, and it
> will be reoccuring for the maintainers other Free Software which will
> then choose to pick non-copyleft compilers entirely instead of playing
> mix-and-match games or writing their own parsers.

This is a GCC problem, and GCC as a project should solve it.  It could
also be serious enough to become a general problem for the GNU
Project, in which case the FSF should work together with the GCC
project on resolving it.

Emacs cannot do anything constructive to speed up that resolution.
The only constructive way open to us for now is to use CEDET; let's
not wait and do it sooner rather than later.  If we do that now, then
by the time GCC developers get their act together, we will either have
an alternative solution against which to measure theirs, or at least
the necessary front-end for interfacing with them, and enough knowhow
to present them with suitable requirements and to integrate their
solution into Emacs.  In both cases, we win.

The alternative is to effectively do nothing, or, worse, continue
these flame wars.

Re: Emacs contributions, C and Lisp

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Kastrup <dak@gnu.org>
>> Date: Sat, 01 Mar 2014 12:04:04 +0100
>> 
>> > I actually don't understand why are we talking about compilers as the
>> > only viable infrastructure for the features being discussed.
>> 
>> It's basically the UNIX philosophy to employ dedicated tools for the job
>> they have been written for.
>
> I'm not at all sure this philosophy is applicable to the issue at
> hand, or to complex interactive environments the kind of Emacs in
> general.  We are a long way from the classic batch-mode Unix pipeline,
> where (AFAIK) this philosophy originated.

[...]

Well, I tried summarizing my view of the situation, and you summarized
yours.  We arrive at different conclusions from what appears like the
same information.  But I suppose we have a sufficient amount of
summaries to have people wanting to take some action see what may and
may not work to take them where.

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Sat, 01 Mar 2014 14:21:40 +0100
> 
> >> We already have elaborate infrastructure for this kind of features:
> >> CEDET and in particular Semantic.  Moreover, we included CEDET in
> >> Emacs several releases ago precisely _because_ we wanted to move in
> >> the direction of making Emacs a better IDE.  If we need more of what
> >> is in ECB and the upstream CEDET, we can bring that in.  This is IMO
> >> the preferred direction: make Emacs features depend on nothing but
> >> Emacs packages.
> 
> I know that you don't trust me when I say that this is foolish

It is foolish to call an existing implementation impossible to
achieve.

> so I'll recommend to you to ask on gcc-devel: "how much work it is
> to implement the necessary features from scratch (say on Elisp) for
> providing C++ smart code completion as Clang does?"

It is already implemented.

> You can also take a look at GCC or Clang sources to get an idea of the
> involved complexity (not that the sources reflects all the complexity.)

Irrelevant, because we don't need to compile C++ sources, we need
something else.

Re: Emacs contributions, C and Lisp

[Thien-Thi Nguyen]

[-- Attachment #1: Type: text/plain, Size: 460 bytes --]

() David Kastrup <dak@gnu.org>
() Sat, 01 Mar 2014 14:09:14 +0100

   I shudder to think what for.  I'd have expected
   them to have torn out all of their hair long ago.

There is the compile time and there is the run time.

-- 
Thien-Thi Nguyen
   GPG key: 4C807502
   (if you're human and you know it)
      read my lisp: (responsep (questions 'technical)
                               (not (via 'mailing-list)))
                     => nil

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: Emacs contributions, C and Lisp

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> I know that you don't trust me when I say that this is foolish
>
> It is foolish to call an existing implementation impossible to
> achieve.
>
>> so I'll recommend to you to ask on gcc-devel: "how much work it is
>> to implement the necessary features from scratch (say on Elisp) for
>> providing C++ smart code completion as Clang does?"
>
> It is already implemented.

This is false, as acknowledged by the C++/CEDET developer himself and
easily testable with a few lines of code.

>> You can also take a look at GCC or Clang sources to get an idea of the
>> involved complexity (not that the sources reflects all the complexity.)
>
> Irrelevant, because we don't need to compile C++ sources, we need
> something else.

We don't need the backend, but we need all the other big parts. In the
case of Clang, that's probably more than 70% of its source code (the
backend is provided by LLVM, which is a segregated code base.)

Re: Emacs contributions, C and Lisp

[David Kastrup]

David Kastrup <dak@gnu.org> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: David Kastrup <dak@gnu.org>
>>> Date: Sat, 01 Mar 2014 12:04:04 +0100
>>> 
>>> > I actually don't understand why are we talking about compilers as the
>>> > only viable infrastructure for the features being discussed.
>>> 
>>> It's basically the UNIX philosophy to employ dedicated tools for the job
>>> they have been written for.
>>
>> I'm not at all sure this philosophy is applicable to the issue at
>> hand, or to complex interactive environments the kind of Emacs in
>> general.  We are a long way from the classic batch-mode Unix pipeline,
>> where (AFAIK) this philosophy originated.
>
> [...]
>
> Well, I tried summarizing my view of the situation, and you summarized
> yours.  We arrive at different conclusions from what appears like the
> same information.  But I suppose we have a sufficient amount of
> summaries to have people wanting to take some action see what may and
> may not work to take them where.

Regarding the dump-xref angle: it would appear that the respective
building blocks are described here:

<URL:https://docs.adacore.com/gnatcoll-docs/xref.html>

and that the code is, in one form or the other, in the GCC repository
already.

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[Stefan Monnier]

> The point is to give people a strong motivation to implement the
> necessary support with GCC.

I don't see how avoiding clang support in Emacs provides motivation for
GCC hackers to add this support, that's been lacking for several
years now.


        Stefan "who'd favor a GNU clang effort, extending clang with new
                exciting features distributed under the GPL"

Re: Emacs contributions, C and Lisp

[David Kastrup]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> The point is to give people a strong motivation to implement the
>> necessary support with GCC.
>
> I don't see how avoiding clang support in Emacs provides motivation for
> GCC hackers to add this support, that's been lacking for several
> years now.
>
>
>         Stefan "who'd favor a GNU clang effort, extending clang with new
>                 exciting features distributed under the GPL"

Well, if I understood correctly, part of the reason such support in GCC
is not easily forthcoming is because of technical restrictions that are
a consequence of policy decisions.  So we are shooting ourselves in our
own foot here.

Inherently this boils down to the "viral nature" of the GPL, namely
coercing newly created extensions to fall under the GPL automatically
when distributed, requiring to cover a work "as a whole", and the point
of modularity as a software design goal is to separate works into
reasonably separate aggregates.

This coercive nature of the GPL has bargained us the Objective C
frontend to GCC.  With a high amount of modularity and nice compiler
building blocks, this frontend might have stayed proprietary.

As it stands, Objective C is nowadays very much constrained to fringe
status except on proprietary platforms like those of Apple.  And I doubt
GCC is used to any significant degree for compiling Objective C code.
Which is disappointing.

My personal preference would be to allow general purpose interfaces and
modularity where they provide obvious benefit for Free Software
development of uncoordinated parties.  Of course, where the only
imminent usage (and motivation for factoring out interfaces) would be by
a proprietary program, the mere "it _could_ be used for Free Software"
should not really drive forward a feature, nor should special cases be
added for proprietary software.

But the main problem we are dealing with is that modularity and
general-purpose interfaces weaken the power of the GPL, while at the
same time allowing for more reuse of software and for a better scaling
of developers since they can drive their own projects forward in a less
lockstepped manner.  And the speed and lags of software development
these days make lockstepped development really expensive.

I am not sure that the balance regarding modularity and integration we
are striking these days serves best our goals of promoting Software
Freedom and extending the amount of copylefted software available and
useful.  The easiest way for a large number of people to cooperate, and
there _is_ a large number of people writing copylefted software, is to
make sure this cooperation requires a minimal amount of interaction, and
that means modular and versatile tools.  After all, that's what UNIX,
after which GNU has been moduled, is all about.

But we are obviously not going to turn on a dime regarding policies long
in the making.

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[Tom]

David Engster <deng <at> randomsample.de> writes:

> 
> Eli Zaretskii writes:
> > Isn't it true that all the packages that use clang for completion save
> > the buffer whenever completion is required?  At least the one I tried
> > did just that, which sounds an awful design decision to me.
> 
> Weirdly enough, while clang compiles from stdin without problems, it
> needs a file to perform completions (last I checked). So you either save
> the current file, which I agree is a no-no, or you save the current
> buffer into a temporary file (which is what semantic-clang does, which
> only exists in CEDET upstream).

This saving to file thing always bothered me, so I searched for it and 
found this in the clang api:

http://clang.llvm.org/doxygen/
group__CINDEX__TRANSLATION__UNIT.html#ga2baf83f8c3299788234c8bce55e4472e

According to this the clang_parseTranslationUnit function has an 
unsaved_files parameter and if you click its type then it says it
provides the contents of a file that has not yet been saved to disk:

http://clang.llvm.org/doxygen/structCXUnsavedFile.html


So it looks like clang has an explicit way for working with unsaved
source files which I guess was added to handle this situation in
editors.

Re: Emacs contributions, C and Lisp

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Sat, 01 Mar 2014 16:08:37 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> I know that you don't trust me when I say that this is foolish
> >
> > It is foolish to call an existing implementation impossible to
> > achieve.
> >
> >> so I'll recommend to you to ask on gcc-devel: "how much work it is
> >> to implement the necessary features from scratch (say on Elisp) for
> >> providing C++ smart code completion as Clang does?"
> >
> > It is already implemented.
> 
> This is false, as acknowledged by the C++/CEDET developer himself and
> easily testable with a few lines of code.

What was acknowledged was that CEDET does not implement the full C++
standard, that's all.  It remains to be seen how important that is to
the Emacs users at large; obviously, CEDET developers didn't think
what they had was useless.  Your needs are not the only ones, and not
necessarily representative of those of others.

> We don't need the backend, but we need all the other big parts. In the
> case of Clang, that's probably more than 70% of its source code (the
> backend is provided by LLVM, which is a segregated code base.)

Because Clang was designed and implemented as a compiler, first and
foremost, and not as a CEDET backend.

How many times we will need to go through this before you will
understand that hand-waving and unsubstantiated claims are not
convincing?  These repetitions serve nothing else but discouraging
people for trying different approaches -- is this really your goal and
your agenda?

Re: Emacs contributions, C and Lisp

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Sat, 01 Mar 2014 10:34:12 -0500
> Cc: Juanma Barranquero <lekktu@gmail.com>, dak@gnu.org, emacs-devel@gnu.org
> 
> > The point is to give people a strong motivation to implement the
> > necessary support with GCC.
> 
> I don't see how avoiding clang support in Emacs provides motivation for
> GCC hackers to add this support, that's been lacking for several
> years now.
> 
> 
>         Stefan "who'd favor a GNU clang effort, extending clang with new
>                 exciting features distributed under the GPL"

It is a mystery to me why you don't favor CEDET instead.

Re: Emacs contributions, C and Lisp

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> > It is already implemented.
>> 
>> This is false, as acknowledged by the C++/CEDET developer himself and
>> easily testable with a few lines of code.
>
> What was acknowledged was that CEDET does not implement the full C++
> standard, that's all.

Actually, it doesn't implement the C++ language as it was used in the
mid 1980s, much less so the old 1998 standard. But that was already
stated multiple times, to no avail.

> It remains to be seen how important that is to
> the Emacs users at large; obviously, CEDET developers didn't think
> what they had was useless.

CEDET is much more than a C++ parser. Regardless the value of its C++
parser, CEDET remains useful. CEDET can use Clang (and it already has
some support for it, AFAIK) for the C++ analysis and then bring in the
associated features into Emacs.

You seem to think that, by using Clang, CEDET is no longer necessary,
when the contrary is the truth: by using Clang, CEDET gets a solid
ground for exploiting its own potential.

> Your needs are not the only ones, and not necessarily representative
> of those of others.

Making an strawman does not help to the discussion.

>> We don't need the backend, but we need all the other big parts. In the
>> case of Clang, that's probably more than 70% of its source code (the
>> backend is provided by LLVM, which is a segregated code base.)
>
> Because Clang was designed and implemented as a compiler, first and
> foremost, and not as a CEDET backend.

You are showing your ignorance here. You can go to clang.llvm.org and
see by yourself, on the front page, that you are wrong, but I guess that
you wont to.

From day 1, Clang aimed at modularization, with each component providing
a convenient API for the benefit of external clients.

Just in passing I'll mention that that was one of the main motivations
for creating Clang. Some of today's Clang heavy contributors would have
preferred to do that modularization on GCC instead of starting from
scratch on a new project, but that was forbidden. Hence Clang is, in
great part, a consequence of the GNU policies intended to avoid GCC
usage by non-free software. Ironic, uh?

> How many times we will need to go through this before you will
> understand that hand-waving and unsubstantiated claims are not
> convincing?

You saw real examples where CEDET failed to handle basic cases, you
ignored advice to ask to the experts, you refused to look at the actual
code that implements the stuff we are talking about, you prefer to judge
on your (evidently) very limited knowledge of C++ than on seeing how
others use it, you wont devote some minutes to learn about how difficult
it is to implement, you pull from nowhere false claims such as "Clang
was designed and implemented as a compiler, first and foremost"...

But yeah, I'm the one who resorts to hand-waving and unsubstantiated
claims.

> These repetitions serve nothing else but discouraging people for
> trying different approaches -- is this really your goal and your
> agenda?

Eli, I sincerely hope that you are not being serious with this. I prefer
to mentally see you right now having a good laugh at my stubborn attemps
to educate you.

Although my real intention is to kill your claim that going with CEDET's
C++ parser makes Clang/GCC unnecesary.

Hopefully other members on this community got the idea that what you
propose is not such a great idea.

Re: Emacs contributions, C and Lisp

[David Kastrup]

Óscar Fuentes <ofv@wanadoo.es> writes:

> Just in passing I'll mention that that was one of the main motivations
> for creating Clang. Some of today's Clang heavy contributors would
> have preferred to do that modularization on GCC instead of starting
> from scratch on a new project, but that was forbidden. Hence Clang is,
> in great part, a consequence of the GNU policies intended to avoid GCC
> usage by non-free software. Ironic, uh?

Not particularly.  The GPL has been crafted to use a subset of
restrictions created by copyright law for ensuring a corpus of software
that cannot be used to create software with other restrictions.

That's the core "irony" that the FSF has to deal with and that the GNU
project has been built upon.  Since it operates by balancing opposing
goals, self-defeating elements are an unavoidable part of the overall
strategy which has, overall, shown itself remarkably effective when
compared to the more straightforward non-copyleft approaches to creating
Free Software.

Since the copyleft approach is basically self-contradictory, the line of
contradiction has to be matched to reality in a manner maximizing user
freedom.  Reality is a moving target, and some manner in which one may
redraw the lines may be rather permanent.

So your observation that GCC did not cover every area that Clang is
currently popular for is not really more ironic as the necessity for
creating and maintaining a copyleft universe in the first place.

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Sat, 01 Mar 2014 20:03:45 +0100
> 
> > Your needs are not the only ones, and not necessarily representative
> > of those of others.
> 
> Making an strawman does not help to the discussion.

It's not a strawman at all.  Different users do have different needs
and preferences, there's nothing wrong about that.

> >> We don't need the backend, but we need all the other big parts. In the
> >> case of Clang, that's probably more than 70% of its source code (the
> >> backend is provided by LLVM, which is a segregated code base.)
> >
> > Because Clang was designed and implemented as a compiler, first and
> > foremost, and not as a CEDET backend.
> 
> You are showing your ignorance here.

I forgive you.

> You can go to clang.llvm.org and see by yourself, on the front page,
> that you are wrong, but I guess that you wont to.
> 
> >From day 1, Clang aimed at modularization, with each component providing
> a convenient API for the benefit of external clients.

If you ever managed a large software project (I did), then you know
that the main goal of any project will show in its design and
implementation loud and clear, no matter what secondary goals are
there.  A compiler whose main goal is to be a set of modules will
never be a good compiler.  In fact, no coherent software system can
ever be made successful if it "aims at modularization" as a primary
goal.

Because I think Clang is a good compiler, I'm sure when it serves as a
library, it pulls in gobs of code that isn't strictly necessary.  It
simply cannot be any other way.

> Just in passing I'll mention that that was one of the main motivations
> for creating Clang. Some of today's Clang heavy contributors would have
> preferred to do that modularization on GCC instead of starting from
> scratch on a new project, but that was forbidden. Hence Clang is, in
> great part, a consequence of the GNU policies intended to avoid GCC
> usage by non-free software. Ironic, uh?

You are again going into politics.  Sorry, not interested.

> But yeah, I'm the one who resorts to hand-waving and unsubstantiated
> claims.

We are talking about features Emacs needs to become a better IDE.  All
I said is that there's more than one way, and the one I think we
should explore first is CEDET.  I didn't say it will necessarily
provide the solution, all I said is that we should _try_.  It's really
the logical way any software shop behaves: first see if any existing
package could fit the bill, and only after that do something from
scratch, or buy an external product.  This must make sense to anyone
who is in this business for any significant time.

You, OTOH, claim from the get-go that this way will certainly fail,
and should be dismissed without even trying, and back that up by toy
programs and slogans.  So yeah, you are the one who makes
unsubstantiated claims.

> > These repetitions serve nothing else but discouraging people for
> > trying different approaches -- is this really your goal and your
> > agenda?
> 
> Eli, I sincerely hope that you are not being serious with this. I prefer
> to mentally see you right now having a good laugh at my stubborn attemps
> to educate you.

If you want me to laugh, add a smiley.

But my worries are first and foremost for the lurkers out there.  They
might not laugh even if you do add a smiley.  You don't attract
newcomers by telling them that the job is so hard they shouldn't even
try.

> Although my real intention is to kill your claim that going with CEDET's
> C++ parser makes Clang/GCC unnecesary.

If the CEDET based solution works well enough, Clang/GCC will indeed
be unnecessary.  They could be useful alternatives, though.

> Hopefully other members on this community got the idea that what you
> propose is not such a great idea.

If they have eyes in their heads, they will see that I'm right, I
hope.

Re: Emacs contributions, C and Lisp

[Óscar Fuentes]

David Kastrup <dak@gnu.org> writes:

> Óscar Fuentes <ofv@wanadoo.es> writes:
>
>> Just in passing I'll mention that that was one of the main motivations
>> for creating Clang. Some of today's Clang heavy contributors would
>> have preferred to do that modularization on GCC instead of starting
>> from scratch on a new project, but that was forbidden. Hence Clang is,
>> in great part, a consequence of the GNU policies intended to avoid GCC
>> usage by non-free software. Ironic, uh?
>
> Not particularly.  The GPL has been crafted to use a subset of
> restrictions created by copyright law for ensuring a corpus of software
> that cannot be used to create software with other restrictions.

IIUC what you say does not apply on this specific case, because those
"heavy contributors" I'm talking about are, in essence, Google employees
who are interested on creating tools for themselves. AFAIK the GPL is
not an issue for them on this case and they will be happy to contribute
back those tools to GCC, as they do for Clang.

[snip]

Re: Emacs contributions, C and Lisp

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

[snip]

>> Eli, I sincerely hope that you are not being serious with this. I prefer
>> to mentally see you right now having a good laugh at my stubborn attemps
>> to educate you.
>
> If you want me to laugh, add a smiley.

Sincerely, just after posting I realized that that phrase would be
clearer with a final smiley.

For the rest, well, we both made our points long time ago, so let's
enjoy the weekend on more productive endeavours.

Regards.

[snip]

Re: Emacs contributions, C and Lisp

[David Kastrup]

Óscar Fuentes <ofv@wanadoo.es> writes:

> David Kastrup <dak@gnu.org> writes:
>
>> Óscar Fuentes <ofv@wanadoo.es> writes:
>>
>>> Just in passing I'll mention that that was one of the main motivations
>>> for creating Clang. Some of today's Clang heavy contributors would
>>> have preferred to do that modularization on GCC instead of starting
>>> from scratch on a new project, but that was forbidden. Hence Clang is,
>>> in great part, a consequence of the GNU policies intended to avoid GCC
>>> usage by non-free software. Ironic, uh?
>>
>> Not particularly.  The GPL has been crafted to use a subset of
>> restrictions created by copyright law for ensuring a corpus of software
>> that cannot be used to create software with other restrictions.
>
> IIUC what you say does not apply on this specific case,

You don't understand correctly.

> because those "heavy contributors" I'm talking about are, in essence,
> Google employees who are interested on creating tools for
> themselves. AFAIK the GPL is not an issue for them on this case and
> they will be happy to contribute back those tools to GCC, as they do
> for Clang.

The Google employees are not the ones who have to figure out the
technical consequences of letting the GPL keep serving its purpose with
GCC.  Apparently you believe that all one needs to do for copyleft to
achieve its goals is to write the GPL and/or slap it on arbitrary
software and then retire.  Which would probably have made Richard quite
more popular with a number of people, but that was never an objective.

The basic "irony" requires more than just maintaining the GPL itself, it
also means technical and strategical decisions that serve to make the
GPL extend to derived applications in a useful and court-defensible
manner.

In this particular case, as I understand it basically from hearsay as
I am not involved with GCC, there were several decisions made by Richard
in the past stopping various attempts at modularizing GCC, like
particular forms of plugins.  The GPL can place demands on a derived
work "as a whole" but does not extend its reach to separate programs
that can, thanks to using well-defined interfaces, be considered as not
being part of the same work.

It does not really matter whether or not the Google engineers would have
been willing to contribute under the GPL: their work would have become
part of upstream GCC only when they were willing/able to assign
copyright to GCC.  But providing the interfaces where they would not
have needed to work with the source code of GCC itself would have meant
_exactly_ that they would not even have needed to release their part of
the work under the GPL because it was _separable_.  Whether or not they
would have released under the GPL and/or reassigned copyright anyway,
anybody else would have been able to release works depending on GCC with
parts released under non-free licenses.

That's the "ironic" line that Richard and the FSF are navigating.  And
if you are planning to sway his opinion, it would be smart to first
understand what it is based on.

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[John Yates]

[-- Attachment #1: Type: text/plain, Size: 1021 bytes --]

On Sat, Mar 1, 2014 at 1:02 PM, Eli Zaretskii <eliz@gnu.org> wrote:

> Because Clang was designed and implemented as a compiler, first and
> foremost, and not as a CEDET backend.
>

Actually that is more wrong than right.

First of all clang is not a compiler.  Hence a frequent mistaken assumption
within this thread that there is one clang, comparable more or less to GCC
(or at least GCC's C++ frontend).  In reality there are multiple tools
built using some collection of clang components.  A traditional C++
compiler creating using LLVM is only one tool:

https://github.com/llvm-mirror/clang-tools-extra
http://llvm.org/devmtg/2008-08/Kremenek_StaticAnalyzer.pdf

Clang from the outset has been and remains a toolkit for C-like languages.
 It is true that - as Eli asserts - it was not designed with CEDET per se
in mind.  But it was designed with the hope of serving the full domain of
tools that want to interact in a language aware manner with C-like source
code.  If I am not mistaken that covers CEDET.

/john

[-- Attachment #2: Type: text/html, Size: 1647 bytes --]

Re: Emacs contributions, C and Lisp

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Sat, 01 Mar 2014 21:12:32 +0100
> 
> we both made our points long time ago, so let's enjoy the weekend on
> more productive endeavours.

Alas, due to peculiarities of my locale, my weekend is already over.

Re: Emacs contributions, C and Lisp

[David Engster]

Óscar Fuentes writes:
> Actually, it doesn't implement the C++ language as it was used in the
> mid 1980s, much less so the old 1998 standard. But that was already
> stated multiple times, to no avail.

It is more complicated than that; the support differs wildly between
different areas. I could go into details, but you're not going to help
out anyway, so what's the point.

> CEDET is much more than a C++ parser. Regardless the value of its C++
> parser, CEDET remains useful. CEDET can use Clang (and it already has
> some support for it, AFAIK) for the C++ analysis and then bring in the
> associated features into Emacs.

Semantic's goal is to be a framework to support different programming
languages. Improving the C++ parser often improves Semantic ability to
parse other languages as well. For instance, if you'd be so kind and
implemented storing of overloads in the Semantic database, any language
which has a similar concept will profit from that.

> You saw real examples where CEDET failed to handle basic cases,

Things that look basic can be incredibly complicated to implement, and
vice versa. For instance, in my personal branch, I taught Semantic to
parse

 std::vector<somestruct> foo;
 foo[0]. // complete 'somestruct' members

Looks simple enough, but since you are a C++ language expert, I'm sure
you know all that stuff this involves, like allocator::rebind.

You saw Semantic cannot deal with overloads, which however has almost
nothing to do with the C++ parser. It's a database problem; I already
told you that, but you chose to ignore that bit.

> Hopefully other members on this community got the idea that what you
> propose is not such a great idea.

And then what? Are *you* contributing to some of the clang-based
projects out there? Or do you simply want others to do the work for you,
and you're not happy they're "wasting" their time on CEDET?

I'm not promising anything regarding what features Semantic will be able
to support in the future. I don't have to sell anything here; this is
not work. I do this because it's fun, and because I learn a *ton* of
stuff. Don't discourage others from experiencing that.

-David

Re: Emacs contributions, C and Lisp

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> we both made our points long time ago, so let's enjoy the weekend on
>> more productive endeavours.
>
> Alas, due to peculiarities of my locale, my weekend is already over.

Hope it started earlier than mine, then.

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    It would be interesting to measure (somehow ;-) whether that "Emacs
    leverage" is enough to preserve GCC's importance.

There is no way to measure such things -- they can't be known
in advance.  What's clear is that we can either _try_ to support
GCC or else give up without a fight.  We can't fight later
if we give up now.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    It cannot be Emacs's task to provide selling points for GCC, exactly
    as GCC doesn't provide selling points for Emacs.

It is not unusual to use one GNU package to motivate people
to work on another GNU package.  We should do it more often,
remembering that these packages aim for a larger goal.

    No project can be requested to be that altruistic.

That concept is not applicable because it presumes a project
that exists narrowly for its own success.

GNU Emacs and GCC are part of a larger project, the GNU operating
system, and the GNU operating system has an overarching ethical
purpose: to establish freedom for computer users.  Our policy must be
based on that purpose; it would be irrational neglect to do otherwise.



-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    I don't see how avoiding clang support in Emacs provides motivation for
    GCC hackers to add this support, that's been lacking for several
    years now.

I will work on that side of it too.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Please don't use this list for discussing how to use Clang features
that have no equivalent in GCC.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[Óscar Fuentes]

David Kastrup <dak@gnu.org> writes:

>>> Not particularly.  The GPL has been crafted to use a subset of
>>> restrictions created by copyright law for ensuring a corpus of software
>>> that cannot be used to create software with other restrictions.
>>
>> IIUC what you say does not apply on this specific case,
>
> You don't understand correctly.
>
>> because those "heavy contributors" I'm talking about are, in essence,
>> Google employees who are interested on creating tools for
>> themselves. AFAIK the GPL is not an issue for them on this case and
>> they will be happy to contribute back those tools to GCC, as they do
>> for Clang.
>
> The Google employees are not the ones who have to figure out the
> technical consequences of letting the GPL keep serving its purpose with
> GCC.  Apparently you believe that all one needs to do for copyleft to
> achieve its goals is to write the GPL and/or slap it on arbitrary
> software and then retire.  Which would probably have made Richard quite
> more popular with a number of people, but that was never an objective.
>
> The basic "irony" requires more than just maintaining the GPL itself, it
> also means technical and strategical decisions that serve to make the
> GPL extend to derived applications in a useful and court-defensible
> manner.
>
> In this particular case, as I understand it basically from hearsay as
> I am not involved with GCC, there were several decisions made by Richard
> in the past stopping various attempts at modularizing GCC, like
> particular forms of plugins.  The GPL can place demands on a derived
> work "as a whole" but does not extend its reach to separate programs
> that can, thanks to using well-defined interfaces, be considered as not
> being part of the same work.
>
> It does not really matter whether or not the Google engineers would have
> been willing to contribute under the GPL: their work would have become
> part of upstream GCC only when they were willing/able to assign
> copyright to GCC.  But providing the interfaces where they would not
> have needed to work with the source code of GCC itself would have meant
> _exactly_ that they would not even have needed to release their part of
> the work under the GPL because it was _separable_.  Whether or not they
> would have released under the GPL and/or reassigned copyright anyway,
> anybody else would have been able to release works depending on GCC with
> parts released under non-free licenses.
>
> That's the "ironic" line that Richard and the FSF are navigating.  And
> if you are planning to sway his opinion, it would be smart to first
> understand what it is based on.

[Thanks for the detailed explanation. This is a complex topic, my
English is poor and I have little time, so forgive me if what follows is
not too very well connected.]

I understand that spreading and enforcing copyleft is the raison d'être
of the FSF. It is also true that on the past other not-so purist
strategies were used (GPL+runtime exception, LGPL).

The ironic part I was talking about not only consists on hackers who
right now are contributing to Clang when they could be contributing to
GCC.

Due to its insistence on making things difficult for non-free software
by banning GCC modularization and API opennes, the FSF helped to create
the environment for Clang to arise. So now non-free software has a more
convenient tool than GCC ever could be (friendly API plus non-copyleft
license) *and* on addition GCC's long-term relevance is threatened.

If GCC architecture turned to be more Clang-like, the GPL would still be
a deterrent for non-free software developers, but GCC could be competing
on technical merit with Clang beyond the traditional compiler realm.

The argument about non-free backends using GCC as a front-end does not
hold much water, because that could be happening now. See gcc-xml: if it
falls short of exporting the full tree is probably because the author
was uninterested on doing that, not because the information is missing.

If you have a minute, take a look at how Clang can be used by external
software (apart from the GCC-style of running the driver and collecting
the output on some file/stream) and see how the GPL would be effective
against non-free applications of Clang, if it were GPLed:

http://clang.llvm.org/docs/Tooling.html

There are essentially two methods: link to a Clang library (and then you
have a combined work) or command the driver to load a plugin, which is
also acceptable by the FSF (IIRC GCC and ld support that.) So there is
not too much of an opportunity to make a "separable" work there, unless
you do a stub driver that uses the Clang libraries, dumps the output to
a stream, and then read that stream from the separate, non-free tool.
But, as mentioned, that was always possible with GCC.

Sorry if I'm still missing the crux of your argument, but nevertheless I
hope to have provided some food for thought.

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    Now the case of Windows is obviously different (as Windows is
    proprietary and is more of a platform rather than a tool), but I would
    guess that it would be reasonable to apply the same logic here.

Windows is proprietary software -- evil and unjust.  LLVM is free
software, so it is not in itself evil or unjust.  However,
noncopylefted LLVM's entry into a field dominated by copylefted GCC
has the effect of opening the door to injustice.

Morally, the two are not equivalent.  What the developers of Windows
are doing is flat-out wrong, and would be wrong under any
circumstances.  What the developers of LLVM are doing is foolish given
that we already had GCC: it invites others to do wrong, in ways which
were difficult to do with GCC.

The difference is comparable to the difference between strewing
radioactive material on a city and building a skyscraper there with a
nuclear power plant on top.  The latter is not an attack, but it is a
very bad thing to do.

Thus, strategically, there is not much difference between launching
LLVM into a world with only GCC and launching Windows into a world
with only GNU/Linux.

In practice, however, there's another difference: Windows today is not
new; on the contrary, it is far more widely used than GNU/Linux.

In fact, our policy towards the two is pretty similar.

It is ok to make Emacs run on Windows, but we don't accept any Emacs
features that depend on Windows.  Our policy is, "it runs best on
GNU".  There shouldn't be any features in Emacs that give an advantage
to some other system over the GNU system.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[Eric S. Raymond]

Richard Stallman <rms@gnu.org>:
>                 What the developers of LLVM are doing is foolish given
> that we already had GCC:

LLVM got off the ground because GCC, by policy, refused to provide
interfaces that some toolmakers wanted.  Consequently, those hackers
exercised their freedom by going around GCC rather than through it.

That may lead to an outcome you don't like, but they could with
precisely equal justification call *you* foolish for crippling GCC by
policy.

Generally, if you use the term "foolish" for people who are acting
intelligently to pursue their own objectives rather than yours, you
will mislead yourself and not affect them at all.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: Emacs contributions, C and Lisp

[Dmitry Gutov]

On 01.03.2014 23:31, Richard Stallman wrote:
>      The one I'm maintaining, and which has been mentioned a few times in
>      this and other threads: http://elpa.gnu.org/packages/company.html
>
> What does the mode do?

It provides a modern-ish code completion interface. Take a look at the 
home page. It has a description and a couple of screenshots.

> What job does it do using Clang?

It calls clang to know the possible completions at point, and their 
signatures (it passes the buffer contents through stdin, and the line 
and column of point). No persistent process, so I'd imagine it's not 
very suitable for large projects.

You might want to give company-clang.el a read. It's short enough.

> I gather there are other back ends; what jobs do they do?

They provide completions using different other sources, in different 
major modes.

Re: Emacs contributions, C and Lisp

[Dmitry Gutov]

Eli Zaretskii <eliz@gnu.org> writes:

> Isn't it true that all the packages that use clang for completion save
> the buffer whenever completion is required?  At least the one I tried
> did just that, which sounds an awful design decision to me.  Since
> when do we let a feature decide for the user when to save the buffer
> she is editing?

You probably tried the wrong package, or had an older Clang installed.
Recent Clang doesn't have that restriction.

Re: Emacs contributions, C and Lisp

[David Kastrup]

"Eric S. Raymond" <esr@thyrsus.com> writes:

> Richard Stallman <rms@gnu.org>:
>>                 What the developers of LLVM are doing is foolish given
>> that we already had GCC:
>
> LLVM got off the ground because GCC, by policy, refused to provide
> interfaces that some toolmakers wanted.  Consequently, those hackers
> exercised their freedom by going around GCC rather than through it.
>
> That may lead to an outcome you don't like, but they could with
> precisely equal justification call *you* foolish for crippling GCC by
> policy.

Shrug.  The whole point of the GPL is "crippling by policy", preventing
reuse in proprietary software and thus also affecting legitimate uses in
Free Software.  You can call people foolish who submit themselves to
chemotherapy, because it is making them sick and lets their hair fall
out.  Yes, there are adverse consequences.  But pretending there are no
desired consequences is disingenuous.

> Generally, if you use the term "foolish" for people who are acting
> intelligently to pursue their own objectives rather than yours, you
> will mislead yourself and not affect them at all.

The GPL is a legal tool working by coercion.  It's purpose is to
preclude people thwarting the objectives of free software when making
use of it.  There is no problem with people pursuing different
objectives having to use different tools: the GPL has been designed to
only help those who are willing to accept certain objectives.

Things go wrong only When those _sharing_ the common objectives don't
make use of GCC.  There may be several reasons for that.  And "we are
for free software, but proprietary software does pose no danger to our
goals" is the one that is foolish.

A significant number of Clang developers have worked on Clang because
the restrictions and/or the design of GCC did not meet their goals,
without caring particularly for free software or copyleft.  That GCC did
not meet their requirements may have been partly by design.

The foolishness comes more by those who embrace Clang as being under a
"more free" license than GCC.  Clang development is significantly driven
by proprietary interests, and quite a bit of code that makes use of it
is not getting contributed back.  Supporting the parts of the
infrastructure required to keep things like proprietary compilers for
GPUs running means work without gain.

An extreme case of that was the OpenDarwin project.  They basically were
doing free development work for Apple, with Apple not giving them
anything useful in return, dragging their feet in providing them with
the code that they maintained without anything useful coming from it
that would not have been behind proprietary walls.  After several years
of increasingly disenfranchising and downright condescending behavior by
Apple, OpenDarwin closed shop.

If you let those running the monetized part of the show set the rules,
that's what comes out quite often.  It's basically a microcosmic version
of the U.S. government which makes its decisions based on the flow of
lobbyist payments.  The actual bulk of the total money spent comes from
taxpayers, but the direction where it flows is set by those bribing the
switchguards, the politicians.  Creating and supporting a system that
takes the controls from those actually keeping it running is foolish by
those that hand off control voluntarily to people not serving the same
interests.

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[Eli Zaretskii]

> Date: Sat, 01 Mar 2014 16:28:53 -0500
> From: Richard Stallman <rms@gnu.org>
> CC: dak@gnu.org, emacs-devel@gnu.org
> 
>     It cannot be Emacs's task to provide selling points for GCC, exactly
>     as GCC doesn't provide selling points for Emacs.
> 
> It is not unusual to use one GNU package to motivate people
> to work on another GNU package.  We should do it more often,
> remembering that these packages aim for a larger goal.

I agree with what you say, but that wasn't my point at all, as should
be clear from reading what I wrote in its entirety.

>     No project can be requested to be that altruistic.
> 
> That concept is not applicable because it presumes a project
> that exists narrowly for its own success.
> 
> GNU Emacs and GCC are part of a larger project, the GNU operating
> system, and the GNU operating system has an overarching ethical
> purpose: to establish freedom for computer users.  Our policy must be
> based on that purpose; it would be irrational neglect to do otherwise.

No disagreement here, but again, you removed all context from what I
wrote, which makes the quote sound very different from its original
meaning.  Fortunately, the original text should talk for itself.

Re: Emacs contributions, C and Lisp

[Stephen J. Turnbull]

David Kastrup writes:
 > "Stephen J. Turnbull" <stephen@xemacs.org> writes:

 > > However, this situation is easily enough changed.  The useful programs
 > > from the LLVM project can be forked (AFAIK their license is not
 > > perversely incompatible with the GPL) as GNU projects under the "GPL
 > > v3 or later" permission schema.
 > >
 > > Would you object to that?
 > 
 > I'd object, for basically practical reasons.  You can fork code, but you
 > cannot fork a community.

True.

 > A fork of the LLVM codebase under the GPLv3 makes only sense if you
 > actually add nontrivial nonseparable components under the GPLv3 or
 > the code base can be just swapped out.

Not at all.  This could merely be a distribution fork, like the
Ghostscript dual license scheme, or the various foobar+gnureadline
distributions of individual programs that appeared over the years, or
like some of the various commercial versions of BSD and X11 and TeX
that have appeared over the years.

And of course the code base can just be swapped out.  The point is
simply to make the public point that *this* distribution is copyleft,
and *that one* isn't.  "Defend" their free software for them, as it
were.

I expect Richard to object too (on the grounds that it's still
providing succor to the "enemy" by validating their code, and
"undermining" GCC because GCC's feature support is either less, or a
patch, or nonexistent depending on who you listen to), but I don't
think "redistributing code under a different license" means you have
to "add nontrivial nonseparable components."

 > Now if the upstream is a weak license like X11, at some point of time

It has to be, otherwise sublicensing can't happen.

 > the developers might say "that fork is annoying, let's relicense
 > what we have now under BSD with advertising clause and cut off
 > those others".

Implausible.  Yes, SSLeay.  That example is over a decade old and was
pretty idiosyncratic and poorly-received even then.

OpenOffice vs. LibreOffice ... doesn't that undermine your point?
Apache 2.0 is compatible with GNU GPL 3.0.  I would suppose it's
compatible with LGPL 3.0, since LGPL 3 is GPL 3.  Sure, I suppose the
folks at Apache would be a little miffed at the one-way flow of code,
but they did it to themselves (at fairly high cost of redundant
development, too).  AFAIK there was never any intent by Apache legal
to be GPL incompatible, it was just that they took the step of adding
a patent "poison pill" before the GPL did that made their licenses
GPLv2-incompatible.

 > They'll likely retain the majority of their development community but
 > hang out the fork under GPLv3 to dry.

I really don't see why LLVM would care enough to pervert their
license, any more than Apache does.

Re: Emacs contributions, C and Lisp

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> David Kastrup writes:
>  > "Stephen J. Turnbull" <stephen@xemacs.org> writes:
>
>  > > However, this situation is easily enough changed.  The useful programs
>  > > from the LLVM project can be forked (AFAIK their license is not
>  > > perversely incompatible with the GPL) as GNU projects under the "GPL
>  > > v3 or later" permission schema.
>  > >
>  > > Would you object to that?
>  > 
>  > I'd object, for basically practical reasons.  You can fork code, but you
>  > cannot fork a community.
>
> True.
>
>  > A fork of the LLVM codebase under the GPLv3 makes only sense if you
>  > actually add nontrivial nonseparable components under the GPLv3 or
>  > the code base can be just swapped out.
>
> Not at all.  This could merely be a distribution fork, like the
> Ghostscript dual license scheme, or the various foobar+gnureadline
> distributions of individual programs that appeared over the years, or
> like some of the various commercial versions of BSD and X11 and TeX
> that have appeared over the years.
>
> And of course the code base can just be swapped out.  The point is
> simply to make the public point that *this* distribution is copyleft,
> and *that one* isn't.  "Defend" their free software for them, as it
> were.

But you can't.  There is no point in slapping a license on a
distribution when you don't have standing to sue over license breaches
since you are not holding copyright to any significant part of it.

It only weakens the GPL if you start creating situations where it cannot
be taken seriously and/or enforced.

> OpenOffice vs. LibreOffice ... doesn't that undermine your point?
> Apache 2.0 is compatible with GNU GPL 3.0.  I would suppose it's
> compatible with LGPL 3.0, since LGPL 3 is GPL 3.  Sure, I suppose the
> folks at Apache would be a little miffed at the one-way flow of code,
> but they did it to themselves (at fairly high cost of redundant
> development, too).

If you take a look at R.C.Weir venting off in the comment section of
basically every publication delivering a LibreOffice release
announcement, that "a little miffed" is not a mere hypothetical.

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[Stephen J. Turnbull]

David Kastrup writes:

 > Shrug.  The whole point of the GPL is "crippling by policy",
 > preventing reuse in proprietary software and thus also affecting
 > legitimate uses in Free Software.

It's true it has that effect (although I think you're being overly
melodramatic), but that is *not* the point of GPL.  The point of the
GPL is reciprocity.  That has been shown to work in many businesses,
including some that depend on proprietary licensing for the lion's
share of their profit.  And yes, it has affected legitimate uses, but
those issues are becoming fewer as more and more projects switch to
GPL-compatible policies (including dual licensing).

The "use GNU" policy is merely self-defeating, I think -- stifling
friendly competition is a very bad thing, have we learned nothing from
the Great Planning Experiments of Marx-Lenin-Stalin-ism?  How long
does it take to see that GNU Arch is dead and WannaGNU Bazaar had its
commercial support withdrawn ... er, just about the time it became
really usable for Emacs development?  But OK, the "avoid redevelopment
of every feature" argument is very appealing, and the failure of
communism is merely an analogy so why not try it?

But this anti-LLVM policy?  Let's see if I understand how this works.
We have Groff, so TeX modes are treason.[1]  We have GNU sed and gawk,
so Perl, Python, and Ruby modes gotta go.  Oh yeah, don't forget to
rip out raw X11 and TTY support in Emacs entirely, just link to GTK.

Where does it stop?  Oh, yeah.  It stops with LLVM.  Good thing!  But
the logic escapes me.  (And no, I see no logical reason to only
ostracize a program if it implements the same language standard as
another.  If use cases are similar, that should be enough to argue
against use of the permissively-licensed product in favor of the
copyleft substitute.)

Footnotes: 
[1]  I wonder who might object a bit strongly to that decision, hm?

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

By mentioning just part of the situation, you've created the appearace
that my decision backfired.  Looking at the real goal we see it was
successful.

    LLVM got off the ground because GCC, by policy, refused to provide
    interfaces that some toolmakers wanted.

True.  Note that I set this policy because the other choice would have
immediately opened the door to nonfree compilers based on GCC.

      Consequently, those hackers
    exercised their freedom by going around GCC rather than through it.

Yes, they did, and brought about part of the bad results I tried to
avoid -- around 15 years later.  We delayed them for 15 years!

Not only that, but since Clang only handles C and C++, we have also
reduced the scope of the bad results.  We are still succeeding in
preventing them for other languages.

This was not a permanent total victory, sad to say, but it was a
victory.  It shows that my decision was right.

Furthermore, they did not HAVE to release their program under a
pushover license.  Thus, there was a chance for an even greater
partial victory.

Over all, I made the right decision.  Perhaps it could have been
a little better.

    Generally, if you use the term "foolish" for people who are acting
    intelligently to pursue their own objectives rather than yours, you
    will mislead yourself and not affect them at all.

If "people who" refers to the LLVM developers, it makes no difference
since I'm not addressing them anyway.  I'm talking to people working
on the GNU Project about our goals.

When I say that releasing LLVM in these circumstances was foolish, I
mean that in terms of freedom as a goal.  Of course there are people
with other views.  There may be some who think computers are satanic
and programmers should be burned at the stake.  But since we're
talking about a GNU Project decision, what other views might imply is
beside the point.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I guess I did not understand your point.  I saw those two statements
which seemed wrong, and that was all I could get.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[Eli Zaretskii]

> Date: Sun, 02 Mar 2014 12:42:53 -0500
> From: Richard Stallman <rms@gnu.org>
> Cc: dak@gnu.org, emacs-devel@gnu.org
> 
> I guess I did not understand your point.  I saw those two statements
> which seemed wrong, and that was all I could get.

My point, in a nutshell, was and remains that we should direct our
efforts in this area towards CEDET, which is already part of Emacs.

Re: Emacs contributions, C and Lisp

[Dmitry Gutov]

Eli Zaretskii <eliz@gnu.org> writes:

> My point, in a nutshell, was and remains that we should direct our
> efforts in this area towards CEDET, which is already part of Emacs.

CEDET can also use external parsing tools.

Have you noticed David Engster arguing towards being allowed to use
Clang/LLVM?

Re: Emacs contributions, C and Lisp

[Óscar Fuentes]

Richard Stallman <rms@gnu.org> writes:

> By mentioning just part of the situation, you've created the appearace
> that my decision backfired.  Looking at the real goal we see it was
> successful.

I agree with ESR: your decision backfired. Furthermore, the worse is yet
to come.

>     LLVM got off the ground because GCC, by policy, refused to provide
>     interfaces that some toolmakers wanted.
>
> True.  Note that I set this policy because the other choice would have
> immediately opened the door to nonfree compilers based on GCC.

This is not correct. The key point that enabled LLVM/Clang for
propietary software is the license, not its architecture. With the
LLVM/Clang architecture, if it was GPLed, you either end with a combined
work or having to create a Free driver for interfacing with the non-free
part. The later was always possible with GCC.

>       Consequently, those hackers
>     exercised their freedom by going around GCC rather than through it.
>
> Yes, they did, and brought about part of the bad results I tried to
> avoid -- around 15 years later.  We delayed them for 15 years!

LLVM was usable circa 2003. Clang started in 2007 and was usable as a C
compiler on 2009, IIRC.

BTW, who are "them"?

> Not only that, but since Clang only handles C and C++,

Clang handles C, C++ and Objective C/C++. There are projecs for adding
Fortran.

> we have also
> reduced the scope of the bad results.  We are still succeeding in
> preventing them for other languages.

LLVM acts as the backend for a large number of compilers for multiple
languages. Most of them are Free Software.

It's also very attractive for researchers and teachers, which creates a
cumulative effect. From experience, I can say that hacking LLVM is far
easier, pleasant and productive than working with GCC. Too bad that
proposals for modernizing GCC's code base were rejected for so long.

> This was not a permanent total victory, sad to say, but it was a
> victory.  It shows that my decision was right.

Putting GCC on the way to irrelevance is not a victory.

[snip]

Re: Emacs contributions, C and Lisp

[Stephen J. Turnbull]

David Kastrup writes:

 > But you can't.  There is no point in slapping a license on a
 > distribution when you don't have standing to sue over license breaches
 > since you are not holding copyright to any significant part of it.
 > 
 > It only weakens the GPL if you start creating situations where it cannot
 > be taken seriously and/or enforced.

I see.  So the widespread use of GPL in projects that don't collect
assignments is another excuse to declare a piece of software an enemy
of the movement.

Seriously, I disagree.  Sure, somebody who really wants to take it
proprietary can do so.  But they can do that with most GPL'ed software
too by making their derivative a webapp.

 > > OpenOffice vs. LibreOffice ... doesn't that undermine your point?
 > 
 > If you take a look at R.C.Weir venting off in the comment section of
 > basically every publication delivering a LibreOffice release
 > announcement, that "a little miffed" is not a mere hypothetical.

Once again, if he's really venting about the license (and not about
"who is the real successor to Sun OpenOffice.org"), that's crazy.  If
Apache didn't want to enable one-way code flow, they wouldn't use a
permissive license.  (Who is R.C.Weir, anyway?  I seem to recall a
Grateful Dead guitarist by that name....)

Re: Emacs contributions, C and Lisp

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> David Kastrup writes:
>
>  > But you can't.  There is no point in slapping a license on a
>  > distribution when you don't have standing to sue over license breaches
>  > since you are not holding copyright to any significant part of it.
>  > 
>  > It only weakens the GPL if you start creating situations where it cannot
>  > be taken seriously and/or enforced.
>
> I see.  So the widespread use of GPL in projects that don't collect
> assignments is another excuse to declare a piece of software an enemy
> of the movement.
>
> Seriously, I disagree.

Since I cannot even figure out what your strawman is supposed to refer
to, I am not sure what you disagree with.

>  > > OpenOffice vs. LibreOffice ... doesn't that undermine your point?
>  > 
>  > If you take a look at R.C.Weir venting off in the comment section
>  > of basically every publication delivering a LibreOffice release
>  > announcement, that "a little miffed" is not a mere hypothetical.
>
> Once again, if he's really venting about the license (and not about
> "who is the real successor to Sun OpenOffice.org"), that's crazy.

It's rather hard to tell what it is _supposed_ to be about.  But it is
not hard to see the license-originating asymmetry of the overall
situation as a driving factor.

> If Apache didn't want to enable one-way code flow, they wouldn't use a
> permissive license.

They are fine with one-way flow into proprietary products.  They tend to
be less than enthused about GPLed or LGPLed reuse.

It's like selling a donkey stallion of your own good breed to a mule
breeding farm and you find they mate him with donkeys rather than mares,
creating breeding donkeys competing with your own business.

In other terms: proprietary products don't compete with their community
and their ideas of freedom.

I understand what this is about, but it does demonstrate a sore spot of
the model.

> (Who is R.C.Weir, anyway?  I seem to recall a Grateful Dead guitarist
> by that name....)

I think he is something like the principal maintainer of Apache
OpenOffice or close to that, but I don't have all that much of a clue.

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 02 Mar 2014 20:20:39 +0200
> Cc: dak@gnu.org, rms@gnu.org, emacs-devel@gnu.org
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > My point, in a nutshell, was and remains that we should direct our
> > efforts in this area towards CEDET, which is already part of Emacs.
> 
> CEDET can also use external parsing tools.

I meant Semantic, not external parsers.

Re: Emacs contributions, C and Lisp

[Stephen J. Turnbull]

David Kastrup writes:
 > "Stephen J. Turnbull" <stephen@xemacs.org> writes:
 > > David Kastrup writes:

 > >  > It only weakens the GPL if you start creating situations where
 > >  > it cannot be taken seriously and/or enforced.
 > >
 > > I see.  So the widespread use of GPL in projects that don't collect
 > > assignments is another excuse to declare a piece of software an enemy
 > > of the movement.
 > >
 > > Seriously, I disagree.
 > 
 > Since I cannot even figure out what your strawman is supposed to
 > refer to, I am not sure what you disagree with.

The argument for the FSF assignment policy is that it makes the
license much easier to enforce.  Ie, it is the position of the FSF
that failing to collect assignments creates "situations where [the
GPL] cannot be taken seriously and/or enforced."  If the GPL can't be
enforced, the software can be proprietized -- and so such projects,
like LLVM, are "failing to defend software freedom".  So Emacs
shouldn't provide modes to support their unique features according to
RMS's logic AIUI.

But I don't see how it can weaken the GPL.  Some users will ignore the
permission notice, but they do so at their peril -- the whole app
can't be defended, but any parts original to the copyleft project can,
and the licensor is under no obligation to make disentangling the two
simple (for most upstream licenses -- like the GPL at most you must
indicate that the program is modified, not what you did).

The GPL *itself* isn't weakened, at least not in the U.S.: it's well-
established that failure to pursue a copyright infringement does not
constitute a license -- the copyright holder can change her mind and
put infringers in the dock at any time.  If users copy other than as
provided by the GPL, they likely have no license at all (AFAIK
permissive licenses don't have clauses like GPL sec. 10).

And there is a potential benefit.  Most users will open the file, see
"GPL", and automatically obey the terms of the GPL for code they copy
into their own projects.  OK, some may feel cheated if they discover
the file is substantially identical to permissively licensed upstream
code, but those folks already resent the GPL -- what's lost?

 > > If Apache didn't want to enable one-way code flow, they wouldn't
 > > use a permissive license.
 > 
 > They are fine with one-way flow into proprietary products.  They
 > tend to be less than enthused about GPLed or LGPLed reuse.

Sure.  But they aren't stopping it by adding an EAY clause to the
license.  At worst, they jawboning for their philosophy of freedom in
the style practiced by RMS as well.  Eg, he has not added the Affero
clause to the GPL itself, but instead made a separate license.  Still,
he recommends the AGPL, and denounces closed cloud-based applications
using GPLed software every chance he gets.

I just don't see a strong argument that a GPLed fork is legally
impractical, just RMS's argument that it wouldn't be effective because
it would permit the same kind of proprietary combination that his
policy toward refactoring GCC is intended to prevent.

Re: Emacs contributions, C and Lisp

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> David Kastrup writes:
>  > "Stephen J. Turnbull" <stephen@xemacs.org> writes:
>  > > David Kastrup writes:
>
>  > >  > It only weakens the GPL if you start creating situations where
>  > >  > it cannot be taken seriously and/or enforced.
>  > >
>  > > I see.  So the widespread use of GPL in projects that don't collect
>  > > assignments is another excuse to declare a piece of software an enemy
>  > > of the movement.
>  > >
>  > > Seriously, I disagree.
>  > 
>  > Since I cannot even figure out what your strawman is supposed to
>  > refer to, I am not sure what you disagree with.
>
> The argument for the FSF assignment policy is that it makes the
> license much easier to enforce.

No, that it makes the license enforceable at all.  Only the copyright
holder can enforce the license for a piece of code.

> Ie, it is the position of the FSF that failing to collect assignments
> creates "situations where [the GPL] cannot be taken seriously and/or
> enforced."

Wrong.  The GPL can _always_ be enforced by the copyright holder.
Collecting the assignments makes sure that the FSF has the full ability
to enforce any violation of the whole code base even when a non-trivial
part of the code's history can be traceable to someone affiliated with
the defendant.

The problem is not that the GPL would or would not be taken seriously,
but rather the ability of the FSF to act as copyright holder.  Since the
FSF is not directly responsible for the majority of the code it has
copyrights on, acting as a warden without assignments would be
difficult.

> If the GPL can't be enforced,

The GPL can be enforced.  The problem would rather be that the choice
whether to enforce it or not would not lie with the FSF.  If you take a
look at the enforcement situation with the Linux kernel, a large ratio
of enforcement cases is done by Harald Welte over the Netfilter code.
Throw out the network filtering (and isn't it currently being replaced
anyway?), and you have a good chance to use the Linux kernel in
proprietary settings without anybody bothering to sue.

That's underwhelming.

> the software can be proprietized -- and so such projects, like LLVM,
> are "failing to defend software freedom".  So Emacs shouldn't provide
> modes to support their unique features according to RMS's logic AIUI.

> But I don't see how it can weaken the GPL.

It doesn't weaken the GPL.  It weakens GCC, and with it, the GNU
project.  Part of the strength of the GNU project lies with GCC setting
language and performance standards, and it's bad if the most thorough
modes we make available with Emacs are not able to support the GCC
dialects.

The Linux ecosystem (for a lack of a better work) is tightly connected
to GNU, not just because of the general GNU flavor of the user space
(including glibc and the core utilities) but also because the kernel is
compiled using GCC, both for feature and performance reasons.  And I
think that it may by now be possible to compile the Linux kernel with
Clang (which got implementions of all of the extensions including
assembler templates that the kernel used).  So in this case GCC and the
GNU project were in the situation of creating a de facto standard.  Once
Emacs supports what Clang provides rather than what GCC provides, not
even Emacs will support any new features of GCC.

There is a rather tenuous hold based on licenses, technical
dependencies, public perception (including "coolness") and so on over
where semi-open systems like "Android" go, choose to go, and are able to
go.  GCC plays a central role in where people place their focus, their
loyalties, and their goals.  It's one of the actually active ties to the
GNU project that the Linux kernel has.

So we want to have GCC stay technically competitive in order to retain a
non-licensing based connection into those kind of projects and, not
least of all, the Linux kernel.  On the other hand, staying technically
competitive is not self-serving but it serves to have a wider audience
for our non-technical goals.  Giving up the latter in order to have a
stronger stand for the former is sort of pointless.  Sort of what one
calls a "dying throw" or "parting shot".

When put to the choice, we'd rather give up on Android than GNU.  And us
having to take that choice at some point of time is what Clang/LLVM are
working towards.

Once Clang/LLVM become the compiler for Android, it is a matter of time
and hipness before it becomes the default compiler for the Linux kernel.

Once the kernel is bootstrapped using Clang/LLVM, it is a matter of time
before the userland of big distributions is compiled using Clang/LLVM as
well, with GCC becoming optional.

At some point of time, it will become harder to compile the Linux kernel
with GCC out of the box, and bootstrapping a full GNU system will mean
that we need to revert to the HURD, mostly relying on using Linux
drivers.  At least until someone sues us for combining GPLv2 drivers
with a GPLv3+ kernel.

All of this has nothing to do with "the GPL being weakened".  It's the
weakening of GCC as a vital part of the GNU-derived software universe
that's the issue.

And if you are saying "that's too pessimistic.  All of this _could_
happen, but that's by no means sure.": the whole point of being behind
GCC is to do what is in our power to push against this happening.  Our
power may be small compared to those of the universe, but that has not
kept us from making a difference by applying it aimedly and timely
enough.

"We can still patch up our roof if and when a storm comes" does not
work.  And "oh, it's on the horizon, but maybe it'll pass.  Let's see if
it does, then we don't need to patch the roof." is, well...

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Reciprocity is an aspect of the point of the GNU GPL, but its main
point is _defending users' freedom_.

With the GNU GPL (and copyleft in general), we make sure that all
copies of all versions of our code _respect users' freedom_.  We make
sure that if people develop nonfree software, if we can't stop them,
at least we deny them the benefit of our help.

We also oblige them, in some circumstances, to contribute their
improvements to our community, and that's the reciprocity aspect.

    stifling friendly competition is a very bad thing,

By calling LLVM "friendly competition" you misrepresent the issue at
stake.  You're wasting your time, asking me to change my mind based on
ignoring what's at stake.

In any case, we are not in a position to stifle LLVM, so you're
discussing an imaginary issue.

     have we learned nothing from
    the Great Planning Experiments of Marx-Lenin-Stalin-ism?

Lots of projects make plans and implement them.  If you equate making
plans to Communist dictatorship, you've gone off the deep end.  And
that's not to mention that your accusation is so nasty that we
shouldn't be on speaking terms.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    But by the same reasoning you are applying to clang/LLVM, Emacs' must
    not support any feature from Subversion (which is Apache-licensed)
    unless the same feature is present

You're making the tacit basic assumption that the right policy
is a function only of the _structure_ of a situation.
That's not the way we make these decisions.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    The argument for the FSF assignment policy is that it makes the
    license much easier to enforce.

Yes, but...

      Ie, it is the position of the FSF
    that failing to collect assignments creates "situations where [the
    GPL] cannot be taken seriously and/or enforced."

that is putting it too strongly.  This does not make it impossible, but
it makes things harder in some ways.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[Stephen J. Turnbull]

David Kastrup writes:
 > "Stephen J. Turnbull" <stephen@xemacs.org> writes:
 >> David Kastrup writes:
 >>> "Stephen J. Turnbull" <stephen@xemacs.org> writes:
 >>>> David Kastrup writes:

 >>>>> It only weakens the GPL

[big snip and back to David]

 > The GPL can _always_ be enforced by the copyright holder.
 > Collecting the assignments makes sure that the FSF has the full
 > ability to enforce

You could have stopped there.  *You* said "weaken the GPL" when what
you really meant was "weaken the FSF."  OK, conceded.

But you continue:

 >> If the GPL can't be enforced,
 > 
 > The GPL can be enforced.

In theory, yes, each of a large collection of copyright-holders can
enforce the GPL on their little bit.  In practice, as you say:

 > That's underwhelming.

That is, such a collective's threat of action to enforce is not
credible, and the costs of defending in the unlikely event that such
challenge arises becomes an ordinary business risk.

Aside: I wonder if perhaps you could defend something like XEmacs or
the Linux kernel or Python with a class action suit.  I'll have to ask
a lawyer.  RMS might want to ask his, too, since the FSF owns enough
of XEmacs, and probably other non-GNU software, to be a plausible
class representative for them.  Or maybe he already has. :-)

And then your main point:

 > It doesn't weaken the GPL.  It weakens GCC, and with it, the GNU
 > project.  Part of the strength of the GNU project lies with GCC
 > setting language and performance standards,

Except that last is now an exaggeration.  Obviously GCC no longer sets
performance standards, and since the reason I switched my default to
Clang was to get better error messages, in at least one aspect it
seems to be lagging on language standard-setting.  (But see discussion
below, especially fn. 1.)

This is what happens when you have no real competition.

 > and it's bad if the most thorough modes we make available with
 > Emacs are not able to support the GCC dialects.

Now that, as Richard would say, is a misrepresentation.  Of course
nothing prevents Emacs from supporting GCC dialects, certainly not a
fiat from Mt. Olympus.  Of course Emacs *should* support them.  And if
such support was lacking or weak, I think there's some chance that
Richard would come around and encourage development of that mode.
Don't you?

AFAICS, the issue at hand is supporting unique features of free
software whose licenses do not defend freedom according to Richard.
There are a lot of such licenses, including those used by TeX, Perl,
Python, Ruby, ncurses, X11, and Apache.  Why doesn't the logic that
applies to LLVM apply to them too?

 > So in this case GCC and the GNU project were in the situation of
 > creating a de facto standard.  Once Emacs supports what Clang
 > provides rather than what GCC provides, not even Emacs will support
 > any new features of GCC.

Hm.  You've said that twice, now.  This is a leap that I cannot
follow.  Would you explain?  I understand the rest of your argument
that *compiler users* could abandon GCC en masse.  What I don't see is
why that would prevent Emacs from supporting unique GCC features any
more than it prevents Emacs from supporting a zillion true niche
applications (eg, modes for .Xresource and RGB.txt manipulation).

Furthermore, my personal estimate is that GCC's capacity for relevant
innovation has been sapped by its dominant position and by the focus
on internal issues that comes with high-stakes standard-setting.  And
furthermore, refactoring to make internal data structures available to
cooperating applications has been forbidden for a decade or so.  Just
where are those "unsupportable" new features going to come from?  And
what is going to make them radical enough from the users' (and Emacs's)
point of view that support is non-trivial?

 > GCC plays a central role in where people place their focus, their
 > loyalties, and their goals.

Not in my experience.  I tried Clang *once* because an XEmacs user
posted about warnings it was issuing, discovered that its error
messages and warnings were way more useful than GCC's, and switched
immediately.[1]  Perhaps kernel developers and a few others pushing the
envelope of C performance or C++ conformance experience GCC as central,
but for most of us it's just a tool, quite replaceable, I suspect.
glibc and Emacs are far more iconic, and of course glibc is universal
to the GNU System.

 > It's one of the actually active ties to the GNU project that the
 > Linux kernel has.

That's because the kernel developers have not been supported by GNU.
As usual with GNU, it was "my way or the highway."  Linus took the
highway, and almost all kernel developers went with him.  There had
to be a better way to handle that.  There has to be a better way to
handle LLVM.  I'm not saying "mistakes were made"[2], but surely the
result was undesirable.  And I'm saying "we really don't want a
central component of the system developed outside of the 'House of
GNU' again."[3]

 > So we want to have GCC stay technically competitive

No, at least based on current behavior, you don't want *GCC* to *do*
anything.  You want *LLVM* to *fail to innovate* in the same way that
GCC is *prohibited* from innovating.  The effect is the same, but your
statement suggests that GCC is being defended, when in actual fact
what is happening here is that Emacs is being commanded to attack LLVM
(in the sense of "trade war", ie, withdraw cooperation from an external
entity whose competence threatens a protected local industry).

This is only to the detriment of Emacs and LLVM users.  It does not
hinder LLVM's attempt to achieve compiler dominance, and may even
en(cou)rage some because they see it as mere spitefulness from GNU,
and evidence that GCC really is lagging, even as viewed by GNU itself.

 > in order to retain a non-licensing based connection into those kind
 > of projects and, not least of all, the Linux kernel.  On the other
 > hand, staying technically competitive is not self-serving but it
 > serves to have a wider audience for our non-technical goals.
 > Giving up the latter in order to have a stronger stand for the
 > former is sort of pointless.  Sort of what one calls a "dying
 > throw" or "parting shot".

That justifies redoubling efforts to innovate in GCC, while respecting
the self-imposed limitation on acceptable areas for innovation.  It
doesn't justify "trade war".

 > When put to the choice, we'd rather give up on Android than GNU.
 > And us having to take that choice at some point of time is what
 > Clang/LLVM are working towards.

I rather think you overstate the importance of GNU in the thinking of
LLVM developers.

 > Once Clang/LLVM become the compiler for Android, it is a matter of
 > time and hipness before it becomes the default compiler for the
 > Linux kernel.
 > 
 > Once the kernel is bootstrapped using Clang/LLVM, it is a matter of
 > time before the userland of big distributions is compiled using
 > Clang/LLVM as well, with GCC becoming optional.
 >
 > At some point of time, it will become harder to compile the Linux
 > kernel with GCC out of the box, and bootstrapping a full GNU system
 > will mean that we need to revert to the HURD, mostly relying on
 > using Linux drivers.

A sad future history indeed.  And you think that refusal to integrate
Emacs modes that use Clang to improve the Emacs user experience is the
finger in the dike that will save Holland, er, the GNU Project?

The obvious fact is that GNU can't stop the process you've described
-- it all happens outside of GNU!  You can only hope it will fall
apart due to its own internal inconsistencies.  Unfortunately, the
process you describe is pretty clearly the GNU Project falling apart
due to its own internal inconsistency.  Specifically, it wants to be
universal but denies the clearly expressed aspirations and preferences
of the rest of the free-software-producing world and their users.

And now, it's denying the aspirations of some of its own members,
merely to spite those with different beliefs about defending freedom.

GNU's beliefs are essential to the free software movement.  Nobody can
deny that.  But this decision is mere spite, without real effect, and 

 > At least until someone sues us for combining GPLv2 drivers with a
 > GPLv3+ kernel.

Surely GNU will not take that risk.  If that ever happens, RMS and the
GNU Project lose so much face that it's "game over".  If the HURD is
already doing so, isn't it time for a revolution?  (Preferably led by
RMS himself.)

 > And if you are saying "that's too pessimistic.  All of this _could_
 > happen, but that's by no means sure."

No.  I'm saying you're way too *optimistic*.  The story you tell is
way too plausible to be ignored, and the current response is way too
weak.  GNU outlived Zawinski's challenge, compromised with Drepper,
and surmounted the EGCS challenge, but this time could easily be
different (I'd put a small wager on no effective response and GCC
gradually falling into disuse except for building HURD-based systems).

What's different now?  *Big* business has learned to cooperate
effectively with free software projects, both idiosyncratically (eg,
the Linux kernel) and systematically (via organizations like the
Apache Foundation).

Looking around more widely, the GNU Project is doing nothing effective
to catch up to commerce-friendly free software leaders[4], and
resorting to shooting spitballs at front runners (Linux and LLVM) and
coddling con men ("GNU" Bazaar, whose members, with the presumed
exception of the Invisible Maintainer, have far less allegiance to GNU
ideals than I do).

It's funny.  I could happily live in the GNU System as described in
the Manifesto, plus the Linux kernel.  I could get all of my current
work (with the exception of web publication and reading HTML mail and
*Office docs -- even submission of *Office docs can usually be avoided
by using plain text or PDF and w3.el is sufficient for the browsing I
*need* to do -- including both $DAYJOB and FS support and development)
done with today's versions of the base system, applications, and
libraries listed there.  But I look out at the world, and I see that
others want a lot more.  Some of it (eg, web browsing and publication,
mailing lists, and issue tracking) is convenient for my activities --
but with the exception of GNU Mailman, *none* of that additional
convenience can be achieved with GNU projects.

Somebody needs to figure out whether that last sentence is too true in
general to be ignored.  (Eg, my desktop is XEmacs, so I don't include
GNOME -- surely a convenience to many -- in my list of "conveniences".)
And if so, why it happened and what to do about it.

Hey, GNU!  WTF?!

 > the whole point of being behind GCC is to do what is in our power
 > to push against this happening.  Our power may be small compared to
 > those of the universe, but that has not kept us from making a
 > difference by applying it aimedly and timely enough.

Being behind GCC?  What have you contributed to GCC lately?  Besides
apologia for shooting spitballs at Clang, that is.  If you wanted to
*effectively* support, GCC you'd be ignoring me and posting ideas and
encouragement and maybe even patches on GCC lists instead, no?

The whole thing reminds me of my host country.  "Alas, alack, a 'Lost
Decade' for the Japanese economy.  We must reform!"  Oops, we can now
make that two decades -- and counting, "Abe-nomics" not withstanding
(Band-Aids #1 and #2 have been applied, but the painful surgery has
not been done, and pretty clearly the administration intends to avoid
it).  And now they're shooting spitballs at TPP and justifying "attack
as the best defense" military strategy.

I don't know what to do about either Japan or GNU.  I've presented my
ideas, but (in both cases) they run counter to the natural instincts
and/or principles of the leadership.


Footnotes: 
[1]  The effect may be temporary, of course: since the architecture is
different, the errors and warnings issued are likely to be different.
Long experience with GCC means those are *gone* (I mean, 100%) for
older versions of GCC.  Clang may just be reporting on issues that it
catches that have been present for a decade.  I'll have to go back
and see which does a better job once we're "Clang clean".  However, I
bet that Clang's architecture gives it a lasting advantage here.

[2]  Perhaps the approach taken to trying to cooperate with Linus et
cie. was the best possible, especially given the important differences
on licensing.  I grant those weren't reconcilable by concessions from
the GNU side.

[3]  Well, to be honest I don't care specifically about the compiler.
I do think that a strong GNU (= free software movement) is important,
maybe essential, to a healthy FLOSS community.  I don't think it has
to be *dominant and universal*, though.  Bottom line: if GNU thinks
"owning" compiler space is important to its goals, I'm happy to assume
that's important in my discussion.

[4]  The GNU Project has no entrants in kernels (HURD? give me a
break), scripting languages (Guile? ditto), web servers, or web
frameworks, the areas that scare Microsoft and power Google.  It's
most innovative on the desktop (GNOME), but still way behind the
industry leaders (Microsoft in installations and Apple for overall
design) and facing intense competition even for GNU System desktops
(KDE).  It doesn't have a distribution to speak of, not even a package
manager.  And now its long dominance in system software toolchains
(GCC et cie.) is seriously challenged.

Re: Emacs contributions, C and Lisp

[Stephen J. Turnbull]

Richard Stallman writes:

 > Reciprocity is an aspect of the point of the GNU GPL, but its main
 > point is _defending users' freedom_.

Conceded.

 > With the GNU GPL (and copyleft in general), we make sure that all
 > copies of all versions of our code _respect users' freedom_.

That's a nice euphemism for *dis*respect for the users.  You treat
them like children, fearing they will abuse their freedom by choosing
bondage to proprietary software rather than choosing less capable free
software, or even just saying no to the unique benefits of some
proprietary software.  The contrast with your support for removal of
all restraint on various other kinds of dangerous freedoms (where the
most at risk are real children rather than possibly childish adults)
is striking.

 >     stifling friendly competition is a very bad thing,
 > 
 > By calling LLVM "friendly competition" you misrepresent the issue at
 > stake.  You're wasting your time, asking me to change my mind based on
 > ignoring what's at stake.

I apologize for using the word "friendly"; I should remember that your
sense of humor doesn't extend that far.  But I am not misrepresenting
what's at stake any more than you are -- I'm simply presenting a side
you wish we could ignore.

 > In any case, we are not in a position to stifle LLVM, so you're
 > discussing an imaginary issue.

Then what the heck are you trying to do?  The results are increasing
enmity with the LLVM developers and unhappiness among Emacs
developers.  There are no practical benefits of this policy.  The
claim that a really minor potential attraction of Clang/LLVM will be
erased is surely false.  Eclipse and vim, among others, will take
advantage of it.  So at that point, what you have succeeded in doing
is reducing the attractiveness of Emacs, too.  And that's the end of
the "benefits".

I see no good to come of this.  It's unfortunate that GCC is
prohibited from emulating this particular feature of LLVM, but hey,
why not just suck it up, man?  That's *your* *arbitrary* prohibition,
for reasons that apply specifically to GCC, not to Emacs.

So be arbitrary again, and let Emacs implement completion based on
features provided by free software (that happens to compete with GCC,
but this feature could benefit GCC users, too!)  Or let somebody else
implement the feature, and Emacs collect papers and distribute it.

 >     have we learned nothing from the Great Planning Experiments of
 >     Marx-Lenin-Stalin-ism?
 > 
 > Lots of projects make plans and implement them.  If you equate making
 > plans to Communist dictatorship, you've gone off the deep end.  And
 > that's not to mention that your accusation is so nasty that we
 > shouldn't be on speaking terms.

For heaven's sake, Richard, remember I'm an economist and consider
history.  I *did* not mean the "dictatorships of the proletariat".
Note that those dictatorships have mutated (sometimes non-Communist,
sometimes unreconstructed) but survived.

I *did* mean the "great planning experiments" which did stifle
competition.  OK, you're not in a position to stifle LLVM, but you
certainly wish it didn't exist, don't you?  You wish that all that
effort would flow into GCC, don't you?

But (like the Communist planning organs) GCC is a highly political
community, because the stakes are so great.  GNU has played its part
in that politicization (by delaying implementation of plug-ins and
exportation of useful internal data structures for example), but the
fact that GCC is both an important input and partly shaped by the ISO
standardization process is even more important in its politicization.

Now, the "great experiments" did channel the entire efforts of
(national) communities into allegedly rational plans, and *look what
happened to those communities*.  "Post hoc ergo propter hoc," I know,
but even a fallacy can give true results sometimes.  GCC's decade of
dominance, attracting many of the best and brightest compiler writers,
may have done the same thing to it.

And (as I pointed out to David elsewhere), though GNU has surmounted
challenges from XEmacs, separatists in glibc, and EGCS, LLVM may very
well be different because *big* business and projects producing free
software have learned to cooperate well.  LLVM is by far a stronger
challenge to GNU than anything it has experienced except the Linux
kernel itself.

If LLVM does achieve the kind of dominance that GCC has enjoyed for
the past several years, and (as David fears) it becomes the compiler
of choice for GNU/Linux distros (not to mention Android/iOS/OSX), then
what?  Perhaps at that point you'll look at the situation and decide
GNU needs a competitive compiler suite, and allow plug-ins and export
of internal data structures to compete with (even conform to!) LLVM
APIs.  Perhaps you can deny the very *possibility* of such a future.

But if not, why not *consider* using "license judo" on LLVM?

The idea is to preempt that future by (1) allowing Emacs to support
Clang/LLVM features if that suits the Emacs contributors, (2) removing
the absolute block on "export" and the implied refactoring into
multiple executables (and adding a "free software compatible" API as
you recommended for the kernel modules and GCC plugins, if that is
feasible), (3) encourage the GCC community to adopt Clang/LLVM
architecture, API, and implementations, and (4) watch designs,
implementations, and eventually contributors flow *out* of LLVM and
*into* GCC because of the licenses.  In the end, one of two things is
likely to happen: LLVM withers, or LLVM adopts the GPL in order to
stem the one-way flow of innovations.  Either way GNU (if not GCC)
wins.

Note that I do not claim that there is *no* innovation in recent GCC;
merely that it's (a) slower than it might otherwise be because the
incentive of strong competition has been lacking and specifically
slower than in the newly arisen competition, and (b) less effective
than it might be because the absence of competitors (not to mention
administrative restriction on kinds of innovation) means that bold
experiments are far fewer.  The strategy above of course depends on
there being plenty of innovation in GCC still -- thus by efficiently
copying Clang/LLVM it can catch up where it needs to, and innovation
unique to GCC will then attract both commercial support and volunteer
developers.

Caveat: I don't claim there's no risk to that strategy, nor that it
conforms to your principles (though I think it might).  I do insist
that there is an equally big risk to your strategy, minor annoyances
along the way, and I wonder whether it's worth adhering slavishly to
principle only to see that principle end up in the dustbin of history.

Re: Emacs contributions, C and Lisp

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> David Kastrup writes:
>  > "Stephen J. Turnbull" <stephen@xemacs.org> writes:
>  >> David Kastrup writes:
>  >>> "Stephen J. Turnbull" <stephen@xemacs.org> writes:
>  >>>> David Kastrup writes:
>
>  >>>>> It only weakens the GPL
>
> [big snip and back to David]
>
>  > The GPL can _always_ be enforced by the copyright holder.
>  > Collecting the assignments makes sure that the FSF has the full
>  > ability to enforce
>
> You could have stopped there.  *You* said "weaken the GPL" when what
> you really meant was "weaken the FSF."  OK, conceded.

It weakens the GPL when the FSF applies it to software where it cannot
hope to successfully defend the GPL in court.

> And then your main point:
>
>  > It doesn't weaken the GPL.  It weakens GCC, and with it, the GNU
>  > project.  Part of the strength of the GNU project lies with GCC
>  > setting language and performance standards,
>
> Except that last is now an exaggeration.  Obviously GCC no longer sets
> performance standards,

If you want to believe the summaries of Phoronix' Michael Larabel rather
than his numbers...

> and since the reason I switched my default to Clang was to get better
> error messages, in at least one aspect it seems to be lagging on
> language standard-setting.  (But see discussion below, especially
> fn. 1.)

You are confusing "being helpful with figuring out compliance with C++
standards" with "introducing its own features that are so good that
they are eventually accepted into standards".

GCC has had variable length arrays (also multidimensional) decades
before they made it into C99, and C++ still hasn't caught up.

>  > and it's bad if the most thorough modes we make available with
>  > Emacs are not able to support the GCC dialects.
>
> Now that, as Richard would say, is a misrepresentation.  Of course
> nothing prevents Emacs from supporting GCC dialects, certainly not a
> fiat from Mt. Olympus.  Of course Emacs *should* support them.

Of course Emacs could not possibly hope to support them if its language
parsing relied on Clang.  Which is what this thread is about.

> And if such support was lacking or weak, I think there's some chance
> that Richard would come around and encourage development of that mode.
> Don't you?

How can Richard encourage Clang developers to develop better parsing of
GCC-only features?  I think you overestimate his influence.

> AFAICS, the issue at hand is supporting unique features of free
> software whose licenses do not defend freedom according to Richard.

The issue at hand is relying on Clang as infrastructure for
cross-referencing/completing features in C/C++ modes.  The issue is
_not_ to "support unique features" but rather to have Emacs operations
_depend_ on unique features of Clang.  It's not the first time I've
corrected this statement.

> There are a lot of such licenses, including those used by TeX, Perl,
> Python, Ruby, ncurses, X11, and Apache.  Why doesn't the logic that
> applies to LLVM apply to them too?

Because Emacs does not rely on those to the detriment of GNU-internal
solutions of the GCC calibre and importance.

>  > So in this case GCC and the GNU project were in the situation of
>  > creating a de facto standard.  Once Emacs supports what Clang
>  > provides rather than what GCC provides, not even Emacs will support
>  > any new features of GCC.
>
> Hm.  You've said that twice, now.  This is a leap that I cannot
> follow.  Would you explain?

If Emacs only understands C/C++ by using Clang for its parsing, the
dialects it will support will be those of Clang.  This can't be that
hard to understand.

> I understand the rest of your argument that *compiler users* could
> abandon GCC en masse.  What I don't see is why that would prevent
> Emacs from supporting unique GCC features any more than it prevents
> Emacs from supporting a zillion true niche applications (eg, modes for
> .Xresource and RGB.txt manipulation).

Emacs does not get to choose which language features it supports when it
uses Clang for understanding source files.

> Furthermore, my personal estimate is that GCC's capacity for relevant
> innovation has been sapped by its dominant position and by the focus
> on internal issues that comes with high-stakes standard-setting.  And
> furthermore, refactoring to make internal data structures available to
> cooperating applications has been forbidden for a decade or so.  Just
> where are those "unsupportable" new features going to come from?

There has been no ban on "features".  Instead, the ban has been on
generic infrastructure making it easy to hack up features without
requiring changes to GCC.

Each individual feature can be supported by custom code in GCC.  The
cost of sacrificing modularity and generic interfaces is that
development and deployment of those features is then coupled to upstream
GCC development and its release cycles.

If the purpose of a discussion is to readjust the basis for
decision-making, constant misrepresentation of the salient points is
leading nowhere.  The way to tip a scale is not blowing it up.

> No, at least based on current behavior, you don't want *GCC* to *do*
> anything.  You want *LLVM* to *fail to innovate* in the same way that
> GCC is *prohibited* from innovating.  The effect is the same, but your
> statement suggests that GCC is being defended, when in actual fact
> what is happening here is that Emacs is being commanded to attack LLVM
> (in the sense of "trade war", ie, withdraw cooperation from an
> external entity whose competence threatens a protected local
> industry).

That's total rubbish and you know it.  Emacs is not "being commanded to
attack LLVM".  It is in no position to do anything of that sort.  And it
is most certainly not an "attack" if Emacs developers are told that
Emacs' C/C++ modes should not be equipped with features that can only
work by using Clang internally, consequently restricting the supported
languages and more important language dialects to those of Clang as well
as introducing a permanent dependency on a compiler not under our
control.

> This is only to the detriment of Emacs and LLVM users.

It is to the detriment of Emacs users who don't care about GCC and
ultimately the fate of the GNU project.  Yes.  The GNU project was never
about making everybody happy.  It was about making sure its users have
free software available where they have access to the full sources, and
making sure that this is not just a temporary state.  Whether or not its
users are interested in it.

It turns out that most aren't.  Which is hardly a surprise if you take a
look at the takeup of operating systems like Windows and MacOSX.

That GNU and GNU/Linux nowadays don't look ridiculous in popularity
contests does not mean that this is the metric we now should be
pursuing.  It is a dead end now as much as before.

> That justifies redoubling efforts to innovate in GCC, while respecting
> the self-imposed limitation on acceptable areas for innovation.  It
> doesn't justify "trade war".

It is no war if we decide what we are going to use ourselves as a
critical part of our infrastructure.  There is just no point in pulling
down our defenses in order to drag a pretty wooden horse in.

>  > When put to the choice, we'd rather give up on Android than GNU.
>  > And us having to take that choice at some point of time is what
>  > Clang/LLVM are working towards.
>
> I rather think you overstate the importance of GNU in the thinking of
> LLVM developers.

I think you utterly fail to comprehend that it does not matter at all
what LLVM developers think.  We have to take into account the effect of
their actions, not their motivations.

>
>  > Once Clang/LLVM become the compiler for Android, it is a matter of
>  > time and hipness before it becomes the default compiler for the
>  > Linux kernel.
>  > 
>  > Once the kernel is bootstrapped using Clang/LLVM, it is a matter of
>  > time before the userland of big distributions is compiled using
>  > Clang/LLVM as well, with GCC becoming optional.
>  >
>  > At some point of time, it will become harder to compile the Linux
>  > kernel with GCC out of the box, and bootstrapping a full GNU system
>  > will mean that we need to revert to the HURD, mostly relying on
>  > using Linux drivers.
>
> A sad future history indeed.  And you think that refusal to integrate
> Emacs modes that use Clang to improve the Emacs user experience

To improve the Emacs user experience for those users that are willing to
forego GCC as compiler

> is the finger in the dike that will save Holland, er, the GNU Project?

The GNU project started by sticking up a finger in a sea of propretary
software.  That's where it started, and it will end when nobody is
willing to stick up for it.  Complacency and resignation are not the way
to anything.

> The obvious fact is that GNU can't stop the process you've described
> -- it all happens outside of GNU!

Which means that if we cannot stop the processes outside of GNU, we can
at least keep things as they should be inside of GNU.  That is nothing
that can be taken away from us unless we give it up voluntarily.

> Specifically, it wants to be universal but denies the clearly
> expressed aspirations and preferences of the rest of the
> free-software-producing world and their users.

The GNU project never aspired "to be universal".  It has always been an
enclave because of a world that has gone mad in its laws and business
processes.  The FSF has a mission to make the world less mad, and it
would be great if at some point of time the GNU project would become
unnecessary because software was free anyway.  But making the world less
mad and maintaining GNU are two separate endeavors, and they will not
become the same in our life time.

> And now, it's denying the aspirations of some of its own members,
> merely to spite those with different beliefs about defending freedom.

Shrug.  What do you hope to achieve with inflammatory spins like that?
Trying to work with those resources one has under one's own control and
reducing external dependencies where feasible is Economics 101.  If I
tend to my own orchard instead of buying apples at the market, is that
merely to spite those merchants at the market with different beliefs
about planting trees?

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> Richard Stallman writes:
>
>  > Reciprocity is an aspect of the point of the GNU GPL, but its main
>  > point is _defending users' freedom_.
>
> Conceded.
>
>  > With the GNU GPL (and copyleft in general), we make sure that all
>  > copies of all versions of our code _respect users' freedom_.
>
> That's a nice euphemism for *dis*respect for the users.  You treat
> them like children,

Are we still talking about a world where the dominant government form is
representative democracy (in the U.S., the self-declared bastion of
freedom, even filtered through an additional layer of electors) rather
than anarchy?  The closest you can get to an actual democracy nowadays
probably is Switzerland, and it's not like that kind of system would be
popular elsewhere.  And it's still to a good degree based on
representatives.

A large part of public life is regulated by laws, conventions, taxation
simply because people cannot be bothered to think about the consequences
of their actions.

The GNU project does not treat its users like children, but it also does
not bet its fate on everybody behaving in a responsible manner by
default.  If it did, there would not have been the GPL to start with.
Or even the necessity to start the GNU project at all.

> fearing they will abuse their freedom by choosing bondage to
> proprietary software rather than choosing less capable free software,
> or even just saying no to the unique benefits of some proprietary
> software.

The GNU project is not democratic.  The FSF sets and pursues its
policies and everyone is free to join or not for whatever reason he
wants.  That does not mean that the FSF is under any obligation to cater
for any particular reason.

Indeed, it has remained remarkably constant in its original goals and
aims even though users of GNU software have easily increased more than a
millionfold from GNU's starts.

Of course, this attracts attention leading to alternative options with
watered-down guarantees.  And their availability in turn impacts the
adoption numbers of GNU.

Of course, an economist will cry "_now_ is the time to sell out, it
doesn't get any better than that".  Which actually tends to be the end
result of almost every revolution, sometimes by eating its leaders
alive, sometimes waiting for them to fade from life or significance
first.

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[Stephen J. Turnbull]

David Kastrup writes:

A lot of misrepresentations.  Two are important to Emacs:

 > And it is most certainly not an "attack" if Emacs developers are
 > told that Emacs' C/C++ modes should not be equipped with features
 > that can only work by using Clang internally, consequently
 > restricting the supported languages

This was the fifth time in your post you've made the unsupported claim
that use of Clang involves restricting the languages supported by
Emacs to those supported by Clang.  Please document people advocating
that no other technology than Clang should be used to implement "smart
completion".  Surely Óscar advocates Clang (or other AST-producing
front end) as the most accurate way to do that, but I didn't see him
(or anyone else) anywhere say that no other techniques should be
allowed.

In any case, Richard's argument nowhere depends on *exclusive* use of
Clang.  He simply doesn't want unique features of Clang supported, no
matter how good Emacs's support for GCC is.

 > The GNU project was never about making everybody happy.  It was
 > about making sure its users have free software available where they
 > have access to the full sources, and making sure that this is not
 > just a temporary state.

Um, no, that clearly is not what the GNU Project is about.  It is not
good enough to merely provide some free software, even a complete
(whatever that means) system built from free software.  The goal of
the GNU Project quite clearly is the same as the FSF's: the
elimination of proprietary software, starting with all derivatives of
GNU software.  That is clear from the choice of license.

The goal you propose here is the one espoused by permissive license
advocates.

Re: Emacs contributions, C and Lisp

[David Kastrup]

"Stephen J. Turnbull" <turnbull@sk.tsukuba.ac.jp> writes:

> David Kastrup writes:
>
> A lot of misrepresentations.  Two are important to Emacs:
>
>  > And it is most certainly not an "attack" if Emacs developers are
>  > told that Emacs' C/C++ modes should not be equipped with features
>  > that can only work by using Clang internally, consequently
>  > restricting the supported languages
>
> This was the fifth time in your post you've made the unsupported claim
> that use of Clang involves restricting the languages supported by
> Emacs to those supported by Clang.

Language support implemented by letting Emacs call an external parser.
That's what this thread is about.  If the only external parser that can
be called for this purpose is Clang, the languages and dialects
supported by this feature will be those implemented by Clang.

> Please document people advocating that no other technology than Clang
> should be used to implement "smart completion".

We are not talking "should be used".  We are talking "can currently be
used" for functionality that has apparently already slipped into ELPA.

> Surely Óscar advocates Clang (or other AST-producing front end) as the
> most accurate way to do that, but I didn't see him (or anyone else)
> anywhere say that no other techniques should be allowed.

If it is "the most accurate" way, how is that _not_ promoting the use of
Clang for Emacs functionality.

> In any case, Richard's argument nowhere depends on *exclusive* use of
> Clang.  He simply doesn't want unique features of Clang supported, no
> matter how good Emacs's support for GCC is.

Sigh.  I don't know _how_ often I have to repeat this.  The problem is
not "supporting features of Clang", the problem is _requiring_ features
of Clang for supporting features of Emacs.

We don't want Emacs features that _depend_ on Clang.

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[Óscar Fuentes]

David Kastrup <dak@gnu.org> writes:

[snip]

>> In any case, Richard's argument nowhere depends on *exclusive* use of
>> Clang.  He simply doesn't want unique features of Clang supported, no
>> matter how good Emacs's support for GCC is.
>
> Sigh.  I don't know _how_ often I have to repeat this.  The problem is
> not "supporting features of Clang", the problem is _requiring_ features
> of Clang for supporting features of Emacs.
>
> We don't want Emacs features that _depend_ on Clang.

It's clear that that is not RMS intention. Otherwise, for the specific
case of C++ smart completion, CEDET could default to its own parser and
provide Clang as an option. But that's not allowed by RMS. He doesn't
want Clang on Emacs unless *GCC* provides the same features. Which is an
unfair scenario for GCC because, contrary to Clang, it wasn't intended
to provide those features, and past attempts to steer its development
towards those goals were rejected by the same person who now spurs them
to match Clang's library-like features :-/

Re: Emacs contributions, C and Lisp

[Stephen J. Turnbull]

David Kastrup writes:

 > Sigh.  I don't know _how_ often I have to repeat this.  The problem
 > is not "supporting features of Clang", the problem is _requiring_
 > features of Clang for supporting features of Emacs.

No, the real problem is that our English dialects differ.  As far as I
can tell from your "corrections" I use "support feature" where you use
"require feature", I use "implement feature" where you use "support
feature", but we mean almost exactly the same thing.

Re: Emacs contributions, C and Lisp

[David Kastrup]

Óscar Fuentes <ofv@wanadoo.es> writes:

> David Kastrup <dak@gnu.org> writes:
>
> [snip]
>
>>> In any case, Richard's argument nowhere depends on *exclusive* use of
>>> Clang.  He simply doesn't want unique features of Clang supported, no
>>> matter how good Emacs's support for GCC is.
>>
>> Sigh.  I don't know _how_ often I have to repeat this.  The problem is
>> not "supporting features of Clang", the problem is _requiring_ features
>> of Clang for supporting features of Emacs.
>>
>> We don't want Emacs features that _depend_ on Clang.
>
> It's clear that that is not RMS intention. Otherwise, for the specific
> case of C++ smart completion, CEDET could default to its own parser and
> provide Clang as an option.

That would be native-parser supported editing _only_ for Clang, not for
GCC.

> But that's not allowed by RMS. He doesn't want Clang on Emacs unless
> *GCC* provides the same features. Which is an unfair scenario for GCC
> because, contrary to Clang, it wasn't intended to provide those
> features, and past attempts to steer its development towards those
> goals were rejected by the same person who now spurs them to match
> Clang's library-like features :-/

He doesn't spur them to match "library-like features".  At the moment
smart completion is what is involved here specifically.  That's quite
not the same.  Personally I think that the time delay associated with
adding features individually is prohibitively large.

But it's impossible to get things in perspective if everybody insists on
misrepresenting Richard and ascribing absurdities to him.

If you refuse to see the issues that are to be balanced, you can't
complain when your input on choosing the balance is discarded as
worthless.

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> David Kastrup writes:
>
>  > Sigh.  I don't know _how_ often I have to repeat this.  The problem
>  > is not "supporting features of Clang", the problem is _requiring_
>  > features of Clang for supporting features of Emacs.
>
> No, the real problem is that our English dialects differ.  As far as I
> can tell from your "corrections" I use "support feature" where you use
> "require feature", I use "implement feature" where you use "support
> feature", but we mean almost exactly the same thing.

Quite unlikely.  "dialects" don't differ to a degree where causations
are diametrically opposed.

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[Óscar Fuentes]

David Kastrup <dak@gnu.org> writes:

> Óscar Fuentes <ofv@wanadoo.es> writes:
>
>> David Kastrup <dak@gnu.org> writes:
>>
>> [snip]
>>
>>>> In any case, Richard's argument nowhere depends on *exclusive* use of
>>>> Clang.  He simply doesn't want unique features of Clang supported, no
>>>> matter how good Emacs's support for GCC is.
>>>
>>> Sigh.  I don't know _how_ often I have to repeat this.  The problem is
>>> not "supporting features of Clang", the problem is _requiring_ features
>>> of Clang for supporting features of Emacs.
>>>
>>> We don't want Emacs features that _depend_ on Clang.
>>
>> It's clear that that is not RMS intention. Otherwise, for the specific
>> case of C++ smart completion, CEDET could default to its own parser and
>> provide Clang as an option.
>
> That would be native-parser supported editing _only_ for Clang, not for
> GCC.

Exactly. So this is not about Emacs features depending on Clang, but
about GCC not having Clang features. Unless you say that "C++ parsing"
and "C++ native parsing" are two different features. Emacs is just a
sacrificial lamb.

>> But that's not allowed by RMS. He doesn't want Clang on Emacs unless
>> *GCC* provides the same features. Which is an unfair scenario for GCC
>> because, contrary to Clang, it wasn't intended to provide those
>> features, and past attempts to steer its development towards those
>> goals were rejected by the same person who now spurs them to match
>> Clang's library-like features :-/
>
> He doesn't spur them to match "library-like features".  At the moment
> smart completion is what is involved here specifically.  That's quite
> not the same.

Smart completion is the tip of the iceberg. If GCC limits itself to
providing only smart completion, other features that depend on having
access to parsing and semantic information will still be missing from
GCC.

>  Personally I think that the time delay associated with
> adding features individually is prohibitively large.
>
> But it's impossible to get things in perspective if everybody insists on
> misrepresenting Richard and ascribing absurdities to him.
>
> If you refuse to see the issues that are to be balanced, you can't
> complain when your input on choosing the balance is discarded as
> worthless.

David, with all due respect: I think that people like you and RMS who
don't know the issue at all from the user's POV (and even loudly despise
C++) have no right for telling anyone that they do not understand the
matter at hand. As a lifetime C++ programmer who follows GCC, Emacs and
Clang communities since a very long time, it is obvious to me that RMS
stance will keep GCC and Emacs back for no gain on any aspect, including
user's freedom. It is possible that I'm missing some nuances of your
political POV, but you are uninterested on the whole technical landscape
and its implications. And please don't say that technical issues are
secondary here, because understanding those is fundamental for choosing
the correct policy. You are like catholic priests dictating how people
should deal with sexuality. See what they achieve.

Re: Emacs contributions, C and Lisp

[Florian Weimer]

* Eric S. Raymond:

> LLVM got off the ground because GCC, by policy, refused to provide
> interfaces that some toolmakers wanted.  Consequently, those hackers
> exercised their freedom by going around GCC rather than through it.

There might be some truth to that, but that the main LLVM contributor
is deeply unhappy about the GPL, version 3, turned out a more
important factor IMHO.  I'm not following LLVM development closely.
Are there really that many contributions from people who use Clang for
building IDEs?  (Rafael's work probably doesn't depend that much on
modularity.)  I just don't believe in claims along the lines of "if
you change X, you'll attract new contributors, producing useful
features for your users".  That almost never happens, whether it's a
good policy from a software freedom point of view or not.

And it certainly wasn't downright refusal on the GCC part, it's just
that it was (and still is) difficult to get an annotated AST
approximating the source code out of the C++ front end.

Anyway, it is difficult to pitch GCC against LLVM in terms of software
freedom because it used to be common practice among GCC developers to
sell themselves into slavery and work on what are, for all intents and
purposes, proprietary forks, and the practice probably continues today
to a lesser extent.

Re: Emacs contributions, C and Lisp

[Stephen J. Turnbull]

David Kastrup writes:
 > "Stephen J. Turnbull" <stephen@xemacs.org> writes:
 > 
 > > David Kastrup writes:
 > >
 > >  > Sigh.  I don't know _how_ often I have to repeat this.  The problem
 > >  > is not "supporting features of Clang", the problem is _requiring_
 > >  > features of Clang for supporting features of Emacs.
 > >
 > > No, the real problem is that our English dialects differ.  As far as I
 > > can tell from your "corrections" I use "support feature" where you use
 > > "require feature", I use "implement feature" where you use "support
 > > feature", but we mean almost exactly the same thing.
 > 
 > Quite unlikely.  "dialects" don't differ to a degree where causations
 > are diametrically opposed.

As I wrote above and you quote, IMO causation is not opposed at all.
We simply use certain crucial words ("support", "require", "feature")
in very different ways when speaking of relationships of one software
program to another.  It isn't easy for me, but if you will concede
that basic equivalence of causation, I'll try to use your dialect's
terminology to express it in the future.

Re: Emacs contributions, C and Lisp

[David Kastrup]

Óscar Fuentes <ofv@wanadoo.es> writes:

> David Kastrup <dak@gnu.org> writes:
>
>> But it's impossible to get things in perspective if everybody insists on
>> misrepresenting Richard and ascribing absurdities to him.
>>
>> If you refuse to see the issues that are to be balanced, you can't
>> complain when your input on choosing the balance is discarded as
>> worthless.
>
> David, with all due respect: I think that people like you and RMS who
> don't know the issue at all from the user's POV (and even loudly despise
> C++) have no right for telling anyone that they do not understand the
> matter at hand. As a lifetime C++ programmer who follows GCC, Emacs and
> Clang communities since a very long time,

I don't think that there is any further point to answer you if you
insist on considering everybody with a different viewpoint as clueless.
You may have been following the GCC and Emacs communities since a very
long time, but Richard has been founding and leading them as well as the
GNU project exactly because he has been seeing the issue from the users'
point of view.  It is an aspect of users he is catering for that you
refuse to see.  That is perfectly within your rights.

But you cannot meaningfully contribute feedback for decisions that are
focused about maintaining the longterm viability of free software if you
refuse to consider the reasoning that lead to the existence of the GNU
project in the first place.

> it is obvious to me that RMS stance will keep GCC and Emacs back for
> no gain on any aspect, including user's freedom.

The user is free to install software against the long-term interest of
the GNU project without the blessing of GNU infrastructure like its
servers and other official distribution channels.  His freedom is not
hampered by us not lying about what is and what is not in GNU's
long-term interests.

Making Emacs depend on Clang for features clearly is not.

> It is possible that I'm missing some nuances of your political POV,
> but you are uninterested on the whole technical landscape and its
> implications. And please don't say that technical issues are secondary
> here, because understanding those is fundamental for choosing the
> correct policy. You are like catholic priests dictating how people
> should deal with sexuality. See what they achieve.

You manage to stoop to insinuations of child molestation.  Impressive.
It is probably my fault for not cutting off the "discussion" sooner, but
then I lack Richard's experience about how to deal with getting ranted
at.  I am sure that there are other people who will just love to discuss
matters with you in that style, but I'm out.

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[Thien-Thi Nguyen]

() David Kastrup <dak@gnu.org>
() Tue, 04 Mar 2014 21:32:04 +0100

   > No, the real problem is that our English dialects differ.

   Quite unlikely.  "dialects" don't differ to a degree where
   causations are diametrically opposed.

Well, to be pedantic, "support" and "require" both do not have
causal connotations.  That is, in both constructs:

  A supports B.
  A requires B.

one cannot generally infer that A "causes" B (or vice versa).
That A is impacted (perhaps negatively) by B (or its lack) is
another matter entirely.

I think a lot of misunderstanding arises when we forget this.

-- 
Thien-Thi Nguyen
   GPG key: 4C807502
   (if you're human and you know it)
      read my lisp: (responsep (questions 'technical)
                               (not (via 'mailing-list)))
                     => nil

Re: Emacs contributions, C and Lisp

[John Yates]

[-- Attachment #1: Type: text/plain, Size: 822 bytes --]

On Wed, Mar 5, 2014 at 12:20 AM, David Kastrup <dak@gnu.org> wrote:

> Óscar Fuentes <ofv@wanadoo.es> writes:
> >  You are like catholic priests dictating how people
> > should deal with sexuality. See what they achieve.
>
> You manage to stoop to insinuations of child molestation.


David,

If you reread Oscar's posting carefully I hope that you will see that he
did not insinuate child molestation.  That you had such a response speaks
volumes regarding the poor image the catholic priesthood has currently in
the public eye (and perhaps a tiny bit regarding your willingness to
demonize Oscar).  My interpretation was that, at least in North America and
Western Europe, catholic teachings on sexuality are largely ignored (e.g.
pre-marital sex, birth control, divorce, homosexuality, etc).

/john

[-- Attachment #2: Type: text/html, Size: 1259 bytes --]

Re: Emacs contributions, C and Lisp

[Óscar Fuentes]

David Kastrup <dak@gnu.org> writes:

>> David, with all due respect: I think that people like you and RMS who
>> don't know the issue at all from the user's POV (and even loudly despise
>> C++) have no right for telling anyone that they do not understand the
>> matter at hand. As a lifetime C++ programmer who follows GCC, Emacs and
>> Clang communities since a very long time,
>
> I don't think that there is any further point to answer you if you
> insist on considering everybody with a different viewpoint as clueless.

Handwaving. You are proud of keeping away from C++. But at the same you
feel entitled to stablish policies that depend on how it is practiced,
while sneering at the insiders that say that you are wrong all the way.

Something for your consideration, that is self-evident for anyone
working the C/C++ compiler scene but that you are missing completely: it
doesn't make any sense to worry about non-free software using GCC as a
front-end/back-end/whatever because choosing Clang for that is a
no-brainer.

[snip]

> Making Emacs depend on Clang for features clearly is not.

Here you go again with "making Emacs depend on Clang." You simply don't
want to listen.

>> You are like catholic priests dictating how people
>> should deal with sexuality. See what they achieve.
>
> You manage to stoop to insinuations of child molestation.  Impressive.

If you don't want to continue this discussion, it is fine with me. But
trying to sidestep the discussion by depicting me as if I were
slandering you with outrageous charges (when it is obvious that's not
the case) is something I wouldn't expect from you. But it is a technique
I've seen RMS to use often, so I guess you are taking him as a model.

[snip]

Re: Emacs contributions, C and Lisp

[David Kastrup]

John Yates <john@yates-sheets.org> writes:

> On Wed, Mar 5, 2014 at 12:20 AM, David Kastrup <dak@gnu.org> wrote:
>
>> Óscar Fuentes <ofv@wanadoo.es> writes:
>> >  You are like catholic priests dictating how people
>> > should deal with sexuality. See what they achieve.
>>
>> You manage to stoop to insinuations of child molestation.
>
>
> David,
>
> If you reread Oscar's posting carefully I hope that you will see that he
> did not insinuate child molestation.  That you had such a response speaks
> volumes regarding the poor image the catholic priesthood has currently in
> the public eye (and perhaps a tiny bit regarding your willingness to
> demonize Oscar).  My interpretation was that, at least in North America and
> Western Europe, catholic teachings on sexuality are largely ignored (e.g.
> pre-marital sex, birth control, divorce, homosexuality, etc).

Your interpretation of "largely ignored by the public" does not fit his
attack since we are talking about _inner_ effects of the policies of the
GNU project, not their effect on the public.  It also does not fit his
attack since the priests would not fare any better than "largely being
ignored" if they tried not to teach their church's morals at all.

Now it may well be that Óscar does not bother to think through what he
writes.  The end result is the same: as he does not consider either
treating his discussion partners or their points with basic respect,
there is nothing to be gained by submitting oneself to his abuse.

His conclusions are foregone, and the longer one attempts to reason with
him, the lower his esteem of people disagreeing with him gets and the
more abusive he becomes.

If you actually stoop low enough to interpret his "analogy", the
conclusion you end up with is that he claims that neither Richard nor
I know what programming and/or free software is like while talking about
it.

I am not interested in the next escalation.  The current one is more
than pathetic enough.

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 395 bytes --]

On 03/05/2014 06:07 AM, David Kastrup wrote:
> If you actually stoop low enough to interpret his "analogy", the
> conclusion you end up with is that he claims that neither Richard nor
> I know what programming and/or free software is like while talking about
> it.

The analogy is perfectly applicable when it comes to modern C++
programming, and you've made a point of eschewing C++.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 901 bytes --]

Re: Emacs contributions, C and Lisp

[David Kastrup]

Daniel Colascione <dancol@dancol.org> writes:

> On 03/05/2014 06:07 AM, David Kastrup wrote:
>> If you actually stoop low enough to interpret his "analogy", the
>> conclusion you end up with is that he claims that neither Richard nor
>> I know what programming and/or free software is like while talking about
>> it.
>
> The analogy is perfectly applicable when it comes to modern C++
> programming, and you've made a point of eschewing C++.

If you'd bother checking the bug trackers, you'd find that I tracked
down and reported several code generation problems with C++ to GCC in
the last years.  One of them was actually critical enough that it
contributed to delaying a scheduled Fedora release.

It's not all that surprising since I am fulltime project lead for GNU
LilyPond which has a C++ core.  You can probably guess who wrote the
patches to the LilyPond code base in order to make it to compile under
Clang.

May I kindly suggest that you pick another theme for your ad hominem
attacks in order to lend your arguments more weight?  It would appear
that they are in sore need of it.

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

     > With the GNU GPL (and copyleft in general), we make sure that all
     > copies of all versions of our code _respect users' freedom_.

    That's a nice euphemism for *dis*respect for the users.  You treat
    them like children, fearing they will abuse their freedom by choosing
    bondage to proprietary software rather than choosing less capable free
    software, or even just saying no to the unique benefits of some
    proprietary software.

Choosing proprietary software rather than less convenient free
software is something that users of all ages regularly do.  When I am
concerned that users might fall prey to proprietary extended versions
of GCC, I am treating them like real adults with real adults'
weaknesses.

I have taken measures to prevent proprietary extended versions of GCC
from existing.  If they don't exist, people don't fall prey to them.

What's significant about this point is that it shows that your
disagreement is not really with this specific decision about Emacs.
Rather, you're against the broader goal which this specific decision
is meant to achieve.  What bothers you is not the possibility that
this Emacs decision might fail, but that it might succeed.

     > By calling LLVM "friendly competition" you misrepresent the issue at
     > stake.  You're wasting your time, asking me to change my mind based on
     > ignoring what's at stake.

    I apologize for using the word "friendly"; I should remember that your
    sense of humor doesn't extend that far.

I am concerned with the serious distortion, not with the veil of humor
that was meant to augment its misleading effect.

You oppose some basic goals of the GNU Project; you are trying to
interfere with our efforts to achieve them.  One way you do this is by
attacking decisions about how to implement them, claiming the decision
will backfire.

Every strategic decision has an upside and a downside.  You exaggerate
the downside and downplay the upside, and thus "prove" that it will
backfire.  But that is not a proof, it is just spin.

Since nobody can see the future, such decisions are judgment calls.  I
don't trust your judgment about how to achieve our goals because you
don't want to achieve them.

I discuss my strategy decisions with people I trust, people who
understand and agree with the broader basic decisions of the GNU
Project.  When they say I am making a mistake, I pay attention,
because I know they aim for the same kind of success.  When you make
the same claim, I discount it because you don't want us to achieve
that success.  It may well be that you call a decision a mistake
precisely because it would achieve our goals.

I've announced my decision on the issue of Emacs and clang.  I'm
willing to explain the reasons for it in response to serious questions
from people who support the goal.  Your arguments, which oppose more
basic points, belong on gnu-misc-discuss, not here.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I am not going to change my decision because of minor bad consequences
or apparent inconsistencies, because those are less important than
the what the decision aims to achieve.

Maybe it is possible to improve some details of the decision.  If so,
I don't reject that out of hand.  However, I am going to think long
and hard to make sure a proposed change does not reintroduce the
problem I am trying to exclude.

For instance, suppose a feature is implemented to use CEDET.  If so,
why would we want code to implement the same feature using Clang, even
as an option?  If no one would ever use that option, installing it is
a waste of time and a confusion for the users; we are better off
without it.  However, if there is a reason people might see an
advantage in using that option, evidently its use of Clang constitutes
a real problem.

Let's implement the change with CEDET, if that works well.
If it can work better using a real compiler, let's use GCC,
not Clang.

    unfair scenario for GCC because, contrary to Clang, it wasn't intended
    to provide those features,

The only sort of "fairness to GCC" that matters is to promote its use
now, so that it can promote copyleft on compilers now, so that it can
discourage nonfree compilers now.

			       and past attempts to steer its development
    towards those goals were rejected by the same person who now spurs them
    to match Clang's library-like features :-/

In the past, I made decisions so as to promote free compilers and
prevent nonfree compilers from displacing them.  I do that now, and
will do so in the future.  If the methods I use today differ from the
methods I used in the past, that should not be shocking.

The contrast between past and present policies, even if ironic,
tragic, or whatever, is no argument for anything.  Whenever two things
differ, an incomplete or misleading description of them can make that
difference look like a horrible "inconsistency" than absolutely must
be corrected.  One must learn to disregard that sort of argument when
it pops up.

I am not infallible, but that's no reason I should not do my best.  I
will continue to make decisions in accord with the goals of the GNU
Project.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[Stefan Monnier]

> I have taken measures to prevent proprietary extended versions of GCC
> from existing.  If they don't exist, people don't fall prey to them.

But if as a result of those measures, all development moves from GCC to
Clang/LLVM, this will be a pyrrhic victory :-(


        Stefan

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    But if as a result of those measures, all development moves from GCC to
    Clang/LLVM, this will be a pyrrhic victory :-(

That's what life is like.  If you fight, you might lose.  If you
surrender, you lose for certain.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[Jambunathan K]

Richard Stallman <rms@gnu.org> writes:

> For instance, suppose a feature is implemented to use CEDET.

If I look back at some of David Engster's post I realize that

1. CEDET has a C++ parser.  (Not of the hand-written sort that can do
   pure magic but of some down-to-earth variety that gets the job done.)

2. It has some modules that interfaces with Clang.

3. For it's completion features to improve, it's "database" support has
   to improve.

I would like to hear some specifics that a GSoC-like student would be
excited to hear about.

I am interested in pure technicalities and some posts or articles that
articulate the hole and some starting points for how that hole may be
filled.

> If no one would ever use that option

IME, there are two sorts of programmers.

1. Those who write entire modules.
   
2. Those who iherit codebases and try to gain insights in to it
   (invariably when the original authors have moved on to other
   contracts or companies.)

The programmers of first variety have little use for refactoring tools.
They can do better than what a refactoring tool can do or they do some
thing that refactoring things can never hope to do.

The programmers of the second sort - the corporate programmers - often
have access to commercial editors - superior to Emacs for task at hand -
with site-wide licenses.  (i.e., Emacs is virtually non-existent save
for minor exceptions.)

There are programmers who neither fall squarely in (1) or (2) but
vaguely fall close to the (2)-end of the spectrum.

I have always felt that Emacs felt seriously short in the "code insight"
space.  Lots of adhoc tools here and there, but anything which can be
"solely relied upon" for the particularly situation in.

In short, I want to see some improvements in CEDET.  I may even take up
some of the improvements on my own shoulders.

Re: Emacs contributions, C and Lisp

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 4761 bytes --]

On 03/10/2014 07:40 PM, Richard Stallman wrote:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
>     But if as a result of those measures, all development moves from GCC to
>     Clang/LLVM, this will be a pyrrhic victory :-(
> 
> That's what life is like.  If you fight, you might lose.  If you
> surrender, you lose for certain.

Suicidal charges on entrenched positions make for great drama, but poor
results; it's better to run and fight another day. Many smart,
experienced people (far more so than I am) have made this point, and it
might be helpful not to immediately dismiss [1] their advice

The GNU project found early success in creating tools that plugged into
non-free ecosystems. GNU tools were better than the proprietary
alternatives, so it's natural that people chose to use them. Users of
these tools found the FSF's political message, agreed with it, wrote
more free software, and proselytized. Eventually, there was enough free
software to build a whole operating system. Free software would have
never achieved its present success if the GNU project had built an
incompatible operating system and stocked it with inferior tools. If it
had, GNU would have been another Plan 9, and we'd have been stuck with
far more proprietary software.

Today, there's a new ecosystem. Clang isn't really a program that
competes with GCC. Instead, it's an entire self-contained universe of
development tools, one that includes a compiler and linker, of course,
but also a debugger, code formatter, static analyzer, JIT system, and
various source-to-source translators, indexers, and IDEs. The ecosystem
approach to toolchain development has tremendous technical advantages
and is a real advance in the state of the art. It's an idea, not a
program; you can fight it no better than Canute could fight the tide.

This new ecosystem is actually a tremendous opportunity for copyleft
free software. Right now, the LLVM world is full of defenseless
permissively-licensed programs, and proprietary components (e.g., GPU
compilers) are encroaching upon it. This situation is unfortunate, but
we can salvage it.

Instead of decrying Clang, we should integrate Emacs and GCC into it as
tightly as possible. Entice users with technical superiority, then help
them understand the other benefits of using software that defend their
freedoms. Start the same positive feedback loop that helped GNU succeed
in the first place. While this strategy will temporarily allow
proprietary software makers to benefit from the work invested in free
software, free components will gradually supplant proprietary ones. The
same logic that applies to competing programs in general apples in the
special case of interoperating LLVM-based components. After free
software fills all the important niches in this ecosystem, users will
once again be spared the choice between free and useful software.

If the FSF pursues its current strategy, however, and current trends
continue, the LLVM world and the traditional one will drift apart, and
in a few years, users will face this choice with increasing frequency.
Do you really think users are going to forgo useful, gratis features
because you tell them they're being naughty? Do you really think
developers will choose inferior tools, and thereby work harder, die
sooner, and provide less for their children all because you scold them
for not following some  moral principle that, from their perspective,
produces only acrimony and missing features? In the past, the GNU
project did an excellent job of demonstrating not only the moral, but
also the practical benefits of free software. Right now, you're doing a
poor job of demonstrating how it's in the interest of developers to
choose copyleft software and doing a disservice to your own cause.


[1] I want to preempt the charge of working to undermine free software.
You should assume that we're all working in good faith. On numerous
occasions (e.g., <E1WN5YT-0003gl-Eh@fencepost.gnu.org>), you've argued
those who disagreed with your decisions could be doing so only out of a
secret desire to undermine free software. Statements like "I don't trust
your judgment about how to achieve our goals because you don't want to
achieve them." are circular nonsense. Supporting the goal of ubiquitous
free software is not synonymous with supporting your particular strategy
for achieving it, and frankly, it's disturbing that you think that it
is. Leaders who adopt this sort of attitude seldom end up on the right
side of history.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 901 bytes --]

Re: Emacs contributions, C and Lisp

[Stephen J. Turnbull]

Richard Stallman writes:

 > Your arguments, which oppose more basic points, belong on
 > gnu-misc-discuss, not here.

Point taken.  I will, however, correct some blatant misstatements.

 > Rather, you're against the broader goal which this specific decision
 > is meant to achieve.

This is false.

 > What bothers you is not the possibility that this Emacs decision
 > might fail, but that it might succeed.

That's also false.

 > One way you do this is by attacking decisions about how to
 > implement them, claiming the decision will backfire.

That's not my claim, assuming that by "backfire" you mean Stefan's
worry about a Pyrrhic victory.

Regards,

Re: Emacs contributions, C and Lisp

[Thien-Thi Nguyen]

[-- Attachment #1: Type: text/plain, Size: 811 bytes --]

() Richard Stallman <rms@gnu.org>
() Mon, 10 Mar 2014 22:40:19 -0400

       But if as a result of those measures, all development moves from
       GCC to Clang/LLVM, this will be a pyrrhic victory :-(

   That's what life is like.  If you fight, you might lose.  If you
   surrender, you lose for certain.

However, if you command concurrent efforts, you may withdraw from some
fronts in order to push "forward" on other fronts.  I think the current
situation w/ GCC reflects a "one-eye victory" (to use a Go analogy),
which basically means that what is defended is not guaranteed to live.

--
Thien-Thi Nguyen
   GPG key: 4C807502
   (if you're human and you know it)
      read my lisp: (responsep (questions 'technical)
                               (not (via 'mailing-list)))
                     => nil

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: Emacs contributions, C and Lisp

[Jambunathan K]

Daniel Colascione <dancol@dancol.org> writes:

> Do you really think developers will choose inferior tools, and thereby
> work harder, die sooner, and provide less for their children all

With a single sweeping statement you have overlooked the staunch
loyalist, the staunch detractor and all the in betweens.  Even in this
very thread I can categorize people and their position in to these three
buckets with little or no effort.

As an Emacs user and a contributor, it is best to articulate how you are
personally impacted by a decision that GNU project makes.  You have
little stake in whether GNU succeeds or not.  (When I am saying this, I
assume that you don't confuse your wish for GNU's success to mean that
you have pledged your personal resources for it's success).

I am sure you realize that FSF folks have a belly.  Their income is from
the work that they do.

If you feel strongly that GNU project is wrong, the best way to do it is
re-consider your donation - in terms of code or money or voluntary
involvement in the community.

Unless you get your foot in to the FSF or GNU's board-room, it is
unlikely that your arguments in this list will make a significant
impact.

Remember the old maxim:

The fox that fattens is the one that is fed.  Keep feeding GNU.  It will
continue to fatten irrespective of whether some other beast is fattening
or not.

Re: Emacs contributions, C and Lisp

[Jambunathan K]

David Kastrup <dak@gnu.org> writes:

> "dialects"

I find the way you construct your sentences - the way the words are
stringed together - is a bit "atypical".  Not wrong.  But atypical.
(Instead of "atypical" I could have used "original". But I have some
serious reservations).

Mr. Turnbull is a keen observer.  He did well by shifting his attention
away from moot points to the specific words and sentences used to
support it.

> causations

Let there be no doubt.  Your use of causation threw me off as well.

The Trail leading to a Mountain doesn't cause the Mountain.

Water or Nutrients don't cause a Tree.

Instead of saying "Causation", I would have really liked to see you
articulate your position as:

1. We want a trail here.
2. We don't want a trail here.
3. We don't want to fill their water tanks.
4. Our water first goes to our taps before it hits the neighbour's.

Essentially you are not talking of causation per-se, but conditions that
favor or hinder a particular result.  If you want another word, I would
suggest something along the lines of "catalyst".

1. This work will catalyze our effort.
2. This work will catalyze their effort.

You get the idea.

Re: Emacs contributions, C and Lisp

[Stefan Monnier]

>     But if as a result of those measures, all development moves from GCC to
>     Clang/LLVM, this will be a pyrrhic victory :-(
> That's what life is like.  If you fight, you might lose.  If you
> surrender, you lose for certain.

No.  There can be other measures that don't lead to a loss and don't
lead to a pyrrhic victory either.

But I think you'd first need to define what you'd consider as a loss.
AFAICT, you seem to consider "Using GCC in a proprietary product" as
a loss.  That's only true if the alternative was "making that product
Free".  If the alternative is "Using Clang in a proprietary product"
then I think "Using GCC in a proprietary product" is definitely not
a loss (especially if the GPLv3 works as it should, making it possible
for the user to replace/fix/improve the GCC part of the proprietary
product).


        Stefan

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

      If the alternative is "Using Clang in a proprietary product"
    then I think "Using GCC in a proprietary product" is definitely not
    a loss

It is more or less the same loss.  The case I'm concerned about is that
it become normal to use GCC with proprietary add-ons.

	   (especially if the GPLv3 works as it should, making it possible
    for the user to replace/fix/improve the GCC part of the proprietary
    product).

This would not make the proprietary combination any less bad.  What it
would do is make sure we would only need to replace the proprietary
parts.  However, that won't automatically make the job replacing them
get done.

Nowadays GCC does allow plug-ins -- we came up with a safe way to do
it (or at least I hope it's safe).  The issue now is to convince
people to work on improvements to GCC.  I hope that some people will
be motivated to work on them so as to get certain features into Emacs.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

      I think the current
    situation w/ GCC reflects a "one-eye victory" (to use a Go analogy),
    which basically means that what is defended is not guaranteed to live.

Nothing is guaranteed to live, but we are going to fight for it, not
give up.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[Jambunathan K]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>     But if as a result of those measures, all development moves from GCC to
>>     Clang/LLVM, this will be a pyrrhic victory :-(
>> That's what life is like.  If you fight, you might lose.  If you
>> surrender, you lose for certain.
>
> No.  There can be other measures that don't lead to a loss and don't
> lead to a pyrrhic victory either.
>
> But I think you'd first need to define what you'd consider as a loss.
> AFAICT, you seem to consider "Using GCC in a proprietary product" as
> a loss.  That's only true if the alternative was "making that product
> Free".  If the alternative is "Using Clang in a proprietary product"
> then I think "Using GCC in a proprietary product" is definitely not
> a loss (especially if the GPLv3 works as it should, making it possible
> for the user to replace/fix/improve the GCC part of the proprietary
> product).

Stefan,

As a layman user I would like to understand what this discussion is
about.

Is it about:

1. Making new features available within Emacs?

   - completion
   - refactoring
    
2. Making new features through *a specific* means.

   - llvm only
   - gcc only
    
3. Improving and stregthening the overall ecosystem.  Hyper-linking to
   other Free-software out there.

   - co-opeation with GCC
   - co-opeation with LLVM


If the focus of this thread is (1), it is better to invite CEDET
developers and ask for their inputs.  It bothers me that CEDET is not
really taking off.

As for (2) and (3), I have no observations to make.  It is better to
clarify (for the benefit of spectators) which of the three aspects is
being discussed - (1), (2) or (3).

From my own observations, it seems like (3) is taken up specifically wrt
llvm.

Shouldn't (1) merit equal attention? Implication of (3) is more
strategic and impacts the overally ecosystem but that of (1) is more of
a domestic affair.  Wouldn't it be easy to push for (1) which is less
controversial?

I am open to someone educating me and others.

>         Stefan

Re: Emacs contributions, C and Lisp

[Stephen J. Turnbull]

Jambunathan K writes:

 > Is it about:
 > 
 > 1. Making new features available within Emacs?
 > 
 >    - completion

Yes.

 >    - refactoring

Different posters have different views on that.

 > 2. Making new features through *a specific* means.
 > 
 >    - llvm only
 >    - gcc only

I don't think anybody advocates anything-only at this point.  Some
posters clearly believe LLVM *first* is a good strategy, because it
provides a very detailed description of the semantics of identifiers
out of the box.  However, Richard has declared LLVM may not be used
for development of these features[1], so this issue is moot.

David's point about LLVM support being implicitly LLVM-only is about
the pragmatic outcome of any implementation using LLVM: to some extent
it is very likely to detect LLVM-only C/C++ extensions and provide
appropriate "advice" to Emacs completion, but it would be rather
difficult to serve GCC-only extensions in the same way.  (Did I get
that right this time, David?)  If that is "first", then (at least for
a while) the effect will be LLVM-only in this sense.[2]

 > 3. Improving and stregthening the overall ecosystem.

Always, for any GNU project.  (But beware, Richard deprecates terms
like "ecosystem" and "ecology", I forget exactly why.  Something about
GNU not being an emergent outcome of random evolution, but rather
being designed and organized specifically to support software freedom,
IIRC.)

 >    Hyper-linking to other Free-software out there.

Dunno what you mean by that.

 >    - co-opeation with GCC

Richard wants that, both in this case and as a general matter of
greater unity within GNU as a whole.  I'm not sure if it's realistic
to expect much direct cooperation in this case, the skill sets are
rather different.

 >    - co-opeation with LLVM

Not as a project, that's clearly out because of the licensing.
Importing bits of LLVM code into GNU projects, yes, but not the other
way around.

 > If the focus of this thread is (1), it is better to invite CEDET
 > developers and ask for their inputs.

They're already here, and have been providing their input all along.
The consensus of CEDET supporters seems to be that LLVM tools might
potentially be somewhat more accurate, but that existing CEDET
features plus some additional heuristics should give excellent
results.

 > Shouldn't (1) merit equal attention?

I would say that's David Kastrup's point (except that he would say
that's really the only thing to focus on, use of GCC or CEDET is just
a question of of the inclination of the developers who actually do the
work and the available features).

I suspect David K would also advocate sticking specifically to
"completion" as the feature in question.  Trying to generalize the set
of features to be supported leads pretty quickly to relatively
abstract discussion of the advantages of LLVM's architecture for
building code analysis toolkits (whether in the compiler suite or in
Emacs Lisp), which doesn't help to get people to start working on the
features.

Footnotes: 
[1]  Of course to the extent that LLVM provides the same UI as GCC,
users cannot be prevented for using LLVM instead of GCC.  But no LLVM-
specific options or I/O formats may be supported.

[2]  Don't anybody bother objecting that the common subset of syntax
would serve both LLVM and GCC users.  Neither David nor Richard
objects to that, but they do not consider it a mitigating circumstance.

Re: Emacs contributions, C and Lisp

[Bastien]

Thien-Thi Nguyen <ttn@gnu.org> writes:

> I think the current
> situation w/ GCC reflects a "one-eye victory" (to use a Go analogy),
> which basically means that what is defended is not guaranteed to
> live.

I like this analogy.  It also suggests the way to win is to consider
both fighting from "inside" (by developing the discussed feature with
GCC or Emacs or both), and from "outside" (by making it less obvious
for Clang users that Cland is the best choice.)

Just 2 lurking-in-this-thread-for-a-minute cents,

-- 
 Bastien

Re: Emacs contributions, C and Lisp

[Stefan Monnier]

> It is more or less the same loss.  The case I'm concerned about is that
> it become normal to use GCC with proprietary add-ons.

My understanding is that people are satisfied with the current GCC
plugin licensing.  What is at stake here is the development of a stable
output that gives enough info for things like refactoring.

IOW the issue is not disallowing proprietary add-ons (we're all pretty
happy to disallow them, IIUC), but making it impossible to use GCC
within a proprietary product (e.g. a proprietary IDE).

> Nowadays GCC does allow plug-ins -- we came up with a safe way to do
> it (or at least I hope it's safe).  The issue now is to convince
> people to work on improvements to GCC.

Maybe that's the issue for GCC, but for Emacs the issue is to get detailed
info out of GCC, which is a different problem.  My understanding is that
you're opposed to GCC providing this useful info because that info would
need to be complete enough to be usable as input to a proprietary
compiler backend.


        Stefan

Re: Emacs contributions, C and Lisp

[Stefan Monnier]

> I like this analogy.  It also suggests the way to win is to consider
> both fighting from "inside" (by developing the discussed feature with
> GCC or Emacs or both), and from "outside" (by making it less obvious
> for Clang users that Cland is the best choice.)

Or, we could change Clang's color, by working hard on GPL extensions
of Clang.


        Stefan

Re: Emacs contributions, C and Lisp

[David Kastrup]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> It is more or less the same loss.  The case I'm concerned about is that
>> it become normal to use GCC with proprietary add-ons.
>
> My understanding is that people are satisfied with the current GCC
> plugin licensing.  What is at stake here is the development of a stable
> output that gives enough info for things like refactoring.
>
> IOW the issue is not disallowing proprietary add-ons (we're all pretty
> happy to disallow them, IIUC), but making it impossible to use GCC
> within a proprietary product (e.g. a proprietary IDE).

I don't think that there is much that we can do here.  It's always going
to be able to use GCC from within proprietary IDEs.  And any feature
that we are going to implement for making GCC do work for Emacs as an
IDE will definitely be open to proprietary IDEs equally.

What we do have is the problem of timing.  Richard is asking the GCC
developers, as far as I understand, what is doable now in order to get
support for smart completion into Emacs.  This will always be an option
for integration on-demand, and it means that there are no large time
windows where some feature is actually _only_ getting used by
proprietary applications/IDEs, giving them a competitive advantage over
our own environments.

On the other hand, "on-demand" introduces large time delays and
dependencies and makes spontaneous hacking lots harder if possible at
all.

>> Nowadays GCC does allow plug-ins -- we came up with a safe way to do
>> it (or at least I hope it's safe).  The issue now is to convince
>> people to work on improvements to GCC.
>
> Maybe that's the issue for GCC, but for Emacs the issue is to get
> detailed info out of GCC, which is a different problem.  My
> understanding is that you're opposed to GCC providing this useful info
> because that info would need to be complete enough to be usable as
> input to a proprietary compiler backend.

My long-term approach to integrating Emacs and GCC components more
tightly while still retaining a competitive technological argument over
the coupling into proprietary components is to tie things together not
at the basic I/O layer and not via dynamical linking or mapped memory,
but rather by having shared data structures and a common execution
model.

For Emacs/GCC, the way to do that would likely by making both run
through GUILE.  In the not so long-term perspective, data exchange would
be performed using simple LISP/Scheme data structures or dumps.  That
would at least favor integration and processing with Emacs LISP and make
it fast to exchange data.

Going in that direction would also facilitate using the existing parsing
frameworks for just-in-time compilation experiments.

Again, anything that is layered through independent modules will be
susceptible to reuse in proprietary components.  Having a GUILE runtime
as the executor will, however, at least make it harder to exploit the
modularity without either adopting Emacs-friendly data exchange methods
or getting into muddy water regarding the independence of modules
according to the expected court interpretations.

At any rate, the most feasible compromise between advantages for hacking
within our own copylefted software universe and not enabling approaches
bypassing our preferred technologies and/or principles would be to me to
move to a common reasonably integrateable platform at least on the
driver/control layer of compilation.  One probably still would want some
sort of separation for C-programmed components to avoid GCC bugs
crashing Emacs (which would be nightmarish to debug).  But it might be
possible to design systems ping-ponging Scheme data structures across
process boundaries.

That kind of setup would have the advantage that the political decisions
for accommodating integration of GCC/Emacs workings would not imply
having to deal with unexpected additional hurdles for tackling
integration tasks, but with unexpectedly fewer hurdles.

Putting GCC info out in LISP/Scheme reader compatible form would be a
first such step.  But that would also imply that typical batch
processing _not_ using Emacs would likely preferably be done using GUILE
(presumably preferredly) or some other LISP system (rep?).

It's clear that we are not going to find a satisfactory solution for
everything in a rush, even though that's what many approaches are being
advertised for.  But at least navigating the compromise between reuse
and modularity within GNU and reuse and modularity within proprietary
frameworks might be a bit nicer when we focus basing the reuse and
modularity on technology that is more on our home turf than "industry
standards".

Modularity will always be modularity, and consequently open to other
applications including non-free ones.  But we can at least try to make
the flavor of modularity one that might appeal more to people moving in
the GNU universe than outside.

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[Florian Weimer]

* Stefan Monnier:

>> It is more or less the same loss.  The case I'm concerned about is that
>> it become normal to use GCC with proprietary add-ons.
>
> My understanding is that people are satisfied with the current GCC
> plugin licensing.  What is at stake here is the development of a stable
> output that gives enough info for things like refactoring.
>
> IOW the issue is not disallowing proprietary add-ons (we're all pretty
> happy to disallow them, IIUC), but making it impossible to use GCC
> within a proprietary product (e.g. a proprietary IDE).

Eh, isn't the latter already extremely common in the embedded area?
These IDEs do not offer refactoring, but they still invoke GCC to
perform the compilation.  They might even using debugging information
to implement some form of source code browsing.  (And in many cases,
you can't redistribute the GCC binaries you received, but that's a
different story.)

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

      (And in many cases,
    you can't redistribute the GCC binaries you received, but that's a
    different story.)

That sounds like a GPL violation.  Can you identify a specific case?

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    Suicidal charges on entrenched positions make for great drama, but poor
    results; it's better to run and fight another day.

I have thought carefully about the likelihood of various outcomes.
However, none of them are likely to result in killing any of us.
This colorful exaggeration shows you've got it wrong.

     you've argued
    those who disagreed with your decisions could be doing so only out of a
    secret desire to undermine free software. Statements like "I don't trust
    your judgment about how to achieve our goals because you don't want to
    achieve them." are circular nonsense.

Our community is full of people who work on free software and disagree
with the free software movement.  Often they say so, sometimes very
loudly.  What you are saying here shows your picture of the general
situation is very misguided.  Arguments based on that are not cogent.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp (was: Re: /srv/bzr/emacs/trunk r101338: ...)

[Michał Nazarewicz]

2014-02-17 21:29 GMT+01:00 Eli Zaretskii <eliz@gnu.org>:
> What red tape?  Emacs is about the most red-tape-less project as you
> can find, as far as the procedure of admitting a patch is considered.

I'm a bit late to the party, and haven't yet gone through the whole thread,
but let me chip in my 2 rappens anyway. ;)

What I found a bit discouraging is lack of responses to my patches. Admittedly,
after re-reading etc/CONTRIBUTE I discovered I was sending them to the wrong
place (emacs-devel instead of bug-gnu-emacs) but still it would be nice if
someone yelled at me for using the wrong mailing list.

Re: Emacs contributions, C and Lisp

[Michał Nazarewicz]

2014-02-19 18:57 GMT+01:00 Glenn Morris <rgm@gnu.org>:
> For example, you are proving that here by commenting on a CONTRIBUTE
> version that I can tell is not the version in trunk. :)

And I must admit I welcomed the removal of “use ‘Context Diff’ format”
requirement, since now I don't need to feel bad about doing:

  git format-patch HEAD~
  git send-email … 0001-*

Since the whole repository is being moved to git, I think the above method
should be advertised as it makes sending patches much easier and many
developers using git are already familiar with it.

Re: Emacs contributions, C and Lisp

[Phillip Lord]

Michał Nazarewicz <mina86@mina86.com> writes:
> 2014-02-17 21:29 GMT+01:00 Eli Zaretskii <eliz@gnu.org>:
>> What red tape?  Emacs is about the most red-tape-less project as you
>> can find, as far as the procedure of admitting a patch is considered.
>
> I'm a bit late to the party, and haven't yet gone through the whole thread,
> but let me chip in my 2 rappens anyway. ;)
>
> What I found a bit discouraging is lack of responses to my patches. Admittedly,
> after re-reading etc/CONTRIBUTE I discovered I was sending them to the wrong
> place (emacs-devel instead of bug-gnu-emacs) but still it would be nice if
> someone yelled at me for using the wrong mailing list.

Something like the gitorious merge request facilities would help here.
Then you would be able to see that your request had gone in. Mailing
lists are good for some things but maintaining a database is probably
not one of them.

Phil

Re: Emacs contributions, C and Lisp

[Michal Nazarewicz]

On Wed, Feb 19 2014, David Kastrup <dak@gnu.org> wrote:
> With regard to copyright assignments, you pretend that it is some magic
> ritual of initiation.  There is nobody who'd be more glad than the FSF
> if this kind of paperwork was without merit and unneeded.

There are people who would argue that this kind of paperwork is in fact
unneeded.  I admit it has merit, and I understand why lawyers want it,
but it's not at all clear that it is worth creating this burden.

SFC for example runs its GPL Compliance Project for Linux Developers
representing just a small fraction of Linux copyright holders.  Even GNU
does not require copyright assignment to all its projects.

I may be wrong in my assessment, I'm not a lawyer after all, but I bet
*majority* of developers, myself included, see CA as an unneeded burden.
We can only guess how many people dropped their patches after reading CA
is required.

And individuals are not even the hardest part.  I became a maintainer of
auto-dim-other-buffers.el[1].  I would love to have it in GNU ELPA, but
frankly, I won't even bother asking other contributors of to the project
to sign a CA, and it's not even a complicated case -- there are only two
other people involved -- again we can only guess how many projects are
not contributed to Emacs because they were started outside of GNU Emacs
and now people don't want to deal with CAs.

And we did not even start with the fact that some people oppose CAs as
a matter of principle.

[1] https://github.com/mina86/auto-dim-other-buffers.el

-- 
Best regards,                                         _     _
.o. | Liege of Serenely Enlightened Majesty of      o' \,=./ `o
..o | Computer Science,  Michał “mina86” Nazarewicz    (o o)
ooo +--<mpn@google.com>--<xmpp:mina86@jabber.org>--ooO--(_)--Ooo--

Re: Emacs contributions, C and Lisp

[David Kastrup]

Michal Nazarewicz <mina86@mina86.com> writes:

> On Wed, Feb 19 2014, David Kastrup <dak@gnu.org> wrote:
>> With regard to copyright assignments, you pretend that it is some
>> magic ritual of initiation.  There is nobody who'd be more glad than
>> the FSF if this kind of paperwork was without merit and unneeded.
>
> There are people who would argue that this kind of paperwork is in
> fact unneeded.  I admit it has merit, and I understand why lawyers
> want it, but it's not at all clear that it is worth creating this
> burden.
>
> SFC for example runs its GPL Compliance Project for Linux Developers
> representing just a small fraction of Linux copyright holders.

Which means that a large ratio of Linux code can be misappropriated into
proprietary products without anybody bothering to complain.  In Germany,
basically _all_ Linux license violations are prosecuted based on the
netfilter code, and if I am not mistaken, that code is slated to be
replaced by a different implementation anyway.

> Even GNU does not require copyright assignment to all its projects.

So?

> I may be wrong in my assessment, I'm not a lawyer after all,

The FSF is in constant contact with lawyers, and the GPL is a legal
tool.

> but I bet *majority* of developers, myself included, see CA as an
> unneeded burden.

The copyright assignment is not for the sake of the developers (well, it
_does_ free them of the responsibility to go after copyright violations
themselves).

> We can only guess how many people dropped their patches after reading
> CA is required.

Sure.

> And individuals are not even the hardest part.  I became a maintainer
> of auto-dim-other-buffers.el[1].  I would love to have it in GNU ELPA,
> but frankly, I won't even bother asking other contributors of to the
> project to sign a CA, and it's not even a complicated case -- there
> are only two other people involved -- again we can only guess how many
> projects are not contributed to Emacs because they were started
> outside of GNU Emacs and now people don't want to deal with CAs.

"not contributed to Emacs" is a red herring since you can always
distribute them yourself.  They are not distributed along with the core
Emacs, and the core Emacs is copyrighted by the FSF: it's the principal
editor and IDE of the GNU project.

> And we did not even start with the fact that some people oppose CAs as
> a matter of principle.
>
> [1] https://github.com/mina86/auto-dim-other-buffers.el

And other people oppose the GPL as a matter of principle, and most
people opposing a copyright assignment would not dream of going after
violations of their copyright themselves, thus implicitly opposing the
GPL as well.  A GPL that is not enforced is pointless; you could be
using the MIT license in the first place.  The FSF retains several
copyright-assigned projects on which it will make sure the GPL has
teeth.

So in essence you are asking the FSF to stop being the FSF.  You are not
the first one to do so, and you will not be the last one.  Do you really
think that after 25 years of the GPL's use in FSF-governed projects you
are bringing something new to the table here?

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[Stefan Monnier]

> What I found a bit discouraging is lack of responses to my patches.

I do think lack of response is the worst part of our handling of patches.

> Admittedly, after re-reading etc/CONTRIBUTE I discovered I was sending
> them to the wrong place (emacs-devel instead of bug-gnu-emacs) but
> still it would be nice if someone yelled at me for using the wrong
> mailing list.

emacs-devel is not really the wrong list for that.  bug-gnu-emacs is
better, but emacs-devel is fine.
The problem was simply lack of response, pure and simple.  This reflects
on the lack of time/energy on our part to deal with those patches.


        Stefan

Re: Emacs contributions, C and Lisp

[Michal Nazarewicz]

[-- Attachment #1: Type: text/plain, Size: 1132 bytes --]

On Sat, Mar 01 2014, Eli Zaretskii wrote:
> IMNSHO, it is bad design to ask users to install a particular
> compiler, be it GCC or clang, just to be able to have decent editing
> capabilities for a program source.  What next? shall we require
> LibreOffice to be able to edit text files conveniently?

Emacs already uses ?spell for spell checking, GPG for signatures, and
probably many more tools I'm not aware of.  I also think it's safe to
assume that if someone needs a decent editing capabilities for a program
source, she has tools that are able to compile that program source
already installed.

(I'm not saying that requiring a compiler for completion is the only
possible option, by the way.  I just wanna say, that not everything has
to be implemented in Emacs Lisp and for one, I hope no one will try to
reimplement GPG in Emacs Lisp).

-- 
Best regards,                                         _     _
.o. | Liege of Serenely Enlightened Majesty of      o' \,=./ `o
..o | Computer Science,  Michał “mina86” Nazarewicz    (o o)
ooo +--<mpn@google.com>--<xmpp:mina86@jabber.org>--ooO--(_)--Ooo--

[-- Attachment #2.1: Type: text/plain, Size: 0 bytes --]



[-- Attachment #2.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 835 bytes --]

Re: Emacs contributions, C and Lisp

[Eli Zaretskii]

> From: phillip.lord@newcastle.ac.uk (Phillip Lord)
> Cc: Eli Zaretskii <eliz@gnu.org>, <dgutov@yandex.ru>, <emacs-devel@gnu.org>,
>         Jorgen Schaefer <forcer@forcix.cx>
> Date: Thu, 27 Mar 2014 10:19:30 +0000
> 
> > What I found a bit discouraging is lack of responses to my patches. Admittedly,
> > after re-reading etc/CONTRIBUTE I discovered I was sending them to the wrong
> > place (emacs-devel instead of bug-gnu-emacs) but still it would be nice if
> > someone yelled at me for using the wrong mailing list.
> 
> Something like the gitorious merge request facilities would help here.
> Then you would be able to see that your request had gone in. Mailing
> lists are good for some things but maintaining a database is probably
> not one of them.

What we really need is to have in place an efficient and effective
procedures for patch review and application.  Without that, patches
will collect dust on gitorious as they will in the archives of this or
another mailing list.

So I think you are putting the carriage ahead of the horse.  We need
first to have "patch pilots" and maybe some support rules, and then
those people will pick up the tools they like best for doing their
job.  Having a database of patches that no one looks upon will lead us
nowhere.

Re: Emacs contributions, C and Lisp

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:
>> Something like the gitorious merge request facilities would help here.
>> Then you would be able to see that your request had gone in. Mailing
>> lists are good for some things but maintaining a database is probably
>> not one of them.
>
> What we really need is to have in place an efficient and effective
> procedures for patch review and application.  Without that, patches
> will collect dust on gitorious as they will in the archives of this or
> another mailing list.


Yep, but the flip side is that it is easier to see which patches are
gathering dust. That's always a good motivator. I mean, it's like a todo
list; yes, you still have to do the things on the todo list, and if you
don't it comes big and pointless. But, I have found my todo handling is
a little more efficient since I discovered org-mode to replace either
post-it notes or my (every failing) memory.


> So I think you are putting the carriage ahead of the horse.  We need
> first to have "patch pilots" and maybe some support rules, and then
> those people will pick up the tools they like best for doing their
> job.  Having a database of patches that no one looks upon will lead us
> nowhere.

Again, I think this is partly true, but my experience is that it's
easier to pull a branch and switch to it, than it is to handle patches.

Processes are important. But so are tools. It's why I use Emacs rather
than ed for instance.

Phil

Re: Emacs contributions, C and Lisp

[Eli Zaretskii]

> From: Michal Nazarewicz <mina86@mina86.com>
> Cc: dak@gnu.org, emacs-devel@gnu.org
> Date: Thu, 27 Mar 2014 16:10:29 +0100
> 
> On Sat, Mar 01 2014, Eli Zaretskii wrote:
> > IMNSHO, it is bad design to ask users to install a particular
> > compiler, be it GCC or clang, just to be able to have decent editing
> > capabilities for a program source.  What next? shall we require
> > LibreOffice to be able to edit text files conveniently?
> 
> Emacs already uses ?spell for spell checking, GPG for signatures, and
> probably many more tools I'm not aware of.  I also think it's safe to
> assume that if someone needs a decent editing capabilities for a program
> source, she has tools that are able to compile that program source
> already installed.

You are misreading what I wrote, see above.  I said it was
unreasonable to ask users to install some _particular_ compiler, not
_a_ compiler.

That would be similar to only support in ispell.el one particular
speller and not the rest.

> (I'm not saying that requiring a compiler for completion is the only
> possible option, by the way.  I just wanna say, that not everything has
> to be implemented in Emacs Lisp and for one, I hope no one will try to
> reimplement GPG in Emacs Lisp).

Strawman.  No one suggested to "implement everything in Emacs Lisp".

Re: Emacs contributions, C and Lisp

[Eli Zaretskii]

> From: phillip.lord@newcastle.ac.uk (Phillip Lord)
> Cc: <mina86@mina86.com>, <dgutov@yandex.ru>, <emacs-devel@gnu.org>,
>         <forcer@forcix.cx>
> Date: Thu, 27 Mar 2014 17:08:14 +0000
> 
> Processes are important. But so are tools.

Indeed, but tools should be selected by those who do the job.

Re: Emacs contributions, C and Lisp

[Michal Nazarewicz]

[-- Attachment #1: Type: text/plain, Size: 508 bytes --]

On Thu, Mar 27 2014, Eli Zaretskii wrote:
> You are misreading what I wrote, see above.  I said it was
> unreasonable to ask users to install some _particular_ compiler, not
> _a_ compiler.

Sorry about that.  Disregard my comment.

-- 
Best regards,                                         _     _
.o. | Liege of Serenely Enlightened Majesty of      o' \,=./ `o
..o | Computer Science,  Michał “mina86” Nazarewicz    (o o)
ooo +--<mpn@google.com>--<xmpp:mina86@jabber.org>--ooO--(_)--Ooo--

[-- Attachment #2.1: Type: text/plain, Size: 0 bytes --]



[-- Attachment #2.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 835 bytes --]

Re: Emacs contributions, C and Lisp

[Michal Nazarewicz]

[-- Attachment #1: Type: text/plain, Size: 582 bytes --]

On Thu, Mar 27 2014, Phillip Lord wrote:
> Again, I think this is partly true, but my experience is that it's
> easier to pull a branch and switch to it, than it is to handle patches.

At the same time, I much prefer reviewing patches in my mail client
(which, by the way, is Emacs) than on some website. 

-- 
Best regards,                                         _     _
.o. | Liege of Serenely Enlightened Majesty of      o' \,=./ `o
..o | Computer Science,  Michał “mina86” Nazarewicz    (o o)
ooo +--<mpn@google.com>--<xmpp:mina86@jabber.org>--ooO--(_)--Ooo--

[-- Attachment #2.1: Type: text/plain, Size: 0 bytes --]



[-- Attachment #2.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 835 bytes --]

Re: Emacs contributions, C and Lisp

[Stephen J. Turnbull]

Eli Zaretskii writes:

 > What we really need is to have in place an efficient and effective
 > procedures for patch review and application.  Without that, patches
 > will collect dust on gitorious as they will in the archives of this or
 > another mailing list.

No, what is really needed is "all of the above".  Patch pilots[1] are
needed, or gitorius-like facilities will provide a high-tech way for
patches to collect barnacles.  A facility to make it easy to see that
patches have not been addressed (triage, comment) is necessary, or
patches will fall through the cracks more than necessary.  Reviewers
and committers are needed to actually process the patches.  A facility
that makes it The One Obvious Way To Do It ["it" == submit patches] is
necessary, or the facility for finding "orphan" patches won't be used
to submit patches.  And a facility to make it trivial (ie, electronic)
to submit a contribution assignment[2], so that submitted patches
won't languish for lack of an assignment/[3]

My suggestion is that while you're right, and Michal is right, and I'm
right, and ..., *your* best response here (as an Emacs maintainer) is
the same as always:

 patches [provision of needed facilities or reviewer effort] welcome!

Ie, just provide a list of needed facilities and encourage people to
provide them.  Just like ESR has done the work to port the repo to git.


Footnotes: 
[1]  Perhaps informally, as in current Emacs procedure.

[2]  And somewhat less trivial to submit a "future" assignment, so
that people won't complain that they didn't realize all their Emacs
code would automatically become FSF property upon integration into
Emacs.  Most want that of course, but there are jerks out there.
They'll complain anyway, but you want to be able to killfile them
after responding "you were warned in big red letters and it's not
default" exactly once.

[3]  If the user isn't a refusenik, nothing technological will help
with that.

Re: Emacs contributions, C and Lisp

[Stephen J. Turnbull]

Aside to Richard: the last 3 paragraphs may be of use to you/the
FSF/the movement.

Michal Nazarewicz writes:

 > There are people who would argue that this kind of paperwork is in fact
 > unneeded.  I admit it has merit, and I understand why lawyers want it,
 > but it's not at all clear that it is worth creating this burden.

Doesn't matter, because it is clear to RMS as well as the formal Emacs
maintainers.  (I suspect clarity on this matter is a prerequisite for
Emacs maintainership, but either way, there's an actual consensus
among Those Who Decide -- only if the lawyers change their minds would
this change, but our arguments won't affect the lawyers' advice.)

Also, you should note that you're bucking the tide here.  I believe
that Linus has conceded that assignments can be helpful, and many
projects such as Python require a contributor agreement, which is just
as much paperwork, even if it doesn't require giving up copyright.
The number of successful projects that require paperwork in some form
is growing.

 > SFC for example runs its GPL Compliance Project for Linux
 > Developers representing just a small fraction of Linux copyright
 > holders.

Good for them!  Here's hoping the fraction gets larger!

 > Even GNU does not require copyright assignment to all its projects.

GNU includes a lot of things that are not GNU products.  RMS's
penchant for abbreviation notwithstanding, the proper name of most GNU
distributions is "GNU/BSD/TeX/X11/Linux/Perl/..., oh screw it, My Name
Is Legion".  Because all GNU distributions include such "borrowed"
components (see the GNU Manifesto), requiring assignments is difficult
in practice, except on a project by project basis.

 > I may be wrong in my assessment, I'm not a lawyer after all, but I
 > bet *majority* of developers, myself included, see CA as an
 > unneeded burden.

So what?  The GNU Emacs project sees it as necessary to protect their
distribution of Emacs.  As David points out that doesn't stop you from
distributing Emacs including your code, or just your code, yourself.

 > And individuals are not even the hardest part.  I became a
 > maintainer of auto-dim-other-buffers.el[1].  I would love to have
 > it in GNU ELPA, but frankly, I won't even bother asking other
 > contributors of to the project to sign a CA

Why not?  All they can do is say "No".  You don't have to push it
hard, but asking is easy.

I think that it should be easier to make the assignment than it
currently is, but that's another matter.

 > And we did not even start with the fact that some people oppose CAs
 > as a matter of principle.

So what?  Whenever my principles come into conflict with yours, I have
to act in accordance with my own principles, no?  Of course I should
respect you for acting in accordance with yours.  If we can't
cooperate for that reason it's a shame, but life is like that.  The
alternative is the Crusades (or jihad, if you prefer).

N.B.  It's hard for me to understand the "in principle" part.  For all
practical purposes your code remains your code; you are free to do
anything with it that you were free to do before.  The ownership
transfer is *necessary* to protect *your* right to do anything you
like with that code.  Under U.S. law, at least, it is not possible to
license somebody else to administer your copyright unless that license
is exclusive, which means you can't do anything else with the code for
the term of that license.  What the FSF copyright assignment does is
to assume ownership, then grant you a *nonexclusive* license to do
*anything* you want with your code, including relicensing it as a
proprietary product.

Ie, just as the GPL is legal judo turning copyright *itself* on its
head, the FSF copyright assignment is legal judo turning copyright
*ownership* on its head.

I can understand not assigning code that doesn't depend on copyleft
code, or where you believe you might be able to get an appropriate
license from the owner of the copyleft code.  But not assigning code
that is derivative of Emacs ... where's the principle in that?
"What's mine is mine, and I'm not giving it up"?  Unless you are very
rich, rich enough to live off interest, you have to give something up
to live: your work, or the products of that work.

P.S. to Richard: I don't recall seeing this argument expressed before.
Maybe it's new?  Either way, I hereby dedicate the previous three
paragraphs of this message, beginning with "N.B.", to the public
domain -- if it's useful to you, feel free.  If your lawyers want
paperwork, let me know.  Or you can do a clean-room reimplementation. :-)

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

It is always reasonable for one GNU package to depend on another
GNU package, when there is a practical reason for things to be that way.
-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[Michal Nazarewicz]

[-- Attachment #1: Type: text/plain, Size: 2144 bytes --]

> On Wed, Feb 19 2014, David Kastrup <dak@gnu.org> wrote:
>> With regard to copyright assignments, you pretend that it is some magic
>> ritual of initiation.  There is nobody who'd be more glad than the FSF
>> if this kind of paperwork was without merit and unneeded.

On Thu, Mar 27 2014, Michal Nazarewicz wrote:
> There are people who would argue that this kind of paperwork is in fact
> unneeded.  I admit it has merit, and I understand why lawyers want it,
> but it's not at all clear that it is worth creating this burden.

All right, let me take a step back because it seems my argument was
worded too strongly and spawn a discussion that I did not intend.

What I'm trying to say is that copyright assignment *is* a red tape.  It
*does* prevent patches from going into Emacs.

One can put fingers in their ears and claim this in not the case but
such behaviour would be hurtful to Emacs.  The other option is to admit
the issue, but say GNU project chosen to go with CAs anyway, and stop
the discussion here.  This is better, but I don't think the discussion
needs to stop there.

So here are some ideas that would reduce the cost of a first patch:
- Allow the first 100 patches without a CA.
- Allow electronically-signed CA.
- Allow electronically-signed CA for the first 100 patches, and then
  require dead-tree CA.
- Allow multi-project CAs (perhaps a form with a list of check-boxes).

A lawyer will tell you[1] that it increases risk and may make
enforcement harder, but your job is not to do whatever lawyers tell you,
but weight the risks against benefits, and I *strongly* believe that in
the case of Emacs benefits of allowing the first 100 patches w/o a CA
*greatly* exceed risks.

[1] By “you” I mean whoever would be in position to make a decision to
    change the CA process, which most likely means Richard.

-- 
Best regards,                                         _     _
.o. | Liege of Serenely Enlightened Majesty of      o' \,=./ `o
..o | Computer Science,  Michał “mina86” Nazarewicz    (o o)
ooo +--<mpn@google.com>--<xmpp:mina86@jabber.org>--ooO--(_)--Ooo--

[-- Attachment #2.1: Type: text/plain, Size: 0 bytes --]



[-- Attachment #2.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 835 bytes --]

Re: Emacs contributions, C and Lisp

[David Kastrup]

Michal Nazarewicz <mina86@mina86.com> writes:

>> On Wed, Feb 19 2014, David Kastrup <dak@gnu.org> wrote:
>>> With regard to copyright assignments, you pretend that it is some magic
>>> ritual of initiation.  There is nobody who'd be more glad than the FSF
>>> if this kind of paperwork was without merit and unneeded.
>
> On Thu, Mar 27 2014, Michal Nazarewicz wrote:
>> There are people who would argue that this kind of paperwork is in fact
>> unneeded.  I admit it has merit, and I understand why lawyers want it,
>> but it's not at all clear that it is worth creating this burden.
>
> All right, let me take a step back because it seems my argument was
> worded too strongly and spawn a discussion that I did not intend.

Sigh.  It did not "spawn a discussion".  It merely led to the thousandth
iteration of the same old points that have been valid for the last 25
years.

Again: what makes you think that you can contribute anything new to it?

> What I'm trying to say is that copyright assignment *is* a red tape.
> It *does* prevent patches from going into Emacs.

What makes you think that nobody understood that?

> One can put fingers in their ears and claim this in not the case

Can you _please_ stop attacking the same strawmen all over?  Nobody
claims that this is not the case.

> but such behaviour would be hurtful to Emacs.

You don't seem to be able to get it into your head that some decisions
involve tradeoffs.  Naturally, there is a downside.  But you are
mistaken in your insistence that everybody buy yourself must have been
too stupid in the last 25 years to ever notice.

> The other option is to admit the issue, but say GNU project chosen to
> go with CAs anyway, and stop the discussion here.  This is better, but
> I don't think the discussion needs to stop there.

There is no discussion.  A discussion is something which people enter
with the will to leave it with different opinions than they started
with.  Since this topic has been talked over backwards and forwards and
discussed with lawyers for more than 25 years and is inherently tied to
copyright law and the way the GPL works, you cannot expect the people
who are in it to go into your "discussion" with the expectation to hear
something new.  And apparently you yourself do not accept any point of
view except your own and will rather put up strawmen than accept that
anybody has thought this through.

> So here are some ideas that would reduce the cost of a first patch:
> - Allow the first 100 patches without a CA.
> - Allow electronically-signed CA.
> - Allow electronically-signed CA for the first 100 patches, and then
>   require dead-tree CA.
> - Allow multi-project CAs (perhaps a form with a list of check-boxes).
>
> A lawyer will tell you[1] that it increases risk and may make
> enforcement harder, but your job is not to do whatever lawyers tell
> you,

Do you even understand that the GPL is a _legal_ tool and works through
legal processes?

It's like telling a bank "your job is not to do whatever security
experts tell you but what makes your customers happy" with regard to
electronic banking.

> but weight the risks against benefits, and I *strongly* believe that
> in the case of Emacs benefits of allowing the first 100 patches w/o a
> CA *greatly* exceed risks.

Emacs is the first application _ever_ released under the GPL.  The GPL
was _created_ for it.  By Richard Stallman, 25 years ago, in
consultation with lawyers.  The processes the FSF follows are discussed
with some of the best law professors in the field of copyright.  That
has been the case for decades.

If you come to the table with "I am not a lawyer but I think you should
do everything differently because I think I know better", do you really
expect _any_ outcome from this "discussion" other than that you are
going to convince yourself to be dealing with unreasonable people?

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1075 bytes --]

On 03/28/2014 10:27 AM, David Kastrup wrote:
>> So here are some ideas that would reduce the cost of a first patch:
>> - Allow the first 100 patches without a CA.
>> - Allow electronically-signed CA.
>> - Allow electronically-signed CA for the first 100 patches, and then
>>   require dead-tree CA.
>> - Allow multi-project CAs (perhaps a form with a list of check-boxes).
>>
>> A lawyer will tell you[1] that it increases risk and may make
>> enforcement harder, but your job is not to do whatever lawyers tell
>> you,
> 
> Do you even understand that the GPL is a _legal_ tool and works through
> legal processes?
> 
> It's like telling a bank "your job is not to do whatever security
> experts tell you but what makes your customers happy" with regard to
> electronic banking.

I arrange financial transactions all the time without touching a single
bit of paper. Electronic signatures have become much more prevalent over
the past few years. Maybe it's time to revisit this specific issue. Have
you asked the lawyers about this subject recently?


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 901 bytes --]

Re: Emacs contributions, C and Lisp

[Glenn Morris]

Daniel Colascione wrote:

> I arrange financial transactions all the time without touching a single
> bit of paper. Electronic signatures have become much more prevalent over
> the past few years. Maybe it's time to revisit this specific issue. Have
> you asked the lawyers about this subject recently?

Oh hey, look (from August 2013):

https://www.fsf.org/blogs/licensing/fsf-to-begin-accepting-gpg-signed-assignments-from-the-u-s

So yes, amazingly, people do think about these things.

Anyway, can we please stop endlessly rehashing FSF/GNU policy on the
Emacs development list?

Re: Emacs contributions, C and Lisp

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 614 bytes --]

On 03/28/2014 12:01 PM, Glenn Morris wrote:
> Daniel Colascione wrote:
> 
>> I arrange financial transactions all the time without touching a single
>> bit of paper. Electronic signatures have become much more prevalent over
>> the past few years. Maybe it's time to revisit this specific issue. Have
>> you asked the lawyers about this subject recently?
> 
> Oh hey, look (from August 2013):
> 
> https://www.fsf.org/blogs/licensing/fsf-to-begin-accepting-gpg-signed-assignments-from-the-u-s
> 
> So yes, amazingly, people do think about these things.

That's great news. Thanks for the pointer.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 901 bytes --]

Re: Emacs contributions, C and Lisp

[Stefan Monnier]

> So here are some ideas that would reduce the cost of a first patch:
> - Allow the first 100 patches without a CA.

We currently only allow upto "20 lines worth" of code without CA.

I don't think Richard would want to increase that limit (IIRC the limit
is based not on "the risk is low enough" but on the fact that it's
small enough to be "uncopyrightable" from a legal viewpoint).

But even if we want to increase that limit, it comes with the burden of
keeping track of those things to make sure we don't go over the limit.

Also, this limit is only useful if it completely avoids a CA, not if it
just delays it.  If someone contributed 50 patches, he's likely to end
up wanting/able to contribute more than 100.

Of course, there is an important exception: the contributors has stopped
contributing, we can't even find him, and we want to include code of his
part of some package that's been distributed outside Emacs for
many years.  I'd be happy if we could find a way to accept such cases
(with the understanding that if the contributor ever shows up again,
he'll then have to sign a CA before we can accept his code).

> - Allow electronically-signed CA.

IIUC, this is now the case if you live in the US (and I know the FSF
lawyers are working at making this exception work in more cases).

> - Allow electronically-signed CA for the first 100 patches, and then
>   require dead-tree CA.

Nah, this wouldn't worth the trouble.  I can't think of a single case
where someone would have been able/willing to sign a CA electronically
but refused to sign the paper version.

> - Allow multi-project CAs (perhaps a form with a list of check-boxes).

IIRC someone managed to sign a copyright assignment for "ANY" GNU
project, and I think it'd be nice if it can be done more generally.
But I had the impression the FSF discouraged it strongly, tho I can't
remember why.  Maybe I'm just mistaken.


        Stefan

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    GNU includes a lot of things that are not GNU products.  RMS's
    penchant for abbreviation notwithstanding, the proper name of most GNU
    distributions is "GNU/BSD/TeX/X11/Linux/Perl/..., oh screw it, My Name
    Is Legion".

http://www.gnu.org/gnu/linux-and-gnu.html#many for why that assertion
is not valid.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[Glenn Morris]

Richard Stallman wrote:

> http://www.gnu.org/gnu/linux-and-gnu.html#many for why that assertion
> is not valid.

ITYM http://www.gnu.org/gnu/gnu-linux-faq.html#many

Re: Emacs contributions, C and Lisp

[Stephen J. Turnbull]

Glenn Morris writes:
 > Richard Stallman wrote:
 > 
 > > http://www.gnu.org/gnu/linux-and-gnu.html#many for why that assertion
 > > is not valid.
 > 
 > ITYM http://www.gnu.org/gnu/gnu-linux-faq.html#many

Whichever he means, the FAQ itself sanctions my opinion:

    If you feel even more strongly about giving credit where it is
    due, you might feel that some secondary contributors also deserve
    credit in the system's name. If so, far be it from us to argue
    against it. If you feel that X11 deserves credit in the system's
    name, and you want to call the system GNU/X11/Linux, please do. If
    you feel that Perl simply cries out for mention, and you want to
    write GNU/Linux/Perl, go ahead.

I mentioned GNU, I even put it first.  'Nuff said.

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    ITYM http://www.gnu.org/gnu/gnu-linux-faq.html#many

Yes, oops.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

You wrote

    Whichever he means, the FAQ itself sanctions my opinion:

and then quoted this:

    If you feel even more strongly about giving credit where it is due,
    you might feel that some secondary contributors also deserve credit in
    the system's name.  If so, far be it from us to argue against it.  If
    you feel that X11 deserves credit in the system's name, and you want
    to call the system GNU/X11/Linux, please do.  If you feel that Perl
    simply cries out for mention, and you want to write GNU/Linux/Perl, go
    ahead.</p>

You quoted part of the entry, out of context, misrepresenting its
point.  I've included the whole entry below so people can see this is
true.

This discussion is a response to your fallacious call for people to
deny us credit for our work.


<dt id="many">Many other projects contributed to
    the system as it is today; it includes TeX, X11, Apache, Perl, and many
    more programs.  Don't your arguments imply we have to give them credit
    too?  (But that would lead to a name so long it is
    absurd.) <span class="anchor-reference-id">(<a href="#many">#many</a>)</span></dt>


<dd>
What we say is that you ought to give the system's principal developer
a share of the credit.  The principal developer is the GNU Project,
and the system is basically GNU.
<p>
If you feel even more strongly about giving credit where it is due,
you might feel that some secondary contributors also deserve credit in
the system's name.  If so, far be it from us to argue against it.  If
you feel that X11 deserves credit in the system's name, and you want
to call the system GNU/X11/Linux, please do.  If you feel that Perl
simply cries out for mention, and you want to write GNU/Linux/Perl, go
ahead.</p>
<p>
Since a long name such as GNU/X11/Apache/Linux/TeX/Perl/Python/FreeCiv
becomes absurd, at some point you will have to set a threshold and
omit the names of the many other secondary contributions.  There is no
one obvious right place to set the threshold, so wherever you set it,
we won't argue against it.</p>
<p>
Different threshold levels would lead to different choices of name for
the system.  But one name that cannot result from concerns of fairness
and giving credit, not for any possible threshold level, is &ldquo;Linux&rdquo;.
It can't be fair to give all the credit to one secondary contribution
(Linux) while omitting the principal contribution (GNU).</p>
</dd>


-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

We are thinking about allowing somewhat larger contributions without
a copyright assignment; but I think we will still want some sort of
papers to deal with the issue of whether the person has an employer
who might claim copyright.  We need to work out the details of this.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[Stephen J. Turnbull]

Richard Stallman writes:

 > This discussion is a response to your fallacious call for people to
 > deny us credit for our work.

Richard, I did no such thing.  I called for you to give the same
credit to others you demand for yourself.  That FAQ admits that
"GNU/Linux" is an abbreviation, and goes on to provide some
justification for giving primacy to GNU.

But that justification is misdirected, given that the GNU Project with
an uppercase "P" does almost no development.  More important from a PR
standpoint, these days, it basically leaves even distribution up to
others -- I don't know how long it's been since I heard the term "gold
tape", and I don't think I even heard once about "gold CDs", let alone
"gold DVDs" or "gold BDs".  The *real* contribution of GNU, as
acknowledged by several of the BSD folks, was the *insight* that "hey
-- we don't have to deliver just 'a pile of useful software', we can
deliver a whole working system *as free software*!"  And the GNU
Project proceeded to *assemble* the work of others, as a *free
software* operating system, nearly complete.[1]

As many discussions on this list and others have shown, much of the
development work for GNU projects is done by people who feel no
allegiance to the GNU Project per se.  They feel allegiance to
particular project, and often to the Free Software Movement.
Commercial interests such as Canonical hitchhike on the GNU name, but
fail to even pay lip service on the web sites for "GNU projects" they
maintain until prodded.

And this is one of the big difficulties *you* face in advocating
"GNU/Linux".  These days, the "GNU System" is nothing but a "pile of
useful software".  You yourself encourage folks with no particular
desire to promote GNU to slap the GNU label on their products.  The
actual curating of the GNU System, as a system, is done by the
distros.  Development of almost all of the end-user-visible software
is done anywhere but in gnu.org facilities.  And the core GNU people
(at the GNU Project level, rather than the GNU project level) such as
yourself have a core mission, not of developing the GNU system, but of
political advocacy.  Compare that with the BSD distros, where the
"BSD" name lives on in common parlance: FreeBSD, NetBSD, OpenBSD, and
a legion of minor variants.  Their leaders concerned themselves with
the systems as a whole, starting from the kernel and working out.  GNU
was -- and is! -- stuck with a system with a hole, and worse, that
hole is the kernel.

It's ironic that you claim the right to name a system whose raison
d'etre is "make the software your own!" based on the initial push in
1985, whose centralizing impetus was basically moribund by 1995 --
core projects like GCC and glibc spawning forks and threatening to
bolt the GNU fold entirely, the kernel itself never considering
becoming a GNU project.  Well, the rest of the world made the GNU
System its own, and chose to denote it by the component that they
found inspirational.

Please note that everything written above is *observation*, not
*justification*.  You're welcome to promote the name "GNU/Linux",
because it's meaningful and true.  But I wish you'd drop the "but we
built it!" argument and find something genuine to say.  Your petulance
in insisting that GNU alone deserves credit for the whole makes it
embarrassing to use that term outside of GNU channels.  I still use it
because it makes a couple of points for me (and about me).  But I have
to be on guard, and I often am called on to justify it, when I do.


Footnotes: 
[1]  This is not to deny that many members of GNU projects rejoice in
their association with the GNU Project.  Nor that core members of the
GNU Project, starting with yourself, deliberately set about to create
crucial missing components such as GCC and glibc, and extremely useful
utilities such as Emacs, gdb, and Make.  The point is that as leader
of the GNU Project nowadays (and for the last decade or two) you busy
yourself not with ensuring that the GNU System is excellent, but with
political advocacy and *preventing* addition of software that you
consider dangerous to software freedom to the GNU System.  How ironic
in the context of a claim that GNU is the primary *developer* of the
software distributed by so-called "Linux distributors"!

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

     > This discussion is a response to your fallacious call for people to
     > deny us credit for our work.

    Richard, I did no such thing.  I called for you to give the same
    credit to others you demand for yourself.

Six of one, and half dozen of the other.  You made a point as part of
a larger argument, and http://www.gnu.org/gnu/gnu-linux-faq.html#many
responds to both.

    But that justification is misdirected, given that the GNU Project with
    an uppercase "P" does almost no development.

The GNU Project does development in the same way it always did, by
recruiting volunteers to work on projects.  But it is a tangent
anyway.  Whether these volunteers are loyal to our ideals or to GNU
itself is a tangent too.  I won't try to address all the tangents and
spin you brought up.

People have made a thousand versions of GNU/Linux, because it is free
software.  The GNU Project is the main developer of the system as a
whole; what we did in the 1980s, alone, is enough to make that true.
GNU contributed more to the system than any other one project.

Arguing about this is outside the purpose of this list.
We have a list for such discussion: gnu-misc-discuss.
Please take it there, and stop raising it here.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[Stephen J. Turnbull]

Richard Stallman writes:

 >      > This discussion is a response to your fallacious call for
 >      > people to deny us credit for our work.
 > 
 >     Richard, I did no such thing.  I called for you to give the same
 >     credit to others you demand for yourself.
 > 
 > Six of one, and half dozen of the other.

I'm happy to leave that to the readers to judge.  You might want to
ask the opinions of some you trust.

 > Arguing about this is outside the purpose of this list.

I know that.  I'd be happy to remain mute, if you would "correct" me
without using insulting and factive rhetoric (such as the gratuitious
slur "fallacious" quoted above), and attributing motives to me which I
do not in fact possess.

 > We have a list for such discussion: gnu-misc-discuss.
 > Please take it there, and stop raising it here.

Excuse me for the correction, but I didn't raise it.  You did.

I made the point that there is a wide variety of packages and
development organizations participating in GNU, many of which are
otherwise independent of GNU, in the context of the question "isn't
GNU inconsistent when Emacs requires assignments but many other GNU
projects don't?"

You took exception to my expression and made several statements on
*this* mailing list, some of which are evidently false according to
your own FAQ, in terms detrimental to my reputation.  I defended
myself here, where those statements were made.

Re: Emacs contributions, C and Lisp

[Michal Nazarewicz]

[-- Attachment #1: Type: text/plain, Size: 1041 bytes --]

On Fri, Mar 28 2014, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> Also, this limit is only useful if it completely avoids a CA, not if it
> just delays it.  If someone contributed 50 patches, he's likely to end
> up wanting/able to contribute more than 100.

That's exactly the point.  Make first few patches as easy to submit as
possible, and once the contributor is familiar with the process, she
won't be intimidated by a CA so much.

I'm aware of the “20 lines worth of code” limit, because my very first
patch fell into that category.  And I was glad I did not have to sign
the CA.

CA may not be that big of an issue, but for someone who's about to send
her first patch to an unknown group of people it's yet another obstacle
to overcome.

-- 
Best regards,                                         _     _
.o. | Liege of Serenely Enlightened Majesty of      o' \,=./ `o
..o | Computer Science,  Michał “mina86” Nazarewicz    (o o)
ooo +--<mpn@google.com>--<xmpp:mina86@jabber.org>--ooO--(_)--Ooo--

[-- Attachment #2.1: Type: text/plain, Size: 0 bytes --]



[-- Attachment #2.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 835 bytes --]

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

     > We have a list for such discussion: gnu-misc-discuss.
     > Please take it there, and stop raising it here.

    Excuse me for the correction, but I didn't raise it.  You did.

"It" means attacking the GNU Project's general positions, policies,
goals, etc.

    I made the point that there is a wide variety of packages and
    development organizations participating in GNU, many of which are
    otherwise independent of GNU, in the context of the question "isn't
    GNU inconsistent when Emacs requires assignments but many other GNU
    projects don't?"

Yes, you did -- as an argument against a GNU Project position.
Earlier I explained why the argument was invalid.  Now the point
is that such arguments do not belong on this list.

We have a list for that sort of discussion: gnu-misc-discuss.
Please take it there.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[Stephen J. Turnbull]

Richard Stallman writes:

 > "It" means attacking the GNU Project's general positions, policies,
 > goals, etc.

[snip]

 > We have a list for that sort of discussion: gnu-misc-discuss.

I am not discussing GNU whatevers, I am discussing your attacks on me,
*on emacs-devel*, and I will answer where those attacks take place.

I am not hostile to GNU, I do not "attack" its general whatevers.  I
disagree with particular whatevers where I think they are self-
defeating (in this case, the FAQ concerning the appropriate way to
refer to the system distributed by the distros).

Grant me that and I will make efforts to avoid snide remarks, like the
one that occasioned this subthread, in the future.

Re: Emacs contributions, C and Lisp

[Stephen J. Turnbull]

Stephen J. Turnbull writes:

 > Grant me that and I will make efforts to avoid snide remarks, like the
 > one that occasioned this subthread, in the future.

Please read the above as "Grant me that, please.  I will make efforts
to avoid snide remarks."  The former is not a condition for the latter.

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    I am not discussing GNU whatevers, I am discussing your attacks on me,
    *on emacs-devel*, and I will answer where those attacks take place.

Perhaps you were doing the latter, but what matters is that you were
doing the former.  Attacking the GNU project's positions and
statements is unwelcome on this list.

That your words could also be described in some other way
does not make them ok.

    I do not "attack" its general whatevers.  I
    disagree with particular whatevers where I think they are self-
    defeating

By "general" I mean not specifically about Emacs.  This list is for
discussing Emacs development.  I mention general GNU policies here
when they are relevant to the discussion here or to Emacs development.

When I mention them, it is ok to ask me to explain.  But if you
disagree with them and want to argue about whether they are proper,
you should do that on gnu-misc-discuss.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[Stefan Monnier]

Can you guys take this off-list, please?


        Stefan

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    Can you guys take this off-list, please?

Upholding the GNU rules for using this list
is entirely proper use of the list.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I let this drop back in March -- please forgive me.

  > Maybe that's the issue for GCC, but for Emacs the issue is to get detailed
  > info out of GCC, which is a different problem.  My understanding is that
  > you're opposed to GCC providing this useful info because that info would
  > need to be complete enough to be usable as input to a proprietary
  > compiler backend.

My hope is that we can work out a kind of "detailed output" that is
enough for what Emacs wants, but not enough for misuse of GCC front ends.

I don't want to discuss the details on the list, because I think that
would mean 50 messages of misunderstanding and tangents for each
message that makes progress.  Instead, is there anyone here who would
like to work on this in detail?


-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[David Engster]

Richard Stallman writes:
> I let this drop back in March -- please forgive me.
>
>   > Maybe that's the issue for GCC, but for Emacs the issue is to get detailed
>   > info out of GCC, which is a different problem.  My understanding is that
>   > you're opposed to GCC providing this useful info because that info would
>   > need to be complete enough to be usable as input to a proprietary
>   > compiler backend.
>
> My hope is that we can work out a kind of "detailed output" that is
> enough for what Emacs wants, but not enough for misuse of GCC front ends.

Anyone can write a GCC plugin that simply outputs the AST in some
form. It's not that hard. The plugin itself would have to export the
symbol 'plugin_is_GPL_compatible', but of course you can't control
what's done with the output. No on bothers with this though, because
everyone just uses libclang.

> I don't want to discuss the details on the list, because I think that
> would mean 50 messages of misunderstanding and tangents for each
> message that makes progress.  Instead, is there anyone here who would
> like to work on this in detail?

I'm already working on it.

-David

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > My hope is that we can work out a kind of "detailed output" that is
  > > enough for what Emacs wants, but not enough for misuse of GCC front ends.

  > Anyone can write a GCC plugin that simply outputs the AST in some
  > form.

We are not talking about the same thing.  What I have in mind is to
output considerably LESS than the AST -- just enough to do the
completion and other operations that Emacs needs, and NO MORE.

To figure out just what Emacs needs, that's the task I am talking
about.

  >   Instead, is there anyone here who would
  > > like to work on this in detail?

  > I'm already working on it.

Are you working on outputting just the data that Emacs needs for
completion?

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[David Engster]

Richard Stallman writes:
>   > > My hope is that we can work out a kind of "detailed output" that is
>   > > enough for what Emacs wants, but not enough for misuse of GCC front ends.
>
>   > Anyone can write a GCC plugin that simply outputs the AST in some
>   > form.
>
> We are not talking about the same thing.  What I have in mind is to
> output considerably LESS than the AST -- just enough to do the
> completion and other operations that Emacs needs, and NO MORE.

I understand that. What I'm saying is: For almost five years now (since
gcc 4.5 introduced plugins), access to GCC's AST is wide open for
everyone. However, in all that time (and to my knowledge) no one has
used that to feed non-free backends, and that is in my opinion enough
evidence that your worries are unfounded. They might have been valid in
the past, but not since LLVM and clang have joined the scene.

> To figure out just what Emacs needs, that's the task I am talking
> about.

If you want to support things like completions, refactoring, symbol
searches, etc., we need full access to the AST from inside Emacs.

>
>   >   Instead, is there anyone here who would
>   > > like to work on this in detail?
>
>   > I'm already working on it.
>
> Are you working on outputting just the data that Emacs needs for
> completion?

Completion is only the starting point. If that works, I hope more stuff
will follow. In any case, I will need the full AST. If you are saying
that you won't allow that, then this is a non-starter. Dealing with
GCC's AST is hard enough already; if I have to additionally worry what
information I'm allowed to feed to Emacs, than I'll simply stop working
on this.

-David

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I understand that. What I'm saying is: For almost five years now (since
  > gcc 4.5 introduced plugins), access to GCC's AST is wide open for
  > everyone. However, in all that time (and to my knowledge) no one has
  > used that to feed non-free backends, and that is in my opinion enough
  > evidence that your worries are unfounded. They might have been valid in
  > the past, but not since LLVM and clang have joined the scene.

Since LLVM and Clang are not copylefted, they invite nonfree extensions.
They are a gaping hole in the defensive wall around our city.

  > > To figure out just what Emacs needs, that's the task I am talking
  > > about.

  > If you want to support things like completions, refactoring, symbol
  > searches, etc., we need full access to the AST from inside Emacs.

With all due respect, it is so important to avoid the full AST
that I'm not going to give up on it just because someone claims
that is necessary.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[Eli Zaretskii]

> Date: Mon, 05 Jan 2015 12:50:42 -0500
> From: Richard Stallman <rms@gnu.org>
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
>   > If you want to support things like completions, refactoring, symbol
>   > searches, etc., we need full access to the AST from inside Emacs.
> 
> With all due respect, it is so important to avoid the full AST
> that I'm not going to give up on it just because someone claims
> that is necessary.

What would you need to know or hear to make up your mind on this
issue?  IOW, what information is missing for you to agree or disagree
with David on this?

Re: Emacs contributions, C and Lisp

[Perry E. Metzger]

On Mon, 05 Jan 2015 12:50:42 -0500 Richard Stallman <rms@gnu.org>
wrote:
>   > I understand that. What I'm saying is: For almost five years
>   > now (since gcc 4.5 introduced plugins), access to GCC's AST is
>   > wide open for everyone. However, in all that time (and to my
>   > knowledge) no one has used that to feed non-free backends, and
>   > that is in my opinion enough evidence that your worries are
>   > unfounded. They might have been valid in the past, but not
>   > since LLVM and clang have joined the scene.
> 
> Since LLVM and Clang are not copylefted, they invite nonfree
> extensions. They are a gaping hole in the defensive wall around our
> city.
> 
>   > > To figure out just what Emacs needs, that's the task I am
>   > > talking about.
> 
>   > If you want to support things like completions, refactoring,
>   > symbol searches, etc., we need full access to the AST from
>   > inside Emacs.
> 
> With all due respect, it is so important to avoid the full AST
> that I'm not going to give up on it just because someone claims
> that is necessary.

So, I've been doing a bunch of complicated refactoring work in
conjunction with my current academic research, and I've been forced
to use clang/LLVM because I need the full AST. For example, I have to
be able to go in and do things like finding all instances of
particular kinds of pointer arithmetic and turn them into stylized
forms of array indexing (the details of why are too complicated for
even a fairly long email message.) It is only possible to do this
because I have a full AST available.

clang/LLVM also provide a very friendly set of libraries to make it
easy for people to go in, walk the AST, pattern match on it, refactor
it, etc. -- the tools are being used extensively in industry for
refactoring support.

I understand your feelings on the whole thing, but the whole reason I
use LLVM for my current work is this capability. gcc just won't give
me appropriate hooks.

The libclang AST manipulation functionality was originally created
because Apple wanted it to enable sophisticated IDE and refactoring
capabilities in XCode's editor. I have wanted to have all that stuff
in Emacs forever, but it is hard without having access to tools that
generate a full AST of the code being examined. For Emacs to be able
to compete with non-free tools, it is important that it have access
to similar capabilities as the non-free tools have.

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Emacs contributions, C and Lisp

[David Engster]

Richard Stallman writes:
>   > I understand that. What I'm saying is: For almost five years now (since
>   > gcc 4.5 introduced plugins), access to GCC's AST is wide open for
>   > everyone. However, in all that time (and to my knowledge) no one has
>   > used that to feed non-free backends, and that is in my opinion enough
>   > evidence that your worries are unfounded. They might have been valid in
>   > the past, but not since LLVM and clang have joined the scene.
>
> Since LLVM and Clang are not copylefted, they invite nonfree extensions.
> They are a gaping hole in the defensive wall around our city.

It seems we are unable to communicate. My paragraph above was not about
Clang and LLVM, but that the door to GCC's AST is already open. That
ship has sailed.

>   > > To figure out just what Emacs needs, that's the task I am talking
>   > > about.
>
>   > If you want to support things like completions, refactoring, symbol
>   > searches, etc., we need full access to the AST from inside Emacs.
>
> With all due respect, it is so important to avoid the full AST
> that I'm not going to give up on it just because someone claims
> that is necessary.

And with the same respect, I choose to not invest more time on this. It
was you who told me to abandon libclang and choose GCC instead. And now
that I'm working on that, I only get confronted with vague restrictions
like "you may only export what you need for completions".

Of course, the nice thing about free software is that I don't need your
approval. However, it is clear to me now that I cannot depend on GCC's
plugin infrastructure, as there's the very real risk that you shut that
door as soon as someone uses it to export the AST.

-David

Re: Emacs contributions, C and Lisp

[Stefan Monnier]

>> > To figure out just what Emacs needs, that's the task I am talking
>> > about.
>> If you want to support things like completions, refactoring, symbol
>> searches, etc., we need full access to the AST from inside Emacs.
> With all due respect, it is so important to avoid the full AST
> that I'm not going to give up on it just because someone claims
> that is necessary.

Give up on what, exactly?

Richard, this is absurd!
I'm convinced there is a misunderstanding.

David wants to write a GCC plugin that outputs the complete AST.
There's absolutely nothing stopping him (or anyone else) doing that
since GCC's current plugin interface does allow for that (and has been
doing so for the last 5 years, apparently).

Are you saying that if David writes this, and writes code for Emacs to
make use of it, you're going to use your leverage to try and make sure
this code doesn't make it into Emacs?
How would the Free Software movement benefit from that?


        Stefan

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > With all due respect, it is so important to avoid the full AST
  > > that I'm not going to give up on it just because someone claims
  > > that is necessary.

  > What would you need to know or hear to make up your mind on this
  > issue?

I'd like to see an investigation that establishes just what data is
needed for each purpose.


-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[Perry E. Metzger]

On Tue, 06 Jan 2015 13:24:44 -0500 Richard Stallman <rms@gnu.org>
wrote:
> [[[ To any NSA and FBI agents reading my email: please consider
> ]]] [[[ whether defending the US Constitution against all
> enemies,     ]]] [[[ foreign or domestic, requires you to follow
> Snowden's example. ]]]
> 
>   > > With all due respect, it is so important to avoid the full AST
>   > > that I'm not going to give up on it just because someone
>   > > claims that is necessary.
> 
>   > What would you need to know or hear to make up your mind on this
>   > issue?
> 
> I'd like to see an investigation that establishes just what data is
> needed for each purpose.

Say you want to do a global refactoring of a function that alters the
parameters it takes -- the sort of thing modern IDEs do very
well. You can't do that very effectively without the AST. Moreover,
say you want to globally find all instances where a particular
function's return is not tested for NULL and have the user manually
check that they're not bugs -- this is the sort of thing modern
development tools can do, and which you can't do without an AST.

The Go programming language, during its development phase, relied
heavily on a tool that updated old APIs and syntactic conventions
into new ones to permit the code base to be modified constantly from
old to new when serious changes were made to the language and its
libraries.

Those are pretty straightforward -- there are far more remarkable
sorts of high level refactorings that now get done pretty
straightforwardly with AST aware tools. The guys at Google who build
a bunch of the LLVM libAST stuff have demos of doing very serious C++
refactorings in which large amounts of code gets rewritten to
accommodate new paradigms.

Programmers like programmable tools. That's one of the reasons Emacs
is so successful. Being able to program your editor to do massive
transformations of your source code is a really cool thing -- Emacs
could actually be far better at this than XCode, Eclipse and all the
rest if it could actually get access to AST information.


Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Emacs contributions, C and Lisp

[Eli Zaretskii]

> Date: Tue, 6 Jan 2015 14:39:33 -0500
> From: "Perry E. Metzger" <perry@piermont.com>
> Cc: Eli Zaretskii <eliz@gnu.org>, monnier@iro.umontreal.ca,
> 	deng@randomsample.de, emacs-devel@gnu.org
> 
> Say you want to do a global refactoring of a function that alters the
> parameters it takes -- the sort of thing modern IDEs do very
> well.

Please also describe why the AST is needed for completion, which was
the original topic discussed here.

Re: Emacs contributions, C and Lisp

[Perry E. Metzger]

On Tue, 06 Jan 2015 22:37:01 +0200 Eli Zaretskii <eliz@gnu.org> wrote:
> > Date: Tue, 6 Jan 2015 14:39:33 -0500
> > From: "Perry E. Metzger" <perry@piermont.com>
> > Cc: Eli Zaretskii <eliz@gnu.org>, monnier@iro.umontreal.ca,
> > 	deng@randomsample.de, emacs-devel@gnu.org
> > 
> > Say you want to do a global refactoring of a function that alters
> > the parameters it takes -- the sort of thing modern IDEs do very
> > well.
> 
> Please also describe why the AST is needed for completion, which was
> the original topic discussed here.

Oh, well for C and C++, given that you can't know the scope of an
identifier (or even what it is -- see the awful C typedef problem in
parsing) without pretty good scope and symbol table info, I had
thought that was obvious. If it isn't, I'm happy to explain in
detail, but I had thought that Richard had written a bunch of the
early GCC stuff himself and understood that.

In a modern IDE, you can complete anything -- struct members or
object member variables/functions, type names, etc., etc., and you
need what are essentially compiler data structures to get that stuff,
and you need the AST to understand the context in which you're
requesting the completion. You can't even know that the variable
"foo" is an object of class "baz" and thus that when you hit complete
on foo.con that it should look for the members that start with "con"
in objects of class "baz" (say "construct()" or some such).

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Emacs contributions, C and Lisp

[Eli Zaretskii]

> Date: Tue, 6 Jan 2015 15:45:39 -0500
> From: "Perry E. Metzger" <perry@piermont.com>
> Cc: rms@gnu.org, monnier@iro.umontreal.ca, deng@randomsample.de,
>  emacs-devel@gnu.org
> 
> > Please also describe why the AST is needed for completion, which was
> > the original topic discussed here.
> 
> Oh, well for C and C++, given that you can't know the scope of an
> identifier (or even what it is -- see the awful C typedef problem in
> parsing) without pretty good scope and symbol table info, I had
> thought that was obvious. If it isn't, I'm happy to explain in
> detail, but I had thought that Richard had written a bunch of the
> early GCC stuff himself and understood that.

I think that this is only necessary for OO languages, so you can
complete on C identifiers without the AST.  C++, OTOH, is another
matter.  So I think an explanation with examples will help here.

Thanks.

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > With all due respect, it is so important to avoid the full AST
  > > that I'm not going to give up on it just because someone claims
  > > that is necessary.

  > Give up on what, exactly?

Give up on the idea of generating and using data that is less than the
full AST.

  > David wants to write a GCC plugin that outputs the complete AST.

We do not want to promote that sort of plugin.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[Stefan Monnier]

>> David wants to write a GCC plugin that outputs the complete AST.
> We do not want to promote that sort of plugin.

What exactly does that mean?  IIUC, it's *easy* to write such a plugin,
so the only reason nobody's done it so far is because it's even easier
to do it with clang, so noone but a true Free Software advocate would
bother writing/using such a plugin.
IOW discouraging such plugins only hurts *us*.


        Stefan

Re: Emacs contributions, C and Lisp

[David Engster]

Richard Stallman writes:
>   > David wants to write a GCC plugin that outputs the complete AST.
>
> We do not want to promote that sort of plugin.

After reading a bit about how GCC plugins came into being, I stumbled
upon

  https://www.gnu.org/licenses/gcc-exception-3.1-faq.html

which has this:

    When we first considered adding a plugin infrastructure to GCC, we
    were deeply concerned about the possibility that someone would write
    a plugin that would merely save GCC's internal, low-level
    compilation data structures to disk. With that done, other software
    would be able to optimize or otherwise improve that code without
    being directly connected to GCC. It may have been difficult for us
    to argue that those programs should be subject to the GPL's
    copyleft, so we wanted to discourage these sorts of arrangements.

    We do that by excluding such output from the definition of Target
    Code. Because of this, even if someone writes a plugin that saves
    this information to disk, any programs that change the structures
    before GCC writes out Target Code will be involved in the
    Compilation Process. If that program is proprietary, the exception
    will not be available to any software compiled with it; the object
    code that GCC ultimately creates will have to be distributed under
    the terms of the GPL.

Now I'm even more confused why you'd have a problem with exporting the
AST.

-David

Re: Emacs contributions, C and Lisp

[Eli Zaretskii]

> Date: Tue, 06 Jan 2015 23:26:48 -0500
> From: Richard Stallman <rms@gnu.org>
> Cc: deng@randomsample.de, emacs-devel@gnu.org
> 
> Give up on the idea of generating and using data that is less than the
> full AST.
> 
>   > David wants to write a GCC plugin that outputs the complete AST.
> 
> We do not want to promote that sort of plugin.

But if access to the AST is necessary for providing important
features, such as intelligent completion and refactoring in OO
languages, then you are saying that Emacs users can never have those
important features, unless they go out and use non-free or less-free
or GNU-unfriendly software instead of GCC.

Without these feature, Emacs is much less powerful an editor and a
programming environment than it could have been.  And the GCC plugin
architecture already requires GPL compliance, so having such a plugin
should not in itself pose any danger to Free Software and the GNU
project.

Please help us understand why we as a community should pay such a high
price where seemingly there's no real problem.

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > In a modern IDE, you can complete anything -- struct members or
  > object member variables/functions, type names, etc., etc., and you
  > need what are essentially compiler data structures to get that stuff,
  > and you need the AST to understand the context in which you're
  > requesting the completion.

I am not convinced this requires the entire AST.
I think it could be done with a lot less.

It certainly can be done with less.  For instance, if the code says

  foo.x + bar.y

or

  foo.x - bar.y

it makes no difference for completion what that operator is.  Clearly
there is information from the AST that need not be included in order
to fully support completion.

I think you propose "entire AST" because it is conceptually satisfying
to you, not because all the data in the AST are necessary.

What data ARE necessary?

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[Eli Zaretskii]

> Date: Wed, 07 Jan 2015 14:25:01 -0500
> From: Richard Stallman <rms@gnu.org>
> CC: eliz@gnu.org, monnier@iro.umontreal.ca, deng@randomsample.de,
> 	emacs-devel@gnu.org
> 
> It certainly can be done with less.  For instance, if the code says
> 
>   foo.x + bar.y
> 
> or
> 
>   foo.x - bar.y
> 
> it makes no difference for completion what that operator is.

In an object-oriented language that supports operator overloading, the
+ or - can do anything, and accept only specific data types under
certain constraints.  For example, if foo.x is of a certain data type,
the candidates for the right-hand operand might be restricted to a
small subset of what a + or a - can generally support.  If you
complete to a large list of candidates here disregarding the
constraints, you'd leave a lot of users unhappy.

How do you know that kind of information without parsing the whole
program yourself?

Re: Emacs contributions, C and Lisp

[David Kastrup]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > In a modern IDE, you can complete anything -- struct members or
>   > object member variables/functions, type names, etc., etc., and you
>   > need what are essentially compiler data structures to get that stuff,
>   > and you need the AST to understand the context in which you're
>   > requesting the completion.
>
> I am not convinced this requires the entire AST.
> I think it could be done with a lot less.
>
> It certainly can be done with less.  For instance, if the code says
>
>   foo.x + bar.y
>
> or
>
>   foo.x - bar.y
>
> it makes no difference for completion what that operator is.

In C++, I am afraid you are wrong.  It depends on what types operator+
and operator- may be defined for.  If you define operator+ on a
half-group, you won't generally have operator- available.

Operator overloading is a reality in C++.  For example, << and >> are
seriously overloaded for the <iostream> standard library.

> Clearly there is information from the AST that need not be included in
> order to fully support completion.

Probably.  However, if you don't want to cause major pain when
reevaluating your requirements, it is important that you don't need to
change/recompile several _differently_ maintained and versioned pieces
of software in lockstep to adapt to changed requirements.  If you can
shake plugins interfacing to Elisp out from GCC without having them tied
to a particular version of GCC basically on the fly, you don't need to
have the whole AST available in a physical form.

But the plugin interface then needs to be general and convenient enough
to get any aspect of it on-demand.  And with regard to enabling
undesirable uses of it, I don't see that this buys us significant
savings over a non-Emacs specific AST dump.

> I think you propose "entire AST" because it is conceptually satisfying
> to you, not because all the data in the AST are necessary.
>
> What data ARE necessary?

It depends on the level of sophistication in the editing modes.  If we
want the editing modes to be able to evolve without being constrained to
the GCC release cycles, we need either rather generically useful data,
or generically useful GCC extension interfaces.

Keeping all of our babies while throwing out all of the bath water for
evolving notions of babies and bathwater is not likely to work well.

I don't really see that we have much of a chance not to be hobbling us
ourselves more in the process than currently hypothetical leeches.

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > >   foo.x - bar.y
  > >
  > > it makes no difference for completion what that operator is.

  > In C++, I am afraid you are wrong.  It depends on what types operator+
  > and operator- may be defined for.  If you define operator+ on a
  > half-group, you won't generally have operator- available.

Could you explain how that is so?

If + and - accept different types,
how does that affect completion?

Eli tried to explain

  > In an object-oriented language that supports operator overloading, the
  > + or - can do anything, and accept only specific data types under
  > certain constraints.  For example, if foo.x is of a certain data type,
  > the candidates for the right-hand operand might be restricted to a
  > small subset of what a + or a - can generally support.  If you
  > complete to a large list of candidates here disregarding the
  > constraints, you'd leave a lot of users unhappy.

but he didn't give the substance, so I still don't follow.

I have never used C++.

  > But the plugin interface then needs to be general and convenient enough
  > to get any aspect of it on-demand.  And with regard to enabling
  > undesirable uses of it, I don't see that this buys us significant
  > savings over a non-Emacs specific AST dump.

Are you thinking of this only in terms of the amount of work?
I'm concerned with avoiding dangers.

If you want to convince me generating the whole AST is safe, you have
to understand my concern and take it seriously.  You can't do this by
brushing off the issue.  That will only convince me that you haven't seen
the issue.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  >     We do that by excluding such output from the definition of Target
  >     Code. Because of this, even if someone writes a plugin that saves
  >     this information to disk, any programs that change the structures
  >     before GCC writes out Target Code will be involved in the
  >     Compilation Process. If that program is proprietary, the exception
  >     will not be available to any software compiled with it; the object
  >     code that GCC ultimately creates will have to be distributed under
  >     the terms of the GPL.

  > Now I'm even more confused why you'd have a problem with exporting the
  > AST.

These conditions on libgcc will be effective if a plug-in is used,
provided compilation continues thru the GCC back end, because the GCC
back end makes code that depends on libgcc.

But if the plug-in writes the whole program as an AST and compilation
uses some other back-end, it probably won't use libgcc, so those
conditions on libgcc won't apply at all.

I am very concerned lest the GCC back ends to be replaced with LLVM,
which permits proprietary changes and proprietary back ends.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[Óscar Fuentes]

Richard Stallman <rms@gnu.org> writes:

> But if the plug-in writes the whole program as an AST and compilation
> uses some other back-end, it probably won't use libgcc, so those
> conditions on libgcc won't apply at all.
>
> I am very concerned lest the GCC back ends to be replaced with LLVM,
> which permits proprietary changes and proprietary back ends.

That already happened long time ago:

http://dragonegg.llvm.org/

It is now abandoned, mostly because Clang is a better front-end than
GCC.

One key fact that was repeatedly mentioned but you insist on ignoring is
that, from the technical POV, Clang+LLVM is far more attractive than GCC
for any compiler-related task, and on top of that it has no legal
hurdles, so only a clueless proprietary developer would choose GCC as a
basis for his compiler-related product.

The bad news for you is that people that are creating Free Software or
are learning compiler development are using Clang+LLVM too. GCC is only
a reasonable option if you veto Clang+LLVM for political reasons, but
even then you insist on adding difficulties to the work of those who are
amenable enough to play by your rules.

Re: Emacs contributions, C and Lisp

[Óscar Fuentes]

Richard Stallman <rms@gnu.org> writes:

[snip]

> Eli tried to explain
>
>   > In an object-oriented language that supports operator overloading, the
>   > + or - can do anything, and accept only specific data types under
>   > certain constraints.  For example, if foo.x is of a certain data type,
>   > the candidates for the right-hand operand might be restricted to a
>   > small subset of what a + or a - can generally support.  If you
>   > complete to a large list of candidates here disregarding the
>   > constraints, you'd leave a lot of users unhappy.
>
> but he didn't give the substance, so I still don't follow.
>
> I have never used C++.

Possibly this explanation is more familiar to you: C++ has type
deduction for declarations:

auto some_var = foo(bar);

and when the compiler resolves `foo' it can choose an overload depending
on complex attributes of `bar' (such as if `bar' is an instance of a
class that implements certain method.) As each overload can return a
different type, only a C++ compiler can know which type `some_var'
finally is.

This way C++ is a bit like Lisp. In fact, the latest C++ standard
requires the existence of a compile-time interpreter that covers a fair
subset of the C++ language. The same way that it is useful to inspect
Lisp from an IDE, a C++ IDE can benefit a lot from doing that. Actually,
it can benefit more than Lisp, because the presence of types (explicit
or implicit) provides rich information that allows fine-grained features
such as batch code transformations and real-time code analysis.

[snip]

Re: Emacs contributions, C and Lisp

[David Engster]

Richard Stallman writes:
> I am very concerned lest the GCC back ends to be replaced with LLVM,
> which permits proprietary changes and proprietary back ends.

As Oscar already mentioned, this has already happened, although this
effort is now largely abandoned. It might be revived of course, but
there's not much incentive to do so. Instead, people rather right new
frontends. For instance, there was already work done on 'flang', a
Fortran frontend, and new languages like Mozilla's "Rust" use LLVM as
their backend right from the beginning.

That aside, I would dump the AST in some form of LISP structure and
probably for other tools as JSON and XML (I don't want this to be an
Emacs-only thing). For feeding LLVM however, one would much rather use
one of GCC's intermediate representations, to which LLVM was designed to
be more or less compatible.

-David

Re: Emacs contributions, C and Lisp

[Perry E. Metzger]

On Wed, 07 Jan 2015 21:46:14 -0500 Richard Stallman <rms@gnu.org>
wrote:
> Are you thinking of this only in terms of the amount of work?
> I'm concerned with avoiding dangers.

Here is a real danger: people are ignoring free software as a way to
get their work done because non-free software provides them with
capabilities that free software does not. The danger is even worse
because the people using non-free software to do their work in this
instance are programmers, who are learning that if you want to be
productive, you use software like XCode, and not software like Emacs.
In the longer term, this could very well reduce the pool of
developers who are part of the free software ecosystem.

Equally bad from your point of view, and unrelated to editors, I'll
personally testify that the academic programming languages community
is pouring all its effort into LLVM from every side, and this is
because LLVM can be used in a modular manner without someone having
figured out in advance what sort of work someone might have wanted to
do.

I can think of a half dozen projects at the University of
Pennsylvania (my university), VeLLVM, Ironclad C++, and my own work
on C safety being just three, where no one used GCC as a platform
specifically because of the more open architecture in LLVM. You
can find literally dozens of projects done at other universities,
such as John Regehr's work on integer overflow detection in C and
C++, where LLVM was the only platform considered for the work because
it is considered impossible to use GCC in a modular fashion.

A whole generation of compiler experts is being trained to use LLVM
and only LLVM for this sort of work, where 15 years ago they would
have used GCC.

The long term result of all of this may very well be to do exactly the
opposite of what you want -- to convince compiler researchers that
LLVM is the only serious platform for their work, and even worse, to
convince developers in general that free software is too hard to use
and that non-free software is the way for them to get their work done.

> If you want to convince me generating the whole AST is safe, you
> have to understand my concern and take it seriously.

We do indeed understand I think, and take it seriously. The issue is a
real one. I think the distinction is the rest of us see the tradeoff
differently than you do.

We do understand that the choice may not be perfectly "safe" in the
sense you mean. People may very well do what you suggest and build
non-free compiler components on top of free compilers, just as they
run proprietary software on top of GNU/Linux.

However, allowing people run some proprietary software on top of
GNU/Linux has driven literally every other Unix-like operating system
into the ground while slowly eroding the proprietary software
itself. Sure, Oracle runs on GNU/Linux, but almost every database
system out there is a free database, not something like Oracle.

The LGPL gives up a powerful tool in exchange for an important result.
It is true that the Gnu C library could have prevented itself
from being linked in to proprietary software, and this is nearly the
same situation as you face with GCC, but this would have been
counterproductive, causing proprietary software users to avoid
GNU/Linux entirely. Instead, they got absorbed into the ecosystem and
ultimately it was the proprietary software that was heavily damaged,
not FSF's goals.

GCC is often used to compile proprietary software, but allowing GCC to
be used that way also meant that the closed source compiler platforms
were heavily damaged.

The current situation is one in which there are lots of proprietary
IDEs that use hooks onto compilers, some into proprietary ones and
some into LLVM, to let people do all sorts of very useful things. It
is also a situation where loads of people want to do hacks on top of
compiler infrastructure and have found that LLVM is the only easy way
to do that. Yes, there is a trade-off here, but we think it is
important enough to permit, because for practical purposes the result
of not allowing the usage will be to push people further in a
direction we do not want.

> You can't do this by brushing off the issue.  That will only
> convince me that you haven't seen the issue.

I think most of us understand the issue as you see it. I think the
distinction is that most of us believe the trade-off is important. Yes,
this may indeed mean that some proprietary software ends up being
based on GCC just as some proprietary software is based on GNU/Linux,
but the overall impact will be positive, and the risk is much lower
than the reward.

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Emacs contributions, C and Lisp

[Stefan Monnier]

> A whole generation of compiler experts is being trained to use LLVM
> and only LLVM for this sort of work, where 15 years ago they would
> have used GCC.

Indeed.

> The long term result of all of this may very well be to do exactly the
> opposite of what you want -- to convince compiler researchers that
> LLVM is the only serious platform for their work, and even worse, to
> convince developers in general that free software is too hard to use
> and that non-free software is the way for them to get their work done.

Let's not forget that LLVM is Free.  It's not as Free as GCC since it
doesn't use a copyleft license, but it's not proprietary.

> I think most of us understand the issue as you see it. I think the
> distinction is that most of us believe the trade-off is important. Yes,
> this may indeed mean that some proprietary software ends up being
> based on GCC just as some proprietary software is based on GNU/Linux,
> but the overall impact will be positive, and the risk is much lower
> than the reward.

I think this "much" is an understatement.


        Stefan

Re: Emacs contributions, C and Lisp

[David Kastrup]

"Perry E. Metzger" <perry@piermont.com> writes:

> The long term result of all of this may very well be to do exactly the
> opposite of what you want -- to convince compiler researchers that
> LLVM is the only serious platform for their work, and even worse, to
> convince developers in general that free software is too hard to use
> and that non-free software is the way for them to get their work done.

LLVM is not non-free software.  It is non-copyleft software.  The
compiler researchers you are talking about are using free software, but
their work is ultimately also enabling non-free software that can no
longer be used in open research.

For the academics, we are not talking about the difficulties of using
free software vs non-free software but rather about using (and
consequently also producing) copylefted software vs non-copylefted
software: actual non-free software is hard to work with and write about
anyway.

Academia of course is also set up to serve industries with "standard"
production, marketing, and licensing models.

>> If you want to convince me generating the whole AST is safe, you have
>> to understand my concern and take it seriously.
>
> We do indeed understand I think, and take it seriously. The issue is a
> real one. I think the distinction is the rest of us see the tradeoff
> differently than you do.

Most of the "if only $x, then magically $y" scenarios don't really work
as thoroughly as people imagine and end up a tempest in a teapot.
I suspect that there would be a lot more catching up to do apart from
unencumbering module/data interfaces into GCC.

However, we _do_ apparently have people chomping at the bit to improve
the integration of GCC into Emacs as a programming editor which have
serious problems getting a workable roadmap agreed on, and the mere
necessity for getting everything agreed upon in advance is a real
detriment: most work requires exploring a number of technical dead ends
before finding a productive path, and when each of those dead ends
requires a big upfront fight, the perceived cost in disappointment and
lost efforts is much higher.

We are losing contact with academia, that's true.  I don't see a
short-term fix for that.  But we are losing also contact with willing
hackers who'd actually want to work on Emacs/GCC integration and who'd
be perfectly willing to just leave the legal considerations in the hand
of the FSF but who are barred from working on the implementation.

And that's really worrying.  I think there's too much baby in the
bathwater we are throwing out here.

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Thu, 08 Jan 2015 16:03:16 +0100
> Cc: eliz@gnu.org, emacs-devel@gnu.org, Richard Stallman <rms@gnu.org>,
> 	deng@randomsample.de, monnier@iro.umontreal.ca
> 
> We are losing contact with academia, that's true.  I don't see a
> short-term fix for that.  But we are losing also contact with willing
> hackers who'd actually want to work on Emacs/GCC integration and who'd
> be perfectly willing to just leave the legal considerations in the hand
> of the FSF but who are barred from working on the implementation.
> 
> And that's really worrying.  I think there's too much baby in the
> bathwater we are throwing out here.

I agree, but I think we should get back on track discussing the
technical issues and aspects of building infrastructure for a good
modern IDE using some help from GCC.

Suppose we did have a way getting the necessary information from GCC
that is not in the form of an AST: what such an information package
would need to include to allow the functionality users expect from a
modern IDE?  Could someone come up with a list of information items
that would be needed?  Given such a list, we could try to discuss
various alternatives of getting the info from GCC.

Re: Emacs contributions, C and Lisp

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

[snip]

> Suppose we did have a way getting the necessary information from GCC
> that is not in the form of an AST: what such an information package
> would need to include to allow the functionality users expect from a
> modern IDE?  Could someone come up with a list of information items
> that would be needed?  Given such a list, we could try to discuss
> various alternatives of getting the info from GCC.

This looks like an excellent approach for wasting time while
compromising the future usefulness of the resulting work at the same
time.

The AST is right there, we just need a method for accessing it on an
effective way from Emacs. You propose to transform or filter the AST.
This means coming with an intermediate step for throwing away
information. Obviously devising and implementing such intermediate step
is far from trivial, but the worst part of it is the "throwing away
information" aspect. The field of compiler-based IDE features is a
thriving field, a magnet for inventive individuals. This is not about
copying the features of some other IDE, it is about providing a basis
por making Emacs the home of those inventive individuals that could put
Emacs on the top of IDEs again. Having an intentionally crippled
compiler-provided info set is sending a "go away" message.

Re: Emacs contributions, C and Lisp

[Perry E. Metzger]

On Thu, 08 Jan 2015 17:21:02 +0200 Eli Zaretskii <eliz@gnu.org> wrote:
> I agree, but I think we should get back on track discussing the
> technical issues and aspects of building infrastructure for a good
> modern IDE using some help from GCC.
> 
> Suppose we did have a way getting the necessary information from GCC
> that is not in the form of an AST: what such an information package
> would need to include to allow the functionality users expect from a
> modern IDE?

Here are several sorts of things I really want to be able to do in a
refactoring editor:

a) Take all instances of a call of the form a(c, d, e) and turn them
into a call of the form b(e, d, NULL, c), regardless of what sort of
syntactic mess may be associated with the calls -- whether they're on
multiple lines or not, whether one of the parameters is a complicated
expression that is impossible for regular expressions to parse, etc.
I also want to only do this for the instances of the call "a" which
are at a particular scope level -- inner definitions of "a" which
mask the outer definition should not be touched. All this can possibly
be done without an AST, but it is harder, especially given how
maddening parsing parameters with regexes can be.

b) To do versions of a), in an environment like C++ where functions
are overloaded, so it is necessary to distinguish not only scope but
also the types of all of the passed parameters so I can know what
overload is in question. The passed parameters might be difficult to
type -- they be embedded inside macros, or they might themselves be
overloaded functions, and pure syntactic analysis cannot distinguish
their types, you need more information. Again, perhaps less than an
AST could help here, but it would be a big pain.

c) To handle something like completing foo.<complete> = complex_call.
This requires knowledge of the return type of complex_call, which in
a language like C++ is a difficult thing to know based merely on
syntax.

d) To handle c) but in a situation where "foo" is replaced by a macro
which has to be expanded and the types of whose return needs
sophisticated evaluation even to know which object or struct is being
discussed. Again, this is very hard to do purely syntactically or
only with object table access.

e) To be able to take every instance where a particular
definition (including overload) of foo() is called without checking
it for a NULL return value and to offer the user the chance to
replace it with the text in which foo() is replaced by a conditional
that checks foo for NULL. This is (again) difficult to do in the
general case without ASTs because it is insufficient just to have
symbol tables, you need to actually know what flavor of foo() this
is, and to be able to judge the context in which foo() is called.
(This is not a theoretical example, I have really wanted to be able
to do this. Yes, it would require a line or two of elisp even with
an AST being available and a proper API, but that's the whole point of
having Emacs!)

I can come up with more and richer examples pretty quickly.

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Emacs contributions, C and Lisp

[David Engster]

David Engster writes:
> Richard Stallman writes:
>> I am very concerned lest the GCC back ends to be replaced with LLVM,
>> which permits proprietary changes and proprietary back ends.
>
> As Oscar already mentioned, this has already happened, although this
> effort is now largely abandoned. It might be revived of course, but
> there's not much incentive to do so. Instead, people rather right new
> frontends. For instance, there was already work done on 'flang', a
> Fortran frontend, and new languages like Mozilla's "Rust" use LLVM as
> their backend right from the beginning.

To back this up: GCC ships with frontends for C/C++/ObjC, Fortran, Java,
Go, and Ada. AFAIK, the Java frontend is in maintenance mode and not
actively developed anymore, and I'm not sure about Go's status, since
the Go people have their own compiler nowadays.

LLVM already has native frontends for C/C++/ObjC (clang), Java, and Go,
and a maybe halfway finished one for Fortran. So it seems the only GCC
frontend that LLVM is actually missing is Ada.

-David

Re: Emacs contributions, C and Lisp

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Thu, 08 Jan 2015 16:49:20 +0100
> 
> You propose to transform or filter the AST. This means coming with
> an intermediate step for throwing away information. Obviously
> devising and implementing such intermediate step

You've misread what I wrote.  I said nothing about doing anything with
the AST, certainly not throwing away information in it.

What I proposed was to come up with a set of requirements for a
hypothetical data package that GCC would need to deliver to support
the kind of functionality we would like to have.  Once we have these
requirements, we can see how they map into the AST (or into something
else, if someone has other proposals).  E.g., if it turns out that we
need 99% of what is already in the AST, the argument for using it will
be much more substantiated.

> The field of compiler-based IDE features is a thriving field, a
> magnet for inventive individuals. This is not about copying the
> features of some other IDE, it is about providing a basis por making
> Emacs the home of those inventive individuals that could put Emacs
> on the top of IDEs again. Having an intentionally crippled
> compiler-provided info set is sending a "go away" message.

I agree with every word, and didn't intend to imply anything against
that.

Re: Emacs contributions, C and Lisp

[Eli Zaretskii]

> Date: Thu, 8 Jan 2015 10:49:46 -0500
> From: "Perry E. Metzger" <perry@piermont.com>
> Cc: David Kastrup <dak@gnu.org>, monnier@iro.umontreal.ca, rms@gnu.org,
> 	deng@randomsample.de, emacs-devel@gnu.org
> 
> I can come up with more and richer examples pretty quickly.

I think concrete examples would help, yes.  Especially related to OO
languages.

Re: Emacs contributions, C and Lisp

[Perry E. Metzger]

On Thu, 08 Jan 2015 09:26:08 -0500 Stefan Monnier
<monnier@IRO.UMontreal.CA> wrote:
> > The long term result of all of this may very well be to do
> > exactly the opposite of what you want -- to convince compiler
> > researchers that LLVM is the only serious platform for their
> > work, and even worse, to convince developers in general that free
> > software is too hard to use and that non-free software is the way
> > for them to get their work done.
> 
> Let's not forget that LLVM is Free.  It's not as Free as GCC since
> it doesn't use a copyleft license, but it's not proprietary.

Yes. I was referring more to the young hackers who are starting to
use proprietary IDEs in preference to open editors because of the
lack of modern tooling in Emacs.

> > I think most of us understand the issue as you see it. I think the
> > distinction is that most of us believe the trade-off is
> > important. Yes, this may indeed mean that some proprietary
> > software ends up being based on GCC just as some proprietary
> > software is based on GNU/Linux, but the overall impact will be
> > positive, and the risk is much lower than the reward.
> 
> I think this "much" is an understatement.

Indeed.

I hope RMS understands that we do indeed appreciate that people will
probably then base some proprietary programs on GCC and that we don't
like this, but that we believe the trade-off is very important, and
that the bad result probably cannot be prevented.

That is to say: the bad part of this likely cannot be
prevented, but by not permitting modular use of GCC we could act to
prevent a good result as well (that is, we could prevent Emacs from
being able to provide modern IDE features), and *that* would be a
tragedy.

There is also a significant opportunity here that could be lost. If
GCC were modular, but the best libraries for dealing with the
intermediate data formats like the AST were GPLed, then code linked to
those libraries would be GPLed. Yes, one could do a non-free
re-implementation of such libraries as well, but one can also build a
whole new compiler that isn't GPLed (LLVM, anyone?) If you tempt
people with good functionality, the path of least resistance will be
to bring more software under the GPL mantle, not less.

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > auto some_var = foo(bar);

  > and when the compiler resolves `foo' it can choose an overload depending
  > on complex attributes of `bar' (such as if `bar' is an instance of a
  > class that implements certain method.) As each overload can return a
  > different type, only a C++ compiler can know which type `some_var'
  > finally is.

To handle that, it is sufficient that the compiler say what type
some_var was ultimately given.  Of course, the data must include
the data types of all identifiers.

What more is needed, and why?

Some data that I am sure is NOT needed is the entire statement-level
structure.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I can think of a half dozen projects at the University of
  > Pennsylvania (my university), VeLLVM, Ironclad C++, and my own work
  > on C safety being just three, where no one used GCC as a platform
  > specifically because of the more open architecture in LLVM.

This is very unfortunate, but I don't know that anything can be done
about it.

LLVM is a real technical step forward, yoked to a step backward in
defending our freedom.  If it were just the former, I would be
enthusiastic for it.  However, the latter is more important, so we
must do our best to resist it, even temporarily.

  > The long term result of all of this may very well be to do exactly the
  > opposite of what you want -- to convince compiler researchers that
  > LLVM is the only serious platform for their work,

Why do you say "may very well be"?  According to your previous
paragraph, they are already convinced, so there is no way to
make that any worse.

						      and even worse, to
  > convince developers in general that free software is too hard to use
  > and that non-free software is the way for them to get their work done.

I cannot follow you there.  Which non-free software are you talking
about?  Could there be a misunderstanding here?

LLVM is free software.  It is undefended by copyleft, ideal for Apple
to make it proprietary.  That's why it is a big step backwards.
Nonetheless the version that the researchers work on is free.  So I
don't see how they could derive the conclusion you suggest.

You think to make me stop resisting by telling me resistance is
useless, that what I am defending is lost.  At the same time, you are
telling me that my efforts to defend it really endanger it.  (Those
two can't both be true.)

What this says to me is that you might be exaggerating how bad things
are in the aim of convincing me.

I will read the second part of your message later.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > This looks like an excellent approach for wasting time while
  > compromising the future usefulness of the resulting work at the same
  > time.

You consider it a waste of time, because you see the value in what
I am trying to protect.

This difference in values is why I am not going to heed your conclusions.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  >   E.g., if it turns out that we
  > need 99% of what is already in the AST, the argument for using it will
  > be much more substantiated.

For me, the concern is rather,
what IS in the AST that we DON'T need.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > LLVM already has native frontends for C/C++/ObjC (clang), Java, and Go,
  > and a maybe halfway finished one for Fortran. So it seems the only GCC
  > frontend that LLVM is actually missing is Ada.

You have just mentioned two of them -- Ada and Fortran.
There is also Pascal.

You and several others are trying to pressure me to decide to make
GCC output the full AST.  I have seen insults and harassment.
I have seen distortions.

This is not the way to convince me.  It is the way to make me resent
your behavior.

Beyond that, your analysis is one-sided and tendentious.  It would be
wrong to decide based on a one-sided analysis.  That is not the way to
reach a valid conclusion.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[Óscar Fuentes]

Richard Stallman <rms@gnu.org> writes:

> To handle that, it is sufficient that the compiler say what type
> some_var was ultimately given.  Of course, the data must include
> the data types of all identifiers.
>
> What more is needed, and why?
>
> Some data that I am sure is NOT needed is the entire statement-level
> structure.

Ok, I'll try again: let's suppose an extension that highlights chunks of
code that might have side effects, or that could use multi-threading
features, or that could result on calls to certain functions, or some
other condition specified by the user. There is no doubt about the value
of such extension, and it is technically possible. How could you do that
without accessing the full AST?

Re: Emacs contributions, C and Lisp

[Perry E. Metzger]

On Thu, 08 Jan 2015 18:59:24 -0500 Richard Stallman <rms@gnu.org>
wrote:
> Some data that I am sure is NOT needed is the entire statement-level
> structure.

There are refactorings that are impossible without statement level
structure, such as semantic patches, that is, refactorings that are
aware of higher level information such as "has this value that
resulted from bar() being called been checked against NULL since the
last call to foo(), and if not, bracket the variable it was assigned
to in the following if statement...".

Yes, people now have the opportunity to do such things, and because
they can, they do. It is a powerful tool to use on a large code base.
Programmers like being able to apply software to their programs to
help them automate their work -- that's as old as computers
practically.

So, such high level global refactorings are a very desirable tool to
the programmer. I have been forced up to now to use systems other than
Emacs to perform such refactorings (specifically, LLVM derived
systems -- and yes, I do such refactorings in the real world), but I
would vastly prefer to write them in elisp and perform them inside of
Emacs except in unusual circumstances.

Again, I recognize the problem you wish to avoid, but I think that
you are restricting the freedom of the users too much to avoid
something that is now happening regardless.

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Emacs contributions, C and Lisp

[Perry E. Metzger]

On Thu, 08 Jan 2015 19:02:13 -0500 Richard Stallman <rms@gnu.org>
wrote:
>   > E.g., if it turns out that we
>   > need 99% of what is already in the AST, the argument for using
>   > it will be much more substantiated.
> 
> For me, the concern is rather, what IS in the AST that we DON'T
> need.

I am sad to say that I can't think of something the compiler might
know that would not, at least some of the time, be very valuable to
an IDE to know as well. I say I am sad because it would be much
simpler to have this discussion otherwise.

We live in an era where people expect, more and more, to be able to
perform substantial source code to source code transformations using
programs. To restrict access to the AST is to declare that some such
transformations are to be impossible.

I recognize that you fear the use of GCC in proprietary systems, and
I recognize that this is a serious problem and legitimate concern. I
agree that it would be vastly preferable to get what programmers need
without opening up GCC this way.

However, as with the need to allow proprietary software to use GNU
libc and to allow proprietary software to run on GNU/Linux even though
it would be nicer to live in a world where that was not a practical
requirement, it is, de facto, likely to be necessary to allow such
breeches for the greater good of the free software movement.

In any case, LLVM will end up becoming the tool of choice for all
compiler writers -- has already, in fact, as I've said I've got
little choice about using it since GCC does not have the hooks into
the AST I need for my own compiler work -- and the question of
GCC's large share of the market will become impossible to defend
anyway.

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Emacs contributions, C and Lisp

[Perry E. Metzger]

On Thu, 08 Jan 2015 19:01:27 -0500 Richard Stallman <rms@gnu.org>
wrote:
>   > and even worse, to
>   > convince developers in general that free software is too hard
>   > to use and that non-free software is the way for them to get
>   > their work done.
> 
> I cannot follow you there.  Which non-free software are you talking
> about?  Could there be a misunderstanding here?
> 
> LLVM is free software.

I'm referring to IDEs and refactoring tools. I now know a large number
of people, even people who used to be Emacs enthusiasts and would
otherwise prefer to continue using Emacs, who use such tools because
they are simply to convenient. They reduce their work load too much,
make it much easier to produce software they need to write at higher
speed.


Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Emacs contributions, C and Lisp

[Stefan Monnier]

> You consider it a waste of time, because you see the value in what
> I am trying to protect.

> This difference in values is why I am not going to heed your conclusions.

To me there is blindingly clear evidence that what you're trying to
protect doesn't exist any more, and that it's just alienating those rare
individuals who do share our values enough that they haven't already
moved on to LLVM/clang.

And since the GCC license already fully allows writing a plugin which
gives all the info we need, I strongly support and encourage everyone to
go ahead, write such a plugin, link it to CEDET (or any other Emacs
package if you so prefer), and I'll do everything I can to get this
included into Emacs.

With all due respect, Richard, I really think here you're simply wrong,
and I'd be willing to consider a fork if that's what it takes.


        Stefan

Re: Emacs contributions, C and Lisp

[Stefan Monnier]

> LLVM is a real technical step forward, yoked to a step backward in
> defending our freedom.  If it were just the former, I would be
> enthusiastic for it.  However, the latter is more important, so we
> must do our best to resist it, even temporarily.

Why resist?  We should instead embrace&extend.  With GPL'd extensions,
of course.


        Stefan

Re: Emacs contributions, C and Lisp

[John Yates]

[-- Attachment #1: Type: text/plain, Size: 1167 bytes --]

On Thu, Jan 8, 2015 at 7:01 PM, Richard Stallman <rms@gnu.org> wrote:

>
>   > The long term result of all of this may very well be to do exactly the
>   > opposite of what you want -- to convince compiler researchers that
>   > LLVM is the only serious platform for their work,
>
> Why do you say "may very well be"?  According to your previous
> paragraph, they are already convinced, so there is no way to
> make that any worse.
>

Richard,

That almost comes across as you playing intentionally dumb.  The paragraph
you quote explicitly uses the term 'researchers'.  While some compiler and
language aware tools research happens in industry surely the largest amount
happens in academe.  Academic researchers constitute a continually renewing
flow with those graduate students who move on into - typically to less
'researchy' - industry roles being replaced by fresh, impressionable talent.

Yes, we may be able to little to reshape attitudes of those who have left
academe.  But are you therefore suggesting that the battle for hearts and
minds of potential future researchers is unalterably predetermined and so
utterly lost that we should not even try?

/john

[-- Attachment #2: Type: text/html, Size: 1684 bytes --]

Re: Emacs contributions, C and Lisp

[Paul Nathan]

[-- Attachment #1: Type: text/plain, Size: 2052 bytes --]

Much of this discussion is passe, I am afraid. It is quite clear that
Clang/LLVM are leaps and bounds ahead in the tooling field; I see arguments
about if the front door should be locked, when it's plain that the back
door of the house has been opened and there is a thriving auction of the
goods from the porch.

If GCC is to compete on the tooling field, it will have to provide superior
functionality that will not only meet Clang's bar, but exceed it
significantly. I invite interested parties to play with XCode, Visual
Studio, Eclipse, and IntelliJ to begin to understand exactly what the
competition is.

I would far rather see a libre software product at the top of the field,
but it requires adaptation to the current world. Let the data be free,
please.


On Thu, Jan 8, 2015 at 6:51 PM, John Yates <john@yates-sheets.org> wrote:

> On Thu, Jan 8, 2015 at 7:01 PM, Richard Stallman <rms@gnu.org> wrote:
>
>>
>>   > The long term result of all of this may very well be to do exactly the
>>   > opposite of what you want -- to convince compiler researchers that
>>   > LLVM is the only serious platform for their work,
>>
>> Why do you say "may very well be"?  According to your previous
>> paragraph, they are already convinced, so there is no way to
>> make that any worse.
>>
>
> Richard,
>
> That almost comes across as you playing intentionally dumb.  The paragraph
> you quote explicitly uses the term 'researchers'.  While some compiler and
> language aware tools research happens in industry surely the largest amount
> happens in academe.  Academic researchers constitute a continually renewing
> flow with those graduate students who move on into - typically to less
> 'researchy' - industry roles being replaced by fresh, impressionable talent.
>
> Yes, we may be able to little to reshape attitudes of those who have left
> academe.  But are you therefore suggesting that the battle for hearts and
> minds of potential future researchers is unalterably predetermined and so
> utterly lost that we should not even try?
>
> /john
>
>

[-- Attachment #2: Type: text/html, Size: 2949 bytes --]

Re: Emacs contributions, C and Lisp

[Eli Zaretskii]

> Date: Thu, 08 Jan 2015 18:59:24 -0500
> From: Richard Stallman <rms@gnu.org>
> Cc: emacs-devel@gnu.org
> 
>   > auto some_var = foo(bar);
> 
>   > and when the compiler resolves `foo' it can choose an overload depending
>   > on complex attributes of `bar' (such as if `bar' is an instance of a
>   > class that implements certain method.) As each overload can return a
>   > different type, only a C++ compiler can know which type `some_var'
>   > finally is.
> 
> To handle that, it is sufficient that the compiler say what type
> some_var was ultimately given.  Of course, the data must include
> the data types of all identifiers.

A single source line can reference more than one identifier that have
the same string as their name, but different semantics and types.  A
list of all the identifiers and their types is not enough to handle
that; you need to know the type of each 'foo' in its exact place.
Even in C, a single string can mean very different things in the same
program unit, due to different namespaces, like variables/functions vs
struct tags vs struct members vs labels.

Further, to support refactoring, one needs to have access to the
semantics of arguments of each function, otherwise the refactored code
will fail to compile at best and introduce subtle bugs at worst.  The
list of identifiers and data types won't be enough here as well.

> Some data that I am sure is NOT needed is the entire statement-level
> structure.

I suggested to write down requirements for what _is_ needed.  I
suspect that the full amount of syntactic and semantic information
that is needed for the features we are discussing is very close to the
AST.  After all, what other ways are known to mankind for expressing
this kind of information?

Since no one seems to be willing to bite the bullet, how about the
list of features, such as completion and refactoring, that we would
like to enable?  Would someone who feels they know enough about this
please post such a list, with a short (1-2 sentences) description of
each feature?

The reason I think such a list would be useful is that I'm not sure
Richard is aware of the kind of features we are talking about, which
makes this discussion less efficient and effective than it could be.

Re: Emacs contributions, C and Lisp

[Eli Zaretskii]

> Date: Thu, 08 Jan 2015 19:02:13 -0500
> From: Richard Stallman <rms@gnu.org>
> CC: ofv@wanadoo.es, emacs-devel@gnu.org
> 
>   >   E.g., if it turns out that we
>   > need 99% of what is already in the AST, the argument for using it will
>   > be much more substantiated.
> 
> For me, the concern is rather,
> what IS in the AST that we DON'T need.

That concern will be addressed by the same procedure I proposed.  You
can look at that 1% (or 5% or 10% or whatever) and decide then whether
it's worth to avoid giving it up.  But to make that decision you need
to know what is and what isn't needed, there's no way around that for
an intelligent decision.

Re: Emacs contributions, C and Lisp

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Fri, 09 Jan 2015 01:23:10 +0100
> 
> Ok, I'll try again: let's suppose an extension that highlights chunks of
> code that might have side effects, or that could use multi-threading
> features, or that could result on calls to certain functions, or some
> other condition specified by the user. There is no doubt about the value
> of such extension, and it is technically possible. How could you do that
> without accessing the full AST?

Thanks for these examples.  I think it would help to explain how would
a program (such as a Lisp program) know whether some code has these
side effects, given the information in the AST.  IOW, what parts of
the AST information will be used for each one of those, and how?

Re: Emacs contributions, C and Lisp

[Eli Zaretskii]

> Date: Thu, 8 Jan 2015 19:43:42 -0500
> From: "Perry E. Metzger" <perry@piermont.com>
> Cc: Óscar Fuentes <ofv@wanadoo.es>, emacs-devel@gnu.org
> 
> There are refactorings that are impossible without statement level
> structure, such as semantic patches, that is, refactorings that are
> aware of higher level information such as "has this value that
> resulted from bar() being called been checked against NULL since the
> last call to foo(), and if not, bracket the variable it was assigned
> to in the following if statement...".

Can you show a concrete example of such refactoring?

TIA

Re: Emacs contributions, C and Lisp

[David Kastrup]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   >   E.g., if it turns out that we need 99% of what is already in the
>   > AST, the argument for using it will be much more substantiated.
>
> For me, the concern is rather,
> what IS in the AST that we DON'T need.

A technical framework sculpted around making a particular limitation
unavoidable is likely to end up cumbersome.  It's definitely a good idea
to provide a mechanism providing only the information necessary for a
particular task.  But predefining those bunches of compiler information
a programmer is allowed to access or not is just putting a block on what
functionality programmers may be inspired to implement on top of GCC.

We don't want to end up with micromanaging restriction management of the
"I'm sorry Dave, but I am afraid I can't do that" kind.  GCC allows
compiling binaries that are proprietary: at some point of time we draw a
line of where we try exerting control over what people use tools for.

It is a greater problem for us to definitely block free applications
from being developed, either completely or by putting up prohibitely
high administrative or technical hurdles, than it is to accept the
possibility of non-free ones here.

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[Perry E. Metzger]

On Fri, 09 Jan 2015 10:48:39 +0200 Eli Zaretskii <eliz@gnu.org> wrote:
> > Date: Thu, 8 Jan 2015 19:43:42 -0500
> > From: "Perry E. Metzger" <perry@piermont.com>
> > Cc: Óscar Fuentes <ofv@wanadoo.es>, emacs-devel@gnu.org
> > 
> > There are refactorings that are impossible without statement level
> > structure, such as semantic patches, that is, refactorings that
> > are aware of higher level information such as "has this value that
> > resulted from bar() being called been checked against NULL since
> > the last call to foo(), and if not, bracket the variable it was
> > assigned to in the following if statement...".
> 
> Can you show a concrete example of such refactoring?

How much detail do you want?

An obvious example of something of the flavor is this: you add an
error return code to a function, and you then want to add code
throughout the program to check for the error return code, but only
if the error is possible from context. (For example, your call might
have an optional parameter, and the error return might tell
you that the optional parameter is invalid, but you only want to
check for such a return if the optional parameter has been passed at
all.) I could make up a synthetic example but I think most programmers
have seen such things.

A second, inverse example might be this: you find a bug caused by
failing to check an error return code, and you would like to go
through all your code looking for similar instances of the same call.
There are thousands of such calls and almost all of them are correct,
so you only want to see the ones where the error return code hasn't
been checked for. Your search is then not on some short string but
rather on a fairly arbitrary pattern over the AST. Again, I could
make up a synthetic example, but I think most programmers have seen
such things before. If it will help, though, I can invent an example.


Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Emacs contributions, C and Lisp

[David Engster]

Richard Stallman writes:
> You and several others are trying to pressure me to decide to make
> GCC output the full AST.  I have seen insults and harassment.

Not from me, and I haven't seen anything like it on this thread.

> This is not the way to convince me.  It is the way to make me resent
> your behavior.

I've no idea what I've done to earn your resentment. I think my behavior
was entirely reasonable, given that I've started with this only because
you asked to base our tooling efforts on GCC. Anyway, you don't have to
worry that I'll continue with this.

-David

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  >   > This looks like an excellent approach for wasting time while
  >   > compromising the future usefulness of the resulting work at the same
  >   > time.

I wrote

  > You consider it a waste of time, because you see the value in what
  > I am trying to protect.

I meant to write

  > You consider it a waste of time, because you don't see the value in what
  > I am trying to protect.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Ok, I'll try again: let's suppose an extension that highlights chunks of
  > code that might have side effects, or that could use multi-threading
  > features, or that could result on calls to certain functions,

Those certainly don't need the statement structure.

								  or some
  > other condition specified by the user.

That's not specific, that's handwaving.

					   There is no doubt about the value
  > of such extension,

How can we say anything about the worth of something totally unspecified?

To reach your desired conclusion, you will leap huge logical
crevasses.


-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > There are refactorings that are impossible

We were talking about completion -- you are changing the subject,
taking my statements out of context to make a false attack.

This is an example of your general approach.  You (this means several
people) are not trying to help me make the right decision.  Rather you
are trying to pressure me to do what you want, at the expense of
something I consider important.

The result of this is that I don't trust your judgment about anything
related to this issue.

I will try to find out more about these refactoring practices --
privately, with people I have confidence in, that have no axe to
grind.

To approach the issue without prejudice, I will need to prevent
resentment for your pressure campaign from influencing me.  To help me
overcome it, you would do well to drop the issue right now.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[Perry E. Metzger]

On Fri, 09 Jan 2015 12:39:19 -0500 Richard Stallman <rms@gnu.org>
wrote:
>   > There are refactorings that are impossible
> 
> We were talking about completion -- you are changing the subject,
> taking my statements out of context to make a false attack.

We are not just talking about completion. We are talking about
allowing Emacs to compete effectively with XCode, IntelliJ, and other
modern integrated development environments. Such systems permit not
only naive code completion (itself very difficult in a language like
C++ without having an AST present) but complicated refactorings, and
they are permitting more and more complicated refactorings with
every passing year.

Note that I'm also not persuaded that the AST is not needed for
code completion in context when the underlying language has operator
overloading and function overloading. I could perhaps be persuaded
that there might be very specialized APIs possible to get around
that, but then the sorts of refactorings you would want to perform
become impossible anyway.

> This is an example of your general approach.  You (this means
> several people) are not trying to help me make the right decision.
> Rather you are trying to pressure me to do what you want, at the
> expense of something I consider important.

I am attempting to persuade. I believe everyone here is convinced
and understands that you find it important to protect the AST, and
we understand why. However, having free development tools that can
compete with proprietary ones is also something many of us also
consider very important.

There is no shame in our explaining quite forcefully that this is
very important and that there are few real alternatives to a real AST.
 
> I will try to find out more about these refactoring practices --
> privately, with people I have confidence in, that have no axe to
> grind.

I would suggest that you instead attempt to actually use some of the
modern tools out there so that you can intuitively understand, for
yourself, the sorts of things Emacs is currently missing, without
any sort of intermediary at all. You may find it enlightening.

That said, to restrict yourself to *current* refactoring practices
would also be to miss part of the point -- tools of this sort are
constantly improving, and even if a limited API might handle this
week's needs, it may not handle next week's or next year's.

By forbidding the editor from having advanced knowledge of the parse
tree, you are intentionally crippling the ability of smart people to
build better and better free software tools. You are preventing smart
hackers from going in and making the system as good as they can make
it, from expressing their creativity in such a way as to build the
best development environment they know how.

Dare I say there is a freedom issue here? Many people -- you have
heard from the in this thread -- would very much like to have
information that the compiler has available inside their editor. You,
as the software vendor, for purposes that I will admit are well
intentioned, wish to prohibit that. Naturally, people find this
frustrating, for the same reason you might yourself find it
frustrating.

You have asked us to see this from your point of view, and many of us
have tried hard to do that. I beseech you to look at it from the
point of view of other people as well.

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Emacs contributions, C and Lisp

[David Kastrup]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > There are refactorings that are impossible
>
> We were talking about completion -- you are changing the subject,
> taking my statements out of context to make a false attack.

I think that is an unfair characterization: the strictly limited idea of
completion in this thread has admittedly moved a bit to the background
in the course of the discussion, but I don't think that this has been
either a conscious strategy nor has it been Perry's "doing".

> This is an example of your general approach.  You (this means several
> people) are not trying to help me make the right decision.  Rather you
> are trying to pressure me to do what you want, at the expense of
> something I consider important.

I think the salient point here was that we currently have a problem with
implementing reliable and powerful versions of completion with awkwardly
complex languages like C++ that, for better or worse, are _supposed_ to
be well-supported by GNU.

However, the effort of tackling this problem is quite similar to a
number of other problems that _are_ being successfully addressed by IDEs
other than Emacs, and I think that if addressing them by GNU software is
to be an option, we need to provide the freedom for experimenting with
integration of GCC without close supervision.

Admittedly, the imminent project right now seems to be completion.  I am
somewhat afraid that by the time we have finished the spec books for
getting, more or less, the permission for implementing the technical
measures on track, there will be nobody bothering to pick them up.

> The result of this is that I don't trust your judgment about anything
> related to this issue.
>
> I will try to find out more about these refactoring practices --
> privately, with people I have confidence in, that have no axe to
> grind.

Shrug.  I don't see that the "axe to grind" here is one directed against
the goals of the GNU project.  Some people might not be interested in
the full view of all goals, and some might come to different conclusions
about them.  In the end, we can only implement one strategy, but that
does not make people making different proposals an enemy of the project.

> To approach the issue without prejudice, I will need to prevent
> resentment for your pressure campaign from influencing me.  To help me
> overcome it, you would do well to drop the issue right now.

That might well be the case: you probably know yourself best.  It's
still another constraint that might not be easy to properly factor in
for some project members, and the facilities needed for that are
non-technical and not necessarily a strong skill of typical competent
hackers.

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[Perry E. Metzger]

On Fri, 09 Jan 2015 12:39:15 -0500 Richard Stallman <rms@gnu.org>
wrote:
>   > Ok, I'll try again: let's suppose an extension that highlights
>   > chunks of code that might have side effects, or that could use
>   > multi-threading features, or that could result on calls to
>   > certain functions,
> 
> Those certainly don't need the statement structure.

It is difficult to figure out things such as whether global variables
have been assigned in a block without access to the AST. I'm sure a
very specific API could be crafted to permit it, of course, but then
someone else will want to check whether or not a variable is written
within a block protected by a lock variable. That too can be done
perhaps with a very specific API, but then someone else may want to
highlight all overloaded forms of the [] operator and not the
non-overloaded ones. One can keep going on.

By the way, when hacking on C++, I would often *very much* like to
have overloaded vs. non-overloaded operators highlighted differently.
It would make "unexpected" operator behavior a lot easier to be
aware of in advance -- one complaint in using the language is that
one is often unaware that an operator has been overloaded, so one
does not think to check what its behavioral differences from the
default may be. So, this isn't a hypothetical. Again, this is
something I'm sure a very specialized API could permit, but how many
such very specialized APIs does one want to add?

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Emacs contributions, C and Lisp

[Eli Zaretskii]

> Date: Fri, 09 Jan 2015 12:39:19 -0500
> From: Richard Stallman <rms@gnu.org>
> Cc: ofv@wanadoo.es, emacs-devel@gnu.org
> 
>   > There are refactorings that are impossible
> 
> We were talking about completion -- you are changing the subject,

This is the same subject.  Completion, refactoring, and a few more
features are all parts of what a modern IDE needs to offer.  We are
talking about creating infrastructure for such features.

Re: Emacs contributions, C and Lisp

[Karl Fogel]

Richard Stallman <rms@gnu.org> writes:
>We were talking about completion -- you are changing the subject,
>taking my statements out of context to make a false attack.
>
>This is an example of your general approach.  You (this means several
>people) are not trying to help me make the right decision.  Rather you
>are trying to pressure me to do what you want, at the expense of
>something I consider important.
>
>The result of this is that I don't trust your judgment about anything
>related to this issue.
>
>I will try to find out more about these refactoring practices --
>privately, with people I have confidence in, that have no axe to
>grind.
>
>To approach the issue without prejudice, I will need to prevent
>resentment for your pressure campaign from influencing me.  To help me
>overcome it, you would do well to drop the issue right now.

Richard, this is a very unfair characterization of what Perry has been saying.  He's not changing the subject by talking about features other than completion.  For quite a while now this thread's topic has been about all the features that modern IDEs provide, and that Emacs currently has trouble providing (with the data handed to it by GCC).  Perry's points are made in that context.  If you're having trouble tracking the context, that's okay, just say so -- people will be glad to help.  But please don't make these kinds of accusations against Perry.

Perry has very patiently explained how he *does* understand your goals and shares them, and he has explained why he thinks his proposed course of action serves those goals better.  He's actually gone into much more detail than really should be needed, since for some reason you have not been willing to acknowledge that at least *in principle* his point might have merit.

But not just in principle.  I think he is actually right: the cause of freedom would be better served by the course he and others here are advocating.

Everyone here understands the tradeoff:  there is an inherent tension between promoting freedom by building a protective wall around our city, and promoting freedom by interoperating with non-free environments so that people in those environments have a chance to experience freedom.  Historically, the FSF has used both strategies -- so you, too, understand this tradeoff.

For some reason you refuse to acknowledge that others are cognizant of this tradeoff.  They have been very patiently making a detailed argument for why, in this particular case, you are choosing the wrong side of that tradeoff -- the side that will be *less* effective at accomplishing our shared goal.  They make this argument, with impressive clarity, and then you accuse them of bad faith.

This would be poor behavior even if those people were wrong.  I think they're actually right, though, which makes it even worse, because now our goal is being damaged too.

In a later message, Perry wrote one thing I disagree with:

> Dare I say there is a freedom issue here? Many people -- you have
> heard from the in this thread -- would very much like to have
> information that the compiler has available inside their editor. You,
> as the software vendor, for purposes that I will admit are well
> intentioned, wish to prohibit that. Naturally, people find this
> frustrating, for the same reason you might yourself find it
> frustrating.

There is no "freedom issue" in the sense he meant.  GCC is still Free Software, and anyone is free to fork it -- by which I mean, any one is free to try to create a socially and technically credible copy of the GCC project, one that attracts a majority of GCC developers as contributors, and is not restricted by your decisions.

We know it's possible; after all, it's happened before.  But the fact that no one has yet stepped up to do it in this case is their problem, not yours.  You're not denying anyone's freedom.  However, you're making such a fork more likely, and for poor reasons.  If you took Perry's advice, freedom would be better served.

-Karl

Re: Emacs contributions, C and Lisp

[Eric Ludlam]

On 01/09/2015 10:09 AM, David Engster wrote:
> Richard Stallman writes:
>> You and several others are trying to pressure me to decide to make
>> GCC output the full AST.  I have seen insults and harassment.
>
> Not from me, and I haven't seen anything like it on this thread.
>
>> This is not the way to convince me.  It is the way to make me resent
>> your behavior.
>
> I've no idea what I've done to earn your resentment. I think my behavior
> was entirely reasonable, given that I've started with this only because
> you asked to base our tooling efforts on GCC. Anyway, you don't have to
> worry that I'll continue with this.

This conversation seems unnecessarily final.  Richard has valid concerns 
that need details, but since the AST (which I know almost nothing about) 
is so huge (based on parser's I've written), no matter how many details 
we may think of that are needed, someone else can think of a bit that 
could indeed be unnecessary.

I wrote the first run for the "smart completion" engine that currently 
ships in Emacs, the parts of CEDET that includes EDE and Semantic. 
While I personally think it is pretty awesome, it really isn't hard to 
fool it which is where a lot of this GCC interest comes from.   It took 
many years of my part-time work (and contributions from others like 
David) to assemble what is there now into a robust well tested system.

The basic pieces of the system which is implemented in Emacs Lisp 
consists of a parser generator plus some parsers written in a bison-like 
syntax including a C++ parser.  Due to limitations of Emacs' 
performance, only the parts of the language that handle definitions are 
implemented.  (ie - tags for functions, variables, structures, etc.) 
The parser outputs a tag table with lots of details.   Having a full 
parser generator and the parser is what makes this convenient to do. 
Hacks like etags, GNU Global, etc can't produce enough information for 
the next step.

The next step is the completion engine.  This is where regexp hacks 
exist to "parse" a statement like:

   i = foo.bar.substring

which peels it apart into a variable "i", a notion of assignment, and 
("foo" "bar" "substring") via several assumptions, such as that users 
don't write code like this:

   i /* some variable */
   = /* equals */
   foo. /* mystruct */
   bar.substring

The engine then goes and looks up i in reverse to see what it is.  It 
then looks up foo in various tables that get built of known symbols, 
derives the data type, and thus members of foo.  It iterates down 
through the "." symbols dereferencing each symbol by data type to get to 
the next step.  This depends on the fact that most projects compile all 
their headers "the same way" so that tables parsed from some header 
included in this C file will have the same symbols when included in a 
different C file.

With that background, there are a couple options for a GCC plugin.  One 
option would be to have one plugin that outputs tags compatible with 
some standard.  Naturally I suggest the one already in Emacs.  A second 
plugin could be used to figure out all the state I mentioned earlier 
when looking up symbols, and provide completions directly (ie - a list 
of text strings to offer as completions.)  That plugin would ONLY be 
used for completion, and all the internal logic couldn't be reused for 
another purpose in Emacs.

The alternative is to dump out the AST into an Emacs friendly form, and 
write the above logic in Emacs instead.  This is convenient because 
Emacs is easy to hack, and gcc plugins (based on what I've been reading) 
are really complicated.  In terms of "get up and running quickly", 
dumping a big scary data structure out of a scary environment into a 
friendly easy to hack environment is a desirable path for us, and as 
Richard points out, for non-free software.

I personally think that if there were a good way to bridge the gap so 
that gcc could directly output tags for the existing Semantic engine, 
then there is an incremental benefit of nearly perfect tag generation 
for the existing tool AND a performance boost.  It won't solve the whole 
problem though.   To solve the rest of it, we'd need a gcc plugin to 
parse a file up to a chosen point.  For a file with 1000 lines of code 
and included headers, gcc needs the WHOLE AST to make sense of "the last 
line", or the part that needs the completion.  This is because we can't 
guess at what isn't needed until you've actually processed it all.  This 
is where the boundary between gcc and Emacs comes in.   In theory, the 
GCC plugin could process the AST and output ONLY the completions, or 
ONLY whatever was asked for (local types, scope information, refactoring 
data or what not.)  An alternative might be to output a subset of the 
AST for processing in Emacs that is local to the the completion area, 
and depend on our old Emacs code to do the type lookup, etc.  This would 
improve the current completion, but still could be fooled based on the 
quality of the Emacs data which is now, by definition, incomplete.  In 
the past (call it year 2000) people thought my smart completion was lame 
(ie - inaccurate) and slow compared to dynamic abbreviation completion 
where claims of "dabbrevs is good enough" were stated.   This proposal 
could be "good enough" for this single feature.

So, I've laid out some scenarios that are "not full AST" friendly. 
There are some benefits (performance), and tradeoffs (difficulty).  Even 
so, we've only touched on one feature.  There are lots of other features 
in the existing Semantic tool already in Emacs derived from having a 
parser built right into Emacs, such as highlighting code with syntax 
errors (but only the code for definitions, not the logic.)  I have a 
long list of other things I'd love to do to such as redo font-lock with 
the many "hints" about what your code is doing that only the compiler 
could know, but can't because writing a parser from scratch is actually 
pretty hard and error prone regardless of doing so in Emacs.  Many folks 
have touched on those features in a myriad of other thread replies in 
this mailing list.  I've taken my best stab at some of them that seemed 
attainable, but feel I've gone as far as I can aside from some 
incremental improvements, or just adding new languages.

I've been very thankful for David's help with the CEDET project, and the 
many improvements in our existing smart completion engine he's made. 
For myself, and I imagine David, having gone through that and persevered 
simulating a compiler for so long to try and get these features only to 
have dabbrev people scoff on one side and clang users sneer on the other 
is disheartening.  The hope of having a "real compiler" to lean on could 
open so many doors for us we just can't get to right now it is hard not 
to be discouraged by non technical issues.

I would hope that David, who is looking into the gcc plugin route, and 
Richard can find a reasonable compromise that enables Emacs to have data 
from gcc that would enabled our existing tools to grow in its accuracy, 
and would encourage contributions from folks who do not have the skills 
to hack gcc plugins create new features.  I suspect that isn't possible 
until someone learns more about gcc's AST and thinks about what a good 
abstraction model for Emacs is, and how it could be applied to the 
existing pretty good smart completion system.

Eric

Re: Emacs contributions, C and Lisp

[David Engster]

Eric Ludlam writes:
> On 01/09/2015 10:09 AM, David Engster wrote:
>> Richard Stallman writes:
>>> You and several others are trying to pressure me to decide to make
>>> GCC output the full AST.  I have seen insults and harassment.
>
>>
>> Not from me, and I haven't seen anything like it on this thread.
>>
>>> This is not the way to convince me.  It is the way to make me resent
>>> your behavior.
>>
>> I've no idea what I've done to earn your resentment. I think my behavior
>> was entirely reasonable, given that I've started with this only because
>> you asked to base our tooling efforts on GCC. Anyway, you don't have to
>> worry that I'll continue with this.
>
> This conversation seems unnecessarily final.

I'm afraid it is final for me. Maybe someone else will step up.

> Richard has valid concerns that need details, but since the AST (which
> I know almost nothing about) is so huge (based on parser's I've
> written), no matter how many details we may think of that are needed,
> someone else can think of a bit that could indeed be unnecessary.

I'm not willing to work under some elaborate rule set what I may and may
not do with the AST. It is way too easy to make a mistake, and a few
years ago I've learned the hard way that there's no such thing as an
"honest mistake" when it comes to software's freedom. Also, what happens
if someone else takes my code and simply rewrites it to dump everything?
It would be trivial, since the real work is walking the AST and
extracting its information. The only solution for defending against this
is to restrict access to the AST in GCC's plugin interface, rendering my
plugin unusable. Also, I seem to be unable to discuss with Richard
without him calling out my behavior, so I'm clearly not the right person
for this job.

-David

Re: Emacs contributions, C and Lisp

[David Kastrup]

David Engster <deng@randomsample.de> writes:

> Also, I seem to be unable to discuss with Richard without him calling
> out my behavior, so I'm clearly not the right person for this job.

If getting along perfectly with Richard, in particular in electronic
media, was a precondition to coding GNU software, the project would be
in a pretty poor state.

It is perfectly reasonable if you choose to step down from arguing this
case once you see yourself as having no additional points to make.  Mere
repetition tends to achieve little but cause annoyance.

However, the skills of persuading humans and computers are at best
loosely related, and so it would be a pity if people stepped down from a
team effort when they find that they do not get everywhere they'd like
in one area all on their own.

Richard said he'll discuss the issue with people he trusts.  And
frankly, this sounds like a responsible thing to do even if it is
frustrating to people who'd like to see a more prompt reaction to their
input.  It would be a pity if by the time this process comes to a
conclusion there is nobody interested in making it count any more.

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Dare I say there is a freedom issue here?

Now you're really stretching things.

With proprietary software, the developers' decisions about what
features to implement, or not implement, are restrictions on the
users.  The point of free software is that our decisions -- whatever
we may decide -- are not restrictions.

What I see is that you do are trying to pressure me in a certain
direction and you're willing to stretch things to do it.

In the messages arguing for the full AST I have often seen strained
arguments and gaps.  This is not the way to convince me.  Rather, it
stimulates resistance.  It makes me skeptical about everything those
people say.  Where I do not see a gap or exaggeration, that doesn't
mean there isn't one.

Changing the subject -- making arguments about refactoring while I am
trying to understand about completion -- doesn't impress me favorably.
It interferes with my effort to understand the issue.

Here's how all this appears to me.  I am considering choosing a course
that seems dangerous, and someone asserts that it is obligatory, a
forced move.  He presents arguments I cannot follow, about a feature I
have never seen, which he claims is very important but I don't know
that.  The arguments cite facts about whose veracity I have no
independent knowledge.

I cannot evaluate those arguments.  Under the circumstances, I can
either follow his judgment on faith, or ignore it.

I might follow his judgment, if I have the fullest confidence in him
at all levels.  I would have to be sure he has evaluated the whole
issue from all sides.  I would have to confident that he would have
looked hard for some way to avoid the dangerous course; that if there
is a way, he would have found it and recommended it.

I don't have that sort of confidence in people who cut corners to
lobby for the dangerous course.  I suspect you have not tried hard
enough to find a way to avoid it.

What I intend to do is investigate these issues thoroughly _one by
one_ to see what options exist for each, and what is good or bad about
them.  I will think about refactoring when I understand it well enough
to be able to judge arguments myself.  First I will learn about it
from people who are not trying to pressure me about it.

You can help me do this, when I get to it, by giving me factual
answers to the questions I will ask.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Richard, this is a very unfair characterization of what Perry has
  > been saying.  He's not changing the subject by talking about
  > features other than completion.

I said he took my words out of context.  I was responding to someone
else's message; we were talking about completion, nothing else.  He
reinterpreted my statement as about refactoring, which was the wrong
context for it.

It was also changing the subject.  The topic of this discussion is
completion.  Despite the messages some have posted about refactoring,
that's not the subject I am discussing.

Occasionally bringing up a different but related topic is not a bad
thing, but to do so persistently gets in the way.

  > Perry has very patiently explained how he *does* understand your
  > goals and shares them,

No matter how patient it was, it did not get at the issue at hand.

I believe he supports the free software goals at the deepest level.
The divergence comes at a level less deep.  He (along with others) is
lobbying for one decision, rather than helping me consider the
question carefully and make the right decision.

Indeed, that sort of lobby campaign urges me to skip the careful
thought and hastily adopt their conclusion.  I insist on doing this
slowly and carefully.

I need to go through these issues systematically, to ascertain what
options really exist and what their consequences are.  Right now I am
trying to understand the issue of completion.

Refactoring is a different issue.  I won't look at that issue until
(1) I am done with the completion issue -- and (2) I have learned the
basic facts about refactoring so that I CAN think about it.

Getting back to completion, some tried to dismiss the idea that
completion can be done with less than the full AST.  The discussion
convinces me that it can be done with somewhat less, but I want to
understand what data is needed for various kinds of completion, and
why.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[Dmitry Gutov]

On 01/10/2015 10:29 PM, Richard Stallman wrote:

> Getting back to completion, some tried to dismiss the idea that
> completion can be done with less than the full AST.  The discussion
> convinces me that it can be done with somewhat less

Like Eric said, it *can* be done with a lot less, the question is 
whether we really do want to do it this way.

If the plugin implements the "meat" of the completion logic (probably in 
C), Emacs could serve as a "dumb client", only forwarding completion 
queries to the plugin, and interpreting the responses.

The downsides:

1. Any new feature, significant change, or a bugfix would probably have 
to touch the plugin code, which exludes contributions from a significant 
portion of the user base who would rather contribute Emacs Lisp code.

2. CEDET integration would basically be out of the question, while it's 
the blessed Emacs code assistance suite, which we would want to see 
developed.

The same should be true for refactoring.

Re: Emacs contributions, C and Lisp

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1745 bytes --]

On 01/10/2015 11:29 AM, Richard Stallman wrote:
> With proprietary software, the developers' decisions about what
> features to implement, or not implement, are restrictions on the
> users.  The point of free software is that our decisions -- whatever
> we may decide -- are not restrictions.
> 
> What I see is that you do are trying to pressure me in a certain
> direction 

What you call "pressure", I see as advocacy. There is nothing wrong with
advancing arguments in favor of one's position.

> What I intend to do is investigate these issues thoroughly _one by
> one_ to see what options exist for each, and what is good or bad about
> them.  I will think about refactoring when I understand it well enough
> to be able to judge arguments myself.  

Does Clang's popularity inform your decision-making process? I've seen
no evidence in this thread that you're considering the broader social
context.

Do we need to repeat this process every few years and every time someone
thinks of a new way to integrate GCC and Emacs? I very much like the
example upstream of highlighting overloaded operators. Another feature I
want very much is the ability to fontify, in C++ and Java, is to
distinguish local, member, and global variable references using
different font-lock faces. This task isn't "completion", but 1) requires
access to the AST (for lexical context), and 2) is already present in
IntelliJ, which is also free software.

> First I will learn about it
> from people who are not trying to pressure me about it.

The trouble with asking people who already agree with you is that they
already agree with you. My sense is that you're merely equating
positions contrary to your own with "pressure".


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Emacs contributions, C and Lisp

[Perry E. Metzger]

On Sat, 10 Jan 2015 14:29:53 -0500 Richard Stallman <rms@gnu.org>
wrote:
> Here's how all this appears to me.  I am considering choosing a
> course that seems dangerous, and someone asserts that it is
> obligatory, a forced move.  He presents arguments I cannot follow,
> about a feature I have never seen, which he claims is very
> important but I don't know that.  The arguments cite facts about
> whose veracity I have no independent knowledge.

I am happy to provide

1) Detailed explanations of anything you do not understand, regardless
of the number of iterations needed so that you feel you have a
comfortable working knowledge of the topic.
2) Detailed references to commercial products that provide the sorts
of features I'm describing so you may verify for yourself that these
features exist and are useful without needing to trust me on anything
at all.
3) Detailed and patient assistance of any other sort so you may
determine, for yourself, if what I am saying is true.

No one wants you to have to trust other people's opinions blindly.

> Changing the subject -- making arguments about refactoring while I
> am trying to understand about completion -- doesn't impress me
> favorably. It interferes with my effort to understand the issue.

We have, however, a much fuller set of concerns than
completion. Refactoring *is* a real concern.

Here, for example, is a list of the sorts of refactorings that
IntelliJ, a proprietary editor, makes available for Java
programming. You can read them and decide for yourself how many of
these could be done without access to an AST -- I suspect the answer
is "only a few, and with difficulty".

https://www.jetbrains.com/idea/help/refactoring-source-code.html

(If you don't know what any of these things mean, we can help explain.)

Here is a set of refactorings that the commercial DevExpress add-on to
Visual Studio provides:

https://www.devexpress.com/Products/CodeRush/refactor_pro.xml#autolist5

There's quite a lot of them as you can see.

Now, it is my contention that Emacs could provide much more general
sorts of tools, not merely competing with such a proprietary systems
but far surpassing them.

The commercial tools rarely provide the end user with the ability to
reprogram the ways the tools work. They provide only for doing what
the developer imagined the user might want to do. One of the reasons
that the Clang crowd now has libAST is because they found they needed
to write custom code to do many refactorings they wanted to do, and
this provided them with a way to do that.

However, Emacs could get us beyond all of this. Emacs has, after all,
elisp available. In addition to canned refactorings, users could build
their own. They could get access to the AST matching API in elisp
directly, and with some work one could even allow for interactive
development and application of matchers/transformers. The potential
here for really cool developer tools is high.

I recognize that you don't want me to "change the subject" to
refactoring, but I don't see this as a change of subject. The concern
isn't as such code completion, that's just a detail. The underlying
concern is being able to make Emacs the very best programmer's editor
it can possibly be. Modern development environments have astonishing
power, and there is an enormous desire on the part of certain parts of
the Emacs community to have that power or even more available in
Emacs.

If you find that you do not understand any comment being made, I am
happy to expand and restate until you do feel you understand it. I am
sure that others feel the same way. If you wish to be able to verify
anything I claim independently, I can provide references and pointers
to programs so you can check that I am telling you the truth without
having to trust me at all, and I'm sure others will also happily
provide such information.


Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Emacs contributions, C and Lisp

[David Kastrup]

"Perry E. Metzger" <perry@piermont.com> writes:

> Here, for example, is a list of the sorts of refactorings that
> IntelliJ, a proprietary editor, makes available for Java
> programming. You can read them and decide for yourself how many of
> these could be done without access to an AST -- I suspect the answer
> is "only a few, and with difficulty".

I am pretty sure that the answer is "every single one of them".
However, if you form the _superset_ of all the information required for
implementing every single one of them, it will no longer be markedly
different from the AST.

Instead of dumping we could write a query interface for Emacs into GCC
where Emacs can ask some GCC module questions.  But programs other than
Emacs would be able to use it.  We could dump the AST in Lisp form.  But
one of the points of Lisp is that its data is simple to read by
computer.

Now let's step back and put all the details aside for a moment: whatever
we are afraid of others implementing as a compellingly useful
proprietary application built for some proprietary tool could be
implemented as a compellingly useful free application built for Emacs or
other free tools.

For better or worse, Emacs _is_ a framework that can be used for tying
separate independent components into a compelling and cohering whole.

So it does not really make all that much sense allowing or prohibiting
certain kinds of applications, but rather allowing them in lockstep with
people doing the actual work on Emacs, just to avoid others making a
headstart with our own tools.

One problem with that approach is that it requires micromanagement.  Why
is that bad?  Let's explain with a joke:

   The physics faculty goes to the university president asking for more
   funding for a particle accelerator.  The president is oblivious "All
   that money!  Can't you take the mathematics department as model?  We
   just need to provide them with pencils, paper, and wastebaskets, and
   they are able to work for years.  And the philosophy department does
   not even need wastebaskets."

Now for programmers, and possibly more so programmers of free software,
the wastebasket is indeed one of the most important resources.
Micromanaging means that one needs to manage before even getting to the
wastebasket stage.  That's expensive.  Also potentially embarrassing: my
wastebasket is actually rather private; I don't actually enjoy parading
my mistakes and misestimates in full detail in public.  They are part of
my own creative process.

So since anything useful in a free program may be useful in a
proprietary one as well, and since Emacs is an independent application
and we cannot really make it reach into GCC in a manner where they
become a single application with regard to licensing, the moment where
we need to think about enabling some possible interaction is when people
are seriously working on an actual implementation.  That's when it
actually makes sense to step aside and let the horses out of the stable.
And we can't avoid getting the proprietary ones out at the same time
than the free ones: the best we can do is keeping the doors locked until
our own horses are at the start, so they at least don't get a headstart.

But at least we can make the racetrack as friendly to our own horses as
we can.  Like dumping the information in a form that Emacs can directly
read.

As to the question "does it need to be the full annotated syntax tree"?
For almost any single application, I am pretty sure the answer is "no".
But it is one natural starting point for figuring out just _what_
information a particular application may want to need.  Another starting
point is the information of visible identifiers and data types at every
source point location.  Compressing that into a reasonably concise
offline data structure is a challenge of its own.

At the current point of time, I don't really see that we can hope to
make people enjoying working with and on free software without handing
them control of their own wastebasket and letting themselves figure out
where they can go with all the information they can ask for.  Yes, all
that information is equally useful for proprietary applications.  We
cannot really stop that by micromanagement.  There might be some
licensing hack that could discriminate free/proprietary but I cannot
really think of one, and if there was one I'd be surprised if it would
not have to be much more invasive than the GPL.

So I'd really be happy if the decision-making following this discussion
would result in more than a solution for contextual completion.  It
really calls for a general strategy for the question of how to make
Emacs, the ultimate application glue, actually get access to everything
that free software programmers want to glue together.  Even if it means
that other people can replicate the process using non-free glue.

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[Eric Ludlam]

On 01/10/2015 02:59 PM, Dmitry Gutov wrote:
> On 01/10/2015 10:29 PM, Richard Stallman wrote:
>
>> Getting back to completion, some tried to dismiss the idea that
>> completion can be done with less than the full AST.  The discussion
>> convinces me that it can be done with somewhat less
>
> Like Eric said, it *can* be done with a lot less, the question is
> whether we really do want to do it this way.
>
> If the plugin implements the "meat" of the completion logic (probably in
> C), Emacs could serve as a "dumb client", only forwarding completion
> queries to the plugin, and interpreting the responses.
>
> The downsides:
>
> 1. Any new feature, significant change, or a bugfix would probably have
> to touch the plugin code, which exludes contributions from a significant
> portion of the user base who would rather contribute Emacs Lisp code.

I agree with this, but

> 2. CEDET integration would basically be out of the question, while it's
> the blessed Emacs code assistance suite, which we would want to see
> developed.

I think there are fine ways to integrate tools at many different levels 
into CEDET.  Some examples include GNU Global (an external tagger), or 
javap for extracting symbol info from .jar files a source file may 
reference.  It can use exuberent ctags as a C++ parser in place of the 
built in one too (though substandard).  GCC could substitute in at any 
of those levels emitting tags in the form CEDET understands.  It could 
replace the local context parser (which is regexp based).   CEDET has a 
way to swap in alternate implementations at many different levels on a 
per language basis for this purpose.

The difference is that each custom use of gcc would be some new plugin 
implementation instead of new Emacs Lisp code interpreting some single 
output.

Eric

Re: Emacs contributions, C and Lisp

[Stefan Monnier]

> plugin unusable. Also, I seem to be unable to discuss with Richard
> without him calling out my behavior,

Nobody seems to be able to do that on this subject.  It seems to be an
emotional subject for Richard.

> so I'm clearly not the right person for this job.

No, I think it's got nothing to do with you,


        Stefan

Re: Emacs contributions, C and Lisp

[Stefan Monnier]

> It is perfectly reasonable if you choose to step down from arguing this
> case once you see yourself as having no additional points to make.

He's not stepping down from the discussion.  He was actually working on
the actual GCC plugin, and that's what he's "stepping down from".


        Stefan

Re: Emacs contributions, C and Lisp

[David Kastrup]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>> plugin unusable. Also, I seem to be unable to discuss with Richard
>> without him calling out my behavior,
>
> Nobody seems to be able to do that on this subject.  It seems to be an
> emotional subject for Richard.

Entering a discussion with one's conclusion and an immediate call for
action is not all that much of a discussion.  It does not become one by
five people doing the same.  Any action is for Richard to decide, to
justify, and to deal with the consequences.

Now it's obvious that he was not prepared for the perceived urgency of
his action, and neither were others.

>> so I'm clearly not the right person for this job.
>
> No, I think it's got nothing to do with you,

The main problem here was bad timing I think.  A lot of work had already
been invested before Richard got an idea that there was something to
think about.

This can easily happen without somebody consciously planning to pass
something under Richard's radar since most people are happy to leave the
policing to Richard and to not bother themselves with it.  But then it
can happen that he notices something requiring a strategy at a point of
time where this is rather inconvenient.

It's not that the respective work is lost: since we are talking about
free software, the main effect of the decision is whether such projects
make it into the core distribution of Emacs and/or GCC, or whether there
will need to be separate distribution channels outside of the FSF's
control.

I don't think it will end there, but obviously that's Richard's choice
to make.  And I would be rather unhappy if Richard's obvious need for
thinking time and breathing room to come to a decision about a
consistent and tenable strategy in that regard would cause people to
abandon work they'd otherwise would have been happy to do.

-- 
David Kastrup

Re: Emacs contributions, C and Lisp

[Thien-Thi Nguyen]

[-- Attachment #1: Type: text/plain, Size: 2118 bytes --]

() "Perry E. Metzger" <perry@piermont.com>
() Sat, 10 Jan 2015 17:07:54 -0500

   I recognize that you don't want me to "change the subject" to
   refactoring, but I don't see this as a change of subject. The
   concern isn't as such code completion, that's just a detail.

Right, i see this discussion as kind of like trying to decide
how to get to the a southeast destination on the gridded city
streets: one can go south then east, or east then south:
  
|   A──────────────y         A -- you are here
|   │              │         B -- destination
|   │              │
|   │              │         x -- completion supported
|   │              │         y -- AST exposed
|   │              │
|   x──────────────B

At the destination, Emacs and GCC will be friendly and fruitful
and a new era of meta-to-the-Nth free software hacking begins.
M-x praise-gnu-i-can-hardly-wait RET!

I think the route through ‘x’, determining the minimal required
design for supporting only completion and implementing that,
makes for a lot more work, later (segment ‘xB’).  That will be a
pain not only in quantity, but also in quality, in the sense of
conceptual backtracking, design retrofitting, and other "undo".

I think it would be wise to traverse ‘Ay’ first, even though
that's a lot of work, too, and more importantly, fraught w/ the
danger of (professional :-D) proprietary misappropriation.  The
latter risk is mitigated because traversing ‘yB’ will be fast.
True, it will be fast also for the (pro) prop m.prop. forces,
but i think free software hackers are more nimble, confident,
and capable of achieving a multiplicity of superior results.

That last statement is both evidence- and faith-based.  The
reader is invited to percolate their experience to taste...

-- 
Thien-Thi Nguyen
   GPG key: 4C807502
   (if you're human and you know it)
      read my lisp: (responsep (questions 'technical)
                               (not (via 'mailing-list)))
                     => nil

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Emacs contributions, C and Lisp

[Mario Lang]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> > To figure out just what Emacs needs, that's the task I am talking
>>> > about.
>>> If you want to support things like completions, refactoring, symbol
>>> searches, etc., we need full access to the AST from inside Emacs.
>> With all due respect, it is so important to avoid the full AST
>> that I'm not going to give up on it just because someone claims
>> that is necessary.
>
> Give up on what, exactly?
>
> Richard, this is absurd!
> I'm convinced there is a misunderstanding.

While I also think this attempt to cage data is rather absurd, I think
there is likely another misunderstanding.  What Clang currently seems to
do for editor integration seems to match Richard's vision.

While they have functionality to dump the full AST, clang-format,
clang-query, clang-rename and the completion interface don't expose a full AST at
all.  The idea behind the Clang tools is that they parse the translation
units on their own, and do manipulations on the data internally.  You
pass parameters to these tools, like a position in a file you want to
see completions for, or a range of locations to reformat, or a matcher
expression to search for particular AST constructs.

If GNU were to match that functionality, it would not rely on exporting
a full AST.  Walking a full AST in elisp just to do completion seems
like a performance hog anyway.  While, on the other hand, reparsing the
TU from disk via an external tool is also likely not the fastest
approach.  So while performance is likely an issue with both approaches,
I just wanted to point out that AFAICS, full ASTs are not really used in
the Clang ecosystem for editor integration, they are more like a debug
tool for devs.

Same applies to clang-rename.  You pass in a source location, and a new
name.  And it parses all the involved TUs, and does the change on disk,
or serialize the changes to a sort of diff format for you to review
and apply.

So the approach *they* are taking is to write external tools that do the
various refactoring jobs.  These tools can easily be called from *any*
editor, not needing to duplicate the various refactoring features
everywhere.  I think this approach is rather good, and it does *not*
rely on fully exporting any AST to the various editors, because, as
already mentioned, the goal is to *not* duplicate refactoring features
in the various editing environments.

If GNU were to match this, the work would go into tools based on GCC.
The Emacs integration should be rather simple, since we are basically
just calling a few external commands to have all the work done.

-- 
CYa,
  ⡍⠁⠗⠊⠕

Re: Emacs contributions, C and Lisp

[Óscar Fuentes]

Mario Lang <mlang@delysid.org> writes:

> While I also think this attempt to cage data is rather absurd, I think
> there is likely another misunderstanding.  What Clang currently seems to
> do for editor integration seems to match Richard's vision.
>
> While they have functionality to dump the full AST, clang-format,
> clang-query, clang-rename and the completion interface don't expose a full AST at
> all.

Those are tools that use the Clang library. It's purpose is not to
expose the innards of the compiler to other tools, but to do an specific
task. The topic of this thread is to interface Emacs with GCC for
building the tools in Emacs. The Clang tools are quite inflexible
(actually, they are first attempts at using the Clang library; demos
with some real-world applicability, if you wish.)

[snip]

> If GNU were to match that functionality, it would not rely on exporting
> a full AST.  Walking a full AST in elisp just to do completion seems
> like a performance hog anyway.

I don't think so. You don't need to walk the full AST for every task.
And, as any other data set, the AST can be indexed in various ways. The
performance hog is the generation of the AST, updating it every few
seconds as the user types text. Possibly not the full AST, but only the
subset that corresponds to the area being edited. IIRC Clang has support
for this, dunno about GCC.

> While, on the other hand, reparsing the
> TU from disk via an external tool is also likely not the fastest
> approach.  So while performance is likely an issue with both approaches,
> I just wanted to point out that AFAICS, full ASTs are not really used in
> the Clang ecosystem for editor integration, they are more like a debug
> tool for devs.

I'm not sure what to make of your last phrase. Clang is not in the
bussiness of text editors. OTOH, an editor that uses libClang for a
feature might not resort to querying the AST if libClang provides
higher-level APIs that do the job.

> Same applies to clang-rename.  You pass in a source location, and a new
> name.  And it parses all the involved TUs, and does the change on disk,
> or serialize the changes to a sort of diff format for you to review
> and apply.

It is a mistake to depend on external binaries for features that can be
effectively implemented on Emacs.

> So the approach *they* are taking is to write external tools that do the
> various refactoring jobs.

Again, *they* are not on the text editor business. However, *they* are
interested on showing-off their product and on tools that can work on
batch mode (clang-format, clang-check)

> These tools can easily be called from *any*
> editor, not needing to duplicate the various refactoring features
> everywhere.

Uh? Do you realize that you are on emacs-devel, don't you? We don't care
about how to arrange *external* projects for making them useful for
*other* editors. Moreover, the tooling you propose would be a true
gateway for propietary editors into using GCC for competing with Emacs.

> I think this approach is rather good, and it does *not*
> rely on fully exporting any AST to the various editors, because, as
> already mentioned, the goal is to *not* duplicate refactoring features
> in the various editing environments.
>
> If GNU were to match this, the work would go into tools based on GCC.
> The Emacs integration should be rather simple, since we are basically
> just calling a few external commands to have all the work done.

See above. We don't control GCC, the hackers here are not GCC experts,
we don't want to hack on GCC for every feature we want to add to Emacs,
we don't want to create tools that can be seamlessly used by propietary
products for competing with Emacs... the list goes on and on.

Re: Emacs contributions, C and Lisp

[Achim Gratz]

Mario Lang writes:
> While I also think this attempt to cage data is rather absurd, I think
> there is likely another misunderstanding.  What Clang currently seems to
> do for editor integration seems to match Richard's vision.

That may be the case, but the contention RMS is having revolves around
the ability to use this facility to make a non-free backend (or at least
that's my understanding of it).  An overt dump of the AST is ostensibly
not necessary to cross that threshold.

> While they have functionality to dump the full AST, clang-format,
> clang-query, clang-rename and the completion interface don't expose a full AST at
> all.  The idea behind the Clang tools is that they parse the translation
> units on their own, and do manipulations on the data internally.  You
> pass parameters to these tools, like a position in a file you want to
> see completions for, or a range of locations to reformat, or a matcher
> expression to search for particular AST constructs.
[…]

Looking at the documentation, the base for those tools is essentially a
graph execution engine working on the AST and it could very well be used
to produce a code generator (or backend) if that hasn't been done
already.

> If GNU were to match this, the work would go into tools based on GCC.
> The Emacs integration should be rather simple, since we are basically
> just calling a few external commands to have all the work done.

The plugin interface to GCC already exists and is apparently capable
enough to allow the implementation of a full AST dump.  In fact it seems
that is already implemented in GCC MELT and other GCC plugins certainly
also have the depth to do this.  One step further, DragonEgg
specifically implements a backend (using LLVM tools) that uses GCC as a
front end only, so I guess the only open question is whether or not that
back-end is free or non-free.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: Emacs contributions, C and Lisp

[David Engster]

Mario Lang writes:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>>>> > To figure out just what Emacs needs, that's the task I am talking
>>>> > about.
>>>> If you want to support things like completions, refactoring, symbol
>>>> searches, etc., we need full access to the AST from inside Emacs.
>>> With all due respect, it is so important to avoid the full AST
>>> that I'm not going to give up on it just because someone claims
>>> that is necessary.
>>
>> Give up on what, exactly?
>>
>> Richard, this is absurd!
>> I'm convinced there is a misunderstanding.
>
> While I also think this attempt to cage data is rather absurd, I think
> there is likely another misunderstanding.  What Clang currently seems to
> do for editor integration seems to match Richard's vision.
>
> While they have functionality to dump the full AST, clang-format,
> clang-query, clang-rename and the completion interface don't expose a
> full AST at all. The idea behind the Clang tools is that they parse
> the translation units on their own, and do manipulations on the data
> internally.  You pass parameters to these tools, like a position in a
> file you want to see completions for, or a range of locations to
> reformat, or a matcher expression to search for particular AST
> constructs.

There is one important point that needs to be mentioned: all those
tools, as well as the clang binary itself, are not really
"external". They all simply call functions from libclang. You probably
know this, but this has already confused people here.

Therefore, to do what libclang does, you would have to implement all
that functionality inside GCC. I don't doubt that it is possible - it's
just software, after all. However, you would need experienced GCC
hackers to do that, since you need to hook yourself deep into the parser
to do what libclang does. AFAIK, this is not possible with a mere GCC
plugin.

Unfortunately, GCC hackers are a rare breed. I'm not one of them, and I
don't plan to become one. So my goal is to get GCCs internal data into a
hacking-friendly environment, which essentially means "out of GCC". My
understanding is that due to clang's modular architecture and because it
is implemented as a library, it is much easier to hook yourself into it
than it is in GCC.

For instance, implementing something like completions inside GCC is not
trivial at all. There was actually a patch which tried to implement this
(search in their mailing list for 'gccsense'). It was realized by
introducing a "phantom token" into the lexer, and when the parser would
reach that token, it would spill out the possible completions based on
the current AST. This patch was rejected because it is arguably a hack
and would be very hard to maintain. I actually pointed Richard to this
patch over two years ago; he wanted to investigate, and I never heard
back.

> If GNU were to match that functionality, it would not rely on exporting
> a full AST.  Walking a full AST in elisp just to do completion seems
> like a performance hog anyway.

Walking the full AST in Elisp is out of the question, yes. Some stuff
must be done inside the plugin. For instance, matching certain
expressions in the AST could be done there. And whatever Emacs does must
be done lazily, meaning only looking at those parts of the AST that are
currently in scope; and with heavy caching, of course. Fortunately, much
of that infrastructure is already present in CEDET.

I would also envision external tools. For instance, CEDET already relies
on GNU Global for finding references. One could extend GNU Global to use
GCC's AST instead of parsing the files itself, and introduce new queries
specific to C++.

> So the approach *they* are taking is to write external tools that do the
> various refactoring jobs.

Again: not "really" external. It's all libclang.

> These tools can easily be called from *any* editor, not needing to
> duplicate the various refactoring features everywhere.  I think this
> approach is rather good, and it does *not* rely on fully exporting any
> AST to the various editors, because, as already mentioned, the goal is
> to *not* duplicate refactoring features in the various editing
> environments.
>
> If GNU were to match this, the work would go into tools based on GCC.

But those tools would need the AST! This is precisely what Richard does
not want. So the only solution is to do it inside GCC. I'm not opposed
to that approach at all, but it's not like this is a new idea. GCC is
very hard to hack on, and my guess is that there's simply no manpower to
implement this.

-David

Re: Emacs contributions, C and Lisp

[Perry E. Metzger]

On Sun, 11 Jan 2015 19:49:36 +0100 Mario Lang <mlang@delysid.org>
wrote:
> What Clang
> currently seems to do for editor integration seems to match
> Richard's vision.
> 
> While they have functionality to dump the full AST, clang-format,
> clang-query, clang-rename and the completion interface don't expose
> a full AST at all.

This is getting far afield, but:

Clang/LLVM's libclang provides significantly more than that, and
libAST provides a general AST match and rewrite interface.

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Emacs contributions, C and Lisp

[Mario Lang]

Óscar Fuentes <ofv@wanadoo.es> writes:

> Mario Lang <mlang@delysid.org> writes:
>
>> While I also think this attempt to cage data is rather absurd, I think
>> there is likely another misunderstanding.  What Clang currently seems to
>> do for editor integration seems to match Richard's vision.
>>
>> While they have functionality to dump the full AST, clang-format,
>> clang-query, clang-rename and the completion interface don't expose a full AST at
>> all.
>
> Those are tools that use the Clang library. It's purpose is not to
> expose the innards of the compiler to other tools, but to do an specific
> task. The topic of this thread is to interface Emacs with GCC for
> building the tools in Emacs. The Clang tools are quite inflexible
> (actually, they are first attempts at using the Clang library; demos
> with some real-world applicability, if you wish.)

My point was exactly that.  A solution to the aledged problem of not
wanting to leak a full AST is to have specific tools do specific jobs.

Yes, I know these tools would be a part of GCC.  However, since the
discussion seems to involve overall GNU strategics, I think it isn't
wrong at all to consider pushing the work to another project, if that
solves a strategic problem.

>> If GNU were to match that functionality, it would not rely on exporting
>> a full AST.  Walking a full AST in elisp just to do completion seems
>> like a performance hog anyway.
>
> I don't think so. You don't need to walk the full AST for every task.
> And, as any other data set, the AST can be indexed in various ways. The
> performance hog is the generation of the AST, updating it every few
> seconds as the user types text. Possibly not the full AST, but only the
> subset that corresponds to the area being edited. IIRC Clang has support
> for this, dunno about GCC.

I can certainly see that some people would want to *have* the AST in
Lisp, to be able to do clever hacks on top of that.  I am not
questioning the potential usefulness of that.  All I wanted to point out
was, that there is a solution to the AST export problem.  We might not
be able to do clever on-the-fly hacks, but with tools like gcc-query and
gcc-rename we could at least get much further then we currently are.

>> While, on the other hand, reparsing the
>> TU from disk via an external tool is also likely not the fastest
>> approach.  So while performance is likely an issue with both approaches,
>> I just wanted to point out that AFAICS, full ASTs are not really used in
>> the Clang ecosystem for editor integration, they are more like a debug
>> tool for devs.
>
> I'm not sure what to make of your last phrase. Clang is not in the
> bussiness of text editors. OTOH, an editor that uses libClang for a
> feature might not resort to querying the AST if libClang provides
> higher-level APIs that do the job.

Clang is not a text editor, but with Clang tooling, it definitley tries
to offer IDE integration.

>> Same applies to clang-rename.  You pass in a source location, and a new
>> name.  And it parses all the involved TUs, and does the change on disk,
>> or serialize the changes to a sort of diff format for you to review
>> and apply.
>
> It is a mistake to depend on external binaries for features that can be
> effectively implemented on Emacs.

True.  However, if we can't export the AST for some reason, external
tools would be a useful alternative approach.

>> So the approach *they* are taking is to write external tools that do the
>> various refactoring jobs.
>
> Again, *they* are not on the text editor business. However, *they* are
> interested on showing-off their product and on tools that can work on
> batch mode (clang-format, clang-check)

That's not quite true, if you also consider clang-rename and clang-query.

>> These tools can easily be called from *any*
>> editor, not needing to duplicate the various refactoring features
>> everywhere.
>
> Uh? Do you realize that you are on emacs-devel, don't you? We don't care
> about how to arrange *external* projects for making them useful for
> *other* editors. Moreover, the tooling you propose would be a true
> gateway for propietary editors into using GCC for competing with Emacs.

I am deeply sorry for having proposed a reusable solution.  What was I
thinking.  EOT.

-- 
CYa,
  ⡍⠁⠗⠊⠕

Re: Emacs contributions, C and Lisp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > What you call "pressure", I see as advocacy.

It is both.

I am not going to decide based on anyone's "advocacy".  I intend to
look at the issues carefully and see what conclusions really follow
from them.

  > Does Clang's popularity inform your decision-making process?

Not in the way you want it to.

You demand I draw certain specific conclusions from that, and take
certain actions.  I can see other possible conclusions to draw from
the same facts.

  > I've seen
  > no evidence in this thread that you're considering the broader social
  > context.

You won't see it here.  Your kind of "advocacy" is no help to me in
making the right decision.  I'm not going to take counsel from people
who are trying to pressure me.

  > Do we need to repeat this process every few years and every time someone
  > thinks of a new way to integrate GCC and Emacs?

You don't "need" to do anything about this.  It is not your
responsibility, and you are not by boss.

I have now quoted three successive insults from your message.

I'm not interested in discussing a difficult decision with someone who
treats me as you do.  Rather, I will wait, forget about what you have
said, and then make an unprejudiced decision after consulting others
whose advice I value.  This will take time.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs contributions, C and Lisp

[Helmut Eller]

On Sun, Jan 11 2015, David Engster wrote:

> AFAIK, this is not possible with a mere GCC
> plugin.

So what was your plan (or code, if any) before you abandoned this
project?

Helmut

Re: Emacs contributions, C and Lisp

[David Engster]

Helmut Eller writes:
> On Sun, Jan 11 2015, David Engster wrote:
>
>> AFAIK, this is not possible with a mere GCC
>> plugin.
>
> So what was your plan (or code, if any) before you abandoned this
> project?

The first step would have been to replace our existing C++ parser with
the AST that is produced by GCC. The plugin would output the same LISP
structures that Semantic uses. My work so far was mainly to investigate
how C++ types are actually stored in the AST. Especially the template
stuff is pretty weird, and documentation is sparse. Fortunately, the
headers are pretty well commented, but it still involves a *lot* of
trial and error.

The actual "semantic" part of parsing C++ would still be handled by
Emacs' Semantic package. For instance, it would calculate
completions. So obviously, those completions wouldn't match those from
libclang w.r.t. to accuracy, but they would be *much* better than they
are now, especially because the preprocessor is already handled, which
is currently one of Semantic's main problems. Also, type inference would
already be done by GCC, so you would see the resulting type from 'auto'
and such.

One main problem would be how to parse the file you're currently working
on, since it usually unparseable. I don't have a good answer for
that. We would have our internal parser as fallback, but I would think
that we could also try to internally make the file parseable before
sending it to GCC. It would probably be very messy and involve a lot of
closing braces.

So in a nutshell, for people familiar with clang: my approach is more
like using libtooling/libast than libclang.

All this would have already occupied my spare time for many, many
months, so after this, my plans are very vague. People would probably
want "perfect" completions similar to libclang and not the one by
Semantic. These would have to be provided by GCC itself. Maybe the
'gccsense' patch could serve as a starting point here, but this would
need support from an experienced GCC hacker.

My plan was also to make this plugin usable for other tools. That means,
it should not only output LISP structures, but alternatively also JSON
and possibly XML. For instance, an external tool could build a symbol
database for providing references. This could also serve as a starting
point for doing refactoring. For more complicated tasks, the plugin
could provide an AST matcher which you can query with certain
expressions.

My plan was also to investigate how much work it would be to serve the
AST for other frontends, like Fortran, Go, Java and Ada. I think this
would be the prime advantage over libclang, which is restricted to the
C-family. I haven't looked into it, but I would think that it shouldn't
be too difficult.

-David

Re: Emacs contributions, C and Lisp

[Helmut Eller]

On Mon, Jan 12 2015, David Engster wrote:

> Helmut Eller writes:
>> On Sun, Jan 11 2015, David Engster wrote:
>>
>>> AFAIK, this is not possible with a mere GCC
>>> plugin.
>>
>> So what was your plan (or code, if any) before you abandoned this
>> project?

Thanks for the detailed answer!

> The first step would have been to replace our existing C++ parser with
> the AST that is produced by GCC. The plugin would output the same LISP
> structures that Semantic uses.

I'm a bit confused because at one side you seem to say that certain
things are not possible with plugins but at the other side you seem to
think that plugins can dump enough information to make these things
possible.

> My work so far was mainly to investigate
> how C++ types are actually stored in the AST. Especially the template
> stuff is pretty weird, and documentation is sparse. Fortunately, the
> headers are pretty well commented, but it still involves a *lot* of
> trial and error.

I can imagine that templates are complicated.  I tried to implement a
find-definition command as a GCC plugin.  My first approach was to
search the smallest subtree that contains a particular source location.
That didn't work out because GCC doesn't record "source ranges" so it's
difficult to know if a tree covers a particular location.  Another
problem is that identifiers are resolved early eg. "x + y" produces a
PLUS_EXPR (with the source location pointing to the + sign) but the
arguments are pointers to the VAR_DECLs of x and y and the source
location of those VAR_DECLs is typically a few lines earlier.

In a second attempt I made Emacs insert a custom #pragma at the place
where we want to search for a definition; similar to the gccsense
approach.  Plugins can register pragmas and that way have access to the
lexer.  That kinda works but the problem is that pragmas are only
allowed in certain places (eg. at the end of a statement) and Emacs has
to guess where those places are.

> The actual "semantic" part of parsing C++ would still be handled by
> Emacs' Semantic package. For instance, it would calculate
> completions. So obviously, those completions wouldn't match those from
> libclang w.r.t. to accuracy, but they would be *much* better than they
> are now, especially because the preprocessor is already handled, which
> is currently one of Semantic's main problems. Also, type inference would
> already be done by GCC, so you would see the resulting type from 'auto'
> and such.

Is the idea is to let GCC output some "global" information like type
declarations to enable better "local" parsing of function bodies in
Emacs?  Or do you want to do pretty much all parsing in GCC?

> One main problem would be how to parse the file you're currently working
> on, since it usually unparseable. I don't have a good answer for
> that. We would have our internal parser as fallback, but I would think
> that we could also try to internally make the file parseable before
> sending it to GCC. It would probably be very messy and involve a lot of
> closing braces.

Completion seems to be difficult problem. find-definition looks simpler
and perhaps better as a first step.

> So in a nutshell, for people familiar with clang: my approach is more
> like using libtooling/libast than libclang.

I know libclang only a bit and libtooling/libast not at all.  What I
failed to implement is like libclang's clang_getCursor.

[...]
> My plan was also to make this plugin usable for other tools. That means,
> it should not only output LISP structures, but alternatively also JSON
> and possibly XML. For instance, an external tool could build a symbol
> database for providing references. This could also serve as a starting
> point for doing refactoring. For more complicated tasks, the plugin
> could provide an AST matcher which you can query with certain
> expressions.

In general I think the heavy lifting should be done in GCC+plugin and
Emacs should only do the "easy" stuff like displaying the result.  But
for performance and other reasons it might be necessary to do at least
some parsing in Emacs too.

Helmut

Re: Emacs contributions, C and Lisp

[David Engster]

Helmut Eller writes:
> On Mon, Jan 12 2015, David Engster wrote:
>> The first step would have been to replace our existing C++ parser with
>> the AST that is produced by GCC. The plugin would output the same LISP
>> structures that Semantic uses.
>
> I'm a bit confused because at one side you seem to say that certain
> things are not possible with plugins but at the other side you seem to
> think that plugins can dump enough information to make these things
> possible.

I'm not sure how familiar you are with CEDET. We already have the
infrastructure to parse local expressions and calculate completions
based on a database of "tags", which are structures generated from our
own C++ parser. At a first step, I wanted to replace only the parser,
meaning the part which creates the AST. The actual "rules" of C++ are
coded in Semantic (to various degree).

>> My work so far was mainly to investigate
>> how C++ types are actually stored in the AST. Especially the template
>> stuff is pretty weird, and documentation is sparse. Fortunately, the
>> headers are pretty well commented, but it still involves a *lot* of
>> trial and error.
>
> I can imagine that templates are complicated.  I tried to implement a
> find-definition command as a GCC plugin.  My first approach was to
> search the smallest subtree that contains a particular source location.
> That didn't work out because GCC doesn't record "source ranges" so it's
> difficult to know if a tree covers a particular location.  Another
> problem is that identifiers are resolved early eg. "x + y" produces a
> PLUS_EXPR (with the source location pointing to the + sign) but the
> arguments are pointers to the VAR_DECLs of x and y and the source
> location of those VAR_DECLs is typically a few lines earlier.
>
> In a second attempt I made Emacs insert a custom #pragma at the place
> where we want to search for a definition; similar to the gccsense
> approach.  Plugins can register pragmas and that way have access to the
> lexer.  That kinda works but the problem is that pragmas are only
> allowed in certain places (eg. at the end of a statement) and Emacs has
> to guess where those places are.

Indeed, the main difficulty here is to find the correct location in the
AST when you only have line/column information. But when using CEDET,
your source file will already be parsed, so Semantic has type
information for your symbols, meaning it would already know the types
from "x" and "y". You could directly ask the GCC plugin for the
definition of that actual type (it probably wouldn't even have to call
the plugin, because Semantic has a database for types).

>> The actual "semantic" part of parsing C++ would still be handled by
>> Emacs' Semantic package. For instance, it would calculate
>> completions. So obviously, those completions wouldn't match those from
>> libclang w.r.t. to accuracy, but they would be *much* better than they
>> are now, especially because the preprocessor is already handled, which
>> is currently one of Semantic's main problems. Also, type inference would
>> already be done by GCC, so you would see the resulting type from 'auto'
>> and such.
>
> Is the idea is to let GCC output some "global" information like type
> declarations to enable better "local" parsing of function bodies in
> Emacs?  Or do you want to do pretty much all parsing in GCC?

Here's how Semantic currently does it: when you load a file, it will
first parse only declarations and function signatures, so something like
a "shallow parse" (with depth=0). When you put your cursor in a
function, it calls the parser again and asks him to only parse the
function's body, which would be a depth=1 parse. I wanted to try to do a
similar thing with the GCC plugin; that means, by default it would do a
"shallow parse", skipping things like function bodies and only output
their signatures. Then later, you would pass parameters like a
function's name for which you'd like the detailed AST, or only things
like local variables or similar.

>> My plan was also to make this plugin usable for other tools. That means,
>> it should not only output LISP structures, but alternatively also JSON
>> and possibly XML. For instance, an external tool could build a symbol
>> database for providing references. This could also serve as a starting
>> point for doing refactoring. For more complicated tasks, the plugin
>> could provide an AST matcher which you can query with certain
>> expressions.
>
> In general I think the heavy lifting should be done in GCC+plugin and
> Emacs should only do the "easy" stuff like displaying the result.  But
> for performance and other reasons it might be necessary to do at least
> some parsing in Emacs too.

Again, I'm not very familiar with GCC, which is why I wanted to do as
much in Elisp as possible, meaning to re-use as much from CEDET as
possible. My primary goal was to make the C++ parser more accurate and
faster.

-David