bug#22241: 25.0.50; etags Ruby parser problems

bug#22241: 25.0.50; etags Ruby parser problems

[Dmitry Gutov]

In GNU Emacs 25.0.50.5 (x86_64-unknown-linux-gnu, GTK+ Version 3.16.7).

It's great that we've incorporated some Ruby support, but it has some
apparent problems:

- Constants are not indexed.

- Class methods (def self.foo) are given the wrong name ("self."
  shouldn't be included).

- "class << self" blocks are given a separate entry.

- Qualified tag names are never generated.

Take this example file:

module A
  class B
    ABC = 4

    def foo!
    end

    def self._bar?(abc)
    end

    class << self
      def qux=(tee)
      end
    end
  end
end

It should have this unqualified index:

A
B
ABC
foo!
_bar?
qux=

And the qualified names should look like this:

A
A::B
A::B::ABC
A::B#foo!
A::B.bar?
A::B.qux=

Lastly, it would be great if the parser recognized some built-in
code-generating methods. Example:

def A
  attr_reader :foo
  attr_writer :bar
  attr_accessor :tee
  alias_method :qux, :tee
end

should become (the unqualified version):

A
foo
bar=
tee
tee=
qux

All attr_* methods can take a variable number of arguments. The parser
should take each argument, check that it's a symbol and not a variable
(starts with :), and if so, record the corresponding method name.

Thanks!

bug#22241: 25.0.50; etags Ruby parser problems

[Dmitry Gutov]

On 12/26/2015 05:59 AM, Dmitry Gutov wrote:

> And the qualified names should look like this:
>
> ...
> A::B.bar?

Sorry, this should be "A::B._bar?".

bug#22241: 25.0.50; etags Ruby parser problems

[Dmitry Gutov]

And looking at the existing examples:

def ModuleExample.singleton_module_method

should translate to "singleton_module_method" as unqualified name, and 
"ModuleExample.singleton_module_method" as qualified name.

bug#22241: 25.0.50; etags Ruby parser problems

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sat, 26 Dec 2015 05:59:34 +0200
> 
> It's great that we've incorporated some Ruby support, but it has some
> apparent problems:

I don't speak Ruby.  So please give a more detailed spec for the
features you want added.  I wrote some questions below, but I'm quite
sure there are more questions I should ask, but don't know about.  So
please provide as complete specification for each feature as you
possibly can, TIA.

> - Constants are not indexed.

What is the full syntax of a "constant"?  Is it just

  IDENTIFIER "=" INTEGER-NUMBER

?  Is whitespace significant?  What about newlines?

> - Class methods (def self.foo) are given the wrong name ("self."
>   shouldn't be included).

Is it enough to remove a single "self.", case-sensitive, at the
beginning of an identifier?  Can there be more than one, like
"self.self.SOMETHING"?  Your other example, i.e.

  def ModuleExample.singleton_module_method

indicates that anything up to and including the period should be
removed, is that correct?  Is there only one, or can there be many?
Should they all be removed for an unqualified name?

> - "class << self" blocks are given a separate entry.

What should be done instead?  Can't a class be named "<<"?

> - Qualified tag names are never generated.

(Etags never promised qualified names except for C and derived
languages, and also in Java.)

How to know when a module's or a class's scope ends?  Is it enough to
count "end" lines?  Can I assume that "end" will always appear by
itself on a line?  Can I disregard indentation of "end" (and of
everything else) when I determine where a scope begins and ends?

> A
> A::B
> A::B::ABC
> A::B#foo!
> A::B.bar?
> A::B.qux=

Why did 'foo!' get a '#' instead of a '.', as for '_bar'?  Why doesn't
"class << self" count as a class scope, and add something to qualified
names?

> Lastly, it would be great if the parser recognized some built-in
> code-generating methods. Example:
> 
> def A
>   attr_reader :foo
>   attr_writer :bar
>   attr_accessor :tee
>   alias_method :qux, :tee
> end
> 
> should become (the unqualified version):
> 
> A
> foo
> bar=
> tee
> tee=
> qux
> 
> All attr_* methods can take a variable number of arguments. The parser
> should take each argument, check that it's a symbol and not a variable
> (starts with :), and if so, record the corresponding method name.

Why did 'bar' and 'tee' git a '=' appended?  Are there any other such
"append rules"?

bug#22241: 25.0.50; etags Ruby parser problems

[Dmitry Gutov]

On 01/23/2016 07:38 PM, Eli Zaretskii wrote:

> I don't speak Ruby.  So please give a more detailed spec for the
> features you want added.  I wrote some questions below, but I'm quite
> sure there are more questions I should ask, but don't know about.  So
> please provide as complete specification for each feature as you
> possibly can, TIA.

There's no actual up-to-date language spec, and when in doubt, I fire up 
the REPL and try things out (and forget many of the results afterwards). 
So there's no "detailed spec" in my head. Let me just try my best 
answering your questions, for now.

>> - Constants are not indexed.
>
> What is the full syntax of a "constant"?  Is it just
>
>    IDENTIFIER "=" INTEGER-NUMBER

Pretty much. IDENTIFIER should be ALL_CAPS, or CamelCase, with 
underscores allowed.

INTEGER-NUMBER should be just EXPRESSION, because it can be any 
expression, possibly a multiline one.

CamelCase constants usually are assigned some "anonymous class" value, 
like in the following example:

SpecialError = Class.new(StandardError)

(Which is a metaprogramming-y way to define the class SpecialError).

But you probably shouldn't worry about ALL_CAPS vs CamelCase distinction 
here, and just treat them the same.

> ?  Is whitespace significant?  What about newlines?

No spaces around "=" is fine. Spaces can also be replaced by tabs. A 
newline before "=" is not allowed.

>> - Class methods (def self.foo) are given the wrong name ("self."
>>    shouldn't be included).
>
> Is it enough to remove a single "self.", case-sensitive, at the
> beginning of an identifier?  Can there be more than one, like
> "self.self.SOMETHING"?

One one "self." is allowed. When you remove it, you should record that 
SOMETHING is a method defined on the current class (or module). In Java 
terms, say, it would be like "static" method.

The upshot is, it can be called on the class itself, but not on its 
instance:

irb(main):001:0> class C
irb(main):002:1> def self.foo
irb(main):003:2> 3
irb(main):004:2> end
irb(main):005:1> end
=> nil
irb(main):006:0> C.foo
=> 3
irb(main):007:0> C.new.foo
NoMethodError: undefined method `foo' for #<C:0x000000020141e8>

So the qualified name of that method should be "C.foo", as opposed to 
"C#foo" for an instance method.

> Your other example, i.e.
>
>    def ModuleExample.singleton_module_method
>
> indicates that anything up to and including the period should be
> removed, is that correct?

More or less. This is an "explicit syntax", which is equivalent to using 
"self.". These two declarations are equivalent:

module ModuleExample
   def ModuleExample.foo
   end
end

module ModuleExample
   def self.foo
   end
end

> Is there only one, or can there be many?

There can be only one dot there. There could be a method resolution 
operator (::) in there, I suppose, but I'm not sure if you want to add 
support for that right now, or ever.

> Should they all be removed for an unqualified name?

Yes.

>> - "class << self" blocks are given a separate entry.
>
> What should be done instead?  Can't a class be named "<<"?

A class cannot be named "<<". You should not add that line to the index, 
but record that the method definitions inside the following scope are 
defined on the current class or module. These are equivalent:

class C
   def self.foo
   end
end

class C
   class << self
     def foo
     end
   end
end

>> - Qualified tag names are never generated.
>
> (Etags never promised qualified names except for C and derived
> languages, and also in Java.)

OK, that would be a nice bonus, but we can live without it. ctags 
doesn't define qualified names either.

Without qualified names, I suppose you should treat

def self.foo
end

and

def foo
end

and

def Class.foo
end

the same. Only record those as "foo".

> How to know when a module's or a class's scope ends?  Is it enough to
> count "end" lines?

Hmm, maybe? I'm guessing etags doesn't really handle heredoc syntax, or 
multiline strings defined with percent literals (examples here: 
https://en.wikibooks.org/wiki/Ruby_Programming/Syntax/Literals#.22Here_document.22_notation)

The result shouldn't be too bad if you do that, anyway. Except:

> Can I assume that "end" will always appear by
> itself on a line?

Unfortunately, no. It can also be on the same line, after a semicolon 
(or on any other line, I suppose, but nobody writes Ruby like that). 
Examples:

class SpecialError < StandardError; end

or

class MyStruct < Struct.new(:a, :b, :c); end

(One could also stick a method definition inside that, but I haven't 
seen that in practice yet). So, either:

- 'end' is on a separate line (after ^[ \t]*).
- class/module Name[< ]...; end$

'end' can also be followed by "# some comment" in both cases.

> Can I disregard indentation of "end" (and of
> everything else) when I determine where a scope begins and ends?

Probably, yes.

Indentation is not significant in Ruby, but heredocs can mess up the 
detection of 'end' keywords, so we could use indentation as a way to 
detect where each scope ends. But if etags doesn't normally do that, 
let's not go there now.

>> A
>> A::B
>> A::B::ABC
>> A::B#foo!
>> A::B.bar?
>> A::B.qux=
>
> Why did 'foo!' get a '#' instead of a '.', as for '_bar'?

It's common to use '#' in the qualified names of instance methods, in 
Java, Ruby and JS docstrings. '.' is used for class methods (static 
methods, in Java), or methods defined on other singleton objects.

Examples:

http://usejsdoc.org/tags-inline-link.html (search for '#' there)
http://stackoverflow.com/questions/5915992/javadoc-writing-links-to-methods
http://docs.ruby-lang.org/en/2.1.0/RDoc/Markup.html#class-RDoc::Markup-label-Links 
(the documentation also says to use ":: for class methods", but let's 
not do that)

 > Why doesn't
 > "class << self" count as a class scope, and add something to qualified
 > names?

It just served to turn 'qux=' into a class (static) method.

>> should become (the unqualified version):
>>
>> A
>> foo
>> bar=
>> tee
>> tee=
>> qux
>>
>> All attr_* methods can take a variable number of arguments. The parser
>> should take each argument, check that it's a symbol and not a variable
>> (starts with :), and if so, record the corresponding method name.
>
> Why did 'bar' and 'tee' git a '=' appended?

Because 'attr_writer :bar' effectively expands to

def bar=(val)
   @bar = val
end

and 'attr_accessor :tee' expands into

def tee
   @tee
end

def tee=(val)
   @tee = val
end

> Are there any other such "append rules"?

There are other macros (any code can define a macro), but let's not 
worry about them now.

bug#22241: 25.0.50; etags Ruby parser problems

[Eli Zaretskii]

> Cc: 22241@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sat, 23 Jan 2016 21:23:57 +0300
> 
> On 01/23/2016 07:38 PM, Eli Zaretskii wrote:
> 
> > I don't speak Ruby.  So please give a more detailed spec for the
> > features you want added.  I wrote some questions below, but I'm quite
> > sure there are more questions I should ask, but don't know about.  So
> > please provide as complete specification for each feature as you
> > possibly can, TIA.
> 
> There's no actual up-to-date language spec, and when in doubt, I fire up 
> the REPL and try things out (and forget many of the results afterwards). 
> So there's no "detailed spec" in my head. Let me just try my best 
> answering your questions, for now.

Thanks.  I have a couple of follow-ups.

> >> - Constants are not indexed.
> >
> > What is the full syntax of a "constant"?  Is it just
> >
> >    IDENTIFIER "=" INTEGER-NUMBER
> 
> Pretty much. IDENTIFIER should be ALL_CAPS, or CamelCase, with 
> underscores allowed.
> 
> INTEGER-NUMBER should be just EXPRESSION, because it can be any 
> expression, possibly a multiline one.

So I guess I will leave constants out for now: etags has no notion of
expressions.

> >> - "class << self" blocks are given a separate entry.
> >
> > What should be done instead?  Can't a class be named "<<"?
> 
> A class cannot be named "<<". You should not add that line to the index, 
> but record that the method definitions inside the following scope are 
> defined on the current class or module.

Is the telltale part "<<" or "self" (or both)?  If it's "<<", then are
there other such tokens that "invalidate" a class?

> > How to know when a module's or a class's scope ends?  Is it enough to
> > count "end" lines?
> 
> Hmm, maybe? I'm guessing etags doesn't really handle heredoc syntax, or 
> multiline strings defined with percent literals (examples here: 
> https://en.wikibooks.org/wiki/Ruby_Programming/Syntax/Literals#.22Here_document.22_notation)
> 
> The result shouldn't be too bad if you do that, anyway. Except:
> 
> > Can I assume that "end" will always appear by
> > itself on a line?
> 
> Unfortunately, no. It can also be on the same line, after a semicolon 
> (or on any other line, I suppose, but nobody writes Ruby like that). 
> Examples:
> 
> class SpecialError < StandardError; end
> 
> or
> 
> class MyStruct < Struct.new(:a, :b, :c); end

Looks complicated, but I will look into this.  I hope no identifier
can be named "end", as in

  def foo
    bar = end
  end

?

> >> A
> >> A::B
> >> A::B::ABC
> >> A::B#foo!
> >> A::B.bar?
> >> A::B.qux=
> >
> > Why did 'foo!' get a '#' instead of a '.', as for '_bar'?
> 
> It's common to use '#' in the qualified names of instance methods

What part of the source makes 'foo!' an instance method?

> > Why did 'bar' and 'tee' git a '=' appended?
> 
> Because 'attr_writer :bar' effectively expands to
> 
> def bar=(val)
>    @bar = val
> end
> 
> and 'attr_accessor :tee' expands into
> 
> def tee
>    @tee
> end
> 
> def tee=(val)
>    @tee = val
> end

So you are saying that attr_writer and attr_accessor cause the '=' to
be appended?

Thanks.

bug#22241: 25.0.50; etags Ruby parser problems

[Dmitry Gutov]

On 01/23/2016 09:59 PM, Eli Zaretskii wrote:

> So I guess I will leave constants out for now: etags has no notion of
> expressions.

That would be a noticeable omission. Can't you just look for

^[ \t]([A-Z][a-z0-9_])[ \t]*=[ \t]*

? Then record the first group, and simply don't look at what's being 
assigned. The right hand value is an expression, and you need to skip 
over paired {} and do-end's, but the etags parser has to do that anyway, 
right?

> Is the telltale part "<<" or "self" (or both)?  If it's "<<", then are
> there other such tokens that "invalidate" a class?

It's "class << self" as a whole. Instead of self, there could be a 
variable, or a class name, but let's ignore those cases for now.

If we see "class <<" - it's not a class definition. If it's followed by 
"self", record the methods inside the scope as class methods. If it's 
followed by something other than "self"... maybe even skip the following 
scope altogether.

> Looks complicated, but I will look into this.  I hope no identifier
> can be named "end", as in
>
>    def foo
>      bar = end
>    end
>
> ?

No variable can be named 'end'. But 'end' can be a method name (so 
foo.end is valid syntax). You should also be on the lookout for :end or 
end:, that's an :end Symbol, not a keyword.

In practice, the 'end' keyword is almost always either preceded by ^[ 
\t]* or by ;[ \t]*.

>> It's common to use '#' in the qualified names of instance methods
>
> What part of the source makes 'foo!' an instance method?

An instance method is a "normal" method, i.e. a method you can call on 
an "instance" of a class. Example:

class C
   def foo
   end
end

(That's C#foo).

c = C.new # instantiate
c.foo # call

>> Because 'attr_writer :bar' effectively expands to
>>
>> def bar=(val)
>>     @bar = val
>> end
>>
>> and 'attr_accessor :tee' expands into
>>
>> def tee
>>     @tee
>> end
>>
>> def tee=(val)
>>     @tee = val
>> end
>
> So you are saying that attr_writer and attr_accessor cause the '=' to
> be appended?

They generate a method with the name bar=, yes.

To clarify the meaning of this: you can't have '=' in a name of a 
variable, only at the end of a method name. And if you have 'bar=(val)' 
defined in class C, it gets called during assignment:

class C
   def bar=(val)
     @bar = val
   end

   def foo
     @bar * 3
   end
end

c = C.new
c.bar = 4
c.foo # => 12

So attr_writer, attr_reader and attr_accessor generate "accessor" 
methods for the instance variables in the given class.

bug#22241: 25.0.50; etags Ruby parser problems

[Eli Zaretskii]

> Cc: 22241@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sat, 23 Jan 2016 22:29:02 +0300
> 
> On 01/23/2016 09:59 PM, Eli Zaretskii wrote:
> 
> > So I guess I will leave constants out for now: etags has no notion of
> > expressions.
> 
> That would be a noticeable omission. Can't you just look for
> 
> ^[ \t]([A-Z][a-z0-9_])[ \t]*=[ \t]*
> 
> ? Then record the first group, and simply don't look at what's being 
> assigned.

That's possible, but is it good enough?  Does the above regexp
necessarily mean it's a constant?

> > Is the telltale part "<<" or "self" (or both)?  If it's "<<", then are
> > there other such tokens that "invalidate" a class?
> 
> It's "class << self" as a whole. Instead of self, there could be a 
> variable, or a class name, but let's ignore those cases for now.
> 
> If we see "class <<" - it's not a class definition.

OK.

Thanks for the other info, I will work on this.

bug#22241: 25.0.50; etags Ruby parser problems

[Dmitry Gutov]

On 01/23/2016 11:48 PM, Eli Zaretskii wrote:

>> ^[ \t]([A-Z][a-z0-9_])[ \t]*=[ \t]*

                         ^ I missed a * there.

>> ? Then record the first group, and simply don't look at what's being
>> assigned.
>
> That's possible, but is it good enough?  Does the above regexp
> necessarily mean it's a constant?

I think so. The important point is that its name begins with a capital 
letter.

And we should probably recognize assignments like these:

ModuleExample::CONSTANT = 5

The qualified name "ModuleExample::CONSTANT" if at the top level, 
unqualified name is "CONSTANT". When inside classes, modules or methods, 
only record the unqualified name; maybe disregard these assignments when 
inside methods altogether.

Thanks in advance.

bug#22241: 25.0.50; etags Ruby parser problems

[Eli Zaretskii]

> Cc: 22241@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 24 Jan 2016 00:43:21 +0300
> 
> On 01/23/2016 11:48 PM, Eli Zaretskii wrote:
> 
> >> ^[ \t]([A-Z][a-z0-9_])[ \t]*=[ \t]*
> 
>                          ^ I missed a * there.
> 
> >> ? Then record the first group, and simply don't look at what's being
> >> assigned.
> >
> > That's possible, but is it good enough?  Does the above regexp
> > necessarily mean it's a constant?
> 
> I think so. The important point is that its name begins with a capital 
> letter.
> 
> And we should probably recognize assignments like these:
> 
> ModuleExample::CONSTANT = 5
> 
> The qualified name "ModuleExample::CONSTANT" if at the top level, 
> unqualified name is "CONSTANT". When inside classes, modules or methods, 
> only record the unqualified name; maybe disregard these assignments when 
> inside methods altogether.

OK, thanks.  I will see what I can do with this.

bug#22241: 25.0.50; etags Ruby parser problems

[Eli Zaretskii]

> Cc: 22241@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sat, 23 Jan 2016 21:23:57 +0300
> 
> >> - "class << self" blocks are given a separate entry.
> >
> > What should be done instead?  Can't a class be named "<<"?
> 
> A class cannot be named "<<". You should not add that line to the index, 
> but record that the method definitions inside the following scope are 
> defined on the current class or module. These are equivalent:
> 
> class C
>    def self.foo
>    end
> end
> 
> class C
>    class << self
>      def foo
>      end
>    end
> end

What about the following snippet: what tags, if any, should it produce
for the 'class' line?

class << a
  def inspect
    '"bar"'
  end
end

Exuberant ctags doesn't produce anything for that line, FWIW.

Also, in the above example, what should be the class-qualified name of
'inspect'?

Thanks.

bug#22241: 25.0.50; etags Ruby parser problems

[Eli Zaretskii]

> Date: Sun, 24 Jan 2016 17:44:44 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 22241@debbugs.gnu.org
> 
> > Cc: 22241@debbugs.gnu.org
> > From: Dmitry Gutov <dgutov@yandex.ru>
> > Date: Sun, 24 Jan 2016 00:43:21 +0300
> > 
> > On 01/23/2016 11:48 PM, Eli Zaretskii wrote:
> > 
> > >> ^[ \t]([A-Z][a-z0-9_])[ \t]*=[ \t]*
> > 
> >                          ^ I missed a * there.
> > 
> > >> ? Then record the first group, and simply don't look at what's being
> > >> assigned.
> > >
> > > That's possible, but is it good enough?  Does the above regexp
> > > necessarily mean it's a constant?
> > 
> > I think so. The important point is that its name begins with a capital 
> > letter.
> > 
> > And we should probably recognize assignments like these:
> > 
> > ModuleExample::CONSTANT = 5
> > 
> > The qualified name "ModuleExample::CONSTANT" if at the top level, 
> > unqualified name is "CONSTANT". When inside classes, modules or methods, 
> > only record the unqualified name; maybe disregard these assignments when 
> > inside methods altogether.
> 
> OK, thanks.  I will see what I can do with this.

Please take a look at the results of commit 25b79d7 on the emacs-25
branch.  I think I implemented everything except the optional name
qualification.  I hope the results are good enough.  If you agree,
please close the bug.

Of course, if there are still bugs, or the implementation doesn't
catch some use cases, please show them.

Thanks.

bug#22241: 25.0.50; etags Ruby parser problems

[Dmitry Gutov]

On 01/30/2016 01:52 PM, Eli Zaretskii wrote:

> What about the following snippet: what tags, if any, should it produce
> for the 'class' line?
>
> class << a
>    def inspect
>      '"bar"'
>    end
> end

No tags on the class line. It's not a declaration either, it's another 
way to define a method named 'inspect' on the value of 'a'. 'a' must be 
a local variable.

> Also, in the above example, what should be the class-qualified name of
> 'inspect'?

Depends on the value of a. Which would be pretty hard for etags to 
track, hence my earlier suggestion to skip it:

     If it's followed by something other than "self"...
     maybe even skip the following scope altogether.

bug#22241: 25.0.50; etags Ruby parser problems

[Dmitry Gutov]

On 01/30/2016 03:21 PM, Eli Zaretskii wrote:

> Please take a look at the results of commit 25b79d7 on the emacs-25
> branch.  I think I implemented everything except the optional name
> qualification.  I hope the results are good enough.  If you agree,
> please close the bug.
>
> Of course, if there are still bugs, or the implementation doesn't
> catch some use cases, please show them.

Thank you.

First, it doesn't seem like it handles attr_reader, attr_writter, 
attr_accessor and method_alias, like I asked at the end of the bug 
report. That line of discussion was somehow dropped.

Second: a minor thing. If I remove the space after '=', 'ABC =4' doesn't 
get recorded.

Third, this is tangential, but I don't think anybody uses the .ruby 
extension for Ruby files (you can see it's not in auto-mode-alist). But 
maybe someone somewhere will use it for something else, and etags will 
erroneously parse that file as Ruby?

On the other hand, you might want to add *.ru, *.rbw, Rakefile and 
Thorfile to the list of Ruby file names.

I'll be sure to let you know if I notice something else.

bug#22241: 25.0.50; etags Ruby parser problems

[Eli Zaretskii]

> Cc: 22241@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 31 Jan 2016 01:06:07 +0300
> 
> First, it doesn't seem like it handles attr_reader, attr_writter, 
> attr_accessor and method_alias, like I asked at the end of the bug 
> report. That line of discussion was somehow dropped.

Right, that part is not implemented.  Perhaps later.  Is it terribly
important?

> Second: a minor thing. If I remove the space after '=', 'ABC =4' doesn't 
> get recorded.

Will fix.

> Third, this is tangential, but I don't think anybody uses the .ruby 
> extension for Ruby files (you can see it's not in auto-mode-alist). But 
> maybe someone somewhere will use it for something else, and etags will 
> erroneously parse that file as Ruby?

I found that on the Internet, I can try to find that again.

> On the other hand, you might want to add *.ru, *.rbw, Rakefile and 
> Thorfile to the list of Ruby file names.

That's easy to add.  Should we?

> I'll be sure to let you know if I notice something else.

Thanks.

bug#22241: 25.0.50; etags Ruby parser problems

[Dmitry Gutov]

On 01/31/2016 06:37 AM, Eli Zaretskii wrote:

> Right, that part is not implemented.  Perhaps later.  Is it terribly
> important?

Exuberant Ctags doesn't do it. I suppose it's rather a missing feature 
than a bug.

It's fine if it's not in 25.1, but let's keep the bug open until it's 
implemented.

>> Third, this is tangential, but I don't think anybody uses the .ruby
>> extension for Ruby files (you can see it's not in auto-mode-alist). But
>> maybe someone somewhere will use it for something else, and etags will
>> erroneously parse that file as Ruby?
>
> I found that on the Internet, I can try to find that again.

Please do.

>> On the other hand, you might want to add *.ru, *.rbw, Rakefile and
>> Thorfile to the list of Ruby file names.
>
> That's easy to add.  Should we?

I believe so. *.ru is a bit questionable (it's not an official Ruby 
extension, and it might be used by some other file formats), but it's 
used by a very popular Ruby library for web application init scripts, 
and those can contain different definitions (in practice, mostly 
constants, although it can have functions defined, if the application is 
tiny, or somehow exotic). *.rbw is the Windows extension for Ruby 
programs that don't need the cmd window. Rakefile and Thorfile can also 
contain definitions (usually constants, but not necessarily just them, 
in the former, and classes and methods in the latter).

bug#22241: 25.0.50; etags Ruby parser problems

[Eli Zaretskii]

> Date: Sun, 31 Jan 2016 05:37:03 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 22241@debbugs.gnu.org
> 
> > Second: a minor thing. If I remove the space after '=', 'ABC =4' doesn't 
> > get recorded.
> 
> Will fix.

But that was a trap, wasn't it?  What can legitimately follow the '+',
in addition to whitespace?  (It's amazing, but among all the gazillion
references to Ruby, I cannot easily find a formal description of its
syntax.)  According to this rare gem:

  https://en.wikibooks.org/wiki/Ruby_Programming/Syntax

(assuming I understand what it says), the RHS can be any literal, and
also any constant expression, is that right?  If so, either (a) we
recognize only '^[ \t]([A-Z][a-z0-9_])*[ \t]*=' and get potential
false positives on the likes of

   ABC == SOMETHING
   ABC =< WHATEVER

etc. (are these possible?); or (b) you tell me which characters can
potentially follow the '=' in an assignment of a constant.  My current
best guess for the latter is this:

    " # % \' ( + - < ? [ {
    0 1 2 3 4 5 6 7 8 9
    A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

And if we go the latter way, there are still multi-line expressions
that I think are way too much.

Ugh!

bug#22241: 25.0.50; etags Ruby parser problems

[Eli Zaretskii]

> Cc: 22241@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 31 Jan 2016 08:43:15 +0300
> 
> On 01/31/2016 06:37 AM, Eli Zaretskii wrote:
> 
> > Right, that part is not implemented.  Perhaps later.  Is it terribly
> > important?
> 
> Exuberant Ctags doesn't do it. I suppose it's rather a missing feature 
> than a bug.
> 
> It's fine if it's not in 25.1, but let's keep the bug open until it's 
> implemented.

Ah, so it _is_ important.  But then I'd need a complete specification
of what is needed.  (And I already smell a tip of an iceberg.)  Again,
the references are scarce and incomplete, but I already understand
that it could be either of the following

  attr_WHATEVER :foo
  SOMETHING ; attr_WHATEVER :foo ; ...
  attr_WHATEVER :foo, :bar; ...

Is that true?  Are there any other forms, or can the symbol be
followed only by a comma, a semi-colon, or whitespece?  And what ends
a line like that -- a newline, or can it be continued on the next
line?

> >> Third, this is tangential, but I don't think anybody uses the .ruby
> >> extension for Ruby files (you can see it's not in auto-mode-alist). But
> >> maybe someone somewhere will use it for something else, and etags will
> >> erroneously parse that file as Ruby?
> >
> > I found that on the Internet, I can try to find that again.
> 
> Please do.

Couldn't find it.  And it isn't important enough to argue, just tell
which file-name extensions to consider Ruby and I will do it.

bug#22241: 25.0.50; etags Ruby parser problems

[Dmitry Gutov]

On 01/31/2016 09:01 PM, Eli Zaretskii wrote:

> But that was a trap, wasn't it?

Almost every feature is a trap, if one considers it long enough. :)

> What can legitimately follow the '+',
> in addition to whitespace?  (It's amazing, but among all the gazillion
> references to Ruby, I cannot easily find a formal description of its
> syntax.)

And there isn't one! Ruby is magical that way.

> According to this rare gem:
>
>    https://en.wikibooks.org/wiki/Ruby_Programming/Syntax
>
> (assuming I understand what it says), the RHS can be any literal, and
> also any constant expression, is that right?  If so, either (a) we
> recognize only '^[ \t]([A-Z][a-z0-9_])*[ \t]*=' and get potential
> false positives on the likes of
>
>     ABC == SOMETHING
>     ABC =< WHATEVER

=< is not a valid operator. You must be thinking of <=.

> etc. (are these possible?); or (b) you tell me which characters can
> potentially follow the '=' in an assignment of a constant.

Why not do it like this:

If 'ABC =' is followed by any character, except for '=' and '>', you 
record it as a tag "ABC".

>      " # % \' ( + - < ? [ {
>      0 1 2 3 4 5 6 7 8 9
>      A B C D E F G H I J K L M N O P Q R S T U V W X Y Z
>
> And if we go the latter way, there are still multi-line expressions
> that I think are way too much.

What about them? Ideally, you'd skip over multi-line expressions, but 
you'd have to do that whether you record constants or not.

bug#22241: 25.0.50; etags Ruby parser problems

[Dmitry Gutov]

On 01/31/2016 09:11 PM, Eli Zaretskii wrote:

> Ah, so it _is_ important.

It kind of is. But I can open a separate bug for it, if you want.

> But then I'd need a complete specification
> of what is needed.  (And I already smell a tip of an iceberg.)  Again,
> the references are scarce and incomplete, but I already understand
> that it could be either of the following
>
>    attr_WHATEVER :foo
>    SOMETHING ; attr_WHATEVER :foo ; ...
>    attr_WHATEVER :foo, :bar; ...
>
> Is that true?  Are there any other forms, or can the symbol be
> followed only by a comma, a semi-colon, or whitespece?

The newline might also be preceded by a comment, I suppose.

But really, if recognizing attr_WHATEVER when it's just one of the 
instructions on a line presents a noticeable difficulty, you can 
disregard that case: nobody really does that in practice. Or we can 
disregard it at least until somebody complains.

So you would handle

attr_WHATEVER :foo, :bar # comment

and probably

attr_WHATEVER :bar;

(the semicolon is redundant, but hey, it shouldn't be too hard to support)

and the most difficult realistic case I can imagine looks like this:

attr_WHATEVER :foo, :bar, # comment
               :qux, :tee

> And what ends
> a line like that -- a newline, or can it be continued on the next
> line?

If there's a comma at the end of the current line, the argument list 
continues on the next one.

> Couldn't find it.  And it isn't important enough to argue, just tell
> which file-name extensions to consider Ruby and I will do it.

Let's go with my original suggestions, then:

.rb .ru .rbw Rakefile Thorfile

Thanks!

bug#22241: 25.0.50; etags Ruby parser problems

[Eli Zaretskii]

> Cc: 22241@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 1 Feb 2016 11:24:38 +0300
> 
> Why not do it like this:
> 
> If 'ABC =' is followed by any character, except for '=' and '>', you 
> record it as a tag "ABC".

If you say so.  Done.

bug#22241: 25.0.50; etags Ruby parser problems

[Eli Zaretskii]

> Cc: 22241@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 1 Feb 2016 11:40:46 +0300
> 
> On 01/31/2016 09:11 PM, Eli Zaretskii wrote:
> 
> > Ah, so it _is_ important.
> 
> It kind of is. But I can open a separate bug for it, if you want.
> 
> > But then I'd need a complete specification
> > of what is needed.  (And I already smell a tip of an iceberg.)  Again,
> > the references are scarce and incomplete, but I already understand
> > that it could be either of the following
> >
> >    attr_WHATEVER :foo
> >    SOMETHING ; attr_WHATEVER :foo ; ...
> >    attr_WHATEVER :foo, :bar; ...
> >
> > Is that true?  Are there any other forms, or can the symbol be
> > followed only by a comma, a semi-colon, or whitespece?
> 
> The newline might also be preceded by a comment, I suppose.
> 
> But really, if recognizing attr_WHATEVER when it's just one of the 
> instructions on a line presents a noticeable difficulty, you can 
> disregard that case: nobody really does that in practice. Or we can 
> disregard it at least until somebody complains.
> 
> So you would handle
> 
> attr_WHATEVER :foo, :bar # comment
> 
> and probably
> 
> attr_WHATEVER :bar;
> 
> (the semicolon is redundant, but hey, it shouldn't be too hard to support)
> 
> and the most difficult realistic case I can imagine looks like this:
> 
> attr_WHATEVER :foo, :bar, # comment
>                :qux, :tee

OK, this is all implemented, except...

> > And what ends
> > a line like that -- a newline, or can it be continued on the next
> > line?
> 
> If there's a comma at the end of the current line, the argument list 
> continues on the next one.

...this.  If supporting such split definitions is important, it will
need a slightly more complex code.

> Let's go with my original suggestions, then:
> 
> .rb .ru .rbw Rakefile Thorfile

Also done (and doing so exposed a real bug in etags).

Please take a look.

bug#22241: 25.0.50; etags Ruby parser problems

[Dmitry Gutov]

On 02/02/2016 09:16 PM, Eli Zaretskii wrote:

>> So you would handle
>>
>> attr_WHATEVER :foo, :bar # comment
>>
>> and probably
>>
>> attr_WHATEVER :bar;

> OK, this is all implemented, except...

Thank you.

>> If there's a comma at the end of the current line, the argument list
>> continues on the next one.
>
> ...this.  If supporting such split definitions is important, it will
> need a slightly more complex code.

It's basically a multiline function call. Not sure how frequently that 
is used with attr_* in practice, but in our big project at work, just 
one out of 190 attr_* declarations is multiline.

So, it happens, but in the vast majority of cases the arguments stay on 
one line. Some projects (like Rails) choose to make several calls 
instead, as a stylistic choice.

I can't really say yet if the lack of support for multiline calls is a 
significant problem, but it is an omission.

Whether to implement it now, or close this bug and wait until another 
bug report, is up to you.

>> .rb .ru .rbw Rakefile Thorfile
>
> Also done (and doing so exposed a real bug in etags).

Thanks! Looks good.

bug#22241: 25.0.50; etags Ruby parser problems

[Eli Zaretskii]

> Cc: 22241@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Tue, 2 Feb 2016 22:59:35 +0300
> 
> >> If there's a comma at the end of the current line, the argument list
> >> continues on the next one.
> >
> > ...this.  If supporting such split definitions is important, it will
> > need a slightly more complex code.
> 
> It's basically a multiline function call. Not sure how frequently that 
> is used with attr_* in practice, but in our big project at work, just 
> one out of 190 attr_* declarations is multiline.
> 
> So, it happens, but in the vast majority of cases the arguments stay on 
> one line. Some projects (like Rails) choose to make several calls 
> instead, as a stylistic choice.
> 
> I can't really say yet if the lack of support for multiline calls is a 
> significant problem, but it is an omission.
> 
> Whether to implement it now, or close this bug and wait until another 
> bug report, is up to you.

I implemented that now.

bug#22241: 25.0.50; etags Ruby parser problems

[Dmitry Gutov]

On 02/03/2016 07:26 PM, Eli Zaretskii wrote:

> I implemented that now.

Thanks!

One nitpick:

attr_WHATEVER :foo, bar

will generate the methods for 'foo', and whatever the value of bar is. 
But bar is not very likely to have the value :bar, so we should only 
generate the tags for 'foo'.

IOW, skip the arguments that are not Ruby Symbols (don't start with a 
colon). The current implementation treats them the same.

The test example looks a bit odd as well, but I'll comment on that in a 
separate email.

bug#22241: 25.0.50; etags Ruby parser problems

[Eli Zaretskii]

> Cc: 22241@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Thu, 4 Feb 2016 02:21:53 +0300
> 
> One nitpick:
> 
> attr_WHATEVER :foo, bar
> 
> will generate the methods for 'foo', and whatever the value of bar is. 
> But bar is not very likely to have the value :bar, so we should only 
> generate the tags for 'foo'.
> 
> IOW, skip the arguments that are not Ruby Symbols (don't start with a 
> colon). The current implementation treats them the same.

Yes, because etags is not supposed to do sensible things with
syntactically incorrect programs.

I'm not sure I understand the rules, though: is the "no colon, don't
tag" rule valid for any symbol following the attr_WHATEVER, or is that
applicable only to the 2nd, 3rd, etc. symbols?  IOW, what, if
anything, should be tagged here:

  attr_reader foo

bug#22241: 25.0.50; etags Ruby parser problems

[Dmitry Gutov]

On 02/04/2016 06:43 AM, Eli Zaretskii wrote:

>> IOW, skip the arguments that are not Ruby Symbols (don't start with a
>> colon). The current implementation treats them the same.
>
> Yes, because etags is not supposed to do sensible things with
> syntactically incorrect programs.

This is syntactically correct:

class C
   [:foo, :bar].each do |name|
     attr_reader name
   end
end

> I'm not sure I understand the rules, though: is the "no colon, don't
> tag" rule valid for any symbol following the attr_WHATEVER, or is that
> applicable only to the 2nd, 3rd, etc. symbols?

Any of the arguments.

> IOW, what, if
> anything, should be tagged here:
>
>    attr_reader foo

Just skip it as well.

bug#22241: 25.0.50; etags Ruby parser problems

[Eli Zaretskii]

> Cc: 22241@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Thu, 4 Feb 2016 11:24:52 +0300
> 
> > I'm not sure I understand the rules, though: is the "no colon, don't
> > tag" rule valid for any symbol following the attr_WHATEVER, or is that
> > applicable only to the 2nd, 3rd, etc. symbols?
> 
> Any of the arguments.
> 
> > IOW, what, if
> > anything, should be tagged here:
> >
> >    attr_reader foo
> 
> Just skip it as well.

Done.

bug#22241: 25.0.50; etags Ruby parser problems

[Dmitry Gutov]

On 02/04/2016 08:24 PM, Eli Zaretskii wrote:

>> Just skip it as well.
>
> Done.

Thanks a lot, marking this as done.