[el-search] How to search string excluding docstring?

[el-search] How to search string excluding docstring?

[Chunyang Xu]

Hi Michael Heerdegen and other Emacs users,

For example, with the following contents in a buffer

(defun foo ()
  "foo docstring"
  (message "foo string"))

(string "foo") matches both "foo docstring" and "foo string", I want a
way to exclude the docstring, by "docstring", I just mean the string
being checking should not be the fourth element in

  (defun _ _ docstring . _)

I am thinking a pattern like

  (and (pred stringp) (guard (not (docstring-p))) (string "foo"))

but I have no idea how to define 'docstring-p'.

Any tips?

Re: [el-search] How to search string excluding docstring?

[Jean-Christophe Helary]

> On Dec 25, 2017, at 19:27, Chunyang Xu <mail@xuchunyang.me> wrote:
> 
> (defun foo ()
>  "foo docstring"
>  (message "foo string"))
> 
>  (and (pred stringp) (guard (not (docstring-p))) (string "foo"))
> 
> but I have no idea how to define 'docstring-p'.
> 
> Any tips?

The definition of a docstring is that it is located right below the defun. Sorry I can't help much more than that...


Jean-Christophe Helary
-----------------------------------------------
@brandelune http://mac4translators.blogspot.com

Re: [el-search] How to search string excluding docstring?

[Emanuel Berg]

Chunyang Xu wrote:

> but I have no idea how to define
> 'docstring-p'.

Docstrings are in `font-lock-doc-face' so Emacs
already knows how to determine that. Find out
how that is done, and then invoke or reuse that
code from your function.

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: [el-search] How to search string excluding docstring?

[Skip Montanaro]

While you consider a solution, it might be worthwhile to think about
generalizing beyond ELisp. Python, for example, uses the same
structural placement of docstrings for modules, classes, functions,
and methods.

Skip

Re: [el-search] How to search string excluding docstring?

[Chunyang Xu]

Emanuel Berg writes:

> Chunyang Xu wrote:
>
>> but I have no idea how to define
>> 'docstring-p'.
>
> Docstrings are in `font-lock-doc-face' so Emacs
> already knows how to determine that. Find out
> how that is done, and then invoke or reuse that
> code from your function.

It almost solves my problem via:

  (and (pred stringp)
       (guard (not (eq 'font-lock-doc-face (get-char-property (point) 'face))))
       (string "foo"))

but it doesn't work for files which is not already opened, I guess it is
because font-lock-mode is only active when I open the file. For example,

  M-x el-search-directory RET
  (and (pred stringp)
       (guard (not (eq 'font-lock-doc-face (get-char-property (point) 'face))))
       (string "^GNU Emacs")) RET
  M-x el-search-occur RET

also matches process-event-p's docstring (from levents.el.gz):

  (defun process-event-p (obj)
    "True if the argument is a process-output event object.
  GNU Emacs 19 does not currently generate process-output events."
    (eq (car-safe obj) 'process))

and after I open levents.el.gz and try the 'M-x el-search-directory'
with the exact same argument again, the match disappears.

Besides I am still interested in using "pattern match" to solve the
issue. I think it is possible though I don't know how yet.

Re: [el-search] How to search string excluding docstring?

[Emanuel Berg]

Chunyang Xu wrote:

> It almost solves my problem via:
>
>   (and (pred stringp)
>        (guard (not (eq 'font-lock-doc-face (get-char-property (point) 'face))))
>        (string "foo"))
>
> but it doesn't work for files which is not
> already opened, I guess it is because
> font-lock-mode is only active when I open the
> file.

Font lock is run by the idle timer. So if you
are productive enough and do things all the
time, you won't get it :)

If the file isn't opened, it is just a bunch of
data. Font lock isn't encoded in the files,
thanks heaven.

You will have to make a loop that opens all
files... perhaps it'll be a challenge to have
font loke come to life each time tho?

> Besides I am still interested in using
> "pattern match" to solve the issue. I think
> it is possible though I don't know how yet.

Pattern match, like they have for arguments in
Erlang and SML? There are Lisp modules for
that. Still, I'd start with finding out how
font lock does it...

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: [el-search] How to search string excluding docstring?

[Michael Heerdegen]

Chunyang Xu <mail@xuchunyang.me> writes:

>   (and (pred stringp)
>        (guard (not (eq 'font-lock-doc-face (get-char-property (point) 'face))))
>        (string "foo"))
>
> but it doesn't work for files which is not already opened, I guess it is
> because font-lock-mode is only active when I open the file.

Exactly.  It's intentional, because applying the major mode and
fontifying is by a factor n slower than el-searching with most patterns.


Michael.

Re: [el-search] How to search string excluding docstring?

[Michael Heerdegen]

Chunyang Xu <mail@xuchunyang.me> writes:

> Hi Michael Heerdegen and other Emacs users,
>
> For example, with the following contents in a buffer
>
> (defun foo ()
>   "foo docstring"
>   (message "foo string"))
>
> (string "foo") matches both "foo docstring" and "foo string", I want a
> way to exclude the docstring, by "docstring", I just mean the string
> being checking should not be the fourth element in
>
>   (defun _ _ docstring . _)
>
> I am thinking a pattern like
>
>   (and (pred stringp) (guard (not (docstring-p))) (string "foo"))
>
> but I have no idea how to define 'docstring-p'.

The general task is context-sensitive matching.  You have no chance with
the current version of el-search, because all patterns there are not
context aware, as you have noticed.

I want to add that, but because there are some pitfalls, I've not yet
uploaded something like this.

Here is what I currently have:

#+begin_src emacs-lisp
(defun el-search--try-find-parent-beg ()
  ;; try to find beg of parent exp heuristically
  (unless (looking-at "^(")
    (let ((opoint (point)))
      (or (and (search-backward-regexp "[;\(\)\"]" (max (- (point) 300) (point-min)))
               (looking-at "\(")
               (and (or (eobp)) (not (= (char-before) ?\\)))
               (not (looking-back ";" (line-beginning-position))))
          (progn (goto-char opoint)
                 nil)))))

(el-search-defpattern parent (pattern &optional n)
  "Matches when PATTERN matches the (Nth) parent of the current expression.
N defaults to 1.  Slow."
  (let ((parent-end (make-symbol "parent-end"))
        (counter (make-symbol "ctr")))
    `(and
      (let ,counter (or ,n 1))
      (let (,'\` ((,'\, ,pattern)))
        (save-excursion
          (condition-case nil
              (progn
                (while (>= (cl-decf ,counter) 0)
                  (or
                   (el-search--try-find-parent-beg)
                   (cond
                    ((eq (char-before) ?`) (backward-char 1))
                    ((eq (char-before) ?,) (backward-char 1))
                    ((and (eq (char-before (1- (point))) ?,)
                          (eq (char-before) ?@))
                     (backward-char 2))
                    ((eq (char-before) ?') (backward-char 1))
                    ((and (eq (char-before (1- (point))) ?#)
                          (eq (char-before) ?'))
                     (backward-char 2))
                    (t (when-let* ((,parent-end (scan-lists (point) 1 1)))
                         (goto-char ,parent-end)
                         (backward-list))))))
                (list (read (current-buffer))))
            (scan-error nil)))))))
#+end_src

That should enable you to solve your task.  Note it's experimental, and
I only had a quick look again now - want to go hiking!

One thing you can definitely _not_ do with this is to use it recursively
- i.e. (parent (parent PATTERN)) won't work (that's why I added the
optional N argument) - `parent' is only valid when applied to the
current expression.

Please tell me if it works for you.

BTW, another not yet uploaded pattern for matching strings you may find
useful is

#+begin_src emacs-lisp
(el-search-defpattern string-lines (pattern)
  "Match any string whose line number is matched by PATTERN.

Examples: (string-lines 1) matches one-line strings.
\(string-lines (pred (>= 5))\) matches strings consisting of not
more than 5 lines."
  (let ((string (make-symbol "string")))
    `(and (string)
          ,string
          (let ,pattern
            (with-temp-buffer
              (insert ,string)
              (count-lines (point-min) (point-max)))))))
#+end_src

I don't know if it's useful enough to upload it.


Regards,

Michael.

Re: [el-search] How to search string excluding docstring?

[Emanuel Berg]

Michael Heerdegen wrote:

> The general task is context-sensitive
> matching. You have no chance with the current
> version of el-search, because all patterns
> there are not context aware, as you
> have noticed.

Tell us again why one can't just use grep to
search Elisp? :)

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: [el-search] How to search string excluding docstring?

[Chunyang Xu]

Michael Heerdegen writes:

> Chunyang Xu <mail@xuchunyang.me> writes:
>
>> Hi Michael Heerdegen and other Emacs users,
>>
>> For example, with the following contents in a buffer
>>
>> (defun foo ()
>>   "foo docstring"
>>   (message "foo string"))
>>
>> (string "foo") matches both "foo docstring" and "foo string", I want a
>> way to exclude the docstring, by "docstring", I just mean the string
>> being checking should not be the fourth element in
>>
>>   (defun _ _ docstring . _)
>>
>> I am thinking a pattern like
>>
>>   (and (pred stringp) (guard (not (docstring-p))) (string "foo"))
>>
>> but I have no idea how to define 'docstring-p'.
>
> The general task is context-sensitive matching.  You have no chance with
> the current version of el-search, because all patterns there are not
> context aware, as you have noticed.
>
> I want to add that, but because there are some pitfalls, I've not yet
> uploaded something like this.
>
> Here is what I currently have:
>
> #+begin_src emacs-lisp
> (defun el-search--try-find-parent-beg ()
>   ;; try to find beg of parent exp heuristically
>   (unless (looking-at "^(")
>     (let ((opoint (point)))
>       (or (and (search-backward-regexp "[;\(\)\"]" (max (- (point) 300) (point-min)))
>                (looking-at "\(")
>                (and (or (eobp)) (not (= (char-before) ?\\)))
>                (not (looking-back ";" (line-beginning-position))))
>           (progn (goto-char opoint)
>                  nil)))))
>
> (el-search-defpattern parent (pattern &optional n)
>   "Matches when PATTERN matches the (Nth) parent of the current expression.
> N defaults to 1.  Slow."
>   (let ((parent-end (make-symbol "parent-end"))
>         (counter (make-symbol "ctr")))
>     `(and
>       (let ,counter (or ,n 1))
>       (let (,'\` ((,'\, ,pattern)))
>         (save-excursion
>           (condition-case nil
>               (progn
>                 (while (>= (cl-decf ,counter) 0)
>                   (or
>                    (el-search--try-find-parent-beg)
>                    (cond
>                     ((eq (char-before) ?`) (backward-char 1))
>                     ((eq (char-before) ?,) (backward-char 1))
>                     ((and (eq (char-before (1- (point))) ?,)
>                           (eq (char-before) ?@))
>                      (backward-char 2))
>                     ((eq (char-before) ?') (backward-char 1))
>                     ((and (eq (char-before (1- (point))) ?#)
>                           (eq (char-before) ?'))
>                      (backward-char 2))
>                     (t (when-let* ((,parent-end (scan-lists (point) 1 1)))
>                          (goto-char ,parent-end)
>                          (backward-list))))))
>                 (list (read (current-buffer))))
>             (scan-error nil)))))))
> #+end_src
>
> That should enable you to solve your task.  Note it's experimental, and
> I only had a quick look again now - want to go hiking!
>
> One thing you can definitely _not_ do with this is to use it recursively
> - i.e. (parent (parent PATTERN)) won't work (that's why I added the
> optional N argument) - `parent' is only valid when applied to the
> current expression.
>
> Please tell me if it works for you.

It does work for me, via

  (and (string "foo") (not (parent `(defun . ,_))))

I also realize another way, which is not general at all but seeming
working. To check if the current string is docstring

(save-excursion
  (backward-up-list)
  (pcase (read (current-buffer))
    (`(defun ,_ ,_ ,(and (pred stringp) docstring) . ,_)
     (string= docstring string))))

then to search docstring

  (and (string) string (guard ...))

It has a bug, for example, when search the following, it matches both
the first and the second "foo" (the second is not wanted)

  (defun hello ()
    "foo"
    "foo"
    (message "foo"))

But it can be ignored because the second "foo" is useless so we don't
write code like this.

So my current approach of searching string excluding docstring is

  (defun el-search--docstring-p (string)
    (save-excursion
      (backward-up-list)
      (pcase (read (current-buffer))
        (`(defun ,_ ,_ ,(and (pred stringp) docstring) . ,_)
         (string= docstring string)))))

  (and (string "foo") s (guard (not (el-search--docstring-p s))))

I plan to support defvar/defcustom etc later.

> BTW, another not yet uploaded pattern for matching strings you may find
> useful is
>
> #+begin_src emacs-lisp
> (el-search-defpattern string-lines (pattern)
>   "Match any string whose line number is matched by PATTERN.
>
> Examples: (string-lines 1) matches one-line strings.
> \(string-lines (pred (>= 5))\) matches strings consisting of not
> more than 5 lines."
>   (let ((string (make-symbol "string")))
>     `(and (string)
>           ,string
>           (let ,pattern
>             (with-temp-buffer
>               (insert ,string)
>               (count-lines (point-min) (point-max)))))))
> #+end_src
>
> I don't know if it's useful enough to upload it.

Thanks. I don't have such a need yet but it shows me a simple example of
using 'el-search-defpattern'.

>
> Regards,
>
> Michael.

Re: [el-search] How to search string excluding docstring?

[Michael Heerdegen]

Emanuel Berg <moasen@zoho.com> writes:

> Tell us again why one can't just use grep to search Elisp? :)

There are tasks where grep isn't enough, like "find all uses of cl stuff
where I have forgotten the cl- prefix", "find all definitions that have
no docstring", "find all function "foo" calls that make use the third
optional arg", etc.

It can also do list-level query-replace.  Some examples: "I've changed
my function "bar" so that the second and third argument are now
transposed.  Let's adopt every use of it and transpose the expressions
at the second and third argument position."  Or "automatically transform
any nested if's into an equivalent `cond'", or "find any `if'
expressions that have no `else' and make them a "when", and such stuff.


Michael.

Re: [el-search] How to search string excluding docstring?

[Michael Heerdegen]

Chunyang Xu <mail@xuchunyang.me> writes:

> I also realize another way, which is not general at all but seeming
> working. To check if the current string is docstring
>
> (save-excursion
>   (backward-up-list)
>   (pcase (read (current-buffer))
>     (`(defun ,_ ,_ ,(and (pred stringp) docstring) . ,_)
>      (string= docstring string))))
>
> then to search docstring
>
>   (and (string) string (guard ...))
>
> It has a bug, for example, when search the following, it matches both
> the first and the second "foo" (the second is not wanted)
>
>   (defun hello ()
>     "foo"
>     "foo"
>     (message "foo"))

Yes, that's one of the pitfalls you can run into.  One could try to find
out at which list position the current expression is - I've not really
needed something like this, however.

> So my current approach of searching string excluding docstring is
>
>   (defun el-search--docstring-p (string)
>     (save-excursion
>       (backward-up-list)
>       (pcase (read (current-buffer))
>         (`(defun ,_ ,_ ,(and (pred stringp) docstring) . ,_)
>          (string= docstring string)))))
>
>   (and (string "foo") s (guard (not (el-search--docstring-p s))))

Yes, that should work fine (unless you have any top-level strings, but
that is never the case for code buffers).  You could simplify it a bit
by using

 (pred (string= string))

in the pattern.


Regards,

Michael.

Re: [el-search] How to search string excluding docstring?

[tomas]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Mon, Dec 25, 2017 at 03:08:31PM +0100, Emanuel Berg wrote:

[...]

> Pattern match, like they have for arguments in
> Erlang and SML?

Pattern match like pcase, I guess.

Cheers
- -- t
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlpBO6MACgkQBcgs9XrR2ka9IwCeJ+d1Qqm1K3IvNAjSV1qXVNtx
C5gAn1Kui6Qi90CNWpHOjkXNGwznjfAF
=oZm0
-----END PGP SIGNATURE-----

Re: [el-search] How to search string excluding docstring?

[Michael Heerdegen]

Skip Montanaro <skip.montanaro@gmail.com> writes:

> While you consider a solution, it might be worthwhile to think about
>  generalizing beyond ELisp. Python, for example, uses the same
>  structural placement of docstrings for modules, classes, functions,
>  and methods.

If there was a tool to produce abstract syntax trees efficiently,
preferably with positions, it would be possible to extend el-search to
the supported languages.

Semantic had something like that, but AFAICT it has been obsoleted.


Regards,

Michael.

Re: [el-search] How to search string excluding docstring?

[Emanuel Berg]

Michael Heerdegen wrote:

>> Tell us again why one can't just use grep to
>> search Elisp? :)
>
> There are tasks where grep isn't enough, like
> "find all uses of cl stuff where I have
> forgotten the cl- prefix", "find all
> definitions that have no docstring", "find
> all function "foo" calls that make use the
> third optional arg", etc.
>
> It can also do list-level query-replace.
> Some examples: "I've changed my function
> "bar" so that the second and third argument
> are now transposed. Let's adopt every use of
> it and transpose the expressions at the
> second and third argument position." Or
> "automatically transform any nested if's into
> an equivalent `cond'", or "find any `if'
> expressions that have no `else' and make them
> a "when", and such stuff.

Your software can do that? Impressive! Still,
I'd do that manually. But that's just me.
OTOH I have written tons of Elisp "for the sake
of Elisp". So I enjoy that. Looking at it from
a spaceship tho one might consider it a case of
built-up creativity/frustration. There aren't
any real challanges left [1] so people write
code for the sake or writing code, with the
purpose of writing even more code later.
It gives pleasure and is interesting but I, who
almost exclusively did just that, know there is
even more pleasure to do stuff that has a less
recursive application surface... Like it is
more interesting to write very advanced Elisp.
But to build a ramp so people can get their
bikes up onto a deck without bumping up
a stairways. Building the ramp might be trivial
in comparison but nonetheless every time you
see or use it it'll make you feel happy. So it
is complicated.

[1] Of course there are real challenges left!
    They are just harder to find. It is an open
    question if all fiddling makes you more
    perceptive of seeing them, *or* if it makes
    you less so as you get stuck in all the
    fiddling details.

Some random thoughts on a general level, not to
be taken as an assault on anyone's project or
style in particular...


          pink elephants and lemonade
        dear Jessie hears the laughter
         running thru the love parade


-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: [el-search] How to search string excluding docstring?

[Emanuel Berg]

t wrote:

>> Pattern match, like they have for arguments
>> in Erlang and SML?
>
> Pattern match like pcase, I guess.

Isn't that the same? AFAIK there isn't any
native pattern matching i Lisp, but one can do
it with macros...

I don't like it personally because it makes
writing code neurotic keeping track of all
those cases, rather than just examine what the
value is and act accordingly...

Whenever I use `pcase' it isn't really pattern
matching in my eyes, i.e. in the Erlang/SML
sense, more like a way of doing case with
strings. I guess it could be used in other ways
as well. Even tho technically it is the
same thing.

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: [el-search] How to search string excluding docstring?

[tomas]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Mon, Dec 25, 2017 at 07:11:19PM +0100, Emanuel Berg wrote:
> t wrote:
> 
> >> Pattern match, like they have for arguments
> >> in Erlang and SML?
> >
> > Pattern match like pcase, I guess.
> 
> Isn't that the same?

No. Regexps are built to match regular languages. Pcase is
built to match S-expressions, which are beyond regular.
Well-nested parentheses are more than a regular language.

Start with the Chomsky hierarchy[1], and delve as deep as
you like :-)

> AFAIK there isn't any
> native pattern matching i Lisp, but one can do
> it with macros...

whatever you understand by "native pattern matching".

> I don't like it personally because it makes
> writing code neurotic keeping track of all
> those cases, rather than just examine what the
> value is and act accordingly...

It's not a psychological question, but rather a mathematical
one: there are things you can't do with regexps.

> Whenever I use `pcase' it isn't really pattern
> matching in my eyes, i.e. in the Erlang/SML
> sense, more like a way of doing case with
> strings.

There lies your error: you are matching S-expressions, not
strings. The "case" part isn't the exciting one :-)

> I guess it could be used in other ways
> as well. Even tho technically it is the
> same thing.

Hmmm.

Cheers
[1] https://en.wikipedia.org/wiki/Chomsky_hierarchy

- -- tomás
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlpBTgwACgkQBcgs9XrR2kbuAwCdGeCGed8FArdnqKkpQ2sLLnym
SUsAn2Kd5MuqW6LzPW6ZZP0RzXQNlEoA
=F3k5
-----END PGP SIGNATURE-----

Re: [el-search] How to search string excluding docstring?

[Emanuel Berg]

t wrote:

>>>> Pattern match, like they have for
>>>> arguments in Erlang and SML?
>>>
>>> Pattern match like pcase, I guess.
>>
>> Isn't that the same?
>
> No. Regexps are built to match regular
> languages. Pcase is built to match
> S-expressions, which are beyond regular.
> Well-nested parentheses are more than
> a regular language.

Regexps, you mean that is what they use to
implement pattern match in Erlang and SML?! Or
why did you bring it up?

I lost all my Erlang and SML (literally) but
I found some Haskell, the most neurotic of all
languages, which rely the most on pattern
matching virtually all the time - here is what
I mean:

    data Prop = DoNotCare | Pp Char
    instance Eq Prop where
      DoNotCare == DoNotCare = True
      Pp a      == Pp b      = a == b
      _         == _         = False
      a         /= b         = not (a == b)

Is this really what you mean? Surly that isn't
implemented as mere regexps?!

From: http://user.it.uu.se/~embe8573/candelim/ce.hs

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: [el-search] How to search string excluding docstring?

[Emanuel Berg]

t wrote:

> Start with the Chomsky hierarchy, and delve
> as deep as you like :-)

I had that on Automata Theory which I completed
2010-01-13. That was only 7y 11m 12d (total
2903d) ago which is why I remember it so well.

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: [el-search] How to search string excluding docstring?

[tomas]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Mon, Dec 25, 2017 at 08:36:41PM +0100, Emanuel Berg wrote:
> t wrote:
> 
> >>>> Pattern match, like they have for
> >>>> arguments in Erlang and SML?
> >>>
> >>> Pattern match like pcase, I guess.

[...]

> I lost all my Erlang and SML (literally) but
> I found some Haskell, the most neurotic of all
> languages, which rely the most on pattern
> matching virtually all the time - here is what

[...]

> Is this really what you mean? Surly that isn't
> implemented as mere regexps?!

I'd be very surprised if it were.

Regards
- -- t
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlpBa6MACgkQBcgs9XrR2ka/kgCfXr+GmyO5FrZhhUWkEmVveFL9
ff4AnRiVSFq7GPuBSiAbQSpm4yd3yBWU
=4ZPe
-----END PGP SIGNATURE-----

Re: [el-search] How to search string excluding docstring?

[Emanuel Berg]

t wrote:

>> Is this really what you mean? Surly that
>> isn't implemented as mere regexps?!
>
> I'd be very surprised if it were.

So why did you bring it up to begin with?

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: [el-search] How to search string excluding docstring?

[Emanuel Berg]

First check if it is a string at all, then

    (lisp-font-lock-syntactic-face-function (syntax-ppss))

If you don't want to use font lock, try reuse
code from that function.

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: [el-search] How to search string excluding docstring?

[Emanuel Berg]

YT wrote:

>>> Is this really what you mean? Surly that
>>> isn't implemented as mere regexps?!
>>
>> I'd be very surprised if it were.
>
> So why did you bring it up to begin with?

OK, so now I understand how regexps entered
this discussion! "You can't use grep with
regexps", because it isn't a Lisp parser so it
can't answer the kind of questions that require
you to crunch it down token by token!"

And that's true! Regexps are sometimes
underestimated. Because they cannot do
everything, it doesn't mean they cannot do
a lot, still. But it will never be a perfect
fallout and the result may also be confusing as
semantically the same Lisp, and the same
regexp, can still yield different results.

So you can use grep to search your Elisp, but
not to query your Elisp and expect 100%
consistent/correct/complete answers. If this is
the goal, even to use font lock, which probably
itself uses some half baked method to find out,
to use font lock would be a clever hack
at best.

The reason you can still use grep or font lock
or any other such method is that in the world
of editing code, these are just tools to do
things. An imperfect tool can still help you do
things. And a perfect tool, which is 100 times
more advanced than the imperfect tool, will not
be 100 times more helpful to you.

If we then switch worlds from the
practical/editing to the abstract/theoretical,
to parse the code, or eval it like Lisp does,
this is already capable of telling if something
is a docstring or not. Just do C-h
f forward-char RET !

However here there is a collision between the
two worlds. Because where is point?
Over which element to begin with, when the
query is executed? If one knows that, one could
simply get the defun as a list (which it
already is) and ask, "is the 4th element
a string?". If it is, and if point's at it in
the buffer, then yes it is a docstring.

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: [el-search] How to search string excluding docstring?

[Jean-Christophe Helary]

> On Dec 26, 2017, at 22:34, Emanuel Berg <moasen@zoho.com> wrote:
> 
> If one knows that, one could
> simply get the defun as a list (which it
> already is) and ask, "is the 4th element
> a string?".

Ok, so we're back to my suggestion. docstring-p can be defined as the string coming after the defun statement.

Jean-Christophe Helary
-----------------------------------------------
@brandelune http://mac4translators.blogspot.com

Re: [el-search] How to search string excluding docstring?

[Emanuel Berg]

Jean-Christophe Helary wrote:

> Ok, so we're back to my suggestion.
> docstring-p can be defined as the string
> coming after the defun statement.

Technically, `defun' is a macro.

Talking broadly, this

    (do-something-with me and you)

is a function or a form.

But yes :)

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: [el-search] How to search string excluding docstring?

[Chunyang Xu]

Emanuel Berg writes:

> However here there is a collision between the
> two worlds. Because where is point?
> Over which element to begin with, when the
> query is executed? If one knows that, one could
> simply get the defun as a list (which it
> already is) and ask, "is the 4th element
> a string?". If it is, and if point's at it in
> the buffer, then yes it is a docstring.

I think point is at the beginning of the string, the following assumes
this to work. I compare point with the point of docstring (if any) to
see if it is a docstring.

#+begin_src emacs-lisp
;; FIXME: Update this list every time el-search starts. How?
(defvar el-search--symbols-with-doc-string
  (let (symbols)
    (mapatoms
     (lambda (sym)
       (and (fboundp sym)
            (get sym 'doc-string-elt)
            (push sym symbols))))
    symbols)
  "A list of symbols which support doc-string.")

(defun el-search--doc-string-p ()
  "Return t if point is at docstring."
  (pcase (save-excursion
           (backward-up-list)
           (read (current-buffer)))
    (`(,(and symbol (guard (memq symbol el-search--symbols-with-doc-string)))
       ,_ . ,_)
     (let ((op (point)))
       (save-excursion
         (backward-up-list)
         (forward-char)
         (ignore-errors
           (forward-sexp (1+ (get symbol 'doc-string-elt)))
           (backward-sexp)
           (= op (point))))))))

(el-search-defpattern doc-string (&rest regexps)
  "Match any documentation string that is matched by all REGEXPS."
  `(and (string ,@regexps) (guard (el-search--doc-string-p))))

(el-search-defpattern s (&rest regexps)
  "Match any string (excluding doc string) that is matched by all REGEXPS"
  `(and (string ,@regexps) (guard (not (el-search--doc-string-p)))))
#+end_src

Re: [el-search] How to search string excluding docstring?

[Emanuel Berg]

Chunyang Xu wrote:

> I think point is at the beginning of the
> string, the following assumes this to work.

Perhaps that can be improved easily.

First, check if point is at a string.

If it is, search backwards to the initial
string delimiter (and make sure it isn't
quoted).

Then, check if that string is in
a docstring position.

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: [el-search] How to search string excluding docstring?

[Michael Heerdegen]

Chunyang Xu <mail@xuchunyang.me> writes:

> ;; FIXME: Update this list every time el-search starts. How?

For efficiency reasons I use in el-search a primitive type of caches
that know when they need to refresh themselves.  In this situation, this
would look like

#+begin_src emacs-lisp
;; -*- lexical-binding: t -*-

(defun el-search--documented-function-p (name)
  (gethash name (el-search--get-documented-function-table)))

(let ((helper (el-search-with-short-term-memory
               (lambda (_load-history)
                 (message "Recomputing table of documented functions...")
                 (let ((table (make-hash-table)))
                   (mapatoms
                    (lambda (sym)
                      (and (fboundp sym)
                           (get sym 'doc-string-elt)
                           (puthash sym t table))))
                   table)))))
  (defun el-search--get-documented-function-table ()
    (funcall helper load-history)))
#+end_src

The cache is a hash-table (faster lookup) that refreshes when queried
and `load-history' changed.  This is not optimal when you evaluate
definitions by hand, of course.  I could provide a
`el-search-last-search-start-time' for that purpose, or a
`el-search-start-hook' - though, I don't really want hooks to be used to
implement pattern matching.


FWIW, if you happen to want this (hack!), with a similar technique you
can let el-search load searched buffers or files by side effect:

#+begin_src emacs-lisp
(el-search-defpattern load ()
  "Match anything, load current buffer or file as side effect.
Any buffer or file is loaded at most once.

This is useful only in rare cases.  Use with caution!!!"
  (declare (heuristic-matcher #'el-search--load-matcher))
  (let ((load-matcher (el-search--load-matcher)))
    `(guard (funcall ',load-matcher (current-buffer) nil))))

(defun el-search--load-matcher ()
  (let ((test (el-search-with-short-term-memory
               (lambda (file-name-or-buffer)
                 (when-let ((file (if (bufferp file-name-or-buffer)
                                      (buffer-file-name file-name-or-buffer)
                                    file-name-or-buffer)))
                   (with-demoted-errors "Error: %S" (load file)))))))
    (lambda (file-name-or-buffer _) (funcall test file-name-or-buffer))))
#+end_src


HTH,

Michael.