use Elisp to improve your Elisp - some code issues

use Elisp to improve your Elisp - some code issues

[Emanuel Berg]

I just wrote some Elisp which can be used on a set of
files to identify for example the construct

    (if a a b)

if you want to replace those for

    (or a b)

See the comments for the issues!

Issue one is how to best create a temporary buffer to
display the results.

Issue two is to not kill buffers that were already
open at invocation - I can solve that by checking if
there is such a buffer, but I suspect there is
a better way to do these kind of things all in the
background, rather than the `find-file' and then
conditionally `kill-buffer' combo.

Third (minor) issue is the annoying message that
`downcase' does. Isn't there a (shut-up (do-stuff))?

Other comments also appreciated, as always.

;; This file: http://user.it.uu.se/~embe8573/conf/emacs-init/search-regexp-in-files.el

(defun files-as-list (file-regexp)
  (split-string
   (with-temp-buffer
     (call-process-shell-command
      (format "ls %s" file-regexp) nil t) ; no INFILE, temp BUFFER
     (buffer-substring (point-min) (point-max)) )))

(defun search-regexp-in-files (file-regexp regexp)
  (let ((paths       (files-as-list file-regexp))
        (regexp-hits "regexp-hits") ; unlikely, but if there is another such buffer
        (hits        nil))
    (with-current-buffer regexp-hits (erase-buffer)) ; then we can't have this
    (dolist (p paths)
      (let ((buffer (find-file p)))
        (with-current-buffer buffer
          (goto-char (point-min))
          (while (re-search-forward regexp nil t) ; no BOUND, NOERROR
            (setq hits t)
            (let ((hit-line (downcase (what-line))))
              (with-current-buffer regexp-hits
                (insert (format "file: %s (%s)\n" p hit-line)))))
          (kill-buffer buffer) ))) ; what if the buffer was open already?
                                   ; we only want to kill buffers that we opened
    (if hits (switch-to-buffer regexp-hits)
      (message "No hits!") )))

;; use this to test
(when nil

  ;; find "kill" - should be some hits even for pacifists
  (search-regexp-in-files "~/.emacs.d/emacs-init/*.el" "kill")

  ;; find the construct (if a a b) if you want to replace it with (or a b)
  ;; if it works, when applied to this file, it should find the example above!
  (search-regexp-in-files (buffer-file-name)
   "([[:space:]\n]*if[[:space:]\n]+\\(.*\\)[[:space:]\n]+\\1[[:space:]\n]+\\(.*\\))"
   )
  )

(provide 'search-regexp-in-files)

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: use Elisp to improve your Elisp - some code issues

[Marcin Borkowski]

On 2015-07-31, at 02:22, Emanuel Berg <embe8573@student.uu.se> wrote:

> I just wrote some Elisp which can be used on a set of
> files to identify for example the construct
>
>     (if a a b)
>
> if you want to replace those for
>
>     (or a b)

I like the idea!  Even though basically the % m Q combo in Dired already
does this (well, more or less).  Or multi-occur.  Or
multi-occur-in-matching-buffers.  (These two operate on buffers, not
files, OTOH.)  (Projectile has similar things, too, I guess, though
I don't se it.  And Icicles takes that idea to the next level.)

> See the comments for the issues!

Well, I also glanced at the code itself;-).

> Issue one is how to best create a temporary buffer to
> display the results.

`pop-to-buffer'?  Also, why not name the buffer with earmuffs?

> Issue two is to not kill buffers that were already
> open at invocation - I can solve that by checking if
> there is such a buffer, but I suspect there is
> a better way to do these kind of things all in the
> background, rather than the `find-file' and then
> conditionally `kill-buffer' combo.

I wouldn't use find-file in Lisp programs.  Related:
http://emacs.stackexchange.com/questions/2868/whats-wrong-with-find-file-noselect
(the solution given there would also solve your problem, I guess - the
downside being that if the file is already visited, you get "data
duplication").

I'd write a macro (with-file-visited file &rest body) which checks
whether `file' is visited by some buffer or not.  If yes, it would just
use `with-current-buffer', and if not, it would use the
`with-temp-buffer' / `insert-file-contents' combo.  This way, you would
both be efficient and have a decent level of abstraction at the same
time.

> Third (minor) issue is the annoying message that
> `downcase' does. Isn't there a (shut-up (do-stuff))?

What about `line-number-at-pos' instead of `what-line'?

Also,

(require 'cl)
(cl-flet ((message (&rest args))) ...).

> Other comments also appreciated, as always.

Why use regexen for that?  It is not, ekhm, the best solution.  ;-P
Shouldn't you use some kind of pattern-matching/destructuring thingy?

In fact, I wanted to write a similar thing, and that was exactly when
I decided that I absolutely *have* to understand precisely how the
backtick works.  (Now I think I do, but not before I implemented
a subset of Elisp in Elisp...)

(I haven't written that thing yet, but it's somewhere on the agenda.)

Best,

-- 
Marcin Borkowski
http://octd.wmi.amu.edu.pl/en/Marcin_Borkowski
Faculty of Mathematics and Computer Science
Adam Mickiewicz University

Re: use Elisp to improve your Elisp - some code issues

[Pascal J. Bourguignon]

Emanuel Berg <embe8573@student.uu.se> writes:

> I just wrote some Elisp which can be used on a set of
> files to identify for example the construct
>
>     (if a a b)
>
> if you want to replace those for
>
>     (or a b)
>
> See the comments for the issues!
>
> Issue one is how to best create a temporary buffer to
> display the results.

It's not so much a temporary buffer than a buffer that is not backed by
a file, that you want.

A temporary buffer, created by with-temporary-buffer would be deleted
right away, and not visible to the user.

You want your buffer to be visible by the user.
The convention for this type of buffer is to name it with stars around
"*Results*".

If you want such a buffer but not visible to the user, add a prefix
space:

" *PrivateResults*"

You can see the current "invisible" buffer with C-x b SPC TAB
or all the buffers with (buffer-list)


> Issue two is to not kill buffers that were already
> open at invocation - I can solve that by checking if
> there is such a buffer, but I suspect there is
> a better way to do these kind of things all in the
> background, rather than the `find-file' and then
> conditionally `kill-buffer' combo.

I don't think there's another way.  

In my opinion, it's not too important a feature; in my own with-file
macro (used by with-files), I didn't check for pre-existing buffers.


> Third (minor) issue is the annoying message that
> `downcase' does. Isn't there a (shut-up (do-stuff))?

I never noted any message from downcase; what do you get?


Perhaps you'd want to use:

(defun what-line-message ()
  "Return a formated string containing the current buffer line number and narrowed line number of point."
  (let ((start (point-min))
        (n (line-number-at-pos)))
    (if (= start 1)
        (format "Line %d" n)
        (save-excursion
         (save-restriction
          (widen)
          (format "line %d (narrowed line %d)"
                  (+ n (line-number-at-pos start) -1) n))))))

(what-line-message)
--> "line 79 (narrowed line 72)"

instead of what-line?

You could send a patch to emacs to replace what-line with:

(defun what-line ()
  (interactive)
  (message "%s" (what-line-message)))

and the what-line-message function.  In anycase, it is a basic precept
to never mix I/O with computing in a single function.


-- 
__Pascal Bourguignon__                 http://www.informatimago.com/
“The factory of the future will have only two employees, a man and a
dog. The man will be there to feed the dog. The dog will be there to
keep the man from touching the equipment.” -- Carl Bass CEO Autodesk

Re: use Elisp to improve your Elisp - some code issues

[Stefan Monnier]

> (require 'cl)
> (cl-flet ((message (&rest args))) ...).

`cl' does not provide cl-flet.  `cl-lib' does.


        Stefan

Re: use Elisp to improve your Elisp - some code issues

[Navy Cheng]

This is my code after your discuss. 

(defun search-regexp-in-files (wildcards regexp)
    "Search for regular expression REGEXP in files whose name matches WILDCARDS.                                                       
The results will be print to buffer *PrivateResults*."
    (let ((result (or (get-buffer "*PrivateResults*")       ; There will be only one result buffer  
		      (generate-new-buffer "*PrivateResults*")))
	  (file-list (file-expand-wildcards wildcards))     ; All matched files
	  (buffer-to-kill nil))                            ; Record the buffer to be killed
      
      (if (eq file-list nil)
	  (with-current-buffer result
	    (insert "Wrong file name!"))
	(dolist (file-path file-list)
	  (setq file (file-name-nondirectory file-path))    ; Get the buffer name
	  (let ((buffer (or (progn                          ; File buffer is exist
			      (setq buffer-to-kill nil)
			      (get-buffer file))
			    (progn                          ; Create buffer for a file
			      (setq buffer-to-kill file)
			      (find-file-noselect file)))))
	    (with-current-buffer buffer
	      (goto-char (point-min))
	      (while (re-search-forward regexp nil t)       ; no BOUND, NOERROR
		(let ((hit-line (downcase (what-line))))
		  (with-current-buffer result
		    (insert (format "file: %s (%s)\n" buffer hit-line))))))
	    (with-current-buffer result
	      (if (eq (buffer-size result) 0)
		  (insert "No hits!"))))
	  (if buffer-to-kill
	      (kill-buffer buffer-to-kill))))               ; Kill the buffer-to-kill
      
      (switch-to-buffer result)))

On Fri, Jul 31, 2015 at 04:30:20AM +0200, Marcin Borkowski wrote:
> 
> On 2015-07-31, at 02:22, Emanuel Berg <embe8573@student.uu.se> wrote:
> 
> > I just wrote some Elisp which can be used on a set of
> > files to identify for example the construct
> >
> >     (if a a b)
> >
> > if you want to replace those for
> >

Re: use Elisp to improve your Elisp - some code issues

[Sam Halliday]

On Friday, 31 July 2015 01:24:24 UTC+1, Emanuel Berg  wrote:
> I just wrote some Elisp which can be used on a set of
> files to identify for example the construct
> 
>     (if a a b)
> 
> if you want to replace those for
> 
>     (or a b)
> 
> See the comments for the issues!

This is really great! We have plans to do something similar in ENSIME (for Scala, and soon Java, development).

The Scala/Java support is of no relevance to this group, so I shall just say that we have an external process that we communicate with and it can provide async suggestions of code that can be changed.

It sounds like what you're doing here and we aim to achieve (albeit it a long burn) would benefit from a common interface of "code suggestion" --- both a format for the changes (universal diff?) and the method to show and accept the changes.

FYI we're tracking this on our github issue tracker https://github.com/ensime/ensime-server/issues/848 so please keep us updated if you come up with anything reusable. This issue is for the server side support, but the data that is returned is going to have to be driven by the text editor requirements.


Best regards,
Sam

Re: use Elisp to improve your Elisp - some code issues

[Emanuel Berg]

"Pascal J. Bourguignon" <pjb@informatimago.com>
writes:

> It's not so much a temporary buffer than a buffer
> that is not backed by a file, that you want.

Indeed, good clarification.

> You want your buffer to be visible by the user.
> The convention for this type of buffer is to name it
> with stars around "*Results*".

OK, done.

> If you want such a buffer but not visible to the
> user, add a prefix space:
>
> " *PrivateResults*"
>
> You can see the current "invisible" buffer with C-x
> b SPC TAB or all the buffers with (buffer-list)

Interesting, but of course the buffer should be
accessible the ordinary way as you might fix one
occurence, get back to the hit list, examine the next
hit, and so on.

>> Issue two is to not kill buffers that were already
>> open at invocation - I can solve that by checking
>> if there is such a buffer, but I suspect there is
>> a better way to do these kind of things all in the
>> background, rather than the `find-file' and then
>> conditionally `kill-buffer' combo.
>
> I don't think there's another way.
>
> In my opinion, it's not too important a feature; in
> my own with-file macro (used by with-files),
> I didn't check for pre-existing buffers.

Here it is very important! Because say that you have
a couple of Elisp files open. Then it strikes you you
can do (or a b) instead of (if a a b) while working on
one of them. So you think, did I do that in any of my
other Elisp files? You apply the tool - boom, all
buffers killed! See the code (last) for how the
situation can be solved (?).

> I never noted any message from downcase; what do
> you get?

Messages :)

In the form of: "Line X". But it wasn't `downcase' but
`what-line'!

> In anycase, it is a basic precept to never mix I/O
> with computing in a single function.

Indeed, and I want data, not I/O. `line-number-at-pos'
did it as mentioned by anther poster.

The code:

;; This file: http://user.it.uu.se/~embe8573/conf/emacs-init/search-regexp-in-files.el

(defun files-as-list (file-regexp)
  (split-string
   (with-temp-buffer
     (call-process-shell-command
      (format "ls %s" file-regexp) nil t) ; no INFILE, temp BUFFER
     (buffer-substring (point-min) (point-max)) )))

(require 'cl)
(defalias 'cl-set-xor 'cl-set-exclusive-or)

(defun search-regexp-in-files (file-regexp regexp)
  (let ((paths       (files-as-list file-regexp))
        (regexp-hits "*regexp-hits*")
        (hits        nil) )
    (get-buffer-create regexp-hits)
    (let ((buffers     (buffer-list))) ; get list to see if we opened the file -
      (with-current-buffer regexp-hits (erase-buffer))
      (dolist (p paths)
        (let ((buffer     (find-file p)) ; here -
              (kill-later (cl-set-xor buffers (buffer-list))) ) ; by comparing!
          (with-current-buffer buffer
            (goto-char (point-min))
            (while (re-search-forward regexp nil t) ; no BOUND, NOERROR
              (setq hits t)
              (let ((hit-line (line-number-at-pos)))
                (with-current-buffer regexp-hits
                  (insert (format "%s (%s)\n" p hit-line)))))
            (when kill-later (kill-buffer buffer) ))))
      (if hits
          (progn
            (switch-to-buffer regexp-hits)
            (set-buffer-modified-p nil)
            (goto-char (point-min)) )
        (message "No hits!") ))))

;; use this to test
(when nil

  ;; find "kill" - should be some hits even for pacifists
  (search-regexp-in-files "~/.emacs.d/emacs-init/*.el" "kill")

  ;; find the construct (if a a b) if you want to replace it with (or a b)
  ;; if it works, when applied to this file, it should find the example above!
  (search-regexp-in-files (buffer-file-name)
   "([[:space:]\n]*if[[:space:]\n]+\\(.*\\)[[:space:]\n]+\\1[[:space:]\n]+\\(.*\\))"
   )
  )

(provide 'search-regexp-in-files)

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: use Elisp to improve your Elisp - some code issues

[Emanuel Berg]

Sam Halliday <sam.halliday@gmail.com> writes:

> This is really great! We have plans to do something
> similar in ENSIME (for Scala, and soon Java,
> development).

Well, thank you :) The idea is great, however it is
not a new one since it is basically find + egrep, with
find here being a zsh pattern matching ls (I'm unsure
bash has "**/*", so that example may not work work tho
many others will), and egrep being
`re-search-forward'. But yeah, it is a nice little
thing :)

> The Scala/Java support is of no relevance to this
> group, so I shall just say that we have an external
> process that we communicate with and it can provide
> async suggestions of code that can be changed.
>
> It sounds like what you're doing here and we aim to
> achieve (albeit it a long burn) would benefit from
> a common interface of "code suggestion" --- both
> a format for the changes (universal diff?) and the
> method to show and accept the changes.

More or less. The tool (code) just searches for
regexps in the files whose names match another regexp.
So it is just a tool to find things in files.
However if you express style things you don't like as
regexps, it can be used as to improve the code (if you
change them as well). See the examples last in the
file - if I come up with more, I'll add them to the
same file.

One example would be what we have mentioned a couple
of times by now and that is

    (if a b)
    (if (not a) b)

which should be

    (when a b)
    (unless a b)

Anyone has an regexp for that?

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: use Elisp to improve your Elisp - some code issues

[Marcin Borkowski]

On 2015-08-01, at 06:20, Emanuel Berg <embe8573@student.uu.se> wrote:

> One example would be what we have mentioned a couple
> of times by now and that is
>
>     (if a b)
>     (if (not a) b)
>
> which should be
>
>     (when a b)
>     (unless a b)
>
> Anyone has an regexp for that?

Well, since s-expressions and XML are (more or less) isomorphic, and
parsing XML and HTML is (more or less) the same thing, I guess the Most
Famous StackOverflow Answer applies. ;-P

http://stackoverflow.com/a/1732454/1181665

Best,

-- 
Marcin Borkowski
http://octd.wmi.amu.edu.pl/en/Marcin_Borkowski
Faculty of Mathematics and Computer Science
Adam Mickiewicz University

Re: use Elisp to improve your Elisp - some code issues

[Pascal J. Bourguignon]

Emanuel Berg <embe8573@student.uu.se> writes:

>   ;; find the construct (if a a b) if you want to replace it with (or a b)
>   ;; if it works, when applied to this file, it should find the example above!
>   (search-regexp-in-files (buffer-file-name)
>    "([[:space:]\n]*if[[:space:]\n]+\\(.*\\)[[:space:]\n]+\\1[[:space:]\n]+\\(.*\\))"

Perhaps you have notied some discrepancy here.
Check: https://en.wikipedia.org/wiki/Chomsky_hierarchy

You may use read, or forward-sexp / backward-sexp and down-lisp /
up-list, to walk the sexps of the buffer in such a way that now you can
check for source patterns instead of text patterns.

Not that in this case you may want to use it, but it might still be
better to write:

    (map-sexp-in-files
       (lambda (form)
         (cond
           ((atom form) form)
           ((and (eql (first form) 'if)
                 (eql (second form) (third form))) ; eql ensures we don't
                 ;;  substitute expressions that may have side effects.
             `(or ,(second form) ,@(cdddr form)))
           (t form)))
       (list  (buffer-file-name)))

or:

    (require 'pjb-pmatch)
    (map-sexp-in-files
       (lambda (form)
         (match-case form
            ((if (!v a) (!v a) (!! (!x e)))
              `(or ,a ,e))
            (otherwise form)))
        (list (buffer-file-name)))


-- 
__Pascal Bourguignon__                 http://www.informatimago.com/
“The factory of the future will have only two employees, a man and a
dog. The man will be there to feed the dog. The dog will be there to
keep the man from touching the equipment.” -- Carl Bass CEO Autodesk

Re: use Elisp to improve your Elisp - some code issues

[Pascal J. Bourguignon]

Marcin Borkowski <mbork@mbork.pl> writes:

> On 2015-08-01, at 06:20, Emanuel Berg <embe8573@student.uu.se> wrote:
>
>> One example would be what we have mentioned a couple
>> of times by now and that is
>>
>>     (if a b)
>>     (if (not a) b)
>>
>> which should be
>>
>>     (when a b)
>>     (unless a b)
>>
>> Anyone has an regexp for that?
>
> Well, since s-expressions and XML are (more or less) isomorphic, and
> parsing XML and HTML is (more or less) the same thing, I guess the Most
> Famous StackOverflow Answer applies. ;-P
>
> http://stackoverflow.com/a/1732454/1181665

Indeed.  This is  Compiler 101.  If you didn't have that lecture at your
university, you should request a refund, because you've been swindled.

-- 
__Pascal Bourguignon__                 http://www.informatimago.com/
“The factory of the future will have only two employees, a man and a
dog. The man will be there to feed the dog. The dog will be there to
keep the man from touching the equipment.” -- Carl Bass CEO Autodesk

Re: use Elisp to improve your Elisp - some code issues

[Emanuel Berg]

"Pascal J. Bourguignon" <pjb@informatimago.com>
writes:

> Perhaps you have notied some discrepancy here.
> Check:
> https://en.wikipedia.org/wiki/Chomsky_hierarchy

It would be more illustrative if you provide
a counterexample of the same construct which cannot be
caught by that or another regexp. It will be
interesting to see if that example is anything I would
put in my code, ever.

But it works. That regexp did identify several such
occurrences in my Elisp, which I have changed since.
It is yet to be found, if it failed to find any - if
that indeed happened. Either case, the code still
got better.

Perhaps the discrepancy gets smaller the better you
know your own style? Gnothi seauton - always
a good thing.

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: use Elisp to improve your Elisp - some code issues

[Emanuel Berg]

"Pascal J. Bourguignon" <pjb@informatimago.com>
writes:

> Indeed. This is Compiler 101. If you didn't have
> that lecture at your university, you should request
> a refund, because you've been swindled.

This is the difference between blue-collar engineering
and white-collar science: All the while the
white-collar people are letting themself be
incapacitated by absolutist ideas and ideals which
teaches them what they can't do, the blue-collar
people are pushing the envelope every day doing new
things which little by little improve their game.

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: use Elisp to improve your Elisp - some code issues

[Marcin Borkowski]

On 2015-08-01, at 14:48, Emanuel Berg <embe8573@student.uu.se> wrote:

> incapacitated by absolutist ideas and ideals which
> teaches them what they can't do, [...]

Like, "keyboard macros are wrong, Lisp is good"?

;-)

-- 
Marcin Borkowski
http://octd.wmi.amu.edu.pl/en/Marcin_Borkowski
Faculty of Mathematics and Computer Science
Adam Mickiewicz University

Re: use Elisp to improve your Elisp - some code issues

[Emanuel Berg]

Now the code is even better. Now you can go to a hit
with one keystroke - RET - and iterate with the
favorites "i" and "k".

;; This file: http://user.it.uu.se/~embe8573/conf/emacs-init/search-regexp-in-files.el

(require 'cl)
(defalias 'cl-set-xor 'cl-set-exclusive-or)

(defun files-as-list (file-regexp)
  (split-string
   (with-temp-buffer
     (call-process-shell-command
      (format "ls %s" file-regexp) nil t) ; no INFILE, temp BUFFER
     (buffer-substring (point-min) (point-max)) )))

(defun find-file-goto-line (file line)
  (interactive)
  (find-file file)
  (goto-char (point-min))
  (forward-line (1- line)) )

(defun regexp-hits-find-hit ()
  (interactive)
  (beginning-of-line)
  (let ((file (thing-at-point 'filename nil))) ; NO-PROPERTIES
    (end-of-line)
    (search-backward "(")
    (forward-char 1)
    (let ((line (thing-at-point 'number nil)))
      (find-file-goto-line file line) )))

(defun set-regexp-hits-keys ()
  (local-set-key "\r" 'regexp-hits-find-hit)
  (local-set-key "i"  'previous-line)
  (local-set-key "k"  'forward-line) )

(defun search-regexp-in-files (file-regexp regexp)
  (let ((paths       (files-as-list file-regexp))
        (regexp-hits "*regexp-hits*")
        (hits        nil) )
    (get-buffer-create regexp-hits)
    (let ((buffers     (buffer-list))) ; get list to see if we opened the file -
      (with-current-buffer regexp-hits
        (erase-buffer)
        (set-regexp-hits-keys) )
      (dolist (p paths)
        (let ((buffer     (find-file p)) ; here -
              (kill-later (cl-set-xor buffers (buffer-list))) ) ; by comparing!
          (with-current-buffer buffer
            (goto-char (point-min))
            (while (re-search-forward regexp nil t) ; no BOUND, NOERROR
              (setq hits t)
              (let ((hit-line (line-number-at-pos)))
                (with-current-buffer regexp-hits
                  (insert (format "%s (%s)\n" p hit-line)))))
            (when kill-later (kill-buffer buffer) ))))
      (if hits
          (progn
            (switch-to-buffer regexp-hits)
            (set-buffer-modified-p nil)
            (goto-char (point-min)) )
        (message "No hits!") ))))

;; use this to test
(when nil

  ;; find "kill" - should be some hits even for pacifists
  (search-regexp-in-files "~/.emacs.d/emacs-init/*.el" "kill")

  ;; find the construct (if a a b) if you want to replace it with (or a b)
  ;; if it works, when applied to this file, it should find the example above!
  (search-regexp-in-files (buffer-file-name)
   "([[:space:]\n]*if[[:space:]\n]+\\(.*\\)[[:space:]\n]+\\1[[:space:]\n]+\\(.*\\))"
   )
  )

(provide 'search-regexp-in-files)

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: use Elisp to improve your Elisp - some code issues

[Emanuel Berg]

Marcin Borkowski <mbork@mbork.pl> writes:

>> incapacitated by absolutist ideas and ideals which
>> teaches them what they can't do, [...]
>
> Like, "keyboard macros are wrong, Lisp is good"?

How did those two entities ever end up in the same,
short sentence?

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: use Elisp to improve your Elisp - some code issues

[Pascal J. Bourguignon]

Emanuel Berg <embe8573@student.uu.se> writes:

> "Pascal J. Bourguignon" <pjb@informatimago.com>
> writes:
>
>> Perhaps you have notied some discrepancy here.
>> Check:
>> https://en.wikipedia.org/wiki/Chomsky_hierarchy
>
> It would be more illustrative if you provide
> a counterexample of the same construct which cannot be
> caught by that or another regexp. It will be
> interesting to see if that example is anything I would
> put in my code, ever.

It's easy.  Try to simplify (- x x) for any sexp x, (assuming x doesn't
side effect.  Just something like (+ 2 y) or (+ (* a x x) (* b x) c).



-- 
__Pascal Bourguignon__                 http://www.informatimago.com/
“The factory of the future will have only two employees, a man and a
dog. The man will be there to feed the dog. The dog will be there to
keep the man from touching the equipment.” -- Carl Bass CEO Autodesk

Re: use Elisp to improve your Elisp - some code issues

[Michael Heerdegen]

"Pascal J. Bourguignon" <pjb@informatimago.com> writes:

> You may use read, or forward-sexp / backward-sexp and down-lisp /
> up-list, to walk the sexps of the buffer in such a way that now you can
> check for source patterns instead of text patterns.

I had the same idea some time ago.  Of course such an approach is then
limited to Elisp.

Anyway, by coincidence I've been implementing exactly that the last
days, using `pcase' patterns (see emacs-dev thread "Search Elisp buffers
with pcase patterns" - comments very welcome!).


Michael.

Re: use Elisp to improve your Elisp - some code issues

[Emanuel Berg]

"Pascal J. Bourguignon" <pjb@informatimago.com>
writes:

>> It would be more illustrative if you provide
>> a counterexample of the same construct which cannot
>> be caught by that or another regexp. It will be
>> interesting to see if that example is anything
>> I would put in my code, ever.
>
> It's easy. Try to simplify (- x x) for any sexp x,
> (assuming x doesn't side effect. Just something like
> (+ 2 y) or (+ (* a x x) (* b x) c).

...?

The example was finding (if a a b). Didn't you say
regexps couldn't identify all such cases because of
lack of expressiveness with respect to the more
expressive Lisp?

Finding (- x x) should be very easy! And

    ;; find (- x x) - this is line 64
    (search-regexp-in-files "~/.emacs.d/emacs-init/**/*.el" "(- \\(.*\\) \\1)")

indeed shows the result:

    /home/incal/.emacs.d/emacs-init/search-regexp-in-files.el (64)

(By the way, is there a less bulky way to get the
equivalence of [[:space:]\n]* (and ditto +) at all
places where the punctuation and whitespaces now serve
as delimiters?)

If you mean post-evaluation obviously regexps don't
evaluate anything and they don't know Lisp.

But if you or anyone else do this is Lisp - why not in
the byte-compiler? - and it is an improvement either
with respect to finding all occurrences *or* doing
this post-evaluation (I don't know which you mean)
then I'd use that in an instant rather than my stuff
for analyzing Elisp. I don't see how such things can
be identified post-evaluation because that depends on
evaluation, but perhaps if you only use calculation
with fixed digits it works, tho how many people do how
much of that I wonder.

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: use Elisp to improve your Elisp - some code issues

[Pascal J. Bourguignon]

Emanuel Berg <embe8573@student.uu.se> writes:

> "Pascal J. Bourguignon" <pjb@informatimago.com>
> writes:
>
>>> It would be more illustrative if you provide
>>> a counterexample of the same construct which cannot
>>> be caught by that or another regexp. It will be
>>> interesting to see if that example is anything
>>> I would put in my code, ever.
>>
>> It's easy. Try to simplify (- x x) for any sexp x,
>> (assuming x doesn't side effect. Just something like
>> (+ 2 y) or (+ (* a x x) (* b x) c).
>
> ...?
>
> The example was finding (if a a b). Didn't you say
> regexps couldn't identify all such cases because of
> lack of expressiveness with respect to the more
> expressive Lisp?

No.  

I said that if you went beyond those simple case, your regexp solution
would break lamentably.



-- 
__Pascal Bourguignon__                 http://www.informatimago.com/
“The factory of the future will have only two employees, a man and a
dog. The man will be there to feed the dog. The dog will be there to
keep the man from touching the equipment.” -- Carl Bass CEO Autodesk

Re: use Elisp to improve your Elisp - some code issues

[Emanuel Berg]

"Pascal J. Bourguignon" <pjb@informatimago.com>
writes:

> No.
>
> I said that if you went beyond those simple case,
> your regexp solution would break lamentably.

This is an interface to regexps so it sure doesn't go
anywhere beyond what can be expressed by regexps.

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: use Elisp to improve your Elisp - some code issues

[Sam Halliday]

On Saturday, 1 August 2015 05:22:35 UTC+1, Emanuel Berg  wrote:
> The tool (code) just searches for regexps in the files whose names match another regexp.

OK, I don't think that would extend itself to use elsewhere unfortunately. I'd imagined that you had a "detector phase" and then a "suggestion phase" whereas it sounds like it's all in one go with a regex replace.

If there was a way to factor out the suggestion phase into a standalone component (which provided a diff and some metadata) then we could definitely make use of that in ENSIME in many places. e.g. Something non-intrusive like just add a symbol to the gutter and require some further user action before showing the diff (which might be multi-file) and allowing the user to accept or reject.

Re: use Elisp to improve your Elisp - some code issues

[Robert Thorpe]

Emanuel Berg <embe8573@student.uu.se> writes:

> "Pascal J. Bourguignon" <pjb@informatimago.com>
> writes:
>
>> No.
>>
>> I said that if you went beyond those simple case,
>> your regexp solution would break lamentably.
>
> This is an interface to regexps so it sure doesn't go
> anywhere beyond what can be expressed by regexps.

I agree with Pascal & Marcin.

Why not use the "read" function to read in the code of the file.  Then
you have everything as a tree.  You can use car and cdr to walk the tree
and find the relevant function calls.  Then you're in the right place in
the Chomsky heirachy.

> (if a a b)

Suppose "a" and "b" are expressions rather than variables.  Suppose
that the two instances of "a" have different whitespace and comments
between them.  In that case they can't be differentiated by regex.
Regex can't count parenthesis either.

Another possibility is editing the byte-compiler source to provide more warnings.

BR,
Robert Thorpe

Re: use Elisp to improve your Elisp - some code issues

[Rusi]

On Sunday, August 2, 2015 at 9:06:33 PM UTC+5:30, Robert Thorpe wrote:
> Emanuel Berg  writes:
> 
> > "Pascal J. Bourguignon" 
> > writes:
> >
> >> No.
> >>
> >> I said that if you went beyond those simple case,
> >> your regexp solution would break lamentably.
> >
> > This is an interface to regexps so it sure doesn't go
> > anywhere beyond what can be expressed by regexps.
> 
> I agree with Pascal & Marcin.
> 
> Why not use the "read" function to read in the code of the file.  Then
> you have everything as a tree.  You can use car and cdr to walk the tree

If something like this 
https://common-lisp.net/project/cl-match/doc/clmatch.htm
is there it sure makes life more pleasant than car/cdr.

[Disclaimer: I dont know much about common-lisp nor about whether these
things are there in elisp.
Talking from (ancient) scheme experience
]

Re: use Elisp to improve your Elisp - some code issues

[Pascal J. Bourguignon]

Robert Thorpe <rt@robertthorpeconsulting.com> writes:

> Emanuel Berg <embe8573@student.uu.se> writes:
>
>> "Pascal J. Bourguignon" <pjb@informatimago.com>
>> writes:
>>
>>> No.
>>>
>>> I said that if you went beyond those simple case,
>>> your regexp solution would break lamentably.
>>
>> This is an interface to regexps so it sure doesn't go
>> anywhere beyond what can be expressed by regexps.
>
> I agree with Pascal & Marcin.
>
> Why not use the "read" function to read in the code of the file.  Then
> you have everything as a tree.  You can use car and cdr to walk the tree
> and find the relevant function calls.  Then you're in the right place in
> the Chomsky heirachy.

Careful, there's a reason why not to use read and instead to use
forward-sexp and other emacs editing function.

read will drop comments, newlines, indentation, etc.

Since most of the case, you will transform only some form or subform,
you may want to keep the rest of the source file intact.  Therefore
using primarily forward-sexp/down-list/up-list to navigate the source is
better. You may then use read-from-string+buffer-substring to transform
a sexp identified with forward-sexp, to process it, and insert+pp to
format it back into the buffer.


On the other hand, you may consider that comments have nothing to do in
source databases, and that pp is authoritative, and therefore use read
and pp to build an editor that will maintain the source database
automatically for yourself.  Then you always work at the sexp level.

It's probably to be able to work at the sexp level that docstrings were
invented, since even in LISP 1.5, comments existed (in the form of
comment cards).




-- 
__Pascal Bourguignon__                 http://www.informatimago.com/
“The factory of the future will have only two employees, a man and a
dog. The man will be there to feed the dog. The dog will be there to
keep the man from touching the equipment.” -- Carl Bass CEO Autodesk

Re: use Elisp to improve your Elisp - some code issues

[Ian Zimmerman]

On 2015-08-02 02:06 +0200, Emanuel Berg wrote:

> (By the way, is there a less bulky way to get the equivalence of
> [[:space:]\n]* (and ditto +) at all places where the punctuation and
> whitespaces now serve as delimiters?)

Look at the sregex module.

-- 
Please *no* private copies of mailing list or newsgroup messages.
Rule 420: All persons more than eight miles high to leave the court.

Re: use Elisp to improve your Elisp - some code issues

[Tassilo Horn]

Hi,

just on Saturday, Michael Heerdegen announced his el-search.el on
emacs-devel:

  https://lists.gnu.org/archive/html/emacs-devel/2015-08/msg00022.html

That doesn't work with regexes on strings but instead the Lisp code is
properly read, and then you can use pattern matching in terms of `pcase'
patterns for searching and replacing.

Bye,
Tassilo

Re: use Elisp to improve your Elisp - some code issues

[Emanuel Berg]

Sam Halliday <sam.halliday@gmail.com> writes:

> OK, I don't think that would extend itself to use
> elsewhere unfortunately.

The program is an interface to regexps, using one such
on the filesystem to delimit the search area files,
then applying another on those files' contents - and
last, reporting the result.

It is find + grep in another form, as said.

The use cases for such a tool are virtually countless
and there are many such implementations and
interfaces around.

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: use Elisp to improve your Elisp - some code issues

[Emanuel Berg]

Robert Thorpe <rt@robertthorpeconsulting.com> writes:

> I agree with Pascal & Marcin.
>
> Why not use the "read" function to read in the code
> of the file. Then you have everything as a tree.
> You can use car and cdr to walk the tree and find
> the relevant function calls. Then you're in the
> right place in the Chomsky heirachy.
>
> Suppose "a" and "b" are expressions rather than
> variables. Suppose that the two instances of "a"
> have different whitespace and comments between them.
> In that case they can't be differentiated by regex.
> Regex can't count parenthesis either.
>
> Another possibility is editing the byte-compiler
> source to provide more warnings.

If you read the code, you find the two most important
functions are `regexp-hits-find-hit' and
`search-regexp-in-files'. regexp means regexps!
That there are examples - which work - applied on Lisp
files don't change that.

Now, it doesn't say *one word* about re-parsing or
otherwise analyzing the code with respect imbecile,
ten million times accursed computer science theory.

If you want to do that - just do it! See if I will
come and have 42 degrees of expedition fever and
purposely misunderstand and be all negativistic.
Of course, I will not - and why do you think that is?

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: use Elisp to improve your Elisp - some code issues

[John Mastro]

> If you read the code, you find the two most important
> functions are `regexp-hits-find-hit' and
> `search-regexp-in-files'. regexp means regexps!
> That there are examples - which work - applied on Lisp
> files don't change that.
>
> Now, it doesn't say *one word* about re-parsing or
> otherwise analyzing the code with respect imbecile,
> ten million times accursed computer science theory.
>
> If you want to do that - just do it! See if I will
> come and have 42 degrees of expedition fever and
> purposely misunderstand and be all negativistic.
> Of course, I will not - and why do you think that is?

Where you see negativity I think you could also see people who found
your idea interesting and are discussing ways it could be taken further.
There's no slight in that - on the contrary, I'd say it's a compliment!

-- 
john