Re: cleaning up a big regexp

Re: cleaning up a big regexp

[Joost Kremers]

Tory S. Anderson wrote:
> Using gnus I have a growing regexp that represents the criteria for bulk email and splits accordingly:

[...]

> Is there a way to clean this up to make it both more readable and more
> easily editable? It seems like keeping some kind of list would be the
> way to do it, instead of an ever-lengthening string.

There's the function `regexp-opt', which takes a list of strings and
returns a regular expression that will match any of those strings.
Perhaps you can use that?


-- 
Joost Kremers                                   joostkremers@fastmail.fm
Selbst in die Unterwelt dringt durch Spalten Licht
EN:SiS(9)

Re: cleaning up a big regexp

[Tory S. Anderson]

That command would shrink the regexp and make it more efficient, but my goal here is to be able to more easily read and append to it, rather than optimize it. I would like to be able to have a list, maybe something like:

("^From:.*" ((".*@bulk1.com")
          (".*@bulk2.com")
          (".*@bulk3.com")))

I'm new enough to [e]lisp that I'm not sure what list->string concatenation functions would do the trick here. 

Joost Kremers <joost.m.kremers@gmail.com> writes:

> Tory S. Anderson wrote:
>> Using gnus I have a growing regexp that represents the criteria for bulk email and splits accordingly:
>
> [...]
>
>> Is there a way to clean this up to make it both more readable and more
>> easily editable? It seems like keeping some kind of list would be the
>> way to do it, instead of an ever-lengthening string.
>
> There's the function `regexp-opt', which takes a list of strings and
> returns a regular expression that will match any of those strings.
> Perhaps you can use that?

Re: cleaning up a big regexp

[Michael Albinus]

Joost Kremers <joost.m.kremers@gmail.com> writes:

> There's the function `regexp-opt', which takes a list of strings and
> returns a regular expression that will match any of those strings.
> Perhaps you can use that?

regexp-opt cannot handle meta characters like "*". The OP showed such a regexp.

Maybe somthing like this works (untested):

(setq my-gnus-bulk-regexp
      (concat
       "^\\(From:.*@"
       (regexp-opt
	'("maillist.codeproject.com"
	  "papajohns-specials.com"
	  "qomail.quikorder.com"
	  "linkedin.com"
	  "facebookmail.com"
	  "plus.google.com"
	  "twitter.com"
	  "youtube.com"
	  "linguistlist.org"
	  "sportsauthority.com")
	'par)
       "\\)\\|\\(To:.*torysanderson@gmail.com\\)"))

Best regards, Michael.

Re: cleaning up a big regexp

[Tory S. Anderson]

Ok. Applying the advice I've received so far, I have the following (which doesn't quite evaluate). Clearly my syntax is wrong. 

(setq my-gnus-bulk-from-address-list '("@maillist.codeproject.com"
				      "@papajohns-specials.com"
				      "@qomail.quikorder.com"
				      "@linkedin.com"
				      "@facebookmail.com"
				      "@plus.google.com"
				      "@twitter.com"
				      "@youtube.com"
				      "@linguistlist.org"
				      "sportsauthority.com")) ;; list of bulkmail addresses
(setq my-gnus-bulk-from-address-regexp (mapconcat `regexp-quote my-gnus-bulk-from-address-list "\\|")) ;; make OR

(setq my-gnus-bulk-from-regexp
      (regexp-opt (mapconcat `(concat "^From:.*") my-gnus-bulk-from-address-regexp "\\|"))) ;; apply "From.*" to the start of each address

Re: cleaning up a big regexp

[Michael Albinus]

torys.anderson@gmail.com (Tory S. Anderson) writes:

> (setq my-gnus-bulk-from-address-regexp (mapconcat `regexp-quote my-gnus-bulk-from-address-list "\\|")) ;; make OR

Use a single apostroph "'" here.

> (setq my-gnus-bulk-from-regexp
>       (regexp-opt (mapconcat `(concat "^From:.*") my-gnus-bulk-from-address-regexp "\\|"))) ;; apply "From.*" to the start of each address

That's wrong. You `concat' just one element, not needed. `regexp-opt'
cannot be applied over meta characters, like "*". Even if you quote
them, it doesn't work.

Best regards, Michael.

Re: cleaning up a big regexp

[Stefan Monnier]

> (mapconcat `regexp-quote my-gnus-bulk-from-address-list "\\|"))

A.k.a (regexp-opt my-gnus-bulk-from-address-list)


-- Stefan

Re: cleaning up a big regexp

[Michael Albinus]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> (mapconcat `regexp-quote my-gnus-bulk-from-address-list "\\|"))
>
> A.k.a (regexp-opt my-gnus-bulk-from-address-list)

Not when the strings contain meta characters like "*". I repeat it again
and again, because I've entered this trap very recently. See commit
trunk r117880, where I refuse to tell that I have been an idiot.

> -- Stefan

Best regards, Michael.

Thanks! Re: cleaning up a big regexp

[Tory S. Anderson]

Okay. In my code I now have: 
(setq my-gnus-bulk-from-address-list '("@maillist.codeproject.com"
				      "@papajohns-specials.com"
				      "@qomail.quikorder.com"
				      "@linkedin.com"
				      "@facebookmail.com"
				      "@plus.google.com"
				      "@twitter.com"
				      "@youtube.com"
				      "@linguistlist.org"
				      "@sportsauthority.com")) ;; list of bulkmail addresses
(setq my-bulk-from (mapconcat (lambda (x) (concat "^From:.*" x)) my-gnus-bulk-from-address-list "\\|"))

(setq nnmail-split-methods
       '(("mail.bulk" my-bulk-from)


Looks much better and seems to work. It appends "From:.*" to each string and then combines them into a single string. Any suggestions for improvement are welcome; big thanks for the suggestions that helped me figure this out. regexp-opt will probably make this more efficient (especially for big lists), but I haven't figured out how to plug it in yet. 


Michael Albinus <michael.albinus@gmx.de> writes:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>>> (mapconcat `regexp-quote my-gnus-bulk-from-address-list "\\|"))
>>
>> A.k.a (regexp-opt my-gnus-bulk-from-address-list)
>
> Not when the strings contain meta characters like "*". I repeat it again
> and again, because I've entered this trap very recently. See commit
> trunk r117880, where I refuse to tell that I have been an idiot.
>
>> -- Stefan
>
> Best regards, Michael.

Re: cleaning up a big regexp

[Stefan Monnier]

>>> (mapconcat `regexp-quote my-gnus-bulk-from-address-list "\\|"))
>> A.k.a (regexp-opt my-gnus-bulk-from-address-list)
> Not when the strings contain meta characters like "*".

No, no: even if they do.  Read again: he mapconcats `regexp-quote'.


        Stefan

Regexp in nnmail-split-methods (was Re: cleaning up a big regexp)

[Tory S. Anderson]

Okay; I've run into one more problem that is probabably (hopefully) a simple fix. Running the following code, I get a nice regexp in my "my-bulk-from" var. However, if I simply set that var as the criteria for "mail.bulk" splitting, all my split-methods break and all my mail ends up in my main inbox. If, on the other hand, I simply paste the value of that variable (rather than referencing the variable), it works as expected. I expect I'm missing a quote or something; more importantly, I'm missing some understanding. 

Why does `("mail.bulk" my-bulk-from)` fail but `("mail.bulk" "BIG REGEXP")` work, when the value of my-bulk-from is BIG REGEXP?



(setq my-gnus-bulk-from-address-list '("@maillist.codeproject.com"
				      "@papajohns-specials.com"
				      "@qomail.quikorder.com"
				      "@linkedin.com"
				      "@facebookmail.com"
				      "@plus.google.com"
				      "@twitter.com"
				      "@youtube.com"
				      "@linguistlist.org"
				      "@sportsauthority.com")) ;; list of bulkmail addresses
(setq my-bulk-from (concat "^From:.*" (regexp-opt my-gnus-bulk-from-address-list)))

(setq nnmail-split-methods
;       '(("mail.bulk" my-bulk-from) ;; breaks my split-methods
	 ("mail.bulk" "^From:.*\\(?:@\\(?:facebookmail\\.com\\|lin\\(?:guistlist\\.org\\|kedin\\.com\\)\\|\\(?:maillist\\.codeproject\\|p\\(?:apajohns-specials\\|lus\\.google\\)\\|qomail\\.quikorder\\|sportsauthority\\|twitter\\|youtube\\)\\.com\\)\\)")

Re: Regexp in nnmail-split-methods (was Re: cleaning up a big regexp)

[sokobania.01]

Le mardi 16 septembre 2014 13:05:16 UTC+2, Tory S. Anderson a écrit :
> (setq nnmail-split-methods
> 
> ;       '(("mail.bulk" my-bulk-from) ;; breaks my split-methods
> 
> 	 ("mail.bulk" "^From:.*\\(?:@\\(?:facebookmail\\.com\\|lin\\(?:guistlist\\.org\\|kedin\\.com\\)\\|\\(?:maillist\\.codeproject\\|p\\(?:apajohns-specials\\|lus\\.google\\)\\|qomail\\.quikorder\\|sportsauthority\\|twitter\\|youtube\\)\\.com\\)\\)")

You have to use a backquote (or backtick) and a comma to replace the symbol my-bulk-from by its value in the list you create:
(setq nnmail-split-methods
       `(("mail.bulk" ,my-bulk-from) ;; Won't break my split-methods
         ("mail.bulk" "^From:.*\\(?:@\\(?:facebookmail\\.com\\|lin\\(?:guistlist\\.org\\|kedin\\.com\\)\\|\\(?:maillist\\.codeproject\\|p\\(?:apajohns-specials\\|lus\\.google\\)\\|qomail\\.quikorder\\|sportsauthority\\|twitter\\|youtube\\)\\.com\\)\\)")

This is some lispish stuff...
HTH
)jack(

RESOLVED Re: Regexp in nnmail-split-methods (was Re: cleaning up a big regexp)

[Tory S. Anderson]

That did the trick. Thanks!

sokobania.01@gmail.com writes:

> Le mardi 16 septembre 2014 13:05:16 UTC+2, Tory S. Anderson a écrit :
>> (setq nnmail-split-methods
>> 
>> ;       '(("mail.bulk" my-bulk-from) ;; breaks my split-methods
>> 
>> 	 ("mail.bulk" "^From:.*\\(?:@\\(?:facebookmail\\.com\\|lin\\(?:guistlist\\.org\\|kedin\\.com\\)\\|\\(?:maillist\\.codeproject\\|p\\(?:apajohns-specials\\|lus\\.google\\)\\|qomail\\.quikorder\\|sportsauthority\\|twitter\\|youtube\\)\\.com\\)\\)")
>
> You have to use a backquote (or backtick) and a comma to replace the symbol my-bulk-from by its value in the list you create:
> (setq nnmail-split-methods
>        `(("mail.bulk" ,my-bulk-from) ;; Won't break my split-methods
>          ("mail.bulk" "^From:.*\\(?:@\\(?:facebookmail\\.com\\|lin\\(?:guistlist\\.org\\|kedin\\.com\\)\\|\\(?:maillist\\.codeproject\\|p\\(?:apajohns-specials\\|lus\\.google\\)\\|qomail\\.quikorder\\|sportsauthority\\|twitter\\|youtube\\)\\.com\\)\\)")
>
> This is some lispish stuff...
> HTH
> )jack(

cleaning up a big regexp

[Tory S. Anderson]

Using gnus I have a growing regexp that represents the criteria for bulk email and splits accordingly:

(setq my-gnus-bulk-regexp "\\(^From:.*\\(@maillist.codeproject.com\\|@papajohns-specials.com\\|@qomail.quikorder.com\\|@linkedin.com\\|@facebookmail.com\\|@plus.google.com\\|@twitter.com\\|@youtube.com\\|@linguistlist.org\\|sportsauthority.com\\)\\)\\|\\(^To:.*torysanderson@gmail.com\\)")

;; TODO make bulk-list
(setq nnmail-split-methods
       '(("mail.bulk" my-gnus-bulk-regexp)
        ...
 

Is there a way to clean this up to make it both more readable and more easily editable? It seems like keeping some kind of list would be the way to do it, instead of an ever-lengthening string. 

Re: cleaning up a big regexp

[Eric Abrahamsen]

torys.anderson@gmail.com (Tory S. Anderson) writes:

> Using gnus I have a growing regexp that represents the criteria for bulk email and splits accordingly:
>
> (setq my-gnus-bulk-regexp "\\(^From:.*\\(@maillist.codeproject.com\\|@papajohns-specials.com\\|@qomail.quikorder.com\\|@linkedin.com\\|@facebookmail.com\\|@plus.google.com\\|@twitter.com\\|@youtube.com\\|@linguistlist.org\\|sportsauthority.com\\)\\)\\|\\(^To:.*torysanderson@gmail.com\\)")
>
> ;; TODO make bulk-list
> (setq nnmail-split-methods
>        '(("mail.bulk" my-gnus-bulk-regexp)
>         ...
>  
>
> Is there a way to clean this up to make it both more readable and more easily editable? It seems like keeping some kind of list would be the way to do it, instead of an ever-lengthening string. 

Check out the regexp-opt function, which takes an arbitrary number of
strings and gloms them into a big regexp. At least it looks nicer in
your .gnus.el file.

Re: cleaning up a big regexp

[Thorsten Jolitz]

torys.anderson@gmail.com (Tory S. Anderson) writes:

> Using gnus I have a growing regexp that represents the criteria for bulk email and splits accordingly:
>
> (setq my-gnus-bulk-regexp )
>
> ;; TODO make bulk-list
> (setq nnmail-split-methods
>        '(("mail.bulk" my-gnus-bulk-regexp)
>         ...
>  
>
> Is there a way to clean this up to make it both more readable and more
> easily editable? It seems like keeping some kind of list would be the
> way to do it, instead of an ever-lengthening string. 

I have a little library called drx.el on github
(https://github.com/tj64/drx ) which isn't as extreme as rx.el in
avoiding regexp strings, but is pretty flexible when it comes to avoid
all the plumbing/glue-code between the actual regexp strings.

Furthermore it abstracts from 3 common regexp elements: "^", "$" and
"\\*", so it lets you e.g. write a regexp for Org-mode:


#+BEGIN_SRC emacs-lisp :results raw
  (require 'drx)
  (format "%S"
   (drx "foo" t '(t nil (2)) t '(nil t (2))
         "bar" "loo"))
#+END_SRC

#+results:
"^\\(\\*\\(\\*\\)\\{2\\}\\)foo\\(bar\\)\\(loo\\)\\{2\\}$"

and reuse it for css mode with different outline stars:

#+BEGIN_SRC emacs-lisp :results raw
  (require 'drx)
  (let ((drx-BOL "^/\\*  ")
        (drx-EOL " \\*/$")
	(drx-STAR "@"))
  (format "%S"
   (drx "foo" t '(t nil (2)) t '(nil t (2))
         "bar" "loo")))
#+END_SRC

#+results:
"^/\\*  \\(@\\(@\\)\\{2\\}\\)foo\\(bar\\)\\(loo\\)\\{2\\} \\*/$"

Here is something similar to your example regexp:

#+BEGIN_SRC emacs-lisp :results raw
  (require 'drx)
  (format "%S"
   (drx 
    (drx "From:.*" t nil nil 'group
        (drx "@maillist.codeproject.com" nil nil nil 'alt    
             "@papajohns-specials.com"
             "@qomail.quikorder.com"
             "@linkedin.com"
             "@facebookmail.com"
             "@plus.google.com"
             "@twitter.com"
             "@youtube.com"
             "@linguistlist.org"
             "sportsauthority.com"))
    nil nil nil 'alt
    (drx "To:.*torysanderson@gmail.com" nil nil nil t)))
#+END_SRC

#+results:
"\\(^\\(From:.*\\(@maillist.codeproject.com\\|@papajohns-specials.com\\|@qomail.quikorder.com\\|@linkedin.com\\|@facebookmail.com\\|@plus.google.com\\|@twitter.com\\|@youtube.com\\|@linguistlist.org\\|sportsauthority.com\\)\\)\\|\\(To:.*torysanderson@gmail.com\\)\\)"

-- 
cheers,
Thorsten