How to highlight-regexp across multiple lines

How to highlight-regexp across multiple lines

[Angus Comber]

I was looking at this question on stackoverflow and it seems regex can
select across multiple lines generally but how would I do so for
highlight-regexp in emacs?

http://stackoverflow.com/questions/159118/how-do-i-match-any-character-across-multiple-lines-in-a-regular-expression


Eg I want to highlight ALL text from aPDU... to the final closing brace:

aPDU-rose : retResult : {
  invokeID 2,
  operValue {
    operationValue local : 71,
    result {
      crossRefIdentifier '40 3f'H
    }
  }
}


How would I do that using highlight-regexp?

highlight-regexp aPDU-rose.* hi-yellow  would get me the first line - but
how to get everything including last } character?

Re: How to highlight-regexp across multiple lines

[Thorsten Jolitz]

Angus Comber <anguscomber@gmail.com> writes:

> I was looking at this question on stackoverflow and it seems regex can
> select across multiple lines generally but how would I do so for
> highlight-regexp in emacs?
>
> http://stackoverflow.com/questions/159118/how-do-i-match-any-character-across-multiple-lines-in-a-regular-expression
>
>
> Eg I want to highlight ALL text from aPDU... to the final closing brace:
>
> aPDU-rose : retResult : {
>   invokeID 2,
>   operValue {
>     operationValue local : 71,
>     result {
>       crossRefIdentifier '40 3f'H
>     }
>   }
> }
>
>
> How would I do that using highlight-regexp?
>
> highlight-regexp aPDU-rose.* hi-yellow  would get me the first line - but
> how to get everything including last } character?

This regexp did match the aPDU in the gnus *Article* buffer:

,--------------------
| "^aPDU[^\\000]+[}]+?$"
`--------------------

-- 
cheers,
Thorsten

Re: How to highlight-regexp across multiple lines

[Angus Comber]

That's brilliant.

The only thing is...

This example isn't highlighted:

aPDU-rose : invoke : {
  invokeID 5,
  operationValue local : 71,
  argument {
    monitorObject device : dialingNumber : "401"
  }
}

or this one:

aPDU-rose : retResult : {
  invokeID 2,
  operValue {
    operationValue local : 71,
    result {
      crossRefIdentifier '40 3f'H
    }
  }
}

Any ideas why?

What is [^\\000] ?


On 3 April 2014 16:08, Thorsten Jolitz <tjolitz@gmail.com> wrote:

> Angus Comber <anguscomber@gmail.com> writes:
>
> > I was looking at this question on stackoverflow and it seems regex can
> > select across multiple lines generally but how would I do so for
> > highlight-regexp in emacs?
> >
> >
> http://stackoverflow.com/questions/159118/how-do-i-match-any-character-across-multiple-lines-in-a-regular-expression
> >
> >
> > Eg I want to highlight ALL text from aPDU... to the final closing brace:
> >
> > aPDU-rose : retResult : {
> >   invokeID 2,
> >   operValue {
> >     operationValue local : 71,
> >     result {
> >       crossRefIdentifier '40 3f'H
> >     }
> >   }
> > }
> >
> >
> > How would I do that using highlight-regexp?
> >
> > highlight-regexp aPDU-rose.* hi-yellow  would get me the first line - but
> > how to get everything including last } character?
>
> This regexp did match the aPDU in the gnus *Article* buffer:
>
> ,--------------------
> | "^aPDU[^\\000]+[}]+?$"
> `--------------------
>
> --
> cheers,
> Thorsten
>
>
>

Re: How to highlight-regexp across multiple lines

[Angus Comber]

It seems that it matches the aPDU messages provided there is no 0 in the
text.  I am puzzled by the [^\\000] - where can I find documentation on it?

Is it not null or something like that?  But then why does it not capture
string "0"?


On 3 April 2014 16:08, Thorsten Jolitz <tjolitz@gmail.com> wrote:

> Angus Comber <anguscomber@gmail.com> writes:
>
> > I was looking at this question on stackoverflow and it seems regex can
> > select across multiple lines generally but how would I do so for
> > highlight-regexp in emacs?
> >
> >
> http://stackoverflow.com/questions/159118/how-do-i-match-any-character-across-multiple-lines-in-a-regular-expression
> >
> >
> > Eg I want to highlight ALL text from aPDU... to the final closing brace:
> >
> > aPDU-rose : retResult : {
> >   invokeID 2,
> >   operValue {
> >     operationValue local : 71,
> >     result {
> >       crossRefIdentifier '40 3f'H
> >     }
> >   }
> > }
> >
> >
> > How would I do that using highlight-regexp?
> >
> > highlight-regexp aPDU-rose.* hi-yellow  would get me the first line - but
> > how to get everything including last } character?
>
> This regexp did match the aPDU in the gnus *Article* buffer:
>
> ,--------------------
> | "^aPDU[^\\000]+[}]+?$"
> `--------------------
>
> --
> cheers,
> Thorsten
>
>
>

Re: How to highlight-regexp across multiple lines

[Thorsten Jolitz]

Angus Comber <anguscomber@gmail.com> writes:

> It seems that it matches the aPDU messages provided there is no 0 in the
> text.  I am puzzled by the [^\\000] - where can I find documentation on it?

Thats stolen from the Org-mode sources, and I asked the same question
before ;)

Here is Carsten Dominik's answer:

,----------------------------------------------------------------------------
| > PS 
| > Can anybody explain this marvelous construct in the regexp:
| > 
| > ,---------
| > | [^\\000]
| > `---------
| 
| This is just a cheep way to match any character at all, because \000 should
| not be part of any string (in C it indicates the end of a string).
| In principle you could put any character you are sure will not turn up,
| but \000 seems to be the safest choice.  It is
| faster (I think) than "\\(.\\|\n\\)*" because the first will
| just run fast and streight with a table lookup while the
| latter need to always alternate between two alternatives.
| I have not timed it, though.
`----------------------------------------------------------------------------

> Is it not null or something like that?  But then why does it not capture
> string "0"?

This happened to me to, although I think it shouldn't.

But when I used it with M-x regexp-builder it worked. With M-x reb-copy
(and then paste), the [^\\000] is transformed to a different
representation - maybe you could try reb-copy & paste it into your
function and see if it works then?

> On 3 April 2014 16:08, Thorsten Jolitz <tjolitz@gmail.com> wrote:
>
>> Angus Comber <anguscomber@gmail.com> writes:
>>
>> > I was looking at this question on stackoverflow and it seems regex can
>> > select across multiple lines generally but how would I do so for
>> > highlight-regexp in emacs?
>> >
>> >
>> http://stackoverflow.com/questions/159118/how-do-i-match-any-character-across-multiple-lines-in-a-regular-expression
>> >
>> >
>> > Eg I want to highlight ALL text from aPDU... to the final closing brace:
>> >
>> > aPDU-rose : retResult : {
>> >   invokeID 2,
>> >   operValue {
>> >     operationValue local : 71,
>> >     result {
>> >       crossRefIdentifier '40 3f'H
>> >     }
>> >   }
>> > }
>> >
>> >
>> > How would I do that using highlight-regexp?
>> >
>> > highlight-regexp aPDU-rose.* hi-yellow  would get me the first line - but
>> > how to get everything including last } character?
>>
>> This regexp did match the aPDU in the gnus *Article* buffer:
>>
>> ,--------------------
>> | "^aPDU[^\\000]+[}]+?$"
>> `--------------------
>>
>> --
>> cheers,
>> Thorsten
>>
>>
>>
>

-- 
cheers,
Thorsten

Re: How to highlight-regexp across multiple lines

[Angus Comber]

Strangely, this works better:

aPDU[^W]+[}]$


[^W] is definitely mystifying.  Others eg [^some other letter] can work -
but only if your target block doesn't contain that letter.  And some of
them don't work at all - puzzling.

Even then, its mysterious because in a fairly large log file I have ALL
aPDU's were highlighted correctly except this text where the selected text
was from the start of the first to the end of the second one.

aPDU-rose : invoke : {
  invokeID 11,
  operationValue local : 21,
  argument {
    crossRefIdentifier '40 2f'H,
    eventSpecificInfo callEvent : connectionClearedEvent : {
      droppedConnection {
        call '02 01 19 0f'H,
        device staticID : dialingNumber : "403"
      },
      releasingDevice deviceIdentifier : deviceIdentifier : dialingNumber :
"403",
      localConnectionInfo connect,
      cause normalClearing
    }
  }
}
   -- monitor: 402
   -- CSTA Fwk : set control party
   -- setting cntrlDN=403 (for 1 parties)
   -- event ignored
  +++ Calls dump +++
    1 connids, 1 threads, 1 calls
    1:876002435e991001@2d8f990 2d8fc98:1[d403.2d90a08:1/EA-O
d402.2d91e28:2/EA-D]
  --- Calls dump ---
18:36:28.116 --- Input ---
18:36:28.125 +++ Input +++
18:36:28.125 DataFromSwitch 14 bytes
0000-000d 00 0c a2 0a 02 01 1e 30 05 02 01 05 05 00       |.......0......  |
  Decoded:
aPDU-rose : retResult : {
  invokeID 30,
  operValue {
    operationValue local : 5,
    result noData : NULL
  }
}



On 3 April 2014 20:21, Thorsten Jolitz <tjolitz@gmail.com> wrote:

> Angus Comber <anguscomber@gmail.com> writes:
>
> > It seems that it matches the aPDU messages provided there is no 0 in the
> > text.  I am puzzled by the [^\\000] - where can I find documentation on
> it?
>
> Thats stolen from the Org-mode sources, and I asked the same question
> before ;)
>
> Here is Carsten Dominik's answer:
>
>
> ,----------------------------------------------------------------------------
> | > PS
> | > Can anybody explain this marvelous construct in the regexp:
> | >
> | > ,---------
> | > | [^\\000]
> | > `---------
> |
> | This is just a cheep way to match any character at all, because \000
> should
> | not be part of any string (in C it indicates the end of a string).
> | In principle you could put any character you are sure will not turn up,
> | but \000 seems to be the safest choice.  It is
> | faster (I think) than "\\(.\\|\n\\)*" because the first will
> | just run fast and streight with a table lookup while the
> | latter need to always alternate between two alternatives.
> | I have not timed it, though.
>
> `----------------------------------------------------------------------------
>
> > Is it not null or something like that?  But then why does it not capture
> > string "0"?
>
> This happened to me to, although I think it shouldn't.
>
> But when I used it with M-x regexp-builder it worked. With M-x reb-copy
> (and then paste), the [^\\000] is transformed to a different
> representation - maybe you could try reb-copy & paste it into your
> function and see if it works then?
>
> > On 3 April 2014 16:08, Thorsten Jolitz <tjolitz@gmail.com> wrote:
> >
> >> Angus Comber <anguscomber@gmail.com> writes:
> >>
> >> > I was looking at this question on stackoverflow and it seems regex can
> >> > select across multiple lines generally but how would I do so for
> >> > highlight-regexp in emacs?
> >> >
> >> >
> >>
> http://stackoverflow.com/questions/159118/how-do-i-match-any-character-across-multiple-lines-in-a-regular-expression
> >> >
> >> >
> >> > Eg I want to highlight ALL text from aPDU... to the final closing
> brace:
> >> >
> >> > aPDU-rose : retResult : {
> >> >   invokeID 2,
> >> >   operValue {
> >> >     operationValue local : 71,
> >> >     result {
> >> >       crossRefIdentifier '40 3f'H
> >> >     }
> >> >   }
> >> > }
> >> >
> >> >
> >> > How would I do that using highlight-regexp?
> >> >
> >> > highlight-regexp aPDU-rose.* hi-yellow  would get me the first line -
> but
> >> > how to get everything including last } character?
> >>
> >> This regexp did match the aPDU in the gnus *Article* buffer:
> >>
> >> ,--------------------
> >> | "^aPDU[^\\000]+[}]+?$"
> >> `--------------------
> >>
> >> --
> >> cheers,
> >> Thorsten
> >>
> >>
> >>
> >
>
> --
> cheers,
> Thorsten
>
>
>

Re: How to highlight-regexp across multiple lines

[Thorsten Jolitz]

Angus Comber <anguscomber@gmail.com> writes:

> Strangely, this works better:
>
> aPDU[^W]+[}]$

nice, because I was looking for an alternative to [^\\000] since I want
to match Org regexps that actually do contain quite a lot \\000's. 

> [^W] is definitely mystifying.  Others eg [^some other letter] can work -
> but only if your target block doesn't contain that letter.  And some of
> them don't work at all - puzzling.

its indeed hard to find info about this stuff, I would need more info too.

> Even then, its mysterious because in a fairly large log file I have ALL
> aPDU's were highlighted correctly except this text where the selected text
> was from the start of the first to the end of the second one.
>
> aPDU-rose : invoke : {
>   invokeID 11,
>   operationValue local : 21,
>   argument {
>     crossRefIdentifier '40 2f'H,
>     eventSpecificInfo callEvent : connectionClearedEvent : {
>       droppedConnection {
>         call '02 01 19 0f'H,
>         device staticID : dialingNumber : "403"
>       },
>       releasingDevice deviceIdentifier : deviceIdentifier : dialingNumber :
> "403",
>       localConnectionInfo connect,
>       cause normalClearing
>     }
>   }
> }
>    -- monitor: 402
>    -- CSTA Fwk : set control party
>    -- setting cntrlDN=403 (for 1 parties)
>    -- event ignored
>   +++ Calls dump +++
>     1 connids, 1 threads, 1 calls
>     1:876002435e991001@2d8f990 2d8fc98:1[d403.2d90a08:1/EA-O
> d402.2d91e28:2/EA-D]
>   --- Calls dump ---
> 18:36:28.116 --- Input ---
> 18:36:28.125 +++ Input +++
> 18:36:28.125 DataFromSwitch 14 bytes
> 0000-000d 00 0c a2 0a 02 01 1e 30 05 02 01 05 05 00       |.......0......  |
>   Decoded:
> aPDU-rose : retResult : {
>   invokeID 30,
>   operValue {
>     operationValue local : 5,
>     result noData : NULL
>   }
> }

maybe you could experiment with non-greedy variants of the operators: 

,-------------------------------------------------------------------
| *?, +?, ??
|     are non-greedy variants of the operators above. The normal
|     operators '*', '+', '?' match as much as they can, as long as
|     the overall regexp can still match. With a following '?', they
|     will match as little as possible.
`-------------------------------------------------------------------


> On 3 April 2014 20:21, Thorsten Jolitz <tjolitz@gmail.com> wrote:
>
>> Angus Comber <anguscomber@gmail.com> writes:
>>
>> > It seems that it matches the aPDU messages provided there is no 0
>> > in the
>> > text.  I am puzzled by the [^\\000] - where can I find documentation on
>> it?
>>
>> Thats stolen from the Org-mode sources, and I asked the same question
>> before ;)
>>
>> Here is Carsten Dominik's answer:
>>
>>
>> ,----------------------------------------------------------------------------
>> | > PS
>> | > Can anybody explain this marvelous construct in the regexp:
>> | >
>> | > ,---------
>> | > | [^\\000]
>> | > `---------
>> |
>> | This is just a cheep way to match any character at all, because \000
>> should
>> | not be part of any string (in C it indicates the end of a string).
>> | In principle you could put any character you are sure will not turn up,
>> | but \000 seems to be the safest choice.  It is
>> | faster (I think) than "\\(.\\|\n\\)*" because the first will
>> | just run fast and streight with a table lookup while the
>> | latter need to always alternate between two alternatives.
>> | I have not timed it, though.
>>
>> `----------------------------------------------------------------------------
>>
>> > Is it not null or something like that?  But then why does it not
>> > capture
>> > string "0"?
>>
>> This happened to me to, although I think it shouldn't.
>>
>> But when I used it with M-x regexp-builder it worked. With M-x reb-copy
>> (and then paste), the [^\\000] is transformed to a different
>> representation - maybe you could try reb-copy & paste it into your
>> function and see if it works then?
>>
>> > On 3 April 2014 16:08, Thorsten Jolitz <tjolitz@gmail.com> wrote:
>> >
>> >> Angus Comber <anguscomber@gmail.com> writes:
>> >>
>> >> > I was looking at this question on stackoverflow and it seems
>> >> > regex can
>> >> > select across multiple lines generally but how would I do so for
>> >> > highlight-regexp in emacs?
>> >> >
>> >> >
>> >>
>> http://stackoverflow.com/questions/159118/how-do-i-match-any-character-across-multiple-lines-in-a-regular-expression
>> >> >
>> >> >
>> >> > Eg I want to highlight ALL text from aPDU... to the final closing
>> brace:
>> >> >
>> >> > aPDU-rose : retResult : {
>> >> >   invokeID 2,
>> >> >   operValue {
>> >> >     operationValue local : 71,
>> >> >     result {
>> >> >       crossRefIdentifier '40 3f'H
>> >> >     }
>> >> >   }
>> >> > }
>> >> >
>> >> >
>> >> > How would I do that using highlight-regexp?
>> >> >
>> >> > highlight-regexp aPDU-rose.* hi-yellow  would get me the first line -
>> but
>> >> > how to get everything including last } character?
>> >>
>> >> This regexp did match the aPDU in the gnus *Article* buffer:
>> >>
>> >> ,--------------------
>> >> | "^aPDU[^\\000]+[}]+?$"
>> >> `--------------------
>> >>
>> >> --
>> >> cheers,
>> >> Thorsten
>> >>
>> >>
>> >>
>> >
>>
>> --
>> cheers,
>> Thorsten
>>
>>
>>
>

-- 
cheers,
Thorsten

Re: How to highlight-regexp across multiple lines

[Nicolas Richard]

Angus Comber <anguscomber@gmail.com> writes:

> I was looking at this question on stackoverflow and it seems regex can
> select across multiple lines generally but how would I do so for
> highlight-regexp in emacs?
>
> http://stackoverflow.com/questions/159118/how-do-i-match-any-character-across-multiple-lines-in-a-regular-expression
>
>
> Eg I want to highlight ALL text from aPDU... to the final closing brace:

interactively, you enter the string it as follows:
a P D U [ ^ C-q 0 RET ] + C-q C-j } RET

in elisp it gives:
(highlight-regexp "aPDU[^\000]+?\n}" (quote hi-yellow))

C-q 0 RET inserts a character which has '0' as octal (or whatevervalue
read-quoted-char-radix is) representation, that's the NUL character that
Thorsten mentionned. This is equivalent to "\000" in elisp.

C-q C-j inserts a new line character (simply hitting RET would terminate
the prompt, so that's not an option). This is equivalent to "\n" in elisp.

The idea is : highlight *everything* lazily from the string "aPDU" up to
the first sequence "newline-closing brace".

HTH,

-- 
Nico.

Re: How to highlight-regexp across multiple lines

[Nicolas Richard]

Nicolas Richard <theonewiththeevillook@yahoo.fr> writes:
> interactively, you enter the string it as follows:
> a P D U [ ^ C-q 0 RET ] + C-q C-j } RET

oops, I forgot '?' right after '+' there.

-- 
Nico.

Re: How to highlight-regexp across multiple lines

[Angus Comber]

It works fine using re-builder as Thorsten explained.  Mine looks like this
in re-builder pane:

"aPDU[^^@]+?
}
"

But that gave me the hint - the [^^@]

Now this works perfectly:

(highlight-regexp "aPDU[^^@]+[\}]$" (quote hi-yellow))

I am thinking this means aPDU followed by anything that is not a null
character (ie EVERYTHING) followed by }\n

I am still a little puzzled though because I have multiple closing braces
followed by linefeeed in the aPDU's.  But anyway, it works!



On 4 April 2014 11:49, Nicolas Richard <theonewiththeevillook@yahoo.fr>wrote:

> Angus Comber <anguscomber@gmail.com> writes:
>
> > I was looking at this question on stackoverflow and it seems regex can
> > select across multiple lines generally but how would I do so for
> > highlight-regexp in emacs?
> >
> >
> http://stackoverflow.com/questions/159118/how-do-i-match-any-character-across-multiple-lines-in-a-regular-expression
> >
> >
> > Eg I want to highlight ALL text from aPDU... to the final closing brace:
>
> interactively, you enter the string it as follows:
> a P D U [ ^ C-q 0 RET ] + C-q C-j } RET
>
> in elisp it gives:
> (highlight-regexp "aPDU[^\000]+?\n}" (quote hi-yellow))
>
> C-q 0 RET inserts a character which has '0' as octal (or whatevervalue
> read-quoted-char-radix is) representation, that's the NUL character that
> Thorsten mentionned. This is equivalent to "\000" in elisp.
>
> C-q C-j inserts a new line character (simply hitting RET would terminate
> the prompt, so that's not an option). This is equivalent to "\n" in elisp.
>
> The idea is : highlight *everything* lazily from the string "aPDU" up to
> the first sequence "newline-closing brace".
>
> HTH,
>
> --
> Nico.
>

Re: How to highlight-regexp across multiple lines

[Thorsten Jolitz]

Angus Comber <anguscomber@gmail.com> writes:

> (highlight-regexp "aPDU[^^@]+[\}]$" (quote hi-yellow))

> I am still a little puzzled though because I have multiple closing braces
> followed by linefeeed in the aPDU's.  But anyway, it works!

Probably because + is greedy and looks for the longest possible
match. You could check if it still works with non-greedy +?.

-- 
cheers,
Thorsten