Re: error in replace-match: "args out of range"

Re: error in replace-match: "args out of range"

[Stefan Monnier]

>   eval((replace-match head-text-nested t nil head-text 4))

There's your problem right there: replace-match operates on the
match-data "returned" (i.e. set up) by the previous regexp match.
Now regexp matches happen *all the time* in Emacs, so the match-data
gets changed continuously.  So your above command will, maybe,
sometimes, operate on the match-data you want, but other times it will
operate on the match-data corresponding to some other regexp-match
you're not even aware of.

The only way to use replace-match reliably is to do it in the same piece
of code that did the regex-match and make sure nothing of importance
happens in between.  Going back to the top-level is not "nothing of
importance" since it'll run various hooks, may run pending timers,
process filters, plus cause redisplay which may call font-lock, etc...


        Stefan

Re: error in replace-match: "args out of range"

[ken]

On 03/28/2011 11:38 AM Stefan Monnier wrote:
>>   eval((replace-match head-text-nested t nil head-text 4))
> 
> There's your problem right there: replace-match operates on the
> match-data "returned" (i.e. set up) by the previous regexp match.
> Now regexp matches happen *all the time* in Emacs, so the match-data
> gets changed continuously.  So your above command will, maybe,
> sometimes, operate on the match-data you want, but other times it will
> operate on the match-data corresponding to some other regexp-match
> you're not even aware of.
> 
> The only way to use replace-match reliably is to do it in the same piece
> of code that did the regex-match and make sure nothing of importance
> happens in between.  Going back to the top-level is not "nothing of
> importance" since it'll run various hooks, may run pending timers,
> process filters, plus cause redisplay which may call font-lock, etc...
> 
> 
>         Stefan

Stefan,

Thanks for your reply.

I know that the error occurs on that line.  What I'm trying to figure
out is what to change on that line to correct the problem... resolve the
error.  I understand that another search-- one invoked since the
subexpressions were set for the above-cited replace-match-- will render
ineffective those subexpressions.  However, this replace-match (and
others used in the same function) occurs after the relevant search and
there is no other search occurring in between.  In fact, the relevant
search (the one where the relevant replace-match sets needed
subexpressions) is the *last* one in this function.

Moreover, I've run this function through edebug and when I evaluate the
individual variables inside the above-cited replace-match, those
variables do evaluate to the desired values.

Now I'm trying to understand what you're saying in your last sentence
above.  Are you saying that another search invoked by emacs and outside
of my code could alter the values of subexpressions of my last search?
If so, this would counsel against ever using replace-match *ever* in any
code.  For this reason I don't think it's what you're saying, or what is
actually confounding my replace-match.  Besides this, as said,
evaluating the values of the variables (excepting of course the final
arg consisting of the subexpression number) shows that those variables
contain expected values.

Let me add that I've altered the args to the above replace-match and get
the same error message.  Strangely the out-of-range args are "-1 -1".
What does that particular couple of numbers signify?

Thanks again for the reply.

Re: error in replace-match: "args out of range"

[Stefan Monnier]

> Now I'm trying to understand what you're saying in your last sentence
> above.  Are you saying that another search invoked by emacs and outside
> of my code could alter the values of subexpressions of my last search?

I'm just saying that

M-: (some-regexp-search) RET
M-: (replace-match ...) RET

won't always do what you want, because a whole lot of code will run
between the two.


        Stefan

Re: error in replace-match: "args out of range"

[ken]

On 03/28/2011 04:32 PM Stefan Monnier wrote:
>> Now I'm trying to understand what you're saying in your last sentence
>> above.  Are you saying that another search invoked by emacs and outside
>> of my code could alter the values of subexpressions of my last search?
> 
> I'm just saying that
> 
> M-: (some-regexp-search) RET
> M-: (replace-match ...) RET
> 
> won't always do what you want, because a whole lot of code will run
> between the two.
> 
> 
>         Stefan

Hmm.  I'm hearing "this is faith-based functionality."  Does it have to
do with the fact that all processes running from one emacs session have
the one and the same PID?  I recall reading a discussion on this list
(which was largely outside my capacity) multiple years ago about some
problem which 'would be resolved when threading is implemented.'  Is
this an issue of that sort?

Re: error in replace-match: "args out of range"

[PJ Weisberg]

On 4/3/11, ken <gebser@mousecar.com> wrote:
> On 03/28/2011 04:32 PM Stefan Monnier wrote:
>> I'm just saying that
>>
>> M-: (some-regexp-search) RET
>> M-: (replace-match ...) RET
>>
>> won't always do what you want, because a whole lot of code will run
>> between the two.
>>
>>
>>         Stefan
>
> Hmm.  I'm hearing "this is faith-based functionality."  Does it have to
> do with the fact that all processes running from one emacs session have
> the one and the same PID?  I recall reading a discussion on this list
> (which was largely outside my capacity) multiple years ago about some
> problem which 'would be resolved when threading is implemented.'  Is
> this an issue of that sort?

It sounds more like the sort of issue that would be *created* when
threading was implemented.

Based on what Stefan said, a more helpful question might be: what
happens between the call to match-string and the call to
replace-match?  Are they in the same function, and close together?
What can match-string tell you about the string you're trying to
replace?

-PJ

Re: error in replace-match: "args out of range"

[ken]

On 04/03/2011 07:19 PM PJ Weisberg wrote:
> On 4/3/11, ken <gebser@mousecar.com> wrote:
>> On 03/28/2011 04:32 PM Stefan Monnier wrote:
>>> I'm just saying that
>>>
>>> M-: (some-regexp-search) RET
>>> M-: (replace-match ...) RET
>>>
>>> won't always do what you want, because a whole lot of code will run
>>> between the two.
>>>
>>>
>>>         Stefan
>> Hmm.  I'm hearing "this is faith-based functionality."  Does it have to
>> do with the fact that all processes running from one emacs session have
>> the one and the same PID?  I recall reading a discussion on this list
>> (which was largely outside my capacity) multiple years ago about some
>> problem which 'would be resolved when threading is implemented.'  Is
>> this an issue of that sort?
> 
> It sounds more like the sort of issue that would be *created* when
> threading was implemented.

Probably this discussion won't solve anything, so let's leave it for
another time.


> 
> Based on what Stefan said, a more helpful question might be: what
> happens between the call to match-string and the call to
> replace-match?  Are they in the same function, and close together?
> What can match-string tell you about the string you're trying to
> replace?
> 
> -PJ
> 

Here's the relevant code fragment:

    (if (re-search-forward aname-re-str head-text-end-pt t)
	(progn	
	  (setq aname-str (match-string 2))
	  (setq head-text-nested (match-string 4))
	  (if (equal head-text-nested "")
	      (progn
		(setq head-text-nested (create-heading-text))
		(if (equal aname-str "")
		    (progn
		      (setq aname-str (create-aname head-text-nested))
		      (replace-match head-text-nested t nil nil 4)
		      (replace-match aname-str t nil nil 2))
		  (replace-match head-text-nested t t nil 4))) ;ERROR:

The top line above is the search.  The last line is the replace-match
which fails.  The couple replace-match lines above it might also fail...
haven't tested those yet.

create-aname is nothing but a set of setq's and a read-from-minibuffer.
 create-heading-text is just a read-from-minibuffer.

So, yes, the search and replace-match are in the same function.  Are
they "close together"?  I'd say yes, but that's a judgment call really.
 The string I want to replace in this particular instance is a null string.

Re: error in replace-match: "args out of range"

[Tim X]

ken <gebser@mousecar.com> writes:

> On 04/03/2011 07:19 PM PJ Weisberg wrote:
>> On 4/3/11, ken <gebser@mousecar.com> wrote:
>>> On 03/28/2011 04:32 PM Stefan Monnier wrote:
>>>> I'm just saying that
>>>>
>>>> M-: (some-regexp-search) RET
>>>> M-: (replace-match ...) RET
>>>>
>>>> won't always do what you want, because a whole lot of code will run
>>>> between the two.
>>>>
>>>>
>>>>         Stefan
>>> Hmm.  I'm hearing "this is faith-based functionality."  Does it have to
>>> do with the fact that all processes running from one emacs session have
>>> the one and the same PID?  I recall reading a discussion on this list
>>> (which was largely outside my capacity) multiple years ago about some
>>> problem which 'would be resolved when threading is implemented.'  Is
>>> this an issue of that sort?
>> 
>> It sounds more like the sort of issue that would be *created* when
>> threading was implemented.
>
> Probably this discussion won't solve anything, so let's leave it for
> another time.
>
>
>> 
>> Based on what Stefan said, a more helpful question might be: what
>> happens between the call to match-string and the call to
>> replace-match?  Are they in the same function, and close together?
>> What can match-string tell you about the string you're trying to
>> replace?
>> 
>> -PJ
>> 
>
> Here's the relevant code fragment:
>
>     (if (re-search-forward aname-re-str head-text-end-pt t)
> 	(progn	
> 	  (setq aname-str (match-string 2))
> 	  (setq head-text-nested (match-string 4))
> 	  (if (equal head-text-nested "")
> 	      (progn
> 		(setq head-text-nested (create-heading-text))
> 		(if (equal aname-str "")
> 		    (progn
> 		      (setq aname-str (create-aname head-text-nested))
> 		      (replace-match head-text-nested t nil nil 4)
> 		      (replace-match aname-str t nil nil 2))
> 		  (replace-match head-text-nested t t nil 4))) ;ERROR:
>
> The top line above is the search.  The last line is the replace-match
> which fails.  The couple replace-match lines above it might also fail...
> haven't tested those yet.
>
> create-aname is nothing but a set of setq's and a read-from-minibuffer.
>  create-heading-text is just a read-from-minibuffer.
>
> So, yes, the search and replace-match are in the same function.  Are
> they "close together"?  I'd say yes, but that's a judgment call really.
>  The string I want to replace in this particular instance is a null string.
>

Providing the code alway helps. While Stephan's points are valid and a
common error people make, I don't think it is the problem you are
seeing.

It would help to see the regexp you are using to test this. Note that
match-string returns nil if the subexpression you are asking for did not
exist. If the subexpression did not exist, what would replace-match
replace as there is no match?

Note also, you can get rid of some of those progns i.e.

     (when (re-search-forward aname-re-str head-text-end-pt t)
 	  (setq aname-str (match-string 2))
 	  (setq head-text-nested (match-string 4))
 	  (when (string= head-text-nested "")
 		(setq head-text-nested (create-heading-text))
                (if (not (string= aname-str ""))
 		     (replace-match head-text-nested t t nil 4)
                  (setq aname-str (create-aname head-text-nested))
 		  (replace-match head-text-nested t nil nil 4)
 		  (replace-match aname-str t nil nil 2))))

Note that the 'else' part of an if form has an implicit progn, so by
revesing the condition test, we can eliminate that progn. The 'unless'
form can be used instead of 'when' if you want to reverse the logic of
what you are doing. 

(Note above is untested, but intention should be clear)

My guess is that it does not work because replace-match has no idea what
it is yhou want to match as the subexpression was not found i.e. is not
in the string, so it has no way to know where in the string to do the
replace.

Tim


-- 
tcross (at) rapttech dot com dot au

Re: error in replace-match: "args out of range"

[ken]

On 04/04/2011 06:21 PM Tim X wrote:
> ken <gebser@mousecar.com> writes:
> 
>> On 04/03/2011 07:19 PM PJ Weisberg wrote:
>>> On 4/3/11, ken <gebser@mousecar.com> wrote:
>>>> On 03/28/2011 04:32 PM Stefan Monnier wrote:
>>>>> I'm just saying that
>>>>>
>>>>> M-: (some-regexp-search) RET
>>>>> M-: (replace-match ...) RET
>>>>>
>>>>> won't always do what you want, because a whole lot of code will run
>>>>> between the two.
>>>>>
>>>>>
>>>>>         Stefan
>>>> 
>>....
>>
>> So, yes, the search and replace-match are in the same function.  Are
>> they "close together"?  I'd say yes, but that's a judgment call really.
>>  The string I want to replace in this particular instance is a null string.
>>
> 
> Providing the code alway helps. While Stephan's points are valid and a
> common error people make, I don't think it is the problem you are
> seeing.

Yes, providing the actual code is generally a good idea... it gives
everyone something concrete to look at.  But the function I was writing
was too long to expect anyone to read and digest it in just a few
minutes; it would also require a lot of explanation from me to help
others to understand it; three or four ancillary functions play roles
and so they would have to be included also; all of this required a
specific kind of text file to work it on, so I would have need to
include that also; and besides all this, the code was a very unappealing
mess to look at, with all kinds of cruft commented out, sundry comments
inserted here and there, and some code not formatted in the traditional
fashion.  In short, it would have been too much work for everyone.  So I
thought just posting the directly relevant code fragments would be more
effective.  When there's a working package, a version 0.0001, I'll post
it on the web so anyone who's curious or wants to use it can do so.


> 
> It would help to see the regexp you are using to test this. Note that
> match-string returns nil if the subexpression you are asking for did not
> exist. If the subexpression did not exist, what would replace-match
> replace as there is no match?

Tim, this is a valid point generally, but not in this instance.  The
code included conditionals for each search and so if the search failed
to find the sought RE, a different block of code would be executed.  And
there was a block of code to cover each of the five possible conditions.


> 
> Note also, you can get rid of some of those progns i.e.
> 
>      (when (re-search-forward aname-re-str head-text-end-pt t)
>  	  (setq aname-str (match-string 2))
>  	  (setq head-text-nested (match-string 4))
>  	  (when (string= head-text-nested "")
>  		(setq head-text-nested (create-heading-text))
>                 (if (not (string= aname-str ""))
>  		     (replace-match head-text-nested t t nil 4)
>                   (setq aname-str (create-aname head-text-nested))
>  		  (replace-match head-text-nested t nil nil 4)
>  		  (replace-match aname-str t nil nil 2))))
> 
> Note that the 'else' part of an if form has an implicit progn, so by
> revesing the condition test, we can eliminate that progn. The 'unless'
> form can be used instead of 'when' if you want to reverse the logic of
> what you are doing. 

Yeah, inverting the conditionals does remove the progn statements and
makes the code shorter.  I read quite a bit about conditionals in some
of the docs... and someone here on this list explained about progn a bit
for me also.  But I wrote the code how I was thinking through the
problem, much in the way Kernighan and Ritchie say happens naturally...
I think it was in the "Introduction" to their book, "The C Programming
Language".  That "progn" statement does seem clunky... it took some
getting used to using an entire word just to denote the beginning of a
block of text.  But I've adapted.  In fact, now it's like a huge flag
telling me, 'this is where the block of text begins which executes when
the if statement above is true.'  It makes it easier to find those if's
 too.  Sometimes I almost wish there was a corresponding "<progn" to
just as boldly show me where the end of the code block is.  But it's too
late to design the syntax for lisp :) and I'd probably get tired of
"<progn" after a time anyway.


> 
> (Note above is untested, but intention should be clear)
> 
> My guess is that it does not work because replace-match has no idea what
> it is yhou want to match as the subexpression was not found i.e. is not
> in the string, so it has no way to know where in the string to do the
> replace.

When the subexpression isn't found, then the search fails.  When the
search fails, then a different code block executes, one which doesn't
require the existence of that subexpression.

In addition, I wrote a simple diagnostic function (a couple versions of
it actually) to show me what, if anything, was matched and inserted this
function at various points in the code.  This confirmed that the
searches and conditionals were functioning as I wanted.  Something else
was wiping out the subexpressions.  I eventually traced the problem to
the read-from-minibuffer function.  For some reason it wipes out
previously established subexpressions.  I wasn't expecting that at all.


> 
> Tim

Tim, thanks for all the input... and everyone else too.  Don't worry...
there's quite a bit more code I need to write, so I'll most likely be
posting more questions/problems in the upcoming.  (Is that something
anybody's worried about? :)  )

Re: error in replace-match: "args out of range"

[Tim X]

ken <gebser@mousecar.com> writes:

> On 04/04/2011 06:21 PM Tim X wrote:
>> ken <gebser@mousecar.com> writes:
>> 
>>> On 04/03/2011 07:19 PM PJ Weisberg wrote:
>>>> On 4/3/11, ken <gebser@mousecar.com> wrote:
>>>>> On 03/28/2011 04:32 PM Stefan Monnier wrote:
>>>>>> I'm just saying that
>>>>>>
>>>>>> M-: (some-regexp-search) RET
>>>>>> M-: (replace-match ...) RET
>>>>>>
>>>>>> won't always do what you want, because a whole lot of code will run
>>>>>> between the two.
>>>>>>
>>>>>>
>>>>>>         Stefan
>>>>> 
>>>....
>>>
>>> So, yes, the search and replace-match are in the same function.  Are
>>> they "close together"?  I'd say yes, but that's a judgment call really.
>>>  The string I want to replace in this particular instance is a null string.
>>>
>> 
>> Providing the code alway helps. While Stephan's points are valid and a
>> common error people make, I don't think it is the problem you are
>> seeing.
>
> Yes, providing the actual code is generally a good idea... it gives
> everyone something concrete to look at.  But the function I was writing
> was too long to expect anyone to read and digest it in just a few
> minutes; it would also require a lot of explanation from me to help
> others to understand it; three or four ancillary functions play roles
> and so they would have to be included also; all of this required a
> specific kind of text file to work it on, so I would have need to
> include that also; and besides all this, the code was a very unappealing
> mess to look at, with all kinds of cruft commented out, sundry comments
> inserted here and there, and some code not formatted in the traditional
> fashion.  In short, it would have been too much work for everyone.  So I
> thought just posting the directly relevant code fragments would be more
> effective.  When there's a working package, a version 0.0001, I'll post
> it on the web so anyone who's curious or wants to use it can do so.
>

An approach that can work is to boil down the problem to the smallest
bit of code that will reproduce the error and post that. To some extent,
I guess that is what you did. When asking for help with code related
problems, it really is the only way to go. 


>
>> 
>> It would help to see the regexp you are using to test this. Note that
>> match-string returns nil if the subexpression you are asking for did not
>> exist. If the subexpression did not exist, what would replace-match
>> replace as there is no match?
>
> Tim, this is a valid point generally, but not in this instance.  The
> code included conditionals for each search and so if the search failed
> to find the sought RE, a different block of code would be executed.  And
> there was a block of code to cover each of the five possible conditions.
>

The reason I asked for a copy of the regexp is that regexps can be
tricky and it really helps to also have, in addtion to the code, the
regexp being used as input when the code generates the error. 

For example, you can have a regexp with one or more  subexpression that does not
match while your overall regexp did match. According to the docs, in
that case, match-string would return nil when queried for the
subexpressions and not "". Note that "" and nil are not
the same. Depending on your regexp, both aname-str and head-text-nested
could have the value nil. However, your conditional statements that use
these variables only check for "", which is not the same as nil and
therefore could be returning true/false unexpectedly and causing the
wrong branch to be executed. 

This may or may not be an issue, depending on your regexp. 

>
>> 
>> Note also, you can get rid of some of those progns i.e.
>> 
>>      (when (re-search-forward aname-re-str head-text-end-pt t)
>>  	  (setq aname-str (match-string 2))
>>  	  (setq head-text-nested (match-string 4))
>>  	  (when (string= head-text-nested "")
>>  		(setq head-text-nested (create-heading-text))
>>                 (if (not (string= aname-str ""))
>>  		     (replace-match head-text-nested t t nil 4)
>>                   (setq aname-str (create-aname head-text-nested))
>>  		  (replace-match head-text-nested t nil nil 4)
>>  		  (replace-match aname-str t nil nil 2))))
>> 
>> Note that the 'else' part of an if form has an implicit progn, so by
>> revesing the condition test, we can eliminate that progn. The 'unless'
>> form can be used instead of 'when' if you want to reverse the logic of
>> what you are doing. 
>
> Yeah, inverting the conditionals does remove the progn statements and
> makes the code shorter.  I read quite a bit about conditionals in some
> of the docs... and someone here on this list explained about progn a bit
> for me also.  But I wrote the code how I was thinking through the
> problem, much in the way Kernighan and Ritchie say happens naturally...
> I think it was in the "Introduction" to their book, "The C Programming
> Language".  That "progn" statement does seem clunky... it took some
> getting used to using an entire word just to denote the beginning of a
> block of text.  But I've adapted.  In fact, now it's like a huge flag
> telling me, 'this is where the block of text begins which executes when
> the if statement above is true.'  It makes it easier to find those if's
>  too.  Sometimes I almost wish there was a corresponding "<progn" to
> just as boldly show me where the end of the code block is.  But it's too
> late to design the syntax for lisp :) and I'd probably get tired of
> "<progn" after a time anyway.

Another way to look at it is if you think you need progn, then maybe you
actually need a different conditional. This often makes your code even
more readable. For example (when something do-this-block) or (unless
something do-this-block) rather than (if something progn do-this-block)
or (if not something progn do this block). 

progn does have a role to play, but less than many realise. When first
learning lisp, it is one of those 'flags' you can use to alert you there
are better ways to express what you are trying to say. It also has the
advantage of less indentation and lines of code on the screen, making it
easier to 'see' the code in one glance. For list, this is quite
important as it is the ability to see the code and in particular, the
indentation to get a sense of the code.

>
>
>> 
>> (Note above is untested, but intention should be clear)
>> 
>> My guess is that it does not work because replace-match has no idea what
>> it is yhou want to match as the subexpression was not found i.e. is not
>> in the string, so it has no way to know where in the string to do the
>> replace.
>
> When the subexpression isn't found, then the search fails.  When the
> search fails, then a different code block executes, one which doesn't
> require the existence of that subexpression.
>

This was my point of warning. It is quite possible to have
regexps where sub expressions are not found, but the overall regexp
succeeds. This means that the regexp-search-* will succeed, but the
functions to extract the subexpressions will return nil. As we dind't
have an example of the regexp you are using, it is not possible to know
in this case if this is an issue or not. If your code will allow the
user to define the regexp, you would need to test for nil and not just "".

> In addition, I wrote a simple diagnostic function (a couple versions of
> it actually) to show me what, if anything, was matched and inserted this
> function at various points in the code.  This confirmed that the
> searches and conditionals were functioning as I wanted.  Something else
> was wiping out the subexpressions.  I eventually traced the problem to
> the read-from-minibuffer function.  For some reason it wipes out
> previously established subexpressions.  I wasn't expecting that at all.
>

I guess that was Stefan's point. You can not know what other functions
do 'under the hood' and therefore don't know when regexp searches are
performed and when the match data may be overridden. Therefore, you must
always assume the data is volatile and either use it immediately or
store the relevant details in local variables that you control or use
something like save-match-data etc

> Tim, thanks for all the input... and everyone else too.  Don't worry...
> there's quite a bit more code I need to write, so I'll most likely be
> posting more questions/problems in the upcoming.  (Is that something
> anybody's worried about? :)  )
>

Writing code is the only way to learn and in general, most people seem
happy to help anyone who is showing effort to learn and understand.
Besides, this is usernet, anyone who isn't interested can just skip
those messages.

Tim


-- 
tcross (at) rapttech dot com dot au

Re: error in replace-match: "args out of range"

[ken]

On 04/07/2011 07:25 PM Tim X wrote:
> ken <gebser@mousecar.com> writes:
> 
>> On 04/04/2011 06:21 PM Tim X wrote:
>>> ken <gebser@mousecar.com> writes:
>>>
>>>> On 04/03/2011 07:19 PM PJ Weisberg wrote:
>>>>> On 4/3/11, ken <gebser@mousecar.com> wrote:
>>>>>> On 03/28/2011 04:32 PM Stefan Monnier wrote:
>>>>>>> I'm just saying that
>>>>>>>
>>>>>>> M-: (some-regexp-search) RET
>>>>>>> M-: (replace-match ...) RET
>>>>>>>
>>>>>>> won't always do what you want, because a whole lot of code will run
>>>>>>> between the two.
>>>>>>>
>>>>>>>
>>>>>>>         Stefan
>>>> ....
>>>>
>>>> So, yes, the search and replace-match are in the same function.  Are
>>>> they "close together"?  I'd say yes, but that's a judgment call really.
>>>>  The string I want to replace in this particular instance is a null string.
>>>>
>>> ....
> 
>>> It would help to see the regexp you are using to test this. Note that
>>> match-string returns nil if the subexpression you are asking for did not
>>> exist. If the subexpression did not exist, what would replace-match
>>> replace as there is no match?
>> Tim, this is a valid point generally, but not in this instance.  The
>> code included conditionals for each search and so if the search failed
>> to find the sought RE, a different block of code would be executed.  And
>> there was a block of code to cover each of the five possible conditions.
>>
> 
> The reason I asked for a copy of the regexp is that regexps can be
> tricky and it really helps to also have, in addtion to the code, the
> regexp being used as input when the code generates the error. 
> 
> For example, you can have a regexp with one or more  subexpression that does not
> match while your overall regexp did match. According to the docs, in
> that case, match-string would return nil when queried for the
> subexpressions and not "". Note that "" and nil are not
> the same. Depending on your regexp, both aname-str and head-text-nested
> could have the value nil. However, your conditional statements that use
> these variables only check for "", which is not the same as nil and
> therefore could be returning true/false unexpectedly and causing the
> wrong branch to be executed. 
> 
> This may or may not be an issue, depending on your regexp. 
> 
>>> ....
>>>
>>> My guess is that it does not work because replace-match has no idea what
>>> it is yhou want to match as the subexpression was not found i.e. is not
>>> in the string, so it has no way to know where in the string to do the
>>> replace.
>> When the subexpression isn't found, then the search fails.  When the
>> search fails, then a different code block executes, one which doesn't
>> require the existence of that subexpression.
>>
> 
> This was my point of warning. It is quite possible to have
> regexps where sub expressions are not found, but the overall regexp
> succeeds. 

I'm having a hard time with that concept.  Is there a small bit of
example code you could show it with?


> This means that the regexp-search-* will succeed, but the
> functions to extract the subexpressions will return nil. As we dind't
> have an example of the regexp you are using, it is not possible to know
> in this case if this is an issue or not. If your code will allow the
> user to define the regexp, you would need to test for nil and not just "".
> 
>> ....

Re: error in replace-match: "args out of range"

[PJ Weisberg]

On 4/8/11, ken <gebser@mousecar.com> wrote:
> On 04/07/2011 07:25 PM Tim X wrote:
>> This was my point of warning. It is quite possible to have
>> regexps where sub expressions are not found, but the overall regexp
>> succeeds.
>
> I'm having a hard time with that concept.  Is there a small bit of
> example code you could show it with?

"I am\\( not\\)? a fish."

-PJ

Re: error in replace-match: "args out of range"

[ken]

On 04/09/2011 02:44 AM PJ Weisberg wrote:
> On 4/8/11, ken <gebser@mousecar.com> wrote:
>> On 04/07/2011 07:25 PM Tim X wrote:
>>> This was my point of warning. It is quite possible to have
>>> regexps where sub expressions are not found, but the overall regexp
>>> succeeds.
>> I'm having a hard time with that concept.  Is there a small bit of
>> example code you could show it with?
> 
> "I am\\( not\\)? a fish."
> 
> -PJ

And on what sort of string would that RE match but the subexpression
have the value nil (as opposed to the null string "")?

Re: error in replace-match: "args out of range"

[David Kastrup]

ken <gebser@mousecar.com> writes:

> On 04/09/2011 02:44 AM PJ Weisberg wrote:
>> On 4/8/11, ken <gebser@mousecar.com> wrote:
>>> On 04/07/2011 07:25 PM Tim X wrote:
>>>> This was my point of warning. It is quite possible to have
>>>> regexps where sub expressions are not found, but the overall regexp
>>>> succeeds.
>>> I'm having a hard time with that concept.  Is there a small bit of
>>> example code you could show it with?
>> 
>> "I am\\( not\\)? a fish."
>> 
>> -PJ
>
> And on what sort of string would that RE match but the subexpression
> have the value nil (as opposed to the null string "")?

Huh?  Isn't that rather obvious?

(list (string-match "I am\\( not\\)? a fish." "I am a fish!")
      (match-string 1)) => (0 nil)

-- 
David Kastrup

Re: error in replace-match: "args out of range"

[PJ Weisberg]

On 4/9/11, David Kastrup <dak@gnu.org> wrote:
> ken <gebser@mousecar.com> writes:
>
>> On 04/09/2011 02:44 AM PJ Weisberg wrote:
>>> On 4/8/11, ken <gebser@mousecar.com> wrote:
>>>> On 04/07/2011 07:25 PM Tim X wrote:
>>>>> This was my point of warning. It is quite possible to have
>>>>> regexps where sub expressions are not found, but the overall regexp
>>>>> succeeds.
>>>> I'm having a hard time with that concept.  Is there a small bit of
>>>> example code you could show it with?
>>>
>>> "I am\\( not\\)? a fish."
>>>
>>> -PJ
>>
>> And on what sort of string would that RE match but the subexpression
>> have the value nil (as opposed to the null string "")?
>
> Huh?  Isn't that rather obvious?
>
> (list (string-match "I am\\( not\\)? a fish." "I am a fish!")
>       (match-string 1)) => (0 nil)

Notice that the expression " not" can NEVER match an empty string.
The ? allows it to appear zero times, so the overall expression
matches even though the subexpression doesn't.

-PJ

Re: error in replace-match: "args out of range"

[Perry Smith]

On Apr 9, 2011, at 3:04 PM, PJ Weisberg wrote:

> On 4/9/11, David Kastrup <dak@gnu.org> wrote:
>> ken <gebser@mousecar.com> writes:
>> 
>>> On 04/09/2011 02:44 AM PJ Weisberg wrote:
>>>> On 4/8/11, ken <gebser@mousecar.com> wrote:
>>>>> On 04/07/2011 07:25 PM Tim X wrote:
>>>>>> This was my point of warning. It is quite possible to have
>>>>>> regexps where sub expressions are not found, but the overall regexp
>>>>>> succeeds.
>>>>> I'm having a hard time with that concept.  Is there a small bit of
>>>>> example code you could show it with?
>>>> 
>>>> "I am\\( not\\)? a fish."
>>>> 
>>>> -PJ
>>> 
>>> And on what sort of string would that RE match but the subexpression
>>> have the value nil (as opposed to the null string "")?
>> 
>> Huh?  Isn't that rather obvious?
>> 
>> (list (string-match "I am\\( not\\)? a fish." "I am a fish!")
>>      (match-string 1)) => (0 nil)
> 
> Notice that the expression " not" can NEVER match an empty string.
> The ? allows it to appear zero times, so the overall expression
> matches even though the subexpression doesn't.

Also, "" does not match the subexpression.  I don't know but that may be key as well.

Re: error in replace-match: "args out of range"

[Stefan Monnier]

> Here's the relevant code fragment:

>     (if (re-search-forward aname-re-str head-text-end-pt t)
> 	(progn	
> 	  (setq aname-str (match-string 2))
> 	  (setq head-text-nested (match-string 4))
> 	  (if (equal head-text-nested "")
> 	      (progn
> 		(setq head-text-nested (create-heading-text))
> 		(if (equal aname-str "")
> 		    (progn
> 		      (setq aname-str (create-aname head-text-nested))
> 		      (replace-match head-text-nested t nil nil 4)
> 		      (replace-match aname-str t nil nil 2))
> 		  (replace-match head-text-nested t t nil 4))) ;ERROR:

An obvious part to heck is create-heading-text: if it might ever call
a regexp-matching function, then you have a problem.


        Stefan

Re: error in replace-match: "args out of range" [SOLVED]

[ken]

On 04/04/2011 09:56 PM Stefan Monnier wrote:
>> Here's the relevant code fragment:
> 
>>     (if (re-search-forward aname-re-str head-text-end-pt t)
>> 	(progn	
>> 	  (setq aname-str (match-string 2))
>> 	  (setq head-text-nested (match-string 4))
>> 	  (if (equal head-text-nested "")
>> 	      (progn
>> 		(setq head-text-nested (create-heading-text))
>> 		(if (equal aname-str "")
>> 		    (progn
>> 		      (setq aname-str (create-aname head-text-nested))
>> 		      (replace-match head-text-nested t nil nil 4)
>> 		      (replace-match aname-str t nil nil 2))
>> 		  (replace-match head-text-nested t t nil 4))) ;ERROR:
> 
> An obvious part to heck is create-heading-text: if it might ever call
> a regexp-matching function, then you have a problem.
> 
> 
>         Stefan

Stefan,

You pointed to the right function... well, it was actually both.  But
the problem wasn't a search, but with the read-from-minibuffer function.
 For some reason, invoking that wipes out all the subexpressions.

Re: error in replace-match: "args out of range" [SOLVED]

[ken]

On 04/05/2011 11:39 PM ken wrote:
> ....
> the problem wasn't a search, but with the read-from-minibuffer function.
>  For some reason, invoking that wipes out all the subexpressions.

I don't recall reading about this.  If it's not already in the official
documentation, how do we add it in?

Re: error in replace-match: "args out of range" [SOLVED]

[PJ Weisberg]

M-x report-emacs-bug is probably the best way.  The docstrings for
match-string and replace-match should emphasize that ANY code that
deals with strings--notably code that gives control to the user--is
liable to replace the "last matched" string, so those functions should
be called right away after you do the match.

On 4/7/11, ken <gebser@mousecar.com> wrote:
>
> On 04/05/2011 11:39 PM ken wrote:
>> ....
>> the problem wasn't a search, but with the read-from-minibuffer function.
>>  For some reason, invoking that wipes out all the subexpressions.
>
> I don't recall reading about this.  If it's not already in the official
> documentation, how do we add it in?
>
>
>


-- 

-PJ

Re: error in replace-match: "args out of range" [SOLVED]

[David Combs]

In article <mailman.14.1302203359.27822.help-gnu-emacs@gnu.org>,
PJ Weisberg  <pj@irregularexpressions.net> wrote:
>M-x report-emacs-bug is probably the best way.  The docstrings for
>match-string and replace-match should emphasize that ANY code that
>deals with strings--notably code that gives control to the user--is
>liable to replace the "last matched" string, so those functions should
>be called right away after you do the match.
>
>On 4/7/11, ken <gebser@mousecar.com> wrote:
>>
>> On 04/05/2011 11:39 PM ken wrote:
>>> ....
>>> the problem wasn't a search, but with the read-from-minibuffer function.
>>>  For some reason, invoking that wipes out all the subexpressions.
>>
>> I don't recall reading about this.  If it's not already in the official
>> documentation, how do we add it in?

Or maybe a "with-last-matched-string-preserved" or something like it?

David

Re: error in replace-match: "args out of range" [SOLVED]

[Stefan Monnier]

> Or maybe a "with-last-matched-string-preserved" or something like it?

You mean save-match-data?


        Stefan

Re: error in replace-match: "args out of range" [SOLVED]

[David Combs]

In article <jwv39l8443e.fsf-monnier+gnu.emacs.help@gnu.org>,
Stefan Monnier  <monnier@iro.umontreal.ca> wrote:
>> Or maybe a "with-last-matched-string-preserved" or something like it?
>
>You mean save-match-data?
>
>
>        Stefan

Yes indeed.  Thanks!

David

Re: error in replace-match: "args out of range" [SOLVED]

[Stefan Monnier]

>> the problem wasn't a search, but with the read-from-minibuffer function.
>> For some reason, invoking that wipes out all the subexpressions.
> I don't recall reading about this.  If it's not already in the official
> documentation, how do we add it in?

There are two problems:

- you expect the doc to mention the functions that wipe-out the
  match-data (whereas what really happens is that only a few functions
  can be assumed to preserve that data and all others should be assumed
  to wipe it).  Hopefully this thread will fix this ;-)
- Emacs does not document which functions can be trusted not to wipe the
  match-data :-(


        Stefan

error in replace-match: "args out of range"

[ken]

replace-match works fine in a couple instances in one function, but this
instance (in the same function) it doesn't.  I get the error msg below
about the arguments being out of range.  I'm guessing the problem is
with the 4th arg, the one beginning: #("<a ... yes?  There's a lot of
"(fontified ..." cruft in it... is that confusing replace-match...?  Or
might I have just the totally wrong type arg there?

Debugger entered--Lisp error: (args-out-of-range -1 -1)
  replace-match("Text for Heading" t nil #("<a
name=\"Testing-TOC-Header\"></a>" 0 1 (fontified t face
font-lock-function-name-face) 1 2 (fontified t face html-tag-face) 2 3
(fontified t) 3 8 (fontified t face font-lock-variable-name-face) 8 9
(fontified t face font-lock-string-face) 9 27 (fontified t face
font-lock-string-face) 27 28 (fontified t face font-lock-string-face) 28
29 (fontified t face font-lock-function-name-face) 29 30 (fontified t
face font-lock-function-name-face) 30 32 (fontified t face
html-tag-face) 32 33 (fontified t face font-lock-function-name-face)) 4)
  eval((replace-match head-text-nested t nil head-text 4))


-- 
Anything is easy if you know how to do it.

Re: error in replace-match: "args out of range"

[ken]

On 03/27/2011 08:02 AM ken wrote:
> replace-match works fine in a couple instances in one function, but this
> instance (in the same function) it doesn-- 't.  I get the error msg below
> about the arguments being out of range.  I'm guessing the problem is
> with the 4th arg, the one beginning: #("<a ... yes?  There's a lot of
> "(fontified ..." cruft in it... is that confusing replace-match...?  Or
> might I have just the totally wrong type arg there?
> 
> Debugger entered--Lisp error: (args-out-of-range -1 -1)
>   replace-match("Text for Heading" t nil #("<a
> name=\"Testing-TOC-Header\"></a>" 0 1 (fontified t face
> font-lock-function-name-face) 1 2 (fontified t face html-tag-face) 2 3
> (fontified t) 3 8 (fontified t face font-lock-variable-name-face) 8 9
> (fontified t face font-lock-string-face) 9 27 (fontified t face
> font-lock-string-face) 27 28 (fontified t face font-lock-string-face) 28
> 29 (fontified t face font-lock-function-name-face) 29 30 (fontified t
> face font-lock-function-name-face) 30 32 (fontified t face
> html-tag-face) 32 33 (fontified t face font-lock-function-name-face)) 4)
>   eval((replace-match head-text-nested t nil head-text 4))
> 
> 

Still groping for a resolution....   Here's something that might be
relevant.

The immediately preceding re-search-forward was successful.  However,
the subexpression which matched (#4) is a null string, i.e., "".  So the
question: Will replace-match replace an empty string with specified
text...? in effect *inserting* text at the location of the specified
(albeit nil) subexpression?


-- 
Anything is easy if you know how to do it.

Re: error in replace-match: "args out of range"

[Alan Mackenzie]

Hi, Ken.

ken <gebser@mousecar.com> wrote:
> 
> On 03/27/2011 08:02 AM ken wrote:
>> replace-match works fine in a couple instances in one function, but this
>> instance (in the same function) it doesn-- 't.  I get the error msg below
>> about the arguments being out of range.  I'm guessing the problem is
>> with the 4th arg, the one beginning: #("<a ... yes?  There's a lot of
>> "(fontified ..." cruft in it... is that confusing replace-match...?  Or
>> might I have just the totally wrong type arg there?

>> Debugger entered--Lisp error: (args-out-of-range -1 -1)
>>   replace-match("Text for Heading" t nil #("<a
>> name=\"Testing-TOC-Header\"></a>" 0 1 (fontified t face
>> font-lock-function-name-face) 1 2 (fontified t face html-tag-face) 2 3
>> (fontified t) 3 8 (fontified t face font-lock-variable-name-face) 8 9
>> (fontified t face font-lock-string-face) 9 27 (fontified t face
>> font-lock-string-face) 27 28 (fontified t face font-lock-string-face) 28
>> 29 (fontified t face font-lock-function-name-face) 29 30 (fontified t
>> face font-lock-function-name-face) 30 32 (fontified t face
>> html-tag-face) 32 33 (fontified t face font-lock-function-name-face)) 4)
>>   eval((replace-match head-text-nested t nil head-text 4))



> Still groping for a resolution....   Here's something that might be
> relevant.

> The immediately preceding re-search-forward was successful.  However,
> the subexpression which matched (#4) is a null string, i.e., "".  So the
> question: Will replace-match replace an empty string with specified
> text...? in effect *inserting* text at the location of the specified
> (albeit nil) subexpression?

You'll be far more likely to get help if you give some basic details.
WhIch Emacs version are you using (M-x emacs-version)?  What major mode
are you in?  What minor modes (if any) are active?  What did you ACTUALLY
DO (exactly which keys did you press) to cause the error.  Can you make
the error happen in a small enough buffer to post?  If so, do it!

The idea is to help somebody reproduce the error with the minimum amount
of work.

Incidentally, there's a section on bug reporting in the Emacs Manual on
page "Bugs".  To get to it, do C-h i, m emacs, g bugs.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: error in replace-match: "args out of range"

[ken]

On 03/27/2011 03:07 PM Alan Mackenzie wrote:
> Hi, Ken.
> 
> ken <gebser@mousecar.com> wrote:
>> On 03/27/2011 08:02 AM ken wrote:
>>> replace-match works fine in a couple instances in one function, but this
>>> instance (in the same function) it doesn-- 't.  I get the error msg below
>>> about the arguments being out of range.  I'm guessing the problem is
>>> with the 4th arg, the one beginning: #("<a ... yes?  There's a lot of
>>> "(fontified ..." cruft in it... is that confusing replace-match...?  Or
>>> might I have just the totally wrong type arg there?
> 
>>> Debugger entered--Lisp error: (args-out-of-range -1 -1)
>>>   replace-match("Text for Heading" t nil #("<a
>>> name=\"Testing-TOC-Header\"></a>" 0 1 (fontified t face
>>> font-lock-function-name-face) 1 2 (fontified t face html-tag-face) 2 3
>>> (fontified t) 3 8 (fontified t face font-lock-variable-name-face) 8 9
>>> (fontified t face font-lock-string-face) 9 27 (fontified t face
>>> font-lock-string-face) 27 28 (fontified t face font-lock-string-face) 28
>>> 29 (fontified t face font-lock-function-name-face) 29 30 (fontified t
>>> face font-lock-function-name-face) 30 32 (fontified t face
>>> html-tag-face) 32 33 (fontified t face font-lock-function-name-face)) 4)
>>>   eval((replace-match head-text-nested t nil head-text 4))
> 
> 
>> Still groping for a resolution....   Here's something that might be
>> relevant.
> 
>> The immediately preceding re-search-forward was successful.  However,
>> the subexpression which matched (#4) is a null string, i.e., "".  So the
>> question: Will replace-match replace an empty string with specified
>> text...? in effect *inserting* text at the location of the specified
>> (albeit nil) subexpression?
> 


Hi, Alan,

Sorry, I forget that you guys aren't here in my livingroom and uptodate
on what I'm doing.  So, yeah, there's a bit of context I should have given.

> You'll be far more likely to get help if you give some basic details.
> WhIch Emacs version are you using (M-x emacs-version)?  

v.23.1.1  YSU, I know.

> What major mode
> are you in?  What minor modes (if any) are active?  What did you ACTUALLY
> DO (exactly which keys did you press) to cause the error.  

This is an emacs script I'm writing (so in emacs-lisp mode); sometimes I
run this in edebug.  The script is meant to run on HTML files of
whatever mode; I'm not requiring that it be run in any particular mode,
just valid html.  I use html-helper-mode, but that's probably irrelevant
to this problem.


> Can you make
> the error happen in a small enough buffer to post?  If so, do it!
> 
> The idea is to help somebody reproduce the error with the minimum amount
> of work.

That would be hard.  The little bit of code above is buried in the
middle of a function with enough going on inside that it would take too
much explanation... people would run away from all I'd have to write
about it.


> 
> Incidentally, there's a section on bug reporting in the Emacs Manual on
> page "Bugs".  To get to it, do C-h i, m emacs, g bugs.

Thanks, I'll keep that in mind.  I'm not saying this is a bug though.  I
don't know if what I'm getting is normal behavior or not.  I'm more
asking after non-documented behavior of replace-match.  Or maybe I
should be using a different function.


Thanks for the pointers.