bug#33167: 26; Doc string of `region-extract-function'

bug#33167: 26; Doc string of `region-extract-function'

[Drew Adams]

I don't understand these parts of the description:

 - nil: return the content as a string.

What does that mean, for a noncontiguous region?  Are the buffer
substrings for all of the region segments (zones) concatenated together?
If so, say that.

 - anything else: delete the region and return its content
   as a string, after filtering it with `filter-buffer-substring', which
   is called with METHOD as its 3rd argument.

So first the region is deleted, and the deleted content (see previous)
is filtered via `(filter-buffer-substring BEG END METHOD)' and then
returned.

Some of what I don't understand:

1. What are the BEG and END args passed to `filter-buffer-substring'?
   Is BEG the smallest car of any of the zones in the noncontiguous
   region, and END the largest cdr of any of the zones?

2. `filter-buffer-substring' calls the value of
   `filter-buffer-substring-function' with the same 3 args.  But what
   can that function do with BEG and END (which are what?)?  It's
   presumably a function that expects to use a single stretch of buffer
   text from BEG to END.  But here we're talking about a noncontiguous 

3. The 3rd arg to `filter-buffer-substring' just deletes the region from
   BEG to END if it is non-nil, so it seems like passing that non-nil
   3rd arg is useless, as the region gets deleted anyway, by
   `region-extract-function'.

4. The use of `filter-buffer-substring' is also unclear.  It is passed
   BEG and END (and METHOD, but see #3, above).  And it filters the
   buffer text between BEG and END.  But see #1 above - are BEG and END
   buffer positions that make sense for the whole region text?  Just
   what happens here?

This is quite unclear to me.  And following the rabbit hole from
`region-extract-function' down to `filter-buffer-substring' and then to
`filter-buffer-substring-function' does not make things more clear.

Or is what happens perhaps that EACH element of the noncontiguous
region, that is, each zone (BEG<N> . END<N>) of the list ((BEG1 . END1)
...) gets filtered by `filter-buffer-substring', passing its BEG END and
EMTHOD - so that a mapcar is applied?  In that case, how are the
resulting buffer substrings assembled - are they concatenated to get the
return value?

In GNU Emacs 26.1 (build 1, x86_64-w64-mingw32)
 of 2018-05-30
Repository revision: 07f8f9bc5a51f5aa94eb099f3e15fbe0c20ea1ea
Windowing system distributor `Microsoft Corp.', version 10.0.16299
Configured using:
 `configure --without-dbus --host=x86_64-w64-mingw32
 --without-compress-install 'CFLAGS=-O2 -static -g3''

bug#33167: 26; Doc string of `region-extract-function'

[Eli Zaretskii]

> Date: Fri, 26 Oct 2018 08:32:04 -0700 (PDT)
> From: Drew Adams <drew.adams@oracle.com>
> 
>  - nil: return the content as a string.
> 
> What does that mean, for a noncontiguous region?  Are the buffer
> substrings for all of the region segments (zones) concatenated together?
> If so, say that.

No, the value is a list of strings in this case.  I fixed the doc
string to say that.

> 1. What are the BEG and END args passed to `filter-buffer-substring'?
>    Is BEG the smallest car of any of the zones in the noncontiguous
>    region, and END the largest cdr of any of the zones?

They are the first and the last positions of the region for
filter-buffer-substring to act upon.  That function is not supposed to
process non-contiguous regions.

> 2. `filter-buffer-substring' calls the value of
>    `filter-buffer-substring-function' with the same 3 args.  But what
>    can that function do with BEG and END (which are what?)?  It's
>    presumably a function that expects to use a single stretch of buffer
>    text from BEG to END.  But here we're talking about a noncontiguous 

No, we are talking about contiguous regions when this function is
concerned.

> 3. The 3rd arg to `filter-buffer-substring' just deletes the region from
>    BEG to END if it is non-nil, so it seems like passing that non-nil
>    3rd arg is useless, as the region gets deleted anyway, by
>    `region-extract-function'.

Not sure what is the problem here.

> 4. The use of `filter-buffer-substring' is also unclear.  It is passed
>    BEG and END (and METHOD, but see #3, above).  And it filters the
>    buffer text between BEG and END.  But see #1 above - are BEG and END
>    buffer positions that make sense for the whole region text?  Just
>    what happens here?

See above.  It is not the job of filter-buffer-substring to DTRT when
the region is non-contiguous, it is the job of its callers.  See
rect.el for one example.

> This is quite unclear to me.  And following the rabbit hole from
> `region-extract-function' down to `filter-buffer-substring' and then to
> `filter-buffer-substring-function' does not make things more clear.

It isn't supposed to.  If you want to see how non-contiguous regions
are used in this context, you need to look in places that do so.

> Or is what happens perhaps that EACH element of the noncontiguous
> region, that is, each zone (BEG<N> . END<N>) of the list ((BEG1 . END1)
> ...) gets filtered by `filter-buffer-substring', passing its BEG END and
> METHOD

Yes!

> - so that a mapcar is applied?

Not literally, but the result is a list, yes.

> In that case, how are the resulting buffer substrings assembled -
> are they concatenated to get the return value?

No, you get a list.  Again, see rect.el.

Bottom line: I fixed the doc string of region-extract-function to say
what are the values when the region is non-contiguous, and how
filter-buffer-substring is invoked in that case.  But I don't think
there's anything else that should be done here, because the details of
using this facility for non-contiguous regions is entirely up to the
Lisp program which implements such a feature.

Thanks.

bug#33167: 26; Doc string of `region-extract-function'

[Drew Adams]

> >  - nil: return the content as a string.
> >
> > What does that mean, for a noncontiguous region?  Are the buffer
> > substrings for all of the region segments (zones) concatenated
> together?
> > If so, say that.
> 
> No, the value is a list of strings in this case.  I fixed the doc
> string to say that.

Good.  Thanks.  So with nil the value is a list of strings,
not a string that is "the content" of the noncontiguous region.

> > 1. What are the BEG and END args passed to `filter-buffer-substring'?
> >    Is BEG the smallest car of any of the zones in the noncontiguous
> >    region, and END the largest cdr of any of the zones?
> 
> They are the first and the last positions of the region for
> filter-buffer-substring to act upon.  That function is not supposed to
> process non-contiguous regions.

Yes, but where do they come from, for that call?  How do they
relate to, or how are they derived from, the noncontiguous
region? Are they point and mark?  Something else?  That's the
question.

The only arg to the function that is the value of
`region-extract-function` is METHOD.  The "region's content"
is presumably the content of the noncontiguous region.  And
presumably it is that that is "the content" that is returned
as a string.  How is that string derived from the noncontiguous
region?  That's the question.

No connection is made in this doc, AFAICT, between the
noncontiguous region and the BEG and END that are passed
to `filter-buffer-substring'.

> > 2. `filter-buffer-substring' calls the value of
> >    `filter-buffer-substring-function' with the same 3 args.  But what
> >    can that function do with BEG and END (which are what?)?  It's
> >    presumably a function that expects to use a single stretch of
> >    buffer text from BEG to END.  But here we're talking about a 
> >    noncontiguous
> 
> No, we are talking about contiguous regions when this function is
> concerned.

Sorry, but I don't understand.  What contiguous regions
are we talking about?  How/where in this doc did you
get from a noncontiguous region to contiguous regions?

I haven't see your update to the doc string, but so far
your description here doesn't give the impression that
these questions have been cleared up.

> > 3. The 3rd arg to `filter-buffer-substring' just deletes the region
> >    from BEG to END if it is non-nil, so it seems like passing that
> >    non-nil 3rd arg is useless, as the region gets deleted anyway, by
> >    `region-extract-function'.
> 
> Not sure what is the problem here.

Not sure what you're saying.  Are you saying that I'm wrong
that a non-nil 3rd arg just deletes the region?  Or are you
saying that I'm wrong that that non-nil 3rd arg is useless
because the region is deleted anyway?  This doc is not clear
to me.

> > 4. The use of `filter-buffer-substring' is also unclear.  It is passed
> >    BEG and END (and METHOD, but see #3, above).  And it filters the
> >    buffer text between BEG and END.  But see #1 above - are BEG and
> >    END buffer positions that make sense for the whole region text?
> >    Just what happens here?
> 
> See above.  It is not the job of filter-buffer-substring to DTRT when
> the region is non-contiguous, it is the job of its callers.  See
> rect.el for one example.

See above.  What are BEG and END that get passed to it?
The doc should be able to make clear what the behavior
is, without someone needing to investigate the code of
rect.el.  Plus, noncontiguous regions are more general
than rectangles.  The doc should make clear what happens
in the general case.

> > This is quite unclear to me.  And following the rabbit hole from
> > `region-extract-function' down to `filter-buffer-substring' and then
> > to `filter-buffer-substring-function' does not make things more clear.
> 
> It isn't supposed to.  If you want to see how non-contiguous regions
> are used in this context, you need to look in places that do so.

The doc should make clear in general terms what extracting
a noncontiguous region is about, and just what the value of
`region-extract-function' does: what its arguments are and
what its return value is (as a function of those arguments).

If someone has to look at the (very few, and particular
to rectangles, BTW) code where `region-extract-function'
is currently used to try to understand it, and the doc
doesn't help but at best misleads, then we might as well
have no doc for it.

I did what you suggest before filing the bug report:
checked all of the existing occurrences.  I felt that
the doc is wrong wrt what really happens, and I feel
that there's a need for a general description of this
variable which helps.

> > Or is what happens perhaps that EACH element of the noncontiguous
> > region, that is, each zone (BEG<N> . END<N>) of the list ((BEG1 .
> > END1) ...) gets filtered by `filter-buffer-substring', passing
> > its BEG END and METHOD
> 
> Yes!

Then please say that in the doc.  The function iterates
over the zones of the noncontiguous region, doing ___.

> > - so that a mapcar is applied?
> 
> Not literally, but the result is a list, yes.

If you haven't already done so, please say that in the doc.

> > In that case, how are the resulting buffer substrings assembled -
> > are they concatenated to get the return value?
> 
> No, you get a list.  Again, see rect.el.
> 
> Bottom line: I fixed the doc string of region-extract-function to say
> what are the values when the region is non-contiguous, and how
> filter-buffer-substring is invoked in that case.  But I don't think
> there's anything else that should be done here, because the details of
> using this facility for non-contiguous regions is entirely up to the
> Lisp program which implements such a feature.

Sounds like you've already corrected some, maybe all, of
what was incorrect.  See above, in case you were unclear
about some of the problems with it and you want to make
some of that clearer.

Thanks for looking at this.

bug#33167: 26; Doc string of `region-extract-function'

[Eli Zaretskii]

> Date: Sat, 27 Oct 2018 08:27:41 -0700 (PDT)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: 33167-done@debbugs.gnu.org
> 
> > > 1. What are the BEG and END args passed to `filter-buffer-substring'?
> > >    Is BEG the smallest car of any of the zones in the noncontiguous
> > >    region, and END the largest cdr of any of the zones?
> > 
> > They are the first and the last positions of the region for
> > filter-buffer-substring to act upon.  That function is not supposed to
> > process non-contiguous regions.
> 
> Yes, but where do they come from, for that call?  How do they
> relate to, or how are they derived from, the noncontiguous
> region? Are they point and mark?  Something else?  That's the
> question.

It's entirely up to the region-extract-function that handles such
regions.  The default one doesn't.  So I don't see why these questions
need to be answered in the doc string of filter-buffer-substring, they
don't belong there and would only confuse.

> The only arg to the function that is the value of
> `region-extract-function` is METHOD.  The "region's content"
> is presumably the content of the noncontiguous region.  And
> presumably it is that that is "the content" that is returned
> as a string.  How is that string derived from the noncontiguous
> region?  That's the question.

It isn't.  The region passed to filter-buffer-substring must always be
contiguous.

> > > 2. `filter-buffer-substring' calls the value of
> > >    `filter-buffer-substring-function' with the same 3 args.  But what
> > >    can that function do with BEG and END (which are what?)?  It's
> > >    presumably a function that expects to use a single stretch of
> > >    buffer text from BEG to END.  But here we're talking about a 
> > >    noncontiguous
> > 
> > No, we are talking about contiguous regions when this function is
> > concerned.
> 
> Sorry, but I don't understand.  What contiguous regions
> are we talking about?  How/where in this doc did you
> get from a noncontiguous region to contiguous regions?

That depends on the implementation of region-extract-function that
supports non-contiguous regions: it must somehow call
filter-buffer-substring-function in a way that this function gets only
contiguous regions of text.  So this question cannot be answered in
the doc string of filter-buffer-substring-function.

> I haven't see your update to the doc string, but so far
> your description here doesn't give the impression that
> these questions have been cleared up.

Well, I suggest to read that first:

  http://git.savannah.gnu.org/cgit/emacs.git/commit/?h=emacs-26&id=df64da8eb845c9f07ee93bfbf28af41a01a2e83f

> > > 3. The 3rd arg to `filter-buffer-substring' just deletes the region
> > >    from BEG to END if it is non-nil, so it seems like passing that
> > >    non-nil 3rd arg is useless, as the region gets deleted anyway, by
> > >    `region-extract-function'.
> > 
> > Not sure what is the problem here.
> 
> Not sure what you're saying.

You say that passing the 3rd arg is "useless", and I don't understand
why.

> Are you saying that I'm wrong that a non-nil 3rd arg just deletes
> the region?  Or are you saying that I'm wrong that that non-nil 3rd
> arg is useless because the region is deleted anyway?

I'm saying that I don't understand what you wanted to say here.  At
all.

> > > 4. The use of `filter-buffer-substring' is also unclear.  It is passed
> > >    BEG and END (and METHOD, but see #3, above).  And it filters the
> > >    buffer text between BEG and END.  But see #1 above - are BEG and
> > >    END buffer positions that make sense for the whole region text?
> > >    Just what happens here?
> > 
> > See above.  It is not the job of filter-buffer-substring to DTRT when
> > the region is non-contiguous, it is the job of its callers.  See
> > rect.el for one example.
> 
> See above.

See above.

> What are BEG and END that get passed to it?

The doc string already tells that: the beginning and the end of the
region to be filtered.

> The doc should be able to make clear what the behavior
> is, without someone needing to investigate the code of
> rect.el.

It does.  What it does NOT have to do is tell how one would use this
function while implementing a replacement for the default
region-extract-function that can handle non-contiguous regions.

> Plus, noncontiguous regions are more general than rectangles.

Exactly.  So each such implementation has to figure out how to call
filter-buffer-substring in a way that makes sense and passes only
contiguous regions to filter-buffer-substring.

> The doc should make clear what happens in the general case.

The doc string of filter-buffer-substring needs to tell what that
function does.  And it does precisely that.

> The doc should make clear in general terms what extracting
> a noncontiguous region is about, and just what the value of
> `region-extract-function' does: what its arguments are and
> what its return value is (as a function of those arguments).

It does now, please see the new doc string.

> I did what you suggest before filing the bug report:
> checked all of the existing occurrences.  I felt that
> the doc is wrong wrt what really happens, and I feel
> that there's a need for a general description of this
> variable which helps.

The doc string was indeed incomplete before my changes; it is IMO
complete now.

> > > Or is what happens perhaps that EACH element of the noncontiguous
> > > region, that is, each zone (BEG<N> . END<N>) of the list ((BEG1 .
> > > END1) ...) gets filtered by `filter-buffer-substring', passing
> > > its BEG END and METHOD
> > 
> > Yes!
> 
> Then please say that in the doc.

NO!  Because that's just one possible implementation, for one possible
kind of non-contiguous regions.

> > > - so that a mapcar is applied?
> > 
> > Not literally, but the result is a list, yes.
> 
> If you haven't already done so, please say that in the doc.

If you haven't already done so, please read the new doc string.  I
think I covered everything that should be covered.

bug#33167: 26; Doc string of `region-extract-function'

[Drew Adams]

> > > > 1. What are the BEG and END args passed to `filter-buffer-
> > > >    substring'?
> > > >    Is BEG the smallest car of any of the zones in the
> > > >    noncontiguous
> > > >    region, and END the largest cdr of any of the zones?
> > >
> > > They are the first and the last positions of the region for
> > > filter-buffer-substring to act upon.  That function is not supposed
> to
> > > process non-contiguous regions.
> >
> > Yes, but where do they come from, for that call?  How do they
> > relate to, or how are they derived from, the noncontiguous
> > region? Are they point and mark?  Something else?  That's the
> > question.
> 
> It's entirely up to the region-extract-function that handles such
> regions.  The default one doesn't.  So I don't see why these questions
> need to be answered in the doc string of filter-buffer-substring, they
> don't belong there and would only confuse.

OK.  I guess you're saying that the `region-extract-function'
must invoke `filter-buffer-substring' on the content of each
piece of the noncontiguous region, but it can pass whatever
BEG and END positions it wants, to `filter-buffer-substring'.
There is no specified relation between those values it passes
and the BEG and END of each piece.

That makes little sense to me, so far.

I see that in the rectangle case it does decide what BEG and
END values to use, and these do not necessarily correspond to
the limits of the pieces.

In any case, that the use of the var for rectangles might
filter each line of the region in the way that it does says
nothing about what another value of the variable, in some
other context might do.

But if the spec of the variable imposes nothing on the BEG
and END values passed for filtering, then why does it impose
a requirement that `filter-buffer-substring' must be called?

The doc string of `region-extract-function' says that it
returns the content of the region as a string, after
filtering with `filter-buffer-substring' (in the "anything
else" case).

(You have presumably corrected that to say that it returns
a list of such strings - after each is filtered.)

If everything were up to the function that is the value of
`region-extract-function' then there would be no need, and
it would be incorrect, to state that it must filter buffer
content by calling `filter-buffer-substring'.

If, however, it is a _requirement_ that it calls
`filter-buffer-substring' then we need to know what args
it passes for BEG and END, in addition to knowing that it
needs to pass METHOD as the 3rd arg.

Otherwise, the need to use `filter-buffer-substring' is
a hollow requirement (no requirement at all).  If there is
no limit on what it can pass as BEG and END then I do not
see why we say that it must call `filter-buffer-substring'.
Whether it calls that function or not, it can, it seems,
do anything it likes, as long as it returns, for each
piece, some subsequence of that piece.

Worse: If there is no requirement on what it can pass
as BEG and END - not necessarily any relation between
those args passed and the limits of the pieces of the
noncontiguous region, then it can return any list of
subsequences of the buffer - not necessarily related
to the noncontiguous region at all.

I truly do not understand the intended (specified)
behavior of a function that can be the value of this
variable.  It's not enough to look at the rectangle
example - that doesn't specify what, in general, can
and cannot be a legitimate value of the variable.

My impression, so far, is that the actual requirement
for the function (definition of a proper value for the
variable),for the "anything else" case, is just that
the strings in the returned list must be subsequences
of the pieces of the noncontiguous region.

> > Not sure what you're saying.
> 
> You say that passing the 3rd arg is "useless", and
> I don't understand why.

I said:

 The 3rd arg to `filter-buffer-substring' just deletes the
 region from BEG to END if it is non-nil, so it seems like
 passing that non-nil 3rd arg is useless, as the region
 gets deleted anyway, by `region-extract-function'.

In the "anything else" case the `region-extract-function'
must delete the region - the whole noncontiguous region.
How it does that is its business presumably.

Since it deletes the whole noncontig region anyway, why
must it also pass METHOD to `filter-buffer-substring'?
The only reason to pass a non-nil value as 3rd arg is to
delete the text between the first two args passed.

But see above.  You've now said, IIUC, that it can pass
_any_ BEG and END values to `filter-buffer-substring' -
those values need not have anything to do with the region
or its pieces.  So is the point of passing METHOD to
`filter-buffer-substring' to be able to delete additional
text, which is not in the (noncontiguous) region?  What's
it about?

> > What are BEG and END that get passed to it?
> 
> The doc string already tells that: the beginning and
> the end of the region to be filtered.

No, the doc string says nothing about the BEG and END
values that get passed to `filter-buffer-substring'.
And you pointed out that those values passed need not
have any relation to the BEG and END limits of the pieces
of the noncontiguous region.  And it's certainly true,
e.g. for rectangles, that they need not be _equal_ to
the piece limits.

>>>> Or is what happens perhaps that EACH element of
>>>> the noncontiguous region, that is, each zone
>>>> (BEG<N> . END<N>) of the list ((BEG1 . END1) ...)
>>>> gets filtered by `filter-buffer-substring', passing
>>>> its BEG END and METHOD
> > >
> > > Yes!
> >
> > Then please say that in the doc.
> 
> NO!  Because that's just one possible implementation,
> for one possible kind of non-contiguous regions.

I didn't ask about one possible kind.  I asked if
that's what happens for any noncontiguous region.
You replied "Yes!", so I requested that that info be
added to the doc.

Now, and above IIUC, you've said that it's not true
for all non-contig regions.  From what you've said
above, it can pass _any_ BEG and END values it wants
for any given piece of the noncontig region.

But at least the doc can say, and it does now say,
that `filter-buffer-substring' is called once for
each piece.  What it does not say is that it need
not be called with the limits of that piece - it
can be called with any two buffer positions
(apparently).

I find it hard to believe that we would both (1)
insist that the function (in the "anything else"
case) must call `filter-buffer-substring' and
(2) not specify some required relation between the
limits passed to `filter-buffer-substring' and the
limits of the region piece.

That's what I've understood now, from your last
message.  Would you please confirm or correct that
understanding of what you've said?

I can see that the function should iterate over
the pieces of the region.  I can see that it might
be good to let it remove some bits of those pieces,
i.e., filter them a la `filter-buffer-substring'.

I guess I can even imagine that it might even be
useful for a function to return subsequences of the
buffer text that have no relation to the pieces of
the noncontiguous region, but that's a stretch.

But in that last case, I see no reason to impose
the use of `filter-buffer-substring' to accomplish
that.

And in any case, once the spec says that the
function (in the "anything else" case) must delete
the noncontiguous region, I see no reason why the
function must pass METHOD to `filter-buffer-substring',
unless you really want to allow the function to delete
not only the noncontiguous region but also any other
text in the buffer.

I'm trying to understand what the requirement is
for a function that is the value of
`region-extract-function'.

Confirming that it iterates over the pieces of
a noncontiguous region cleared up one problem.
But the use by `region-extract-function' of
`filter-buffer-substring' is less clear than ever,
I'm afraid.

bug#33167: 26; Doc string of `region-extract-function'

[Eli Zaretskii]

> Date: Sat, 27 Oct 2018 12:02:55 -0700 (PDT)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: 33167@debbugs.gnu.org
> 
> But if the spec of the variable imposes nothing on the BEG
> and END values passed for filtering, then why does it impose
> a requirement that `filter-buffer-substring' must be called?

So that applications could rely on that, and could provide their own
filter functions, if they want.

bug#33167: 26; Doc string of `region-extract-function'

[Drew Adams]

> > But if the spec of the variable imposes nothing on the BEG
> > and END values passed for filtering, then why does it impose
> > a requirement that `filter-buffer-substring' must be called?
> 
> So that applications could rely on that, and could provide their own
> filter functions, if they want.

I see.  That makes sense.  Thx.

Is it really the case that the value of `region-extract-function'
can pass any limits it wants to `filter-buffer-substring', even
limits that have no relation to the pieces of the noncontiguous
region?

So the only thing specified is that `filter-buffer-substring'
gets called once for each each piece?  I.e., all that's specified
is that if the nc region has N pieces then `filter-buffer-substring'
is called N times - nothing more?

bug#33167: 26; Doc string of `region-extract-function'

[Eli Zaretskii]

> Date: Sat, 27 Oct 2018 12:30:31 -0700 (PDT)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: 33167@debbugs.gnu.org
> 
> Is it really the case that the value of `region-extract-function'
> can pass any limits it wants to `filter-buffer-substring', even
> limits that have no relation to the pieces of the noncontiguous
> region?

I'd say doing so would make no sense to me.

> So the only thing specified is that `filter-buffer-substring'
> gets called once for each each piece?  I.e., all that's specified
> is that if the nc region has N pieces then `filter-buffer-substring'
> is called N times - nothing more?

I think it should be called once on each such piece, yes.