Pretest begins end-June

Pretest begins end-June

[Chong Yidong]

Stefan and I have decided on the end of June for the beginning of the
Emacs 24.1 pretest.  We're hoping for an early 2012 release, but, as
always, that will depend on the code.

The pretest process for 24.1 will be similar to that of 23.1.  A few
days before the pretest (I'll give a more precise date later), the trunk
will go into feature freeze, disallowing new features and other major
changes (e.g. new gnulib modules).  Packages can still be added to
elpa.gnu.org, independent of developments on the trunk.  We'll use the
trunk for most of the pretest, switching to a release branch only
shortly before the release.

Please plan accordingly.  Thanks.

Re: Pretest begins end-June

[Eli Zaretskii]

> From: Chong Yidong <cyd@stupidchicken.com>
> Date: Mon, 30 May 2011 12:07:03 -0400
> 
> Stefan and I have decided on the end of June for the beginning of the
> Emacs 24.1 pretest.  We're hoping for an early 2012 release, but, as
> always, that will depend on the code.

That schedule is way too early for the bidi stuff.  There are still 2
major jobs I need to do (reordering display strings and support for
truncated lines) and one minor one (a couple of bug reports regarding
cursor motion around invisible text and before-/after-strings).
There's no chance in the world that they will be ready in a month.

I also understand that Handa-san is working on modifying uni-bidi.el
and uni-mirror.el so that they could be used by bidi.c, instead of the
current separate tables.  I would like to have that done before we
enter feature freeze.

If you must start the pretest so early, then either 24.1 will need to
be released with the bidi flag off, or we will have significant
potentially destabilizing changes during the pretest (which
contradicts the feature freeze and will generally prolong the
pretest).

I'm sorry for my slow progress, but that's the best I can do working
alone on this.

Please tell me what is your decision on that.

Re: Pretest begins end-June

[Stefan Monnier]

>> Stefan and I have decided on the end of June for the beginning of the
>> Emacs 24.1 pretest.  We're hoping for an early 2012 release, but, as
>> always, that will depend on the code.
> That schedule is way too early for the bidi stuff.  There are still 2
> major jobs I need to do (reordering display strings and support for
> truncated lines) and one minor one (a couple of bug reports regarding
> cursor motion around invisible text and before-/after-strings).
> There's no chance in the world that they will be ready in a month.

I know it's hard to guess the future, but do you have some approximate
idea of when we could reasonably expect to get these fixed?

Other question: do these affect the behavior of Emacs when
bidi-reorder-display is non-nil but there's no R2L in sight?

> I also understand that Handa-san is working on modifying uni-bidi.el
> and uni-mirror.el so that they could be used by bidi.c, instead of the
> current separate tables.  I would like to have that done before we
> enter feature freeze.

That would be good, indeed.

> If you must start the pretest so early, then either 24.1 will need to
> be released with the bidi flag off,

I definitely want bidi-reorder-display to be t by default for Emacs-24.

> or we will have significant potentially destabilizing changes during
> the pretest (which contradicts the feature freeze and will generally
> prolong the pretest).

Maybe we could let bidi-reorder-display be t, while leaving some R2L
reordering (such as display strings) unimplemented?

> I'm sorry for my slow progress, but that's the best I can do working
> alone on this.

No reason to be sorry.  We're all glad you're doing it at all.


        Stefan

Re: Pretest begins end-June

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Chong Yidong <cyd@stupidchicken.com>,  emacs-devel@gnu.org
> Date: Mon, 30 May 2011 16:20:26 -0300
> 
> >> Stefan and I have decided on the end of June for the beginning of the
> >> Emacs 24.1 pretest.  We're hoping for an early 2012 release, but, as
> >> always, that will depend on the code.
> > That schedule is way too early for the bidi stuff.  There are still 2
> > major jobs I need to do (reordering display strings and support for
> > truncated lines) and one minor one (a couple of bug reports regarding
> > cursor motion around invisible text and before-/after-strings).
> > There's no chance in the world that they will be ready in a month.
> 
> I know it's hard to guess the future, but do you have some approximate
> idea of when we could reasonably expect to get these fixed?

The first part of display string support -- reordering portions of
text covered by display properties -- is already done and tested on my
local branch, I was planning on merging that this week.

The other part -- reordering the strings themselves -- is mostly local
to bidi.c, but also involves some changes in xdisp.c, particularly in
cursor motion and push_it/pop_it.  It's a large job, and will probably
take a month (i.e. 4 to 5 weekends) at least, maybe a little more.
The uncertainty here is significant: this area of the display engine
is notoriously under-documented and full of surprises, so it's
possible I'll need to change the design half way through because I
find out something I'm not aware of now.  It happened before.

Truncated lines will take something like 2 weekends to get to some
reasonably usable state, but that's without the radical changes
requested by Richard here:

  http://lists.gnu.org/archive/html/emacs-devel/2010-02/msg00167.html

At the time, you (Stefan) thought that the current "inverted
scrolling" was ``OK for now''.  OTOH, if we want to implement the
"rigid hscroll" that everybody agrees is TRT, we need to discuss the
details here first, because we never did back then.  Whatever the
details, I expect this to be a complete rewrite of the whole hscroll
thing, which reaches deep into the core parts of redisplay.

Bug-fixing could be left for after the pretest starts, because I
expect the pretest to be long, and so enough time to fix the bugs.

So in sum, it sounds like I'll need 6 to 7 weeks from now, barring any
unforeseen obstacles, personal disasters, etc.

Alternatively, we could start the pretest end of June as you plan, but
let me commit these changes as they become ready.  I expect that
turning bidi-display-reordering on for everyone even with today's code
base will expose quite a few bugs that I don't know about, so there's
some sense in doing that even though some features are not yet there.
In the past, a pretest version was supposed to be quite stable, but
maybe nowadays with so many people using the development code, it is
no longer such a significant consideration, if people can tolerate
transient breakage even during pretest.

> Other question: do these affect the behavior of Emacs when
> bidi-reorder-display is non-nil but there's no R2L in sight?

Of course, they do: when bidi-display-reordering is non-nil, the
display engine always goes through bidi.c, it doesn't know that the
result will be a simple linear iteration.  The same goes for cursor
positioning in set_cursor_from_row: the old Emacs 23 vintage code is
simply gone from there, so any bugs there will be acutely visible.
From the user perspective, the behavior in a plain L2R buffer should
be like in Emacs 23, but that's assuming there are no bugs...

> Maybe we could let bidi-reorder-display be t, while leaving some R2L
> reordering (such as display strings) unimplemented?

I won't recommend that except as the last resort, if it turns out that
the above estimates are way too optimistic, and/or the pretest is
otherwise finished quicker than I think it will be.

Re: Pretest begins end-June

[Stefan Monnier]

> So in sum, it sounds like I'll need 6 to 7 weeks from now, barring any
> unforeseen obstacles, personal disasters, etc.

I guess we could push the feature freeze to end of July.  But any new
feature introduced after late June will need to get permission.
How does that sound?

The feature freeze is not a code freeze, so it's OK if you need some
minor redesign to some part of the bidi/xdisp code as long as it is
necessary to fix a bug rather than add a new feature.  The distinction
between bug and new feature in this context should mostly be "is it
a regression w.r.t Emacs-23".


        Stefan

Re: Pretest begins end-June

[Kenichi Handa]

In article <jwvtyccxanq.fsf-monnier+emacs@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca> writes:

> I know it's hard to guess the future, but do you have some approximate
> idea of when we could reasonably expect to get these fixed?

For this work to finish:

> I also understand that Handa-san is working on modifying uni-bidi.el
> and uni-mirror.el so that they could be used by bidi.c, instead of the
> current separate tables.  I would like to have that done before we
> enter feature freeze.

I think the end of July is a good guess.  By the way, what
I'm going to do is to make all (or most of) uni-*.el data
more easily available to C code (and modify uni-mirror.el to
have mirror charatcter codes).

---
Kenichi Handa
handa@m17n.org

Re: Pretest begins end-June

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: cyd@stupidchicken.com,  emacs-devel@gnu.org
> Date: Mon, 30 May 2011 19:31:51 -0300
> 
> > So in sum, it sounds like I'll need 6 to 7 weeks from now, barring any
> > unforeseen obstacles, personal disasters, etc.
> 
> I guess we could push the feature freeze to end of July.  But any new
> feature introduced after late June will need to get permission.
> How does that sound?

End of July will certainly give me a deadline I can hope to meet.

I assume that by "new feature" you mean anything beyond the 2 I
mentioned (bidirectional display strings and hscroll in buffers with
bidi text), is that right?

Re: Pretest begins end-June

[David Kastrup]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> Stefan and I have decided on the end of June for the beginning of the
>>> Emacs 24.1 pretest.  We're hoping for an early 2012 release, but, as
>>> always, that will depend on the code.
>> That schedule is way too early for the bidi stuff.  There are still 2
>> major jobs I need to do (reordering display strings and support for
>> truncated lines) and one minor one (a couple of bug reports regarding
>> cursor motion around invisible text and before-/after-strings).
>> There's no chance in the world that they will be ready in a month.
>
> I know it's hard to guess the future, but do you have some approximate
> idea of when we could reasonably expect to get these fixed?
>
> Other question: do these affect the behavior of Emacs when
> bidi-reorder-display is non-nil but there's no R2L in sight?

Emacs is a desktop environment.  R2L will come "in sight" unintendedly
via Mail, via opening files in the wrong encoding, via Usenet, via
interpreting binary garbage and so on and so on.

-- 
David Kastrup

Re: Pretest begins end-June

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

> The first part of display string support -- reordering portions of
> text covered by display properties -- is already done and tested on my
> local branch, I was planning on merging that this week.
>
> The other part -- reordering the strings themselves -- is mostly local
> to bidi.c, but also involves some changes in xdisp.c, particularly in
> cursor motion and push_it/pop_it.  It's a large job, and will probably
> take a month (i.e. 4 to 5 weekends) at least, maybe a little more.
> The uncertainty here is significant: this area of the display engine
> is notoriously under-documented and full of surprises, so it's
> possible I'll need to change the design half way through because I
> find out something I'm not aware of now.  It happened before.

To give some perspective to this: one of the early adopters of display
properties in Emacs 21 pretest had been preview-latex (an AUCTeX
plugin).  And I sent about two bug reports per week to Gerd Möllmann for
months until all weird cases had been sorted out.  IIRC, by far the
worst contender for confusing the display engine was when the text
hidden by the display string contained newlines, presumably because
display matrix accounting and optimization got befuddled by it.

Newlines.  And now we are talking about R2L.

-- 
David Kastrup

Re: Pretest begins end-June

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Tue, 31 May 2011 09:50:31 +0200
> 
> IIRC, by far the worst contender for confusing the display engine
> was when the text hidden by the display string contained newlines,

I'm not surprised.  Newlines in the buffer serve as anchors for the
display engine when it moves non-linearly, e.g. goes back by N lines.
Handling the case where a newline does not indicate a beginning of a
new screen line is painful and tricky.

> Newlines.  And now we are talking about R2L.

Well, at least with R2L users can turn off the reordering if it
happens to screw them up too badly.

Re: Pretest begins end-June

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Kastrup <dak@gnu.org>
>> Date: Tue, 31 May 2011 09:50:31 +0200
>> 
>> IIRC, by far the worst contender for confusing the display engine
>> was when the text hidden by the display string contained newlines,
>
> I'm not surprised.  Newlines in the buffer serve as anchors for the
> display engine when it moves non-linearly, e.g. goes back by N lines.
> Handling the case where a newline does not indicate a beginning of a
> new screen line is painful and tricky.
>
>> Newlines.  And now we are talking about R2L.
>
> Well, at least with R2L users can turn off the reordering if it
> happens to screw them up too badly.

Unless the screwup asserts itself in the form of an infloop.  Anyway, as
opposed to normal R2L rendering, it is rather unlikely that display
strings and overlays will happen without some explicit user request
being involved.

-- 
David Kastrup

Re: Pretest begins end-June

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Tue, 31 May 2011 11:11:03 +0200
> 
> > Well, at least with R2L users can turn off the reordering if it
> > happens to screw them up too badly.
> 
> Unless the screwup asserts itself in the form of an infloop.

I meant turn off for the next session ;-)

> Anyway, as opposed to normal R2L rendering, it is rather unlikely
> that display strings and overlays will happen without some explicit
> user request being involved.

For some value of "explicit", unfortunately.  As you know only too
well, even TAB-completion in the minibuffer uses an overlay with an
after-string when it displays the "[Complete, but not unique]"
message.  And visiting an image file uses a display property.  Etc.,
etc.

Re: Pretest begins end-June

[Stefan Monnier]

>> > So in sum, it sounds like I'll need 6 to 7 weeks from now, barring any
>> > unforeseen obstacles, personal disasters, etc.
>> I guess we could push the feature freeze to end of July.  But any new
>> feature introduced after late June will need to get permission.
>> How does that sound?

> End of July will certainly give me a deadline I can hope to meet.

Good.

> I assume that by "new feature" you mean anything beyond the 2 I
> mentioned (bidirectional display strings and hscroll in buffers with
> bidi text), is that right?

I was mostly thinking about non-bidi features, but yes, that also
applies to not-already-agreed new features of bidi.


        Stefan

Re: Pretest begins end-June

[Chong Yidong]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> I guess we could push the feature freeze to end of July.  But any new
>>> feature introduced after late June will need to get permission.
>>> How does that sound?
>>
>> I assume that by "new feature" you mean anything beyond the 2 I
>> mentioned (bidirectional display strings and hscroll in buffers with
>> bidi text), is that right?
>
> I was mostly thinking about non-bidi features, but yes, that also
> applies to not-already-agreed new features of bidi.

So we'll use July to get started on stabilizing the non-bidi parts of
the code, and the 24.1 documentation.  That should be fine.

Re: Pretest begins end-June

[martin rudalics]

 > Please plan accordingly.

I intend to gradually deploy my window code changes in the next days.
I'll start with the window tree functions, continue with moving the
resize routines to Elisp, and finally apply the `display-buffer'
changes.  The final step will break a number of functions using buffer
display routines, notably in org, gud, calculator and calendar code,
which I cannot fix alone because I often don't understand the authors'
intentions.  It very likely will also break external packages which I
don't know like Drew's Icicles.  So I'll need some help there.

If there are any objections please tell me.  The code can be visited at
and built from the window-pub branch.  New items are described in the
NEWS file, the windows section in the Elisp reference manual has been
completely rewritten to reflect all changes.

martin

Re: Pretest begins end-June

[Stefan Monnier]

> I intend to gradually deploy my window code changes in the next days.
> I'll start with the window tree functions, continue with moving the
> resize routines to Elisp, and finally apply the `display-buffer'
> changes.

That was unexpected, but I think it's a good idea.

> The final step will break a number of functions using buffer
> display routines, notably in org, gud, calculator and calendar code,
> which I cannot fix alone because I often don't understand the authors'
> intentions.  It very likely will also break external packages which I
> don't know like Drew's Icicles.  So I'll need some help there.

Can you explain the kind of breakage and the reason why the change has
to introduce these incompatibilities?


        Stefan

Re: Pretest begins end-June

[martin rudalics]

 >> The final step will break a number of functions using buffer
 >> display routines, notably in org, gud, calculator and calendar code,
 >> which I cannot fix alone because I often don't understand the authors'
 >> intentions.  It very likely will also break external packages which I
 >> don't know like Drew's Icicles.  So I'll need some help there.
 >
 > Can you explain the kind of breakage and the reason why the change has
 > to introduce these incompatibilities?

The breakage occurs because `display-buffer' now expects that _all_
directives wrt where and how to display the buffer must be passed via a
list in the second argument (which was formerly called `other-window'
and is now called `specifiers') and the user customization of the
behavior of `display-buffer' occurs independently in a new option called
`display-buffer-alist'.

The reason for this change was one of your earlier postings

 > Rather than let-binding some Lisp-manipulated "config" var, I was
 > thinking of passing special parameters to display-buffer ...

and Juri Linkov's subsequent proposal

 > Why not use the existing argument `other-window'
 > of `pop-to-buffer' and `display-buffer'?

Hence, if an application now wants to pop up a buffer in another frame
it must do something like

(pop-to-buffer buffer-or-name 'other-frame)

instead of

(let ((pop-up-frames t))
   (pop-to-buffer buffer))

The breakage will usually cause a call like the later to simply follow
the user customizations for how to display "buffer".  I changed most of
the trivial calls already but some are rather special.

martin

Re: Pretest begins end-June

[Stefan Monnier]

> The breakage occurs because `display-buffer' now expects that _all_
> directives wrt where and how to display the buffer must be passed via a
> list in the second argument (which was formerly called `other-window'
> and is now called `specifiers') and the user customization of the
> behavior of `display-buffer' occurs independently in a new option called
> `display-buffer-alist'.

This is a good change, yes, thank you.

> (let ((pop-up-frames t))
>   (pop-to-buffer buffer))

The part I don't understand is why we can't preserve backward
compatibility for such code.


        Stefan

RE: Pretest begins end-June

[Drew Adams]

> Hence, if an application now wants to pop up a buffer in
> another frame it must do something like
> (pop-to-buffer buffer-or-name 'other-frame)
> instead of (let ((pop-up-frames t)) (pop-to-buffer buffer))

Which means that an app that calls another function that does (pop-to-buffer
buffer) cannot control the behavior.

What you describe is no big deal for code that calls `pop-to-buffer' directly.
But it makes it difficult if not impossible to control the behavior when code
calls other code that in turn calls `pop-to-buffer'.  Without access to changing
the latter code, one's goose is pretty much cooked.

`pop-up-frames' is designed to be used as a dynamic ("special") var.  Dynamic
binding can be very useful, as I'm sure you know.  The change you describe works
against that usefulness.

> I changed most of the trivial calls already but some are
> rather special.

Are you talking about just changing calls to `pop-to-buffer' in the Emacs
sources?  Many, probably most, `pop-to-buffer' calls are out there in the wider
Emacs world, not just in the Emacs-Dev sources.

More importantly, what you describe apparently limits the use and usefulness of
`pop-up-frames' (essentially eliminating it) - it's not just about existing
code.

Please provide some way to dynamically bind _some_ var to control the behavior.
IOW, please restore the flexibility (control) you are apparently taking away.

RE: Pretest begins end-June

[Drew Adams]

> > (let ((pop-up-frames t)) (pop-to-buffer buffer))
> 
> The part I don't understand is why we can't preserve backward
> compatibility for such code.

It is not just, or even most importantly, about backward compatibility.  Being
able to determine/regulate the behavior with a dynamic binding is an important
feature.  This change would apparently remove/limit that possibility.

Re: Pretest begins end-June

[martin rudalics]

 >> (let ((pop-up-frames t))
 >>   (pop-to-buffer buffer))
 >
 > The part I don't understand is why we can't preserve backward
 > compatibility for such code.

Because it's not quite easy to decide in general which value should
prevail.  When `display-buffer' encounters a setting like this it would
have to know whether it comes from a user who has set this in .emacs for
Emacs < 24 or from an application that has not yet adopted the new
behavior.

In the former case, I would have to merge the request to pop up a new
frame with the actual customization of `display-buffer-alist'.  Now
`pop-up-frames' is probably not very problematic in this context, but
the options for popping up and reusing windows have been expanded
considerably and most of these would get lost.  Note: They would get
lost because `display-buffer' would _have to assume_ that a specific
behavior was requested by the application although that application did
_not_ rebind the corresponding variable in the first place.

It might be possible to use some heuristics like "`pop-up-windows' is by
default t, so if `display-buffer' sees that this is t it will disregard
it and consult the value of `display-buffer-alist' instead".  But doing
such guesses doesn't strike me as very robust :-(

martin

Re: Pretest begins end-June

[martin rudalics]

 >>> (let ((pop-up-frames t)) (pop-to-buffer buffer))
 >> The part I don't understand is why we can't preserve backward
 >> compatibility for such code.
 >
 > It is not just, or even most importantly, about backward compatibility.  Being
 > able to determine/regulate the behavior with a dynamic binding is an important
 > feature.  This change would apparently remove/limit that possibility.

Unfortunately not.  But it's an attempt to discourage that possibility
by providing a suitable alternative - passing an explicit argument.

IMHO an application may request (or better "suggest") specific behavior
and set private variables.  But it should not change or rebind any user
option.

martin

Re: Pretest begins end-June

[Stefan Monnier]

>>> (let ((pop-up-frames t))
>>> (pop-to-buffer buffer))
>> 
>> The part I don't understand is why we can't preserve backward
>> compatibility for such code.

> Because it's not quite easy to decide in general which value should
> prevail.  When `display-buffer' encounters a setting like this it would
> have to know whether it comes from a user who has set this in .emacs for
> Emacs < 24 or from an application that has not yet adopted the new
> behavior.

Let me put the question differently:
*how* can we support code that rebinds pop-up-frames?

I think preserving backward compatibility for this case is important,
because it is used in many packages, not all of which are included in
Emacs.

> In the former case, I would have to merge the request to pop up a new
> frame with the actual customization of `display-buffer-alist'.  Now
> `pop-up-frames' is probably not very problematic in this context, but
> the options for popping up and reusing windows have been expanded
> considerably and most of these would get lost.  Note: They would get
> lost because `display-buffer' would _have to assume_ that a specific
> behavior was requested by the application although that application did
> _not_ rebind the corresponding variable in the first place.

IIUC, the problems are:
1- detect that pop-up-frames was set.
2- decide whether pop-up-frames was set by user or let-bound by the caller.
3- for each of those two cases, take this request into account.
4- same for pop-up-windows.

Case 1 is easy (set the default value to `unset' and you're done).
Case 2 is more difficult.  Of course, we could add a new primitive that
walks the specpdl stack to decide if a var is let-bound or not, but that
doesn't sound very appealing.
Case 3 doesn't sound too hard; IIUC it involves losing some
functionality but that functionality is absent from Emacs-23 anyway.
Do we really need to solve case 2?  Probably not.

> It might be possible to use some heuristics like "`pop-up-windows' is by
> default t, so if `display-buffer' sees that this is t it will disregard
> it and consult the value of `display-buffer-alist' instead".  But doing
> such guesses doesn't strike me as very robust :-(

If we declare pop-up-frames and pop-up-windows as obsolete, I think it's
OK to have an heuristic simulation of its semantics as long as it
handles the known cases well enough.


        Stefan

RE: Pretest begins end-June

[Drew Adams]

> > Being able to determine/regulate the behavior with a dynamic 
> > binding is an important feature.  This change would apparently
> > remove/limit that possibility.
> 
> Unfortunately not.  But it's an attempt to discourage that possibility
> by providing a suitable alternative - passing an explicit argument.

You said that binding the var would no longer work.
That is a removal/limit of that possibility.

And you do not provide a suitable alternative.  You are missing/ignoring the
general case, where the code that does the display is not necessarily available
to modify textually.

Your "suitable alternative" is workable only in the minority of cases where the
call to `pop-to-buffer' is directly available and can be edited.  You propose
source-code editing as a replacement for the programmatic control of dynamic
binding.

> IMHO an application may request (or better "suggest") 
> specific behavior and set private variables.  

Your example substitute code does not "request" or "suggest" how to display,
IIUC.  It imposes how to display, the same as the code it would replace.

Or if not, then it is not at all a suitable alternative and we are not even
talking apples and oranges but rather apples and orangutans.

> But it should not change or rebind any user option.

Obviously I disagree.  You're apparently OK with an application deciding whether
to pop up a frame, but not OK with doing that by binding a dynamic variable,
because in this case the var is a user option.

That's hypocritical and silly.  If an application controls the display, then it
has taken control for that away from the user, regardless of how it does so.

Bypassing `pop-up-frames' to do that is no more respectful of the user's general
preference as expressed by that var than is binding the var.  What is important
are the resulting behavior and the user's preferences.

Sometimes an application can be right to decide how something is displayed, in
spite of a user's general preference.  Developers need to DTRT, thinking about
the user and her preferences to decide what TRT is in any given context.
Respecting users does not necessarily mean blindly applying their general
preferences in all contexts.

If your code violates a user's general preference as defined by her
`pop-up-frames' value, it is irrelevant how that violation was implemented.

And this is more general than a question about user options.  It is about
dynamic variables in general.  Removing the ability to bind a dynamic var in
order to control behavior in code that is not directly modifiable is limiting
and, frankly, unlispy.  (No flames about lexical-only Lisps, please.)

Re: Pretest begins end-June

[Stefan Monnier]

> That's hypocritical and silly.

That is not very constructive, now is it.
I suggest instead you first check to see what Martin's code does (it
provides a lot more and finer control to the end-user).  And if you
still find cases that don't work well enough, then please report those
concrete situations.


        Stefan

Re: Pretest begins end-June

[martin rudalics]

 > Let me put the question differently:
 > *how* can we support code that rebinds pop-up-frames?

It's a bit like having the cake and eating it too.

 > I think preserving backward compatibility for this case is important,
 > because it is used in many packages, not all of which are included in
 > Emacs.

Agreed.

 > IIUC, the problems are:
 > 1- detect that pop-up-frames was set.
 > 2- decide whether pop-up-frames was set by user or let-bound by the caller.
 > 3- for each of those two cases, take this request into account.
 > 4- same for pop-up-windows.
 >
 > Case 1 is easy (set the default value to `unset' and you're done).

I'm not 100% sure whether this could lead to difficulties (IIUC I would
have to hardcode the Emacs 23 default values of `split-height-threshold'
and `pop-up-windows' in `display-buffer') but let's agree.

 > Case 2 is more difficult.  Of course, we could add a new primitive that
 > walks the specpdl stack to decide if a var is let-bound or not, but that
 > doesn't sound very appealing.

Not really.

 > Case 3 doesn't sound too hard; IIUC it involves losing some
 > functionality but that functionality is absent from Emacs-23 anyway.
 > Do we really need to solve case 2?  Probably not.

Suppose a user has set `split-height-threshold' to some value for use in
Emacs 23 and in Emacs 24 wants to use a new functionality of
`display-buffer-alist' say apply `fit-window-to-buffer' for adjusting
the window height.  What shall `display-buffer' do?  Respect the value
of `split-height-threshold'?  Adjust the height of the window?  Do both?

 > If we declare pop-up-frames and pop-up-windows as obsolete, I think it's
 > OK to have an heuristic simulation of its semantics as long as it
 > handles the known cases well enough.

I think I won't have great problems providing an acceptable heuristic
for `pop-up-frames'.  But I'm afraid that packages outside Emacs have
some very unknown cases in store for me.

martin

Re: Pretest begins end-June

[martin rudalics]

 > You said that binding the var would no longer work.
 > That is a removal/limit of that possibility.

The application can bind the variable `display-buffer-alist' instead.

 > And you do not provide a suitable alternative.  You are missing/ignoring the
 > general case, where the code that does the display is not necessarily available
 > to modify textually.

I don't understand you here.

 > Your "suitable alternative" is workable only in the minority of cases where the
 > call to `pop-to-buffer' is directly available and can be edited.  You propose
 > source-code editing as a replacement for the programmatic control of dynamic
 > binding.

Actually I followed Stefan's advice which I repeat here once more:

   Rather than let-binding some Lisp-manipulated "config" var, I was
   thinking of passing special parameters to display-buffer (I'd rather
   avoid dynamic scoping whenever possible).

So you can see that my first approach was to use conventional binding.

 >> IMHO an application may request (or better "suggest")
 >> specific behavior and set private variables.
 >
 > Your example substitute code does not "request" or "suggest" how to display,
 > IIUC.  It imposes how to display, the same as the code it would replace.

But not by manipulating a user option.  Moreover, users can _always_
override the application, if they want to.

 > Or if not, then it is not at all a suitable alternative and we are not even
 > talking apples and oranges but rather apples and orangutans.
 >
 >> But it should not change or rebind any user option.
 >
 > Obviously I disagree.  You're apparently OK with an application deciding whether
 > to pop up a frame, but not OK with doing that by binding a dynamic variable,
 > because in this case the var is a user option.

Yes (if we replace the term "deciding" by the term "suggesting").

 > That's hypocritical and silly.  If an application controls the display, then it
 > has taken control for that away from the user, regardless of how it does so.

No.  A let-binding affects every `display-buffer' call nested in it.  An
argument affects only the associated call.

 > Bypassing `pop-up-frames' to do that is no more respectful of the user's general
 > preference as expressed by that var than is binding the var.  What is important
 > are the resulting behavior and the user's preferences.
 >
 > Sometimes an application can be right to decide how something is displayed, in
 > spite of a user's general preference.  Developers need to DTRT, thinking about
 > the user and her preferences to decide what TRT is in any given context.
 > Respecting users does not necessarily mean blindly applying their general
 > preferences in all contexts.

Let's agree that developers and users may be both wrong, sometimes.

 > If your code violates a user's general preference as defined by her
 > `pop-up-frames' value, it is irrelevant how that violation was implemented.
 >
 > And this is more general than a question about user options.  It is about
 > dynamic variables in general.  Removing the ability to bind a dynamic var in
 > order to control behavior in code that is not directly modifiable is limiting
 > and, frankly, unlispy.  (No flames about lexical-only Lisps, please.)

We wouldn't agree anyway.

martin

Re: Pretest begins end-June

[Stefan Monnier]

>> Let me put the question differently:
>> *how* can we support code that rebinds pop-up-frames?
> It's a bit like having the cake and eating it too.

Yup.

>> Case 1 is easy (set the default value to `unset' and you're done).
> I'm not 100% sure whether this could lead to difficulties (IIUC I would

Neither am I, but we know that doing nothing will also lead to difficulties.

>> Case 3 doesn't sound too hard; IIUC it involves losing some
>> functionality but that functionality is absent from Emacs-23 anyway.
>> Do we really need to solve case 2?  Probably not.

> Suppose a user has set `split-height-threshold' to some value for use in
> Emacs 23 and in Emacs 24 wants to use a new functionality of
> `display-buffer-alist' say apply `fit-window-to-buffer' for adjusting
> the window height.  What shall `display-buffer' do?  Respect the value
> of `split-height-threshold'?  Adjust the height of the window?  Do both?

fir-window-to-buffer is unlikely to be something that you want to apply
to *all* cases, so if set it should take precedence over
split-height-threshold which is meant as a default behavior.

> I think I won't have great problems providing an acceptable heuristic
> for `pop-up-frames'.  But I'm afraid that packages outside Emacs have
> some very unknown cases in store for me.

As I said: not doing anything is guaranteed to bring us problems, so
even if the heuristic doesn't solve all cases, it should not be too hard
to come up with one that solves more problems than it introduces.


        Stefan

Re: Pretest begins end-June

[martin rudalics]

 > Which means that an app that calls another function that does (pop-to-buffer
 > buffer) cannot control the behavior.

Intentionally so.  Nested within that call there may be other
`display-buffer' calls of which the caller is not aware.  These calls
should not be affected by any bindings.  I do not have the slightest
sentiment for my `pop-up-frames' nil `debug-on-error' t setting have

(let ((pop-up-frames t))
   (error "???"))

pop up a new frame.

 > `pop-up-frames' is designed to be used as a dynamic ("special") var.  Dynamic
 > binding can be very useful, as I'm sure you know.  The change you describe works
 > against that usefulness.

I suppose that `pop-up-frames' was designed as an option for the user to
control the behavior of buffer display functions.  The history of this
and all related options is that of building a tower as follows:

- Initially the user was provided the option `pop-up-frames' to control
   the creation of new frames.

- Applications stole the customizations by let-binding this variable to
   whatever they considered appropriate.

- The user was given back the option via `special-display-popup-frame'.

- Applications stole the customizations again by binding
   `special-display-buffer-names' and `special-display-regexps' to nil.

So where would you go from here?

 >> I changed most of the trivial calls already but some are
 >> rather special.
 >
 > Are you talking about just changing calls to `pop-to-buffer' in the Emacs
 > sources?  Many, probably most, `pop-to-buffer' calls are out there in the wider
 > Emacs world, not just in the Emacs-Dev sources.

That's why I raised this issue in my first mail.

 > More importantly, what you describe apparently limits the use and usefulness of
 > `pop-up-frames' (essentially eliminating it) - it's not just about existing
 > code.
 >
 > Please provide some way to dynamically bind _some_ var to control the behavior.
 > IOW, please restore the flexibility (control) you are apparently taking away.

That flexibility has been given back to the user.

martin

Re: Pretest begins end-June

[martin rudalics]

 > fir-window-to-buffer is unlikely to be something that you want to apply
 > to *all* cases, so if set it should take precedence over
 > split-height-threshold which is meant as a default behavior.

OK.  I'll use this as one possible guideline.

 > As I said: not doing anything is guaranteed to bring us problems, so
 > even if the heuristic doesn't solve all cases, it should not be too hard
 > to come up with one that solves more problems than it introduces.

I'll see what I can do in this regard.  Since, in the worst case, the
user can always override the application, I shall try to make this more
application-friendly than intended.  Maybe I also add a compatibility
option the user can set - something like "if this is non-nil, the value
of an obsolete buffer display variable is respected if it's different
from the default." - and make it by default non-nil for the time being.

martin

Re: Pretest begins end-June

[Eli Zaretskii]

> Date: Tue, 31 May 2011 00:12:30 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: cyd@stupidchicken.com, emacs-devel@gnu.org
> 
> The first part of display string support -- reordering portions of
> text covered by display properties -- is already done and tested on my
> local branch, I was planning on merging that this week.

Done (revision 104482 on the trunk).

Re: Pretest begins end-June

[Stefan Monnier]

>> The first part of display string support -- reordering portions of
>> text covered by display properties -- is already done and tested on my
>> local branch, I was planning on merging that this week.
> Done (revision 104482 on the trunk).

1 down,


        Stefan

23.4 [was Re: Pretest begins end-June]

[Glenn Morris]

Chong Yidong wrote:

> Stefan and I have decided on the end of June for the beginning of the
> Emacs 24.1 pretest.  We're hoping for an early 2012 release, but, as
> always, that will depend on the code.

In light of this, I wonder if you could comment on whether you expect
there to be a 23.4?

Since the above plan is a relatively short time-scale, and since there
are no show-stopping problems with 23.3 (so far), I'm hoping we can get
away without a 23.4.

Re: 23.4 [was Re: Pretest begins end-June]

[Stefan Monnier]

>> Stefan and I have decided on the end of June for the beginning of the
>> Emacs 24.1 pretest.  We're hoping for an early 2012 release, but, as
>> always, that will depend on the code.
> In light of this, I wonder if you could comment on whether you expect
> there to be a 23.4?
> Since the above plan is a relatively short time-scale, and since there
> are no show-stopping problems with 23.3 (so far), I'm hoping we can get
> away without a 23.4.

That's also my hope.


        Stefan

Re: 23.4 [was Re: Pretest begins end-June]

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 721 bytes --]

On 6/8/2011 8:35 AM, Stefan Monnier wrote:
>>> Stefan and I have decided on the end of June for the beginning of the
>>> Emacs 24.1 pretest.  We're hoping for an early 2012 release, but, as
>>> always, that will depend on the code.
>> In light of this, I wonder if you could comment on whether you expect
>> there to be a 23.4?
>> Since the above plan is a relatively short time-scale, and since there
>> are no show-stopping problems with 23.3 (so far), I'm hoping we can get
>> away without a 23.4.
> 
> That's also my hope.

Is it okay to make changes to the emacs-23 branch just in case? If we
*do* release a 23.4, marking lexical-binding as a safe local variable
would make some annoyances go away.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 258 bytes --]

Re: 23.4 [was Re: Pretest begins end-June]

[Stefan Monnier]

>>>> Stefan and I have decided on the end of June for the beginning of the
>>>> Emacs 24.1 pretest.  We're hoping for an early 2012 release, but, as
>>>> always, that will depend on the code.
>>> In light of this, I wonder if you could comment on whether you expect
>>> there to be a 23.4?
>>> Since the above plan is a relatively short time-scale, and since there
>>> are no show-stopping problems with 23.3 (so far), I'm hoping we can get
>>> away without a 23.4.
>> That's also my hope.
> Is it okay to make changes to the emacs-23 branch just in case?

Of course.


        Stefan

window-safely-shrinkable-p [was Re: Pretest begins end-June]

[Glenn Morris]

martin rudalics wrote:

> The final step will break a number of functions using buffer display
> routines, notably in org, gud, calculator and calendar code, which I
> cannot fix alone because I often don't understand the authors' intentions.

Re calendar: me neither (probably), but you can ask. :)

The first thing I notice is that the function
window-safely-shrinkable-p, used by calendar.el, seems to have vanished
(with no ChangeLog entry) in r104562.

Re: window-safely-shrinkable-p [was Re: Pretest begins end-June]

[martin rudalics]

 > The first thing I notice is that the function
 > window-safely-shrinkable-p, used by calendar.el, seems to have vanished
 > (with no ChangeLog entry) in r104562.

Arrrgh ...

Can you please try replacing

     (when in-calendar-window
       ;; The second test used to be window-full-width-p.
       ;; Not sure what it was/is for, except perhaps some way of saying
       ;; "try not to mess with existing configurations".
       ;; If did the wrong thing on wide frames, where we have done a
       ;; horizontal split in calendar-basic-setup.
       (if (or (one-window-p t) (not (window-safely-shrinkable-p)))
           ;; Don't mess with the window size, but ensure that the first
           ;; line is fully visible.
           (set-window-vscroll nil 0)
         ;; Adjust the window to exactly fit the displayed calendar.
         (fit-window-to-buffer nil nil calendar-minimum-window-height))

by

     (when in-calendar-window
       (if (window-iso-combined-p)
	  ;; Adjust the window to exactly fit the displayed calendar.
	  (fit-window-to-buffer nil nil calendar-minimum-window-height)
	;; For a full height window or a window that is horizontally
	;; combined don't fit height to that of its buffer.
	(set-window-vscroll nil 0))

and tell me whether it works?

Thanks, martin

Re: window-safely-shrinkable-p

[Glenn Morris]

martin rudalics wrote:

>     (when in-calendar-window
>       (if (window-iso-combined-p)
> 	  ;; Adjust the window to exactly fit the displayed calendar.
> 	  (fit-window-to-buffer nil nil calendar-minimum-window-height)
> 	;; For a full height window or a window that is horizontally
> 	;; combined don't fit height to that of its buffer.
> 	(set-window-vscroll nil 0))
>
> and tell me whether it works?


I can only do very limited testing right now, but it seems ok, thanks.
Feel free to install this and and any other related changes to calendar
window handling. I'm looking forward to the improvements/simplifications
that your work should bring, window handling has been messy in the past.

Re: window-safely-shrinkable-p [was Re: Pretest begins end-June]

[Stefan Monnier]

> Can you please try replacing

Please provide a backward compatibility implementation of any function
that's found missing: if Emacs's packages use it, it's likely that
outside packages use it as well.


        Stefan

Re: window-safely-shrinkable-p

[martin rudalics]

 > I can only do very limited testing right now, but it seems ok, thanks.
 > Feel free to install this

Installed (mainly because IMHO `window-safely-shrinkable-p' was not
really safe in the first place).  Please check whether it DTRT.

 > and and any other related changes to calendar
 > window handling. I'm looking forward to the improvements/simplifications
 > that your work should bring, window handling has been messy in the past.

I hope that my future `display-buffer' changes will remain compatible
with calendar.  I'll then have some trivial changes for calendar ready
to install but will contact you before applying them.  We'll have to
discuss the more tricky cases after that.

martin

Re: window-safely-shrinkable-p [was Re: Pretest begins end-June]

[martin rudalics]

 > Please provide a backward compatibility implementation of any function
 > that's found missing:

The change in question was not intentional (which is also explained by
the fact that there was no corresponding ChangeLog entry).  I have
restored the original function but declared it as obsolete because I
think it doesn't do the right thing in more complex configurations.

 > if Emacs's packages use it, it's likely that
 > outside packages use it as well.

I hope this doesn't hold for window--.. functions.

martin

Re: Pretest begins end-June

[Dan Nicolaescu]

Chong Yidong <cyd@stupidchicken.com> writes:

> Stefan and I have decided on the end of June for the beginning of the
> Emacs 24.1 pretest.  We're hoping for an early 2012 release, but, as
> always, that will depend on the code.

Is there a plan for things that need to be finished?

For example given that lexical scoping is one of the new things in this
release it would be nice if more things could use it.  The "term",
"language" and "international" subdirectories could be converted with
very little effort.

Also how about making sure that more programming modes are derived from
prog-mode?  This patch can take care of a few of them:

=== modified file 'lisp/progmodes/ld-script.el'
--- lisp/progmodes/ld-script.el 2011-05-12 16:46:53 +0000
+++ lisp/progmodes/ld-script.el 2011-06-13 06:10:51 +0000
@@ -168,7 +168,7 @@
   "Default font-lock-keywords for `ld-script-mode'.")
 
 ;;;###autoload
-(define-derived-mode ld-script-mode nil "LD-Script"
+(define-derived-mode ld-script-mode prog-mode "LD-Script"
    "A major mode to edit GNU ld script files"
   (set (make-local-variable 'comment-start) "/* ")
   (set (make-local-variable 'comment-end)   " */")

=== modified file 'lisp/progmodes/mixal-mode.el'
--- lisp/progmodes/mixal-mode.el        2011-05-23 17:57:17 +0000
+++ lisp/progmodes/mixal-mode.el        2011-06-13 06:10:44 +0000
@@ -1103,7 +1103,7 @@ Assumes that file has been compiled with
     (error "mixvm.el needs to be loaded to run `mixvm'")))
 
 ;;;###autoload
-(define-derived-mode mixal-mode fundamental-mode "mixal"
+(define-derived-mode mixal-mode prog-mode "mixal"
   "Major mode for the mixal asm language."
   (set (make-local-variable 'comment-start) "*")
   (set (make-local-variable 'comment-start-skip) "^\\*[ \t]*")

=== modified file 'lisp/progmodes/ps-mode.el'
--- lisp/progmodes/ps-mode.el   2011-04-22 18:44:26 +0000
+++ lisp/progmodes/ps-mode.el   2011-06-13 06:11:12 +0000
@@ -485,7 +485,7 @@ If nil, use `temporary-file-directory'."
 ;; PostScript mode.
 
 ;;;###autoload
-(define-derived-mode ps-mode fundamental-mode "PostScript"
+(define-derived-mode ps-mode prog-mode "PostScript"
   "Major mode for editing PostScript with GNU Emacs.
 
 Entry to this mode calls `ps-mode-hook'.

=== modified file 'lisp/progmodes/python.el'
--- lisp/progmodes/python.el    2011-05-24 03:38:35 +0000
+++ lisp/progmodes/python.el    2011-06-13 06:32:06 +0000
@@ -2420,7 +2420,7 @@ without confirmation."
 (defvar python-mode-running)            ;Dynamically scoped var.
 
 ;;;###autoload
-(define-derived-mode python-mode fundamental-mode "Python"
+(define-derived-mode python-mode prog-mode "Python"
   "Major mode for editing Python files.
 Turns on Font Lock mode unconditionally since it is currently required
 for correct parsing of the source.

Re: Pretest begins end-June

[Eli Zaretskii]

> From: Dan Nicolaescu <dann@gnu.org>
> Date: Mon, 13 Jun 2011 12:29:39 -0400
> Cc: emacs-devel@gnu.org
> 
> Also how about making sure that more programming modes are derived from
> prog-mode?  This patch can take care of a few of them:

Thank you.  I think this should be installed ASAP, if for no other
reason, then because bidi-display-reordering will be eventually turned
on by default.  prog-mode makes sure every mode derived from it will
have the paragraph direction forced to be left-to-right, to avoid
messing the display if someone writes a comment or string in some R2L
script in a strategic place.

Re: Pretest begins end-June

[Chong Yidong]

Dan Nicolaescu <dann@gnu.org> writes:

> For example given that lexical scoping is one of the new things in this
> release it would be nice if more things could use it.  The "term",
> "language" and "international" subdirectories could be converted with
> very little effort.

Apart from those files that were heavily using lexical-let---which are
already converted---it doesn't seem worth making the conversion a
release goal for 24.1.

On the other hand, if any Emacs developer wants to do the work of
converting files to lexical binding during the remaining time to
pretest, it will probably do no harm.  Stefan, WDYT?

> Also how about making sure that more programming modes are derived from
> prog-mode?  This patch can take care of a few of them:

Yes, please commit.

Re: Pretest begins end-June

[Stefan Monnier]

> Is there a plan for things that need to be finished?

Nothing very specific, no.

> For example given that lexical scoping is one of the new things in this
> release it would be nice if more things could use it.  The "term",
> "language" and "international" subdirectories could be converted with
> very little effort.

This is not a priority, no.  The main purpose of Emacs-24's lexical
binding support is to make it possible to use efficient
lexical binding.  Maybe for Emacs-25, we will want to encourage
conversion more explicitly, but not for now.  Note for example that
debugging lexically scoped code is not as well supported.

> Also how about making sure that more programming modes are derived from
> prog-mode?

That would be nice, yes.  Feel free to install such changes.


        Stefan

Re: window-safely-shrinkable-p [was Re: Pretest begins end-June]

[Stefan Monnier]

> The change in question was not intentional (which is also explained by
> the fact that there was no corresponding ChangeLog entry).  I have
> restored the original function but declared it as obsolete because I
> think it doesn't do the right thing in more complex configurations.

Thanks.

>> if Emacs's packages use it, it's likely that
>> outside packages use it as well.
> I hope this doesn't hold for window--.. functions.

So do I,


        Stefan

deriving from prog-mode (was: Re: Pretest begins end-June)

[Dan Nicolaescu]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Also how about making sure that more programming modes are derived from
>> prog-mode?
>
> That would be nice, yes.  Feel free to install such changes.

Cited patch checked in.

It looks like the only remaining stragglers in lisp/progmodes are
cperl-mode.el (which, AFAIR, is not maintained here) and 
delphi.el (which passes an extra `skip-initial-parsing' to `delphi-mode'.

Re: deriving from prog-mode

[Chong Yidong]

Dan Nicolaescu <dann@gnu.org> writes:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>>> Also how about making sure that more programming modes are derived from
>>> prog-mode?
>>
>> That would be nice, yes.  Feel free to install such changes.
>
> Cited patch checked in.
>
> It looks like the only remaining stragglers in lisp/progmodes are
> cperl-mode.el (which, AFAIR, is not maintained here) and delphi.el
> (which passes an extra `skip-initial-parsing' to `delphi-mode'.

I've fixed delphi-mode.  Not sure about the best thing for cperl-mode;
due to its maintenance baggage, maybe we should just add
bidi-paragraph-direction to the function body instead of using
define-derived-mode.

Re: deriving from prog-mode

[Dan Nicolaescu]

Chong Yidong <cyd@stupidchicken.com> writes:

> Dan Nicolaescu <dann@gnu.org> writes:
>
>> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>
>>>> Also how about making sure that more programming modes are derived from
>>>> prog-mode?
>>>
>>> That would be nice, yes.  Feel free to install such changes.
>>
>> Cited patch checked in.
>>
>> It looks like the only remaining stragglers in lisp/progmodes are
>> cperl-mode.el (which, AFAIR, is not maintained here) and delphi.el
>> (which passes an extra `skip-initial-parsing' to `delphi-mode'.
>
> I've fixed delphi-mode.  Not sure about the best thing for cperl-mode;
> due to its maintenance baggage, maybe we should just add
> bidi-paragraph-direction to the function body instead of using
> define-derived-mode.

If you go that way, please also make it run `prog-mode-hook`.

Re: deriving from prog-mode

[Stefan Monnier]

> If you go that way, please also make it run `prog-mode-hook`.

I'd rather only do that via the use of define-derived-mode.
As for whether we should use define-derived-mode for cperl-mode.el,
I have no opinion on it (I generally prefer refrain from changing
cperl-mode.el, but maybe it's not that bad in this case).


        Stefan

Re: deriving from prog-mode

[Chong Yidong]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> If you go that way, please also make it run `prog-mode-hook`.
>
> I'd rather only do that via the use of define-derived-mode.
> As for whether we should use define-derived-mode for cperl-mode.el,
> I have no opinion on it (I generally prefer refrain from changing
> cperl-mode.el, but maybe it's not that bad in this case).

I went ahead and changed it to use define-derived-mode.