The new text-mode menu and the cursor in -nw mode

The new text-mode menu and the cursor in -nw mode

[Mario Lang]

Hi.

The new text mode menu is nice, something I have been waiting for since
a long time.  However, I have an accessibility issue with it: When I hit
F10 to go to the menu, the menu pops up and the selected item is
highlighted with a change of color, but the terminal cursor stays at the
place in the currently selected buffer.  I can see that this makes sense
for some use cases (you can still see where point is while navigating
the menu), but it is quite a problem to (some) blind users of Emacs.

Screen readers on UNIX usually do only track the hardware cursor
position, but they do not track background color changes.  So my screen
reader has currently no way to follow the currently selected menu item,
which makes the menu as such, rather useless because it is very
time-inefficient for me.

Similar to my recent request about allowing blink-matching-paren being
set to 'jump, and therefore getting the old hardware-cursor behaviour
back, I'd like to ask if there is a way to enhance the new text mode
menu with a user option to have the terminal cursor jump to the
currently selected menu item?  

-- 
CYa,
  ⡍⠁⠗⠊⠕

Re: The new text-mode menu and the cursor in -nw mode

[Eli Zaretskii]

> From: Mario Lang <mlang@delysid.org>
> Date: Mon, 28 Apr 2014 14:21:46 +0200
> 
> The new text mode menu is nice, something I have been waiting for since
> a long time.  However, I have an accessibility issue with it: When I hit
> F10 to go to the menu, the menu pops up and the selected item is
> highlighted with a change of color, but the terminal cursor stays at the
> place in the currently selected buffer.  I can see that this makes sense
> for some use cases (you can still see where point is while navigating
> the menu), but it is quite a problem to (some) blind users of Emacs.

Actually, we turn off the cursor while a menu is active, because
otherwise it can show through the menu, which is ugly.  We turn it
back on only when the menu pops down, i.e. the user either makes a
selection or quits the menu.

> Screen readers on UNIX usually do only track the hardware cursor
> position, but they do not track background color changes.  So my screen
> reader has currently no way to follow the currently selected menu item,
> which makes the menu as such, rather useless because it is very
> time-inefficient for me.
> 
> Similar to my recent request about allowing blink-matching-paren being
> set to 'jump, and therefore getting the old hardware-cursor behaviour
> back, I'd like to ask if there is a way to enhance the new text mode
> menu with a user option to have the terminal cursor jump to the
> currently selected menu item?  

Please tell more, as I don't yet understand the requirements (and
don't have Unix screen readers anywhere near me to try this myself).
E.g., is it OK to have the cursor turned off when the menu is
displayed?  IOW, are you saying that only the cursor coordinates
matter, not whether it is visible or not?

Next, what happens when some sub-menu is selected?  When the user
does this, we redraw large portions of the screen (to remove the old
menu and display the new one), and each screen line that is partially
or fully redrawn causes cursor motion -- will this cause those
portions that are redrawn to be read, and if so, is that OK?

Also, moving to a different menu item will generally display the
corresponding help echo in the echo area -- does it matter whether the
help echo is read before or after the newly highlighted menu item?

Finally, what exactly is read by the screen reader, given a cursor
position?  That is, how does the reader know when to stop reading
stuff off the screen?  I'm asking this because the TTY menus overlay
the buffer text at some arbitrary place, so where the menu item ends,
there's some random text, which typically starts in the middle of a
word.  If the reader will read that, you will hear a terrible
gibberish.

Thanks.

Re: The new text-mode menu and the cursor in -nw mode

[Mario Lang]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Mario Lang <mlang@delysid.org>
>> Date: Mon, 28 Apr 2014 14:21:46 +0200
>> 
>> The new text mode menu is nice, something I have been waiting for since
>> a long time.  However, I have an accessibility issue with it: When I hit
>> F10 to go to the menu, the menu pops up and the selected item is
>> highlighted with a change of color, but the terminal cursor stays at the
>> place in the currently selected buffer.  I can see that this makes sense
>> for some use cases (you can still see where point is while navigating
>> the menu), but it is quite a problem to (some) blind users of Emacs.
>
> Actually, we turn off the cursor while a menu is active, because
> otherwise it can show through the menu, which is ugly.  We turn it
> back on only when the menu pops down, i.e. the user either makes a
> selection or quits the menu.

As far as I am concerned, this is only a visual trick, see below.

>> Screen readers on UNIX usually do only track the hardware cursor
>> position, but they do not track background color changes.  So my screen
>> reader has currently no way to follow the currently selected menu item,
>> which makes the menu as such, rather useless because it is very
>> time-inefficient for me.
>> 
>> Similar to my recent request about allowing blink-matching-paren being
>> set to 'jump, and therefore getting the old hardware-cursor behaviour
>> back, I'd like to ask if there is a way to enhance the new text mode
>> menu with a user option to have the terminal cursor jump to the
>> currently selected menu item?  
>
> Please tell more, as I don't yet understand the requirements (and
> don't have Unix screen readers anywhere near me to try this myself).
> E.g., is it OK to have the cursor turned off when the menu is
> displayed?  IOW, are you saying that only the cursor coordinates
> matter, not whether it is visible or not?

Exactly.  Typical screen readers do not differenciate between
visiable and non-visible cursors.  What they do, is to position
the viewing area around the (visible or not) hardware cursor.

So, if we keep the hardware cursor invisible, that is perfectly fine to
me.   What I need is the hardware cursor positioned at the currently
selected item in the menu.  Either on the first character, or the
character preceding it, would be most convenient.

> Next, what happens when some sub-menu is selected?  When the user
> does this, we redraw large portions of the screen (to remove the old
> menu and display the new one), and each screen line that is partially
> or fully redrawn causes cursor motion -- will this cause those
> portions that are redrawn to be read, and if so, is that OK?

It would be ideal to only reposition the cursor when the drawing has
been complete.

> Also, moving to a different menu item will generally display the
> corresponding help echo in the echo area -- does it matter whether the
> help echo is read before or after the newly highlighted menu item?

I don't think it matters much, although it might be better to display
the echo are change *after* the menu change has been redrawn.

> Finally, what exactly is read by the screen reader, given a cursor
> position?  That is, how does the reader know when to stop reading
> stuff off the screen?  I'm asking this because the TTY menus overlay
> the buffer text at some arbitrary place, so where the menu item ends,
> there's some random text, which typically starts in the middle of a
> word.  If the reader will read that, you will hear a terrible
> gibberish.

I am a braille user.  A braille display typically consists of 40 or 80
characters.  When the hardware cursor position changes, my braille
display is typically updated around that cursor position.  As a braille
user, I can make sense of text that is mixed, i.e., I can understand
that a part of the text is related to the menu item, and the rest is
related to the text that was displayed before the menu overlayed this
area of the screen.  Speech users typically can also cope with this
situation.  However, as mention in this thread, what is important to
both groups, is
that the hardware cursor position is update if the currently
"highlighted" area of the screen changes, so that the screen reader can
"follow" the hightlight around on the screen.


> Thanks.

Thanks for asking, and answering so swiftly.

-- 
CYa,
  ⡍⠁⠗⠊⠕ | Debian Developer <URL:http://debian.org/>
  .''`. | Get my public key via finger mlang/key@db.debian.org
 : :' : | 1024D/7FC1A0854909BCCDBE6C102DDFFC022A6B113E44
 `. `'
   `-      <URL:http://delysid.org/>  <URL:http://www.staff.tugraz.at/mlang/>

Re: The new text-mode menu and the cursor in -nw mode

[Eli Zaretskii]

> From: Mario Lang <mlang@delysid.org>
> Date: Mon, 28 Apr 2014 20:14:37 +0200
> 
> > Next, what happens when some sub-menu is selected?  When the user
> > does this, we redraw large portions of the screen (to remove the old
> > menu and display the new one), and each screen line that is partially
> > or fully redrawn causes cursor motion -- will this cause those
> > portions that are redrawn to be read, and if so, is that OK?
> 
> It would be ideal to only reposition the cursor when the drawing has
> been complete.

I don't think we can do that.  I'm not an expert, but I think text
terminals must move the cursor to where they are going to write stuff,
before actually writing it.

> > Finally, what exactly is read by the screen reader, given a cursor
> > position?  That is, how does the reader know when to stop reading
> > stuff off the screen?  I'm asking this because the TTY menus overlay
> > the buffer text at some arbitrary place, so where the menu item ends,
> > there's some random text, which typically starts in the middle of a
> > word.  If the reader will read that, you will hear a terrible
> > gibberish.
> 
> I am a braille user.  A braille display typically consists of 40 or 80
> characters.  When the hardware cursor position changes, my braille
> display is typically updated around that cursor position.  As a braille
> user, I can make sense of text that is mixed, i.e., I can understand
> that a part of the text is related to the menu item, and the rest is
> related to the text that was displayed before the menu overlayed this
> area of the screen.

The menu is displayed in different colors.  But since you say that
colors are "lost in translation", I wonder whether it is indeed as
easy as you describe to understand where the menu item ends and the
overlaid text begins.

> However, as mention in this thread, what is important to both
> groups, is that the hardware cursor position is update if the
> currently "highlighted" area of the screen changes, so that the
> screen reader can "follow" the hightlight around on the screen.

That's exactly what bothers me: updating a menu might, and generally
will, change more than one place on the screen.  As a trivial example,
moving to the next menu item will redraw both the one which was
highlighted before and the one that is to be highlighted now.  Screen
readers will probably read both (and the help echo in between them),
and I'm not sure the user will understand what is going on, not
without some training.

Re: The new text-mode menu and the cursor in -nw mode

[Mario Lang]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Mario Lang <mlang@delysid.org>
>> Date: Mon, 28 Apr 2014 20:14:37 +0200
>> 
>> > Next, what happens when some sub-menu is selected?  When the user
>> > does this, we redraw large portions of the screen (to remove the old
>> > menu and display the new one), and each screen line that is partially
>> > or fully redrawn causes cursor motion -- will this cause those
>> > portions that are redrawn to be read, and if so, is that OK?
>> 
>> It would be ideal to only reposition the cursor when the drawing has
>> been complete.
>
> I don't think we can do that.  I'm not an expert, but I think text
> terminals must move the cursor to where they are going to write stuff,
> before actually writing it.

Yes, sorry, I was unclear on that.  I know what you describe.  I ment to
say: After the screen has been drawn, set the cursor to the beginning of
the highlighted area.

>> > Finally, what exactly is read by the screen reader, given a cursor
>> > position?  That is, how does the reader know when to stop reading
>> > stuff off the screen?  I'm asking this because the TTY menus overlay
>> > the buffer text at some arbitrary place, so where the menu item ends,
>> > there's some random text, which typically starts in the middle of a
>> > word.  If the reader will read that, you will hear a terrible
>> > gibberish.
>> 
>> I am a braille user.  A braille display typically consists of 40 or 80
>> characters.  When the hardware cursor position changes, my braille
>> display is typically updated around that cursor position.  As a braille
>> user, I can make sense of text that is mixed, i.e., I can understand
>> that a part of the text is related to the menu item, and the rest is
>> related to the text that was displayed before the menu overlayed this
>> area of the screen.
>
> The menu is displayed in different colors.  But since you say that
> colors are "lost in translation", I wonder whether it is indeed as
> easy as you describe to understand where the menu item ends and the
> overlaid text begins.

Trust me, it is, this is something that blind users have to cope with,
and we know how to do that.

>> However, as mention in this thread, what is important to both
>> groups, is that the hardware cursor position is update if the
>> currently "highlighted" area of the screen changes, so that the
>> screen reader can "follow" the hightlight around on the screen.
>
> That's exactly what bothers me: updating a menu might, and generally
> will, change more than one place on the screen.  As a trivial example,
> moving to the next menu item will redraw both the one which was
> highlighted before and the one that is to be highlighted now.  Screen
> readers will probably read both (and the help echo in between them),
> and I'm not sure the user will understand what is going on, not
> without some training.

I am not really sure about speech based screen readers, but some have
ways to cope with very fast screen updates, and present the changes in a
"meaningful" way.  All I know, as a braille user, is that the hardware
cursor position is essential to indicate the highlighted area.  Screen
update ordering is totally irrelevant to me, since the screen updates
happen way to fast to actually "see" them.  However, what I need is the
hardware cursor on the highlighted item, such that my screen reader
shows the text "around" the highlight area, instead of going stuck in
the middle of the currently active window.

-- 
CYa,
  ⡍⠁⠗⠊⠕ | Debian Developer <URL:http://debian.org/>
  .''`. | Get my public key via finger mlang/key@db.debian.org
 : :' : | 1024D/7FC1A0854909BCCDBE6C102DDFFC022A6B113E44
 `. `'
   `-      <URL:http://delysid.org/>  <URL:http://www.staff.tugraz.at/mlang/>

Re: The new text-mode menu and the cursor in -nw mode

[Eli Zaretskii]

> From: Mario Lang <mlang@delysid.org>
> Date: Mon, 28 Apr 2014 21:14:35 +0200
> 
> >> However, as mention in this thread, what is important to both
> >> groups, is that the hardware cursor position is update if the
> >> currently "highlighted" area of the screen changes, so that the
> >> screen reader can "follow" the hightlight around on the screen.
> >
> > That's exactly what bothers me: updating a menu might, and generally
> > will, change more than one place on the screen.  As a trivial example,
> > moving to the next menu item will redraw both the one which was
> > highlighted before and the one that is to be highlighted now.  Screen
> > readers will probably read both (and the help echo in between them),
> > and I'm not sure the user will understand what is going on, not
> > without some training.
> 
> I am not really sure about speech based screen readers, but some have
> ways to cope with very fast screen updates, and present the changes in a
> "meaningful" way.  All I know, as a braille user, is that the hardware
> cursor position is essential to indicate the highlighted area.  Screen
> update ordering is totally irrelevant to me, since the screen updates
> happen way to fast to actually "see" them.  However, what I need is the
> hardware cursor on the highlighted item, such that my screen reader
> shows the text "around" the highlight area, instead of going stuck in
> the middle of the currently active window.

Please try the patch below.  If it gives good results, I will install
it.

Thanks.

--- src/term.c~0	2014-03-21 08:34:40 +0200
+++ src/term.c	2014-04-29 17:20:24 +0300
@@ -2897,6 +2897,13 @@ tty_menu_display (tty_menu *menu, int x,
 	  menu_help_paneno = pn - 1;
 	  menu_help_itemno = j;
 	}
+      /* Take note of the coordinates of the active menu item, to
+	 display the cursor there.  */
+      if (mousehere)
+	{
+	  row = y + i;
+	  col = x;
+	}
       display_tty_menu_item (menu->text[j], max_width, face, x, y + i,
 			     menu->submenu[j] != NULL);
     }
@@ -3177,6 +3184,7 @@ tty_menu_activate (tty_menu *menu, int *
   bool first_time;
   Lisp_Object selectface;
   int first_item = 0;
+  int col, row;
 
   /* Don't allow non-positive x0 and y0, lest the menu will wrap
      around the display.  */
@@ -3364,6 +3372,11 @@ tty_menu_activate (tty_menu *menu, int *
 			    faces, x, y, first_item, 1);
 	  tty_hide_cursor (tty);
 	  fflush (tty->output);
+	  /* The call to display help-echo below will move the cursor,
+	     so remember its current position as computed by
+	     tty_menu_display.  */
+	  col = cursorX (tty);
+	  row = cursorY (tty);
 	}
 
       /* Display the help-echo message for the currently-selected menu
@@ -3373,6 +3386,10 @@ tty_menu_activate (tty_menu *menu, int *
 	{
 	  help_callback (menu_help_message,
 			 menu_help_paneno, menu_help_itemno);
+	  /* Move the cursor to the beginning of the current menu
+	     item, so that screen readers and other accessibility aids
+	     know where the active region is.  */
+	  cursor_to (sf, row, col);
 	  tty_hide_cursor (tty);
 	  fflush (tty->output);
 	  prev_menu_help_message = menu_help_message;

Re: The new text-mode menu and the cursor in -nw mode

[Mario Lang]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Mario Lang <mlang@delysid.org>
>> Date: Mon, 28 Apr 2014 21:14:35 +0200
>> 
>> >> However, as mention in this thread, what is important to both
>> >> groups, is that the hardware cursor position is update if the
>> >> currently "highlighted" area of the screen changes, so that the
>> >> screen reader can "follow" the hightlight around on the screen.
>> >
>> > That's exactly what bothers me: updating a menu might, and generally
>> > will, change more than one place on the screen.  As a trivial example,
>> > moving to the next menu item will redraw both the one which was
>> > highlighted before and the one that is to be highlighted now.  Screen
>> > readers will probably read both (and the help echo in between them),
>> > and I'm not sure the user will understand what is going on, not
>> > without some training.
>> 
>> I am not really sure about speech based screen readers, but some have
>> ways to cope with very fast screen updates, and present the changes in a
>> "meaningful" way.  All I know, as a braille user, is that the hardware
>> cursor position is essential to indicate the highlighted area.  Screen
>> update ordering is totally irrelevant to me, since the screen updates
>> happen way to fast to actually "see" them.  However, what I need is the
>> hardware cursor on the highlighted item, such that my screen reader
>> shows the text "around" the highlight area, instead of going stuck in
>> the middle of the currently active window.
>
> Please try the patch below.  If it gives good results, I will install
> it.

That is exactly what I was looking for, thank you!
Tested on trunk.  While I can see that this might be counted as a
feature by some, I'd like to argue that this change should be installed
to the release branch as well.  Thanks for a very quick response.

> Thanks.
>
> --- src/term.c~0	2014-03-21 08:34:40 +0200
> +++ src/term.c	2014-04-29 17:20:24 +0300
> @@ -2897,6 +2897,13 @@ tty_menu_display (tty_menu *menu, int x,
>  	  menu_help_paneno = pn - 1;
>  	  menu_help_itemno = j;
>  	}
> +      /* Take note of the coordinates of the active menu item, to
> +	 display the cursor there.  */
> +      if (mousehere)
> +	{
> +	  row = y + i;
> +	  col = x;
> +	}
>        display_tty_menu_item (menu->text[j], max_width, face, x, y + i,
>  			     menu->submenu[j] != NULL);
>      }
> @@ -3177,6 +3184,7 @@ tty_menu_activate (tty_menu *menu, int *
>    bool first_time;
>    Lisp_Object selectface;
>    int first_item = 0;
> +  int col, row;
>
>    /* Don't allow non-positive x0 and y0, lest the menu will wrap
>       around the display.  */
> @@ -3364,6 +3372,11 @@ tty_menu_activate (tty_menu *menu, int *
>  			    faces, x, y, first_item, 1);
>  	  tty_hide_cursor (tty);
>  	  fflush (tty->output);
> +	  /* The call to display help-echo below will move the cursor,
> +	     so remember its current position as computed by
> +	     tty_menu_display.  */
> +	  col = cursorX (tty);
> +	  row = cursorY (tty);
>  	}
>
>        /* Display the help-echo message for the currently-selected menu
> @@ -3373,6 +3386,10 @@ tty_menu_activate (tty_menu *menu, int *
>  	{
>  	  help_callback (menu_help_message,
>  			 menu_help_paneno, menu_help_itemno);
> +	  /* Move the cursor to the beginning of the current menu
> +	     item, so that screen readers and other accessibility aids
> +	     know where the active region is.  */
> +	  cursor_to (sf, row, col);
>  	  tty_hide_cursor (tty);
>  	  fflush (tty->output);
>  	  prev_menu_help_message = menu_help_message;

-- 
CYa,
  ⡍⠁⠗⠊⠕ | Debian Developer <URL:http://debian.org/>
  .''`. | Get my public key via finger mlang/key@db.debian.org
 : :' : | 1024D/7FC1A0854909BCCDBE6C102DDFFC022A6B113E44
 `. `'
   `-      <URL:http://delysid.org/>  <URL:http://www.staff.tugraz.at/mlang/>

Re: The new text-mode menu and the cursor in -nw mode

[Eli Zaretskii]

> From: Mario Lang <mlang@delysid.org>
> Date: Tue, 29 Apr 2014 17:20:58 +0200
> 
> > Please try the patch below.  If it gives good results, I will install
> > it.
> 
> That is exactly what I was looking for, thank you!

Thanks for testing.

> Tested on trunk.  While I can see that this might be counted as a
> feature by some, I'd like to argue that this change should be installed
> to the release branch as well.  Thanks for a very quick response.

It's not a new feature, it's fix of a new feature introduced for
v24.4, so I always intended to install it on the emacs-24 branch.  Now
done.