tabulated-list-mode needs incremental search option

tabulated-list-mode needs incremental search option

[Jean Louis]

* Chong Yidong <cyd@stupidchicken.com> [2020-11-10 10:25]:
> Hi Jean Louis,
> 
> > But now when you gave me references on how to filter the view then I
> > can make function for my particular case myself to enter one or more
> > words to filter by words. That is workaround, not real solution.
> > ...
> > In my opinion the above is enough for me. You said how / k filtering
> > is implemented, even I used it but I forgot it. You reminded me, so
> > now I can filter in similar way myself.
> >
> > If it happens that you decide to improve it, I have idea how you could
> > do it: simply concatenating all tabulated items with space between and
> > then using matching words on that list.
> 
> Glad the interim solution can be done.
> 
> I think the incremental typing functionality is best implemented as its
> own minor mode.  It should not be tied to tabulated-list-mode.
> 
> This could be modelled after Emacs' completion functions.  In
> completion, you start typing and Emacs draws in information from the
> buffer to try and complete what you have in the minibuffer.  In the new
> incremental processing mode, you start typing and Emacs alters the
> buffer to reflect what's in the minibuffer.  Similar to completion, it
> would not be mode-specific, but different modes would be able to
> customize the effects in a way that makes sense for that mode.
> 
> Could you file a feature request in the Emacs bug tracker, or send an
> email to emacs-devel to open a discussion?  It's possible there's
> already related functionality that I'm not current aware of, e.g. in an
> external package.

I will Cc: this email to emacs-devel.

In general I am advising that every application with choices offers
among others the narrowing incremental search.

Be it by invoking a key binding first or directly by simply
typing. Because there are various key bindings in almost every mode it
is better to start narrowing incremental seach when user press special
key like / or something.

To come out of narrowed list one could press other key binding and see
the original list.

In my program I am using (setq tabulated-list-entries entries) and it
is common in tabulated-list-mode.

In my program I am using (setq tabulated-list-entries entries) and it
is common in tabulated-list-mode.

So I guess it could be implemented as you said in a derived mode
invoked by programmer or program where:

- user press some key to start a real-time narrowing incremental
  search

- program remembers the original `tabulated-list-entries' as
  ORIGINAL-DATA

- program can concatenate parts of each entry into strings (or maybe
  use other method) during the incremental narrowing search

- user starts typing and lines shown in tabulated-list-mode get
  narrowed to selected lines

- program obtains the ID numbers and displays only those ID numbers by
  using (setq tabulated-list-entries NARROWED-DATA) and redisplays

- use could press ENTER and narrowing incremental search stops at that
  point, user is facing new tabulated list and can handle it in usual
  manner.

- user may press a key to go back to go to original
  `tabulated-list-entries' and program does (setq
  tabulated-list-entries ORIGINAL-DATA) and re-displays

Jean

Re: tabulated-list-mode needs incremental search option

[Stefan Monnier]

> In general I am advising that every application with choices offers
> among others the narrowing incremental search.

Very well, but to me this is much too vague to know what it means.
AFAICT, if every application uses `completing-read` when it has
a choice, then it does offer "narrowing incremental search" because
`icomplete-mode` (among other minibuffer completion schemes) does offer
something that narrow the search incrementally.


        Stefan

Re: tabulated-list-mode needs incremental search option

[Jean Louis]

* Stefan Monnier <monnier@iro.umontreal.ca> [2020-11-10 16:18]:
> > In general I am advising that every application with choices offers
> > among others the narrowing incremental search.
> 
> Very well, but to me this is much too vague to know what it means.

Drew also reminded me to express me better. The above statement refers
to general applications and not only to Emacs.

I am referring to real-time collection narrowing just as
icomplete-mode or ido-mode or ivy, helm.

> AFAICT, if every application uses `completing-read` when it has
> a choice, then it does offer "narrowing incremental search" because
> `icomplete-mode` (among other minibuffer completion schemes) does offer
> something that narrow the search incrementally.

I am using tabulated-list-mode to classify hyperdocuments and browse
trees of information. That will arrive to GNU ELPA when polished. But
first must come the `emacs-libpq' PostgreSQL database dynamic module
which developers are working on it.

One way to search is filtering results like in M-x list-packages with
/ a for example to filter by archive. That uses tabulated-list-mode

Filtering the final result of matches is useful in any application
offering larger selections.

Completion frameworks are excellent improvement for Emacs to narrow
the collection of items. Yet not all of them are developing with good
principles of work.

We think of minibuffer being minibuffer. Of course it can expand and
become as large as main buffer. Yet historical and practical use is
that we look and focus into minibuffer, it is down there, and it is
mini.

Above minibuffer there is modeline. People do look into modeline
including me.

What hapens when using ivy-mode is that modeline jumps up-down when
completing. And I need to use external function (without knowing the
license) to accommodate ivy to behave more useful.

Some things are apparently pretty static in Emacs. For example Help
menu and modeline. Does anybody expects Help menu to jump down to
middle of screen?

In the same way I do not expect modeline to jump in the middle of
screen. Minibuffer completion that raises modeline does not enhance
legibility, it enhances confusion.

Good example how is that solved ind `M-x helm' may be seen here:
https://emacs-helm.github.io/helm/images/helm-buffers-list.gif

- minibuffer stays where it is, it does not enlarge
- modline does not jump up and down
- separate window is used for completion

More about Helm:  https://emacs-helm.github.io/helm/

icomplete-mode is far from being complete in the above context of
making the legible visual non-disturbing selection.

Here is short demonstration and comparison, video (17M):
https://gnu.support/images/2020/11/2020-11-10/2020-11-10-20:10:49.ogv

The demonstration is showing:

- how Hyperscope for Emacs is using tabulated-list-mode to browse the
  tree and sets of hyperdocuments

- entering sub-tree is done with arrow left, going up the tree is
  arrow right

- if selection or collection is very long, browsing is tedious, it
  could be 15000 items, I have 19000 items now.

- basic filtering is implemented, yet this does not replace the real
  time incremental matching (aka ivy, helm)

- querying with ivy or helm or any completing-read function is
  possible. What is missing here is ivy-occur so when narrowing have
  been performed that user gets narrowed list.

- there is comparison between ivy where modeline is jumping up and
  down because minibuffer is enlarging. Those elements of user
  interface considered static (even if they are not) are not supposed
  to jump up and down, it confuses user. I do use external function to
  remedy it and hope that ivy will get it internally. Manual does not
  warn users that modeline is to jump up and down when using
  minibuffer. 

- helm mode shows that modeline remains where it is expected to be
  just as it is explained to users in the Emacs Manual. 

As tabulated-mode-list can be used for selection of items, for listing
of items, it is good to have real-time incremental narrowing
function.

Difference between the tabulated-mode-list feature (narrowing
incremental real time matchin) and minibuffer completing framework
functions is that the first uses full screen and does not mess the
modeline and general user interface.

In this application it would be better to remain in full screen and to
narrow by using minibuffer.

When using completing frameworks there are visual disturbances to
screen:

- screen is divided in half either by modeline jumping up or by split
  window
  
- minibuffer is not mini any more

- modeline jumps up and down

With the new feature I am proposing we would enable Emacs Users to
create full screen menus and selections from menu.

Jean

Re: tabulated-list-mode needs incremental search option

[Juri Linkov]

> In general I am advising that every application with choices offers
> among others the narrowing incremental search.

The term "incremental search" is too ambiguous to use here.  You had a better
alternative below: "incremental matching" or "incremental narrowing".
And "incremental completion" would be too specific to icomplete.el.

> What hapens when using ivy-mode is that modeline jumps up-down when
> completing.

The modeline doesn't jump up-down when a list of completions
is displayed in the pop-up buffer *Completions*.

RE: tabulated-list-mode needs incremental search option

[Drew Adams]

> > In general I am advising that every application with choices offers
> > among others the narrowing incremental search.
> 
> The term "incremental search" is too ambiguous to use here.  You had a
> better alternative below: "incremental matching" or "incremental narrowing".
> And "incremental completion" would be too specific to icomplete.el.
> 
> > What hapens when using ivy-mode is that modeline jumps up-down when
> > completing.
> 
> The modeline doesn't jump up-down when a list of completions
> is displayed in the pop-up buffer *Completions*.

+1 to all that Juri said here.

I believe that Jean is talking about some kind of dynamic,
incremental filtering.  Vanilla Emacs has such a thing for
incremental search (which is maybe what inspired Jean's use
of that term) and for Ido and Icomplete.  In none of those
cases does the filtering affect/use *Completions*.

And I think that what Jean is maybe talking about is
something general, which could be applied in multiple
contexts - completion, but not just completion.

Re: tabulated-list-mode needs incremental search option

[Jean Louis]

* Juri Linkov <juri@linkov.net> [2020-11-11 22:24]:
> > In general I am advising that every application with choices offers
> > among others the narrowing incremental search.
> 
> The term "incremental search" is too ambiguous to use here.  You had a better
> alternative below: "incremental matching" or "incremental narrowing".
> And "incremental completion" would be too specific to icomplete.el.
> 
> > What hapens when using ivy-mode is that modeline jumps up-down when
> > completing.
> 
> The modeline doesn't jump up-down when a list of completions
> is displayed in the pop-up buffer *Completions*.

How do I set it?

Re: tabulated-list-mode needs incremental search option

[Jean Louis]

* Juri Linkov <juri@linkov.net> [2020-11-11 22:24]:
> > In general I am advising that every application with choices offers
> > among others the narrowing incremental search.
> 
> The term "incremental search" is too ambiguous to use here.

I understand.

Please see here how other people refer to incremental search:
https://en.wikipedia.org/wiki/Incremental_search

Quote:

In computing, incremental search, incremental find or real-time
suggestions is a user interface interaction method to progressively
search for and filter through text. As the user types text, one or
more possible matches for the text are found and immediately presented
to the user. This immediate feedback often allows the user to stop
short of typing the entire word or phrase they were looking for. The
user may also choose a closely related option from the presented list.

To make it more specific I was often referring to Emacs completion
functions, built-in or outside packages. It is in that context.

My references are always in the context of built-in or modified
`completing-read' function and related functions used by helm, ivy,
ido, fido, icomplete, selectrum, etc.

I do not use the context of C-s or M-C-s which is also incremental
search.

> You had a better alternative below: "incremental matching" or
> "incremental narrowing".  And "incremental completion" would be too
> specific to icomplete.el.

When user chooses a file, user is completing file name and works on
specific file.

For my work now I am in 2 options, maybe I do look for specific item,
in that sense I am not really completing the item but jumping to the
item within a list in tabulated-list-mode. There is no action
specified on that item. In the second option I am just looking for the
subtree where that item belongs as maybe I wish to add new sets or
subtrees or other items relevant to that item. I do use
completing-read but maybe to enact some actions only closely related
to completed item. Maybe technically I "complete" but result is quite
different.

Re: tabulated-list-mode needs incremental search option

[Juri Linkov]

>> The term "incremental search" is too ambiguous to use here.  You had a better
>> alternative below: "incremental matching" or "incremental narrowing".
>> And "incremental completion" would be too specific to icomplete.el.

Actually, "incremental narrowing" is also too specific to 'C-x n n'.
But as Drew pointed out, the best term would be "incremental filtering".

>> > What hapens when using ivy-mode is that modeline jumps up-down when
>> > completing.
>>
>> The modeline doesn't jump up-down when a list of completions
>> is displayed in the pop-up buffer *Completions*.
>
> How do I set it?

You don't need to set it, this is the default behavior of completion.
It still doesn't allow selecting a completion from the *Completions* buffer
without leaving the minibuffer, but this shortcoming should be eliminated.

Re: tabulated-list-mode needs incremental search option

[Jean Louis]

* Juri Linkov <juri@linkov.net> [2020-11-12 10:35]:
> >> The term "incremental search" is too ambiguous to use here.  You had a better
> >> alternative below: "incremental matching" or "incremental narrowing".
> >> And "incremental completion" would be too specific to icomplete.el.
> 
> Actually, "incremental narrowing" is also too specific to 'C-x n n'.
> But as Drew pointed out, the best term would be "incremental
> filtering".

Fine for me to call it anyhow as long as we agree that we know what we
talk about.

> >> The modeline doesn't jump up-down when a list of completions
> >> is displayed in the pop-up buffer *Completions*.
> >
> > How do I set it?
> 
> You don't need to set it, this is the default behavior of completion.
> It still doesn't allow selecting a completion from the *Completions* buffer
> without leaving the minibuffer, but this shortcoming should be eliminated.

That would substantially change default habits and behavior. 

Re: tabulated-list-mode needs incremental search option

[Jean Louis]

I actually wish to say for that feauture:

real time filtering

Please see emacs-tangents as I am discussing with Thien-Thi about it.
https://lists.gnu.org/archive/html/emacs-tangents/2020-11/index.html

Other question that you may know and I do not know it related to
tabulated-list-mode. How could I get it to return the value of the
tabulated list ID after closing the buffer, and when invoked from
other program or buffer.

Here is what I am trying to do, to make it as menu system where I can
feed entries to be selected:

(defun hyperscope-choice (entries &optional buffer-name)
  "This is meant to be te general choice UI with tabulated list mode"
  (interactive)
  (let* ((buffer-name (if buffer-name buffer-name "*HyperScope*")))
    (pop-to-buffer buffer-name)
    (read-only-mode 0)
    (erase-buffer)
    (read-only-mode 1)        
    (hyperscope-choice-mode)
    (hl-line-mode)
    (setq tabulated-list-entries entries)
    (tabulated-list-print t)))

(defun hyperscope-choice-sql (sql &optional buffer-name)
  (interactive)
  (let* ((buffer-name (if buffer-name buffer-name "*HyperScope*"))
	 (entries (hyperscope-sql-prepare-entries sql)))
    (hyperscope-choice entries buffer-name)))

;; (setq idb (hyperscope-choice-sql "SELECT contacts_id, get_full_contacts_name(contacts_id), '' FROM contacts WHERE contacts_lastname ~* 'louis'" "Choice"))

;; (hyperscope-sql-prepare-entries "SELECT contacts_id, get_full_contacts_name(contacts_id), '' FROM contacts WHERE contacts_lastname ~* 'louis'")

(define-key hyperscope-choice-mode-map (kbd "<RET>") 'hyperscope-get-id)

So this function here I do not know how to make it:

- to kill the buffer and return the value back. I cannot kill the
  buffer this way, I have not yet discovered how.

(defun hyperscope-get-id ()
  "Returns the ID from hyperscope-choice-mode"
  (interactive)
  (let ((id (tabulated-list-get-id)))
    (when id
      (set-register 100 id) ;; character d, I am trying to enter value
                            ;; in register, to obtain it later.
      (kill-this-buffer))))

But buffer does not get killed that I may return to previous program.


* Jean Louis <bugs@gnu.support> [2020-11-10 13:37]:
> * Chong Yidong <cyd@stupidchicken.com> [2020-11-10 10:25]:
> > Hi Jean Louis,
> > 
> > > But now when you gave me references on how to filter the view then I
> > > can make function for my particular case myself to enter one or more
> > > words to filter by words. That is workaround, not real solution.
> > > ...
> > > In my opinion the above is enough for me. You said how / k filtering
> > > is implemented, even I used it but I forgot it. You reminded me, so
> > > now I can filter in similar way myself.
> > >
> > > If it happens that you decide to improve it, I have idea how you could
> > > do it: simply concatenating all tabulated items with space between and
> > > then using matching words on that list.
> > 
> > Glad the interim solution can be done.
> > 
> > I think the incremental typing functionality is best implemented as its
> > own minor mode.  It should not be tied to tabulated-list-mode.
> > 
> > This could be modelled after Emacs' completion functions.  In
> > completion, you start typing and Emacs draws in information from the
> > buffer to try and complete what you have in the minibuffer.  In the new
> > incremental processing mode, you start typing and Emacs alters the
> > buffer to reflect what's in the minibuffer.  Similar to completion, it
> > would not be mode-specific, but different modes would be able to
> > customize the effects in a way that makes sense for that mode.
> > 
> > Could you file a feature request in the Emacs bug tracker, or send an
> > email to emacs-devel to open a discussion?  It's possible there's
> > already related functionality that I'm not current aware of, e.g. in an
> > external package.
> 
> I will Cc: this email to emacs-devel.
> 
> In general I am advising that every application with choices offers
> among others the narrowing incremental search.
> 
> Be it by invoking a key binding first or directly by simply
> typing. Because there are various key bindings in almost every mode it
> is better to start narrowing incremental seach when user press special
> key like / or something.
> 
> To come out of narrowed list one could press other key binding and see
> the original list.
> 
> In my program I am using (setq tabulated-list-entries entries) and it
> is common in tabulated-list-mode.
> 
> In my program I am using (setq tabulated-list-entries entries) and it
> is common in tabulated-list-mode.
> 
> So I guess it could be implemented as you said in a derived mode
> invoked by programmer or program where:
> 
> - user press some key to start a real-time narrowing incremental
>   search
> 
> - program remembers the original `tabulated-list-entries' as
>   ORIGINAL-DATA
> 
> - program can concatenate parts of each entry into strings (or maybe
>   use other method) during the incremental narrowing search
> 
> - user starts typing and lines shown in tabulated-list-mode get
>   narrowed to selected lines
> 
> - program obtains the ID numbers and displays only those ID numbers by
>   using (setq tabulated-list-entries NARROWED-DATA) and redisplays
> 
> - use could press ENTER and narrowing incremental search stops at that
>   point, user is facing new tabulated list and can handle it in usual
>   manner.
> 
> - user may press a key to go back to go to original
>   `tabulated-list-entries' and program does (setq
>   tabulated-list-entries ORIGINAL-DATA) and re-displays
> 
> Jean