bug#45260: 28.0.50; Maybe flex should only sort when there is no sorting metadata?

bug#45260: 28.0.50; Maybe flex should only sort when there is no sorting metadata?

[Omar Antolín Camarena]

Hi! This is isn't quite a bug report, but more like a technical question
and suggestion.

Briefly, the flex completion style does its own sorting of candidates by
something called "completion-score". It does this by adjusting the
metadata of the completion table to introduce new display-sort-function
and cycle-sort-function properties that sort by completion-score. If the
completion table already had a display-sort-function or a
cycle-sort-function, then the orginal function is called first and then
the result is sorted by completion-score.

I was a little surprised by this last behavior and think the following
is more natural: if the original completion table has *no*
display-sort-function then a new one that does sorting by
completion-score is introduced, but if the original table does have sort
functions, keep those intact! My reasoning is that if a completion table
bothers specifying sort functions, it is probably because they sort
things in a useful order and it's a bad idea to mess with it.

Here's the concrete example where I came across this behavior. Daniel
Mendler is working on a library called consult [1], which provides a
series of commands that rely on completing-read. One of the commands is
consult-line, which prompts you for a line from your current buffer and
jumps to it. The idea, of course, is that depending on your
completion-styles, specifying a line in the buffer with completion might
require typing very little.

The consult-line function sets the display-sort-function and
cycle-sort-function to #'identity in the metadata of the table it hands
to completing read. This is done so that the completions buffer or
icomplete display the matching lines in the order those lines occur in
the buffer. Daniel noticed [2] that under fido-mode the lines became
disordered! I set out to investigate and eventually figured out that
this was because fido-mode overrides the user's setting of
completion-styles setting it to '(flex), and the flex completion styles
uses:

(put 'flex 'completion--adjust-metadata 'completion--flex-adjust-metadata)

to install the completion--flex-adjust-metadata function which modifies
the completion table metadata to compose any existing sort-functions
with one that sorts by completion-score.

Now, I do understand the purpose of flex's sorting by completion-score
and think it does a fair job of making flex more useful. I only wonder
if always sorting by completion-score is a good idea, and propose
instead sorting by completion-score only in the absence of sort-function
metadata.

So, my questions are:

1. Why does flex sort even if there is display/cycle-sort-functions
present? Is there some good reason or use case I'm not seeing?

2. If not, would you consider change completion--flex-adjust-metadata so that it only does the sorting by completion-score in case there is no display/cycle-sort-function metadata present?

[1] Consult, https://github.com/minad/consult
[2] "Candidate order is changed when filtering in fido-mode", https://github.com/minad/consult/issues/57


In GNU Emacs 28.0.50 (build 1, x86_64-w64-mingw32)
 of 2019-12-28 built on W550S
Windowing system distributor 'Microsoft Corp.', version 10.0.19041
System Description: Microsoft Windows 10 Home (v10.0.2004.19041.685)

Recent messages:
Auto-saving...done
Starting new Ispell process c:/Users/Omar/Utils/hunspell/bin/hunspell.exe with default dictionary...done
Checking spelling of OCCURRENCE...
OCCURRENCE is correct
Mark set [2 times]
Auto-saving...done
Saving file c:/Users/Omar/advent-of-code/2020/day15.pl...
Wrote c:/Users/Omar/advent-of-code/2020/day15.pl
Discard unstaged changes in 2020/day15.pl? (y or n) y
Reverting buffer ‘day15.pl’.

Configured using:
 'configure --without-compress-install --without-dbus --with-modules
 'CFLAGS=-O2 -g3''

Configured features:
XPM JPEG TIFF GIF PNG RSVG SOUND NOTIFY W32NOTIFY ACL GNUTLS LIBXML2
HARFBUZZ ZLIB TOOLKIT_SCROLL_BARS MODULES THREADS JSON PDUMPER LCMS2 GMP

Important settings:
  value of $LANG: en_US
  locale-coding-system: cp1252

Major mode: Lisp Interaction

Minor modes in effect:
  magit-auto-revert-mode: t
  global-git-commit-mode: t
  async-bytecomp-package-mode: t
  shell-dirtrack-mode: t
  show-paren-mode: t
  electric-pair-mode: t
  beginend-global-mode: t
  beginend-prog-mode: t
  global-dot-mode: t
  dot-mode: t
  marginalia-mode: t
  minibuffer-electric-default-mode: t
  minibuffer-depth-indicate-mode: t
  gcmh-mode: t
  override-global-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ox hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ox
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ox-texinfo hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ox-texinfo
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ox-publish hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ox-publish
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ox-org hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ox-org
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ox-odt hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ox-odt
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ox-md hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ox-md
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ox-man hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ox-man
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ox-latex hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ox-latex
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ox-icalendar hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ox-icalendar
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ox-html hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ox-html
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ox-beamer hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ox-beamer
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ox-ascii hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ox-ascii
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/org hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/org
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/org-version hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/org-version
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/org-timer hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/org-timer
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/org-tempo hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/org-tempo
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/org-table hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/org-table
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/org-src hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/org-src
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/org-protocol hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/org-protocol
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/org-plot hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/org-plot
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/org-pcomplete hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/org-pcomplete
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/org-num hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/org-num
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/org-mouse hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/org-mouse
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/org-mobile hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/org-mobile
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/org-macs hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/org-macs
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/org-macro hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/org-macro
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/org-loaddefs hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/org-loaddefs
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/org-list hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/org-list
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/org-lint hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/org-lint
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/org-keys hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/org-keys
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/org-install hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/org-install
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/org-inlinetask hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/org-inlinetask
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/org-indent hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/org-indent
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/org-id hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/org-id
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/org-habit hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/org-habit
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/org-goto hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/org-goto
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/org-footnote hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/org-footnote
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/org-feed hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/org-feed
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/org-faces hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/org-faces
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/org-entities hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/org-entities
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/org-element hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/org-element
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/org-duration hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/org-duration
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/org-datetree hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/org-datetree
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/org-ctags hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/org-ctags
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/org-crypt hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/org-crypt
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/org-compat hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/org-compat
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/org-colview hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/org-colview
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/org-clock hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/org-clock
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/org-capture hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/org-capture
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/org-attach hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/org-attach
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/org-attach-git hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/org-attach-git
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/org-archive hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/org-archive
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/org-agenda hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/org-agenda
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ol hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ol
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ol-w3m hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ol-w3m
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ol-rmail hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ol-rmail
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ol-mhe hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ol-mhe
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ol-irc hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ol-irc
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ol-info hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ol-info
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ol-gnus hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ol-gnus
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ol-eww hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ol-eww
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ol-eshell hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ol-eshell
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ol-docview hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ol-docview
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ol-bibtex hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ol-bibtex
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ol-bbdb hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ol-bbdb
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-vala hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-vala
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-tangle hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-tangle
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-table hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-table
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-stan hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-stan
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-sqlite hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-sqlite
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-sql hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-sql
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-shen hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-shen
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-shell hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-shell
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-sed hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-sed
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-screen hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-screen
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-scheme hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-scheme
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-sass hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-sass
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-ruby hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-ruby
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-ref hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-ref
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-R hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-R
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-python hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-python
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-processing hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-processing
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-plantuml hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-plantuml
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-picolisp hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-picolisp
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-perl hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-perl
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-org hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-org
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-octave hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-octave
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-ocaml hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-ocaml
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-mscgen hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-mscgen
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-maxima hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-maxima
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-matlab hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-matlab
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-makefile hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-makefile
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-lua hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-lua
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-lob hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-lob
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-lisp hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-lisp
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-lilypond hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-lilypond
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-ledger hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-ledger
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-latex hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-latex
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-js hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-js
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-java hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-java
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-J hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-J
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-io hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-io
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-hledger hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-hledger
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-haskell hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-haskell
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-groovy hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-groovy
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-gnuplot hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-gnuplot
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-fortran hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-fortran
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-forth hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-forth
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-exp hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-exp
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-eval hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-eval
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-eshell hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-eshell
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-emacs-lisp hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-emacs-lisp
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-ebnf hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-ebnf
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-dot hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-dot
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-ditaa hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-ditaa
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-css hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-css
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-core hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-core
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-coq hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-coq
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-comint hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-comint
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-clojure hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-clojure
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-calc hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-calc
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-C hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-C
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-awk hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-awk
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-asymptote hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-asymptote
c:/Users/Omar/.emacs.d/elpa/org-plus-contrib-20201214/ob-abc hides c:/Users/Omar/emax64/share/emacs/28.0.50/lisp/org/ob-abc

Features:
(shadow sort bbdb-message mailalias bbdb-mua bbdb-com bbdb bbdb-site
timezone mail-extr emacsbug sendmail ispell bug-reference magit-extras
isearch-extras misearch multi-isearch misc misc-text executable vc-git
perl-mode mule-util face-remap magit-submodule magit-obsolete
magit-blame magit-stash magit-reflog magit-bisect magit-push magit-pull
magit-fetch magit-clone magit-remote magit-commit magit-sequence
magit-notes magit-worktree magit-tag magit-merge magit-branch
magit-reset magit-files magit-refs magit-status magit magit-repos
magit-apply magit-wip magit-log which-func magit-diff smerge-mode diff
diff-mode magit-core magit-autorevert autorevert filenotify magit-margin
magit-transient magit-process magit-mode git-commit transient magit-git
magit-section magit-utils crm log-edit message rmc puny dired
dired-loaddefs format-spec rfc822 mml mml-sec epa derived epg epg-config
gnus-util rmail rmail-loaddefs text-property-search time-date mm-decode
mm-bodies mm-encode mail-parse rfc2231 rfc2047 rfc2045 mm-util
ietf-drums mail-prsvr mailabbrev mail-utils gmm-utils mailheader
pcvs-util add-log with-editor async-bytecomp advice async shell
pcomplete server dash email-config markdown-mode rx color noutline
outline comint ansi-color pdf-loader paren elec-pair ace-link avy ring
beginend dot-mode marginalia embark ffap thingatpt icomplete-vertical
icomplete orderless minibuf-eldef mb-depth block-undo imenu
modus-vivendi-theme modus-operandi-theme modus-themes edmacro kmacro
gcmh diminish cl-extra help-mode use-package use-package-ensure
use-package-delight use-package-diminish use-package-bind-key bind-key
easy-mmode use-package-core finder-inf tex-site slime-autoloads info
package easymenu browse-url url-handlers url-parse auth-source cl-seq
eieio eieio-core cl-macs eieio-loaddefs password-cache json subr-x map
url-vars seq byte-opt gv bytecomp byte-compile cconv cl-loaddefs cl-lib
tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type
mwheel dos-w32 ls-lisp disp-table term/w32-win w32-win w32-vars
term/common-win tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode elisp-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch timer
select scroll-bar mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame minibuffer cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite charscript charprop case-table epa-hook
jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice loaddefs
button faces cus-face macroexp files text-properties overlay sha1 md5
base64 format env code-pages mule custom widget hashtable-print-readable
backquote threads w32notify w32 lcms2 multi-tty make-network-process
emacs)

Memory information:
((conses 16 405692 178817)
 (symbols 48 25554 3)
 (strings 32 112339 21643)
 (string-bytes 1 3103871)
 (vectors 16 37642)
 (vector-slots 8 472577 106426)
 (floats 8 255 730)
 (intervals 56 765 635)
 (buffers 1000 18))

bug#45260: 28.0.50; Maybe flex should only sort when there is no sorting metadata?

[João Távora]

Omar Antolín Camarena <omar@matem.unam.mx> writes:

> 1. Why does flex sort even if there is display/cycle-sort-functions
> present? Is there some good reason or use case I'm not seeing?

When I designed flex I wanted it to sort by flexy stuff and keep the
relative order specified by earlier sorting when there are ties.  I've
not seen an example of a table where its original sorting mattered more
_when_ there is some search pattern.  When there is no search pattern,
flex sorting doesn't kick in (because there is nothing to work from).

But you can probably design some other "omar-flex" completion style that
doesn't do this and behaves the way you want.

João

bug#45260: 28.0.50; Maybe flex should only sort when there is no sorting metadata?

[Dmitry Gutov]

On 17.08.2021 13:41, João Távora wrote:
> I've
> not seen an example of a table where its original sorting mattered more
> _when_  there is some search pattern.

Do you have examples of completion tables which do specify sorting, but 
where flex sorting obviously should take precedence?

bug#45260: 28.0.50; Maybe flex should only sort when there is no sorting metadata?

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 677 bytes --]

On Tue, Aug 17, 2021, 13:43 Dmitry Gutov <dgutov@yandex.ru> wrote:

> On 17.08.2021 13:41, João Távora wrote:
> > I've
> > not seen an example of a table where its original sorting mattered more
> > _when_  there is some search pattern.
>
> Do you have examples of completion tables which do specify sorting, but
> where flex sorting obviously should take precedence?
>

No, do you? Love to see them, or the converse. But does it matter? Flex is
for searching with patterns, stable sorted by scores, solving ties with
whatever sort order came first. Dmitry/flex, or Flix, or Flax, may be
something else, you understand? It's by design like this.

João

>

[-- Attachment #2: Type: text/html, Size: 1221 bytes --]

bug#45260: 28.0.50; Maybe flex should only sort when there is no sorting metadata?

[Dmitry Gutov]

On 17.08.2021 17:05, João Távora wrote:
> On Tue, Aug 17, 2021, 13:43 Dmitry Gutov <dgutov@yandex.ru 
> <mailto:dgutov@yandex.ru>> wrote:
> 
>     On 17.08.2021 13:41, João Távora wrote:
>      > I've
>      > not seen an example of a table where its original sorting
>     mattered more
>      > _when_  there is some search pattern.
> 
>     Do you have examples of completion tables which do specify sorting, but
>     where flex sorting obviously should take precedence?
> 
> 
> No, do you? Love to see them, or the converse. But does it matter? Flex 
> is for searching with patterns, stable sorted by scores, solving ties 
> with whatever sort order came first. Dmitry/flex, or Flix, or Flax, may 
> be something else, you understand? It's by design like this.

If there aren't any such significant cases, flex could do what Omar is 
suggesting, solving the reported problem with no real downside. I would 
admit it to be a hackish move, but no more so than the recent commit 
ab23fa4.

Having multiple similar styles is of course an option, but it comes with 
code duplication and associated inconveniences (how many such styles 
should Emacs include OOTB?).

That could be solved with some more general approach, like have both 
styles and completion tables return sorting scores (with documented 
possible values and meanings), and do the combined sorting somewhere in 
the layer above. Then the user could choose the combining strategy, 
configured separately.

bug#45260: 28.0.50; Maybe flex should only sort when there is no sorting metadata?

[Omar Antolín Camarena]

João said:

>  I've not seen an example of a table where its original sorting
>  mattered more _when_ there is some search pattern.

I'd argue the case I mentioned in the bug report is a good example. That
was the case of the consult-line command from Daniel Mendler's consult
package. That commands prompts for a line to go to using completions.
The completion candidates thus are all the lines in the current buffer.
The lines are prefixed with their line numbers in an affixation-function
and the display-sort-function is set to identity, so that the candidates
appear in the order they occur in the buffer.

The flex completion style is a pretty reasonable one to use with that
command, and it is a powerful way to find a line in a buffer by typing
realtively few characters in the line. But if you use a completion UI
that displays the candidates automatically while you type, such as
icomplete-mode (preferably icomplete-vertical-mode since these
candidates are full lines!), it is jarring that flex jumbles the line
order. I believe this to be a natural example where you want to keep the
candidates' order.

But also feel free to close this issue if you strongly disagree with it.
I'm not that invested since I do have what João suggested: an
"omar-flex" style that doesn't sort (through a completion-style I wrote
called orderless which includes a flex substyle, and to be honest I
hardly ever use flex, either João's version or the version in
orderless). Also, if I really want to use flex it is easy enough to
disable the sorting for specific commands such as consult-line:

(defun keep-flex-from-sorting (fn &rest args)
  (put 'flex 'completion--adjust-metadata nil)
  (unwind-protect
      (apply fn args)
    (put 'flex 'completion--adjust-metadata 'completion--flex-adjust-metadata)))

(advice-add 'consult-line :around #'keep-flex-from-sorting)

Since flex's sorting is what you want for the vast majorit of commands,
I think this advice approach is a perfectly serviceable work around.

[Since in Emacs I can usually work around any issue without needing any
upstream changes, I feel my debating skills have deteriorated: I don't
have to convince people my suggestions are a good idea! So I just
suggest but then don't insist. ;) ]

-- 
Omar

bug#45260: 28.0.50; Maybe flex should only sort when there is no sorting metadata?

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 17.08.2021 17:05, João Távora wrote:
>> On Tue, Aug 17, 2021, 13:43 Dmitry Gutov <dgutov@yandex.ru
>> <mailto:dgutov@yandex.ru>> wrote:
>>     On 17.08.2021 13:41, João Távora wrote:
>>      > I've
>>      > not seen an example of a table where its original sorting
>>     mattered more
>>      > _when_  there is some search pattern.
>>     Do you have examples of completion tables which do specify
>> sorting, but
>>     where flex sorting obviously should take precedence?
>> No, do you? Love to see them, or the converse. But does it matter?
>> Flex is for searching with patterns, stable sorted by scores,
>> solving ties with whatever sort order came first. Dmitry/flex, or
>> Flix, or Flax, may be something else, you understand? It's by design
>> like this.
>
> If there aren't any such significant cases, flex could do what Omar is
> suggesting, solving the reported problem with no real downside.

OK. Let's see the code and then it's easier to evaluate if there's "no
real downside".  If it's indeed true, than no problem.

If it helps, I'll just repeat that flex is a pattern-matching scoreing
thing.  You type "foo" and if the first element is "fabrobazo" and the
last one is "foobarbaz", the last one will sort before the first.  A guy
who made something called "hotfuzz" which is the same principle,
apparently, even better I guess.  https://github.com/axelf4/hotfuzz.
Anyway, I'd like it to stay that way even if the table says "fabrobazo"
is more important than "foobarbaz".  When there's no flexy stuff
happening, that's fine, when flex kicks in, it takes over.

> I would admit it to be a hackish move, but no more so than the recent
> commit ab23fa4.

I must admit to tire easily of adjectives like "hackish", "improper",
"dirty", "faux" along with "not my cup of tee" and "there will be subtle
bugs" and judgements based on some vague principia.  These things matter
close to 0 to me.  I need to hear criticism in the form of "there's a
bug when I do xyz", "it becomes 123% slower", or "it duplicates x lines
of code found in lalaland.el".  That's really the only useful criticism
of my programs that I can address.

In this case ab23fa4, bug#49888 directly solves a pretty solid and very
well described bug report that pointed to a vacuum in the flex
completion style.  It reuses functions
`minibuffer--sort-by-length-alpha' and `minibuffer--sort-by-position'
that someone else wrote to maximize code reuse.  To concrete things: can
code reuse and abstractions be improved even further there?  I wanted
to, but there were the problems noted in the FIXME.  You're welcome to
fix those, if you can.  Sorting takes time and patternless flex became
somewhat slower, but it'll become much faster soon, hopefully.

> Having multiple similar styles is of course an option, but it comes
> with code duplication and associated inconveniences (how many such
> styles should Emacs include OOTB?).

Code duplication if you make duplicated.  If you don't make it
duplicated there won't be duplication.  Different styles already reuse
lots of code.  How many styles?  As many as you think are worth
including, I guess.

> That could be solved with some more general approach, like have both
> styles and completion tables return sorting scores (with documented
> possible values and meanings), and do the combined sorting somewhere
> in the layer above. Then the user could choose the combining strategy,
> configured separately.

Sure, I guess, let's see some code.  I trust you to make lots of good
code that generally solves problems well and cares about backward
compatibility.  So let's do that and lay off the vague rethoric, please.

João

bug#45260: 28.0.50; Maybe flex should only sort when there is no sorting metadata?

[João Távora]

Omar Antolín Camarena <omar@matem.unam.mx> writes:

>>  I've not seen an example of a table where its original sorting
>>  mattered more _when_ there is some search pattern.
> I'd argue the case I mentioned in the bug report is a good example. That
> was the case of the consult-line command from Daniel Mendler's consult
> package.

I don't know consult.  Is it a completion table?  For what?

> icomplete-mode (preferably icomplete-vertical-mode since these
> candidates are full lines!), it is jarring that flex jumbles the line
> order.

I see.  A table where the candidates are the lines of a file.  Well, not
jarring the way I look at flex, not the way _I_ use it.  My personal
expectation is to bubble up the tightest matches to the top.  If you're
interested where I picked up the idea, it's from many years of Common
Lisp programming with lots (close to a million IIRC) of named symbols.

Anyway, sounds like a job for a flex-sort-but-only-sometimes completion
style.  Or a flex sorting algorithm with another formula that produces
more ties.

Then, in a file with these 5 lines:

1: fabricate organic ornitopters
2: bar
3: and also using fooey
4: baz
5: using foo

With the pattern "foo", my personal expectation, and the way flex works
right now IINM, is to see 5, 3, 1.

Do you prefer to see 3, 5, 1?  Or keep 1, 3, 5 as presumably the table
stipulates?  Or something else?  Probably your decision in a new 'omar
style.

João

bug#45260: 28.0.50; Maybe flex should only sort when there is no sorting metadata?

[Dmitry Gutov]

On 17.08.2021 19:38, João Távora wrote:
> Dmitry Gutov<dgutov@yandex.ru>  writes:
> 
>> On 17.08.2021 17:05, João Távora wrote:
>>> On Tue, Aug 17, 2021, 13:43 Dmitry Gutov <dgutov@yandex.ru
>>> <mailto:dgutov@yandex.ru>> wrote:
>>>      On 17.08.2021 13:41, João Távora wrote:
>>>       > I've
>>>       > not seen an example of a table where its original sorting
>>>      mattered more
>>>       >_when_   there is some search pattern.
>>>      Do you have examples of completion tables which do specify
>>> sorting, but
>>>      where flex sorting obviously should take precedence?
>>> No, do you? Love to see them, or the converse. But does it matter?
>>> Flex is for searching with patterns, stable sorted by scores,
>>> solving ties with whatever sort order came first. Dmitry/flex, or
>>> Flix, or Flax, may be something else, you understand? It's by design
>>> like this.
>> If there aren't any such significant cases, flex could do what Omar is
>> suggesting, solving the reported problem with no real downside.
> OK. Let's see the code and then it's easier to evaluate if there's "no
> real downside".  If it's indeed true, than no problem.

Please go ahead and fix your own code in any way you like.

Here's an example of user confusion that the current behavior invokes:

https://github.com/company-mode/company-mode/discussions/1223#discussioncomment-1387929

bug#45260: 28.0.50; Maybe flex should only sort when there is no sorting metadata?

[João Távora]

On Tue, Sep 28, 2021 at 12:12 AM Dmitry Gutov <dgutov@yandex.ru> wrote:

> Please go ahead and fix your own code in any way you like.
>
> Here's an example of user confusion that the current behavior invokes:
>
> https://github.com/company-mode/company-mode/discussions/1223#discussioncomment-1387929

That's a very large discussion with lots of company-mode-specific details.
Can you post an example of Emacs -Q where Emacs does something that
would you would not like it to do?  That's a bare minimum for discussion
to resume here.

If you just wish to haphazardly shoot some issue reference to vaguely
blame my work for your company-mode troubles or frustration, I think
that's a very petty thing to do.  If not, please collaborate.

I think I explained that flex has certain semantics regarding sorting.
Yes, those are the semantics that I like when I created it.
I think I've also explained that if you don't like those semantics you
can utilize functions to create a new completion style with semantics
that you like.  I even offered to help you do that.  But then again,
I have no idea what you are bringing up in that link.

João

bug#45260: 28.0.50; Maybe flex should only sort when there is no sorting metadata?

[Dmitry Gutov]

On 28.09.2021 02:25, João Távora wrote:
> On Tue, Sep 28, 2021 at 12:12 AM Dmitry Gutov <dgutov@yandex.ru> wrote:
> 
>> Please go ahead and fix your own code in any way you like.
>>
>> Here's an example of user confusion that the current behavior invokes:
>>
>> https://github.com/company-mode/company-mode/discussions/1223#discussioncomment-1387929
> 
> That's a very large discussion with lots of company-mode-specific details.
> Can you post an example of Emacs -Q where Emacs does something that
> would you would not like it to do?  That's a bare minimum for discussion
> to resume here.

It's an example of a user trying to write a completion table (to be 
consumed by company-capf, as they usually are) with sorting logic and 
tripping over the flex style overriding it.

> If you just wish to haphazardly shoot some issue reference to vaguely
> blame my work for your company-mode troubles or frustration, I think
> that's a very petty thing to do.  If not, please collaborate.

The problem is more or less understood: flex overriding the sorting 
logic even if it's defined in the table.

There is no point in me retelling the discussion. It is itself a piece 
of evidence for the problem existing. Maybe not categorical evidence, 
but evidence nevertheless.

You should read it (the last few messages will suffice).

> I think I explained that flex has certain semantics regarding sorting.
> Yes, those are the semantics that I like when I created it.
> I think I've also explained that if you don't like those semantics you
> can utilize functions to create a new completion style with semantics
> that you like.  I even offered to help you do that.  But then again,
> I have no idea what you are bringing up in that link.

flex is popular enough. People use it. I end up having to diagnose 
people's problems.

Having many similar-but-slightly-different styles available is unlikely 
to reduce my support burdens.

bug#45260: 28.0.50; Maybe flex should only sort when there is no sorting metadata?

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 28.09.2021 02:25, João Távora wrote:
>
> It's an example of a user trying to write a completion table (to be
> consumed by company-capf, as they usually are)

Hm? A table is consumed by any capf-capable frontend. That's not limited
to `company-capf'.

> with sorting logic and tripping over the flex style overriding it.

But where is the "tripping" if, to the best of my knowledge (and given
that you chose this specific Emacs bug), the complaint is about
something that the flex style was designed to do from the very
beginning?  That thing is sort, first and foremost, by how well the
pattern entered by the user matches each candidate.  Doesn't this user
like that?  If not, it's fine.  But not appreciating a behaviour doesn't
count as "tripping over" to me.

> You should read it (the last few messages will suffice).

I tried to, it's full of references to company internals that I can't
evaluate.  Is the user writing a table specifically for company?  Or
modifying company?  Or both?  I must understand it in terms of a simpler
example.

> flex is popular enough. People use it. I end up having to diagnose
> people's problems.

OK, I sympathize with that.  I can see you indeed invest many hours.

> Having many similar-but-slightly-different styles available is
> unlikely to reduce my support burdens.

But "reducing your support burdens" cannot come at the cost of breaking
existing behaviour.  Look at the simple 5-line examples I gave to Omar
(to which there was no reply, unfortunately).  I expect those examples
to sort in a certain way, and I guess Omar would to like them some other
way.

What is the problem with you telling those users to use 'pikachu'
instead of 'flex'?  Or tell them to set a hypothetical variable
flex-prefer-table-sorting to t?

I also suggest that people: "if you're writing a completion table, make
sure you try it without company as well".

João

bug#45260: 28.0.50; Maybe flex should only sort when there is no sorting metadata?

[João Távora]

João Távora <joaotavora@gmail.com> writes:

> I also suggest that people: "if you're writing a completion table, make
> sure you try it without company as well".

Duh, I meant to write "suggest that you tell these people" (that are
burdening you).

João

bug#45260: 28.0.50; Maybe flex should only sort when there is no sorting metadata?

[Dmitry Gutov]

On 28.09.2021 03:36, João Távora wrote:
> Dmitry Gutov <dgutov@yandex.ru> writes:
> 
>> On 28.09.2021 02:25, João Távora wrote:
>>
>> It's an example of a user trying to write a completion table (to be
>> consumed by company-capf, as they usually are)
> 
> Hm? A table is consumed by any capf-capable frontend. That's not limited
> to `company-capf'.

Right. But some frontend is required.

So the references to company-capf in that discussion are necessary, but 
incidental.

>> with sorting logic and tripping over the flex style overriding it.
> 
> But where is the "tripping" if, to the best of my knowledge (and given
> that you chose this specific Emacs bug), the complaint is about
> something that the flex style was designed to do from the very
> beginning?  That thing is sort, first and foremost, by how well the
> pattern entered by the user matches each candidate.  Doesn't this user
> like that?  If not, it's fine.  But not appreciating a behaviour doesn't
> count as "tripping over" to me.

Since we're talking about a beginner learning the c-a-p-f API, saying 
"it is intended to work this way" is rather off base. I think the 
question is rather which choice will ultimately lead to less friction.

While still leading to the expected behavior in all practical cases, of 
course.

>> You should read it (the last few messages will suffice).
> 
> I tried to, it's full of references to company internals that I can't
> evaluate.  Is the user writing a table specifically for company?  Or
> modifying company?  Or both?  I must understand it in terms of a simpler
> example.

They're asking how to write a completion table which does sorting.

It later turned out that they have the flex completion style set up in 
personal configuration. If you follow the discussion, you can note this 
conflict wasn't particularly easy to discover.

And the practice of asking for "full reproduction starting with emacs 
-Q" in such threads feels fairly out of place.

>> Having many similar-but-slightly-different styles available is
>> unlikely to reduce my support burdens.
> 
> But "reducing your support burdens" cannot come at the cost of breaking
> existing behaviour.  Look at the simple 5-line examples I gave to Omar
> (to which there was no reply, unfortunately).  I expect those examples
> to sort in a certain way, and I guess Omar would to like them some other
> way.

But would it? Break existing behavior in any scenario that matters?

Do we know of any completion table which specifies 
display-sort-function, where we're sure we want 'flex' to re-sort anyway?

> What is the problem with you telling those users to use 'pikachu'
> instead of 'flex'?  Or tell them to set a hypothetical variable
> flex-prefer-table-sorting to t?

Discovery.

> I also suggest that people: "if you're writing a completion table, make
> sure you try it without company as well".

Not something I'm eager to recommend to somebody just starting out. It 
wouldn't help here either, I believe.

bug#45260: 28.0.50; Maybe flex should only sort when there is no sorting metadata?

[João Távora]

On Tue, Sep 28, 2021 at 1:38 PM Dmitry Gutov <dgutov@yandex.ru> wrote:

> So the references to company-capf in that discussion are necessary, but
> incidental.

So it should be possible to describe the problem in simpler terms.  Which
is what I'm asking you to do.  In the simplest terms _possible_ of course.

> Since we're talking about a beginner learning the c-a-p-f API, saying
> "it is intended to work this way" is rather off base. I think the
> question is rather which choice will ultimately lead to less friction.
>
> While still leading to the expected behavior in all practical cases, of
> course.

Yes, and the expected behaviour is what happened.

It's not about the beginner developer here, it's about the user.
The beginner developer of the c-a-p-f API has to do more work
than the user to understand what the purpose is of the things
she is invoking.  If she invokes 'flex' in some program, she must
be aware of what she's doing, of how 'flex' works.

> >> You should read it (the last few messages will suffice).
> >
> > I tried to, it's full of references to company internals that I can't
> > evaluate.  Is the user writing a table specifically for company?  Or
> > modifying company?  Or both?  I must understand it in terms of a simpler
> > example.
>
> They're asking how to write a completion table which does sorting.
>
> It later turned out that they have the flex completion style set up in
> personal configuration. If you follow the discussion, you can note this
> conflict wasn't particularly easy to discover.
>
> And the practice of asking for "full reproduction starting with emacs
> -Q" in such threads feels fairly out of place.

I'm sorry.   We need a common ground to be able to communicate technical
problems.  I don't know if it's unfashionable in "such threads" but
I'm pretty sure
that you're accustomed to seeing such recipes in the Emacs bug tracker.

> But would it? Break existing behavior in any scenario that matters?

See my reply to Omar.

> > What is the problem with you telling those users to use 'pikachu'
> > instead of 'flex'?  Or tell them to set a hypothetical variable
> > flex-prefer-table-sorting to t?
>
> Discovery.

So you're talking about changing defaults.  But flex is not even the default,
so I don't understand.  Why are people using it if they're not content with
what it does?

Do a 'flix' style, as I've suggested multiple times! Look,  if you're right and
it represents much less friction it should rapidly become popular and your
burdens will be solved.  It probably take few lines of code.

> > I also suggest that people: "if you're writing a completion table, make
> > sure you try it without company as well".
> Not something I'm eager to recommend to somebody just starting out. It
> wouldn't help here either, I believe.

So you decide that it's a good idea for a beginner to mix company internals
into her adventures.... Questionable, I'd say.

João

bug#45260: 28.0.50; Maybe flex should only sort when there is no sorting metadata?

[Dmitry Gutov]

On 28.09.2021 19:09, João Távora wrote:
> On Tue, Sep 28, 2021 at 1:38 PM Dmitry Gutov <dgutov@yandex.ru> wrote:
> 
>> So the references to company-capf in that discussion are necessary, but
>> incidental.
> 
> So it should be possible to describe the problem in simpler terms.  Which
> is what I'm asking you to do.  In the simplest terms _possible_ of course.

I have already done this here. Twice, I think?

And more, if we count the underlying technical issue that is the 
headline of this report.

> It's not about the beginner developer here, it's about the user.

The user-developer line is fairly blurred in this particular case.

> The beginner developer of the c-a-p-f API has to do more work
  than the user to understand what the purpose is of the things
> she is invoking.  If she invokes 'flex' in some program, she must
> be aware of what she's doing, of how 'flex' works.

That seems optimistic. Especially since 'flex' is a rare exception among 
other completion styles, in that it affects sorting (through an opaque 
mechanism), and not just filtering.

>> But would it? Break existing behavior in any scenario that matters?
> 
> See my reply to Omar.

Omar brought in a scenario of something the user does, starting with the 
UI, and the full use case.

You responded with an abstract piece of behavior.

In your scenario, which UI is the user going through? What completion 
table would they be querying?

How much effort would it be for such UI to switch to a completion table 
that doesn't specify the sorting function, so that 'flex' still applies 
its sorting, to the satisfaction of your hypothetical user, while now 
satisfying Omar's user as well?

>>> What is the problem with you telling those users to use 'pikachu'
>>> instead of 'flex'?  Or tell them to set a hypothetical variable
>>> flex-prefer-table-sorting to t?
>>
>> Discovery.
> 
> So you're talking about changing defaults.  But flex is not even the default,
> so I don't understand.  Why are people using it if they're not content with
> what it does?
> 
> Do a 'flix' style, as I've suggested multiple times! Look,  if you're right and
> it represents much less friction it should rapidly become popular and your
> burdens will be solved.  It probably take few lines of code.

I didn't do 'flex' and I don't do 'flix'. I'm not the 
person/place/piece-of-software people come to for completions styles. I 
have to deal with existing styles. Even if just one of them has a 
problem, that can bring a question or a bug report on my doorstep.

Furthermore, 'flex' being in the core implicitly gives it a strong 
recommendation over alternatives.

>>> I also suggest that people: "if you're writing a completion table, make
>>> sure you try it without company as well".
>> Not something I'm eager to recommend to somebody just starting out. It
>> wouldn't help here either, I believe.
> 
> So you decide that it's a good idea for a beginner to mix company internals
> into her adventures.... Questionable, I'd say.

See the thread.

bug#45260: 28.0.50; Maybe flex should only sort when there is no sorting metadata?

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

> And more, if we count the underlying technical issue that is the
> headline of this report.

Underlying technical isse?  As I see it it's a simple feature request.
It even starts with the word "Maybe".  It started with a description of
the "Consult" system, which was very clear.  Then I understood the
problem and decided that it goes against some fundamental flex
behaviour.  That is all.

Anyhoo, here's my poor man's consult directly from an Emacs -Q
*scratch*:

   ;; This buffer is for text that is not saved, and for Lisp evaluation.
   ;; To create a file, visit it with C-x C-f and enter text in its buffer.
   (require 'cl-lib)
    
   (defun confabulate ()
     (interactive)
     (let* ((lines (cl-loop for i from 1
                            for s in (split-string (buffer-string) "\n")
                            collect (format "%s: %s" i s)))
            (choice (completing-read
                     "Hey, choose a line: "
                     (lambda (string pred action)
                       (if (eq action 'metadata)
                           `(metadata (cycle-sort-function . ,#'identity))
                         (complete-with-action action lines string pred)))
                     nil t)))
       (goto-line (string-to-number choice))))

Eval this and M-x confabulate.  Pick your favourite UI (vertico?) and
style (orderless?)  Does this work how you want?  I guess it does, no
flex involved.

But I use flex and icomplete in my .emacs, and I like the way it works
for me, just fine and quite a useful little snippet.

I type 'oos' and I get line 7 as my first candidate.  Omar would prefer
line 1, I guess.  I don't.  I suppose orderless gives line 1.  I'm fine
with that.

> That seems optimistic. Especially since 'flex' is a rare exception
> among other completion styles, in that it affects sorting (through an
> opaque mechanism), and not just filtering.

I don't care if it's an exception, it's also the only one to do scoring
of completions.  Quite an important part of it.  Without it, it's pretty
useless.  You think the original table order is _more_ important and
should override scoring.  I don't get that, but I certainly won't judge
you for wanting it.  That behaviour is ONE cusotmization variable away.
Or a new completion style, which isn't complicated.  Do you want me to
add this customization variable, yes or no?

> Omar brought in a scenario of something the user does, starting with
> the UI, and the full use case.
> You responded with an abstract piece of behavior.

Abstract?  I gave a super-clear example, with a three-number answer, and
now have even coded it in Elisp.  Really, I'm not the one with the
"non-categorical evidence" here!

> I didn't do 'flex' and I don't do 'flix'. I'm not the
> person/place/piece-of-software people come to for completions
> styles. I have to deal with existing styles. Even if just one of them
> has a problem, that can bring a question or a bug report on my
> doorstep.

If it's not your bug and you don't know how to answer the question, then
just say that to the user.  I sympathize with your care for beginner
users, I've said that.  But here there's nothing I can do.  Flex works
as designed, to the best of my knowledge.

João