Re: Community improvements to the Emacs Widget Library manual?

Re: Community improvements to the Emacs Widget Library manual?

[Mauro Aranda]

Bryce Carson <bovine@cyberscientist.ca> writes:

 > I searched and read the Customize mailing list. It has more spam than it
 > does messages from Custom.el or Widget.el users.

I didn't know about that mailing list.  I don't feel like it's relevant
as of today.

 > Would anyone like to collaborate to improve The Emacs Widget Library
 > manual? Are there any active Emacs Lisp hackers that actually
 > understand this library at a deep level?

I do think I understand it quite a bit.  I've fixed some bugs, in the
code and in the documentation, and I have some techniques for debugging
the code which sometimes turns out to be a mess of nested widgets
creation.

And sure, I'd love to collaborate on improving the documentation, be it
the info manual that gets shipped with Emacs or with tutorials about how
to use it to get the most out of it, without starting to hate the
library.

 > I'm writing a package using the library, and I'm still quite lost while
 > reading the manual. It's been days that I've spent with the manual, so it
 > is not one or two quick readings. I have studied it, yet I am 
bewildered at
 > times.

I'm absolutely willing to help.  If you have the code somewhere, or if
you want to tell me what are the things you're finding difficult, I can
try to help you out.  And maybe that can give me some ideas about what
to improve in the Widget documentation.

 > One aspect that is confusing is widget definition with widget-specific
 > argument handling. Built-in widgets handle arguments after the
 > keyword-value pairs in widget-specific ways. How do end users create
 > such behaviour in their own widgets? The answer is a function value for
 > the widget-create keyword, but this is a difficult thing to implement.

Please tell more about the difficulties you have encountered. If you
can show a piece of code to show what you expected and what really
happens, then better yet.

 > TL;DR:
 >
 > The Emacs Widget Library manual could use a re-write, preferably
 > following the Diataxis documentation framework if possible. Does
 > anyone want to collaborate over the long-term, creating a study group
 > and editing the manual to a high standard to benefit the community and
 > ourselves?

I'm not sure about a re-write.  But yes, I feel like there's more room
for improvement.  And count me in as one possible collaborator to
improve it.

RE: [External] : Re: Community improvements to the Emacs Widget Library manual?

[Drew Adams]

1. You _are_ the resident expert, Mauro.
Maybe, and hopefully, there will be some
more to come someday.  A public thread
about this effort might just encourage
people to learn and expertize themselves.
Progress encourages participation and
invites more progress.

2. I can't guarantee I'll have time to
help, and any such help would likely be
limited to (hopefully) helpful questions
and some text editing/suggestions.  But
if you think that could help more than
hinder, please cc me in whatever threads
you like.

Thanks for working on this, both of you.
It would sure be great to give Customize
a boost in general understanding and
perhaps usability, by way of its doc (at
least).

I know you're talking here about _widget_
(not Customize) doc, but it's mainly the
Widgetry (TM) that makes Customize code
difficult for many of us.

I've long felt that more users, including
Lispers of all levels, could benefit by
using and leveraging Customize more.

But when trying to encourage that, the
response is often either or both (1) the
UI needs improvement (usually stated less
politely) and (2) the Lisp code behind
widgets is obtuse.

Re: [External] : Re: Community improvements to the Emacs Widget Library manual?

[Mauro Aranda]

Drew Adams <drew.adams@oracle.com> writes:

 > 2. I can't guarantee I'll have time to
 > help, and any such help would likely be
 > limited to (hopefully) helpful questions
 > and some text editing/suggestions.  But
 > if you think that could help more than
 > hinder, please cc me in whatever threads
 > you like.

Will do :-).  Your comments have always been helpful to me to spot
weirdness (or plain bugs) in the Widget code.

Re: Community improvements to the Emacs Widget Library manual?

[Bryce]

Mauro Aranda <maurooaranda@gmail.com> writes:

> Bryce Carson <bovine@cyberscientist.ca> writes:
>
>  > I searched and read the Customize mailing list. It has more spam than it
>  > does messages from Custom.el or Widget.el users.
>
> I didn't know about that mailing list.  I don't feel like it's relevant
> as of today.

It is definitely defunct. I wanted to find some help for using Widget,
so I began subscribing to all sorts of mailing lists related to Emacs
and Lisp with Thunderbird. At the moment I am attempting to get
comfortable with Rmail, but I need to send a message to the Emacs Help
list for an issue I'm having with it (feel free to read it there! I'm
just letting you know that getting up to speed with how mailing lists
work is one of the reasons I haven't replied sooner).

>  > Would anyone like to collaborate to improve The Emacs Widget Library
>  > manual? Are there any active Emacs Lisp hackers that actually
>  > understand this library at a deep level?
>
> I do think I understand it quite a bit.  I've fixed some bugs, in the
> code and in the documentation, and I have some techniques for debugging
> the code which sometimes turns out to be a mess of nested widgets
> creation.

I'd love to hear about that debugging technique. Twice I've witnessed
debug (not edebug) get itself into a circle of evaluating the commands
that pertain to evaluating commands that pertain to evaluating
commands... sorry, I'm sure you understand. Do you know if that may have
had anything to do with Widget's recursive bits? I am not good at using
the debugger, I could do with more practice.

I let debug walk into a recursive loop when trying to debug an error
which occured when I wanted to save the value of a custom variable using
the widget type I defined. (I settled on _merely_ defining a custom type
and using Customize to handle all user interaction for my application,
rather than setting up a buffer using Widgets myself and managing
_everything_).

If I encounter the same error (I have shiny new ones now) I will ask for
specific help resolving it.

> And sure, I'd love to collaborate on improving the documentation, be it
> the info manual that gets shipped with Emacs or with tutorials about how
> to use it to get the most out of it, without starting to hate the
> library.

I'd like to see as much as possible end up in the Widget manual itself.
The manuals that ship with Emacs for some modes are gargantuan, there is
no reason other than politics to not ship any tutorials we write (if
they are quality).

The most one could get out of the library is making a Widgetry RAD
(WidgeTRY™?), I imagine. I can see the possibility for that when I read
the manual and consider the command/event loop of Emacs, and the global
keymap mentioned in the manual.

>  > I'm writing a package using the library, and I'm still quite lost while
>  > reading the manual. It's been days that I've spent with the manual, so it is
>  > not one or two quick readings. I have studied it, yet I am bewildered at
>  > times.
>
> I'm absolutely willing to help.  If you have the code somewhere, or if
> you want to tell me what are the things you're finding difficult, I can
> try to help you out.  And maybe that can give me some ideas about what
> to improve in the Widget documentation.

I haven't made any code public yet because I am still deciding on how I
want to publish it. I may provide the code on my GitHub, or I might
provide it on my personal website to churn up some traffic.

The biggest issue I've had with the library is creating a working SEXP
widget. I wanted to match _any_ symoblic expression that would create an
object of a specific type (using :validate). I have settled on something
much more reasonable (a choice-menu) instead, especially while I dogfood
and bootstrap the application.

The application is a literate programming interface. It's closing in on
0.1.0, but because I am writing the interface using Noweb (I am writing
it literately) it is taking longer, because I am nothing more than a
would-be acolyte of LaTeX. I will ask for help soon once I have a
specific question; before I was not making any notes of errors or where
the project was at, and I also lost some Git history of the project due
to accidentally deleteing it (however, I recovered it with some handy
libre software; I can recall the amusing story, if wanted, but don't
fret I aliased rm to remind me to use trash instead).

>  > One aspect that is confusing is widget definition with widget-specific
>  > argument handling. Built-in widgets handle arguments after the
>  > keyword-value pairs in widget-specific ways. How do end users create
>  > such behaviour in their own widgets? The answer is a function value for
>  > the widget-create keyword, but this is a difficult thing to implement.
>
> Please tell more about the difficulties you have encountered. If you
> can show a piece of code to show what you expected and what really
> happens, then better yet.

I responded to Michael in detail concerning :widget-create and the
handling of further (non-keyword) arguments.

To quote the library,

    | This is the general syntax of a type specification:
    |
    |      NAME ::= (NAME [KEYWORD ARGUMENT]... ARGS)
    |           |   NAME
    |
    |    Where, NAME is a widget name, KEYWORD is the name of a
    | property, ARGUMENT is the value of the property, and ARGS are
    | interpreted in a widget specific way.

ARGS in the general type specification is not one of the pairs of
:keyword value pairs, it is anything else. I forget where it is stated,
but I believe the manual says that any non-keyword value pair arguments
must follow the pairs (if any pairs exist). How to define the handling
of such ARGS in a custom widget is never exampled in the manual; it
certainly isn't a simple (widget-args) function call away.

When replying to Michael, I gave the example of the ITEM widget, which
doesn't handle any of the §5 keyword arguments in a special way, but
will initialize :value on behalf of the user with VALUE (see §5.9).

The description of the ITEM widget, while on the topic, should probably
be expanded to elaborate on what last sentence means. When matching
against widgets, is a (widget-find type) function being used? It would
be nice if there was a Widget Object Model, where all widgets are rooted
in the buffer they belong to, but that's a lofty, far off goal like a
WidgeTRY™ RAD for the Emacs Widget Library.

>  > The Emacs Widget Library manual could use a re-write [...]
>
> I'm not sure about a re-write.  But yes, I feel like there's more room
> for improvement.  And count me in as one possible collaborator to
> improve it.

Great! I hope the possibility is realized.

Considering the age of the library (twenty-three years! [if the
earlier copyright year is correct]), and the support for embedded
graphics like GTK+ widgets (so far only WebkitGTK…) _has_ improved,
perhaps this statement could be removed or edited in some way? At least
the part about the _automatic_ improvement in graphical appearance. I
feel like that is not correct, but I could easily be wrong (if these are
merely _goals_ for the library, not the state of affairs).

    7. As support for embedded graphics improve, the widget library will
       be extended to use the GUI features.  This means that your code
       using the widget library will also use the new graphic features
       automatically.

Re: Community improvements to the Emacs Widget Library manual?

[Mauro Aranda]

Bryce <bovine@cyberscientist.ca> writes:

 > Mauro Aranda <maurooaranda@gmail.com> writes:
 >
 >> Bryce Carson <bovine@cyberscientist.ca> writes:

 >> I'm absolutely willing to help.  If you have the code somewhere, or if
 >> you want to tell me what are the things you're finding difficult, I can
 >> try to help you out.  And maybe that can give me some ideas about what
 >> to improve in the Widget documentation.
 >
 > I haven't made any code public yet because I am still deciding on how I
 > want to publish it. I may provide the code on my GitHub, or I might
 > provide it on my personal website to churn up some traffic.
 >
 > The biggest issue I've had with the library is creating a working SEXP
 > widget. I wanted to match _any_ symoblic expression that would create an
 > object of a specific type (using :validate). I have settled on something
 > much more reasonable (a choice-menu) instead, especially while I dogfood
 > and bootstrap the application.

It sounds to me like you want to use a custom widget type whose parent
is the 'restricted-sexp widget, and define a :match function to do the
matching.   Do you recall if you tried to use that widget, or what
widget did you try to use? If you did, do you recall what problems you
ran into?

 >>  > One aspect that is confusing is widget definition with 
widget-specific
 >>  > argument handling. Built-in widgets handle arguments after the
 >>  > keyword-value pairs in widget-specific ways. How do end users create
 >>  > such behaviour in their own widgets? The answer is a function 
value for
 >>  > the widget-create keyword, but this is a difficult thing to 
implement.
 >>
 >> Please tell more about the difficulties you have encountered. If you
 >> can show a piece of code to show what you expected and what really
 >> happens, then better yet.
 >
 > I responded to Michael in detail concerning :widget-create and the
 > handling of further (non-keyword) arguments.
 >
 > To quote the library,
 >
 >     | This is the general syntax of a type specification:
 >     |
 >     |      NAME ::= (NAME [KEYWORD ARGUMENT]... ARGS)
 >     |           |   NAME
 >     |
 >     |    Where, NAME is a widget name, KEYWORD is the name of a
 >     | property, ARGUMENT is the value of the property, and ARGS are
 >     | interpreted in a widget specific way.
 >
 > ARGS in the general type specification is not one of the pairs of
 > :keyword value pairs, it is anything else. I forget where it is stated,
 > but I believe the manual says that any non-keyword value pair arguments
 > must follow the pairs (if any pairs exist). How to define the handling
 > of such ARGS in a custom widget is never exampled in the manual; it
 > certainly isn't a simple (widget-args) function call away.

The widget specific way is handled via a :convert-widget function.

Creation of a widget takes two steps: conversion and creation.

Conversion means to take a specification of the widget to be created
(type and properties) and convert that specification into another one,
suitable for creating the widget.  So if you want to handle those extra
args in a specific way, or to manipulate the widget before creation, you
need a :convert-widget function, that widget-convert (which is
undocumented in the manual) will make sure to call.  It will call the
most specific :convert-widget function first, and then the
:convert-widget of the parents.

You can see an example of this in the following code:

(defun completion--update-styles-options (widget)
   "Function to keep updated the options in 
`completion-category-overrides'."
   (let ((lst (mapcar (lambda (x)
                        (list 'const (car x)))
              completion-styles-alist)))
     (widget-put widget :args (mapcar #'widget-convert lst))
     widget))

(defconst completion--styles-type
   `(repeat :tag "insert a new menu to add more styles"
            (choice :convert-widget completion--update-styles-options)))

Here completion--styles-type holds a type definition for a defcustom,
and the idea is to make the choices available build dynamically.
Without a custom :convert-widget function, the :args would've been
defined in a static way as soon as the type gets defined.  So we use a
:convert-widget function that builds the choices, and puts them into the
:args property, as required by the choice widget.  Then, when creating
it, all the choices in :args show up in the buffer.

 > When replying to Michael, I gave the example of the ITEM widget, which
 > doesn't handle any of the §5 keyword arguments in a special way, but
 > will initialize :value on behalf of the user with VALUE (see §5.9).

Yes.  And that's another undocumented thing.  The :convert-widget
function for the 'item widget is widget-value-convert-widget, which is
documented.

 > The description of the ITEM widget, while on the topic, should probably
 > be expanded to elaborate on what last sentence means. When matching
 > against widgets, is a (widget-find type) function being used?

No, the last sentence "This widget will only match the specified value."
means that the :match function will return non-nil if the value passed
to it is equal to the :value upon creation.

 > It would be nice if there was a Widget Object Model, where all widgets
 > are rooted in the buffer they belong to, but that's a lofty, far off
 > goal like a WidgeTRY™ RAD for the Emacs Widget Library.

Yes, it'd be nice if we could show the hierarchy.  For now, all we have
is widget-browse, and navigate through the super widgets.

 >>  > The Emacs Widget Library manual could use a re-write [...]
 >>
 >> I'm not sure about a re-write.  But yes, I feel like there's more room
 >> for improvement.  And count me in as one possible collaborator to
 >> improve it.
 >
 > Great! I hope the possibility is realized.
 >
 > Considering the age of the library (twenty-three years! [if the
 > earlier copyright year is correct]), and the support for embedded
 > graphics like GTK+ widgets (so far only WebkitGTK…) _has_ improved,
 > perhaps this statement could be removed or edited in some way? At least
 > the part about the _automatic_ improvement in graphical appearance. I
 > feel like that is not correct, but I could easily be wrong (if these are
 > merely _goals_ for the library, not the state of affairs).
 >
 >     7. As support for embedded graphics improve, the widget library will
 >        be extended to use the GUI features.  This means that your code
 >        using the widget library will also use the new graphic features
 >        automatically.

I don't know if that item is relevant anymore.  Maybe should be moved to
the Wishlist section, which surely needs an update.

Re: Community improvements to the Emacs Widget Library manual?

[Michael Heerdegen]

Mauro Aranda <maurooaranda@gmail.com> writes:

> Creation of a widget takes two steps: conversion and creation.
>
> Conversion means to take a specification of the widget to be created
> (type and properties) and convert that specification into another one,
> suitable for creating the widget.  So if you want to handle those extra
> args in a specific way, or to manipulate the widget before creation, you
> need a :convert-widget function, that widget-convert (which is
> undocumented in the manual) will make sure to call.  It will call the
> most specific :convert-widget function first, and then the
> :convert-widget of the parents.

This would be a nice paragraph to add to the manual I think.

> You can see an example of this in the following code:
>
> (defun completion--update-styles-options (widget)
>   "Function to keep updated the options in
> `completion-category-overrides'."
>   (let ((lst (mapcar (lambda (x)
>                        (list 'const (car x)))
>              completion-styles-alist)))
>     (widget-put widget :args (mapcar #'widget-convert lst))
>     widget))
>
> (defconst completion--styles-type
>   `(repeat :tag "insert a new menu to add more styles"
>            (choice :convert-widget completion--update-styles-options)))
>
> Here completion--styles-type holds a type definition for a defcustom,
> and the idea is to make the choices available build dynamically.
> Without a custom :convert-widget function, the :args would've been
> defined in a static way as soon as the type gets defined.  So we use a
> :convert-widget function that builds the choices, and puts them into the
> :args property, as required by the choice widget.  Then, when creating
> it, all the choices in :args show up in the buffer.

And this is a nice example for it!



BTW, one thing I often wondered about (I think it's missing in the
manual): one often sees code using list values like

`(widget-type :keyword1 ,(lambda ...) ...)

What is that?  Already a widget?  A widget definition?  What functions
accept such list values, and how do I create a real widget using such
values?


Michael.

RE: [External] : Re: Community improvements to the Emacs Widget Library manual?

[Drew Adams]

> I've also noticed that I have three separate threads
> of emails related to this subject, some with [External]

That "[External]" is my fault, or rather the
fault of my (company's) mail server.  It gets
added automatically for any msgs that are
outside the company.  Sorry about that.  

Re: [External] : Re: Community improvements to the Emacs Widget Library manual?

[Bryce Carson]

On 7/13/23 17:27, Drew Adams wrote:

>> I've also noticed that I have three separate threads
>> of emails related to this subject, some with [External]
> That "[External]" is my fault, or rather the
> fault of my (company's) mail server.  It gets
> added automatically for any msgs that are
> outside the company.  Sorry about that.
I see! No worries, my university does similar things.

Re: Community improvements to the Emacs Widget Library manual?

[Bryce]

[...]

To quote the library,

    | This is the general syntax of a type specification:
    |
    |      NAME ::= (NAME [KEYWORD ARGUMENT]... ARGS)
    |           |   NAME
    |
    |    Where, NAME is a widget name, KEYWORD is the name of a
    | property, ARGUMENT is the value of the property, and ARGS are
    | interpreted in a widget specific way.

When I first approached the manual I had a difficult time understanding
the full meaning of this meta-language. I did not know what BNF or EBNF
are, and even after having a friend explain it to me I still struggled
to realize that NAME on the second bar is an alternative right-hand
side.

I'm a competent self-taught programmer (and I have a year of formal
education in CS under my belt), but I wasn't familialr with BNF. I now
know it is widely used when describing a formal language grammar, but we
should include something in the manual to at least make it clear that
this is what follows.

We /might/ elect to include a new manual in Emacs overall, or write a
new GNU manual to describe the concepts of Backus-Naur forms (BNF), or
Extended BNF. GNU Bison has a section (§1.1) in its manual which
describes some of the basic concepts of BNF and context-free grammars.
We might simply refer uses to that manual, or link to an external
reference document with a GPL compatible license which could be
redistributed by the FSF or another party friendly to GNU Emacs.


    | 5 Basic Types
    | *************
    |
    | This is the general syntax of a type specification:
    |
    |      NAME ::= (NAME [KEYWORD ARGUMENT]... ARGS)
    |           |   NAME
    |
    |    Where, NAME is a widget name, KEYWORD is the name of a property,
    | ARGUMENT is the value of the property, and ARGS are interpreted in a
    | widget specific way.

I feel there is some ambiguity in the description of this syntax as
well. NAME is said to be a widget name (the symbol which refers to the
type being defined), but in every following use of the syntax the
uppercase word "TYPE" is repeated verbatim, except for ITEM in §5.9.

Shouldn't all type specifications read as follows?

    | 5.5 The ‘editable-field’ Widget
    | ===============================
    |
    | Syntax:
    |
    |      EDITABLE-FIELD ::= (editable-field [KEYWORD ARGUMENT]... [ VALUE ])
    |
    |    The VALUE, if present, is used to initialize the ‘:value’ property.
    | The value should be a string, which will be inserted in the field.  This
    | widget will match all string values.

TYPE is said to be a type name, so it should be the name of the type
being specified throughout the manual. If I've misunderstood the grammar
or the description, we should instead correct the description in §5
(Basic Types) and fix ITEMs type.

Re: Community improvements to the Emacs Widget Library manual?

[Eli Zaretskii]

> From: Bryce <bovine@cyberscientist.ca>
> Cc: maurooaranda@gmail.com, emacs-devel@gnu.org
> Date: Sun, 23 Jul 2023 17:06:45 -0600
> 
> [...]
> 
> To quote the library,
> 
>     | This is the general syntax of a type specification:
>     |
>     |      NAME ::= (NAME [KEYWORD ARGUMENT]... ARGS)
>     |           |   NAME
>     |
>     |    Where, NAME is a widget name, KEYWORD is the name of a
>     | property, ARGUMENT is the value of the property, and ARGS are
>     | interpreted in a widget specific way.
> 
> When I first approached the manual I had a difficult time understanding
> the full meaning of this meta-language. I did not know what BNF or EBNF
> are, and even after having a friend explain it to me I still struggled
> to realize that NAME on the second bar is an alternative right-hand
> side.
> 
> I'm a competent self-taught programmer (and I have a year of formal
> education in CS under my belt), but I wasn't familialr with BNF. I now
> know it is widely used when describing a formal language grammar, but we
> should include something in the manual to at least make it clear that
> this is what follows.
> 
> We /might/ elect to include a new manual in Emacs overall, or write a
> new GNU manual to describe the concepts of Backus-Naur forms (BNF), or
> Extended BNF. GNU Bison has a section (§1.1) in its manual which
> describes some of the basic concepts of BNF and context-free grammars.
> We might simply refer uses to that manual, or link to an external
> reference document with a GPL compatible license which could be
> redistributed by the FSF or another party friendly to GNU Emacs.

A cross-reference to another manual where BNF is explained is TRT,
IMO.