Re: project.el semantics

[John Wiegley <jwiegley@gmail.com>]

>>>>> John Wiegley <jwiegley@gmail.com> writes:

> I thank you for your nearly endless patience while we dance this dance. I
> will write up a description of higher-level semantics later today, against
> which we can look at the various API options.

What is a "project"?

A project, semantically, is a set of sets: {{A, ...}, {B, ...}, ...}, where
each member set defines a "view" of the entities of the project. Example
views:

  - All files related to the project, in any way
  - All files containing source code (i.e., no generated content)
  - All files under version control
  - All generated files and directories
  - The arguments one would provide to GNU find to produce a similar list of
    files to any of the above
  - The directories and files relating to the project, with some indication of
    whether those directories should be recursed (and how)
  - etc., etc.

There are potentially infinitely many ways to view a project, depending on
what the users wishes to do. Examples of such uses include:

  - Build a TAGS file
  - Run a query-replace operation
  - Populate a dired buffer with the project file paths
  - Run a find-grep on the project's contents
  - Provide data to a higher-level package like Projectile
  - Delete all generated files
  - Build an ignore list for dired-omit-files
  - etc., etc.

The main question is, how do we determine this "set of sets", since the fully
generated set is potentially infinite?

To achieve this, we take the "project set" to be intensionally defined, and
provide an API for querying along the multiple views. The data returned from
these views should be generative (using stream.el), so that element-by-element
actions do not require excess allocation up front.

This API should be sufficient to define the set of utility functions that
operate in terms of it, and of taking whatever actions we wish in relation to
a "project".

Having described it this way, it seems there at least three sub-languages
involved: (1) A description language to define projects; (2) a query language
to define the desired view of a project; and (3) a selection language to
determine how the results of the query should be returned. In SQL terms, it
might look like:

    SELECT (selection) FROM (definition) WHERE (query);

For convenience, the DEFINITION can often be determined from context, such as
by querying Git or VC.

For convenience, the SELECTION can often be reduced to small set of options,
such as "all file paths" or "files and directory roots".

For convenience, the QUERY can be reduced to a set of possibilities, like
"source code", or "ignored files", or "library files", etc.

While the surface API might encode these conveniences as short-hands for what
are likely to be common uses, the underlying API should allow the general form
of a rich query. The project is, in a sense, a database of entities, and it's
hard to pre-determine all possible useful queries of such data.

How these conveniences are implemented is completely open; `cl-defgeneric'
methods that encode the QUERY&SELECTION against an auto-determined DEFINITION
is just fine, and is how project.el is written today.

What is presently missing is the lower-level query API, and the freedom to
specify the DEFINITION, QUERY and SELECTION in more flexible ways.

*That said*, I'd be willing to postone such an API until a future version,
since, from the user's perpective, it could be seen as an implementation
detail behind the convenience APIs.

So maybe this has come full circle, but if so, I hope it adds some clarity.

John