Re: Larger GC thresholds for non-interactive Emacs

[Lynn Winebarger <owinebar@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 13547 bytes --]

On Fri, Jun 24, 2022, 2:54 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Lynn Winebarger <owinebar@gmail.com>
> >
> > > > It would help if there were some taxonomy of features/design
> > > > points for
> > > > reference.  Is the bug database being used for that?
> > >
> > > I don't think so (IIUC what you mean by that), and I don't really see
> > > how bugs could serve in that role.
> >
> > I agree, but some of the bug reports are classified as something along
> > the lines of "feature request" or "wishlist".
>
> If you look at those, you will see that almost all of them request
> minor features that have no impact on the design whatsoever.  It is
> very rare to have a feature request on the bug tracker that changes
> the design or introduces new designs.  I would even dare to say it
> never happened, definitely not on my watch.
>



> > I haven't found any other database tracking anything corresponding
> > to features, proposed or otherwise.
>
> We do have etc/TODO.  Did you look there?
>

Yes, but it's not indexed so there's no good way to reference particular
items the way bugs are.


> > Once something is in place, no matter how bare-bones, the key would be
> > maintainers mandating that any substantive introduction or revision of
> > features be tied to appropriate entries for the corresponding
> > design/spec documentation, in the same way and for the same reason
> > coding standards are enforced.
>
> I think you are greatly underestimating the efforts required to
> maintain documentation of this kind, let alone the effort for creating
> it basically from scratch.
>

I just have a lot of sympathy for the points you made in that portable
dumper thread, regarding the cost of accepting substantial contributions
from transitory contributors.
Anyone can create a new feature or improvement and maintain their own
variant, but it's a lot easier for them if the project takes over
maintenance and breaking it becomes a rejection test for future changes.
Everything you say below is sensible.  You and the other (current and
future) core maintainers are the ones who have to make the trade-off, after
all.  I'm going to add some comments below, though, because I think my
perspective on what is useful is different.  If they are obvious or too
idiosyncratic to be useful in the context of attracting new
contributors/maintainers, feel free to ignore them.

>
> Programmers usually don't like writing documentation, and most of them
> cannot write good documentation even if they did want, in large part
> because that take years of practice actually doing it.  In a project
> like Emacs, it is nigh impossible to force contributors and developers
> do what they are reluctant to do in the first place.  Without everyone
> updating the documentation as they change code, any documentation
> about design and internals will quickly become outdated, and thus not
> just useless, but downright harmful.  We have trouble keeping even our
> user documentation up-to-date.
>

Is it harmful?  I suppose it depends on who's relying on it.  Diving into
the source of a project with Emacs's complexity with a particular design
change in mind,  personally I prefer to have some guideposts knowing that
they may be dated, then ask knowledgeable people if there seems to be a
disagreement between the doc and the code.  When that happens on a public
forum like this list, then there's some reference.

It's true, I have a low bar.  It might not seem like it, but my preference
is to explore a question as much as I can on my own before bothering the
experts.  In the best case, the question that stumps me also stumps the
expert - then I know I've done adequate study.   With Emacs (and often
other free software projects), it's just hard to tell what I should be
responsible for studying prior to asking questions.  Beyond the code
itself, of course.


Are you aware of _any_ Free Software project that has any useful
>
documentation of this kind?


I know some "Open Source" projects (eg chromium) that have useful docs.
Also projects with academic roots (Larceny Scheme is still my go-to) tend
to have them.  I'm assuming that in this context, the OSS projects, at
least, are not "Free Software project[s]" in the sense you intended, even
if the software produced by those projects happens to be Free.
I don't do development for a living, so when I first looked into the code
of those corporate projects, the general culture around demanding
uniformity of build systems and supported configurations was quite a
culture shock after being accustomed to the approach of Free Software
projects.  So I'm assuming you intend to refer to projects that have
freedom as an organizing principle, and hence exhibit a self-selection bias
toward diverse and idiosyncratic preferences and goals among their
contributors.  Or something along those lines?

Every project I ever knew or was involved
> with which tried ended up abandoning that.  A case in point is GDB,
> which once had a "GDB Internals" manual -- it was always outdated, and
> was eventually scrapped because the maintainers decided they could not
> and didn't want to invest the effort.


I remember thinking it was a shame.  But, my bar for usefulness is fairly
low.

XEmacs tried in its time to
> have the internals documented, but that was basically a one-man
> effort, and even in it whole chapters were never written.  Etc. etc.
>
> I think there's some conclusion waiting here.
>

I think we have different problem scopes in mind.  I'd take some kind of
indexed wiki as a substantial improvement, particularly if there were links
between tags in the code as well as files and directories cross-referencing
features and VC logs, with the primary archive of discussion happening
there rather than an email list (as with discussion around bugs).  I'd just
like to be able to determine answers to questions like:  why is preventing
the allocation of lisp objects by mmap necessary?  What purpose does
ralloc.c serve?  Do they still serve a purpose?, and know whether I've done
enough self-study to justify asking the experts.

> That's assuming the maintainers consider such documentation
> > important enough for enabling future potential contributors and
> > maintainers to hold otherwise useful contributions in limbo.
>
> And you think we don't?  Of course we do!  We also want to release
> Emacs much more frequently, and we want it to have no bugs, and a few
> other things.  But somehow, each such goal cannot be reached for some
> boring practical reasons, none of them related to their importance in
> our eyes.
>

I did not mean to imply you don't consider them important (I can see it
might read that way).  I meant "important enough [relative to all the
competing priorities]".  I don't know of any other way of expressing
importance except relative to competing priorities subject to
resource constraints.


> We do in general consider it somewhat more important to develop Emacs
> and accept contributions than to document its internals, that's true.
> Which is why I said up-thread that without someone who'd volunteer to
> do this job (thus dedicating his/her time almost exclusively to it),
> this will probably never happen, given the current state of our
> resources, which are barely enough to maintain the status quo and move
> forward at some reasonable pace.
>
> > >  Most of the design changes and
> > > redesigns in Emacs were developed without any bug report, simply
> > > because those who did the job knew that a particular group of problems
> > > needs to be taken care of.
> >
> > It's not like there isn't any discussion or justification of features
> > offered prior to code being integrated into the main branch. It's more
> > a challenge of how to weave what's there into a coherent account of
> > what's going on in the code.
>
> IME, you are wrong in that assumption.  The significant changes in
> Emacs design are almost never publicly discussed, not in the way that
> would allow someone to glean the design from them.
>

Then,  I should understand Daniel Colascione's approach with the portable
dumper preview thread and the more recent one on a revised gc are either
outliers - or that I'm misreading their scope relative to the discussion of
design issues they provide? [ That's a straight question to resolve my
ignorance, not intended to be snarky. ]

> It would just be easier to automate some sort of design note
> > extraction from the git log if references to mails could be associated
> > with relevant features.  I've never used org, but maybe there's some
> > syntax that would be useful?  Or maybe some notation from supercite
> > for precise pulling of relevant text from list archives?
>
> I wish this were true.  It isn't.  The discussions and the commit log
> messages rarely describe the design, and in many cases barely describe
> even the particular implementation.  In a project where people with
> write access can commit changes without any review, I don't know how
> can anything else to be expected.  We basically rely on each
> individual to do the job perfectly and contribute to what you want to
> see documented.  The results are before our eyes, and they shouldn't
> surprise anyone.
>

True enough.


>
> > > > https://github.com/rocky/elisp-bytecode
> > > > That is really useful documentation that would ideally be in the
> emacs docs
> > > > or etc directory.
> > >
> > > That's not design description, though.
> >
> > You probably have a more nuanced view than me on this.  It's true,
> > that document is focused on the specification (the "what") rather than
> > the (detailed) "how" and "why" - is that what you mean?
>
> Of course.
>
> > Either way, if you want to understand how the operational semantics
> > of emacs lisp work in practice, a document of that sort is
> > invaluable.  Without that, a document explaining the "why" isn't
> > going to be able to be very concrete.
>
> I agree, but the "what" is usually already available in the comments
> to the code, though not everywhere and for every significant feature.
> The "why" and "how", by contrast are almost completely missing.
>

In the particular case of the bytecode spec, at least it gives me the sense
of the invariants that are being maintained by the implementation.  Trying
to reverse engineer what those invariants are from a giant C switch
statement is always tricky, because they tend to be expressed with
boilerplate code, and then sometimes you have clever use of the fact that
cases are non-exclusive and "fall through" without an explicit break, etc.
 That's where it intersects with "design" for me.

OTOH, the same switch statement could be written as a dispatch table of
higher-order functions exploiting proper tail-recursion in a way I would
feel comfortable enough with to not feel the need to lean on a spec.  But
that assumes those higher-order functions clearly express the invariants
they impose.  It's a personal preference, I know.

To summarize: I'm not sure we should continue this discussion, because
> I don't see where is it going and what could it possibly change in
> practice.  I agree with the value of having all of that documented,
> I'm just saying it's a large job that needs dedicated individuals, and
> I don't see how that could be replaced by any semi-automated means.
>

I was surprised my off-the-cuff remark about trawling the archives
generated any response in the first place, to be honest.

One thing (I think) some of those OSS IDE projects do well is eat their own
dogfood in terms of features for projects/documentation/collaborative
development models generally.  AFAICT, Emacs has support (either in core or
in non-core packages) for a lot of different tools/approaches/etc to these
issues, but doesn't seem to take advantage of them for emacs development
itself, beyond the integrated bug reporting and version control systems.
There's support for all these project formats and tags - why isn't there a
standard choice of those imposed on the emacs projects itself, e.g. with a
centralized repo of pre-generated tags and auto-generated extraction of
documentation/data structure definitions?  Maybe I just haven't read that
section of the manual or the right document in "etc" yet - I freely admit
my ignorance.  As it is, I've just assumed that the core maintainer(s) have
weighed the trade-offs and decided imposing that kind of constraint, even
among the core maintainers, would cost more than it is worth (in attracting
contributors and maintainers).

If there was an "emacs-devel" mode where the tags (and files would be tags
considered as entries in a directory "document") linked to a database
cross-referenced with the feature database, the VC logs, and a wiki-ish
interface that could record both documentation and "talk".  Sort of like
"help" mode, but with a wikipedia-ish feel.  The contributors to that piece
of code and maintainers covering the related features would be
auto-subscribed to the entry, and any discussion would at least get
recorded in a way linked to the thing being discussed.  I'm not sure
contributors would want to be auto-subscribed like that, but then maybe
their contribution should be reviewed by someone who would be.

It's just an idea - whether it's useful or practical for the purpose of
facilitating the recruitment of new contributors/maintainers, given that
some culture change would likely be required, is not something I have the
perspective to judge.  I do know I would find something like what I've
described very useful.

Lynn

[-- Attachment #2: Type: text/html, Size: 17766 bytes --]