Re: master f8fed41 2/3: image-dired: Improve XDG compliance

Re: master f8fed41 2/3: image-dired: Improve XDG compliance

[Eli Zaretskii]

I can't say I'm happy with creeping of XDG specifics into our
application code.  It's too platform-specific, so it gets in the way
of either adhering to platform-specific directories on each platform,
or having completely platform-independent identical directories on all
platforms.  Both of these alternatives could be useful, but XDG
provides neither: it only caters to a single platform, and leaves the
rest to manual coding.  Here's the result:

  (defcustom image-dired-main-image-directory
    (or (xdg-user-dir "PICTURES") "~/pics/")

Why "~/pics/"?  What platform behaves like that?  Probably none.

XDG is also prone to fashion changes.  What will we do when it goes
out of fashion, and we have dozens of xdg-user-dir calls in our
application code? change all of them to some latest desktop fashion?

To me, xdg.el was supposed to be a platform-specific layer on which to
base platform-independent abstractions.  Those abstractions should
have other platform-specific layers for other platforms, and the
abstractions should then be used in application code.  So when
fashions change on this or that platform, we need to change only the
code that implements the abstractions.  Our handling of XDG in
startup.el to find the user's Emacs home directory fits this
description; code like the above does not.

Can we agree not to use xdg.el in application code, and instead
develop those abstractions and start using them?  For example, how
about having a function user-pics-directory, which will return the
appropriate file name for the underlying platform?  Doing that will
not diminish our support for XDG (as long as it lasts), but will allow
us to have application code XDG-clean, so to speak.

OK?

Re: master f8fed41 2/3: image-dired: Improve XDG compliance

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

> XDG is also prone to fashion changes.  What will we do when it goes
> out of fashion, and we have dozens of xdg-user-dir calls in our
> application code? change all of them to some latest desktop fashion?

What do you mean by this? Shouldn't XDG as a specification prevent this
from being an issue?

-- 
	Philip Kaludercic

Re: master f8fed41 2/3: image-dired: Improve XDG compliance

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: Stefan Kangas <stefan@marxist.se>,  emacs-devel@gnu.org
> Date: Mon, 25 Oct 2021 16:14:33 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > XDG is also prone to fashion changes.  What will we do when it goes
> > out of fashion, and we have dozens of xdg-user-dir calls in our
> > application code? change all of them to some latest desktop fashion?
> 
> What do you mean by this? Shouldn't XDG as a specification prevent this
> from being an issue?

Sorry, I don't understand the second question.  What can "XDG as a
specification" prevent?  It cannot prevent XDG ceasing to be the
fashion, that's entirely out of control of any specification.
Fashions some and go, and when they come, users rightfully expect us
to support the latest fashions, and we generally comply.

Re: master f8fed41 2/3: image-dired: Improve XDG compliance

[Stefan Monnier]

Philip Kaludercic [2021-10-25 16:14:33] wrote:
> Eli Zaretskii <eliz@gnu.org> writes:
>> XDG is also prone to fashion changes.  What will we do when it goes
>> out of fashion, and we have dozens of xdg-user-dir calls in our
>> application code? change all of them to some latest desktop fashion?
> What do you mean by this?

The YEH spec from the Wayland++ guys is quite different, for one.

> Shouldn't XDG as a specification prevent this
> from being an issue?

It did back when people cared about it, but freedesktop.org has
been inaccesible for a few years now,


        Stefan from the future

Re: master f8fed41 2/3: image-dired: Improve XDG compliance

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: Stefan Kangas <stefan@marxist.se>,  emacs-devel@gnu.org
>> Date: Mon, 25 Oct 2021 16:14:33 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > XDG is also prone to fashion changes.  What will we do when it goes
>> > out of fashion, and we have dozens of xdg-user-dir calls in our
>> > application code? change all of them to some latest desktop fashion?
>> 
>> What do you mean by this? Shouldn't XDG as a specification prevent this
>> from being an issue?
>
> Sorry, I don't understand the second question.  What can "XDG as a
> specification" prevent?  

I misunderstood what you meant by fashion, assuming you meant how XDG
was implemented was a matter of fashion.

> It cannot prevent XDG ceasing to be the fashion, that's entirely out
> of control of any specification.  Fashions some and go, and when they
> come, users rightfully expect us to support the latest fashions, and
> we generally comply.

If anything, I think XDG is becoming more and more popular since 2003,
and I wouldn't know of any other serious contender to replace XDG.

Generally I'd agree that it would be better to provide better support
for the problem XDG solves, without having to use XDG directly for
cross-platform compliance, but XDG is certainly preferable to
hard-coding directories as is still done in Emacs.

-- 
	Philip Kaludercic

Re: master f8fed41 2/3: image-dired: Improve XDG compliance

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Philip Kaludercic [2021-10-25 16:14:33] wrote:
>> Eli Zaretskii <eliz@gnu.org> writes:
>>> XDG is also prone to fashion changes.  What will we do when it goes
>>> out of fashion, and we have dozens of xdg-user-dir calls in our
>>> application code? change all of them to some latest desktop fashion?
>> What do you mean by this?
>
> The YEH spec from the Wayland++ guys is quite different, for one.

I have never heard of this, nor can I find any information about YEH.
Do you have a link or some other pointer?

-- 
	Philip Kaludercic

Re: master f8fed41 2/3: image-dired: Improve XDG compliance

[Gregory Heytings]

>>>> XDG is also prone to fashion changes.  What will we do when it goes 
>>>> out of fashion, and we have dozens of xdg-user-dir calls in our 
>>>> application code? change all of them to some latest desktop fashion?
>>>
>>> What do you mean by this?
>>
>> The YEH spec from the Wayland++ guys is quite different, for one.
>
> I have never heard of this, nor can I find any information about YEH. Do 
> you have a link or some other pointer?
>

Read the signature: "Stefan from the future" ;-)

Re: master f8fed41 2/3: image-dired: Improve XDG compliance

[Stefan Monnier]

Philip Kaludercic [2021-10-25 17:55:33] wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> Philip Kaludercic [2021-10-25 16:14:33] wrote:
>>> Eli Zaretskii <eliz@gnu.org> writes:
>>>> XDG is also prone to fashion changes.  What will we do when it goes
>>>> out of fashion, and we have dozens of xdg-user-dir calls in our
>>>> application code? change all of them to some latest desktop fashion?
>>> What do you mean by this?
>> The YEH spec from the Wayland++ guys is quite different, for one.
> I have never heard of this, nor can I find any information about YEH.

My Gnus from the future inferred those links from me, as usual.
Maybe you're stuck with an old MUA from 2021 or so?


        Stefan

Re: master f8fed41 2/3: image-dired: Improve XDG compliance

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: stefan@marxist.se,  emacs-devel@gnu.org
> Date: Mon, 25 Oct 2021 17:52:56 +0000
> 
> If anything, I think XDG is becoming more and more popular since 2003,
> and I wouldn't know of any other serious contender to replace XDG.

Yes, XDG is the current fashion.  As were the ones before it.

> Generally I'd agree that it would be better to provide better support
> for the problem XDG solves, without having to use XDG directly for
> cross-platform compliance, but XDG is certainly preferable to
> hard-coding directories as is still done in Emacs.

I wasn't suggesting we hard-code directories, I suggested something
quite different.

Re: master f8fed41 2/3: image-dired: Improve XDG compliance

[Po Lu]

Philip Kaludercic <philipk@posteo.net> writes:

> If anything, I think XDG is becoming more and more popular since 2003,
> and I wouldn't know of any other serious contender to replace XDG.

That is what we used to think about the "standards" of CDE (DT) and
Motif.  So these standards are definitely subject to changes of fashion.

For instance, many Freedesktop people already consider X-Windows legacy
software, and are in the process of replacing it with Wayland.

Re: master f8fed41 2/3: image-dired: Improve XDG compliance

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

> > If anything, I think XDG is becoming more and more popular since 2003,
> > and I wouldn't know of any other serious contender to replace XDG.
>
> Yes, XDG is the current fashion.  As were the ones before it.

Which ones would that be?

Default for image-dired-main-image-directory

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

>   (defcustom image-dired-main-image-directory
>     (or (xdg-user-dir "PICTURES") "~/pics/")
>
> Why "~/pics/"?  What platform behaves like that?  Probably none.

(I will reply to your other points separately, this is a bit of a tangent.)

That's the old default, so if you don't use XDG this code will behave
exactly as before.  So this is a step forward on GNU/Linux, and
everyone else is unaffected (unless they have specifically opted in).

We could use something more reasonable than "~/pics" of course, which
just seems to be what someone happened to be using at some point.

I suggest "~/Pictures", which is at least used by XDG.

Re: master f8fed41 2/3: image-dired: Improve XDG compliance

[Eli Zaretskii]

> From: Stefan Kangas <stefan@marxist.se>
> Date: Tue, 26 Oct 2021 12:24:11 +0200
> Cc: Philip Kaludercic <philipk@posteo.net>, Emacs developers <emacs-devel@gnu.org>
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > > If anything, I think XDG is becoming more and more popular since 2003,
> > > and I wouldn't know of any other serious contender to replace XDG.
> >
> > Yes, XDG is the current fashion.  As were the ones before it.
> 
> Which ones would that be?

Is it important?

Re: Default for image-dired-main-image-directory

[Eli Zaretskii]

> From: Stefan Kangas <stefan@marxist.se>
> Date: Tue, 26 Oct 2021 12:25:51 +0200
> Cc: Emacs developers <emacs-devel@gnu.org>
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >   (defcustom image-dired-main-image-directory
> >     (or (xdg-user-dir "PICTURES") "~/pics/")
> >
> > Why "~/pics/"?  What platform behaves like that?  Probably none.
> 
> (I will reply to your other points separately, this is a bit of a tangent.)
> 
> That's the old default, so if you don't use XDG this code will behave
> exactly as before.  So this is a step forward on GNU/Linux, and
> everyone else is unaffected (unless they have specifically opted in).

It is a step forward for GNU/Linux, maybe, but it is a step back for
Emacs, for the reasons I explained.

> We could use something more reasonable than "~/pics" of course, which
> just seems to be what someone happened to be using at some point.
> 
> I suggest "~/Pictures", which is at least used by XDG.

How about if we DTRT right away, and provide reasonable defaults for
all the non-XDG platforms (assuming we prefer platform-specific
defaults in this particular case)?  I see no real reason to install
half-solutions.

Re: master f8fed41 2/3: image-dired: Improve XDG compliance

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

> > > Yes, XDG is the current fashion.  As were the ones before it.
> >
> > Which ones would that be?
>
> Is it important?

It is not very important to me, but it seems crucial for the argument
that the XDG Base Directory Specification is merely "the current
fashion".

AFAIK, there was no standard before the XDG BDS which it could
replace, there exists no competing standard, and so far there has been
no standard proposed to replace it.  Is that wrong?

Re: Default for image-dired-main-image-directory

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

> How about if we DTRT right away, and provide reasonable defaults for
> all the non-XDG platforms (assuming we prefer platform-specific
> defaults in this particular case)?  I see no real reason to install
> half-solutions.

FYI, there is no-littering package [1] trying to provide unified
directory structure within .emacs.d.  It follows the same idea of
unified set of abstract directory variables for different types of data:
- etc-directory
- var-directory

Essentially, it is similar to XDG directory set, but all the directories
are inside .emacs.d by default.

I think that putting things in a reasonably-names directories inside
.emacs.d can be a meaningful default, unless there is an OS-specific
standard (XDG on Linux, de-facto standard on Windows or Mac).

The question is what kind of directories are actually useful within
Emacs.

[1] https://github.com/emacscollective/no-littering

Best,
Ihor

Re: master f8fed41 2/3: image-dired: Improve XDG compliance

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

> XDG is also prone to fashion changes.  What will we do when it goes
> out of fashion, and we have dozens of xdg-user-dir calls in our
> application code? change all of them to some latest desktop fashion?

Yes, that's what we should do in that case.  Such work will be much
easier if we have already adapted our code to use the xdg
specification: if you move xdg.el to "lisp/obsolete" the byte-compiler
will warn about any places where we use it.  But we have no easy way
to find strings like "~/pics", and therefore can't easily update them.

In other words, if we expect the standard to change very soon, it is
already a step forward to update hardcoded strings like "~/pics" like
I did in image-dired.el.

However, I don't really see any reason to expect a replacement to the
XDG Base Directory Specification (XDG BDS) any time soon.  As Philip
points out, it has only become more ubiquitous over the almost 20
years since its inception, and no replacement is on the horizon.
Therefore, this all seems very academic to me, and I think the points
below about other platforms are more immediately relevant.

> Can we agree not to use xdg.el in application code, and instead
> develop those abstractions and start using them?  For example, how
> about having a function user-pics-directory, which will return the
> appropriate file name for the underlying platform?  Doing that will
> not diminish our support for XDG (as long as it lasts), but will allow
> us to have application code XDG-clean, so to speak.

I have no objections to the general plan you propose, and I encourage
anyone with a particular interest in such support to work on it.  (It
will be mainly of interest to macOS or MS-Windows users, I think?  But
they should probably already be using a lot of software that uses the
XDG BDS.  I note that I have several BDS mandated directories on my
macOS laptop.)

However, I do not see any logic in stopping work on XDG BDS support
until such a library or functions has been implemented.  Just like
above, any work we do in conforming better to the XDG BDS can only
serve to make the "platform independent" or "XDG BDS independent" plan
you propose easier to implement.

Re: master f8fed41 2/3: image-dired: Improve XDG compliance

[Eli Zaretskii]

> From: Stefan Kangas <stefan@marxist.se>
> Date: Tue, 26 Oct 2021 15:28:27 +0200
> Cc: "Philip K." <philipk@posteo.net>, Emacs developers <emacs-devel@gnu.org>
> 
> AFAIK, there was no standard before the XDG BDS which it could
> replace, there exists no competing standard, and so far there has been
> no standard proposed to replace it.  Is that wrong?

The original standard was to use files that begin with a dot in the
user's home directory, with system-wide defaults in /etc or /usr/etc.

Then came the standard (copy-cat from Windows) that each type of files
had its own directory under the home directory -- so we have
~/Pictures, ~/Documents, ~/Desktop, ~/Mail, etc.

And now we have XDG.

I'm sure they will invent something else in a few years, which will of
course be the best thing since sliced bread.

Re: Default for image-dired-main-image-directory

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

> How about if we DTRT right away, and provide reasonable defaults for
> all the non-XDG platforms

Which are the "non-XDG platforms" more exactly?

Why is "~/Pictures" not a reasonable default on those platforms?

Re: Default for image-dired-main-image-directory

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: Stefan Kangas <stefan@marxist.se>,  emacs-devel@gnu.org
> Date: Tue, 26 Oct 2021 21:42:22 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > How about if we DTRT right away, and provide reasonable defaults for
> > all the non-XDG platforms (assuming we prefer platform-specific
> > defaults in this particular case)?  I see no real reason to install
> > half-solutions.
> 
> FYI, there is no-littering package [1] trying to provide unified
> directory structure within .emacs.d.  It follows the same idea of
> unified set of abstract directory variables for different types of data:
> - etc-directory
> - var-directory
> 
> Essentially, it is similar to XDG directory set, but all the directories
> are inside .emacs.d by default.

That's a different philosophy, and since it contradicts XDG, which
many people want, I don't think we can go that way, except as an
opt-in feature (which will then become a maintenance burden, as it
will be yet another, third, set of defaults).

> I think that putting things in a reasonably-names directories inside
> .emacs.d can be a meaningful default, unless there is an OS-specific
> standard (XDG on Linux, de-facto standard on Windows or Mac).

Yes, every modern platform has standardized places for this stuff.  If
we decided to support XDG on GNU/Linux, it follows that we prefer
platform-specific behavior, not identical standard directories on all
platforms.

So we need something like system-config.el, which would use xdg.el for
XDG platforms, and something else for the rest.

Re: master f8fed41 2/3: image-dired: Improve XDG compliance

[Eli Zaretskii]

> From: Stefan Kangas <stefan@marxist.se>
> Date: Tue, 26 Oct 2021 16:01:50 +0200
> Cc: Emacs developers <emacs-devel@gnu.org>
> 
> However, I do not see any logic in stopping work on XDG BDS support
> until such a library or functions has been implemented.  Just like
> above, any work we do in conforming better to the XDG BDS can only
> serve to make the "platform independent" or "XDG BDS independent" plan
> you propose easier to implement.

I'm not against extending xdg.el, I'm against seeing more and more
xdg-FOO stuff in the application code.  That's not why we agreed to
include it in Emacs.

Re: Default for image-dired-main-image-directory

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

> That's a different philosophy, and since it contradicts XDG, which
> many people want, I don't think we can go that way, except as an
> opt-in feature (which will then become a maintenance burden, as it
> will be yet another, third, set of defaults).

Exactly.  I think "no-litter" solves the problem of "how do I keep all
my stuff in .emacs.d without making too much of a mess of it".  It
doesn't do a bad job at solving that problem, but it is IMO exactly
the wrong problem to solve.

> So we need something like system-config.el, which would use xdg.el for
> XDG platforms, and something else for the rest.

Sounds good to me.

Re: master f8fed41 2/3: image-dired: Improve XDG compliance

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

> I'm against seeing more and more
> xdg-FOO stuff in the application code.  That's not why we agreed to
> include it in Emacs.

I don't see why: as I explained, this will make implementing your idea
easier, not harder.  Right?

Re: master f8fed41 2/3: image-dired: Improve XDG compliance

[Eli Zaretskii]

> From: Stefan Kangas <stefan@marxist.se>
> Date: Tue, 26 Oct 2021 16:01:50 +0200
> Cc: Emacs developers <emacs-devel@gnu.org>
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > XDG is also prone to fashion changes.  What will we do when it goes
> > out of fashion, and we have dozens of xdg-user-dir calls in our
> > application code? change all of them to some latest desktop fashion?
> 
> Yes, that's what we should do in that case.

I'm quite astonished to read this, coming from an active Emacs
developer.  It's basically an antithesis of good maintainability.

> Such work will be much
> easier if we have already adapted our code to use the xdg
> specification: if you move xdg.el to "lisp/obsolete" the byte-compiler
> will warn about any places where we use it.  But we have no easy way
> to find strings like "~/pics", and therefore can't easily update them.

I didn't propose to have hard-coded strings like "~/pics", not at
all.  I proposed to have a couple of higher-level APIs that will hide
system-dependent stuff, including hard-coded strings if some systems
need that.  xdg.el will then be one layer lower, and application code
will not call it directly.

> In other words, if we expect the standard to change very soon, it is
> already a step forward to update hardcoded strings like "~/pics" like
> I did in image-dired.el.

These "semi-standards" change quite frequently, and we should be
prepared for the next change in advance, even if we don't know when
will that happen.

> However, I don't really see any reason to expect a replacement to the
> XDG Base Directory Specification (XDG BDS) any time soon.  As Philip
> points out, it has only become more ubiquitous over the almost 20
> years since its inception, and no replacement is on the horizon.
> Therefore, this all seems very academic to me, and I think the points
> below about other platforms are more immediately relevant.

Good maintainability is not an academic subject, it is pretty much in
our face here.

Re: master f8fed41 2/3: image-dired: Improve XDG compliance

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

> The original standard was to use files that begin with a dot in the
> user's home directory, with system-wide defaults in /etc or /usr/etc.

I'd say that ad-hoc mess is just what happens organically if you have
no standard.  But sure, that was all we got from Unix, so I guess that
I will have to agree that we have seen two standards since 1969.

> I'm sure they will invent something else in a few years, which will of
> course be the best thing since sliced bread.

Well, sure.  OTOH, it took 15 years for us to warm up to the XDG BDS
enough to introduce xdg.el - and it is still at least somewhat
controversial - so I expect that we will not see any rush to implement
the new standard either.  ;-)

Re: master f8fed41 2/3: image-dired: Improve XDG compliance

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

> I proposed to have a couple of higher-level APIs that will hide
> system-dependent stuff, including hard-coded strings if some systems
> need that.  xdg.el will then be one layer lower, and application code
> will not call it directly.

OK, I think we agree that this would be a good thing.

The contentious point seems to be what we should do until we have such
a library.  I think that it is better to update hardcoded strings to
use xdg.el, since it will make the above idea easier to implement.

However, if we have a plan for a higher-level API in a reasonable
time-frame (for example, before Emacs 29 is released), then I am more
than happy to accept that we might as well wait with any such changes
until that work is done.

Re: Default for image-dired-main-image-directory

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> Essentially, it is similar to XDG directory set, but all the directories
>> are inside .emacs.d by default.
>
> That's a different philosophy, and since it contradicts XDG, which
> many people want, I don't think we can go that way, except as an
> opt-in feature (which will then become a maintenance burden, as it
> will be yet another, third, set of defaults).

Not necessarily.  no-littering provides a unified directory structure to
be used by other packages. Yes, it defaults to structure inside .emacs.d
(and thus do not need to bother with OS-specific conventions), but
that's not required:

(defvar no-littering-etc-directory
  (expand-file-name (convert-standard-filename "etc/") user-emacs-directory)
  "The directory where packages place their configuration files.
This variable has to be set before `no-littering' is loaded.")

From point of view of a package author, I usually have difficulty
deciding where to put files. Package install dir? Directly into
.emacs.d? User home dir? I would prefer to have a convention about
default dirs. I would prefer Emacs to handle variations of directory
structure in different OS.

no-littering proposes a set of Emacs-specific conventions about where to
put package files:

>> (A) File names
>> 
>> File names are based on the name of the respective Emacs lisp variables
>> and the name of the respective Emacs package.
>> 
>> The name of the respective Emacs package should serve as the prefix of
>> the file name, unless the file is in a subdirectory in which case the
>> name of the subdirectory serves as the prefix.
>> 
>> If the name of the package and the prefix of the variable do not match,
>> then we prefer the name of the package.
>> 
>> If the name of a path variable ends with -file, -default-file,
>> -directory, -default-directory, or something similar, then that suffix
>> is usually dropped from the file name.
>> 
>> If applicable, the appropriate extension is added to the file name so
>> that files are visited using the appropriate major-modes and also to
>> provide a hint about the kind of data stored in the file. E.g. if a file
>> contains an S-expression, then the suffix should be *.el.
>> 
>> B) File location and subdirectories If a package has only one data file,
>> then that is usually placed in no-littering-var-directory itself.
>> Likewise if a package has only one config file, then that is placed in
>> no-littering-etc-directory itself.
>> 
>> If a package has multiple data (or config files), then those files are
>> placed in a subdirectory of no-littering-var-directory (or
>> no-littering-etc-directory).
>> 
>> If a subdirectory is used for a package's data (or config) file
>> variables, then the name of the directory should match the name of the
>> package in most cases. The subdirectory name may serve as the package
>> prefix of the file name.
>> 
>> If a package provides a "framework" for other packages to use, then we
>> may reuse its directories for other packages that make use of that
>> framework or otherwise "extend" the "main package". E.g. we place all
>> helm related files in helm/.
>> 
>> If a package only defines a single variable that specifies a data (or
>> config) directory, then the directory name should nevertheless be just
>> the package name. E.g. the path used for sx-cache-directory from the sx
>> package is sx/cache/, not sx-cache/.
>> 
>> However if the name of the directory variable implies that the package
>> won't ever define any data (or config) files that won't be placed in
>> that directory, then we use a top-level directory. E.g. when the name of
>> the variable is <package>-directory, in which case we would use just
>> <package>/ as the path.

Best,
Ihor

Re: Default for image-dired-main-image-directory

[Eli Zaretskii]

> From: Stefan Kangas <stefan@marxist.se>
> Date: Tue, 26 Oct 2021 16:14:42 +0200
> Cc: Emacs developers <emacs-devel@gnu.org>
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > How about if we DTRT right away, and provide reasonable defaults for
> > all the non-XDG platforms
> 
> Which are the "non-XDG platforms" more exactly?
> 
> Why is "~/Pictures" not a reasonable default on those platforms?

Because it goes against the platform conventions.  Or let me put it
this way: for the same reason we don't want to use ~/Pictures on
systems that use the XDG conventions.

Re: master f8fed41 2/3: image-dired: Improve XDG compliance

[Eli Zaretskii]

> From: Stefan Kangas <stefan@marxist.se>
> Date: Tue, 26 Oct 2021 16:24:30 +0200
> Cc: Emacs developers <emacs-devel@gnu.org>
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I'm against seeing more and more
> > xdg-FOO stuff in the application code.  That's not why we agreed to
> > include it in Emacs.
> 
> I don't see why: as I explained, this will make implementing your idea
> easier, not harder.  Right?

No.  What will happen is we will see the proliferation of the likes of
this:

  (cond
    ((eq system-type 'windows-nt
       (or (w32-read-registry
            'HKCU
	    "SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Explorer\\User Shell Folders" "Personal")
	    "~/../My Documents"
	    "~/../../Documents"))
    ((eq system-type 'darwin "~/Documents"))
    (t (xdg-documents-directory)))

and similarly for other standard directories, in our application code.
I really hope this is not what we want to see there.  My idea was to
hide all this platform-specific ugliness under a standard API, such as
(user-std-directory 'DOCUMENTS) or maybe a set of APIs like
(user-documents-directory), (user-music-directory), etc.  I thought it
will be a no-brainer that using the latter will be both cleaner, more
readable, and easier in maintenance; am I the only one who thinks so?

Re: master f8fed41 2/3: image-dired: Improve XDG compliance

[Eli Zaretskii]

> From: Stefan Kangas <stefan@marxist.se>
> Date: Tue, 26 Oct 2021 16:45:04 +0200
> Cc: Emacs developers <emacs-devel@gnu.org>
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I proposed to have a couple of higher-level APIs that will hide
> > system-dependent stuff, including hard-coded strings if some systems
> > need that.  xdg.el will then be one layer lower, and application code
> > will not call it directly.
> 
> OK, I think we agree that this would be a good thing.
> 
> The contentious point seems to be what we should do until we have such
> a library.  I think that it is better to update hardcoded strings to
> use xdg.el, since it will make the above idea easier to implement.
> 
> However, if we have a plan for a higher-level API in a reasonable
> time-frame (for example, before Emacs 29 is released), then I am more
> than happy to accept that we might as well wait with any such changes
> until that work is done.

It is so easy to provide this API (or set of APIs) piecemeal that I
don't see why we would need to wait for anything.  Let's simply start
a file called, say, std-dirs.el or somesuch, and add support there for
each directory we bump into.  For example, in the specific case that
triggered this, just add a function there that calls

  (or (xdg-user-dir "PICTURES") "~/pics/")

and call that function in image-dired.el.  Before long, we will have
everything we need, and more.  If you don't want to provide support
for anything except XDG, you can initially leave the fixed strings
there as the alternatives, and someone else will pick up where you
left off and fill in the blanks.

The only decision we need to make is whether we want a single function
that accepts an argument telling it the purpose of the standard
directory (in this case, I'd use 'PICTURES, a symbol), or maybe a
separate function for each kind of directory, like user-pictures-dir.

Re: master f8fed41 2/3: image-dired: Improve XDG compliance

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

> No.  What will happen is we will see the proliferation of the likes of
> this:
>
>   (cond
>     ((eq system-type 'windows-nt
>        (or (w32-read-registry
>             'HKCU
>             "SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Explorer\\User Shell Folders" "Personal")
>             "~/../My Documents"
>             "~/../../Documents"))
>     ((eq system-type 'darwin "~/Documents"))
>     (t (xdg-documents-directory)))
>
> and similarly for other standard directories, in our application code.

That looks like a nightmare, yes.

> I really hope this is not what we want to see there.  My idea was to
> hide all this platform-specific ugliness under a standard API, such as
> (user-std-directory 'DOCUMENTS) or maybe a set of APIs like
> (user-documents-directory), (user-music-directory), etc.

I'm not seeing any other volunteers, so let me have a crack at it.

Re: Default for image-dired-main-image-directory

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: stefan@marxist.se,  emacs-devel@gnu.org
> Date: Tue, 26 Oct 2021 22:58:09 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Essentially, it is similar to XDG directory set, but all the directories
> >> are inside .emacs.d by default.
> >
> > That's a different philosophy, and since it contradicts XDG, which
> > many people want, I don't think we can go that way, except as an
> > opt-in feature (which will then become a maintenance burden, as it
> > will be yet another, third, set of defaults).
> 
> Not necessarily.  no-littering provides a unified directory structure to
> be used by other packages. Yes, it defaults to structure inside .emacs.d
> (and thus do not need to bother with OS-specific conventions), but
> that's not required:

You say "not necessarily", but then proceed to describe details that
actually agree with what I said.

It is fine to provide such a feature, I'm just saying that XDG
requires different directories, and so these two arrangements
contradict each other.

Re: master f8fed41 2/3: image-dired: Improve XDG compliance

[Eli Zaretskii]

> From: Stefan Kangas <stefan@marxist.se>
> Date: Tue, 26 Oct 2021 18:13:41 +0200
> Cc: Emacs developers <emacs-devel@gnu.org>
> 
> > I really hope this is not what we want to see there.  My idea was to
> > hide all this platform-specific ugliness under a standard API, such as
> > (user-std-directory 'DOCUMENTS) or maybe a set of APIs like
> > (user-documents-directory), (user-music-directory), etc.
> 
> I'm not seeing any other volunteers, so let me have a crack at it.

Thanks, and let me know if I can help.

Re: master f8fed41 2/3: image-dired: Improve XDG compliance

[Eli Zaretskii]

> Date: Tue, 26 Oct 2021 19:12:31 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> Let's simply start a file called, say, std-dirs.el or somesuch

Or maybe user-dir.el.  (We can bikeshed the name later.)

Re: master f8fed41 2/3: image-dired: Improve XDG compliance

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: stefan@marxist.se,  emacs-devel@gnu.org
>> Date: Mon, 25 Oct 2021 17:52:56 +0000
>> 
>> If anything, I think XDG is becoming more and more popular since 2003,
>> and I wouldn't know of any other serious contender to replace XDG.
>
> Yes, XDG is the current fashion.  As were the ones before it.

What time frames are we talking about? Fashion can be anything that
remains popular for only a few months up until something that dominates
the scene for years or decades. XDG appears to lean towards the latter.

>> Generally I'd agree that it would be better to provide better support
>> for the problem XDG solves, without having to use XDG directly for
>> cross-platform compliance, but XDG is certainly preferable to
>> hard-coding directories as is still done in Emacs.
>
> I wasn't suggesting we hard-code directories, I suggested something
> quite different.

I understood that, my point was that using XDG is preferable to that,
and anything that abstracts over XDG might be even better.

-- 
	Philip Kaludercic

Re: master f8fed41 2/3: image-dired: Improve XDG compliance

[Stefan Monnier]

> The only decision we need to make is whether we want a single function
> that accepts an argument telling it the purpose of the standard
> directory (in this case, I'd use 'PICTURES, a symbol), or maybe a
> separate function for each kind of directory, like user-pictures-dir.

I'd vote for a single defgeneric function.


        Stefan

Re: master f8fed41 2/3: image-dired: Improve XDG compliance

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Stefan Kangas <stefan@marxist.se>,  emacs-devel@gnu.org
> Date: Tue, 26 Oct 2021 15:17:52 -0400
> 
> > The only decision we need to make is whether we want a single function
> > that accepts an argument telling it the purpose of the standard
> > directory (in this case, I'd use 'PICTURES, a symbol), or maybe a
> > separate function for each kind of directory, like user-pictures-dir.
> 
> I'd vote for a single defgeneric function.

What would be generic about it?

Re: master f8fed41 2/3: image-dired: Improve XDG compliance

[Po Lu]

Stefan Kangas <stefan@marxist.se> writes:

> I'd say that ad-hoc mess is just what happens organically if you have
> no standard.  But sure, that was all we got from Unix, so I guess that
> I will have to agree that we have seen two standards since 1969.

There was the CDE, and all of its associated conventions, that
Freedesktop was at least partially created to compete with.

And aside from that, I'm told Sun's Open Windows also had its own set of
quirks and conventions.

Re: Default for image-dired-main-image-directory

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

> You say "not necessarily", but then proceed to describe details that
> actually agree with what I said.
>
> It is fine to provide such a feature, I'm just saying that XDG
> requires different directories, and so these two arrangements
> contradict each other.

Probably I misunderstood what XDG stadard describes. I have read the XDG
description at freedesktop.org [1]. My understanding is that XDG does
not provide any recommendations on the file naming and subdirectories
inside home, config, state, bin, data, runtime, and cache dirs. XDG only
prescribes using the listed dirs to store different types of data.

From a Emacs package perspective, just following XDG is still not clear
enough.  Should all Emacs packages store data inside "XDG_DIR/emacs"
subdirectory or maybe "XDG_DIR/emacs/package-name"? What should be the
file names? The details I quoted are providing a possible clarification
about these kinds of questions, except that no-littering assumes
XDG_DIR=.emacs.d.

[1] https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html

Best,
Ihor

Re: Default for image-dired-main-image-directory

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: stefan@marxist.se,  emacs-devel@gnu.org
> Date: Wed, 27 Oct 2021 21:04:40 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > You say "not necessarily", but then proceed to describe details that
> > actually agree with what I said.
> >
> > It is fine to provide such a feature, I'm just saying that XDG
> > requires different directories, and so these two arrangements
> > contradict each other.
> 
> Probably I misunderstood what XDG stadard describes. I have read the XDG
> description at freedesktop.org [1]. My understanding is that XDG does
> not provide any recommendations on the file naming and subdirectories
> inside home, config, state, bin, data, runtime, and cache dirs. XDG only
> prescribes using the listed dirs to store different types of data.

Sure, but none of those is called .emacs.d or anything close.

> >From a Emacs package perspective, just following XDG is still not clear
> enough.  Should all Emacs packages store data inside "XDG_DIR/emacs"
> subdirectory or maybe "XDG_DIR/emacs/package-name"? What should be the
> file names? The details I quoted are providing a possible clarification
> about these kinds of questions, except that no-littering assumes
> XDG_DIR=.emacs.d.

Look at xdg.el and at startup.el where we already use XDG-compliant
directories.  What you say above is neither compliant with XDG nor
with any current Emacs convention; it's an entirely new arrangement.
I think that's one too many.

Re: Default for image-dired-main-image-directory

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> >From a Emacs package perspective, just following XDG is still not clear
>> enough.  Should all Emacs packages store data inside "XDG_DIR/emacs"
>> subdirectory or maybe "XDG_DIR/emacs/package-name"? What should be the
>> file names? The details I quoted are providing a possible clarification
>> about these kinds of questions, except that no-littering assumes
>> XDG_DIR=.emacs.d.
>
> Look at xdg.el and at startup.el where we already use XDG-compliant
> directories.  What you say above is neither compliant with XDG nor
> with any current Emacs convention; it's an entirely new arrangement.
> I think that's one too many.

What I propose is not an alternative to XDG. It is complient with XDG in
a sense defined in "Referencing this specification" section of the XDG
standard. We just put things under appropriate XDG_DIR instead of
.emacs.d.

It is indeed a new arrangement for Emacs. I feel that having a
convention about sub-directories and file names would be useful. The
very existence of no-littering package indicates that some other people
also prefer having a convention. If you think that it is not necessary,
it is fine. Other people can jump in and support/oppose my suggestion.

Best,
Ihor

XDG config dir [Was: Re: Default for image-dired-main-image-directory]

[Alexandre Garreau]

Le mercredi 27 octobre 2021, 15:42:11 CEST Ihor Radchenko a écrit :
> Eli Zaretskii <eliz@gnu.org> writes:
> >> >From a Emacs package perspective, just following XDG is still not
> >> >clear
> >> 
> >> enough.  Should all Emacs packages store data inside "XDG_DIR/emacs"
> >> subdirectory or maybe "XDG_DIR/emacs/package-name"? What should be
> >> the
> >> file names? The details I quoted are providing a possible
> >> clarification
> >> about these kinds of questions, except that no-littering assumes
> >> XDG_DIR=.emacs.d.
> > 
> > Look at xdg.el and at startup.el where we already use XDG-compliant
> > directories.  What you say above is neither compliant with XDG nor
> > with any current Emacs convention; it's an entirely new arrangement.
> > I think that's one too many.
> 
> What I propose is not an alternative to XDG. It is complient with XDG in
> a sense defined in "Referencing this specification" section of the XDG
> standard. We just put things under appropriate XDG_DIR instead of
> .emacs.d.

Oh btw that makes me think to something I always wondered: would there be 
a way to make so that to have .config/emacs instead of .emacs.d? at least 
when .config/xdg-something exists, for instance

That would reduce the clutter inside home directory, that’s partially 
solved by that xdg .config directory (although I find regrettable it’s not 
.local/etc, but a such symlink can solve that).

Btw there’s a whole bunch of directories autoconf computes at configuration 
time… where are these accessible from emacs lisp? shouldn’t emacs use the 
same naming convention for all of these? if they’re not accessible yet, 
maybe exposing them would be a good way to start…

Something like (std-dir 'user 'music) or (std-dir 'system 'icons) would be 
more complex than music-user-dir (and less consistent with already 
existing variables such as emacs-user-dir), but be useful in case a dir 
needs to be computed programmatically, for instance to do (std-dir 'user 
(case file-ext (("jpg" "png") 'picture) ("mkv" 'video) ("ogg" 'music))) 
instead of rewriting “-user-dir” each time (and imho allows greater 
flexibility since you could reuse the symbol to trigger something else 
before passing it to std-dir).

OTOH a such system couldn’t be searched with variable autocompletion, but 
on the other hand, given autocompletion/searching, the most useful naming 
scheme wouldn’t be a normal english name but something reversed like dir-
std-user-music or dir-std-system-icons, while a doc for std-dir could 
specify all of that comprehensively in just-one-place™ (but it would be a 
less standard way to search for a variable).

…although emacs supports autocompletion starting by the end or middle of a 
variable, so I don’t know, but I like the more complex system.

Re: XDG config dir [Was: Re: Default for image-dired-main-image-directory]

[Ihor Radchenko]

Alexandre Garreau <galex-713@galex-713.eu> writes:

> Oh btw that makes me think to something I always wondered: would there be 
> a way to make so that to have .config/emacs instead of .emacs.d? at least 
> when .config/xdg-something exists, for instance

AFAIK, emacs does support ~/.config/emacs. As long, as you do not have
~/.emacs and ~/.emacs.d and do have ~/.config/emacs directory, Emacs is
using the ~/.config/emacs (or $XDG_CONFIG_HOME/emacs).

> That would reduce the clutter inside home directory, that’s partially 
> solved by that xdg .config directory (although I find regrettable it’s not 
> .local/etc, but a such symlink can solve that).

According to XDG spec, you just need to set $XDG_CONFIG_HOME environment
variable.  ~/.config is only used when the variable is not set:

From
https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html:

>> $XDG_CONFIG_HOME defines the base directory relative to which user-specific configuration files should be stored. If $XDG_CONFIG_HOME is either not set or empty, a default equal to $HOME/.config should be used.

Best,
Ihor

Re: XDG config dir [Was: Re: Default for image-dired-main-image-directory]

[Phil Sainty]

On 2021-10-28 05:53, Alexandre Garreau wrote:
> Oh btw that makes me think to something I always wondered: would there 
> be
> a way to make so that to have .config/emacs instead of .emacs.d? at 
> least
> when .config/xdg-something exists, for instance

Like this?

https://git.savannah.gnu.org/cgit/emacs.git/tree/etc/NEWS?h=emacs-27&id=348b2aed0c8c3630be4c15c8e70acf7ea9c023af#n224

Re: XDG config dir [Was: Re: Default for image-dired-main-image-directory]

[Alexandre Garreau]

Le jeudi 28 octobre 2021, 05:48:56 CEST Phil Sainty a écrit :
> On 2021-10-28 05:53, Alexandre Garreau wrote:
> > Oh btw that makes me think to something I always wondered: would there
> > be
> > a way to make so that to have .config/emacs instead of .emacs.d? at
> > least
> > when .config/xdg-something exists, for instance
> 
> Like this?
> 
> https://git.savannah.gnu.org/cgit/emacs.git/tree/etc/NEWS?h=emacs-27&id=
> 348b2aed0c8c3630be4c15c8e70acf7ea9c023af#n224

oh yes! I just appear to be using emacs 26 (debian stable), so I won’t be 
able to benefit from that immediately (or until my next recompile of emacs) 
^^

I find sad that it’s still .emacs.d (or, well, .emacs isn’t it?) that will 
be created by default, but well, I imagine that’s configurable per-site in 
/etc

Re: XDG config dir [Was: Re: Default for image-dired-main-image-directory]

[Stefan Kangas]

Alexandre Garreau <galex-713@galex-713.eu> writes:

> oh yes! I just appear to be using emacs 26 (debian stable), so I won’t be
> able to benefit from that immediately (or until my next recompile of emacs)

Debian stable/bullseye has Emacs 27.1:

    https://packages.debian.org/stable/emacs

user-directory: New library to find user {conf,data,state,...} files

[Stefan Kangas]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> The only decision we need to make is whether we want a single function
>> that accepts an argument telling it the purpose of the standard
>> directory (in this case, I'd use 'PICTURES, a symbol), or maybe a
>> separate function for each kind of directory, like user-pictures-dir.
>
> I'd vote for a single defgeneric function.

I hacked something quick together and pushed it to
features/user-directory, mostly so that we can decide on the basic API.
It needs more work, but the interface should hopefully be clear.

The one I propose is:

    (user-directory 'cache)
    => "/home/skangas/.cache"

    (user-file 'cache "foo")
    => "/home/skangas/.cache/foo"

There's also a defcustom `user-directory-alist' for overrides:

    (let ((user-directory-alist '((downloads "/foo"))))
      (user-directory 'downloads))
    => "/foo"

I thought about having defuns like `user-directory-cache', which gives
us autocompletion, but I feel like the symbol approach is more
straightforward and means we can avoid code like this:

    (intern (concat "user-directory-" (symbol-name sym)))

This is implemented with a defgeneric, as suggested by Stefan Monnier
(thanks!), but we could easily convert it to regular defuns if we need
to preload it.

Re: user-directory: New library to find user {conf,data,state,...} files

[Lars Ingebrigtsen]

Stefan Kangas <stefan@marxist.se> writes:

> The one I propose is:
>
>     (user-directory 'cache)
>     => "/home/skangas/.cache"
>
>     (user-file 'cache "foo")
>     => "/home/skangas/.cache/foo"

Looks nice to me.  I haven't looked at the branch, but there's one
problem that many of the things that try to store data have: They used
to store the data in ~/some-strange-location, but now want to follow XDG
or whatever.

So it would be nice if we could just say

(user-file 'cache "foo" :legacy "~/Groovy/foo")

and it'd return the legacy file name if it exists, but otherwise use the
new one.  Or something like that -- the :legacy should probably also be
allowed to be a list of names for things that have moved around a lot.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: user-directory: New library to find user {conf, data, state, ...} files

[Stefan Kangas]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Looks nice to me.

Thanks.

> So it would be nice if we could just say
>
> (user-file 'cache "foo" :legacy "~/Groovy/foo")
>
> and it'd return the legacy file name if it exists, but otherwise use the
> new one.  Or something like that -- the :legacy should probably also be
> allowed to be a list of names for things that have moved around a lot.

I forgot to mention it, but `user-file' is supposed to support an
OLD-NAME argument just like `locate-user-emacs-file' does.  Making it
accept either a string or a list sounds like a good addition.

Re: user-directory: New library to find user {conf, data, state, ...} files

[Stefan Kangas]

Stefan Kangas <stefan@marxist.se> writes:

> This is implemented with a defgeneric, as suggested by Stefan Monnier
> (thanks!), but we could easily convert it to regular defuns if we need
> to preload it.

By the way, I am getting the defmethods in the reverse order in
`C-h f user-directory' than the one in which they were defined.  Is that
expected?  Should I put the most important defmethods at the bottom
of the file if I want them listed at the top?

Re: user-directory: New library to find user {conf, data, state, ...} files

[Stefan Kangas]

Stefan Kangas <stefan@marxist.se> writes:

> I hacked something quick together and pushed it to
> features/user-directory, mostly so that we can decide on the basic API.
> It needs more work, but the interface should hopefully be clear.
>
> The one I propose is:
>
>     (user-directory 'cache)
>     => "/home/skangas/.cache"
>
>     (user-file 'cache "foo")
>     => "/home/skangas/.cache/foo"
>
> There's also a defcustom `user-directory-alist' for overrides:
>
>     (let ((user-directory-alist '((downloads "/foo"))))
>       (user-directory 'downloads))
>     => "/foo"

OK, I think this is ready for merging to master so that we can start
experimenting with it, obviously based on the assumption that this is
tentative and open for change.  I guess someone will want to look at
adding MS-Windows support to this, for example.

Note that on GNU/Linux, even if the user has their configuration file in
"~/.emacs.d/", the library currently prefers the XDG paths here:

    (user-file 'cache "image-dired")
    => "~/.cache/emacs/image-dired"

This is a feature.  If users prefer not to have a clean "~/.emacs.d/"
they can say:

    (setq user-directory-alist '((cache . user-emacs-directory)
                                 (config . user-emacs-directory)
                                 (data . user-emacs-directory)
                                 (runtime . user-emacs-directory)
                                 (state . user-emacs-directory)))

I also think we should consider adding some heuristics to `user-file' to
be able to find the old name if it exists.  I considered some
alternatives, but I thought I'd see what people think first.  Currently,
application code would say:

    (user-file 'cache "image-dired"
                       (locate-user-emacs-file "image-dired"))

Which could, if we want to be smart, be simplified to:

    (user-file 'cache "image-dired" "image-dired")

Or even:

    (user-file 'cache "image-dired")

Unfortunately, in this case I think it's harder to be smart:

    (user-file 'data "bookmarks"
                     (locate-user-emacs-file "bookmarks" ".emacs.bmk"))

But perhaps we can think of something even for that.

Re: user-directory: New library to find user {conf, data, state, ...} files

[Eli Zaretskii]

> From: Stefan Kangas <stefan@marxist.se>
> Date: Sun, 7 Nov 2021 11:31:50 -0800
> Cc: emacs-devel@gnu.org
> 
> Note that on GNU/Linux, even if the user has their configuration file in
> "~/.emacs.d/", the library currently prefers the XDG paths here:
> 
>     (user-file 'cache "image-dired")
>     => "~/.cache/emacs/image-dired"
> 
> This is a feature.  If users prefer not to have a clean "~/.emacs.d/"
> they can say:
> 
>     (setq user-directory-alist '((cache . user-emacs-directory)
>                                  (config . user-emacs-directory)
>                                  (data . user-emacs-directory)
>                                  (runtime . user-emacs-directory)
>                                  (state . user-emacs-directory)))

This contradicts how Emacs by default finds the init file: its
preference is the other way around, and it sets user-emacs-directory
according to the directory from which it loaded the init file.  I
don't think I understand why this package should behave differently by
default (it could do that as an opt-in option, of course, if that is
considered useful to someone).

Re: user-directory: New library to find user {conf, data, state, ...} files

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

>> Note that on GNU/Linux, even if the user has their configuration file in
>> "~/.emacs.d/", the library currently prefers the XDG paths here:
>>
>>     (user-file 'cache "image-dired")
>>     => "~/.cache/emacs/image-dired"
>
> This contradicts how Emacs by default finds the init file: its
> preference is the other way around, and it sets user-emacs-directory
> according to the directory from which it loaded the init file.  I
> don't think I understand why this package should behave differently by
> default (it could do that as an opt-in option, of course, if that is
> considered useful to someone).

It should because it is a change for the better.  We have a chance to
decide what this should look like for the foreseeable future, and I
don't think we are bound by past decisions if they do not produce
optimal (or even good) results.

Here's my take on it:

The `user-emacs-directory' directory currently contains a grab-bag of,
well, everything, which among other things makes it hard for a user to
know what is safe to delete and what is not, which files you would need
to backup, etc.  You basically need to manually inspect the files and
consider their content one by one.

The big use-case here, though, is that we want to be able to easily
synchronize our Emacs configuration file between several machines.  You
don't want to synchronize things like your completion history or
recently opened files, because it is not very useful, and you will have
an infinite amounts of conflicts.  (It's somewhat easier with version
control, which is what I use, but less so if you want to use rsync or
unison.)

The solution is to proscribe that:

The categories data, cache and state files belong in specific `data',
`state' and `cache' directories.  This means that e.g. my bookmarks file
goes to the data directory, the recentf and saveplace file goes to
`state', while my image-dired thumbnails go to `cache'.  As for runtime
files, I note that `server-socket-dir' already uses XDG_RUNTIME_DIR (but
gets it manually).

IOW, all this should be orthogonal to where your init file is placed.
You can put it in any of the accepted locations, including the XDG one
if you like.  (Personally, I keep mine in "~/.emacs.d/init.el" and will
continue to do so.)  You should still be able to get all of the above,
by default and with no hassle.

What I propose is obviously inspired by the XDG BDS, but with the Emacs
twist that its interface is not tied to that standard, it's completely
customizable, and it can easily grow to be platform independent.

With all that being said, we must take great care not to cause
unnecessary churn.  In particular, we should absolutely not break
anything that's currently working.  This means that we should respect
locations of already existing files, and ensure they continue working
seamlessly.

PS. Note that (user-directory 'config) in particular already just falls
    back to use whatever `user-emacs-directory' was set to.  There is no
    need to duplicate what is happening in startup.el, or to try to
    outsmart it.

Re: user-directory: New library to find user {conf, data, state, ...} files

[Stefan Kangas]

Stefan Kangas <stefan@marxist.se> writes:

> I also think we should consider adding some heuristics to `user-file' to
> be able to find the old name if it exists.  I considered some
> alternatives, but I thought I'd see what people think first.  Currently,
> application code would say:
>
>     (user-file 'cache "image-dired"
>                        (locate-user-emacs-file "image-dired"))
>
> Which could, if we want to be smart, be simplified to:
>
>     (user-file 'cache "image-dired" "image-dired")

I have implemented the above simplification now, see my recent commit to
features/user-directory.  We could take it even further if we want to of
course, to support the below as well:

> Or even:
>
>     (user-file 'cache "image-dired")
>
> Unfortunately, in this case I think it's harder to be smart:
>
>     (user-file 'data "bookmarks"
>                      (locate-user-emacs-file "bookmarks" ".emacs.bmk"))
>
> But perhaps we can think of something even for that.

Re: user-directory: New library to find user {conf, data, state, ...} files

[Eric S Fraga]

On Sunday,  7 Nov 2021 at 21:04, Stefan Kangas wrote:
> The big use-case here, though, is that we want to be able to easily
> synchronize our Emacs configuration file between several machines.  You
> don't want to synchronize things like your completion history or
> recently opened files, because it is not very useful, and you will have
> an infinite amounts of conflicts.  

+1

especially true for eln-cache?

-- 
Eric S Fraga via Emacs 28.0.60 & org 9.5 on Debian 11.1

Re: user-directory: New library to find user {conf, data, state, ...} files

[Eli Zaretskii]

> From: Stefan Kangas <stefan@marxist.se>
> Date: Sun, 7 Nov 2021 21:04:42 -0600
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Note that on GNU/Linux, even if the user has their configuration file in
> >> "~/.emacs.d/", the library currently prefers the XDG paths here:
> >>
> >>     (user-file 'cache "image-dired")
> >>     => "~/.cache/emacs/image-dired"
> >
> > This contradicts how Emacs by default finds the init file: its
> > preference is the other way around, and it sets user-emacs-directory
> > according to the directory from which it loaded the init file.  I
> > don't think I understand why this package should behave differently by
> > default (it could do that as an opt-in option, of course, if that is
> > considered useful to someone).
> 
> It should because it is a change for the better.  We have a chance to
> decide what this should look like for the foreseeable future, and I
> don't think we are bound by past decisions if they do not produce
> optimal (or even good) results.

The problem is that changes which break users' configurations will not
be appreciated.

> The `user-emacs-directory' directory currently contains a grab-bag of,
> well, everything, which among other things makes it hard for a user to
> know what is safe to delete and what is not, which files you would need
> to backup, etc.  You basically need to manually inspect the files and
> consider their content one by one.
> 
> The big use-case here, though, is that we want to be able to easily
> synchronize our Emacs configuration file between several machines.  You
> don't want to synchronize things like your completion history or
> recently opened files, because it is not very useful, and you will have
> an infinite amounts of conflicts.  (It's somewhat easier with version
> control, which is what I use, but less so if you want to use rsync or
> unison.)
> 
> The solution is to proscribe that:
> 
> The categories data, cache and state files belong in specific `data',
> `state' and `cache' directories.  This means that e.g. my bookmarks file
> goes to the data directory, the recentf and saveplace file goes to
> `state', while my image-dired thumbnails go to `cache'.  As for runtime
> files, I note that `server-socket-dir' already uses XDG_RUNTIME_DIR (but
> gets it manually).
> 
> IOW, all this should be orthogonal to where your init file is placed.
> You can put it in any of the accepted locations, including the XDG one
> if you like.  (Personally, I keep mine in "~/.emacs.d/init.el" and will
> continue to do so.)  You should still be able to get all of the above,
> by default and with no hassle.

My problem is that Emacs computes user-emacs-directory from the place
where it loaded the init file, and then uses the value of
user-emacs-directory to look for other files.  Search the sources for
user-emacs-directory and you will see how we use it.  If the changes
you propose will modify those users of user-emacs-directory to use
user-file machinery instead, and if user-file machinery prefers the
XDG directories, Emacs will not find the files it found previously,
and some of the user customizations will mysteriously stop working.

> With all that being said, we must take great care not to cause
> unnecessary churn.  In particular, we should absolutely not break
> anything that's currently working.  This means that we should respect
> locations of already existing files, and ensure they continue working
> seamlessly.

That is fine, but I'm not sure how you can achieve that goal and still
prefer the XDG directories.

> PS. Note that (user-directory 'config) in particular already just falls
>     back to use whatever `user-emacs-directory' was set to.  There is no
>     need to duplicate what is happening in startup.el, or to try to
>     outsmart it.

What do you mean by "falls back"?  To respect the current behavior,
the value of user-emacs-directory should be used in preference to
everything else.  But if we do that, then in which cases will the XDG
directories be used, since user-emacs-directory always exists and is
defined.

Re: user-directory: New library to find user {conf, data, state, ...} files

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

> The problem is that changes which break users' configurations will not
> be appreciated.

Correct, and fully agreed.  I believe we can avoid this.

> > With all that being said, we must take great care not to cause
> > unnecessary churn.  In particular, we should absolutely not break
> > anything that's currently working.  This means that we should respect
> > locations of already existing files, and ensure they continue working
> > seamlessly.
>
> That is fine, but I'm not sure how you can achieve that goal and still
> prefer the XDG directories.

This is the key point, indeed.

The new name will be preferred only when there is no old file.  If it
exists, the old name will be used.  (This is based on how
'locate-user-emacs-file' works.)  See below.

> > PS. Note that (user-directory 'config) in particular already just falls
> >     back to use whatever `user-emacs-directory' was set to.  There is no
> >     need to duplicate what is happening in startup.el, or to try to
> >     outsmart it.
>
> What do you mean by "falls back"?  To respect the current behavior,
> the value of user-emacs-directory should be used in preference to
> everything else.

Sorry, I should have said "use", not "falls back": it returns the
value of 'user-emacs-directory' and does nothing else.

> But if we do that, then in which cases will the XDG
> directories be used, since user-emacs-directory always exists and is
> defined.

The 'config case is different from the 'cache, 'state and 'data cases,
where the XDG directories will be preferred instead.  To be more
precise, they will be preferred _only_ if we don't specify an old
name, or if we did specify an old name but no such file exists.

To give an example, let's assume that 'user-emacs-directory' is "~/.emacs.d/".

1. If "~/.emacs.d/old" exists, then we get:

    (user-file 'config "new" "old")
    => "~/.emacs.d/old"

    (user-file 'cache "new" "old")
    => "~/.emacs.d/old"

2. However, if "~/.emacs.d/old" does not exist:

    (user-file 'config "new" old")
    => "~/.emacs.d/new"

    (user-file 'cache "new" old")
    => "~/.cache/new"

3. Finally, let's consider the bookmark case, where a user might be
using the very old name "~/.emacs.bmk":

    (user-file 'data "bookmarks" (locate-user-emacs-file "bookmarks"
".emacs.bmk"))
    => "~/.emacs.bmk"

It's true that this requires discipline on behalf of application
developers when calling 'user-directory': they need to provide an "old
name" in addition to a "new name".  I don't see any way around that.
(See my separate email about the state of the API so far, and some
possible simplifications, especially for the third example above.)

I'd consider any failure to use a previously existing file as a bug.
Bugs will inevitably exist, but I don't believe fixing them will be
fundamentally impossible, or necessarily even hard.

Another consideration is that the interface of course has to be
helpful: it has to be easy to use correctly, and hard to use
incorrectly.  I think what it looks like now is not too bad in that
sense, but I'm very open to suggestions about how to do better.

Re: user-directory: New library to find user {conf, data, state, ...} files

[Stefan Kangas]

Stefan Kangas <stefan@marxist.se> writes:

> > Unfortunately, in this case I think it's harder to be smart:
> >
> >     (user-file 'data "bookmarks"
> >                      (locate-user-emacs-file "bookmarks" ".emacs.bmk"))
> >
> > But perhaps we can think of something even for that.

Maybe this could be simplified to

    (user-file 'data "bookmarks" "bookmarks" "~/.emacs.bmk")

or even

    (user-file 'data "bookmarks" "~/.emacs.bmk")

?

Re: user-directory: New library to find user {conf, data, state, ...} files

[Eli Zaretskii]

> From: Stefan Kangas <stefan@marxist.se>
> Date: Mon, 8 Nov 2021 16:26:15 +0100
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, Emacs developers <emacs-devel@gnu.org>
> 
> > That is fine, but I'm not sure how you can achieve that goal and still
> > prefer the XDG directories.
> 
> This is the key point, indeed.
> 
> The new name will be preferred only when there is no old file.  If it
> exists, the old name will be used.  (This is based on how
> 'locate-user-emacs-file' works.)  See below.

"Old file" or "old directory"?  That's an important difference, IMO.

> > What do you mean by "falls back"?  To respect the current behavior,
> > the value of user-emacs-directory should be used in preference to
> > everything else.
> 
> Sorry, I should have said "use", not "falls back": it returns the
> value of 'user-emacs-directory' and does nothing else.
> 
> > But if we do that, then in which cases will the XDG
> > directories be used, since user-emacs-directory always exists and is
> > defined.
> 
> The 'config case is different from the 'cache, 'state and 'data cases,
> where the XDG directories will be preferred instead.

Which of the existing files we have in Emacs will be in one of the 3
latter directories?  Please search the tree for uses of
user-emacs-directory and tell me whether any of those will/should be
in one of those 3.  If none of them should be there, then perhaps my
problem is smaller than I feared, but I'd still want to talk about the
files which will be in those 3.  And also perhaps we should see if
some popular 3rd party packages place files under
user-emacs-directory, and consider those as well.

> 1. If "~/.emacs.d/old" exists, then we get:
> 
>     (user-file 'config "new" "old")
>     => "~/.emacs.d/old"
> 
>     (user-file 'cache "new" "old")
>     => "~/.emacs.d/old"

Not sure I understand why "new" and "old" are different strings.
Presumably, code which uses user-file will look for a particular
basename, so why do we need 2 separate arguments?

> 3. Finally, let's consider the bookmark case, where a user might be
> using the very old name "~/.emacs.bmk":
> 
>     (user-file 'data "bookmarks" (locate-user-emacs-file "bookmarks"
> ".emacs.bmk"))
>     => "~/.emacs.bmk"

And here I don't understand why the caller need to call
locate-user-emacs-file explicitly? why couldn't user-file call it
internally?

Hmm... actually, does the above mean that locate-user-emacs-file will
forever use the old semantics?  Perhaps it should call user-file
instead?

Re: user-directory: New library to find user {conf, data, state, ...} files

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

>> The new name will be preferred only when there is no old file.  If it
>> exists, the old name will be used.  (This is based on how
>> 'locate-user-emacs-file' works.)  See below.
>
> "Old file" or "old directory"?  That's an important difference, IMO.

The above applies to looking for the file name with `user-file'.

When looking for the directory, the new one will always be preferred.
The old name in all cases is just `user-emacs-directory', so it doesn't
make sense to look for it.

I expect that most application code will use `user-file' rather than
`user-directory'.

> Which of the existing files we have in Emacs will be in one of the 3
> latter directories?  Please search the tree for uses of
> user-emacs-directory and tell me whether any of those will/should be
> in one of those 3.  If none of them should be there, then perhaps my
> problem is smaller than I feared, but I'd still want to talk about the
> files which will be in those 3.  And also perhaps we should see if
> some popular 3rd party packages place files under
> user-emacs-directory, and consider those as well.

I have looked over all of `user-emacs-directory' in our tree.  I found
some cases where 'locate-user-emacs-file' was not used, so I fixed them.
Now, if we filter out some irrelevant ones, I see this:

cache

  ./startup.el547:      (push (expand-file-name "eln-cache/"
user-emacs-directory)

data

  ./net/eww.el101:(defcustom eww-bookmarks-directory user-emacs-directory
  ./startup.el401:	 (concat user-emacs-directory "auto-save.list/_s"))
  ./startup.el403:	 (concat user-emacs-directory "auto-save-list/.saves-")))

state

  ./desktop.el259:(defcustom desktop-path (list user-emacs-directory "~")
  ./desktop.el1062:                                 user-emacs-directory)))
  ./desktop.el1255:             user-emacs-directory))))
  ./progmodes/gdb-mi.el654:(defcustom
gdb-window-configuration-directory user-emacs-directory

(The "desktop directory" should probably just be a "desktop file".
Similarly for the eww bookmarks and the gdb variable window
configuration.  But that's a separate issue.)

>> 1. If "~/.emacs.d/old" exists, then we get:
>>
>>     (user-file 'config "new" "old")
>>     => "~/.emacs.d/old"
>>
>>     (user-file 'cache "new" "old")
>>     => "~/.emacs.d/old"
>
> Not sure I understand why "new" and "old" are different strings.
> Presumably, code which uses user-file will look for a particular
> basename, so why do we need 2 separate arguments?

The handling of them is currently different: "new" (NAME) is only
considered in the new directory, while "old" (OLD-NAME) is considered in
the new directory and in `user-emacs-directory'.

If we don't need to make that distinction, this can indeed be simplified
to just one argument.  I'm starting to lean towards the conclusion that
it is indeed okay to simplify this.

>> 3. Finally, let's consider the bookmark case, where a user might be
>> using the very old name "~/.emacs.bmk":
>>
>>     (user-file 'data "bookmarks" (locate-user-emacs-file "bookmarks"
>> ".emacs.bmk"))
>>     => "~/.emacs.bmk"
>
> And here I don't understand why the caller need to call
> locate-user-emacs-file explicitly? why couldn't user-file call it
> internally?

I have made a simplification on the branch, so the above is now:

    (user-file 'data "bookmarks" "bookmarks" "~/.emacs.bmk")

With the further simplification of NAME and OLD-NAME discussed above,
this would be just:

    (user-file 'data "bookmarks" "~/.emacs.bmk")

> Hmm... actually, does the above mean that locate-user-emacs-file will
> forever use the old semantics?  Perhaps it should call user-file
> instead?

Sure, that is definitely something we could consider.  The main idea
here is that application code should use `user-file' directly though, to
be more explicit about what kind of file it is that they want to store.

Re: user-directory: New library to find user {conf, data, state, ...} files

[Eli Zaretskii]

> From: Stefan Kangas <stefan@marxist.se>
> Date: Mon, 8 Nov 2021 23:05:57 -0800
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> The new name will be preferred only when there is no old file.  If it
> >> exists, the old name will be used.  (This is based on how
> >> 'locate-user-emacs-file' works.)  See below.
> >
> > "Old file" or "old directory"?  That's an important difference, IMO.
> 
> The above applies to looking for the file name with `user-file'.
> 
> When looking for the directory, the new one will always be preferred.
> The old name in all cases is just `user-emacs-directory', so it doesn't
> make sense to look for it.

That's not true, unless we are mis-communicating.  I have quite a few
directories under ~/.emacs.d/, and most of them are important.  Are
you saying we will now look for them all according to the XDG spec?

> I expect that most application code will use `user-file' rather than
> `user-directory'.

If some Lisp program needs to ensure a sub-directory of
user-emacs-directory exists, it will want to use user-directory, I
think.

> > Which of the existing files we have in Emacs will be in one of the 3
> > latter directories?  Please search the tree for uses of
> > user-emacs-directory and tell me whether any of those will/should be
> > in one of those 3.  If none of them should be there, then perhaps my
> > problem is smaller than I feared, but I'd still want to talk about the
> > files which will be in those 3.  And also perhaps we should see if
> > some popular 3rd party packages place files under
> > user-emacs-directory, and consider those as well.
> 
> I have looked over all of `user-emacs-directory' in our tree.  I found
> some cases where 'locate-user-emacs-file' was not used, so I fixed them.
> Now, if we filter out some irrelevant ones, I see this:
> 
> cache
> 
>   ./startup.el547:      (push (expand-file-name "eln-cache/"
> user-emacs-directory)

Isn't that a problem?  It would mean we'd need to preload xdg.el and
user-file, and all the stuff they need to work, right?  Also, the XDG
spec says about "cache":

  $XDG_CACHE_HOME defines the base directory relative to which
  user-specific non-essential data files should be stored.

I don't think files in eln-cache can be classified as "non-essential".
They are important for using Emacs; removing them will make Emacs slow
and sluggish for quite some time after startup.

> data
> 
>   ./net/eww.el101:(defcustom eww-bookmarks-directory user-emacs-directory
>   ./startup.el401:	 (concat user-emacs-directory "auto-save.list/_s"))
>   ./startup.el403:	 (concat user-emacs-directory "auto-save-list/.saves-")))
> 
> state
> 
>   ./desktop.el259:(defcustom desktop-path (list user-emacs-directory "~")
>   ./desktop.el1062:                                 user-emacs-directory)))
>   ./desktop.el1255:             user-emacs-directory))))
>   ./progmodes/gdb-mi.el654:(defcustom
> gdb-window-configuration-directory user-emacs-directory

And you are saying we will redirect all those to the XDG tree?  Users
have files in these places, and they need to trust Emacs to find those
files when it is restarted.  How can we change where we look for them
without breaking configurations?

> (The "desktop directory" should probably just be a "desktop file".
> Similarly for the eww bookmarks and the gdb variable window
> configuration.  But that's a separate issue.)

No, desktop.el is explicitly programmed to be able to use a "path",
i.e. a list of directories.  You can have several different desktop
files in several different directories, and use each one for its own
project or type of activity.

> >> 1. If "~/.emacs.d/old" exists, then we get:
> >>
> >>     (user-file 'config "new" "old")
> >>     => "~/.emacs.d/old"
> >>
> >>     (user-file 'cache "new" "old")
> >>     => "~/.emacs.d/old"
> >
> > Not sure I understand why "new" and "old" are different strings.
> > Presumably, code which uses user-file will look for a particular
> > basename, so why do we need 2 separate arguments?
> 
> The handling of them is currently different: "new" (NAME) is only
> considered in the new directory, while "old" (OLD-NAME) is considered in
> the new directory and in `user-emacs-directory'.
> 
> If we don't need to make that distinction, this can indeed be simplified
> to just one argument.  I'm starting to lean towards the conclusion that
> it is indeed okay to simplify this.

I see no reason not to.  We are not going to change the base names as
well, are we?

>     (user-file 'data "bookmarks" "bookmarks" "~/.emacs.bmk")
> 
> With the further simplification of NAME and OLD-NAME discussed above,
> this would be just:
> 
>     (user-file 'data "bookmarks" "~/.emacs.bmk")

Once again, why not just one argument?

Re: user-directory: New library to find user {conf, data, state, ...} files

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

>> When looking for the directory, the new one will always be preferred.
>> The old name in all cases is just `user-emacs-directory', so it doesn't
>> make sense to look for it.
>
> That's not true, unless we are mis-communicating.  I have quite a few
> directories under ~/.emacs.d/, and most of them are important.  Are
> you saying we will now look for them all according to the XDG spec?

Aha, yes we are talking about two different things.

If you need some file in `user-emacs-directory' ("file" in the Unix
sense here, i.e. it could also be a directory) you would use
`user-file'.  So to get "~/.emacs.d/auto-save-list/" you would say:

    (user-file 'state "auto-save-list")
    => "~/.local/state/emacs/auto-save-list"

`user-directory' only returns the top-level directory.  There is no way
to give it a name of a file.

    (user-directory 'state)
    => "/home/skangas/.local/state/emacs"

However, your question makes me think that we should make `user-file'
the only public function here, to avoid confusion.  I think it's rare to
need the directory itself, and if someone does they could just use

    user-emacs-directory
    => "~/.emacs.d/"

or

    (user-file 'cache "")
    => "~/.cache/emacs"

So I think we should rename the library to user-file.el, and make
`user-file' the only non-internal and user-facing function.  That makes
it simpler, and more clear.

>>   ./startup.el547:      (push (expand-file-name "eln-cache/"
>> user-emacs-directory)
>
> Isn't that a problem?  It would mean we'd need to preload xdg.el and
> user-file, and all the stuff they need to work, right?  Also, the XDG
> spec says about "cache":

Yes, if we want to move the eln-cache to the cache directory, we need to
do something about preloading.  The relevant XDG code could be moved
from xdg.el and xdg.el could depend on the preloaded location; this is
about 30-40 lines of code.  We would also need to rewrite the code I
have in user-directory.el to not use cl-lib.el.

>   $XDG_CACHE_HOME defines the base directory relative to which
>   user-specific non-essential data files should be stored.
>
> I don't think files in eln-cache can be classified as "non-essential".
> They are important for using Emacs; removing them will make Emacs slow
> and sluggish for quite some time after startup.

Removing anything from a cache will always make performance worse until
you can recache it, but the character of a cache is that it can always
be recreated from some original data.

So my understanding is that "non-essential" here is provided to contrast
with the data file, and simply means that it is not (highly) important
not to lose those files.  I.e. it is just a way to further specify that
this is the directory for cached files.

> And you are saying we will redirect all those to the XDG tree?  Users
> have files in these places, and they need to trust Emacs to find those
> files when it is restarted.  How can we change where we look for them
> without breaking configurations?

For each file NAME: (1) look for NAME in the XDG directory, if NAME
exists there and is readable return it, otherwise (2) look for it in
`user-emacs-directory', if NAME exists there and is readable return it,
otherwise (3) return the file name of NAME in the XDG directory.  This
is the same basic approach that we also use in `locate-user-emacs-file'.

Is that what you mean?

>>     (user-file 'data "bookmarks" "~/.emacs.bmk")
>
> Once again, why not just one argument?

For backwards compatibility.  The "bookmarks" file used to be in
"~/.emacs.bmk" (before 2010 or so?).

We currently locate it using this call:

    (locate-user-emacs-file "bookmarks" ".emacs.bmk")

Re: user-directory: New library to find user {conf, data, state, ...} files

[Eli Zaretskii]

> From: Stefan Kangas <stefan@marxist.se>
> Date: Tue, 9 Nov 2021 12:38:52 -0800
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> When looking for the directory, the new one will always be preferred.
> >> The old name in all cases is just `user-emacs-directory', so it doesn't
> >> make sense to look for it.
> >
> > That's not true, unless we are mis-communicating.  I have quite a few
> > directories under ~/.emacs.d/, and most of them are important.  Are
> > you saying we will now look for them all according to the XDG spec?
> 
> Aha, yes we are talking about two different things.
> 
> If you need some file in `user-emacs-directory' ("file" in the Unix
> sense here, i.e. it could also be a directory) you would use
> `user-file'.  So to get "~/.emacs.d/auto-save-list/" you would say:
> 
>     (user-file 'state "auto-save-list")
>     => "~/.local/state/emacs/auto-save-list"
> 
> `user-directory' only returns the top-level directory.  There is no way
> to give it a name of a file.
> 
>     (user-directory 'state)
>     => "/home/skangas/.local/state/emacs"

I'm saying that user-directory should also honor user-emacs-directory,
because we have no way of knowing the intent of the caller of
user-directory: it could be that the caller wants then to get to some
file under it.

> However, your question makes me think that we should make `user-file'
> the only public function here, to avoid confusion.  I think it's rare to
> need the directory itself, and if someone does they could just use
> 
>     user-emacs-directory
>     => "~/.emacs.d/"
> 
> or
> 
>     (user-file 'cache "")
>     => "~/.cache/emacs"
> 
> So I think we should rename the library to user-file.el, and make
> `user-file' the only non-internal and user-facing function.  That makes
> it simpler, and more clear.

Could be, indeed.  (But please make user-file accept nil instead of an
empty string.)

> >>   ./startup.el547:      (push (expand-file-name "eln-cache/"
> >> user-emacs-directory)
> >
> > Isn't that a problem?  It would mean we'd need to preload xdg.el and
> > user-file, and all the stuff they need to work, right?  Also, the XDG
> > spec says about "cache":
> 
> Yes, if we want to move the eln-cache to the cache directory, we need to
> do something about preloading.  The relevant XDG code could be moved
> from xdg.el and xdg.el could depend on the preloaded location; this is
> about 30-40 lines of code.  We would also need to rewrite the code I
> have in user-directory.el to not use cl-lib.el.

That's a bummer, because we didn't intend to preload xdg.el.  And
since it prefers the "traditional" setups anyway, I wonder whether
preloading it would be justified, as in most cases its code will be
trivially bypassed.

> >   $XDG_CACHE_HOME defines the base directory relative to which
> >   user-specific non-essential data files should be stored.
> >
> > I don't think files in eln-cache can be classified as "non-essential".
> > They are important for using Emacs; removing them will make Emacs slow
> > and sluggish for quite some time after startup.
> 
> Removing anything from a cache will always make performance worse until
> you can recache it, but the character of a cache is that it can always
> be recreated from some original data.
> 
> So my understanding is that "non-essential" here is provided to contrast
> with the data file, and simply means that it is not (highly) important
> not to lose those files.  I.e. it is just a way to further specify that
> this is the directory for cached files.

That's not true for the *.eln files: they cannot be "simply lost".
Note that the XDG spec says "data files", which *.eln aren't: they are
code.  I think it would be wrong to put *.eln files there for the same
reasons it would be wrong to put *.elc files there.

> > And you are saying we will redirect all those to the XDG tree?  Users
> > have files in these places, and they need to trust Emacs to find those
> > files when it is restarted.  How can we change where we look for them
> > without breaking configurations?
> 
> For each file NAME: (1) look for NAME in the XDG directory, if NAME
> exists there and is readable return it, otherwise (2) look for it in
> `user-emacs-directory', if NAME exists there and is readable return it,
> otherwise (3) return the file name of NAME in the XDG directory.  This
> is the same basic approach that we also use in `locate-user-emacs-file'.

Why not start with (2)?  It will save us one step.

> Is that what you mean?
> 
> >>     (user-file 'data "bookmarks" "~/.emacs.bmk")
> >
> > Once again, why not just one argument?
> 
> For backwards compatibility.  The "bookmarks" file used to be in
> "~/.emacs.bmk" (before 2010 or so?).
> 
> We currently locate it using this call:
> 
>     (locate-user-emacs-file "bookmarks" ".emacs.bmk")

If the 2nd argument is optional for alternative names, that's OK, I
think.

Thanks.