Relative directories for database-mail_root and -path?

Relative directories for database-mail_root and -path?

[Ralph Seichter]

Looking at [1], I am trying to figure out if/how the semantic equivalent
of the pseudo configuration

	[database]
	mail_root = ~/.maildir
	path = ~/.local/share/notmuch/default

can be achieved? The docs don't mention that '~' has any special
meaning, and my tests agree. How then does one specify directories
relative to the user's $HOME ? Simply dropping the ~/ prefix and using
.maildir or .local does not seem to work either. I found the latter
being suggested in some Internet search result or other, and thought I'd
give it a try, but no dice.

-Ralph

[1] https://notmuchmail.org/doc/latest/man1/notmuch-config.html

Re: Relative directories for database-mail_root and -path?

[Pengji Zhang]

"Ralph Seichter" <ralph@ml.seichter.de> writes:

> Looking at [1], I am trying to figure out if/how the semantic
> equivalent of the pseudo configuration
>
> 	[database]
> 	mail_root = ~/.maildir
> 	path = ~/.local/share/notmuch/default
>
> can be achieved? The docs don't mention that '~' has any special
> meaning, and my tests agree. How then does one specify directories
> relative to the user's $HOME ? Simply dropping the ~/ prefix and using
> .maildir or .local does not seem to work either. I found the latter
> being suggested in some Internet search result or other, and thought
> I'd give it a try, but no dice.

I suppose dropping the prefix should work? In 'lib/config.cc':

--8<---------------cut here---------------start------------->8---
static char *
_expand_path (void *ctx, const char *key, const char *val)
{
    char *expanded_val;

    if ((strcmp (key, "database.path") == 0 ||
	 strcmp (key, "database.mail_root") == 0 ||
	 strcmp (key, "database.hook_dir") == 0 ||
	 strcmp (key, "database.backup_path") == 0 ) &&
	val[0] != '/')
	expanded_val = talloc_asprintf (ctx, "%s/%s", getenv ("HOME"), val);
    else
	expanded_val = talloc_strdup (ctx, val);

    return expanded_val;
}
--8<---------------cut here---------------end--------------->8---

So if the value does not start with '/', it is considered relative to
the home directory.

What does 'notmuch config list' show for you?

Pengji

Re: Relative directories for database-mail_root and -path?

[Ralph Seichter]

* Pengji Zhang:

>> Simply dropping the ~/ prefix and using .maildir or .local does not
>> seem to work either. [...]
>
> I suppose dropping the prefix should work? In 'lib/config.cc' [...]

And it does indeed work, thank you for looking into it. I just tried
again, this time with a freshly minted config file.

  [database]
  mail_root=.maildir
  path=.local/share/notmuch/default

"notmuch config list" then results in:

  database.backup_dir=/home/alice/.local/share/notmuch/default/backups
  database.hook_dir=/home/alice/.local/share/notmuch/default/hooks
  database.mail_root=/home/alice/.maildir
  database.path=/home/alice/.local/share/notmuch/default

I don't know what exactly I did wrong during yesterday's experiments,
but I am sure it was user error. My bad.

As for finding information about notmuch's approach to relative paths, I
mentioned that I read a hint in a search result. I think it was a GitHub
issue for "afew", so it was an indirect reference to a notmuch feature.

Is the no-/-means-relative-to-HOME made explicit in notmuch's own
documentation, and I have overlooked it (not unheard of), or is this
something to be learned only by examining the source code?

-Ralph

Re: Relative directories for database-mail_root and -path?

[David Bremner]

"Ralph Seichter" <ralph@ml.seichter.de> writes:
>
> Is the no-/-means-relative-to-HOME made explicit in notmuch's own
> documentation, and I have overlooked it (not unheard of), or is this
> something to be learned only by examining the source code?
>

For better or worse, most things like that are documented in
notmuch-config(1), and that one is too.

d

Re: Relative directories for database-mail_root and -path?

[Ralph Seichter]

* David Bremner:

> For better or worse, most things like that are documented in
> notmuch-config(1), and that one is too.

I made quite a hash of this, didn't I? I even quoted the manpage's URL
in my original question, because that was naturally the first place I
checked, but I missed the relevant paragraph:

  The available configuration items are described below. Non-absolute
  paths are presumed relative to $HOME for items in section database.

I don't understand how I fumbled things so thoroughly yesterday. My
apologies for wasting your time.

-Ralph

Re: Relative directories for database-mail_root and -path?

[Tomi Ollila]

On Sat, Oct 26 2024, Ralph Seichter wrote:

> * David Bremner:
>
>> For better or worse, most things like that are documented in
>> notmuch-config(1), and that one is too.
>
> I made quite a hash of this, didn't I? I even quoted the manpage's URL
> in my original question, because that was naturally the first place I
> checked, but I missed the relevant paragraph:
>
>   The available configuration items are described below. Non-absolute
>   paths are presumed relative to $HOME for items in section database.
>
> I don't understand how I fumbled things so thoroughly yesterday. My
> apologies for wasting your time.

IT happens to everyone :D

Anyway all this means our documentation is tolerable enough not
needing any updates on this matter ;D

Tomi

>
> -Ralph

Re: Relative directories for database-mail_root and -path?

[Carl Worth]

On Sat, Oct 26 2024, Ralph Seichter wrote:
> I made quite a hash of this, didn't I? I even quoted the manpage's URL
> in my original question, because that was naturally the first place I
> checked, but I missed the relevant paragraph:

No apology needed!

I just went to double-check the man page myself. I first read the
details of 'database.mail_root' and then 'database.path'. When I didn't
find the detail there, I also read the section below on 'DATABASE
LOCATION'.

I still hadn't found anything about relative paths, but since I had the
email from David telling me this was documented there, I went back and
searched (computationally, not visually) for "relative" and finally
found the sentence you found as well:

>   The available configuration items are described below. Non-absolute
>   paths are presumed relative to $HOME for items in section database.

It's in its own sentence before the details of any specific
configuration item.

So I did just demonstrate that it's an easy thing to miss. That's not a
short man page, and it will generally be used as a reference (looking
for details of one specific entry, not reading the entire page straight
through).

In spite of the redundancy, perhaps it would make sense to add:

	A non-absolute path is presumed relative to $HOME.

to the descriptions of each of:

	database.backup_dir
        database.hook_dir
        database.mail_root
        database.path

(And if someone does this, they should also check whether git.path
deserves the same treatment.)

-Carl

Re: Relative directories for database-mail_root and -path?

[Ralph Seichter]

* Carl Worth:

> No apology needed!

Thank you all who commented and were understanding. You made me feel
less foolish.

> In spite of the redundancy, perhaps it would make sense to add:
> A non-absolute path is presumed relative to $HOME. [...]

Good idea. That text, or perhaps the equivalent of a

  <a href="#paths">see: paths</a>

link pointing to the existing paragraph containing the detailed
explanation of absolute versus relative paths? Like you mentioned, I
jumped into the manpage searching for database.path specifics, including
DATABASE LOCATION. Scrolling upwards to an unmarked paragraph did not
occur to me at the time.


By the way, it is certainly not a technical requirement, but supporting
the "~" prefix as an alias for $HOME would be an idiomatic way to permit
users to explicitly mark paths as relative to their home directory.
Interpreting the absence of a "/" prefix obviously works fine from a
technical standpoint, but added "~" support would enhance readability,
I think.

-Ralph

Re: Relative directories for database-mail_root and -path?

[Tomi Ollila]

On Mon, Oct 28 2024, Ralph Seichter wrote:

> * Carl Worth:
>
>> No apology needed!
>
> Thank you all who commented and were understanding. You made me feel
> less foolish.
>
>> In spite of the redundancy, perhaps it would make sense to add:
>> A non-absolute path is presumed relative to $HOME. [...]

There is certain "format" in those entries in notmuch config *manual*
pages. I think having the line

A non-absolute path is presumed relative to $HOME.

separately works there (before these 'History:' and 'Default:' lines
(and that could look pretty good).

Maintenance burden is minimal there (should one ever need to edit
these lines, those are in close proximity to each other and grep(1)
finds all entries quickly).

>
> Good idea. That text, or perhaps the equivalent of a
>
>   <a href="#paths">see: paths</a>
>
> link pointing to the existing paragraph containing the detailed
> explanation of absolute versus relative paths? Like you mentioned, I
> jumped into the manpage searching for database.path specifics, including
> DATABASE LOCATION. Scrolling upwards to an unmarked paragraph did not
> occur to me at the time.

That could work in html versions of the documents.

>
> By the way, it is certainly not a technical requirement, but supporting
> the "~" prefix as an alias for $HOME would be an idiomatic way to permit
> users to explicitly mark paths as relative to their home directory.
> Interpreting the absence of a "/" prefix obviously works fine from a
> technical standpoint, but added "~" support would enhance readability,
> I think.

One could make ~/ work there (which would be nice -- and patches tolerated,
says an old saying), that is just more work :D

>
> -Ralph

Tomi

[PATCH] Add support for ~/foo style relative paths

[Ralph Seichter]

In addition to the existing implicit syntax of non-absolute paths
being interpreted as relative to users' $HOME directories, also
allow the explicit syntax "~/some/relative/path". This well known
variant improves readability of configuration files.

Signed-off-by: Ralph Seichter <ralph@seichter.de>
---
 lib/config.cc | 18 ++++++++++++------
 1 file changed, 12 insertions(+), 6 deletions(-)

diff --git a/lib/config.cc b/lib/config.cc
index acb397ec..2537b451 100644
--- a/lib/config.cc
+++ b/lib/config.cc
@@ -400,15 +400,21 @@ notmuch_config_pairs_destroy (notmuch_config_pairs_t *pairs)
 static char *
 _expand_path (void *ctx, const char *key, const char *val)
 {
-    char *expanded_val;
+    char *expanded_val = NULL;
 
-    if ((strcmp (key, "database.path") == 0 ||
+    if (strcmp (key, "database.path") == 0 ||
 	 strcmp (key, "database.mail_root") == 0 ||
 	 strcmp (key, "database.hook_dir") == 0 ||
-	 strcmp (key, "database.backup_path") == 0 ) &&
-	val[0] != '/')
-	expanded_val = talloc_asprintf (ctx, "%s/%s", getenv ("HOME"), val);
-    else
+	 strcmp (key, "database.backup_path") == 0) {
+	if (val[0] == '~')
+	    /* Explicit syntax for "~/some/relative/path" */
+	    expanded_val = talloc_asprintf (ctx, "%s%s", getenv ("HOME"), &val[1]);
+	else if (val[0] != '/')
+	    /* Implicit syntax for "some/relative/path" */
+	    expanded_val = talloc_asprintf (ctx, "%s/%s", getenv ("HOME"), val);
+    }
+    if (! expanded_val)
+	/* Catchall, no special expansion */
 	expanded_val = talloc_strdup (ctx, val);
 
     return expanded_val;
-- 
2.47.0

Re: [PATCH] Add support for ~/foo style relative paths

[David Bremner]

Ralph Seichter <ralph@ml.seichter.de> writes:
> +	 strcmp (key, "database.backup_path") == 0) {
> +	if (val[0] == '~')
> +	    /* Explicit syntax for "~/some/relative/path" */
> +	    expanded_val = talloc_asprintf (ctx, "%s%s", getenv ("HOME"), &val[1]);

what should we do if val[1] != '/'?  I think that is what derailed us
last time. Do we think it's worth checking the passwd database for user
names? what about things like '~.foo' ?

> +	else if (val[0] != '/')
> +	    /* Implicit syntax for "some/relative/path" */
> +	    expanded_val = talloc_asprintf (ctx, "%s/%s", getenv ("HOME"), val);
> +    }
> +    if (! expanded_val)
> +	/* Catchall, no special expansion */
>  	expanded_val = talloc_strdup (ctx, val);
>  
>      return expanded_val;

If we do decide to make this change, obviously we'll need to update the
documentation and (maybe less obviously) tests.

Re: [PATCH] Add support for ~/foo style relative paths

[Tomi Ollila]

On Thu, Oct 31 2024, David Bremner wrote:

> Ralph Seichter <ralph@ml.seichter.de> writes:
>> +	 strcmp (key, "database.backup_path") == 0) {
>> +	if (val[0] == '~')
>> +	    /* Explicit syntax for "~/some/relative/path" */
>> +	    expanded_val = talloc_asprintf (ctx, "%s%s", getenv ("HOME"), &val[1]);
>
> what should we do if val[1] != '/'?  I think that is what derailed us
> last time. Do we think it's worth checking the passwd database for user
> names? what about things like '~.foo' ?

IMO:
	if (val[0] == '~' && val[1] == '/')

i.e. expand leading ~/ as $HOME/ -- otherwise expand as previously.

Tomi

>> +	else if (val[0] != '/')
>> +	    /* Implicit syntax for "some/relative/path" */
>> +	    expanded_val = talloc_asprintf (ctx, "%s/%s", getenv ("HOME"), val);
>> +    }
>> +    if (! expanded_val)
>> +	/* Catchall, no special expansion */
>>  	expanded_val = talloc_strdup (ctx, val);
>>  
>>      return expanded_val;
>
> If we do decide to make this change, obviously we'll need to update the
> documentation and (maybe less obviously) tests.

Re: [PATCH] Add support for ~/foo style relative paths

[Ralph Seichter]

* David Bremner:

> what should we do if val[1] != '/'?

My patch assumes that the answer to this question is "nothing". The
string "~xyz" would expand to "/home/alicexyz" instead of, say,
"/home/xyz" if notmuch is run by user alice.

It is not a very useful expansion, but it does what it says on the tin,
and saves the extra effort of parsing /etc/passwd: Tilde expands to the
process owner's home directory, no matter what character follows next.

Mind you, I don't object to being able to reference other users' home
directories. I just don't quite see how useful that ability would be,
given the private nature of email. Am I missing an obvious usecase here?

-Ralph

Re: [PATCH] Add support for ~/foo style relative paths

[David Bremner]

"Ralph Seichter" <ralph@ml.seichter.de> writes:

> Mind you, I don't object to being able to reference other users' home
> directories. I just don't quite see how useful that ability would be,
> given the private nature of email. Am I missing an obvious usecase here?

I don't see an obvious use case either, I'm just thinking about user
expectations and error reporting.

Re: [PATCH] Add support for ~/foo style relative paths

[Tomi Ollila]

On Fri, Nov 01 2024, David Bremner wrote:

> "Ralph Seichter" <ralph@ml.seichter.de> writes:
>
>> Mind you, I don't object to being able to reference other users' home
>> directories. I just don't quite see how useful that ability would be,
>> given the private nature of email. Am I missing an obvious usecase here?
>
> I don't see an obvious use case either, I'm just thinking about user
> expectations and error reporting.

$ man dash | grep -B3 -C5 'tilde expansion'

  Tilde Expansion (substituting a user's home directory)
       A  word beginning with an unquoted tilde character (~) is sub‐
       jected to tilde expansion.  All the characters up to  a  slash
       (/)  or  the end of the word are treated as a username and are
       replaced with the user's home directory.  If the  username  is
       missing (as in ~/foobar), the tilde is replaced with the value
       of the HOME variable (the current user's home directory).

IMO only deviation from this can be *not* supporting ~user. ~foo
expanding as ${HOME}foo is inconsistent and unexpected...

Tomi

PS. did you know that e.g. ssh does not expand ~ -> getenv("HOME"),
but reads the real directory from passwd database.

PPS. bash(1) also expands ~+ as $PWD and ~- as $OLDPWD (and zsh(1)),
     again I suggest *not* supporting these (either ;D):
  $ bash -c 'echo ~+'
  /home/too/vc/ext/notmuch
  $ bash -c 'echo ~+/a-dir'
  /home/too/vc/ext/notmuch/a-dir
  $ bash -c 'echo ~+snafu'
  ~+snafu\r

Re: [PATCH] Add support for ~/foo style relative paths

[Ralph Seichter]

* Tomi Ollila:

> IMO only deviation from this can be *not* supporting ~user. ~foo
> expanding as ${HOME}foo is inconsistent and unexpected...

Fair enough. Still, supporting only the "~/" prefix for ${HOME}/ but
interpreting "~bob" as ${HOME}/~bob is just as unexpected, don't you
think?


> PS. did you know that e.g. ssh does not expand ~ -> getenv("HOME"),
> but reads the real directory from passwd database.

If I had to guess, I'd say this was probably done with security concerns
in mind? Avoid possible shenanigans with a process' environment.

-Ralph

Re: [PATCH] Add support for ~/foo style relative paths

[Tomi Ollila]

On Fri, Nov 01 2024, Ralph Seichter wrote:

> * Tomi Ollila:
>
>> IMO only deviation from this can be *not* supporting ~user. ~foo
>> expanding as ${HOME}foo is inconsistent and unexpected...
>
> Fair enough. Still, supporting only the "~/" prefix for ${HOME}/ but
> interpreting "~bob" as ${HOME}/~bob is just as unexpected, don't you
> think?

Well, if shells or emacs cannot do tilde expansion then those just
keept it as is:

  $ HOME=/where/ever sh -c 'echo ~'
  /where/ever

  $ HOME=/where/ever sh -c 'echo ~too'
  /home/too

  $ HOME=/where/ever sh -c 'echo ~nouser'
  ~nouser

a bit shorter for emacs to fit better...

  $ HOME=/x emacs --batch --eval '(message "%s" (expand-file-name "~"))'
  /x

  $ HOME=/x emacs --batch --eval '(message "%s" (expand-file-name "~too"))'
  /home/too

  $ HOME=/x emacs --batch --eval '(message "%s" (expand-file-name "~na"))'
  /home/too/~na

this may happen with zsh -- I don't think it breaks "the rule"

  $ HOME=/where/ever zsh -c 'echo ~nouser'
  zsh:1: no such user or named directory: nouser

>> PS. did you know that e.g. ssh does not expand ~ -> getenv("HOME"),
>> but reads the real directory from passwd database.
>
> If I had to guess, I'd say this was probably done with security concerns
> in mind? Avoid possible shenanigans with a process' environment.
>
> -Ralph

Re: [PATCH] Add support for ~/foo style relative paths

[Ralph Seichter]

* Tomi Ollila:

> Well, if shells or emacs cannot do tilde expansion then those just
> keept it as is [...]

That's thorough work, and I am convinced. Shall we agree on supporting
only regex "^~/" as a representation of ${HOME}/ and continue treating
matches for regex "^(~[^/])" as ${HOME}/\1 then?

As for the implementation: My draft patch was mostly meant to get the
ball rolling. I don't know enough about the notmuch development
ecosystem to feel comfortable adding tests and suchlike, so if one of
you guys were willing to handle this whole tilde expansion feature, I'd
really appreciate that.

-Ralph