Unicode Paths

Unicode Paths

[Martin Owens]

Hello Again,

I notice in the lib code notmuch_database_open(),
notmuch_database_create() these functions use const char *path for the
directory path input. Is this unicode safe?

The python bindings (and ctype docs) seem to suggest using something
called 'wchar_t *' for accepting unicode but that's for C not C++.

Is this something that should be patched?

Best Regards, Martin Owens

Re: Unicode Paths

[Kan-Ru Chen]

Martin Owens <doctormo@gmail.com> writes:

> Hello Again,
>
> I notice in the lib code notmuch_database_open(),
> notmuch_database_create() these functions use const char *path for the
> directory path input. Is this unicode safe?
>
> The python bindings (and ctype docs) seem to suggest using something
> called 'wchar_t *' for accepting unicode but that's for C not C++.
>
> Is this something that should be patched?

I think as long as the path does not contain embedded null character
then it is safe. Most posix filesystem does not allow null character in
the filename so you cannot use UTF-16 or UTF-32 to encode the unicode
path.

-- 
Kanru

Re: Unicode Paths

[Martin Owens]

It looks like the python variables do include null, my investigations
show that the problem also effects tag names.

The symptoms can be seen when trying to use the python interface and
using unicode tag names or paths. Instead of seeing 'mytag1' we see 'm'
and instead of '/my/path/to/mail' we see '/' thus causing issues were
the db amusingly was trying to write to root.

I'll see if there is a way to remove the nulls from the strings in the
python bindings.

Martin,

On Wed, 2011-09-14 at 12:38 +0800, Kan-Ru Chen wrote:
> I think as long as the path does not contain embedded null character
> then it is safe. Most posix filesystem does not allow null character
> in
> the filename so you cannot use UTF-16 or UTF-32 to encode the unicode
> path. 

Re: Unicode Paths

[Austin Clements]

On Tue, Sep 13, 2011 at 11:55 PM, Martin Owens <doctormo@gmail.com> wrote:
> Hello Again,
>
> I notice in the lib code notmuch_database_open(),
> notmuch_database_create() these functions use const char *path for the
> directory path input. Is this unicode safe?
>
> The python bindings (and ctype docs) seem to suggest using something
> called 'wchar_t *' for accepting unicode but that's for C not C++.
>
> Is this something that should be patched?

char* is the correct type for paths on POSIX systems.  The *meaning*
of those bytes is a more complicated matter and depends on your locale
settings.  On old systems it was generally ASCII, on modern systems
it's generally UTF-8, and it can be many other things.  However, as a
consequence of UNIX's C heritage, it is *always* terminated with a
NULL byte and cannot contain embedded NULL's.  Any encoding that
doesn't satisfy this would not be a valid encoding for file names (you
couldn't even pass such a file name to the open() system call, because
it expects a NULL-terminated byte string).

wchar_t is another matter entirely.  wchar_t is the type used by C to
represent wide strings internally, which generally (but not
necessarily!) means it stores a Unicode code point.  However, this
isn't an encoding, and different compilers can give wchar_t different
meanings, so wchar_t strings aren't generally appropriate for storing
or sharing between processes or with the kernel.

Re: Unicode Paths

[Sebastian Spaeth]

[-- Attachment #1: Type: text/plain, Size: 1663 bytes --]

On Thu, 15 Sep 2011 13:52:12 -0400, Austin Clements <amdragon@mit.edu> wrote:
> On Tue, Sep 13, 2011 at 11:55 PM, Martin Owens <doctormo@gmail.com> wrote:
> > Hello Again,
> >
> > I notice in the lib code notmuch_database_open(),
> > notmuch_database_create() these functions use const char *path for the
> > directory path input. Is this unicode safe?
> >
> > The python bindings (and ctype docs) seem to suggest using something
> > called 'wchar_t *' for accepting unicode but that's for C not C++.
> >
> > Is this something that should be patched?
> 
> char* is the correct type for paths on POSIX systems.  The *meaning*
> of those bytes is a more complicated matter and depends on your locale
> settings.  On old systems it was generally ASCII, on modern systems
> it's generally UTF-8, and it can be many other things.  However, as a
> consequence of UNIX's C heritage, it is *always* terminated with a
> NULL byte and cannot contain embedded NULL's.

Right, that's what we are doing, passing in utf-8 encoded unicode
strings to char*, which should be just fine if that is what the
underlying OS uses.

> wchar_t is another matter entirely.  wchar_t is the type used by C to
> represent wide strings internally, which generally (but not
> necessarily!) means it stores a Unicode code point.  However, this
> isn't an encoding, and different compilers can give wchar_t different
> meanings, so wchar_t strings aren't generally appropriate for storing
> or sharing between processes or with the kernel.

Mmh, I remember I attempted to user wchar_t to pass in unicode objects
directly and it had failed miserably.

Sebastian

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]