[PATCH] Add "scandir" procedure

[PATCH] Add "scandir" procedure

[nalaginrut]

Hi guys!
I found there isn't "scandir" in current Guile. And we may use "ftw" to
instead. I guess "ftw" traverse all sub-directoies. Yes, we may use nftw
to filter the level we don't need, but "ftw" seems always traverse all
sub-directories. If my guess is correct, I believe it's too slow for
someone, for a instance, me. :-)
But the efficiency is not the case.
The reason is that when I was trying to write a web server ,I need to
print directory content in HTML back to the client. And I found "ftw"
won't return "." and "..", moreover ,the result isn't sorted.
And I think "scandir" which provided by POSIX is a better solution. Many
dynamic language like PHP own a "scandir" implementation. I believe
Guile should have one. So I wrote an implementation into filesys.c.

There's a little difference of usage between C and Guile version. But
it's trivial if you're familiar with the C version.

The usage is like this:
-------------------------------
scandir dir [filter [sort]]
-------------------------------

"dir" must be a string which owned by a directory. You may run with one
arg directly:
------------------
(scandir "mmr")
==>("." ".." "a" "b" "c" "dd" "dir1" "dir2")
------------------

As you see, it returned a list contained all contents of directory
"mmr". So, you can do whatever you like with a list.

And you may use filter:
------------------------
(scandir "mmr"
   (lambda (fname)
       (if (string-contains fname "dir")
             #f
             #t)))
==> ("." ".." "a" "b" "c" "dd")
------------------------
It's easy to filter the file you want. Return #f will skip current file,
#t will add it to the result list.

And the third arg "sort" has only two possibility: 
'asort stands for alphasort
or
'vsort stands for versionsort
Call "scandir" with "sort" unspecified you'll get 'asort in default. If
you can't understand that, you may checkout the C version manual: 
----------------
man 3 scandir
----------------

Besides, there's a clumsy documentation in the REPL:
-------------
,d scandir
-------------

PS: There must be some bugs in this implementation, like portable
problem. I believe I didn't add some critical macro in it. But I think
there're lots of people know more than me in the mail-list. So I submit
it, and waiting for any advices or patches of patch. :-)
Anyway, I've tested it. It's function complete. Use it as you like!

My patch will be submitted in the next mail, I found some guy(like me)
doesn't like to open an attachment of obscure origin. ;-)


-- 
GNU Powered it
GPL Protected it
GOD Blessed it

HFG - NalaGinrut

--hacker key--
v4sw7CUSMhw6ln6pr8OSFck4ma9u8MLSOFw3WDXGm7g/l8Li6e7t4TNGSb8AGORTDLMen6g6RASZOGCHPa28s1MIr4p-x hackerkey.com
---end key---

Re: [PATCH] Add "scandir" procedure

[Nala Ginrut]

From a79c0db1ef1a932443e41ba6185b0d8c1a5774ed Mon Sep 17 00:00:00 2001
From: Nala Ginrut <NalaGinrut@gmail.com>
Date: Sun, 28 Aug 2011 03:19:15 +0800
Subject: [PATCH] Add scandir procedure

---
 libguile/_scm.h    |    3 ++
 libguile/filesys.c |   98 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 101 insertions(+), 0 deletions(-)

diff --git a/libguile/_scm.h b/libguile/_scm.h
index 48fb2cc..cd3f82c 100644
--- a/libguile/_scm.h
+++ b/libguile/_scm.h
@@ -166,6 +166,9 @@
 #define off_t_or_off64_t                CHOOSE_LARGEFILE(off_t,off64_t)
 #define open_or_open64                  CHOOSE_LARGEFILE(open,open64)
 #define readdir_or_readdir64            CHOOSE_LARGEFILE(readdir,readdir64)
+#define scandir_or_scandir64		CHOOSE_LARGEFILE(scandir,scandir64)
+#define alphasort_or_alphasort64	CHOOSE_LARGEFILE(alphasort,alphasort64)
+#define versionsort_or_versionsort64	CHOOSE_LARGEFILE(versionsort,versionsort64)
 #if SCM_HAVE_READDIR64_R == 1
 # define readdir_r_or_readdir64_r       CHOOSE_LARGEFILE(readdir_r,readdir64_r)
 #else
diff --git a/libguile/filesys.c b/libguile/filesys.c
index f600328..d4bf2f4 100644
--- a/libguile/filesys.c
+++ b/libguile/filesys.c
@@ -1670,6 +1670,104 @@ SCM_DEFINE (scm_opendir, "opendir", 1, 0, 0,
 }
 #undef FUNC_NAME
 
+SCM_DEFINE (scm_scandir, "scandir", 1, 2, 0,
+	    (SCM dir, SCM filter, SCM sort),
+	    "Return a list which contains all files and directories' name of the "
+	    "@var{dir}. The @var{sort} is the sort method of the result.\n"
+	    "The second arg @var{filter} is a proc with 2 args and return #t "
+	    "for keeping this result, and vice versa.\n"
+	    "If @var{sort} is unspecified, @code{alplasort} would be default.\n"
+	    "The optional @var{sort} must be symbol and could be:"
+	    "@defvar{asort} ==> for alphasort\n"
+	    "@defvar{vsort} ==> for versionsort")
+#define FUNC_NAME s_scm_scandir
+{
+  struct dirent_or_dirent64 **rdent;
+  int sort_type = 0;
+  int has_filter = 0;
+  int n = 0 ,i = 0;
+  SCM flag;
+  SCM ret = SCM_EOL;
+  SCM *prev;
+  SCM str;
+
+  SCM_VALIDATE_STRING (1, dir);
+
+  if (!SCM_UNBNDP (filter))
+    {
+      SCM_ASSERT (scm_is_true (scm_procedure_p (filter)),
+		  filter ,SCM_ARG2 ,FUNC_NAME);
+      has_filter = 1;
+    }
+  
+  if (!SCM_UNBNDP (sort))
+    {
+      SCM_ASSERT (scm_symbol_p(sort) ,sort ,SCM_ARG3 ,FUNC_NAME);
+      
+      if (scm_is_true (scm_eq_p (sort, scm_from_latin1_symbol ("asort"))))
+	sort_type = 0;	
+      else if (scm_is_true (scm_eq_p (sort, scm_from_latin1_symbol ("vsort"))))
+	sort_type = 1;
+      else
+	scm_error (scm_from_latin1_symbol ("invalid-sort-type"),
+		   "scandir", "Scandir got an invalid sort type: ~S",
+		   scm_list_1 (sort), scm_list_1 (sort));
+    }
+
+  scm_dynwind_begin (0);
+  errno = 0;
+
+  switch (sort_type)
+    {	
+    case 0: 
+      SCM_SYSCALL(n = scandir_or_scandir64 (scm_to_locale_string (dir), 
+					    &rdent, NULL, alphasort_or_alphasort64));
+      break;
+    case 1:
+      SCM_SYSCALL(n = scandir_or_scandir64 (scm_to_locale_string (dir), 
+					    &rdent, NULL, versionsort_or_versionsort64))	;
+      break;
+    default:
+      SCM_SYSERROR;
+    }
+
+  if (has_filter)
+    {
+      for (prev = &ret;i<n;i++)
+	{
+	  str = rdent[i]? scm_from_locale_stringn (rdent[i]->d_name ,NAMLEN (rdent[i])) : SCM_EOF_VAL;
+	  flag = scm_call_1 (filter ,str);
+	  free (rdent[i]);
+
+	  if (scm_is_true (flag))
+	    {
+	      *prev = scm_cons (str ,SCM_EOL);
+	      prev = SCM_CDRLOC (*prev);
+	    }
+	  
+	}
+    }
+  else
+    {
+      for (prev = &ret;i<n;i++)
+	{
+	  str = rdent[i]? scm_from_locale_stringn (rdent[i]->d_name ,NAMLEN (rdent[i])) : SCM_EOF_VAL;
+	  *prev = scm_cons (str ,SCM_EOL);
+	  prev = SCM_CDRLOC (*prev);
+	  free (rdent[i]);
+	}
+    }
+
+  if (errno != 0)
+    SCM_SYSERROR;
+
+  scm_dynwind_end ();
+
+  free (rdent);
+  
+  return ret;
+}
+#undef FUNC_NAME
 
 /* FIXME: The glibc manual has a portability note that readdir_r may not
    null-terminate its return string.  The circumstances outlined for this
-- 
1.7.0.4

Re: [PATCH] Add "scandir" procedure

[Ludovic Courtès]

Hi!

nalaginrut <nalaginrut@gmail.com> skribis:

> I found there isn't "scandir" in current Guile. And we may use "ftw" to
> instead. I guess "ftw" traverse all sub-directoies. Yes, we may use nftw
> to filter the level we don't need, but "ftw" seems always traverse all
> sub-directories. If my guess is correct, I believe it's too slow for
> someone, for a instance, me. :-)

I was actually planning to push ‘file-system-fold’, a functional
alternative to ‘ftw’, which would be along the lines of this (from
<http://git.savannah.gnu.org/cgit/libchop.git/tree/utils/chop-backup#n46>):

  (define (file-system-fold enter? leaf down up skip init file-name)
    "Traverse the directory at FILE-NAME, recursively.  Enter sub-directories
  only when (ENTER? PATH STAT RESULT) returns true.  When a sub-directory is
  entered, call (DOWN PATH STAT RESULT), where PATH is the path of the
  sub-directory and STAT the result of (lstat PATH); when it is left, call (UP
  PATH STAT RESULT).  For each file in a directory, call (LEAF PATH STAT
  RESULT).  Return the result of these successive applications.  When ENTER?
  returns no, call (SKIP PATH STAT RESULT)."

    ...)

I think it would allow you to do what you want, right?  Like this:

  (define (directory-contents file-name)
    "Return the list of regular files and directories in the directory
  at FILE-NAME as a list of name/stat pairs."
    (file-system-fold (lambda (path stat result)         ; enter?
                        (eq? path file-name))
                      alist-cons                         ; leaf
                      (lambda (path stat result) result) ; down
                      (lambda (path stat result) result) ; up
                      alist-cons                         ; skip
                      '()
                      file-name))

What do you think?

Thanks,
Ludo’.

Re: [PATCH] Add "scandir" procedure

[Nala Ginrut]

[-- Attachment #1: Type: text/plain, Size: 562 bytes --]

hi, ludo!
Sorry for late reply. Because I thought web gmail would tag this thread as
high priority, but it didn't. So I missed the newest message. :-(

Yes, that's what I want.
And I have a question between C and guile, should we add more useful
functions in scheme or just wrap the POSIX C functions?
I chose to wrap the POSIX C function because "scandir" is a common function.
Maybe some guys who're not familiar with Guile would find "scandir" first as
he did under other languages.
But file-system-fold seems more flexible and portable. How about its speed?

[-- Attachment #2: Type: text/html, Size: 628 bytes --]

Re: [PATCH] Add "scandir" procedure

[Ludovic Courtès]

Hi,

Nala Ginrut <nalaginrut@gmail.com> skribis:

> Yes, that's what I want.

Great.  Please remind me to push it in 2.0 if I don’t do it in a timely
fashion.

> And I have a question between C and guile, should we add more useful
> functions in scheme or just wrap the POSIX C functions?

I guess it’s largely a matter of taste, but I’d prefer the former,
(unless it’s not doable from Scheme), because Scheme is more hackable
and because writing things in Scheme allows you to come up with more
idiomatic interfaces.

> I chose to wrap the POSIX C function because "scandir" is a common
> function.  Maybe some guys who're not familiar with Guile would find
> "scandir" first as he did under other languages.  But file-system-fold
> seems more flexible and portable. How about its speed?

I don’t have any figures.  However, when traversing a file system
hierarchy, the bottleneck is often I/O, so I wouldn’t worry about it.

Technically I think ‘file-system-fold’ does the minimum number of system
calls that it really needs to do.

Thanks,
Ludo’.

Re: [PATCH] Add "scandir" procedure

[Nala Ginrut]

[-- Attachment #1: Type: text/plain, Size: 427 bytes --]

Well, I think both of them should be added into Guile. :-)
I confess 'file-system-fold' is more hackable and interesting, but it
contains 6 args.
I don't mean 6 args is  improper. But maybe some guys just want an easy
thing, and maybe Guile newbies need a easy start with 'scandir'.
'scandir' is a POSIX function, so many people will think it exists in Guile.
We have 'opendir' and 'readdir', but no 'scandir', that's strange.

[-- Attachment #2: Type: text/html, Size: 519 bytes --]

Re: [PATCH] Add "scandir" procedure

[Ludovic Courtès]

Hi,

Nala Ginrut <nalaginrut@gmail.com> skribis:

> I don't mean 6 args is  improper. But maybe some guys just want an easy
> thing, and maybe Guile newbies need a easy start with 'scandir'.

Oh, I just meant to say you could build ‘scandir’ (and others) atop
‘file-system-fold’.

The scandir(3) C function is really meant to be a higher-order function
but its interface is awkward and mixes different concerns (traversal and
filtering.)  Using something akin to the ‘directory-contents’ procedure
I posted earlier, one could then use SRFI-1 to do actual filtering.

> 'scandir' is a POSIX function, so many people will think it exists in Guile.
> We have 'opendir' and 'readdir', but no 'scandir', that's strange.

Yes, you’re right.  However, I feel that we can come up with something
both simpler and more expressive, as sketched above.

WDYT?

Thanks,
Ludo'.

Re: [PATCH] Add "scandir" procedure

[Nala Ginrut]

[-- Attachment #1: Type: text/plain, Size: 1618 bytes --]

On Fri, Sep 2, 2011 at 5:01 AM, Ludovic Courtès <ludo@gnu.org> wrote:

> Hi,
>
> Nala Ginrut <nalaginrut@gmail.com> skribis:
>
> > I don't mean 6 args is  improper. But maybe some guys just want an easy
> > thing, and maybe Guile newbies need a easy start with 'scandir'.
>
> Oh, I just meant to say you could build ‘scandir’ (and others) atop
> ‘file-system-fold’.
>
> The scandir(3) C function is really meant to be a higher-order function
> but its interface is awkward and mixes different concerns (traversal and
> filtering.)  Using something akin to the ‘directory-contents’ procedure
> I posted earlier, one could then use SRFI-1 to do actual filtering.
>
> > 'scandir' is a POSIX function, so many people will think it exists in
> Guile.
> > We have 'opendir' and 'readdir', but no 'scandir', that's strange.
>
> Yes, you’re right.  However, I feel that we can come up with something
> both simpler and more expressive, as sketched above.
>

yeah~I realized that the idea about a bunch of  gorgeous high-order
functions around one base function 'file-system-fold' is more creative and
flexible. It becomes happier to add new features into Guile than wrap a C
function. Anyway, C is more painful. ;-)
So, have you already implemented others like 'directory-content' instead of
'scandir' ? Or should we submit some patches for it?
What about the sort method? And I hope it won't dismiss "." and ".." in the
result.
Let me retell my need, I need the function returns a sorted list which
contained "." and "..". Is it easy to implement it with 'file-system-fold'?

[-- Attachment #2: Type: text/html, Size: 2504 bytes --]

Re: [PATCH] Add "scandir" procedure

[Nala Ginrut]

[-- Attachment #1: Type: text/plain, Size: 694 bytes --]

On Fri, Sep 2, 2011 at 5:01 PM, Ludovic Courtès <ludo@gnu.org> wrote:

> Hi,
>
> Nala Ginrut <nalaginrut@gmail.com> skribis:
>
> > Let me retell my need, I need the function returns a sorted list which
> > contained "." and "..". Is it easy to implement it with
> 'file-system-fold'?
>
> Currently ‘file-system-fold’ dismisses dot and dot-dot; why do you need
> them?
>
> Ludo’.
>


Maybe "." and ".." is unnecessary , I just want to list a directory in HTML
to client. Anyway, I can generate a "parent directory" to point to last
level dir.
And how about the sort? I need a sorted result. 'scandir' do it
automatically, will 'file-system-fold' provide a easy way?

[-- Attachment #2: Type: text/html, Size: 1151 bytes --]

Re: [PATCH] Add "scandir" procedure

[Ludovic Courtès]

Nala Ginrut <nalaginrut@gmail.com> skribis:

> And how about the sort? I need a sorted result. 'scandir' do it
> automatically, will 'file-system-fold' provide a easy way?

You can always use ‘sort’ & co.

Ludo’.

Re: [PATCH] Add "scandir" procedure

[Andy Wingo]

On Tue 30 Aug 2011 18:06, ludo@gnu.org (Ludovic Courtès) writes:

> I was actually planning to push ‘file-system-fold’, a functional
> alternative to ‘ftw’, which would be along the lines of this (from
> <http://git.savannah.gnu.org/cgit/libchop.git/tree/utils/chop-backup#n46>):
>
>   (define (file-system-fold enter? leaf down up skip init file-name)
>     "Traverse the directory at FILE-NAME, recursively.  Enter sub-directories
>   only when (ENTER? PATH STAT RESULT) returns true.  When a sub-directory is
>   entered, call (DOWN PATH STAT RESULT), where PATH is the path of the
>   sub-directory and STAT the result of (lstat PATH); when it is left, call (UP
>   PATH STAT RESULT).  For each file in a directory, call (LEAF PATH STAT
>   RESULT).  Return the result of these successive applications.  When ENTER?
>   returns no, call (SKIP PATH STAT RESULT)."
>
>     ...)

I see that you haven't pushed this yet.  Want to do so?

I would be happy with a `scandir' implementation on top of this
interface.  We should find a way to push most POSIX things out to a
module, though!

Andy
-- 
http://wingolog.org/

Re: [PATCH] Add "scandir" procedure

[Ludovic Courtès]

Hi,

Andy Wingo <wingo@pobox.com> skribis:

> On Tue 30 Aug 2011 18:06, ludo@gnu.org (Ludovic Courtès) writes:
>
>> I was actually planning to push ‘file-system-fold’, a functional
>> alternative to ‘ftw’, which would be along the lines of this (from
>> <http://git.savannah.gnu.org/cgit/libchop.git/tree/utils/chop-backup#n46>):
>>
>>   (define (file-system-fold enter? leaf down up skip init file-name)
>>     "Traverse the directory at FILE-NAME, recursively.  Enter sub-directories
>>   only when (ENTER? PATH STAT RESULT) returns true.  When a sub-directory is
>>   entered, call (DOWN PATH STAT RESULT), where PATH is the path of the
>>   sub-directory and STAT the result of (lstat PATH); when it is left, call (UP
>>   PATH STAT RESULT).  For each file in a directory, call (LEAF PATH STAT
>>   RESULT).  Return the result of these successive applications.  When ENTER?
>>   returns no, call (SKIP PATH STAT RESULT)."
>>
>>     ...)
>
> I see that you haven't pushed this yet.  Want to do so?

Yes, I’m looking into this now.

I was planning to make it part of (ice-9 ftw); WDYT?

If there’s anything else you’d like to discuss about the API or
implementation, please let me know.

> I would be happy with a `scandir' implementation on top of this
> interface.

Yes.  The mixture of concerns in that function is not brilliant, but
OTOH it’s probably good to have a function people are familiar with.

Thanks,
Ludo’.

Re: [PATCH] Add "scandir" procedure

[Andy Wingo]

Heya,

On Sat 10 Dec 2011 15:52, ludo@gnu.org (Ludovic Courtès) writes:

> Andy Wingo <wingo@pobox.com> skribis:
>
>> On Tue 30 Aug 2011 18:06, ludo@gnu.org (Ludovic Courtès) writes:
>>
>>> I was actually planning to push ‘file-system-fold’, a functional
>>> alternative to ‘ftw’, which would be along the lines of this (from
>>> <http://git.savannah.gnu.org/cgit/libchop.git/tree/utils/chop-backup#n46>):
>>>
>>>   (define (file-system-fold enter? leaf down up skip init file-name)
>>>     "Traverse the directory at FILE-NAME, recursively.  Enter sub-directories
>>>   only when (ENTER? PATH STAT RESULT) returns true.  When a sub-directory is
>>>   entered, call (DOWN PATH STAT RESULT), where PATH is the path of the
>>>   sub-directory and STAT the result of (lstat PATH); when it is left, call (UP
>>>   PATH STAT RESULT).  For each file in a directory, call (LEAF PATH STAT
>>>   RESULT).  Return the result of these successive applications.  When ENTER?
>>>   returns no, call (SKIP PATH STAT RESULT)."
>>>
>>>     ...)
>>
>> I see that you haven't pushed this yet.  Want to do so?
>
> Yes, I’m looking into this now.
>
> I was planning to make it part of (ice-9 ftw); WDYT?

Sounds fine to me.  I didn't take a close look at the implementation,
but the idea of the function sounded sane to me.

>> I would be happy with a `scandir' implementation on top of this
>> interface.
>
> Yes.  The mixture of concerns in that function is not brilliant, but
> OTOH it’s probably good to have a function people are familiar with.

Yeah, I have the same thoughts.  OTOH, if file-system-fold goes into
(ice-9 ftw), where would a `scandir' go?  There also?  I guess that's
OK.

Just thinking out loud :)

Happy hacking,

Andy
-- 
http://wingolog.org/

Re: [PATCH] Add "scandir" procedure

[Ludovic Courtès]

Hello!


Andy Wingo <wingo@pobox.com> skribis:

> On Sat 10 Dec 2011 15:52, ludo@gnu.org (Ludovic Courtès) writes:

[...]

>> Yes.  The mixture of concerns in that function is not brilliant, but
>> OTOH it’s probably good to have a function people are familiar with.
>
> Yeah, I have the same thoughts.  OTOH, if file-system-fold goes into
> (ice-9 ftw), where would a `scandir' go?  There also?

Yes.

> I guess that's OK.

Someday maybe we’ll have a (posix) module, but until then that’s fine.
:-)

Thanks,
Ludo’.

Re: [PATCH] Add "scandir" procedure

[Ludovic Courtès]

Hello!

I finally committed ‘file-system-fold’ (and its new friend
‘file-system-tree’), but I have yet to write ‘scandir’ (unless someone
else does ;-)).

Feedback welcome!

Thanks,
Ludo’.

Re: [PATCH] Add "scandir" procedure

[Ludovic Courtès]

Hi Nala!

I just added a ‘scandir’ implementation in terms of ‘file-system-fold’:

  http://git.sv.gnu.org/cgit/guile.git/commit/?h=stable-2.0&id=1629429d63170b1c5a19e72e838cab331c7eba8b

Please let me know what you think of it!

Thanks,
Ludo’.

Re: [PATCH] Add "scandir" procedure

[Nala Ginrut]

[-- Attachment #1: Type: text/plain, Size: 1415 bytes --]

hi Ludo, nice job!
And I tested it, it works.
But I did find some problems. Now there're three:
1. I think file-system-fold based scandir tried to traverse the whole
directories include sub-directories. It's rather slow for a deep one if I
just
    want a files list under 0 level directory tree;
2. New scandir will crash while encounters a Chinese file name. This will
be eliminated by using (setlocale LC_ALL "zh_CN.UTF-8").
    I think it's the same problem we faced in another thread. There's
something locale problem in Guile. Of course, we have a temporary solution
in recent commit;
3. It returns weird result. E.g (scandir "mmr")
==>  ("." "." "." ".." ".." ".." "aa.c" "exclude" "ml" "myecl")
There're two sub-directories, so two more dot & dot-dot added.

Anyway, I think new scandir's cool. Though it's little slow than my C wrap
version, it's more hackable.
Maybe I'll keep C wrap scandir in my project for a time. But it's
reasonable to change it to the new one in the future but depends on it's
efficiency.



On Mon, Dec 19, 2011 at 4:30 AM, Ludovic Courtès <ludo@gnu.org> wrote:

> Hi Nala!
>
> I just added a ‘scandir’ implementation in terms of ‘file-system-fold’:
>
>
> http://git.sv.gnu.org/cgit/guile.git/commit/?h=stable-2.0&id=1629429d63170b1c5a19e72e838cab331c7eba8b
>
> Please let me know what you think of it!
>
> Thanks,
> Ludo’.
>
>
>

[-- Attachment #2: Type: text/html, Size: 2131 bytes --]

Re: [PATCH] Add "scandir" procedure

[Ludovic Courtès]

Hi Nala!

Thanks for testing!

Nala Ginrut <nalaginrut@gmail.com> skribis:

> 1. I think file-system-fold based scandir tried to traverse the whole
> directories include sub-directories. It's rather slow for a deep one if I
> just
>     want a files list under 0 level directory tree;

The code had initially approximately 1 typo per line, and I think I’ve
fixed most of them now.  ;-)

So ‘scandir’ does not enter sub-directories.  If it does, that’s another
bug.  :-)

> 2. New scandir will crash while encounters a Chinese file name. This will
> be eliminated by using (setlocale LC_ALL "zh_CN.UTF-8").
>     I think it's the same problem we faced in another thread. There's
> something locale problem in Guile. Of course, we have a temporary solution
> in recent commit;

Yes, Guile views file names as strings and decodes them from the current
locale encoding.  So if there are file names encoded differently, then
scm_from_locale_string, called by ‘readdir’, throws a decoding-error.

That’s unfortunate, I’m not sure what to do.  I think GLib/GIO issues a
warning in such cases, while still being able to handle the file.  We
could imagine ‘readdir’ returning a raw bytevector when decoding fails,
and ‘open-file’ & co. could accept it as input.  But that’s really ugly.

I think Mark had some ideas about it, which would be worth checking.

> 3. It returns weird result. E.g (scandir "mmr")
> ==>  ("." "." "." ".." ".." ".." "aa.c" "exclude" "ml" "myecl")

One of the typos that got fixed, hopefully.  :-)

Can you check again?

> Anyway, I think new scandir's cool. Though it's little slow than my C wrap
> version

Because it uses ‘file-system-fold’, it does one ‘stat’ call for each
file, which the C version doesn’t do.  That should be the only
efficiency difference.

Maybe we could change our ‘scandir’ to return a list of file name/stat
pairs since we have the info anyway?

Thanks,
Ludo’.

Re: [PATCH] Add "scandir" procedure

[Nala Ginrut]

[-- Attachment #1: Type: text/plain, Size: 2420 bytes --]

On Tue, Dec 20, 2011 at 5:38 AM, Ludovic Courtès <ludo@gnu.org> wrote:

> Hi Nala!
>
> Thanks for testing!
>
> Nala Ginrut <nalaginrut@gmail.com> skribis:
>
> > 1. I think file-system-fold based scandir tried to traverse the whole
> > directories include sub-directories. It's rather slow for a deep one if I
> > just
> >     want a files list under 0 level directory tree;
>
> The code had initially approximately 1 typo per line, and I think I’ve
> fixed most of them now.  ;-)
>
> So ‘scandir’ does not enter sub-directories.  If it does, that’s another
> bug.  :-)
>
> > 2. New scandir will crash while encounters a Chinese file name. This will
> > be eliminated by using (setlocale LC_ALL "zh_CN.UTF-8").
> >     I think it's the same problem we faced in another thread. There's
> > something locale problem in Guile. Of course, we have a temporary
> solution
> > in recent commit;
>
> Yes, Guile views file names as strings and decodes them from the current
> locale encoding.  So if there are file names encoded differently, then
> scm_from_locale_string, called by ‘readdir’, throws a decoding-error.
>
> That’s unfortunate, I’m not sure what to do.  I think GLib/GIO issues a
> warning in such cases, while still being able to handle the file.  We
> could imagine ‘readdir’ returning a raw bytevector when decoding fails,
> and ‘open-file’ & co. could accept it as input.  But that’s really ugly.
>
> I think Mark had some ideas about it, which would be worth checking.
>
> > 3. It returns weird result. E.g (scandir "mmr")
> > ==>  ("." "." "." ".." ".." ".." "aa.c" "exclude" "ml" "myecl")
>
> One of the typos that got fixed, hopefully.  :-)
>
> Can you check again?
>
> > Anyway, I think new scandir's cool. Though it's little slow than my C
> wrap
> > version
>
> Because it uses ‘file-system-fold’, it does one ‘stat’ call for each
> file, which the C version doesn’t do.  That should be the only
> efficiency difference.
>
> Maybe we could change our ‘scandir’ to return a list of file name/stat
> pairs since we have the info anyway?
>
>
Well~it's a good idea I never thought about. And I've tested new commit,
it's wonderful.
Maybe you should consider this name/stat, now the new scandir is almost 4
times slower than C wrapper version.
Anyway, name/stat list would be a smart and easy solution. ;-)

[-- Attachment #2: Type: text/html, Size: 3144 bytes --]

Re: [PATCH] Add "scandir" procedure

[Nala Ginrut]

[-- Attachment #1: Type: text/plain, Size: 2747 bytes --]

Oh, I forget to tell that C wrapper version doesn't have locale problem.
And according to the commit log, seems we may expect it completely fixed in
2.0.4?

On Tue, Dec 20, 2011 at 11:23 AM, Nala Ginrut <nalaginrut@gmail.com> wrote:

>
>
> On Tue, Dec 20, 2011 at 5:38 AM, Ludovic Courtès <ludo@gnu.org> wrote:
>
>> Hi Nala!
>>
>> Thanks for testing!
>>
>> Nala Ginrut <nalaginrut@gmail.com> skribis:
>>
>> > 1. I think file-system-fold based scandir tried to traverse the whole
>> > directories include sub-directories. It's rather slow for a deep one if
>> I
>> > just
>> >     want a files list under 0 level directory tree;
>>
>> The code had initially approximately 1 typo per line, and I think I’ve
>> fixed most of them now.  ;-)
>>
>> So ‘scandir’ does not enter sub-directories.  If it does, that’s another
>> bug.  :-)
>>
>> > 2. New scandir will crash while encounters a Chinese file name. This
>> will
>> > be eliminated by using (setlocale LC_ALL "zh_CN.UTF-8").
>> >     I think it's the same problem we faced in another thread. There's
>> > something locale problem in Guile. Of course, we have a temporary
>> solution
>> > in recent commit;
>>
>> Yes, Guile views file names as strings and decodes them from the current
>> locale encoding.  So if there are file names encoded differently, then
>> scm_from_locale_string, called by ‘readdir’, throws a decoding-error.
>>
>> That’s unfortunate, I’m not sure what to do.  I think GLib/GIO issues a
>> warning in such cases, while still being able to handle the file.  We
>> could imagine ‘readdir’ returning a raw bytevector when decoding fails,
>> and ‘open-file’ & co. could accept it as input.  But that’s really ugly.
>>
>> I think Mark had some ideas about it, which would be worth checking.
>>
>> > 3. It returns weird result. E.g (scandir "mmr")
>> > ==>  ("." "." "." ".." ".." ".." "aa.c" "exclude" "ml" "myecl")
>>
>> One of the typos that got fixed, hopefully.  :-)
>>
>> Can you check again?
>>
>> > Anyway, I think new scandir's cool. Though it's little slow than my C
>> wrap
>> > version
>>
>> Because it uses ‘file-system-fold’, it does one ‘stat’ call for each
>> file, which the C version doesn’t do.  That should be the only
>> efficiency difference.
>>
>> Maybe we could change our ‘scandir’ to return a list of file name/stat
>> pairs since we have the info anyway?
>>
>>
> Well~it's a good idea I never thought about. And I've tested new commit,
> it's wonderful.
> Maybe you should consider this name/stat, now the new scandir is almost 4
> times slower than C wrapper version.
> Anyway, name/stat list would be a smart and easy solution. ;-)
>
>

[-- Attachment #2: Type: text/html, Size: 3652 bytes --]

Re: [PATCH] Add "scandir" procedure

[Ludovic Courtès]

Nala Ginrut <nalaginrut@gmail.com> skribis:

> Oh, I forget to tell that C wrapper version doesn't have locale problem.

That’s because the C version, in particular alphasort/strcoll, silently
ignores locale decoding errors.

Thanks,
Ludo’.

Re: [PATCH] Add "scandir" procedure

[Ludovic Courtès]

Hello, friends of scandir!  :-)

Commit be96155b508d220efe6f419d7743cf39744ee47c adds an ‘error’
parameter to ‘file-system-fold’ so that ‘opendir’ and ‘stat’ errors can
be handled gracefully by the caller (instead of having to guess in the
‘skip’ procedure whether an error occurred.)

Comments welcome!

Ludo’.