documentation.scm close files

documentation.scm close files

[Kevin Ryde]

[-- Attachment #1: Type: text/plain, Size: 957 bytes --]

In documentation.scm I noticed file-commentary and
find-documentation-in-file don't close their files before returning.
I think it'd be good to do so explicitly,

        * documentation.scm (file-commentary, find-documentation-in-file): Use
        call-with-input-file, so as to close ports when done.

(Patch below, un-indented to make the changes clear.)

I know ports get closed when gc'ed, but I'd suggest it does the system
and the program no good to keep stuff open long after finished with.

As it stands, in fact, by artificially lowering the number of files
available an error can be induced (on my i386 debian at least),

	ulimit -n 30
	guile -s foo.scm

with foo.scm containing

	(use-modules (ice-9 documentation))
	(while #t (file-commentary "/dev/null"))
or
	(use-modules (ice-9 documentation))
	(while #t (search-documentation-files 'car "/dev/null"))

both giving

	ERROR: In procedure open-file:
	ERROR: Too many open files: "/dev/null"


[-- Attachment #2: documentation.scm.with-input.diff --]
[-- Type: text/plain, Size: 1713 bytes --]

--- documentation.scm.~1.8.~	2003-04-07 08:04:58.000000000 +1000
+++ documentation.scm	2003-05-06 13:44:39.000000000 +1000
@@ -115,8 +115,9 @@
                    default-scrub
                    (let ((v (caddr cust)))
                      (cond ((procedure? v) v)
-                           (else default-scrub)))))
-        (port (open-input-file filename)))
+                           (else default-scrub))))))
+    (call-with-input-file filename
+      (lambda (port)
     (let loop ((line (read-delimited "\n" port))
                (doc "")
                (parse-state 'before))
@@ -132,7 +133,7 @@
                       (if (and (eq? 'in new-state) (eq? 'in parse-state))
                           (string-append doc (scrub line) "\n")
                           doc)
-                      new-state)))))))
+                      new-state)))))))))
 
 \f
 
@@ -151,8 +152,9 @@
 
 (define (find-documentation-in-file name file)
   (and (file-exists? file)
-       (let ((port (open-input-file file))
-	     (name (symbol->string name)))
+       (call-with-input-file file
+	 (lambda (port)
+       (let ((name (symbol->string name)))
 	 (let ((len (string-length name)))
 	   (read-delimited entry-delimiter port) ;skip to first entry
 	   (let loop ((entry (read-delimited entry-delimiter port)))
@@ -166,7 +168,7 @@
 			 (memq (string-ref entry len) '(#\newline)))
 		    ;; cut away name tag and extra surrounding newlines
 		    (substring entry (+ len 2) (- (string-length entry) 2)))
-		   (else (loop (read-delimited entry-delimiter port)))))))))
+		   (else (loop (read-delimited entry-delimiter port)))))))))))
 
 (define (search-documentation-files name . files)
   (or-map (lambda (file)

[-- Attachment #3: Type: text/plain, Size: 142 bytes --]

_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: documentation.scm close files

[Marius Vollmer]

Kevin Ryde <user42@zip.com.au> writes:

> I know ports get closed when gc'ed, but I'd suggest it does the system
> and the program no good to keep stuff open long after finished with.

Yes, good catch.  Maybe we should even make the GC emit warnings when
it sweeps open ports.  Closing a port can raise errors and they can't
really be handled from within the GC.

-- 
GPG: D5D4E405 - 2F9B BCCC 8527 692A 04E3  331E FAF8 226A D5D4 E405


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: documentation.scm close files

[Kevin Ryde]

Marius Vollmer <mvo@zagadka.de> writes:
>
> Maybe we should even make the GC emit warnings when
> it sweeps open ports.  Closing a port can raise errors and they can't
> really be handled from within the GC.

I suppose if a program has forgotten about a port it's not really
interested in errors from it any more :-).  Might have been a read
error that made it bail out in the first place.

I'm thinking to add some words to the manual to say why it's good to
close explicitly, and the consequences of not doing so.  Perhaps in
the "File Ports" node, with cross references from fdes, pipe and
socket stuff.


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: documentation.scm close files

[Kevin Ryde]

I wrote:
>
> I'm thinking to add some words to the manual to say why it's good to
> close explicitly,

As threatened, but in the form of a revision of the "Ports" node.  The
first three paras are existing words slightly warmed-over, the rest is
new.


Ports
=====

Sequential input/output in Scheme is represented by operations on a
"port".  This chapter explains the operations that Guile provides for
working with ports.

   Ports are created by opening, for instance `open-file' for a file
(*note File Ports::).  Characters can be read from an input port and
written to an output port, or both on an input/output port.  A port can
be closed (*note Closing::) when no longer required, after which any
attempt to read or write is an error.

   The formal definition of a port is very generic: an input port is
simply "an object which can deliver characters on demand," and an
output port is "an object which can accept characters."  Because this
definition is so loose, it is easy to write functions that simulate
ports in software.  "Soft ports" and "string ports" are two interesting
and powerful examples of this technique.  (*note Soft Ports::, and
*Note String Ports::.)

   Ports are garbage collected in the usual way (*note Memory
Management::), and will be closed at that time if not already closed.
In this case any errors occuring in the close will not be reported.
Usually a program will want to explicitly close so as to be sure all
its operations have been successful.  Of course if a program has
abandoned something due to an error or other condition then closing
problems are probably not of interest.

   It is strongly recommended that file ports be explicitly closed when
no longer required.  Most systems have limits on how many files can be
open, both on a per-process and a system-wide basis.  A program that
uses many files should take care not to to upset its own or other
program's operations by hitting those limits.  The same applies to
similar system resources such as pipes and sockets.

   If program flow means it's hard to be certain when to close, then an
acceptable substitute may be to call `gc' at suitable times to pick up
unreferenced ports.  Note that when a system limit is encountered Guile
makes no attempt to automatically garbage collect and retry.  There
would be little value in doing so, since it would remove just one of
many problems for a program trying to operate near system limits all
the time.


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: documentation.scm close files

[tomas]

On Thu, May 22, 2003 at 08:40:09AM +1000, Kevin Ryde wrote:

 [...]

> As threatened, but in the form of a revision of the "Ports" node.  The
> first three paras are existing words slightly warmed-over, the rest is
> new.

I like your text very much.However...

>    It is strongly recommended that file ports be explicitly closed when
> no longer required.  Most systems have limits on how many files can be
> open, both on a per-process and a system-wide basis.  A program that
> uses many files should take care not to to upset its own or other
> program's operations by hitting those limits.  The same applies to
> similar system resources such as pipes and sockets.

it seems important to me to make even clearer that the garbage collector
is triggered when we are short on memory (something along the lines
of ``the system will trigger the garbage collector based on memory
usage, but it won't notice if it is running low on file descriptors,
so don't depend on the garbage collector to automatically close unused
files''). Of course, using your clear language :-)

(As an anecdotal note, we once witnessed one Java application where
the one resource left to the garbage collector was the number of
Oracle client licenses. There was plenty of memory, so...)

Regardsand thanks for the good work!

-- tomas


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: documentation.scm close files

[Marius Vollmer]

Kevin Ryde <user42@zip.com.au> writes:

> As threatened, but in the form of a revision of the "Ports" node.  The
> first three paras are existing words slightly warmed-over, the rest is
> new.

Cool.  What about saying something about 'with-input-from', etc.
That's probably the easiest way to ensure that ports are closed even
in the case of errors.

-- 
GPG: D5D4E405 - 2F9B BCCC 8527 692A 04E3  331E FAF8 226A D5D4 E405


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: documentation.scm close files

[Kevin Ryde]

Marius Vollmer <mvo@zagadka.de> writes:
>
> What about saying something about 'with-input-from', etc.
> That's probably the easiest way to ensure that ports are closed even
> in the case of errors.

Except that with-input-from-file and call-with-input-file don't
guarantee to close on error, do they?  :-)

Due to being let's not dynamic-wind of course, if I'm looking at the
right code in r4rs.scm.  Presumably this is deliberate, so
continuations can be captured and still use the port.


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: documentation.scm close files

[Kevin Ryde]

tomas@fabula.de writes:
>
> it seems important to me to make even clearer that the garbage collector
> is triggered when we are short on memory

Yep, good point.  New words below.

> ``the system will trigger the garbage collector based on memory
> usage, but it won't notice if it is running low on file descriptors,

I guess some sort of file descriptor threshold scheme might be cute.
Like garbage collect after every 10 or 20 files net opened (ie. opens
less explicit closes).

I suppose to be reliable such a scheme would need to be carefully
applied to all functions opening or closing fds.  Not much value if
it's not universal.



   It is strongly recommended that file ports be explicitly closed when
no longer required.  Most systems have limits on how many files can be
open, both on a per-process and a system-wide basis.  A program that
uses many files should take care not to hit those limits.  The same
applies to similar system resources such as pipes and sockets.

   Note that automatic garbage collection is triggered only by memory
consumption, not by file or other resource usage, so a program cannot
rely on that to keep it away from system limits.  There would be little
value in Guile attempting this, since the consequences of even
approaching limits tend to be serious both for a program's own
operation (eg. libraries accessing files), and for other processes (eg.
jobs failing intermittently).

   An explicit call to `gc' can of course be relied on to pick up
unreferenced ports.  If program flow makes it hard to be certain when
to close then this may be an acceptable way to control resource usage.


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: documentation.scm close files

[tomas]

On Sat, May 24, 2003 at 11:50:24AM +1000, Kevin Ryde wrote:
> tomas@fabula.de writes:
> >
> > it seems important to me to make even clearer that the garbage collector
> > is triggered when we are short on memory
> 
> Yep, good point.  New words below.

Wow. Thanks.

> I guess some sort of file descriptor threshold scheme might be cute.
> Like garbage collect after every 10 or 20 files net opened (ie. opens
> less explicit closes).
> 
> I suppose to be reliable such a scheme would need to be carefully
> applied to all functions opening or closing fds.  Not much value if
> it's not universal.

But file descriptors are just one of many possible resources
which might run low and which might be recovered from gc.
(that's what I was trying to hint at with this strange Oracle
anecdote). Would it be worth implementing a generalized resource
watcher framework? Are fds important enough to implement a
special case for them? May be...

Regards
-- tomas


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: documentation.scm close files

[Marius Vollmer]

Kevin Ryde <user42@zip.com.au> writes:

> Marius Vollmer <mvo@zagadka.de> writes:
> >
> > What about saying something about 'with-input-from', etc.
> > That's probably the easiest way to ensure that ports are closed even
> > in the case of errors.
> 
> Except that with-input-from-file and call-with-input-file don't
> guarantee to close on error, do they?  :-)

Yep, you are right!  I was talking too fast.

> Due to being let's not dynamic-wind of course, if I'm looking at the
> right code in r4rs.scm.  Presumably this is deliberate, so
> continuations can be captured and still use the port.

Yes.  I really start to think that call/cc does more harm than good.
It is perfect for self contained illustrations of computer science
stuff, but can cause a lot of fuzziness in larger, multi module
systems.

-- 
GPG: D5D4E405 - 2F9B BCCC 8527 692A 04E3  331E FAF8 226A D5D4 E405


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: documentation.scm close files

[Kevin Ryde]

I  toned down the words about the consequences of nearing fd limits,
and applied this change.

I was tempted to put the bits about file resources in the file ports
node, but decided it followed on nicely enough from the closing and
garbage collection in the ports node.


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

doco with-input-from-file, with-output-to-file (was: documentation.scm close files)

[Kevin Ryde]

Marius Vollmer <mvo@zagadka.de> writes:
>
> What about saying something about 'with-input-from', etc.

Speaking of these, I wonder if it could be a documented feature that
they use dynamic-wind to restore the current-input-port.  Currently
the docs follow the r5rs and don't say.

The behaviour in respect of the port itself would still be
unspecified, but getting current-input-port saved and restored nicely
seems a pretty sensible thing.  (Unless there's any prospect of an
r6rs going in a different direction. :-)


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel