LISP Questions - random, random elements and memory management

LISP Questions - random, random elements and memory management

[Jeff Clough]

Hello,

I've started making real forays into Emacs Lisp over the last few
days, making good progress (I think I'm finally starting to "get it"),
and I have some questions.

random - Is there some way I can tell that (random t) has already been
done?  I know I can detect if my own code has done this via setting
setting J Random Variable, but I'm looking for something more robust.

get-random-element - Is there a function somewhere in Emacs that,
given a list (or sequence in general, perhaps), returns a random
element of the list?  Something like this?

(defun get-random-element (list)
  "Returns a random element of LIST."
  (if (not (and (list) (listp list)))
      (nth (random (1- (1+ (length list)))) list)
    (error "Argument to get-random-element not a list or the list is empty")))

Obviously I have this functionality now, but I'm looking for something
that Joe Average is already going to have in his Emacs.  (PS, if the
code above sucks, please tell me why.)

Lastly, I have a function that takes two lists, pulls a random element
from each and concats the elements to form a string (a random name
generator).  It works just swell with the lists as defconsts, but one
of the lists is quite large (88,000 elements today) and burns about a
megabyte of RAM.  It takes above half a second to evaluate the
defconst, but then using the list is zippy (even nth'ing near the end
of the list takes no time).

What is the done thing in Emacs Lisp to keep this data out of memory
until it's needed that also lets the function stay zippy?  Bonus
points if I can keep the data and function bundled up in the same .el
file.

I've taken a look at autoload, provide and require, but I'm not
certain I understand them fully or how they would be used to solve
this problem.

Any help would be most appreciated!

Jeff

Re: LISP Questions - random, random elements and memory management

[Jeff Clough]

From: Jeff Clough <jeff@chaosphere.com>
Date: Wed, 18 Nov 2009 07:54:28 -0500 (EST)

> (defun get-random-element (list)
>   "Returns a random element of LIST."
>   (if (not (and (list) (listp list)))
>       (nth (random (1- (1+ (length list)))) list)
>     (error "Argument to get-random-element not a list or the list is empty")))
> 

Yay for not being awake!  The above should be this:

(defun get-random-element (list)
  "Returns a random element of LIST."
  (if (and list (listp list))
      (nth (random (1- (1+ (length list)))) list)
    (error "Argument to get-random-element not a list or the list is empty")))



Jeff

Re: LISP Questions - random, random elements and memory management

[Kevin Rodgers]

Jeff Clough wrote:
> From: Jeff Clough <jeff@chaosphere.com>
> Date: Wed, 18 Nov 2009 07:54:28 -0500 (EST)
> 
>> (defun get-random-element (list)
>>   "Returns a random element of LIST."
>>   (if (not (and (list) (listp list)))
>>       (nth (random (1- (1+ (length list)))) list)
>>     (error "Argument to get-random-element not a list or the list is empty")))
>>
> 
> Yay for not being awake!  The above should be this:
> 
> (defun get-random-element (list)
>   "Returns a random element of LIST."
>   (if (and list (listp list))
>       (nth (random (1- (1+ (length list)))) list)
>     (error "Argument to get-random-element not a list or the list is empty")))

Is (1- (1+ (length list))) not exactly the same as (length list)?

-- 
Kevin Rodgers
Denver, Colorado, USA

Re: LISP Questions - random, random elements and memory management

[Kevin Rodgers]

Jeff Clough wrote:
> Lastly, I have a function that takes two lists, pulls a random element
> from each and concats the elements to form a string (a random name
> generator).  It works just swell with the lists as defconsts, but one
> of the lists is quite large (88,000 elements today) and burns about a
> megabyte of RAM.  It takes above half a second to evaluate the
> defconst, but then using the list is zippy (even nth'ing near the end
> of the list takes no time).
> 
> What is the done thing in Emacs Lisp to keep this data out of memory
> until it's needed that also lets the function stay zippy?  Bonus
> points if I can keep the data and function bundled up in the same .el
> file.

The data *is* in memory.  The function is zippy because nth is fast, in
turn because cdr is fast (and implemented in C).  And perhaps the cons
cells for the list are allocated in adjacent memory, if it is
constructed all at once.

Of course you can put the data (defconst forms) and function (defun) in
the same .el file!

> I've taken a look at autoload, provide and require, but I'm not
> certain I understand them fully or how they would be used to solve
> this problem.

Me neither.

-- 
Kevin Rodgers
Denver, Colorado, USA

Re: LISP Questions - random, random elements and memory management

[Jeff Clough]

From: Kevin Rodgers <kevin.d.rodgers@gmail.com>
Date: Wed, 18 Nov 2009 20:03:51 -0700

> The data *is* in memory.  The function is zippy because nth is fast,
> in
> turn because cdr is fast (and implemented in C).  And perhaps the cons
> cells for the list are allocated in adjacent memory, if it is
> constructed all at once.
> 
> Of course you can put the data (defconst forms) and function (defun)
> in
> the same .el file!

Um, actually I was looking for a way that would allow me to keep the
data *out* of memory until it was needed, then toss it on the floor
when I was done, that way I'm only eating the RAM *some* of the time
instead of all the time.  With my existing scheme, the data is in
memory as soon as the .el file is loaded.

And yes, as your earlier message indicates, the relevant line would
be:

(nth (random (length list)) list)

Overall update:

In absence of reaching enlightenment with the other points, I think
I'm just going to use the above line in my name generator instead of
worrying about defining a new function, put a note in the source about
calling (random t) and not worry about the memory issue for now.

Jeff

Re: LISP Questions - random, random elements and memory management

[Barry Margolin]

In article <mailman.11040.1258631785.2239.help-gnu-emacs@gnu.org>,
 Jeff Clough <jeff@chaosphere.com> wrote:

> From: Kevin Rodgers <kevin.d.rodgers@gmail.com>
> Date: Wed, 18 Nov 2009 20:03:51 -0700
> 
> > The data *is* in memory.  The function is zippy because nth is fast,
> > in
> > turn because cdr is fast (and implemented in C).  And perhaps the cons
> > cells for the list are allocated in adjacent memory, if it is
> > constructed all at once.
> > 
> > Of course you can put the data (defconst forms) and function (defun)
> > in
> > the same .el file!
> 
> Um, actually I was looking for a way that would allow me to keep the
> data *out* of memory until it was needed, then toss it on the floor
> when I was done, that way I'm only eating the RAM *some* of the time
> instead of all the time.  With my existing scheme, the data is in
> memory as soon as the .el file is loaded.

How about putting it in a text file that you load into a temporary 
buffer when you need it?  Then kill the buffer when you're done with it.

-- 
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***

Re: LISP Questions - random, random elements and memory management

[Jeff Clough]

From: Barry Margolin <barmar@alum.mit.edu>
Date: Thu, 19 Nov 2009 08:44:24 -0500

> How about putting it in a text file that you load into a temporary 
> buffer when you need it?  Then kill the buffer when you're done with it.

I was thinking about this.  The only real issue is with figuring out a
good lifecycle for the buffer that wouldn't leave the user confused
("Hey, where did this buffer come from?") or result in the file being
loaded/unloaded frequently enough to cause delays.

At this point, after dicking around in the Emacs Lisp manual, I'm
going to keep this as a "load" in my .emacs and assume that anyone who
wants to use the code will either do the same, or set up an autoload
expression if they care about memory usage.  When I know more about
provide/require, I'll probably revisit it.

Thanks for the suggestion, though!

Jeff

Re: LISP Questions - random, random elements and memory management

[Kevin Rodgers]

Jeff Clough wrote:
> From: Barry Margolin <barmar@alum.mit.edu>
> Date: Thu, 19 Nov 2009 08:44:24 -0500
> 
>> How about putting it in a text file that you load into a temporary 
>> buffer when you need it?  Then kill the buffer when you're done with it.

Isn't the point to read the words into a list?  So assuming the file looks
like:

("word1" "word2" "word3" ... "wordN")

Then this kills the buffer automatically but keeps the list in memory:

(defvar word-list nil)

(with-temp-buffer
   (insert-file-contents-literally "wordlist.el")
   (setq word-list (read (current-buffer))))

And when he's done with the data, this should release it:

(setq word-list nil)
(garbage-collect)

> I was thinking about this.  The only real issue is with figuring out a
> good lifecycle for the buffer that wouldn't leave the user confused
> ("Hey, where did this buffer come from?") or result in the file being
> loaded/unloaded frequently enough to cause delays.

Instead of freeing the data immediately after using it, use run-at-time
to create a timer to do so some time in the future.  If in the meantime,
it needs to be used, just cancel the timer (a new timer will be created
when this use completes).

You can tell whether you need to reload the list or not by testing
whether word-list is nil.

> At this point, after dicking around in the Emacs Lisp manual, I'm
> going to keep this as a "load" in my .emacs and assume that anyone who
> wants to use the code will either do the same, or set up an autoload
> expression if they care about memory usage.  When I know more about
> provide/require, I'll probably revisit it.

You could use provide/require/unload-feature, but it would be a level
of indirection (the feature symbol) that just hides what you're really
trying to do.

-- 
Kevin Rodgers
Denver, Colorado, USA

Re: LISP Questions - random, random elements and memory management

[Barry Margolin]

In article <mailman.11114.1258689162.2239.help-gnu-emacs@gnu.org>,
 Kevin Rodgers <kevin.d.rodgers@gmail.com> wrote:

> Jeff Clough wrote:
> > From: Barry Margolin <barmar@alum.mit.edu>
> > Date: Thu, 19 Nov 2009 08:44:24 -0500
> > 
> >> How about putting it in a text file that you load into a temporary 
> >> buffer when you need it?  Then kill the buffer when you're done with it.
> 
> Isn't the point to read the words into a list?  So assuming the file looks
> like:
> 
> ("word1" "word2" "word3" ... "wordN")

I thought the point was to select a word at random.  Putting it into a 
list was just part of his initial implementation of this.

If he reads it into a buffer, instead of picking a random list element, 
he can pick a random line number.

-- 
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***

Re: LISP Questions - random, random elements and memory management

[Colin S. Miller]

Jeff,

Jeff Clough wrote:
> From: Barry Margolin <barmar@alum.mit.edu>
> Date: Thu, 19 Nov 2009 08:44:24 -0500
> 
>> How about putting it in a text file that you load into a temporary 
>> buffer when you need it?  Then kill the buffer when you're done with it.
 > I was thinking about this.  The only real issue is with figuring out a
 > good lifecycle for the buffer that wouldn't leave the user confused
 > ("Hey, where did this buffer come from?")

If a buffer's name starts with a space, then it is hidden from the user
unless they do C-x b SPC TAB  or similar.

HTH,
Colin S. Miller


-- 
Replace the obvious in my email address with the first three letters of the hostname to reply.

Re: LISP Questions - random, random elements and memory management

[Jeff Clough]

From: "Colin S. Miller" <no-spam-thank-you@csmiller.demon.co.uk>
Date: Thu, 19 Nov 2009 17:40:48 +0000

> If a buffer's name starts with a space, then it is hidden from the
> user
> unless they do C-x b SPC TAB  or similar.

Now that is interesting.  I missed that in the manual, but see it now.
The documentation seems to indicate that if that buffer is visiting a
file, it still shows up, but I assume I can get around that by using
insert-file-contents with "visit" as nil.

Definately food for thought.  Thanks.

Jeff