Using guile as an extension language for GNU make

Using guile as an extension language for GNU make

[Paul Smith]

Hi all.

I've been experimenting with using Guile as an extension language to GNU
make (optionally built-in of course).  I wonder if experts here can give
me their thoughts and opinions.

The "interesting" thing about this integration is that GNU make is
essentially a big string parser, so most of the internal representations
of things are kept as strings.  So I need to be converting from C
strings into Guile SCM objects (that's easy) and from Guile SCM objects
into strings... this is slightly more complex because make strings have
no quotes.  What I've been doing is checking if the converted object has
quotes and if so, removing them.  However I don't really know if this is
the best method or not.

What I've done so far:

      * Modified GNU make's main to be invoked from scm_boot_guile(), if
        Guile is enabled.

      * Created a new GNU make function, $(guile ...), where the
        argument is passed to Guile for expansion and the result is
        turned into a string and used as the result; the code looks like
        this:
                func_guile (char *o, char **argv, const char *funcname UNUSED)
                {
                  if (argv[0] && argv[0][0] != '\0')
                    {
                      char *str = scm_to_locale_string (scm_object_to_string (scm_c_eval_string (argv[0]),
                                                                              SCM_UNDEFINED));
                      char *s = str;
                      unsigned int l = strlen (s);
                
                      if (s[0] == '"' && s[l-1] == '"')
                        {
                          s[l-1] = '\0';
                          ++s;
                          l -= 2;
                        }
                
                      o = variable_buffer_output (o, s, l);
                      free (str);
                    }
                
                  return o;
                }

      * Created two new functions and registered them with Guile:
        (make-expand <string>) which takes a string argument and expands
        it as a make expression, so it can be something like
        (make-expand "$(VAR)") for example to get the value of the make
        variable VAR.  And (make-eval <string>) which takes a string
        argument and evaluates it as a makefile snippet; this is
        essentially the same as running (make-expand "$(eval <string>)")
        just shorter to type.  This lets you define make constructs like
        rules and variables from within the Guile interpreter.  The code
        looks like this:
                SCM
                guile_eval_wrapper (SCM obj)
                {
                  char *str = scm_to_locale_string (scm_object_to_string (obj, SCM_UNDEFINED));
                  /* We want to avoid the surrounding double-quotes if present. */
                  char *s = str;
                  unsigned int l = strlen (s);
                
                  if (s[0] == '"' && s[l-1] == '"')
                    {
                      s[l-1] = '\0';
                      ++s;
                    }
                
                  DB (DB_BASIC, (_("guile: Evaluating '%s'\n"), s));
                  eval_buffer (s);
                
                  free (str);
                  return scm_from_locale_string ("");
                }
                
                SCM
                guile_expand_wrapper (SCM obj)
                {
                  SCM ret;
                  char *str = scm_to_locale_string (scm_object_to_string (obj, SCM_UNDEFINED));
                  /* We want to avoid the surrounding double-quotes if present. */
                  char *s = str;
                  unsigned int l = strlen (s);
                
                  if (s[0] == '"' && s[l-1] == '"')
                    {
                      s[l-1] = '\0';
                      ++s;
                    }
                
                  DB (DB_BASIC, (_("guile: Expanding '%s'\n"), s));
                  s = allocated_variable_expand (s);
                  ret = scm_from_locale_string (s);
                
                  free (str);
                  free (s);
                  return ret;
                }
                
                void
                set_up_guile ()
                {
                  /* Register a subr for GNU make's eval capability.  */
                  scm_c_define_gsubr ("make-eval", 1, 0, 0, guile_eval_wrapper);
                  scm_c_define_gsubr ("make-expand", 1, 0, 0, guile_expand_wrapper);
                }

Any thoughts, suggestions, improvements, or consideration you all have,
as Guile experts, would be very welcome.

Cheers!

-- 
-------------------------------------------------------------------------------
 Paul D. Smith <psmith@gnu.org>          Find some GNU make tips at:
 http://www.gnu.org                      http://make.mad-scientist.net
 "Please remain calm...I may be mad, but I am a professional." --Mad Scientist

Re: Using guile as an extension language for GNU make

[Ludovic Courtès]

Hi Paul,

Paul Smith <psmith@gnu.org> skribis:

> I've been experimenting with using Guile as an extension language to GNU
> make (optionally built-in of course).

Sounds like great news!  :-)

> What I've done so far:
>
>       * Modified GNU make's main to be invoked from scm_boot_guile(), if
>         Guile is enabled.

Ideally, when Guile support is enabled, GNU make would be turned into a
Guile extension (a shared library and its companion Scheme module that
loads it with ‘load-extension’) that would expose make’s functionality.

The main advantage is that make could then be used by “normal” Guile
programs:

  (use-modules (make))

  (define c->o
    (make-rule (make-target "foo.o")
               (make-prerequisites '("foo.c" "foo.h"))
               (list (make-expand "$(CC)") "-c -Wall"
                     (make-expand "$^") " -o "
                     (make-expand "$@"))))

  (eval-make-rule c->o)

Or:

  (use-modules (make))

  (define mf (parse-makefile "/foo/GNUmakefile"))
  (format #t "the targets are: ~a~%" (makefile-targets mf))

  ;; Imagine code that extracts the complete DAG, computes the critical
  ;; path length, and schedules tasks...

The ‘make’ executable would be a Guile script like:

  (use-modules (make))

  (apply run-make (command-line))

This is more intrusive than embedding libguile in make, but it’s also
more fruitful.  On the general patterns of embedding vs. extending, see
the excellent <http://www.twistedmatrix.com/users/glyph/rant/extendit.html>.

>       * Created a new GNU make function, $(guile ...), where the
>         argument is passed to Guile for expansion and the result is
>         turned into a string and used as the result; the code looks like
>         this:
>                 func_guile (char *o, char **argv, const char *funcname UNUSED)
>                 {
>                   if (argv[0] && argv[0][0] != '\0')
>                     {
>                       char *str = scm_to_locale_string (scm_object_to_string (scm_c_eval_string (argv[0]),
>                                                                               SCM_UNDEFINED));
>                       char *s = str;
>                       unsigned int l = strlen (s);
>                 
>                       if (s[0] == '"' && s[l-1] == '"')

There are two problems I can think of here:

  - string unquoting is actually more complex than this (recall that
    ‘object->string’ merely calls ‘write’):

--8<---------------cut here---------------start------------->8---
(call-with-output-string
  (lambda (p)
    (set-port-encoding! p "ISO-8859-1")
    (set-port-conversion-strategy! p 'substitute)
    (write "\"λ\" is a Greek letter" p)))

=> "\"\\\"\\u03bb\\\" is a Greek letter\""
--8<---------------cut here---------------end--------------->8---

  - ‘scm_c_eval_string’ can return any Scheme objects, some of which
    have meaningless representations as strings:

--8<---------------cut here---------------start------------->8---
(object->string (current-module))
=> "#<directory (guile-user) 2405090>"
--8<---------------cut here---------------end--------------->8---

The latter is probably the most serious question.  I think you would
really want to constrain expressions passed in $(guile ...) to return a
string, and not any other type of objects.

In that case, you could solve the first problem by using (display ...)
instead of (write ...).

>       * Created two new functions and registered them with Guile:
>         (make-expand <string>) which takes a string argument and expands
>         it as a make expression, so it can be something like
>         (make-expand "$(VAR)") for example to get the value of the make
>         variable VAR.  And (make-eval <string>) which takes a string
>         argument and evaluates it as a makefile snippet; this is
>         essentially the same as running (make-expand "$(eval <string>)")
>         just shorter to type.  This lets you define make constructs like
>         rules and variables from within the Guile interpreter.  The code
>         looks like this:

Looks good to me.

Thanks!

Ludo’.

Embedding vs. Extending (was: Re: Using guile as an extension language for GNU make)

[Paul Smith]

On Sun, 2011-09-18 at 14:10 +0200, Ludovic Courts wrote:
> Ideally, when Guile support is enabled, GNU make would be turned into
> a Guile extension (a shared library and its companion Scheme module
> that loads it with ‘load-extension’) that would expose make’s
> functionality.

I'm not sure I'm interested in going to that extreme.  GNU make has one
overarching, guiding concept: it's a critical component of a GNU-based
POSIX environment.  All POSIX-standard makefiles must work in GNU make,
and there are hundreds of thousands of already-existing makefiles, both
for POSIX and GNU versions of make.  Obviously asking people to start
writing makefiles in Scheme would not be appropriate.

A technically acceptable option would be to build GNU make in two forms:
first a standalone application that worked as now, and second a
"library" that could be linked as a Guile extension.

However, from what I've read of Guile that would be an immense amount of
work: GNU make was created over 20 years ago and has a lot of
not-completely-clean features and implementation details which rely on
it being a stand-alone program.  There's massive amounts of global
memory usage, not even a nod to threading capabilities or locking, and
features like automatically re-exec'ing itself in some situations.

Finally, while it's a cool idea I'm not sure there's a compelling need
for this.  Have lots of people wanted to be able to define make-type
rules and invoke make-like algorithms in Guile programs?  Coming from
the GNU make side, I've never heard of such a request.  Of course
sometimes the usages aren't clear before the capability exists;
nevertheless there are so many things to do to GNU make proper that I
can't justify the effort this would take, for the apparent return
involved.  If someone else were interested in it I'd be happy to work
with them on cleanups to the GNU make codebase that would enable this,
as long as they were generally appropriate.


My main purpose is to add some kind of scripting capability to GNU make
to augment the current functions capability.  So many people want me to
add new functions to GNU make, and they're very useful functions, but
I'm not interested in creating yet another complete scripting language.
I'd rather choose an existing language and allow GNU make makefiles to
take advantage of it.  Guile seems like a natural choice for many
reasons.

-- 
-------------------------------------------------------------------------------
 Paul D. Smith <psmith@gnu.org>          Find some GNU make tips at:
 http://www.gnu.org                      http://make.mad-scientist.net
 "Please remain calm...I may be mad, but I am a professional." --Mad Scientist

Re: Embedding vs. Extending

[Ludovic Courtès]

Hi!

Paul Smith <psmith@gnu.org> skribis:

> On Sun, 2011-09-18 at 14:10 +0200, Ludovic Courts wrote:
>> Ideally, when Guile support is enabled, GNU make would be turned into
>> a Guile extension (a shared library and its companion Scheme module
>> that loads it with ‘load-extension’) that would expose make’s
>> functionality.

[...]

> A technically acceptable option would be to build GNU make in two forms:
> first a standalone application that worked as now, and second a
> "library" that could be linked as a Guile extension.

Yes.

> However, from what I've read of Guile that would be an immense amount of
> work: GNU make was created over 20 years ago and has a lot of
> not-completely-clean features and implementation details which rely on
> it being a stand-alone program.  There's massive amounts of global
> memory usage, not even a nod to threading capabilities or locking, and
> features like automatically re-exec'ing itself in some situations.

Yeah, widespread use of global variables and such is likely to make a
hypothetical ‘make’ library much less useful.

> Finally, while it's a cool idea I'm not sure there's a compelling need
> for this.

Here’s an example: DMake [0] is concerned with makefile task scheduling
on distributed architectures.  To do that, it needs to know the DAG of
tasks defined in a makefile.  To achieve that, it ends up parsing the
output of ‘make -ptq’; it works, but it’s fragile (has to use the C
locale, is sensitive to formatting changes, etc.), and suboptimal.

Imagine if this could be achieved simply by having DMake directly call
make’s library functions to get the data it needs.

[0] http://dmake.ligforge.imag.fr/


Another example: many projects have make-like functionality built-in.
For instance, Java compilers (ahem...) have dependency tracking
built-in; Rubber [1] automatically infers dependencies from LaTeX source
files and runs the right actions in the right order; Guile’s
auto-compilation feature is a simple .scm → .go rule; and so on.

[1] https://launchpad.net/rubber


I could probably come up with others if you’re curious.  ;-)


> My main purpose is to add some kind of scripting capability to GNU make
> to augment the current functions capability.

And this is a worthy goal too!

Thanks,
Ludo’.

Re: Using guile as an extension language for GNU make

[Paul Smith]

On Sun, 2011-09-18 at 14:10 +0200, Ludovic Courts wrote:
> There are two problems I can think of here:
> 
>   - string unquoting is actually more complex than this (recall that
>     ‘object->string’ merely calls ‘write’):
>   - ‘scm_c_eval_string’ can return any Scheme objects, some of which
>     have meaningless representations as strings:
> 
> The latter is probably the most serious question.  I think you would
> really want to constrain expressions passed in $(guile ...) to return
> a string, and not any other type of objects.
> 
> In that case, you could solve the first problem by using (display ...)
> instead of (write ...).

There's no question that the string conversion is a hack: that's mainly
why I'm asking here :-).

I don't want to restrict it to just strings; for example I'd have to
support numbers as well (one of GNU make's main issues right now is that
there is no math capability whatsoever).  I'd like to be able to
possibly convert other things like t and nil (in GNU make's terms,
"false" is the empty string and "true" is any non-empty string).  Maybe
there are other things as well (what about symbols like 'foobar?), I
don't know.

On Sun, 2011-09-18 at 17:30 +0200, Thien-Thi Nguyen wrote:
> The double-quote stripping is kind of hacky.  I would create a port
> and ‘display’ the result of ‘scm_c_eval_string’ to it.  Perhaps you
> could expose a ‘write’ variant, as well, for complete user control.

As above, the double-quote stripping is intensely hacky :-).  I'll look
into the concepts of display and write and see what I can figure out.
Any suggestions or pointers to example code are welcome.

> Similarly for the other funcs.  Overall, i get the vague impression
> that points of exposure could be improved -- made more orthogonal, w/
> fewer (but more powerful) funcs, but that's probably simply my
> ignorance of Make internals speaking.

Well, I'm only exposing two functions, so "fewer" would be ... one, I
guess :-).  It's true that one of them (make-expand) is a superset of
the other; with that single function you could do pretty much anything I
can think of; I'm not sure there's any more powerful capability I can
export.  I'm not sure that's best though.

If anyone has any specific thoughts I'm very interested (that's why I'm
writing).

-- 
-------------------------------------------------------------------------------
 Paul D. Smith <psmith@gnu.org>          Find some GNU make tips at:
 http://www.gnu.org                      http://make.mad-scientist.net
 "Please remain calm...I may be mad, but I am a professional." --Mad Scientist

Re: Using guile as an extension language for GNU make

[Ludovic Courtès]

Hi Paul!

Paul Smith <psmith@gnu.org> skribis:

> On Sun, 2011-09-18 at 14:10 +0200, Ludovic Courts wrote:
>> There are two problems I can think of here:
>> 
>>   - string unquoting is actually more complex than this (recall that
>>     ‘object->string’ merely calls ‘write’):
>>   - ‘scm_c_eval_string’ can return any Scheme objects, some of which
>>     have meaningless representations as strings:
>> 
>> The latter is probably the most serious question.  I think you would
>> really want to constrain expressions passed in $(guile ...) to return
>> a string, and not any other type of objects.
>> 
>> In that case, you could solve the first problem by using (display ...)
>> instead of (write ...).
>
> There's no question that the string conversion is a hack: that's mainly
> why I'm asking here :-).

Heh, understood.  :-)

> I don't want to restrict it to just strings; for example I'd have to
> support numbers as well (one of GNU make's main issues right now is that
> there is no math capability whatsoever).  I'd like to be able to
> possibly convert other things like t and nil (in GNU make's terms,
> "false" is the empty string and "true" is any non-empty string).  Maybe
> there are other things as well (what about symbols like 'foobar?), I
> don't know.

OK, makes sense.  What about defining the Scheme data types that would
be acceptable to pass back at the make level?  I think you pretty much
listed them above: strings, numbers, and booleans.

In your code, you would type-check them along these lines:

  char *c str;

  [...]

  if (scm_is_string (obj))
    c_str = scm_to_locale_string (obj); /* No quoting issue here. */
  else if (scm_is_number (obj))
    c_str = scm_to_locale_string (object_to_string (obj));
  else if (scm_is_false (obj))
    c_str = strdup ("#f");
  else if (scm_is_eq (obj, SCM_BOOL_T))
    c_str = strdup ("#t");
  else
    scm_wrong_type_arg (__func__, 0, obj);

> As above, the double-quote stripping is intensely hacky :-).  I'll look
> into the concepts of display and write and see what I can figure out.
> Any suggestions or pointers to example code are welcome.

See info "(guile) Scheme Write".

Thanks,
Ludo’.

Re: Using guile as an extension language for GNU make

[Thien-Thi Nguyen]

() Paul Smith <psmith@gnu.org>
() Sat, 17 Sep 2011 20:10:16 -0400

    char *str = scm_to_locale_string (scm_object_to_string (scm_c_eval_string (argv[0]),
                                                            SCM_UNDEFINED));
    char *s = str;
    unsigned int l = strlen (s);

    if (s[0] == '"' && s[l-1] == '"')
      {
        s[l-1] = '\0';
        ++s;
        l -= 2;
      }

    o = variable_buffer_output (o, s, l);
    free (str);

The double-quote stripping is kind of hacky.  I would create a port and
‘display’ the result of ‘scm_c_eval_string’ to it.  Perhaps you could
expose a ‘write’ variant, as well, for complete user control.

Similarly for the other funcs.  Overall, i get the vague impression that
points of exposure could be improved -- made more orthogonal, w/ fewer
(but more powerful) funcs, but that's probably simply my ignorance of Make
internals speaking.

Re: Using guile as an extension language for GNU make

[Paul Smith]

On Sun, 2011-09-18 at 17:30 +0200, Thien-Thi Nguyen wrote:
> The double-quote stripping is kind of hacky.  I would create a port
> and ‘display’ the result of ‘scm_c_eval_string’ to it.

Thanks for the hints.  I've reworked my code to implement a generic "SCM
to make string" function; currently it looks like this:

        char *
        cvt_scm_to_str (SCM obj)
        {
          char *str;
        
          if (scm_is_bool (obj) && scm_is_true (obj))
            str = xstrdup ("t");
          else if (scm_is_number (obj) || scm_is_string (obj) || scm_is_symbol (obj))
            {
              SCM port = scm_open_output_string ();
        
              scm_display (obj, port);
              str = scm_to_locale_string (scm_get_output_string (port));
              scm_close_output_port (port);
            }
          else
            str = xstrdup ("");
        
          return str;
        }

That's better than what I had before, but I still have some concerns.
For example, what if a Guile call wanted to return a list?  I can use
display as above, but the list will be enclosed in parentheses, which is
not how make displays lists.  Is there a clean way to handle this?  I
could write a function then invoke it with scm_map() (right?) but this
seems like it might be work.  Also what if the data structure is more
complex, where some elements of the list are lists themselves, etc.?  I
can "flatten" the entire thing out, I suppose.

Or I could ignore them as above and require the Guile scripting to
convert the list into a string before returning it.

-- 
-------------------------------------------------------------------------------
 Paul D. Smith <psmith@gnu.org>          Find some GNU make tips at:
 http://www.gnu.org                      http://make.mad-scientist.net
 "Please remain calm...I may be mad, but I am a professional." --Mad Scientist

Re: Using guile as an extension language for GNU make

[Thien-Thi Nguyen]

() Paul Smith <psmith@gnu.org>
() Sun, 18 Sep 2011 15:28:00 -0400

   [new impl]

   That's better than what I had before, but I still have some concerns.
   For example, what if a Guile call wanted to return a list?  I can use
   display as above, but the list will be enclosed in parentheses, which is
   not how make displays lists.  Is there a clean way to handle this?

"Clean" in what way?  Everything depends on what the expected (valid) use
of this facility would be.  Could you give some examples (w/ failure cases)?

   I could write a function then invoke it with scm_map() (right?) but this
   seems like it might be work.

Everything is work.  Even play is work (but hopefully more fun/interesting).

   Also what if the data structure is more complex, where some elements of the
   list are lists themselves, etc.?  I can "flatten" the entire thing out, I
   suppose.

Yes.

   Or I could ignore them as above and require the Guile scripting to
   convert the list into a string before returning it.

Right.  The protocol is internal; you can choose 60/40 or 40/60.

I think concomitant w/ this particular hacking it would nice to implement
some of the make functions (e.g., ‘patsubst’) in Scheme.  Enough of that
and you will end up {in,con}verting the "embedding" effort to an "extending"
effort as suggested by Ludo -- YHBW!  :-D

Re: Using guile as an extension language for GNU make

[Paul Smith]

On Mon, 2011-09-19 at 02:28 +0200, Thien-Thi Nguyen wrote:
> "Clean" in what way?  Everything depends on what the expected (valid) use
> of this facility would be.  Could you give some examples (w/ failure cases)?

In make, everything is just words: broken up on whitespace.  So for
example, maybe someone writes a Guile function that computes a complex
set of prerequisites for a target:

	target: $(guile (...some Guile program...))

The list of prerequisites is a whitespace-separated list of target
names.  If the Guile program returned a string like "dep1 dep2 dep3"
then that would be fine with what I have.  But it seems like it might be
nice to allow the Guile script to return it as a list instead.  Then I'd
like to convert a list like '(dep1 dep2 dep3) into a string "dep1 dep2
dep3" (not "(dep1 dep2 dep3)" as display would do).

But of course each element of the list could be something more complex,
as well.  So it gets tricky.

>    I could write a function then invoke it with scm_map() (right?) but this
>    seems like it might be work.
> 
> Everything is work.  Even play is work (but hopefully more fun/interesting).

Yes but implementing it in C, with the memory management etc., would be
a lot of (not fun/interesting) work.

Hm.  I guess I could write a little Guile program to do it for me :-).

> I think concomitant w/ this particular hacking it would nice to implement
> some of the make functions (e.g., ‘patsubst’) in Scheme.  Enough of that
> and you will end up {in,con}verting the "embedding" effort to an "extending"
> effort as suggested by Ludo -- YHBW!  :-D

Well, since Guile is not required and I want GNU make to continue to
work as-is on systems where Guile is not available, I won't be rewriting
core features in Guile.   Yet?!?!  :-).

-- 
-------------------------------------------------------------------------------
 Paul D. Smith <psmith@gnu.org>          Find some GNU make tips at:
 http://www.gnu.org                      http://make.mad-scientist.net
 "Please remain calm...I may be mad, but I am a professional." --Mad Scientist

Re: Using guile as an extension language for GNU make

[Hans Aberg]

On 19 Sep 2011, at 17:14, Paul Smith wrote:

> In make, everything is just words: broken up on whitespace.  So for
> example, maybe someone writes a Guile function that computes a complex
> set of prerequisites for a target:
> 
> 	target: $(guile (...some Guile program...))
> 
> The list of prerequisites is a whitespace-separated list of target
> names.  If the Guile program returned a string like "dep1 dep2 dep3"
> then that would be fine with what I have.  But it seems like it might be
> nice to allow the Guile script to return it as a list instead.  Then I'd
> like to convert a list like '(dep1 dep2 dep3) into a string "dep1 dep2
> dep3" (not "(dep1 dep2 dep3)" as display would do).
> 
> But of course each element of the list could be something more complex,
> as well.  So it gets tricky.

How does 'make' store a list of words separated by strings internally? If Guile converts its lists to such strings, the behavior is likely to be fragile. It would better if 'make' converted those strings internally to lists, and the Guile extension interaction with those lists.

Hans

Re: Using guile as an extension language for GNU make

[Paul Smith]

On Mon, 2011-09-19 at 21:41 +0200, Hans Aberg wrote:
> On 19 Sep 2011, at 17:14, Paul Smith wrote:
> 
> > In make, everything is just words: broken up on whitespace.  So for
> > example, maybe someone writes a Guile function that computes a complex
> > set of prerequisites for a target:
> > 
> > 	target: $(guile (...some Guile program...))
> > 
> > The list of prerequisites is a whitespace-separated list of target
> > names.  If the Guile program returned a string like "dep1 dep2 dep3"
> > then that would be fine with what I have.  But it seems like it might be
> > nice to allow the Guile script to return it as a list instead.  Then I'd
> > like to convert a list like '(dep1 dep2 dep3) into a string "dep1 dep2
> > dep3" (not "(dep1 dep2 dep3)" as display would do).
> > 
> > But of course each element of the list could be something more complex,
> > as well.  So it gets tricky.
> 
> How does 'make' store a list of words separated by strings internally?
> If Guile converts its lists to such strings, the behavior is likely to
> be fragile. It would better if 'make' converted those strings
> internally to lists, and the Guile extension interaction with those
> lists.

Hi Hans;

The first stage of make is reading in all the makefiles.  As part of
this, variables and functions are expanded (one line at a time) and the
result is a string.

The $(guile ...) make function, like all other make functions and
variables, is expanded in this way, so the return value of the guile
function must be converted to a string and appended to the string buffer
make is constructing.  Once all of the input is expanded and the
expansion is added to the buffer, then make will go back and parse it
(exactly how this is done is very context-dependent).

The short is, I agree that Guile would not want to try to convert a list
into some kind of internal make "list of words" data structure; in fact
that just wouldn't work because at the time it runs there is no such
thing.  All we have is a buffer with whitespace-separated words.

Rather, I need to define a translation from any Guile data type I want
to support into a make-appropriate string (char* buffer) so it can be
appended to make's read buffer, then parsed by make.  For any Guile data
type I can't or don't want to translate, I'll either throw a make error
or else just expand to the empty string.

So far, I have these translations (see the C code I posted earlier):

t		=> "t" (for make conditionals, non-empty is true)
nil		=> ""  (for make conditionals, empty is false)
"string"	=> "string"
'symbol		=> "symbol"
1234		=> "1234"

I can see that it would be nice to be able to translate:

'(a b c)	=> "a b c"

But what about more complex structures like lists of lists?  What about
simple pairs; should '(a . b) => "a b" as well?  Lists of words are just
about all make knows about so maybe the answer is yes.

And finally, currently I have all unknown types expanding to the empty
string but now I'm thinking it would be better to start out more
restrictive and throw errors.  This would ensure that people write their
Guile scripts correctly (giving valid return values) from the start, and
would let me, in the future, expand the supported types without breaking
anything.

Re: Using guile as an extension language for GNU make

[Hans Aberg]

On 19 Sep 2011, at 23:56, Paul Smith wrote:

> The first stage of make is reading in all the makefiles.  As part of
> this, variables and functions are expanded (one line at a time) and the
> result is a string.

Its parser is handwritten, and perhaps not easy to tweak, and get 'make' variables and stuff into Guile.

> The $(guile ...) make function, like all other make functions and
> variables, is expanded in this way, so the return value of the guile
> function must be converted to a string and appended to the string buffer
> make is constructing.  Once all of the input is expanded and the
> expansion is added to the buffer, then make will go back and parse it
> (exactly how this is done is very context-dependent).

There is the pretty-print module.
  (use-modules (ice-9 pretty-print))

Perhaps you might check how it is implemented.

Hans

Re: Using guile as an extension language for GNU make

[Hans Aberg]

On 19 Sep 2011, at 23:56, Paul Smith wrote:

> Rather, I need to define a translation from any Guile data type I want
> to support into a make-appropriate string (char* buffer) so it can be
> appended to make's read buffer, then parsed by make.  For any Guile data
> type I can't or don't want to translate, I'll either throw a make error
> or else just expand to the empty string.
> 
> So far, I have these translations (see the C code I posted earlier):
> 
> t		=> "t" (for make conditionals, non-empty is true)
> nil		=> ""  (for make conditionals, empty is false)
> "string"	=> "string"
> 'symbol		=> "symbol"
> 1234		=> "1234"
> 
> I can see that it would be nice to be able to translate:
> 
> '(a b c)	=> "a b c"
> 
> But what about more complex structures like lists of lists?  What about
> simple pairs; should '(a . b) => "a b" as well?  Lists of words are just
> about all make knows about so maybe the answer is yes.
> 
> And finally, currently I have all unknown types expanding to the empty
> string but now I'm thinking it would be better to start out more
> restrictive and throw errors.  This would ensure that people write their
> Guile scripts correctly (giving valid return values) from the start, and
> would let me, in the future, expand the supported types without breaking
> anything.

Perhaps you should have some Guile pretty-printing object that can be manipulated from the makefile.

Hans

Re: Using guile as an extension language for GNU make

[Mark H Weaver]

Hi Paul,

Thanks for working on this! :)

Paul Smith <psmith@gnu.org> writes:
> So far, I have these translations (see the C code I posted earlier):
>
> t		=> "t" (for make conditionals, non-empty is true)
> nil		=> ""  (for make conditionals, empty is false)

Note that in modern Scheme, t and nil are just ordinary symbols, and not
booleans.  #t and #f are the boolean values, though Guile also includes
#nil in order to support other Lisps (notably Emacs Lisp).  The proper
way to test for either #nil or #f from C is to use "scm_is_false".

> "string"	=> "string"
> 'symbol	=> "symbol"
> 1234		=> "1234"
>
> I can see that it would be nice to be able to translate:
>
> '(a b c)	=> "a b c"

This all sounds great.

> But what about more complex structures like lists of lists?  What about
> simple pairs; should '(a . b) => "a b" as well?  Lists of words are just
> about all make knows about so maybe the answer is yes.

I'd recommend limiting the automatic conversion of list structure to
only proper lists of atoms, and to otherwise raise an exception.  For
unusual cases, it's easy enough to flatten the list (or even convert it
to a string) from within Scheme.

> And finally, currently I have all unknown types expanding to the empty
> string but now I'm thinking it would be better to start out more
> restrictive and throw errors.  This would ensure that people write their
> Guile scripts correctly (giving valid return values) from the start, and
> would let me, in the future, expand the supported types without breaking
> anything.

I agree wholeheartedly.  If you are too permissive, it may cause
programming errors to go unnoticed.  It's better to limit the automatic
conversions to common cases that are clearly useful.  The other cases
can be handled easily from Scheme.

As for the suggestion to look at the pretty-printing library: I don't
see why that would be useful.  The purpose of that library is primarily
to intelligently choose where to insert newlines in nested list
structure, so that lines don't grow too long for the terminal display,
and to add indentation as would be found in typical Scheme source code.
It seems to me that these functions are undesirable for this purpose.

     Best,
      Mark

Re: Using guile as an extension language for GNU make

[Hans Aberg]

On 21 Sep 2011, at 04:42, Mark H Weaver wrote:

>> And finally, currently I have all unknown types expanding to the empty
>> string but now I'm thinking it would be better to start out more
>> restrictive and throw errors.  This would ensure that people write their
>> Guile scripts correctly (giving valid return values) from the start, and
>> would let me, in the future, expand the supported types without breaking
>> anything.
> 
> I agree wholeheartedly.  If you are too permissive, it may cause
> programming errors to go unnoticed.  It's better to limit the automatic
> conversions to common cases that are clearly useful.  The other cases
> can be handled easily from Scheme.
> 
> As for the suggestion to look at the pretty-printing library: I don't
> see why that would be useful.  The purpose of that library is primarily
> to intelligently choose where to insert newlines in nested list
> structure, so that lines don't grow too long for the terminal display,
> and to add indentation as would be found in typical Scheme source code.
> It seems to me that these functions are undesirable for this purpose.

What I had in my mind is a more general "pretty-printing" object, which can override normal printing. For each object, one should be able define new printing. Perhaps Guile would need to have some kind of extension to achieve that.

Hans

Re: Using guile as an extension language for GNU make

[Thien-Thi Nguyen]

() Paul Smith <psmith@gnu.org>
() Mon, 19 Sep 2011 11:14:34 -0400

   In make, everything is just words: broken up on whitespace.  So for
   example, maybe someone writes a Guile function that computes a complex
   set of prerequisites for a target:

           target: $(guile (...some Guile program...))

Even before thinking about the return value (which, scanning ahead,
you seem to have hit upon a workable plan), i have some questions
about the "some Guile program":

- Is that just a procedure call or can it be definition(s) + expression(s)?
  e.g., single proc call:
    $(guile (compute-complex-prerequisites "$@"))
  e.g., complex:
    $(guile (use-modules (srfi srfi-13))
            (define (ext filename)
              (string-append filename ".ext"))
            (define (ext-all ls)
              (map ext ls))
            (define normalize  ;;; nb: alias
              string-tokenize)
            (define (compute-complex-prerequisites s)
              (ext-all (normalize s)))
            ;; comment: do it!
            (trace 'compute-complex-prerequisites "$@")
            (compute-complex-prerequisites "$@"))

- (if the former) How would ‘compute-complex-prerequisites’ be defined?

- Would stuff like ‘normalize’ and ‘trace’ be provided (builtin)?

- What happens if the stuff inside the double quotes includes
  $(call...) or $(eval...)?  What about those constructs elsewhere?

   Then I'd like to convert a list like '(dep1 dep2 dep3) into a string "dep1
   dep2 dep3" (not "(dep1 dep2 dep3)" as display would do).

   But of course each element of the list could be something more complex,
   as well.  So it gets tricky.

In Scheme, w/ SRFI 13, you could express this as:

(define (as-string x)
  (cond ((not x) "")
        ((unspecified? x) "")
        ((eq? #t x) "t")
        ((string? x) x)
        ((null? x) "")
        (else (object->string x))))
        
(define (space-sep x)
  (let ((acc '()))
    (define (walk x)
      (cond ((pair? x)
             (walk (car x))
             (walk (cdr x)))
            ((null? x))
            (else
             (set! acc (cons x acc)))))
    (walk x)
    (string-join (map as-string (reverse! acc)))))
 
This is not so tricky, i think.

   Yes but implementing it in C, with the memory management etc., would be
   a lot of (not fun/interesting) work.

   Hm.  I guess I could write a little Guile program to do it for me :-).

Yes, that's the idea!

   Well, since Guile is not required and I want GNU make to continue to
   work as-is on systems where Guile is not available, I won't be rewriting
   core features in Guile.   Yet?!?!  :-).

IMHO ‘patsubst’ is not a core feature (sez the spewful ignoramus).  Rather,
i think rewriting ‘patsubst’ would be a good exercise to help you formulate
and answer questions about small technical details that will eventually serve
as a foundation for rewriting core features.  Why not give it a try?

Re: Using guile as an extension language for GNU make

[Paul Smith]

On Tue, 2011-09-20 at 18:17 +0200, Thien-Thi Nguyen wrote:
> () Paul Smith <psmith@gnu.org>
> () Mon, 19 Sep 2011 11:14:34 -0400
> 
>    In make, everything is just words: broken up on whitespace.  So for
>    example, maybe someone writes a Guile function that computes a complex
>    set of prerequisites for a target:
> 
>            target: $(guile (...some Guile program...))
> 
> Even before thinking about the return value (which, scanning ahead,
> you seem to have hit upon a workable plan), i have some questions
> about the "some Guile program":
> 
> - Is that just a procedure call or can it be definition(s) + expression(s)?
>   e.g., single proc call:
>     $(guile (compute-complex-prerequisites "$@"))
>   e.g., complex:
>     $(guile (use-modules (srfi srfi-13))
>             (define (ext filename)
>               (string-append filename ".ext"))
>             (define (ext-all ls)
>               (map ext ls))
>             (define normalize  ;;; nb: alias
>               string-tokenize)
>             (define (compute-complex-prerequisites s)
>               (ext-all (normalize s)))
>             ;; comment: do it!
>             (trace 'compute-complex-prerequisites "$@")
>             (compute-complex-prerequisites "$@"))

I showed the code in my original post, but the way I've implemented it
is that the argument to the make "guile" function (that is, everything
after the "$(guile " to the closing paren--it's handy that Guile parens
will virtually always match, but you can use ${guile ...} instead if you
prefer), will first be make-expanded (so all verbatim instances of "$"
in the Guile script will need to be escaped as "$$"--it doesn't seem
like Guile uses "$" very much so this shouldn't be too painful) and then
the result will be passed to scm_c_eval_string().

As far as I understand it, that means the Guile program can be as
complex as you want.  HOWEVER, you will need to use backslashes to
escape newlines in your example above.

You can also use make's define/endef if you want to write long Guile
programs and avoid the need for backslashes, then you can run $(guile
$(DEFINEDVARIABLE)).

Then make will convert the SCM returned from scm_c_eval_string() into a
string buffer as discussed below etc., and use that as the expansion of
the $(guile ...) function.

> - Would stuff like ‘normalize’ and ‘trace’ be provided (builtin)?

You are now running far beyond my understanding of Guile :-).  If it
works in the environment I describe above then it will work.  If not,
then I'll need more details.

However I just tried your example above and I got an error "unbound
variable: trace" so there's something missing here.  I'll have to read
about that.

> - What happens if the stuff inside the double quotes includes
>   $(call...) or $(eval...)?  What about those constructs elsewhere?

As with all GNU make expansions, quotes are ignored.  The ONLY special
character, to make, in an expansion is "$".  Just as with call and eval,
if you do not escape the "$" then it will be expanded BEFORE the Guile
function is invoked.  So, if you are using (make-expand ...) etc. in
your Guile program you'll need to escape all the "$" there.

There is definitely some complexity here and you have to FIRMLY
understand make's expansion rules, just as you do when you use call and
eval make functions, or even when writing shell script rules.  Some
debugging capability, to show the script we are passing to the guile
function, seems like it would be quite useful.

I suppose it's possible that we could introduce a new, more advanced
parser for the guile function that took into account quotes.  But I'm
nervous about this: it's completely different than other expansions, and
it might be just as annoying as doing it the other way.

> In Scheme, w/ SRFI 13, you could express this as:
> 
> (define (as-string x)
>   (cond ((not x) "")
>         ((unspecified? x) "")
>         ((eq? #t x) "t")
>         ((string? x) x)
>         ((null? x) "")
>         (else (object->string x))))
>         
> (define (space-sep x)
>   (let ((acc '()))
>     (define (walk x)
>       (cond ((pair? x)
>              (walk (car x))
>              (walk (cdr x)))
>             ((null? x))
>             (else
>              (set! acc (cons x acc)))))
>     (walk x)
>     (string-join (map as-string (reverse! acc)))))
>  
> This is not so tricky, i think.

Heh, cute!  I haven't done much Lisp of any kind since I wrote a number
of elisp packages, years ago.  I think I'll have to dust off my Scheme.

>    Well, since Guile is not required and I want GNU make to continue to
>    work as-is on systems where Guile is not available, I won't be rewriting
>    core features in Guile.   Yet?!?!  :-).
> 
> IMHO ‘patsubst’ is not a core feature (sez the spewful ignoramus).  Rather,
> i think rewriting ‘patsubst’ would be a good exercise to help you formulate
> and answer questions about small technical details that will eventually serve
> as a foundation for rewriting core features.  Why not give it a try?

Maybe as an exercise, but patsubst IS a core feature, in that it has
existed forever and there are thousands of makefiles that use it...
basically rewriting existing capabilities so they're not available if
Guile is not present changes things from "GNU make with Guile optional"
to "GNU make with Guile required".  I'm not ready to go there.

-- 
-------------------------------------------------------------------------------
 Paul D. Smith <psmith@gnu.org>          Find some GNU make tips at:
 http://www.gnu.org                      http://make.mad-scientist.net
 "Please remain calm...I may be mad, but I am a professional." --Mad Scientist

Re: Using guile as an extension language for GNU make

[Paul Smith]

On Tue, 2011-09-20 at 13:31 -0400, Paul Smith wrote:
> > In Scheme, w/ SRFI 13, you could express this as:
> >  
> > This is not so tricky, i think.
> 
> Heh, cute!

Thinking about this more it occurs to me that I will likely need a
module defined for any GNU make specific procedures I write to avoid
namespace pollution.

What do people think would be an appropriate module name?

I was thinking something like "gnu make" would be OK, but maybe there
are already conventions for applications/application-specific module
hierarchies that I should be following?

-- 
-------------------------------------------------------------------------------
 Paul D. Smith <psmith@gnu.org>          Find some GNU make tips at:
 http://www.gnu.org                      http://make.mad-scientist.net
 "Please remain calm...I may be mad, but I am a professional." --Mad Scientist

Re: Using guile as an extension language for GNU make

[Thien-Thi Nguyen]

[-- Attachment #1: Type: text/plain, Size: 617 bytes --]

() Paul Smith <psmith@gnu.org>
() Tue, 20 Sep 2011 15:02:09 -0400

   I was thinking something like "gnu make" would be OK, but maybe there
   are already conventions for applications/application-specific module
   hierarchies that I should be following?

None that i am aware of.  If the module is completely internal,
you can basically choose whatever pleases you.  In the example
previously posted, i chose ‘(make-user)’ to ape '(guile-user)'.

BTW, appended is version 0.2.  It fixes an infloop and flips the
logic from blacklist-plus-default-ok to whitelist-plus-default-wtf.
Too much fun...


[-- Attachment #2: procs-for-gnu-make-hacking.scm --]
[-- Type: application/x-scheme, Size: 1456 bytes --]

Re: Using guile as an extension language for GNU make

[Thien-Thi Nguyen]

[-- Attachment #1: Type: text/plain, Size: 4827 bytes --]

() Paul Smith <psmith@gnu.org>
() Tue, 20 Sep 2011 13:31:28 -0400

   I showed the code in my original post, but the way I've implemented it
   is that the argument to the make "guile" function (that is, everything
   after the "$(guile " to the closing paren--it's handy that Guile parens
   will virtually always match, but you can use ${guile ...} instead if you
   prefer), will first be make-expanded (so all verbatim instances of "$"
   in the Guile script will need to be escaped as "$$"--it doesn't seem
   like Guile uses "$" very much so this shouldn't be too painful) and then
   the result will be passed to scm_c_eval_string().

Ah, ok.

   As far as I understand it, that means the Guile program can be as
   complex as you want.  HOWEVER, you will need to use backslashes to
   escape newlines in your example above.

Distasteful.

   You can also use make's define/endef if you want to write long Guile
   programs and avoid the need for backslashes, then you can run $(guile
   $(DEFINEDVARIABLE)).

   Then make will convert the SCM returned from scm_c_eval_string() into a
   string buffer as discussed below etc., and use that as the expansion of
   the $(guile ...) function.

OK, sounds much better.

   > - Would stuff like ‘normalize’ and ‘trace’ be provided (builtin)?

   You are now running far beyond my understanding of Guile :-).  If it
   works in the environment I describe above then it will work.  If not,
   then I'll need more details.

   However I just tried your example above and I got an error "unbound
   variable: trace" so there's something missing here.  I'll have to read
   about that.

Yeah, ‘trace’ does not exist.  I think it (and other facilities) should
be defined by make, by (perhaps lazily, to avoid penalizing non-users)
loading a file, say, builtin.scm.  The minimum contents might be:

  (define-module (make-user)
    ;; builtins from C
    ;; #:use-module (make-internals)
    #:use-module (guile-user)
    #:use-module (srfi srfi-13)
    #:use-module (srfi srfi-14))
  
  (define DEBUG? (getenv "DEBUG"))
  ;; TODO: Set based on ‘-d’ exposed from C.
  
  (define (fse s . args)
    (apply simple-format (current-error-port) s args))
  
  (define (trace name . etc)
    (cond (DEBUG? (fse "hey: ~A ~S~%" name etc))))
  
  ;;; Add other convenience procs here.

Note that this doesn't export anything; the evaluation would need to
arrange to do its work with this as the "current module".  But, that's
just an idea.  You can also ‘(define-module (guile-user) ...)’ to keep
things simple initially...

   > - What happens if the stuff inside the double quotes includes
   >   $(call...) or $(eval...)?  What about those constructs elsewhere?

   [standard expansion explanation]

OK, got it.

   I suppose it's possible that we could introduce a new, more advanced
   parser for the guile function that took into account quotes.  But I'm
   nervous about this: it's completely different than other expansions, and
   it might be just as annoying as doing it the other way.

I agree, that's probably to be avoided.

   > (define (as-string x) ...)
   > (define (space-sep x) ...)

   Heh, cute!  I haven't done much Lisp of any kind since I wrote a number
   of elisp packages, years ago.  I think I'll have to dust off my Scheme.

Yes, don't delay, just do it!  Playing some more, i realized the above
version 0.0 is suboptimal.  Below is version 0.1 -- perhaps it could go
into builtins.scm.

   Maybe as an exercise, but patsubst IS a core feature, in that it has
   existed forever and there are thousands of makefiles that use it...
   basically rewriting existing capabilities so they're not available if
   Guile is not present changes things from "GNU make with Guile optional"
   to "GNU make with Guile required".  I'm not ready to go there.

Right, i see the sticking point is the "re" in "rewriting".  For me, that
"re" does not imply replacement, just iteration.  My "playing at working"
MO: i skim working code and write (and integrate side-by-side-wise) a
workalike implementation based on what i think are the original's intended
design, then compare the two under load, selecting the impl to use based
on some runtime switch (typically an env var).  This leads to better
questions as understanding (and/or frustration) grows.  Even if the second
impl is inferior (ends up trashed or permanently disabled), i gain
something from the experience (although i lose time, oh well).  Anyway,
that's what i was suggesting.

I chose ‘patsubst’ because its design seems straightforward; it would be
easy to validate another impl.  No worries, i shall harp no more on this.

____________________________________________________

[-- Attachment #2: procs-for-gnu-make-hacking.scm --]
[-- Type: application/x-scheme, Size: 1019 bytes --]