About Guile crypto support

About Guile crypto support

[Nala Ginrut]

As mentioned in another thread about digest algorithm support in Guile,
my plan is use part of implementation of libgcrypt and make a wrapper,
then put into libguile.
But now I found weinholt's Scheme industria lib, which contains all
mainstream crypto(not only digest) algorithm.
http://weinholt.se/industria/manual/crypto.html#crypto
So what's your opinion, guys?
Would you prefer C implementation or Scheme way?

Thanks!

Re: About Guile crypto support

[Thien-Thi Nguyen]

[-- Attachment #1: Type: text/plain, Size: 550 bytes --]

() Nala Ginrut <nalaginrut@gmail.com>
() Sun, 03 Feb 2013 20:55:46 +0800

   Would you prefer C implementation or Scheme way?

The less C the better, generally, IMHO.

-- 
Thien-Thi Nguyen ..................................... GPG key: 4C807502
.                  NB: ttn at glug dot org is not me                   .
.                 (and has not been since 2007 or so)                  .
.                        ACCEPT NO SUBSTITUTES                         .
........... please send technical questions to mailing lists ...........

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: About Guile crypto support

[Daniel Hartwig]

Hello

On 3 February 2013 20:55, Nala Ginrut <nalaginrut@gmail.com> wrote:
> As mentioned in another thread about digest algorithm support in Guile,
> my plan is use part of implementation of libgcrypt and make a wrapper,
> then put into libguile.
> But now I found weinholt's Scheme industria lib, which contains all
> mainstream crypto(not only digest) algorithm.
> http://weinholt.se/industria/manual/crypto.html#crypto

As mentioned on that page, there are some issues that apply to any
pure Scheme implementation:

> Beware that if you're using some of these libraries for sensitive
> data, let's say passwords, then there is probably no way to make
> sure a password is ever gone from memory. There is no guarantee that
> the passwords will not be swapped out to disk or transmitted by
> radio.

Libgcrypt provides a means to specify that some data should be stored
in secured memory, which will never be swapped to disk. Doing
something similar in Guile may be problematic, at least with a
Scheme-only implementation.

> So what's your opinion, guys?
> Would you prefer C implementation or Scheme way?

As gcrypt is mature, reimplementing it in either C or Scheme just for
Guile does not seem useful — on it's own.  An FFI wrapper or extension
benefits from upstream security and maintenance efforts.


If you have a particular interest in learning about crypto. algorithms,
by all means port or write your own implementation using whichever
language.  If you want it to have a Scheme interface, then Scheme
seems a logical choice to use.

If your goal is only to provide crypto. support to Guile programs,
then time is better spent providing a wrapper to the existing library.
 Concerns about adding an external dependency do not hold much weight
next to the advantages of directly using the library; “don't repeat
yourself”, and all that.

Perhaps you are aware that there an extension for gcrypt under
development, with modules for the hash and randomize functions:
<https://gitorious.org/gcrypt-guile/>.


Regards

Re: About Guile crypto support

[Nala Ginrut]

On Mon, 2013-02-04 at 09:14 +0800, Daniel Hartwig wrote:
> Hello
> 
> On 3 February 2013 20:55, Nala Ginrut <nalaginrut@gmail.com> wrote:
> > As mentioned in another thread about digest algorithm support in Guile,
> > my plan is use part of implementation of libgcrypt and make a wrapper,
> > then put into libguile.
> > But now I found weinholt's Scheme industria lib, which contains all
> > mainstream crypto(not only digest) algorithm.
> > http://weinholt.se/industria/manual/crypto.html#crypto
> 
> As mentioned on that page, there are some issues that apply to any
> pure Scheme implementation:
> 

If it's worthy, I can do some modification.

> > Beware that if you're using some of these libraries for sensitive
> > data, let's say passwords, then there is probably no way to make
> > sure a password is ever gone from memory. There is no guarantee that
> > the passwords will not be swapped out to disk or transmitted by
> > radio.
> 
> Libgcrypt provides a means to specify that some data should be stored
> in secured memory, which will never be swapped to disk. Doing
> something similar in Guile may be problematic, at least with a
> Scheme-only implementation.
> 
> > So what's your opinion, guys?
> > Would you prefer C implementation or Scheme way?
> 
> As gcrypt is mature, reimplementing it in either C or Scheme just for
> Guile does not seem useful — on it's own.  An FFI wrapper or extension
> benefits from upstream security and maintenance efforts.
> 
> 
> If you have a particular interest in learning about crypto. algorithms,
> by all means port or write your own implementation using whichever
> language.  If you want it to have a Scheme interface, then Scheme
> seems a logical choice to use.
> 

Well, no ;-)

> If your goal is only to provide crypto. support to Guile programs,
> then time is better spent providing a wrapper to the existing library.
>  Concerns about adding an external dependency do not hold much weight
> next to the advantages of directly using the library; “don't repeat
> yourself”, and all that.
> 

That's my aim, nowadays a language should provide md5/sha1 at least
since they are very common.

> Perhaps you are aware that there an extension for gcrypt under
> development, with modules for the hash and randomize functions:
> <https://gitorious.org/gcrypt-guile/>.
> 

I have a similar project too:
https://gitorious.org/nacre/libgcrypt-guile

Write a lib-wrapper is another story, guys who needs more specific
feature(efficiency/security) could use these packages from guildhall.
But my opinion is to provide the common digest API in ice-9, many guys
asked such a questions, and I suggested them use my libgcrypt-guile, but
I can't answer why Guile doesn't has these common API. So I decide to
add them.

What do you think?

> 
> Regards
> 

Re: About Guile crypto support

[Daniel Hartwig]

On 4 February 2013 11:12, Nala Ginrut <nalaginrut@gmail.com> wrote:
>> If your goal is only to provide crypto. support to Guile programs,
>> then time is better spent providing a wrapper to the existing library.
>>  Concerns about adding an external dependency do not hold much weight
>> next to the advantages of directly using the library; “don't repeat
>> yourself”, and all that.
>>
>
> That's my aim, nowadays a language should provide md5/sha1 at least
> since they are very common.
>
>> Perhaps you are aware that there an extension for gcrypt under
>> development, with modules for the hash and randomize functions:
>> <https://gitorious.org/gcrypt-guile/>.
>>
>
> I have a similar project too:
> https://gitorious.org/nacre/libgcrypt-guile
>

The other project is much further along, providing direct bindings to
gcrypt and has a convenience form (quick-hash ALGORITHM DATA).
Suggest to continue working there rather than duplicating efforting.

Taking a quick look at your code, there are some problems that
immediately stand out.  The calls to gcry_control should not be in
mda, being repeated on every call; libgcrypt documents even specify
that language bindings should not handle init, which is a task for the
application.

> Write a lib-wrapper is another story, guys who needs more specific
> feature(efficiency/security) could use these packages from guildhall.
> But my opinion is to provide the common digest API in ice-9, many guys
> asked such a questions, and I suggested them use my libgcrypt-guile, but
> I can't answer why Guile doesn't has these common API. So I decide to
> add them.
>
> What do you think?

Personally I would work with the existing gcrypt-guile project.  It is
LGPL and anyone who needs such functionality can easily import it.  As
mentioned above, it already provides the quick-hash form which I
believe is what you are looking for.

I also don't any pressing need to move this in to the core Guile
distribution when it is freely available as an addon.  But I see that
we disagreed on this same point with your colorized module as well :-)

Regards

Re: About Guile crypto support

[Nala Ginrut]

On Mon, 2013-02-04 at 11:35 +0800, Daniel Hartwig wrote:
> On 4 February 2013 11:12, Nala Ginrut <nalaginrut@gmail.com> wrote:
> >> If your goal is only to provide crypto. support to Guile programs,
> >> then time is better spent providing a wrapper to the existing library.
> >>  Concerns about adding an external dependency do not hold much weight
> >> next to the advantages of directly using the library; “don't repeat
> >> yourself”, and all that.
> >>
> >
> > That's my aim, nowadays a language should provide md5/sha1 at least
> > since they are very common.
> >
> >> Perhaps you are aware that there an extension for gcrypt under
> >> development, with modules for the hash and randomize functions:
> >> <https://gitorious.org/gcrypt-guile/>.
> >>
> >
> > I have a similar project too:
> > https://gitorious.org/nacre/libgcrypt-guile
> >
> 
> The other project is much further along, providing direct bindings to
> gcrypt and has a convenience form (quick-hash ALGORITHM DATA).
> Suggest to continue working there rather than duplicating efforting.
> 
> Taking a quick look at your code, there are some problems that
> immediately stand out.  The calls to gcry_control should not be in
> mda, being repeated on every call; libgcrypt documents even specify
> that language bindings should not handle init, which is a task for the
> application.
> 

libgcrypt-guile is old and earlier than grcypt-guile one year. 
I need digest things and there's no guildhall, so I have to write a
buggy one. I'm glad there's a better choice for it. ;-) 

> > Write a lib-wrapper is another story, guys who needs more specific
> > feature(efficiency/security) could use these packages from guildhall.
> > But my opinion is to provide the common digest API in ice-9, many guys
> > asked such a questions, and I suggested them use my libgcrypt-guile, but
> > I can't answer why Guile doesn't has these common API. So I decide to
> > add them.
> >
> > What do you think?
> 
> Personally I would work with the existing gcrypt-guile project.  It is
> LGPL and anyone who needs such functionality can easily import it.  As
> mentioned above, it already provides the quick-hash form which I
> believe is what you are looking for.
> 
> I also don't any pressing need to move this in to the core Guile
> distribution when it is freely available as an addon.  But I see that
> we disagreed on this same point with your colorized module as well :-)
> 

Well, I think there's a little difference to add module to core between
us. My opinion is to add very common thing to core, though sometimes
these things maybe not a very proper core API.
But anyway, considering guildhall works in the long term, these two
opinions difference are trivial. ;-)

I don't against anything about gcrypt-guile, it's fine.
But I think adding common digest APIs into core is a considerable topic.
If folks need more powerful crypto things, finding other more expert
packages in the guildhall is a better choice. But most of the time, they
just need md5/sha since doing some hash work is very common nowadays.
Users don't have to download & compile another project, or install
guildhall for that if they just need md5. They will choice a pipe for
'md5sum' instead. I do think Guile should provide convenience for such
issue. 

What do you think?

> Regards

Re: About Guile crypto support

[Ludovic Courtès]

Nala Ginrut <nalaginrut@gmail.com> skribis:

> As mentioned in another thread about digest algorithm support in Guile,
> my plan is use part of implementation of libgcrypt and make a wrapper,
> then put into libguile.

We probably don’t want Guile to depend on libgcrypt.

So, instead, I’d suggest choosing the best of the 10 gcrypt FFI bindings
already mentioned ;-), and putting it in the guildhall.

If you want to go further, you (or its authors) could submit it for
inclusion in libgcrypt proper.

Thanks,
Ludo’.

Re: About Guile crypto support

[Nala Ginrut]

On Tue, 2013-02-05 at 00:03 +0100, Ludovic Courtès wrote:
> Nala Ginrut <nalaginrut@gmail.com> skribis:
> 
> > As mentioned in another thread about digest algorithm support in Guile,
> > my plan is use part of implementation of libgcrypt and make a wrapper,
> > then put into libguile.
> 
> We probably don’t want Guile to depend on libgcrypt.
> 

No, I didn't mean to use libgcrypt directly, I just suggested reuse part
of libgcrypt code(only the common digest algorithm) and make wrapper,
then put the C code in libguile.

> So, instead, I’d suggest choosing the best of the 10 gcrypt FFI bindings
> already mentioned ;-), and putting it in the guildhall.
> 
> If you want to go further, you (or its authors) could submit it for
> inclusion in libgcrypt proper.
> 

The gcrypt-guile project is doing so, I'll help it if I can. 
But my original thought is orthogonal with gcrypt-guile, just put some
common digest algorithm in libguile rather than a full-stack crypto-lib.

My suggest opposed by many guys, since they don't think md5/sha are very
common things. ;-)
But I'm dealing with server & web framework development, so I need
digest so much, and maybe it's no so common.
I'll let this topic alone, till others found they have same
requirements, or just forget about it. ;-D

> Thanks,
> Ludo’.
> 
> 

Re: About Guile crypto support

[Noah Lavine]

[-- Attachment #1: Type: text/plain, Size: 2043 bytes --]

Hello,

I was just thinking about this, and I was wondering, can you hash an
arbitrary Guile object? And if so, what do you hash? (I mean, algorithms
like SHA-1 are defined on sequences of bits, as I understand it. So what
collection of bits do you hash?) And is the hash recursive? (I.e. is it an
equal?-hash, an eqv?-hash, or an eq?-hash.)

If I understand the conversation correctly, the answer is yes, and that you
hash the bit representation that Guile uses internally, and it is an
equal?-hash. Is that accurate?

Thanks,
Noah


On Mon, Feb 4, 2013 at 9:43 PM, Nala Ginrut <nalaginrut@gmail.com> wrote:

> On Tue, 2013-02-05 at 00:03 +0100, Ludovic Courtès wrote:
> > Nala Ginrut <nalaginrut@gmail.com> skribis:
> >
> > > As mentioned in another thread about digest algorithm support in Guile,
> > > my plan is use part of implementation of libgcrypt and make a wrapper,
> > > then put into libguile.
> >
> > We probably don’t want Guile to depend on libgcrypt.
> >
>
> No, I didn't mean to use libgcrypt directly, I just suggested reuse part
> of libgcrypt code(only the common digest algorithm) and make wrapper,
> then put the C code in libguile.
>
> > So, instead, I’d suggest choosing the best of the 10 gcrypt FFI bindings
> > already mentioned ;-), and putting it in the guildhall.
> >
> > If you want to go further, you (or its authors) could submit it for
> > inclusion in libgcrypt proper.
> >
>
> The gcrypt-guile project is doing so, I'll help it if I can.
> But my original thought is orthogonal with gcrypt-guile, just put some
> common digest algorithm in libguile rather than a full-stack crypto-lib.
>
> My suggest opposed by many guys, since they don't think md5/sha are very
> common things. ;-)
> But I'm dealing with server & web framework development, so I need
> digest so much, and maybe it's no so common.
> I'll let this topic alone, till others found they have same
> requirements, or just forget about it. ;-D
>
> > Thanks,
> > Ludo’.
> >
> >
>
>
>
>

[-- Attachment #2: Type: text/html, Size: 2713 bytes --]

Re: About Guile crypto support

[Nala Ginrut]

hi Noah!

On Mon, 2013-02-04 at 21:57 -0500, Noah Lavine wrote:
> Hello,
> 
> 
> I was just thinking about this, and I was wondering, can you hash an
> arbitrary Guile object? And if so, what do you hash? (I mean,
> algorithms like SHA-1 are defined on sequences of bits, as I
> understand it. So what collection of bits do you hash?) And is the
> hash recursive? (I.e. is it an equal?-hash, an eqv?-hash, or an
> eq?-hash.)
> 
> 
> If I understand the conversation correctly, the answer is yes, and
> that you hash the bit representation that Guile uses internally, and
> it is an equal?-hash. Is that accurate?
> 
According to our talking about this topic, I'll explain something ;-P
My goal is not to hash arbitrary object. As I said, I'm dealing with
Server-design and web-framework, so what I need is to hash bytevectors
and strings. Since Guile2 uses character-string instead of byte-string,
I have to mention strings separately.
> 
Thanks!

> Thanks,
> Noah
> 
> 
> On Mon, Feb 4, 2013 at 9:43 PM, Nala Ginrut <nalaginrut@gmail.com>
> wrote:
>         On Tue, 2013-02-05 at 00:03 +0100, Ludovic Courtès wrote:
>         > Nala Ginrut <nalaginrut@gmail.com> skribis:
>         >
>         > > As mentioned in another thread about digest algorithm
>         support in Guile,
>         > > my plan is use part of implementation of libgcrypt and
>         make a wrapper,
>         > > then put into libguile.
>         >
>         > We probably don’t want Guile to depend on libgcrypt.
>         >
>         
>         
>         No, I didn't mean to use libgcrypt directly, I just suggested
>         reuse part
>         of libgcrypt code(only the common digest algorithm) and make
>         wrapper,
>         then put the C code in libguile.
>         
>         > So, instead, I’d suggest choosing the best of the 10 gcrypt
>         FFI bindings
>         > already mentioned ;-), and putting it in the guildhall.
>         >
>         > If you want to go further, you (or its authors) could submit
>         it for
>         > inclusion in libgcrypt proper.
>         >
>         
>         
>         The gcrypt-guile project is doing so, I'll help it if I can.
>         But my original thought is orthogonal with gcrypt-guile, just
>         put some
>         common digest algorithm in libguile rather than a full-stack
>         crypto-lib.
>         
>         My suggest opposed by many guys, since they don't think
>         md5/sha are very
>         common things. ;-)
>         But I'm dealing with server & web framework development, so I
>         need
>         digest so much, and maybe it's no so common.
>         I'll let this topic alone, till others found they have same
>         requirements, or just forget about it. ;-D
>         
>         > Thanks,
>         > Ludo’.
>         >
>         >
>         
>         
>         
> 
> 

Re: About Guile crypto support

[Ludovic Courtès]

Nala Ginrut <nalaginrut@gmail.com> skribis:

> On Tue, 2013-02-05 at 00:03 +0100, Ludovic Courtès wrote:

[...]

>> We probably don’t want Guile to depend on libgcrypt.
>> 
>
> No, I didn't mean to use libgcrypt directly, I just suggested reuse part
> of libgcrypt code(only the common digest algorithm) and make wrapper,
> then put the C code in libguile.

[...]

> The gcrypt-guile project is doing so, I'll help it if I can. 
> But my original thought is orthogonal with gcrypt-guile, just put some
> common digest algorithm in libguile rather than a full-stack crypto-lib.

We could actually use the Gnulib crypto modules.  There’s a
duplication/convenience trade-off: we’d provide useful functionality by
default, at the expense of duplicating C code in our tarballs (2500 SLOC
for Gnulib’s sha*.[ch] and md5.[ch].)

Opinions?

I think I’d be more inclined to have good bindings in libgcrypt proper,
but I’d like to hear what others think.

Ludo’.

Re: About Guile crypto support

[Daniel Hartwig]

On 5 February 2013 23:48, Ludovic Courtès <ludo@gnu.org> wrote:
>> The gcrypt-guile project is doing so, I'll help it if I can.
>> But my original thought is orthogonal with gcrypt-guile, just put some
>> common digest algorithm in libguile rather than a full-stack crypto-lib.
>
> We could actually use the Gnulib crypto modules.  There’s a
> duplication/convenience trade-off: we’d provide useful functionality by
> default, at the expense of duplicating C code in our tarballs (2500 SLOC
> for Gnulib’s sha*.[ch] and md5.[ch].)
>
> Opinions?

Avoiding duplication and feature creep /in the core/ is highly
desirable.  Guildhall makes it convenient enough to pull in additional
features; guile-lib has md5 and industria provides also sha and
others.

A small core with addons has some obvious benefits, for the platform.
Maintainers may disagree, though I notice that historic choices to
include some modules (such as sxml, web) in the core were made before
the advent of Guildhall.  I wonder where these modules would reside if
they were introduced in the current situation …?

>
> I think I’d be more inclined to have good bindings in libgcrypt proper,

Yes, this solution will have many benefits.  The currently available
bindings are, well, not so great.  However, the API is small enough
that building a proper set will not take much effort.  Curiously, some
parts make use of S-expressions.

Regards

Re: About Guile crypto support

[Daniel Hartwig]

On 6 February 2013 12:18, Daniel Hartwig <mandyke@gmail.com> wrote:
> Avoiding duplication and feature creep /in the core/ is highly
> desirable.  Guildhall makes it convenient enough to pull in additional
> features; guile-lib has md5 and industria provides also sha and
> others.

During yesterday's IRC discussion the idea was raised to somehow
provide a “batteries included” package that would include the core and
a selection of “useful” addons.  I suppose that is similar to the
original purpose of guile-lib.

In apt terms, that would probably just be a meta-package depending on
guile and a few others.

Re: About Guile crypto support

[Ludovic Courtès]

Daniel Hartwig <mandyke@gmail.com> skribis:

> On 5 February 2013 23:48, Ludovic Courtès <ludo@gnu.org> wrote:
>>> The gcrypt-guile project is doing so, I'll help it if I can.
>>> But my original thought is orthogonal with gcrypt-guile, just put some
>>> common digest algorithm in libguile rather than a full-stack crypto-lib.
>>
>> We could actually use the Gnulib crypto modules.  There’s a
>> duplication/convenience trade-off: we’d provide useful functionality by
>> default, at the expense of duplicating C code in our tarballs (2500 SLOC
>> for Gnulib’s sha*.[ch] and md5.[ch].)
>>
>> Opinions?
>
> Avoiding duplication and feature creep /in the core/ is highly
> desirable.  Guildhall makes it convenient enough to pull in additional
> features; guile-lib has md5 and industria provides also sha and
> others.

Scheme implementations of these would be impractical in many cases (too
slow).

> A small core with addons has some obvious benefits, for the platform.
> Maintainers may disagree, though I notice that historic choices to
> include some modules (such as sxml, web) in the core were made before
> the advent of Guildhall.  I wonder where these modules would reside if
> they were introduced in the current situation …?

Some modules were deliberately moved from guile-lib to Guile to improve
integration, and to come with more standard APIs by default.

I’m not sure that having guildhall changes much in that respect.

>> I think I’d be more inclined to have good bindings in libgcrypt proper,
>
> Yes, this solution will have many benefits.  The currently available
> bindings are, well, not so great.  However, the API is small enough
> that building a proper set will not take much effort.

Yeah, apparently there are several half-baked bindings around.  Let’s
just polish one of them, and submit it for inclusion in libgcrypt.

Ludo’.

Re: About Guile crypto support

[Daniel Hartwig]

On 9 February 2013 00:21, Ludovic Courtès <ludo@gnu.org> wrote:
>
> Yeah, apparently there are several half-baked bindings around.  Let’s
> just polish one of them, and submit it for inclusion in libgcrypt.

I have already submitted some additions to one, but I may just continue
in a new repository.

By the way, I very much like the conventions used in the GnuTLS
bindings.  The enums in particular make a lot of sense for a security
library, with the extreme type safety they provide.  I will pursue a
similar approach.

One question.  With the current state of FFI, do you think it matters
much whether the bulk of the bindings are done in C or FFI?

Regards

Re: About Guile crypto support

[Ludovic Courtès]

Hi,

Daniel Hartwig <mandyke@gmail.com> skribis:

> By the way, I very much like the conventions used in the GnuTLS
> bindings.  The enums in particular make a lot of sense for a security
> library, with the extreme type safety they provide.  I will pursue a
> similar approach.

Yeah, I think it’s helpful.

> One question.  With the current state of FFI, do you think it matters
> much whether the bulk of the bindings are done in C or FFI?

I think it depends on the amount of public C structs, enums, inlines,
and constants, and how often they are changed.  When there are too many
of them and they are subject to change, it might be easier to use C
(though that can be worked around from the FFI by calling the C
compiler, as in [0].)

My impression is that libgcrypt uses mostly opaque pointer types and has
a stable API, so the using FFI should be just fine.

An issue with the FFI is distros where .la and .so files are only
available in the -dev package, because then ‘dynamic-link’ won’t work
unless that -dev package is installed (as recently discussed on
guile-user.)

Thanks,
Ludo’.

[0] http://git.sv.gnu.org/cgit/libchop.git/tree/guile2/chop/internal.scm#n130

Re: About Guile crypto support

[Andy Wingo]

On Sat 09 Feb 2013 16:12, ludo@gnu.org (Ludovic Courtès) writes:

> An issue with the FFI is distros where .la and .so files are only
> available in the -dev package, because then ‘dynamic-link’ won’t work
> unless that -dev package is installed (as recently discussed on
> guile-user.)

I have the feeling that we should implement our own dynamic-link
function without libltdl.  It would eliminate a dependency and allow us
to use other search path rules, like ones that could deal with this
case.  I think the situation would actually be better on other
architectures because we wouldn't have to deal with bugs like this one:

  http://thread.gmane.org/gmane.lisp.guile.bugs/5269

Andy
-- 
http://wingolog.org/

Re: About Guile crypto support

[Mark H Weaver]

Hi Andy,

Andy Wingo <wingo@pobox.com> writes:

> On Sat 09 Feb 2013 16:12, ludo@gnu.org (Ludovic Courtès) writes:
>
>> An issue with the FFI is distros where .la and .so files are only
>> available in the -dev package, because then ‘dynamic-link’ won’t work
>> unless that -dev package is installed (as recently discussed on
>> guile-user.)
>
> I have the feeling that we should implement our own dynamic-link
> function without libltdl.  It would eliminate a dependency and allow us
> to use other search path rules, like ones that could deal with this
> case.  I think the situation would actually be better on other
> architectures because we wouldn't have to deal with bugs like this one:
>
>   http://thread.gmane.org/gmane.lisp.guile.bugs/5269

The problems we're having with libltdl are likely affecting many other
projects.  Wouldn't it be better to fix these problems in libltdl, to
the benefit of all its users, than for each of its users to duplicate
its functionality within their own projects?

More generally, I'm concerned with the direction we are being pressured
into by those who complain about the number of dependencies.  We ought
to look for better solutions than duplicating library functionality
within Guile's own source code.  Imagine if every program did that.
That way lies madness.

IMO, we ought to look for better solutions for those who complain about
dependencies.  One idea is to provide precompiled versions of Guile for
the major platforms (i.e. MinGW, MacOS and possibly also GNU/Linux) with
all dependencies included, for use by libguile-based projects that wish
to provide precompiled bundles for their users.  It might also make
sense to provide something along the lines of jhbuild to make the build
job easier for those who want more flexibility.

What do you think?

    Mark

Re: About Guile crypto support

[Noah Lavine]

[-- Attachment #1: Type: text/plain, Size: 2474 bytes --]

Mark,

I agree with everything you said about dependencies. I think the real
solution is something like what you said - sharing code, but bundling.

One way to push that farther would be to distribute tarballs that include
the complete source of some libraries, and somehow making a combined build
system that configures and builds all of them. I would nominate libgc for
this, since we keep hitting trouble with new or old libgc versions.

Noah

On Sat, Feb 9, 2013 at 12:50 PM, Mark H Weaver <mhw@netris.org> wrote:

> Hi Andy,
>
> Andy Wingo <wingo@pobox.com> writes:
>
> > On Sat 09 Feb 2013 16:12, ludo@gnu.org (Ludovic Courtès) writes:
> >
> >> An issue with the FFI is distros where .la and .so files are only
> >> available in the -dev package, because then ‘dynamic-link’ won’t work
> >> unless that -dev package is installed (as recently discussed on
> >> guile-user.)
> >
> > I have the feeling that we should implement our own dynamic-link
> > function without libltdl.  It would eliminate a dependency and allow us
> > to use other search path rules, like ones that could deal with this
> > case.  I think the situation would actually be better on other
> > architectures because we wouldn't have to deal with bugs like this one:
> >
> >   http://thread.gmane.org/gmane.lisp.guile.bugs/5269
>
> The problems we're having with libltdl are likely affecting many other
> projects.  Wouldn't it be better to fix these problems in libltdl, to
> the benefit of all its users, than for each of its users to duplicate
> its functionality within their own projects?
>
> More generally, I'm concerned with the direction we are being pressured
> into by those who complain about the number of dependencies.  We ought
> to look for better solutions than duplicating library functionality
> within Guile's own source code.  Imagine if every program did that.
> That way lies madness.
>
> IMO, we ought to look for better solutions for those who complain about
> dependencies.  One idea is to provide precompiled versions of Guile for
> the major platforms (i.e. MinGW, MacOS and possibly also GNU/Linux) with
> all dependencies included, for use by libguile-based projects that wish
> to provide precompiled bundles for their users.  It might also make
> sense to provide something along the lines of jhbuild to make the build
> job easier for those who want more flexibility.
>
> What do you think?
>
>     Mark
>
>

[-- Attachment #2: Type: text/html, Size: 3253 bytes --]

Re: About Guile crypto support

[Ludovic Courtès]

Mark H Weaver <mhw@netris.org> skribis:

> Andy Wingo <wingo@pobox.com> writes:
>
>> On Sat 09 Feb 2013 16:12, ludo@gnu.org (Ludovic Courtès) writes:
>>
>>> An issue with the FFI is distros where .la and .so files are only
>>> available in the -dev package, because then ‘dynamic-link’ won’t work
>>> unless that -dev package is installed (as recently discussed on
>>> guile-user.)
>>
>> I have the feeling that we should implement our own dynamic-link
>> function without libltdl.  It would eliminate a dependency and allow us
>> to use other search path rules, like ones that could deal with this
>> case.  I think the situation would actually be better on other
>> architectures because we wouldn't have to deal with bugs like this one:
>>
>>   http://thread.gmane.org/gmane.lisp.guile.bugs/5269
>
> The problems we're having with libltdl are likely affecting many other
> projects.  Wouldn't it be better to fix these problems in libltdl,

Agreed.

On one hand, that will take more time than rolling our own.  On the
other, that will save us many headaches (incompatibility with previous
Guile versions, maintenance of our own stuff, proliferation of search
path rules, etc.)

Ludo’.

Re: About Guile crypto support

[Andy Wingo]

Howdy,

On Sat 09 Feb 2013 18:50, Mark H Weaver <mhw@netris.org> writes:

> Andy Wingo <wingo@pobox.com> writes:
>
>> I have the feeling that we should implement our own FOO function
>> without libBAR.
>
> Wouldn't it be better to fix these problems in libBAR, to the benefit
> of all its users, than for each of its users to duplicate its
> functionality within their own projects?

This is a nice description of the advantages of shared libraries in
general, but to be fair we should list the disadvantages as well:

  - Coordination cost (working with upstream on bugs/features)
  - QA cost (does the user have a bug-free libfoo or not?)
  - Runtime cost (another shared library)
  - Extensibility cost (does using the library prevent us from making
    extensions to functionality in the area that the library solves?)

So I think the discussion about external dependencies should be, "does
this dependency pay for itself?"  For libgc, gmp, libffi, and
libunistring, the answer for me is "yes, definitely".  For ltdl, my
instinct is "no".

Just MHO :)

Andy
-- 
http://wingolog.org/

Re: About Guile crypto support

[Thien-Thi Nguyen]

[-- Attachment #1: Type: text/plain, Size: 1070 bytes --]

() Andy Wingo <wingo@pobox.com>
() Mon, 11 Feb 2013 09:20:55 +0100

     - Coordination cost (working with upstream on bugs/features)

   For ltdl, my instinct is "no".

   Just MHO :)

I imagine no one likes to hear the curmudgeon spew, but i feel it is my
duty (to GNU, Guile, and my fellow programmers) to trot out a reminder:
Live by the snub, die by the snub.

When someone applies the same instinct a few years down the line, the
libltdl reimpls (multiple, one per Guile release) will most likely
suffer from the same pronouncement.  It will be a lot of work for
everyone involved to return to the true upstream from the usurper, and
that work will not be done gladly.

-- 
Thien-Thi Nguyen ..................................... GPG key: 4C807502
.                  NB: ttn at glug dot org is not me                   .
.                 (and has not been since 2007 or so)                  .
.                        ACCEPT NO SUBSTITUTES                         .
........... please send technical questions to mailing lists ...........

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: About Guile crypto support

[Mike Gran]

> I have the feeling that we should implement our own dynamic-link
> function without libltdl.  It would eliminate a dependency and allow us
> to use other search path rules, like ones that could deal with this
> case.  I think the situation would actually be better on other
> architectures because we wouldn't have to deal with bugs like this one:

Right now, the Guile project does a poor job of testing Guile on
multiple platforms.  Hydra does check that i686-linux, x86_64-linux
and x86_64-darwin compile, but, that's about the extent of multi-platform
testing.  (There are Hydra jobs for Cygwin and Solaris, but, they are
broken; and FreeBSD hasn't built successfully in a year or so on 2.0)

Guile has some talented hackers, and maybe they could write a better
dynamic linker than libtool, but, history suggests that Guile might
lose interest in maintaining such code for non-GNU systems.

-Mike

Re: About Guile crypto support

[Ludovic Courtès]

Mike Gran <spk121@yahoo.com> skribis:

> Right now, the Guile project does a poor job of testing Guile on
> multiple platforms.  Hydra does check that i686-linux, x86_64-linux
> and x86_64-darwin compile, but, that's about the extent of multi-platform
> testing.  (There are Hydra jobs for Cygwin and Solaris, but, they are
> broken; and FreeBSD hasn't built successfully in a year or so on 2.0)

Actually --without-threads builds work on FreeBSD, but there’s a known
problem in the tracker about threading there.

I see the Darwin machines at hydra.nixos.org are currently down, I’ll
ping the admins (Darwin builds have been fairly reliable over the
years.)

I admit that Solaris and Cygwin are more problematic, though I did some
work in the past to help with those.

> Guile has some talented hackers, and maybe they could write a better
> dynamic linker than libtool, but, history suggests that Guile might
> lose interest in maintaining such code for non-GNU systems.

I tend to agree.  This is typically where ltdl is supposed to help.

Ludo’.

Re: About Guile crypto support

[Nala Ginrut]

On Sat, 2013-02-09 at 16:12 +0100, Ludovic Courtès wrote:
> Hi,
> 
> Daniel Hartwig <mandyke@gmail.com> skribis:
> 
> > By the way, I very much like the conventions used in the GnuTLS
> > bindings.  The enums in particular make a lot of sense for a security
> > library, with the extreme type safety they provide.  I will pursue a
> > similar approach.
> 
> Yeah, I think it’s helpful.
> 
> > One question.  With the current state of FFI, do you think it matters
> > much whether the bulk of the bindings are done in C or FFI?
> 
> I think it depends on the amount of public C structs, enums, inlines,
> and constants, and how often they are changed.  When there are too many
> of them and they are subject to change, it might be easier to use C
> (though that can be worked around from the FFI by calling the C
> compiler, as in [0].)
> 
> My impression is that libgcrypt uses mostly opaque pointer types and has
> a stable API, so the using FFI should be just fine.
> 
> An issue with the FFI is distros where .la and .so files are only
> available in the -dev package, because then ‘dynamic-link’ won’t work
> unless that -dev package is installed (as recently discussed on
> guile-user.)unanimous
> 

This could be a real issue since almost all mainstream distros packaging
policy force *.so be put in -devel packages. Though openSUSE/debian adds
the exception for Guile, I believe it's so hard to do that for every
packages uses Guile. 
Considering Guile would exists in every GNU project (in principle), the
issue may break the packaging policy totally. 

@andy: But I do like to have our own dynamic-link without libltdl, which
will be interesting and a study chance for me . ;-)
And maybe it'll be blamed for reinventing wheels?

> Thanks,
> Ludo’.
> 
> [0] http://git.sv.gnu.org/cgit/libchop.git/tree/guile2/chop/internal.scm#n130
> 

Re: About Guile crypto support

[Greg Troxel]

[-- Attachment #1: Type: text/plain, Size: 1329 bytes --]


Nala Ginrut <nalaginrut@gmail.com> writes:

> On Sat, 2013-02-09 at 16:12 +0100, Ludovic Courtès wrote:
>> 
>> Daniel Hartwig <mandyke@gmail.com> skribis:
>> An issue with the FFI is distros where .la and .so files are only
>> available in the -dev package, because then ‘dynamic-link’ won’t work
>> unless that -dev package is installed (as recently discussed on
>> guile-user.)unanimous
>
> This could be a real issue since almost all mainstream distros packaging
> policy force *.so be put in -devel packages. Though openSUSE/debian adds
> the exception for Guile, I believe it's so hard to do that for every
> packages uses Guile. 
> Considering Guile would exists in every GNU project (in principle), the
> issue may break the packaging policy totally. 

(First, "all mainstream distros" is only talking about Linux.)

This .so=>devel does not make sense to me.   I thought the point was
that -devel split things that people who wanted to compile against the
package needed, but not things needed to run.  So if a .so is used by a
program that has been compiled, then it needs to be in the non-devel
package.  I would expect that .so generally belongs in the non-devel
package, and that the -devel package would have .a and .h.

FWIW, BSD packaging systems do not have this -devel notion

[-- Attachment #2: Type: application/pgp-signature, Size: 194 bytes --]

Re: About Guile crypto support

[Daniel Hartwig]

On 11 February 2013 23:23, Greg Troxel <gdt@ir.bbn.com> wrote:
> (First, "all mainstream distros" is only talking about Linux.)
>
> This .so=>devel does not make sense to me.   I thought the point was
> that -devel split things that people who wanted to compile against the
> package needed, but not things needed to run.  So if a .so is used by a
> program that has been compiled, then it needs to be in the non-devel
> package.  I would expect that .so generally belongs in the non-devel
> package, and that the -devel package would have .a and .h.
>
> FWIW, BSD packaging systems do not have this -devel notion

[Assuming a Debian-centric view.]

To be clear, the “.so” files shipped in -dev packages are just
symlinks.  The real “.so.X.Y” are shipped in the corresponding library
package, as makes sense.

Nala Ginrut wrote earlier:
> This could be a real issue since almost all mainstream distros packaging
> policy force *.so be put in -devel packages. Though openSUSE/debian adds
> the exception for Guile, I believe it's so hard to do that for every
> packages uses Guile.

What do you mean, “adds the exception for Guile”?  The guile-2.0-dev
package contains the same /symlink/ as other -dev packages do.  The real
.so is in guile-2.0-libs.  I do not see how that is different to any
other library/dev package pair.

Re: About Guile crypto support

[Nala Ginrut]

[-- Attachment #1: Type: text/plain, Size: 1681 bytes --]

在 2013-2-12 AM10:10，"Daniel Hartwig" <mandyke@gmail.com>写道：
>
> On 11 February 2013 23:23, Greg Troxel <gdt@ir.bbn.com> wrote:
> > (First, "all mainstream distros" is only talking about Linux.)
> >
> > This .so=>devel does not make sense to me.   I thought the point was
> > that -devel split things that people who wanted to compile against the
> > package needed, but not things needed to run.  So if a .so is used by a
> > program that has been compiled, then it needs to be in the non-devel
> > package.  I would expect that .so generally belongs in the non-devel
> > package, and that the -devel package would have .a and .h.
> >
> > FWIW, BSD packaging systems do not have this -devel notion
>
> [Assuming a Debian-centric view.]
>
> To be clear, the “.so” files shipped in -dev packages are just
> symlinks.  The real “.so.X.Y” are shipped in the corresponding library
> package, as makes sense.
>

Yes, I'm talking about this *.so link. Not .so.X.Y

> Nala Ginrut wrote earlier:
> > This could be a real issue since almost all mainstream distros packaging
> > policy force *.so be put in -devel packages. Though openSUSE/debian adds
> > the exception for Guile, I believe it's so hard to do that for every
> > packages uses Guile.
>
> What do you mean, “adds the exception for Guile”?

Put that link .so in guile rather than guile-devel is the exception I
mentioned. The regular packaging policy not allow it.

 The guile-2.0-dev
> package contains the same /symlink/ as other -dev packages do.  The real
> .so is in guile-2.0-libs.  I do not see how that is different to any
> other library/dev package pair.
>

[-- Attachment #2: Type: text/html, Size: 2143 bytes --]

Re: About Guile crypto support

[Daniel Hartwig]

On 12 February 2013 12:20, Nala Ginrut <nalaginrut@gmail.com> wrote:
> Put that link .so in guile rather than guile-devel is the exception I
> mentioned. The regular packaging policy not allow it.
>

[Again, referring only to Debian.]

Right.  This applies only to libguilereadline-v-18.so, not
libguile-2.0.so.  I had overlooked the former.

Debian policy explicitly says that packaging like this is allowed, as
some libraries and dynamic modules require it in their typical use.

It is true that this would be a /minor/ issue for a pure FFI approach,
as libgcrypt.so would not be available without the -dev package.  In
this case, the binary package for the /Guile module/ just depends on
libgcrypt-dev, no problem.

Anyway, to use FFI, extension, or a hybrid approach is a decision for
whoever actually begins to write these proper bindings.

Regards

Dynamic FFI vs Static FFI (was Re: About Guile crypto support)

[Mark H Weaver]

Greg Troxel <gdt@ir.bbn.com> writes:
> This .so=>devel does not make sense to me.   I thought the point was
> that -devel split things that people who wanted to compile against the
> package needed, but not things needed to run.

It seems to me that the dynamic FFI performs, at run time, the same jobs
that a C compiler performs at compile time.  If at some point we add
support for accessing preprocessor macros and type definitions (which
seems important), then we'll need the header files as well.  So it makes
sense to me that a full-featured dynamic FFI fundamentally depends on
information that's only available in the *-dev packages.

At some point, it might make sense to create a more static FFI that
works more like a C compiler does, splitting the job into compile-time
and run-time phases.  This static FFI would be strictly less powerful
than the dynamic FFI, in a similar sense to how syntactic record APIs
are less powerful than procedural ones.  However, the static FFI would
be sufficient in most cases, and would have some advantages.

Thoughts?

     Mark

Re: Dynamic FFI vs Static FFI (was Re: About Guile crypto support)

[Ludovic Courtès]

Mark H Weaver <mhw@netris.org> skribis:

> It seems to me that the dynamic FFI performs, at run time, the same jobs
> that a C compiler performs at compile time.  If at some point we add
> support for accessing preprocessor macros and type definitions (which
> seems important), then we'll need the header files as well.  So it makes
> sense to me that a full-featured dynamic FFI fundamentally depends on
> information that's only available in the *-dev packages.

I don’t think you would parse header files at run-time.  I’d rather do
this at macro expansion time.

Overall, I think it’s good to pay attention to what popular distros do,
while at the same time keeping in mind that we’re not bound by their
policies, and could very well come up with use cases that are hindered
by these policies.

> At some point, it might make sense to create a more static FFI that
> works more like a C compiler does, splitting the job into compile-time
> and run-time phases.  This static FFI would be strictly less powerful
> than the dynamic FFI, in a similar sense to how syntactic record APIs
> are less powerful than procedural ones.  However, the static FFI would
> be sufficient in most cases, and would have some advantages.

In my mind the “static FFI” is the C API, and the dynamic FFI is
(system foreign).

To me, the main advantage of the latter is its simplicity of use and
deployment.

Thanks,
Ludo’.

Re: Dynamic FFI vs Static FFI (was Re: About Guile crypto support)

[Mark H Weaver]

ludo@gnu.org (Ludovic Courtès) writes:

> Mark H Weaver <mhw@netris.org> skribis:
>
>> At some point, it might make sense to create a more static FFI that
>> works more like a C compiler does, splitting the job into compile-time
>> and run-time phases.  This static FFI would be strictly less powerful
>> than the dynamic FFI, in a similar sense to how syntactic record APIs
>> are less powerful than procedural ones.  However, the static FFI would
>> be sufficient in most cases, and would have some advantages.
>
> In my mind the “static FFI” is the C API, and the dynamic FFI is
> (system foreign).
>
> To me, the main advantage of the latter is its simplicity of use and
> deployment.

Okay, but here I'm using "Static FFI" to mean something very different
than the C API: I'm talking about a pure scheme-based API that would be
quite similar to the API our current dynamic FFI, except that a lot of
the work would be done at compilation time (probably during macro
expansion).

I don't care what terminology we use to describe it, but that's the idea
that I intended to start a discussion about.

    Mark

Re: Dynamic FFI vs Static FFI (was Re: About Guile crypto support)

[Ludovic Courtès]

Mark H Weaver <mhw@netris.org> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:
>
>> Mark H Weaver <mhw@netris.org> skribis:
>>
>>> At some point, it might make sense to create a more static FFI that
>>> works more like a C compiler does, splitting the job into compile-time
>>> and run-time phases.  This static FFI would be strictly less powerful
>>> than the dynamic FFI, in a similar sense to how syntactic record APIs
>>> are less powerful than procedural ones.  However, the static FFI would
>>> be sufficient in most cases, and would have some advantages.
>>
>> In my mind the “static FFI” is the C API, and the dynamic FFI is
>> (system foreign).
>>
>> To me, the main advantage of the latter is its simplicity of use and
>> deployment.
>
> Okay, but here I'm using "Static FFI" to mean something very different
> than the C API: I'm talking about a pure scheme-based API that would be
> quite similar to the API our current dynamic FFI, except that a lot of
> the work would be done at compilation time (probably during macro
> expansion).

Ah, OK, sorry for the confusion!

Ludo’.

Re: Dynamic FFI vs Static FFI (was Re: About Guile crypto support)

[William ML Leslie]

On 13 February 2013 05:24, Mark H Weaver <mhw@netris.org> wrote:
> Okay, but here I'm using "Static FFI" to mean something very different
> than the C API: I'm talking about a pure scheme-based API that would be
> quite similar to the API our current dynamic FFI, except that a lot of
> the work would be done at compilation time (probably during macro
> expansion).

Maybe even something like LuaJIT FFI or python's CFFI - that can parse
C code at compile time to produce the data structures for the dynamic
FFI?

-- 
William Leslie

Re: Dynamic FFI vs Static FFI (was Re: About Guile crypto support)

[Nala Ginrut]

On Thu, 2013-02-14 at 18:21 +1100, William ML Leslie wrote:
> On 13 February 2013 05:24, Mark H Weaver <mhw@netris.org> wrote:
> > Okay, but here I'm using "Static FFI" to mean something very different
> > than the C API: I'm talking about a pure scheme-based API that would be
> > quite similar to the API our current dynamic FFI, except that a lot of
> > the work would be done at compilation time (probably during macro
> > expansion).
> 
> Maybe even something like LuaJIT FFI or python's CFFI - that can parse
> C code at compile time to produce the data structures for the dynamic
> FFI?
> 

I think that would be better. 
The reason why I don't use FFI is that we lack of convenient C struct
parsing mechanism. Though it's not so hard to implement, designing
proper APIs maybe take sometime. 
Anyway, I'd like to see a prototype, or maybe I could provide a naive
one. ;-)