CPAN-type thing: specifications, wishes, thoughts?

CPAN-type thing: specifications, wishes, thoughts?

[Paul Raccuglia]

Hi. Since the idea of creating something like CPAN is something folks
wants, I figure it would be good to collect as many thoughts and
information as possible in one place.

(I'd also appreciate links to relevant threads)

My thoughts, specifically, were for the client to be fairly similar in
function to apt-get. (from the user's perspective)
The core commands would be:

INSTALL package_name   -- queries a server (set in configuration
files), gets an absolute link to the package, and a list of
dependencies. Downloads the package, and then any dependencies, and
takes care of unpacking, verifying, compiling, storing, etc. anything
that needs to be done. The package would have to have meta-data
attached in some fashion.

REMOVE package_name -- just has to remove all the relevant files, and
clean up any references to the package in the storage system. May want
to check dependencies and alert the user.

UPDATE [package_name] -- could be called to check all packages (by not
specifying a package name) or to update a specific package. Warn the
user of any dependencies that could be broken.

My thought was that the package manager could, itself, be a package
(but, hopefully, one that would be included by default). I wouldn't
imagine that the current code base would need to be modified a whole
lot.

I was thinking that most of this project could be written in Guile.
Does that make sense?

Some thoughts I heard were:
using stowfs to handle storing the packages intelligently
use dorodango to handle most of the acquisition


For the server design:
I was envisioning a website to navigate packages (like
http://packages.debian.org/stable/). My thought is to do everything
over HTTP (seems like that would work well with Dorodango). Doesn't
seem like a hugely complicated issue?

Questions about the server:
How would the central repository be maintained? Give trusted
developers the ability to make changes to their personal packages at
will?
How will packages be nominated / selected for inclusion?


That's all I can think of, right now. I'm sure there's more but I want
to start a discussion. Thanks,
Paul R

Re: CPAN-type thing: specifications, wishes, thoughts?

[Noah Lavine]

Hello all,

I'm afraid this email is coming much later in the planning process
than it should, and quite possibly we won't be able to do any of this
for SoC, and that's fine with me. But I was thinking about what we
could do that would be a "killer feature" for Guile's CPAN - something
that isn't already done by apt-get, dorodango, and every other package
manager. One answer is that having a big collection of Guile packages
*is* a killer feature, but we could have an apt repository of those if
we wanted one.

I think the answer is that the killer feature for a large repository
of packages is having the ability to painlessly bundle a piece of
software you've been writing and all of its dependencies in one
easy-to-install thing for users. After all, it's easy when you're
developing on your own machine to download a bunch of packages and use
them to do whatever you need to do, but if you then want to send that
to other people, you've got to collect the source code (or at least
.go files) for all of them, put them in a folder, make sure the
load-path will work out, and then most importantly, do all of that
again every time a new version of one of your dependencies comes out.
I think the feature that is missing is the ability to say "take my
software and package it so that its only external dependency is
POSIX", or something similar.

The implementation of such a thing isn't especially deep, but I bet
there will be a lot of little things that need to be done just right
for it to work. I think this could be a part of a package manager that
also does the other things Paul was listing.

How does this idea sound?

Noah

On Tue, Apr 12, 2011 at 11:01 PM, Paul Raccuglia <praccu@gmail.com> wrote:
> Hi. Since the idea of creating something like CPAN is something folks
> wants, I figure it would be good to collect as many thoughts and
> information as possible in one place.
>
> (I'd also appreciate links to relevant threads)
>
> My thoughts, specifically, were for the client to be fairly similar in
> function to apt-get. (from the user's perspective)
> The core commands would be:
>
> INSTALL package_name   -- queries a server (set in configuration
> files), gets an absolute link to the package, and a list of
> dependencies. Downloads the package, and then any dependencies, and
> takes care of unpacking, verifying, compiling, storing, etc. anything
> that needs to be done. The package would have to have meta-data
> attached in some fashion.
>
> REMOVE package_name -- just has to remove all the relevant files, and
> clean up any references to the package in the storage system. May want
> to check dependencies and alert the user.
>
> UPDATE [package_name] -- could be called to check all packages (by not
> specifying a package name) or to update a specific package. Warn the
> user of any dependencies that could be broken.
>
> My thought was that the package manager could, itself, be a package
> (but, hopefully, one that would be included by default). I wouldn't
> imagine that the current code base would need to be modified a whole
> lot.
>
> I was thinking that most of this project could be written in Guile.
> Does that make sense?
>
> Some thoughts I heard were:
> using stowfs to handle storing the packages intelligently
> use dorodango to handle most of the acquisition
>
>
> For the server design:
> I was envisioning a website to navigate packages (like
> http://packages.debian.org/stable/). My thought is to do everything
> over HTTP (seems like that would work well with Dorodango). Doesn't
> seem like a hugely complicated issue?
>
> Questions about the server:
> How would the central repository be maintained? Give trusted
> developers the ability to make changes to their personal packages at
> will?
> How will packages be nominated / selected for inclusion?
>
>
> That's all I can think of, right now. I'm sure there's more but I want
> to start a discussion. Thanks,
> Paul R
>
>

Re: CPAN-type thing: specifications, wishes, thoughts?

[CRLF0710]

Something like the Planets of Racket-lang? Should be able to install
offline, though~

2011/4/18 Noah Lavine <noah.b.lavine@gmail.com>:
> Hello all,
>
> I'm afraid this email is coming much later in the planning process
> than it should, and quite possibly we won't be able to do any of this
> for SoC, and that's fine with me. But I was thinking about what we
> could do that would be a "killer feature" for Guile's CPAN - something
> that isn't already done by apt-get, dorodango, and every other package
> manager. One answer is that having a big collection of Guile packages
> *is* a killer feature, but we could have an apt repository of those if
> we wanted one.
>
> I think the answer is that the killer feature for a large repository
> of packages is having the ability to painlessly bundle a piece of
> software you've been writing and all of its dependencies in one
> easy-to-install thing for users. After all, it's easy when you're
> developing on your own machine to download a bunch of packages and use
> them to do whatever you need to do, but if you then want to send that
> to other people, you've got to collect the source code (or at least
> .go files) for all of them, put them in a folder, make sure the
> load-path will work out, and then most importantly, do all of that
> again every time a new version of one of your dependencies comes out.
> I think the feature that is missing is the ability to say "take my
> software and package it so that its only external dependency is
> POSIX", or something similar.
>
> The implementation of such a thing isn't especially deep, but I bet
> there will be a lot of little things that need to be done just right
> for it to work. I think this could be a part of a package manager that
> also does the other things Paul was listing.
>
> How does this idea sound?
>
> Noah
>
> On Tue, Apr 12, 2011 at 11:01 PM, Paul Raccuglia <praccu@gmail.com> wrote:
>> Hi. Since the idea of creating something like CPAN is something folks
>> wants, I figure it would be good to collect as many thoughts and
>> information as possible in one place.
>>
>> (I'd also appreciate links to relevant threads)
>>
>> My thoughts, specifically, were for the client to be fairly similar in
>> function to apt-get. (from the user's perspective)
>> The core commands would be:
>>
>> INSTALL package_name   -- queries a server (set in configuration
>> files), gets an absolute link to the package, and a list of
>> dependencies. Downloads the package, and then any dependencies, and
>> takes care of unpacking, verifying, compiling, storing, etc. anything
>> that needs to be done. The package would have to have meta-data
>> attached in some fashion.
>>
>> REMOVE package_name -- just has to remove all the relevant files, and
>> clean up any references to the package in the storage system. May want
>> to check dependencies and alert the user.
>>
>> UPDATE [package_name] -- could be called to check all packages (by not
>> specifying a package name) or to update a specific package. Warn the
>> user of any dependencies that could be broken.
>>
>> My thought was that the package manager could, itself, be a package
>> (but, hopefully, one that would be included by default). I wouldn't
>> imagine that the current code base would need to be modified a whole
>> lot.
>>
>> I was thinking that most of this project could be written in Guile.
>> Does that make sense?
>>
>> Some thoughts I heard were:
>> using stowfs to handle storing the packages intelligently
>> use dorodango to handle most of the acquisition
>>
>>
>> For the server design:
>> I was envisioning a website to navigate packages (like
>> http://packages.debian.org/stable/). My thought is to do everything
>> over HTTP (seems like that would work well with Dorodango). Doesn't
>> seem like a hugely complicated issue?
>>
>> Questions about the server:
>> How would the central repository be maintained? Give trusted
>> developers the ability to make changes to their personal packages at
>> will?
>> How will packages be nominated / selected for inclusion?
>>
>>
>> That's all I can think of, right now. I'm sure there's more but I want
>> to start a discussion. Thanks,
>> Paul R
>>
>>
>
>



-- 
CrLF.0710

Re: CPAN-type thing: specifications, wishes, thoughts?

[Tristan Colgate-McFarlane]

Something that works through a http proxy (if HTTP is going to be sued),
would be nice too. Failing that, an easy way of creating and using local
mirrors and repos would be really useful.

That's my pet hate with racket.

On Mon, 2011-04-18 at 15:56 +0800, CRLF0710 wrote: 
> Something like the Planets of Racket-lang? Should be able to install
> offline, though~

Re: CPAN-type thing: specifications, wishes, thoughts?

[Andreas Rottmann]

Paul Raccuglia <praccu@gmail.com> writes:

[ Note that I'm writing here with my dorodango developer hat on, not so
  much as a potential co-mentor. ]

>
> My thoughts, specifically, were for the client to be fairly similar in
> function to apt-get. (from the user's perspective)
> The core commands would be:
>
> INSTALL package_name   -- queries a server (set in configuration
> files), gets an absolute link to the package, and a list of
> dependencies. Downloads the package, and then any dependencies, and
> takes care of unpacking, verifying, compiling, storing, etc. anything
> that needs to be done. The package would have to have meta-data
> attached in some fashion.
>
> REMOVE package_name -- just has to remove all the relevant files, and
> clean up any references to the package in the storage system. May want
> to check dependencies and alert the user.
>
> UPDATE [package_name] -- could be called to check all packages (by not
> specifying a package name) or to update a specific package. Warn the
> user of any dependencies that could be broken.
>
This, in slight variation, is all already implemented in Dorodango; you
may want to have a look at its manual[0].

[0] http://home.gna.org/dorodango/manual/

> My thought was that the package manager could, itself, be a package
> (but, hopefully, one that would be included by default).
>
Dorodango is a package; the way you install it works as follows: you
download a tarball containing dorodango and its dependencies. After
unpacking the tarball, you run 'setup', which sets up the load path for
the implementation and invokes it on the doro.sps Scheme program, which
then proceeds to install dorodango and all of its dependencies.

> I wouldn't imagine that the current code base would need to be
> modified a whole lot.
>
I think the plan was to develop the package manager as an external
project, and then integrate it into Guile proper, once its "ready".  I
imagine Guile core would not need to be modified at all, perhaps modulo
some missing features, should that become necessary.

> I was thinking that most of this project could be written in Guile.
>
I think all of it can be written in Scheme, except for some low-level
code interfacing with things like libzip (if ZIP is adopted as a package
format).

> Does that make sense?
>
> Some thoughts I heard were:
> using stowfs to handle storing the packages intelligently
> use dorodango to handle most of the acquisition
>
From my (obviously strongly biased view), it would be preferable to
start with the current dorodango codebase, make sure it works well with
Guile (there should not be much work left), and work from there.

>
> For the server design: I was envisioning a website to navigate
> packages (like http://packages.debian.org/stable/). My thought is to
> do everything over HTTP (seems like that would work well with
> Dorodango). Doesn't seem like a hugely complicated issue?
>
I started on something like this (i.e. a web interface for dorodango
repositories), I'll put the little code I have online and notify you.

> Questions about the server:
> How would the central repository be maintained? Give trusted
> developers the ability to make changes to their personal packages at
> will?
>
This is indeed a good question; the way it works in Debian is that all
developers can make changes to any package (by doing a signed upload via
FTP), but there are social rules about when it is OK to upload a package
one is not the designated maintainer for; see Non-Maintainer Uploads
(NMU)[1]. There are also many collaboratively and team-maintained
packages where there's not a single maintainer.  Perhaps a similiar
strategy would work for Guile's repository as well.

[1] http://www.debian.org/doc/developers-reference/pkgs.html#nmu

> How will packages be nominated / selected for inclusion?
>
I'd imagine a discussion on guile-devel beforehand would work.

Regards, Rotty
-- 
Andreas Rottmann -- <http://rotty.yi.org/>

Re: CPAN-type thing: specifications, wishes, thoughts?

[Andreas Rottmann]

Noah Lavine <noah.b.lavine@gmail.com> writes:

> Hello all,
>
> I'm afraid this email is coming much later in the planning process
> than it should, and quite possibly we won't be able to do any of this
> for SoC, and that's fine with me. But I was thinking about what we
> could do that would be a "killer feature" for Guile's CPAN - something
> that isn't already done by apt-get, dorodango, and every other package
> manager. One answer is that having a big collection of Guile packages
> *is* a killer feature, but we could have an apt repository of those if
> we wanted one.
>
> I think the answer is that the killer feature for a large repository
> of packages is having the ability to painlessly bundle a piece of
> software you've been writing and all of its dependencies in one
> easy-to-install thing for users. After all, it's easy when you're
> developing on your own machine to download a bunch of packages and use
> them to do whatever you need to do, but if you then want to send that
> to other people, you've got to collect the source code (or at least
> .go files) for all of them, put them in a folder, make sure the
> load-path will work out, and then most importantly, do all of that
> again every time a new version of one of your dependencies comes out.
> I think the feature that is missing is the ability to say "take my
> software and package it so that its only external dependency is
> POSIX", or something similar.
>
Well, IIRC, it's not a feature that "dorodango does not have" ;-):
dordango's package container format (referred to as a "bundle") can
contain multiple packages.  So it is possible to pack all your
dependencies in a single ZIP, and install them in one go using that
bundle.  There's a bit of text about that in the dorodango manual, but
here is how it works practically:

% doro show-bundle my-app.zip # to view packages in the bundle
% doro install --bundle=my-app-full.zip my-app # install my-app and all dependencies

> The implementation of such a thing isn't especially deep, but I bet
> there will be a lot of little things that need to be done just right
> for it to work.
>
Support for this has to be present in the conception of the
package/bundle format, so indeed it's important that taking this feature
into account from the beginning.

> I think this could be a part of a package manager that also does the
> other things Paul was listing.
>
Indeed!

Regards, Rotty
-- 
Andreas Rottmann -- <http://rotty.yi.org/>

Re: CPAN-type thing: specifications, wishes, thoughts?

[Andreas Rottmann]

CRLF0710 <crlf0710@gmail.com> writes:

> Something like the Planets of Racket-lang? Should be able to install
> offline, though~
>
FWIW, dorodango supports local filesystem-based repositories, so you can
just mirror your favorite repository, and then install offline.

Regards, Rotty
-- 
Andreas Rottmann -- <http://rotty.yi.org/>

Re: CPAN-type thing: specifications, wishes, thoughts?

[Noah Lavine]

> Well, IIRC, it's not a feature that "dorodango does not have" ;-):
> dordango's package container format (referred to as a "bundle") can
> contain multiple packages.  So it is possible to pack all your
> dependencies in a single ZIP, and install them in one go using that
> bundle.  There's a bit of text about that in the dorodango manual, but
> here is how it works practically:
>
> % doro show-bundle my-app.zip # to view packages in the bundle
> % doro install --bundle=my-app-full.zip my-app # install my-app and all dependencies

Ah, that's very cool! I actually meant support for installing the
program without even dorodango installed - using only what comes with
it. But I imagine you could create some sort of bundle with the 'doro
install' code plus the app bundle.

Eventually you'd want cross-platform bundling support too, but that
gets more difficult. I imagine that software like LilyPond or GnuCash
would be interested in a convenient way to use new Guile modules.

> Support for this has to be present in the conception of the
> package/bundle format, so indeed it's important that taking this feature
> into account from the beginning.

I'm glad you've been thinking it through, then.

Noah

Re: CPAN-type thing: specifications, wishes, thoughts?

[Andreas Rottmann]

Andreas Rottmann <a.rottmann@gmx.at> writes:

>> I was thinking that most of this project could be written in Guile.
>>
> I think all of it can be written in Scheme, except for some low-level
> code interfacing with things like libzip (if ZIP is adopted as a package
> format).
>
Let me restate: *all* of it can/should be written in Scheme, as Guile
2.0 has a dynamic FFI which allows to build language bindings without
touching C code; however, there are some considerations when using the
dynamic FFI:

- You are essentially encoding the ABI of the library you are making a
  binding for into your Scheme code.  If the ABI changes, you need to
  adapt your Scheme code.

- Due to a bug in libltdl[0], it is currently not possible to specify
  the full path (including ABI designation, e.g. libzip.so.1 instead of
  libzip.so) when loading the shared library.  This means the shared
  library can change ABI without resulting in the Scheme code to fail in
  a clear way.

[0] http://lists.gnu.org/archive/html/bug-guile/2011-03/msg00138.html

Regards, Rotty
-- 
Andreas Rottmann -- <http://rotty.yi.org/>