various packaging issues

various packaging issues

[Dave Love]

I was looking at making packages for HPC-related stuff I'm familiar with
and I'm wondering how to translate features of the rpm or dpkg packaging
that I understand.  I couldn't see answers in the doc or immediately
find examples; apologies if I've missed something obvious.

* I should specify a disjunction of licences like "BSD or GPLv2" rather
  than the conjunction, which seems to be represented as a list (though
  the doc doesn't say what a list represents).  I.e. in Fedora-ish
  rpm-speak, I have A or B, rather than C and D.  Is that possible?

* RPM specs commonly have multiple sources specified, e.g. source and
  doc or example tarballs, or configuration files of some kind which
  aren't patches.  Is there a way to do that?

* What if I effectively have multiple build systems?  For instance a
  python interface needs to be built with an explicit
  "python... setup.py" with an overall autotools build.  While I figured
  out how to build and install in that situation, at least for one
  python, it looks messy.  I wonder what the recommended way is,
  particularly to provide python2 and python3 modules, or for other
  things like Java or Perl components.

Re: various packaging issues

[Ludovic Courtès]

Hello Dave,

Glad to see another Guix user in HPC!

Dave Love <fx@gnu.org> skribis:

> * I should specify a disjunction of licences like "BSD or GPLv2" rather
>   than the conjunction, which seems to be represented as a list (though
>   the doc doesn't say what a list represents).  I.e. in Fedora-ish
>   rpm-speak, I have A or B, rather than C and D.  Is that possible?

We cannot specify it currently, so what we do is just provide a list of
license objects as the ‘license’ field and explain the meaning of the
list in a comment.

Clearly this can be improved, though we have to pay attention of
overengineering in this domain.

> * RPM specs commonly have multiple sources specified, e.g. source and
>   doc or example tarballs, or configuration files of some kind which
>   aren't patches.  Is there a way to do that?

The ‘source’ field can be only one thing.  However, you can always
specify additional origin objects as inputs.  There are multiple
examples of that.

> * What if I effectively have multiple build systems?  For instance a
>   python interface needs to be built with an explicit
>   "python... setup.py" with an overall autotools build.  While I figured
>   out how to build and install in that situation, at least for one
>   python, it looks messy.  I wonder what the recommended way is,
>   particularly to provide python2 and python3 modules, or for other
>   things like Java or Perl components.

Most of the time there’s one build system that drives the others, so
you’d just specify that one.

If you really need bits of one build system and bits of another, you
still have to choose the “main” one, and then you can manually pick
build phases from the other build system(s) you’re interested in.  I
think there are a few examples of that, like things that combine
‘glib-or-gtk-build-system’ with ‘python-build-system’.  That’s not a
great answer, but that’s what we have so far.

For Python 2 vs. 3, there’s the ‘package-with-python2’ procedure, which
you can see in (gnu packages python).

HTH!

Ludo’.

Re: various packaging issues

[Dave Love]

ludo@gnu.org (Ludovic Courtès) writes:

> Hello Dave,
>
> Glad to see another Guix user in HPC!

Not exactly, I'm afraid.  This is largely academic research as I have to
deal with and contribute to a typical mess of "module" stuff (which
isn't at all modular, or course).

[I'm surprised that something with HPC interest doesn't have
Openfabrics support, which is what I was looking at.]

> Dave Love <fx@gnu.org> skribis:
>
>> * I should specify a disjunction of licences like "BSD or GPLv2" rather
>>   than the conjunction, which seems to be represented as a list (though
>>   the doc doesn't say what a list represents).  I.e. in Fedora-ish
>>   rpm-speak, I have A or B, rather than C and D.  Is that possible?
>
> We cannot specify it currently, so what we do is just provide a list of
> license objects as the ‘license’ field and explain the meaning of the
> list in a comment.
>
> Clearly this can be improved, though we have to pay attention of
> overengineering in this domain.

It seems to me that you need to be able to treat licences similarly to
Fedora and Debian.  They're not obviously over-engineered, and the
treatment of licensing must be constrained by legalities, which may or
may not allow simple engineering :-/.

>> * RPM specs commonly have multiple sources specified, e.g. source and
>>   doc or example tarballs, or configuration files of some kind which
>>   aren't patches.  Is there a way to do that?
>
> The ‘source’ field can be only one thing.  However, you can always
> specify additional origin objects as inputs.  There are multiple
> examples of that.

Oh, thanks, but I still don't see how to use it.  I'd be grateful if
someone can point me to examples; I couldn't find any when looking for
packages I thought might have them.

>> * What if I effectively have multiple build systems?  For instance a
>>   python interface needs to be built with an explicit
>>   "python... setup.py" with an overall autotools build.  While I figured
>>   out how to build and install in that situation, at least for one
>>   python, it looks messy.  I wonder what the recommended way is,
>>   particularly to provide python2 and python3 modules, or for other
>>   things like Java or Perl components.
>
> Most of the time there’s one build system that drives the others, so
> you’d just specify that one.

Part of the issue is how to specify the dependencies^Winputs that a
build system would comprise.  I.e. in Fedora-land, you buildrequire
things like python-devel so that you have what's necessary, which might
change.

> If you really need bits of one build system and bits of another, you
> still have to choose the “main” one, and then you can manually pick
> build phases from the other build system(s) you’re interested in.  I
> think there are a few examples of that, like things that combine
> ‘glib-or-gtk-build-system’ with ‘python-build-system’.  That’s not a
> great answer, but that’s what we have so far.

OK, I'll look.  I did make it work by checking which version of python
was available, but it wasn't neat.

> For Python 2 vs. 3, there’s the ‘package-with-python2’ procedure, which
> you can see in (gnu packages python).

Yes, but that seems to deal with packages just for python modules,
rather than modules which are just a component of something bigger,
which would go in python2/python3 sub-packages in Fedora.  I guess there
are examples, but again I couldn't find one.

For what it's worth, packaging seems to be rather harder than suggested
by the propaganda, and looks as if it's often going to need Scheming
with the Guix framework, which is a bit of a barrier.  I realize I'm
very rusty, and never knew Guile.  (Just an observation from someone
leaping in, without meaning to be critical.)

> HTH!

At least a bit, thanks.  I'll look again and maybe send doc fragments.

Re: various packaging issues

[Ricardo Wurmus]

Dave Love <fx@gnu.org> writes:

>>> * RPM specs commonly have multiple sources specified, e.g. source and
>>>   doc or example tarballs, or configuration files of some kind which
>>>   aren't patches.  Is there a way to do that?
>>
>> The ‘source’ field can be only one thing.  However, you can always
>> specify additional origin objects as inputs.  There are multiple
>> examples of that.
>
> Oh, thanks, but I still don't see how to use it.  I'd be grateful if
> someone can point me to examples; I couldn't find any when looking for
> packages I thought might have them.

One example is “kentutils” in (gnu packages bioinformatics).  It has
“samtabix” among its native-inputs and these additional sources are
unpacked in a separate build phase “prepare-samtabix”.

-- 
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: various packaging issues

[Dave Love]

Ricardo Wurmus <rekado@elephly.net> writes:

> One example is “kentutils” in (gnu packages bioinformatics).  It has
> “samtabix” among its native-inputs and these additional sources are
> unpacked in a separate build phase “prepare-samtabix”.

Ah, thanks!

Has anyone done or considered a rough idiom conversion guide from RPM
specs (maybe also for dpkg, but that's less prescribed)?  I can see
possible problems, but it might be useful.

Re: various packaging issues

[Ludovic Courtès]

Hi,

Dave Love <fx@gnu.org> skribis:

> Ricardo Wurmus <rekado@elephly.net> writes:
>
>> One example is “kentutils” in (gnu packages bioinformatics).  It has
>> “samtabix” among its native-inputs and these additional sources are
>> unpacked in a separate build phase “prepare-samtabix”.
>
> Ah, thanks!
>
> Has anyone done or considered a rough idiom conversion guide from RPM
> specs (maybe also for dpkg, but that's less prescribed)?  I can see
> possible problems, but it might be useful.

Not that I know of, but why not.

Honestly it’s unclear to me that this would bring much to
<https://www.gnu.org/software/guix/manual/html_node/Defining-Packages.html>
and related sections.  Call it propaganda if you want, but I still find
this much simpler than Debian packaging (I’m not too familiar with RPM.)

Ludo’.

Re: various packaging issues

[Dave Love]

ludo@gnu.org (Ludovic Courtès) writes:

> Honestly it’s unclear to me that this would bring much to
> <https://www.gnu.org/software/guix/manual/html_node/Defining-Packages.html>
> and related sections.

I'm seeing it from the point of view of view of someone who's quite
experienced with Fedora and Debian packaging and has past experience of
Scheme, but not Guile, and doesn't have time to become an expert.  I'm
guessing I'm not so unusual, but maybe that's wrong.

Some things which are documented aren't very easy to spot.  For others
it seems as if you have to go looking for examples.  I think would help
considerably if they were compared as idioms explicitly with what you'd
do for Fedora, in particular (to the extent there are correspondences).
Although doing a simple program initially was quite easy, I've struggled
with a fairly straightforward library which is just the first step to
provide openfabrics interfaces.

I'll have a go at writing down a few things when I have a chance; they
may turn out to be based on misconceptions, but that's worth finding
out.  This is meant to be constructive, of course!

> Call it propaganda if you want,

Apologies, that wasn't badly; I'm used to any sort of
why-you-should-use-it material being called "propaganda" non-seriously,
even if I wrote it.

> but I still find
> this much simpler than Debian packaging (I’m not too familiar with RPM.)

-- 
The ultimate computing slogan is "Your Mileage May Vary".
  -- Richard O'Keefe

Re: various packaging issues

[Ludovic Courtès]

Dave Love <fx@gnu.org> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:
>
>> Honestly it’s unclear to me that this would bring much to
>> <https://www.gnu.org/software/guix/manual/html_node/Defining-Packages.html>
>> and related sections.
>
> I'm seeing it from the point of view of view of someone who's quite
> experienced with Fedora and Debian packaging and has past experience of
> Scheme, but not Guile, and doesn't have time to become an expert.  I'm
> guessing I'm not so unusual, but maybe that's wrong.
>
> Some things which are documented aren't very easy to spot.  For others
> it seems as if you have to go looking for examples.  I think would help
> considerably if they were compared as idioms explicitly with what you'd
> do for Fedora, in particular (to the extent there are correspondences).
> Although doing a simple program initially was quite easy, I've struggled
> with a fairly straightforward library which is just the first step to
> provide openfabrics interfaces.
>
> I'll have a go at writing down a few things when I have a chance; they
> may turn out to be based on misconceptions, but that's worth finding
> out.  This is meant to be constructive, of course!

Sure, that would be useful!  I may lack hindsight to determine what
should be added or restructured in the manual to be readily useful.

I wrote “Defining Packages” for an audience relatively new to packaging
(or so I thought) and to Scheme.  We updated it based on feedback, but a
fresh look at it can certainly lead to improvements.

Ludo’.

Re: various packaging issues

[Ludovic Courtès]

Dave Love <fx@gnu.org> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:

[...]

>> Dave Love <fx@gnu.org> skribis:
>>
>>> * I should specify a disjunction of licences like "BSD or GPLv2" rather
>>>   than the conjunction, which seems to be represented as a list (though
>>>   the doc doesn't say what a list represents).  I.e. in Fedora-ish
>>>   rpm-speak, I have A or B, rather than C and D.  Is that possible?
>>
>> We cannot specify it currently, so what we do is just provide a list of
>> license objects as the ‘license’ field and explain the meaning of the
>> list in a comment.
>>
>> Clearly this can be improved, though we have to pay attention of
>> overengineering in this domain.
>
> It seems to me that you need to be able to treat licences similarly to
> Fedora and Debian.  They're not obviously over-engineered, and the
> treatment of licensing must be constrained by legalities, which may or
> may not allow simple engineering :-/.

Sure, agreed.

>> For Python 2 vs. 3, there’s the ‘package-with-python2’ procedure, which
>> you can see in (gnu packages python).
>
> Yes, but that seems to deal with packages just for python modules,

Right, it stops at the first non-python-build-system dependency.  That
said, ‘package-mapping’ allows you to do arbitrary graph rewriting, so
that might be what you’re looking for.

Ludo’.

Re: various packaging issues

[Adonay Felipe Nogueira]

For the "package with multiple origins" scenario, provided *only one*
package needs the same set of origins, one could do something *similar*
to this:

--8<---------------cut here---------------start------------->8---
(define-public major-package
  (let ((extra-origin
	 (let ((extra-origin-release "1.1"))
	   (origin ORIGIN-BODY))))
    (package
     (name "major-package")
     (native-inputs
      `(("extra-origin" ,extra-origin)
      PACKAGE-NATIVE-INPUTS-REST))
     PACKAGE-BODY-REST)))               
--8<---------------cut here---------------end--------------->8---

This defines and exports the 'major-package symbol/variable as a public
part of the current module.

Inside 'major-package you have the local 'extra-origin variable/symbol
that is only available to the BODY part of (let ((VARIABLE1 INIT1) ...)
BODY).

I have also inserted another (let ((VARIABLE1 INIT1) ...) BODY) inside
'extra-origin, but if you don't need local varaibles/symbols inside the
'extra-origin, you can simply extract the BODY part of this (let
((VARIABLE1 INIT1) ...) BODY) and discard the rest.

Back to the BODY part of the topmost (let ((VARIABLE1 INIT1) ...) BODY),
I made the 'package definition of 'major-package. And assigned the local
'extra-origin as one of its 'native-inputs (I could have placed it
elsewhere, but I didn't test this example with 'inputs nor with
'propagated-inputs).

About packaging and Scheming: Interesting, I'm not a developer, but I
find the contrary. In fact, nowadays I try to study and code more in
Scheme and Emacs Lisp than in other programing languages, also because
most of the others have limitations. As you also said, it's a matter of
learning and patience, :).

-- 
- [[https://libreplanet.org/wiki/User:Adfeno]]
- Palestrante e consultor sobre /software/ livre (não confundir com
  gratis).
- "WhatsApp"? Ele não é livre, por isso não uso. Iguais a ele prefiro
  GNU Ring, ou Tox. Quer outras formas de contato? Adicione o vCard
  que está no endereço acima aos teus contatos.
- Pretende me enviar arquivos .doc, .ppt, .cdr, ou .mp3? OK, eu
  aceito, mas não repasso. Entrego apenas em formatos favoráveis ao
  /software/ livre. Favor entrar em contato em caso de dúvida.

Re: various packaging issues

[Adonay Felipe Nogueira]

About the license field notation, I don't know if this is indeed what
the GNU Guix project interpretes about it, but for me I always see a
list of licenses as being connected with a series of "or"'s.

One thing to note however, and I often find it simpler to do, is that:
at least in *my* opinion, the license field is used to list the license
in which the resulting package is under, not to list each and every
license found inside the package nor to list each and every license of
the package's dependencies/recommendations/suggestions.

-- 
- [[https://libreplanet.org/wiki/User:Adfeno]]
- Palestrante e consultor sobre /software/ livre (não confundir com
  gratis).
- "WhatsApp"? Ele não é livre, por isso não uso. Iguais a ele prefiro
  GNU Ring, ou Tox. Quer outras formas de contato? Adicione o vCard
  que está no endereço acima aos teus contatos.
- Pretende me enviar arquivos .doc, .ppt, .cdr, ou .mp3? OK, eu
  aceito, mas não repasso. Entrego apenas em formatos favoráveis ao
  /software/ livre. Favor entrar em contato em caso de dúvida.

Re: various packaging issues

[Dave Love]

Adonay Felipe Nogueira <adfeno@openmailbox.org> writes:

> About the license field notation, I don't know if this is indeed what
> the GNU Guix project interpretes about it, but for me I always see a
> list of licenses as being connected with a series of "or"'s.

That appears to be wrong, and certainly wouldn't be the usual case.

> One thing to note however, and I often find it simpler to do, is that:
> at least in *my* opinion, the license field is used to list the license
> in which the resulting package is under, not to list each and every
> license found inside the package nor to list each and every license of
> the package's dependencies/recommendations/suggestions.

In many cases the package isn't under a single licence -- that's the
point.  A few packages have a choice of licence, like MIT or GPL, but
they're relatively rare.  I wouldn't expect to list licences of all the
dependencies; certainly that's not what you do with Debian and Fedora
packaging.

It's important not to go by your (non-legal, I assume) opinion when it
come to legal matters like copyright.  For instance, I've had a GNU
contribution blocked because of Moglen's opinion on it when I thought
the situation was clear with respect to GPL enforcement as I'd
explicitly engineered it that way.

Re: various packaging issues

[Adonay Felipe Nogueira]

Regarding the meaning of the license list: I see what you mean
now. Thanks. :)

About own opinions and blocked contributions: Since you commented on a
contribution to a GNU project, I wonder if the issue was taken to the
FSF's Licensing and Compliance Lab and also to copyright-related
lawyers? Remember that the second one is usually the best option if they
are indeed lawyers known not to be biased towards non-free software
(this also includes some "open source" practices).

Optionally, if you really want to, we could discuss the blocked
contribution either privately or with the libreplanet-discuss mailing
list. Currently, the description you gave me seems to vague do draw any
consideration. Keep in mind that it's often a good idea to re-read what
was the resulting consideration of that blocking and perhaps even
discuss the points in more detail and with more people.

Re: various packaging issues

[Dave Love]

Adonay Felipe Nogueira <adfeno@openmailbox.org> writes:


> About own opinions and blocked contributions: Since you commented on a
> contribution to a GNU project, I wonder if the issue was taken to the
> FSF's Licensing and Compliance Lab and also to copyright-related
> lawyers? Remember that the second one is usually the best option if they
> are indeed lawyers known not to be biased towards non-free software
> (this also includes some "open source" practices).

I don't think the compliance lab existed in 2002, but it was a
properly-assigned contribution to GNU by an active maintainer.  I wasn't
complaining, just providing an example of what I thought was a strange
opinion on the defensibility of something specifically engineered for
legal reasons that wasn't even the first example in GNU; but I am not a
lawyer.

Sorry, I obviously shouldn't assume people know who Moglen is:
<https://softwarefreedom.org/about/team/#moglen>.

Re: various packaging issues

[Adonay Felipe Nogueira]

Yes, we do know who he is! :)

Still, that issue is somewhat not clear to me, neither is the submission
which started it all back in 2002.


-- 
- [[https://libreplanet.org/wiki/User:Adfeno]]
- Palestrante e consultor sobre /software/ livre (não confundir com
  gratis).
- "WhatsApp"? Ele não é livre. Por favor, use o GNU Ring ou o Tox.
- Contato: [[https://libreplanet.org/wiki/User:Adfeno#vCard]]
- Arquivos comuns aceitos (apenas sem DRM): Corel Draw, Microsoft
  Office, MP3, MP4, WMA, WMV.
- Arquivos comuns aceitos e enviados: CSV, GNU Dia, GNU Emacs Org, GNU
  GIMP, Inkscape SVG, JPG, LibreOffice (padrão ODF), OGG, OPUS, PDF
  (apenas sem DRM), PNG, TXT, WEBM.