[PATCH] gnu: Add Nmap.

[PATCH] gnu: Add Nmap.

["Taylan Ulrich Bayırlı/Kammer"]

Many distros separate Zenmap from the Nmap package.  Should we do the
same?  If yes, I'll need some help.  This recipe builds it together
with the rest of the suite.

===File /home/tub/media/src/guix/0001-gnu-Add-Nmap.patch====
From d2d36938b679818f42a66590f25766cfc321245f Mon Sep 17 00:00:00 2001
From: Taylan Ulrich B <taylanbayirli@gmail.com>
Date: Sat, 22 Nov 2014 22:47:26 +0100
Subject: [PATCH] gnu: Add Nmap.

* gnu/packages/nmap.scm: New file.
* gnu-system.am (GNU_SYSTEM_MODULES): Add nmap.scm.
---
 gnu-system.am         |  1 +
 gnu/packages/nmap.scm | 62 +++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 63 insertions(+)
 create mode 100644 gnu/packages/nmap.scm

diff --git a/gnu-system.am b/gnu-system.am
index d3b822c..a4094b2 100644
--- a/gnu-system.am
+++ b/gnu-system.am
@@ -186,6 +186,7 @@ GNU_SYSTEM_MODULES =				\
   gnu/packages/ncurses.scm			\
   gnu/packages/netpbm.scm			\
   gnu/packages/nettle.scm			\
+  gnu/packages/nmap.scm				\
   gnu/packages/node.scm				\
   gnu/packages/noweb.scm			\
   gnu/packages/ntp.scm				\
diff --git a/gnu/packages/nmap.scm b/gnu/packages/nmap.scm
new file mode 100644
index 0000000..2137ee7
--- /dev/null
+++ b/gnu/packages/nmap.scm
@@ -0,0 +1,62 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2014 Taylan Ulrich Bayırlı/Kammer <taylanbayirli@gmail.com>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (gnu packages nmap)
+  #:use-module (guix packages)
+  #:use-module (guix download)
+  #:use-module (guix build-system gnu)
+  #:use-module ((guix licenses) #:prefix l:)
+  #:use-module (gnu packages openssl)
+  #:use-module ((gnu packages admin) #:select (libpcap))
+  #:use-module (gnu packages pcre)
+  #:use-module (gnu packages lua)
+  #:use-module (gnu packages python))
+
+(define-public nmap
+  (package
+    (name "nmap")
+    (version "6.47")
+    (source (origin
+              (method url-fetch)
+              (uri (string-append "http://nmap.org/dist/nmap-" version
+                                  ".tar.bz2"))
+              (sha256
+               (base32
+                "14d53aji4was68c01pf105n5ylha257wmdbx40ddiqiw42g1x8cg"))))
+    (build-system gnu-build-system)
+    (inputs `(("openssl" ,openssl)
+              ("libpcap" ,libpcap)
+              ("pcre" ,pcre)
+              ("lua" ,lua)
+              ("python-2" ,python-2)
+              ;; FIXME: Add liblinear here once it's packaged.  (Nmap uses its
+              ;; own version when it can't find it.)
+              ))
+    (arguments
+     ;; Tests require network access, which build processes don't have.
+     `(#:tests? #f))
+    (synopsis "Network discovery and security auditing tool")
+    (description
+     "Nmap (\"Network Mapper\") is a network discovery and security auditing
+tool.  It is also useful for tasks such as network inventory, managing service
+upgrade schedules, and monitoring host or service uptime.  It also provides an
+advanced netcat implementation (ncat), a utility for comparing scan
+results (ndiff), a packet generation and response analysis tool (nping), and
+the Zenmap GUI and results viewer.")
+    (home-page "http://nmap.org/")
+    (license l:gpl2)))
-- 
2.1.2

============================================================

Re: [PATCH] gnu: Add Nmap.

[Andreas Enge]

On Sat, Nov 22, 2014 at 11:05:35PM +0100, "Taylan Ulrich Bayırlı/Kammer" wrote:
> Many distros separate Zenmap from the Nmap package.  Should we do the
> same?  If yes, I'll need some help.  This recipe builds it together
> with the rest of the suite.

Our general policy is to build as intended by upstream with as little changes
as possible. So unless there is a compelling reason to separate the two,
I would not do it.

Andreas

Re: [PATCH] gnu: Add Nmap.

[Taylan Ulrich Bayırlı/Kammer]

Andreas Enge <andreas@enge.fr> writes:

> On Sat, Nov 22, 2014 at 11:05:35PM +0100, "Taylan Ulrich Bayırlı/Kammer" wrote:
>> Many distros separate Zenmap from the Nmap package.  Should we do the
>> same?  If yes, I'll need some help.  This recipe builds it together
>> with the rest of the suite.
>
> Our general policy is to build as intended by upstream with as little
> changes as possible. So unless there is a compelling reason to
> separate the two, I would not do it.

Hm, zenmap would pull in X.Org and all sorts of GUI things, which would
be very annoying for someone who wants nmap on a headless server,
embedded system, or the like.

Well, I didn't need to add any such inputs, and it built zenmap anyway,
but of course it won't run without those.  Does that mean I made a
mistake, or is it fine that users have to know what run-time
dependencies to install for certain components of packages to work?

It seems Debian recently split of ndiff as well, after which the nmap
package itself won't depend on Python either, and only has very slim
dependencies.  (I *did* have to input python-2 for it to build ndiff and
zenmap...)

This topic seems to tie in well with Gammel Holte's recent post on the
ML regarding how optional run-time dependencies will be handled in Guix.
Maybe we should wait for the outcome of that.  Or at least I'll need to
read up on Nix/Guix dependency handling, because I'm confused on what
should and shouldn't be an input.

Taylan

Re: [PATCH] gnu: Add Nmap.

[Ludovic Courtès]

taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") skribis:

> Andreas Enge <andreas@enge.fr> writes:
>
>> On Sat, Nov 22, 2014 at 11:05:35PM +0100, "Taylan Ulrich Bayırlı/Kammer" wrote:
>>> Many distros separate Zenmap from the Nmap package.  Should we do the
>>> same?  If yes, I'll need some help.  This recipe builds it together
>>> with the rest of the suite.
>>
>> Our general policy is to build as intended by upstream with as little
>> changes as possible. So unless there is a compelling reason to
>> separate the two, I would not do it.
>
> Hm, zenmap would pull in X.Org and all sorts of GUI things, which would
> be very annoying for someone who wants nmap on a headless server,
> embedded system, or the like.

Indeed.

This is normally dealt with by using multiple outputs (info "(guix)
Packages with Multiple Outputs").  An example of that is Git: the Tcl
GUIs are moved to a separate output, and so is git-svn support, such
that the main output does not depend on Tcl, libx11, Subversion, etc.

Would it work here?

Thanks,
Ludo’.

Re: [PATCH] gnu: Add Nmap.

[Taylan Ulrich Bayırlı/Kammer]

ludo@gnu.org (Ludovic Courtès) writes:

> This is normally dealt with by using multiple outputs (info "(guix)
> Packages with Multiple Outputs").  An example of that is Git: the Tcl
> GUIs are moved to a separate output, and so is git-svn support, such
> that the main output does not depend on Tcl, libx11, Subversion, etc.

OK, will have a go at this.

It seems Zenmap doesn't need X11/GTK libraries (rather headers) at build
time because it only uses a Python GTK module.  This raises two general
questions for me:

1) Is it OK if users have to install additional packages for a given
   component of a package to work, or should all dependencies, even if
   purely run-time, be inputs?

2) If purely-run-time dependencies are inputs, won't that trigger
   unnecessary rebuilds of the package when a run-time dependency is
   updated?

(In this special case only question 2 applies, because the whole of
Zenmap is useless without GTK, not just a component of it.)

After some pondering, I would say:

1) There should be a way to run-time-depend on another package without
   it being a build input at all.  (The installation of these needn't be
   forced on the user, though in some cases like Zenmap it's senseless
   not to do so; we could have "dependencies" and "recommendations" and
   possibly more, like in Debian.)

2) When interface files of a dylib are needed during compilation of a
   static lang (e.g. C headers), a special for-building package should
   be used as input, thus the actual dylib can be updated without
   causing rebuilds.  (Until ABI compatibility breaks I guess.)

3) Similarly, when a program is needed purely at build-time, like Bash
   or SCons, a special for-building package should be used as input,
   thus the actual program can be updated without causing rebuilds.
   (The for-building package would be updated only when doing so will
   improve the builds, like when a newer GCC version optimizes better.)

4) The for-building packages in #3 should obviously not be installed on
   the user's machine (unless they build locally instead of using binary
   substitutes), meaning "build inputs" and "run-time dependencies" are
   fully orthogonal.

In the Nix manual I see the following, which possibly fixes #4 without
needing to separate build inputs from run-time dependencies in recipes:

"Runtime dependencies are found by scanning binaries for the hash parts
of Nix store paths (such as r8vvq9kq…). This sounds risky, but it works
extremely well."

Perhaps we already do that?  But otherwise, it seems like we could save
gigawatts of electricity over time with #1-3.  Please tell me if I'm
missing something obvious. :-)

Taylan

Re: [PATCH] gnu: Add Nmap.

[Ludovic Courtès]

taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") skribis:

> ludo@gnu.org (Ludovic Courtès) writes:
>
>> This is normally dealt with by using multiple outputs (info "(guix)
>> Packages with Multiple Outputs").  An example of that is Git: the Tcl
>> GUIs are moved to a separate output, and so is git-svn support, such
>> that the main output does not depend on Tcl, libx11, Subversion, etc.
>
> OK, will have a go at this.

Thanks, that seems like the best option.

> It seems Zenmap doesn't need X11/GTK libraries (rather headers) at build
> time because it only uses a Python GTK module.  This raises two general
> questions for me:
>
> 1) Is it OK if users have to install additional packages for a given
>    component of a package to work, or should all dependencies, even if
>    purely run-time, be inputs?

Yes.  (There may be rare exceptions, but this is not one of them.)

Here you want users to be able to run ‘zenmap’ (or whatever the command
is called) and have it Just Work, regardless of whether they happen to
have installed Python and Python-GTK as well.

So that means that the zenmap (or nmap) package basically needs to
“close over” these inputs, probably using ‘wrap-program’.

> 2) If purely-run-time dependencies are inputs, won't that trigger
>    unnecessary rebuilds of the package when a run-time dependency is
>    updated?

It’s rarely be “purely run-time”.  Normally, at the very least there’s a
configure script that makes sure at build-time that the dependencies are
satisfied, and a test suite that makes sure things work.

> After some pondering, I would say:
>
> 1) There should be a way to run-time-depend on another package without
>    it being a build input at all.

The whole functional approach things means that bindings are static
(again, there may be exceptions, but zenmap has nothing exceptional
here.)

> 2) When interface files of a dylib are needed during compilation of a
>    static lang (e.g. C headers), a special for-building package should
>    be used as input, thus the actual dylib can be updated without
>    causing rebuilds.  (Until ABI compatibility breaks I guess.)

You’re describing an imperative packaging system.  This is fine, but it
defeats the whole idea of functional packaging.  :-)

See <http://nixos.org/docs/papers.html> and
<http://gnu.org/s/guix/#documentation> for the rationale.

> 3) Similarly, when a program is needed purely at build-time, like Bash
>    or SCons, a special for-building package should be used as input,
>    thus the actual program can be updated without causing rebuilds.
>    (The for-building package would be updated only when doing so will
>    improve the builds, like when a newer GCC version optimizes better.)

Build tools are listed in ‘native-inputs’.

> "Runtime dependencies are found by scanning binaries for the hash parts
> of Nix store paths (such as r8vvq9kq…). This sounds risky, but it works
> extremely well."
>
> Perhaps we already do that?

Yes.  That means that run-time dependencies are a strict subset of
build-time dependencies.

I hope this clarifies things.  Otherwise let me know!

Thanks,
Ludo’.

Re: [PATCH] gnu: Add Nmap.

[Taylan Ulrich Bayırlı/Kammer]

ludo@gnu.org (Ludovic Courtès) writes:

> So that means that the zenmap (or nmap) package basically needs to
> “close over” these inputs, probably using ‘wrap-program’.

Thanks for the pointer; I'll look into 'wrap-program'.

>> 2) If purely-run-time dependencies are inputs, won't that trigger
>>    unnecessary rebuilds of the package when a run-time dependency is
>>    updated?
>
> It’s rarely be “purely run-time”.  Normally, at the very least there’s
> a configure script that makes sure at build-time that the dependencies
> are satisfied, and a test suite that makes sure things work.

You're right; now that I think of it, it's probably a bug that Nmap's
build process doesn't check for the presence of a GTK module.  I'll talk
to their developers to clarify.  (ML thread should pop up at the bottom
of http://seclists.org/nmap-dev/2014/q4/index.html soon.)

>> 1) There should be a way to run-time-depend on another package
>>    without it being a build input at all.
>
> The whole functional approach things means that bindings are static

That's a neat analogy. :-) However, we needn't recompile C code when
there's an update to some dynlang library the program uses, do we?

Then again, we want to re-run a potential test suite that might be using
that library.  I guess it's OK to be "conservative" for now, until One
Day™ we can be more precise and separate testing from building...

(If that day comes though, we better find a way to automatically discern
run-time-only inputs.  (Regarding test suites to be "run-time" too.)
Otherwise we'll have a lot to do.)

Also, configure scripts that check for such dynlang libraries' presence
might become a problem.  I imagine one could provide some for-building
version of said library that's merely of the same major version, to
"shut up" the configure script, since it only gets in the way in this
case.  (Complaining about the lack of a run-time dependency, at
build-time...  Yes, above I said *not* doing this is probably a bug in
Nmap's build process, but for us it would become an annoyance.)
Alternatively, the script might simply take a flag telling it to enable
a feature even if it can't find a run-time dependency; I don't how many
configure scripts have this.

>> 2) When interface files of a dylib are needed during compilation of a
>>    static lang (e.g. C headers), a special for-building package should
>>    be used as input, thus the actual dylib can be updated without
>>    causing rebuilds.  (Until ABI compatibility breaks I guess.)
>
> You’re describing an imperative packaging system.  This is fine, but
> it defeats the whole idea of functional packaging.  :-)
>
> See <http://nixos.org/docs/papers.html> and
> <http://gnu.org/s/guix/#documentation> for the rationale.

Same thing as above really.  Until the ABI of a dylib changes, updates
to it needn't ignite recompilation of C code which uses that lib; it
won't affect the compiler output anyway.  (Tell me if I'm technically
wrong; not an expert on C compilation etc.)  But again, test suites...

It does sound like breaking functional purity, but is it really so when
an input has *no* effect on a build process other than the test suite?
Once the test suite is separated, the real package and for-building
package become operationally equivalent under the given build process,
so providing one in place of the other would be a correctness preserving
optimization. :-)

>> 3) Similarly, when a program is needed purely at build-time, like Bash
>>    or SCons, a special for-building package should be used as input,
>>    thus the actual program can be updated without causing rebuilds.
>>    (The for-building package would be updated only when doing so will
>>    improve the builds, like when a newer GCC version optimizes better.)
>
> Build tools are listed in ‘native-inputs’.

Ah, thanks for clarifying.  (BTW 'native-inputs' seems undocumented.)

In that case, we should already be able to do at least this, no?  (The
other ideas are sadly high up in the air.)  It could have prevented
world-rebuilds upon the bash security patches, for example.  (Given it's
acceptable to use a buggy (even security-holy) Bash for some ./configure
scripts and the like, so long as it doesn't break specifically them.)

>> "Runtime dependencies are found by scanning binaries for the hash
>> parts of Nix store paths (such as r8vvq9kq…). This sounds risky, but
>> it works extremely well."
>>
>> Perhaps we already do that?
>
> Yes.  That means that run-time dependencies are a strict subset of
> build-time dependencies.

I see.  One confusion remains: wouldn't that mean it would drop a
dependency such as the Python GTK module in our case, because its hash
doesn't appear anywhere, and instead it's only referenced by module name
which is expected to be in some Python load path at run-time?


Thanks a lot and sorry for taking your time!  I might be dumping out
many dubious ideas.

Taylan

Re: [PATCH] gnu: Add Nmap.

[Ludovic Courtès]

taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") skribis:

> ludo@gnu.org (Ludovic Courtès) writes:

[...]

>> The whole functional approach things means that bindings are static
>
> That's a neat analogy. :-)

It’s not just an analogy, it’s really what happens.

> However, we needn't recompile C code when there's an update to some
> dynlang library the program uses, do we?

Yes we do.

> Also, configure scripts that check for such dynlang libraries' presence
> might become a problem.  I imagine one could provide some for-building
> version of said library that's merely of the same major version, to
> "shut up" the configure script, since it only gets in the way in this
> case.  (Complaining about the lack of a run-time dependency, at
> build-time...  Yes, above I said *not* doing this is probably a bug in
> Nmap's build process, but for us it would become an annoyance.)
> Alternatively, the script might simply take a flag telling it to enable
> a feature even if it can't find a run-time dependency; I don't how many
> configure scripts have this.

Most of the time, packages check at configure-time whether their
perquisites are available.  Then, their test suite use these specific
dependencies.  Finally, in the case of DSOs, the RUNPATH makes sure that
these specific dependencies are used at run time.

Then there’s the case of dynamic languages with no RUNPATH equivalent.
In that case we arrange with ‘propagated-inputs’ or with wrappers to
make sure the “right” dependencies are used at run time.

I’m not sure if that answers your concerns, does it?

> Same thing as above really.  Until the ABI of a dylib changes, updates
> to it needn't ignite recompilation of C code which uses that lib; it
> won't affect the compiler output anyway.  (Tell me if I'm technically
> wrong; not an expert on C compilation etc.)

At the very least, the RUNPATH will be different.

But then, keep in mind we’re aiming for complete reproducibility.  So we
can’t rely on things like “oh, developers say this version is
ABI-compatible with the previous one.”

> It does sound like breaking functional purity, but is it really so when
> an input has *no* effect on a build process other than the test suite?

That’s an important effect: it means that the test suite is known to
work this particular input.

> Once the test suite is separated, the real package and for-building
> package become operationally equivalent under the given build process,
> so providing one in place of the other would be a correctness preserving
> optimization. :-)

Yeah, I see where you’re going.  :-)  Eelco Dolstra’s PhD thesis on Nix
had that idea of “equivalence classes”.  It’s not very practical, though.

> In that case, we should already be able to do at least this, no?  (The
> other ideas are sadly high up in the air.)  It could have prevented
> world-rebuilds upon the bash security patches, for example.  (Given it's
> acceptable to use a buggy (even security-holy) Bash for some ./configure
> scripts and the like, so long as it doesn't break specifically them.)

See “Security Updates” in the manual for that.

>>> "Runtime dependencies are found by scanning binaries for the hash
>>> parts of Nix store paths (such as r8vvq9kq…). This sounds risky, but
>>> it works extremely well."
>>>
>>> Perhaps we already do that?
>>
>> Yes.  That means that run-time dependencies are a strict subset of
>> build-time dependencies.
>
> I see.  One confusion remains: wouldn't that mean it would drop a
> dependency such as the Python GTK module in our case, because its hash
> doesn't appear anywhere, and instead it's only referenced by module name
> which is expected to be in some Python load path at run-time?

Correct.  That’s why we’d either use ‘propagated-inputs’ or
‘wrap-program’, depending on the situation.

I hope this clarifies things.

So, where were we with this nmap patch?  :-)

Ludo’.

Re: [PATCH] gnu: Add Nmap.

[Taylan Ulrich Bayırlı/Kammer]

ludo@gnu.org (Ludovic Courtès) writes:

> [large snip]
>
> I hope this clarifies things.

Yes!  Thanks for your patience.  It now dawns on me that our primary aim
isn't byte-equal build process outputs (in terms of file contents); it's
"immutable run-time behavior" of a package.  That is to say, if
/gnu/store/...-foo/bin/foo works fine today, it should work fine any
day; it's unacceptable to let it load a different .so than its
designated one, unless we define some very strict criteria for the .so
and prove that a replacement satisfies them...  Similarly for some
Python module the program uses.  I was chasing a "reproducible builds"
red herring.

That being said, I still think it's wasteful that C code gets recompiled
when there's an update to a dynamic component.

A rough idea for the future: save some intermediate build output (say
from the 'build' phase), then when there's an update to a run-time
component such as a DSO or dynlang module, make a new package that
merely receives some tweaks to the saved intermediate output, such as
fixing RUNPATH, re-patching shebang lines, etc.  Users will still need
to download these new packages; that in turn could be improved via
binary diffs one day.

> So, where were we with this nmap patch?  :-)

Heh, indeed.

I thought I'd wait for feedback from the Nmap ML but I guess that's not
necessary.  I'll continue working on it tomorrow.

Taylan

Re: [PATCH] gnu: Add Nmap.

[Eric Bavier]

[-- Attachment #1: Type: text/plain, Size: 757 bytes --]

On Tue, Nov 25, 2014 at 5:25 PM, Taylan Ulrich Bayırlı/Kammer <
taylanbayirli@gmail.com> wrote:

>
> That being said, I still think it's wasteful that C code gets recompiled
> when there's an update to a dynamic component.
>
> A rough idea for the future: save some intermediate build output (say
> from the 'build' phase)
>

I added the ccache package a little while ago.  I had wondered whether it
would be possible to run all our builds through ccache so that updates to
core packages might not be as much of a time burden (at the cost of some
extra disk space).  Might be something interesting to try.

`~Eric

-- 
Please avoid sending me Word or PowerPoint attachments.
See http://www.gnu.org/philosophy/no-word-attachments.html

[-- Attachment #2: Type: text/html, Size: 1244 bytes --]