bug#33300: hplip 3.18.9 contains non-free binary blobs

bug#33300: hplip 3.18.9 contains non-free binary blobs

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 2090 bytes --]

Hello,

The tarball of hplip 3.18.9 contains several .so files that it installs
as-is:

--8<---------------cut here---------------start------------->8---
ludo@ribbon ~/src/guix$ (cd /tmp; tar xf $(guix build -S hplip))
ludo@ribbon ~/src/guix$ find /tmp/hplip-3.18.9 -name \*.so
/tmp/hplip-3.18.9/prnt/plugins/hbpl1-arm32.so
/tmp/hplip-3.18.9/prnt/plugins/hbpl1-x86_64.so
/tmp/hplip-3.18.9/prnt/plugins/lj-x86_32.so
/tmp/hplip-3.18.9/prnt/plugins/hbpl1-arm64.so
/tmp/hplip-3.18.9/prnt/plugins/hbpl1-x86_32.so
/tmp/hplip-3.18.9/prnt/plugins/lj-arm64.so
/tmp/hplip-3.18.9/prnt/plugins/lj-x86_64.so
/tmp/hplip-3.18.9/prnt/plugins/lj-arm32.so
/tmp/hplip-3.18.9/prnt/hpcups/libImageProcessor-x86_32.so
/tmp/hplip-3.18.9/prnt/hpcups/libImageProcessor-x86_64.so
--8<---------------cut here---------------end--------------->8---

I tried removing them with a snippet (patch attached), but installation
eventually fails while trying to link against libImageProcessor, which
is now missing.

In <https://bugs.launchpad.net/hplip/+bug/1785230/> people suggest that
3.18.6 is the last known-good version.  Indeed that version does not
have the obnoxious libImageProcessor, so that should be fine.  It does
come with the non-free binary plug-ins, though these are just plugins so
removing them will be easier:

--8<---------------cut here---------------start------------->8---
ludo@ribbon ~/src/guix$ find /tmp/hplip-3.18.6/ -name \*.so
/tmp/hplip-3.18.6/prnt/plugins/hbpl1-arm32.so
/tmp/hplip-3.18.6/prnt/plugins/hbpl1-x86_64.so
/tmp/hplip-3.18.6/prnt/plugins/lj-x86_32.so
/tmp/hplip-3.18.6/prnt/plugins/hbpl1-arm64.so
/tmp/hplip-3.18.6/prnt/plugins/hbpl1-x86_32.so
/tmp/hplip-3.18.6/prnt/plugins/lj-arm64.so
/tmp/hplip-3.18.6/prnt/plugins/lj-x86_64.so
/tmp/hplip-3.18.6/prnt/plugins/lj-arm32.so
--8<---------------cut here---------------end--------------->8---

Thus, I propose to:

  1. Revert to 3.18.6 (we’ll upgrade if and when hplip becomes free
     again.)

  2. Add a snippet to remove the non-free plugins.

Thoughts?

Thanks,
Ludo’.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 1154 bytes --]

diff --git a/gnu/packages/cups.scm b/gnu/packages/cups.scm
index 4259648c69..c0ac365691 100644
--- a/gnu/packages/cups.scm
+++ b/gnu/packages/cups.scm
@@ -410,6 +410,19 @@ device-specific programs to convert and print many types of files.")
                '(begin
                   (substitute* "prnt/hpcups/genPCLm.cpp"
                     (("boolean") "bool"))
+
+                  ;; Starting from version 3.18.9, hplip comes with binary
+                  ;; blobs under prnt/hpcups and prnt/plugins.  Remove them.
+                  (for-each delete-file (find-files "." "\\.so$"))
+
+                  ;; This trick changes the behavior of the
+                  ;; 'install-data-hook' target so that it doesn't install the
+                  ;; binary blobs.
+                  (substitute* "Makefile.in"
+                    (("^UNAME =.*")
+                     "UNAME = free-software-only-thanks\n")
+                    (("prnt/hpcups/libImageProcessor-([[:graph:]]+)\\.so")
+                     ""))
                   #t))))
     (build-system gnu-build-system)
     (home-page "https://developers.hp.com/hp-linux-imaging-and-printing")

bug#33300: hplip 3.18.9 contains non-free binary blobs

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 4378 bytes --]

On Wed, Nov 07, 2018 at 11:19:35AM +0100, Ludovic Courtès wrote:
> Hello,
> 
> The tarball of hplip 3.18.9 contains several .so files that it installs
> as-is:
> 
> --8<---------------cut here---------------start------------->8---
> ludo@ribbon ~/src/guix$ (cd /tmp; tar xf $(guix build -S hplip))
> ludo@ribbon ~/src/guix$ find /tmp/hplip-3.18.9 -name \*.so
> /tmp/hplip-3.18.9/prnt/plugins/hbpl1-arm32.so
> /tmp/hplip-3.18.9/prnt/plugins/hbpl1-x86_64.so
> /tmp/hplip-3.18.9/prnt/plugins/lj-x86_32.so
> /tmp/hplip-3.18.9/prnt/plugins/hbpl1-arm64.so
> /tmp/hplip-3.18.9/prnt/plugins/hbpl1-x86_32.so
> /tmp/hplip-3.18.9/prnt/plugins/lj-arm64.so
> /tmp/hplip-3.18.9/prnt/plugins/lj-x86_64.so
> /tmp/hplip-3.18.9/prnt/plugins/lj-arm32.so
> /tmp/hplip-3.18.9/prnt/hpcups/libImageProcessor-x86_32.so
> /tmp/hplip-3.18.9/prnt/hpcups/libImageProcessor-x86_64.so
> --8<---------------cut here---------------end--------------->8---
> 
> I tried removing them with a snippet (patch attached), but installation
> eventually fails while trying to link against libImageProcessor, which
> is now missing.
> 
> In <https://bugs.launchpad.net/hplip/+bug/1785230/> people suggest that
> 3.18.6 is the last known-good version.  Indeed that version does not
> have the obnoxious libImageProcessor, so that should be fine.  It does
> come with the non-free binary plug-ins, though these are just plugins so
> removing them will be easier:
> 
> --8<---------------cut here---------------start------------->8---
> ludo@ribbon ~/src/guix$ find /tmp/hplip-3.18.6/ -name \*.so
> /tmp/hplip-3.18.6/prnt/plugins/hbpl1-arm32.so
> /tmp/hplip-3.18.6/prnt/plugins/hbpl1-x86_64.so
> /tmp/hplip-3.18.6/prnt/plugins/lj-x86_32.so
> /tmp/hplip-3.18.6/prnt/plugins/hbpl1-arm64.so
> /tmp/hplip-3.18.6/prnt/plugins/hbpl1-x86_32.so
> /tmp/hplip-3.18.6/prnt/plugins/lj-arm64.so
> /tmp/hplip-3.18.6/prnt/plugins/lj-x86_64.so
> /tmp/hplip-3.18.6/prnt/plugins/lj-arm32.so
> --8<---------------cut here---------------end--------------->8---
> 
> Thus, I propose to:
> 
>   1. Revert to 3.18.6 (we’ll upgrade if and when hplip becomes free
>      again.)

Not bad for an interm solution. Debian already has 3.18.10 packaged, and
based on their versioning scheme it still contains non-free artifacts.

> 
>   2. Add a snippet to remove the non-free plugins.
> 
> Thoughts?

Here's what I have right now:

;; Delete non-free blobs$
(delete-file "prnt/hpcups/libImageProcessor-x86_64.so")$
(delete-file "prnt/hpcups/libImageProcessor-x86_32.so")$
(delete-file-recursively "prnt/plugins")$
(substitute* "Makefile.am"$
  (("dist_printplugins_DATA") "# dist_printplugins_DATA"))

and a patch from debian to address imageprocessor:
  https://salsa.debian.org/printing-team/hplip/raw/debian/3.18.10+dfsg0-1/debian/patches/0025-Remove-all-ImageProcessor-functionality-which-is-clo.patch

> 
> Thanks,
> Ludo’.
> 

> diff --git a/gnu/packages/cups.scm b/gnu/packages/cups.scm
> index 4259648c69..c0ac365691 100644
> --- a/gnu/packages/cups.scm
> +++ b/gnu/packages/cups.scm
> @@ -410,6 +410,19 @@ device-specific programs to convert and print many types of files.")
>                 '(begin
>                    (substitute* "prnt/hpcups/genPCLm.cpp"
>                      (("boolean") "bool"))
> +
> +                  ;; Starting from version 3.18.9, hplip comes with binary
> +                  ;; blobs under prnt/hpcups and prnt/plugins.  Remove them.
> +                  (for-each delete-file (find-files "." "\\.so$"))
> +
> +                  ;; This trick changes the behavior of the
> +                  ;; 'install-data-hook' target so that it doesn't install the
> +                  ;; binary blobs.
> +                  (substitute* "Makefile.in"
> +                    (("^UNAME =.*")
> +                     "UNAME = free-software-only-thanks\n")
> +                    (("prnt/hpcups/libImageProcessor-([[:graph:]]+)\\.so")
> +                     ""))
>                    #t))))
>      (build-system gnu-build-system)
>      (home-page "https://developers.hp.com/hp-linux-imaging-and-printing")


-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#33300: hplip 3.18.9 contains non-free binary blobs

[Tobias Geerinckx-Rice]

Ludo',

How horrid.

Ludovic Courtès wrote:
> I tried removing them with a snippet (patch attached), but 
> installation
> eventually fails while trying to link against libImageProcessor, 
> which
> is now missing.

If I correctly read the Debian maintainer's post in the bug you 
linked[0], it's possible to revert only the libImageProcessor 
infec^Waddition. If it's all right with everyone, I'd like to give 
that a try first. Say by tomorrow? Or do you want to revert first 
& ask such questions later?

> +                  ;; This trick changes the behavior of the
> +                  ;; 'install-data-hook' target so that it 
> doesn't install the
> +                  ;; binary blobs.
> +                  (substitute* "Makefile.in"
> +                    (("^UNAME =.*")
> +                     "UNAME = free-software-only-thanks\n")

Nice. I wish it worked.

Aside, -ish: looks like most distributions there found out about 
this file due to some failing sanity check. Perhaps we could add 
our own, in ‘guix lint’ or at build time, to warn about ELF files 
and other suspicious binaries in post-snippet sourceballs?

No idea if it's worth the trouble/performance hit/false-positive 
rate, of course. That's for the ner^Wgods to decide.

Kind regards,

T G-R

[0]: https://bugs.launchpad.net/hplip/+bug/1785230/comments/6

bug#33300: hplip 3.18.9 contains non-free binary blobs

[Ludovic Courtès]

Hello!

Efraim Flashner <efraim@flashner.co.il> skribis:

> Here's what I have right now:
>
> ;; Delete non-free blobs$
> (delete-file "prnt/hpcups/libImageProcessor-x86_64.so")$
> (delete-file "prnt/hpcups/libImageProcessor-x86_32.so")$
> (delete-file-recursively "prnt/plugins")$

I’d suggest simply something along the lines of what I tried earlier:

              ;; Starting from version 3.18.9, hplip comes with binary
              ;; blobs under prnt/hpcups and prnt/plugins.  Remove them.
              (for-each delete-file (find-files "." "\\.so$"))

> (substitute* "Makefile.am"$
>   (("dist_printplugins_DATA") "# dist_printplugins_DATA"))

Rather “Makefile.in”, to avoid depending on Automake.

> and a patch from debian to address imageprocessor:
>   https://salsa.debian.org/printing-team/hplip/raw/debian/3.18.10+dfsg0-1/debian/patches/0025-Remove-all-ImageProcessor-functionality-which-is-clo.patch

With this patch we should be able to keep 3.18.9, so it looks better
than reverting.

Could you send a patch for this?  (If not Tobias said he can work on it
soon. :-)).

Thanks!

Ludo’.

bug#33300: hplip 3.18.9 contains non-free binary blobs

[Ludovic Courtès]

Hi!

Tobias Geerinckx-Rice <me@tobias.gr> skribis:

> Ludovic Courtès wrote:
>> I tried removing them with a snippet (patch attached), but
>> installation
>> eventually fails while trying to link against libImageProcessor,
>> which
>> is now missing.
>
> If I correctly read the Debian maintainer's post in the bug you
> linked[0], it's possible to revert only the libImageProcessor
> infec^Waddition. If it's all right with everyone, I'd like to give
> that a try first.

Indeed, the Debian patch Efraim linked to does exactly that.  So it
should be easy to solve.  Let’s see if Efraim or another one of us can
get it done soon!

> Aside, -ish: looks like most distributions there found out about this
> file due to some failing sanity check. Perhaps we could add our own,
> in ‘guix lint’ or at build time, to warn about ELF files and other
> suspicious binaries in post-snippet sourceballs?

Commit b17004f9f9541acbd07b45e35222e431427bfde0 added a -Wl,-rpath flag;
perhaps that was due to address an error in libImageProcessor.so
detected by ‘validate-runpath’?

That said, we could have a post-unpack phase that fails when ELF files
are found.  The problem is that there are exceptions, in particular
“yogurt software” (compilers, mostly).  So we’d have to manually fix
every exception.

> No idea if it's worth the trouble/performance hit/false-positive rate,
> of course. That's for the ner^Wgods to decide.

Yeah I wonder if it would be fruitful.

Ludo’.

bug#33300: Automatically detecting binaries in source tarballs

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 1722 bytes --]

Hi,

I think it would be good to have guix check for closed-source binaries after
unpacking, automatically (including jar files with class files in them).

Even when I know that they are there, I sometimes forget to delete them.  In
the long run it could even auto-delete those, but I guess only after a looong
time of integration.

> > Aside, -ish: looks like most distributions there found out about this
> > file due to some failing sanity check. Perhaps we could add our own,
> > in ‘guix lint’ or at build time, to warn about ELF files and other
> > suspicious binaries in post-snippet sourceballs?  

That would be great.

> Commit b17004f9f9541acbd07b45e35222e431427bfde0 added a -Wl,-rpath flag;
> perhaps that was due to address an error in libImageProcessor.so
> detected by ‘validate-runpath’?
> 
> That said, we could have a post-unpack phase that fails when ELF files
> are found.  The problem is that there are exceptions, in particular
> “yogurt software” (compilers, mostly).  So we’d have to manually fix
> every exception.
> 
> > No idea if it's worth the trouble/performance hit/false-positive rate,
> > of course. That's for the ner^Wgods to decide.  
> 
> Yeah I wonder if it would be fruitful.

Marking known-good binaries (whitelisting) is still better than hoping
we notice some closed-source binary (blacklisting).

It would be a conspicious reminder of what we still have to do - as
opposed to the situation now where it's mostly in someone's head
(if at all).

Once we finish the bootstrapping effort, the source tarballs won't
need to contain any binaries anymore anyway :)

I wonder just how many whitelist entries that would be, though.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

bug#33300: Automatically detecting binaries in source tarballs

[Ludovic Courtès]

Hello,

Danny Milosavljevic <dannym@scratchpost.org> skribis:

> I think it would be good to have guix check for closed-source binaries after
> unpacking, automatically (including jar files with class files in them).

Oh right, jars are certainly quite common, more than .so files.

>> > No idea if it's worth the trouble/performance hit/false-positive rate,
>> > of course. That's for the ner^Wgods to decide.  
>> 
>> Yeah I wonder if it would be fruitful.
>
> Marking known-good binaries (whitelisting) is still better than hoping
> we notice some closed-source binary (blacklisting).
>
> It would be a conspicious reminder of what we still have to do - as
> opposed to the situation now where it's mostly in someone's head
> (if at all).

Yeah, that makes sense.

What about adding such a phase in %standard-phases in core-updates-next?
I guess it could check for files that match ‘elf-file?’ or ‘ar-file?’
and for *.jar.  WDYT?

We must make add a keyword parameter in ‘gnu-build-system’ to make it
easy to disable it and/or to skip specific files.

Any takers?

Thanks,
Ludo’.

bug#33300: Automatically detecting binaries in source tarballs

[Björn Höfling]

[-- Attachment #1: Type: text/plain, Size: 1858 bytes --]

On Thu, 08 Nov 2018 09:50:23 +0100
ludo@gnu.org (Ludovic Courtès) wrote:

> Hello,
> 
> Danny Milosavljevic <dannym@scratchpost.org> skribis:
> 
> > I think it would be good to have guix check for closed-source
> > binaries after unpacking, automatically (including jar files with
> > class files in them).  
> 
> Oh right, jars are certainly quite common, more than .so files.
> 
> >> > No idea if it's worth the trouble/performance hit/false-positive
> >> > rate, of course. That's for the ner^Wgods to decide.    
> >> 
> >> Yeah I wonder if it would be fruitful.  
> >
> > Marking known-good binaries (whitelisting) is still better than
> > hoping we notice some closed-source binary (blacklisting).
> >
> > It would be a conspicious reminder of what we still have to do - as
> > opposed to the situation now where it's mostly in someone's head
> > (if at all).  
> 
> Yeah, that makes sense.
> 
> What about adding such a phase in %standard-phases in
> core-updates-next? I guess it could check for files that match
> ‘elf-file?’ or ‘ar-file?’ and for *.jar.  WDYT?
> 
> We must make add a keyword parameter in ‘gnu-build-system’ to make it
> easy to disable it and/or to skip specific files.

That is definitively a good idea.

One of my review-tasks is this:

[] Binaries included? If yes, created a snipped?
   find . -name "*.rar" -or -name "*.pdf" -or -name "*.bin" -or -name "*.pdf" -or -name "*.dsy" -or -name "*.jar" -or -name "*.exe" 

Should this be a phase of the build system? Or just a linter, that was
my first idea?

If it is a build-system-phase, it should probably go to core-updates
and beforehand someone must rebuild the world. I'm sure at least for
Java there are some JARs remaining and I had the plan to fold-packages
through them, but that had low priority.

Björn

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

bug#33300: Automatically detecting binaries in source tarballs

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 2377 bytes --]

On Fri, Nov 09, 2018 at 12:11:34AM +0100, Björn Höfling wrote:
> On Thu, 08 Nov 2018 09:50:23 +0100
> ludo@gnu.org (Ludovic Courtès) wrote:
> 
> > Hello,
> > 
> > Danny Milosavljevic <dannym@scratchpost.org> skribis:
> > 
> > > I think it would be good to have guix check for closed-source
> > > binaries after unpacking, automatically (including jar files with
> > > class files in them).  
> > 
> > Oh right, jars are certainly quite common, more than .so files.
> > 
> > >> > No idea if it's worth the trouble/performance hit/false-positive
> > >> > rate, of course. That's for the ner^Wgods to decide.    
> > >> 
> > >> Yeah I wonder if it would be fruitful.  
> > >
> > > Marking known-good binaries (whitelisting) is still better than
> > > hoping we notice some closed-source binary (blacklisting).
> > >
> > > It would be a conspicious reminder of what we still have to do - as
> > > opposed to the situation now where it's mostly in someone's head
> > > (if at all).  
> > 
> > Yeah, that makes sense.
> > 
> > What about adding such a phase in %standard-phases in
> > core-updates-next? I guess it could check for files that match
> > ‘elf-file?’ or ‘ar-file?’ and for *.jar.  WDYT?
> > 
> > We must make add a keyword parameter in ‘gnu-build-system’ to make it
> > easy to disable it and/or to skip specific files.
> 
> That is definitively a good idea.
> 
> One of my review-tasks is this:
> 
> [] Binaries included? If yes, created a snipped?
>    find . -name "*.rar" -or -name "*.pdf" -or -name "*.bin" -or -name "*.pdf" -or -name "*.dsy" -or -name "*.jar" -or -name "*.exe" 

also "*.so" or "*.a" I assume.

For python we'd want to grep the source files for "Generated by Cython"

> 
> Should this be a phase of the build system? Or just a linter, that was
> my first idea?

I'd go with a phase

> 
> If it is a build-system-phase, it should probably go to core-updates
> and beforehand someone must rebuild the world. I'm sure at least for
> Java there are some JARs remaining and I had the plan to fold-packages
> through them, but that had low priority.
> 
> Björn



-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#33300: Automatically detecting binaries in source tarballs

[Ludovic Courtès]

Hi,

Efraim Flashner <efraim@flashner.co.il> skribis:

> On Fri, Nov 09, 2018 at 12:11:34AM +0100, Björn Höfling wrote:

[...]

>> One of my review-tasks is this:
>> 
>> [] Binaries included? If yes, created a snipped?
>>    find . -name "*.rar" -or -name "*.pdf" -or -name "*.bin" -or -name "*.pdf" -or -name "*.dsy" -or -name "*.jar" -or -name "*.exe" 
>
> also "*.so" or "*.a" I assume.
>
> For python we'd want to grep the source files for "Generated by Cython"

Indeed.  The “risk”, if we make the list too long, is that we’ll find that
we’re not doing so well in many cases.  We should make sure we can
address all these issues.

>> Should this be a phase of the build system? Or just a linter, that was
>> my first idea?
>
> I'd go with a phase

Same here.  It’d be inconvenient to implement in ‘guix lint’ because
we’d first need to extract the tarball etc.

Thanks,
Ludo’.

bug#33300: hplip 3.18.9 contains non-free binary blobs

[Ludovic Courtès]

Closing this bug, which Efraim addressed in commit
b44b1f08f6945ea8370746cfdadb44c7dea9ea3e.

Thanks!

Ludo’.