More robust substitute*

More robust substitute*

[Andreas Enge]

[-- Attachment #1: Type: text/plain, Size: 2932 bytes --]

Hello,

in texlive, there are lots of scripts to be installed in share; thus, the 
patch-shebang phase does not catch them. I tried the following:

   (substitute* (find-files "." "")
     (("^#!/bin/sh") (string-append "#!" (which "bash"))))

This chokes on binary files, in this case on a "Compiled terminfo entry" 
with the following message:
...
In unknown file:
   ?: 1 [regexp-exec # ...]
In ice-9/boot-9.scm:
 106: 0 [#<procedure 2a43840 at ice-9/boot-9.scm:97:6 (thrown-k . args)> 
misc-error ...]

ice-9/boot-9.scm:106:20: In procedure #<procedure 2a43840 at 
ice-9/boot-9.scm:97:6 (thrown-k . args)>:
ice-9/boot-9.scm:106:20: string contains #\nul character: "\x1a\x01!
\x00%\x00!\x00\x82\x016\x01vt100|vt100-am|vt100am|DEC 
vt100\x00\x00\x01\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00P\x00\b\x00\x18\x00ÿÿÿÿÿÿ\x03\x00ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ!
\x005\x007\x00.\x01-\x00\x91\x00ÿÿÿÿÿÿk\x00S\x00\x95\x00ÿÿH\x00ÿÿÿÿ^\x00ÿÿ|
\x00ÿÿ\x89\x00ÿÿÿÿÿÿÿÿ#\x00(\x00ÿÿÿÿÿÿÿÿÿÿÿÿÅ\x00$\x01)\x01ÿÿÿÿ\x18\x01ÿÿ\x8d\x00ÿÿñ\x00õ\x00ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ\xa0\x00ÿÿÿÿÿÿÿÿÿÿ¦\x00ÿÿÿÿÿÿÿÿ²\x00ÿÿ¶\x00º\x00¾\x00ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ¢\x00ÿÿÿÿÿÿª\x00ÿÿÿÿÿÿ®\x00é\x00\x1c\x01ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ3\x01ÿÿÿÿÿÿU\x00ÿÿÿÿÿÿJ\x00b\x00ÿÿ\x80\x00ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿù\x00ÿÿÊ\x00Â\x00ÿÿ\x15\x01\x9e\x00æ\x00ÿÿ\x9b\x00ÿÿ\x99\x00ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿvt100|
vt100-am|vt100am|DEC 
vt100\x00\a\x00\x1b[5m\x00\x1b[1m\x00\x1b[H\x1b[2J\x00\r\x00\x1b[%i%p1%d;
%p2%dr\x00\b\x00\x1b[%p1%dD\x00\n"

Would there be a way to make this more robust? Or to obtain a "find-file" 
that returns only text files?

Andreas

[-- Attachment #2: Type: text/html, Size: 6401 bytes --]

Re: More robust substitute*

[Ludovic Courtès]

Hello!

Andreas Enge <andreas@enge.fr> skribis:

> in texlive, there are lots of scripts to be installed in share; thus, the 
> patch-shebang phase does not catch them.

Do you know why it doesn’t catch them?  The ‘patch-source-shebangs’
phase patches all the files found under “.”, recursively.  Did you by
any chance disable that phase?  Could you check in the log?

> I tried the following:
>
>    (substitute* (find-files "." "")
>      (("^#!/bin/sh") (string-append "#!" (which "bash"))))
>
> This chokes on binary files,

Yes, it’s not supposed to be used on binary files.

Instead, you should use the ‘patch-shebang’ procedure, which is
specifically written to gracefully handle that situation (see
guix/build/utils.scm).

HTH,
Ludo’.

Re: More robust substitute*

[Andreas Enge]

[-- Attachment #1: Type: text/plain, Size: 1445 bytes --]

Am Samstag, 2. Februar 2013 schrieb Ludovic Courtès:
> Andreas Enge <andreas@enge.fr> skribis:
> > in texlive, there are lots of scripts to be installed in share; thus,
> > the patch-shebang phase does not catch them.
> Do you know why it doesn’t catch them?  The ‘patch-source-shebangs’
> phase patches all the files found under “.”, recursively.

I am not speaking about patch-source-shebangs, but patch-shebangs. Texlive 
consists of three source packages. The first one contains the source for 
the binaries, the other two contain files to be installed under "share". I 
unpack these after the installation procedure and copy them to "share".

But if patch-source-shebangs works recursively, I should maybe unpack 
everything first and let it do its magic. I will give it a try.

In any case, I have a working texlive! It contains over 100000 files (that 
are symlinked from the user profile...) and takes over 3GB, but it works!

Actually, I think one does not need to symlink the files from the user 
profile; tex has its own way of finding files via the binary "kpsewhich" 
(linked from the user profile), which points directly to the nix store:
$ kpsewhich article.sty
/nix/store/2cc4xyivn5f52gywl5mnz6fi90bj24xh-texlive-2012/share/texmf-
dist/tex/latex/base/article.sty

So maybe by splitting into two or three packages, one could hide the data 
in the nix store. I will think about it.

Andreas

[-- Attachment #2: Type: text/html, Size: 4876 bytes --]

Re: More robust substitute*

[Ludovic Courtès]

Hello!

Andreas Enge <andreas@enge.fr> skribis:

> Am Samstag, 2. Februar 2013 schrieb Ludovic Courtès:
>> Andreas Enge <andreas@enge.fr> skribis:
>> > in texlive, there are lots of scripts to be installed in share; thus,
>> > the patch-shebang phase does not catch them.
>> Do you know why it doesn’t catch them?  The ‘patch-source-shebangs’
>> phase patches all the files found under “.”, recursively.
>
> I am not speaking about patch-source-shebangs, but patch-shebangs.

Ah right, the ‘patch-shebangs’ phase just looks at files in ‘bin’ and
‘sbin’ (see gnu-build-system.scm).

Perhaps you can add a phase somewhere that does along the lines of:

  (lambda* (#:key outputs #:allow-other-keys)
    (for-each (match-lambda
                ((_ . dir)
                 (for-each patch-shebang
                           (find-files (string-append dir "/share")
                                       ".*"))))
              outputs)
    #t)

> In any case, I have a working texlive! It contains over 100000 files (that 
> are symlinked from the user profile...) and takes over 3GB, but it works!

Woow, congratulations!  :-)

I’m not sure if that would help here, but do you know about
“multiple-output derivations”?  It’s used for Libtool for instance: the
‘outputs’ field there means that binaries go into one directory, and the
rest goes into another directory (you see them when typing ‘guix-build
libtool’).

So if there are several kinds of files, you may want to separate them in
different outputs.

> Actually, I think one does not need to symlink the files from the user 
> profile; tex has its own way of finding files via the binary "kpsewhich" 
> (linked from the user profile), which points directly to the nix store:
> $ kpsewhich article.sty
> /nix/store/2cc4xyivn5f52gywl5mnz6fi90bj24xh-texlive-2012/share/texmf-
> dist/tex/latex/base/article.sty
>
> So maybe by splitting into two or three packages, one could hide the data 
> in the nix store. I will think about it.

Hmm, intriguing.  :-)

Thanks!

Ludo’, coming back from FOSDEM.

Re: More robust substitute*

[Andreas Enge]

[-- Attachment #1: Type: text/plain, Size: 793 bytes --]

Am Sonntag, 3. Februar 2013 schrieb Ludovic Courtès:
> Ah right, the ‘patch-shebangs’ phase just looks at files in ‘bin’ and
> ‘sbin’ (see gnu-build-system.scm).

Indeed, I am unpacking the additional tarballs and using patch-source-
shebangs on them, so your suggestion was really useful.

> I’m not sure if that would help here, but do you know about
> “multiple-output derivations”?  It’s used for Libtool for instance: the
> ‘outputs’ field there means that binaries go into one directory, and the
> rest goes into another directory (you see them when typing ‘guix-build
> libtool’).
> 
> So if there are several kinds of files, you may want to separate them in
> different outputs.

See my other message - it is actually all done!

Andreas

[-- Attachment #2: Type: text/html, Size: 3903 bytes --]