bootstrapping SED-4.5, circular dependency?

bootstrapping SED-4.5, circular dependency?

[Jan Nieuwenhuizen]

Hi!

Now that we have successfully bootstrapped GuixSD without using Gcc,
Binutils, GNU C lib, I am working to remove other binary bootstrap
seeds; such as SED and GZIP.

I have bootstrapped GNU make and Bash and was looking into building
GZIP, only to find out that it depends on a SED feature that our
bootstrap-gash does not provide.

So, I figured that SED should be built before GZIP, but I found that
SED-4.5 depends on the same SED feature that GZIP needs...apparently
a circular dependency?

Here is what I get:
--8<---------------cut here---------------start------------->8---
configure flags: ("CONFIG_SHELL=/gnu/store/md6pc5frcqnmndgwmnwk3xkmh4bx71jc-bash-mesboot-4.4.23/bin/bash" "SHELL=/gnu/store/md6pc5frcqnmndgwmnwk3xkmh4bx71jc-bash-mesboot-4.4.23/bin/bash" "--prefix=/gnu/store/hy9dq7jhmnjsgbaq9rkg59h9ssgqqzhw-sed-mesboot-4.5" "--enable-fast-install" "--build=x86_64-unknown-linux-gnu")
Backtrace:
           5 (apply-smob/1 #<catch-closure 9e5260>)
In ice-9/boot-9.scm:
    705:2  4 (call-with-prompt _ _ #<procedure default-prompt-handle?>)
In ice-9/eval.scm:
    619:8  3 (_ #(#(#<directory (guile-user) a670a0>)))
In gash/commands/sed.scm:
   132:30  2 (sed . _)
In ice-9/boot-9.scm:
   222:17  1 (map1 ("/^.*\\/\\([^/][^/]*\\)\\/*$/{\n\t    s//\\1/\n?"))
In unknown file:
           0 (scm-error misc-error #f "~A" ("SED: command not supp?") ?)

ERROR: In procedure scm-error:
SED: command not supported: "/^.*\\/\\([^/][^/]*\\)\\/*$/{\n\t    s//\\1/\n\t    q\n\t  }\n\t  /^X\\/\\(\\/\\/\\)$/{\n\t    s//\\1/\n\t    q\n\t  }\n\t  /^X\\/\\(\\/\\).*/{\n\t    s//\\1/\n\t    q\n\t  }\n\t  s/.*/./; q"
--8<---------------cut here---------------end--------------->8---

The SED throwing this error is the minimal SED from Gash, that I
implemented in GNU Guile.

Do you know whether bootstrapping of GZIP and/or SED is supported?  Can
you advise me on my next step (looking to bootstrap an earlier SED --
which version? or an earlier GZIP -- which version?)

Thanks!
janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: bootstrapping SED-4.5, circular dependency?

[Assaf Gordon]

Hello,

On 2018-12-08 6:46 a.m., Jan Nieuwenhuizen wrote:
> Now that we have successfully bootstrapped GuixSD without using Gcc,
> Binutils, GNU C lib, I am working to remove other binary bootstrap
> seeds; such as SED and GZIP.
> 
> I have bootstrapped GNU make and Bash and was looking into building
> GZIP, only to find out that it depends on a SED feature that our
> bootstrap-gash does not provide.
> 
> So, I figured that SED should be built before GZIP, but I found that
> SED-4.5 depends on the same SED feature that GZIP needs...apparently
> a circular dependency?

Technically speaking, these kind of dependencies (e.g. needing SED
to build a package) arise from autotools, not directly from sed's build 
commands.


> Here is what I get:
[...]
> ERROR: In procedure scm-error:
> SED: command not supported: "/^.*\\/\\([^/][^/]*\\)\\/*$/{\n\t    s//\\1/\n\t    q\n\t  }\n\t  /^X\\/\\(\\/\\/\\)$/{\n\t    s//\\1/\n\t    q\n\t  }\n\t  /^X\\/\\(\\/\\).*/{\n\t    s//\\1/\n\t    q\n\t  }\n\t  s/.*/./; q"

The failed sed program is (transcribed from the above error message)
====
/^.*\/\([^/][^/]*\)\/*$/{
	    s//\1/
	    q
	  }
   /^X\/\(\/\/\)$/{
	    s//\1/
	    q
   }
   /^X\/\(\/\).*/{
	    s//\1/
	    q
   }
   s/.*/./; q
====

It is a very standard sed program, which should (and does) work on every
existing SED implementation (posix-compliant of course).

> The SED throwing this error is the minimal SED from Gash, that I
> implemented in GNU Guile.

I would recommend adjusting your sed implementation to support
these commands, because you'll encounter them in many packages
that use autotools.

> Do you know whether bootstrapping of GZIP and/or SED is supported? 

It is supported in the sense that almost any old sed implementation
can be used (e.g. you can build GNU sed if you only have BSD sed
or AIX sed or busybox sed).

> Can
> you advise me on my next step (looking to bootstrap an earlier SED --
> which version? or an earlier GZIP -- which version?)

Digging a bit further, I see this specific sed program originates
from autoconf, from the M4 code used to implement basename and dirname,
e.g. here:
https://git.savannah.gnu.org/cgit/autoconf.git/tree/lib/m4sugar/m4sh.m4#n922
https://git.savannah.gnu.org/cgit/autoconf.git/tree/lib/m4sugar/m4sh.m4#n970

So it means every autotools package that uses these macros (and there
are many) will cause the same issue.

The best solution is likely to implement these sed commands in your
sed implementation (if you need help in testing I'm happy to try,
though I know very little about guile).

Second best would likely be to investigate with autotools developers
how to avoid this. Perhaps have a specific flag to autoconf to skip
generation of these portability m4 macros, since you know guix's 
basename/dirname are good enough ?


As a last resort, I can think of a really ugly hack:
create a "fake sed" wrapper script in guile,
that checks if the given SED script is one that you don't implement
(e.g. the one above). If it is, since we know all it does is 
basename/dirname - just call a real dirname/basename instead of
using sed's regexes. Otherwise, pass the parameters to the real
guile-sed.
Put this guile-sed-wrapper in $PATH, and it might just work...


In a completely different approach, since you are trying to bootstrap:
building OpenBSD's sed binary is very easy (needs just a C compiler and 
works fine on GNU/Linux) - perhaps building it
would be faster as a first step to get a working sed, then use it
to build other packages (I can help with that if you want, send me an 
email).


regards,
  - assaf

Re: bootstrapping SED-4.5, circular dependency?

[Ludovic Courtès]

Hello,

Assaf Gordon <assafgordon@gmail.com> skribis:

> Digging a bit further, I see this specific sed program originates
> from autoconf, from the M4 code used to implement basename and dirname,
> e.g. here:
> https://git.savannah.gnu.org/cgit/autoconf.git/tree/lib/m4sugar/m4sh.m4#n922
> https://git.savannah.gnu.org/cgit/autoconf.git/tree/lib/m4sugar/m4sh.m4#n970

IIUC, ‘configure’ scripts first try to determine whether they have a
working ‘basename’ and ‘dirname’ with:

--8<---------------cut here---------------start------------->8---
if (basename -- /) >/dev/null 2>&1 && test "X`basename -- / 2>&1`" = "X/"; then
  as_basename=basename
else
  as_basename=false
fi

if (as_dir=`dirname -- /` && test "X$as_dir" = X/) >/dev/null 2>&1; then
  as_dirname=dirname
else
  as_dirname=false
fi
--8<---------------cut here---------------end--------------->8---

Jan, could it be that the ‘basename’ and ‘dirname’ implementations in
Gash fail this test?  It might be that sed is not needed if you get
these right.

Thanks,
Ludo’.

Re: bootstrapping SED-4.5, circular dependency?

[Jan Nieuwenhuizen]

Assaf Gordon writes:

Hello Assaf,

> Technically speaking, these kind of dependencies (e.g. needing SED
> to build a package) arise from autotools, not directly from sed's
> build commands.

Ah yes, course.  That is a big difference.

>> Here is what I get:
> [...]
>> ERROR: In procedure scm-error:
>> SED: command not supported: "/^.*\\/\\([^/][^/]*\\)\\/*$/{\n\t    s//\\1/\n\t    q\n\t  }\n\t  /^X\\/\\(\\/\\/\\)$/{\n\t    s//\\1/\n\t    q\n\t  }\n\t  /^X\\/\\(\\/\\).*/{\n\t    s//\\1/\n\t    q\n\t  }\n\t  s/.*/./; q"
>
> The failed sed program is (transcribed from the above error message)
> ====
> /^.*\/\([^/][^/]*\)\/*$/{
> 	    s//\1/
> 	    q
> 	  }
>   /^X\/\(\/\/\)$/{
> 	    s//\1/
> 	    q
>   }
>   /^X\/\(\/\).*/{
> 	    s//\1/
> 	    q
>   }
>   s/.*/./; q
> ====
>
> It is a very standard sed program, which should (and does) work on every
> existing SED implementation (posix-compliant of course).

Right, now I see.  I really only catered for plain `-e s,,,' commands.

> I would recommend adjusting your sed implementation to support
> these commands, because you'll encounter them in many packages
> that use autotools.

That's sounds like the right thing to do, thanks!

> Digging a bit further, I see this specific sed program originates
> from autoconf, from the M4 code used to implement basename and dirname,
> e.g. here:
> https://git.savannah.gnu.org/cgit/autoconf.git/tree/lib/m4sugar/m4sh.m4#n922
> https://git.savannah.gnu.org/cgit/autoconf.git/tree/lib/m4sugar/m4sh.m4#n970
>
> So it means every autotools package that uses these macros (and there
> are many) will cause the same issue.
>
> The best solution is likely to implement these sed commands in your
> sed implementation (if you need help in testing I'm happy to try,
> though I know very little about guile).

That is even more helpful!  While I still think we should support the
simple SED script above, there could be a problem with basename/dirname.
Solving that might even sidestep this SED feature.  Many options now!

> As a last resort, I can think of a really ugly hack:
> create a "fake sed" wrapper script in guile,
> that checks if the given SED script is one that you don't implement
> (e.g. the one above). If it is, since we know all it does is
> basename/dirname - just call a real dirname/basename instead of
> using sed's regexes. Otherwise, pass the parameters to the real
> guile-sed.
> Put this guile-sed-wrapper in $PATH, and it might just work...
>
>
> In a completely different approach, since you are trying to bootstrap:
> building OpenBSD's sed binary is very easy (needs just a C compiler
> and works fine on GNU/Linux) - perhaps building it
> would be faster as a first step to get a working sed, then use it
> to build other packages (I can help with that if you want, send me an
> email).

Many thanks, also for these two last resort options!  I'm sure we'll
manage to bootstrap SED.  I'll let you know how it goes.

Greetings,
janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: bootstrapping SED-4.5, circular dependency?

[Jan Nieuwenhuizen]

Ludovic Courtès writes:

Hello,

> Jan, could it be that the ‘basename’ and ‘dirname’ implementations in
> Gash fail this test?  It might be that sed is not needed if you get
> these right.

Ah, yes!  I have added some test to Gash and this

    basename -- /

failed, producing the empty string, where coreutils' basename returns
`/'.

Gash "simply" (there is som if'ing involved) calls Guile's basename,
which disagrees here and returns "" for "/".

I'm assuming that this is expected behaviour for Guile?... so I
special-cased this for Gash' basename.

Greetings,
janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: bootstrapping SED-4.5, circular dependency?

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 1005 bytes --]

Hi!

Jan Nieuwenhuizen <janneke@gnu.org> skribis:

> Ludovic Courtès writes:
>
> Hello,
>
>> Jan, could it be that the ‘basename’ and ‘dirname’ implementations in
>> Gash fail this test?  It might be that sed is not needed if you get
>> these right.
>
> Ah, yes!  I have added some test to Gash and this
>
>     basename -- /
>
> failed, producing the empty string, where coreutils' basename returns
> `/'.
>
> Gash "simply" (there is som if'ing involved) calls Guile's basename,
> which disagrees here and returns "" for "/".
>
> I'm assuming that this is expected behaviour for Guile?... so I
> special-cased this for Gash' basename.

Looking at
<http://pubs.opengroup.org/onlinepubs/9699919799/functions/basename.html>,
I think it’s a bug of Guile’s ‘basename’ (which is not implemented in
terms of libc’s ‘basename’.)

The fix might be as simple as the patch below, WDYT?

In the meantime Gash should work around the bug.

Thanks,
Ludo’.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 811 bytes --]

diff --git a/libguile/filesys.c b/libguile/filesys.c
index e1aeeed1b..c1cd50e21 100644
--- a/libguile/filesys.c
+++ b/libguile/filesys.c
@@ -1602,11 +1602,20 @@ SCM_DEFINE (scm_basename, "basename", 1, 1, 0,
   c_filename = scm_to_utf8_string (filename);
   scm_dynwind_free (c_filename);
 
+  if (strcmp (c_filename, "/") == 0
+      || strcmp (c_filename, "//") == 0)
+    /* As per
+       <http://pubs.opengroup.org/onlinepubs/9699919799/functions/basename.html>,
+       "/" and "//" are treated specially.  */
+    res = scm_from_utf8_string ("/");
+  else
+    {
       c_last_component = last_component (c_filename);
       if (!c_last_component)
         res = filename;
       else
         res = scm_from_utf8_string (c_last_component);
+    }
   scm_dynwind_end ();
 
   if (!SCM_UNBNDP (suffix) &&