* bug#63024: Crash during `guix import pypi -r' @ 2023-04-22 19:59 Timo Wilken 2023-04-24 8:33 ` Simon Tournier 0 siblings, 1 reply; 5+ messages in thread From: Timo Wilken @ 2023-04-22 19:59 UTC (permalink / raw) To: 63024 The `guix import pypi -r ...' command frequently crashes for me. When it does, it always crashes on the third download, the one for the first dependency's source distribution, like so: --8<---------------cut here---------------start------------->8--- $ guix import pypi -r calcure Starting download of /tmp/guix-file.7aaMby From https://files.pythonhosted.org/packages/61/e3/c960430852d4322d42e02421690958f47fa2b42aaf4f3befb25c1c8baf21/calcure-2.8.6.tar.gz... …8.6.tar.gz 42KiB 2.1MiB/s 00:00 [##################] 100.0% Starting download of /tmp/guix-file.mPnbsd From https://files.pythonhosted.org/packages/b5/a8/8ba240c952a13b3b2bee64356dc1a1515c496100dc4c9edb6a6648927864/calcure-2.8.6-py3-none-any.whl... …y3-none-any.whl 49KiB 1.9MiB/s 00:00 [##################] 100.0% Starting download of /tmp/guix-file.XzIPz0 From https://files.pythonhosted.org/packages/24/91/77edc7626400922d44019753c39e0fd911176ce3f77c3f85ac832d5d70b7/holidays-0.23.tar.gz... In procedure connect*: Connection timed out Starting download of /tmp/guix-file.XzIPz0 From https://web.archive.org/web/20230422212928/https://files.pythonhosted.org/packages/24/91/77edc7626400922d44019753c39e0fd911176ce3f77c3f85ac832d5d70b7/holidays-0.23.tar.gz... In procedure connect*: Connection timed out Trying to use Disarchive to assemble /tmp/guix-file.XzIPz0... could not find its Disarchive specification failed to download "/tmp/guix-file.XzIPz0" from "https://files.pythonhosted.org/packages/24/91/77edc7626400922d44019753c39e0fd911176ce3f77c3f85ac832d5d70b7/holidays-0.23.tar.gz" Backtrace: 10 (primitive-load "/home/timo/.config/guix/current/bin/gu…") In guix/ui.scm: 2300:7 9 (run-guix . _) 2263:10 8 (run-guix-command _ . _) In guix/scripts/import.scm: 89:11 7 (guix-import . _) In ice-9/boot-9.scm: 1752:10 6 (with-exception-handler _ _ #:unwind? _ # _) In guix/scripts/import/pypi.scm: 97:21 5 (_) In guix/import/utils.scm: 638:3 4 (recursive-import _ #:repo->guix-package _ #:guix-name . #) 600:31 3 (topological-sort _ #<procedure 7f668bf7a8d0 at guix/i…> …) In srfi/srfi-1.scm: 586:17 2 (map1 (("holidays" #f) ("jdatetime" #f) ("ics" #f))) In guix/import/utils.scm: 630:33 1 (lookup-node "holidays" #f) In ice-9/boot-9.scm: 1685:16 0 (raise-exception _ #:continuable? _) ice-9/boot-9.scm:1685:16: In procedure raise-exception: Wrong number of values returned to continuation (expected 2) --8<---------------cut here---------------end--------------->8--- However, that URL is valid -- running a manual `guix download' afterwards works fine: --8<---------------cut here---------------start------------->8--- $ guix download 'https://files.pythonhosted.org/packages/24/91/77edc7626400922d44019753c39e0fd911176ce3f77c3f85ac832d5d70b7/holidays-0.23.tar.gz' Starting download of /tmp/guix-file.vsuTGB From https://files.pythonhosted.org/packages/24/91/77edc7626400922d44019753c39e0fd911176ce3f77c3f85ac832d5d70b7/holidays-0.23.tar.gz... ….23.tar.gz 374KiB 2.5MiB/s 00:00 [##################] 100.0% /gnu/store/jk0v685hhwirl16la349c7i5is640181-holidays-0.23.tar.gz 0yxs1lzbqykl25yf1hcrqjb7qqw69f2zgf7991slnkmk98p1bsxw --8<---------------cut here---------------end--------------->8--- ^ permalink raw reply [flat|nested] 5+ messages in thread
* bug#63024: Crash during `guix import pypi -r' 2023-04-22 19:59 bug#63024: Crash during `guix import pypi -r' Timo Wilken @ 2023-04-24 8:33 ` Simon Tournier 0 siblings, 0 replies; 5+ messages in thread From: Simon Tournier @ 2023-04-24 8:33 UTC (permalink / raw) To: Timo Wilken, 63024 Hi, On Sat, 22 Apr 2023 at 21:59, "Timo Wilken" <guix@twilken.net> wrote: > The `guix import pypi -r ...' command frequently crashes for me. When > it does, it always crashes on the third download, the one for the > first dependency's source distribution, like so: Thanks for the report. I am merging with #62334 [1]. 1: http://issues.guix.gnu.org/issue/62334 Cheers, simon ^ permalink raw reply [flat|nested] 5+ messages in thread
* bug#62334: Network is unreachable only for recursive pypi import @ 2023-03-21 14:17 Greg Hogan 2023-05-02 21:48 ` bug#63024: Crash during `guix import pypi -r' Ludovic Courtès 0 siblings, 1 reply; 5+ messages in thread From: Greg Hogan @ 2023-03-21 14:17 UTC (permalink / raw) To: 62334 The following package import fails when importing the dependency, but succeeds when directly importing that dependency. I can also manually download the tar without issue. --8<---------------cut here---------------start------------->8--- $ guix describe Generation 38 Mar 21 2023 13:38:52 (current) guix 38b64d4 repository URL: https://git.savannah.gnu.org/git/guix.git branch: master commit: 38b64d47ed3dfaeb63b859e7a8834e477ffed3a1 --8<---------------cut here---------------end--------------->8--- --8<---------------cut here---------------start------------->8--- $ guix import pypi -r cdlib Starting download of /tmp/guix-file.oeSQon From https://files.pythonhosted.org/packages/eb/6d/97167dce848b65023a272e2ffd04b2e462612efdb3538d16e2b8b2221a15/cdlib-0.2.6.tar.gz... ….6.tar.gz 191KiB 41.0MiB/s 00:00 ▕██████████████████▏ 100.0% Starting download of /tmp/guix-file.H0dsHq From https://files.pythonhosted.org/packages/56/37/80bdc21fbb88d87bca352cdf1ea0b9e2492737dd5775f32a04ea242f842e/cdlib-0.2.6-py3-none-any.whl... …3-none-any.whl 223KiB 74.0MiB/s 00:00 ▕██████████████████▏ 100.0% Starting download of /tmp/guix-file.xReGb4 From https://files.pythonhosted.org/packages/49/01/21337857631a97ab551bb2e3b5691ed6b1b9586011aa6a5355b9694a37fa/demon-2.0.6.tar.gz... In procedure connect: Network is unreachable Starting download of /tmp/guix-file.xReGb4 From https://web.archive.org/web/20230321140958/https://files.pythonhosted.org/packages/49/01/21337857631a97ab551bb2e3b5691ed6b1b9586011aa6a5355b9694a37fa/demon-2.0.6.tar.gz... In procedure connect*: Connection timed out Trying to use Disarchive to assemble /tmp/guix-file.xReGb4... could not find its Disarchive specification failed to download "/tmp/guix-file.xReGb4" from "https://files.pythonhosted.org/packages/49/01/21337857631a97ab551bb2e3b5691ed6b1b9586011aa6a5355b9694a37fa/demon-2.0.6.tar.gz" Backtrace: 10 (primitive-load "/home/ec2-user/.config/guix/current/bi…") In guix/ui.scm: 2300:7 9 (run-guix . _) 2263:10 8 (run-guix-command _ . _) In guix/scripts/import.scm: 89:11 7 (guix-import . _) In ice-9/boot-9.scm: 1752:10 6 (with-exception-handler _ _ #:unwind? _ # _) In guix/scripts/import/pypi.scm: 97:21 5 (_) In guix/import/utils.scm: 638:3 4 (recursive-import _ #:repo->guix-package _ #:guix-name . #) 600:31 3 (topological-sort _ #<procedure 7faf595b7270 at guix/i…> …) In srfi/srfi-1.scm: 586:17 2 (map1 (("demon" #f) ("nf1" #f) ("eva-lcd" #f) ("b…" …) …)) In guix/import/utils.scm: 630:33 1 (lookup-node "demon" #f) In ice-9/boot-9.scm: 1685:16 0 (raise-exception _ #:continuable? _) ice-9/boot-9.scm:1685:16: In procedure raise-exception: Wrong number of values returned to continuation (expected 2) --8<---------------cut here---------------end--------------->8--- --8<---------------cut here---------------start------------->8--- $ guix import pypi demon Starting download of /tmp/guix-file.MOR39A From https://files.pythonhosted.org/packages/49/01/21337857631a97ab551bb2e3b5691ed6b1b9586011aa6a5355b9694a37fa/demon-2.0.6.tar.gz... ….6.tar.gz 7KiB 17.3MiB/s 00:00 ▕██████████████████▏ 100.0% Starting download of /tmp/guix-file.RuwPtL From https://files.pythonhosted.org/packages/96/85/5f9146e3f8d7324d8741cd3bea9b99926d5ee75571b9610c661b1bbd06f5/demon-2.0.6-py3-none-any.whl... …3-none-any.whl 7KiB 17.6MiB/s 00:00 ▕██████████████████▏ 100.0% (package (name "python-demon") (version "2.0.6") (source (origin (method url-fetch) (uri (pypi-uri "demon" version)) (sha256 (base32 "0nd370yjfb3jirmxd0wa03c75dl7jac4xfsq1rvp5483mxddirb0")))) (build-system python-build-system) (propagated-inputs (list python-networkx python-tqdm)) (home-page "https://github.com/GiulioRossetti/DEMON") (synopsis "Community Discovery algorithm") (description "Community Discovery algorithm") (license license:bsd-2)) --8<---------------cut here---------------end--------------->8--- ^ permalink raw reply [flat|nested] 5+ messages in thread
* bug#63024: Crash during `guix import pypi -r' 2023-03-21 14:17 bug#62334: Network is unreachable only for recursive pypi import Greg Hogan @ 2023-05-02 21:48 ` Ludovic Courtès 2023-05-03 9:04 ` bug#62334: " Ludovic Courtès 2023-05-04 11:13 ` Ludovic Courtès 0 siblings, 2 replies; 5+ messages in thread From: Ludovic Courtès @ 2023-05-02 21:48 UTC (permalink / raw) To: Greg Hogan; +Cc: 63024, 62334, Josselin Poiret, Simon Tournier Hello! Greg Hogan <code@greghogan.com> skribis: > Starting download of /tmp/guix-file.xReGb4 >>From https://web.archive.org/web/20230321140958/https://files.pythonhosted.org/packages/49/01/21337857631a97ab551bb2e3b5691ed6b1b9586011aa6a5355b9694a37fa/demon-2.0.6.tar.gz... > In procedure connect*: Connection timed out I can very much reproduce the bug with “guix import pypi -r cdlib” on 39ba8a10971f15264966823e8696d63c2995df86 and thereabouts. Turns out it’s a fun story! (Josselin, I’m sure you’ll love it.) First, the person who wrote ‘connect*’ *cough* hadn’t carefully read the Guile manual, which reads (info "(guile) Ports and File Descriptors"): Note that ‘select’ may return early for other reasons, for example due to pending interrupts. Clearly, ‘select’ is returning early, so we should check that and loop. So far, so good. Now, why is ‘select’ returning immediately? Because the underlying select(2) call is passed an extra file descriptor, the “sleep pipe”, as you can see in ‘scm_std_select’ in libguile, and it’s that file descriptor, number 3, that’s active at each select(2) call: --8<---------------cut here---------------start------------->8--- 28825 23:06:04 pselect6(15, [3], [14], [14], {tv_sec=10, tv_nsec=0}, NULL) = 1 (in [3], left {tv_sec=9, tv_nsec=999991063}) <0.000079> 28825 23:06:04 read(3, "A", 1) = 1 <0.000044> --8<---------------cut here---------------end--------------->8--- Why so much activity on that “sleep pipe”, you ask? Well, once we’ve filtered the read(3, …) calls, a pattern appears: --8<---------------cut here---------------start------------->8--- 14335:28825 23:06:04 read(3, "A", 1) = 1 <0.000044> 14344:28825 23:06:04 read(3, "r", 1) = 1 <0.000121> 14353:28825 23:06:04 read(3, "c", 1) = 1 <0.000107> 14362:28825 23:06:04 read(3, "h", 1) = 1 <0.000040> 14436:28825 23:06:04 read(3, "i", 1) = 1 <0.000049> 14445:28825 23:06:04 read(3, "v", 1) = 1 <0.000029> 14454:28825 23:06:04 read(3, "e", 1) = 1 <0.000019> 14463:28825 23:06:04 read(3, ":", 1) = 1 <0.000017> 14472:28825 23:06:04 read(3, " ", 1) = 1 <0.000019> 14481:28825 23:06:04 read(3, " ", 1) = 1 <0.000020> 14490:28825 23:06:04 read(3, "/", 1) = 1 <0.000019> 14499:28825 23:06:04 read(3, "t", 1) = 1 <0.000015> 14508:28825 23:06:04 read(3, "m", 1) = 1 <0.000018> 15001:28825 23:06:04 read(3, "p", 1) = 1 <0.000015> 15010:28825 23:06:04 read(3, "/", 1) = 1 <0.000016> 15019:28825 23:06:04 read(3, "g", 1) = 1 <0.000016> 15028:28825 23:06:04 read(3, "u", 1) = 1 <0.000015> 15037:28825 23:06:04 read(3, "i", 1) = 1 <0.000015> 15046:28825 23:06:04 read(3, "x", 1) = 1 <0.000015> 15055:28825 23:06:04 read(3, "-", 1) = 1 <0.000015> --8<---------------cut here---------------end--------------->8--- See? Who’s chatting over our sleep pipe? --8<---------------cut here---------------start------------->8--- 4806:28831 23:06:03 dup2(0, 3) = 3 <0.000011> 4807:28831 23:06:03 dup2(3, 0) = 0 <0.000011> 4808:28831 23:06:03 dup2(4, 1) = 1 <0.000012> 4809:28831 23:06:03 dup2(5, 2) = 2 <0.000010> 4810:28831 23:06:03 close(3) = 0 <0.000010> 4811:28831 23:06:03 close(4) = 0 <0.000011> 4812:28831 23:06:03 close(5) = 0 <0.000010> 4813:28831 23:06:03 close(6) = 0 <0.000010> 4814:28831 23:06:03 close(7) = 0 <0.000010> 4815:28831 23:06:03 close(8) = 0 <0.000011> 4816:28831 23:06:03 close(9) = 0 <0.000010> 4817:28831 23:06:03 close(10) = 0 <0.000010> 4818:28831 23:06:03 close(11) = 0 <0.000011> 4819:28831 23:06:03 close(12) = 0 <0.000010> 4820:28831 23:06:03 close(13) = 0 <0.000010> 4821:28831 23:06:03 close(14) = 0 <0.000011> 4822:28831 23:06:03 close(15) = -1 EBADF (Bad file descriptor) <0.000011> 4823:28831 23:06:03 prlimit64(0, RLIMIT_NOFILE, NULL, {rlim_cur=1024, rlim_max=4*1024}) = 0 <0.000011> 4824:28831 23:06:03 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 <0.000010> 4825:28831 23:06:03 execve("/home/ludo/src/guix/scripts/unzip", ["unzip", "/tmp/guix-file.bmholm", "-d", "/tmp/guix-directory.Ok2DAA", "cdlib-0.2.6.dist-info/METADATA"], 0x18a7db0 /* 85 v […] 4938:28831 23:06:03 write(1, "Archive: /tmp/guix-file.bmholm\n", 32) = 32 <0.000018> --8<---------------cut here---------------end--------------->8--- ‘unzip’! And all its friends invoked from (guix import pypi): --8<---------------cut here---------------start------------->8--- (call-with-temporary-directory (lambda (dir) (parameterize ((current-error-port (%make-void-port "rw+")) (current-output-port (%make-void-port "rw+"))) (if (string=? "zip" (file-extension source-url)) (invoke "unzip" archive "-d" dir) (invoke "tar" "xf" archive "-C" dir))) …)) --8<---------------cut here---------------end--------------->8--- Looks like we shoudn’t dup(4, 1) in the child process, because 4 is the other end of our sleep pipe. :-) The problem exists both in 3.0.9 and Guile ‘main’. Looks related to the file descriptor shuffling code in ‘do_spawn’. WDYT, Josselin? Ludo’. ^ permalink raw reply [flat|nested] 5+ messages in thread
* bug#62334: bug#63024: Crash during `guix import pypi -r' 2023-05-02 21:48 ` bug#63024: Crash during `guix import pypi -r' Ludovic Courtès @ 2023-05-03 9:04 ` Ludovic Courtès 2023-05-03 10:08 ` bug#63024: " Simon Tournier 2023-05-04 11:13 ` Ludovic Courtès 1 sibling, 1 reply; 5+ messages in thread From: Ludovic Courtès @ 2023-05-03 9:04 UTC (permalink / raw) To: Josselin Poiret; +Cc: 63024, 62334, Greg Hogan, Simon Tournier [-- Attachment #1: Type: text/plain, Size: 215 bytes --] Hey, Ludovic Courtès <ludo@gnu.org> skribis: > Looks like we shoudn’t dup(4, 1) in the child process, because 4 is the > other end of our sleep pipe. :-) How about this patch, Josselin? Ludo’. [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: Type: text/x-patch, Size: 2418 bytes --] diff --git a/libguile/posix.c b/libguile/posix.c index 3adc743c4..2d55d985c 100644 --- a/libguile/posix.c +++ b/libguile/posix.c @@ -1388,11 +1388,27 @@ do_spawn (char *exec_file, char **exec_argv, char **exec_env, } /* Move the fds out of the way, so that duplicate fds or fds equal - to 0, 1, 2 don't trample each other */ + to 0, 1, 2 don't trample each other. Since 'system*' might give + us -1 for IN, OUT, or ERR, open /dev/null when that's the case. */ + + if (in < 0) + posix_spawn_file_actions_addopen (&actions, fd_slot[0], + "/dev/null", O_RDONLY | O_CLOEXEC, 0); + else + posix_spawn_file_actions_adddup2 (&actions, in, fd_slot[0]); + + if (out < 0) + posix_spawn_file_actions_addopen (&actions, fd_slot[1], + "/dev/null", O_WRONLY | O_CLOEXEC, 0); + else + posix_spawn_file_actions_adddup2 (&actions, out, fd_slot[1]); + + if (err < 0) + posix_spawn_file_actions_addopen (&actions, fd_slot[2], + "/dev/null", O_WRONLY | O_CLOEXEC, 0); + else + posix_spawn_file_actions_adddup2 (&actions, err, fd_slot[2]); - posix_spawn_file_actions_adddup2 (&actions, in, fd_slot[0]); - posix_spawn_file_actions_adddup2 (&actions, out, fd_slot[1]); - posix_spawn_file_actions_adddup2 (&actions, err, fd_slot[2]); posix_spawn_file_actions_adddup2 (&actions, fd_slot[0], 0); posix_spawn_file_actions_adddup2 (&actions, fd_slot[1], 1); posix_spawn_file_actions_adddup2 (&actions, fd_slot[2], 2); diff --git a/test-suite/tests/posix.test b/test-suite/tests/posix.test index d5cf47cda..18dad8902 100644 --- a/test-suite/tests/posix.test +++ b/test-suite/tests/posix.test @@ -374,7 +374,17 @@ (system* "sh" "-c" "echo bong >&2")))))))) (and (zero? (status:exit-val status)) - (call-with-input-file file get-string-all))))) + (call-with-input-file file get-string-all)))) + + (pass-if-equal "https://bugs.gnu.org/63024" + 0 + (if (file-exists? "/proc/self/fd/0") ;on GNU/Linux? + (parameterize ((current-output-port (%make-void-port "w0"))) + (system* "guile" "-c" + (object->string + '(exit (string=? "/dev/null" + (readlink "/proc/self/fd/1")))))) + (throw 'unresolved)))) ;; ;; spawn ^ permalink raw reply related [flat|nested] 5+ messages in thread
* bug#63024: bug#62334: bug#63024: Crash during `guix import pypi -r' 2023-05-03 9:04 ` bug#62334: " Ludovic Courtès @ 2023-05-03 10:08 ` Simon Tournier 2023-05-04 11:10 ` Ludovic Courtès 0 siblings, 1 reply; 5+ messages in thread From: Simon Tournier @ 2023-05-03 10:08 UTC (permalink / raw) To: Ludovic Courtès, Josselin Poiret; +Cc: 63024, 62334, Greg Hogan Hi Ludo, On Wed, 03 May 2023 at 11:04, Ludovic Courtès <ludo@gnu.org> wrote: > diff --git a/libguile/posix.c b/libguile/posix.c > index 3adc743c4..2d55d985c 100644 > --- a/libguile/posix.c > +++ b/libguile/posix.c Does it mean patch the current Guile or a new release of Guile? Cheers, simon ^ permalink raw reply [flat|nested] 5+ messages in thread
* bug#63024: Crash during `guix import pypi -r' 2023-05-03 10:08 ` bug#63024: " Simon Tournier @ 2023-05-04 11:10 ` Ludovic Courtès 0 siblings, 0 replies; 5+ messages in thread From: Ludovic Courtès @ 2023-05-04 11:10 UTC (permalink / raw) To: Simon Tournier; +Cc: Josselin Poiret, 63024, Greg Hogan, 62334 Hi, Simon Tournier <zimon.toutoune@gmail.com> skribis: > On Wed, 03 May 2023 at 11:04, Ludovic Courtès <ludo@gnu.org> wrote: > >> diff --git a/libguile/posix.c b/libguile/posix.c >> index 3adc743c4..2d55d985c 100644 >> --- a/libguile/posix.c >> +++ b/libguile/posix.c > > Does it mean patch the current Guile or a new release of Guile? It’s a patch against Guile ‘main’. That gives an incentive to push a new Guile release. Ludo’. ^ permalink raw reply [flat|nested] 5+ messages in thread
* bug#63024: Crash during `guix import pypi -r' 2023-05-02 21:48 ` bug#63024: Crash during `guix import pypi -r' Ludovic Courtès 2023-05-03 9:04 ` bug#62334: " Ludovic Courtès @ 2023-05-04 11:13 ` Ludovic Courtès 1 sibling, 0 replies; 5+ messages in thread From: Ludovic Courtès @ 2023-05-04 11:13 UTC (permalink / raw) To: Greg Hogan; +Cc: 63024, 62334, Josselin Poiret, Simon Tournier Ludovic Courtès <ludo@gnu.org> skribis: > First, the person who wrote ‘connect*’ *cough* hadn’t carefully read the > Guile manual, which reads (info "(guile) Ports and File Descriptors"): > > Note that ‘select’ may return early for other reasons, for example due > to pending interrupts. > > Clearly, ‘select’ is returning early, so we should check that and loop. Pushed a fix for that in fc6c96c88a0e4ad0b9e48272e5f97ffaa6eec36e. Ludo’. ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2023-05-04 11:14 UTC | newest] Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2023-04-22 19:59 bug#63024: Crash during `guix import pypi -r' Timo Wilken 2023-04-24 8:33 ` Simon Tournier -- strict thread matches above, loose matches on Subject: below -- 2023-03-21 14:17 bug#62334: Network is unreachable only for recursive pypi import Greg Hogan 2023-05-02 21:48 ` bug#63024: Crash during `guix import pypi -r' Ludovic Courtès 2023-05-03 9:04 ` bug#62334: " Ludovic Courtès 2023-05-03 10:08 ` bug#63024: " Simon Tournier 2023-05-04 11:10 ` Ludovic Courtès 2023-05-04 11:13 ` Ludovic Courtès
Code repositories for project(s) associated with this public inbox https://git.savannah.gnu.org/cgit/guix.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).