Quicklisp importer bug in tarball->extract->parse

Quicklisp importer bug in tarball->extract->parse

[swedebugia]

[-- Attachment #1.1: Type: text/plain, Size: 1878 bytes --]

Hi

I continued working on it and have run into a bug with extracting and
parsing files in the tarball (like the pypi importer does)

This is really putting my lisp skills to the test I feel like i'm in
over my head here. 😛

The idea is to
- first extract information from quicklisp - done
- then extract the tarball - done
- then find the 'name'.asd-file in the extracted directory - fails
- then parse with PEG - todo
- then print the result - todo

Does anyone have a clue what is wrong or how to debug (find-files)?

Here is the code: https://gitlab.com/swedebugia/guix/tree/quicklisp

Here is the output:

Starting download of /tmp/guix-file.Q7fTrf
From https://beta.quicklisp.org/dist/quicklisp.txt...
 quicklisp.txt  408B                  202KiB/s 00:00
[##################] 100.0%

Starting download of /tmp/guix-file.N8gfUL
From http://beta.quicklisp.org/dist/quicklisp/2019-03-07/releases.txt...
 releases.txt  423KiB                 1.1MiB/s 00:00
[##################] 100.0%

Starting download of /tmp/guix-file.B6vRet
From https://beta.quicklisp.org/dist/quicklisp.txt...
 quicklisp.txt  408B                  223KiB/s 00:00
[##################] 100.0%

Starting download of /tmp/guix-file.QA07gb
From http://beta.quicklisp.org/dist/quicklisp/2019-03-07/releases.txt...
 releases.txt  423KiB                 768KiB/s 00:01
[##################] 100.0%

;;; (asd "1am.asd")

Starting download of /tmp/guix-file.4QyAk4
From
http://beta.quicklisp.org/archive/1am/2014-11-06/1am-20141106-git.tgz...
 …06-git.tgz  3KiB                    4.2MiB/s 00:00
[##################] 100.0%

;;; (dirname "1am-20141106-git")

;;; (system-files "1am.asd")

;;; (dir "/tmp/guix-directory.DnfwBX")

;;; (file-name "1am.asd")

;;; (find-files ())
warning: Failed to extract requirements files

-- 
Cheers Swedebugia


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: Quicklisp importer bug in tarball->extract->parse

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 122 bytes --]

Hi,

thanks for your work!  I'll look into it Very Soon™.

Cheers!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Quicklisp importer bug in tarball->extract->parse

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1620 bytes --]

Hi!

I've had time to look at it a little.

If you want to debug, you can, for instance, put a breakpoint in
guess-requirements-from-source, then enter the following in your guile
prompt:

  ,backtrace #:full? #t

or shorter

  ,bt #:full? #t

That will print the backtrace will all local variables.


There is nothing wrong with find-files: the "dir" directory is empty.
This is because the extraction failed: 
check the exit-code of system* before proceeding, you'll see what
happens ;)

--8<---------------cut here---------------start------------->8---
(system* "tar" "xf" tarball "-C" dir file-name)
--8<---------------cut here---------------end--------------->8---

The above will error out if "file-name" is not found.  For 1am:

Queried file-name:

  "1am.asd"

Tarball actual file:

  "1am-20141106-git/1am.asd"

I don't think we can predict the root folder, and more generally
projects have different trees.
So I suggest we extract the whole tarball and proceed from there.  So
your code should work by just removing "file-name":

--8<---------------cut here---------------start------------->8---
(system* "tar" "xf" tarball "-C" dir)
--8<---------------cut here---------------end--------------->8---

Makes sense?

Unrelated comment: If I'm not mistaken, you are fetching the meta-file
and the index-file on every query.  I suggest you keep those in memories
as soon as possible (memoize them), this will allow you to iterate much
faster when developing.

It seems that you are on the right track, great job!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Quicklisp importer bug in tarball->extract->parse

[Ricardo Wurmus]

Pierre Neidhardt <mail@ambrevar.xyz> writes:

> The above will error out if "file-name" is not found.  For 1am:
>
> Queried file-name:
>
>   "1am.asd"
>
> Tarball actual file:
>
>   "1am-20141106-git/1am.asd"
>
> I don't think we can predict the root folder, and more generally
> projects have different trees.

We can add “--strip-components=1” and do the whole thing in
“with-directory-excursion”.  This way we’ll always know the directory
name.  You can also extract individual files in the archive by wildcard,
so you don’t need to know the exact name.

--
Ricardo

Re: Quicklisp importer bug in tarball->extract->parse

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 394 bytes --]

Ricardo Wurmus <rekado@elephly.net> writes:

> We can add “--strip-components=1” and do the whole thing in
> “with-directory-excursion”.  This way we’ll always know the directory
> name.  You can also extract individual files in the archive by wildcard,
> so you don’t need to know the exact name.

Didn't know, that's great!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Quicklisp importer bug in tarball->extract->parse

[swedebugia]

[-- Attachment #1.1: Type: text/plain, Size: 4158 bytes --]

Hi

Thanks a lot for taking the time to look at this.

On 2019-04-11 16:31, Pierre Neidhardt wrote:
> Hi!
> 
> I've had time to look at it a little.
> 
> If you want to debug, you can, for instance, put a breakpoint in
> guess-requirements-from-source, 

How exactly do you do this?
I looked in the guile manual and found the module (system vm trap-state)
and tried calling add-trap-at-procedure-call! like this:

(add-trap-at-procedure-call!

  (system* "tar" "xf" tarball "-C" dir "--wildcards" (string-append "*"
file-name)))))

But it failed with:
;;; (file-name "1am.asd")

Backtrace:

          13 (apply-smob/1 #<catch-closure 1480820>)

In ice-9/boot-9.scm:

    705:2 12 (call-with-prompt _ _ #<procedure default-prompt-handler (k
proc)>)
In ice-9/eval.scm:

    619:8 11 (_ #(#(#<directory (guile-user) 14de140>)))

In ice-9/boot-9.scm:

   2312:4 10 (save-module-excursion _)

  3831:12  9 (_)

In guix/import/quicklisp.scm:

    404:3  8 (_)

In guix/utils.scm:

    618:8  7 (call-with-temporary-output-file _)

    632:8  6 (call-with-temporary-directory _)

In guix/import/quicklisp.scm:

   374:32  5 (_ "/tmp/guix-directory.Z4chBm")

In srfi/srfi-1.scm:

   592:17  4 (map1 ("1am.asd"))

In guix/import/quicklisp.scm:

   380:41  3 (_ _)

In system/vm/trap-state.scm:

   213:15  2 (add-trap-at-procedure-call! 0 _)

In system/vm/traps.scm:

   137:45  1 (trap-at-procedure-call 0 #<procedure 3777e00 at
system/vm/trap-state.scm:128:2 (frame)> #:our-frame? _)

In unknown file:

           0 (scm-error misc-error #f "~A ~S" ("Not a VM program" 0) #f)



ERROR: In procedure scm-error:

Not a VM program 0


> There is nothing wrong with find-files: the "dir" directory is empty.
> This is because the extraction failed: 
> check the exit-code of system* before proceeding, you'll see what
> happens ;)
> 
> --8<---------------cut here---------------start------------->8---
> (system* "tar" "xf" tarball "-C" dir file-name)
> --8<---------------cut here---------------end--------------->8---
> 
> The above will error out if "file-name" is not found.  For 1am:
> 
> Queried file-name:
> 
>   "1am.asd"
> 
> Tarball actual file:
> 
>   "1am-20141106-git/1am.asd"
> 
> I don't think we can predict the root folder, and more generally
> projects have different trees.
> So I suggest we extract the whole tarball and proceed from there.  So
> your code should work by just removing "file-name":
> 
> --8<---------------cut here---------------start------------->8---
> (system* "tar" "xf" tarball "-C" dir)
> --8<---------------cut here---------------end--------------->8---
> 
> Makes sense?

Yes. I implemented Ricardos wildcard-suggestion:

(system* "tar" "xf" tarball "-C" dir "--wildcards" (string-append "*"
file-name))

> 
> Unrelated comment: If I'm not mistaken, you are fetching the meta-file
> and the index-file on every query.  I suggest you keep those in memories
> as soon as possible (memoize them), this will allow you to iterate much
> faster when developing.

I already tried by adding:


 (memoize

  (let* ( ;;(name "circular-streams")

         (name "1am")

         (source-url (ql-extract 'url name))

         (asd (ql-extract 'system-files name)))

    (peek 'asd asd)

    (call-with-temporary-output-file

     (lambda (temp port)

       (begin (url-fetch source-url temp)

              (guess-requirements-from-source source-url asd temp)

              )))))

and

(let* ( ;;(name "circular-streams")

       (name "1am")

       (source-url (memoize (ql-extract 'url name)))

       (asd (memoize (ql-extract 'system-files name))))

  (peek 'asd asd)

  (call-with-temporary-output-file

   (lambda (temp port)

     (begin (url-fetch source-url temp)

            (guess-requirements-from-source source-url asd temp)

            ))))

But it still fetches the index-file 2 times regardless where I put it.
Any ideas?

> It seems that you are on the right track, great job!

Thanks for the encouragement!

-- 
Cheers Swedebugia


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: Quicklisp importer bug in tarball->extract->parse

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1444 bytes --]

swedebugia <swedebugia@riseup.net> writes:

> Hi
>
> Thanks a lot for taking the time to look at this.
>
> On 2019-04-11 16:31, Pierre Neidhardt wrote:
>> Hi!
>> 
>> I've had time to look at it a little.
>> 
>> If you want to debug, you can, for instance, put a breakpoint in
>> guess-requirements-from-source, 
>
> How exactly do you do this?

See

--8<---------------cut here---------------start------------->8---
,help debug
--8<---------------cut here---------------end--------------->8---

from the Guile REPL.
In particular

--8<---------------cut here---------------start------------->8---
,break PROCEDURE
--8<---------------cut here---------------end--------------->8---

should do what you want.

>  (memoize
> ...

`memoize' takes a procedure as argument, not an arbitrary expression.
For instance in the following

--8<---------------cut here---------------start------------->8---
(let ((mfoo (memoize foo)))
  (mfoo 17)
  (mfoo 17))
--8<---------------cut here---------------end--------------->8---

`foo' will be called only once.

You don't have to use `memoize' though, you can also store the value you
want in a local variable and check if it's set or not.  We can always
refine the code later.

>> It seems that you are on the right track, great job!
>
> Thanks for the encouragement!

And thanks for giving this a shot! :)

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]