From: Attila Lendvai <attila@lendvai.name>
To: 55242@debbugs.gnu.org
Cc: Attila Lendvai <attila@lendvai.name>
Subject: [bug#55242] [PATCH 07/10] guix: import: go: More resilience wrt network errors; add logging.
Date: Tue, 3 May 2022 13:42:58 +0200 [thread overview]
Message-ID: <20220503114301.9524-7-attila@lendvai.name> (raw)
In-Reply-To: <20220503114301.9524-2-attila@lendvai.name>
This also adds logging statements to various parts of the go importer.
* guix/import/go.scm (go-module->guix-package*): Catch exceptions and retry in
case of network errors. Add ability to enable printing a full backtrace when
debugging.
(go-module->guix-package): Tolerate the inability to fetch the synopsis and
description.
---
guix/import/go.scm | 104 +++++++++++++++++++++++++++++++++++++--------
1 file changed, 86 insertions(+), 18 deletions(-)
diff --git a/guix/import/go.scm b/guix/import/go.scm
index a08005d090..6871132a70 100644
--- a/guix/import/go.scm
+++ b/guix/import/go.scm
@@ -23,7 +23,15 @@
;;; You should have received a copy of the GNU General Public License
;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+;;; goproxy protocol:
+;;; https://golang.org/ref/mod#module-proxy
+;;; https://docs.gomods.io/intro/protocol/
+;;; https://roberto.selbach.ca/go-proxies/
+
(define-module (guix import go)
+ #:use-module (git)
+ #:use-module (git structs)
+ #:use-module (git errors)
#:use-module (guix build-system go)
#:use-module (guix git)
#:use-module (guix hash)
@@ -51,6 +59,8 @@ (define-module (guix import go)
#:autoload (guix build utils) (mkdir-p dump-port)
#:autoload (gcrypt hash) (hash-algorithm sha256)
#:use-module (ice-9 format)
+ #:use-module (ice-9 control)
+ #:use-module (ice-9 exceptions)
#:use-module (ice-9 match)
#:use-module (ice-9 peg)
#:use-module (ice-9 rdelim)
@@ -629,10 +639,14 @@ (define (go-import->module-meta content-text)
(make-module-meta root-path (string->symbol vcs)
(strip-.git-suffix/maybe repo-url)))))
;; <meta name="go-import" content="import-prefix vcs repo-root">
- (let* ((meta-data (http-fetch* (format #f "https://~a?go-get=1" module-path)))
+ (let* ((url (format #f "https://~a?go-get=1" module-path))
+ (meta-data (http-fetch* url #:accept-all-response-codes? #true))
(select (sxpath `(// (meta (@ (equal? (name "go-import"))))
- // content))))
- (match (select (html->sxml meta-data #:strict? #t))
+ // content)))
+ (sxml (html->sxml meta-data #:strict? #t))
+ (selected (select sxml)))
+ (log.debug "The fetched meta-data from ~A is:~%~S" url selected)
+ (match selected
(() #f) ;nothing selected
((('content content-text) ..1)
(or
@@ -809,22 +823,76 @@ (define* (go-module->guix-package module-path #:key
dependencies+versions
dependencies))))
-(define go-module->guix-package*
- (lambda args
- ;; Disable output buffering so that the following warning gets printed
- ;; consistently.
- (setvbuf (current-error-port) 'none)
- (let ((package-name (match args ((name _ ...) name))))
- (guard (c ((http-get-error? c)
- (warning (G_ "Failed to import package ~s.
-reason: ~s could not be fetched: HTTP error ~a (~s).
+(define (go-module->guix-package* . args)
+ ;; Disable output buffering so that the following warning gets printed
+ ;; consistently.
+ (setvbuf (current-error-port) 'none)
+ (setvbuf (current-warning-port) 'none)
+ (let* ((package-name (match args ((name _ ...) name)))
+ ;; Use report-all-errors? for debugging purposes only, because
+ ;; e.g. getaddrinfo is not reentrant and therefore we must unwind
+ ;; before retrying.
+ (report-all-errors? #false)
+ (report-network-error
+ (lambda (reason)
+ (warning (G_ "Failing to import package ~S.
+reason: ~A.~%")
+ package-name reason))))
+ (let loop ((attempts 0))
+ (when (> attempts 0)
+ (sleep 3)
+ (log.info "~%Retrying, attempt ~s." attempts))
+ (cond
+ ((> attempts 60)
+ (warning (G_ "Giving up on importing package ~s.
This package and its dependencies won't be imported.~%")
- package-name
- (uri->string (http-get-error-uri c))
- (http-get-error-code c)
- (http-get-error-reason c))
- (values #f '())))
- (apply go-module->guix-package args)))))
+ package-name)
+ (values #f '()))
+ (else
+ (guard (c ((http-get-error? c)
+ (report-network-error
+ (format #f "~s could not be fetched: HTTP error ~a (~s)"
+ (uri->string (http-get-error-uri c))
+ (http-get-error-code c)
+ (http-get-error-reason c)))
+ (loop (+ 1 attempts)))
+ ((eq? (exception-kind c)
+ 'getaddrinfo-error)
+ (report-network-error "DNS lookup failed")
+ (loop (+ 1 attempts)))
+ ((and (eq? (exception-kind c)
+ 'git-error)
+ (eq? (git-error-class (first (exception-args c)))
+ GITERR_NET))
+ (report-network-error "network error coming from git")
+ (loop (+ 1 attempts)))
+ (else
+ (let ((port (current-warning-port)))
+ (format port "Unexpected error, will skip ~S.~%reason: "
+ package-name)
+ ;; Printing a backtrace here is not very useful: it is
+ ;; cut off because GUARD unwinds.
+ (print-exception port (stack-ref (make-stack #t) 1)
+ c (exception-args c))
+ (display-backtrace (make-stack #t) port))
+ ;; give up on this entry
+ (values #f '())))
+ (with-exception-handler
+ (lambda (c)
+ (when report-all-errors?
+ (let ((port (current-warning-port)))
+ (format port "*** exception while importing:~%")
+ (print-exception port (stack-ref (make-stack #t) 1)
+ c (exception-args c))
+ (format port "*** printing backtrace:~%")
+ (display-backtrace (make-stack #t) port)
+ ;; DISPLAY-BACKTRACE can fail, so it's better to make its
+ ;; exit also visible.
+ (format port "*** done printing backtrace~%")))
+ (raise-continuable c))
+ (lambda ()
+ (apply go-module->guix-package args))
+ #:unwind? (not report-all-errors?))))))))
(define* (go-module-recursive-import package-name
#:key (goproxy "https://proxy.golang.org")
--
2.35.1
next prev parent reply other threads:[~2022-05-03 11:44 UTC|newest]
Thread overview: 45+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-05-03 11:16 [bug#55242] [PATCH 01/10] guix: import: Print the number of packages at the end Attila Lendvai
2022-05-03 11:42 ` [bug#55242] [PATCH 02/10] guix: import: go: Rename go.pkg.dev-info to pkg.go.dev-info Attila Lendvai
2022-05-03 11:42 ` [bug#55242] [PATCH 03/10] guix: import: go: Add mockup logging facility Attila Lendvai
2022-05-03 16:23 ` Maxime Devos
2022-06-14 19:46 ` [bug#55242] [PATCH 01/10] guix: import: Print the number of packages at the end Maxim Cournoyer
2022-05-03 11:42 ` [bug#55242] [PATCH 04/10] guix: import: go: Fix the interpretation of the replace directive Attila Lendvai
2022-05-03 16:25 ` Maxime Devos
2022-05-03 16:26 ` Maxime Devos
2022-05-03 16:27 ` Maxime Devos
2022-05-03 16:28 ` Maxime Devos
2022-05-03 11:42 ` [bug#55242] [PATCH 05/10] guix: import: go: Harden sxml->texi conversion Attila Lendvai
2022-05-03 16:32 ` Maxime Devos
2022-05-03 11:42 ` [bug#55242] [PATCH 06/10] guix: import: go: Add a local duplicate of http-fetch Attila Lendvai
2022-05-03 11:42 ` Attila Lendvai [this message]
2022-05-03 16:12 ` [bug#55242] [PATCH 07/10] guix: import: go: More resilience wrt network errors; add logging Maxime Devos
2022-05-03 17:00 ` Attila Lendvai
2022-05-03 17:38 ` Maxime Devos
2022-05-09 12:34 ` Attila Lendvai
2022-05-09 17:45 ` Maxime Devos
2022-05-09 20:02 ` Attila Lendvai
2022-05-09 20:08 ` Maxime Devos
2022-05-03 16:17 ` Maxime Devos
2022-05-03 16:36 ` Maxime Devos
2022-05-03 16:37 ` Maxime Devos
2022-05-03 11:42 ` [bug#55242] [PATCH 08/10] guix: import: go: Modules in a subdir and prefixed tags Attila Lendvai
2022-05-03 16:42 ` Maxime Devos
2022-05-03 16:48 ` Maxime Devos
2022-05-03 16:50 ` Maxime Devos
2022-05-09 12:50 ` Attila Lendvai
2022-05-03 16:51 ` Maxime Devos
2022-05-03 16:53 ` Maxime Devos
2022-05-03 16:55 ` Maxime Devos
2022-05-03 16:56 ` Maxime Devos
2022-05-03 16:59 ` Maxime Devos
2022-05-03 17:00 ` Maxime Devos
2022-05-03 17:01 ` Maxime Devos
2022-05-03 11:43 ` [bug#55242] [PATCH 09/10] guix: import: go: module-name -> module-path to be consistent Attila Lendvai
2022-05-03 11:43 ` [bug#55242] [PATCH 10/10] guix: import: go: Better handling of /v2 in the module path Attila Lendvai
2022-05-09 20:39 ` Maxime Devos
2022-05-03 16:22 ` [bug#55242] [PATCH 02/10] guix: import: go: Rename go.pkg.dev-info to pkg.go.dev-info Maxime Devos
2022-05-03 16:21 ` [bug#55242] [PATCH 01/10] guix: import: Print the number of packages at the end Maxime Devos
2022-05-03 17:29 ` Maxime Devos
2022-05-09 12:37 ` Attila Lendvai
2022-05-14 7:09 ` Maxime Devos
2022-05-14 7:15 ` Maxime Devos
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://guix.gnu.org/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20220503114301.9524-7-attila@lendvai.name \
--to=attila@lendvai.name \
--cc=55242@debbugs.gnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/guix.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).