From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id QFk3Ecthq18+cAAA0tVLHw (envelope-from ) for ; Wed, 11 Nov 2020 04:00:11 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id +BMeDcthq1+8NwAAbx9fmQ (envelope-from ) for ; Wed, 11 Nov 2020 04:00:11 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id B705C9401BD for ; Wed, 11 Nov 2020 04:00:10 +0000 (UTC) Received: from localhost ([::1]:51284 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kchIn-0005FA-Ap for larch@yhetil.org; Tue, 10 Nov 2020 23:00:09 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:53356) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kchIg-0005Ew-KD for guix-patches@gnu.org; Tue, 10 Nov 2020 23:00:02 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:56419) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kchIg-0001nD-7i for guix-patches@gnu.org; Tue, 10 Nov 2020 23:00:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1kchIg-0001cs-1m for guix-patches@gnu.org; Tue, 10 Nov 2020 23:00:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#44567] [PATCH] publish: Improve HTTP performance when not using --cache. Resent-From: Maxim Cournoyer Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Wed, 11 Nov 2020 04:00:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 44567 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 44567@debbugs.gnu.org Cc: Ludovic =?UTF-8?Q?Court=C3=A8s?= , Maxim Cournoyer X-Debbugs-Original-To: guix-patches@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.16050671586174 (code B ref -1); Wed, 11 Nov 2020 04:00:01 +0000 Received: (at submit) by debbugs.gnu.org; 11 Nov 2020 03:59:18 +0000 Received: from localhost ([127.0.0.1]:39732 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kchHy-0001bW-5Z for submit@debbugs.gnu.org; Tue, 10 Nov 2020 22:59:18 -0500 Received: from lists.gnu.org ([209.51.188.17]:40048) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kchHv-0001bN-RC for submit@debbugs.gnu.org; Tue, 10 Nov 2020 22:59:16 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:53288) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kchHv-00058C-4J for guix-patches@gnu.org; Tue, 10 Nov 2020 22:59:15 -0500 Received: from mail-qt1-x842.google.com ([2607:f8b0:4864:20::842]:32793) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kchHt-0001WH-41; Tue, 10 Nov 2020 22:59:14 -0500 Received: by mail-qt1-x842.google.com with SMTP id i12so501765qtj.0; Tue, 10 Nov 2020 19:59:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=XaeK7A8gB+CRFWk7fmbBt3z4ezhaDDguNFUKEmtT7pU=; b=qCf8b5wbptyFNlTuFFsAmBuBvjMV38+gu/djoO+frLmIhkd02J/kkugiqS2HZd44rn SvMaeZ3n77AkLA1QhnxnyKlD5V7uo31yQiZiPCmTA1fjM7LSLbqG1CkKOiP0JncU80Hw CQn8p/ZigYuVOhFuAWkK7xd+CQHrXBAIKOtZZJL+64nryEQTJP5bwP2EBRj3qa6hkWR2 c/nfRXl8+tfnupV8YXuJHjlTveOo1eIip53Ead4h7sbjodn82K710WynimoOFhF84QRD WOXmjMIhLbPNMryzvRzO1QQPHV+tKm3HykkEy3CZ0djREzSUZmf3p2RzTJV9UuorY08y oh3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=XaeK7A8gB+CRFWk7fmbBt3z4ezhaDDguNFUKEmtT7pU=; b=N0E4+mH+Mhv5LJhuKizb/y0r7MT59MvETaAiVt23PinBLaLqaszPy0TC/QI3uXCwOB BLVhRkGmEl3F3tcm6+4411PSIFBiisXbN1igFIimlhAM8EcdNh5iHtb8vhqjNcMUC3rt 2JWs1l/LV1ybHs5Cqr521O9yP5Fw2sZu46YEOCPF81gEVbRTCHi/f9iyZST68QDv7Syn Yj9bvOWdhZgxlVaTz2IP9CZ2iXvZ37y37kuTbIPT2NPK+n6SC1qzOz/cGgO+RXUyeu7+ dGgTOKXIafiK3wX34pPpUIzYjMM5P4VicpfEAFejEiEoMsl/fLbs5+zmkEHBPC3xRF/a Tf0A== X-Gm-Message-State: AOAM533JK8ld77ULL1KTmFjJMGIkdd+cnlvipvsK3hdK+hqPIOWpl26T Ehmy8sPOOiBxxzvlR0QKywt8m0TT6rtZXg== X-Google-Smtp-Source: ABdhPJwZBAJfqRl1n2GhG9iwpvamVxkjT8pttDeITee/gQK92rAIRJge7dB8Gb7BFty49E71d4hMCg== X-Received: by 2002:aed:30e2:: with SMTP id 89mr22038831qtf.259.1605067150996; Tue, 10 Nov 2020 19:59:10 -0800 (PST) Received: from localhost.localdomain (dsl-10-133-153.b2b2c.ca. [72.10.133.153]) by smtp.gmail.com with ESMTPSA id d12sm986767qtp.77.2020.11.10.19.59.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Nov 2020 19:59:10 -0800 (PST) From: Maxim Cournoyer Date: Tue, 10 Nov 2020 22:57:27 -0500 Message-Id: <20201111035727.11184-1-maxim.cournoyer@gmail.com> X-Mailer: git-send-email 2.28.0 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Received-SPF: pass client-ip=2607:f8b0:4864:20::842; envelope-from=maxim.cournoyer@gmail.com; helo=mail-qt1-x842.google.com X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: 0.2 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-Spam-Score: -2.3 (--) X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Scanner: ns3122888.ip-94-23-21.eu Authentication-Results: aspmx1.migadu.com; dkim=fail (headers rsa verify failed) header.d=gmail.com header.s=20161025 header.b=qCf8b5wb; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Spam-Score: 1.59 X-TUID: bs2dGDqZuEGQ This change harmonizes the way we configure the buffer sizes and the socket options, so that we don't forget to change it at one place like it happened in commit 5e3d169945935b53325e6b738a307ba286751259. Using a greater socket buffer size reportedly improves throughput tenfold. The solution was found by Ludovic Courtès. * guix/scripts/publish.scm (%default-buffer-size) (%default-socket-options): New variables. * guix/scripts/publish.scm (configure-socket): New procedure. (compress-nar): Use %default-buffer-size for the buffer size, increased from 128 to 208 KiB. (nar-response-port): Likewise, increased from 64 to 208 KiB. (http-write): Use configure-socket to set socket options. (open-server-socket): Likewise. --- guix/scripts/publish.scm | 23 +++++++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-) diff --git a/guix/scripts/publish.scm b/guix/scripts/publish.scm index e8faf379e2..8a07f2300e 100644 --- a/guix/scripts/publish.scm +++ b/guix/scripts/publish.scm @@ -2,6 +2,7 @@ ;;; Copyright © 2015 David Thompson ;;; Copyright © 2020 by Amar M. Singh ;;; Copyright © 2015, 2016, 2017, 2018, 2019, 2020 Ludovic Courtès +;;; Copyright © 2020 Maxim Cournoyer ;;; ;;; This file is part of GNU Guix. ;;; @@ -250,6 +251,18 @@ usage." ("WantMassQuery" . 0) ("Priority" . 100))) +(define %default-buffer-size + (* 208 1024)) + +(define %default-socket-options + (list (list SO_SNDBUF %default-buffer-size))) + +(define* (configure-socket socket #:key (level SOL_SOCKET) + (options %default-socket-options)) + "Apply multiple option tuples in OPTIONS to SOCKET, using LEVEL." + (for-each (cut apply setsockopt socket level <>) + options)) + (define (signed-string s) "Sign the hash of the string S with the daemon's key. Return a canonical sexp for the signature." @@ -569,7 +582,7 @@ requested using POOL." (lambda (port) (write-file item port)) #:level (compression-level compression) - #:buffer-size (* 128 1024)) + #:buffer-size %default-buffer-size) (rename-file (string-append nar ".tmp") nar)) ('lzip ;; Note: the file port gets closed along with the lzip port. @@ -858,7 +871,7 @@ or if EOF is reached." ;; 'make-gzip-output-port' wants a file port. (make-gzip-output-port (response-port response) #:level level - #:buffer-size (* 64 1024))) + #:buffer-size %default-buffer-size)) (($ 'lzip level) (make-lzip-output-port (response-port response) #:level level)) @@ -883,6 +896,7 @@ blocking." client)) (port (begin (force-output client) + (configure-socket client) (nar-response-port response compression)))) ;; XXX: Given our ugly workaround for in ;; 'render-nar', BODY here is just the file name of the store item. @@ -912,7 +926,7 @@ blocking." size) client)) (output (response-port response))) - (setsockopt client SOL_SOCKET SO_SNDBUF (* 128 1024)) + (configure-socket client) (if (file-port? output) (sendfile output input size) (dump-port input output)) @@ -1057,7 +1071,8 @@ methods, return the applicable compression." (define (open-server-socket address) "Return a TCP socket bound to ADDRESS, a socket address." (let ((sock (socket (sockaddr:fam address) SOCK_STREAM 0))) - (setsockopt sock SOL_SOCKET SO_REUSEADDR 1) + (configure-socket sock #:options (cons (list SO_REUSEADDR 1) + %default-socket-options)) (bind sock address) sock)) -- 2.28.0