From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id IImGKmcDEWGdpAAAgWs5BA (envelope-from ) for ; Mon, 09 Aug 2021 12:28:55 +0200 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id SIgnJmcDEWF/UAAAB5/wlQ (envelope-from ) for ; Mon, 09 Aug 2021 10:28:55 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 4FBB420FA1 for ; Mon, 9 Aug 2021 12:28:55 +0200 (CEST) Received: from localhost ([::1]:50234 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mD2Wc-0001zf-B6 for larch@yhetil.org; Mon, 09 Aug 2021 06:28:54 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:36084) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mD2WR-0001xe-0S for guix-devel@gnu.org; Mon, 09 Aug 2021 06:28:43 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:36808) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mD2WP-0005aE-EO; Mon, 09 Aug 2021 06:28:42 -0400 Received: from [2a01:e0a:19b:d9a0:f2f7:a404:c3d3:f8b4] (port=44256 helo=meije) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mD2WP-0005uY-5Q; Mon, 09 Aug 2021 06:28:41 -0400 From: Mathieu Othacehe To: guix-devel Subject: Substitute timeouts Date: Mon, 09 Aug 2021 12:28:39 +0200 Message-ID: <875ywec3oo.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1628504935; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type:list-id:list-help:list-unsubscribe: list-subscribe:list-post; bh=LyTpPppNkGUAt4VAoDNZ4MPebPG4pCZSfHDGc9CMJaM=; b=W5835kc6YaLrww4Z/C8nh67JsFANmrORAvUpsZg2ZCa/OjhqvhMg5mr3akRAYzjdzphWGo 8KFBkTa2bsLQROPhyucWiYvz7W5eZFsLiDGAWNFLw3t2XIrjOQBDc1DtaRJT4jIXKqIMR1 Xu0e2jRhPho8RPl5zISdy3JIZLD4XRePKNQpA99/tRDPJTvevP2ng0QYzo2I2XeRamlJEr KOUJ/nJL7hxUqVEMGddKI+0buNJ0yGMrqGTTsv0Eb6f2due7oylXEcTzJD1ZTganSs3vbt 9fdlcWaRBecsymvALVROkygR0p+jYegy0tPpBy6y84wKQi54ZDcw6Htc3oRpSA== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1628504935; a=rsa-sha256; cv=none; b=BUBtIrmZLq4Bg18mTuEaRlGWejaSjoQkmo+8AVlITcOW6lKqtre3NMavPVBSdGx5GTpBGj KJTzV6a9T8NNkwckmpgwZrfJ28SX0tzXpKVdwL6VIxupHrgxI1nopDFW0G84jvKsfezJwB 4n2SN5qn2jKeyF96aV30s70T9FtKEC3zPUXe8rjXFKPyRCdBV/TkgM/V8v45g5WSHksGTY 8fdHqZigqrA1/JVBMbvLyDi9Z7gp2JiTjxWT3e0LEKP+gduTsbFEi03QLeBevGJOf3YVmF YQRtfahLtDsp3GzSBLMeOkdpqmAU2amcRs4wPZifMrcGnQmbmTIUlwxRl++PUg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Spam-Score: -2.91 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Queue-Id: 4FBB420FA1 X-Spam-Score: -2.91 X-Migadu-Scanner: scn1.migadu.com X-TUID: GOnsIG9zSTiz --=-=-= Content-Type: text/plain Hello, I have been investigating a problem that is visible both on the main guix publish server at https://ci.guix.gnu.org[1] and on the Cuirass build farm[2]. This error comes from the fact that the publish server does not accept the "guix substitute" connection requests within the %fetch-timeout duration of 5 seconds. The main guix publish server is using a cache. If a requested narinfo is not in the cache, it will be baked and the client receives a 404 error. Since ecaa102a58ad3ab0b42e04a3d10d7c761c05ec98 and the introduction of the bypass mechanism, small store items are directly returned. This means that the "narinfo-string" procedure can be called directly in the main publish thread. Running perf on the main publish server reveals that this procedure can be really expensive under IO pressure (GC running for example) because it opens a lot of files. I have observed that the "read-derivation-from-file" call can take up to 600 ms. If multiple clients were to ask narinfo of several items not yet cached, under IO pressure, I think that the publish server could become unresponsive and cause the timeout errors. The fact that Cuirass triggers the baking of successfully built derivations probably doesn't help here. Now regarding the timeout errors that are much more frequent on the Cuirass build farm, the cause varies a bit. The Cuirass publish server running on Berlin does not use a cache. This means that the "narinfo-string" procedure is called for each request, in the main thread. To fix those issues, a solution could be to run the "narinfo-string" in a separate thread, but it will make the publish server code even harder to understand. My proposition would be to get rid of the bypass mechanism and instead implement a retry when some substitutes are reported as being baked, as proposed by Miguel[3]. I think this is the most reasonable solution. This way, users won't receive 404 errors and start building substitutes that are being baked[4]. It will also allow the Cuirass build farm to use directly the main guix publish server, simplifying the current CI setup. There's a proposed patch attached, WDYT? Thanks, Mathieu [1]: https://issues.guix.gnu.org/49089 [2]: https://issues.guix.gnu.org/48468 [3]: https://issues.guix.gnu.org/44193#2 [4]: http://issues.guix.gnu.org/33370 --=-=-= Content-Type: application/octet-stream Content-Disposition: attachment; filename=patch Content-Transfer-Encoding: base64 RnJvbSBlM2M1MGM2ZDcwYWRmNzFiOGU4NGU2ODk1YjYyNWYzZDQzMmIxMGRkIE1vbiBTZXAgMTcg MDA6MDA6MDAgMjAwMQpGcm9tOiBNYXRoaWV1IE90aGFjZWhlIDxvdGhhY2VoZUBnbnUub3JnPgpE YXRlOiBNb24sIDkgQXVnIDIwMjEgMTE6MjU6NDEgKzAyMDAKU3ViamVjdDogW1BBVENIIDEvM10g bmFyaW5mbzogQWRkIG5hcmluZm8tcHJvbWlzZSBzdXBwb3J0LgoKKiBndWl4L25hcmluZm8uc2Nt ICg8bmFyaW5mby1wcm9taXNlPik6IE5ldyByZWNvcmQuCiglbWFrZS1uYXJpbmZvLXByb21pc2Us IG5hcmluZm8tcHJvbWlzZT8sIG5hcmluZm8tcHJvbWlzZS1wYXRoKTogTmV3CnByb2NlZHVyZXMu Ci0tLQogZ3VpeC9uYXJpbmZvLnNjbSB8IDExICsrKysrKysrKysrCiAxIGZpbGUgY2hhbmdlZCwg MTEgaW5zZXJ0aW9ucygrKQoKZGlmZiAtLWdpdCBhL2d1aXgvbmFyaW5mby5zY20gYi9ndWl4L25h cmluZm8uc2NtCmluZGV4IDcyZTBmNzVmZGEuLmMwZDQzMzhhMzUgMTAwNjQ0Ci0tLSBhL2d1aXgv bmFyaW5mby5zY20KKysrIGIvZ3VpeC9uYXJpbmZvLnNjbQpAQCAtNTEsNiArNTEsMTAgQEAKICAg ICAgICAgICAgIG5hcmluZm8tc2lnbmF0dXJlCiAgICAgICAgICAgICBuYXJpbmZvLWNvbnRlbnRz CiAKKyAgICAgICAgICAgICVtYWtlLW5hcmluZm8tcHJvbWlzZQorICAgICAgICAgICAgbmFyaW5m by1wcm9taXNlPworICAgICAgICAgICAgbmFyaW5mby1wcm9taXNlLXBhdGgKKwogICAgICAgICAg ICAgbmFyaW5mby1oYXNoLWFsZ29yaXRobSt2YWx1ZQogCiAgICAgICAgICAgICBuYXJpbmZvLWhh c2gtPnNoYTI1NgpAQCAtODksNiArOTMsMTMgQEAKICAgOzsgZm9yIG1vcmUgaW5mb3JtYXRpb24u CiAgIChjb250ZW50cyAgICAgbmFyaW5mby1jb250ZW50cykpCiAKKzs7IFRoaXMgZGVub3RlcyBh IG5hcmluZm8gdGhhdCBpcyBjdXJyZW50bHkgYmVpbmcgYmFrZWQgYnkgdGhlIHB1Ymxpc2ggc2Vy dmVyCis7OyBhbmQgZm9yIHdoaWNoIGFuIEhUVFAgMjAyIHJldHVybiBjb2RlIGhhcyBiZWVuIGlz c3VlZC4KKyhkZWZpbmUtcmVjb3JkLXR5cGUgPG5hcmluZm8tcHJvbWlzZT4KKyAgKCVtYWtlLW5h cmluZm8tcHJvbWlzZSBwYXRoKQorICBuYXJpbmZvLXByb21pc2U/CisgIChwYXRoICAgICAgICAg bmFyaW5mby1wcm9taXNlLXBhdGgpKQorCiAoZGVmaW5lIChuYXJpbmZvLWhhc2gtYWxnb3JpdGht K3ZhbHVlIG5hcmluZm8pCiAgICJSZXR1cm4gdHdvIHZhbHVlczogdGhlIGhhc2ggYWxnb3JpdGht IHVzZWQgYnkgTkFSSU5GTyBhbmQgaXRzIHZhbHVlIGFzIGEKIGJ5dGV2ZWN0b3IuIgotLSAKMi4z Mi4wCgoKRnJvbSA3ZGMwZjE5YWNhMWNmN2RhMGYyNTg2YmIzODZiOWI2ODM3NjczNTgxIE1vbiBT ZXAgMTcgMDA6MDA6MDAgMjAwMQpGcm9tOiBNYXRoaWV1IE90aGFjZWhlIDxvdGhhY2VoZUBnbnUu b3JnPgpEYXRlOiBNb24sIDkgQXVnIDIwMjEgMTE6Mjc6MDIgKzAyMDAKU3ViamVjdDogW1BBVENI IDIvM10gc2NyaXB0czogcHVibGlzaDogU2VuZCBhbiBIVFRQIDIwMiByZXR1cm4gY29kZSB3aGVu CiBiYWtpbmcgYSBzdWJzdGl0dXRlLgoKKiBndWl4L3NjcmlwdHMvcHVibGlzaC5zY20gKGJha2lu Zyk6IE5ldyBwcm9jZWR1cmUuCihyZW5kZXItbmFyaW5mby9jYWNoZWQpOiBVc2UgaXQgdG8gcmV0 dXJuIGFuIEhUVFAgMjAyIHJldHVybiBjb2RlIHdoZW4gYQpzdWJzdGl0dXRlIGlzIGJlaW5nIGJh a2VkLgotLS0KIGd1aXgvc2NyaXB0cy9wdWJsaXNoLnNjbSB8IDEyICsrKysrKysrKy0tLQogMSBm aWxlIGNoYW5nZWQsIDkgaW5zZXJ0aW9ucygrKSwgMyBkZWxldGlvbnMoLSkKCmRpZmYgLS1naXQg YS9ndWl4L3NjcmlwdHMvcHVibGlzaC5zY20gYi9ndWl4L3NjcmlwdHMvcHVibGlzaC5zY20KaW5k ZXggZjM1ZjgxZGMzNC4uYTI2ZDgzMjEyYiAxMDA2NDQKLS0tIGEvZ3VpeC9zY3JpcHRzL3B1Ymxp c2guc2NtCisrKyBiL2d1aXgvc2NyaXB0cy9wdWJsaXNoLnNjbQpAQCAtMzc5LDYgKzM3OSwxMyBA QCBSZWZlcmVuY2VzOiB+YX4lIgogICAgICAgICAgICAgICAgICAgICAgIChjYW5vbmljYWwtc2V4 cC0+c3RyaW5nIChzaWduZWQtc3RyaW5nIGluZm8pKSkpKQogICAgIChmb3JtYXQgI2YgIn5hU2ln bmF0dXJlOiAxO35hO35hfiUiIGluZm8gKGdldGhvc3RuYW1lKSBzaWduYXR1cmUpKSkKIAorKGRl ZmluZSogKGJha2luZyByZXF1ZXN0CisgICAgICAgICAgICAgICAgICM6a2V5IChwaHJhc2UgIldl J3JlIGJha2luZyBpdCIpKQorICAiUmVuZGVyIDIwMiByZXNwb25zZSBmb3IgUkVRVUVTVC4iCisg ICh2YWx1ZXMgKGJ1aWxkLXJlc3BvbnNlICM6Y29kZSAyMDIpCisgICAgICAgICAgKHN0cmluZy1h cHBlbmQgcGhyYXNlICI6ICIKKyAgICAgICAgICAgICAgICAgICAgICAgICAodXJpLXBhdGggKHJl cXVlc3QtdXJpIHJlcXVlc3QpKSkpKQorCiAoZGVmaW5lKiAobm90LWZvdW5kIHJlcXVlc3QKICAg ICAgICAgICAgICAgICAgICAgIzprZXkgKHBocmFzZSAiUmVzb3VyY2Ugbm90IGZvdW5kIikKICAg ICAgICAgICAgICAgICAgICAgdHRsKQpAQCAtNTkxLDkgKzU5OCw4IEBAIHJlcXVlc3RlZCB1c2lu ZyBQT09MLiIKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAjOnR0bCAzMDAgICAgICAg ICAgO3RlbXBvcmFyeQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICM6bmFyLXBhdGgg bmFyLXBhdGgKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAjOmNvbXByZXNzaW9ucyBj b21wcmVzc2lvbnMpCi0gICAgICAgICAgICAgICAobm90LWZvdW5kIHJlcXVlc3QKLSAgICAgICAg ICAgICAgICAgICAgICAgICAgIzpwaHJhc2UgIldlJ3JlIGJha2luZyBpdCIKLSAgICAgICAgICAg ICAgICAgICAgICAgICAgIzp0dGwgMzAwKSkpICAgICAgICAgIDtzaG91bGQgYmUgYXZhaWxhYmxl IHdpdGhpbiA1bQorICAgICAgICAgICAgICAgOzsgU2VlOiBodHRwczovL2lzc3Vlcy5ndWl4Lmdu dS5vcmcvNDQxOTMuCisgICAgICAgICAgICAgICAoYmFraW5nIHJlcXVlc3QpKSkKICAgICAgICAg ICAoZWxzZQogICAgICAgICAgICAobm90LWZvdW5kIHJlcXVlc3QgIzpwaHJhc2UgIiIgIzp0dGwg bmVnYXRpdmUtdHRsKSkpKSkKIAotLSAKMi4zMi4wCgoKRnJvbSAzZjIxZjYwYTY1ZjRiNjZjM2M3 NTMxNjBkYjFhNjhmMmRjNDg1ZDJjIE1vbiBTZXAgMTcgMDA6MDA6MDAgMjAwMQpGcm9tOiBNYXRo aWV1IE90aGFjZWhlIDxvdGhhY2VoZUBnbnUub3JnPgpEYXRlOiBNb24sIDkgQXVnIDIwMjEgMTE6 MzY6MjQgKzAyMDAKU3ViamVjdDogW1BBVENIIDMvM10gc3Vic3RpdHV0ZXM6IFJldHJ5IHRvIGZl dGNoIG5hcmluZm9zIGJlaW5nIGJha2VkLgoKZ3VpeC9zdWJzdGl0dXRlcy5zY20gKCVuYXItYmFr aW5nLWRlbGF5KTogTmV3IHZhcmlhYmxlLgooZmV0Y2gtbmFyaW5mb3MpOiBSZXR1cm4gPG5hcmlu Zm8tcHJvbWlzZT4gcmVjb3JkcyBmb3IgdGhlIG5hcmluZm9zIHRoYXQgYXJlCmN1cnJlbnRseSBi ZWluZyBmZXRjaGVkLgoobG9va3VwLW5hcmluZm9zKTogUmV0cnkgdG8gZmV0Y2ggdGhlIG5hcmlu Zm9zIHRoYXQgYXJlIGN1cnJlbnRseSBiZWluZyBiYWtlZCwKYWZ0ZXIgYSBkZWxheS4gRG8gbm90 IHJldHVybiB1bnRpbCB0aGVyZSBhcmUgbm8gbW9yZSBuYXJpbmZvcyBiZWluZyBiYWtlZC4KLS0t CiBndWl4L3N1YnN0aXR1dGVzLnNjbSB8IDU4ICsrKysrKysrKysrKysrKysrKysrKysrKysrKysr KysrLS0tLS0tLS0tLS0tCiAxIGZpbGUgY2hhbmdlZCwgNDMgaW5zZXJ0aW9ucygrKSwgMTUgZGVs ZXRpb25zKC0pCgpkaWZmIC0tZ2l0IGEvZ3VpeC9zdWJzdGl0dXRlcy5zY20gYi9ndWl4L3N1YnN0 aXR1dGVzLnNjbQppbmRleCBhNWM1NTRhY2ZmLi4zYTJmZGIxMjRkIDEwMDY0NAotLS0gYS9ndWl4 L3N1YnN0aXR1dGVzLnNjbQorKysgYi9ndWl4L3N1YnN0aXR1dGVzLnNjbQpAQCAtNzksNiArNzks MTEgQEAKICAgOzsgTGlrZXdpc2UsIGJ1dCBmb3IgdHJhbnNpZW50IGVycm9ycyBzdWNoIGFzIDUw NCAoIkdhdGV3YXkgdGltZW91dCIpLgogICAoKiA1IDYwKSkKIAorKGRlZmluZSAlbmFyLWJha2lu Zy1kZWxheQorICA7OyBOdW1iZXIgb2Ygc2Vjb25kcyB0byB3YWl0IGJlZm9yZSBhc2tpbmcgYWdh aW4gYSBuYXJpbmZvIHRoYXQgaGFzIGJlZW4KKyAgOzsgZGVjbGFyZWQgYXMgYmVpbmcgYmFrZWQg KDIwMikuCisgIDMwKQorCiAoZGVmaW5lICVuYXJpbmZvLWNhY2hlLWRpcmVjdG9yeQogICA7OyBB IGxvY2FsIGNhY2hlIG9mIG5hcmluZm9zLCB0byBhdm9pZCBnb2luZyB0byB0aGUgbmV0d29yay4g IE1vc3Qgb2YgdGhlCiAgIDs7IHRpbWUsICdndWl4IHN1YnN0aXR1dGUnIGlzIGNhbGxlZCBieSBn dWl4LWRhZW1vbiBhcyByb290IGFuZCBzdG9yZXMgaXRzCkBAIC0xODAsNyArMTg1LDkgQEAgaWYg ZmlsZSBkb2Vzbid0IGV4aXN0LCBhbmQgdGhlIG5hcmluZm8gb3RoZXJ3aXNlLiIKICAgICAgICAg ICAgICAgICAgICAgICAgICAob3Blbi1jb25uZWN0aW9uIGd1aXg6b3Blbi1jb25uZWN0aW9uLWZv ci11cmkpCiAgICAgICAgICAgICAgICAgICAgICAgICAgKG1ha2UtcHJvZ3Jlc3MtcmVwb3J0ZXIK ICAgICAgICAgICAgICAgICAgICAgICAgICAgKGNvbnN0IHByb2dyZXNzLXJlcG9ydGVyL3NpbGVu dCkpKQotICAiUmV0cmlldmUgYWxsIHRoZSBuYXJpbmZvcyBmb3IgUEFUSFMgZnJvbSB0aGUgY2Fj aGUgYXQgVVJMIGFuZCByZXR1cm4gdGhlbS4iCisgICJSZXRyaWV2ZSBhbGwgdGhlIG5hcmluZm9z IGZvciBQQVRIUyBmcm9tIHRoZSBjYWNoZSBhdCBVUkwgYW5kIHJldHVybiB0aGVtCithcyBhIGxp c3Qgb2YgPG5hcmluZm8+IGFuZCA8bmFyaW5mby1wcm9taXNlPiByZWNvcmRzLiAgVGhlIDxuYXJp bmZvLXByb21pc2VzPgorcmVjb3JkcyBkZW5vdGUgbmFyaW5mb3MgdGhhdCBhcmUgY3VycmVudGx5 IGJlaW5nIGJha2VkIGJ5IHRoZSBwdWJsaXNoIHNlcnZlci4iCiAgIChkZWZpbmUgcHJvZ3Jlc3Mt cmVwb3J0ZXIKICAgICAobWFrZS1wcm9ncmVzcy1yZXBvcnRlciAobGVuZ3RoIHBhdGhzKQogICAg ICAgICAgICAgICAgICAgICAgICAgICAgICM6dXJsIHVybCkpCkBAIC0yMjMsMTEgKzIzMCwxNSBA QCBpZiBmaWxlIGRvZXNuJ3QgZXhpc3QsIGFuZCB0aGUgbmFyaW5mbyBvdGhlcndpc2UuIgogICAg ICAgICAgICAgKGlmIGxlbgogICAgICAgICAgICAgICAgIChnZXQtYnl0ZXZlY3Rvci1uIHBvcnQg bGVuKQogICAgICAgICAgICAgICAgIChyZWFkLXRvLWVvZiBwb3J0KSkKLSAgICAgICAgICAgIChj YWNoZS1uYXJpbmZvISB1cmwgKGhhc2gtcGFydC0+cGF0aCBoYXNoLXBhcnQpICNmCi0gICAgICAg ICAgICAgICAgICAgICAgICAgICAgKGlmIChvciAoPSA0MDQgY29kZSkgKD0gMjAyIGNvZGUpKQot ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB0dGwKLSAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgJW5hcmluZm8tdHJhbnNpZW50LWVycm9yLXR0bCkpCi0gICAgICAgICAgICBy ZXN1bHQpKSkpCisgICAgICAgICAgICAoaWYgKD0gY29kZSAyMDIpCisgICAgICAgICAgICAgICAg KGNvbnMgKCVtYWtlLW5hcmluZm8tcHJvbWlzZSAoaGFzaC1wYXJ0LT5wYXRoIGhhc2gtcGFydCkp CisgICAgICAgICAgICAgICAgICAgICAgcmVzdWx0KQorICAgICAgICAgICAgICAgIChiZWdpbgor ICAgICAgICAgICAgICAgICAgKGNhY2hlLW5hcmluZm8hIHVybCAoaGFzaC1wYXJ0LT5wYXRoIGhh c2gtcGFydCkgI2YKKyAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAoaWYgKD0gNDA0 IGNvZGUpCisgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHR0bAorICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAlbmFyaW5mby10cmFuc2llbnQtZXJyb3It dHRsKSkKKyAgICAgICAgICAgICAgICAgIHJlc3VsdCkpKSkpKQogCiAgIChkZWZpbmUgKGRvLWZl dGNoIHVyaSkKICAgICAoY2FzZSAoYW5kPT4gdXJpIHVyaS1zY2hlbWUpCkBAIC0zMDAsNyArMzEx LDkgQEAgZm9yIFBBVEguIgogICAgICAgICAgICAgICAgICAgICAgICAgICAobWFrZS1wcm9ncmVz cy1yZXBvcnRlcgogICAgICAgICAgICAgICAgICAgICAgICAgICAgKGNvbnN0IHByb2dyZXNzLXJl cG9ydGVyL3NpbGVudCkpKQogICAiUmV0dXJuIHRoZSBuYXJpbmZvcyBmb3IgUEFUSFMsIGludm9r aW5nIHRoZSBzZXJ2ZXIgYXQgQ0FDSEUgd2hlbiBubwotaW5mb3JtYXRpb24gaXMgYXZhaWxhYmxl IGxvY2FsbHkuIgoraW5mb3JtYXRpb24gaXMgYXZhaWxhYmxlIGxvY2FsbHkuICBJZiBzb21lIG5h cmluZm9zIGFyZSBjdXJyZW50bHkgYmVpbmcgYmFrZWQsCityZXRyeSB0byBmZXRjaCBhZnRlciBh IGRlbGF5LiAgRG8gbm90IHJldHVybiB1bnRpbCB0aGVyZSBhcmUgbm8gbW9yZSBuYXJpbmZvcwor YmVpbmcgYmFrZWQuIgogICAobGV0LXZhbHVlcyAoKChjYWNoZWQgbWlzc2luZykKICAgICAgICAg ICAgICAgICAoZm9sZDIgKGxhbWJkYSAocGF0aCBjYWNoZWQgbWlzc2luZykKICAgICAgICAgICAg ICAgICAgICAgICAgICAobGV0LXZhbHVlcyAoKCh2YWxpZD8gdmFsdWUpCkBAIC0zMTMsMTQgKzMy NiwyOSBAQCBpbmZvcm1hdGlvbiBpcyBhdmFpbGFibGUgbG9jYWxseS4iCiAgICAgICAgICAgICAg ICAgICAgICAgICcoKQogICAgICAgICAgICAgICAgICAgICAgICAnKCkKICAgICAgICAgICAgICAg ICAgICAgICAgcGF0aHMpKSkKLSAgICAodmFsdWVzIChpZiAobnVsbD8gbWlzc2luZykKLSAgICAg ICAgICAgICAgICBjYWNoZWQKLSAgICAgICAgICAgICAgICAobGV0ICgobWlzc2luZyAoZmV0Y2gt bmFyaW5mb3MgY2FjaGUgbWlzc2luZwotICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAjOm9wZW4tY29ubmVjdGlvbiBvcGVuLWNvbm5lY3Rpb24KLSAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIzptYWtlLXByb2dyZXNzLXJl cG9ydGVyCi0gICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIG1h a2UtcHJvZ3Jlc3MtcmVwb3J0ZXIpKSkKLSAgICAgICAgICAgICAgICAgIChhcHBlbmQgY2FjaGVk IChvciBtaXNzaW5nICcoKSkpKSkKLSAgICAgICAgICAgIChsZW5ndGggbWlzc2luZykpKSkKKyAg ICAodmFsdWVzCisgICAgIChpZiAobnVsbD8gbWlzc2luZykKKyAgICAgICAgIGNhY2hlZAorICAg ICAgICAgKGxldCogKChtaXNzaW5nIChmZXRjaC1uYXJpbmZvcyBjYWNoZSBtaXNzaW5nCisgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICM6b3Blbi1jb25uZWN0aW9uIG9w ZW4tY29ubmVjdGlvbgorICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAj Om1ha2UtcHJvZ3Jlc3MtcmVwb3J0ZXIKKyAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgbWFrZS1wcm9ncmVzcy1yZXBvcnRlcikpCisgICAgICAgICAgICAgICAgKG5hcmlu Zm9zIChmaWx0ZXIgbmFyaW5mbz8gbWlzc2luZykpCisgICAgICAgICAgICAgICAgKHByb21pc2Vz IChtYXAgbmFyaW5mby1wcm9taXNlLXBhdGgKKyAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAoZmlsdGVyIG5hcmluZm8tcHJvbWlzZT8gbWlzc2luZykpKSkKKyAgICAgICAgICAgKGFwcGVu ZCBjYWNoZWQgbmFyaW5mb3MKKyAgICAgICAgICAgICAgICAgICAoaWYgKG51bGw/IHByb21pc2Vz KQorICAgICAgICAgICAgICAgICAgICAgICAnKCkKKyAgICAgICAgICAgICAgICAgICAgICAgKGJl Z2luCisgICAgICAgICAgICAgICAgICAgICAgICAgKGluZm8gKEdfICJTb21lIHN1YnN0aXR1dGVz IGFyZSBiZWluZyBiYWtlZCxcCisgcmV0cnlpbmcgaW4gfmEgc2Vjb25kcy5+JSIpCisgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgJW5hci1iYWtpbmctZGVsYXkpCisgICAgICAgICAgICAg ICAgICAgICAgICAgKHNsZWVwICVuYXItYmFraW5nLWRlbGF5KQorICAgICAgICAgICAgICAgICAg ICAgICAgIChsb29rdXAtbmFyaW5mb3MgY2FjaGUgcHJvbWlzZXMKKyAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICM6b3Blbi1jb25uZWN0aW9uIG9wZW4tY29ubmVjdGlv bgorICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIzptYWtlLXByb2dy ZXNzLXJlcG9ydGVyCisgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBt YWtlLXByb2dyZXNzLXJlcG9ydGVyKSkpKSkpCisgICAgIChsZW5ndGggbWlzc2luZykpKSkKIAog KGRlZmluZSogKGxvb2t1cC1uYXJpbmZvcy9kaXZlcnNlIGNhY2hlcyBwYXRocyBhdXRob3JpemVk PwogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICM6a2V5IChvcGVuLWNvbm5lY3Rp b24KLS0gCjIuMzIuMAoK --=-=-=--