From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2.migadu.com ([2001:41d0:303:e16b::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms13.migadu.com with LMTPS id UAgsKx+N3WYqbAAAe85BDQ:P1 (envelope-from ) for ; Sun, 08 Sep 2024 11:40:15 +0000 Received: from aspmx1.migadu.com ([2001:41d0:303:e16b::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2.migadu.com with LMTPS id UAgsKx+N3WYqbAAAe85BDQ (envelope-from ) for ; Sun, 08 Sep 2024 13:40:15 +0200 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=fail ("body hash did not verify") header.d=tethera.net header.s=2024 header.b=TcqzyOLc; spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 135.181.149.255 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1725795615; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-owner:list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=iE0rPXq8xOLcUXILpf3bNAfwp9JM0N9TMAoDPTuTlrs=; b=EyvXcDuoqA8icikF7360HiTAAfW9XAXp4fO02VyyHMgfFm2xr26OnyWjRreKG5u25nSxyQ 9+l6KB1JOsRne/MNN8BbFNxqM1zH2Nzx7U3bF2wuAb15kFrnhoNE7N2C+s1gXXZWzBtHEP NwqIeR0BhQsNChYqDh61/s+reBLmHXuuNwTYaWSdhl/H0G18acsVD23PQfC2UzifkXCh4o t65b8HuAk1HqWveSnH1yx3lvztqF50o8AlVmOnMTetNn3vSnovHQMDfCJDO5h793kaNqgM yf0+3MEAouSHKUJayilLBndl1Lq40n9rIyyV6V2QVIRkDJ7P68xHhPsRyaKEoQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("body hash did not verify") header.d=tethera.net header.s=2024 header.b=TcqzyOLc; spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 135.181.149.255 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org; dmarc=none ARC-Seal: i=1; s=key1; d=yhetil.org; t=1725795615; a=rsa-sha256; cv=none; b=CltV7G4VZ/qzSaC7zidBkv+PH4XjnnhrIR8rI843GcQQUUzGMhSPlsbIl9Yip597NQvMKe S33rbEOmHwbhlkgWKu5judusTk/YQ+tJQtzoZYiR3+GTM52EaWSYjU2rS/gXr5Fg7ARA9c i+8ICD4yW6RUwCKgWBiaLeRV3y++9YffPyR/I6ReWD3IK+5thAS4m59by45wzEJHcOpw4o AmV96yi1D7F5mZy2SutX82l2L0ieEFXqNeGldpF6BYO4vR/8eiksjKItGoFgDotCZgNmck u1vfxMxBjkMib7tYuaGjiYfWgvuhYNGjU5Ff0vjOFYzi9Ik4OWP8eOLNoXK8Eg== Received: from mail.notmuchmail.org (yantan.tethera.net [135.181.149.255]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 7FEF5F119 for ; Sun, 08 Sep 2024 13:40:15 +0200 (CEST) Received: from yantan.tethera.net (localhost [127.0.0.1]) by mail.notmuchmail.org (Postfix) with ESMTP id A51F25E503; Sun, 8 Sep 2024 11:39:54 +0000 (UTC) Received: from phubs.tethera.net (phubs.tethera.net [192.99.9.157]) by mail.notmuchmail.org (Postfix) with ESMTPS id 364305E292 for ; Sun, 8 Sep 2024 11:39:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tethera.net; i=@tethera.net; q=dns/txt; s=2024; t=1725795588; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=KD2q1P9DTwhyf51IlVdmyCVpi9t5RoOGX8xOsY1hzws=; b=TcqzyOLcWScAlEAdvb7segMhuHuTf8VJ0sfMlBEv5DlmkyP/BgQ/RvGn1vbvTLlwAnhs2 DRaignVFYfCipf1FYh9oHZrrtm4l3vlzqGAhnhdtBF5L6p7CqYC4wj9CazM1R+IFwQFovNI QAHMlbXI9qIITZLNBnuOINWJYAx2Fc8VngqufpjiFWgFHo3slubTT9/ACwHhUs91jK+DvST sYU25F84GfgjEmgCuPsCR0gY6odmlGq0hoeXBDrOTpcfXmG3FBE9K0P59drrpPJp+SMtE1h f6RklrQrLP09iT27K3XBpCZR6DCz7wbqUwLnyNS3jEwQu+t3mJI6EI4WbnbQ== Received: from tethera.net (fctnnbsc51w-159-2-211-58.dhcp-dynamic.fibreop.nb.bellaliant.net [159.2.211.58]) by phubs.tethera.net (Postfix) with ESMTPS id D86AE180184; Sun, 8 Sep 2024 08:39:48 -0300 (ADT) Received: (nullmailer pid 286552 invoked by uid 1000); Sun, 08 Sep 2024 11:39:45 -0000 From: David Bremner To: notmuch@notmuchmail.org Subject: [PATCH v2 4/4] cli/git-remote: add export command Date: Sun, 8 Sep 2024 08:27:32 -0300 Message-ID: <20240908113937.286108-5-david@tethera.net> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240908113937.286108-1-david@tethera.net> References: <20240908113937.286108-1-david@tethera.net> MIME-Version: 1.0 Message-ID-Hash: 5MHDV5KU5KS6EDDL2YAFVIEO3YZLU6EQ X-Message-ID-Hash: 5MHDV5KU5KS6EDDL2YAFVIEO3YZLU6EQ X-MailFrom: bremner@tethera.net X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-notmuch.notmuchmail.org-0; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.3 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Migadu-Country: DE X-Migadu-Flow: FLOW_IN X-Migadu-Queue-Id: 7FEF5F119 X-Migadu-Scanner: mx13.migadu.com X-Migadu-Spam-Score: -1.35 X-Spam-Score: -1.35 X-TUID: 2O7Fi8l1RluU This enables the push command, and the helper is now feature complete. --- git-remote-notmuch.c | 136 +++++++++++++++++++++++++++++ performance-test/M07-git-remote.sh | 4 + performance-test/T08-git-remote.sh | 41 +++++++++ test/T860-git-remote.sh | 68 +++++++++++++++ 4 files changed, 249 insertions(+) diff --git a/git-remote-notmuch.c b/git-remote-notmuch.c index 39ba6bf3..de689474 100644 --- a/git-remote-notmuch.c +++ b/git-remote-notmuch.c @@ -280,6 +280,140 @@ cmd_import (notmuch_database_t *notmuch, store_lastmod (notmuch, nm_dir); } +static GString * +read_data () +{ + ssize_t nread; + size_t bytes; + size_t data_size; + + g_auto (GStrv) tokens = NULL; + + ASSERT ((nread = getline (&buffer, &buffer_len, stdin) != -1)); + + tokens = tokenize_buffer (); + + str2ul (tokens[1], &data_size); + + buffer = realloc (buffer, data_size + 1); + bytes = fread (buffer, 1, data_size, stdin); + ASSERT (bytes == data_size); + + buffer_len = data_size; + + return g_string_new_len (buffer, buffer_len); +} + +static void +free_string (GString *str) +{ + g_string_free (str, true); +} + +static void +cmd_export (notmuch_database_t *notmuch, const char *nm_dir) +{ + ssize_t nread; + + g_autoptr (GHashTable) blobs = NULL; + + ASSERT (blobs = g_hash_table_new_full ((GHashFunc) g_str_hash, + (GEqualFunc) g_str_equal, + g_free, (GDestroyNotify) free_string) + ); + + while ((nread = getline (&buffer, &buffer_len, stdin)) != -1) { + flog ("\texport %s\n", buffer); + if (STRNCMP_LITERAL (buffer, "done") == 0) { + break; + } else if (STRNCMP_LITERAL (buffer, "blob") == 0) { + GString *data; + g_auto (GStrv) tokens = NULL; + + + flog ("export blob\n"); + buffer_line (stdin); + + tokens = tokenize_buffer (); + + data = read_data (); + + flog ("\tmark%s\n", tokens[1]); + g_hash_table_insert (blobs, g_strdup (tokens[1]), data); + buffer_line (stdin); + } else if (STRNCMP_LITERAL (buffer, "commit") == 0) { + char *mid = NULL; + size_t mid_len = 0; + + flog ("export commit\n"); + + /* mark for commit (ignored) */ + buffer_line (stdin); + /* author (ignored) */ + buffer_line (stdin); + /* committer (ignored) */ + buffer_line (stdin); + + /* commit message (ignored) */ + (void) read_data (); + + while (strlen (buffer) > 0) { + g_autoptr (GString) mark = NULL; + g_autoptr (GString) path = NULL; + const GString *blob; + g_autofree char *basename = NULL; + notmuch_message_t *message; + const char *tok; + size_t tok_len; + size_t max_tok_len; + tag_op_list_t *tag_ops; + g_auto (GStrv) tokens = NULL; + + buffer_line (stdin); + if (strlen (buffer) == 0) + break; + + tokens = tokenize_buffer (); + flog ("looking for blob |%s|\n", tokens[2]); + ASSERT (blob = g_hash_table_lookup (blobs, tokens[2])); + + basename = g_path_get_dirname (tokens[3] + 6); + ASSERT (HEX_SUCCESS == + hex_decode (notmuch, basename, &mid, &mid_len)); + ASSERT (NOTMUCH_STATUS_SUCCESS == + notmuch_database_find_message (notmuch, mid, &message)); + ASSERT (message); + + ASSERT (NOTMUCH_STATUS_SUCCESS == + notmuch_message_freeze (message)); + + tag_ops = tag_op_list_create (message); + tok = blob->str; + max_tok_len = blob->len; + tok_len = 0; + while ((tok_len < max_tok_len) && + (tok = strsplit_len (tok + tok_len, '\n', &tok_len)) != NULL) { + const char *tag = talloc_strndup (message, tok, tok_len); + ASSERT (0 == tag_op_list_append (tag_ops, tag, false)); + } + + ASSERT (NOTMUCH_STATUS_SUCCESS == + tag_op_list_apply (message, tag_ops, TAG_FLAG_REMOVE_ALL)); + + ASSERT (NOTMUCH_STATUS_SUCCESS == + notmuch_message_thaw (message)); + + notmuch_message_destroy (message); + } + puts ("ok refs/heads/master"); + } + + } + store_lastmod (notmuch, nm_dir); + puts (""); +} + + /* stubs since we cannot link with notmuch.o */ const notmuch_opt_desc_t notmuch_shared_options[] = { { } @@ -408,6 +542,8 @@ main (int argc, char *argv[]) if (STRNCMP_LITERAL (s, "capabilities") == 0) cmd_capabilities (); + else if (STRNCMP_LITERAL (s, "export") == 0) + cmd_export (db, nm_dir); else if (STRNCMP_LITERAL (s, "import") == 0) cmd_import (db, nm_dir, uuid, lastmod); else if (STRNCMP_LITERAL (s, "list") == 0) diff --git a/performance-test/M07-git-remote.sh b/performance-test/M07-git-remote.sh index efce18a6..526cd856 100755 --- a/performance-test/M07-git-remote.sh +++ b/performance-test/M07-git-remote.sh @@ -6,6 +6,7 @@ test_description='search' mkdir repo export GIT_DIR=`pwd`/repo +MAKE_EXPORT_PY=$NOTMUCH_SRCDIR/test/make-export.py memory_start @@ -13,4 +14,7 @@ echo "import refs/heads/master" > import.in memory_run "import" "git-remote-notmuch origin notmuch:// >import.out export.in +memory_run "export" "git-remote-notmuch origin notmuch:// >export.out > $path + fi + done + + cd $olddir +} + time_start time_run 'clone --bare' "git clone --quiet --bare -b master notmuch::default default.git" time_run 'clone' "git clone --quiet -b master notmuch:// repo" +time_run "push (no changes)" "git -C repo push --quiet origin master" + +add_tags repo 10 +git -C repo add -u +git -C repo commit --quiet -m'add tags to 10% of messages' +time_run "push (10% changed)" "git -C repo push --quiet origin master" + +add_tags repo 4 +git -C repo add -u +git -C repo commit --quiet -m'add tags to 25% of messages' +time_run "push (25% changed)" "git -C repo push --quiet origin master" + +add_tags repo 2 +git -C repo add -u +git -C repo commit --quiet -m'add tags to 50% of messages' +time_run "push (50% changed)" "git -C repo push --quiet origin master" + +hash=$(git -C repo hash-object --stdin -w < /dev/null) +# replace all files with empty files +git -C repo ls-tree -r HEAD | sed "s/blob [^\t]*/blob $hash/" \ + | git -C repo update-index --index-info +git -C repo commit --quiet -m'zero tags' 2>>log.txt 1>&2 + +time_run "push (rem. all tags)" "git -C repo push --quiet origin master" + time_done diff --git a/test/T860-git-remote.sh b/test/T860-git-remote.sh index 97b8e4f9..014ffc2b 100755 --- a/test/T860-git-remote.sh +++ b/test/T860-git-remote.sh @@ -112,4 +112,72 @@ zznew EOF test_expect_equal_file EXPECTED repo/$TAG_FILE +test_begin_subtest 'export runs' +run_helper < OUTPUT +export +blob +mark :1 +data 10 +tag1 +tag2 + +commit refs/heads/master +mark :2 +author Notmuch Test Suite 1234 +0000 +committer Notmuch Test Suite 1234 +0000 +data 8 +ignored +M 100644 :1 $TAG_FILE + +done + +EOF +cat < EXPECTED +ok refs/heads/master + +EOF +test_expect_equal_file EXPECTED OUTPUT + +# this test depends on the previous one +test_begin_subtest 'export modifies database' +notmuch dump id:4EFC743A.3060609@april.org | tail -n 1 > OUTPUT +cat < EXPECTED ++tag1 +tag2 -- id:4EFC743A.3060609@april.org +EOF +test_expect_equal_file EXPECTED OUTPUT + +test_begin_subtest 'restore via export' +notmuch dump > BEFORE +python3 $MAKE_EXPORT_PY > export.in +notmuch tag +transient -- id:4EFC743A.3060609@april.org +run_helper < export.in > OUTPUT +notmuch dump id:4EFC743A.3060609@april.org | tail -n 1 > OUTPUT +cat < EXPECTED ++tag1 +tag2 -- id:4EFC743A.3060609@april.org +EOF +test_expect_equal_file EXPECTED OUTPUT + +test_begin_subtest "push updates database" +git -C repo push origin master +notmuch dump id:4EFC743A.3060609@april.org | tail -n 1 > OUTPUT +cat < EXPECTED ++tag1 +tag2 -- id:4EFC743A.3060609@april.org +EOF +test_expect_equal_file EXPECTED OUTPUT + +test_begin_subtest "adding tag via repo" +cat<repo/$TAG_FILE +tag1 +tag2 +tag3 +EOF +git -C repo add $TAG_FILE +git -C repo commit -m 'testing push' +git -C repo push origin master +notmuch dump id:4EFC743A.3060609@april.org | tail -n 1 > OUTPUT +cat < EXPECTED ++tag1 +tag2 +tag3 -- id:4EFC743A.3060609@april.org +EOF +test_expect_equal_file EXPECTED OUTPUT + test_done -- 2.43.0