From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1.migadu.com ([2001:41d0:403:58f0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms13.migadu.com with LMTPS id SISeNVZLz2Z0BgAA62LTzQ:P1 (envelope-from ) for ; Wed, 28 Aug 2024 16:07:51 +0000 Received: from aspmx1.migadu.com ([2001:41d0:403:58f0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1.migadu.com with LMTPS id SISeNVZLz2Z0BgAA62LTzQ (envelope-from ) for ; Wed, 28 Aug 2024 18:07:50 +0200 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=fail ("body hash did not verify") header.d=tethera.net header.s=2024 header.b=JJVvbmxB; dmarc=none; spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 135.181.149.255 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1724861270; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-owner:list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=W+/HqMTnmJcUSzwgs46yMf0oIADSdppvBF3o8UPO2Eg=; b=hUANl6rnfeFK3U7NaJ8Ag1jwdNgxI0ukbSNRo32rC3sSbckdYkWZY3Jj8ooPok+l2BssLb mLaXPYOE4SDBhvGUSGkIdnWXaXvcNtxr16mcbROQktyciOUClbmY3rGYyMja9l5a5cKS3H co0slaJ+x3zSiyMH38kbtxZI5x9710Sz8QWx7whazDtXg1ZnzoYARtkDHSCfY40krqW4c5 0Ra5O6airuqG+oxZfCbLNwU4DADdpk0pTKyC3sCxiaMsU7elbF/M29OvM0RxRD8B3wT/PF R3V/PSqVu+7HkDctghXYf1363mxvaRPyOFu9vD3oL7/6gMeZPHeru1zP4wKIhg== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1724861270; a=rsa-sha256; cv=none; b=iAp+dC8N6yd23IRwWMtGCXpq8lUU6YCZq2DiOcSiyRYU4WQDrl/ERq0I+DOytrASLNqMtW sFWw5jmtPwMFwF3ahYUu3muPNmV8jC9Uia6VfJc6+CrMQADcM1Sai6+Kvm0E0kA/xCBL+8 sl7jmMbmKDlvKcgsEWPXMq5A/hNXuqpDhNp62FrhCMEiJE38BqTGtv33YgVImcmj778FN8 wLjPKyLyodXiM9x+E+DaTm1uAijFKYUvLl1RNXktFgUGFttvbUcGGenC4nvbXVsPvjiP0j O06cRU4oHIM8iiosw2kB73Wswy6YeSvynqd0grp/EhXPb1sKj0L7+n8k+X5ubw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("body hash did not verify") header.d=tethera.net header.s=2024 header.b=JJVvbmxB; dmarc=none; spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 135.181.149.255 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org Received: from mail.notmuchmail.org (yantan.tethera.net [135.181.149.255]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id B42B773F20 for ; Wed, 28 Aug 2024 18:07:50 +0200 (CEST) Received: from yantan.tethera.net (localhost [127.0.0.1]) by mail.notmuchmail.org (Postfix) with ESMTP id 91B265F7FE; Wed, 28 Aug 2024 16:07:30 +0000 (UTC) Received: from phubs.tethera.net (phubs.tethera.net [192.99.9.157]) by mail.notmuchmail.org (Postfix) with ESMTPS id C47C85F7EE for ; Wed, 28 Aug 2024 16:07:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tethera.net; i=@tethera.net; q=dns/txt; s=2024; t=1724861245; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=gwClRCmjUDvapg30Qg6OoFhzS5r2hru6QrgqYp7q2Ww=; b=JJVvbmxBuagEc1WvBdTm8WlUbe35pVrZ+vDzDvnbGcjtbSQituFAlMObxBpoeJV5WS42a 7LPqv/4cKYPvioPn4hONl80JQNK5kVyxMsi8ZHg/N/bhzpovilj6KWfU67sPNrnd4JtmviC NTugJqFPH6o7QHD/7eS3hLTVVZUcUEyUoH4JnL6f7Q+9A2JNNvT/U7rBYHCieN6Pdkg2++f xdWjTXoLEvWuJlKh49hB0of5/09QU9w3/xqUGQAHgAJ09RXv7JGJRjd65SWC9Ro+S5osGqW k5gOqhntsqpdSTLxrAhif+d82Sw8oOtxk1bOEvYyResBz6RdmX0w9Uzw5eRg== Received: from tethera.net (unknown [207.194.98.244]) by phubs.tethera.net (Postfix) with ESMTPS id 0761B18006E; Wed, 28 Aug 2024 13:07:24 -0300 (ADT) Received: (nullmailer pid 869228 invoked by uid 1000); Wed, 28 Aug 2024 16:07:16 -0000 From: David Bremner To: notmuch@notmuchmail.org Subject: [PATCH 4/5] cli/git-remote: add export command Date: Wed, 28 Aug 2024 08:45:57 -0700 Message-ID: <20240828160710.866567-5-david@tethera.net> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240828160710.866567-1-david@tethera.net> References: <20240828160710.866567-1-david@tethera.net> MIME-Version: 1.0 Message-ID-Hash: NIRBYHWPOTX7GTZDQG6QSRMZ4TENWQLV X-Message-ID-Hash: NIRBYHWPOTX7GTZDQG6QSRMZ4TENWQLV X-MailFrom: bremner@tethera.net X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-notmuch.notmuchmail.org-0; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.3 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Migadu-Flow: FLOW_IN X-Migadu-Country: DE X-Migadu-Spam-Score: 1.44 X-Spam-Score: 1.44 X-Migadu-Queue-Id: B42B773F20 X-Migadu-Scanner: mx11.migadu.com X-TUID: onPe+g+cAc5P This enables the push command, and the helper is now feature complete. --- git-remote-notmuch.c | 157 ++++++++++++++++++++++++++++++++++++++++ test/T860-git-remote.sh | 69 ++++++++++++++++++ test/make-export.py | 44 +++++++++++ 3 files changed, 270 insertions(+) create mode 100644 test/make-export.py diff --git a/git-remote-notmuch.c b/git-remote-notmuch.c index a4ed98e2..35579004 100644 --- a/git-remote-notmuch.c +++ b/git-remote-notmuch.c @@ -224,6 +224,161 @@ cmd_import (notmuch_database_t *notmuch) { store_lastmod(notmuch, nm_dir); } +static GString * +get_tok (const char *line, size_t index) { + + const char *tok = line; + size_t tok_len =0; + + for (size_t count = 0; count<=index; count++) { + ASSERT(tok = strsplit_len (tok+tok_len, ' ', &tok_len)); + } + + return g_string_new_len (tok, tok_len); +} + +static GString * +read_data (char **line_p, size_t *len_p) { + ssize_t nread; + size_t bytes; + size_t data_size; + const char *tok; + size_t tok_len =0; + + ASSERT (line_p); + ASSERT (len_p); + ASSERT((nread = getline(line_p, len_p, stdin) != -1)); + + tok=*line_p; + for (size_t count = 0; count<=1; count++) { + ASSERT(tok = strsplit_len (tok+tok_len, ' ', &tok_len)); + } + + /* TODO: don't ignore tok_len */ + ASSERT(sscanf (tok, "%zu", &data_size) == 1); + + *line_p = realloc (*line_p, data_size+1); + bytes = fread (*line_p, 1, data_size, stdin); + ASSERT (bytes == data_size); + + *len_p = data_size; + + return g_string_new_len (*line_p, *len_p); +} + +static void free_string (GString *str) { + g_string_free (str, true); +} + +static void +cmd_export (notmuch_database_t *notmuch) +{ + ssize_t nread; + size_t len = 0; + char *line = NULL; + GHashTable* blobs; + + ASSERT(blobs = g_hash_table_new_full ((GHashFunc)g_string_hash, + (GEqualFunc)g_string_equal, + (GDestroyNotify)free_string, (GDestroyNotify)free_string)); + + while ((nread = getline(&line, &len, stdin)) != -1) { + flog ("\texport %s\n", line); + if (STRNCMP_LITERAL (line, "done") == 0) { + break; + } + else if (STRNCMP_LITERAL (line, "blob") == 0) { + GString *mark; + GString *data; + + flog ("export blob\n"); + read_one_line (stdin, &line, &len); + mark = get_tok (line, 1); + + data = read_data (&line, &len); + + g_hash_table_insert (blobs, mark, data); + read_one_line (stdin, &line, &len); + /* TODO free things */ + } + + else if (STRNCMP_LITERAL (line, "commit") == 0) { + char *mid = NULL; + size_t mid_len = 0; + + flog ("export commit\n"); + + /* mark for commit (ignored) */ + read_one_line (stdin, &line, &len); + /* author (ignored) */ + read_one_line (stdin, &line, &len); + /* committer (ignored) */ + read_one_line (stdin, &line, &len); + + /* commit message (ignored) */ + (void)read_data (&line, &len); + + while (strlen (line) > 0) { + GString *mark, *path; + GBytes *blob; + char *basename; + notmuch_message_t *message; + char *tag; + const char *tok; + size_t tok_len; + size_t max_tok_len; + + read_one_line (stdin, &line, &len); + if (strlen (line) ==0) + break; + + ASSERT(mark = get_tok (line, 2)); + ASSERT(blob = g_hash_table_lookup (blobs, mark)); + free_string (mark); + + ASSERT(path = get_tok (line, 3)); + + basename = g_path_get_dirname (path->str+6); + ASSERT(HEX_SUCCESS == + hex_decode (notmuch, basename, &mid, &mid_len)); + g_free (basename); + free_string (path); + + ASSERT(NOTMUCH_STATUS_SUCCESS == + notmuch_database_find_message (notmuch, mid, &message)); + ASSERT(message); + + + ASSERT(NOTMUCH_STATUS_SUCCESS == + notmuch_message_freeze (message)); + + ASSERT(NOTMUCH_STATUS_SUCCESS == + notmuch_message_remove_all_tags (message)); + + tok = g_bytes_get_data (blob, &max_tok_len); + tok_len = 0; + while ((tok_len < max_tok_len) && + (tok = strsplit_len (tok + tok_len, '\n', &tok_len)) != NULL) { + tag = strndup (tok, tok_len); + ASSERT(NOTMUCH_STATUS_SUCCESS == + notmuch_message_add_tag (message, tag)); + free (tag); + } + + ASSERT(NOTMUCH_STATUS_SUCCESS == + notmuch_message_thaw (message)); + + notmuch_message_destroy (message); + } + puts("ok refs/heads/master"); + } + + } + store_lastmod(notmuch, nm_dir); + puts(""); + g_hash_table_destroy (blobs); +} + int main (int argc, char *argv[]) { @@ -296,6 +451,8 @@ main (int argc, char *argv[]) if (STRNCMP_LITERAL (s, "capabilities")== 0) cmd_capabilities (); + else if (STRNCMP_LITERAL (s, "export") == 0) + cmd_export (db); else if (STRNCMP_LITERAL (s, "import") == 0) cmd_import (db); else if (STRNCMP_LITERAL (s, "list") == 0) diff --git a/test/T860-git-remote.sh b/test/T860-git-remote.sh index d5809e6b..6dc55391 100755 --- a/test/T860-git-remote.sh +++ b/test/T860-git-remote.sh @@ -21,6 +21,7 @@ export GIT_COMMITTER_EMAIL="notmuch@example.com" export GIT_REMOTE_NM_DEBUG="s" export GIT_REMOTE_NM_LOG=grn-log.txt EXPECTED=$NOTMUCH_SRCDIR/test/git-remote.expected-output +MAKE_EXPORT_PY=$NOTMUCH_SRCDIR/test/make-export.py TAG_FILE="87/b1/4EFC743A.3060609@april.org/tags" @@ -78,4 +79,72 @@ zznew EOF test_expect_equal_file EXPECTED repo/$TAG_FILE +test_begin_subtest 'export runs' +run_helper < OUTPUT +export +blob +mark :1 +data 10 +tag1 +tag2 + +commit refs/heads/master +mark :2 +author Notmuch Test Suite 1234 +0000 +committer Notmuch Test Suite 1234 +0000 +data 8 +ignored +M 100644 :1 $TAG_FILE + +done + +EOF +cat < EXPECTED +ok refs/heads/master + +EOF +test_expect_equal_file EXPECTED OUTPUT + +# this test depends on the previous one +test_begin_subtest 'export modifies database' +notmuch dump id:4EFC743A.3060609@april.org | tail -n 1 > OUTPUT +cat < EXPECTED ++tag1 +tag2 -- id:4EFC743A.3060609@april.org +EOF +test_expect_equal_file EXPECTED OUTPUT + +test_begin_subtest 'restore via export' +notmuch dump > BEFORE +python3 $MAKE_EXPORT_PY > export.in +notmuch tag +transient -- id:4EFC743A.3060609@april.org +run_helper < export.in > OUTPUT +notmuch dump id:4EFC743A.3060609@april.org | tail -n 1 > OUTPUT +cat < EXPECTED ++tag1 +tag2 -- id:4EFC743A.3060609@april.org +EOF +test_expect_equal_file EXPECTED OUTPUT + +test_begin_subtest "push updates database" +git -C repo push origin master +notmuch dump id:4EFC743A.3060609@april.org | tail -n 1 > OUTPUT +cat < EXPECTED ++tag1 +tag2 -- id:4EFC743A.3060609@april.org +EOF +test_expect_equal_file EXPECTED OUTPUT + +test_begin_subtest "adding tag via repo" +cat<repo/$TAG_FILE +tag1 +tag2 +tag3 +EOF +git -C repo add $TAG_FILE +git -C repo commit -m 'testing push' +git -C repo push origin master +notmuch dump id:4EFC743A.3060609@april.org | tail -n 1 > OUTPUT +cat < EXPECTED ++tag1 +tag2 +tag3 -- id:4EFC743A.3060609@april.org +EOF +test_expect_equal_file EXPECTED OUTPUT + test_done diff --git a/test/make-export.py b/test/make-export.py new file mode 100644 index 00000000..3837dc3a --- /dev/null +++ b/test/make-export.py @@ -0,0 +1,44 @@ +# generate a test input for the 'export' subcommand of the +# git-remote-notmuch helper + +from notmuch2 import Database +from time import time +from hashlib import sha1 + +def hexencode(str): + output_charset = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+-_@=.," + out = "" + for char in str: + if not char in output_charset: + out+= f"%{ord(char):x}" + else: + out+= char + return out + +db = Database(config=Database.CONFIG.SEARCH) + +count=1 +print("export") +mark={} + +for msg in db.messages(""): + mark[msg.messageid]=count + blob="" + for tag in msg.tags: + blob += f"{tag}\n" + print (f"blob\nmark :{count}"); + print (f"data {len(blob)}\n{blob}") + count=count+1 + +print (f"\ncommit refs/heads/master\nmark :{count+1}") +ctime = int(time()) +print (f"author Notmuch Test Suite {ctime} +0000") +print (f"committer Notmuch Test Suite {ctime} +0000") +print (f"data 8\nignored") + +for msg in db.messages(""): + digest = sha1(msg.messageid.encode('utf8')).hexdigest() + filename = hexencode(msg.messageid) + print (f"M 100644 :{mark[msg.messageid]} {digest[0:2]}/{digest[2:4]}/{filename}/tags") + +print("\ndone\n") -- 2.43.0