From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id 8OcEHg2YFGHUSgEAgWs5BA (envelope-from ) for ; Thu, 12 Aug 2021 05:39:57 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id V/B7GQ2YFGE+VQAAbx9fmQ (envelope-from ) for ; Thu, 12 Aug 2021 03:39:57 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 77FC91BD5 for ; Thu, 12 Aug 2021 05:39:56 +0200 (CEST) Received: from localhost ([::1]:58978 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mE1ZS-0006MY-2L for larch@yhetil.org; Wed, 11 Aug 2021 23:39:54 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:40454) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mE1Yk-0006MO-GQ for emacs-orgmode@gnu.org; Wed, 11 Aug 2021 23:39:10 -0400 Received: from mail-ot1-x32a.google.com ([2607:f8b0:4864:20::32a]:42911) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mE1Yi-0006L1-3r for emacs-orgmode@gnu.org; Wed, 11 Aug 2021 23:39:10 -0400 Received: by mail-ot1-x32a.google.com with SMTP id d10-20020a9d4f0a0000b02904f51c5004e3so6055093otl.9 for ; Wed, 11 Aug 2021 20:39:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=HW0LRbJ6zqEINhDo2z67u9L58lZDb9yO44CfPMBfIeI=; b=G/v1/5ndlIqES6FZmOURdM8SlKYeoEgu19GqyZdCgtUgUMcnxTmWXjSzOYYANyI8iX P9kUgaINzuIfRXf/JC0qoBjcbt6ZREPJYi0aT+vKjZFY/gtqW2igt7A69D724+tKpAM/ 0ydedoswP0V2mZtUQDNqOHgGe80ntW8nAFVX/foe94hSK89daHKvM0gwAD6lQh1i0+tV izeqOsW4R/SQMhvJugBPP+4DvPqMiCU9/NXgXHUaR7CprpwMgdX8+/ICTglcS/9n6o2T Mz4GU8zxsFB7LYeqgY+1Xpbl4bNyYNd1BBcl+nuBOBMQkz//vyjPVLW9/iVuh/gc2foH J0ug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=HW0LRbJ6zqEINhDo2z67u9L58lZDb9yO44CfPMBfIeI=; b=ZKuQNXEXZH05rHoDQ7Pq1Ct/SqS5g1ekQ/rFwZ0qyQ4NIvylAe4fECt4RXqO3AUW2Z o7b+7ovW/iQnkgTeohMgpJ3CArUKjUXLS/18YVAZW++Y/wyjGvWWhysq++yJzdqdkhdW CuQT15hQRHdOP96W5R4otBg2uFYYXHcJJMCiviJlj06wwpFhZb6EaDioCSX0b3217Jp/ ECmgEX3l6oGzt49mYYachwJ0YM99Y1hOiAEUxdifZXXQCyQJDz75VOnqFPjJbYD1BpID SAF1Wgl3s0ZX+AbdtLa6smi7tccTimrwiqMTDQmCw+hNekk3i9HSv3cW8LwONu5ncaBT 2QEA== X-Gm-Message-State: AOAM531XOXCXdtmUIOhIFd+W3j3BGLGR0FVWqBAeqA6ceJE1Kq8uuhuh /CIZ4n1hhRRqXcdgFR/6/GejNPmw2H3MukiMSCY= X-Google-Smtp-Source: ABdhPJw26ba9J002nafw3XMtUFBKRZ4CmZgxa0lgvoRcU/MZOMdBDHRXbfAzSf3K6hYEYno287qPT9+EqBTreXJ6lHI= X-Received: by 2002:a9d:7618:: with SMTP id k24mr1749821otl.235.1628739546127; Wed, 11 Aug 2021 20:39:06 -0700 (PDT) MIME-Version: 1.0 References: <87wnos8vqi.fsf@localhost> <87tujw8orc.fsf@localhost> In-Reply-To: From: Tim Cross Date: Thu, 12 Aug 2021 13:38:55 +1000 Message-ID: Subject: Re: archiving speed [was Re: Tips on maintaining history in Org Mode] To: Samuel Wales Content-Type: multipart/alternative; boundary="0000000000001422c205c9547a31" Received-SPF: pass client-ip=2607:f8b0:4864:20::32a; envelope-from=theophilusx@gmail.com; helo=mail-ot1-x32a.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-orgmode@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Org-mode , Ihor Radchenko , David Masterson Errors-To: emacs-orgmode-bounces+larch=yhetil.org@gnu.org Sender: "Emacs-orgmode" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1628739596; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=HW0LRbJ6zqEINhDo2z67u9L58lZDb9yO44CfPMBfIeI=; b=an/W5uqegwN/KPDRRndRxfB+ilu4nwjFw5juJYM3xRq1SQAZ6Dx5DcCb62s0R5SkTgx0ce 2bSh/GKFFWVnfL3oP4rLbfhLIqfK7vX5ux/fyijpXIp+CfO+JNb0pyfdS+3abopCoF4a39 jwq5Xo2iOf6pC2OUQg92AOCQ8+w23BmbOYY2JXO0lwpYrIbFMOtwD7JxTHLbQGL/69CS2z sG9RFB15KX7KfNgg0B2JfxHogm593NNKDl8saJ95sxQH7C1MbnBfzcDWhsEMJ6fY1oxJE/ KawMq3/zTqsBKL+3NEP36jkGNUrV/kyl/8S/7/oOPoSGRCWOuACSNpL/HiSuDg== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1628739596; a=rsa-sha256; cv=none; b=iGOhrXkM+SpZ2ikDPOdVxlLvNXC9WFjEPEQHjv4YWsqkxo6EHeg3CCjqHQQ7oS2LU9ZUYH zQTIrf/GVX4rKFVTMrYviC7cIkCBDFnUxroTJNfldxPdv59GHqJEQeM8S3Zxd33Y+x/KpO vXtzZSTY0CQxzZcQeGb74x6Wl3bKaNO3jWgFJQEXOGK2nsHYu+qLWTOXLvWKNyVpW/ggKp j95G8cDLZ3BVEp32Gbqf4Dq0lMEX/yiEaGx6d9imCnRMa2h13Mk/FgLgN6o0RO0dXKxMuy IByr2GGCCvpc0+2U5fBQyrRECq1V8hFxNQ+E1ugVEiLPr0zlTVodWOXOOOIf8A== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20161025 header.b="G/v1/5nd"; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of emacs-orgmode-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=emacs-orgmode-bounces@gnu.org X-Migadu-Spam-Score: -1.31 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20161025 header.b="G/v1/5nd"; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of emacs-orgmode-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=emacs-orgmode-bounces@gnu.org X-Migadu-Queue-Id: 77FC91BD5 X-Spam-Score: -1.31 X-Migadu-Scanner: scn0.migadu.com X-TUID: w1sKeOVICeco --0000000000001422c205c9547a31 Content-Type: text/plain; charset="UTF-8" I think the problem with just using append to file is that it won't preserve the shape of the file. For example, if I had a file with * Notes ** Note 1 blah blah ** Note 2 blah blah * Tasks ** DONE task 1 ** TODO Task 2 and I decide to archive note 1 and task 1, I would like them to both appear under the same headings and with the same level. If the process just uses append to file, I can have this for the first archiving i.e. * Noes ** Note 1 * Tasks ** DONE task 1 but then later, I decide to archive note 2, if append file is used, I will end up with * Notes ** Note 1 * Taks ** DONE task 1 * Notes ** Note 2 which is not what I want. I want * Notes ** Note 1 ** Note 2 * Tasks ** DONE Task 1 So, if we want to preserve hierarchies in our archive files and not have everything jumbled up together, the system has to parse the file. If you are also using something like Categories, then even more work needs to be odne to update the category lists. What I tend to do is mark items with the ARCHIVE tag and leave them in the file and then every few months, move archived data to archive files. It can still get slow, but I don't do it often, so it isn't too much of a hassle. On Thu, 12 Aug 2021 at 08:23, Samuel Wales wrote: > thanks for the clarification. are you saying that, for every archived > entry, it calculates teh category property, using the original org > file, in order to add a category property to just one archived entry? > > that would certainly slow down more and more, but it sends me back to > my question about whether append to file would work. > i.e. build the single entry in a temporary buffer then write that > region to a file on disk. > > On 8/10/21, Ihor Radchenko wrote: > > Samuel Wales writes: > > > >> i should clarify. bulk archiving slows down even with /nonexistent/ > >> (have not tried empty) archives. as part of normal and expected > >> operation, bulk creates the archive for the first entry, and then > >> subsequent entries are added. those get slower and slower. > > > > That's what I suspected. I also see this and my suggestion helped > > archiving speed in my case. > > > >> i use (olpath category itags). i will try (file time) when i can, if > >> that still applies. my brain needs to be more operational. > > > > When you use category, every time you modify the original file (not the > > archive!), Org mode re-calculates *all* the categories in the original > > Org file. It happens for every single archived heading. If your original > > Org file is large, re-calculations make things extremely slow. > > > > Best, > > Ihor > > > > > -- > The Kafka Pandemic > > Please learn what misopathy is. > > https://thekafkapandemic.blogspot.com/2013/10/why-some-diseases-are-wronged.html > > -- regards, Tim -- Tim Cross --0000000000001422c205c9547a31 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
I think the problem with just using append to file is that= it won't preserve the shape of the file. For example, if I had a file = with=C2=A0

* Notes
** Note 1
=C2=A0 = =C2=A0blah blah
** Note 2 blah blah

* Ta= sks
** DONE task 1
** TODO Task 2

<= div>and I decide to archive note 1 and task 1, I would like them to both ap= pear under the same headings and with the same level. If the process just u= ses append to file, I can have this for the first archiving i.e.=C2=A0

* Noes
** Note 1

* = Tasks
** DONE task 1

but then later, I d= ecide to archive note 2, if append file is used, I will end up with=C2=A0

* Notes
** Note 1

* Taks
** DONE task 1

* Notes
** Note 2

which is not what I want. I want
=

* Notes
** Note 1
** Note 2

* Tasks
** DONE Task 1

So, if we want to preserve hierarchies in our archive files and not have e= verything jumbled up together, the system has to parse the file. If you are= also using something like Categories, then even more work needs to be odne= to update the category=C2=A0lists.=C2=A0

What I t= end to do is mark items with the ARCHIVE tag and leave them in the file and= then every few months, move archived data to archive files.=C2=A0 It can s= till get slow, but I don't do it often, so it isn't too much of a h= assle.


On Thu, 12 Aug 2021 at 08:23, Samuel Wales <<= a href=3D"mailto:samologist@gmail.com">samologist@gmail.com> wrote:<= br>
thanks for the c= larification.=C2=A0 are you saying that, for every archived
entry, it calculates teh category property, using the original org
file, in order to add a category property to just one archived entry?

that would certainly slow down more and more, but it sends me back to
my question about whether append to file would work.
i.e. build the single entry in a temporary buffer then write that
region to a file on disk.

On 8/10/21, Ihor Radchenko <yantar92@gmail.com> wrote:
> Samuel Wales <samologist@gmail.com> writes:
>
>> i should clarify.=C2=A0 bulk archiving slows down even with /nonex= istent/
>> (have not tried empty) archives.=C2=A0 as part of normal and expec= ted
>> operation, bulk creates the archive for the first entry, and then<= br> >> subsequent entries are added.=C2=A0 those get slower and slower. >
> That's what I suspected. I also see this and my suggestion helped<= br> > archiving speed in my case.
>
>> i use (olpath category itags).=C2=A0 i will try (file time) when i= can, if
>> that still applies.=C2=A0 my brain needs to be more operational. >
> When you use category, every time you modify the original file (not th= e
> archive!), Org mode re-calculates *all* the categories in the original=
> Org file. It happens for every single archived heading. If your origin= al
> Org file is large, re-calculations make things extremely slow.
>
> Best,
> Ihor
>


--
The Kafka Pandemic

Please learn what misopathy is.
https://thekafkapand= emic.blogspot.com/2013/10/why-some-diseases-are-wronged.html



--
regards,

Tim

--
Tim Cross

--0000000000001422c205c9547a31--