From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Adam Porter Newsgroups: gmane.emacs.devel Subject: Re: Turning on savehist-mode by default Date: Fri, 22 Dec 2023 05:52:15 -0600 Message-ID: References: <461a0999-6005-4aa4-a4b3-988123855f92@alphapapa.net> <7942be6a-de0e-493e-85a7-97bfbbe41026@alphapapa.net> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="18878"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Mozilla Thunderbird Cc: eliz@gnu.org, emacs-devel@gnu.org, sbaugh@catern.com To: Stefan Kangas , luangruo@yahoo.com Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Fri Dec 22 12:52:51 2023 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1rGe5D-0004fC-2Q for ged-emacs-devel@m.gmane-mx.org; Fri, 22 Dec 2023 12:52:51 +0100 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rGe4m-0006iP-AA; Fri, 22 Dec 2023 06:52:24 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rGe4k-0006cD-L5 for emacs-devel@gnu.org; Fri, 22 Dec 2023 06:52:22 -0500 Original-Received: from shrimp.cherry.relay.mailchannels.net ([23.83.223.164]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rGe4i-0005bF-Hv; Fri, 22 Dec 2023 06:52:22 -0500 X-Sender-Id: dreamhost|x-authsender|adam@alphapapa.net Original-Received: from relay.mailchannels.net (localhost [127.0.0.1]) by relay.mailchannels.net (Postfix) with ESMTP id 9190F142CB7; Fri, 22 Dec 2023 11:52:17 +0000 (UTC) Original-Received: from pdx1-sub0-mail-a278.dreamhost.com (unknown [127.0.0.6]) (Authenticated sender: dreamhost) by relay.mailchannels.net (Postfix) with ESMTPA id 214B91417D8; Fri, 22 Dec 2023 11:52:17 +0000 (UTC) ARC-Seal: i=1; s=arc-2022; d=mailchannels.net; t=1703245937; a=rsa-sha256; cv=none; b=lLXbK6jSrojruokXBBhEYEsprAaI2atU/jxCMktPTyFhSR53CU18/gU9bjuAZ37v5pIyvw 1VLbo7hZkw7q21H2hR/uJFDy8kmK+gGsb7pkS82LUAT3u6GiphGiK1OZE5XQUgtww95R1A kFHnvWlM8/3MZ6SMGIBLELNTbxtLGhl1nzz+IfBwQNfQ7zu5JSAiJRSGSbpz+lD01Z9Da1 gFtICPk2y47xY8kiT7oO0N3TxJpu+ao7wIgnrjH8mEm/mFyNgPKdHkrhQjQdve3D4FlPWT 3Al8CBklVrrteCmWoO2SV0JkoQy18N+1Hb2hclAGseABjVBjaI72YLk7uVyfkQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=mailchannels.net; s=arc-2022; t=1703245937; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=EfCyicDVdnytDDnNkgpHko9QFohVu1ra0pgFxCzRUjM=; b=yy84c4OVpiW0Rsv7iqeHD8OKcjjzPDpwHAMrmEXomtd01nTiV4Y1r6WypN+GdySBJyEwgb xqbsFfSx/AFDyy8bpqBpoMgn2AZWNzKEwT3/eWuy5PykdznZUOIZAfws2bue78lerUREhb boI8MG65DMs038b6YLLJbsmwU+9LCPoYNl+AfGWgNwtkQZxZSsvtWZcOZiBIxATN7p+EHu 6NeCgx+GGFykxR8K4p1fr8ObMNCdQ9Jfb354/gY9vBmblagsSfIRUUjpviyIJTqsm8J5Gr 1Ln90ztijWWIejdlWP4YgBgsSRYBZKEi8rGv7s8shjLg75Vf+Lk8G2ss5VxGIQ== ARC-Authentication-Results: i=1; rspamd-856c7f878f-fhcmf; auth=pass smtp.auth=dreamhost smtp.mailfrom=adam@alphapapa.net X-Sender-Id: dreamhost|x-authsender|adam@alphapapa.net X-MC-Relay: Neutral X-MailChannels-SenderId: dreamhost|x-authsender|adam@alphapapa.net X-MailChannels-Auth-Id: dreamhost X-Glossy-Ski: 0914328c64820aac_1703245937408_4087718718 X-MC-Loop-Signature: 1703245937408:1089488838 X-MC-Ingress-Time: 1703245937408 Original-Received: from pdx1-sub0-mail-a278.dreamhost.com (pop.dreamhost.com [64.90.62.162]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384) by 100.125.117.212 (trex/6.9.2); Fri, 22 Dec 2023 11:52:17 +0000 Original-Received: from [10.66.0.162] (unknown [91.193.232.98]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: adam@alphapapa.net) by pdx1-sub0-mail-a278.dreamhost.com (Postfix) with ESMTPSA id 4SxQdX1xzvz8m; Fri, 22 Dec 2023 03:52:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=alphapapa.net; s=dreamhost; t=1703245936; bh=EfCyicDVdnytDDnNkgpHko9QFohVu1ra0pgFxCzRUjM=; h=Date:Subject:To:Cc:From:Content-Type:Content-Transfer-Encoding; b=jcfaH1CU3v7jU65w5avaQ8g5h7dgv3uznJNs7Y+FyndCTK/CaF4RqM0eaGVHpBYYf Cgvtv5txoKipNN1ZXp8SZ68lXGE2Mdqx9+CYqY73UdaEsvG7xG8iLj7zgUEgEStZuQ LwNwq3BqDK8MdPK1QLVjHtrmOZu9SeTGMHdeyFiQkZbJqGOhXWElZMfmgwbypiAer4 CA8kD9pY8Kjn1NAQqMVx3Tgh26cDwzVwuSXnLU5JLWJ4OX43D7gjpr6RoK5emZpYko AjHo6lR3P8T9FSgxXCijnDCMLmfq5x+1LnkgFsYCpAamqOZW7wo8MCZ1n7vK3McaOI hffChThM1WBVg== Content-Language: en-US In-Reply-To: Received-SPF: neutral client-ip=23.83.223.164; envelope-from=adam@alphapapa.net; helo=shrimp.cherry.relay.mailchannels.net X-Spam_score_int: -12 X-Spam_score: -1.3 X-Spam_bar: - X-Spam_report: (-1.3 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_NEUTRAL=0.779, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.devel:314070 Archived-At: On 12/22/23 04:45, Stefan Kangas wrote: > Adam Porter writes: > >> It doesn't seem like an easy problem. My guess as to the most "correct" >> solution would be to limit how much time can be spent serializing the >> data, both as a whole and individual elements, and to cancel or >> interrupt the serialization of the whole or a part if the limit is >> exceeded--the goal being to avoid UI pauses. Then the limit could be >> disabled when desired (e.g. when exiting Emacs, wanting to be sure the >> data is saved). But doing that would require some bookkeeping in the >> background, which seems like it might be excessive complexity for what >> should be a simple feature. > > Thinking about this a bit more, why is this not to be considered a bug > in ement? > > It does seem like that package is quite unusual in that it stores both > non-serializable and large data in the -history variable. Perhaps we > should simply say that you should not do that? There seems to be a misunderstanding. No code in Ement stores any data in the command-history variable. As I linked earlier, please see . What happens is that the arguments to interactive commands are automatically saved in the command-history variable by Emacs. In Ement's case, those interactive arguments can include session and room structs, which store events received in the user's rooms. Over time, as a session remains connected for a while (e.g. days or weeks), or if the user is in many, busy rooms, the amount of data can grow large (into the hundreds of megabytes when serialized). When a user enables savehist-mode, it automatically serializes the command-history variable, which includes not only the list of commands, but, as I mentioned, the arguments passed to them. As it does so, it writes the objects to a buffer (which, in this case, amounted to over 400 MB for one user), then tries to read the data back to determine whether it's readable (which I assume would allocate the equivalent amount of memory again), and then, if it is readable, it serializes it again as it writes it to disk. By default, savehist writes this data every 5 minutes. So, for some of these users, every 5 minutes savehist was serializing 800 MB and reading 400 MB (and writing 400 MB to disk)--as well as whenever Emacs exited. This was bewildering until the cause was finally noticed, after months of these weird reports, when someone finally took the time to leave the profiler running and look at the report. Of course, it only happened to the few users who activated savehist-mode, and none of them attributed the problem to savehist, which doesn't appear to have any relation to Ement whatsoever. So I had to implement a special workaround in Ement to remove any Ement commands from the command-history variable before savehist can try to save it. I hope it's now clear what I've been trying to explain, and why I've done so. If there are or come to be any other packages out there that do something similar with command arguments, and savehist is suddenly enabled when a user upgrades to a new version of Emacs, it's unlikely that users of such packages will attribute this weird behavior to savehist. And as the command-history variable, and perhaps others, grow over time, as savehist continues to save and restore them, performance may gradually decline. As well, this can cause unnecessary, excessive disk writes, which isn't good for SSDs. But if these issues do not seem likely to be problematic, then I hope that is so. I have no more to say on this topic.