From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Petteri Hintsanen Newsgroups: gmane.emacs.help Subject: Re: Passing buffers to function in elisp Date: Wed, 06 Sep 2023 22:05:13 +0300 Message-ID: References: <87mt56hg4e.fsf@iki.fi> <87bklihln8.fsf@iki.fi> <87pm9yf0ph.fsf@web.de> <87v8jmkfd0.fsf@iki.fi> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="13739"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Gnus/5.13 (Gnus v5.13) Cc: Stefan Monnier To: help-gnu-emacs@gnu.org Original-X-From: help-gnu-emacs-bounces+geh-help-gnu-emacs=m.gmane-mx.org@gnu.org Wed Sep 06 21:05:53 2023 Return-path: Envelope-to: geh-help-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1qdxqa-0003Md-WD for geh-help-gnu-emacs@m.gmane-mx.org; Wed, 06 Sep 2023 21:05:53 +0200 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qdxq9-0001yE-Hl; Wed, 06 Sep 2023 15:05:25 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qdxq7-0001xh-Oa for help-gnu-emacs@gnu.org; Wed, 06 Sep 2023 15:05:23 -0400 Original-Received: from mail.kapsi.fi ([2001:67c:1be8::25]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qdxq5-0000d6-15 for help-gnu-emacs@gnu.org; Wed, 06 Sep 2023 15:05:23 -0400 Original-Received: from kapsi.fi ([2001:67c:1be8::11] helo=lakka.kapsi.fi) by mail.kapsi.fi with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1qdxpz-001HJG-1T; Wed, 06 Sep 2023 22:05:15 +0300 Original-Received: from phintsan by lakka.kapsi.fi with local (Exim 4.94.2) (envelope-from ) id 1qdxpx-00Cx3Y-Su; Wed, 06 Sep 2023 22:05:13 +0300 In-Reply-To: (Stefan Monnier via Users list for the's message of "Fri, 03 Mar 2023 10:19:09 -0500") X-Rspam-Score: -3.1 (---) X-Rspam-Report: Action: no action Symbol: RCVD_TLS_LAST(0.00) Symbol: ARC_NA(0.00) Symbol: BAYES_HAM(-3.00) Symbol: FROM_HAS_DN(0.00) Symbol: TO_DN_SOME(0.00) Symbol: TO_MATCH_ENVRCPT_ALL(0.00) Symbol: R_SPF_NEUTRAL(0.00) Symbol: MIME_GOOD(-0.10) Symbol: DMARC_NA(0.00) Symbol: RCPT_COUNT_TWO(0.00) Symbol: NEURAL_HAM(0.00) Symbol: FROM_EQ_ENVFROM(0.00) Symbol: R_DKIM_NA(0.00) Symbol: MIME_TRACE(0.00) Symbol: ASN(0.00) Symbol: RCVD_COUNT_TWO(0.00) Symbol: MID_RHS_MATCH_FROM(0.00) Message-ID: x2m8r9j5dhy.fsf_-_@iki.fi X-SA-Exim-Connect-IP: 2001:67c:1be8::11 X-SA-Exim-Mail-From: petterih@iki.fi X-SA-Exim-Scanned: No (on mail.kapsi.fi); SAEximRunCond expanded to false Received-SPF: neutral client-ip=2001:67c:1be8::25; envelope-from=petterih@iki.fi; helo=mail.kapsi.fi X-Spam_score_int: -10 X-Spam_score: -1.1 X-Spam_bar: - X-Spam_report: (-1.1 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_NEUTRAL=0.779 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: help-gnu-emacs@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Users list for the GNU Emacs text editor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: help-gnu-emacs-bounces+geh-help-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: help-gnu-emacs-bounces+geh-help-gnu-emacs=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.help:145083 Archived-At: Hello all, It took some time to do these memory optimizations I asked about few months ago. Here are some remarks. >> Also, if I interpreted profiler's hieroglyphs correctly, it told me that >> this setq >> >> (setq stream (vconcat stream (plist-get page :stream))) > > This is a typical a source of unnecessary O(N=C2=B2) complexity: the above > line takes O(N) time, so if you do it O(N) times, you got your > N=C2=B2 blowup. You're usually better off doing > > (push (plist-get page :stream) stream-chunks) > > and then at the end get the `stream` with > > (mapconcat #'identity (nreverse stream-chunks) nil) > or > (apply #'vconcat (nreverse stream-chunks)) I replaced vconcat with push. However it did not have a significant effect (measured with Emacs memory profiler). Perhaps the chunks were quite small after all. In complexity speak, with small N one usually does not need to worry about quadratics. But it is no worse either, so I left it that way. >> I think I can replace vectors with strings, which should, according to >> the elisp manual, "occupy one-fourth the space of a vector of the same >> elements." > > More likely one-eighth nowadays (64 bit machines). Changing vectors to strings did indeed have a significant effect. It is also the right thing to do, because, frankly, much of the data *are* strings. >> Similarly bindat consumes a lot of memory. > > Hmm... IIRC it should not use up very much "auxiliary" memory. IOW > its memory usage should be determined by the amount of data it > returns. So, when producing the bytestring it should be quite > efficient memorywise. This is correct. Bindat is very conservative. I probably misread the profiler report back then and unjustly put part of the blame on bindat. >>> That's definitely something to consider. Another is whether the ELisp >>> code was byte-compiled (if not, then all bets are off, the interpreter >>> itself generates a fair bit of garbage, especially if you use a lot of >>> macros). >> No, it was not byte-compiled. > > Then stop right there and fix this problem. There's absolutely no point > worrying about performance (including memory use) if the code is > not compiled because compilation can change the behavior drastically. This is also absolutely correct. There is no point in profiling non compiled code. Non compiled code gives wildly changing profiles from time to time. >> I'll try byte-compiling after the code is in good enough shape to do >> controlled experiments. > > The compiler is your friend. He can help you get the code in good shape = :-) Truly he does. I have also native compilation enabled. Don't know how much effect it had. I also tried to replace with-temp-buffer forms (such forms are called hundreds of times) with a static buffer for holding temporary data. It produced mixed results. In some limited settings, memory savings were considerable, but in some others cases it blew up memory usage. I cannot explain why that happened. But it seems safest to stick to with-temp-buffer. Nonetheless, the code is now much better.=20 Thank you all for your insights, Petteri