From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED!not-for-mail From: Michael Heerdegen Newsgroups: gmane.emacs.bugs Subject: bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files' Date: Wed, 28 Feb 2018 11:58:49 +0100 Message-ID: <87bmg91ity.fsf@web.de> References: <87inaiss6l.fsf@web.de> <6FCF6ACA-4F29-4B6B-BE9D-D7130C6E9495@gnu.org> <87fu5moe4c.fsf@web.de> <877eqyocro.fsf@web.de> <83zi3uz4nb.fsf@gnu.org> <87lgfd52by.fsf@gmail.com> NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: text/plain X-Trace: blaine.gmane.org 1519815429 26404 195.159.176.226 (28 Feb 2018 10:57:09 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Wed, 28 Feb 2018 10:57:09 +0000 (UTC) User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.0.91 (gnu/linux) Cc: 30626@debbugs.gnu.org To: Noam Postavsky Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Wed Feb 28 11:57:05 2018 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1eqzQ0-0006Qf-IZ for geb-bug-gnu-emacs@m.gmane.org; Wed, 28 Feb 2018 11:57:04 +0100 Original-Received: from localhost ([::1]:43270 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eqzS2-0002ww-QU for geb-bug-gnu-emacs@m.gmane.org; Wed, 28 Feb 2018 05:59:10 -0500 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:38966) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eqzRx-0002we-11 for bug-gnu-emacs@gnu.org; Wed, 28 Feb 2018 05:59:05 -0500 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eqzRu-0005o8-Hl for bug-gnu-emacs@gnu.org; Wed, 28 Feb 2018 05:59:05 -0500 Original-Received: from debbugs.gnu.org ([208.118.235.43]:56814) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eqzRu-0005o2-ER for bug-gnu-emacs@gnu.org; Wed, 28 Feb 2018 05:59:02 -0500 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1eqzRu-0000ce-81 for bug-gnu-emacs@gnu.org; Wed, 28 Feb 2018 05:59:02 -0500 X-Loop: help-debbugs@gnu.org Resent-From: Michael Heerdegen Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Wed, 28 Feb 2018 10:59:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 30626 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: Original-Received: via spool by 30626-submit@debbugs.gnu.org id=B30626.15198155392383 (code B ref 30626); Wed, 28 Feb 2018 10:59:02 +0000 Original-Received: (at 30626) by debbugs.gnu.org; 28 Feb 2018 10:58:59 +0000 Original-Received: from localhost ([127.0.0.1]:36478 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1eqzRr-0000cM-Cb for submit@debbugs.gnu.org; Wed, 28 Feb 2018 05:58:59 -0500 Original-Received: from mout.web.de ([212.227.17.12]:50553) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1eqzRp-0000c7-1h for 30626@debbugs.gnu.org; Wed, 28 Feb 2018 05:58:57 -0500 Original-Received: from drachen.dragon ([188.99.169.170]) by smtp.web.de (mrweb103 [213.165.67.124]) with ESMTPSA (Nemesis) id 0MOzoR-1elEYY3buB-006PZ8; Wed, 28 Feb 2018 11:58:50 +0100 In-Reply-To: <87lgfd52by.fsf@gmail.com> (Noam Postavsky's message of "Tue, 27 Feb 2018 20:29:21 -0500") X-Provags-ID: V03:K0:XkQ6Dh1kC+Kyw/7Jvu7xd3wRLLW9ULo+7fVx1kw4mvGWdh2aLb1 q3PukxVyQ9NfQcU2PS0sYCxUiVLO9mY+OND2o38Vdc8M3FAdlyD/KrLFtwWUzNquEShTK8q hREyRTtNdfzP4lXRz2Fsk3e6s/xqYgbUzHQEXHJ+QnkCWPixefVAKI3DNdczItkD7RBtaqV 2Q1/uUGfFUtVi4rB+jzaw== X-UI-Out-Filterresults: notjunk:1;V01:K0:MSX261JF4Nk=:9myiQqNSVT3vSAAAi0VHVA tYUYEh9PtaTrt7p9roq3PwsyHA6JZYsa4u+ebygitFxrQiCeTMk0xnTdVAs0wy1buaWaGzrWr ieNaGej1jwYjoAqqg6aOT0KnUfT8jtkywqLB0ozCGMX7OUpOEgELMGzfKclB35AixyyR3VFk7 YiRjB43ZUBo22pPMQ+Xv4LV+Kd7N3DgQ1Aiz3BfC2mNMKV8VfKJuuLnfbLAvKKYlaOhc5a+TB drcEEImqQ7p7c/tto1AAXWHW4e2D8TNS9u7++Z/eD9xWiZnAYZ9jXSzfrwpcQGYuT1tbHzZWw G/GVa6uIWS62KC7kxSOMSUYPSLIPzV1QwgNzfsFg1ajRvVrhXwJcMWwBJpeP/kPxZl/2jZ7UC Jk+X8fxCOKEKmHS3vr+2FmRX8uk6xj8CXSf8XLeMA75vJfRuF+A5ILVFTGKvzIB5/VFd/AllF WuVZLXEqtq63ao3CxCe0haMz/qLx82DaXmKLvrn3cE7dJ5PeQOcNK9bR16qOW7ycV+RQ93K6C sYf7MBypSjQ78RXSxlbR81/qxRIeBVJ6rJsm8ytUp7R+FR7xZs2i+tSdEiOZVS7FBOG/gJZjo ZxlnAmCk4xdYm6n4kthT2YZsrqMzepq4I+2uKUSZDtMu+TY7+MJvymrXHEkBGAqbZokmUhzDG 08KSOlsJeEwxM/3/tc2j1NYMg502+KiWW9qaUunZ3k856/5HSuqZBS1NiHJZI+TOfUFhDws2k fG85Wda9azYAA0gsETqlzOzActd6tOjMkEtaqLH73DEtOllCYqLrxauV4ZDaw93bLIAn2t2Y X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 208.118.235.43 X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.org gmane.emacs.bugs:143730 Archived-At: Noam Postavsky writes: > >> #+begin_src emacs-lisp > >> (seq-doseq (_ (stream-range 1 1000000)) nil) > >> #+end_src > >> > >> Note that this is executed as a loop due how to streams are > >> implemented, although the definition of `seq-doseq' looks > >> recursive. > > Doesn't look recursive to me, it expands to a call to seq-do, which uses > a simple loop. I was imprecise, I meant that the streams are defined recursively (most of the time). Though, it's a delayed type of recursion. Anyway, I think that this doesn't matter here. > > Does anyone has a reasonable idea for avoiding the crash in such > > programs? > > I don't have a quick answer for the general case, but I think it's a bug > in stream.el that it's creating such large structures in the first > place. As far as I understand it, the point of streams is to handle > long lists by encoding them as > > (FIRST-VALUE . FUNCTION-TO-PRODUCE-REST-OF-LIST) Yes, that's exactly how it's implemented. When requesting more elements from the stream, that becomes (FIRST-VALUE . (SECOND-VALUE . FUNCTION-TO-PRODUCE-MORE-REST-OF-LIST)) When we loop over the string, the cons whose car is the FIRST-VALUE, let's call it cons1, is immediately thrown away, and we continue with (SECOND-VALUE . FUNCTION-TO-PRODUCE-MORE-REST-OF-LIST) etc. AFAIU the problem is that the cons1 still exists in memory until garbage collection kicks in. When that happens, the cons1 points to a largely recursive cons structure, though this structure is never referenced from Lisp in this form. > so as to avoid allocating large amounts of memory. Is there an easy way > to find out what the large structures are, and where they are coming > from? I think I've answered that. At least, I think so. What I don't understand is that when I force the `seq-doseq' to call `garbace-collect' explicitly every 1000 cycles, or so, it doesn't help: the crash still happens after generating ~ 70 000 elements, or some more, but I can't avoid the crash, no matter how often I call gc. So I'm not sure whether these long lists are the problem or something else. The FUNCTION-TO-PRODUCE-MORE-REST-OF-LIST looks harmless when I print it, even after thousands of iterations, so I would not understand why that could be problematic. streams.el implements things in a way that these rest functions are not deeply nested lambdas. Michael.