From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: Stefan Monnier Newsgroups: gmane.emacs.devel Subject: Re: map-file-lines Date: Mon, 02 Feb 2009 17:40:37 -0500 Message-ID: References: <86wsc87o3c.fsf@lifelogs.com> <86skmw7ig6.fsf@lifelogs.com> NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Trace: ger.gmane.org 1233614458 1371 80.91.229.12 (2 Feb 2009 22:40:58 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Mon, 2 Feb 2009 22:40:58 +0000 (UTC) Cc: emacs-devel@gnu.org To: Ted Zlatanov Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Mon Feb 02 23:42:12 2009 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([199.232.76.165]) by lo.gmane.org with esmtp (Exim 4.50) id 1LU7UY-0004UG-IG for ged-emacs-devel@m.gmane.org; Mon, 02 Feb 2009 23:42:10 +0100 Original-Received: from localhost ([127.0.0.1]:56100 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1LU7TG-0005Ch-0E for ged-emacs-devel@m.gmane.org; Mon, 02 Feb 2009 17:40:50 -0500 Original-Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1LU7T6-0005AG-VX for emacs-devel@gnu.org; Mon, 02 Feb 2009 17:40:41 -0500 Original-Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1LU7T4-00058x-T8 for emacs-devel@gnu.org; Mon, 02 Feb 2009 17:40:40 -0500 Original-Received: from [199.232.76.173] (port=37302 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1LU7T4-00058t-Mq for emacs-devel@gnu.org; Mon, 02 Feb 2009 17:40:38 -0500 Original-Received: from ironport2-out.pppoe.ca ([206.248.154.182]:62371 helo=ironport2-out.teksavvy.com) by monty-python.gnu.org with esmtp (Exim 4.60) (envelope-from ) id 1LU7T4-0007FQ-DY for emacs-devel@gnu.org; Mon, 02 Feb 2009 17:40:38 -0500 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AgMFAJkEh0nO+IQk/2dsb2JhbACBbssahBQGgmw X-IronPort-AV: E=Sophos;i="4.37,367,1231131600"; d="scan'208";a="33229001" Original-Received: from 206-248-132-36.dsl.teksavvy.com (HELO pastel.home) ([206.248.132.36]) by ironport2-out.teksavvy.com with ESMTP; 02 Feb 2009 17:40:37 -0500 Original-Received: by pastel.home (Postfix, from userid 20848) id 3FCB18229; Mon, 2 Feb 2009 17:40:37 -0500 (EST) In-Reply-To: <86skmw7ig6.fsf@lifelogs.com> (Ted Zlatanov's message of "Mon, 02 Feb 2009 13:22:01 -0600") User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.0.60 (gnu/linux) X-detected-operating-system: by monty-python.gnu.org: Genre and OS details not recognized. X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Xref: news.gmane.org gmane.emacs.devel:108644 Archived-At: >>> Emacs Lisp lacks a good way to iterate over all the lines of a file, >>> especially for a large file. SM> I'm not really happy about focusing on "line at a time". It's a useful SM> and common case, but Emacs usually is pretty good about being "line SM> agnostic" (font-lock being an obvious counter example). SM> Providing some kind of stream-processing functionality might be good, SM> tho the need doesn't seem terribly high, since we've managed to avoid it SM> until now. > Without this function, Emacs simply can't handle large files and that's > been requested at least 4 times by users that I can recall. I think a > general solution to the large file Emacs problem would be better, but > line-oriented processing is a classic approach to processing large files > that many Emacs users will probably find familiar. I know about the large-file problem, obviously, but I wonder what kind of UI you expect to provide, in order for it to be able to work just one line at a time. SM> FWIW, another option is to provide an open-file-stream along the same SM> lines as open-network-stream. I.e. the chunks are received via SM> a process filter. > How is that better than insert-file-contents as I use it? It processes just one chunk at a time, with no need to keep the whole file in memory. A naive implementation could look like (start-process "open-file-stream" nil "cat" file). > Are you thinking of a stateful back/forward seek capability? Or do > you mean you'd like it to be asynchronous? Just that the file is received/processed one chunk at a time, so you never need to hold the whole file in memory at any one time. Stefan