From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED!not-for-mail From: Dmitry Gutov Newsgroups: gmane.emacs.bugs Subject: bug#26710: Fwd: 25.2; project-find-regexp makes emacs use 100% cpu Date: Mon, 1 May 2017 05:42:21 +0300 Message-ID: <77b3a404-adac-fd1c-bd99-ad10e2450338@yandex.ru> References: <87a86zu3gf.fsf@hari-laptop.i-did-not-set--mail-host-address--so-tickle-me> <83vapnktcn.fsf@gnu.org> <3d76a3ac-32ad-412d-349d-5904fc964a2b@yandex.ru> <83ziexka0s.fsf@gnu.org> NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-Trace: blaine.gmane.org 1493606593 5344 195.159.176.226 (1 May 2017 02:43:13 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Mon, 1 May 2017 02:43:13 +0000 (UTC) User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:53.0) Gecko/20100101 Thunderbird/53.0 Cc: hariharanrangasamy@gmail.com, control@debbugs.gnu.org, 26710@debbugs.gnu.org To: Eli Zaretskii Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Mon May 01 04:43:08 2017 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1d51Ip-0001G8-5L for geb-bug-gnu-emacs@m.gmane.org; Mon, 01 May 2017 04:43:07 +0200 Original-Received: from localhost ([::1]:46595 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d51Iu-00009h-Kh for geb-bug-gnu-emacs@m.gmane.org; Sun, 30 Apr 2017 22:43:12 -0400 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:54829) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d51In-00009D-Ru for bug-gnu-emacs@gnu.org; Sun, 30 Apr 2017 22:43:06 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1d51Ik-0007io-Pf for bug-gnu-emacs@gnu.org; Sun, 30 Apr 2017 22:43:05 -0400 Original-Received: from debbugs.gnu.org ([208.118.235.43]:50150) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1d51Ik-0007if-MR for bug-gnu-emacs@gnu.org; Sun, 30 Apr 2017 22:43:02 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1d51Ik-0002Ui-Ab for bug-gnu-emacs@gnu.org; Sun, 30 Apr 2017 22:43:02 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Dmitry Gutov Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Mon, 01 May 2017 02:43:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 26710 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: Original-Received: via spool by 26710-submit@debbugs.gnu.org id=B26710.14936065529546 (code B ref 26710); Mon, 01 May 2017 02:43:02 +0000 Original-Received: (at 26710) by debbugs.gnu.org; 1 May 2017 02:42:32 +0000 Original-Received: from localhost ([127.0.0.1]:48347 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1d51IG-0002Ts-E8 for submit@debbugs.gnu.org; Sun, 30 Apr 2017 22:42:32 -0400 Original-Received: from mail-wr0-f171.google.com ([209.85.128.171]:34448) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1d51ID-0002Ta-19; Sun, 30 Apr 2017 22:42:29 -0400 Original-Received: by mail-wr0-f171.google.com with SMTP id l9so58795651wre.1; Sun, 30 Apr 2017 19:42:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=ZqYeSoWgePUpClRvkL+wpfRpEhcNlYhG1g2/k9wMXvY=; b=rLVOQda88NaYXL50Z0NUAKw5YUb55pDziuEUhpRh7FbBthGx3bhRi5yqpTliGV+Llu cKhqbSViTn/FftArQPNb/byxC+0InGEgq/OKycnX/BeEUlA9C8zXqcrlFJDG+3587ejr 8U3LBmjXwbimwZdw4Dr+PClRDPPuQ6bnRs80zjie3mj0AFRN/0eqe49aAWZ6If5RUmEq b2x2GcHI/zwmgxLQyp3mUPm1iGVrfOY0N6F+tDv/TxgHJ3MNUyc14oIoVDDur1l3aBXQ HK+ydsuy3IobRd4ga/WepBU6JeE2C1bIIJiLbgwTzNTlSjJUbQscRVAc4TI+1myHXAlH tZTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:to:cc:references:from:message-id :date:user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=ZqYeSoWgePUpClRvkL+wpfRpEhcNlYhG1g2/k9wMXvY=; b=UNNIIiHPWzUICU/gvj1rX96XaCCKuoDONOLjjaZJm7Qj+jjvsTS/DoCivF74vOW11I iDHViqcrkBRmF/1ESPgMPMvkH/sisaVwdur1onHTu0WAyy89aukoMUdaVGrGpFO6wqIt sr6uKnY4lCtp1gfWsfUqBQ7egF3p8/M+u+LHSYE1iwbSMg9quuUr0wWgIaGesDKG6VfJ qjvbTBZovz0F47pjkMbOb/qFmLlwmAKCFMuI6lHxYAoZutyCbKoTy73y/JzLjhtUFWoX F2Itw/OqU6rvV3idXrKaxDDdyQ+HTuwfXx9eh4frfaFcXPV1H2ovsmLzSCOcvrf0dO8b z01w== X-Gm-Message-State: AN3rC/5ai8HbB712VzADoGV/cMRyjrfU5auonak4GQsHxB7wZg4yrqJW 3TxvjvMz+Z0cyA== X-Received: by 10.223.165.71 with SMTP id j7mr16434604wrb.35.1493606543286; Sun, 30 Apr 2017 19:42:23 -0700 (PDT) Original-Received: from [192.168.1.3] ([185.105.173.156]) by smtp.googlemail.com with ESMTPSA id o77sm14099490wrc.38.2017.04.30.19.42.22 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 30 Apr 2017 19:42:22 -0700 (PDT) In-Reply-To: <83ziexka0s.fsf@gnu.org> Content-Language: en-US X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 208.118.235.43 X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.org gmane.emacs.bugs:132149 Archived-At: On 30.04.2017 21:47, Eli Zaretskii wrote: > I'll try to look at this. According to my profiling, the lion's share > of time is taken by xref--collect-matches, so that's the place to try > concurrency. I think that's too late. By the time xref--collect-matches is called (and it's called for each hit), we've already spent time synchronously waiting for the find-grep invocation to finish. When there are a lot of matches, xref--collect-matches can take some significant time, with opening the buffers and everything. That can be optimized, however, as a separate issue, and I don't think there's anything to parallelize there, since it all happens in Elisp. What we _can_ manage to run in parallel, in the find-grep process in the background, and the post-processing of the results in Elisp. Depending on how matches there are in total, compared to the project size (which affects how long find-grep will run), the second part will still affect the responsiveness of the UI to a smaller or larger degree, but ultimately there's no way around this AFAIK, as long as Elisp threads do not provide parallelism. >> - Being able to hook up in an asynchronous fashion to that sequence in a >> (second?) background thread, to render the search results in the xref >> buffer as soon as they appear. > > For the other thread to be background, it will need to yield from time > to time, otherwise the user experience will be identical to what we > have today, because an un-yielding thread will hold the execution unit > until it does its job completely, and no other thread gets to run > until it does. Here's how I imagine it: - Main thread creates a thread P which invoked the find-grep, somehow creates a "generator" object, yields to the main thread. - The main thread creates the "other thread" which creates a buffer for displaying the xref items and marks it as still loading (e.g. with a spinner in the mode-line). And then repeatedly calls the generator for the next element. There are three choices: 1. An element is returned. Render it into the buffer. 2. No next element available. That should automatically yield from the current thread until it becomes available. That kind of logic should reside somewhere inside the generator, along with storing the reference to the current thread, to resume it later. 3. No more items, the process in P is finished. Mark the xref buffer as completed. The things I'm not clear on are: - How to best "convert" the process buffer into a generator object. - Which generator type to use. Not sure if any of the ones we already have (generator.el, or iterator.el and stream.el in ELPA) will help. - If the thread P is needed at all, or we could just one the main one instead of it. - Whether we should forget all that concurrency nonsense (at least for this problem), and instead of a generator go with callbacks, similar to the API of the dir-status-files VC command. This way, each callback will render a new line, and the last one will mark the buffer as completed.