From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: Lennart Borgman Newsgroups: gmane.emacs.devel Subject: Re: RFC: A framework for task management and execution in Emacs Date: Tue, 13 Jul 2010 15:53:20 +0200 Message-ID: References: <22158_1279004543_ZZh033Ked6tCc.00_1279004541.2415.244.camel@steed.robot-madness> NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Trace: dough.gmane.org 1279029252 20947 80.91.229.12 (13 Jul 2010 13:54:12 GMT) X-Complaints-To: usenet@dough.gmane.org NNTP-Posting-Date: Tue, 13 Jul 2010 13:54:12 +0000 (UTC) Cc: emacs-devel@gnu.org, Jan Moringen To: joakim@verona.se Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Tue Jul 13 15:54:10 2010 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([199.232.76.165]) by lo.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1OYfvr-0003Th-JY for ged-emacs-devel@m.gmane.org; Tue, 13 Jul 2010 15:54:10 +0200 Original-Received: from localhost ([127.0.0.1]:43710 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OYfvq-0006pb-CT for ged-emacs-devel@m.gmane.org; Tue, 13 Jul 2010 09:53:58 -0400 Original-Received: from [140.186.70.92] (port=60955 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OYfvf-0006ho-FI for emacs-devel@gnu.org; Tue, 13 Jul 2010 09:53:51 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OYfvZ-0004yp-RP for emacs-devel@gnu.org; Tue, 13 Jul 2010 09:53:47 -0400 Original-Received: from mail-ey0-f169.google.com ([209.85.215.169]:61445) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OYfvZ-0004yV-CV for emacs-devel@gnu.org; Tue, 13 Jul 2010 09:53:41 -0400 Original-Received: by eydd26 with SMTP id d26so891749eyd.0 for ; Tue, 13 Jul 2010 06:53:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:mime-version:received:in-reply-to :references:from:date:message-id:subject:to:cc:content-type :content-transfer-encoding; bh=yg+Xw3Xv18a7HlSOXsO/C64HWQfjd0Id9Fl6TzAeke8=; b=q4IrVKp2nR21sEkyneJtOCrJRHvHB+iVCXQ9QeLyAvNzwXCsDGQOmx2rLziePzFGaC FkLarS31vBebcgsqQk3bId7CGJ7qCtQ6JjuDvKolv01OWs7vXnDnkrZCe9SuZpFwc1AK awj6MN9HtHqaIJOaSXW3xrwgVVl08in0wcaPs= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type:content-transfer-encoding; b=EQZhkugMAzn/SfRYPVJIHKi85pEeXufke8+T+qXrIYYgbuWTCDcu+vpMLn4hQNvoLy zulxj+O98XrtBpzkzrucWKOUja8T9a/qkbXg2V5s4T979cA2WsQzp+EZxDGo9jNNtMAA dGyGkqmBhVZzzG6Wg2+ZO3QmVHFd3gdPDXb2Q= Original-Received: by 10.213.14.20 with SMTP id e20mr11591099eba.21.1279029220228; Tue, 13 Jul 2010 06:53:40 -0700 (PDT) Original-Received: by 10.213.15.132 with HTTP; Tue, 13 Jul 2010 06:53:20 -0700 (PDT) In-Reply-To: X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 2) X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Xref: news.gmane.org gmane.emacs.devel:127178 Archived-At: On Tue, Jul 13, 2010 at 12:00 PM, wrote: > Jan Moringen writes: > >> Hi, >> >> I would like to start a discussion regarding a topic that has bothered >> me for some time now: management and execution of (long-running) tasks >> in Emacs. Instead of properly explaining what I mean by "task", I will >> just paste a table from my notes that has several examples: >> >> | Component =C2=A0 =C2=A0| Execution =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|= Progress Indication | Cancelable >> |--------------+--------------------+---------------------+----------- >> | Tramp =C2=A0 =C2=A0 =C2=A0 =C2=A0| synchronous =C2=A0 =C2=A0 =C2=A0 = =C2=A0| progress reporter =C2=A0 | no? >> | VC =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 | sync, async =C2=A0 =C2=A0 =C2= =A0 =C2=A0| mode-line =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 | sometimes >> | Gnus =C2=A0 =C2=A0 =C2=A0 =C2=A0 | synchronous =C2=A0 =C2=A0 =C2=A0 = =C2=A0| ? =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 | = ? >> | Compilation =C2=A0| asynchronous =C2=A0 =C2=A0 =C2=A0 | mode-line =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 | yes >> | URL =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0| synchronous, async | progress = reporter =C2=A0 | ? >> | Emacs Jabber | timers, fsm =C2=A0 =C2=A0 =C2=A0 =C2=A0| ? =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 | ? >> | SemanticDB =C2=A0 | idle-timers =C2=A0 =C2=A0 =C2=A0 =C2=A0| custom re= porter =C2=A0 =C2=A0 | on input >> >> Each of these packages performs some kinds of potentially long running >> operations. Originally, the table was intended to illustrate the >> heterogeneous nature of solutions despite similar requirements and >> use-cases. >> >> The (hopefully) upcoming thread support will probably make the situation >> better and worse. Better, because long-running operations could "simply" >> run in their own threads, worse because actually implementing this is >> usually not that simple. The number and complexity of solutions for task >> execution would probably grow further once threads become available. >> >> My proposal is to introduce a set of mechanisms that make task execution >> more uniform for the user and easier for programmers. Here is the full >> set of requirements/aspects I could come up with: >> >> + Scheduling/Execution >> =C2=A0 + Resource Allocation (threads) >> =C2=A0 + Synchronization (order of execution, retrieval of results) >> =C2=A0 + Canceling >> + User Interface >> =C2=A0 + Graphical Indicators >> =C2=A0 + Progress/remaining Time Estimation >> =C2=A0 + Error Reporting >> + Desktop Integration >> =C2=A0 + Power Management inhibition >> =C2=A0 + Desktop-wide Task Management >> =C2=A0 =C2=A0 (Example: >> http://ssickert.wordpress.com/2010/05/09/introducing-my-gsoc project/) >> + Customization >> + Compatibility >> =C2=A0 + Portability >> =C2=A0 + Backwards Compatibility >> >> Since there is so much potential for code duplication, reinventing the >> wheel and divergent user interfaces, I think all of these issue should >> be addressed in a general way. >> >> Other Environments such as Eclipse or the Evolution mail client seem to >> employ such generic mechanisms since their user interfaces contain lists >> of currently running tasks, which can also be interacted with. >> >> At this point, my general question is whether there are any plans or >> related efforts regarding this topic. Of course, I would also like to >> know whether people consider the approach worthwhile at all :) >> >> The second part of this message is about a small prototype framework I >> made to explore the issue practically. >> >> The framework is structured like this: >> >> Interface Layer >> +--------------------------------+-----------------------------------+ >> | =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0| =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 | >> | =C2=A0 =C2=A0 =C2=A0 =C2=A0 User Interface =C2=A0 =C2=A0 =C2=A0 =C2=A0= | =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Macros =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0| >> | =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0| =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 | >> +--------------------------------+-----------------------------------+ >> Backend Layer >> +------------------------------+-------------------------------------+ >> | =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0| =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 | >> | =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Tasks =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 +---+ =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Executi= on =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0| >> | +-----------+ =C2=A0 =C2=A0 +-----------+ =C2=A0| +-----------+ =C2=A0= =C2=A0 +-----------+ | >> | | Strategy1 | ... | StrategyN | =C2=A0| | Strategy1 | ... | StrategyN = | | >> | +-----------+ =C2=A0 =C2=A0 +-----------+ =C2=A0| +-----------+ =C2=A0= =C2=A0 +-----------+ | >> +----------------------------------+---------------------------------+ >> >> These components are discussed below: >> >> *Macros* >> >> Basically just two macros with identical syntax: >> >> ({do,run}-as-task NAME (TASK-SPEC) (EXECUTION-SPEC) >> =C2=A0 [DOCSTRING] >> =C2=A0 BODY) >> >> The difference being that the do- macro returns the result of BODY when >> it is ready while the run- variant immediately returns an object from >> which the result of BODY can be retrieved later. >> >> Examples: >> >> (do-as-task add-numbers () () >> =C2=A0 "Add some numbers." >> =C2=A0 (dolist (i (number-sequence 1 1000)) >> =C2=A0 =C2=A0 (+ 1 2 3) >> =C2=A0 =C2=A0 'my-result)) >> >> + Blocks immediately >> + Body may or may not start executing immediately >> + Returns when the body finishes >> + Returns the value returned by the body >> >> (let ((delayed (run-as-task add-numbers () () >> =C2=A0 "Add some numbers." >> =C2=A0 (dolist (i (number-sequence 1 1000)) >> =C2=A0 =C2=A0 (+ 1 2 3) >> =C2=A0 =C2=A0 =C2=A0 'my-result)))) >> =C2=A0 ;; do other stuff >> =C2=A0 (future-get delayed)) >> >> + Does not block >> + Body may or may not start executing immediately >> + Returns immediately >> + Returns an object that implements a "future" protocol >> + Result of body can be retrieved from returned object >> >> *Tasks Strategies* >> >>>>From the commentary section: >> >> ;; What a task is >> ;; + What gets executed >> ;; + Execution context (functions callable from task code) >> ;; + Meta information (name, description) >> ;; What a task is not >> ;; + how to execute (synchronous vs. asynchronous) >> ;; + when to execute (delayed, lazy etc) >> >> (do-as-task add-numbers (progress) () >> =C2=A0 "Add some numbers, reporting progress." >> =C2=A0 (dolist (i (number-sequence 1 1000)) >> =C2=A0 =C2=A0 (+ 1 2 3) >> =C2=A0 =C2=A0 (progress i)) >> =C2=A0 'my-result) >> >> (do-as-task add-numbers (cancelable) () >> =C2=A0 "Add some numbers, cancelable." >> =C2=A0 =C2=A0(dolist (i (number-sequence 1 1000)) >> =C2=A0 =C2=A0 =C2=A0(+ 1 2 3) >> =C2=A0 =C2=A0 =C2=A0(maybe-cancel)) >> =C2=A0 =C2=A0'my-result) >> >> (do-as-task add-numbers (progress cancelable) () >> =C2=A0 "Add some numbers, reporting progress and cancelable." >> =C2=A0 =C2=A0(dolist (i (number-sequence 1 1000)) >> =C2=A0 =C2=A0 =C2=A0(+ 1 2 3) >> =C2=A0 =C2=A0 =C2=A0(progress i) >> =C2=A0 =C2=A0 =C2=A0(maybe-cancel)) >> =C2=A0 =C2=A0'my-result) >> >> *Execution Strategies* >> >> These control how and when a task is executed. The currently available >> executions are (:thread is just a teaser, of course): >> >> (run-as-task add-numbers () (:execution blocking) >> >> (run-as-task add-numbers () (:execution idle) >> >> (run-as-task add-numbers () (:execution (timeout :delay 5)) >> >> (run-as-task add-numbers () (:execution thread)) >> >> *User Interface* >> >> There is also user interface code, but discussing it here would probably >> provide little insight. >> >> The code of the framework described above is available at >> >> =C2=A0 http://bazaar.launchpad.net/~scymtym/etasks/trunk/ >> >> Feedback and suggestions would be greatly appreciated. > > Sounds very interesting. I have a scanner package called "emsane" for > which I developed "emsane-postop". This is basically an event driven > transaction queue. you push operations on a queue, and when an event > happens, the operation is popped and executed. If the the operation > fails, the transaction fails. > > Does this fit in your framework? I dont have anywhere public to place > the code at yet, so I attach the file instead, so you can have a look. > > > ;; (C) FSF 2010, Joakim Verona, part of the emsane package > > ;;I want some form of process queue because: > ;; - tramp gets swamped with lots of async calls > ;; - some postops are slow, and hundreds of them cant be expected to run = in parallell > ;; emsane was used for a long time withouth a queue, but it put constrain= ts on what could be done > > ;; it wouldve been nice if i didnt have to write it, but i didnt find any= thing. > ;; i fully expect to be pointed to something glaringly obviously already = existing once > ;; i publish the code. > > ;; TODO I want scan-job postprocessing also, for book-level ocr for insta= nce(with ocropus) > > > ;;;tentative stuff > > ;; - operation =C2=A0base class > ;; - transaction class(a list of operations) > ;; - queue class(a list of transactions, a result slot) > > ;;when a image is finished scanned, a new transaction is created, and the= image pushed on the transaction result slot > ;;the section definition pushes a bunch of ops on the tx > ;;maybe someone else also pushes a op(the job? scanner?) > ;;the tx is pushed on the queue > ;;the queue is event driven, these are events: > ;; - pushing a tx on the queue > ;; - an emacs op finishes > ;; - a shell op finishes > ;;if any op ina tx fails, the entire tx fails, otoh other txes are unafec= ted > ;;the workdir is set on the post op buffer, so will work with tramp > ;;there is a set workdir op, so different scan jobs wont trahs each other > > > ;;its possible to define many queues, > ;;a queue is connected to 1 postop buffer > > ;;transactions are independent, so they could in principle be executed in= parallell > ;;however, a queue will only do transactions in sequence > ;;futureish is supporting several queues, then pushing transactions round= robin on them > > ;;emsane-postop-next-op pops an op from the current tx, and executes it > ;;emsane-postop-push-op pushes op on tx > > ;;an op ought to be able to provide environment modifiers, such as changi= ng the flags for the scanner > ;; the use-case for this is for instance a dust-detector that needs the s= canner to scan a little bit more than > ;; the actual document. the op will then split the img in 2 parts, one ac= tual image, and one part used for dust detection. > > (provide 'emsane-postop) > > (defclass emsane-postop-operation () > =C2=A0() > =C2=A0"base class for post operations for image scans") > > (defclass emsane-postop-lisp-operation (emsane-postop-operation) > =C2=A0((operation-lambda :initarg :operation-lambda > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 :do= cumentation "a lambda of the form (lambda (transaction) (do-something-with-= transaction))")) > =C2=A0"A lisp image file operation. for instance for renaming files.") > > (defclass emsane-postop-simple-shell-operation (emsane-postop-operation) > =C2=A0((operation-shell-command :initarg :operation-shell-command > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0:documentation "a simple string to be evaluated by = a shell")) > =C2=A0"a simple file operation done with a shell command") > > (defclass emsane-postop-lifo () > =C2=A0((lifo :initarg :lifo > =C2=A0 =C2=A0 =C2=A0 =C2=A0 :initform '() > =C2=A0 =C2=A0 =C2=A0 =C2=A0 :documentation "a LIFO, Last In First Out")) > =C2=A0"base class for queue and transaction") > > (defclass emsane-postop-queue (emsane-postop-lifo) > =C2=A0((process-buffer :initarg :process-buffer) > =C2=A0 (continue-go-loop :initarg =C2=A0:continue-go-loop :initform t > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 :do= cumentation "flag setable by subprocess, to indicate continuation") > =C2=A0 (default-directory :initarg :default-directory :initform default-d= irectory > =C2=A0 =C2=A0 :documentation "subproces default dir") > =C2=A0 (state :initarg :state :initform nil > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0:documentation =C2=A0"nil if ok, otherw= ise an object indicating some error") > =C2=A0 (error-queue :initarg :error-queue :initform nil =C2=A0:documentat= ion "transactions who failed gets pushed here") > =C2=A0 (current-tx :initarg :current-tx :initform nil) > =C2=A0 (current-op :initarg :current-op :initform nil) > =C2=A0 (error-hooks :initarg :error-hooks :initform nil > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0:documentation "ho= oks to run in the event of a transaction error")) > =C2=A0"a list of transactions to be performed") > > (defmethod emsane-postop-exec ((this emsane-postop-lisp-operation) tx q) > =C2=A0"execute lisp operation" > =C2=A0(let* > =C2=A0 =C2=A0 =C2=A0((default-directory =C2=A0(oref q default-directory))= ) > =C2=A0 =C2=A0(condition-case lispop-error > =C2=A0 =C2=A0 =C2=A0 =C2=A0(progn > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(funcall (oref this :operation-lambda) = tx q) > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(emsane-postop-push q tx);;push backcur= rent tx. will be skipped if op fails > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(emsane-process-buffer-message q "lisp-= op:%s env:%s\n" > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0(oref this :operation-lambda) > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0(emsane-plist2env (oref tx environment)))) > =C2=A0 =C2=A0 =C2=A0(error (emsane-postop-signal-error q lispop-error))))= ) > > (defmethod emsane-postop-exec ((this emsane-postop-simple-shell-operation= ) tx q) > =C2=A0"execute shell operation" > =C2=A0(let* > =C2=A0 =C2=A0 =C2=A0((default-directory =C2=A0(oref q default-directory)) > =C2=A0 =C2=A0 =C2=A0 (cmd (oref this operation-shell-command)) > =C2=A0 =C2=A0 =C2=A0 (proc-buf =C2=A0(oref q :process-buffer)) > =C2=A0 =C2=A0 =C2=A0 (process-environment (emsane-plist2env (oref tx :env= ironment))) > =C2=A0 =C2=A0 =C2=A0 (post-process (start-file-process-shell-command > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0"postop" > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0proc-buf > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0cmd))) > =C2=A0 =C2=A0(set-process-sentinel post-process 'emsane-postop-sentinel) > =C2=A0 =C2=A0(process-put post-process 'queue q) > =C2=A0 =C2=A0(emsane-process-buffer-message q "shell-op:%s env:%s ..." cm= d (emsane-plist2env (oref tx environment))) > =C2=A0 =C2=A0(oset q :continue-go-loop 'waiting-for-shell-op))) > > (defun emsane-plist2env (plist) > =C2=A0"convert a plist to the env var format used by process-environment" > =C2=A0(let* > =C2=A0 =C2=A0 =C2=A0((env '())) > =C2=A0 =C2=A0(while plist > =C2=A0 =C2=A0 =C2=A0(setq env (append env (list (format "%s=3D%s" (first = plist) (second plist))))) > =C2=A0 =C2=A0 =C2=A0(setq plist (cdr (cdr plist)))) > =C2=A0 =C2=A0env)) > > (defun emsane-postop-sentinel (process result) > =C2=A0"called when an image shell postop finishes" > =C2=A0(let* > =C2=A0 =C2=A0 =C2=A0((queue (process-get process 'queue)) > =C2=A0 =C2=A0 =C2=A0 (tx-no-error (=3D 0 (process-exit-status process)))) > =C2=A0 =C2=A0(unless tx-no-error > =C2=A0 =C2=A0 =C2=A0(emsane-postop-signal-error queue result)) > =C2=A0 =C2=A0(emsane-postop-finish-shell-operation queue tx-no-error) > =C2=A0 =C2=A0(emsane-postop-go queue);;now continue processing queue tran= sations > =C2=A0 =C2=A0)) > > (defmethod emsane-postop-signal-error ((this emsane-postop-queue) result) > =C2=A0"error handler" > =C2=A0;;TODO better error handler > =C2=A0;;there are levels of errors: > =C2=A0;; - tx killers, move the tx to an error queue, other tx:es arent a= ffected > =C2=A0;; - queue killers, inhibit further queue opperations, stop scannin= g! > =C2=A0;;TODO call hooks, client should know about error(shut down scanner= processes in this case) > =C2=A0;;(oset this :state result) ;;TODO "state" is currently used as a q= ueue-killer, which doesnt happen atm > > =C2=A0;;the case below is "tx killer", push the broken tx on an error que= ue for later examination, queue chugs on as usual > =C2=A0(unless (object-p (oref this :error-queue)) (oset this :error-queue= (emsane-postop-lifo "errorq"))) ;;TODO move to q initializer > > =C2=A0;;the current tx must be removed from the queue, but, uh, only if w= ere executing a shell op?? > =C2=A0;;this is because a shell op is pushed back onto the queue before i= ts actualy finished. hmmm. > =C2=A0;;see donext. this sucks. > > =C2=A0;;im trying to have the sentinel push back the tx instead > > =C2=A0;; (if (equal (object-class (oref this :current-op)) =C2=A0'emsane-= postop-simple-shell-operation) > =C2=A0;; =C2=A0 =C2=A0 (emsane-postop-dequeue this)) > > > =C2=A0;;TODO :current-tx should be the complete failed transaction, not t= he same as the modified tx on top of the q, as it is now > =C2=A0(emsane-postop-push (oref this :current-tx) (oref this :current-op)= );;push back the failed op on current tx > =C2=A0(emsane-postop-push (oref this :error-queue) (oref this :current-tx= ));;push failed tx on error q > > > =C2=A0(mapc #'funcall (oref this :error-hooks));;using run-hooks turned o= ut not so good here > =C2=A0(let* > =C2=A0 =C2=A0 =C2=A0((msg (format "postop failed. result:%s\n =C2=A0tx:%s= \n =C2=A0op:%s" result (oref this :current-tx) (oref this :current-op)))) > =C2=A0 =C2=A0(emsane-process-buffer-message this msg) > =C2=A0 =C2=A0(message msg)) > =C2=A0) > > (defmethod emsane-postop-push ((this emsane-postop-lifo) object) > =C2=A0"Push object on the LIFO queue. New objects go at the head of the l= ist." > =C2=A0(oset this :lifo (cons object =C2=A0 (oref this :lifo)))) > > (defmethod emsane-postop-push ((this emsane-postop-queue) object) > =C2=A0"add some debugging output" > =C2=A0(call-next-method) > =C2=A0;;(emsane-process-buffer-message this "pushed on queue: %s\n" objec= t) > =C2=A0) > > > (defmethod emsane-postop-dequeue ((this emsane-postop-lifo)) > =C2=A0"Return object from the end of the LIFO =C2=A0queue, and remove the= element." > =C2=A0(unless (emsane-postop-hasnext this) (error "poping an empty queue = is bad")) > =C2=A0(let > =C2=A0 =C2=A0 =C2=A0((rv (car (last (oref this :lifo))))) > =C2=A0 =C2=A0;;(oset this :lifo (nreverse (cdr (nreverse (oref this :lifo= )))));;TODO ugly implementation > =C2=A0 =C2=A0(oset this :lifo (delq rv (oref this :lifo))) > =C2=A0 =C2=A0rv)) > > (defmethod emsane-postop-hasnext ((this emsane-postop-lifo)) > =C2=A0"empty?" > =C2=A0(not (null (oref this :lifo)))) > > (defclass emsane-postop-transaction (emsane-postop-lifo) > =C2=A0((environment :initarg :environment > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0:initform nil > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0:documentation "tr= ansaction environment variables.")) > =C2=A0"a list of operations that must be performed together. contains env= ironment operations can access") > > (defmethod =C2=A0emsane-postop-getenv ((this =C2=A0emsane-postop-transact= ion) varname) > =C2=A0(plist-get (oref this environment) varname)) > > (defmethod =C2=A0emsane-postop-setenv ((this =C2=A0emsane-postop-transact= ion) varname value) > =C2=A0(oset this environment (plist-put (oref this environment) varname v= alue))) > > (defmethod emsane-postop-finish-shell-operation ((this emsane-postop-queu= e) tx-no-error) > =C2=A0"finishup an ongoing shell operation, take care of error situation.= " > =C2=A0(if tx-no-error > =C2=A0 =C2=A0 =C2=A0(progn > =C2=A0 =C2=A0 =C2=A0 =C2=A0(emsane-postop-push this (oref this :current-t= x));;push backcurrent tx if everything went ok. awkward. > =C2=A0 =C2=A0 =C2=A0 =C2=A0(emsane-process-buffer-message this "... DONE!= env:%s\n" =C2=A0(emsane-plist2env (oref (oref this :current-tx) environmen= t))) > =C2=A0 =C2=A0 =C2=A0 =C2=A0) > =C2=A0 =C2=A0(emsane-process-buffer-message this "... FAILED! %s!!!.\n" t= x-no-error)) > =C2=A0(oset this :continue-go-loop t)) > > (defmethod emsane-process-buffer-message ((this emsane-postop-queue) stri= ng &rest objects) > =C2=A0;;TODO should have its own insert marker, so moving the cursor does= nt break output > =C2=A0 =C2=A0(with-current-buffer (oref this :process-buffer) > =C2=A0 =C2=A0 =C2=A0(insert (apply 'format (cons string objects))))) > > (defmethod emsane-process-buffer-insert ((this emsane-postop-queue) strin= g &rest objects) > =C2=A0;;TODO should have its own insert marker, so moving the cursor does= nt break output > =C2=A0 =C2=A0(with-current-buffer (oref this :process-buffer) > =C2=A0 =C2=A0 =C2=A0(insert (apply 'format (cons string objects))))) > > (defmethod emsane-postop-donext ((this emsane-postop-queue)) > =C2=A0"pops an operation from the current transaction in the queue and ex= ecutes it. > continue with the 1st op of the next transaction if the current transacti= on is finished. > if the queue is empty return nil." > =C2=A0;;TODO the method should always be safe to call, regardless of the = queue state, ensure this > =C2=A0;;TODO delete the transaction if the operation fails. > =C2=A0;;should almost work, because if crash, we dont push back th eop > =C2=A0(if (oref this state) (error "the queue is unwell:%s" (oref this st= ate))) > =C2=A0(if (emsane-postop-hasnext this) > =C2=A0 =C2=A0 =C2=A0(let* > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0((tx (emsane-postop-dequeue this)) > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (op)) > =C2=A0 =C2=A0 =C2=A0 =C2=A0(oset this :current-tx tx) > =C2=A0 =C2=A0 =C2=A0 =C2=A0(if (emsane-postop-hasnext tx) > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(progn > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(setq op =C2=A0(emsane-po= stop-dequeue tx)) > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(oset this :current-op op= ) > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(emsane-postop-exec op tx= this)) > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(emsane-postop-donext this) ;;TODO real= ly? recursion doesnt feel right when we might have a complicated error cond= ition... > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0)))) > > (defmethod emsane-postop-go ((this emsane-postop-queue)) > =C2=A0"start or continue executing transactions in queue. > it is supposed to always be safe to call.";;TODO it isnt atm... > =C2=A0(if (oref this state) (error "the queue is unwell:%s" (oref this st= ate))) > =C2=A0;;(emsane-process-buffer-message this "cgloop:%s\n" (oref this :con= tinue-go-loop) =C2=A0) > =C2=A0(unless =C2=A0(equal (oref this :continue-go-loop) 'waiting-for-she= ll-op) > =C2=A0 =C2=A0 =C2=A0(let > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0((continue-go-loop t)) > =C2=A0 =C2=A0 =C2=A0 =C2=A0(while (and continue-go-loop > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(not= (eq 'waiting-for-shell-op continue-go-loop)) > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(ems= ane-postop-hasnext this)) ;;TODO continue-go-loop is madness atm > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(emsane-postop-donext this) > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(setq continue-go-loop (oref this :cont= inue-go-loop)))))) I have a queue implementation for communicating with Firefox. Maybe this could be replaced by this implementation. If someone is interested please look in nXhtml, mozadd.el.