org batch job using emacsclient?

org batch job using emacsclient?

[Stefan Vollmar]

Hello,

we are currently setting up some org-templates for simple web pages (e.g. CV information for members of our institute). So far this seems to work surprisingly well - even for users who use editors other than Emacs (yes, we need to try harder...). We have created a web service so users can upload an org-file and get a preview of the generated HTML page.

In a python script we use (a recipe from this very useful list):

[...]
cmd = emacs \
  + " -Q --batch"
  + " --eval \"(add-to-list 'load-path \\\"" + ORGLIB + "\\\")\"" \
  + " --eval \"(require 'org)\"" \
  + " --eval \"(require 'org-exp)\"" \
  + " --eval \"(setq org-export-headline-levels 2)\"" \
  + " --visit=\"" + orgfile + "\"" \
  + " --funcall org-export-as-html"

However, even on a modern system (SunFire T5140, Solaris 10) generating the HTML output with Emacs 23.1 and the latest org-version takes several seconds (which is disappointing). To improve performance, we want to switch to emacsclient but we are not sure how to adapt the above code to do this. Our first attempt:

cmd = emacsclient + " --eval " \
  + " \"(add-to-list 'load-path \\\"" + ORGLIB + "\\\")\"" \
  + " \"(require 'org)\"" \
  + " \"(require 'org-exp)\"" \
  + " \"(setq org-export-headline-levels 2)\"" \
  + " \"(load-file \\\"" + orgfile + "\\\")\"" \
  + " \"(org-export-as-html)\""

did not work.
Any help is appreciated.

Many thanks in advance.
Warm regards,
 Stefan
-- 
Dr. Stefan Vollmar, Dipl.-Phys.
Max-Planck-Institut für neurologische Forschung
Gleuelerstr. 50, 50931 Köln, Germany
Tel.: +49-221-4726-213  FAX +49-221-4726-298
Tel.: +49-221-478-5713  Mobile: 0160-93874279
Email: vollmar@nf.mpg.de   http://www.nf.mpg.de

Re: org batch job using emacsclient?

[Noorul Islam K M]

Stefan Vollmar <vollmar@nf.mpg.de> writes:

> Hello,
>
> we are currently setting up some org-templates for simple web pages (e.g. CV information for members of our institute). So far this seems to work surprisingly well - even for users who use editors other than Emacs (yes, we need to try harder...). We have created a web service so users can upload an org-file and get a preview of the generated HTML page.
>
> In a python script we use (a recipe from this very useful list):
>
> [...]
> cmd = emacs \
>   + " -Q --batch"
>   + " --eval \"(add-to-list 'load-path \\\"" + ORGLIB + "\\\")\"" \
>   + " --eval \"(require 'org)\"" \
>   + " --eval \"(require 'org-exp)\"" \
>   + " --eval \"(setq org-export-headline-levels 2)\"" \
>   + " --visit=\"" + orgfile + "\"" \
>   + " --funcall org-export-as-html"
>
> However, even on a modern system (SunFire T5140, Solaris 10) generating the HTML output with Emacs 23.1 and the latest org-version takes several seconds (which is disappointing). To improve performance, we want to switch to emacsclient but we are not sure how to adapt the above code to do this. Our first attempt:
>
> cmd = emacsclient + " --eval " \
>   + " \"(add-to-list 'load-path \\\"" + ORGLIB + "\\\")\"" \
>   + " \"(require 'org)\"" \
>   + " \"(require 'org-exp)\"" \
>   + " \"(setq org-export-headline-levels 2)\"" \
>   + " \"(load-file \\\"" + orgfile + "\\\")\"" \
>   + " \"(org-export-as-html)\""
>
> did not work.
> Any help is appreciated.


You are using option -Q with emacs command. That itself should give you
the maximum performance. 

I don't think that you will get any performace improvement if you use
emacsclient.

Thanks
Noorul

Re: org batch job using emacsclient?

[Adam Spiers]

Hi Stefan,

On Mon, Nov 16, 2009 at 09:51:38AM +0100, Stefan Vollmar wrote:

[snipped]

> To improve performance, we want to switch to emacsclient but we are
> not sure how to adapt the above code to do this. Our first attempt:
> 
> cmd = emacsclient + " --eval " \
>   + " \"(add-to-list 'load-path \\\"" + ORGLIB + "\\\")\"" \
>   + " \"(require 'org)\"" \
>   + " \"(require 'org-exp)\"" \
>   + " \"(setq org-export-headline-levels 2)\"" \
>   + " \"(load-file \\\"" + orgfile + "\\\")\"" \
>   + " \"(org-export-as-html)\""
> 
> did not work.
> Any help is appreciated.

Can you elaborate on what you mean by "did not work" - what happened
exactly?

Regards,
Adam

Re: org batch job using emacsclient?

[Magnus Henoch]

Stefan Vollmar <vollmar@nf.mpg.de> writes:

> cmd = emacsclient + " --eval " \
>   + " \"(add-to-list 'load-path \\\"" + ORGLIB + "\\\")\"" \
>   + " \"(require 'org)\"" \
>   + " \"(require 'org-exp)\"" \
>   + " \"(setq org-export-headline-levels 2)\"" \
>   + " \"(load-file \\\"" + orgfile + "\\\")\"" \
>   + " \"(org-export-as-html)\""

emacsclient --eval expects only a single form.  Enclose all that in
(progn ...) and it should work better.

Magnus

Re: org batch job using emacsclient?

[Gregory J. Grubbs]

Stefan Vollmar <vollmar@nf.mpg.de> writes:

> Hello,
>
> we are currently setting up some org-templates for simple web pages (e.g. CV information for members of our institute). So far this seems to work surprisingly well - even for users who use editors other than Emacs (yes, we need to try harder...). We have created a web service so users can upload an org-file and get a preview of the generated HTML page.
>
> In a python script we use (a recipe from this very useful list):
>
> [...]
> cmd = emacs \
>   + " -Q --batch"
>   + " --eval \"(add-to-list 'load-path \\\"" + ORGLIB + "\\\")\"" \
>   + " --eval \"(require 'org)\"" \
>   + " --eval \"(require 'org-exp)\"" \
>   + " --eval \"(setq org-export-headline-levels 2)\"" \
>   + " --visit=\"" + orgfile + "\"" \
>   + " --funcall org-export-as-html"
>
> However, even on a modern system (SunFire T5140, Solaris 10) generating the HTML output with Emacs 23.1 and the latest org-version takes several seconds (which is disappointing). To improve performance, we want to switch to emacsclient but we are not sure how to adapt the above code to do this. Our first attempt:
>
> cmd = emacsclient + " --eval " \
>   + " \"(add-to-list 'load-path \\\"" + ORGLIB + "\\\")\"" \
>   + " \"(require 'org)\"" \
>   + " \"(require 'org-exp)\"" \
>   + " \"(setq org-export-headline-levels 2)\"" \
>   + " \"(load-file \\\"" + orgfile + "\\\")\"" \
>   + " \"(org-export-as-html)\""
>
> did not work.
> Any help is appreciated.


You are trying to load an Org file as an elisp file.  Use find-file or
something like it instead.

The following worked for me from the shell:
--8<---------------cut here---------------start------------->8---
 emacsclient --eval "(progn 
	 (add-to-list 'load-path \"/home/gregj/emacs/org-mode/lisp\")
	 (require 'org)
	 (require 'org-exp)
	 (find-file \"/home/gregj/projects/blogs.org\")
	 (org-export-as-html 2 nil nil nil nil \"/tmp\"))"
--8<---------------cut here---------------end--------------->8---

Re: Re: org batch job using emacsclient?

[Stefan Vollmar]

Gentlemen,

thanks for the many helpful replies!

To start with, it now works for me and I can measure a consistent speedup of 7x (!) for using emacsclient - in an interactive web service (conversion of org-files to HTML) the difference between 0.35 s and 2.40 s (average) is significant.

(1) Having started the emacs server with

emacs --daemon=org

this shell script works for me:

#!/bin/sh
emacsclient -s org
--eval "(progn 
(add-to-list 'load-path \"/opt/org-6.33c/lisp/\")
(require 'org)(require 'org-exp)
(setq org-export-headline-levels 2)
(find-file \"$1\")
(org-export-as-html 2 nil nil nil nil \".\")
(kill-buffer))"

I have added a (kill-buffer) statement for cleaning up (I observed an increasing number of emacsclient processes before), is this a good solution?

(2) A problem remains with (1): it seems that I need to "name" the daemon. Otherwise there could be a conflict with Emacs daemons started by other users. How can I make sure the daemon is running if the script requires it? I read about a "-a" option for emacsclient - how would this work in my case?

Many thanks in advance.
Warm regards,
 Stefan
-- 
Dr. Stefan Vollmar, Dipl.-Phys.
Max-Planck-Institut für neurologische Forschung
Gleuelerstr. 50, 50931 Köln, Germany
Tel.: +49-221-4726-213  FAX +49-221-4726-298
Tel.: +49-221-478-5713  Mobile: 0160-93874279
Email: vollmar@nf.mpg.de   http://www.nf.mpg.de

Re: Re: org batch job using emacsclient?

[Sebastian Rose]

Stefan Vollmar <vollmar@nf.mpg.de> writes:
> Gentlemen,
>
> thanks for the many helpful replies!
>
> To start with, it now works for me and I can measure a consistent speedup of 7x
> (!) for using emacsclient - in an interactive web service (conversion of
> org-files to HTML) the difference between 0.35 s and 2.40 s (average) is
> significant.

That was to be expected :)

> (1) Having started the emacs server with
>
> emacs --daemon=org
>
> this shell script works for me:
>
> #!/bin/sh
> emacsclient -s org
> --eval "(progn
> (add-to-list 'load-path \"/opt/org-6.33c/lisp/\")
> (require 'org)(require 'org-exp)
> (setq org-export-headline-levels 2)
> (find-file \"$1\")
> (org-export-as-html 2 nil nil nil nil \".\")
> (kill-buffer))"
>
> I have added a (kill-buffer) statement for cleaning up (I observed an increasing
> number of emacsclient processes before), is this a good solution?


  Does it help? Then I think it's ok.

  The `usual' way to close a server buffer is `C-x #' which is bound to
  the function `server-edit'.  That's what I use in org-protocol.el. The
  name is somewhat confusing, though.

  Another trick is, to simply not wait for a reply from the server at
  all:

     emacsclient --no-wait ...


> (2) A problem remains with (1): it seems that I need to "name" the daemon.
> Otherwise there could be a conflict with Emacs daemons started by other users.
> How can I make sure the daemon is running if the script requires it? I read
> about a "-a" option for emacsclient - how would this work in my case?

  I would start Emacs in rc.emacs-server like this (but maybe use no
  server at all, see below):

     /path/to/emacs -Q -l /etc/emacs-server/init.el --daemon=SERVER-NAME


     /path/to/emacsclient --no-wait --socket-name=SERVER-NAME xy.org

  You could also use an absolute path for the socket.

  and put all required setup into /etc/emacs-server/init.el. `-Q'
  ensures emacs will start correctly, even if a user without a
  home-directory (e.g. apache user) starts it and thus no ~/.emacs can
  be found (start-stop-daemon switches to the UID you tell it). Also, if
  started as root, emacs will _not_ load root's .emacs.


  Start Emacs in an /etc/init.d/rc.emacs-server script to make sure it
  starts when you restart the server. Use `start-stop-daemon' to do so
  [1] and provide the commands `start', `stop', `restart' and `status'
  maybe. A cron-job could check for a running emacs-server via

     /etc/init.d/rc.emacs-server status

  every 5 minutes or so. If that fails, re-start Emacs.

  Your script could check for the running Emacs, too, and restart it as
  necessary.

  Are you familiar with init.d stuff? There is an example file somewhere
  on Debian...
  http://github.com/SebastianRose/denycc/blob/master/scripts/rc.denycc
  is made from such an example file for Debian 4.0


  Access privileges might be important. Unfortunately I have no good
  concept at hand - I don't thing emacs-server offers one at all,
  actually. Something to read...

  At least I would add something like this to your emacsclient
  shell-script (insert you Apaches EUID for the 99):

  if [ 99 -ne $EUID ];then
      echo "Got to be a native American to execute this command!"
      exit 124
  fi





Best wishes

  Sebastian


== Footnotes: ====

[1] From `man start-stop-daemon' (German version - English version below):

    BESCHREIBUNG
       start-stop-daemon wird zur Steuerung der Erzeugung und Beendigung
       von Prozessen auf Syste‐ mebene verwendet. Durch die Verwendung
       einer der Abgleich-Optionen kann start-stop-daemon so
       konfiguriert werden, dass er existierende Instanzen von einem
       laufenden Prozess finden kann.

       Hinweis: Falls --pidfile nicht angegeben ist, verhält sich
       start-stop-daemon ähnlich zu killall(1).  start-stop-daemon wird
       die Prozesstabelle nach Prozessen durchsuchen, die auf den
       Prozessnamen, uid und/oder gid (falls angegeben) passen. Jeder
       passende Prozess wird --start daran hindern, den Daemon zu
       starten. Allen passenden Prozessen wird das KILL-Sig‐ nal
       geschickt, falls --stop angegeben ist. Für Daemons die langlebige
       Kinder haben, die ein --stop überleben sollen, müssen Sie eine
       PID-Datei angeben.

 * man -L en_US start-stop-daemon:

   DESCRIPTION
       start-stop-daemon is used to control the creation and termination
       of system-level pro‐ cesses.  Using one of the matching options,
       start-stop-daemon can be configured to find existing instances of
       a running process.

       Note: unless --pidfile is specified, start-stop-daemon behaves
       similar to killall(1).  start-stop-daemon will scan the process
       table looking for any processes which match the process name,
       uid, and/or gid (if specified). Any matching process will prevent
       --start from starting the daemon. All matching processes will be
       sent the KILL signal if --stop is specified. For daemons which
       have long-lived children which need to live through a --stop, you
       must specify a pidfile.

Re: Re: org batch job using emacsclient?

[Stefan Vollmar]

Dear Sebastian,

many thanks for your detailed reply!

On 17.11.2009, at 03:25, Sebastian Rose wrote:

>> To start with, it now works for me and I can measure a consistent speedup of 7x
>> (!) for using emacsclient - in an interactive web service (conversion of
>> org-files to HTML) the difference between 0.35 s and 2.40 s (average) is
>> significant.
> 
> That was to be expected :)

I had certainly hoped so. However, this performance is still disappointing considering the "horse power" of modern systems - and this although I checked that all Lisp sources have been compiled. But I will now officially stop whining: < 0.5 s for a typical interactive query is acceptable from a user's point of view.

>> I have added a (kill-buffer) statement for cleaning up (I observed an increasing
>> number of emacsclient processes before), is this a good solution?
> 
>  Does it help? Then I think it's ok.
> 
>  The `usual' way to close a server buffer is `C-x #' which is bound to
>  the function `server-edit'.  That's what I use in org-protocol.el. The
>  name is somewhat confusing, though.

+1

>     /path/to/emacs -Q -l /etc/emacs-server/init.el --daemon=SERVER-NAME
>     /path/to/emacsclient --no-wait --socket-name=SERVER-NAME xy.org

(1) I have changed the code to 

#!/bin/sh
emacsclient --no-wait -s org
--eval "(progn
(add-to-list 'load-path \"/opt/org-6.33c/lisp/\")
(require 'org)(require 'org-exp)
(setq org-export-headline-levels 2)
(find-file \"$1\")
(org-export-as-html 2 nil nil nil nil \".\"))"

omitting the (kill-buffer) is a speed improvement and no emacsclient processes remain - nice! However, what happens to any buffers, temporary or otherwise, created during org conversion - will they accumulate in the server process?

(2) I have not tried but it is probably important to start emacs-server as the same user who is then going to run emacsclient (root vs webservd)?

(3) 
>  Are you familiar with init.d stuff? There is an example file somewhere
>  on Debian...
>  http://github.com/SebastianRose/denycc/blob/master/scripts/rc.denycc
>  is made from such an example file for Debian 4.0

Yes, I still grew up with this. However, this is (slightly) deprecated on Solaris 10: we are supposed to use services instead, http://www.sun.com/bigadmin/content/selfheal/smf-quickstart.jsp (I have been happily using this feature of Solaris 10 but so far not created a new service which would be required here).

Many thanks in advance for any comments.

Warm regards,
 Stefan
-- 
Dr. Stefan Vollmar, Dipl.-Phys.
Max-Planck-Institut für neurologische Forschung
Gleuelerstr. 50, 50931 Köln, Germany
Tel.: +49-221-4726-213  FAX +49-221-4726-298
Tel.: +49-221-478-5713  Mobile: 0160-93874279
Email: vollmar@nf.mpg.de   http://www.nf.mpg.de

Re: Re: org batch job using emacsclient?

[Sebastian Rose]

Stefan Vollmar <vollmar@nf.mpg.de> writes:

> Dear Sebastian,
>
> many thanks for your detailed reply!
>
> On 17.11.2009, at 03:25, Sebastian Rose wrote:
>
>>> To start with, it now works for me and I can measure a consistent speedup of
> 7x
>>> (!) for using emacsclient - in an interactive web service (conversion of
>>> org-files to HTML) the difference between 0.35 s and 2.40 s (average) is
>>> significant.
>> 
>> That was to be expected :)
>
> I had certainly hoped so. However, this performance is still disappointing
> considering the "horse power" of modern systems - and this although I checked
> that all Lisp sources have been compiled. But I will now officially stop
> whining: < 0.5 s for a typical interactive query is acceptable from a user's
> point of view.
>
>>> I have added a (kill-buffer) statement for cleaning up (I observed an
> increasing
>>> number of emacsclient processes before), is this a good solution?
>> 
>>  Does it help? Then I think it's ok.
>> 
>>  The `usual' way to close a server buffer is `C-x #' which is bound to
>>  the function `server-edit'.  That's what I use in org-protocol.el. The
>>  name is somewhat confusing, though.
>
> +1
>
>>     /path/to/emacs -Q -l /etc/emacs-server/init.el --daemon=SERVER-NAME
>>     /path/to/emacsclient --no-wait --socket-name=SERVER-NAME xy.org
>
> (1) I have changed the code to 
>
> #!/bin/sh
> emacsclient --no-wait -s org
> --eval "(progn
> (add-to-list 'load-path \"/opt/org-6.33c/lisp/\")
> (require 'org)(require 'org-exp)
> (setq org-export-headline-levels 2)
> (find-file \"$1\")
> (org-export-as-html 2 nil nil nil nil \".\"))"
>
> omitting the (kill-buffer) is a speed improvement and no emacsclient processes
> remain - nice! However, what happens to any buffers, temporary or otherwise,
> created during org conversion - will they accumulate in the server process?

Hm - I'd guess so. You could find out by simply opening lots of files
the way you use emacsclient.

Doing

  for f in $(find /etc/ -type f -name '*conf'); do
     emacsclient --no-wait "${f}";
  done

makes my emacs process grow quite a bit, as `ps aux | grep emacs' shows
:-D

Maybe even add a function to your emacs-server to log the opening and
closing of files together with a timestamp? E.g. advising `find-file'
and `kill-buffer'.



> (2) I have not tried but it is probably important to start emacs-server as the
> same user who is then going to run emacsclient (root vs webservd)?


I did - it is necessary, since the socket is read- and writeonly for the
owner:

  <sebastian@beteigeuze:/home/sebastian> ls -l /tmp/emacs1000/
  insgesamt 0
  srwx------ 1 sebastian sebastian 0 2009-11-17 12:17 server
  <sebastian@beteigeuze:/home/sebastian> su - emacs22
  Passwort:
  emacs22@beteigeuze:~$ /usr/local/bin/emacsclient --no-wait -s \
                        /tmp/emacs1000/server xy.org
  /usr/local/bin/emacsclient: can't stat /tmp/emacs1000/server: Permission denied
  /usr/local/bin/emacsclient: error accessing socket "/tmp/emacs1000/server"



  



> (3) 
>>  Are you familiar with init.d stuff? There is an example file somewhere
>>  on Debian...
>>  http://github.com/SebastianRose/denycc/blob/master/scripts/rc.denycc
>>  is made from such an example file for Debian 4.0
>
> Yes, I still grew up with this. However, this is (slightly) deprecated on
> Solaris 10: we are supposed to use services instead,
> http://www.sun.com/bigadmin/content/selfheal/smf-quickstart.jsp (I have been
> happily using this feature of Solaris 10 but so far not created a new service
> which would be required here).

Is this in OpenSolaris that way too? I should install it in my VM.

On Linux we now have this /etc/event.d/ - I think I read something about
it... the /etc/init.d/rcX mechanism has changed somehow. Not sure
though. I have put this on my list.



Best wishes

  Sebastian