Emacsclient sends ^J, PicoLisp receives _

Emacsclient sends ^J, PicoLisp receives _

[Thorsten Jolitz]

Hi List, 

I had this problem before but could not even get close to solving it
even with the help of this mailing list, so I try it again.

In a program, I let PicoLisp call Emacsclient to do some work and
receive the result as a nested list (with the syntax transformed from
Emacs Lisp to PicoLisp so it can be read-in right away).

This works quite well in most cases, but sometimes not.

Emacsclient sends this:

#+begin_src picolisp
  ## sent from Emacs
  (section (parent-id 8) "[2013-06-28 Fr 11:01]^J")
  ^J(headline (org-elem-id 12 ...) )
#+end_src

PicoLisp receives this:

#+begin_src picolisp
  ## received from PicoLisp 
   (section (parent-id 8) "[2013-06-28 Fr 11:01]^J")
  _ (headline (org-elem-id 12 ...
#+end_src

here, and only here, ^J is received as an underscore (_) , while
it is received as line-break in all the other cases (as desired).

Now its very hard to tell what might happen here. Who causes this
(Emacsclient? PicoLisp?) and why? It does not look any different from
the many other occurences of ^J in the nested list that is send.

Any hint would be welcome.

-- 
cheers,
Thorsten

Re: Emacsclient sends ^J, PicoLisp receives _

[Thorsten Jolitz]

Thorsten Jolitz <tjolitz@gmail.com> writes:

ok, I figured out that the problem is actually a different one than
described. This is what I see in an Emacs buffer when I'm in
Emacs(client) and let Emacs produce the nested list that will be sent
to PicoLisp:

> Emacsclient sends this:
>
> #+begin_src picolisp
>   ## sent from Emacs
>   (section (parent-id 8) "[2013-06-28 Fr 11:01]^J")
>   ^J(headline (org-elem-id 12 ...) )
> #+end_src

but when I go to the other side and call Emacsclient from PicoLisp and
redirect what is sent into a file, it looks like this:

#+begin_src picolisp
  (section (parent-id 8) \"[2013-06-28 Fr
  11:01]^J\")_^J (headline (org-elem-id 12 ...) )
#+end_src

thus the underscore (_) is already present in the nested list before it
is sent to PicoLisp, so it must be a problem on the Emacs side (and it
is not that ^J is transformed into _, ^J is treated correctly, but for
some reason the unexpected _ is printed in this place). 

So it might rather be a problem with the Org-mode exporter who
produces this nested list from an Org-mode file/buffer. So the
real questions are probably:

 1. what invisible character might be transformed into an _ by the
    exporter?
 2. Why does the exporter behave differently when called from Emacs on
    an open Org-mode buffer (correct output) then when called from an
    external program (incorrect output with strange underscore)?

I will post this on the Org-mode mailing list too, but any hints from
this mailing list are still welcome. 

-- 
cheers,
Thorsten

Re: Emacsclient sends ^J, PicoLisp receives _

[Thorsten Jolitz]

Thorsten Jolitz <tjolitz@gmail.com> writes:

Again following up to myself, I detected another possible culprit for
the 'bad' underscore introduced when sending output from Emacsclient to
PicoLisp:

In emacsclient.c I found this function:

#+begin_src C
  /* In STR, insert a & before each &, each space, each newline, and
     any initial -.  Change spaces to underscores, too, so that the
     return value never contains a space.
  
     Does not change the string.  Outputs the result to S.  */
  static void
  quote_argument (HSOCKET s, const char *str)
  {
    char *copy = (char *) xmalloc (strlen (str) * 2 + 1);
    const char *p;
    char *q;
  
    p = str;
    q = copy;
    while (*p)
      {
        if (*p == ' ')
          {
            ,*q++ = '&';
            ,*q++ = '_';
            p++;
          } [...]
#+end_src

that is called (together with its antipode `unquote_argument') quite a
few times:

,----------------------------------------------------------------------------
| 17 matches for "quote_argument" in buffer: emacsclient.c
|     782:quote_argument (HSOCKET s, const char *str)
|     819:/* The inverse of quote_argument.  Removes quoting in string STR by
|     823:unquote_argument (char *str)
|    1599:          quote_argument (emacs_socket, environ[i]);
|    1604:  quote_argument (emacs_socket, cwd);
|    1618:      quote_argument (emacs_socket, display);
|    1625:      quote_argument (emacs_socket, parent_id);
|    1632:      quote_argument (emacs_socket, frame_parameters);
|    1649:          quote_argument (emacs_socket, tty_name);
|    1651:          quote_argument (emacs_socket, tty_type);
|    1669:              quote_argument (emacs_socket, argv[i]);
|    1681:                  quote_argument (emacs_socket, argv[i]);
|    1708:          quote_argument (emacs_socket, argv[i]);
|    1718:          quote_argument (emacs_socket, str);
|    1786:              str = unquote_argument (p + strlen ("-print "));
|    1796:              str = unquote_argument (p + strlen ("-print-nonl "));
|    1803:              str = unquote_argument (p + strlen ("-error "));
`----------------------------------------------------------------------------

Now I'm not that much of a C programmer, so I'm not sure what is
actually quoted and unquoted.

Is it possible that Emacsclients quotes and unquotes the output string
produced by my Org-exporter when sending it to PicoLisp (replacing
spaces with underscores and vice-versa) and occasionally things go
wrong in the process (i.e. some underscores are left in the final
string arriving at the PicoLisp side)?

Or is this quoting/unquoting done by Emacsclient completely unrelated
to my issue?

-- 
cheers,
Thorsten

Re: Emacsclient sends ^J, PicoLisp receives _

[duthen01]

Hello!
I won't certainly not solve your problem! Sorry!
But I seems that it looks like a bug!
So, I would suggest, if this is possible, that you give the minimal information so that another person can reproduce this behavior/bug!
HTH
)jack(

Re: Emacsclient sends ^J, PicoLisp receives _

[Thorsten Jolitz]

duthen01@gmail.com writes:

> Hello!
> I won't certainly not solve your problem! Sorry!
> But I seems that it looks like a bug!
> So, I would suggest, if this is possible, that you give the minimal
> information so that another person can reproduce this behavior/bug!
> HTH
> )jack(

It is not yet possible unfortunately because I'm not even sure whom to
blame and I'm not able to create a MWE that somebody else can reproduce
without setting up my whole working environment. 

I only know that 

- when I call the Org-mode HTML exporter from PicoLisp, the HTML string
  received by PicoLisp is wrapped into double-quotes (visible in the
  browser) when I use Emacsclient, but not when I use Emacs in batch
  mode (I blame Emacsclient for that problem). 

- when I call my own Org-mode exporter that returns a nested list in
  PicoLisp syntax that should be readable as is by PicoLisp, sometimes
  and in some places there are 'bad underscores' in the return string.
  But I don't see anything special in the buffer positions where this
  happens (not even with whitespace-mode). I tend to blame Emacsclient
  for this problem too (replacing spaces with underscores?), but I'm
  really not sure at all.

Once I figured out what happens (hopefully soon) and it turns out to be
a bug, I will file a report. 

-- 
cheers,
Thorsten