g-client: character coding problem

g-client: character coding problem

[Joseph Fahey]

Hello all,

I am trying to use T. V. Ramah's g-client interface to the Google API,
in particular as an interface to Blogger.

http://emacspeak.blogspot.com/2007/03/emacs-client-for-google-services.html#cooliris

G-client, and more specifically gblogger.el, works great for me... as
long as I stick to ASCII characters. Any accented characters from the
iso-latin-1 subgroup show up incorrectly. I've tried a lot of
different things to alter the coding system, mostly by setting
everything I can to utf-8-unix.

I think I have found the problem, but am not sure how to solve it.
gblogger.el uses xsltproc via shell-command-on-region to prepare a
blog, before sending the buffer using curl. It appears that the
characters are coming back malformed. Here is the function in
g-utils.el:

(defsubst g-xsl-transform-region (start end xsl)
  "Replace region by result of transforming via XSL."
  (declare (special g-xslt-program))
  (let ((coding-system-for-write 'utf-8))
  (shell-command-on-region
   start end
   (format "%s %s - %s"
           g-xslt-program xsl (g-xslt-debug))
   'replace)))

If I run the following code (the elisp in the middle of this xml) in a
utf-8 buffer (-u) I get malformed characters (the accents in the
"content" part):

<entry xmlns='http://www.w3.org/2005/Atom'>
  <generator url="http://purl.org/net/emacs-gblogger/">http://purl.org/net/emacs-gblogger/</generator>
  <author> <name>Me </name> </author>
  <title mode="escaped" type="text/html">être </title>
  <content type='xhtml'>
    <div xmlns="http://www.w3.org/1999/xhtml">
<!--content goes here -->
<p>Être ou ne pas être ? 

  (g-xsl-transform-region 
  (point-min) (point-max) "~/elisp/g-client/blogger-edit-post.xsl")

</p>
    </div>
  </content>
</entry>

I get the same results if I change the code to:

 (let ((coding-system-for-write 'utf-8-unix))
  (g-xsl-transform-region
   (point-min) (point-max) "~/elisp/g-client/blogger-edit-post.xsl")) 

Here is what I get if I do "C-h C":

Coding system for saving this buffer:
  Not set locally, use the default.
Default coding system (for new files):
  u -- utf-8-unix (alias of mule-utf-8-unix)

Coding system for keyboard input:
  nil
Coding system for terminal output:
  u -- utf-8 (alias of mule-utf-8)

Defaults for subprocess I/O:
  decoding: 1 -- iso-latin-1 (alias: iso-8859-1 latin-1)

  encoding: 1 -- iso-latin-1 (alias: iso-8859-1 latin-1)


Priority order for recognizing coding systems when reading files:
  1. iso-latin-1 (alias: iso-8859-1 latin-1)
  2. windows-1252 (alias: cp1252)
  3. mule-utf-8 (alias: utf-8) ... etc.

I suspect that the problem is coming from the defaults for subprocess
I/O, but I'm not sure how to change that.

So... here I am at wit's end. Any ideas would be much appreciated.

thanks

Joe

PS: this is on Linux with GNU Emacs 22.0.94.2. Here is the output from locale:

LANG=fr_FR.UTF-8@euro
LC_CTYPE=fr_FR.UTF-8
LC_NUMERIC="fr_FR.UTF-8@euro"
LC_TIME="fr_FR.UTF-8@euro"
LC_COLLATE="fr_FR.UTF-8@euro"
LC_MONETARY="fr_FR.UTF-8@euro"
LC_MESSAGES=C
LC_PAPER="fr_FR.UTF-8@euro"
LC_NAME="fr_FR.UTF-8@euro"
LC_ADDRESS="fr_FR.UTF-8@euro"
LC_TELEPHONE="fr_FR.UTF-8@euro"
LC_MEASUREMENT="fr_FR.UTF-8@euro"
LC_IDENTIFICATION="fr_FR.UTF-8@euro"
LC_ALL=

Re: g-client: character coding problem

[Hadron]

Joseph Fahey <fahey@fabula.org> writes:

> Hello all,
>
> I am trying to use T. V. Ramah's g-client interface to the Google API,
> in particular as an interface to Blogger.
>
> http://emacspeak.blogspot.com/2007/03/emacs-client-for-google-services.html#cooliris

With emacs 23 I get this error

,----
| Debugger entered--Lisp error: (file-error "Cannot open load file" "g-cus-load")
|   load("g-cus-load")
|   load-library("g-cus-load")
|   byte-code("ÀÁÂÃÄÅ%ˆÆÇ!ˆÆÈ!ˆÉÊË\"ˆÌÁ!‡" [custom-declare-group g nil "Google Client" :group applications load-library "g-loaddefs" "g-cus-load" add-hook calendar-mode-hook gcal-emacs-calendar-setup provide] 6)
|   load("g")
|   load-library("g")
|   eval((load-library "g"))
|   eval-last-sexp-1(nil)
|   eval-last-sexp(nil)
|   call-interactively(eval-last-sexp)
`----

Any ideas?

Re: g-client: character coding problem

[Hadron]

Hadron <hadronquark@gmail.com> writes:

> Joseph Fahey <fahey@fabula.org> writes:
>
>> Hello all,
>>
>> I am trying to use T. V. Ramah's g-client interface to the Google API,
>> in particular as an interface to Blogger.
>>
>> http://emacspeak.blogspot.com/2007/03/emacs-client-for-google-services.html#cooliris
>
> With emacs 23 I get this error
>
> ,----
> | Debugger entered--Lisp error: (file-error "Cannot open load file" "g-cus-load")
> |   load("g-cus-load")
> |   load-library("g-cus-load")
> |   byte-code("ÀÁÂÃÄÅ%ˆÆÇ!ˆÆÈ!ˆÉÊË\"ˆÌÁ!‡" [custom-declare-group g nil "Google Client" :group applications load-library "g-loaddefs" "g-cus-load" add-hook calendar-mode-hook gcal-emacs-calendar-setup provide] 6)
> |   load("g")
> |   load-library("g")
> |   eval((load-library "g"))
> |   eval-last-sexp-1(nil)
> |   eval-last-sexp(nil)
> |   call-interactively(eval-last-sexp)
> `----
>
> Any ideas?

ok, just commenting out the line in g.el which loads g-cus-load gets it
working, but what lisp would load the file only if it exists?

-- 

Re: g-client: character coding problem

[Joseph Fahey]

>>>>> "h" == Hadron  <hadronquark@gmail.com> writes:

    h> Hadron <hadronquark@gmail.com> writes:
    >> Joseph Fahey <fahey@fabula.org> writes:
    >> 
    >>> I am trying to use T. V. Ramah's g-client interface to the
    >>> Google API, in particular as an interface to Blogger.
    >>> 
    >>> http://emacspeak.blogspot.com/2007/03/emacs-client-for-google-services.html#cooliris
    >> 
    >> With emacs 23 I get this error
    >> 
    >> ,---- | Debugger entered--Lisp error: (file-error "Cannot open
    >> load file" "g-cus-load") | load("g-cus-load") |
    >> load-library("g-cus-load") |
    >> byte-code("ÀÁÂÃÄÅ%.ÆÇ!.ÆÈ!.ÉÊË\".ÌÁ!."
    >> [custom-declare-group g nil "Google Client" :group applications
    >> load-library "g-loaddefs" "g-cus-load" add-hook
    >> calendar-mode-hook gcal-emacs-calendar-setup provide] 6) |
    >> load("g") | load-library("g") | eval((load-library "g")) |
    >> eval-last-sexp-1(nil) | eval-last-sexp(nil) |
    >> call-interactively(eval-last-sexp) `----
    >> 
    >> Any ideas?

    h> ok, just commenting out the line in g.el which loads g-cus-load
    h> gets it working, but what lisp would load the file only if it
    h> exists?

That problem has been fixed in the most recent svn version. When I
first installed g-client, the fix was to add your own g-cus-load.el
file that would be empty except for "(provide 'g-cus-load)".

Joe

Re: g-client: character coding problem

[Hadron]

Joseph Fahey <fahey@fabula.org> writes:

>>>>>> "h" == Hadron  <hadronquark@gmail.com> writes:
>
>     h> Hadron <hadronquark@gmail.com> writes:
>     >> Joseph Fahey <fahey@fabula.org> writes:
>     >> 
>     >>> I am trying to use T. V. Ramah's g-client interface to the
>     >>> Google API, in particular as an interface to Blogger.
>     >>> 
>     >>> http://emacspeak.blogspot.com/2007/03/emacs-client-for-google-services.html#cooliris
>     >> 
>     >> With emacs 23 I get this error
>     >> 
>     >> ,---- | Debugger entered--Lisp error: (file-error "Cannot open
>     >> load file" "g-cus-load") | load("g-cus-load") |
>     >> load-library("g-cus-load") |
>     >> byte-code("ÀÁÂÃÄÅ%.ÆÇ!.ÆÈ!.ÉÊË\".ÌÁ!."
>     >> [custom-declare-group g nil "Google Client" :group applications
>     >> load-library "g-loaddefs" "g-cus-load" add-hook
>     >> calendar-mode-hook gcal-emacs-calendar-setup provide] 6) |
>     >> load("g") | load-library("g") | eval((load-library "g")) |
>     >> eval-last-sexp-1(nil) | eval-last-sexp(nil) |
>     >> call-interactively(eval-last-sexp) `----
>     >> 
>     >> Any ideas?
>
>     h> ok, just commenting out the line in g.el which loads g-cus-load
>     h> gets it working, but what lisp would load the file only if it
>     h> exists?
>
> That problem has been fixed in the most recent svn version. When I
> first installed g-client, the fix was to add your own g-cus-load.el
> file that would be empty except for "(provide 'g-cus-load)".
>
> Joe

Why would anyone leave the main release with an error in it that
literally stops emacs loading properly? Strange.

Re: g-client: character coding problem

[Enchanter]

I am using emacs under Windows. So I can not do "make" than How could
I get the auto-generated code? I can not use it now.

Thanks

Re: g-client: character coding problem

[Tim X]

Hadron <hadronquark@gmail.com> writes:

> Joseph Fahey <fahey@fabula.org> writes:
>
>>>>>>> "h" == Hadron  <hadronquark@gmail.com> writes:
>>
>>     h> Hadron <hadronquark@gmail.com> writes:
>>     >> Joseph Fahey <fahey@fabula.org> writes:
>>     >> 
>>     >>> I am trying to use T. V. Ramah's g-client interface to the
>>     >>> Google API, in particular as an interface to Blogger.
>>     >>> 
>>     >>> http://emacspeak.blogspot.com/2007/03/emacs-client-for-google-services.html#cooliris
>>     >> 
>>     >> With emacs 23 I get this error
>>     >> 
>>     >> ,---- | Debugger entered--Lisp error: (file-error "Cannot open
>>     >> load file" "g-cus-load") | load("g-cus-load") |
>>     >> load-library("g-cus-load") |
>>     >> byte-code("ÀÁÂÃÄÅ%.ÆÇ!.ÆÈ!.ÉÊË\".ÌÁ!."
>>     >> [custom-declare-group g nil "Google Client" :group applications
>>     >> load-library "g-loaddefs" "g-cus-load" add-hook
>>     >> calendar-mode-hook gcal-emacs-calendar-setup provide] 6) |
>>     >> load("g") | load-library("g") | eval((load-library "g")) |
>>     >> eval-last-sexp-1(nil) | eval-last-sexp(nil) |
>>     >> call-interactively(eval-last-sexp) `----
>>     >> 
>>     >> Any ideas?
>>
>>     h> ok, just commenting out the line in g.el which loads g-cus-load
>>     h> gets it working, but what lisp would load the file only if it
>>     h> exists?
>>
>> That problem has been fixed in the most recent svn version. When I
>> first installed g-client, the fix was to add your own g-cus-load.el
>> file that would be empty except for "(provide 'g-cus-load)".
>>
>> Joe
>
> Why would anyone leave the main release with an error in it that
> literally stops emacs loading properly? Strange.

Not that strange if you realise g-client has been developed as part of
emacspeak and if you use it with emacspeak, that file is not empty.

BTW, can someone tell me what emacs 23 is? I'm running from CVS and it reports
the version as emacs 22. I've seen a number of references to emacs 23, which
I've not heard of and didn't think existed (given that I would expect the CVS
to be the latest version).

Tim

Tim
-- 
tcross (at) rapttech dot com dot au

Re: g-client: character coding problem

[Tim X]

Enchanter <ensoul.magazine@gmail.com> writes:

> I am using emacs under Windows. So I can not do "make" than How could
> I get the auto-generated code? I can not use it now.
>
> Thanks
>

Its unlikely to work anyway. Emacspeak is only developed/maintained on GNU
Linux based systems and takes advantage of GNU Linux based utilities that are
normally not found on Windows. While the g-client stuff is likely to prove
useful for the general population, Raman's main objective is to enhance the
functionality of emacspeak rather than create a general utility for the wider
community. If it works for a broader user base, great, but don't expect that to
be a priority. . 

Tim

-- 
tcross (at) rapttech dot com dot au

Re: g-client: character coding problem

[Hadron]

Tim X <timx@nospam.dev.null> writes:

> Hadron <hadronquark@gmail.com> writes:
>
>> Joseph Fahey <fahey@fabula.org> writes:
>>
>>>>>>>> "h" == Hadron  <hadronquark@gmail.com> writes:
>>>
>>>     h> Hadron <hadronquark@gmail.com> writes:
>>>     >> Joseph Fahey <fahey@fabula.org> writes:
>>>     >> 
>>>     >>> I am trying to use T. V. Ramah's g-client interface to the
>>>     >>> Google API, in particular as an interface to Blogger.
>>>     >>> 
>>>     >>> http://emacspeak.blogspot.com/2007/03/emacs-client-for-google-services.html#cooliris
>>>     >> 
>>>     >> With emacs 23 I get this error
>>>     >> 
>>>     >> ,---- | Debugger entered--Lisp error: (file-error "Cannot open
>>>     >> load file" "g-cus-load") | load("g-cus-load") |
>>>     >> load-library("g-cus-load") |
>>>     >> byte-code("ÀÁÂÃÄÅ%.ÆÇ!.ÆÈ!.ÉÊË\".ÌÁ!."
>>>     >> [custom-declare-group g nil "Google Client" :group applications
>>>     >> load-library "g-loaddefs" "g-cus-load" add-hook
>>>     >> calendar-mode-hook gcal-emacs-calendar-setup provide] 6) |
>>>     >> load("g") | load-library("g") | eval((load-library "g")) |
>>>     >> eval-last-sexp-1(nil) | eval-last-sexp(nil) |
>>>     >> call-interactively(eval-last-sexp) `----
>>>     >> 
>>>     >> Any ideas?
>>>
>>>     h> ok, just commenting out the line in g.el which loads g-cus-load
>>>     h> gets it working, but what lisp would load the file only if it
>>>     h> exists?
>>>
>>> That problem has been fixed in the most recent svn version. When I
>>> first installed g-client, the fix was to add your own g-cus-load.el
>>> file that would be empty except for "(provide 'g-cus-load)".
>>>
>>> Joe
>>
>> Why would anyone leave the main release with an error in it that
>> literally stops emacs loading properly? Strange.
>
> Not that strange if you realise g-client has been developed as part of
> emacspeak and if you use it with emacspeak, that file is not empty.
>
> BTW, can someone tell me what emacs 23 is? I'm running from CVS and it reports
> the version as emacs 22. I've seen a number of references to emacs 23, which
> I've not heard of and didn't think existed (given that I would expect the CVS
> to be the latest version).
>

I installed from here.

http://peadrop.com/blog/2007/01/06/pretty-emacs/

Re: g-client: character coding problem

[Hadron]

Tim X <timx@nospam.dev.null> writes:

> Hadron <hadronquark@gmail.com> writes:
>
>> Joseph Fahey <fahey@fabula.org> writes:
>>
>>>>>>>> "h" == Hadron  <hadronquark@gmail.com> writes:
>>>
>>>     h> Hadron <hadronquark@gmail.com> writes:
>>>     >> Joseph Fahey <fahey@fabula.org> writes:
>>>     >> 
>>>     >>> I am trying to use T. V. Ramah's g-client interface to the
>>>     >>> Google API, in particular as an interface to Blogger.
>>>     >>> 
>>>     >>> http://emacspeak.blogspot.com/2007/03/emacs-client-for-google-services.html#cooliris
>>>     >> 
>>>     >> With emacs 23 I get this error
>>>     >> 
>>>     >> ,---- | Debugger entered--Lisp error: (file-error "Cannot open
>>>     >> load file" "g-cus-load") | load("g-cus-load") |
>>>     >> load-library("g-cus-load") |
>>>     >> byte-code("ÀÁÂÃÄÅ%.ÆÇ!.ÆÈ!.ÉÊË\".ÌÁ!."
>>>     >> [custom-declare-group g nil "Google Client" :group applications
>>>     >> load-library "g-loaddefs" "g-cus-load" add-hook
>>>     >> calendar-mode-hook gcal-emacs-calendar-setup provide] 6) |
>>>     >> load("g") | load-library("g") | eval((load-library "g")) |
>>>     >> eval-last-sexp-1(nil) | eval-last-sexp(nil) |
>>>     >> call-interactively(eval-last-sexp) `----
>>>     >> 
>>>     >> Any ideas?
>>>
>>>     h> ok, just commenting out the line in g.el which loads g-cus-load
>>>     h> gets it working, but what lisp would load the file only if it
>>>     h> exists?
>>>
>>> That problem has been fixed in the most recent svn version. When I
>>> first installed g-client, the fix was to add your own g-cus-load.el
>>> file that would be empty except for "(provide 'g-cus-load)".
>>>
>>> Joe
>>
>> Why would anyone leave the main release with an error in it that
>> literally stops emacs loading properly? Strange.
>
> Not that strange if you realise g-client has been developed as part of
> emacspeak and if you use it with emacspeak, that file is not empty.

No. It is that strange. The download I took *is* a standalone
package. You have said "why it is broken" rather than justify it not
being corrected. I'm not necessarily criticising, just pointing out that
the effort to replace the package file is very small and would enable a
few people to "just install and use it" rather than have a totally broken
emacs.

It's a neat little package btw!

Re: g-client: character coding problem

[Peter Dyballa]

Am 14.05.2007 um 09:45 schrieb Tim X:

> BTW, can someone tell me what emacs 23 is? I'm running from CVS and  
> it reports
> the version as emacs 22. I've seen a number of references to emacs  
> 23, which
> I've not heard of and didn't think existed (given that I would  
> expect the CVS
> to be the latest version).

It's another branch (unicode-2) of GNU Emacs. It is based on Unicode  
as a means to represent text characters. Therefore a particular  
character has the same code no matter what encoding is used in that  
buffer. You can isearch for à or ï in every buffer and every  
encoding. It also uses anti-aliasing (although I am not sure whether  
I see the effect) and an experimental font-backend, that seems to be  
different from the fontsets used in earlier Emacsen. Another  
difference is the use of font-config library. Recently libotf has  
been introduced into the code, but it still does not use OpenType  
fonts. All the changes that happen to GNU Emacs 22 in CVS can be  
found after some delay in Unicode Emacs 23.

Documentation is not perfect, there is no explicit mentioning of  
Emacs 23 specific functions and you have to search a lot to  
understand how to make use of the additions. It seems to me that the  
extra code for Carbon Emacs (for Mac OS X) is buggy and not supported  
and maintained. Besides, the X client runs quite reliable!

--
Greetings

   Pete

There's no sense in being precise when you don't even know what  
you're talking about.
                                   -- John von Neumann

Re: g-client: character coding problem

[Hadron]

Peter Dyballa <Peter_Dyballa@Web.DE> writes:

> Am 14.05.2007 um 09:45 schrieb Tim X:
>
>> BTW, can someone tell me what emacs 23 is? I'm running from CVS and
>> it reports
>> the version as emacs 22. I've seen a number of references to emacs
>> 23, which
>> I've not heard of and didn't think existed (given that I would
>> expect the CVS
>> to be the latest version).
>
> It's another branch (unicode-2) of GNU Emacs. It is based on Unicode
> as a means to represent text characters. Therefore a particular
> character has the same code no matter what encoding is used in that
> buffer. You can isearch for à or ï in every buffer and every
> encoding. It also uses anti-aliasing (although I am not sure whether
> I see the effect) and an experimental font-backend, that seems to be
> different from the fontsets used in earlier Emacsen. Another
> difference is the use of font-config library. Recently libotf has
> been introduced into the code, but it still does not use OpenType
> fonts. All the changes that happen to GNU Emacs 22 in CVS can be
> found after some delay in Unicode Emacs 23.
>
> Documentation is not perfect, there is no explicit mentioning of Emacs
> 23 specific functions and you have to search a lot to  understand how
> to make use of the additions. It seems to me that the  extra code for
> Carbon Emacs (for Mac OS X) is buggy and not supported  and
> maintained. Besides, the X client runs quite reliable!

Interestingly enough, with exact same .emacs, my "international german
characters" have stopped working in both input ( I have to press umlaut A
3 times before a japanese character appears) and output (any mail I
receive with german characters just shows squares).

Any idea what remedial action to take? is there some sort of x-resource
I need to change to get my german characters back?



-- 

Re: g-client: character coding problem

[Peter Dyballa]

Am 14.05.2007 um 12:31 schrieb Hadron:

> Any idea what remedial action to take? is there some sort of x- 
> resource
> I need to change to get my german characters back?

Restrict your code to all non-23 Emacsen!

--
Mit friedvollen Grüßen

   Pete

"Specifications are for the weak and timid!"

Re: g-client: character coding problem

[Tim X]

Hadron <hadronquark@gmail.com> writes:

>> BTW, can someone tell me what emacs 23 is? I'm running from CVS and it reports
>> the version as emacs 22. I've seen a number of references to emacs 23, which
>> I've not heard of and didn't think existed (given that I would expect the CVS
>> to be the latest version).
>>
>
> I installed from here.
>
> http://peadrop.com/blog/2007/01/06/pretty-emacs/

OK, so emacs 23 is a customized version put together by someone which is
available in a deb package for ubuntu. Now it makes a bit morer sense. So,
really, its not emacs 23 at all, but rather emacs 22 with custom
patches/extensions. 

thanks,

Tim

-- 
tcross (at) rapttech dot com dot au