po file charset via auto-coding-functions

po file charset via auto-coding-functions

[Kevin Ryde]

[-- Attachment #1: Type: text/plain, Size: 2060 bytes --]

This is a proposal to get the coding system for a .po file via
auto-coding-functions, instead of the way textmodes/po.el reads the
file explicitly.

2005-10-20  Kevin Ryde  <user42@zip.com.au>

        * international/mule.el (po-content-type-charset-alist): Moved from
        textmodes.el, add "CHARSET" which is a placeholder from xgettext.
        (po-auto-coding-function): New function.  This gets the right coding
	system when visiting a .po via archive-mode; po-find-file-coding-system
        only worked on a normal file.  charset= regexp from textmodes/po.el.
        (auto-coding-functions): Use po-auto-coding-function.
        * international/mule-conf.el (file-coding-system-alist): Remove
        po-find-file-coding-system.
        * textmodes/po.el: Remove file, no longer used.


One possible problem is that po files can have more than 1024 bytes of
comments before the header info block.  I see fileio.c
Finsert_file_contents only grabs 1024 bytes before calling
set-auto-coding, but I can't tell if/when that happens.  I think a
normal visit or an `archive-extract' has the whole file, so they work.


I used the following bit of code to exercise po-auto-coding-function
on all my .po files.  The function prints messages about bad charsets,
the result is a list of the bad files.

(delq nil
      (mapcar (lambda (filename)
                (with-temp-buffer
                  (insert-file-contents-literally filename)
                  (goto-char (point-min))
                  (if (po-auto-coding-function (- (point-max) (point-min)))
                      nil
                    filename)))
              (delete "" (split-string
                          (shell-command-to-string "locate \\*.po") "\n"))))

Among my files I found two unrecognised:

"TCVN-5712" in gtk 1.2 vietnamese.  Is there a good place to map or
alias that to `tcvn' which emacs knows?

"iso-8859-9e" in gtk 1.2 Azerbaijani turkish, but I don't know what
that charset is or is meant to be.  glibc iconv doesn't seem to
recognise it, so presumably it's unused.



[-- Attachment #2: mule.el.po-coding.diff --]
[-- Type: text/plain, Size: 3146 bytes --]

*** mule.el.~1.226.~	2005-09-29 09:23:59.000000000 +1000
--- mule.el	2005-10-21 06:54:07.785993736 +1000
***************
*** 1586,1592 ****
  		       (symbol :tag "Coding system"))))
  
  ;; See the bottom of this file for built-in auto coding functions.
! (defcustom auto-coding-functions '(sgml-xml-auto-coding-function
  				   sgml-html-meta-auto-coding-function)
    "A list of functions which attempt to determine a coding system.
  
--- 1586,1593 ----
  		       (symbol :tag "Coding system"))))
  
  ;; See the bottom of this file for built-in auto coding functions.
! (defcustom auto-coding-functions '(po-auto-coding-function
!                                    sgml-xml-auto-coding-function
  				   sgml-html-meta-auto-coding-function)
    "A list of functions which attempt to determine a coding system.
  
***************
*** 2203,2208 ****
--- 2204,2254 ----
  
  ;;; Built-in auto-coding-functions:
  
+ (defconst po-content-type-charset-alist
+   '(("ASCII" . undecided)
+     ("ANSI_X3.4-1968" . undecided)
+     ("US-ASCII" . undecided)
+     ;; "charset=CHARSET" is generated by xgettext, and may be present before
+     ;; someone fills in their target charset.  `undecided' should be right.
+     ("CHARSET" . undecided))
+   "Alist of coding system versus GNU libc/libiconv canonical charset name.
+ Contains canonical charset names that don't correspond to coding systems.")
+ 
+ (defun po-auto-coding-function (size)
+   "Determine character encoding of a gettext .po or .pot file.
+ This function is designed for use in `auto-coding-functions'.
+ 
+ A po file starts with msgstr \"\" which has header information, in
+ particular \"Content-Type: text/plain; charset=ASCII\\n\" or whatever."
+ 
+   ;; Skip "#" comment lines and whitespace-only lines, then want
+   ;;     msgstr ""
+   ;;     msgid ""
+   ;; which, up to the next blank line, is the header info.
+   ;;
+   (and (looking-at
+         "\\(#.*\n\\|[ \t\r]*\n\\)*msgid \"\"[ \t\r]*\nmsgstr \"\"[ \t\r]*\n")
+        (let ((limit (+ (point) size)))
+          (save-excursion
+            (goto-char (match-end 0))
+ 
+            ;; Blank line is the end of the header, stop searching there, or
+            ;; at existing `limit' if the file is a header only.
+            (setq limit (or (save-excursion
+                              (re-search-forward "\n[ \t\r]*\n" limit t))
+                            limit))
+ 
+            (and (re-search-forward "^\"Content-Type:[ \t]*text/plain;[ \t]*charset=\\(.*\\)\\\\n\"" limit t)
+                 (let ((charset (match-string 1)))
+ 
+                   (or (cdr (assoc-string charset po-content-type-charset-alist
+                                          t))
+                       (locale-charset-to-coding-system charset)
+                       (progn
+                         (message "Warning: unknown coding system \"%s\""
+                                  charset)
+                         nil))))))))
+ 
  (defun sgml-xml-auto-coding-function (size)
    "Determine whether the buffer is XML, and if so, its encoding.
  This function is intended to be added to `auto-coding-functions'."

[-- Attachment #3: mule-conf.el.po-coding.diff --]
[-- Type: text/plain, Size: 493 bytes --]

*** mule-conf.el.~1.82.~	2005-08-04 06:28:21.000000000 +1000
--- mule-conf.el	2005-10-20 17:45:02.000000000 +1000
***************
*** 502,508 ****
  	;; the beginning of a doc string, work.
  	("\\(\\`\\|/\\)loaddefs.el\\'" . (raw-text . raw-text-unix))
  	("\\.tar\\'" . (no-conversion . no-conversion))
- 	( "\\.po[tx]?\\'\\|\\.po\\." . po-find-file-coding-system)
  	("\\.\\(tex\\|ltx\\|dtx\\|drv\\)\\'" . latexenc-find-file-coding-system)
  	("" . (undecided . nil))))
  
--- 502,507 ----

[-- Attachment #4: Type: text/plain, Size: 142 bytes --]

_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-devel

Re: po file charset via auto-coding-functions

[Kenichi Handa]

In article <87zmp399ue.fsf@zip.com.au>, Kevin Ryde <user42@zip.com.au> writes:

> This is a proposal to get the coding system for a .po file via
> auto-coding-functions, instead of the way textmodes/po.el reads the
> file explicitly.

I agree that if .po file can be handled correctly in the
frame work of auto-coding-functions, it's a good change.

But, does the current method have any problem that can't be
fixed within po.el?  If not, I think we should make such a
change after the current release.

> One possible problem is that po files can have more than 1024 bytes of
> comments before the header info block.  I see fileio.c
> Finsert_file_contents only grabs 1024 bytes before calling
> set-auto-coding, but I can't tell if/when that happens.  I think a
> normal visit or an `archive-extract' has the whole file, so they work.

Yes.

> "TCVN-5712" in gtk 1.2 vietnamese.  Is there a good place to map or
> alias that to `tcvn' which emacs knows?

I've just added this line in lisp/langauge/vietnamese.el.

(define-coding-system-alias 'tcvn-5712 'vietnamese-tcvn)

> "iso-8859-9e" in gtk 1.2 Azerbaijani turkish, but I don't know what
> that charset is or is meant to be.  glibc iconv doesn't seem to
> recognise it, so presumably it's unused.

I have no idea.

---
Kenichi Handa
handa@m17n.org

Re: po file charset via auto-coding-functions

[Richard M. Stallman]

    One possible problem is that po files can have more than 1024 bytes of
    comments before the header info block.  I see fileio.c
    Finsert_file_contents only grabs 1024 bytes before calling
    set-auto-coding, but I can't tell if/when that happens.

I think it ALWAYS happens.  Every call to Finsert_file_contents will
try to determine the coding system from the first 1k and last 3k of
the file.  (Unless it already knows the coding system to use.)

      I think a
    normal visit or an `archive-extract' has the whole file, so they work.

If Finsert_file_contents can't determine the coding system from that
part of the file, then if it is a normal visit (or if the buffer was
previously empty), it will try again after reading all the file.  So I
guess this will work ok in that case.

But if it is inserting the file into a buffer that had other text in
it, or if it is inserting just part of the file, I think it won't
work.

To make it work reliably, therefore, I think the
Vset_auto_coding_function function has to look at more of the file.
If it is looking at a po file, it should do that.

This should not be too hard, since it gets the file name as an argument.

Re: po file charset via auto-coding-functions

[Kevin Ryde]

"Richard M. Stallman" <rms@gnu.org> writes:
>
> I think it ALWAYS happens.  Every call to Finsert_file_contents will
> try to determine the coding system from the first 1k and last 3k of
> the file.  (Unless it already knows the coding system to use.)

I tried this (in my build of the cvs starting from -q -no-site-file),

	(debug-on-entry 'sgml-html-meta-auto-coding-function)
	(find-file "etc/NEWS")  ;; the emacs NEWS file

and got

	Debugger entered--entering a function:
	* sgml-html-meta-auto-coding-function(580713)
	  byte-code("\212eb\210^H\211A^P@       !)\207" [funcs size] 2)
	  find-auto-coding("/down/emacs/etc/NEWS" 580713)
	  set-auto-coding("/down/emacs/etc/NEWS" 580713)
	  insert-file-contents("/down/emacs/etc/NEWS" t)

ie. the size presented to the func (in just one call to it) is the
full 580kbytes (and the current-buffer has the full file contents).
So, like I say, I was unsure when this does or doesn't happen.

> To make it work reliably, therefore, I think the
> Vset_auto_coding_function function has to look at more of the file.
> If it is looking at a po file, it should do that.

Looking at more than 1k of the file will also be wanted for Project
Gutenberg texts.  They have about a ~1k blurb at the start before the
header information that has the coding system spec.

Re: po file charset via auto-coding-functions

[Kevin Ryde]

Kenichi Handa <handa@m17n.org> writes:
>
> But, does the current method have any problem that can't be
> fixed within po.el?

If you visit a po file from tar-mode or archive-mode (with "Ret", ie.
tar-extract or archive-extra), the coding system specified in the file
doesn't take effect.  Sometimes other guessing seems to get the right
answer, but eg. "tcvn" doesn't work from a tar where it does work from
a plain file.

(I hope I'm right that auto-coding-functions is the place to get a
coding system from file contents.)

(Incidentally, latexenc-find-file-coding-system looks like it might be
another candidate for this in the future, except for the way it goes
looking for a top-level file.)

Re: po file charset via auto-coding-functions

[Kenichi Handa]

In article <87ll0ma3ow.fsf@zip.com.au>, Kevin Ryde <user42@zip.com.au> writes:

> If you visit a po file from tar-mode or archive-mode (with "Ret", ie.
> tar-extract or archive-extra), the coding system specified in the file
> doesn't take effect.  Sometimes other guessing seems to get the right
> answer, but eg. "tcvn" doesn't work from a tar where it does work from
> a plain file.

It's a bug of tar-mode.  Thank you for noticing it.  I've
just installed a fix.  But, I can't reproduce it in
archive-mode.  Please send me a minimum *.zip file that
reproduces that bug.

> (I hope I'm right that auto-coding-functions is the place to get a
> coding system from file contents.)

Yes.  But, file-coding-system-alist is also a place to
handle it.

And, functions in auto-coding-functions is called on any
file, but currently, po-find-file-coding-system is called
only on *.po file.  I'm not sure which is better, but at
least there's a difference.

---
Kenichi Handa
handa@m17n.org

Re: po file charset via auto-coding-functions

[Kevin Ryde]

[-- Attachment #1: Type: text/plain, Size: 730 bytes --]

Kenichi Handa <handa@m17n.org> writes:
>
> It's a bug of tar-mode.  Thank you for noticing it.  I've
> just installed a fix.  But, I can't reproduce it in
> archive-mode.  Please send me a minimum *.zip file that
> reproduces that bug.

Sample x.tar and x.zip below, containing gtk 1.2 vi.po in tcvn-5712.
Visiting from the .zip gives an error,

	Opening input file: no such file or directory, /tmp/vi.po

Visiting from the .tar gives no error but leaves C-h C saying
`raw-text-unix' and apparent garbage in the buffer.  (I just
cvs-up'ed, not sure if my build is hitting your last change though.)

Be sure not to have a vi.po in the current directory, or the po
functions will open and use that, making it look like they work.



[-- Attachment #2: x.tar --]
[-- Type: application/x-tar, Size: 20480 bytes --]

[-- Attachment #3: x.zip --]
[-- Type: application/zip, Size: 3044 bytes --]

[-- Attachment #4: Type: text/plain, Size: 142 bytes --]

_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-devel

Re: po file charset via auto-coding-functions

[Kenichi Handa]

In article <87fyqu9ung.fsf@zip.com.au>, Kevin Ryde <user42@zip.com.au> writes:

> Be sure not to have a vi.po in the current directory, or the po
> functions will open and use that, making it look like they work.

Ah!  I see.  That's why archive-mode succeeded in my
environment.  Hmmm, it seems that you are right.  There's no
way to handle a tared/archived file in a function registered
in file-coding-system-alist.  So, we surely need your patch
to detect a coding of *.po file in such a case.

Richard, shall I install his patch?

---
Kenichi Handa
handa@m17n.org

Re: po file charset via auto-coding-functions

[Stefan Monnier]

> environment.  Hmmm, it seems that you are right.  There's no
> way to handle a tared/archived file in a function registered
> in file-coding-system-alist.

Provide a file-name-handler for tar files and archives would work
around that problem.


        Stefan

Re: po file charset via auto-coding-functions

[Richard M. Stallman]

    Ah!  I see.  That's why archive-mode succeeded in my
    environment.  Hmmm, it seems that you are right.  There's no
    way to handle a tared/archived file in a function registered
    in file-coding-system-alist.

Could you explain what that means?
I don't entirely understand.

Re: po file charset via auto-coding-functions

[Kevin Ryde]

Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
> Provide a file-name-handler for tar files and archives would work
> around that problem.

But I guess there's also jka-compr and the gnus mime part display
using this stuff (or should be using it).

archive-mode does add a archive-file-name-handler while extracting,
but it seems to only pretend the file exists.  There must be a good
reason for that, but claiming the file exists makes
po-find-file-coding-system-guts try to open it, which gets an error.

Re: po file charset via auto-coding-functions

[Kevin Ryde]

Kenichi Handa <handa@m17n.org> writes:
>
> And, functions in auto-coding-functions is called on any
> file, but currently, po-find-file-coding-system is called
> only on *.po file.  I'm not sure which is better, but at
> least there's a difference.

I guess if the file contents are unambiguous, or near enough so, then
it may actually be a good thing to ignore the filename.

Maybe the auto-coding-functions could be called with the filename too,
so they could restrict themselves if they felt the need.
find-auto-coding has that available for the calls, or document that
`filename' is bound, or whatever.

Re: po file charset via auto-coding-functions

[Kenichi Handa]

In article <87k6g6e05k.fsf-monnier+emacs@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>  environment.  Hmmm, it seems that you are right.  There's no
>>  way to handle a tared/archived file in a function registered
>>  in file-coding-system-alist.

> Provide a file-name-handler for tar files and archives would work
> around that problem.

Maybe, but I'm not sure.  My gut feeling tells that it's not
easy to setup various handlers for an archive member already
setup in a (narrowed) buffer.  We don't know what kind of
file operation a function in file-coding-system-alist
performs.

By the way, while considering the possibility of using
file-name-handler, I got this idea.

The correct operation in a handler for insert-file-contents
will be to find a buffer pretending to visit the file, and
insert that buffer contents.  And, for that, we have to give
buffer-file-name (e.g. /home/handa/x.tgz!vi.po") not the
filename itself (e.g. vi.po) to
find-operation-coding-system.  I think such a change is safe
because, at least, all current entries in
file-coding-system-alist checks only the tail of a filename.

But, if we have such a change, with a fairly simple change
to po.el, we can fix the current problem.  So, I now propose
the attached change.

---
Kenichi Handa
handa@m17n.org


2005-10-24  Kenichi Handa  <handa@m17n.org>

	* arc-mode.el (archive-set-buffer-as-visiting-file): Give
	buffer-file-name to find-operation-coding-system.

	* tar-mode.el (tar-extract): Give buffer-file-name to
	find-operation-coding-system.

	* textmodes/po.el (po-find-charset): If there exists a buffer
	visiting filename, check the contents of that buffer.
	(po-find-file-coding-system-guts): Check if there exists a buffer
	visiting filename.

Index: arc-mode.el
===================================================================
RCS file: /cvsroot/emacs/emacs/lisp/arc-mode.el,v
retrieving revision 1.68
diff -c -r1.68 arc-mode.el
*** arc-mode.el	16 Oct 2005 17:05:23 -0000	1.68
--- arc-mode.el	24 Oct 2005 01:33:13 -0000
***************
*** 877,883 ****
  	       (let ((file-name-handler-alist
  		      '(("" . archive-file-name-handler))))
  		 (car (find-operation-coding-system 'insert-file-contents
! 						    filename t))))))
        (if (and (not coding-system-for-read)
  	       (not enable-multibyte-characters))
  	  (setq coding
--- 877,883 ----
  	       (let ((file-name-handler-alist
  		      '(("" . archive-file-name-handler))))
  		 (car (find-operation-coding-system 'insert-file-contents
! 						    buffer-file-name t))))))
        (if (and (not coding-system-for-read)
  	       (not enable-multibyte-characters))
  	  (setq coding
Index: tar-mode.el
===================================================================
RCS file: /cvsroot/emacs/emacs/lisp/tar-mode.el,v
retrieving revision 1.103
diff -c -r1.103 tar-mode.el
*** tar-mode.el	22 Oct 2005 01:24:38 -0000	1.103
--- tar-mode.el	24 Oct 2005 01:33:14 -0000
***************
*** 737,743 ****
  				    (funcall set-auto-coding-function
  					     name (- (point-max) (point)))))
  			     (car (find-operation-coding-system
! 				   'insert-file-contents name t))))
  			(multibyte enable-multibyte-characters)
  			(detected (detect-coding-region
  				   (point-min)
--- 737,743 ----
  				    (funcall set-auto-coding-function
  					     name (- (point-max) (point)))))
  			     (car (find-operation-coding-system
! 				   'insert-file-contents buffer-file-name t))))
  			(multibyte enable-multibyte-characters)
  			(detected (detect-coding-region
  				   (point-min)
Index: textmodes/po.el
===================================================================
RCS file: /cvsroot/emacs/emacs/lisp/textmodes/po.el,v
retrieving revision 1.12
diff -c -r1.12 po.el
*** textmodes/po.el	6 Aug 2005 17:41:15 -0000	1.12
--- textmodes/po.el	24 Oct 2005 01:33:14 -0000
***************
*** 44,55 ****
    "Return PO charset value for FILENAME."
    (let ((charset-regexp
  	 "^\"Content-Type:[ \t]*text/plain;[ \t]*charset=\\(.*\\)\\\\n\"")
  	(short-read nil))
      ;; Try the first 4096 bytes.  In case we cannot find the charset value
      ;; within the first 4096 bytes (the PO file might start with a long
      ;; comment) try the next 4096 bytes repeatedly until we'll know for sure
      ;; we've checked the empty header entry entirely.
!     (while (not (or short-read (re-search-forward "^msgid" nil t)))
        (save-excursion
          (goto-char (point-max))
  	(let ((pair (insert-file-contents-literally filename nil
--- 44,59 ----
    "Return PO charset value for FILENAME."
    (let ((charset-regexp
  	 "^\"Content-Type:[ \t]*text/plain;[ \t]*charset=\\(.*\\)\\\\n\"")
+ 	(buf (get-file-buffer filename))
  	(short-read nil))
+     (when buf
+       (set-buffer buf)
+       (goto-char (point-min)))
      ;; Try the first 4096 bytes.  In case we cannot find the charset value
      ;; within the first 4096 bytes (the PO file might start with a long
      ;; comment) try the next 4096 bytes repeatedly until we'll know for sure
      ;; we've checked the empty header entry entirely.
!     (while (not (or short-read (re-search-forward "^msgid" nil t) buf))
        (save-excursion
          (goto-char (point-max))
  	(let ((pair (insert-file-contents-literally filename nil
***************
*** 57,63 ****
  						    (1- (+ (point) 4096)))))
  	  (setq short-read (< (nth 1 pair) 4096)))))
      (cond ((re-search-forward charset-regexp nil t) (match-string 1))
! 	  (short-read nil)
  	  ;; We've found the first msgid; maybe, only a part of the msgstr
  	  ;; value was loaded.  Load the next 1024 bytes; if charset still
  	  ;; isn't available, give up.
--- 61,67 ----
  						    (1- (+ (point) 4096)))))
  	  (setq short-read (< (nth 1 pair) 4096)))))
      (cond ((re-search-forward charset-regexp nil t) (match-string 1))
! 	  ((or short-read buf) nil)
  	  ;; We've found the first msgid; maybe, only a part of the msgstr
  	  ;; value was loaded.  Load the next 1024 bytes; if charset still
  	  ;; isn't available, give up.
***************
*** 74,80 ****
  Do so according to FILENAME's declared charset."
    (and
     (eq operation 'insert-file-contents)
!    (file-exists-p filename)
     (with-temp-buffer
       (let* ((coding-system-for-read 'no-conversion)
  	    (charset (or (po-find-charset filename) "ascii"))
--- 78,84 ----
  Do so according to FILENAME's declared charset."
    (and
     (eq operation 'insert-file-contents)
!    (or (get-file-buffer filename) (file-exists-p filename))
     (with-temp-buffer
       (let* ((coding-system-for-read 'no-conversion)
  	    (charset (or (po-find-charset filename) "ascii"))

Re: po file charset via auto-coding-functions

[Kenichi Handa]

In article <87zmp1f9np.fsf@zip.com.au>, Kevin Ryde <user42@zip.com.au> writes:

> Kenichi Handa <handa@m17n.org> writes:
>> 
>>  And, functions in auto-coding-functions is called on any
>>  file, but currently, po-find-file-coding-system is called
>>  only on *.po file.  I'm not sure which is better, but at
>>  least there's a difference.

> I guess if the file contents are unambiguous, or near enough so, then
> it may actually be a good thing to ignore the filename.

In general, I agree.  But, for that, a function to detect a
coding system has to check if the contents is surely in an
expected format in advance.  For instance, in the case of
po-find-file-coding-system, it has to check if the file is
surely PO file.  But, it seems that the current po.el
doesn't do that but expects that it is called only on PO
file.

> Maybe the auto-coding-functions could be called with the filename too,
> so they could restrict themselves if they felt the need.
> find-auto-coding has that available for the calls, or document that
> `filename' is bound, or whatever.

Yes, if a function in auto-coding-functions can check a
filename too, the above problem disappears (or at least
Emacs can perform the same operation as now).  But, I don't
want to change an API at this moment, especially when we can
fix the problem by the other method (as in my previous
mail).

---
Kenichi Handa
handa@m17n.org

Re: po file charset via auto-coding-functions

[Kevin Ryde]

Kenichi Handa <handa@m17n.org> writes:
>
> But, it seems that the current po.el doesn't do that but expects
> that it is called only on PO file.

Yes, that's so.  That's why I tightened it up with a looking-at to
demand a msgid "" + msgstr "", then the "Content-Type" in that header
block.  It's always possible a non-po file could start that way, but
it should be unlikely.

> especially when we can fix the problem by the other method (as in my
> previous mail).

I think jka-compr is also affected, try visiting a compressed
vi.po.gz.

Re: po file charset via auto-coding-functions

[Kenichi Handa]

In article <E1ETLeE-0006XR-KE@fencepost.gnu.org>, "Richard M. Stallman" <rms@gnu.org> writes:

>     Ah!  I see.  That's why archive-mode succeeded in my
>     environment.  Hmmm, it seems that you are right.  There's no
>     way to handle a tared/archived file in a function registered
>     in file-coding-system-alist.

> Could you explain what that means?
> I don't entirely understand.

Let's assume that a tar file x.tar contains a file vi.po.

The current problem is that on extracting vi.po in tar-mode,
a function registered in file-coding-system-alist is called
with a filename "vi.po".  So, that function can't check the
contents of that file because it can't read the contents by
insert-file-contents.  But, po-find-file-coding-system have
to read the contents to determine a coding system.

The original suggestion by Kevin is to move
po-find-file-coding-system from file-coding-system-alist to
auto-coding-functions because, then,
po-find-file-coding-system can check the file contents even
for an achive file member.

Then, Stefan suggested to provide a proper file-name-handler
in tar-mode and arc-mode so that insert-file-contents should
correctly work even for an arhive file member.

But, I thought it's not easy, and proposed the other simpler
method in the last mail.

---
Kenichi Handa
handa@m17n.org

Re: po file charset via auto-coding-functions

[Kenichi Handa]

In article <87mzkz3c23.fsf@zip.com.au>, Kevin Ryde <user42@zip.com.au> writes:

> Kenichi Handa <handa@m17n.org> writes:
>> 
>>  But, it seems that the current po.el doesn't do that but expects
>>  that it is called only on PO file.

> Yes, that's so.  That's why I tightened it up with a looking-at to
> demand a msgid "" + msgstr "", then the "Content-Type" in that header
> block.  It's always possible a non-po file could start that way, but
> it should be unlikely.

I see.

>>  especially when we can fix the problem by the other method (as in my
>>  previous mail).

> I think jka-compr is also affected, try visiting a compressed
> vi.po.gz.

It seems that this additional patch will fix it.  But I
admit that that this change is rather tricky compared with
the previous ones.  :-(

*** jka-compr.el	08 Aug 2005 10:13:24 +0900	1.87
--- jka-compr.el	24 Oct 2005 11:38:27 +0900	
***************
*** 500,509 ****
  	     (delete-file local-copy)))
  
  	  (unless notfound
  	    (decode-coding-inserted-region
  	     (point) (+ (point) size)
! 	     (jka-compr-byte-compiler-base-file-name file)
! 	     visit beg end replace))
  
  	  (and
  	   visit
--- 500,513 ----
  	     (delete-file local-copy)))
  
  	  (unless notfound
+ 	    (let ((buffer-file-name
+ 		   (concat file "!"
+ 			   (jka-compr-byte-compiler-base-file-name file))))
+ 
  	    (decode-coding-inserted-region
  	     (point) (+ (point) size)
! 	     buffer-file-name
! 	     visit beg end replace)))
  
  	  (and
  	   visit

---
Kenichi Handa
handa@m17n.org

Re: po file charset via auto-coding-functions

[Stefan Monnier]

>>> especially when we can fix the problem by the other method (as in my
>>> previous mail).

Passing the full file name seems like the right thing to do, indeed.

> It seems that this additional patch will fix it.  But I
> admit that that this change is rather tricky compared with
> the previous ones.  :-(

I don't see what's the trick.  So I guess it deserves a comment.


        Stefan

Re: po file charset via auto-coding-functions

[Juri Linkov]

>> I think jka-compr is also affected, try visiting a compressed
>> vi.po.gz.
>
> It seems that this additional patch will fix it.  But I
> admit that that this change is rather tricky compared with
> the previous ones.  :-(

Please also pay attention to the related problem: .tar and .zip don't
support some Emacs features.  For example, after (require 'generic-x)
visiting vi.po or vi.po.gz sets Default-Generic mode, but visiting the
vi.po file from x.tar or x.zip doesn't set this mode.  Or after
(progn (setq-default save-place t) (require 'saveplace)) visiting
vi.po or vi.po.gz remembers point positions, but visiting them
from x.tar or x.zip doesn't.

These features work by putting the hook in `find-file-hook'.
Calling this hook on a visited archive file seems work for these
features, but is it the right way to do?

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: po file charset via auto-coding-functions

[Kenichi Handa]

In article <87k6g3atv0.fsf-monnier+emacs@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>>>  especially when we can fix the problem by the other method (as in my
>>>>  previous mail).

> Passing the full file name seems like the right thing to do, indeed.

What do you mean by "full file name"?  Something like
"/home/handa/x.tar!vi.po" which my change passes?

>>  It seems that this additional patch will fix it.  But I
>>  admit that that this change is rather tricky compared with
>>  the previous ones.  :-(

> I don't see what's the trick.  So I guess it deserves a comment.

It temporarily binds buffer-file-name to
"/home/handa/vi.po.gz!/home/handa/vi.po" and passes that
name so that a function can find the current buffer by
get-file-buffer.  I think such a way of using
buffer-file-name is rather tricky.  Anyway, yes, I'll add
more comments if my change is approved.

---
Kenichi Handa
handa@m17n.org

Re: po file charset via auto-coding-functions

[Kenichi Handa]

In article <8764rmqyhx.fsf@jurta.org>, Juri Linkov <juri@jurta.org> writes:

> Please also pay attention to the related problem: .tar and .zip don't
> support some Emacs features.  For example, after (require 'generic-x)
> visiting vi.po or vi.po.gz sets Default-Generic mode, but visiting the
> vi.po file from x.tar or x.zip doesn't set this mode.  Or after
> (progn (setq-default save-place t) (require 'saveplace)) visiting
> vi.po or vi.po.gz remembers point positions, but visiting them
> from x.tar or x.zip doesn't.

> These features work by putting the hook in `find-file-hook'.
> Calling this hook on a visited archive file seems work for these
> features, but is it the right way to do?

I have no idea.  As I've never used those mode, to work on
it, I must start from investigating how they work in a
normal case.  I'd like to ask someone else who already knows
those code to work on it.  At least, it seems that it
doesn't require "mule" knowledge.

---
Kenichi Handa
handa@m17n.org

Re: po file charset via auto-coding-functions

[Richard M. Stallman]

    The current problem is that on extracting vi.po in tar-mode,
    a function registered in file-coding-system-alist is called
    with a filename "vi.po".  So, that function can't check the
    contents of that file because it can't read the contents by
    insert-file-contents.

I see.

    The correct operation in a handler for insert-file-contents
    will be to find a buffer pretending to visit the file, and
    insert that buffer contents.  And, for that, we have to give
    buffer-file-name (e.g. /home/handa/x.tgz!vi.po") not the
    filename itself (e.g. vi.po) to
    find-operation-coding-system.  I think such a change is safe
    because, at least, all current entries in
    file-coding-system-alist checks only the tail of a filename.

If this means passing "/home/handa/x.tgz!vi.po" as the second argument
to find-operation-coding-system, I think it is not clean.  That
argument is documented to be _the same_ in meaning as the argument you
would pass to OPERATION, but this string isn't the same, isn't a valid
argument to insert-file-contents.

If we want to modify the calling convention of
find-operation-coding-system, we should do it in a cleaner way: by
binding a global variable.  That variable would be nil, normally.

I think the variable's value should be a buffer name.
If the variable is non-nil, it means "the file's contents
are already in this buffer".  And all the functions that handle
find-operation-coding-system for various file types should check
the variable.

I think that is a fairly clean solution.
What do you think?

Re: po file charset via auto-coding-functions

[Richard M. Stallman]

    These features work by putting the hook in `find-file-hook'.
    Calling this hook on a visited archive file seems work for these
    features, but is it the right way to do?

In theory it sounds right.  It will probably fix more bugs than it
causes.  So let's make that change.

Re: po file charset via auto-coding-functions

[Richard Stallman]

[I sent this message a week ago but did not get a response.
Could we get the discussion moving again?]

    The current problem is that on extracting vi.po in tar-mode,
    a function registered in file-coding-system-alist is called
    with a filename "vi.po".  So, that function can't check the
    contents of that file because it can't read the contents by
    insert-file-contents.

I see.

    The correct operation in a handler for insert-file-contents
    will be to find a buffer pretending to visit the file, and
    insert that buffer contents.  And, for that, we have to give
    buffer-file-name (e.g. /home/handa/x.tgz!vi.po") not the
    filename itself (e.g. vi.po) to
    find-operation-coding-system.  I think such a change is safe
    because, at least, all current entries in
    file-coding-system-alist checks only the tail of a filename.

If this means passing "/home/handa/x.tgz!vi.po" as the second argument
to find-operation-coding-system, I think it is not clean.  That
argument is documented to be _the same_ in meaning as the argument you
would pass to OPERATION, but this string isn't the same, isn't a valid
argument to insert-file-contents.

If we want to modify the calling convention of
find-operation-coding-system, we should do it in a cleaner way: by
binding a global variable.  That variable would be nil, normally.

I think the variable's value should be a buffer name.
If the variable is non-nil, it means "the file's contents
are already in this buffer".  And all the functions that handle
find-operation-coding-system for various file types should check
the variable.

I think that is a fairly clean solution.
What do you think?

Re: po file charset via auto-coding-functions

[Richard Stallman]

[I sent this message twice but did not get a response.
Could we get the discussion moving again?]

    The current problem is that on extracting vi.po in tar-mode,
    a function registered in file-coding-system-alist is called
    with a filename "vi.po".  So, that function can't check the
    contents of that file because it can't read the contents by
    insert-file-contents.

I see.

    The correct operation in a handler for insert-file-contents
    will be to find a buffer pretending to visit the file, and
    insert that buffer contents.  And, for that, we have to give
    buffer-file-name (e.g. /home/handa/x.tgz!vi.po") not the
    filename itself (e.g. vi.po) to
    find-operation-coding-system.  I think such a change is safe
    because, at least, all current entries in
    file-coding-system-alist checks only the tail of a filename.

If this means passing "/home/handa/x.tgz!vi.po" as the second argument
to find-operation-coding-system, I think it is not clean.  That
argument is documented to be _the same_ in meaning as the argument you
would pass to OPERATION, but this string isn't the same, isn't a valid
argument to insert-file-contents.

If we want to modify the calling convention of
find-operation-coding-system, we should do it in a cleaner way: by
binding a global variable.  That variable would be nil, normally.

I think the variable's value should be a buffer name.
If the variable is non-nil, it means "the file's contents
are already in this buffer".  And all the functions that handle
find-operation-coding-system for various file types should check
the variable.

I think that is a fairly clean solution.
What do you think?

Re: po file charset via auto-coding-functions

[Stefan Monnier]

> If this means passing "/home/handa/x.tgz!vi.po" as the second argument
> to find-operation-coding-system, I think it is not clean.  That
> argument is documented to be _the same_ in meaning as the argument you
> would pass to OPERATION, but this string isn't the same, isn't a valid
> argument to insert-file-contents.

IIUC currently the argument passed is "vi.po" but that is not the argument
passed to operation either.  So passing "/home/handa/x.tgz!vi.po" instead
won't make it any worse.  As a matter of fact I think passing
"/home/handa/x.tgz!vi.po" is more correct: if we assume that
a suitable file-name-handler is installed it's perfectly correct.


        Stefan

Re: po file charset via auto-coding-functions

[Richard M. Stallman]

    IIUC currently the argument passed is "vi.po" but that is not the argument
    passed to operation either.

Perhaps we should fix that.

    As a matter of fact I think passing
    "/home/handa/x.tgz!vi.po" is more correct: if we assume that
    a suitable file-name-handler is installed it's perfectly correct.

But I don't think we should install such a handler, because it would
be an incompatible change in file name syntax.  And if we don't
install such a handler, the argument isn't correct at all.

Re: po file charset via auto-coding-functions

[Stefan Monnier]

>     IIUC currently the argument passed is "vi.po" but that is not the argument
>     passed to operation either.

> Perhaps we should fix that.

>     As a matter of fact I think passing
>     "/home/handa/x.tgz!vi.po" is more correct: if we assume that
>     a suitable file-name-handler is installed it's perfectly correct.

> But I don't think we should install such a handler, because it would
> be an incompatible change in file name syntax.

Then maybe another syntax should be used, but currently that's the syntax
chosen: this is the value used for buffer-file-name.  In any case even if
you don't like this syntax, it's not "incompatible" since all it takes is to
make sure the file-name-handler makes expand-file-name,
file-name-nondirectory, and friends work correctly.

Myself, I don't care much about which particular syntax is used but
simply that "/full/pseudoname/to/vi.po" is no worse than "vi.po".

> And if we don't install such a handler, the argument isn't correct at all.

But it currently isn't correct either.  We're just passing more info.


        Stefan

Re: po file charset via auto-coding-functions

[Richard M. Stallman]

    >     As a matter of fact I think passing
    >     "/home/handa/x.tgz!vi.po" is more correct: if we assume that
    >     a suitable file-name-handler is installed it's perfectly correct.

    > But I don't think we should install such a handler, because it would
    > be an incompatible change in file name syntax.

    Then maybe another syntax should be used, but currently that's the syntax
    chosen: this is the value used for buffer-file-name.

Does that file name get used for anything except to appear in the C_x
C-b listing and be helpful for the user?  I think it does not.

The previous message said:

	The correct operation in a handler for insert-file-contents
	will be to find a buffer pretending to visit the file, and
	insert that buffer contents.  And, for that, we have to give
	buffer-file-name (e.g. /home/handa/x.tgz!vi.po") not the
	filename itself (e.g. vi.po) to
	find-operation-coding-system.

Can you explain the context of this?  Where is insert-file-contents
being called from, and with what file name argument?
How does it relate to this issue?

If it is simply a matter to call find-operation-coding-system here,
in tar-extract, then I agree it is ok to pass buffer-file-name.

		  ;; We need to mimic the parts of insert-file-contents
		  ;; which determine the coding-system and decode the text.
		  (let ((coding
			 (or coding-system-for-read
			     (and set-auto-coding-function
				  (save-excursion
				    (funcall set-auto-coding-function
					     name (- (point-max) (point)))))
			     (car (find-operation-coding-system
				   'insert-file-contents name t))))

But since that isn't a true valid file name, it needs a comment
to explain why calling find-operation-coding-system in a way 
contrary to its rules nonetheless gives correct results.
And maybe we need to document this also in the place that defines
find-operation-coding-system and the functions it calls, so that people
will not change them in a way that breaks this code.

Re: po file charset via auto-coding-functions

[Kenichi Handa]

Sorry for the late response on this matter.  I was just back
from Hanoi; Vietnamese foods were very good.  :-)

In article <E1EaTYC-0001cx-Rn@fencepost.gnu.org>, "Richard M. Stallman" <rms@gnu.org> writes:

>>      As a matter of fact I think passing
>>      "/home/handa/x.tgz!vi.po" is more correct: if we assume that
>>      a suitable file-name-handler is installed it's perfectly correct.

>>  But I don't think we should install such a handler, because it would
>>  be an incompatible change in file name syntax.

>     Then maybe another syntax should be used, but currently that's the syntax
>     chosen: this is the value used for buffer-file-name.

> Does that file name get used for anything except to appear in the C_x
> C-b listing and be helpful for the user?  I think it does not.

I don't know exactly which command uses it, but it is used
by any operations that call get-file-buffer.

> The previous message said:

> 	The correct operation in a handler for insert-file-contents
> 	will be to find a buffer pretending to visit the file, and
> 	insert that buffer contents.  And, for that, we have to give
> 	buffer-file-name (e.g. /home/handa/x.tgz!vi.po") not the
> 	filename itself (e.g. vi.po) to
> 	find-operation-coding-system.

> Can you explain the context of this?  Where is insert-file-contents
> being called from, and with what file name argument?
> How does it relate to this issue?

Here the "handler" means a function registered in
file-coding-system-alist (it's po-find-file-coding-system in
the current case).

> If it is simply a matter to call find-operation-coding-system here,
> in tar-extract, then I agree it is ok to pass buffer-file-name.

Yes, that is what the change I propsed for an archived file
does.

And, for a compressed file, I proposed this change.

*** jka-compr.el	08 Aug 2005 10:13:24 +0900	1.87
--- jka-compr.el	24 Oct 2005 11:38:27 +0900	
***************
*** 500,509 ****
  	     (delete-file local-copy)))
  
  	  (unless notfound
  	    (decode-coding-inserted-region
  	     (point) (+ (point) size)
! 	     (jka-compr-byte-compiler-base-file-name file)
! 	     visit beg end replace))
  
  	  (and
  	   visit
--- 500,513 ----
  	     (delete-file local-copy)))
  
  	  (unless notfound
+ 	    (let ((buffer-file-name
+ 		   (concat file "!"
+ 			   (jka-compr-byte-compiler-base-file-name file))))
+ 
  	    (decode-coding-inserted-region
  	     (point) (+ (point) size)
! 	     buffer-file-name
! 	     visit beg end replace)))
  
  	  (and
  	   visit

As you see, this binds buffer-file-name temporarily to
something like /home/handa/temp.po.gz!/home/handa/temp.po so
that find-operation-coding-system (called in
decode-coding-inserted-region) can surely find
po-find-file-coding-system to be called, and it can surely
find the current buffer by get-file-buffer.

Do you agree with this change too (of course provided that
more comments are added)?

---
Kenichi Handa
handa@m17n.org

Re: po file charset via auto-coding-functions

[Stefan Monnier]

> As you see, this binds buffer-file-name temporarily to
> something like /home/handa/temp.po.gz!/home/handa/temp.po so
> that find-operation-coding-system (called in
> decode-coding-inserted-region) can surely find
> po-find-file-coding-system to be called, and it can surely
> find the current buffer by get-file-buffer.

Hmm... that's not what I had understood.  The above looks fishy.
Why not pass /home/handa/temp.po to find-operation-coding-system (just as
it is used for auto-mode-alist)?


        Stefan

Re: po file charset via auto-coding-functions

[Kenichi Handa]

In article <87sltt98l6.fsf-monnier+emacs@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>  As you see, this binds buffer-file-name temporarily to
>>  something like /home/handa/temp.po.gz!/home/handa/temp.po so
>>  that find-operation-coding-system (called in
>>  decode-coding-inserted-region) can surely find
>>  po-find-file-coding-system to be called, and it can surely
>>  find the current buffer by get-file-buffer.

> Hmm... that's not what I had understood.  The above looks fishy.

I agree, so I wrote it's more tricky.

> Why not pass /home/handa/temp.po to find-operation-coding-system (just as
> it is used for auto-mode-alist)?

Very unlikely but a user may be visiting that file in a
different buffer.

---
Kenichi Handa
handa@m17n.org

Re: po file charset via auto-coding-functions

[Richard M. Stallman]

    I don't know exactly which command uses it, but it is used
    by any operations that call get-file-buffer.

Most callers of get-file-buffer pass a real file name.  So unless
someone uses a file whose name resembles that of a tar subfile,
get-file-buffer will never return one of these buffers, and I think
that is the correct result.

Thus, looking at my question:

    > Does that file name get used for anything except to appear in the C_x
    > C-b listing and be helpful for the user?  I think it does not.

I think I was right--it is not used for anything else, or at least,
not unless a problem is occurring.

Therefore, we do not want to install any file name handler for this syntax.

    > If it is simply a matter to call find-operation-coding-system here,
    > in tar-extract, then I agree it is ok to pass buffer-file-name.

    Yes, that is what the change I propsed for an archived file
    does.

Ok, please make that change.

    + 	    (let ((buffer-file-name
    + 		   (concat file "!"
    + 			   (jka-compr-byte-compiler-base-file-name file))))
    + 

Binding buffer-file-name is rather unclean.
And I don't see a reason to do it.

    so that find-operation-coding-system (called in
    decode-coding-inserted-region) can surely find
    po-find-file-coding-system to be called, and it can surely
    find the current buffer by get-file-buffer.

decode-coding-inserted-region passes its FILENAME arg to
find-operation-coding-system.  So all you need to do is to pass
this funny file name to decode-coding-inserted-region.
There is no need to bind buffer-file-name.

Why are you concerned about whether get-file-buffer can be used with
this funny file name?

Re: po file charset via auto-coding-functions

[Juri Linkov]

>> As you see, this binds buffer-file-name temporarily to
>> something like /home/handa/temp.po.gz!/home/handa/temp.po so
>> that find-operation-coding-system (called in
>> decode-coding-inserted-region) can surely find
>> po-find-file-coding-system to be called, and it can surely
>> find the current buffer by get-file-buffer.
>
> Hmm... that's not what I had understood.  The above looks fishy.
> Why not pass /home/handa/temp.po to find-operation-coding-system (just as
> it is used for auto-mode-alist)?

Wouldn't it be better to use the same syntax for members of
all supported archivers (gzip, zip, tar)?

Currently `buffer-file-name' of the visited archive member
uses three different syntaxes:

"/home/handa/temp.po.gz"
"/home/handa/temp.zip:vi.po"
"/home/handa/temp.tar!vi.po"

After choosing one syntax, `buffer-file-name' would be:

"/home/handa/temp.po.gz!temp.po"
"/home/handa/temp.zip!vi.po"
"/home/handa/temp.tar!vi.po"

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: po file charset via auto-coding-functions

[Kenichi Handa]

In article <E1Edc6s-0002BI-Ax@fencepost.gnu.org>, "Richard M. Stallman" <rms@gnu.org> writes:

> Binding buffer-file-name is rather unclean.
> And I don't see a reason to do it.

>     so that find-operation-coding-system (called in
>     decode-coding-inserted-region) can surely find
>     po-find-file-coding-system to be called, and it can surely
>     find the current buffer by get-file-buffer.

> decode-coding-inserted-region passes its FILENAME arg to
> find-operation-coding-system.  So all you need to do is to pass
> this funny file name to decode-coding-inserted-region.
> There is no need to bind buffer-file-name.

> Why are you concerned about whether get-file-buffer can be used with
> this funny file name?

A function registered in find-operation-coding-system have
to find which buffer is pretending to visit FILENAME if
FILENAME doesn't exist.  And, get-file-buffer is the only
(or at least the very natural) way for that.

---
Kenichi Handa
handa@m17n.org

Re: po file charset via auto-coding-functions

[Kevin Rodgers]

Juri Linkov wrote:
> Wouldn't it be better to use the same syntax for members of
> all supported archivers (gzip, zip, tar)?
> 
> Currently `buffer-file-name' of the visited archive member
> uses three different syntaxes:
> 
> "/home/handa/temp.po.gz"
> "/home/handa/temp.zip:vi.po"
> "/home/handa/temp.tar!vi.po"
> 
> After choosing one syntax, `buffer-file-name' would be:
> 
> "/home/handa/temp.po.gz!temp.po"
> "/home/handa/temp.zip!vi.po"
> "/home/handa/temp.tar!vi.po"

Or how about a URL-compatible syntax:

arc://home/handa/temp.po.gz#temp.po
arc://home/handa/temp.gip#vi.po
arc://home/handa/temp.tar#vi.po

-- 
Kevin Rodgers

Re: po file charset via auto-coding-functions

[Juri Linkov]

>> Wouldn't it be better to use the same syntax for members of
>> all supported archivers (gzip, zip, tar)?
>> Currently `buffer-file-name' of the visited archive member
>> uses three different syntaxes:
>> "/home/handa/temp.po.gz"
>> "/home/handa/temp.zip:vi.po"
>> "/home/handa/temp.tar!vi.po"
>> After choosing one syntax, `buffer-file-name' would be:
>> "/home/handa/temp.po.gz!temp.po"
>> "/home/handa/temp.zip!vi.po"
>> "/home/handa/temp.tar!vi.po"
>
> Or how about a URL-compatible syntax:
>
> arc://home/handa/temp.po.gz#temp.po
> arc://home/handa/temp.gip#vi.po
> arc://home/handa/temp.tar#vi.po

There is no such standard URI scheme as `arc'.  But perhaps `#' in
`file' is appropriate to address archive members:

file://home/handa/temp.po.gz#temp.po
file://home/handa/temp.gip#vi.po
file://home/handa/temp.tar#vi.po

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: po file charset via auto-coding-functions

[Richard M. Stallman]

    Or how about a URL-compatible syntax:

    arc://home/handa/temp.po.gz#temp.po
    arc://home/handa/temp.gip#vi.po
    arc://home/handa/temp.tar#vi.po

URLs are not file names.  No thanks.

Re: po file charset via auto-coding-functions

[Richard M. Stallman]

Please forgive the delay in my response.

    > Binding buffer-file-name is rather unclean.
    > And I don't see a reason to do it.

    >     so that find-operation-coding-system (called in
    >     decode-coding-inserted-region) can surely find
    >     po-find-file-coding-system to be called, and it can surely
    >     find the current buffer by get-file-buffer.

    > decode-coding-inserted-region passes its FILENAME arg to
    > find-operation-coding-system.  So all you need to do is to pass
    > this funny file name to decode-coding-inserted-region.
    > There is no need to bind buffer-file-name.

    > Why are you concerned about whether get-file-buffer can be used with
    > this funny file name?

    A function registered in find-operation-coding-system have
    to find which buffer is pretending to visit FILENAME if
    FILENAME doesn't exist.  And, get-file-buffer is the only
    (or at least the very natural) way for that.

Could you give me an example or two?
Which filename handler function does this?
Looking at an example, I can understand the issue.

Re: po file charset via auto-coding-functions

[Kenichi Handa]

In article <E1Ereg8-0000Fq-AH@fencepost.gnu.org>, "Richard M. Stallman" <rms@gnu.org> writes:

> Please forgive the delay in my response.
>>  Binding buffer-file-name is rather unclean.
>>  And I don't see a reason to do it.

>>      so that find-operation-coding-system (called in
>>      decode-coding-inserted-region) can surely find
>>      po-find-file-coding-system to be called, and it can surely
>>      find the current buffer by get-file-buffer.

>>  decode-coding-inserted-region passes its FILENAME arg to
>>  find-operation-coding-system.  So all you need to do is to pass
>>  this funny file name to decode-coding-inserted-region.
>>  There is no need to bind buffer-file-name.

>>  Why are you concerned about whether get-file-buffer can be used with
>>  this funny file name?

>     A function registered in find-operation-coding-system have
>     to find which buffer is pretending to visit FILENAME if
>     FILENAME doesn't exist.  And, get-file-buffer is the only
>     (or at least the very natural) way for that.

> Could you give me an example or two?
> Which filename handler function does this?
> Looking at an example, I can understand the issue.

The attached in the patch for po.el I posted.   This change
utilizes get-file-buffer to check if there is a buffer
visiting (or pretending to visit) FILENAME.

---
Kenichi Handa
handa@m17n.org

*** po.el	08 Aug 2005 10:13:42 +0900	1.12
--- po.el	18 Nov 2005 21:08:50 +0900	
***************
*** 44,55 ****
    "Return PO charset value for FILENAME."
    (let ((charset-regexp
  	 "^\"Content-Type:[ \t]*text/plain;[ \t]*charset=\\(.*\\)\\\\n\"")
  	(short-read nil))
      ;; Try the first 4096 bytes.  In case we cannot find the charset value
      ;; within the first 4096 bytes (the PO file might start with a long
      ;; comment) try the next 4096 bytes repeatedly until we'll know for sure
      ;; we've checked the empty header entry entirely.
!     (while (not (or short-read (re-search-forward "^msgid" nil t)))
        (save-excursion
          (goto-char (point-max))
  	(let ((pair (insert-file-contents-literally filename nil
--- 44,59 ----
    "Return PO charset value for FILENAME."
    (let ((charset-regexp
  	 "^\"Content-Type:[ \t]*text/plain;[ \t]*charset=\\(.*\\)\\\\n\"")
+ 	(buf (get-file-buffer filename))
  	(short-read nil))
+     (when buf
+       (set-buffer buf)
+       (goto-char (point-min)))
      ;; Try the first 4096 bytes.  In case we cannot find the charset value
      ;; within the first 4096 bytes (the PO file might start with a long
      ;; comment) try the next 4096 bytes repeatedly until we'll know for sure
      ;; we've checked the empty header entry entirely.
!     (while (not (or short-read (re-search-forward "^msgid" nil t) buf))
        (save-excursion
          (goto-char (point-max))
  	(let ((pair (insert-file-contents-literally filename nil
***************
*** 57,63 ****
  						    (1- (+ (point) 4096)))))
  	  (setq short-read (< (nth 1 pair) 4096)))))
      (cond ((re-search-forward charset-regexp nil t) (match-string 1))
! 	  (short-read nil)
  	  ;; We've found the first msgid; maybe, only a part of the msgstr
  	  ;; value was loaded.  Load the next 1024 bytes; if charset still
  	  ;; isn't available, give up.
--- 61,67 ----
  						    (1- (+ (point) 4096)))))
  	  (setq short-read (< (nth 1 pair) 4096)))))
      (cond ((re-search-forward charset-regexp nil t) (match-string 1))
! 	  ((or short-read buf) nil)
  	  ;; We've found the first msgid; maybe, only a part of the msgstr
  	  ;; value was loaded.  Load the next 1024 bytes; if charset still
  	  ;; isn't available, give up.
***************
*** 74,80 ****
  Do so according to FILENAME's declared charset."
    (and
     (eq operation 'insert-file-contents)
!    (file-exists-p filename)
     (with-temp-buffer
       (let* ((coding-system-for-read 'no-conversion)
  	    (charset (or (po-find-charset filename) "ascii"))
--- 78,84 ----
  Do so according to FILENAME's declared charset."
    (and
     (eq operation 'insert-file-contents)
!    (or (get-file-buffer filename) (file-exists-p filename))
     (with-temp-buffer
       (let* ((coding-system-for-read 'no-conversion)
  	    (charset (or (po-find-charset filename) "ascii"))

Re: po file charset via auto-coding-functions

[Richard M. Stallman]

    > Could you give me an example or two?
    > Which filename handler function does this?
    > Looking at an example, I can understand the issue.

We are failing to communicate.  Your patch is a change in
po-find-charset.  po-find-charset is not a filename handler function.

You wrote:

    >     A function registered in find-operation-coding-system have
    >     to find which buffer is pretending to visit FILENAME if
    >     FILENAME doesn't exist.  And, get-file-buffer is the only
    >     (or at least the very natural) way for that.

What's the name of one function registered in find-operation-coding-system
which needs to do this?

Does that function call po-find-charset?  If so, could you show me
the sequence of function calls that gets from there to po-find-charset?

Re: po file charset via auto-coding-functions

[Kenichi Handa]

In article <E1Es9qt-000650-Ko@fencepost.gnu.org>, "Richard M. Stallman" <rms@gnu.org> writes:

>>  Could you give me an example or two?
>>  Which filename handler function does this?
>>  Looking at an example, I can understand the issue.

> We are failing to communicate.  Your patch is a change in
> po-find-charset.  po-find-charset is not a filename handler function.

The patch contains also a change in
po-find-file-coding-system-guts which is called from
po-find-file-coding-system.  And po-find-file-coding-system
is a function registered in file-coding-system-alist and
thus is called from find-operation-coding-system.  I thought
you meant such a function by "filename handler function" in
the current context.

---
Kenichi Handa
handa@m17n.org