ja-dic.el and SKK-JYSYO.L

ja-dic.el and SKK-JYSYO.L

[Ivan Kanis]

Hello,

SKK-JYSYO.L is out of date, I am trying to update it.

I can't figure out how ja-dic.el is generated from SKK-JYSYO.L. In the
commentary of ja-dic.el is says it has been generated by
skkdic-convert. I can't find this function or program
anywhere. Searching the web didn't help...

I don't think SKK-JYSYO.L needs to be included in the source code, it's
2.7M and doesn't need to be there if ja-dic.el exists. It would make the
tar ball a bit smaller.
-- 
Ivan Kanis
http://kanis.fr

There is a land of the living and a land of the dead and the bridge is
love, the only survival, the only meaning.
    -- Thornton Wilder 

Re: ja-dic.el and SKK-JYSYO.L

[Eli Zaretskii]

> From: Ivan Kanis <expire-by-2010-02-17@kanis.fr>
> Date: Fri, 12 Feb 2010 10:47:16 +0100
> 
> SKK-JYSYO.L is out of date, I am trying to update it.
> 
> I can't figure out how ja-dic.el is generated from SKK-JYSYO.L. In the
> commentary of ja-dic.el is says it has been generated by
> skkdic-convert. I can't find this function or program
> anywhere. Searching the web didn't help...

The function skkdic-convert is in lisp/international/ja-dic-cnv.el, in
the Emacs source tree.

Re: ja-dic.el and SKK-JYSYO.L

[Kenichi Handa]

In article <m3bpfudbvf.fsf@patsy.makina-nantes.net>, Ivan Kanis <expire-by-2010-02-17@kanis.fr> writes:

> SKK-JYSYO.L is out of date, I am trying to update it.

> I can't figure out how ja-dic.el is generated from SKK-JYSYO.L. In the
> commentary of ja-dic.el is says it has been generated by
> skkdic-convert. I can't find this function or program
> anywhere. Searching the web didn't help...

That function is in lisp/international/ja-dic-cnv.el.  When
you put a new SKK-JISYO.L (not SKK-JYSHO.L) under
leim/SKK-DIC, and run make under the leim directory, new
ja-dic.el and ja-dic.elc should be created.  Have you tried
it?

> I don't think SKK-JYSYO.L needs to be included in the source code, it's
> 2.7M and doesn't need to be there if ja-dic.el exists. It would make the
> tar ball a bit smaller.

When we discussed that matter long ago, we decided to
include all source data of input methods because the
compressed sizes are not that huge compared with the whole
tarball of Emacs.  Is the new SKK-JISYO.L much bigger than
the current one?

Perhaps we should consider moving SKK-JISYO.L (and the other
big files) to `admin' directory which is not included in the
tarball.

---
Kenichi Handa
handa@m17n.org

Re: ja-dic.el and SKK-JYSYO.L

[Ivan Kanis]

Kenichi Handa <handa@m17n.org> wrote:

> That function is in lisp/international/ja-dic-cnv.el.  When

I found it 10mn after sending my e-mail, I felt silly...

> you put a new SKK-JISYO.L (not SKK-JYSHO.L) under
> leim/SKK-DIC, and run make under the leim directory, new
> ja-dic.el and ja-dic.elc should be created.  Have you tried
> it?

I put the new SKK-JYSHO.L in my tmp directory and run the command
skkdic-convert on the dictionary. It created ja-dic.el in my home
directory.

However when I eval ja-dic.el I get an error:

Debugger entered--Lisp error: (args-out-of-range ">\x308fk \x5206" 6)
  string-match("[^ ]+" ">\x308fk \x5206" 6)
  (while (string-match "[^ ]+" entry i) (setq candidates (cons ... candidates)) (setq i (match-end 0)))
  (let ((kana ...) (i ...) candidates) (while (string-match "[^ ]+" entry i) (setq candidates ...) (setq i ...)) (cons (skkdic-get-kana-compact-codes kana) candidates))
  skkdic-extract-conversion-data(">\x308fk \x5206")

I think the following call generates the error :

(skkdic-extract-conversion-data ">わk 分")

However when I run it by itself it doesn't generate an error today,
yesterday it did. That is beyond my skill to fix.

Another issue with the new SKK-JYSHO.L is that it has some comments, for
example :

"あおt 煽;火を煽る 呷;酒を呷る"

My guess is that the block after ; should be ignored. I think 呷 should
be added as a candidate. I think I can fix that.

>> I don't think SKK-JYSYO.L needs to be included in the source code, it's
>> 2.7M and doesn't need to be there if ja-dic.el exists. It would make the
>> tar ball a bit smaller.
>
>  new SKK-JISYO.L much bigger than the current one?

The new SKK-JISYO.L is 4.3M.

> Perhaps we should consider moving SKK-JISYO.L (and the other
> big files) to `admin' directory which is not included in the
> tarball.

It sounds like a good idea, smaller tar ball saves bandwith.

Take care,
-- 
Ivan Kanis
http://kanis.fr

We know what we are, but know not what we may be.
    -- William Shakespeare 

Re: ja-dic.el and SKK-JYSYO.L

[Kenichi Handa]

In article <m3d4094dsx.fsf@patsy.makina-nantes.net>, Ivan Kanis <expire-by-2010-02-18@kanis.fr> writes:

> I put the new SKK-JYSHO.L in my tmp directory and run the command
> skkdic-convert on the dictionary. It created ja-dic.el in my home
> directory.

> However when I eval ja-dic.el I get an error:

> Debugger entered--Lisp error: (args-out-of-range ">\x308fk \x5206" 6)
>   string-match("[^ ]+" ">\x308fk \x5206" 6)
>   (while (string-match "[^ ]+" entry i) (setq candidates (cons ... candidates)) (setq i (match-end 0)))
>   (let ((kana ...) (i ...) candidates) (while (string-match "[^ ]+" entry i) (setq candidates ...) (setq i ...)) (cons (skkdic-get-kana-compact-codes kana) candidates))
>   skkdic-extract-conversion-data(">\x308fk \x5206")

I found that the latest SKK-JISYO.L contains entries that
can't be handled by the current ja-dic-cnv.el.  So, I
installed the fix in addition to new SKK-JISYO.L and
re-generated ja-dic.el.  Please try the latest one, or try
the attached patch.

> Another issue with the new SKK-JYSHO.L is that it has some comments, for
> example :

> "あおt 煽;火を煽る 呷;酒を呷る"

> My guess is that the block after ; should be ignored. I think 呷 should
> be added as a candidate. I think I can fix that.

Instead, I downloaded SKK-JISYO.L.unannotated and renamed it
to SKK-JISYO.L.

>>> I don't think SKK-JYSYO.L needs to be included in the source code, it's
>>> 2.7M and doesn't need to be there if ja-dic.el exists. It would make the
>>> tar ball a bit smaller.
> >
> >  new SKK-JISYO.L much bigger than the current one?

> The new SKK-JISYO.L is 4.3M.

But the compressed one is just 400k bigger than the old one.
It is surely big but not that disastrous.  So, ...

> > Perhaps we should consider moving SKK-JISYO.L (and the other
> > big files) to `admin' directory which is not included in the
> > tarball.

> It sounds like a good idea, smaller tar ball saves bandwith.

I'll do that after 23.2.

---
Kenichi Handa
handa@m17n.org

=== modified file 'lisp/international/ja-dic-cnv.el'
--- lisp/international/ja-dic-cnv.el	2010-01-13 08:35:10 +0000
+++ lisp/international/ja-dic-cnv.el	2010-02-16 06:47:31 +0000
@@ -45,15 +45,6 @@
 ;; Name of a file to generate from SKK dictionary.
 (defvar ja-dic-filename "ja-dic.el")
 
-;; To make a generated ja-dic.el smaller.
-(define-coding-system 'iso-2022-7bit-short
- "Like `iso-2022-7bit' but no ASCII designation before SPC."
-  :coding-type 'iso-2022
-  :mnemonic ?J
-  :charset-list 'iso-2022
-  :designation [(ascii t) nil nil nil]
-  :flags '(short 7-bit designation))
-
 (defun skkdic-convert-okuri-ari (skkbuf buf)
   (message "Processing OKURI-ARI entries ...")
   (goto-char (point-min))
@@ -61,24 +52,22 @@
     (insert ";; Setting okuri-ari entries.\n"
 	    "(skkdic-set-okuri-ari\n"))
   (while (not (eobp))
-    (let ((from (point))
-	  to)
-      (end-of-line)
-      (setq to (point))
-
-      (with-current-buffer buf
-	(insert-buffer-substring skkbuf from to)
-	(beginning-of-line)
-	(insert "\"")
-	(search-forward " ")
-	(delete-char 1)			; delete the first '/'
-	(let ((p (point)))
-	  (end-of-line)
-	  (delete-char -1)		; delete the last '/'
-	  (subst-char-in-region p (point) ?/ ? 'noundo))
-	(insert "\"\n"))
+    (if (/= (following-char) ?>)
+	(let ((from (point))
+	      (to (line-end-position)))
+	  (with-current-buffer buf
+	    (insert-buffer-substring skkbuf from to)
+	    (beginning-of-line)
+	    (insert "\"")
+	    (search-forward " ")
+	    (delete-char 1)		; delete the first '/'
+	    (let ((p (point)))
+	      (end-of-line)
+	      (delete-char -1)		; delete the last '/'
+	      (subst-char-in-region p (point) ?/ ? 'noundo))
+	    (insert "\"\n"))))
 
-      (forward-line 1)))
+    (forward-line 1))
   (with-current-buffer buf
     (insert ")\n\n")))
 
@@ -348,7 +337,7 @@
       (erase-buffer)
       (buffer-disable-undo)
       (insert ";;; ja-dic.el --- dictionary for Japanese input method"
-	      " -*-coding: iso-2022-jp; byte-compile-disable-print-circle:t; -*-\n"
+	      " -*-coding: euc-japan; byte-compile-disable-print-circle:t; -*-\n"
 	      ";;\tGenerated by the command `skkdic-convert'\n"
 	      ";;\tDate: " (current-time-string) "\n"
 	      ";;\tOriginal SKK dictionary file: "
@@ -410,7 +399,7 @@
       ;; Save the working buffer.
       (set-buffer buf)
       (set-visited-file-name (expand-file-name ja-dic-filename dirname) t)
-      (set-buffer-file-coding-system 'iso-2022-7bit-short)
+      (set-buffer-file-coding-system 'euc-japan)
       (save-buffer 0))
     (kill-buffer skkbuf)
     (switch-to-buffer buf)))

Re: ja-dic.el and SKK-JYSYO.L

[Stefan Monnier]

>> > Perhaps we should consider moving SKK-JISYO.L (and the other
>> > big files) to `admin' directory which is not included in the tarball.
>> It sounds like a good idea, smaller tar ball saves bandwith.
> I'll do that after 23.2.

If we do that, we need to replace the file by a URL in the tarball (so
the file in `admin' is just a cache).
I think it's important to try and preserve the property that the tarball
contains all the *source* files.  There are currently some files missing
there, admittedly, but I'm not completely happy about it.


        Stefan "who'd be happy with a trivial solution to the above problem
                which just distributes `admin' in the tarball, or better
                yet: make the tarball be a lightweight checkout of the
                repository"

Re: ja-dic.el and SKK-JYSYO.L

[Kenichi Handa]

In article <jwvaav97031.fsf-monnier+emacs@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> > Perhaps we should consider moving SKK-JISYO.L (and the other
>>> > big files) to `admin' directory which is not included in the tarball.
>>> It sounds like a good idea, smaller tar ball saves bandwith.
> > I'll do that after 23.2.

> If we do that, we need to replace the file by a URL in the tarball (so
> the file in `admin' is just a cache).

?? I don't understand exactly what you mean.

> I think it's important to try and preserve the property that the tarball
> contains all the *source* files.  There are currently some files missing
> there, admittedly, but I'm not completely happy about it.

What do you mean by "*source*" here?  Currently
leim/ja-dic/ja-dic.el is generated from SKK-JISYO.L manually
(i.e. by running M-x skkdic-convert) and then committed as
the source of ja-dic.elc.  And ja-dic.elc is automatically
created from ja-dic.el by Makefile.  So, in some senece,
ja-dic.el is the source, and SKK-JISYO.L is an external
information we use to make ja-dic.el.

---
Kenichi Handa
handa@m17n.org

Re: ja-dic.el and SKK-JYSYO.L

[Stefan Monnier]

>> If we do that, we need to replace the file by a URL in the tarball (so
>> the file in `admin' is just a cache).
> ?? I don't understand exactly what you mean.

It's because I was vague: the precise way we replace the file by a URL
is left unspecified.

>> I think it's important to try and preserve the property that the tarball
>> contains all the *source* files.  There are currently some files missing
>> there, admittedly, but I'm not completely happy about it.
> What do you mean by "*source*" here?

Same as in the GPL.  Basically "not auto-generated from some other
file".  IIRC the GPL says something like "the preferred form for
editing".

> Currently leim/ja-dic/ja-dic.el is generated from SKK-JISYO.L manually
> (i.e. by running M-x skkdic-convert) and then committed as the source
> of ja-dic.elc.

The fact that there is a manual step is not enough to make it "source".


        Stefan

Re: ja-dic.el and SKK-JYSYO.L

[Kenichi Handa]

In article <jwvk4uck56u.fsf-monnier+emacs@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> If we do that, we need to replace the file by a URL in the tarball (so
>>> the file in `admin' is just a cache).
> > ?? I don't understand exactly what you mean.

> It's because I was vague: the precise way we replace the file by a URL
> is left unspecified.

How about including .bzr directory (or something) so that a
user who extracted files from the tarball can get all files
just by typing "bzr up" under admin/*?  Is such a thing
possible by bzr?

---
Kenichi Handa
handa@m17n.org

Re: ja-dic.el and SKK-JYSYO.L

[Stefan Monnier]

>>>> If we do that, we need to replace the file by a URL in the tarball (so
>>>> the file in `admin' is just a cache).
>> > ?? I don't understand exactly what you mean.
>> It's because I was vague: the precise way we replace the file by a URL
>> is left unspecified.
> How about including .bzr directory (or something) so that a
> user who extracted files from the tarball can get all files
> just by typing "bzr up" under admin/*?  Is such a thing
> possible by bzr?

It'd have to be "bzr revert" rather than "bzr update" since missing
files are considered as implicit "bzr rm" requests.

I'd rather have a plain file giving URLs (and md5sums) so people who
want to get the source will know where to find it.


        Stefan "As mentioned, I'd even prefer to just keep all those files in
                the tarball"

Re: ja-dic.el and SKK-JYSYO.L

[Kenichi Handa]

In article <jwvaav8ieb1.fsf-monnier+emacs@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>>>> If we do that, we need to replace the file by a URL in the tarball (so
>>>>> the file in `admin' is just a cache).
>>> > ?? I don't understand exactly what you mean.
>>> It's because I was vague: the precise way we replace the file by a URL
>>> is left unspecified.
> > How about including .bzr directory (or something) so that a
> > user who extracted files from the tarball can get all files
> > just by typing "bzr up" under admin/*?  Is such a thing
> > possible by bzr?

> It'd have to be "bzr revert" rather than "bzr update" since missing
> files are considered as implicit "bzr rm" requests.

Ah, I see.

> I'd rather have a plain file giving URLs (and md5sums) so people who
> want to get the source will know where to find it.

URLs change from time to time and are not reliable.  I think
it is better that we keep the copies by ourselves in the bzr
repository or some of FSF's ftp site.  I think the bzr
repository is better because it is appearent to users when
we update files.

---
Kenichi Handa
handa@m17n.org

Re: ja-dic.el and SKK-JYSYO.L

[Alfred M. Szmidt]

	   Stefan "As mentioned, I'd even prefer to just keep all
		   those files in the tarball"

Is there some problem of not doing exactly that?  The file is only
3MiB, compressed it will be small, and the bandwidth saved by not
including it is miniscule.

Re: ja-dic.el and SKK-JYSYO.L

[Stefan Monnier]

> 	   Stefan "As mentioned, I'd even prefer to just keep all
> 		   those files in the tarball"

> Is there some problem of not doing exactly that?

Not that I know.  If we need to save bandwidth, getting rid of the .elc
files would be at least as effective.


        Stefan