A system for localizing documentation strings

A system for localizing documentation strings

[Jean-Christophe Helary]

A function is  written like this (per Robert Chassell's book, I'm  
currently only at chapter 3, sorry if all sounds a little too obvious):

(defun function-name (arguments...)
	"optional-documentation..."
	(interactive argument-passing-info)     ; optional
	body...)

To offer a practical possibility for interactive localization we'd  
need a function that dynamically generates output instead of the  
"optional-documentation..." string. This function would take a number  
of paired arguments:

(docfun
	source-language-1 source-language-1-documentation-string
	source-language-2 source-language-2-documentation-string
	etc ...)

for ex:

(docfun
	EN "optional-documentation in EN..."
	FR "documentation optionnelle en FR...")

docfun would provide the data (language+string) to the display  
devices (help functions etc) that would need to be able to get the  
proper string based on the environment locale (or whatever locale  
data is available). In case the locale matching string is not  
available (not translated), a mechanism to display a prefered  
language, then the default language should be implemented.

In the end, the template for a defun would be:

Template 1:

(defun function-name (arguments...)
	(docfun
		EN "optional-documentation in EN..."
		FR "documentation optionnelle en FR...")
	(interactive argument-passing-info)     ; optional
  	body...)



To simplify the writing, in the case a lot of functions and  
documentation needs to be written (which is always a good thing) we  
could have a doclang function that takes one argument, the language  
in which the documentation has been written so that a typical .el  
file could look like:


Template 2:

(doclang JA) ; optional so keep backward compatibility with the  
current lack of framework

(defun function-name1 (arguments...)
	"このファンクションのドキュメンテーションは日本語だ..."
	(interactive argument-passing-info)     ; optional
  	body...)

(defun function-name2 (arguments...)
	"このファンクションのドキュメンテーションも日本語だ..."
	(interactive argument-passing-info)     ; optional
  	body...)


Now, to offer a real framework for translating, since we _don't want  
to use intermediate formats (like PO etc), although those could be  
available for gettext fans, we'd need another function that takes a  
number of arguments and that could look like:

(transfun function-name
	source-language
	target-language
	reference-function-name ; should be a list
	reference-file ; should be a list)

The function-name declares which function has to be translated
The source-language declares from which language string the source  
should be displayed
The target-language declares to which language the translator is working
the reference-function-name declares which functions should be taken  
as reference for the current translation.
the reference-file declares which files should be taken as reference  
(ideally, PO compendia, TMX files, CSV files etc)

transfun would be a whole different business since it would actually  
provide a real dynamic fuzzy matching engine between the source- 
language strings and the source reference strings. _NOT_ something  
like the "fuzzy" thing gettext provides.

In the case of Template 1 (a function that has already been  
translated to a number of languages), transfun would just add a line  
to the documentation function.

In the case of Template 2 (a function that exists in only one  
language), transfun would also transform Template 2 into Template 1  
to add the documentation at the proper location. If the doclang  
function is not documented, transfun asks what argument should  
doclang have and proceeds.

Now, it would of course be possible to have translation tools support  
the defun template so that they output the target strings to the  
correct position.

Of course, I am only talking about functions here (remember: chapter  
3 of Chassell's book) but I am gessing that most of this "framework"  
could be extended to support _all_ the translatable strings emacs  
displays.


As for the quality of the translation (and it seems it is mostly  
people who have never done translation work before who complain the  
most) I suggest you leave that to the localization groups that are  
used to the processes.

What we need is provide 1) a way for coders to identify the necessary  
strings for the translation 2) a way for translators to add  
translated strings "the emacs way" 3) a modification of the display  
procedures to take the new strings into account.


Jean-Christophe Helary

Re: A system for localizing documentation strings

[Eli Zaretskii]

> From: Jean-Christophe Helary <fusion@mx6.tiki.ne.jp>
> Date: Thu, 26 Jul 2007 10:05:36 +0900
> 
> A function is  written like this (per Robert Chassell's book, I'm  
> currently only at chapter 3, sorry if all sounds a little too obvious):
> 
> (defun function-name (arguments...)
> 	"optional-documentation..."
> 	(interactive argument-passing-info)     ; optional
> 	body...)
> 
> To offer a practical possibility for interactive localization we'd  
> need a function that dynamically generates output instead of the  
> "optional-documentation..." string. This function would take a number  
> of paired arguments:
> 
> (docfun
> 	source-language-1 source-language-1-documentation-string
> 	source-language-2 source-language-2-documentation-string
> 	etc ...)
> 
> for ex:
> 
> (docfun
> 	EN "optional-documentation in EN..."
> 	FR "documentation optionnelle en FR...")

This kind of suggestions and the ensuing discussion should be on
emacs-devel mailing list, not here.

Fwd: A system for localizing documentation strings

[Jean-Christophe Helary]

I was told by Eli Zaretskii that such discussions should better take  
place here, hence my forward from help-gnu-emacs@gnu.org

After I sent the mail this morning, I realized that obviously  
interactive output could/should be handled in similar ways.

I think in the end, we need a way to encapsulate translatable strings  
in objects (functions ?) that will allow the strings to be available  
in a multiplicity of processes: display first, but of course  
translation.

Keep in mind that I am only a translator and that I am only getting  
started with emacs, emacs lisp and emacs localization.

Regards,

Jean-Christophe Helary

============================================
Begin forwarded message:

From: Jean-Christophe Helary
Date: 26 juillet 2007 10:05:36 HNJ
To: Emacs Mailing List <help-gnu-emacs@gnu.org>
Subject: A system for localizing documentation strings

A function is  written like this (per Robert Chassell's book, I'm  
currently only at chapter 3, sorry if all sounds a little too obvious):

(defun function-name (arguments...)
	"optional-documentation..."
	(interactive argument-passing-info)     ; optional
	body...)

To offer a practical possibility for interactive localization we'd  
need a function that dynamically generates output instead of the  
"optional-documentation..." string. This function would take a number  
of paired arguments:

(docfun
	source-language-1 source-language-1-documentation-string
	source-language-2 source-language-2-documentation-string
	etc ...)

for ex:

(docfun
	EN "optional-documentation in EN..."
	FR "documentation optionnelle en FR...")

docfun would provide the data (language+string) to the display  
devices (help functions etc) that would need to be able to get the  
proper string based on the environment locale (or whatever locale  
data is available). In case the locale matching string is not  
available (not translated), a mechanism to display a prefered  
language, then the default language should be implemented.

In the end, the template for a defun would be:

Template 1:

(defun function-name (arguments...)
	(docfun
		EN "optional-documentation in EN..."
		FR "documentation optionnelle en FR...")
	(interactive argument-passing-info)     ; optional
  	body...)



To simplify the writing, in the case a lot of functions and  
documentation needs to be written (which is always a good thing) we  
could have a doclang function that takes one argument, the language  
in which the documentation has been written so that a typical .el  
file could look like:


Template 2:

(doclang JA) ; optional so keep backward compatibility with the  
current lack of framework

(defun function-name1 (arguments...)
	"このファンクションのドキュメンテーションは日本語だ..."
	(interactive argument-passing-info)     ; optional
  	body...)

(defun function-name2 (arguments...)
	"このファンクションのドキュメンテーションも日本語だ..."
	(interactive argument-passing-info)     ; optional
  	body...)


Now, to offer a real framework for translating, since we _don't want  
to use intermediate formats (like PO etc), although those could be  
available for gettext fans, we'd need another function that takes a  
number of arguments and that could look like:

(transfun function-name
	source-language
	target-language
	reference-function-name ; should be a list
	reference-file ; should be a list)

The function-name declares which function has to be translated
The source-language declares from which language string the source  
should be displayed
The target-language declares to which language the translator is working
the reference-function-name declares which functions should be taken  
as reference for the current translation.
the reference-file declares which files should be taken as reference  
(ideally, PO compendia, TMX files, CSV files etc)

transfun would be a whole different business since it would actually  
provide a real dynamic fuzzy matching engine between the source- 
language strings and the source reference strings. _NOT_ something  
like the "fuzzy" thing gettext provides.

In the case of Template 1 (a function that has already been  
translated to a number of languages), transfun would just add a line  
to the documentation function.

In the case of Template 2 (a function that exists in only one  
language), transfun would also transform Template 2 into Template 1  
to add the documentation at the proper location. If the doclang  
function is not documented, transfun asks what argument should  
doclang have and proceeds.

Now, it would of course be possible to have translation tools support  
the defun template so that they output the target strings to the  
correct position.

Of course, I am only talking about functions here (remember: chapter  
3 of Chassell's book) but I am gessing that most of this "framework"  
could be extended to support _all_ the translatable strings emacs  
displays.


As for the quality of the translation (and it seems it is mostly  
people who have never done translation work before who complain the  
most) I suggest you leave that to the localization groups that are  
used to the processes.

What we need is provide 1) a way for coders to identify the necessary  
strings for the translation 2) a way for translators to add  
translated strings "the emacs way" 3) a modification of the display  
procedures to take the new strings into account.


Jean-Christophe Helary

============================================

Re: Fwd: A system for localizing documentation strings

[Kenichi Handa]

In article <AA79A27B-73A6-4323-AB62-FC0F6259A147@mx6.tiki.ne.jp>, Jean-Christophe Helary <fusion@mx6.tiki.ne.jp> writes:

> From: Jean-Christophe Helary
> Date: 26 juillet 2007 10:05:36 HNJ
> To: Emacs Mailing List <help-gnu-emacs@gnu.org>
> Subject: A system for localizing documentation strings

> A function is  written like this (per Robert Chassell's book, I'm  
> currently only at chapter 3, sorry if all sounds a little too obvious):

> (defun function-name (arguments...)
> 	"optional-documentation..."
> 	(interactive argument-passing-info)     ; optional
> 	body...)

> To offer a practical possibility for interactive localization we'd  

Could you please explain what is "interactive localization",
and why gettext doesn't work for it.

---
Kenichi Handa
handa@m17n.org

Re: A system for localizing documentation strings

[Jean-Christophe Helary]

On 26 juil. 07, at 20:54, Kenichi Handa wrote:

>> To offer a practical possibility for interactive localization we'd
>
> Could you please explain what is "interactive localization",
> and why gettext doesn't work for it.

gettext delivers an intermediate file (PO etc) that needs to be  
translated and then put back into the code to reach localization.

What I mean by "interactive localization" is that the translator  
could translate, evaluate the function in which the translation  
occured and the translation would be right there.



Jean-Christophe Helary

Re: A system for localizing documentation strings

[Kenichi Handa]

In article <520B1EA6-0FF5-43E1-BB6D-6AD97A2EFE22@mx6.tiki.ne.jp>, Jean-Christophe Helary <fusion@mx6.tiki.ne.jp> writes:

> On 26 juil. 07, at 20:54, Kenichi Handa wrote:

>>> To offer a practical possibility for interactive localization we'd
> >
> > Could you please explain what is "interactive localization",
> > and why gettext doesn't work for it.

> gettext delivers an intermediate file (PO etc) that needs to be  
> translated and then put back into the code to reach localization.

> What I mean by "interactive localization" is that the translator  
> could translate, evaluate the function in which the translation  
> occured and the translation would be right there.

Thank you for the explanation.  In the case of translating
labels, menus, etc, I see some merit in getting translation
instantly, but, for function/variable docstrings, I don't
understand why it's that important to get the translation
instantly by evaluating a funciton.



---
Kenichi Handa
handa@m17n.org

Re: A system for localizing documentation strings

[Jean-Christophe Helary]

On 27 juil. 07, at 09:30, Kenichi Handa wrote:

> In article <520B1EA6-0FF5-43E1-BB6D-6AD97A2EFE22@mx6.tiki.ne.jp>,  
> Jean-Christophe Helary <fusion@mx6.tiki.ne.jp> writes:
>
>> On 26 juil. 07, at 20:54, Kenichi Handa wrote:
>
>>>> To offer a practical possibility for interactive localization we'd
>>>
>>> Could you please explain what is "interactive localization",
>>> and why gettext doesn't work for it.
>
>> gettext delivers an intermediate file (PO etc) that needs to be
>> translated and then put back into the code to reach localization.
>
>> What I mean by "interactive localization" is that the translator
>> could translate, evaluate the function in which the translation
>> occured and the translation would be right there.
>
> Thank you for the explanation.  In the case of translating
> labels, menus, etc, I see some merit in getting translation
> instantly, but, for function/variable docstrings, I don't
> understand why it's that important to get the translation
> instantly by evaluating a funciton.

I don't know the emacs code. So my idea was that pretty much  
everything was a function.

If we manage to have a process that smoothly handles the functions  
that define labels and menus etc then, as a "side effect" the same  
process will smoothly handle the rest.

Also, getting the translation immediately allows for easier delivery:  
the translation can be checked right away in context, modified if  
necessary and the result is better than if the translation has to be  
later re-imported in the code (or wherever) for checks and  
modifications.


Jean-Christophe Helary

Re: Fwd: A system for localizing documentation strings

[Eli Zaretskii]

> From: Jean-Christophe Helary <fusion@mx6.tiki.ne.jp>
> Date: Thu, 26 Jul 2007 12:29:19 +0900
> 
> I was told by Eli Zaretskii that such discussions should better take  
> place here, hence my forward from help-gnu-emacs@gnu.org

Thanks.

> To offer a practical possibility for interactive localization we'd  
> need a function that dynamically generates output instead of the  
> "optional-documentation..." string. This function would take a number  
> of paired arguments:
> 
> (docfun
> 	source-language-1 source-language-1-documentation-string
> 	source-language-2 source-language-2-documentation-string
> 	etc ...)
> 
> for ex:
> 
> (docfun
> 	EN "optional-documentation in EN..."
> 	FR "documentation optionnelle en FR...")

I don't like this implementation idea, because it would require the
user to byte-compile Lisp files whenever a translation to another
language is added.  This would be very inconvenient, especially for
*.el files that are preloaded when Emacs is built, because that would
mean one must have the sources available nearby, and must run the
build procedure and "make install".  The latter step requires sysadmin
privileges on many platforms, another inconvenience.

I think it's much better to have a separate translation file for each
.el file.  Such a translation file would be loaded on demand when
documentation for symbols defined on that file is requested by the
user.  The translation file needs include only the names of the
symbols and their doc strings for supported languages.  (We could also
have separate translation files for each language, but I think it
won't be necessary, as the number of symbols on a single .el file is
quite small.)

> (transfun function-name
> 	source-language
> 	target-language
> 	reference-function-name ; should be a list
> 	reference-file ; should be a list)
> 
> The function-name declares which function has to be translated
> The source-language declares from which language string the source  
> should be displayed

Why do we need the source language?

> What we need is provide 1) a way for coders to identify the necessary  
> strings for the translation 2) a way for translators to add  
> translated strings "the emacs way" 3) a modification of the display  
> procedures to take the new strings into account.

Your number 3) is not described correctly: it's not the display that
needs to be modified, it's the Emacs documentation commands.  The
documentation commands don't display anything, they just insert the
doc text into a buffer, whether *Help* or minibuffer or something
else.  The Emacs redisplay engine then displays that buffer; however,
if the text in the buffer to be displayed is already in French (say),
that is what you will see after it is displayed.

So what is needed is to modify the documentation commands so that they
will look up the translated text and display that text instead of the
original English doc string.

Also, we should keep in mind that Lisp primitives (those implemented in
C) have their doc strings as C comments, not as C strings.  The
infrastructure developed for Emacs l10n should provide solution for
the primitives as well, and the solution will have to be different
both from your suggestion above and from the traditional gettext-style
message catalog.

Re: A system for localizing documentation strings

[Jean-Christophe Helary]

On 26 juil. 07, at 21:13, Eli Zaretskii wrote:

>> From: Jean-Christophe Helary <fusion@mx6.tiki.ne.jp>
>> Date: Thu, 26 Jul 2007 12:29:19 +0900
>>
>> I was told by Eli Zaretskii that such discussions should better take
>> place here, hence my forward from help-gnu-emacs@gnu.org
>
> Thanks.
>
>> To offer a practical possibility for interactive localization we'd
>> need a function that dynamically generates output instead of the
>> "optional-documentation..." string. This function would take a number
>> of paired arguments:
>>
>> (docfun
>> 	source-language-1 source-language-1-documentation-string
>> 	source-language-2 source-language-2-documentation-string
>> 	etc ...)
>>
>> for ex:
>>
>> (docfun
>> 	EN "optional-documentation in EN..."
>> 	FR "documentation optionnelle en FR...")
>
> I don't like this implementation idea, because it would require the
> user to byte-compile Lisp files whenever a translation to another
> language is added.  This would be very inconvenient, especially for
> *.el files that are preloaded when Emacs is built, because that would
> mean one must have the sources available nearby, and must run the
> build procedure and "make install".  The latter step requires sysadmin
> privileges on many platforms, another inconvenience.

I see 2 types of .el files.

1) Those that come with the emacs distribution.

2) Those that can be installed in user space.

The distribution would come with the translated .el files and the  
updates to the translation could be included in updates of the  
distribution without requiring the user to build the distribution  
when new translations come. Since the translations are in the code  
there is little management necessary (unlike systems where the  
localizations come in separate files).

For the .el files in user space, the author would maintain the file's  
translation and users would be able to compile the file without any  
specific problem.

> I think it's much better to have a separate translation file for each
> .el file.  Such a translation file would be loaded on demand when
> documentation for symbols defined on that file is requested by the
> user.  The translation file needs include only the names of the
> symbols and their doc strings for supported languages.  (We could also
> have separate translation files for each language, but I think it
> won't be necessary, as the number of symbols on a single .el file is
> quite small.)

I don't think it is a good idea because it would put a bigger load on  
the coder who would then have to write keys in the code and then  
values in a separate file. That would impair understanding of the  
code too. Besides for the fact that it would be a major departure  
from the current code workflow.

>> (transfun function-name
>> 	source-language
>> 	target-language
>> 	reference-function-name ; should be a list
>> 	reference-file ; should be a list)
>>
>> The function-name declares which function has to be translated
>> The source-language declares from which language string the source
>> should be displayed
>
> Why do we need the source language?

Because as soon as we have a system that allows for localization we  
can expect to have "native" code written and so we'll have to  
reference the source language that _will_ be different from English.  
We don't want people who don't master English to produce weird  
English in their descriptions because the system implies  
source=English. That would require the faulty English string to be  
rewritten before proceeding to translation.

Also, we can expect to find translators who would be more familiar  
with one source language than the other. If there is a Japanese  
equivalent to an English description. I'd rather use the Japanese as  
source and the English as reference to translate to French for exemple.

In the end, the translation needs to be recorded as a list of paired  
strings for future reference. This is usually called a "translation  
memory" and comes in different formats. A widely used one in the free  
world being PO compendia, another one, mostly used in the translation  
world is TMX (an XML dialect: Translation Memory eXchange). PO kind  
of expects English to be the source, but my understanding is that  
recent developments of gettext are making this a thing of the past.  
TMX is totaly "multilingual".

>> What we need is provide 1) a way for coders to identify the necessary
>> strings for the translation 2) a way for translators to add
>> translated strings "the emacs way" 3) a modification of the display
>> procedures to take the new strings into account.
>
> Your number 3) is not described correctly: it's not the display that
> needs to be modified, it's the Emacs documentation commands.  The
> documentation commands don't display anything, they just insert the
> doc text into a buffer, whether *Help* or minibuffer or something
> else.  The Emacs redisplay engine then displays that buffer; however,
> if the text in the buffer to be displayed is already in French (say),
> that is what you will see after it is displayed.

Ok, sorry for my misunderstanding.

> So what is needed is to modify the documentation commands so that they
> will look up the translated text and display that text instead of the
> original English doc string.

That is correct. Except that we can't expect the English to be the  
original anymore. Hence the necessity to specify what language the  
strings are written in.

> Also, we should keep in mind that Lisp primitives (those  
> implemented in
> C) have their doc strings as C comments, not as C strings.  The
> infrastructure developed for Emacs l10n should provide solution for
> the primitives as well, and the solution will have to be different
> both from your suggestion above and from the traditional gettext-style
> message catalog.

Could that part be concieved separatly ?

I mean, we could start by having a modification in the documentation  
commands and then see what that gives with catalogs (either in  
the .el or separate, because in the end we could consider that the 2  
solutions are valid depending on the scope of the .el file).


Jean-Christophe Helary

Re: A system for localizing documentation strings

[Jason Rumney]

Jean-Christophe Helary wrote:
> I don't think it is a good idea because it would put a bigger load on
> the coder who would then have to write keys in the code and then
> values in a separate file. That would impair understanding of the code
> too. Besides for the fact that it would be a major departure from the
> current code workflow.

It need not be a major departure. The "keys" can be the function and
variable names (where both exist with the same name, we'd need some way
to differentiate). The "default" docstring (English) could remain with
the function or variable definition, so nothing need change in the lisp
files.

Keeping the translations in separate files makes it easier to update
them separately from the Emacs release cycle. Eventually we will reach a
point where the translations don't change separately from Emacs, but for
at least a few Emacs releases there will be new translations and
corrections to translations appearing much quicker than the Emacs
release cycle, and it would be useful if users could take advantage of them.

If the translated doc string format is a lisp form, then we have a lot
of flexibility in how we arrange the files.  Small lisp extensions
distributed separately from Emacs would have the option to include their
translations in the same file as the code, and larger packages would
have the option of having a file per language.

Eg:

(define-docstring-translation '(:function example-function) 'ja "語例")



The other requirement would be for a lisp file to declare where its
translations are, and for the user to be able to add more later:

(provide-translations 'example-feature '(ja es fr) "example-translations")

where example-feature corresponds to the provide statement in the file,
and translations for Japanese, Spanish and French are in
"example-translations.el"

Re: A system for localizing documentation strings

[Jean-Christophe Helary]

On 26 juil. 07, at 23:50, Jason Rumney wrote:

> Jean-Christophe Helary wrote:
>> I don't think it is a good idea because it would put a bigger load on
>> the coder who would then have to write keys in the code and then
>> values in a separate file. That would impair understanding of the  
>> code
>> too. Besides for the fact that it would be a major departure from the
>> current code workflow.
>
> It need not be a major departure. The "keys" can be the function and
> variable names (where both exist with the same name, we'd need some  
> way
> to differentiate). The "default" docstring (English) could remain with
> the function or variable definition, so nothing need change in the  
> lisp
> files.

I think I understand. But you have to consider that as soon as we  
provide a localization framework we offer the possibility to have  
something else but English as the default.

That may not be the case for the core distribution for the next few  
versions to come, but there are no reasons not to consider the  
possibility that a Brazilian developer could write files documented  
in Brazilian Portuguese, which would later be localized to English,  
or anything else.

This is why I also proposed a language identifier for non localized  
files:

> To simplify the writing, in the case a lot of functions and  
> documentation needs to be written (which is always a good thing) we  
> could have a doclang function that takes one argument, the language  
> in which the documentation has been written so that a typical .el  
> file could look like:
>
>
> Template 2:
>
> (doclang JA) ; optional as to keep backward compatibility with the  
> current lack of framework
>
> (defun function-name1 (arguments...)
> 	"このファンクションのドキュメンテーションは日本語だ..."
> 	(interactive argument-passing-info)     ; optional
>  	body...)
>
> (defun function-name2 (arguments...)
> 	"このファンクションのドキュメンテーションも日本語だ..."
> 	(interactive argument-passing-info)     ; optional
>  	body...)

Once a file is localized, the el file should be language "agnostic"  
and I also proposed:

> In case the locale matching string is not available (not  
> translated), a mechanism to display a prefered language, then the  
> default language should be implemented.

Because we should not assume that the preferred default should be  
English (it could be Russian for some Polish users, or Japanese for  
some Chinese users etc)


> Keeping the translations in separate files makes it easier to update
> them separately from the Emacs release cycle.

I don't think that is related. The ease of updating will only depend  
on the tools that will be made available with this framework. The  
difference will be whether the tools can parse a normal .el file or not.

I think departing from the current, how do you call that, "literate  
programming" style would dramatically reduce the efficiency of the  
current workflow: not only is the function documentation available  
interactively but it also is _within_ the code.

I think the "updating" process fear comes from the omnipresence of  
the "gettext" paradigm.

I don't think this paradigm is strong or flexible enough. Let me  
explain.

When a translation is made at a certain point in time, it is stored  
as a "translation memory" (TM) file.

When a documentation update takes place, the TM file automatically  
matches the non modified contents (directly in the .el file) and  
automatically proposes the modified contents since they are not in  
the TM. The translator can use similar source/target pairs as  
reference to translate the modified string.

The process of taking a diff, translating (without context) and  
patching the result _is_ what makes the process cumbersome.

> Eventually we will reach a
> point where the translations don't change separately from Emacs,  
> but for
> at least a few Emacs releases there will be new translations and
> corrections to translations appearing much quicker than the Emacs
> release cycle, and it would be useful if users could take advantage  
> of them.

Yes, but then, I'd argue that each localization community (and they  
would be created right away) would be able to maintain a "localized  
fork" with the same code base as the stable emacs branch while  
releasing builds for their l10n community. Which _is_ the point in  
the end :) The localized builds can merge their localized files for  
each major emacs stable release.

The simplest the localization process will be, the fastest we will  
have localized _quality_ builds: what keeps professional translators  
from participating to free software localization is the complexity of  
the processes involved, _not_ their lack of will to participate.

> If the translated doc string format is a lisp form, then we have a lot
> of flexibility in how we arrange the files.  Small lisp extensions
> distributed separately from Emacs would have the option to include  
> their
> translations in the same file as the code, and larger packages would
> have the option of having a file per language.
>
> Eg:
>
> (define-docstring-translation '(:function example-function) 'ja  
> "語例")
>
>
>
> The other requirement would be for a lisp file to declare where its
> translations are, and for the user to be able to add more later:
>
> (provide-translations 'example-feature '(ja es fr) "example- 
> translations")
>
> where example-feature corresponds to the provide statement in the  
> file,
> and translations for Japanese, Spanish and French are in
> "example-translations.el"

Whatever is elegant and not too geeky will sure see localizers  
flocking to emacs :)


Jean-Christophe Helary

Re: A system for localizing documentation strings

[Jason Rumney]

Jean-Christophe Helary wrote:
> I think I understand. But you have to consider that as soon as we
> provide a localization framework we offer the possibility to have
> something else but English as the default.

I think it is a bad idea to mix default languages. The default will be
used as a fallback, so it should be English where possible, as that is
the most widely understood language in the world today.

If a developer does not speak English, they could use Portuguese for the
doc strings originally, but as soon as English translations are
available, they should become the default, and the Portuguese moved out
to whatever translation format we end up with.

Re: A system for localizing documentation strings

[Jean-Christophe Helary]

On 27 juil. 07, at 00:41, Jason Rumney wrote:

> Jean-Christophe Helary wrote:
>> I think I understand. But you have to consider that as soon as we
>> provide a localization framework we offer the possibility to have
>> something else but English as the default.
>
> I think it is a bad idea to mix default languages. The default will be
> used as a fallback, so it should be English where possible, as that is
> the most widely understood language in the world today.

Obviously English will be the default for English speaking  
communities, Portuguese for Portuguese speaking communities.

Besides, English is not the most widely understood language in the  
world today. And maybe emacs-dev is not the right place to make such  
assumptions anyway. Obviously we have a very strong English speaking  
community here...

> If a developer does not speak English, they could use Portuguese  
> for the
> doc strings originally, but as soon as English translations are
> available, they should become the default, and the Portuguese moved  
> out
> to whatever translation format we end up with.

Obviously this is a fundamental mistake in the design as I wrote  
earlier: if Portuguese developers want to modify the code they  
originally wrote they'd need to refer to a different file to find  
their original comments.

Also, a Lebanese developer who reads French would rather have French  
as the default than any dialect of English.

Either localization is agnostic (ie it considers all languages  
technically equal) or it is flawed.


Jean-Christophe Helary

Re: A system for localizing documentation strings

[Jason Rumney]

Jean-Christophe Helary wrote:
> Obviously English will be the default for English speaking
> communities, Portuguese for Portuguese speaking communities.

I don't think you understand what I mean here by "fallback". We won't
have translations covering all parts of Emacs in all languages
instantly. In the case that a translation in the user's preferred
language(s) is not available, we need to fall back on a default.
Consistently using English as that default makes more sense than getting
whatever language the developer originally wrote the documentation in.

> Besides, English is not the most widely understood language in the
> world today.

Would you care to elaborate? I'm not only talking about first languages
here.

> Also, a Lebanese developer who reads French would rather have French
> as the default than any dialect of English.
So we allow them to specify multiple languages in order of preference:
(setq preferred-doc-language '(ar-LB, ar, fr))
Only if translations for none of those languages exist do we give them
the English doc string.

Re: A system for localizing documentation strings

[Jean-Christophe Helary]

On 27 juil. 07, at 01:15, Jason Rumney wrote:

>> Also, a Lebanese developer who reads French would rather have French
>> as the default than any dialect of English.
> So we allow them to specify multiple languages in order of preference:
> (setq preferred-doc-language '(ar-LB, ar, fr))
> Only if translations for none of those languages exist do we give them
> the English doc string.

This is exactly what I proposed in my original mail. I am glad you  
eventually reached that conclusion.

> In case the locale matching string is not available (not  
> translated), a mechanism to display a prefered language, then the  
> default language should be implemented.


Jean-Christophe Helary

Re: A system for localizing documentation strings

[Jean-Christophe Helary]

On 27 juil. 07, at 01:15, Jason Rumney wrote:

>> Besides, English is not the most widely understood language in the
>> world today.
>
> Would you care to elaborate? I'm not only talking about first  
> languages
> here.

If you are talking about the level of proficiency required to read  
the emacs manual then I am positive you are wrong.

And I think we _are_ talking about emacs related language ability so  
let's keep it there.

Jean-Christophe Helary

Re: A system for localizing documentation strings

[Jason Rumney]

Jean-Christophe Helary wrote:
> If you are talking about the level of proficiency required to read the
> emacs manual then I am positive you are wrong.

I am talking about a language of last resort. Proficient speakers can
indicate their preferences appropriately. Sure, at some proficiency
level Mandarin will overtake English in number of speakers, but the
number of people who understand enough Mandarin to slowly make sense of
some documentation that is not available in languages they are
proficient in is minuscule compared with the number who know a small
amount of English.

Re: A system for localizing documentation strings

[Richard Stallman]

Most programmers all around the world learn English, because most
of what they want to read is written in English.
I don't think that is likely to change any time soon.

Re: A system for localizing documentation strings

[Jean-Christophe Helary]

On 27 juil. 07, at 14:04, Richard Stallman wrote:

> Most programmers all around the world learn English, because most
> of what they want to read is written in English.
> I don't think that is likely to change any time soon.

I think you should take a look at book stores in any city outside the  
English speaking world.

There are enough books on any language to satisfy any programmer from  
beginner to engineer.

Even in the small town when I live in Japan you can find Paul  
Graham's books in Japanese. And more books on Lisp than in the small  
French town where I used to live.

I am amazed English programmers still believe their industry still  
revolves around their native language. This has not been the case for  
more than a decade: exactly when my small town bookstore stopped to  
sell the English versions of O'Reilly's books. Now most of them are  
translated. SUN's books are translated. Knuth's books are translated.

There is absolutely _no_ need whatsoever for Japanese people to learn  
_any_ bit of English to become excellent programmers.


Jean-Christophe Helary

Re: A system for localizing documentation strings

[David Kastrup]

Jean-Christophe Helary <fusion@mx6.tiki.ne.jp> writes:

> I am amazed English programmers still believe their industry still
> revolves around their native language. This has not been the case
> for more than a decade: exactly when my small town bookstore stopped
> to sell the English versions of O'Reilly's books. Now most of them
> are translated. SUN's books are translated. Knuth's books are
> translated.
>
> There is absolutely _no_ need whatsoever for Japanese people to
> learn _any_ bit of English to become excellent programmers.

A colleague of mine encountered some sort of tutorial or whatever that
included PostScript.  The book had been translated into German very
thoroughly, including all the PostScript operators in the examples.
Needless to say, the examples would no longer work.

As long as all reserved words are written in English, and non-ASCII
letters can't even be made identifiers, there is already preselection
happening in the program ecosystem.  And then there is the existing
code base.  Not least of all, that of the GNU project.  With all of
its program comments (which are not treated with i18n).

Yes, users more and more get the option of conversing with their
applications in their native language.  But programmers miss out on a
lot of important code if they lack English.

-- 
David Kastrup

Re: A system for localizing documentation strings

[David Kastrup]

David Kastrup <dak@gnu.org> writes:

> As long as all reserved words are written in English, and non-ASCII
> letters can't even be made identifiers, there is already
> preselection happening in the program ecosystem.

To followup on myself: and APL is hardly the answer.

-- 
David Kastrup

Re: A system for localizing documentation strings

[Eli Zaretskii]

> From: Jean-Christophe Helary <fusion@mx6.tiki.ne.jp>
> Date: Fri, 27 Jul 2007 16:22:46 +0900
> 
> There is absolutely _no_ need whatsoever for Japanese people to learn  
> _any_ bit of English to become excellent programmers.

Handa-san, could you please give us your perspective on this
assertion?

TIA

Re: A system for localizing documentation strings

[Kenichi Handa]

In article <ur6mudrli.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> > From: Jean-Christophe Helary <fusion@mx6.tiki.ne.jp>
> > Date: Fri, 27 Jul 2007 16:22:46 +0900
> > 
> > There is absolutely _no_ need whatsoever for Japanese people to learn  
> > _any_ bit of English to become excellent programmers.

> Handa-san, could you please give us your perspective on this
> assertion?

To become an excellent programmer, one must read a code
written by the other people, and for that, English is
mandatory.  Not only comments, but also function names,
variable names, etc. are mostly based on English words.  If
they are all some random alphabets something like
"lkvkdloa", it's almost impossible to track codes.  In
addition, one have to communicate with people working on the
same fields, and if you are working on a program used
world-widely, the developper community almost always uses
English.

As for computer/programming related books, it's sure that
most important ones are translated in Japanese.  But, most
of them assumes that a reader has some basic knowledge about
English.  Many technical terms are just written in Katakana
(a phonetic transliteration from the orignal English word),
and one has to know the meaning of original English word.

---
Kenichi Handa
handa@m17n.org

Re: A system for localizing documentation strings

[David Kastrup]

Kenichi Handa <handa@m17n.org> writes:

> In article <ur6mudrli.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:
>
>> > From: Jean-Christophe Helary <fusion@mx6.tiki.ne.jp>
>> > Date: Fri, 27 Jul 2007 16:22:46 +0900
>> > 
>> > There is absolutely _no_ need whatsoever for Japanese people to learn  
>> > _any_ bit of English to become excellent programmers.
>
>> Handa-san, could you please give us your perspective on this
>> assertion?
>
> To become an excellent programmer, one must read a code
> written by the other people, and for that, English is
> mandatory.  Not only comments, but also function names,
> variable names, etc. are mostly based on English words.  If
> they are all some random alphabets something like
> "lkvkdloa", it's almost impossible to track codes.

Historically, Basic identifiers were restricted to two letters
(possibly also just one letter and an optional digit, don't remember).

Anyway, anecdotal story: in my youth, I had been dabbling in assembly
language quite a lot: it was the thing to do if you wanted to get
serious tasks done within the 64kB restraint.  I have transferred one
arcade game from that time into C code about 14 years or so after it
had been written.  Identifiers were confined to 6 letters.  In most of
the program parts, the comments were exclusively numerical:
accumulated execution cycles.

The program flow was obvious and understandable.  Of course, I had
written the program myself.

Another experience I remembered is disassembling a Reversi program.
It was a piece of beauty: the index registers of the Z80 were employed
in an obvious way mapping to the boards and the bookkeeping stacks
respectively, and the whole thing worked with heuristic tables and
alpha/beta-pruning.  The control logic was concise and obvious.  All I
had was the binary for understanding, and it was the work of a master,
nothing deliberate in it.

The moral?  None I can discern.

-- 
David Kastrup

Re: A system for localizing documentation strings

[Kenichi Handa]

In article <86ir86kp6l.fsf@lola.quinscape.zz>, David Kastrup <dak@gnu.org> writes:

> > variable names, etc. are mostly based on English words.  If
> > they are all some random alphabets something like
> > "lkvkdloa", it's almost impossible to track codes.

> Historically, Basic identifiers were restricted to two letters
> (possibly also just one letter and an optional digit, don't remember).

> Anyway, anecdotal story: in my youth, I had been dabbling in assembly
> language quite a lot: it was the thing to do if you wanted to get
> serious tasks done within the 64kB restraint.  I have transferred one
> arcade game from that time into C code about 14 years or so after it
> had been written.  Identifiers were confined to 6 letters.  In most of
> the program parts, the comments were exclusively numerical:
> accumulated execution cycles.

> The program flow was obvious and understandable.  Of course, I had
> written the program myself.

> Another experience I remembered is disassembling a Reversi program.
> It was a piece of beauty: the index registers of the Z80 were employed
> in an obvious way mapping to the boards and the bookkeeping stacks
> respectively, and the whole thing worked with heuristic tables and
> alpha/beta-pruning.  The control logic was concise and obvious.  All I
> had was the binary for understanding, and it was the work of a master,
> nothing deliberate in it.

I also had an experience to try to understand the system
code of NEC PC8000 micro computer by disassembing Z80 codes.
At that age, the total codes are at most 32K or 64K-byte.
Good old days...

What we are facing now is collections of sometimes 10-100
times bigger program pieces.  If one spends weeks to
understand just one piece of them, he can never have a time
to write a code.  :-p

---
Kenichi Handa
handa@m17n.org

Re: A system for localizing documentation strings

[Jean-Christophe Helary]

On 27 juil. 07, at 21:07, Kenichi Handa wrote:

> In article <ur6mudrli.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org>  
> writes:
>
>>> From: Jean-Christophe Helary <fusion@mx6.tiki.ne.jp>
>>> Date: Fri, 27 Jul 2007 16:22:46 +0900
>>>
>>> There is absolutely _no_ need whatsoever for Japanese people to  
>>> learn
>>> _any_ bit of English to become excellent programmers.
>
>> Handa-san, could you please give us your perspective on this
>> assertion?
>
> To become an excellent programmer, one must read a code
> written by the other people, and for that, English is
> mandatory.  Not only comments, but also function names,
> variable names, etc. are mostly based on English words.

This is not about learning English, this is about remembering what  
code means. I.e. This is about assigning arbitrary signs some  
signification. As for code itself, there is nothing more remote from  
English than _code_.

>   If
> they are all some random alphabets something like
> "lkvkdloa", it's almost impossible to track codes.  In
> addition, one have to communicate with people working on the
> same fields, and if you are working on a program used
> world-widely, the developper community almost always uses
> English.

You are talking about a special class of developers. Most excellent  
programmers work in companies where they don't need to communicate  
with the "international community". People who maintain complex cobol  
systems in banking or insurance companies have most probably no use  
for English at all.
  What you describe is good for researchers or for high profile  
company employes (who may not be excellent programmers anyway).  And  
as a counter example, I'd argue that the ruby community in Japan has  
nothing to do with English at all, because _they_ are the experts.


> As for computer/programming related books, it's sure that
> most important ones are translated in Japanese.

Not only that but bookstores are full of high level books that are  
directly written in Japanese by Japanese experts.

>   But, most
> of them assumes that a reader has some basic knowledge about
> English.  Many technical terms are just written in Katakana
> (a phonetic transliteration from the orignal English word),
> and one has to know the meaning of original English word.

Of course not. Most words in katakana are already part of the CS  
jargon in Japanese. What you pretend is equivalent to say that to  
read a recipe book in Japanese one must know French because the  
technics used have katakana words that come from French. But the  
truth is that all those words have specific meanings that people who  
are involved with this subset of Japanese know from using them daily - 
without having to understand a single bit of French, except for "bon  
apétit"...


Jean-Christophe Helary

Re: A system for localizing documentation strings

[Richard Stallman]

      What you describe is good for researchers or for high profile  
    company employes (who may not be excellent programmers anyway).  And  
    as a counter example, I'd argue that the ruby community in Japan has  
    nothing to do with English at all, because _they_ are the experts.

That example proves our point.  They may find it useful to write
Japanese to each other, but very few of the rest of the world can read
what they say.

Technical information that is to be useful all around the world
needs to be in English.

Re: A system for localizing documentation strings

[Henrik Enberg]

Kenichi Handa <handa@m17n.org> writes:
> As for computer/programming related books, it's sure that
> most important ones are translated in Japanese.  But, most
> of them assumes that a reader has some basic knowledge about
> English.  Many technical terms are just written in Katakana
> (a phonetic transliteration from the orignal English word),
> and one has to know the meaning of original English word.

It's also worth pointing out that Japan has a population of almost 130
million people, making it economically feasible to translate special
interest books like ones about programming.  For many languages, this
isn't really possible.  So for people like me, whose mother tongue is
Swedish, learing English is even more paramount.

-- 
If animal trapped call 410-844-6286

Re: A system for localizing documentation strings

[Werner LEMBERG]

> > There is absolutely _no_ need whatsoever for Japanese people to
> > learn _any_ bit of English to become excellent programmers.
> 
> Handa-san, could you please give us your perspective on this
> assertion?

I can only speak for Taiwan: Virtually all major computer books by
publishers like O'Reilly are available in Chinese.


    Werner

Re: A system for localizing documentation strings

[Eli Zaretskii]

> Date: Fri, 27 Jul 2007 16:09:44 +0200 (CEST)
> From: Werner LEMBERG <wl@gnu.org>
> Cc: emacs-devel@gnu.org, fusion@mx6.tiki.ne.jp, handa@m17n.org
> 
> 
> > > There is absolutely _no_ need whatsoever for Japanese people to
> > > learn _any_ bit of English to become excellent programmers.
> > 
> > Handa-san, could you please give us your perspective on this
> > assertion?
> 
> I can only speak for Taiwan: Virtually all major computer books by
> publishers like O'Reilly are available in Chinese.

That wasn't what I was asking about.

Re: A system for localizing documentation strings

[David Kastrup]

Jean-Christophe Helary <fusion@mx6.tiki.ne.jp> writes:

> There is absolutely _no_ need whatsoever for Japanese people to
> learn _any_ bit of English to become excellent programmers.

If they are to become excellent programmers contributing to Emacs, I
don't see how they could escape the need.

Code written by a grandmaster of programming, tackling a well-defined
task in the most concise and reasonable way: such code can be read and
understand without reverting to comments.  It is code that speaks to
the programmer in a language of its own, an ancient and lasting, old
and beautiful language built from structure and logic that only can be
found in traces of today's conversation languages.

But the majority of code sits on a rock and grunts.  Being able to
read the comments outlining the creator's intention is essential to
understanding its purpose.  In that code does not differ from art.
The amount of art that is timeless and self-sufficient is limited.

-- 
David Kastrup

Re: A system for localizing documentation strings

[Richard Stallman]

    There is absolutely _no_ need whatsoever for Japanese people to learn  
    _any_ bit of English to become excellent programmers.

You may be right.  I suspect, however, that many of them need to learn
English to read less-popular technical material which is not so widely
translated.

Be that as it may, it doesn't change the situation for us.  No other
language approaches English in its suitability for being read by
programmers (and users) around the world.

It is a GNU standard that names, comments, and manuals be written in
English.

Re: A system for localizing documentation strings

[Eli Zaretskii]

> From: Jean-Christophe Helary <fusion@mx6.tiki.ne.jp>
> Date: Thu, 26 Jul 2007 22:51:54 +0900
> 
> I see 2 types of .el files.
> 
> 1) Those that come with the emacs distribution.
> 
> 2) Those that can be installed in user space.
> 
> The distribution would come with the translated .el files and the  
> updates to the translation could be included in updates of the  
> distribution without requiring the user to build the distribution  
> when new translations come. Since the translations are in the code  
> there is little management necessary (unlike systems where the  
> localizations come in separate files).
> 
> For the .el files in user space, the author would maintain the file's  
> translation and users would be able to compile the file without any  
> specific problem.

It's not that simple, because in practice many .el files that come
with Emacs are maintained by people who also offer the last version
from some Web site.  So in practice, I think we do need to have the
translations on separate files even for bundled packages.  And since
you agree that this should be supported anyway, for the benefit of
unbundled packages, there's no reason not to use this infrastructure
for bundled ones as well.  There are only advantages to this, I don't
see any disadvantages.

> > I think it's much better to have a separate translation file for each
> > .el file.  Such a translation file would be loaded on demand when
> > documentation for symbols defined on that file is requested by the
> > user.  The translation file needs include only the names of the
> > symbols and their doc strings for supported languages.
> 
> I don't think it is a good idea because it would put a bigger load on  
> the coder who would then have to write keys in the code and then  
> values in a separate file.

No, the idea is that Emacs will automatically recognize the strings
for which it needs to look for translations.  The programmers will
need not do anything beyond what they do today: provide a single doc
string.

> > Why do we need the source language?
> 
> Because as soon as we have a system that allows for localization we  
> can expect to have "native" code written and so we'll have to  
> reference the source language that _will_ be different from English.  
> We don't want people who don't master English to produce weird  
> English in their descriptions because the system implies  
> source=English.

I don't see a problem.  Programmers who don't master English can
supply the documentation in their native language, and someone else
will provide the English equivalent to be installed in the .el file
before it is installed; after, all most maintainers who have write
access to the repository have good command of English.  The original
(non-English) documentation gets installed as one of the translated
messages.

> That would require the faulty English string to be  
> rewritten before proceeding to translation.

We do it already anyway: grep the ChangeLog files for the string "doc
fix", and you will see how many fixes like that are installed.

> Also, we can expect to find translators who would be more familiar  
> with one source language than the other. If there is a Japanese  
> equivalent to an English description. I'd rather use the Japanese as  
> source and the English as reference to translate to French for exemple.

You will be able to do that, by looking at the Japanese translation.

> > Also, we should keep in mind that Lisp primitives (those  
> > implemented in
> > C) have their doc strings as C comments, not as C strings.  The
> > infrastructure developed for Emacs l10n should provide solution for
> > the primitives as well, and the solution will have to be different
> > both from your suggestion above and from the traditional gettext-style
> > message catalog.
> 
> Could that part be concieved separatly ?

It will be a separate solution, but it needs to be designed and
implemented together with the one for *.el files, since it doesn't
make sense to have a localized Emacs where all the primitives are
still documented only in English.

Re: A system for localizing documentation strings

[Jean-Christophe Helary]

On 27 juil. 07, at 00:10, Eli Zaretskii wrote:

> It's not that simple, because in practice many .el files that come
> with Emacs are maintained by people who also offer the last version
> from some Web site.  So in practice, I think we do need to have the
> translations on separate files even for bundled packages.  And since
> you agree that this should be supported anyway, for the benefit of
> unbundled packages, there's no reason not to use this infrastructure
> for bundled ones as well.  There are only advantages to this, I don't
> see any disadvantages.

The fact that the code (.el) will only contain the English string  
defeats one of the purposes of the localization.

If I read code and I need to check a separate file all the time to  
see what the French says then I loose a huge amount of time.

I think (but I may be wrong) that you consider anything that is not  
English as "translations" and English as a gold standard.

It is important to _not_ think that way to be able to offer the most  
flexible framework possible.

The "literate programming" style that elisp/emacs has adopted  
_requires_ to be language agnostic as much as possible.

>>> Why do we need the source language?
>>
>> Because as soon as we have a system that allows for localization we
>> can expect to have "native" code written and so we'll have to
>> reference the source language that _will_ be different from English.
>> We don't want people who don't master English to produce weird
>> English in their descriptions because the system implies
>> source=English.
>
> I don't see a problem.  Programmers who don't master English can
> supply the documentation in their native language, and someone else
> will provide the English equivalent to be installed in the .el file
> before it is installed; after, all most maintainers who have write
> access to the repository have good command of English.  The original
> (non-English) documentation gets installed as one of the translated
> messages.

Here again, you see the process as an English based process.

If we are to provide a localization+translation framework we need to  
identify all the languages (including English and its different  
dialects if necessary, that is ISO 639) to provide the most  
flexibility possible. In fact, the biggest mistake of gettext and  
similar l10n processes was to imply that English was to be the gold  
standard of documentation from which everything should be translated.  
Massive l10n activities around the free world have proved that this  
paradigm was overly limiting and recent developments of gettext seem  
to include the possibility to have other languages in source.

The fact that the "main" emacs is centered on English _currently_  
does not say anything about the state of the code in 10 years from now.

>>> Also, we should keep in mind that Lisp primitives (those
>>> implemented in
>>> C) have their doc strings as C comments, not as C strings.  The
>>> infrastructure developed for Emacs l10n should provide solution for
>>> the primitives as well, and the solution will have to be different
>>> both from your suggestion above and from the traditional gettext- 
>>> style
>>> message catalog.
>>
>> Could that part be concieved separatly ?
>
> It will be a separate solution, but it needs to be designed and
> implemented together with the one for *.el files, since it doesn't
> make sense to have a localized Emacs where all the primitives are
> still documented only in English.

I see. Thank you for your comments.


Jean-Christophe Helary

Re: A system for localizing documentation strings

[Jason Rumney]

Jean-Christophe Helary wrote:
> The fact that the code (.el) will only contain the English string
> defeats one of the purposes of the localization.
>
> If I read code and I need to check a separate file all the time to see
> what the French says then I loose a huge amount of time.

Conversely, if I am reading the code for a short function, and cannot
see the documentation at the same time as the code because there are 15
translations in between causing the small function definition to take up
more than one screen, then I lose a huge amount of time.

If you are primarily interested in the documention, you will be using
describe-function anyway. If you are primarily interested in the code,
you don't want an excessive amount of inline documentation getting in
the way.


> I think (but I may be wrong) that you consider anything that is not
> English as "translations" and English as a gold standard.

That is the way it must be for any globally used software. Perhaps as a
Frenchman living in Japan you feel that is unfair, but the fact is that
English is the most widely understood language there is.


> It is important to _not_ think that way to be able to offer the most
> flexible framework possible.

I think flexibility in this regard is important for translators, but not
for end users. So rather than a primative system where translators look
at the code, and write their translations from there, we should come up
with either a special mode to make it easy for translators to enter
translations while looking at the documentation strings in a particular
language (which need not be English), or a set of export and import
functions for interacting with existing tools for translation.

Re: A system for localizing documentation strings

[Jean-Christophe Helary]

On 27 juil. 07, at 01:03, Jason Rumney wrote:

> Jean-Christophe Helary wrote:
>> The fact that the code (.el) will only contain the English string
>> defeats one of the purposes of the localization.
>>
>> If I read code and I need to check a separate file all the time to  
>> see
>> what the French says then I loose a huge amount of time.
>
> Conversely, if I am reading the code for a short function, and cannot
> see the documentation at the same time as the code because there  
> are 15
> translations in between causing the small function definition to  
> take up
> more than one screen, then I lose a huge amount of time.
>
> If you are primarily interested in the documention, you will be using
> describe-function anyway. If you are primarily interested in the code,
> you don't want an excessive amount of inline documentation getting in
> the way.

Very good, so why not propose a device that allows emacs to select a  
language based on the user's preferences for the code display too ?

If the system is properly designed, there are absolutely no reason  
not to be able to propose dynamic updating of the code itself based  
on linguistic preferences.

We need to make emacs easier to use not primarily for one language  
and secondarily for other languages, but for all languages in a smart  
emacs way.

>> I think (but I may be wrong) that you consider anything that is not
>> English as "translations" and English as a gold standard.
>
> That is the way it must be for any globally used software. Perhaps  
> as a
> Frenchman living in Japan you feel that is unfair, but the fact is  
> that
> English is the most widely understood language there is.

I don't care about what is fair or unfair. I am talking about  
technology. Technology can't afford to have linguistic preferences.  
And you can't argue that "defun" is English. It is not.

Globally used software is localized. And there are no "default"  
English strings in the Spanish localization of Word... But that  
localized version serves also as a based for further documentation  
authoring that does _not_ use the original English.

>> It is important to _not_ think that way to be able to offer the most
>> flexible framework possible.
>
> I think flexibility in this regard is important for translators,  
> but not
> for end users. So rather than a primative system where translators  
> look
> at the code, and write their translations from there, we should  
> come up
> with either a special mode to make it easy for translators to enter
> translations while looking at the documentation strings in a  
> particular
> language (which need not be English), or a set of export and import
> functions for interacting with existing tools for translation.

Flexibility is important for coders (who need the documentation to  
code) and for translators.
Coders and translators have equivalent functions in the case at hand:  
they write documentation that the user will access.
The system should put the coders and the translators on an equivalent  
level, hence the system should be language agnostic.
Users will indirectly benefit from that flexibility.

Translators don't have to look at the code. There can be a special  
mode that parses the code according to the translator's preference  
and handles the translated strings for final inclusion in the code.

I wrote earlier:

> (transfun function-name
> 	source-language
> 	target-language
> 	reference-function-name ; should be a list
> 	reference-file ; should be a list)
>
> The function-name declares which function has to be translated
> The source-language declares from which language string the source  
> should be displayed
> The target-language declares to which language the translator is  
> working
> the reference-function-name declares which functions should be  
> taken as reference for the current translation.
> the reference-file declares which files should be taken as  
> reference (ideally, PO compendia, TMX files, CSV files etc)
>
> transfun would be a whole different business since it would  
> actually provide a real dynamic fuzzy matching engine between the  
> source-language strings and the source reference strings. _NOT_  
> something like the "fuzzy" thing gettext provides.
>
> In the case of Template 1 (a function that has already been  
> translated to a number of languages), transfun would just add a  
> line to the documentation function.
>
> In the case of Template 2 (a function that exists in only one  
> language), transfun would also transform Template 2 into Template 1  
> to add the documentation at the proper location. If the doclang  
> function is not documented, transfun asks what argument should  
> doclang have and proceeds.
>
> Now, it would of course be possible to have translation tools  
> support the defun template so that they output the target strings  
> to the correct position.

I see "transfun" (or whatever it would be called) as a real  
translation mode that provides translators with all the modern  
concepts existing in the localization/translation professional tools.  
Not a half-baked "read the code and type the translation" thing.


Jean-Christophe Helary

Re: A system for localizing documentation strings

[Richard Stallman]

    Very good, so why not propose a device that allows emacs to select a  
    language based on the user's preferences for the code display too ?

I could imagine doing this for display, and I wouldn't object to the
feature.  However, it would be very hard for that feature handle the
case of editing the source file.

Re: A system for localizing documentation strings

[Eli Zaretskii]

> From: Jean-Christophe Helary <fusion@mx6.tiki.ne.jp>
> Date: Fri, 27 Jul 2007 00:42:56 +0900
> 
> The fact that the code (.el) will only contain the English string  
> defeats one of the purposes of the localization.

The code is read by programmers, and programmers know English well
enough to cope with that.  More about this later.

> If I read code and I need to check a separate file all the time to  
> see what the French says then I loose a huge amount of time.

I suspect that ``huge amount of time'' is quite an exaggeration.

> I think (but I may be wrong) that you consider anything that is not  
> English as "translations" and English as a gold standard.
> 
> It is important to _not_ think that way to be able to offer the most  
> flexible framework possible.

FWIW, I don't think being dogmatic about these issues will help.  My
considerations are pragmatic: what I suggested is a smaller change,
both in the Emacs infrastructure and in the code.  If we keep adding
requirements that are not necessary to get this feature off the
ground, things will never change for the better, IMO.

So we will have to disagree.  The differing opinions are clear to the
readers, so they can make up their minds; it's no use to continue
reiterating the same arguments time and again.  Eventually, whoever
steps forward to do the actual work will decide on the design and
implementation that he/she likes best, because those who do the job
get to choose the tools and methods.

> The "literate programming" style that elisp/emacs has adopted  
> _requires_ to be language agnostic as much as possible.

Richard will tell for sure, but IMO, Emacs does not try to use the
literate programming paradigm.  A doc string is not documentation of
the code, it is documentation of the _interface_.

What Emacs did, in my view, is provide a way to keep user-level
documentation (user-level, not programmer-level) together with the
code, and provide means for presenting that documentation given the
symbol name.  There are other projects that do similar things (e.g.,
GDB), and they are nowhere near literate programming, either.

> Here again, you see the process as an English based process.

Yes, but only because it's convenient and more practical.

> The fact that the "main" emacs is centered on English _currently_  
> does not say anything about the state of the code in 10 years from now.

Does the fact that we two (and then some), both of us non-native
English speakers, discuss this issue in English -- does this fact tell
you something?  Or do you expect us to converse in Japanese 10 years
from now?  (My Japanese, as Handa-san can witness, is currently
limited to reading Katakana and Hiragana -- but no Kanji! -- at a rate
of 1 word/minute using a dictionary where Japanese words are
transliterated using Latin letters.)

Let's face it: for programmers, English is the most close to the ideal
of the universal language.

Re: A system for localizing documentation strings

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

> Let's face it: for programmers, English is the most close to the
> ideal of the universal language.

Maybe it would raise the geek image if programming language vocabulary
reverted to the old universal Latin language.  It worked for Harry
Potter, didn't it?  And that "old language" is magic is really a
universal theme, from Homeric recitors, the Bible, Druids, over
Tolkien, Le Guin, whoever...

grepo, gripsi, greptum.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Fwd: A system for localizing documentation strings

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

> Also, we should keep in mind that Lisp primitives (those implemented in
> C) have their doc strings as C comments, not as C strings.  The
> infrastructure developed for Emacs l10n should provide solution for
> the primitives as well, and the solution will have to be different
> both from your suggestion above and from the traditional gettext-style
> message catalog.

Why would gettext not work here?

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
PGP key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Fwd: A system for localizing documentation strings

[Eli Zaretskii]

> From: Andreas Schwab <schwab@suse.de>
> Cc: Jean-Christophe Helary <fusion@mx6.tiki.ne.jp>,
> 	emacs-devel@gnu.org
> Date: Thu, 26 Jul 2007 17:36:37 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Also, we should keep in mind that Lisp primitives (those implemented in
> > C) have their doc strings as C comments, not as C strings.  The
> > infrastructure developed for Emacs l10n should provide solution for
> > the primitives as well, and the solution will have to be different
> > both from your suggestion above and from the traditional gettext-style
> > message catalog.
> 
> Why would gettext not work here?

I didn't know gettext supported message strings in comments.  Does it?

If it does, how does the problem of marking the strings with _() in
C/C++ (or rather lack thereof), to get their translations looked up,
get solved?

Re: Fwd: A system for localizing documentation strings

[Andreas Schwab]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> Eli Zaretskii writes:
>
>  > I didn't know gettext supported message strings in comments.  Does
>  > it?
>
> gettext() itself obviously doesn't, since comments aren't executable.
> Extraction tools like xgettext should, they aren't supposed to parse
> the text beyond the _() macro.

It is not required to use xgettext for message extraction, you can just
as easily roll your own.  It should be trivial to extend make-docfile to
emit POT files as well.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
PGP key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Fwd: A system for localizing documentation strings

[Eli Zaretskii]

> From: Andreas Schwab <schwab@suse.de>
> Cc: Eli Zaretskii <eliz@gnu.org>, fusion@mx6.tiki.ne.jp,
> 	emacs-devel@gnu.org
> Date: Fri, 27 Jul 2007 11:53:00 +0200
> 
> "Stephen J. Turnbull" <stephen@xemacs.org> writes:
> 
> > Eli Zaretskii writes:
> >
> >  > I didn't know gettext supported message strings in comments.  Does
> >  > it?
> >
> > gettext() itself obviously doesn't, since comments aren't executable.
> > Extraction tools like xgettext should, they aren't supposed to parse
> > the text beyond the _() macro.
> 
> It is not required to use xgettext for message extraction, you can just
> as easily roll your own.  It should be trivial to extend make-docfile to
> emit POT files as well.

If we don't use xgettext to gather the strings, and if we don't use
the `gettext' function to print the translated strings (because the
original one is not in the code generated from the C sources), then we
are not really using gettext, are we?

Re: Fwd: A system for localizing documentation strings

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

> if we don't use the `gettext' function to print the translated strings

Why not?

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
PGP key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Fwd: A system for localizing documentation strings

[Eli Zaretskii]

> From: Andreas Schwab <schwab@suse.de>
> Date: Fri, 27 Jul 2007 13:43:42 +0200
> Cc: stephen@xemacs.org, fusion@mx6.tiki.ne.jp, emacs-devel@gnu.org
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > if we don't use the `gettext' function to print the translated strings
> 
> Why not?

Sorry, I'm too dumb for your riddles.  If you can spare me more than
two words and explain what design you have in mind, I'd appreciate
that; if not, that is fine with me, too.

Re: Fwd: A system for localizing documentation strings

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andreas Schwab <schwab@suse.de>
>> Date: Fri, 27 Jul 2007 13:43:42 +0200
>> Cc: stephen@xemacs.org, fusion@mx6.tiki.ne.jp, emacs-devel@gnu.org
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > if we don't use the `gettext' function to print the translated strings
>> 
>> Why not?
>
> Sorry, I'm too dumb for your riddles.  If you can spare me more than
> two words and explain what design you have in mind, I'd appreciate
> that; if not, that is fine with me, too.

Please explain why gettext cannot be used.  It is designed for exactly
that job.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
PGP key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Fwd: A system for localizing documentation strings

[Eli Zaretskii]

> From: Andreas Schwab <schwab@suse.de>
> Date: Fri, 27 Jul 2007 14:17:07 +0200
> Cc: emacs-devel@gnu.org
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Andreas Schwab <schwab@suse.de>
> >> Date: Fri, 27 Jul 2007 13:43:42 +0200
> >> Cc: stephen@xemacs.org, fusion@mx6.tiki.ne.jp, emacs-devel@gnu.org
> >> 
> >> Eli Zaretskii <eliz@gnu.org> writes:
> >> 
> >> > if we don't use the `gettext' function to print the translated strings
> >> 
> >> Why not?
> >
> > Sorry, I'm too dumb for your riddles.  If you can spare me more than
> > two words and explain what design you have in mind, I'd appreciate
> > that; if not, that is fine with me, too.
> 
> Please explain why gettext cannot be used.

I didn't say it cannot.  I asked how to use it.

Re: Fwd: A system for localizing documentation strings

[Jan Djärv]

Eli Zaretskii skrev:
>> From: Andreas Schwab <schwab@suse.de>
>> Date: Fri, 27 Jul 2007 13:43:42 +0200
>> Cc: stephen@xemacs.org, fusion@mx6.tiki.ne.jp, emacs-devel@gnu.org
>>
>> Eli Zaretskii <eliz@gnu.org> writes:
>>
>>> if we don't use the `gettext' function to print the translated strings
>> Why not?
> 
> Sorry, I'm too dumb for your riddles.  If you can spare me more than
> two words and explain what design you have in mind, I'd appreciate
> that; if not, that is fine with me, too.

Strings can be read into emacs, like the ones from the DOC file and then 
passed to the gettext functions to be translated.  After all it is just a 
string to string translation.  The strings doesn't need to be present in the 
binary.

	Jan D.

Re: Fwd: A system for localizing documentation strings

[Jason Rumney]

Jan Djärv wrote:
> Strings can be read into emacs, like the ones from the DOC file and
> then passed to the gettext functions to be translated.
It seems a little wasteful to me, to read a string from one docfile just
so we can look it up in another. Why not look it up the doc we want
directly? I don't think we should constrain ourselves with gettext when
it is not the best design for our purpose. Neither the end-users nor
translators care what is under the hood.

Re: Fwd: A system for localizing documentation strings

[Jan Djärv]

Jason Rumney skrev:
> Jan Djärv wrote:
>> Strings can be read into emacs, like the ones from the DOC file and
>> then passed to the gettext functions to be translated.
> It seems a little wasteful to me, to read a string from one docfile just
> so we can look it up in another. Why not look it up the doc we want
> directly?

We could easily make a locale specific DOC-file, like DOC.sv_SE.  But on the 
other hand, looking up strings like this is an interactive action, I'm not 
sure the performance hit is even visible.

 > I don't think we should constrain ourselves with gettext when
> it is not the best design for our purpose.

No, but we must take care of C level strings also.

 > Neither the end-users nor
> translators care what is under the hood.

End users don't care.  As a translator, I do care.  There are existing 
translations in gettext format I'd like to reuse if I was to translate Emacs. 
  Not sure I can take on such a huge job though :-)

	Jan D.

Re: A system for localizing documentation strings

[Jean-Christophe Helary]

On 28 juil. 07, at 00:28, Jan Djärv wrote:

>> translators care what is under the hood.
>
> End users don't care.  As a translator, I do care.  There are  
> existing translations in gettext format I'd like to reuse if I was  
> to translate Emacs.  Not sure I can take on such a huge job though :-)

This is what I suggested already: the translation functions we'd have  
in emacs would allow emacs to be used as a powerful translation  
editor that uses external translation memories for reference (either  
PO compendia or TMX files or CSV data, anything that can easily be  
parsed).

Don't forget that it is not gettext that does the translation.  
gettext only outputs a source file and puts it back in the code once  
translated.

Jean-Christophe Helary

Re: Fwd: A system for localizing documentation strings

[Jason Rumney]

Jan Djärv wrote:
> End users don't care.  As a translator, I do care.  There are existing
> translations in gettext format I'd like to reuse if I was to translate
> Emacs.  Not sure I can take on such a huge job though :-)

As I understand it, there are several aspects to gettext.

xgettext - for extracting strings from code to be translated. This may
suit our needs if we modify it to understand certain Lisp constructs and
the doc-string C comment convention we use. Or it may be easier to
modify make-docfile to handle the doc-strings and other automatically
recognizable strings for translation, and only use xgettext for strings
in the code that need marking up. I don't think it matters which path we
choose, as either way we end up with the same set of strings to give to
translators, and either way, gettext does not suit our needs out of the
box, so the solution will be Emacs specific.

The .PO file format - This is what I think you mean when you say that
translators do care. We can support this independently of whether we use
gettext or not.

The gettext runtime API - for looking up translated strings. This is
what I mean when I say that neither end users nor translators care
whether we use it or not. It seems that the job this does is quite
simple to implement, so I don't think there is a big benefit in sticking
with it where there are obviously better ways of doing it (in the case
of doc-strings at least).

Re: Fwd: A system for localizing documentation strings

[Jan Djärv]

Jason Rumney skrev:
> Jan Djärv wrote:
>> End users don't care.  As a translator, I do care.  There are existing
>> translations in gettext format I'd like to reuse if I was to translate
>> Emacs.  Not sure I can take on such a huge job though :-)
> 
> As I understand it, there are several aspects to gettext.
> 

...

> The .PO file format - This is what I think you mean when you say that
> translators do care. We can support this independently of whether we use
> gettext or not.

Yes, that and the tools that can take similar translations from several 
programs and use them.  Not part of gettext per se.

> 
> The gettext runtime API - for looking up translated strings. This is
> what I mean when I say that neither end users nor translators care
> whether we use it or not. It seems that the job this does is quite
> simple to implement, so I don't think there is a big benefit in sticking
> with it where there are obviously better ways of doing it (in the case
> of doc-strings at least).

The downside is the maintenance burden.

	Jan D.

Re: A system for localizing documentation strings

[Jean-Christophe Helary]

On 28 juil. 07, at 01:09, Jason Rumney wrote:

> Jan Djärv wrote:
>> End users don't care.  As a translator, I do care.  There are  
>> existing
>> translations in gettext format I'd like to reuse if I was to  
>> translate
>> Emacs.  Not sure I can take on such a huge job though :-)
>
> The .PO file format - This is what I think you mean when you say that
> translators do care. We can support this independently of whether  
> we use
> gettext or not.

PO files are made necessary because the applications that are  
localized are not specialized in string export/edition/import. The  
target files in our emacsy process could be generated by emacs  
transparently as the translation occurs. and they could be evaluated  
immediately so that the translator sees the context and corrects  
accordingly.

One of the main issue with localization is that intermediate files  
like PO or XLIFF do not contain enough contextual information. So  
once the (sometimes faulty) translation is loaded it take a lot of  
efforts to correct it. With a localization process within emacs, the  
localized elisp files could be evaluated right away and corrected on  
the spot. That would contribute to greatly improving the quality of  
the translations.

All the non elisp parts of the code should be handled with the proper  
gettext processes. But such parts seem to evolve at a much lower pace  
than the elisp extension parts and would certainly require less  
efforts for localizing them

Jean-Christophe Helary

Re: Fwd: A system for localizing documentation strings

[David Kastrup]

Jan Djärv <jan.h.d@swipnet.se> writes:

> Jason Rumney skrev:
>> Jan Djärv wrote:
>>> Strings can be read into emacs, like the ones from the DOC file and
>>> then passed to the gettext functions to be translated.
>> It seems a little wasteful to me, to read a string from one docfile just
>> so we can look it up in another. Why not look it up the doc we want
>> directly?
>
> We could easily make a locale specific DOC-file, like DOC.sv_SE.
> But on the other hand, looking up strings like this is an
> interactive action, I'm not sure the performance hit is even
> visible.

Well, the DOC strings basically record where to find them in the DOC
file.  This information is unique for each string, and could be used
as a more compact index into the translated string.  _If_ one
recreates the index->string mapping for the translations whenever one
recreates the DOC file.

-- 
David Kastrup

Re: Fwd: A system for localizing documentation strings

[Jason Rumney]

David Kastrup wrote:
> Well, the DOC strings basically record where to find them in the DOC
> file.  This information is unique for each string, and could be used
> as a more compact index into the translated string.  _If_ one
> recreates the index->string mapping for the translations whenever one
> recreates the DOC file.
>   

I was thinking that the function and variable names could be used as the
index key, which would avoid the need to re-index the files.

Re: Fwd: A system for localizing documentation strings

[Kenichi Handa]

In article <je3aza41hf.fsf@sykes.suse.de>, Andreas Schwab <schwab@suse.de> writes:

> "Stephen J. Turnbull" <stephen@xemacs.org> writes:
> > Eli Zaretskii writes:
> >
> >  > I didn't know gettext supported message strings in comments.  Does
> >  > it?
> >
> > gettext() itself obviously doesn't, since comments aren't executable.
> > Extraction tools like xgettext should, they aren't supposed to parse
> > the text beyond the _() macro.

xgettext knows Lisp syntax.  When it is called as this:

% xgettext -L lisp --keyword=_

something like this docstring is extracted correctly.

(defun temp
  (_"Docstring of the function temp.")
  ...)

---
Kenichi Handa
handa@m17n.org

Re: Fwd: A system for localizing documentation strings

[Jan Djärv]

Andreas Schwab skrev:
> "Stephen J. Turnbull" <stephen@xemacs.org> writes:
> 
>> Eli Zaretskii writes:
>>
>>  > I didn't know gettext supported message strings in comments.  Does
>>  > it?
>>
>> gettext() itself obviously doesn't, since comments aren't executable.
>> Extraction tools like xgettext should, they aren't supposed to parse
>> the text beyond the _() macro.
> 
> It is not required to use xgettext for message extraction, you can just
> as easily roll your own.  It should be trivial to extend make-docfile to
> emit POT files as well.

Good thinking.  If we also use the ordianary way to extract strings in the C 
code, what strings would we miss then?  Menus, tooltips?

	Jan D.

Re: Fwd: A system for localizing documentation strings

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Cc: Andreas Schwab <schwab@suse.de>,
>     fusion@mx6.tiki.ne.jp,
>     emacs-devel@gnu.org
> Date: Fri, 27 Jul 2007 11:33:51 +0900
> 
>  > If it does, how does the problem of marking the strings with _() in
>  > C/C++ (or rather lack thereof), to get their translations looked up,
>  > get solved?
> 
> You mark them as dummies so that the extraction tool (typically
> xgettext but Emacs may need a special tool) will put them in the pot
> file.  Then you arrange that they get gettextized at the point of use,
> as usual:
> 
>     /* This work is fiction; any resemblance to real Emacs identifiers
>        is a pure coincidence.  Except `gettext' plays itself. */
>     docstring = gettext (fetch_docstring_from_DOC (index, docfile))
> 
> or
> 
>     (let ((docstring (gettext (documentation-property symbol))))
>       ;; format for display
>       )

Sorry, I'm probably too dumb this morning, because I don't think I
follow.  Or maybe I just don't know enough about the latest
developments in gettext.

Here's how the declaration of an Emacs primitive looks in current Emacs
sources:

   DEFUN ("face-attribute-relative-p", Fface_attribute_relative_p,
	  Sface_attribute_relative_p,
	  2, 2, 0,
	  doc: /* Check whether a face attribute value is relative.
   Specifically, this function returns t if the attribute ATTRIBUTE
   with the value VALUE is relative.

   A relative value is one that doesn't entirely override whatever is
   inherited from another face.  For most possible attributes,
   the only relative value that users see is `unspecified'.
   However, for :height, floating point values are also relative.  */)
	(attribute, value)
	Lisp_Object attribute, value;
   {

As you see, the doc string is just a C comment; after preprocessing,
it is completely gone.  The make-docfile command gathers these
comments and puts them into etc/DOC, but there's no trace of them in
the binary, as you point out.

I would imagine that we can tweak make-docfile to generate a POT file,
as Andreas suggests, but the other part of gettext is to call some
function to replace the original string with its translation, when
that string needs to be presented to the user.  How does one do that,
if the original string is not there to begin with?

What am I missing here?

Re: Fwd: A system for localizing documentation strings

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

> What am I missing here?

documentation is a compiled Lisp function.
(documentation FUNCTION &optional RAW)

Return the documentation string of FUNCTION.
Unless a non-nil second argument RAW is given, the
string is passed through `substitute-command-keys'.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
PGP key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Fwd: A system for localizing documentation strings

[Stephen J. Turnbull]

Eli Zaretskii writes:

 > I would imagine that we can tweak make-docfile to generate a POT file,
 > as Andreas suggests, but the other part of gettext is to call some
 > function to replace the original string with its translation, when
 > that string needs to be presented to the user.  How does one do that,
 > if the original string is not there to begin with?

The original string is in the DOC file, is it not?

 > What am I missing here?

I not sure, but I would guess that it's that gettext() can be called
on the value returned by a function.  gettext() doesn't know or care
whether its argument is a literal string compiled into the binary or
extracted from an external file at runtime.  All that matters is that
neither the process of creating the DOC file nor the process of
extracting a docstring from the DOC file can change one bit of the
string that is used as an index to gettext().

Re: Fwd: A system for localizing documentation strings

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Cc: schwab@suse.de,
>     fusion@mx6.tiki.ne.jp,
>     emacs-devel@gnu.org
> Date: Fri, 27 Jul 2007 21:18:35 +0900
> 
> The original string is in the DOC file, is it not?

Yes.

>  > What am I missing here?
> 
> I not sure, but I would guess that it's that gettext() can be called
> on the value returned by a function.  gettext() doesn't know or care
> whether its argument is a literal string compiled into the binary or
> extracted from an external file at runtime.

Yes, that can be done.

Thanks for showing me the light.

Re: Fwd: A system for localizing documentation strings

[Richard Stallman]

I am sure it will not be hard to make `documentation' handle
translation of doc strings for subrs.  It can be done at the highest
level so it doesn't need to care whether the function is a subr.

Summary (Re: A system for localizing documentation strings)

[Jean-Christophe Helary]

I will try here to make a short summary of the discussions.

I'll start with quoting myself (with a modification in the 3rd  
requirement):

> What we need is provide:
> 1) a way for coders to identify the necessary strings for the  
> translation
> 2) a way for translators to add translated strings "the emacs way"
> 3) a way for users to have the localized strings displayed

We have also identified another requirement:

4) a way for other coders to read the code without having unrelated  
documentation strings coming in the way

This 4th requirement is a very valid one, but the implementation  
proposals (keep English there) seem to not be practical in the end.

In fact, I see the 4th requirement as a subset of the 3rd. After all  
coders are "only" a subgroup of "users".

Since the code comes in elisp, to meet the 4th requirement it would  
"only" be necessary to provide elisp-mode with a specialized "code  
folding" like function for localized strings, enabled by default.  
This function would take into account the user preferences (main  
languages, preferred languages, default language)

This suppose that the strings are all inside the code, but a language  
preference handling function would work just as well for coders if  
the strings were in a separate file.

Regarding the string data, as it is bound to be lisp lists, having  
them inside or outside the code is not a problem and as long as we  
find an elegant way to deal with the 4th requirement it is not even  
relevant to the discussion.



Managing the code/strings -wherever they are- would be handled by the  
2nd requirement: offer a translation function that properly handles  
the string lists.



It looks like emacs primitives would have to be handled differently,  
but as is indicated i the manuals, the primitives are unlikely to  
change very much and it is generally not advised to extend emacs  
through such primitives.

So we can consider strings from primitives an exception to the above  
requirements and handle them with processes that fit better the C  
code structure.


As far as emacs extensions are concerned, and as long as we keep a  
strictly elisp perspective, the above requirements should be enough  
to satisfy any need.


Jean-Christophe Helary

Re: Summary (Re: A system for localizing documentation strings)

[Stefan Monnier]

> This 4th requirement is a very valid one, but the implementation proposals
> (keep English there) seem to not be practical in the end.

I don't see any technical reason to impose English.  AFAICT, what we'd have
is that the text in the elisp file is "the text by default" (could be in
French, English, Klingon, or even just a UUID).  Then when displaying that
text, if the *.mo file contains a translation for that message, display that
translation, otherwise display the default.

So the French .mo wouldn't contain translations for that messages whose
default is already in French, and same thing for the English, Klingon,
UUID, ... "languages".

Of course, the docstrings within the Emacs distribution would most likely
all be English, but authors of third party packages could use any language
they like, just as they can do now.

Isn't that how gettext works already anyway?  What's new here?


        Stefan

Re: Summary (Re: A system for localizing documentation strings)

[David Reitter]

On 27 Jul 2007, at 06:44, Stefan Monnier wrote:
>
> Of course, the docstrings within the Emacs distribution would most  
> likely
> all be English, but authors of third party packages could use any  
> language
> they like, just as they can do now.

Technically yes. But it would make sense to design the system to  
suggest English comments as a minimum standard to promote  
collaboration and code exchange. To give an example: I have been  
including a patch in my distribution, but the discussions around this  
patch are mostly in Japanese, including the change log. It has been  
difficult for me to deal with updates, since I can't read the change  
log. If people publish sources with documentation in Japanese, that  
practically contravenes some very basic ideas of the GPL.

Re: Summary (Re: A system for localizing documentation strings)

[Stefan Monnier]

>> Of course, the docstrings within the Emacs distribution would most  likely
>> all be English, but authors of third party packages could use any
>> language
>> they like, just as they can do now.

> Technically yes.  But it would make sense to design the system to  suggest
> English comments as a minimum standard to promote  collaboration and code
> exchange.

But that's 100% orthogonal to the discussion at hand (i.e. localization of
docstrings).  The exact same problem has existed for years/decades.
So let's forget about this and focus on the localization problem.


        Stefan

Re: Summary (Re: A system for localizing documentation strings)

[Jan Djärv]

Jean-Christophe Helary skrev:
> I will try here to make a short summary of the discussions.
> 
> I'll start with quoting myself (with a modification in the 3rd
> requirement):
> 
>> What we need is provide:
>> 1) a way for coders to identify the necessary strings for the translation

Yes, we need something in elisp that identifies strings to be translated.  For
C gettext mostly uses the macro _(...).  But it is something that gettext must
be able to recognize, and somethin coders easy can add to strings that shall
be translated.

>> 2) a way for translators to add translated strings "the emacs way"

We don't need that.  Translators are organized in teams, and everything uses
gettext.  We do not want a different mechanism here.  Many translators
translates programs they very rarely, if ever, use.  It would be a mistake to
not use gettext.

>> 3) a way for users to have the localized strings displayed

Gettext does that already.  I guess we have to modify print or some low level
C function in Emacs to use gettext.

> 
> We have also identified another requirement:
> 
> 4) a way for other coders to read the code without having unrelated
> documentation strings coming in the way

Translations does not belong in the code, that is why this requirement is
automatically satisfied if gettext is used.

We need a way to identify elisp strings to be translated, maybe gettext must
be modified to do this.  We also need to find where in Emacs gettext shall be
called.  And that is it, but probably hard work nonetheless.

	Jan D.

Re: Summary (Re: A system for localizing documentation strings)

[Jason Rumney]

Jan Djärv wrote:
> We don't need that.  Translators are organized in teams, and everything uses
> gettext.  We do not want a different mechanism here.  Many translators
> translates programs they very rarely, if ever, use.  It would be a mistake to
> not use gettext.
>   

I disagree. It is a mistake to use any system that requires code to be
modified to fit with its conventions before translation can occur. We
cannot guarantee that maintainers of external lisp packages will do the
work that is required. Since we already have mechanisms in place that
extract doc strings, tooltip text and menu text for display, we should
add the necessary hooks at that point to provide translations transparently.

Re: Summary (Re: A system for localizing documentation strings)

[Jan Djärv]

Jason Rumney skrev:
> Jan Djärv wrote:
>> We don't need that.  Translators are organized in teams, and everything uses
>> gettext.  We do not want a different mechanism here.  Many translators
>> translates programs they very rarely, if ever, use.  It would be a mistake to
>> not use gettext.
>>   
> 
> I disagree. It is a mistake to use any system that requires code to be
> modified to fit with its conventions before translation can occur. We
> cannot guarantee that maintainers of external lisp packages will do the
> work that is required. Since we already have mechanisms in place that
> extract doc strings, tooltip text and menu text for display, we should
> add the necessary hooks at that point to provide translations transparently.

We can not guarantee that maintainers of external lisp packages will do
anything, regardless of what mechanism Emacs use.  But the point is that Emacs
is unlikely to be translated by the translation teams that are already outh
there if a completely new mechanism that differs much from gettext is adopted.

	Jan D.

Re: Summary (Re: A system for localizing documentation strings)

[David Kastrup]

Jan Djärv <jan.h.d@swipnet.se> writes:

> Jason Rumney skrev:
>> Jan Djärv wrote:

>>> We don't need that.  Translators are organized in teams, and
>>> everything uses gettext.  We do not want a different mechanism
>>> here.  Many translators translates programs they very rarely, if
>>> ever, use.  It would be a mistake to not use gettext.
>>>   
>> 
>> I disagree. It is a mistake to use any system that requires code to
>> be modified to fit with its conventions before translation can
>> occur. We cannot guarantee that maintainers of external lisp
>> packages will do the work that is required. Since we already have
>> mechanisms in place that extract doc strings, tooltip text and menu
>> text for display, we should add the necessary hooks at that point
>> to provide translations transparently.
>
> We can not guarantee that maintainers of external lisp packages will
> do anything, regardless of what mechanism Emacs use.  But the point
> is that Emacs is unlikely to be translated by the translation teams
> that are already outh there if a completely new mechanism that
> differs much from gettext is adopted.

It must be stressed that translating Emacs is a Herculanean effort.
There is no way it can be achieved by Emacs developers alone: we
really have to rely on non-coding-involved translation teams here, and
it will still be a real beast to do.

-- 
David Kastrup

Re: Summary (Re: A system for localizing documentation strings)

[Jan Djärv]

David Kastrup skrev:

> It must be stressed that translating Emacs is a Herculanean effort.
> There is no way it can be achieved by Emacs developers alone: we
> really have to rely on non-coding-involved translation teams here, and
> it will still be a real beast to do.
> 

I wonder if it is larger than gcc.  Gcc 4.2.0 has 6489 strings for
translation.  Some languages do have them all translated.

	Jan D.

Re: Summary (Re: A system for localizing documentation strings)

[David Kastrup]

Jan Djärv <jan.h.d@swipnet.se> writes:

> David Kastrup skrev:
>
>> It must be stressed that translating Emacs is a Herculanean effort.
>> There is no way it can be achieved by Emacs developers alone: we
>> really have to rely on non-coding-involved translation teams here, and
>> it will still be a real beast to do.
>> 
>
> I wonder if it is larger than gcc.  Gcc 4.2.0 has 6489 strings for
> translation.  Some languages do have them all translated.

Gcc has a job to do that is not focused on talking with the user.  I
should be quite surprised if we are even talking the same order of
magnitude here.

-- 
David Kastrup

Re: Summary (Re: A system for localizing documentation strings)

[Eli Zaretskii]

> Date: Fri, 27 Jul 2007 11:39:58 +0200
> From: =?ISO-8859-1?Q?Jan_Dj=E4rv?= <jan.h.d@swipnet.se>
> Cc: Jason Rumney <jasonr@gnu.org>,
> 	Jean-Christophe Helary <fusion@mx6.tiki.ne.jp>, emacs-devel@gnu.org
> 
> David Kastrup skrev:
> 
> > It must be stressed that translating Emacs is a Herculanean effort.
> > There is no way it can be achieved by Emacs developers alone: we
> > really have to rely on non-coding-involved translation teams here, and
> > it will still be a real beast to do.
> > 
> 
> I wonder if it is larger than gcc.  Gcc 4.2.0 has 6489 strings for
> translation.

I just tried this command in the top-level Emacs directory:

  grep -hR -A1 "^(def" lisp --include="*.el" | grep -v "^--" | grep "^  \""

and got 30594 lines of output.  And this is only the doc strings;
there are built-ins defined in C, prompts, literal `(message FOO)'
messages, calls to `signal' and `error', etc.

So yes, Emacs is _definitely_ larger than GCC.  Much larger, almost by
an order of magnitude.

Re: Summary (Re: A system for localizing documentation strings)

[Jan Djärv]

Eli Zaretskii skrev:

>> I wonder if it is larger than gcc.  Gcc 4.2.0 has 6489 strings for
>> translation.
> 
> I just tried this command in the top-level Emacs directory:
> 
>   grep -hR -A1 "^(def" lisp --include="*.el" | grep -v "^--" | grep "^  \""
> 
> and got 30594 lines of output.  And this is only the doc strings;
> there are built-ins defined in C, prompts, literal `(message FOO)'
> messages, calls to `signal' and `error', etc.
> 
> So yes, Emacs is _definitely_ larger than GCC.  Much larger, almost by
> an order of magnitude.

Thanks for the data.  This might be a problem in itself.  I don't kow how 
large a gettext compiled file can be with reasonable performance.

	Jan D.

Re: Summary (Re: A system for localizing documentation strings)

[Eli Zaretskii]

> Date: Fri, 27 Jul 2007 11:04:25 +0200
> From: =?ISO-8859-1?Q?Jan_Dj=E4rv?= <jan.h.d@swipnet.se>
> Cc: Jean-Christophe Helary <fusion@mx6.tiki.ne.jp>, emacs-devel@gnu.org
> 
> the point is that Emacs is unlikely to be translated by the
> translation teams that are already outh there if a completely new
> mechanism that differs much from gettext is adopted.

What mechanism are we talking about?

Translators take a FOO.pot file, which is just a specially-formatted
text file, copy it to a LANG.po file (where LANG is the target
language acronym), and add a translated string to every original
(usually English) string.  The result is still a text file, which is
submitted to a central repository.  There's no mechanism involved
here, except text editing and email (to submit the result).

Or did I misunderstand you?

Re: Summary (Re: A system for localizing documentation strings)

[Jan Djärv]

Eli Zaretskii skrev:
>> Date: Fri, 27 Jul 2007 11:04:25 +0200
>> From: =?ISO-8859-1?Q?Jan_Dj=E4rv?= <jan.h.d@swipnet.se>
>> Cc: Jean-Christophe Helary <fusion@mx6.tiki.ne.jp>, emacs-devel@gnu.org
>>
>> the point is that Emacs is unlikely to be translated by the
>> translation teams that are already outh there if a completely new
>> mechanism that differs much from gettext is adopted.
> 
> What mechanism are we talking about?
> 
> Translators take a FOO.pot file, which is just a specially-formatted
> text file, copy it to a LANG.po file (where LANG is the target
> language acronym), and add a translated string to every original
> (usually English) string.  The result is still a text file, which is
> submitted to a central repository.  There's no mechanism involved
> here, except text editing and email (to submit the result).
> 
> Or did I misunderstand you?

I thought the suggestion was do do something in elisp that is independent from 
gettext.  That is the "mechanism".  Sorry for being unclear.

	Jan D.

Re: Summary (Re: A system for localizing documentation strings)

[Jean-Christophe Helary]

On 27 juil. 07, at 18:04, Jan Djärv wrote:

> Jason Rumney skrev:
>> Jan Djärv wrote:
>>> We don't need that.  Translators are organized in teams, and  
>>> everything uses
>>> gettext.  We do not want a different mechanism here.  Many  
>>> translators
>>> translates programs they very rarely, if ever, use.  It would be  
>>> a mistake to
>>> not use gettext.
>>>
>>
>> I disagree. It is a mistake to use any system that requires code  
>> to be
>> modified to fit with its conventions before translation can occur. We
>> cannot guarantee that maintainers of external lisp packages will  
>> do the
>> work that is required. Since we already have mechanisms in place that
>> extract doc strings, tooltip text and menu text for display, we  
>> should
>> add the necessary hooks at that point to provide translations  
>> transparently.
>
> We can not guarantee that maintainers of external lisp packages  
> will do
> anything, regardless of what mechanism Emacs use.  But the point is  
> that Emacs
> is unlikely to be translated by the translation teams that are  
> already outh
> there if a completely new mechanism that differs much from gettext  
> is adopted.

This is preposterous. There are plenty of localization teams that  
handle many different processes, source files, translation tools.

Gettext is far from being the only way to localization. And here I  
mean localization as menu items + interface messages + documentation,  
where documentation weights much more than all the other strings  
combined.

Besides, gettext may be good for short sentences (specifically menu  
items and short messages) but documentation translation is a totally  
different process.

Obviously the self-documentation feature that is advertised as one of  
emacs's specificity should be handled in a very emacsy way that fits  
with its paradigm.


Jean-Christophe Helary

Re: Summary (Re: A system for localizing documentation strings)

[Jan Djärv]

Jean-Christophe Helary skrev:
> 
> On 27 juil. 07, at 18:04, Jan Djärv wrote:
> 
>> Jason Rumney skrev:
>>> Jan Djärv wrote:
>>
>> We can not guarantee that maintainers of external lisp packages will do
>> anything, regardless of what mechanism Emacs use.  But the point is 
>> that Emacs
>> is unlikely to be translated by the translation teams that are already 
>> outh
>> there if a completely new mechanism that differs much from gettext is 
>> adopted.
> 
> This is preposterous. There are plenty of localization teams that handle 
> many different processes, source files, translation tools.

I'm sure they are, but within GNU?

> 
> Gettext is far from being the only way to localization. And here I mean 
> localization as menu items + interface messages + documentation, where 
> documentation weights much more than all the other strings combined.
> 
> Besides, gettext may be good for short sentences (specifically menu 
> items and short messages) but documentation translation is a totally 
> different process.

There are longer strings in applications that use gettext than many function 
documentation strings in elisp.  As for manuals, that is another matter.

> 
> Obviously the self-documentation feature that is advertised as one of 
> emacs's specificity should be handled in a very emacsy way that fits 
> with its paradigm.

That is not at all obvious.  For example, the self documentation relies on C 
code to extract the documentation, not lispy at all.

	Jan D.

Re: Summary (Re: A system for localizing documentation strings)

[Jean-Christophe Helary]

On 28 juil. 07, at 00:16, Jan Djärv wrote:
> Jean-Christophe Helary skrev:
>> On 27 juil. 07, at 18:04, Jan Djärv wrote:
>>> Jason Rumney skrev:
>>>> Jan Djärv wrote:
>>>
>>> We can not guarantee that maintainers of external lisp packages  
>>> will do
>>> anything, regardless of what mechanism Emacs use.  But the point  
>>> is that Emacs
>>> is unlikely to be translated by the translation teams that are  
>>> already outh
>>> there if a completely new mechanism that differs much from  
>>> gettext is adopted.
>> This is preposterous. There are plenty of localization teams that  
>> handle many different processes, source files, translation tools.
>
> I'm sure they are, but within GNU?

What does GNU propose to localization teams ?

A process to deliver an intermediary translation format (PO) and a  
process to put that back inside the code.

The teams must do everything else (ie the actual localization)  
themselves in conditions that sometimes have nothing to do with GNU  
tools.

As I wrote earlier, most localized applications are not applications  
specialized in handling strings. This specificity of emacs should be  
used to ease the whole process.

>> Gettext is far from being the only way to localization. And here I  
>> mean localization as menu items + interface messages +  
>> documentation, where documentation weights much more than all the  
>> other strings combined.
>> Besides, gettext may be good for short sentences (specifically  
>> menu items and short messages) but documentation translation is a  
>> totally different process.
>
> There are longer strings in applications that use gettext than many  
> function documentation strings in elisp.  As for manuals, that is  
> another matter.

Still, most projects will use gettext-like processes even for manuals  
(see po4a from Debian) because the gettext centered localization  
processes have created an ecology of software that prefers to use a  
localization file format _even_ for manual translation work.

The translation functions that would come with the localization  
framework wold address such issues and contribute to facilitate the  
actual localization/translation process that is today barely more  
than text editing in specialized (PO aware) editors. A very  
undeserving task.


Jean-Christophe Helary

Re: Summary (Re: A system for localizing documentation strings)

[David Kastrup]

Jan Djärv <jan.h.d@swipnet.se> writes:

> Translations does not belong in the code, that is why this
> requirement is automatically satisfied if gettext is used.
>
> We need a way to identify elisp strings to be translated, maybe
> gettext must be modified to do this.  We also need to find where in
> Emacs gettext shall be called.  And that is it, but probably hard
> work nonetheless.

It is not just that because not just strings are involved: Emacs is
permeated with English.  All the function names are based on English,
M-x is used transparently, menus are identified in the code by name,
apropos of course works on English function names, customization
groups are displayed by symbol, keyboard events are described by
symbols and so on.

Emacs has no layers separating the user from the programmer.  It is,
after all, the greatest prototyping platform for editing ever
invented.  It is quite illusionary to separate "English" from
"international" layers.  But that does not mean that _explanations_
could not be done in English, or that menu entries and help texts
could not be made in English.  Or even customization group names.  But
what about "talking variable names" which are converted into
capitalized space-separated sequences in the customize groups?
Translate or not?  If yes, how does one get to know the actual
variable name?

-- 
David Kastrup

Re: Summary (Re: A system for localizing documentation strings)

[Jan Djärv]

David Kastrup skrev:
> Jan Djärv <jan.h.d@swipnet.se> writes:
> 
>> Translations does not belong in the code, that is why this
>> requirement is automatically satisfied if gettext is used.
>>
>> We need a way to identify elisp strings to be translated, maybe
>> gettext must be modified to do this.  We also need to find where in
>> Emacs gettext shall be called.  And that is it, but probably hard
>> work nonetheless.
> 
> It is not just that because not just strings are involved: Emacs is
> permeated with English.  All the function names are based on English,
> M-x is used transparently, menus are identified in the code by name,
> apropos of course works on English function names, customization
> groups are displayed by symbol, keyboard events are described by
> symbols and so on.
> 
> Emacs has no layers separating the user from the programmer.  It is,
> after all, the greatest prototyping platform for editing ever
> invented.  It is quite illusionary to separate "English" from
> "international" layers.  But that does not mean that _explanations_
> could not be done in English, or that menu entries and help texts
> could not be made in English.  Or even customization group names.  But
> what about "talking variable names" which are converted into
> capitalized space-separated sequences in the customize groups?
> Translate or not?  If yes, how does one get to know the actual
> variable name?
> 

I'd say no.  Once upon a time there was some swedish MS Excel version that had
all functions translated to swedih.  That didn't work so well when trying to
open a spreadsheet from the US...

If we wan't to be able to use an international .emacs everywhere, variables
must have just one name globally.  Even if some people find it sad, english
seems to be the best.

	Jan D.

Re: Summary (Re: A system for localizing documentation strings)

[David Kastrup]

Jan Djärv <jan.h.d@swipnet.se> writes:

> David Kastrup skrev:
>
>> Emacs has no layers separating the user from the programmer.  It
>> is, after all, the greatest prototyping platform for editing ever
>> invented.  It is quite illusionary to separate "English" from
>> "international" layers.  But that does not mean that _explanations_
>> could not be done in English, or that menu entries and help texts
>> could not be made in English.  Or even customization group names.
>> But what about "talking variable names" which are converted into
>> capitalized space-separated sequences in the customize groups?
>> Translate or not?  If yes, how does one get to know the actual
>> variable name?
>> 
>
> I'd say no.  Once upon a time there was some swedish MS Excel
> version that had all functions translated to swedih.  That didn't
> work so well when trying to open a spreadsheet from the US...
>
> If we wan't to be able to use an international .emacs everywhere,
> variables must have just one name globally.  Even if some people
> find it sad, english seems to be the best.

I was not talking about changing the variable names, just their
circumscription.  If we now do
M-x customize-variable RET confirm-kill-emacs RET
we get

    This is a customization buffer.
    `Raised' text indicates buttons; type RET or click mouse-1 to actuate one.
    Editing a setting changes only the text in the buffer.
    Use the setting's State button to set it or save changes in it.
    Saving a change normally works by editing your Emacs init file.
    See Custom file for information on how to save in a different file.

    See Help for more information.

    Operate on all settings in this buffer that are not marked HIDDEN:
     Set for Current Session Save for Future Sessions
     Undo Edits Reset to Saved Erase Customization   Finish

    Confirm Kill Emacs: Hide Value Value Menu Don't confirm
       State: STANDARD.

    How to ask for confirmation when leaving Emacs. Hide Rest
    If nil, the default, don't ask at all.  If the value is non-nil, it should
    be a predicate function such as `yes-or-no-p'.
    Parent groups: Convenience

It does not talk about "confirm-kill-emacs" but rather about "Confirm
Kill Emacs".  One could make this "Nachfrage beim Verlassen von Emacs
(confirm-kill-emacs):".  Also the parent group is called "Convenience"
rather than the actual existing 'convenience group.  Again, one could
make this into "Übergeordnete Gruppe: Praktisches (convenience)".  The
fallback would be capitalization and hyphen removal, but if a language
(including English) offered a better term, then one could show that
and give the "real" name in parentheses.

-- 
David Kastrup

Re: Summary (Re: A system for localizing documentation strings)

[Jan Djärv]

David Kastrup skrev:

> 
> It does not talk about "confirm-kill-emacs" but rather about "Confirm
> Kill Emacs".  One could make this "Nachfrage beim Verlassen von Emacs
> (confirm-kill-emacs):".  Also the parent group is called "Convenience"
> rather than the actual existing 'convenience group.  Again, one could
> make this into "Übergeordnete Gruppe: Praktisches (convenience)".  The
> fallback would be capitalization and hyphen removal, but if a language
> (including English) offered a better term, then one could show that
> and give the "real" name in parentheses.
> 

That is a good idea.

	Jan D.

Re: Summary (Re: A system for localizing documentation strings)

[Jean-Christophe Helary]

On 27 juil. 07, at 18:08, Jan Djärv wrote:
> I'd say no.  Once upon a time there was some swedish MS Excel  
> version that had
> all functions translated to swedih.  That didn't work so well when  
> trying to
> open a spreadsheet from the US...

This is obviously a design problem and has little to do with  
localization proper.

The localized functions must map to some language agnostic key in the  
code and then any localization of the function names (as is the case  
with OpenOffice.org right now) becomes a non problematic task.

But I am not arguing that current function names in emacs should be  
localized. For me they are _already_ arbitrary strings that bear  
little resemblance to English (defun anybody ?).

Jean-Christophe Helary

Re: Summary (Re: A system for localizing documentation strings)

[Lennart Borgman (gmail)]

Jean-Christophe Helary wrote:
> But I am not arguing that current function names in emacs should be 
> localized. For me they are _already_ arbitrary strings that bear little 
> resemblance to English (defun anybody ?).


I think it is a mistake to believe that function names have no meaning. 
"defun" for example is a mnemonic for define function (of course).

I do not know if you are aware of the complexity of code writing? 
Sometimes it is simple, but other times it get very complex and one 
thing that must be avoided then is to make the immediate complexity 
(what you have to keep in your mind when searching and thinking) larger. 
Otherwise the process will be much slower, because you have to resort of 
managing the complexity since it will exceed your immediate memory.

Clear function names are IMO very important to keep complexity low. 
Keeping the doc strings with the functions and variables is another.

Re: Summary (Re: A system for localizing documentation strings)

[Eli Zaretskii]

> Date: Fri, 27 Jul 2007 11:08:43 +0200
> From: =?ISO-8859-1?Q?Jan_Dj=E4rv?= <jan.h.d@swipnet.se>
> Cc: Jean-Christophe Helary <fusion@mx6.tiki.ne.jp>, emacs-devel@gnu.org
> 
> I'd say no.  Once upon a time there was some swedish MS Excel version that had
> all functions translated to swedih.  That didn't work so well when trying to
> open a spreadsheet from the US...
> 
> If we wan't to be able to use an international .emacs everywhere, variables
> must have just one name globally.  Even if some people find it sad, english
> seems to be the best.

Don't forget that Emacs can display any string as some other string.
So, for example, "Help" could be displayed as "Hilfe" in the German
localized version.  There's more than one Emacs feature we could use
for that (text properties, overlays, display tables).

As for "M-x foo", we could have `intern' look up a translated string
first, before actually interning the symbol.

But I agree that localizing the symbol names should be the last piece
of Emacs localization.

Re: Summary (Re: A system for localizing documentation strings)

[Richard Stallman]

I can envision someday connecting M-x to a feature that records
translated command names.  We might want to do that after we have
handled translation for messages and menu items.

But let's not discuss it now.  Translating doc strings and messages is
a big enough first step.  Let's take that step, then worry about the
next step.

Re: Summary (Re: A system for localizing documentation strings)

[Andreas Schwab]

Jan Djärv <jan.h.d@swipnet.se> writes:

> Jean-Christophe Helary skrev:
>> I will try here to make a short summary of the discussions.
>> 
>> I'll start with quoting myself (with a modification in the 3rd
>> requirement):
>> 
>>> What we need is provide:
>>> 1) a way for coders to identify the necessary strings for the translation
>
> Yes, we need something in elisp that identifies strings to be translated.  For
> C gettext mostly uses the macro _(...).  But it is something that gettext must
> be able to recognize, and somethin coders easy can add to strings that shall
> be translated.

make-docfile already has everything we need, except for the POT output.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
PGP key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Summary (Re: A system for localizing documentation strings)

[Jean-Christophe Helary]

On 27 juil. 07, at 17:03, Jan Djärv wrote:

> Jean-Christophe Helary skrev:
>> I will try here to make a short summary of the discussions.
>>
>> I'll start with quoting myself (with a modification in the 3rd
>> requirement):
>>
>>> What we need is provide:
>>> 1) a way for coders to identify the necessary strings for the  
>>> translation
>
> Yes, we need something in elisp that identifies strings to be  
> translated.  For
> C gettext mostly uses the macro _(...).  But it is something that  
> gettext must
> be able to recognize, and somethin coders easy can add to strings  
> that shall
> be translated.

gettext has nothing to do with emacs. emacs code, as elisp, is data  
and must be considered as such by specific elisp functions created  
for localization.

There is absolutely no point in using a non lisp way to extend emacs  
(because we are talking about extending emacs here so that any elisp  
system can be localized within emacs).

>>> 2) a way for translators to add translated strings "the emacs way"
>
> We don't need that.  Translators are organized in teams, and  
> everything uses
> gettext.  We do not want a different mechanism here.  Many translators
> translates programs they very rarely, if ever, use.  It would be a  
> mistake to
> not use gettext.

Of course we need that. We are trying to extend emacs so that it  
provides a localization framework for elisp systems.

I'd say that it would be much more straightforward to use an elisp  
system to do that than to externalize the tasks to gettext.

Besides, translator teams use different tools for different tasks and  
there are plenty of software that does not use gettext because  
translating PO files is a pain in the butt: none of the tools at hand  
offer modern mechanisms to efficiently leverage translation compendia  
dynamically.

>>> 3) a way for users to have the localized strings displayed
>
> Gettext does that already.  I guess we have to modify print or some  
> low level
> C function in Emacs to use gettext.

And there are ways to do that from within emacs with elisp. Why  
externalize things when we have an extendable framework that is  
yearning to be extended ?

>> We have also identified another requirement:
>>
>> 4) a way for other coders to read the code without having unrelated
>> documentation strings coming in the way
>
> Translations does not belong in the code, that is why this  
> requirement is
> automatically satisfied if gettext is used.
>
> We need a way to identify elisp strings to be translated, maybe  
> gettext must
> be modified to do this.  We also need to find where in Emacs  
> gettext shall be
> called.  And that is it, but probably hard work nonetheless.

What are the odds that such tasks would be easier to handle in elisp ?


Jean-Christophe Helary

Re: Summary (Re: A system for localizing documentation strings)

[Eli Zaretskii]

> From: Jean-Christophe Helary <fusion@mx6.tiki.ne.jp>
> Date: Fri, 27 Jul 2007 20:15:16 +0900
> 
> gettext has nothing to do with emacs. emacs code, as elisp, is data  
> and must be considered as such by specific elisp functions created  
> for localization.

But the Emacs Lisp interpreter is a C program, and as such, it could
very well use gettext, at least in principle.

> There is absolutely no point in using a non lisp way to extend emacs  
> (because we are talking about extending emacs here so that any elisp  
> system can be localized within emacs).

While the way to mark translatable strings should be Lisp-friendly,
there's nothing to prevent us from using the translation in low-level
C code.  Why should a Lisp programmer care if the marked string gets
magically translated without any need for Lisp-level changes?

Re: Summary (Re: A system for localizing documentation strings)

[Jason Rumney]

Jean-Christophe Helary wrote:
> I'd say that it would be much more straightforward to use an elisp
> system to do that than to externalize the tasks to gettext.

I wouldn't completely dismiss using gettext in Emacs. There are several
sources of strings that should be translated in Emacs. Some of these
(documentation, menu names, tooltips) we can already reliably identify
and extract for translation without changing anything in the code, so it
doesn't make sense to change our code to fit in with gettext 's way of
doing things for these. But there are many other strings that get
displayed to the user (prompts and messages in the minibuffer), where an
automated process cannot reliably decide whether or not they should be
translated without human intervention. In these cases we will have to
mark them somehow, so we might as well use existing methods such as
those used by gettext (which does already specify a convention for
elispas well as C).

Re: Summary (Re: A system for localizing documentation strings)

[Jean-Christophe Helary]

On 27 juil. 07, at 20:59, Jason Rumney wrote:

> Jean-Christophe Helary wrote:
>> I'd say that it would be much more straightforward to use an elisp
>> system to do that than to externalize the tasks to gettext.
>
> I wouldn't completely dismiss using gettext in Emacs. There are  
> several
> sources of strings that should be translated in Emacs. Some of these
> (documentation, menu names, tooltips) we can already reliably identify
> and extract for translation without changing anything in the code,  
> so it
> doesn't make sense to change our code to fit in with gettext 's way of
> doing things for these.

I am not dismissing gettext.

I am saying that an elisp system to localize the elisp "part" of  
emacs will also be useful for the localization of other elisp  
systems. What needs to be done with gettext (the primitives parts)  
should be done with gettext.


Jean-Christophe Helary

Re: Summary (Re: A system for localizing documentation strings)

[David Kastrup]

Jean-Christophe Helary <fusion@mx6.tiki.ne.jp> writes:

> On 27 juil. 07, at 20:59, Jason Rumney wrote:
>
>> Jean-Christophe Helary wrote:
>>> I'd say that it would be much more straightforward to use an elisp
>>> system to do that than to externalize the tasks to gettext.
>>
>> I wouldn't completely dismiss using gettext in Emacs. There are
>> several
>> sources of strings that should be translated in Emacs. Some of these
>> (documentation, menu names, tooltips) we can already reliably identify
>> and extract for translation without changing anything in the code,
>> so it
>> doesn't make sense to change our code to fit in with gettext 's way of
>> doing things for these.
>
> I am not dismissing gettext.
>
> I am saying that an elisp system to localize the elisp "part" of
> emacs will also be useful for the localization of other elisp
> systems. What needs to be done with gettext (the primitives parts)
> should be done with gettext.

Other elisp systems?  Reminds me of the story of a linguist
researching some island dialect, who had to wrap up his work before
the single person still speaking it was gone.  And one nagging doubt
was that he suspected his sole study subject to speak the language
incorrectly.

Are there any other systems speaking Elisp?

-- 
David Kastrup

Re: Summary (Re: A system for localizing documentation strings)

[Jean-Christophe Helary]

On 27 juil. 07, at 22:42, David Kastrup wrote:

> Jean-Christophe Helary <fusion@mx6.tiki.ne.jp> writes:
>
>> On 27 juil. 07, at 20:59, Jason Rumney wrote:
>>
>>> Jean-Christophe Helary wrote:

>> I am not dismissing gettext.
>>
>> I am saying that an elisp system to localize the elisp "part" of
>> emacs will also be useful for the localization of other elisp
>> systems. What needs to be done with gettext (the primitives parts)
>> should be done with gettext.
>
> Other elisp systems?  Reminds me of the story of a linguist
> researching some island dialect, who had to wrap up his work before
> the single person still speaking it was gone.  And one nagging doubt
> was that he suspected his sole study subject to speak the language
> incorrectly.
>
> Are there any other systems speaking Elisp?

You misunderstood what I wrote.

I see emacs as an elisp environment. Seeing elisp code as emacs  
extensions is one way to conceptualize that. Seeing elisp code as  
systems that run in emacs is a different way to see that.

When I use nxml, I consider I run an XML editing software within emacs.

That is what I mean by "elisp systems".

If elisp code is written to localize elisp code, I think it will be a  
giant step forward: we'll have a full (hopefully) computer aided  
translation tool available directly from emacs !!! This would be  
fantastic !


Jean-Christophe Helary

Re: Summary (Re: A system for localizing documentation strings)

[Richard Stallman]

    Other elisp systems?  Reminds me of the story of a linguist
    researching some island dialect, who had to wrap up his work before
    the single person still speaking it was gone.  And one nagging doubt
    was that he suspected his sole study subject to speak the language
    incorrectly.

That might sound funny, but it is a real problem (sad to say).

Re: Summary (Re: A system for localizing documentation strings)

[David Kastrup]

Richard Stallman <rms@gnu.org> writes:

>     Other elisp systems?  Reminds me of the story of a linguist
>     researching some island dialect, who had to wrap up his work before
>     the single person still speaking it was gone.  And one nagging doubt
>     was that he suspected his sole study subject to speak the language
>     incorrectly.
>
> That might sound funny, but it is a real problem (sad to say).

I do have the suspicion at times that Emacs does not speak correct
Elisp.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Summary (Re: A system for localizing documentation strings)

[Jan Djärv]

Jean-Christophe Helary skrev:
> 
> On 27 juil. 07, at 17:03, Jan Djärv wrote:
> 
> 
> gettext has nothing to do with emacs. emacs code, as elisp, is data and 
> must be considered as such by specific elisp functions created for 
> localization.

Why?  AFAIK, gettext works with other interpreted languages also.

> 
> There is absolutely no point in using a non lisp way to extend emacs 
> (because we are talking about extending emacs here so that any elisp 
> system can be localized within emacs).

Any?  Are there more than one?


> 
>>>> 2) a way for translators to add translated strings "the emacs way"
>>
>> We don't need that.  Translators are organized in teams, and 
>> everything uses
>> gettext.  We do not want a different mechanism here.  Many translators
>> translates programs they very rarely, if ever, use.  It would be a 
>> mistake to
>> not use gettext.
> 
> Of course we need that. We are trying to extend emacs so that it 
> provides a localization framework for elisp systems.
> 

No, we are trying to localize Emacs.  Or that is what I am talking about.


> I'd say that it would be much more straightforward to use an elisp 
> system to do that than to externalize the tasks to gettext.

Elisp is only part of Emacs.  C code must also be handeled.

> 
> Besides, translator teams use different tools for different tasks and 
> there are plenty of software that does not use gettext because 
> translating PO files is a pain in the butt: none of the tools at hand 
> offer modern mechanisms to efficiently leverage translation compendia 
> dynamically.

There are many tools out there, I haven't checked them all.  But I know some 
use translation dictionaries.  AFAIK, GNU uses gettext only for localization.

> 
>>>> 3) a way for users to have the localized strings displayed
>>
>> Gettext does that already.  I guess we have to modify print or some 
>> low level
>> C function in Emacs to use gettext.
> 
> And there are ways to do that from within emacs with elisp. Why 
> externalize things when we have an extendable framework that is yearning 
> to be extended ?

Why reinvent the wheel when there is a localization framework in use already?
There are not ways to this transparently from elisp.  C code also print out 
lisp expressions.

> 
>>> We have also identified another requirement:
>>>
>>> 4) a way for other coders to read the code without having unrelated
>>> documentation strings coming in the way
>>
>> Translations does not belong in the code, that is why this requirement is
>> automatically satisfied if gettext is used.
>>
>> We need a way to identify elisp strings to be translated, maybe 
>> gettext must
>> be modified to do this.  We also need to find where in Emacs gettext 
>> shall be
>> called.  And that is it, but probably hard work nonetheless.
> 
> What are the odds that such tasks would be easier to handle in elisp ?

Zero as make-docfile already can do much of the work.

	Jan D.

Re: Summary (Re: A system for localizing documentation strings)

[Jean-Christophe Helary]

On 28 juil. 07, at 00:07, Jan Djärv wrote:

> Jean-Christophe Helary skrev:
>> On 27 juil. 07, at 17:03, Jan Djärv wrote:
>> gettext has nothing to do with emacs. emacs code, as elisp, is  
>> data and must be considered as such by specific elisp functions  
>> created for localization.
>
> Why?  AFAIK, gettext works with other interpreted languages also.

This is besides the point.

gettext offer tools to extract the strings, deliver PO(T) files,  
merge the strings, compile the PO and then the result has to be built  
against a l10n library to handle the MO live.

There is nothing interactive in the process and all the extraction/ 
conversion/merging/compilation etc are handled by an external process  
because most of the localized application are not specialized at  
handling strings in general. Some are but that is not relevant.

Emacs is a highly specialized device that handles strings. Why bother  
externalize the processes when they could be handled in a cleaner way  
from within emacs, in a few elisp functions, including translation of  
course.

>> There is absolutely no point in using a non lisp way to extend  
>> emacs (because we are talking about extending emacs here so that  
>> any elisp system can be localized within emacs).
>
> Any?  Are there more than one?

Any extension to emacs is an elisp system. Emacs is an environment  
that runs that system. One can see such systems as emacs extensions  
but developers can (and do) chose elisp and emacs because their code  
will then be highly portable. It is the fabled "write once run  
everywhere", in lisp.

>>>>> 2) a way for translators to add translated strings "the emacs way"
>>>
>>> We don't need that.  Translators are organized in teams, and  
>>> everything uses
>>> gettext.  We do not want a different mechanism here.  Many  
>>> translators
>>> translates programs they very rarely, if ever, use.  It would be  
>>> a mistake to
>>> not use gettext.
>> Of course we need that. We are trying to extend emacs so that it  
>> provides a localization framework for elisp systems.
>
> No, we are trying to localize Emacs.  Or that is what I am talking  
> about.

Sure, but the thread did not begin specifically as a suggestion to  
localize emacs, that was a side effect.

The purpose of the original mail (it was not clear and I apologize  
for that) was the following:

1) by providing a localization framework in elisp for elisp  
(including translation aiding functions)
2) we open the door to emacs localization and documentation  
translation from  within emacs _and_ in elisp
3) that contributes to the adoption of emacs in native language  
communities
4) to the adoption of elisp in the same communities
5) to introducing localizers and translators to the fantastic tool  
that is emacs
6) to providing a more general translation framework in emacs that  
can be used for localizing and translating in a number of formats  
besides for elisp code

etc.

>> I'd say that it would be much more straightforward to use an elisp  
>> system to do that than to externalize the tasks to gettext.
>
> Elisp is only part of Emacs.  C code must also be handeled.

Sure, I already replied to that. Parts that can't be handled with  
elisp would _have_ to be handled in gettext. But the C parts in emacs  
are few in number and do not evolve as fast as the elisp parts.

>> Besides, translator teams use different tools for different tasks  
>> and there are plenty of software that does not use gettext because  
>> translating PO files is a pain in the butt: none of the tools at  
>> hand offer modern mechanisms to efficiently leverage translation  
>> compendia dynamically.
>
> There are many tools out there, I haven't checked them all.  But I  
> know some use translation dictionaries.  AFAIK, GNU uses gettext  
> only for localization.

It uses gettext to extract strings and merge back the translations.  
Localization is handled by the PO editors, yet another  
externalization that could be handled by emacs if a proper  
translation function set were provided (I already mentioned dynamic  
string matching to reference files, unlike what gettext does with  
fuzzies, which is static and removes the original context).

gettext is only the mechanism that puts everything back together. It  
is robust but not conceived with the translator in mind at all.

>>>>> 3) a way for users to have the localized strings displayed
>>>
>>> Gettext does that already.  I guess we have to modify print or  
>>> some low level
>>> C function in Emacs to use gettext.
>> And there are ways to do that from within emacs with elisp. Why  
>> externalize things when we have an extendable framework that is  
>> yearning to be extended ?
>
> Why reinvent the wheel when there is a localization framework in  
> use already?

Because the already existing framework is all-encompassing why we  
only need a small subset of its functions. Emacs can identify the  
translatable strings, it can display them, it can register their  
translations it can compile the data and evaluate it dynamically. Why  
bother using an external process when everything can be done more  
elegantly internally ?

Jean-Christophe Helary

Re: Summary (Re: A system for localizing documentation strings)

[Andreas Schwab]

Jean-Christophe Helary <fusion@mx6.tiki.ne.jp> writes:

> On 28 juil. 07, at 00:07, Jan Djärv wrote:
>
>> Jean-Christophe Helary skrev:
>>> On 27 juil. 07, at 17:03, Jan Djärv wrote:
>>> gettext has nothing to do with emacs. emacs code, as elisp, is data and
>>> must be considered as such by specific elisp functions  created for
>>> localization.
>>
>> Why?  AFAIK, gettext works with other interpreted languages also.
>
> This is besides the point.
>
> gettext offer tools to extract the strings, deliver PO(T) files, merge the
> strings, compile the PO

This is the development part.

> and then the result has to be built  against a
> l10n library to handle the MO live.

This is the runtime part.  You should distinguish these two functions
because they can be handled separately.  Even if the latter part (the
runtime) is handled by the gettext library, there is no requirement that
the first part is done using the tools provided by the gettext bundle.

> Emacs is a highly specialized device that handles strings. Why bother
> externalize the processes when they could be handled in a cleaner way
> from within emacs, in a few elisp functions, including translation of
> course.

The use of the gettext runtime is no externalization of processes.
Emacs would just link against libintl (or use the functions in glibc)
and provide a lisp binding.  This just as lispy as anything Emacs
provides.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
PGP key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Summary (Re: A system for localizing documentation strings)

[Jean-Christophe Helary]

On 28 juil. 07, at 01:11, Andreas Schwab wrote:

> Jean-Christophe Helary <fusion@mx6.tiki.ne.jp> writes:
>
>> On 28 juil. 07, at 00:07, Jan Djärv wrote:
>>
>>> Jean-Christophe Helary skrev:
>>>> On 27 juil. 07, at 17:03, Jan Djärv wrote:
>>>> gettext has nothing to do with emacs. emacs code, as elisp, is  
>>>> data and
>>>> must be considered as such by specific elisp functions  created for
>>>> localization.
>>>
>>> Why?  AFAIK, gettext works with other interpreted languages also.
>>
>> This is besides the point.
>>
>> gettext offer tools to extract the strings, deliver PO(T) files,  
>> merge the
>> strings, compile the PO
>
> This is the development part.

This could be easily handled with elisp code for all the elisp parts  
of emacs which would also contribute to provide a simple way to  
localize non emacs related elisp files. I.e. that could be an emacs  
extension for localizing elisp code.

The emacs core parts in C should of course be handled in gettext.


Jean-Christophe Helary

Re: Summary (Re: A system for localizing documentation strings)

[Richard Stallman]

    We don't need that.  Translators are organized in teams, and everything uses
    gettext.  We do not want a different mechanism here.  Many translators
    translates programs they very rarely, if ever, use.  It would be a mistake to
    not use gettext.

I already explained the problem in using gettext from Emacs: how to
decide WHICH .po files to use.  In Emacs it won't work to use just
one, or even a few dozen.  I think this is a hard problem, if we want
results that are usable in practice.

Maybe it can be solved; I encourage people to look for a solution.
However, to argue for use of gettext without solving that problem is
not useful.

One _possible_ advantage in using gettext is that it might be more
efficient in speed and time.  I don't know how it works, but it might
work by mapping directly .po files into memory (using less space and
time than reading it into Lisp), and it might have optimized lookup
within the file.

With gettext it is probably possible (whether or not currently
implemented) to delete a .po file from memory if it goes unused for a
while.  That would be useful in Emacs; we could treat the in-core .po
files as a cache for the ones on disk.  By contrast, this would be very hard
if we read in the translations as Lisp data.

Re: Summary (Re: A system for localizing documentation strings)

[Richard Stallman]

My decision is that we will keep only English doc strings in the
source files of the code of Emacs.  Translations to other languages
will be kept in separate files.

This follows the GNU Project standard which I established some 15
years ago for good reasons that are just as valid today.

Please consider this question settled.

Re: Summary (Re: A system for localizing documentation strings)

[Jean-Christophe Helary]

On 28 juil. 07, at 01:24, Richard Stallman wrote:

> My decision is that we will keep only English doc strings in the
> source files of the code of Emacs.  Translations to other languages
> will be kept in separate files.
>
> This follows the GNU Project standard which I established some 15
> years ago for good reasons that are just as valid today.
>
> Please consider this question settled.

Do you consider making elisp-mode locale aware a possibility ?


Jean-Christophe Helary

Re: Summary (Re: A system for localizing documentation strings)

[Richard Stallman]

    Do you consider making elisp-mode locale aware a possibility ?

I do not understand what "making elisp-mode locale aware" means
in concrete terms.

Re: Summary (Re: A system for localizing documentation strings)

[Jean-Christophe Helary]

On 29 juil. 07, at 11:22, Richard Stallman wrote:

>     Do you consider making elisp-mode locale aware a possibility ?
>
> I do not understand what "making elisp-mode locale aware" means
> in concrete terms.

In concrete terms it means that if the locale language in English and  
the elisp code documentation is in English elisp-mode will display  
the English documentation.

If the locale language is Japanese and the elisp code documentation  
is in Japanese or in English but with a Japanese translation  
available then elisp-mode will display the Japanese.


Jean-Christophe Helary

Re: Summary (Re: A system for localizing documentation strings)

[M Jared Finder]

Jean-Christophe Helary wrote:
> 
> On 29 juil. 07, at 11:22, Richard Stallman wrote:
> 
>>     Do you consider making elisp-mode locale aware a possibility ?
>>
>> I do not understand what "making elisp-mode locale aware" means
>> in concrete terms.
> 
> In concrete terms it means that if the locale language in English and 
> the elisp code documentation is in English elisp-mode will display the 
> English documentation.
> 
> If the locale language is Japanese and the elisp code documentation is 
> in Japanese or in English but with a Japanese translation available then 
> elisp-mode will display the Japanese.

There is no `elisp-mode'.  I believe you mean eldoc-mode:

eldoc-mode is an interactive autoloaded Lisp function in `eldoc'.
(eldoc-mode &optional arg)

Toggle ElDoc mode on or off.
In ElDoc mode, the echo area displays information about a
function or variable in the text where point is.  If point is
on a documented variable, it displays the first line of that
variable's doc string.  Otherwise it displays the argument list
of the function called in the expression point is on.

With prefix arg, turn ElDoc mode on if and only if arg is positive.


I have been watching this discussion from afar, and would LOVE to see a 
translated GNU Emacs appear, especially for User Commands, User 
Variables, and Error Messages.

   -- MJF

Re: Summary (Re: A system for localizing documentation strings)

[Jean-Christophe Helary]

>>> I do not understand what "making elisp-mode locale aware" means
>>> in concrete terms.

>> In concrete terms it means that if the locale language in English  
>> and the elisp code documentation is in English elisp-mode will  
>> display the English documentation.
>> If the locale language is Japanese and the elisp code  
>> documentation is in Japanese or in English but with a Japanese  
>> translation available then elisp-mode will display the Japanese.

> There is no `elisp-mode'.  I believe you mean eldoc-mode:

Thank you for the precision.

I mean the mode that is triggered when .el files are opened.


Jean-Christophe Helary

Re: Summary (Re: A system for localizing documentation strings)

[Richard Stallman]

    In concrete terms it means that if the locale language in English and  
    the elisp code documentation is in English elisp-mode will display  
    the English documentation.

Do you mean the major mode, Lisp mode?

    If the locale language is Japanese and the elisp code documentation  
    is in Japanese or in English but with a Japanese translation  
    available then elisp-mode will display the Japanese.

Do you mean that the translated documentation strings would
appear in the Lisp source files in place of the actual doc
string text?

If so, what would it do if you try to edit the file?

Re: Summary (Re: A system for localizing documentation strings)

[Jean-Christophe Helary]

On 31 juil. 07, at 01:43, Richard Stallman wrote:

>     In concrete terms it means that if the locale language in  
> English and
>     the elisp code documentation is in English elisp-mode will display
>     the English documentation.
>
> Do you mean the major mode, Lisp mode?

This is what I mean.

>     If the locale language is Japanese and the elisp code  
> documentation
>     is in Japanese or in English but with a Japanese translation
>     available then elisp-mode will display the Japanese.
>
> Do you mean that the translated documentation strings would
> appear in the Lisp source files in place of the actual doc
> string text?

This is what I mean.

> If so, what would it do if you try to edit the file?

The edit would modify the file.

If the documentation had been in the oiginal English then an edit  
would modify that original English. If the displayed documentation  
were to be the Japanese string, then the Japanese data would be  
edited (to eventually be later saved).

Although I am sure I don't understand all the implications of what I  
just wrote and all the implication of using gettext for the whole  
localization process, my impression is that by using the compiled  
data provided by gettext we considerably limit ourselves regarding  
how much flexibility we can have with the textual data provided by  
the translations.

This is one of the reason why at first I imagined a kind of  
specialized "code folding" device to display only the locale relevant  
translated documentation strings.

Jean-Christophe Helary

Re: Summary (Re: A system for localizing documentation strings)

[Richard Stallman]

    > If so, what would it do if you try to edit the file?

    The edit would modify the file.

    If the documentation had been in the oiginal English then an edit  
    would modify that original English. If the displayed documentation  
    were to be the Japanese string, then the Japanese data would be  
    edited (to eventually be later saved).

That would be very hard to do.

Re: Summary (Re: A system for localizing documentation strings)

[Jean-Christophe Helary]

On 31 juil. 07, at 12:38, Richard Stallman wrote:

>> If so, what would it do if you try to edit the file?
>
>     The edit would modify the file.
>
>     If the documentation had been in the oiginal English then an edit
>     would modify that original English. If the displayed documentation
>     were to be the Japanese string, then the Japanese data would be
>     edited (to eventually be later saved).
>
> That would be very hard to do.

I understand that.

If automatic display based on locale information is difficult, do you  
think a command that simply overwrites the documentation with an  
available translation would be easier to implement ? Instead of  
overwriting the strings (which may not be a practical solution), the  
command would simply re-generate the code file from the translation  
database stored strings.

In the end, the user would have a choice between the default English  
and any translation available, and even once "overwritten" the files  
could be reverted to their original English state.

What matters is that the coder has easy access to the translation  
database in a way or another _during_ the coding process without  
having to refer to a separate file.


Jean-Christophe Helary

Re: Summary (Re: A system for localizing documentation strings)

[Jan Djärv]

Jean-Christophe Helary skrev:
> 
> On 31 juil. 07, at 12:38, Richard Stallman wrote:
> 
>>> If so, what would it do if you try to edit the file?
>>
>>     The edit would modify the file.
>>
>>     If the documentation had been in the oiginal English then an edit
>>     would modify that original English. If the displayed documentation
>>     were to be the Japanese string, then the Japanese data would be
>>     edited (to eventually be later saved).
>>
>> That would be very hard to do.
> 
> I understand that.
> 
> If automatic display based on locale information is difficult, do you 
> think a command that simply overwrites the documentation with an 
> available translation would be easier to implement ? Instead of 
> overwriting the strings (which may not be a practical solution), the 
> command would simply re-generate the code file from the translation 
> database stored strings.
> 
> In the end, the user would have a choice between the default English and 
> any translation available, and even once "overwritten" the files could 
> be reverted to their original English state.
> 
> What matters is that the coder has easy access to the translation 
> database in a way or another _during_ the coding process without having 
> to refer to a separate file.
> 

I'm not sure how this would work.  Say I have the swedish locale.  If I edit 
an english doc string, how does Emacs know if I'm translating it to swedish or 
correcting the english one?

	Jan D.

Re: Summary (Re: A system for localizing documentation strings)

[Jean-Christophe Helary]

>> If automatic display based on locale information is difficult, do  
>> you think a command that simply overwrites the documentation with  
>> an available translation would be easier to implement ? Instead of  
>> overwriting the strings (which may not be a practical solution),  
>> the command would simply re-generate the code file from the  
>> translation database stored strings.
>> In the end, the user would have a choice between the default  
>> English and any translation available, and even once "overwritten"  
>> the files could be reverted to their original English state.
>> What matters is that the coder has easy access to the translation  
>> database in a way or another _during_ the coding process without  
>> having to refer to a separate file.
>
> I'm not sure how this would work.  Say I have the swedish locale.   
> If I edit an english doc string, how does Emacs know if I'm  
> translating it to swedish or correcting the english one?

Considering the discussions that far, translating a file would be  
done either externally or in a special translation mode.

As long as you are working on the file is this new "language aware"  
lisp major mode, whatever you do there is editing and not translating.

Will this editing affect the translation database ? It should not. As  
far as emacs is concerned, if you overwrite the original file with  
your modifications then the available translated strings are still  
here and the modified ones won't match anything in the DB. If you  
save your file under a new name it loses the connection to the l10n  
database (unless the db is a global one that is made to apply to any  
string but that is a different story).


Jean-Christophe Helary

Re: Summary (Re: A system for localizing documentation strings)

[Richard Stallman]

Let's focus on the basic job: implementing doc string translation
for help commands.

After that is working, those who are interested can try to make
Emacs Lisp mode display the translated doc strings.  I think it won't
work well, but if you think I'm wrong, feel free to prove it.

However, we should not let discussion of future steps distract us
from the first step.  Would someone like to try implementing
translation of doc strings for help commands?

Re: Summary (Re: A system for localizing documentation strings)

[Davis Herring]

> What matters is that the coder has easy access to the translation
> database in a way or another _during_ the coding process without
> having to refer to a separate file.

Would it count as close enough to "without having to refer" if access to
the separate file were automated?  Go to function (perhaps with
`find-function'), then invoke a hypothetical M-x find-doc-translations
(which would prompt for a function but default to the one at point) to go
to its bank of docstrings?  This would be completely non-intrusive, and
trivial to implement.

Davis

-- 
This product is sold by volume, not by mass.  If it appears too dense or
too sparse, it is because mass-energy conversion has occurred during
shipping.

Re: Summary (Re: A system for localizing documentation strings)

[Jean-Christophe Helary]

>> What matters is that the coder has easy access to the translation
>> database in a way or another _during_ the coding process without
>> having to refer to a separate file.
>
> Would it count as close enough to "without having to refer" if  
> access to
> the separate file were automated?  Go to function (perhaps with
> `find-function'), then invoke a hypothetical M-x find-doc-translations
> (which would prompt for a function but default to the one at point)  
> to go
> to its bank of docstrings?  This would be completely non-intrusive,  
> and
> trivial to implement.

Where would be the translated string in that case ?

The idea is that the coder should be able to have the translated  
string in a position equivalent, or similar, to the English string.


Jean-Christophe Helary

Re: Summary (Re: A system for localizing documentation strings)

[Miles Bader]

Jean-Christophe Helary <fusion@mx6.tiki.ne.jp> writes:
>> Would it count as close enough to "without having to refer" if access
>> to the separate file were automated?  Go to function (perhaps with
>> `find-function'), then invoke a hypothetical M-x
>> find-doc-translations (which would prompt for a function but default
>> to the one at point) to go to its bank of docstrings?  This would be
>> completely non-intrusive, and trivial to implement.
>
> Where would be the translated string in that case ?
>
> The idea is that the coder should be able to have the translated  string
> in a position equivalent, or similar, to the English string.

Why?  Who cares as long as it's _convenient_...

Such a could:

  (1) display the translated string in the echo-area when you move the
  cursor over the "key" string in the source code, and

  (2) offer a simple keybinding which pops the cursor into the
  translation file in another window, with point positioned on the
  translated string (similar to how `find-function-at-point' works).

That would make both source browsing and translation work easy, and
would be easier to implement than some kind of automatic overlay
mechanism.

-Miles

-- 
`...the Soviet Union was sliding in to an economic collapse so comprehensive
 that in the end its factories produced not goods but bads: finished products
 less valuable than the raw materials they were made from.'  [The Economist]

Re: Fwd: A system for localizing documentation strings

[Richard Stallman]

    (defun function-name (arguments...)
	    (docfun
		    EN "optional-documentation in EN..."
		    FR "documentation optionnelle en FR...")
	    (interactive argument-passing-info)     ; optional
	    body...)

That format is not very good, because it won't let you add a language
without editing the source of the function.  It would be much better
to add doc translations separate from the function source code.

Contrast this case with advice: advising a function is confusing
because you can't see in the function's definition what it really
does.  It's totally different for doc strings: not being able to see
the translations of the doc string won't hide what the function's code
does.

Meanwhile, we know that various independent groups will want to create
doc translations for various languages, and we want to facilitate their
releasing these translations separately.

A construct such as

(defun-doc-translations FUNCTION LANG TEXT LANG TEXT...)

is the way to do it.  It should work to have multiple
defun-doc-translations forms for the same FUNCTION.  The last
one to define any given LANG should prevail.

(Of course, we want defvar-doc-translations and a few others,
plus perhaps one general one for misc. kinds of doc strings.)

However, to make this function smoothly at the level of the entire
Emacs, we need to make it possible to autoload these out of various
files.  It would be a pain to have to load ALL the Spanish
translations for ALL the files of Emacs (including the hundreds you
never use) just to get any of them.

    The other requirement would be for a lisp file to declare where its
    translations are, and for the user to be able to add more later:

    (provide-translations 'example-feature '(ja es fr) "example-translations")

That might be a good approach--especially if this form does not have
to be IN the source file it applies to.

To make this work, we need to go from a definition to the feature
provided by its source file.  That can be done thru the load-history
information, most of the time.  But not all source files provide a
feature.  We should have some other way to figure out the feature
that belongs to a certain file.

It would be good if the file of translations could have a string
analogous to an autoload cookie to give this information.  That way,
these files could be scanned very quickly so as to produce a list that
tells Emacs which file to load whenever it does want a translation.

    The distribution would come with the translated .el files and the  
    updates to the translation could be included in updates of the  
    distribution without requiring the user to build the distribution  
    when new translations come.

No, this is not maintainable.  We can't afford to put all the
translations of the doc string of `last' into the defun for `last'.
There may someday be dozens of them.

And even if Emacs does have translations dozens of languages, people
will keep supporting more languages.  We do not want them to have to
edit the code source just to install their translations.

Therefore we do not want the translations in the function definition!

    Because as soon as we have a system that allows for localization we  
    can expect to have "native" code written and so we'll have to  
    reference the source language that _will_ be different from English.  

We aim to discourage that.  The primary language of GNU implementation
is English.  All functions should be documented in English, in addition
to whatever other languages are supported.

If someone's English doc string is not well written, that's as if the
code were not well written.  Someone will suggest an improvement.

Jason had it right:

    That is the way it must be for any globally used software. Perhaps as a
    Frenchman living in Japan you feel that is unfair, but the fact is that
    English is the most widely understood language there is.

Re: A system for localizing documentation strings

[Jean-Christophe Helary]

On 27 juil. 07, at 14:04, Richard Stallman wrote:

>     Because as soon as we have a system that allows for  
> localization we
>     can expect to have "native" code written and so we'll have to
>     reference the source language that _will_ be different from  
> English.
>
> We aim to discourage that.  The primary language of GNU implementation
> is English.  All functions should be documented in English, in  
> addition
> to whatever other languages are supported.
>
> If someone's English doc string is not well written, that's as if the
> code were not well written.  Someone will suggest an improvement.

I am sorry if what I wrote was unclear.

I understand that GNU software is English based, just like the GPL  
license is valid only in English.

I have no specific problem with that.


What I intended to propose was a generic system to localize elisp  
files/elisp systems.

Such a system would directly benefit emacs, of course, since emacs is  
the main provider of elisp code. And since emacs is English based it  
makes perfect sense to provide a system that is backward compatible  
with the current "English only" system.


But the system should not "hardcode" English=default into its  
behavior (ie by not identifying English with a language code etc). It  
is very conceivable that emacs extensions are created for a specific  
"market" that corresponds to a language community and the  
localization mechanism should provide developers with a way to use  
their language as their system's default.

> Jason had it right:
>
>     That is the way it must be for any globally used software.  
> Perhaps as a
>     Frenchman living in Japan you feel that is unfair, but the fact  
> is that
>     English is the most widely understood language there is.

Which assertion do you refer to ?

1) Since emacs started from an English speaking community (as a lot  
of major software packages, like MS Office for example) I suppose the  
practical way to deal with localization does involve starting from  
English. I have nothing against that.

2) As a Frenchman living in Japan and in Japanese, and having a fair  
command of English I don't think it is unfair at all, but in a number  
of cases there are practical considerations that make English  
centered processes a waste of time: some elisp code may be designed  
with a relatively limited scope and technically forcing the use of  
English because elisp=emacs=GNU seems a little far fetched.

3) Such a postulate shows little understanding of what "freedom"  
means, especially in the context of technology. It also shows little  
understanding of the necessity of native language computer literacy  
activities to reach the goals set by the FSF in particular.

Jean-Christophe Helary