bug#25443: Unnecessary building of dictionary files

bug#25443: Unnecessary building of dictionary files

[Phillip Lord]

The Emacs build generates a number of .el files from other sources as
part of leim. Some of these sources files appear to be no longer
distributed from their original source and have not been functionally
updated for over a decade.

These generation steps therefore complicate the build and the bootstrap
process, as they generated before emacs is dumped. These steps could,
therefore, be removed, with the currently generated .el files, becoming
the source.

Examples of files in question would be: files in MISC-DICT (such as
CTLau.html), ECDICT.tit, Punct-b5.tit.

Some of the dictionaries are still available from their original
sources, such as, for example SKK-JISYO.L.

We'd need to carry out a full audit before actually doing anything.

Phil

bug#25443: Unnecessary building of dictionary files

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > These generation steps therefore complicate the build and the bootstrap
  > process, as they generated before emacs is dumped. These steps could,
  > therefore, be removed, with the currently generated .el files, becoming
  > the source.

We can't do that.  We have an obligation to distribute the real source.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

bug#25443: Unnecessary building of dictionary files

[Phillip Lord]

Richard Stallman <rms@gnu.org> writes:
>   > These generation steps therefore complicate the build and the bootstrap
>   > process, as they generated before emacs is dumped. These steps could,
>   > therefore, be removed, with the currently generated .el files, becoming
>   > the source.
>
> We can't do that.  We have an obligation to distribute the real source.

Richard

I find myself in the bizarre position of suggesting that you haven't
read the GPL properly...

The "source" files in question are no longer distributed. So, any future
changes would have to come from the emacs developers. If we decided that
the .el files are the place to make these changes, then, they become the
"preferred form for modification". They become "the real source". I see
nothing in the GPL that prevents us from making this change.

I think the situation is analogous to "nato-alphabet" in
"morse.el". It's been taken from somewhere, but there is no source.
This is, of course, assuming that the source is copyrightable -- I have
my doubts. Looks like non-copyrightable data to me.

Unless you mean "an obligation" in some sense other than "because of the
GPL".

Phil

bug#25443: Unnecessary building of dictionary files

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I find myself in the bizarre position of suggesting that you haven't
  > read the GPL properly...

Before asserting that someone experienced is totally confused,
I suggest you double-check your own reasoning.

  > The "source" files in question are no longer distributed.

It is premature to put quotes around "source".  You are arguing that
we have the option of making something else the real source code,
but we haven't done so yet.  At present, those files are the source code.

You say they are "no longer distributed".  That can't be entirely true,
since the GNU Project does still distribute them.

Would you please lay out the facts that you are talking about?
Pressuring for a particular decision is not being helpful.
Please show us the situation clearly so we can see what is right to do.

  > The "source" files in question are no longer distributed. So, any future
  > changes would have to come from the emacs developers. If we decided that
  > the .el files are the place to make these changes, then, they become the
  > "preferred form for modification". They become "the real source". I see
  > nothing in the GPL that prevents us from making this change.

In principle, that is a possible option.  Whether it is a good idea
is another question.

There is no grave problem here.  Let's see what Handa-san says
about this technical question.

  > Unless you mean "an obligation" in some sense other than "because of the
  > GPL".

Anyone distributing software has a moral obligation to distribute the
real souce code.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

bug#25443: Unnecessary building of dictionary files

[Phillip Lord]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > I find myself in the bizarre position of suggesting that you haven't
>   > read the GPL properly...
>
> Before asserting that someone experienced is totally confused,
> I suggest you double-check your own reasoning.

It was just a light-hearted aside. I did say "bizarre", and didn't say
"totally confused".


>   > The "source" files in question are no longer distributed.
>
> It is premature to put quotes around "source".  You are arguing that
> we have the option of making something else the real source code,
> but we haven't done so yet.  At present, those files are the source code.
>
> You say they are "no longer distributed".  That can't be entirely true,
> since the GNU Project does still distribute them.

Indeed. They are distributed as part of Emacs, but not available from
their referenced source.

> Would you please lay out the facts that you are talking about?
> Pressuring for a particular decision is not being helpful.
> Please show us the situation clearly so we can see what is right to do.


Yes. A file (CTLau.html) which looks like this....


<html>
<head>
<title>CTLau : Hanzi input by Cantonese transcription (Lau style)
</title>
<meta http-equiv="content-type" content="text/html; charset=gb2312">
</head>
<body bgcolor=#ffc0c0><pre>
#
# CTLau.html : Hanzi (GB2312-80) input dictionary for Sidney Lau's

# (STUFF REMOVED)

# Revision History:
# This file was originally converted in 1993 from a hanzi input file
# based on Fung Fung Lee's Cantonese phonetic transcription scheme
# proposed in 1988 (http://umunhum.stanford.edu/~lee/chicomp/CPS.html).
# This file was released under the terms of GNU GPL on May 14, 2001.
#
#<hr>
A 阿啊鸦呀丫哑亚吖氩锕
AAI 埃挨嗌哎唉隘嗳
AAN 晏俺
(STUFF REMOVED)
</body>
</html>

Is reformated to look like this (CTLau.el):

;; Quail package ‘chinese-ctlau’
;;   Generated by the command ‘miscdic-convert’
(STUFF REMOVED)
(quail-define-rules
("a" "阿啊鸦呀丫哑亚吖氩锕")
("aai" "埃挨嗌哎唉隘嗳")
("aan" "晏俺")
(STUFF REMOVED)


AFAICT, we cannot update CTLau.html from a downstream source any
longer. And either way, it has not been updated since 2001, except for a
change to formatted in the licence comments.

>   > The "source" files in question are no longer distributed. So, any future
>   > changes would have to come from the emacs developers. If we decided that
>   > the .el files are the place to make these changes, then, they become the
>   > "preferred form for modification". They become "the real source". I see
>   > nothing in the GPL that prevents us from making this change.
>
> In principle, that is a possible option.  Whether it is a good idea
> is another question.
>
> There is no grave problem here.  Let's see what Handa-san says
> about this technical question.


The reason that this came up is that the generation of CTLau.html has to
be performed by "bootstrap-emacs", as the lisp needs to be available
before "emacs" is dumped. Keeping the work performed by bootstrap-emacs
as simple as possible is a useful aim.


>   > Unless you mean "an obligation" in some sense other than "because of the
>   > GPL".
>
> Anyone distributing software has a moral obligation to distribute the
> real souce code.

I agree with this position, but I see no moral or ethical issue in
changing our minds about what the real source code is. I have raised a
technical question; I think it only needs a technical answer. If it is
going to cause any issues for maintainence of leim, then, of course,
it's a bad idea.

Phil

bug#25443: Unnecessary building of dictionary files

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > AFAICT, we cannot update CTLau.html from a downstream source any
  > longer. And either way, it has not been updated since 2001, except for a
  > change to formatted in the licence comments.

If this is because no one has wanted to change it, then it is neither
here nor there.

  > I agree with this position, but I see no moral or ethical issue in
  > changing our minds about what the real source code is. I have raised a
  > technical question; I think it only needs a technical answer.

I agree with that formulation of the issue.

If we have changes to make, would we rather make them on CTLau.html,
or on the processed Elisp-format file?

Has anyone talked with Handa-san?
When did we last hear from him?


-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

bug#25443: Unnecessary building of dictionary files

[Phillip Lord]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > AFAICT, we cannot update CTLau.html from a downstream source any
>   > longer. And either way, it has not been updated since 2001, except for a
>   > change to formatted in the licence comments.
>
> If this is because no one has wanted to change it, then it is neither
> here nor there.

It cannot be updated from the original source, because that source has
gone.


>   > I agree with this position, but I see no moral or ethical issue in
>   > changing our minds about what the real source code is. I have raised a
>   > technical question; I think it only needs a technical answer.
>
> I agree with that formulation of the issue.
>
> If we have changes to make, would we rather make them on CTLau.html,
> or on the processed Elisp-format file?
>
> Has anyone talked with Handa-san?
> When did we last hear from him?

Answering in order: I think either would work, no, and I don't know.

I will contact him now.

Phil

bug#25443: Unnecessary building of dictionary files

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > If this is because no one has wanted to change it, then it is neither
  > > here nor there.

  > It cannot be updated from the original source, because that source has
  > gone.

That's an abrupt change of topic, not related to what I said.  But
never mind.  I would comment on what you said, but its meaning is
vague.

What, concretely, do you mean by the "original source"?
What does it mean to say it "has gone"?

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

bug#25443: Unnecessary building of dictionary files

[Phillip Lord]

On Thu, January 19, 2017 10:40 pm, Richard Stallman wrote:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>
>>> If this is because no one has wanted to change it, then it is neither
>>>  here nor there.
>
>> It cannot be updated from the original source, because that source has
>> gone.
>
> That's an abrupt change of topic, not related to what I said.  But
> never mind.

I am sorry if you feel this.

> I would comment on what you said, but its meaning is vague.
>
> What, concretely, do you mean by the "original source"?
> What does it mean to say it "has gone"?

For example, the file CTLau.html is documented (in the readme) as:

They are verbatim copies of those distributed by the author; they used to
be at <http://umunhum.stanford.edu/~lee/chicomp/>.  Each file contains a
license notice.

(The license notice in question is GPL).

My assumption is that CTLau.html is reformatted as part of the build
because it might be updated from its original source. This would make
sense to me: otherwise, why not reformat it once into a new source file.
If this is the case, then it is no longer true, since the original source
(given at that URL) does not exist.

Again, I am making suppositions here. Shall we wait until handa san has
responded? If he does not respond, we can either decide what to do anyway,
or do nothing at all. A simpler build is nice, but not critical.

Phil

bug#25443: Unnecessary building of dictionary files

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > They are verbatim copies of those distributed by the author; they used to
  > be at <http://umunhum.stanford.edu/~lee/chicomp/>.  Each file contains a
  > license notice.

  > (The license notice in question is GPL).

  > My assumption is that CTLau.html is reformatted as part of the build
  > because it might be updated from its original source. This would make
  > sense to me: otherwise, why not reformat it once into a new source file.

That seems plausible to me.

  > If this is the case, then it is no longer true, since the original source
  > (given at that URL) does not exist.

You may well be right that they are no longer interested in updating
these files.

It follows that we COULD decide to use the Lisp code as the real source,
if it is the format we will want to edit in the future.

However, whether this is a good idea or not
depends on other questions.

  > Again, I am making suppositions here. Shall we wait until handa san has
  > responded?

Yes.  Also I will try to contact him.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

bug#25443: Unnecessary building of dictionary files

[handa]

Hi, sorry for the late response.

In article <E1cV4G2-0001sO-NE@fencepost.gnu.org>, Richard Stallman <rms@gnu.org> writes:

> Again, I am making suppositions here. Shall we wait until handa san has
> responded?

> Yes.  Also I will try to contact him.

I've just replied to Phillip as follows:

In article <87tw8v6yjg.fsf@russet.org.uk>, phillip.lord@russet.org.uk (Phillip Lord) writes:

> I was wondering if you would be able to comment on the Emacs bug
> tracker.

> In summary, some of the dictionary files (like CTLau.html) appear to be
> no longer distributed from the original location. So, I was wondering
> whether we could remove them and just maintain the lisp generated from
> them instead. This would simplify the build which is a good
> thing.

I agree with that.

> Obviously, this is not worth doing if it makes maintaining leim
> harder.

It does not make leim maintenance harder if Makefile is modified
properly.

---
K. Handa
handa@gnu.org

bug#25443: Unnecessary building of dictionary files

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

The crucial question here is whether the Lisp code qualifies as source code.

Supposing we want to change that code one day,
is the Lisp code a good form for editing that program
as the current source form?

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

bug#25443: Unnecessary building of dictionary files

[Phillip Lord]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> The crucial question here is whether the Lisp code qualifies as source code.
>
> Supposing we want to change that code one day, is the Lisp code a good
> form for editing that program as the current source form?

Looking at it, I would say yes. It's a relatively simple syntactic
transformation, and the lisp is readable. Obviously, though, it's not my
code. Handa has said that it would not complicate maintainence.

Richard, you seem most concerned about this -- do you want to have a
look and make decision. It's not a major issue either way.

Phil

bug#25443: Unnecessary building of dictionary files

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Supposing we want to change that code one day, is the Lisp code a good
  > > form for editing that program as the current source form?

  > Looking at it, I would say yes. It's a relatively simple syntactic
  > transformation, and the lisp is readable. Obviously, though, it's not my
  > code. Handa has said that it would not complicate maintainence.

Handa, would you please respond to my question?
You are the expert, and I'm waiting for your response.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

bug#25443: Unnecessary building of dictionary files

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I too think that Lisp code is good.  The maintainability of HTML and
  > Lisp is almost the same.  If there's no chance of getting a new version
  > as HTML, it's useless to keep HTML version.  And we already keep many
  > input method data in Lisp.

Following your judgment, I agree we can switch to the Lisp code as
the source code for that input method.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

bug#25443: Unnecessary building of dictionary files

[Phillip Lord]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > I too think that Lisp code is good.  The maintainability of HTML and
>   > Lisp is almost the same.  If there's no chance of getting a new version
>   > as HTML, it's useless to keep HTML version.  And we already keep many
>   > input method data in Lisp.
>
> Following your judgment, I agree we can switch to the Lisp code as
> the source code for that input method.

Thank you, I will prepare a patch.

Phil