A proposal for removing obsolete packages

A proposal for removing obsolete packages

[Andrew Hyatt]

I'd like to propose a policy for feature removal - I'll just start out
with the policy, and then give the rationalization, and discuss
alternatives.

Currently, obsolete elisp is moved to the obsolete subdir. Attempts to
load them are successful, but come with the warning message "<package>
is obsolete!" I propose that we leave them in the obsolete subdir for
one major version. At the start of every new major version, we remove
everything from that subdir package that has been there since the start
of last major version.

For example, a package that is declared obsolete during the development
of Emacs 25 would be moved to obsolete, and a message would be added to
say that "<package> is obsolete and will be removed in Emacs 27". It
couldn't be removed in Emacs 26 because it didn't start Emacs 25 in
obsolete.

In short, obsolete package added in the middle of a release will be
guaranteed to exist until the next major release, and then one more
major release.

The need for a policy arose while discussing with Eli and others what
should be done with a bug against the obsolete longlines-mode
(https://debbugs.gnu.org/cgi/bugreport.cgi?bug=1452). We have many bugs,
and it was a bit worrying that there was no way to remove a bug even
against functionality that was obsolete. Plus, there was no accepted way
to get anything out of obsolete. Where was the benefit, then, of putting
things in obsolete at all?

To me, it's clear that emacs would be well served by shedding some
little-used functionality. The amount of elisp is enormous, and it
mostly modifies a common user experience, so that each package can
potentially create problems for each other package. By removing the
obsolete packages, we can reduce the amount of potential issues.

What if someone relied on an obsolete package? Mostly, like
longlines-mode, better solutions have come along. Giving people an
entire major version to switch seems pretty reasonable to me. For most
obsolete functionality, we're talking years of existence in the obsolete
subdir. If not, anyone still interested can create an ELPA package that
they maintain on their favorite ELPA server - but not in gnu ELPA.

There might be even better solutions to this problem. I certainly have
not attempted to explore the entire space of solutions. Perhaps the
timing is too long as well - I wouldn't mind making things more
aggressive and instead removing all obsolete packages at the start of
each major version (meaning that an obsolete package would last one
major version or less). This probably would work itself out and obsolete
packages would tend to not be removed at the end of the major version
development cycle.

Please let me know what you think.

Re: A proposal for removing obsolete packages

[Phillip Lord]

Andrew Hyatt <ahyatt@gmail.com> writes:
> There might be even better solutions to this problem. I certainly have
> not attempted to explore the entire space of solutions. Perhaps the
> timing is too long as well - I wouldn't mind making things more
> aggressive and instead removing all obsolete packages at the start of
> each major version (meaning that an obsolete package would last one
> major version or less). This probably would work itself out and obsolete
> packages would tend to not be removed at the end of the major version
> development cycle.
>
> Please let me know what you think.


Having a policy that says "we can remove and will remove packages at the
beginning of each new major release, unless some one really objects"
would be a reasonable middle course.

This would also give people the incentive to make the jump -- I still
use longlines-mode, for example, because I haven't found a replacement
for M-x longlines-show-hard-newlines in visual-line-mode. If it's not
there, I could add it.

Phil

Re: A proposal for removing obsolete packages

[Eli Zaretskii]

> From: Andrew Hyatt <ahyatt@gmail.com>
> Date: Sun, 10 Jan 2016 22:09:27 -0500
> 
> Currently, obsolete elisp is moved to the obsolete subdir. Attempts to
> load them are successful, but come with the warning message "<package>
> is obsolete!" I propose that we leave them in the obsolete subdir for
> one major version. At the start of every new major version, we remove
> everything from that subdir package that has been there since the start
> of last major version.
> 
> For example, a package that is declared obsolete during the development
> of Emacs 25 would be moved to obsolete, and a message would be added to
> say that "<package> is obsolete and will be removed in Emacs 27". It
> couldn't be removed in Emacs 26 because it didn't start Emacs 25 in
> obsolete.
> 
> In short, obsolete package added in the middle of a release will be
> guaranteed to exist until the next major release, and then one more
> major release.

That sounds good to me, thanks.

> There might be even better solutions to this problem. I certainly have
> not attempted to explore the entire space of solutions. Perhaps the
> timing is too long as well - I wouldn't mind making things more
> aggressive and instead removing all obsolete packages at the start of
> each major version (meaning that an obsolete package would last one
> major version or less). This probably would work itself out and obsolete
> packages would tend to not be removed at the end of the major version
> development cycle.

We've repeatedly seen reports from people who jump 2 major releases,
so I think removing sooner than after a full major release cycle is
not something we should start with.

Re: A proposal for removing obsolete packages

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > For example, a package that is declared obsolete during the development
  > of Emacs 25 would be moved to obsolete, and a message would be added to
  > say that "<package> is obsolete and will be removed in Emacs 27". It
  > couldn't be removed in Emacs 26 because it didn't start Emacs 25 in
  > obsolete.

I agree.  But we should not be rigit about deleting it in Emacs 27,
either.  Depending on how the feature is used, we might want to save
it longer.  Features used in Lisp code may need to remain longer.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: A proposal for removing obsolete packages

[Stefan Monnier]

> We've repeatedly seen reports from people who jump 2 major releases,
> so I think removing sooner than after a full major release cycle is
> not something we should start with.

Agreed.  And that matches our past behavior: we generally purge features
marked obsolete more than 2 major versions ago (tho the condition
shouldn't be based on counting major versions but on years, IMO).


        Stefan

Re: A proposal for removing obsolete packages

[Andrew Hyatt]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > For example, a package that is declared obsolete during the development
>   > of Emacs 25 would be moved to obsolete, and a message would be added to
>   > say that "<package> is obsolete and will be removed in Emacs 27". It
>   > couldn't be removed in Emacs 26 because it didn't start Emacs 25 in
>   > obsolete.
>
> I agree.  But we should not be rigit about deleting it in Emacs 27,
> either.  Depending on how the feature is used, we might want to save
> it longer.  Features used in Lisp code may need to remain longer.

Could we instead not move things into obsolete if we didn't think they
were removable?

Also, can you give an example of something that is obsolete but
shouldn't be removed?  That might help me understand your concern.

Re: A proposal for removing obsolete packages

[Andrew Hyatt]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrew Hyatt <ahyatt@gmail.com>
>> Date: Sun, 10 Jan 2016 22:09:27 -0500
>> 
>> Currently, obsolete elisp is moved to the obsolete subdir. Attempts to
>> load them are successful, but come with the warning message "<package>
>> is obsolete!" I propose that we leave them in the obsolete subdir for
>> one major version. At the start of every new major version, we remove
>> everything from that subdir package that has been there since the start
>> of last major version.
>> 
>> For example, a package that is declared obsolete during the development
>> of Emacs 25 would be moved to obsolete, and a message would be added to
>> say that "<package> is obsolete and will be removed in Emacs 27". It
>> couldn't be removed in Emacs 26 because it didn't start Emacs 25 in
>> obsolete.
>> 
>> In short, obsolete package added in the middle of a release will be
>> guaranteed to exist until the next major release, and then one more
>> major release.
>
> That sounds good to me, thanks.
>
>> There might be even better solutions to this problem. I certainly have
>> not attempted to explore the entire space of solutions. Perhaps the
>> timing is too long as well - I wouldn't mind making things more
>> aggressive and instead removing all obsolete packages at the start of
>> each major version (meaning that an obsolete package would last one
>> major version or less). This probably would work itself out and obsolete
>> packages would tend to not be removed at the end of the major version
>> development cycle.
>
> We've repeatedly seen reports from people who jump 2 major releases,
> so I think removing sooner than after a full major release cycle is
> not something we should start with.

Sounds reasonable to me, thanks for the feedback.

Re: A proposal for removing obsolete packages

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > >   > For example, a package that is declared obsolete during the development
  > >   > of Emacs 25 would be moved to obsolete, and a message would be added to
  > >   > say that "<package> is obsolete and will be removed in Emacs 27". It
  > >   > couldn't be removed in Emacs 26 because it didn't start Emacs 25 in
  > >   > obsolete.
  > >
  > > I agree.  But we should not be rigit about deleting it in Emacs 27,
  > > either.  Depending on how the feature is used, we might want to save
  > > it longer.  Features used in Lisp code may need to remain longer.

  > Could we instead not move things into obsolete if we didn't think they
  > were removable?

Moving them to 'obsolete' would be done at the first step, according to
that proposal.  The question I am raising is when to delete them entirely.

  > Also, can you give an example of something that is obsolete but
  > shouldn't be removed?  That might help me understand your concern.

defadvice might be a good example.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: A proposal for removing obsolete packages

[Andrew Hyatt]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > >   > For example, a package that is declared obsolete during the development
>   > >   > of Emacs 25 would be moved to obsolete, and a message would be added to
>   > >   > say that "<package> is obsolete and will be removed in Emacs 27". It
>   > >   > couldn't be removed in Emacs 26 because it didn't start Emacs 25 in
>   > >   > obsolete.
>   > >
>   > > I agree.  But we should not be rigit about deleting it in Emacs 27,
>   > > either.  Depending on how the feature is used, we might want to save
>   > > it longer.  Features used in Lisp code may need to remain longer.
>
>   > Could we instead not move things into obsolete if we didn't think they
>   > were removable?
>
> Moving them to 'obsolete' would be done at the first step, according to
> that proposal.  The question I am raising is when to delete them entirely.
>
>   > Also, can you give an example of something that is obsolete but
>   > shouldn't be removed?  That might help me understand your concern.
>
> defadvice might be a good example.

OK, that sounds like a completely valid concern.  I agree that some
packages that are well used may need to stick around for longer, and
that it should be at the discretion of the maintainer.  I'd hope that
this would be very unusual, though.

One way to make it unusual (and allow things like advice to be placed in
obsolete and removed later) would be if elisp functionality such as
defadvice had their APIs maintained but implemented on top of newer
functionality. For example, a macro in nadvice.el could re-implement
defadvice on top of the replacement functions. That would allow us to
remove the old implementation safely. I haven't thought about this
example very deeply, though, perhaps it is not so easy.

Re: A proposal for removing obsolete packages

[Phillip Lord]

Richard Stallman <rms@gnu.org> writes:
>   > Also, can you give an example of something that is obsolete but
>   > shouldn't be removed?  That might help me understand your concern.
>
> defadvice might be a good example.

Perhaps we need an intermediate directory, called "deprecated". Not
obsolete, should probably still autoload -- putting defadvice into
obsolete will break some packages for this reason alone, but should
produce warnings.

The intention would be that "deprecated" packages would move to
"obsolete" there after.

Phil

Re: A proposal for removing obsolete packages

[Andrew Hyatt]

phillip.lord@russet.org.uk (Phillip Lord) writes:

> Richard Stallman <rms@gnu.org> writes:
>>   > Also, can you give an example of something that is obsolete but
>>   > shouldn't be removed?  That might help me understand your concern.
>>
>> defadvice might be a good example.
>
> Perhaps we need an intermediate directory, called "deprecated". Not
> obsolete, should probably still autoload -- putting defadvice into
> obsolete will break some packages for this reason alone, but should
> produce warnings.
>
> The intention would be that "deprecated" packages would move to
> "obsolete" there after.

What would the difference be between deprecated and obsolete, then?  Is
it just that obsolete would definitely be deleted and deprecated only
might be?

>
> Phil

Re: A proposal for removing obsolete packages

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

"Deprecated" is neither weaker nor stronger than "obsolete".  Rather,
they are different in quality.

"Obsolete" means "This was useful before, but not now, since the
context has changed."  "Deprecated" means "We disapprove of this."

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: A proposal for removing obsolete packages

[Stefan Monnier]

> One way to make it unusual (and allow things like advice to be placed in
> obsolete and removed later) would be if elisp functionality such as
> defadvice had their APIs maintained but implemented on top of newer
> functionality. For example, a macro in nadvice.el could re-implement
> defadvice on top of the replacement functions.

FWIW: defadvice *is* implemented on top of nadvice already.
Emacs-23.4's advice.el is 170KB.
Emacs-24.5's advice.el is 139KB (+ 24KB of nadvice.el).

I originally intended to delay such "re-implementation" of advice.el to
some later time, but I quickly realized that it was not an option
because otherwise you can't mix defadvice and advice-add on the
same function.

The problem of a long transition (as is likely to occur for advice.el)
could be reduced in the following way:
- Rather than completely remove the obsolete package (after N years),
  move it to GNU ELPA.  So people can still get their old code working,
  tho it takes an extra step.
- Implement an nadvice.el forward compatibility package in GNU ELPA
  (probably implemented on top of defadvice), like I did for cl-lib and
  cl-generic, so external packages can start using nadvice.el without
  having to wait for Emacs-24.3 to be considered "too old to support".


        Stefan

Re: A proposal for removing obsolete packages

[Andrew Hyatt]

[-- Attachment #1: Type: text/plain, Size: 1027 bytes --]

On Sat, Jan 16, 2016 at 11:10 AM Richard Stallman <rms@gnu.org> wrote:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> "Deprecated" is neither weaker nor stronger than "obsolete".  Rather,
> they are different in quality.
>
> "Obsolete" means "This was useful before, but not now, since the
> context has changed."  "Deprecated" means "We disapprove of this."
>

Sounds like it might not make sense to transition from deprecated to
obsolete then.

Given the state of the discussion, I can write this up somewhere in the
source - maybe a README in the osbsolete dir, or a note somewhere in admin
dir?  If anyone has any advice on where would be good, let me know.


>
> --
> Dr Richard Stallman
> President, Free Software Foundation (gnu.org, fsf.org)
> Internet Hall-of-Famer (internethalloffame.org)
> Skype: No way! See stallman.org/skype.html.
>
>

[-- Attachment #2: Type: text/html, Size: 1831 bytes --]

Re: A proposal for removing obsolete packages

[John Wiegley]

[-- Attachment #1: Type: text/plain, Size: 722 bytes --]

>>>>> Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> We've repeatedly seen reports from people who jump 2 major releases, so I
>> think removing sooner than after a full major release cycle is not
>> something we should start with.

> Agreed. And that matches our past behavior: we generally purge features
> marked obsolete more than 2 major versions ago (tho the condition shouldn't
> be based on counting major versions but on years, IMO).

I like this suggestion as well, and would like to delay purging until after a
full major release cycle.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 629 bytes --]

Re: A proposal for removing obsolete packages

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

If we want to have two stages of obsolescence, we can find names for
them.  The real question is the substance: do we want two stages, and
if so, what do they mean?

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: A proposal for removing obsolete packages

[Phillip Lord]

Andrew Hyatt <ahyatt@gmail.com> writes:

> phillip.lord@russet.org.uk (Phillip Lord) writes:
>
>> Richard Stallman <rms@gnu.org> writes:
>>>   > Also, can you give an example of something that is obsolete but
>>>   > shouldn't be removed?  That might help me understand your concern.
>>>
>>> defadvice might be a good example.
>>
>> Perhaps we need an intermediate directory, called "deprecated". Not
>> obsolete, should probably still autoload -- putting defadvice into
>> obsolete will break some packages for this reason alone, but should
>> produce warnings.
>>
>> The intention would be that "deprecated" packages would move to
>> "obsolete" there after.
>
> What would the difference be between deprecated and obsolete, then?  Is
> it just that obsolete would definitely be deleted and deprecated only
> might be?


Autoloads. Obsolete does not autoload. So, for example,

M-x longlines-mode

does not work in current emacs. You need to do

M-x load-library   longlines-mode

first. So, for example, making advice.el obsolete would break package
using defadvice unless advice.el were forceably loaded somewhere.

Phil

Re: A proposal for removing obsolete packages

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Agreed. And that matches our past behavior: we generally purge features
  > > marked obsolete more than 2 major versions ago (tho the condition shouldn't
  > > be based on counting major versions but on years, IMO).

  > I like this suggestion as well, and would like to delay purging until after a
  > full major release cycle.

I think we should state that as a lower bound, a minimum period to
wait, not as a default.  To have a default waiting period before
deleting obsolete features would lead us to act rigidly, rather than
considering in each case what is best for the users.



-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: A proposal for removing obsolete packages

[John Wiegley]

[-- Attachment #1: Type: text/plain, Size: 813 bytes --]

>>>>> Richard Stallman <rms@gnu.org> writes:

> I think we should state that as a lower bound, a minimum period to wait, not
> as a default. To have a default waiting period before deleting obsolete
> features would lead us to act rigidly, rather than considering in each case
> what is best for the users.

I'm quite OK with being flexible about when we feel it's time to delete old
code.

To summarize the final position: Once code has been in "obsolete" for a
complete release cycle, removal may be recommended by those who feel it's time
for the code to go. Once it's truly gone, any bugs against it and remaining
documentation can follow.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 629 bytes --]

Re: A proposal for removing obsolete packages

[Andrew Hyatt]

[-- Attachment #1: Type: text/plain, Size: 1798 bytes --]

On Tue, Jan 19, 2016 at 11:29 PM John Wiegley <jwiegley@gmail.com> wrote:

> >>>>> Richard Stallman <rms@gnu.org> writes:
>
> > I think we should state that as a lower bound, a minimum period to wait,
> not
> > as a default. To have a default waiting period before deleting obsolete
> > features would lead us to act rigidly, rather than considering in each
> case
> > what is best for the users.
>
> I'm quite OK with being flexible about when we feel it's time to delete old
> code.
>
> To summarize the final position: Once code has been in "obsolete" for a
> complete release cycle, removal may be recommended by those who feel it's
> time
> for the code to go. Once it's truly gone, any bugs against it and remaining
> documentation can follow.
>

I think that summary has one key difference to what we previously
discussed, and I think that difference is problematic.

My goal in starting this thread was to find a way for emacs to be able to
remove functionality.  But the proposal, as John puts it, would in effect
keep code around forever.  Someone will have to, at the right time, make an
argument that the code should be removed, and be willing to advocate for
it.  This is too high a burden for code that's been in obsolete for years.
I think the burden should be on people to advocate for it's continued
existence.

Specifically, we can, at the start of a major release, put out a list of
all obsolete functionality that the maintainer feels is safe to delete, and
ask if anyone has objections to deleting them.

Making the default to delete instead of keep will have a real difference on
how much code actually gets removed.


> --
> John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
> http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2
>

[-- Attachment #2: Type: text/html, Size: 2529 bytes --]

Re: A proposal for removing obsolete packages

[Eli Zaretskii]

> From: Andrew Hyatt <ahyatt@gmail.com>
> Date: Wed, 20 Jan 2016 17:05:01 +0000
> 
> Specifically, we can, at the start of a major release, put out a list of all
> obsolete functionality that the maintainer feels is safe to delete, and ask if
> anyone has objections to deleting them. 

I think all John (and Richard) are saying is that the person who puts
out that list should him/herself make judgment calls about the items,
and perhaps remove some of them from the list even without asking.

As long as the removal is not done by a program, but by humans, I
think considering each candidate before producing the list is a
reasonable compromise.

Re: A proposal for removing obsolete packages

[John Wiegley]

[-- Attachment #1: Type: text/plain, Size: 1292 bytes --]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

> As long as the removal is not done by a program, but by humans, I think
> considering each candidate before producing the list is a reasonable
> compromise.

Right. I imagine that even without such a list, the process will look
something like this:

  1. Core developer looks at bug.
  2. Sees it is in "obsolete" code.
  3. Sees it has been "obsolete" for a while.
  4. Asks list, "Hey, can I just delete this code and the bug?"
  5. Hears deafening applause of consent.
  6. Happiness is restored.

Our biggest complaint about obsolete code is not its existence, but having to
tend to it; and so, this new policy gives us a way to do decisively deal with
it after a certain period of time.

Likewise, if an obsolete file never has any bugs, I'm not opposed to letting
it stay in "obsolete" forever. And yet, if someone requests that it be removed
after the appropriate amount of time, that's OK too.

The point is that we get to decide, not a mechanistic process, in the case
that someone knows of a good reason to keep that code around for a few
releases more.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 629 bytes --]

Re: A proposal for removing obsolete packages

[Andrés Ramírez]

Hi.

What about byte-compilation.
Does obsolete packages should be byte-compiled when compiling emacs?

BR
On Wed, 20 Jan 2016 14:52:20 -0500,
John Wiegley wrote:
> 
> [1  <text/plain (quoted-printable)>]
> >>>>> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > As long as the removal is not done by a program, but by humans, I think
> > considering each candidate before producing the list is a reasonable
> > compromise.
> 
> Right. I imagine that even without such a list, the process will look
> something like this:
> 
>   1. Core developer looks at bug.
>   2. Sees it is in "obsolete" code.
>   3. Sees it has been "obsolete" for a while.
>   4. Asks list, "Hey, can I just delete this code and the bug?"
>   5. Hears deafening applause of consent.
>   6. Happiness is restored.
> 
> Our biggest complaint about obsolete code is not its existence, but having to
> tend to it; and so, this new policy gives us a way to do decisively deal with
> it after a certain period of time.
> 
> Likewise, if an obsolete file never has any bugs, I'm not opposed to letting
> it stay in "obsolete" forever. And yet, if someone requests that it be removed
> after the appropriate amount of time, that's OK too.
> 
> The point is that we get to decide, not a mechanistic process, in the case
> that someone knows of a good reason to keep that code around for a few
> releases more.
> 
> -- 
> John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
> http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2
> [2 signature.asc <application/pgp-signature (7bit)>]
> No public key for C144D8F4F19FE630 created at 2016-01-20T14:52:20-0500 using RSA

Re: A proposal for removing obsolete packages

[John Wiegley]

>>>>> Andrés Ramírez <sunshavi@fastmail.fm> writes:

> What about byte-compilation.
> Does obsolete packages should be byte-compiled when compiling emacs?

One thing that doing so would tell us is whether the model even loads with the
current version of Emacs. If it doesn't even function, it would be somewhat
strange to include it in the distribution.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: A proposal for removing obsolete packages

[Eli Zaretskii]

> From: Andrés Ramírez <sunshavi@fastmail.fm>
> Date: Wed, 20 Jan 2016 15:38:05 -0500
> 
> What about byte-compilation.
> Does obsolete packages should be byte-compiled when compiling emacs?

Why not?

Re: A proposal for removing obsolete packages

[Andrés Ramírez]

On Wed, 20 Jan 2016 15:52:32 -0500,
Eli Zaretskii wrote:
> 
> > From: Andrés Ramírez <sunshavi@fastmail.fm>
> > Date: Wed, 20 Jan 2016 15:38:05 -0500
> > 
> > What about byte-compilation.
> > Does obsolete packages should be byte-compiled when compiling emacs?
> 
> Why not?
Cos we are trying to get rid of them. 
Regards

Re: A proposal for removing obsolete packages

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  >   Someone will have to, at the right time, make an
  > argument that the code should be removed, and be willing to advocate for
  > it.  This is too high a burden for code that's been in obsolete for years.
  > I think the burden should be on people to advocate for it's continued
  > existence.

(You mean "its", not "it's".)

I disagree.  Deleting something that people once used
should not be semiautomatic.

How about proposing some obsolete features for deletion
and we can think about deleting them in Emacs 25.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: A proposal for removing obsolete packages

[Eli Zaretskii]

> From: Andrés Ramírez <sunshavi@fastmail.fm>
> Cc: ahyatt@gmail.com,
> 	rms@gnu.org,
> 	monnier@iro.umontreal.ca,
> 	emacs-devel@gnu.org
> Date: Wed, 20 Jan 2016 16:03:01 -0500
> 
> On Wed, 20 Jan 2016 15:52:32 -0500,
> Eli Zaretskii wrote:
> > 
> > > From: Andrés Ramírez <sunshavi@fastmail.fm>
> > > Date: Wed, 20 Jan 2016 15:38:05 -0500
> > > 
> > > What about byte-compilation.
> > > Does obsolete packages should be byte-compiled when compiling emacs?
> > 
> > Why not?
> Cos we are trying to get rid of them. 

No, we are not.  When we decide to get rid of them, we delete them.

Re: A proposal for removing obsolete packages

[Andrew Hyatt]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   >   Someone will have to, at the right time, make an
>   > argument that the code should be removed, and be willing to advocate for
>   > it.  This is too high a burden for code that's been in obsolete for years.
>   > I think the burden should be on people to advocate for it's continued
>   > existence.
>
> (You mean "its", not "it's".)
>
> I disagree.  Deleting something that people once used
> should not be semiautomatic.
>
> How about proposing some obsolete features for deletion
> and we can think about deleting them in Emacs 25.

Here are the packages that are eligible for deletion in Emacs 25 (all
obsolete since before Emacs 24):

awk-mode
cl-compat
iso-acc, iso-insert, iso-swed
keyswap
options
resume
scribe
spell
swedish
sym-comp
vc-mcvs

Most of these seem to be unused. I did a Google search for each of them,
but only resume.el and cl-compat had any notable reference. In addition,
vc-mcvs states that it no longer works with vc in Emacs 23, so that
seems like a pretty obvious candidate to go.  And awk-mode is now
implemented by cc-mode, I believe.

I think we should delete all of these in Emacs 25. The one I'm least
sure of deleting is cl-compat, because I don't know enough about the
history of cl compatibility, and how much the current packages replace
this one. But I think we should take a chance - if removing any package
turns out to be a bad idea, we can put it back.

Is there anything here you think would be dangerous to delete?

Re: A proposal for removing obsolete packages

[Andrew Hyatt]

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> Eli Zaretskii <eliz@gnu.org> writes:
>
>> As long as the removal is not done by a program, but by humans, I think
>> considering each candidate before producing the list is a reasonable
>> compromise.
>
> Right. I imagine that even without such a list, the process will look
> something like this:
>
>   1. Core developer looks at bug.
>   2. Sees it is in "obsolete" code.
>   3. Sees it has been "obsolete" for a while.
>   4. Asks list, "Hey, can I just delete this code and the bug?"
>   5. Hears deafening applause of consent.
>   6. Happiness is restored.
>
> Our biggest complaint about obsolete code is not its existence, but having to
> tend to it; and so, this new policy gives us a way to do decisively deal with
> it after a certain period of time.
>
> Likewise, if an obsolete file never has any bugs, I'm not opposed to letting
> it stay in "obsolete" forever. And yet, if someone requests that it be removed
> after the appropriate amount of time, that's OK too.
>
> The point is that we get to decide, not a mechanistic process, in the case
> that someone knows of a good reason to keep that code around for a few
> releases more.

I guess I'm more of a fan of mechanistic process since it forces action
and decisions, but it sounds like I'm the only one who thinks this way.
The process you propose seems fine, and I can or anyone else can
aggressively push for obsolete package deletion under it.

RE: A proposal for removing obsolete packages

[Drew Adams]

> Here are the packages that are eligible for deletion in Emacs 25 (all
> obsolete since before Emacs 24):
> 
> options

FWIW, wrt `options.el':

1. Command `list-options' lists all user options, together with their
current values and their documentation.

That still seems useful to me.  If it is not, we should tell users
what is its specific replacement.

I see only this in the doc string of `list-options':

  "It is now better to use Customize instead."

("instead" is redundant here, BTW.)

And this message is shown at the top of the `list-options' output:

  "This facility is obsolete; we recommend using M-x customize instead."

Really?  Just how do you "use Customize" to get a listing such as
`list-options' provides?  How do you use `M-x customize' to get
such a listing?

I don't think you can get such a listing.  Certainly not with just
`M-x customize'.  And `customize-apropos .*' doesn't give you the
same thing (no complete doc strings, and not just options, etc.).

If I'm right that there is no real substitute provided by Customize
then I think that command `list-options' (renamed, if necessary)
should be kept.  It could be moved to one of the `cus*.el' files,
if you really plan to toss `options.el'.


2. Similarly, I think that command `edit-options' is still useful.

And yes, I'm familiar with Customize, and I use it often.  But I
don't see that it replaces the specific behavior offered by
`options.el'.  If I'm right about `edit-options' not having a
replacement, please consider keeping it too, possibly moving it
to one of the `cus*.el' files.


3. It is true that `edit-options' does not DTRT when an option has
a `:set' function etc.  It simply uses `set' to set the new value.
(This is true also of command `set-variable', BTW.)  To improve it,
we could make it use a Customize function such as
`customize-set-variable', which does DTRT.


4. I might have said the above when `options.el' was considered
for deprecation.  Dunno.

In any case, I've said it now - I don't see why this library needs
to be deprecated, much less removed.  It represents zero maintenance
burden, unless I'm missing something.

Sure, we want to encourage users to use Customize for most of their
user-option needs.  But I don't see the specific features offered
by `options.el' being provided by Customize.  And I think they are
useful features.

For all the user complaints we hear about Customize (and I generally
defend Customize, though I agree that the UI leaves something to be
desired), I do not recall a single complaint about the commands
`list-options' and `edit-options'.  The listing is clear and easy
to use.

Given #3, above, we could decide to keep only `list-options', but I
think a better approach would be to keep both, possibly improving
`edit-options' to take `:set' etc. into account.  Another alternative
would be for the editing keys to just pop to a relevant Customize
buffer for the given option.

`options.el' provides a useful view of the user options.  I think
that such a view is missing with Customize.

Re: A proposal for removing obsolete packages

[Andrew Hyatt]

Drew Adams <drew.adams@oracle.com> writes:

>> Here are the packages that are eligible for deletion in Emacs 25 (all
>> obsolete since before Emacs 24):
>> 
>> options
>
> FWIW, wrt `options.el':
>
> 1. Command `list-options' lists all user options, together with their
> current values and their documentation.
>
> That still seems useful to me.  If it is not, we should tell users
> what is its specific replacement.
>
> I see only this in the doc string of `list-options':
>
>   "It is now better to use Customize instead."
>
> ("instead" is redundant here, BTW.)
>
> And this message is shown at the top of the `list-options' output:
>
>   "This facility is obsolete; we recommend using M-x customize instead."
>
> Really?  Just how do you "use Customize" to get a listing such as
> `list-options' provides?  How do you use `M-x customize' to get
> such a listing?
>
> I don't think you can get such a listing.  Certainly not with just
> `M-x customize'.  And `customize-apropos .*' doesn't give you the
> same thing (no complete doc strings, and not just options, etc.).
>
> If I'm right that there is no real substitute provided by Customize
> then I think that command `list-options' (renamed, if necessary)
> should be kept.  It could be moved to one of the `cus*.el' files,
> if you really plan to toss `options.el'.

I hadn't used options before, but I tried now.  I guess I don't see the
usefulness of the command.  What I thought you were described above
seems useful indeed - a list of everything customized (for those who
don't want to fiddle with elisp).  But list-options instead gives much,
much more than that in a buffer 38k lines long.  What is the use of
this, and why is it more useful than, say, customize-browse?

>
>
> 2. Similarly, I think that command `edit-options' is still useful.
>
> And yes, I'm familiar with Customize, and I use it often.  But I
> don't see that it replaces the specific behavior offered by
> `options.el'.  If I'm right about `edit-options' not having a
> replacement, please consider keeping it too, possibly moving it
> to one of the `cus*.el' files.
>
>
> 3. It is true that `edit-options' does not DTRT when an option has
> a `:set' function etc.  It simply uses `set' to set the new value.
> (This is true also of command `set-variable', BTW.)  To improve it,
> we could make it use a Customize function such as
> `customize-set-variable', which does DTRT.
>
>
> 4. I might have said the above when `options.el' was considered
> for deprecation.  Dunno.
>
> In any case, I've said it now - I don't see why this library needs
> to be deprecated, much less removed.  It represents zero maintenance
> burden, unless I'm missing something.
>
> Sure, we want to encourage users to use Customize for most of their
> user-option needs.  But I don't see the specific features offered
> by `options.el' being provided by Customize.  And I think they are
> useful features.
>
> For all the user complaints we hear about Customize (and I generally
> defend Customize, though I agree that the UI leaves something to be
> desired), I do not recall a single complaint about the commands
> `list-options' and `edit-options'.  The listing is clear and easy
> to use.
>
> Given #3, above, we could decide to keep only `list-options', but I
> think a better approach would be to keep both, possibly improving
> `edit-options' to take `:set' etc. into account.  Another alternative
> would be for the editing keys to just pop to a relevant Customize
> buffer for the given option.
>
> `options.el' provides a useful view of the user options.  I think
> that such a view is missing with Customize.

Re: A proposal for removing obsolete packages

[Andrew Hyatt]

[-- Attachment #1: Type: text/plain, Size: 4193 bytes --]

On Sat, Jan 23, 2016 at 8:02 PM Andrew Hyatt <ahyatt@gmail.com> wrote:

> Drew Adams <drew.adams@oracle.com> writes:
>
> >> Here are the packages that are eligible for deletion in Emacs 25 (all
> >> obsolete since before Emacs 24):
> >>
> >> options
> >
> > FWIW, wrt `options.el':
> >
> > 1. Command `list-options' lists all user options, together with their
> > current values and their documentation.
> >
> > That still seems useful to me.  If it is not, we should tell users
> > what is its specific replacement.
> >
> > I see only this in the doc string of `list-options':
> >
> >   "It is now better to use Customize instead."
> >
> > ("instead" is redundant here, BTW.)
> >
> > And this message is shown at the top of the `list-options' output:
> >
> >   "This facility is obsolete; we recommend using M-x customize instead."
> >
> > Really?  Just how do you "use Customize" to get a listing such as
> > `list-options' provides?  How do you use `M-x customize' to get
> > such a listing?
> >
> > I don't think you can get such a listing.  Certainly not with just
> > `M-x customize'.  And `customize-apropos .*' doesn't give you the
> > same thing (no complete doc strings, and not just options, etc.).
> >
> > If I'm right that there is no real substitute provided by Customize
> > then I think that command `list-options' (renamed, if necessary)
> > should be kept.  It could be moved to one of the `cus*.el' files,
> > if you really plan to toss `options.el'.
>
> I hadn't used options before, but I tried now.  I guess I don't see the
> usefulness of the command.  What I thought you were described above
> seems useful indeed - a list of everything customized (for those who
> don't want to fiddle with elisp).  But list-options instead gives much,
> much more than that in a buffer 38k lines long.  What is the use of
> this, and why is it more useful than, say, customize-browse?
>

Actually, after a few minutes of looking at customize, I see there is both
customize-saved (to customize everything the user has customized with
customize) and customize-rogue (to customize everything else that is
customized outside of customize).  These should, hopefully, solve your
use-case.  Does it?


>
> >
> >
> > 2. Similarly, I think that command `edit-options' is still useful.
> >
> > And yes, I'm familiar with Customize, and I use it often.  But I
> > don't see that it replaces the specific behavior offered by
> > `options.el'.  If I'm right about `edit-options' not having a
> > replacement, please consider keeping it too, possibly moving it
> > to one of the `cus*.el' files.
> >
> >
> > 3. It is true that `edit-options' does not DTRT when an option has
> > a `:set' function etc.  It simply uses `set' to set the new value.
> > (This is true also of command `set-variable', BTW.)  To improve it,
> > we could make it use a Customize function such as
> > `customize-set-variable', which does DTRT.
> >
> >
> > 4. I might have said the above when `options.el' was considered
> > for deprecation.  Dunno.
> >
> > In any case, I've said it now - I don't see why this library needs
> > to be deprecated, much less removed.  It represents zero maintenance
> > burden, unless I'm missing something.
> >
> > Sure, we want to encourage users to use Customize for most of their
> > user-option needs.  But I don't see the specific features offered
> > by `options.el' being provided by Customize.  And I think they are
> > useful features.
> >
> > For all the user complaints we hear about Customize (and I generally
> > defend Customize, though I agree that the UI leaves something to be
> > desired), I do not recall a single complaint about the commands
> > `list-options' and `edit-options'.  The listing is clear and easy
> > to use.
> >
> > Given #3, above, we could decide to keep only `list-options', but I
> > think a better approach would be to keep both, possibly improving
> > `edit-options' to take `:set' etc. into account.  Another alternative
> > would be for the editing keys to just pop to a relevant Customize
> > buffer for the given option.
> >
> > `options.el' provides a useful view of the user options.  I think
> > that such a view is missing with Customize.
>

[-- Attachment #2: Type: text/html, Size: 5390 bytes --]

RE: A proposal for removing obsolete packages

[Drew Adams]

> > Really?  Just how do you "use Customize" to get a listing such as
> > `list-options' provides?  How do you use `M-x customize' to get
> > such a listing?
> >
> > I don't think you can get such a listing.  Certainly not with just
> > `M-x customize'.  And `customize-apropos .*' doesn't give you the
> > same thing (no complete doc strings, and not just options, etc.).

Both of those messages are incorrect in what they tell users now.

> > If I'm right that there is no real substitute provided by Customize
> > then I think that command `list-options' (renamed, if necessary)
> > should be kept.  It could be moved to one of the `cus*.el' files,
> > if you really plan to toss `options.el'.
> 
> I hadn't used options before, but I tried now.  I guess I don't see
> the usefulness of the command.

It gives you a readable, searchable buffer of all of the options,
together with their values and complete doc strings.  To me, that
can be useful.  And I don't see that Customize provides any
replacement for it - nothing at all similar, AFAICT.

> What I thought you were described above
> seems useful indeed - a list of everything customized (for those who
> don't want to fiddle with elisp).

A list of everything that you have customized but not saved is a
different topic.  It is not what `list-options' is for.

But FWIW, the command you describe already exists:
`customize-customized' (though it returns info also about faces).

And command `customize-saved', if by "customized" you meant
customized and saved.

> But list-options instead gives much,
> much more than that in a buffer 38k lines long.

Yes.  It is not about options that you have customized - that's
a different topic.  And it's not just about listing all of the
options with one-line descriptions.  It's about what `list-options'
does: lists all options together with their current values and
complete doc strings.

> What is the use of this, and why is it more useful than, say,
> customize-browse?

See above for the usefulness.

There is no reason to compare it with `customize-browse' - that's
irrelevant, unless you are claiming that it provides the same
functionality and so is a replacement.

Just because `customize-browse' can be useful is no reason that
`list-options' cannot also be useful.

Similarly for `customize-apropos-options', which is more like
what `list-options' does (with a regexp of `.*').

And you cannot search across all (complete) doc strings with
`customize-apropos-options' - you need to open each entry
individually, to see its full doc string.

(Isearch could perhaps be modified to automatically open
`customize-apropos*' entries that have search hits, instead of
ignoring text in closed entries.  But that's not the case today.)

Anyway, you need not find the command useful for it to be useful
to others.  (There are people who do not find `customize-browse'
to be particularly useful...)

Try `apropos-documentation', which is not about options, but
which is another buffer that shows you complete doc strings,
across which you can search etc.  Useful, no?

An improvement to `list-options' could be to have it list only
matches for some apropos input.  But even now by listing all
options it can be useful.

If the `customize-apropos*' commands were improved to show
what `list-options' shows (searchable full doc strings and
current values) then `customize-apropos-options' might provide
a replacement for `list-options'.  But that's not yet the case.

Re: A proposal for removing obsolete packages

[Andrew Hyatt]

Drew Adams <drew.adams@oracle.com> writes:

>> > Really?  Just how do you "use Customize" to get a listing such as
>> > `list-options' provides?  How do you use `M-x customize' to get
>> > such a listing?
>> >
>> > I don't think you can get such a listing.  Certainly not with just
>> > `M-x customize'.  And `customize-apropos .*' doesn't give you the
>> > same thing (no complete doc strings, and not just options, etc.).
>
> Both of those messages are incorrect in what they tell users now.
>
>> > If I'm right that there is no real substitute provided by Customize
>> > then I think that command `list-options' (renamed, if necessary)
>> > should be kept.  It could be moved to one of the `cus*.el' files,
>> > if you really plan to toss `options.el'.
>> 
>> I hadn't used options before, but I tried now.  I guess I don't see
>> the usefulness of the command.
>
> It gives you a readable, searchable buffer of all of the options,
> together with their values and complete doc strings.  To me, that
> can be useful.  And I don't see that Customize provides any
> replacement for it - nothing at all similar, AFAICT.
>
>> What I thought you were described above
>> seems useful indeed - a list of everything customized (for those who
>> don't want to fiddle with elisp).
>
> A list of everything that you have customized but not saved is a
> different topic.  It is not what `list-options' is for.
>
> But FWIW, the command you describe already exists:
> `customize-customized' (though it returns info also about faces).
>
> And command `customize-saved', if by "customized" you meant
> customized and saved.
>
>> But list-options instead gives much,
>> much more than that in a buffer 38k lines long.
>
> Yes.  It is not about options that you have customized - that's
> a different topic.  And it's not just about listing all of the
> options with one-line descriptions.  It's about what `list-options'
> does: lists all options together with their current values and
> complete doc strings.
>
>> What is the use of this, and why is it more useful than, say,
>> customize-browse?
>
> See above for the usefulness.
>
> There is no reason to compare it with `customize-browse' - that's
> irrelevant, unless you are claiming that it provides the same
> functionality and so is a replacement.
>
> Just because `customize-browse' can be useful is no reason that
> `list-options' cannot also be useful.
>
> Similarly for `customize-apropos-options', which is more like
> what `list-options' does (with a regexp of `.*').
>
> And you cannot search across all (complete) doc strings with
> `customize-apropos-options' - you need to open each entry
> individually, to see its full doc string.
>
> (Isearch could perhaps be modified to automatically open
> `customize-apropos*' entries that have search hits, instead of
> ignoring text in closed entries.  But that's not the case today.)
>
> Anyway, you need not find the command useful for it to be useful
> to others.  (There are people who do not find `customize-browse'
> to be particularly useful...)
>
> Try `apropos-documentation', which is not about options, but
> which is another buffer that shows you complete doc strings,
> across which you can search etc.  Useful, no?
>
> An improvement to `list-options' could be to have it list only
> matches for some apropos input.  But even now by listing all
> options it can be useful.
>
> If the `customize-apropos*' commands were improved to show
> what `list-options' shows (searchable full doc strings and
> current values) then `customize-apropos-options' might provide
> a replacement for `list-options'.  But that's not yet the case.

Everything you wrote above is pretty reasonable, and as far as I can
tell, correct.  Thanks for the explanation.

But this last paragraph seems wrong - if I run, for example,
(customize-apropos-options "beyond.*fill"), I see customize option for
2c-beyond-fill-column, complete with the value I've set and the
docstring, all easily searchable.

However, the functionality that appears missing to me is either to be
able to additionally search by docstring, or showing all options
(basically, (customize-apropos-options ".*") should work quickly and
give something useful, neither of which is true for me today.

Re: A proposal for removing obsolete packages

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > iso-acc, iso-insert, iso-swed

I think I had to use one of these to insert some non-ASCII characters
that didn't appear in my input method.  What is the replacement for these?
-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

RE: A proposal for removing obsolete packages

[Drew Adams]

> > (Isearch could perhaps be modified to automatically open
> > `customize-apropos*' entries that have search hits, instead of
> > ignoring text in closed entries.  But that's not the case today.)
...

> > An improvement to `list-options' could be to have it list only
> > matches for some apropos input.  But even now by listing all
> > options it can be useful.
> >
> > If the `customize-apropos*' commands were improved to show
> > what `list-options' shows (searchable full doc strings and
> > current values) then `customize-apropos-options' might provide
> > a replacement for `list-options'.  But that's not yet the case.
> 
> Everything you wrote above is pretty reasonable, and as far as I can
> tell, correct.  Thanks for the explanation.
> 
> But this last paragraph seems wrong - if I run, for example,
> (customize-apropos-options "beyond.*fill"), I see customize option for
> 2c-beyond-fill-column, complete with the value I've set and the
> docstring, all easily searchable.

With `emacs -q' in Emacs 24.5 or the latest Emacs 25 binary I have
(which is quite old now) I get:
(error "No customizable options matching beyond.*fill")

But yes, if there is only one option that matches the pattern then
the doc string is opened (shown completely).  If there is more
than one (a more typical case for use of an apropos command) then
the details are closed, showing only the first line of the doc
string and not showing the value.

> However, the functionality that appears missing to me is either to be
> able to additionally search by docstring, or showing all options
> (basically, (customize-apropos-options ".*") should work quickly and
> give something useful, neither of which is true for me today.

Yes.  Either it should be improved as suggested or `list-options'
should be retained (IMO).  (And `list-options' could itself be
improved as I indicated, so that it does apropos matching.)

Re: A proposal for removing obsolete packages

[Andrew Hyatt]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > iso-acc, iso-insert, iso-swed
>
> I think I had to use one of these to insert some non-ASCII characters
> that didn't appear in my input method.  What is the replacement for these?

You should be able to get the common ones with the C-x 8 commands that
work out of the box.  For example, "C-x 8 ' e" gives é.  "C-x 8 RET"
should give you any unicode character by name.

Re: A proposal for removing obsolete packages

[Stefan Monnier]

> I think I had to use one of these to insert some non-ASCII characters
> that didn't appear in my input method.  What is the replacement for these?

Hard to tell without knowing what "these" are.  But we have lots of
different ways to insert characters, so there's no doubt that there is
a replacement.


        Stefan

Entering Unicode characters

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > You should be able to get the common ones with the C-x 8 commands that
  > work out of the box.

C-x 8 may be a good replacement for iso-acc, iso-insert, iso-swed.

			   "C-x 8 RET"
  > should give you any unicode character by name.

That is useless in practice, because it requires users to know these
long and often obscure names.  When I wanted to do this, I was unable
to do it with C-x 8.  I was totally stumped!  All I could do was try
to find a file which had the character I wanted.

We need a way to enter any Unicode character that is EASY to use
and does not require using its official name.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: A proposal for removing obsolete packages

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Date: Sun, 24 Jan 2016 20:42:44 -0500
> Cc: johnw@gnu.org, monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
>   > iso-acc, iso-insert, iso-swed
> 
> I think I had to use one of these to insert some non-ASCII characters
> that didn't appear in my input method.  What is the replacement for these?

"C-x 8", mainly.

RE: Entering Unicode characters

[Drew Adams]

> > "C-x 8 RET" should give you any unicode character by name.
> 
> That is useless in practice, because it requires users to know these
> long and often obscure names.  When I wanted to do this, I was unable
> to do it with C-x 8.  I was totally stumped!  All I could do was try
> to find a file which had the character I wanted.
> 
> We need a way to enter any Unicode character that is EASY to use
> and does not require using its official name.

It's a _lot_ easier if completion matching is at least substring,
and much easier still if you can combine multiple input patterns
to match (e.g. multiple substrings).

But yes, some of the char names are hard to imagine (and thus
match).  It would help if a user could narrow the choices by
specifying a character category, for example.

Even just providing a set of insertion commands for different
categories could help.  E.g., if you know you want a math symbol
then you complete against only math-symbol names.  Or if you want
a Greek character then you complete only against their names.

Emacs could at least provide such a set of insertion commands,
and let users decide when and whether to bind some of them.
`C-x 8 RET' is a very wide brush to paint with.  It is a great
command, IMO, but its power and generality can limit its usefulness
for quick insertion of a char from a limited set of chars.

Re: Entering Unicode characters

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Date: Mon, 25 Jan 2016 10:37:01 -0500
> Cc: johnw@gnu.org, monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> We need a way to enter any Unicode character that is EASY to use
> and does not require using its official name.

Indeed, such a feature would be a welcome addition.

Re: Entering Unicode characters

[Stefan Monnier]

>> > "C-x 8 RET" should give you any unicode character by name.
>> That is useless in practice, because it requires users to know these
>> long and often obscure names.  When I wanted to do this, I was unable
>> to do it with C-x 8.  I was totally stumped!  All I could do was try
>> to find a file which had the character I wanted.
> It's a _lot_ easier if completion matching is at least substring,

Which it already is by default.

> But yes, some of the char names are hard to imagine (and thus
> match).  It would help if a user could narrow the choices by
> specifying a character category, for example.

I think for the cases corresponding to those obsolete iso-* packages,
even better would be to provide a "base character" and be offered
a choice among all those chars that build on it.


        Stefan

Re: Entering Unicode characters

[Vivek Dasmohapatra]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 447 bytes --]

> Indeed, such a feature would be a welcome addition.

For situations where xcompose is not availble, why not mimic
what xcompose does, albeit with a different prefix?

    AltGr " e → ë
    AltGr / o → ø
    AltGr t e a → ☕

That last one is a local mapping I have, but you get the idea.
They're even already divided up by locale in the

   /usr/share/X11/locale/*/Compose

files, so a lot of the tedious legwork has already been done.

Re: Entering Unicode characters

[Teemu Likonen]

[-- Attachment #1: Type: text/plain, Size: 503 bytes --]

Vivek Dasmohapatra [2016-01-25 17:20:38Z] wrote:

> For situations where xcompose is not availble, why not mimic
> what xcompose does, albeit with a different prefix?
>
>    AltGr " e → ë
>    AltGr / o → ø
>    AltGr t e a → ☕

Input method “rfc1345” is quite similar:

    &e:  →  ë
    &o/  →  ø
    &"6  →  “
    &"9  →  ”

-- 
/// Teemu Likonen   - .-..   <https://github.com/tlikonen> //
// PGP: 4E10 55DC 84E9 DFF6 13D7 8557 719D 69D3 2453 9450 ///

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]

Re: Entering Unicode characters

[Marcin Borkowski]

On 2016-01-25, at 16:37, Richard Stallman <rms@gnu.org> wrote:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > You should be able to get the common ones with the C-x 8 commands that
>   > work out of the box.
>
> C-x 8 may be a good replacement for iso-acc, iso-insert, iso-swed.
>
> 			   "C-x 8 RET"
>   > should give you any unicode character by name.
>
> That is useless in practice, because it requires users to know these
> long and often obscure names.  When I wanted to do this, I was unable
> to do it with C-x 8.  I was totally stumped!  All I could do was try
> to find a file which had the character I wanted.
>
> We need a way to enter any Unicode character that is EASY to use
> and does not require using its official name.

Have you heard about autocompletion?

Sorry, could not resist.  In fact, I agree that C-x 8 RET with stock
Emacs is not the best experience, but either Icicles (which I used
previously) or Ivy (which I use now) make it much, much better.  Icicles
especially, since it displays the characters next to their names.  Ivy
is much faster, OTOH.

For instance, I can type

™ using C-x 8 RET trade RET
© using C-x 8 RET copyr <down> RET
💩 using C-x 8 RET poo RET (though, fortunately, my font does not support
this)

etc.

Also, TeX input method.  I have it as my default-input-method, and it's
really great, provided one is fluent in TeX (I am, quite).

And, do you think that something like a character table, divided into
categories, might be useful?

Best,

-- 
Marcin Borkowski
http://octd.wmi.amu.edu.pl/en/Marcin_Borkowski
Faculty of Mathematics and Computer Science
Adam Mickiewicz University

RE: Entering Unicode characters

[Drew Adams]

> > We need a way to enter any Unicode character that is EASY to use
> > and does not require using its official name.
> 
> Have you heard about autocompletion?
> 
> Sorry, could not resist.  In fact, I agree that C-x 8 RET with stock
> Emacs is not the best experience, but either Icicles (which I used
> previously) or Ivy (which I use now) make it much, much better.  Icicles
> especially, since it displays the characters next to their names.  Ivy
> is much faster, OTOH.

Icicles too can be instantaneous if you turn off the feature that you mention,
of showing the code points and characters themselves next to the name.

To do that, set `icicle-read-char-by-name-multi-completion-flag' to nil.

These are not just annotations, BTW - you can complete against any of
the 3: name, code point, or char, or any combination of them.  Why might
you complete against the character?  To see its name or code point.  Helpful
if the char is easy to enter.  E.g., what are the name and code point of `}'?
Type it and you see these two (equivalent) candidates:

 CLOSING CURLY BRACKET	7D	}
 RIGHT CURLY BRACKET	7D	}

You can also speed things up during cycling if you sacrifice display of
the current candidate's info in the mode line.  To do that, set
`icicle-help-in-mode-line-delay' to 0.

The doc string of `icicle-read-char-by-name' tells you all of this:

,----
| icicle-read-char-by-name is a compiled Lisp function.
| 
| (icicle-read-char-by-name PROMPT &optional NAMES)
| 
| Read a character by its Unicode name or hex number string.
| Display PROMPT and read a string that represents a character by its
| Unicode property `name' or `old-name'.  Return the char as a number.
| 
| You can use completion against the Unicode name of the character.
| 
| In Icicle mode, if `icicle-read-char-by-name-multi-completion-flag' is
| non-nil:
| 
| * The Unicode code point of the char and the char itself appear next
|   to the char name in `*Completions*' - WYSIWYG.
| 
| * The completion candidate is a multi-completion.  Its first part is
|   the char name.  Its second part is the code point, as a hexadecimal
|   numeral.  Its third part is the character.  This means that you can
|   alternatively type the code point or the character to see what the
|   name is.  You can complete the name or the code point, or both.
| 
| * When you cycle among candidates, regardless of whether buffer
|   `*Completions*' is shown, the current character and its code point
|   are shown in the mode line (provided user option
|   `icicle-help-in-mode-line-delay' is greater than zero).  The code
|   point is shown in hexadecimal, octal, and decimal notation.
| 
| If you use a dedicated `*Completions*' frame, then the font used in
| `*Completions*' is the same as the frame from which you invoked
| completion.
| 
| If you use library `doremi-frm.el' then you can increase the font size
| for `*Completions*' dynamically using `C-x -'.
| 
| As an alternative to completing the Unicode name or code point, you
| can just input the code point as a hexidecimal numeral or a number in
| hash notation: #o21430 for octal, #x2318 for hex, or #10r8984 for
| decimal.
| 
| Non-nil optional arg NAMES is an alist of names to use in place of the
| value returned by `icicle-ucs-names'.  It must have the same form as
| such a return value: (CHAR-NAME . CHAR-CODE).
`----

Re: Entering Unicode characters

[Eli Zaretskii]

> Date: Mon, 25 Jan 2016 17:20:38 +0000 (GMT)
> From: Vivek Dasmohapatra <vivek@etla.org>
> cc: rms@gnu.org, ahyatt@gmail.com, johnw@gnu.org, monnier@iro.umontreal.ca, 
>     emacs-devel@gnu.org
> 
> > Indeed, such a feature would be a welcome addition.
> 
> For situations where xcompose is not availble, why not mimic
> what xcompose does, albeit with a different prefix?
> 
>     AltGr " e → ë
>     AltGr / o → ø
>     AltGr t e a → ☕

Sorry, I don't understand what you are proposing concretely.  When
xcompose is not available, what does Emacs see if you press, e.g.,
AltGr " e?  (You can use "C-h l" to find the answer.)  Does it at all
know that you pressed AltGr and not the left Alt (i.e. Meta) key?

And in any case, how is the above different from Emacs input methods?
There, too, a combination of keys produces a character that is not on
the keyboard.

Thanks.

Re: Entering Unicode characters

[Tianxiang Xiong]

Richard Stallman <rms <at> gnu.org> writes:

>   > You should be able to get the common ones with the C-x 8 commands 
that
>   > work out of the box.
> 
> C-x 8 may be a good replacement for iso-acc, iso-insert, iso-swed.
> 
> 			   "C-x 8 RET"
>   > should give you any unicode character by name.
> 
> That is useless in practice, because it requires users to know these
> long and often obscure names.  When I wanted to do this, I was unable
> to do it with C-x 8.  I was totally stumped!  All I could do was try
> to find a file which had the character I wanted.
> 
> We need a way to enter any Unicode character that is EASY to use
> and does not require using its official name.
> 

Seems to me that this is a problem that should be solved by an 
incremental completion framework.

For example, the "helm-ucs" command from the Helm package provides the 
following features:

- List of all symbols by their proper names in a temporary buffer
- Incremental completion; e.g. I can type "Greek theta" and only symbols 
whose offical names contain the words "Greek" and "theta" are included 
in the result set.
- Preview of symbols (so you can easily see what it looks like and 
whether it's supported by your font)
- Hex character code
- Action menu specifying actions to take on a symbo (e.g. insert symbol, 
copy hex code, etc.)

A solution that does not involve the "official" name would only increase 
the cognitive load for the user because they'd have to know the 
"unofficial" name and how to map it to the official name. Considering 
the large number of symbols users may want to insert, this does not seem 
like a good solution.

RE: Entering Unicode characters

[Drew Adams]

> Seems to me that this is a problem that should be solved by an
> incremental completion framework.

What you describe is exactly what Icicles offers, FWIW.

Re: Entering Unicode characters

[Tianxiang Xiong]

Drew Adams <drew.adams <at> oracle.com> writes:

> 
> > Seems to me that this is a problem that should be solved by an
> > incremental completion framework.
> 
> What you describe is exactly what Icicles offers, FWIW.
> 

Yes, I'm sure there are other frameworks that can offer similar behavior 
as Helm. 

The point is that we should be looking at solutions that allow us to 
better manipulate existing data (in this case Unicode symbols), rather 
than creating new data to fit our too-weak tools.

Re: Entering Unicode characters

[Vivek Dasmohapatra]

> AltGr " e?  (You can use "C-h l" to find the answer.)  Does it at all
> know that you pressed AltGr and not the left Alt (i.e. Meta) key?

I wasn't proposing AltGr, merely explaining what xcompose used.

> And in any case, how is the above different from Emacs input methods?

It isn't. I merely meant that there is already a widespread mechanism
with a set of defined mnemonics which could be imported wholesale.

Re: Entering Unicode characters

[Eli Zaretskii]

> Date: Tue, 26 Jan 2016 01:24:31 +0000 (GMT)
> From: Vivek Dasmohapatra <vivek@etla.org>
> cc: rms@gnu.org, ahyatt@gmail.com, johnw@gnu.org, monnier@iro.umontreal.ca, 
>     emacs-devel@gnu.org
> 
> > AltGr " e?  (You can use "C-h l" to find the answer.)  Does it at all
> > know that you pressed AltGr and not the left Alt (i.e. Meta) key?
> 
> I wasn't proposing AltGr, merely explaining what xcompose used.
> 
> > And in any case, how is the above different from Emacs input methods?
> 
> It isn't. I merely meant that there is already a widespread mechanism
> with a set of defined mnemonics which could be imported wholesale.

I see, thanks for explaining.

I think that this kind of approach works well with accented
characters, but not with others, which are the majority in Unicode.

Re: Entering Unicode characters

[Oleh Krehel]

[-- Attachment #1: Type: text/plain, Size: 606 bytes --]

Marcin Borkowski <mbork@mbork.pl> writes:

> On 2016-01-25, at 16:37, Richard Stallman <rms@gnu.org> wrote:

>> That is useless in practice, because it requires users to know these
>> long and often obscure names.  When I wanted to do this, I was unable
>> to do it with C-x 8.  I was totally stumped!  All I could do was try
>> to find a file which had the character I wanted.
>>
> Have you heard about autocompletion?

Here is the current capability of `counsel-unicode-char' which is
included in the GNU ELPA swiper package. Note how the regex boundaries
"\b" are used to closely match the letter "a".


[-- Attachment #2: counsel-unicode-char.png --]
[-- Type: image/png, Size: 130588 bytes --]

[-- Attachment #3: Type: text/plain, Size: 975 bytes --]


This command is meant to do the same as `insert-char' does.

The only issue is that the built-in `ucs-names' variable, which is
offered for completion, includes around 40000 entries. Maybe there
should be a way for the user to turn off the Unicode ranges she's never
interested in (see the division of ranges in the `nxml-unicode-blocks'
variable).  In the included screenshot, I've already disabled some
ranges that aren't interesting for me - my font doesn't have glyphs for
a large portion of these.

Another issue I've found when shortly examining the Unicode table is
that it provides these three categories:

1. Plain letters,

2. Composition characters, like COMBINING GRAVE ACCENT, COMBINING ACUTE
ACCENT etc.,

3 Plain letters pre-combined with the above composition chars.

So maybe the user should decide if she prefers to have 1+2 or 1+3 or
1+2+3, the latter option significantly increasing the number of
completion candidates (by around 2500).

    Oleh

Re: A proposal for removing obsolete packages

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > I think I had to use one of these to insert some non-ASCII characters
  > > that didn't appear in my input method.  What is the replacement for these?

  > Hard to tell without knowing what "these" are.

One of them was the dotless i, for Turkish.  But I'd have the same
problem for any Unicode character that doesn't have its own C-x 8
subcommand.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Entering Unicode characters

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Completion schemes can make it easier to enter long names that you
know.  They are no use for me in entering unusual characters, because 
I don't know their names.

I want a system that lets me choose them by seeing them on the screen.
I want to specify a language or script and see all its characters.

For instance, if I enter 'turkish' it should show me all the
characters used in Turkish.  Then I could pick the dotless i from the
buffer.

Would someone please implement this?

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: A proposal for removing obsolete packages

[Andreas Schwab]

Richard Stallman <rms@gnu.org> writes:

> One of them was the dotless i, for Turkish.  But I'd have the same
> problem for any Unicode character that doesn't have its own C-x 8
> subcommand.

It should be easy to add bindings for the missing characters to the C-x
8 map.  For example, the dotless i and the dotted I could be added as
`C-x 8 . i' and `C-x 8 . I', resp.

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: Entering Unicode characters

[Jean-Christophe Helary]

> 2016/01/26 18:39、Richard Stallman <rms@gnu.org> のメール：
> 
> I want a system that lets me choose them by seeing them on the screen.
> I want to specify a language or script and see all its characters.
> 
> For instance, if I enter 'turkish' it should show me all the
> characters used in Turkish.  Then I could pick the dotless i from the
> buffer.
> 
> Would someone please implement this?

There is an input system imported from smartphones that is being implemented on some desktop OSes.

In my case it is OSX and it works like this: you choose a keyboard layout and then by pressing a key you know can have variants for 2 secs the list of variants is displayed along with a number under them. You can either use the arrows to select and enter the character or hit the corresponding number.

I just selected the Turkish qwerty layout and when I hit g for 2 secs I get ğ as a possibility. When I am in the US layout I don't get this possibility. In US I get èéêëēėę for e, ÿ for y, ûüùúū for u (and similar options of other vowels), I also get ßśš for s. If I do the same with a capital letter (holding Shift + the letter) I get all the possible capital letters for the selected letter, for ex. S offers ŚŠ but no capital ß.

That's a very convenient way to visually check and select characters.

Jean-Christophe Helary 

Re: Entering Unicode characters

[Yuri Khan]

On Tue, Jan 26, 2016 at 3:39 PM, Richard Stallman <rms@gnu.org> wrote:

> I want a system that lets me choose them by seeing them on the screen.
> I want to specify a language or script and see all its characters.
>
> Would someone please implement this?

You might find the GNOME Character Map application satisfies your
needs. You can see all code points known to it, or all characters
covered by a specific font, or any of the above filtered by Unicode
block or by script. It also supports search by character code, name,
and aliases, and some cross-referencing facilities.

(Although I might prefer if it were an input method usable all over X
rather than a standalone application that you have to copy-and-paste
characters from.)

Re: Entering Unicode characters

[Marcin Borkowski]

On 2016-01-26, at 10:39, Richard Stallman <rms@gnu.org> wrote:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> Completion schemes can make it easier to enter long names that you
> know.  They are no use for me in entering unusual characters, because 
> I don't know their names.

True, but often these names contain a word which is more or less obvious
to guess.

> I want a system that lets me choose them by seeing them on the screen.
> I want to specify a language or script and see all its characters.
>
> For instance, if I enter 'turkish' it should show me all the
> characters used in Turkish.  Then I could pick the dotless i from the
> buffer.
>
> Would someone please implement this?

I like this idea, though I do not know enough about Unicode etc to
implement it...

Best,

-- 
Marcin Borkowski
http://octd.wmi.amu.edu.pl/en/Marcin_Borkowski
Faculty of Mathematics and Computer Science
Adam Mickiewicz University

Re: Entering Unicode characters

[Eli Zaretskii]

> From: Jean-Christophe Helary <jean.christophe.helary@gmail.com>
> Date: Tue, 26 Jan 2016 19:55:49 +0900
> 
> There is an input system imported from smartphones that is being implemented on some desktop OSes.
> 
> In my case it is OSX and it works like this: you choose a keyboard layout and then by pressing a key you know can have variants for 2 secs the list of variants is displayed along with a number under them. You can either use the arrows to select and enter the character or hit the corresponding number.
> 
> I just selected the Turkish qwerty layout and when I hit g for 2 secs I get ğ as a possibility. When I am in the US layout I don't get this possibility. In US I get èéêëēėę for e, ÿ for y, ûüùúū for u (and similar options of other vowels), I also get ßśš for s. If I do the same with a capital letter (holding Shift + the letter) I get all the possible capital letters for the selected letter, for ex. S offers ŚŠ but no capital ß.
> 
> That's a very convenient way to visually check and select characters.

I think you are describing how Emacs input method work.  Try this:

  C-u C-\ turkish-postfix RET

then type i and watch the echo area.

The advantage of Emacs input methods is that they don't depend on the
locale.  (The disadvantage is that they only affect Emacs; but for
someone who lives inside Emacs, that's not a problem ;-)

Re: Entering Unicode characters

[Stefan Monnier]

> The advantage of Emacs input methods is that they don't depend on the
> locale.  (The disadvantage is that they only affect Emacs; but for
> someone who lives inside Emacs, that's not a problem ;-)

There's another difference: usually the echo area shows which other
characters you can type, rather than which characters you can get by
typing.  E.g. instead of showing "èéêëēėę", it might show something like
that [`'^"-.,].


        Stefan

Re: Entering Unicode characters

[Eli Zaretskii]

> From: Marcin Borkowski <mbork@mbork.pl>
> Date: Tue, 26 Jan 2016 13:40:11 +0100
> Cc: Tianxiang Xiong <tianxiang.xiong@gmail.com>, emacs-devel@gnu.org
> 
> > I want a system that lets me choose them by seeing them on the screen.
> > I want to specify a language or script and see all its characters.
> >
> > For instance, if I enter 'turkish' it should show me all the
> > characters used in Turkish.  Then I could pick the dotless i from the
> > buffer.
> >
> > Would someone please implement this?
> 
> I like this idea, though I do not know enough about Unicode etc to
> implement it...

I don't think any deep knowledge about Unicode is needed.  What you
need is:

  . a way to display a range of Unicode characters and let the user
    select the one she wants (this should be able to handle a large
    range without annoying delays, e.g., by breaking such long ranges
    into smaller blocks and providing buttons to unhide and hide them)

  . a database of languages and the corresponding Unicode blocks (we
    already have that for scripts, but languages sometimes select
    narrower ranges, in particular in the Latin and Greek script
    parts)

The somewhat tricky part about the last item is that some languages
use non-contiguous portions of a block that is shared with other
languages.  The Cyrillic block is one good examples.  IWBNI the
feature would either refrain from displaying irrelevant characters, or
maybe display them with pale colors, to indicate that they are not
relevant to the language.

TIA

Re: A proposal for removing obsolete packages

[John Wiegley]

[-- Attachment #1: Type: text/plain, Size: 516 bytes --]

>>>>> Andrew Hyatt <ahyatt@gmail.com> writes:

> Here are the packages that are eligible for deletion in Emacs 25 (all
> obsolete since before Emacs 24):

> awk-mode
> iso-acc, iso-insert, iso-swed
> keyswap
> options
> resume
> scribe
> spell
> swedish
> sym-comp
> vc-mcvs

Other than cl-compat, I'm OK with deleting these. Does anyone have any other
objections?

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 629 bytes --]

RE: Entering Unicode characters

[Drew Adams]

> Completion schemes can make it easier to enter long names that you
> know.  They are no use for me in entering unusual characters, because
> I don't know their names.
> 
> I want a system that lets me choose them by seeing them on the screen.
> I want to specify a language or script and see all its characters.
> 
> For instance, if I enter 'turkish' it should show me all the
> characters used in Turkish.  Then I could pick the dotless i from the
> buffer.

I agree that names don't always help.  Sometimes they do, though.
I guess that in this case they don't.

FWIW, if I use the pattern "turk" with either Icicles (for `C-x 8 RET')
or `apropos-char' (library apu.el), I see these chars listed.  As all
but one have "OLD TURKIC" in the name, I don't suppose they are what
you are after.  Dunno.  I agree that matching by language instead of
just char name would be a useful feature.

---

(This is from `apropos-char'.  The first column is the char, which my
font cannot display.  In the list buffer, `RET' provides details about
the char of the current line, and `^' inserts it into the original buffer.)

𐱈‎  OLD TURKIC LETTER ORKHON BASH    68680  0x10c48
𐱇‎  OLD TURKIC LETTER ORKHON OT      68679  0x10c47
𐱆‎  OLD TURKIC LETTER YENISEI AET    68678  0x10c46
𐱅‎  OLD TURKIC LETTER ORKHON AET     68677  0x10c45
𐱄‎  OLD TURKIC LETTER YENISEI AT     68676  0x10c44
𐱃‎  OLD TURKIC LETTER ORKHON AT      68675  0x10c43
𐱂‎  OLD TURKIC LETTER YENISEI ESH    68674  0x10c42
𐱁‎  OLD TURKIC LETTER ORKHON ESH     68673  0x10c41
𐱀‎  OLD TURKIC LETTER YENISEI ASH    68672  0x10c40
𐰿‎  OLD TURKIC LETTER ORKHON ASH     68671  0x10c3f
𐰾‎  OLD TURKIC LETTER ORKHON AES     68670  0x10c3e
𐰽‎  OLD TURKIC LETTER ORKHON AS      68669  0x10c3d
𐰼‎  OLD TURKIC LETTER ORKHON AER     68668  0x10c3c
𐰻‎  OLD TURKIC LETTER YENISEI AR     68667  0x10c3b
𐰺‎  OLD TURKIC LETTER ORKHON AR      68666  0x10c3a
𐰹‎  OLD TURKIC LETTER YENISEI OQ     68665  0x10c39
𐰸‎  OLD TURKIC LETTER ORKHON OQ      68664  0x10c38
𐰷‎  OLD TURKIC LETTER YENISEI IQ     68663  0x10c37
𐰶‎  OLD TURKIC LETTER ORKHON IQ      68662  0x10c36
𐰵‎  OLD TURKIC LETTER YENISEI AQ     68661  0x10c35
𐰴‎  OLD TURKIC LETTER ORKHON AQ      68660  0x10c34
𐰳‎  OLD TURKIC LETTER YENISEI EC     68659  0x10c33
𐰲‎  OLD TURKIC LETTER ORKHON EC      68658  0x10c32
𐰱‎  OLD TURKIC LETTER ORKHON IC      68657  0x10c31
𐰰‎  OLD TURKIC LETTER ORKHON OP      68656  0x10c30
𐰯‎  OLD TURKIC LETTER ORKHON EP      68655  0x10c2f
𐰮‎  OLD TURKIC LETTER YENISEI AENG   68654  0x10c2e
𐰭‎  OLD TURKIC LETTER ORKHON ENG     68653  0x10c2d
𐰬‎  OLD TURKIC LETTER YENISEI ANG    68652  0x10c2c
𐰫‎  OLD TURKIC LETTER YENISEI ENY    68651  0x10c2b
𐰪‎  OLD TURKIC LETTER ORKHON ENY     68650  0x10c2a
𐰩‎  OLD TURKIC LETTER YENISEI ENC    68649  0x10c29
𐰨‎  OLD TURKIC LETTER ORKHON ENC     68648  0x10c28
𐰧‎  OLD TURKIC LETTER YENISEI ENT    68647  0x10c27
𐰦‎  OLD TURKIC LETTER ORKHON ENT     68646  0x10c26
𐰥‎  OLD TURKIC LETTER YENISEI AEN    68645  0x10c25
𐰤‎  OLD TURKIC LETTER ORKHON AEN     68644  0x10c24
𐰣‎  OLD TURKIC LETTER ORKHON AN      68643  0x10c23
𐰢‎  OLD TURKIC LETTER ORKHON EM      68642  0x10c22
𐰡‎  OLD TURKIC LETTER ORKHON ELT     68641  0x10c21
𐰠‎  OLD TURKIC LETTER ORKHON AEL     68640  0x10c20
𐰟‎  OLD TURKIC LETTER YENISEI AL     68639  0x10c1f
𐰞‎  OLD TURKIC LETTER ORKHON AL      68638  0x10c1e
𐰝‎  OLD TURKIC LETTER YENISEI OEK    68637  0x10c1d
𐰜‎  OLD TURKIC LETTER ORKHON OEK     68636  0x10c1c
𐰛‎  OLD TURKIC LETTER YENISEI AEK    68635  0x10c1b
𐰚‎  OLD TURKIC LETTER ORKHON AEK     68634  0x10c1a
𐰙‎  OLD TURKIC LETTER YENISEI AEY    68633  0x10c19
𐰘‎  OLD TURKIC LETTER ORKHON AEY     68632  0x10c18
𐰗‎  OLD TURKIC LETTER YENISEI AY     68631  0x10c17
𐰖‎  OLD TURKIC LETTER ORKHON AY      68630  0x10c16
𐰕‎  OLD TURKIC LETTER YENISEI EZ     68629  0x10c15
𐰔‎  OLD TURKIC LETTER ORKHON EZ      68628  0x10c14
𐰓‎  OLD TURKIC LETTER ORKHON AED     68627  0x10c13
𐰒‎  OLD TURKIC LETTER YENISEI AD     68626  0x10c12
𐰑‎  OLD TURKIC LETTER ORKHON AD      68625  0x10c11
𐰐‎  OLD TURKIC LETTER YENISEI AEG    68624  0x10c10
𐰏‎  OLD TURKIC LETTER ORKHON AEG     68623  0x10c0f
𐰎‎  OLD TURKIC LETTER YENISEI AG     68622  0x10c0e
𐰍‎  OLD TURKIC LETTER ORKHON AG      68621  0x10c0d
𐰌‎  OLD TURKIC LETTER YENISEI AEB    68620  0x10c0c
𐰋‎  OLD TURKIC LETTER ORKHON AEB     68619  0x10c0b
𐰊‎  OLD TURKIC LETTER YENISEI AB     68618  0x10c0a
𐰉‎  OLD TURKIC LETTER ORKHON AB      68617  0x10c09
𐰈‎  OLD TURKIC LETTER YENISEI OE     68616  0x10c08
𐰇‎  OLD TURKIC LETTER ORKHON OE      68615  0x10c07
𐰆‎  OLD TURKIC LETTER ORKHON O       68614  0x10c06
𐰅‎  OLD TURKIC LETTER YENISEI E      68613  0x10c05
𐰄‎  OLD TURKIC LETTER YENISEI I      68612  0x10c04
𐰃‎  OLD TURKIC LETTER ORKHON I       68611  0x10c03
𐰂‎  OLD TURKIC LETTER YENISEI AE     68610  0x10c02
𐰁‎  OLD TURKIC LETTER YENISEI A      68609  0x10c01
𐰀‎  OLD TURKIC LETTER ORKHON A       68608  0x10c00
₺  TURKISH LIRA SIGN                 8378   0x20ba

Re: Entering Unicode characters

[Tianxiang Xiong]

Richard Stallman <rms <at> gnu.org> writes:

> 
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
> Completion schemes can make it easier to enter long names that you
> know.  They are no use for me in entering unusual characters, because 
> I don't know their names.
> 
> I want a system that lets me choose them by seeing them on the screen.
> I want to specify a language or script and see all its characters.
> 
> For instance, if I enter 'turkish' it should show me all the
> characters used in Turkish.  Then I could pick the dotless i from the
> buffer.
> 
> Would someone please implement this?
> 

It seems that what Richard wants is something like Microsoft Word's 
character insertion modal window. See 
http://wordfaqs.mvps.org/images/SpecChars2.gif for an example.

You select a subset of characters in a character set (e.g. Greek 
letters) and are shown a modal window with visual representations of all 
the characters. You can then choose a character to insert into the 
document.

In Emacs, of course, we don't have modal windows, but we can have 
buffers that display the characters line-by-line.

Re: A proposal for removing obsolete packages

[John Wiegley]

>>>>> John Wiegley <jwiegley@gmail.com> writes:

> Other than cl-compat, I'm OK with deleting these. Does anyone have any other
> objections?

According to objections from Drew about options, we'll keep that one as well.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

RE: Entering Unicode characters

[Drew Adams]

> It seems that what Richard wants is something like Microsoft Word's
> character insertion modal window. See
> http://wordfaqs.mvps.org/images/SpecChars2.gif for an example.
> 
> You select a subset of characters in a character set (e.g. Greek
> letters) and are shown a modal window with visual representations of all
> the characters. You can then choose a character to insert into the
> document.
> 
> In Emacs, of course, we don't have modal windows, but we can have
> buffers that display the characters line-by-line.

Yes.  It is the choosing of a subset (e.g. by language or other
category) that is currently missing, in particular.  Add that and
you can use `apu.el' for the display, choosing chars to insert,
get more info about, etc.

`apu.el' has a few ways to filter characters now, but filtering by
language etc. would be helpful.  And as Eli has mentioned before
(off list), better (quicker) than filtering the Unicode set would
be constructing only the needed subset.

Anyway, you can play with `apu.el', to see what it offers now.
http://www.emacswiki.org/emacs/AproposUnicode   ; Description
http://www.emacswiki.org/emacs/download/apu.el  ; Code

Re: Entering Unicode characters

[Stefan Monnier]

> It seems that what Richard wants is something like Microsoft Word's 
> character insertion modal window. See

FWIW, X-Symbol had something along the same lines.


        Stefan

Re: Entering Unicode characters

[Eli Zaretskii]

> From: Tianxiang Xiong <tianxiang.xiong@gmail.com>
> Date: Tue, 26 Jan 2016 17:08:29 +0000 (UTC)
> 
> Richard Stallman <rms <at> gnu.org> writes:
> 
> > I want a system that lets me choose them by seeing them on the screen.
> > I want to specify a language or script and see all its characters.
> > 
> > For instance, if I enter 'turkish' it should show me all the
> > characters used in Turkish.  Then I could pick the dotless i from the
> > buffer.
> > 
> > Would someone please implement this?
> > 
> 
> It seems that what Richard wants is something like Microsoft Word's 
> character insertion modal window. See 
> http://wordfaqs.mvps.org/images/SpecChars2.gif for an example.
> 
> You select a subset of characters in a character set (e.g. Greek 
> letters) and are shown a modal window with visual representations of all 
> the characters. You can then choose a character to insert into the 
> document.

Word lets you choose a _script_ (more accurately, a Unicode block
which supports a specific script), not a language.  E.g., you won't
find "Turkish" there, because there's no such block in Unicode.

Displaying characters for a language is a more fine-grained task.

Re: Entering Unicode characters

[Jean-Christophe Helary]

> 2016/01/26 23:49、Eli Zaretskii <eliz@gnu.org> のメール：
> 
>> I just selected the Turkish qwerty layout and when I hit g for 2 secs I get ğ as a possibility. When I am in the US layout I don't get this possibility. In US I get èéêëēėę for e, ÿ for y, ûüùúū for u (and similar options of other vowels), I also get ßśš for s. If I do the same with a capital letter (holding Shift + the letter) I get all the possible capital letters for the selected letter, for ex. S offers ŚŠ but no capital ß.
>> 
>> That's a very convenient way to visually check and select characters.
> 
> I think you are describing how Emacs input method work.  Try this:
> 
>  C-u C-\ turkish-postfix RET
> 
> then type i and watch the echo area.

What I see is the character I need to type to get something I don't see. That's not what Richard requested:

> 2016/01/26 18:39、Richard Stallman <rms@gnu.org> のメール：
> 
> I want a system that lets me choose them by seeing them on the screen.
> I want to specify a language or script and see all its characters.


> The advantage of Emacs input methods is that they don't depend on the
> locale.  (The disadvantage is that they only affect Emacs; but for
> someone who lives inside Emacs, that's not a problem ;-)

The OSX system does not either. I can use a French input system in a Turkish local with a JIS keyboard. Out of the box (and the fact that it is not available out of the box on GNU/Linux system is one of the main reason why I don't work on them, but that's not relevant here).

Jean-Christophe 

Re: A proposal for removing obsolete packages

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > It should be easy to add bindings for the missing characters to the C-x
  > 8 map.  For example, the dotless i and the dotted I could be added as
  > `C-x 8 . i' and `C-x 8 . I', resp.

It would be good to add some more characters to C-x 8, but we can't make
subcommands for everything in Unicode.

The approach of showing the characters for a specific unicode block,
or a specific language, or specific script, is feasible for a lot more
characters.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Entering Unicode characters

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > You might find the GNOME Character Map application satisfies your
  > needs.

I run Emacs in a console -- GNOME is not available.

I use that input feature in GNOME.  It is useful there.  But it does
not seem to have the dotless i.  It does not seem to aim for
completeness.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Entering Unicode characters

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I think you are describing how Emacs input method work.  Try this:

  >   C-u C-\ turkish-postfix RET

  > then type i and watch the echo area.

That would work for dotless i (though it would also semipermanently
switch my input method).  But it wouldn't work for (1) nonalphabetic
scripts whose input methods require special knowledge, and (2) special
characters such as ¶.

  > In my case it is OSX and it works like this: you choose a keyboard
  > layout and then by pressing a key you know can have variants for 2
  > secs the list of variants is displayed along with a number under
  > them. You can either use the arrows to select and enter the
  > character or hit the corresponding number.

That seems convenient for characters that are variants of letters,
such as dotless i.

However, there are characters that are not variants of letters.  The
scheme I proposed would be applicable to those sets of characters too.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Entering Unicode characters

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > The somewhat tricky part about the last item is that some languages
  > use non-contiguous portions of a block that is shared with other
  > languages.

It would be easy to allow a language name to specify a list
of character codes to display, instead of a range of codes,
when desirable.

    Word lets you choose a _script_ (more accurately, a Unicode block
    which supports a specific script), not a language.  E.g., you won't
    find "Turkish" there, because there's no such block in Unicode.

    Displaying characters for a language is a more fine-grained task.

A list of the characters special to that language would be easy to add.

So this feature could accept language names, and script names, and various
other ways of specifying which collection of characers to offer.

  > You select a subset of characters in a character set (e.g. Greek 
  > letters) and are shown a modal window with visual representations of all 
  > the characters. You can then choose a character to insert into the 
  > document.

Yes, that's it.


-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: A proposal for removing obsolete packages

[Alexis]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider 
> ]]] [[[ whether defending the US Constitution against all 
> enemies,     ]]] [[[ foreign or domestic, requires you to follow 
> Snowden's example. ]]]
>
>   > It should be easy to add bindings for the missing characters 
>   > to the C-x 8 map.  For example, the dotless i and the dotted 
>   > I could be added as `C-x 8 . i' and `C-x 8 . I', resp.
>
> It would be good to add some more characters to C-x 8, but we 
> can't make subcommands for everything in Unicode.
>
> The approach of showing the characters for a specific unicode 
> block, 

This, at least, is already done by the 'charmap' package:

https://github.com/lateau/charmap

(which is also available via MELPA, as well as Marmalade per the 
README.md).

Typing `M-x charmap` asks the user which Unicode block to display.


Alexis.

Re: Entering Unicode characters

[Eli Zaretskii]

> From: Jean-Christophe Helary <jean.christophe.helary@gmail.com>
> Date: Wed, 27 Jan 2016 06:41:57 +0900
> 
> 
> > I think you are describing how Emacs input method work.  Try this:
> > 
> >  C-u C-\ turkish-postfix RET
> > 
> > then type i and watch the echo area.
> 
> What I see is the character I need to type to get something I don't
> see.

??? What I see is the inputs I can type _and_ the characters that will
be inserted when I type each input, including the dotless i character.

That's not what Richard requested:

Maybe so, but then neither is what you proposed, because the Emacs
input methods work in the same way as what you described.

Re: Entering Unicode characters

[Jean-Christophe Helary]

> 2016/01/27 12:35、Eli Zaretskii <eliz@gnu.org> のメール：
> 
>> From: Jean-Christophe Helary <jean.christophe.helary@gmail.com>
>> Date: Wed, 27 Jan 2016 06:41:57 +0900
>> 
>> 
>>> I think you are describing how Emacs input method work.  Try this:
>>> 
>>> C-u C-\ turkish-postfix RET
>>> 
>>> then type i and watch the echo area.
>> 
>> What I see is the character I need to type to get something I don't
>> see.
> 
> ??? What I see is the inputs I can type _and_ the characters that will
> be inserted when I type each input, including the dotless i character.

Sorry, I see that for turkish-postfix, but not for ex. for latin-1-postfix, where the resulting character is not displayed.

Jean-Christophe 

Re: A proposal for removing obsolete packages

[Andreas Schwab]

Richard Stallman <rms@gnu.org> writes:

> It would be good to add some more characters to C-x 8, but we can't make
> subcommands for everything in Unicode.

Neither did iso-acc.

> The approach of showing the characters for a specific unicode block,
> or a specific language, or specific script, is feasible for a lot more
> characters.

Sure, but it's an orthogonal issue, iso-acc didn't offer that either.

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: A proposal for removing obsolete packages

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > This, at least, is already done by the 'charmap' package:

  > https://github.com/lateau/charmap

  > (which is also available via MELPA, as well as Marmalade per the 
  > README.md).

  > Typing `M-x charmap` asks the user which Unicode block to display.

Can we get legal papers for this?

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Entering Unicode characters

[Jean-Christophe Helary]

>>>> I think you are describing how Emacs input method work.  Try this:
>>>> 
>>>> C-u C-\ turkish-postfix RET
>>>> 
>>>> then type i and watch the echo area.
>>> 
>>> What I see is the character I need to type to get something I don't
>>> see.
>> 
>> ??? What I see is the inputs I can type _and_ the characters that will
>> be inserted when I type each input, including the dotless i character.
> 
> Sorry, I see that for turkish-postfix, but not for ex. for latin-1-postfix, where the resulting character is not displayed.

After checking the turkish-postfix input method, I noticed that the only character that is displayed "a la OSX" is the "i". All the other characters for which there are variants are *not* displayed and only their composing characters are. So emacs is actually *not* displaying resulting characters.

What I suggested would be an improvement is, for ex in latin-1-postfix, to have the following display when you hit "a":

a["'/^_`~] 1.ä 2.á 3.å 4.â 5.ª 6.à 7.ã

right now you only have:
a["'/^_`~]

Jean-Christophe 

Re: Entering Unicode characters

[Eli Zaretskii]

> From: Jean-Christophe Helary <jean.christophe.helary@gmail.com>
> Date: Fri, 29 Jan 2016 08:16:10 +0900
> 
> After checking the turkish-postfix input method, I noticed that the only character that is displayed "a la OSX" is the "i". All the other characters for which there are variants are *not* displayed and only their composing characters are. So emacs is actually *not* displaying resulting characters.

That's quail.el "working as designed" for these input methods.  Most
input methods for Far Eastern scripts show the alternatives almost
always.

> What I suggested would be an improvement is, for ex in latin-1-postfix, to have the following display when you hit "a":
> 
> a["'/^_`~] 1.ä 2.á 3.å 4.â 5.ª 6.à 7.ã
> 
> right now you only have:
> a["'/^_`~]

If that's what we want, we need to change quail.el where it decides
whether or not to show the alternative translations.

Perhaps Handa-san could comment on the reasons for this design in
'quail-get-translations': why does it show the list of possible
translations only sometimes?  Is it just an optimization?

Re: Entering Unicode characters

[handa]

In article <831t90ztgm.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> Perhaps Handa-san could comment on the reasons for this design in
> 'quail-get-translations': why does it show the list of possible
> translations only sometimes?  Is it just an optimization?

The deference is that an input method is determinitic or
non-deterministic.

Most input methods are deterministic, i.e. the defined key sequence
reaches to a target character (or a string), but Chinese input methods
are not deterministic, i.e. the defined key sequence reaches only to a
group of candidate characters.  A user has to select one of them by arrow
keys or by an index number.  So, for such an input method, showing
candidate characters is mandatory.

And, chinese input methods do not show possible translations as this:

> a["'/^_`~] 1.ä 2.á 3.å 4.â 5.ª 6.à 7.ã

The above means that when you type ", you get ä, when you type ', you
get á, ...

But chinese input methods show candidates as this (when you type "ni"
with chinese-py):

ni[aenu]  (01/??) 1.你 2.呢 3.尼 4.泥 5.逆 6.倪 7.匿 8.拟 9.腻 0.妮

This means that when you type 1, you get 你, when you type 2, you get
呢, ..., and when you type a (or e, n, u), the different candidates are
shown.

---
K. Handa
handa@gnu.org

Re: Entering Unicode characters

[Eli Zaretskii]

> From: handa <handa@gnu.org>
> Cc: jean.christophe.helary@gmail.com, emacs-devel@gnu.org
> Date: Sat, 30 Jan 2016 00:51:10 +0900
> 
> In article <831t90ztgm.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Perhaps Handa-san could comment on the reasons for this design in
> > 'quail-get-translations': why does it show the list of possible
> > translations only sometimes?  Is it just an optimization?
> 
> The deference is that an input method is determinitic or
> non-deterministic.
> 
> Most input methods are deterministic, i.e. the defined key sequence
> reaches to a target character (or a string), but Chinese input methods
> are not deterministic, i.e. the defined key sequence reaches only to a
> group of candidate characters.  A user has to select one of them by arrow
> keys or by an index number.  So, for such an input method, showing
> candidate characters is mandatory.

So you are saying that the candidates are shown only when selection is
done by typing a digit?

Can we add an optional feature where the candidates will be shown even
in deterministic input methods, although the way to choose the
candidates is not by typing a digit?  For example:

  a["'/^_`~]  " - ä  ' - á  / - å  ^ - â _ - ª   ` - à  ~ - ã

or something similar?

Thanks.

Re: Entering Unicode characters

[Clément Pit--Claudel]

[-- Attachment #1: Type: text/plain, Size: 822 bytes --]

On 01/29/2016 11:14 AM, Eli Zaretskii wrote:
> So you are saying that the candidates are shown only when selection is
> done by typing a digit?
> 
> Can we add an optional feature where the candidates will be shown even
> in deterministic input methods, although the way to choose the
> candidates is not by typing a digit?  For example:
> 
>   a["'/^_`~]  " - ä  ' - á  / - å  ^ - â _ - ª   ` - à  ~ - ã

I like this sugestion: it would be (more) consistent with Chinese input methods:

In chinese-py "1. 你" means "press 1 to insert 你". With you proposal, "~ - ã"
means "press ~ to insert ã".  I like the symmetry (maybe using "→", just like in
query-replace, would look a tiny bit nicer than " - " though?).  I wonder how this
would generalize to e.g. the TeX input method.

Clément.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Entering Unicode characters

[Jean-Christophe Helary]

> 2016/01/30 1:14、Eli Zaretskii <eliz@gnu.org> のメール：
> 
> Can we add an optional feature where the candidates will be shown even
> in deterministic input methods, although the way to choose the
> candidates is not by typing a digit?  For example:
> 
>  a["'/^_`~]  " - ä  ' - á  / - å  ^ - â _ - ª   ` - à  ~ - ã
> 
> or something similar?

The turkish-postfix method seems to offer both, at least with "i".

i[.]         (01/01) 1.ı 2.i

You can get "i" by either typing "." or by typing "1".

For some reason that is not generalized to other characters:

u["^]
i[.]         (01/01) 1.ı 2.i
o["]
a[^]
s[,]
g[^]
c[,]

For chinese-py, Kenichi proposed "ni":
ni[aenu]  (01/??) 1.你 2.呢 3.尼 4.泥 5.逆 6.倪 7.匿 8.拟 9.腻 0.妮

but since the default is 1, you actually don't have to enter 1 to input 你, and similarly, when you go all the way to niang you get 娘 by default and don't need to type 1 for input.

The implementation of turkish-postfix for "i" is superior to what we have on OSX (if offers discoverability, character based input *and* digit based input) but the implementation is not generalized to other letters and other input methods and thus is not satisfactory.

My original comment (description of the new OSX input system)  was a reply to Richard's request to have a way to view "all the characters" of a given language or script:

> I want a system that lets me choose them by seeing them on the screen.
> I want to specify a language or script and see all its characters.
> 
> For instance, if I enter 'turkish' it should show me all the
> characters used in Turkish.  Then I could pick the dotless i from the
> buffer.

Although what I am discussing here is strictly related to current input systems and not to a the new capability that Richard desires, I think the bigger issue here is first discoverability and then input method.

turkish-postfix offers discoverability for "i". chinese-py offers discoverability for all its characters (presumably), neither latin-1-postfix not latin-1-prefix offer any sort of discoverability and also lack predictability (why does /e= œ but /E=Æ ?, also, when you shift to french-prefix you actually do not get all the characters that you could possibly type in French - ±÷ªº¥°½¾¼ etc. - even though french should be a subset of latin-1, with the exception of œ/Œ.)

Then there is the input method when you have discovered the character you want to enter. I personally think that offering 2 options when possible (composing character *and* digit) is the best. Composing characters, and in the case of chinese-py we can argue that letters *are* composing characters for all practical purposes, are available for characters that are input frequently by a given user, and digits are there to input the occasional character. There is no logical need to not have a digit based input for composed characters.


Jean-Christophe

Re: Entering Unicode characters

[Eli Zaretskii]

> From: Jean-Christophe Helary <jean.christophe.helary@gmail.com>
> Date: Sat, 30 Jan 2016 12:03:12 +0900
> 
> Although what I am discussing here is strictly related to current input systems and not to a the new capability that Richard desires, I think the bigger issue here is first discoverability and then input method.
> 
> turkish-postfix offers discoverability for "i". chinese-py offers discoverability for all its characters (presumably), neither latin-1-postfix not latin-1-prefix offer any sort of discoverability and also lack predictability (why does /e= œ but /E=Æ ?, also, when you shift to french-prefix you actually do not get all the characters that you could possibly type in French - ±÷ªº¥°½¾¼ etc. - even though french should be a subset of latin-1, with the exception of œ/Œ.)
> 
> Then there is the input method when you have discovered the character you want to enter. I personally think that offering 2 options when possible (composing character *and* digit) is the best. Composing characters, and in the case of chinese-py we can argue that letters *are* composing characters for all practical purposes, are available for characters that are input frequently by a given user, and digits are there to input the occasional character. There is no logical need to not have a digit based input for composed characters.

Yes, the area of input methods in Emacs needs some long-term loving
care.  They only rarely get any attention, so most methods stay as
they were submitted years ago, and some are no longer up-to-date with
the changes in people expectations.

Would you like to work on making the Emacs input methods better?

Thanks.

Re: Entering Unicode characters

[Jean-Christophe Helary]

> 2016/01/30 16:30、Eli Zaretskii <eliz@gnu.org> のメール：
> 
> Yes, the area of input methods in Emacs needs some long-term loving
> care.  They only rarely get any attention, so most methods stay as
> they were submitted years ago, and some are no longer up-to-date with
> the changes in people expectations.
> 
> Would you like to work on making the Emacs input methods better?

I definitely would. But the only emacslisp I've written so far is the solutions for problem 1 and 2 of the Euler Project...
And I would not know where to start looking for.
I'm thinking that maybe trying to complete the french method so that it includes other characters from latin-1 would be a start ?

Jean-Christophe 

Re: Entering Unicode characters

[Eli Zaretskii]

> From: Jean-Christophe Helary <jean.christophe.helary@gmail.com>
> Date: Sat, 30 Jan 2016 16:36:22 +0900
> 
> > Would you like to work on making the Emacs input methods better?
> 
> I definitely would. But the only emacslisp I've written so far is the solutions for problem 1 and 2 of the Euler Project...
> And I would not know where to start looking for.

If you means start to looking for problems to fix, you have already
identified quite a few.

If you mean start looking for how input methods work under the hood,
the place to start is quail.el, and please ask questions here if you
don't understand something there.  AFAIR, most of it is not hard to
understand.

> I'm thinking that maybe trying to complete the french method so that it includes other characters from latin-1 would be a start ?

Yes, definitely a good start.  Thanks.

Also, I don't see your copyright assignment for Emacs on file.
Without it, we will only be able to accept limited contributions from
you.  Would you like me to send you the assignment form, to start the
legal paperwork rolling?

TIA

Re: Entering Unicode characters

[handa]

In article <83d1skxt0i.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> > Most input methods are deterministic, i.e. the defined key sequence
> > reaches to a target character (or a string), but Chinese input methods
> > are not deterministic, i.e. the defined key sequence reaches only to a
> > group of candidate characters.  A user has to select one of them by arrow
> > keys or by an index number.  So, for such an input method, showing
> > candidate characters is mandatory.

> So you are saying that the candidates are shown only when selection is
> done by typing a digit?

Not exactly.  The selection can also be done by C-f C-b C-n C-p and
arrow keys.  Anyway, candidates are shown only when an input method
doesn't define an exact key sequence to input a specific character.
Chinese-py only defines that "ni" is for inputting one of "你呢尼泥逆倪
匿拟腻妮霓昵溺旎睨鲵坭猊怩伲祢慝铌".  How many of those candidates are
shown depending on your window width.  If the width is too short to show
all candidates, a user have to type arrow keys or C-n, C-p, C-f, C-b to
see all candidates.  In that sense, chinese-py is non-deterministic.

> Can we add an optional feature where the candidates will be shown even
> in deterministic input methods, although the way to choose the
> candidates is not by typing a digit?  For example:

>   a["'/^_`~]  " - ä  ' - á  / - å  ^ - â _ - ª   ` - à  ~ - ã

> or something similar?

Of course adding such a feature is possible, but it's not just a
changing of condition of some code, but a completely new feature.  The
code to get "你呢尼泥逆倪匿拟腻妮霓昵溺旎睨鲵坭猊怩伲祢慝铌" when a user
type "ni" is easy because chinese-py directly defines it.  But to get
"äáåâªàã" when a user type "a" requires completely different code.

I can work on it, but it may take time.

---
K. Handa
handa@gnu.org

Re: Entering Unicode characters

[Eli Zaretskii]

> From: handa <handa@gnu.org>
> Cc: jean.christophe.helary@gmail.com, emacs-devel@gnu.org
> Date: Sat, 30 Jan 2016 21:16:46 +0900
> 
> > Can we add an optional feature where the candidates will be shown even
> > in deterministic input methods, although the way to choose the
> > candidates is not by typing a digit?  For example:
> 
> >   a["'/^_`~]  " - ä  ' - á  / - å  ^ - â _ - ª   ` - à  ~ - ã
> 
> > or something similar?
> 
> Of course adding such a feature is possible, but it's not just a
> changing of condition of some code, but a completely new feature.  The
> code to get "你呢尼泥逆倪匿拟腻妮霓昵溺旎睨鲵坭猊怩伲祢慝铌" when a user
> type "ni" is easy because chinese-py directly defines it.  But to get
> "äáåâªàã" when a user type "a" requires completely different code.
> 
> I can work on it, but it may take time.

I think it would be good to have such a feature, especially for input
methods such latin-ltx.el or uni-input,, which can potentially produce
a lot of characters whose appearance mostly has nothing in common with
what the user types.

Thanks.

Re: A proposal for removing obsolete packages

[Andrew Hyatt]

[-- Attachment #1: Type: text/plain, Size: 834 bytes --]

On Tue, Jan 26, 2016 at 12:32 PM John Wiegley <jwiegley@gmail.com> wrote:

> >>>>> John Wiegley <jwiegley@gmail.com> writes:
>
> > Other than cl-compat, I'm OK with deleting these. Does anyone have any
> other
> > objections?
>
> According to objections from Drew about options, we'll keep that one as
> well.
>

Sounds good.  The only other objections I've heard were from Richard
Stallman about the iso-* packages, but from what I can tell, the objections
have been answered and the conversation about those has moved to other
issues.  Richard, if I'm misunderstanding, let me know.

I'll remove these packages (except for cl-compat and options) tomorrow from
the trunk.


>
> --
> John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
> http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2
>

[-- Attachment #2: Type: text/html, Size: 1468 bytes --]

Re: Entering Unicode characters

[Helmut Eller]

[-- Attachment #1: Type: text/plain, Size: 559 bytes --]

On Tue, Jan 26 2016, Richard Stallman wrote:

> Would someone please implement this?

Here is my shot at this.  Load the attached file and start the browser
with "M-x ubb-browse". That displays the characters of the "Basic Latin"
block.  From there it's probably the easiest to use the menu to explore
the available commands.

I defined a sets of characters for some languages by looking at
Wikipedia.  This would obviously need more work by people who are
familiar with those languages.

The file is also available at: https://github.com/ellerh/ubb

Helmut


[-- Attachment #2: ubb.el --]
[-- Type: application/emacs-lisp, Size: 25671 bytes --]

;;; ubb.el --- Unicode block browser      -*-coding:utf-8; lexical-binding:t-*-

;; Copyright (C) 2016 Free Software Foundation, Inc.

;; GNU Emacs is free software: you can redistribute it and/or modify
;; it under the terms of the GNU General Public License as published by
;; the Free Software Foundation, either version 3 of the License, or
;; (at your option) any later version.

;; GNU Emacs is distributed in the hope that it will be useful,
;; but WITHOUT ANY WARRANTY; without even the implied warranty of
;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;; GNU General Public License for more details.

;; You should have received a copy of the GNU General Public License
;; along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.

;;; Commentary:

;; This is a mode for browsing the characters in Unicode Blocks or
;; other collections of characters, like Emacs' "charsets".
;;
;; Start the browser with "M-x ubb-browse".  That displays the
;; characters of the "Basic Latin" block.  From there it's probably
;; the easiest to use the menu to explore the available commands.
;;
;; The browser displays the current "set" of the current "group".  A
;; set is a collection of codepoints, e.g. the Unicode Block "Box
;; Drawing" is such a set.  Groups are collections of sets, e.g. all
;; Unicode Blocks form a group.
;;
;; Displaying large sets can be slow.  Don't hesitate to press C-g if
;; it takes too long.

;;; Code

(require 'cl-lib)

\f
;;; Some type definitions

;; A ubb--set represents a set of codepoints.
(cl-defstruct (ubb--set (:constructor ubb--make-set% (name ranges))
			(:constructor nil)
			(:predicate nil) (:copier nil))
  ;; NAME is a string for display purposes.
  (name "" :type string :read-only t)
  ;; RANGES is a list of the form ((START . END) ...)
  ;; START and END denote a range of codepoints.
  ;; START is inclusive; END is exclusive.
  (ranges () :type list :read-only t))

(defun ubb--check-range (x)
  (cl-assert (consp x))
  (let ((start (car x)) (end (cdr x)))
    (cl-assert (integerp start))
    (cl-assert (integerp end))
    (cl-assert (<= 0 start))
    (cl-assert (< start end))))

(defun ubb--sort-ranges (ranges)
  (sort (copy-sequence ranges)
	(lambda (r1 r2)
	  (< (car r1) (car r2)))))

(defun ubb--ranges-to-set (ranges)
  (cond ((null ranges) '())
	((null (cdr ranges)) ranges)
	(t
	 (let ((sorted (ubb--sort-ranges ranges)))
	   (nreverse
	    (cl-reduce (lambda (left+m right)
			 (cl-destructuring-bind ((s1 &rest e1) &rest m) left+m
			   (cl-destructuring-bind (s2 &rest e2) right
			     (cl-assert (<= s1 s2))
			     (cond ((<= e1 s2) ; no overlap
				    (cons right left+m))
				   (t
				    (cons (cons (min s1 s2)
						(max e1 e2))
					  m))))))
		       (cdr sorted) :initial-value (list (car sorted))))))))

(defun ubb--make-set (name ranges)
  (cl-check-type name string) (cl-check-type ranges list)
  (mapc #'ubb--check-range ranges)
  (let* ((sorted (ubb--ranges-to-set ranges)))
    (cl-loop for ((_ . end1) (start2 . _)) on sorted
	     while start2
	     do (cl-assert (<= end1 start2)))
    (ubb--make-set% name sorted)))

(defun ubb--set-size (set)
  (cl-loop for (start . end) in (ubb--set-ranges set)
	   sum (- end start)))

;; Call FUN for each codepoint in SET.  FUN receives two arguments:
;; and "index" and the codepoint.
(defun ubb--set-foreachi (fun set)
  (let ((i 0))
    (cl-loop for (start . end) in (ubb--set-ranges set) do
	     (cl-loop for codepoint from start below end do
		      (funcall fun i codepoint)
		      (cl-incf i)))))

(defun ubb--set-member? (set codepoint)
  (cl-loop for (start . end) in (ubb--set-ranges set)
	   thereis (and (<= start codepoint) (< codepoint end))))

;; Return a list of ranges corresponding to set of codepoints in the
;; string STRING.
(defun ubb--string-ranges (string)
  (let* ((sorted (cl-sort (copy-sequence string) #'<))
	 (ranges '())
	 (start nil)
	 (end nil))
    (cl-loop for c across sorted do
	     (cond ((not start)
		    (setq start c)
		    (setq end (1+ c)))
		   ((= c (1- end)))
		   ((= c end) (setq end (1+ c)))
		   (t (push (cons start end) ranges)
		      (setq start c)
		      (setq end (1+ c)))))
    (when start (push (cons start end) ranges))
    (reverse ranges)))

;; A ubb--group is used to represent a set of ubb--sets.
(cl-defstruct (ubb--group (:constructor ubb--make-group)
			  (:copier nil) (:predicate nil))
  ;; NAME is a string for display purposes
  (name "" :type string :read-only t)
  ;; SETS%% is either a function (with no arguments) that should return a
  ;; sequence of ubb--sets that belong to this group, or the cached result
  ;; of calling the function.
  (sets%% (error "Arg missing") :type (or function sequence))
  ;; HEADER is a function which receives a set as argument and
  ;; should the value for `header-line-format'.
  (header (error "Arg missing") :type function :read-only t))

(defun ubb--group-sets (group)
  (let ((sets%% (ubb--group-sets%% group)))
    (cl-etypecase sets%%
      (sequence sets%%)
      (function (setf (ubb--group-sets%% group) (funcall sets%%))))))

\f
;;; Unicode blocks

;; Local copy.
(defvar ubb--blocks-file-name
  (expand-file-name "../admin/unidata/Blocks.txt" data-directory))

;; Fallback if no local copy.
(defvar ubb--blocks-url "http://www.unicode.org/Public/UNIDATA/Blocks.txt")

;; Parse blocks in the format used by the Unicode Character Database.
(defun ubb--parse-blocks ()
  (goto-char (point-min))
  (let ((result '()))
    (while (re-search-forward
	    "^\\([0-9A-F]+\\)\\.\\.\\([0-9A-F]+\\);[ ]*\\([^ ].*\\)$"
	    nil t)
      (let* ((start (string-to-number (match-string 1) 16))
	     (end (1+ (string-to-number (match-string 2) 16)))
	     (name (match-string 3)))
	(push (ubb--make-set name (list (cons start end))) result)))
    (let ((r (cl-coerce (nreverse result) 'vector)))
      (cl-assert (let ((s (elt r 0)))
		   (and (equal (ubb--set-name s)  "Basic Latin")
			(equal (ubb--set-ranges s) '((0 . #x80))))))
      r)))

;; Load block information from a file or if the file isn't present
;; download it from unicode.org.
(defun ubb--load-blocks ()
  (with-temp-buffer
    (cond ((file-exists-p ubb--blocks-file-name)
	   (let ((coding-system-for-read 'binary))
	     (insert-file-contents-literally ubb--blocks-file-name)))
	  (t
	   (insert
	    (with-current-buffer (url-retrieve-synchronously ubb--blocks-url)
	      (re-search-forward "\n\n")
	      (buffer-substring (point) (point-max))))))
    (ubb--parse-blocks)))

(defun ubb--all-blocks ()
  "Return a sequence of all blocks."
  (ubb--load-blocks))

(defun ubb--find-block-by-codepoint (codepoint)
  (cl-find-if (lambda (block)
		(ubb--set-member? block codepoint))
	      (ubb--all-blocks)))

(defun ubb--unicode-block-header (set)
  (cl-destructuring-bind ((start &rest end)) (ubb--set-ranges set)
    (format "Block: %s  %04X..%04X" (ubb--set-name set) start end)))

(defvar ubb--unicode-blocks-group
  (ubb--make-group :name "Unicode blocks"
		   :sets%% #'ubb--all-blocks
		   :header #'ubb--unicode-block-header))

\f
;;; Charsets

;; NOTE: `map-charset-chars' can call the function with overlapping
;; ranges.  Also the cons cell for the range argument is updated so
;; it's a good idea to copy the contents instead of using the cons
;; cell.
(defun ubb--charset-to-set (charset)
  (let ((ranges '()))
    (map-charset-chars (lambda (from+to _)
			 (cl-destructuring-bind (from &rest to) from+to
			   (cl-assert (characterp from))
			   (cl-assert (characterp to))
			   (push (cons from (1+ to)) ranges)))
		       charset)
    (ubb--make-set (or (get-charset-property charset :long-name)
		       (get-charset-property charset :short-name)
		       (format "%s" charset))
		   ranges)))

(defun ubb--charsets-without-aliases ()
  (reverse ; ascii first, please
   (cl-remove-duplicates charset-list :key #'charset-plist)))

(defun ubb--all-charsets ()
  (mapcar #'ubb--charset-to-set (ubb--charsets-without-aliases)))

(defun ubb--charset-header (set)
  (format "Charset: %s" (ubb--set-name set)))

(defvar ubb--charsets-group
  (ubb--make-group :name "Charsets"
		   :sets%% #'ubb--all-charsets
		   :header #'ubb--charset-header))

\f
;;; Scripts

(defun ubb--all-scripts ()
  (let ((script2ranges (make-hash-table))
	(sets '()))
    (map-char-table (lambda (key script)
		      (setf (gethash script script2ranges)
			    (cons (cl-etypecase key
				    (character (cons key (1+ key)))
				    (cons (cons (car key) (1+ (cdr key)))))
				  (gethash script script2ranges))))
		    char-script-table)
    (maphash (lambda (script ranges)
	       (push (ubb--make-set (symbol-name script) ranges)
		     sets))
	     script2ranges)
    (cl-sort sets #'string< :key #'ubb--set-name)))

(defvar ubb--scripts-group
  (ubb--make-group :name "Scripts"
		   :sets%% #'ubb--all-scripts
		   :header (lambda (set)
			     (format "Script: %s" (ubb--set-name set)))))

\f
;;; Unicode categories

(defun ubb--all-unicode-categories ()
  (let ((cat2ranges (make-hash-table))
	(sets '()))
    (map-char-table (lambda (key cat)
		      (setf (gethash cat cat2ranges)
			    (cons (cl-etypecase key
				    (character (cons key (1+ key)))
				    (cons (cons (car key) (1+ (cdr key)))))
				  (gethash cat cat2ranges))))
		    unicode-category-table)
    (maphash (lambda (cat ranges)
	       (let* ((desc (char-code-property-description
			     'general-category cat))
		      (name (format "%s (%s)" cat desc)))
		 (push (ubb--make-set name ranges)
		       sets)))
	     cat2ranges)
    (cl-sort sets #'string< :key #'ubb--set-name)))

(defvar ubb--unicode-categories-group
  (ubb--make-group :name "Unicode categories"
		   :sets%% #'ubb--all-unicode-categories
		   :header (lambda (set)
			     (format "Unicode general category: %s"
				     (ubb--set-name set)))))

\f
;;; Languages

(defun ubb--all-languages ()
  (list (ubb--make-set "English" (ubb--string-ranges "“”‘’"))
	(ubb--make-set "French" (ubb--string-ranges "\
éàèùâêîôûëïüÿçœæ\
ÉÀÈÙÂÊÎÔÛËÏÜŸÇŒÆ\
«»‹›“”‘’€"))
	(ubb--make-set "German" (ubb--string-ranges "äöüßÄÖÜ„“‚‘’»«›‹€"))
	(ubb--make-set "Italian" (ubb--string-ranges "\
àèìòùéóî\
ÀÈÌÒÙÉÓÎ\
“”‘’«»‹›€"))
	(ubb--make-set "Spanish" (ubb--string-ranges "\
ñáéíóúü\
ÑÁÉÍÓÚÜ\
“”‘’«»¿¡"))
	(ubb--make-set "Turkish" (ubb--string-ranges "\
çşğıöü\
ÇŞĞİÖÜ\
“”‘’«»‹›"))
	(ubb--make-set "APL" (append (ubb--string-ranges "\
?⌈⌊⍴∼∣⍳⋆−+×÷,⌹○⍟⌽⊖⍋⍒⍎⍕⍉!−×÷⋆○?∈⌈⌊⍴↑↓⊥⊤∣,\/⍳⌹⌽⊖⍟⍕⍉!¨<≤=≥>≠∨∧⍱⍲/⌿\⍀.∘.")
				     `((,?⌶ . ,(1+ ?⍺)))))))

(defvar ubb--languages-group
  (ubb--make-group :name "Languages"
		   :sets%% #'ubb--all-languages
		   :header (lambda (set)
			     (format "Language: %s" (ubb--set-name set)))))


\f
;;; Games (important)

(defun ubb--all-game-sets ()
  (list (ubb--make-set "Chess" (ubb--string-ranges "♚♛♜♝♞♟♙♘♗♔♖♕"))
	(ubb--make-set "Domino" `((#x1f030 . #x1f0a0)))
	(ubb--make-set "Poker" `(,@(ubb--string-ranges "♠♤♥♡♦♢♣♧")
				 (#x1f0a0 . #x1f100)))))

(defun ubb--game-header (set)
  (format "Game: %s" (ubb--set-name set)))

(defvar ubb--games-group
  (ubb--make-group :name "Games"
		   :sets%% #'ubb--all-game-sets
		   :header #'ubb--game-header))

\f
;;; Groups

(defun ubb--all-groups ()
  (list ubb--unicode-blocks-group
	ubb--charsets-group
	ubb--scripts-group
	ubb--unicode-categories-group
	ubb--languages-group
	ubb--games-group))

(defvar ubb--read-group-name-history (list "Unicode blocks"))

(defun ubb--read-group-name (prompt)
  (let ((hist 'ubb--read-group-name-history)
	(completion-ignore-case t))
    (completing-read prompt
		     (mapcar #'ubb--group-name (ubb--all-groups))
		     nil t nil hist)))

(defun ubb--find-group-by-name (name)
  (cl-find name (ubb--all-groups) :key #'ubb--group-name :test #'equal))

(defvar ubb--read-set-name-history (list))

(defun ubb--read-set-name (group prompt)
  (let ((hist 'ubb--read-set-name-history)
	(completion-ignore-case t)
	(completion-styles (cl-adjoin 'substring completion-styles)))
    (completing-read prompt
		     (mapcar #'ubb--set-name (ubb--group-sets group))
		     nil t nil hist)))

(defun ubb--find-set-by-name (group name)
  (cl-find name (ubb--group-sets group)
	   :key #'ubb--set-name :test #'equal))

\f
;;; Display

(defconst ubb--space " ")
(defconst ubb--thin-space (string #x2009))

;; Insert space around the string from START to END so that the region
;; occupies approximately 2*FONT-WIDTH pixels.  Inserting space is
;; generally a good idea to "neutralize" combining marks.
(defun ubb--insert-space (win start end font-width)
  (let ((pixel-width (car (window-text-pixel-size win start end))))
    (cond ((<= pixel-width font-width)
	   (save-excursion
	     (goto-char start)
	     (insert ubb--space)))
	  ((< pixel-width (* 2 font-width))
	   (save-excursion
	     (goto-char start)
	     (insert ubb--thin-space)))
	  (t
	   ;; give up
	   ))))

(defface ubb-invisible
  '((t :inherit tooltip))
  "Face used for codepoints that would otherwise be invisible/transparent."
  :group 'ubb)

(defun ubb--propertize (string codepoint)
  (let ((s (propertize string 'codepoint codepoint
		       'help-echo #'ubb--help-echo)))
    (cond ((memq (get-char-code-property codepoint 'general-category)
		 '(Cf Zs Zl Zp))
	   (propertize s 'face 'ubb-invisible))
	  (t s))))

(defun ubb--insert-codepoint (win codepoint font-width)
  (let* ((s (cl-case codepoint
	      (?\n "^J")
	      (t (string codepoint))))
	 (s (ubb--propertize s codepoint))
	 (start (point))
	 (_ (insert s))
	 (end (point)))
    (ubb--insert-space win start end font-width)))

;; FIXME: only for compatibility with Emacs 24
(defun ubb--default-font-width ()
  (cond ((fboundp 'default-font-width)
	 (default-font-width))
	(t
	 (frame-char-width))))

(defvar ubb--right-margin 10)
(defvar ubb--left-margin 1)

;; Insert a set of codepoints, trying to create lines of equal width.
;;
;; The window WIN is needed for pixel measuring functions.  Inserting
;; large sets can be slow, so this calls redisplay for every line to
;; give some visual feedback to the user.  Also, the progress is shown
;; in percent in the echo area.
(defun ubb--insert-set (win set)
  (let* ((font-width (ubb--default-font-width))
	 (right-limit (- (window-width win t)
			 (* ubb--right-margin font-width)))
	 (line-start (point))
	 (set-size (ubb--set-size set)))
    (insert-char ?\s ubb--left-margin)
    (ubb--set-foreachi
     (lambda (i codepoint)
       (ubb--insert-codepoint win codepoint font-width)
       (let ((w (car (window-text-pixel-size win line-start (point)))))
	 (when (<= right-limit w)
	   (insert "\n")
	   (setq line-start (point))
	   (insert-char ?\s ubb--left-margin)
	   (message "%.f%%" (* 100.0 (/ (float i) set-size)))
	   (redisplay))))
     set)
    (message nil)))

(defun ubb--clear-codepoint-info ()
  (message nil))


;; Return a short description for codepoint.  This basically the
;; Unicode name.
(defun ubb--short-description (codepoint)
  (let* ((name (get-char-code-property codepoint 'name))
	 (old (get-char-code-property codepoint 'old-name))
	 (cat (get-char-code-property codepoint 'general-category))
	 (catdesc (char-code-property-description 'general-category cat)))
    (format "\"%c\" %s (%s: %s)" codepoint (or name old "[no name]")
	     cat catdesc)))

(defun ubb--show-codepoint-info (codepoint)
  (message "%s" (ubb--short-description codepoint)))

(defun ubb--current-codepoint (&optional noerror)
  (let ((codepoint (get-text-property (point) 'codepoint)))
    (cond (codepoint)
	  (noerror nil)
	  (t (user-error "No codepoint selected")))))

;; This is called from post-command-hook.
(defun ubb--codepoint-sensor ()
  (unless (current-message)
    (let ((codepoint (ubb--current-codepoint t)))
      (cond (codepoint (ubb--show-codepoint-info codepoint))
	    (t (ubb--clear-codepoint-info))))))

;; This called if the mouse pointer hovers around.
(defun ubb--help-echo (_ __ pos)
  (let ((codepoint (get-text-property pos 'codepoint)))
    (if codepoint (ubb--short-description codepoint))))

(defvar ubb--buffer-set)   ; The currently displayed ubb--set
(defvar ubb--buffer-group) ; The group to which ubb--buffer-set belongs

(define-derived-mode ubb-mode fundamental-mode "ubb"
  "Mode for viewing the characters in Unicode blocks and other charsets."
  (setq-local bidi-display-reordering nil)
  (setq-local truncate-lines t)
  (read-only-mode 1)
  (add-hook 'post-command-hook 'ubb--codepoint-sensor nil t))

(defun ubb--buffer-name () "*ubb*")

(defun ubb--get-buffer ()
  (or (get-buffer (ubb--buffer-name))
      (with-current-buffer (get-buffer-create (ubb--buffer-name))
	(ubb-mode)
	(current-buffer))))

(defun ubb--display-set (set)
  (let* ((inhibit-read-only t))
    (erase-buffer)
    (let* ((block-win (display-buffer (current-buffer)))
	   (_ (select-window block-win))
	   (pos (point)))
      (insert "\n")
      (ubb--insert-set block-win set)
      (set-window-point block-win pos)
      (current-buffer))))

;; This is the main entry point.
;;
;; Insert and display the ubb--set SET belonging to ubb--group GROUP.
;; Create a fresh buffer only if needed.  Display the buffer and
;; select its window.  Finally return the buffer.
(defun ubb--browse-set (group set)
  (with-current-buffer (ubb--get-buffer)
    (setq-local ubb--buffer-group group)
    (setq-local ubb--buffer-set set)
    (setq header-line-format (funcall (ubb--group-header group) set))
    (ubb--display-set set)))

\f
;;; Commands

(defun ubb-describe-codepoint-briefly ()
  "Show name and category of the current codepoint."
  (interactive)
  (ubb--show-codepoint-info (ubb--current-codepoint)))

(defun ubb-describe-codepoint ()
  "Describe the current codepoint."
  (interactive)
  (when (ubb--current-codepoint)
    (describe-char (point))))

(defun ubb--next-set (next)
  (let* ((set ubb--buffer-set)
	 (group ubb--buffer-group)
	 (all (ubb--group-sets group))
	 (i (cl-position set all :test #'equal))
	 (j (funcall next i)))
    (cond ((and (<= 0 j) (< j (length all)))
	   (ubb--browse-set group (elt all j)))
	  (t
	   (user-error "No more sets (in group %S)"
		       (ubb--group-name group))))))

(defun ubb-next-set ()
  "Browse the next set of the group."
  (interactive)
  (ubb--next-set #'1+))

(defun ubb-prev-set ()
  "Browse the previous set of the group."
  (interactive)
  (ubb--next-set #'1-))

(defun ubb-select-set-by-name (name)
  "Select the set to browse by name."
  (interactive (list (ubb--read-set-name ubb--buffer-group "Set name: ")))
  (let ((set (or (ubb--find-set-by-name ubb--buffer-group name)
		 (user-error "No set with name: %S" name))))
    (ubb--browse-set ubb--buffer-group set)))

(defun ubb--search-property (prop &optional backward)
  "Search the next text range where PROP is non-nil.
Return the value of PROP.
If BACKWARD is non-nil, search backward."
  (let ((next (cond (backward #'previous-single-char-property-change)
		    (t #'next-single-char-property-change)))
        (start (point))
        (value nil))
    (while (progn
             (goto-char (funcall next (point) prop))
             (not (or (setq value (get-char-property (point) prop))
                      (eobp)
                      (bobp)))))
    (cond (value)
          (t (goto-char start) nil))))

(defun ubb-forward-codepoint ()
  "Move cursor to the next codepoint."
  (interactive)
  (or (ubb--search-property 'codepoint nil)
      (user-error "No more codepoints")))

(defun ubb-backward-codepoint ()
  "Move cursor to the previous codepoint."
  (interactive)
  (or (ubb--search-property 'codepoint t)
      (user-error "No more codepoints")))

(defun ubb-browse-block (block &optional codepoint)
  "Browse the Unicode block BLOCK.
Interactively without prefix arg, prompt for the block name.
With negative prefix arg, use the character at point to find
the corresponding block.
With positive positive arg, prompt for the name or number of the codepoint
\(see `read-char-by-name')."
  (interactive
   (cond ((not current-prefix-arg)
	  (let ((name (ubb--read-set-name ubb--unicode-blocks-group
					  "Block name: ")))
	    (list (or (ubb--find-set-by-name ubb--unicode-blocks-group name)
		      (user-error "No block with named: %S" name)))))
	 (t
	  (let* ((codepoint
		  (cond ((< (prefix-numeric-value current-prefix-arg) 0)
			 (char-after))
			(t
			 (read-char-by-name
			  "Codepoint (Unicode name or hex): "))))
		 (block (or (ubb--find-block-by-codepoint codepoint)
			    (user-error "No block for codepoint: %X"
					codepoint))))
	    (list block codepoint)))))
  (with-current-buffer (ubb--browse-set ubb--unicode-blocks-group block)
    (when codepoint
      (search-forward (string codepoint))
      (backward-char))))

(defun ubb-browse-block-by-codepoint ()
  "Very similar to `ubb-browse-block'.
The only difference is that when invoked without prefix arg,
prompt for the codepoint instead for the Unicode block."
  (interactive)
  (let ((current-prefix-arg (or current-prefix-arg '(4))))
    (call-interactively #'ubb-browse-block)))

(defun ubb--browse-group (group)
  (ubb--browse-set group (elt (ubb--group-sets group) 0)))

(defun ubb-browse-group-by-name (name)
  "Prompt for a group name and display the first set in the group."
  (interactive (list (ubb--read-group-name "Group name: ")))
  (let ((group (or (ubb--find-group-by-name name)
		   (user-error "No group with name: %S" name))))
    (ubb--browse-group group)))

(defun ubb-reset-text-scale ()
  (interactive)
  (text-scale-set 0))

(defun ubb-redraw ()
  "Redraw the current set."
  (interactive)
  (ubb--display-set ubb--buffer-set))

(defun ubb-browse ()
  "Start the character browser."
  (interactive)
  (let ((buffer (get-buffer (ubb--buffer-name))))
    (cond (buffer (with-current-buffer buffer
		    (ubb--browse-set ubb--buffer-group ubb--buffer-set)))
	  (t (ubb--browse-group ubb--unicode-blocks-group)))))

(defun ubb-quit ()
  "Close the UBB window."
  (interactive)
  (quit-restore-window nil 'bury))

\f
;;; Menu

;; Return an uninterned symbol with FUN set as it function.  This is a
;; trick to put closures into menus.
(defun ubb--fake-menu-symbol (fun)
  (let ((sym (make-symbol "fake-menu-filter-symbol")))
    (fset sym fun)
    sym))

(defun ubb--build-set-menu (group)
  (mapcar (lambda (set)
	    (vector (ubb--set-name set)
		    (ubb--fake-menu-symbol
		     (lambda ()
		       (interactive)
		       (ubb--browse-set group set)))))
	  (ubb--group-sets group)))

(defun ubb--set-menu-filter (_others)
  (ubb--build-set-menu ubb--buffer-group))

(defun ubb--build-group-menu ()
  (mapcar (lambda (group)
	    (list (ubb--group-name group)
		  :filter (ubb--fake-menu-symbol
			   (lambda (_) (ubb--build-set-menu group)))))
	  (ubb--all-groups)))

(easy-menu-define nil ubb-mode-map
  "Menu for UBB mode."
  `("Character-Browser"
    ,@(ubb--build-group-menu)
    "--"
    ["Select group by name" ubb-browse-group-by-name]
    ["Select set by name" ubb-select-set-by-name]
    ["Select Unicode block by codepoint" ubb-browse-block-by-codepoint]
    ("Select set in current group"
     ("Sets in current group" :filter ubb--set-menu-filter)
     ["Next set in group" ubb-next-set :key-sequence ">"]
     ["Previous set in group" ubb-prev-set :key-sequence "<"])
    "--"
    ["Describe character briefly" ubb-describe-codepoint-briefly]
    ["Show character details" ubb-describe-codepoint]
    ("Movement"
     ["Move to next character" ubb-forward-codepoint]
     ["Move to previous character" ubb-backward-codepoint])
    ("Zoom"
     ["Increase scale factor" text-scale-increase]
     ["Decrease scale factor" text-scale-decrease]
     ["Reset scale factor" ubb-reset-text-scale])
    "--"
    ["Redraw" ubb-redraw]
    ["Quit" ubb-quit]))

\f
;;; Key bindings

(define-key ubb-mode-map (kbd "<SPC>") #'ubb-describe-codepoint-briefly)
(define-key ubb-mode-map (kbd "D") #'ubb-describe-codepoint)
(define-key ubb-mode-map (kbd "f") #'ubb-forward-codepoint)
(define-key ubb-mode-map (kbd "b") #'ubb-backward-codepoint)
(define-key ubb-mode-map (kbd "n") #'ubb-next-set)
(define-key ubb-mode-map (kbd "p") #'ubb-prev-set)
(define-key ubb-mode-map (kbd ">") #'ubb-next-set)
(define-key ubb-mode-map (kbd "<") #'ubb-prev-set)
(define-key ubb-mode-map (kbd "N") #'ubb-select-set-by-name)
(define-key ubb-mode-map (kbd "G") #'ubb-browse-group-by-name)
(define-key ubb-mode-map (kbd "C") #'ubb-browse-block-by-codepoint)
(define-key ubb-mode-map (kbd "+") #'text-scale-increase)
(define-key ubb-mode-map (kbd "-") #'text-scale-decrease)
(define-key ubb-mode-map (kbd "*") #'ubb-reset-text-scale)
(define-key ubb-mode-map (kbd "g") #'ubb-redraw)
(define-key ubb-mode-map (kbd "q") #'ubb-quit)

(provide 'ubb)

Re: Entering Unicode characters

[Eli Zaretskii]

> From: Helmut Eller <eller.helmut@gmail.com>
> Date: Sat, 06 Feb 2016 12:56:14 +0100
> 
> On Tue, Jan 26 2016, Richard Stallman wrote:
> 
> > Would someone please implement this?
> 
> Here is my shot at this.  Load the attached file and start the browser
> with "M-x ubb-browse". That displays the characters of the "Basic Latin"
> block.  From there it's probably the easiest to use the menu to explore
> the available commands.
> 
> I defined a sets of characters for some languages by looking at
> Wikipedia.  This would obviously need more work by people who are
> familiar with those languages.
> 
> The file is also available at: https://github.com/ellerh/ubb

Thanks, I think this is very good.

One aspect where some improvement is probably needed is when
displaying large ranges of characters.  For example, select
"Scripts->symbol" from the "Character-Browser" menu -- this displays
all the punctuation characters defined by Unicode.  You get 160 lines
that display something like 4K characters, which takes a long time to
display (because Emacs looks for a suitable font), and somewhat hard
to view.  Maybe we should display with those +/- widgets that allow
showing and hiding portions of the characters, so that each 256
characters could be shown or hidden separately?

Also, I think unassigned codepoints should not be shown, and the same
for Surrogates and Private-Area codepoints.  They will never yield
anything but those rectangles with a codepoint in hex inside them.

Finally, I think a command to insert the character at point into
another buffer would be a good addition.

Thanks again for working on this.

Re: Entering Unicode characters

[Helmut Eller]

On Sat, Feb 06 2016, Eli Zaretskii wrote:

> One aspect where some improvement is probably needed is when
> displaying large ranges of characters.  For example, select
> "Scripts->symbol" from the "Character-Browser" menu -- this displays
> all the punctuation characters defined by Unicode.  You get 160 lines
> that display something like 4K characters, which takes a long time to
> display (because Emacs looks for a suitable font), and somewhat hard
> to view.  Maybe we should display with those +/- widgets that allow
> showing and hiding portions of the characters, so that each 256
> characters could be shown or hidden separately?

Maybe. That's definitely more difficult to implement.  In general, there
is no way around that some sets are just big and that some people will
want see all characters.

Maybe some work could be done with "background" timers, e.g. insert 20
lines then wait a bit before starting the next timer and so on.

> Also, I think unassigned codepoints should not be shown, and the same
> for Surrogates and Private-Area codepoints.  They will never yield
> anything but those rectangles with a codepoint in hex inside them.

Not sure about this.  E.g. if somebody want to see an Unicode block he
may well want to see the range defined by the standard.  I also think
that some fonts do assign glyphs in private-use areas.  A part of the
high surrogates seems to be available for private-use as well.

However, using the same glyph for all unassigned codepoints might speed
up things a bit.

> Finally, I think a command to insert the character at point into
> another buffer would be a good addition.

It's not so easy to know where the charactor should be inserted.  Maybe
a marker could be used, but I suspect that that would be confusing.

I had implement a insert-char command that called the browser in a
recursive edit.  It worked but it definitely was confusing.

So my thinking is that the usual commands for copying a character to the
kill ring or copy&pasting it with the mouse is sufficient.

Helmut

Re: Entering Unicode characters

[Eli Zaretskii]

> From: Helmut Eller <eller.helmut@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Sat, 06 Feb 2016 14:59:15 +0100
> 
> > Also, I think unassigned codepoints should not be shown, and the same
> > for Surrogates and Private-Area codepoints.  They will never yield
> > anything but those rectangles with a codepoint in hex inside them.
> 
> Not sure about this.  E.g. if somebody want to see an Unicode block he
> may well want to see the range defined by the standard.  I also think
> that some fonts do assign glyphs in private-use areas.  A part of the
> high surrogates seems to be available for private-use as well.

If those use cases exist, they are rare.  So a customizable option,
off by default, would be useful, I think.

> > Finally, I think a command to insert the character at point into
> > another buffer would be a good addition.
> 
> It's not so easy to know where the charactor should be inserted.

I thought about prompting for it.  It's simple, and gets the job done.

> So my thinking is that the usual commands for copying a character to the
> kill ring or copy&pasting it with the mouse is sufficient.

Yes, but if the prompt for the buffer offers the MRU buffer as the
default, all one needs is type RET RET (the first one to invoke the
command, the second one to accept the default buffer), which is faster
and more efficient, I think.

Re: Entering Unicode characters

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

This looks pretty good.  It needs to be a little more
self-documenting; for instance, it should say how to switch to another
block.  It has a lot of commands:

    (define-key ubb-mode-map (kbd "<SPC>") #'ubb-describe-codepoint-briefly)
    (define-key ubb-mode-map (kbd "D") #'ubb-describe-codepoint)
    (define-key ubb-mode-map (kbd "f") #'ubb-forward-codepoint)
    (define-key ubb-mode-map (kbd "b") #'ubb-backward-codepoint)
    (define-key ubb-mode-map (kbd "n") #'ubb-next-set)
    (define-key ubb-mode-map (kbd "p") #'ubb-prev-set)
    (define-key ubb-mode-map (kbd ">") #'ubb-next-set)
    (define-key ubb-mode-map (kbd "<") #'ubb-prev-set)
    (define-key ubb-mode-map (kbd "N") #'ubb-select-set-by-name)
    (define-key ubb-mode-map (kbd "G") #'ubb-browse-group-by-name)
    (define-key ubb-mode-map (kbd "C") #'ubb-browse-block-by-codepoint)
    (define-key ubb-mode-map (kbd "+") #'text-scale-increase)
    (define-key ubb-mode-map (kbd "-") #'text-scale-decrease)
    (define-key ubb-mode-map (kbd "*") #'ubb-reset-text-scale)
    (define-key ubb-mode-map (kbd "g") #'ubb-redraw)
    (define-key ubb-mode-map (kbd "q") #'ubb-quit)

and it isn't obvious to me what they all do.

Many of the characters don't display on my console,
Have we got a way to tell, in Emacs Lisp, whether a certain
character code can actually display?  I think there is one.

Could this use that to show something meaningful,
for characters that can't really display?  For instance,
a sequence of 2 or 3 characters that somehow stands for
the real character?

For instance, the sequence << could stand for «, and
the sequence i-. could stand for dotless i.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Entering Unicode characters

[Clément Pit--Claudel]

[-- Attachment #1: Type: text/plain, Size: 307 bytes --]

On 02/07/2016 01:33 PM, Richard Stallman wrote:
> Many of the characters don't display on my console,
> Have we got a way to tell, in Emacs Lisp, whether a certain
> character code can actually display?  I think there is one.

Do we? That would be wonderful! But I don't think we do, unfortunately.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Entering Unicode characters

[John Wiegley]

>>>>> Richard Stallman <rms@gnu.org> writes:

> For instance, the sequence << could stand for «, and the sequence i-. could
> stand for dotless i.

Waiting to see what you display for white and black snowman.. :)

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Entering Unicode characters

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Waiting to see what you display for white and black snowman.. :)

I don't think I have ever seen that character -- I don't know
what it looks like.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Entering Unicode characters

[Stefan Monnier]

>> Many of the characters don't display on my console,
>> Have we got a way to tell, in Emacs Lisp, whether a certain
>> character code can actually display?  I think there is one.
> Do we? That would be wonderful! But I don't think we do, unfortunately.

char-displayable-p?


        Stefan

Re: Entering Unicode characters

[Stefan Monnier]

>> For instance, the sequence << could stand for «, and the sequence i-. could
>> stand for dotless i.

lisp/term/internal.el has quite a few such mappings already, so we could
try and reuse that, if we decide we want this feature.

> Waiting to see what you display for white and black snowman.. :)

Sorry, I couldn't see them, they were hiding behind the green bigfoot.


        Stefan

Re: Entering Unicode characters

[Andreas Schwab]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> char-displayable-p?

That is useless for a TTY frame.  It can only check whether the
character is encodable in the terminal coding system, which is trivially
true for UTF-8.  Even for a multi-font display it is only approximate.

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: Entering Unicode characters

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Date: Mon, 08 Feb 2016 08:02:42 -0500
> Cc: eller.helmut@gmail.com, emacs-devel@gnu.org
> 
>   > Waiting to see what you display for white and black snowman.. :)
> 
> I don't think I have ever seen that character -- I don't know
> what it looks like.

Like a small snowman.  (There's no white snowman, AFAIK, it's called
"snowman without snow".)

Re: Entering Unicode characters

[Alexis]

Richard Stallman <rms@gnu.org> writes:

>> Waiting to see what you display for white and black 
>> snowman.. :)
>
> I don't think I have ever seen that character -- I don't know 
> what it looks like.

My guess is that, in response to you wondering:

> Could this use that to show something meaningful, for characters 
> that can't really display?  For instance, a sequence of 2 or 3 
> characters that somehow stands for the real character? 
 
> For instance, the sequence << could stand for «, and the 
> sequence i-. could stand for dotless i. 

John might have been referring to the large swathes of Unicode 
that don't necessarily have an 'obvious' representation in such a 
form. For example:

* CJK-related characters;
* the "Musical symbols" characters;
* many of the "Miscellaneous symbols" characters.

Additionally, there are lots of opportunities for bikeshedding 
over the representations of certain characters. Take ☺, WHITE 
SMILING FACE. Off the top of my head, i can imagine that being 
represented by ':-)', but i frequently encounter people using ':)' 
and '(-:' to represent the 'smiling face' concept instead.

Together, these issues seem to me to suggest that it might be 
non-trivial to attempt to build a table of ASCII representations 
of various Unicode characters. 


Alexis.

Re: Entering Unicode characters

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > char-displayable-p?

  > That is useless for a TTY frame.

It should be possible to make it smarter
if it is told which font the tty is displaying.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Entering Unicode characters

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Like a small snowman.

Without seeing it, I think maybe a combination including 8 might be good
to represent it.  Perhaps `8^ or \8/.

			   (There's no white snowman, AFAIK, it's called
  > "snowman without snow".)

Is that the space character?

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Entering Unicode characters

[Dale Snell]

[-- Attachment #1: Type: text/plain, Size: 1159 bytes --]

On Tue, 09 Feb 2016 07:14:50 -0500, in message
E1aT7By-00056G-8Y@fencepost.gnu.org, Richard Stallman wrote:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
>   > Like a small snowman.
> 
> Without seeing it, I think maybe a combination including 8 might be
> good to represent it.  Perhaps `8^ or \8/.
> 
> 			   (There's no white snowman, AFAIK, it's
> called
>   > "snowman without snow".)
> 
> Is that the space character?
> 

No, it's not a space.  Here they are: "⛄" "⛇".  The first one is
U+26C4, SNOWMAN WITHOUT SNOW (a.k.a., light snow); the second one
is U+26C7 BLACK SNOWMAN (a.k.a., heavy snow).

BTW, the Turkish and Azerbaijani dotless i is U+0131 LATIN SMALL
LETTER DOTLESS I: "ı".  I had to use the search function in the
Gnome character map to find it.  Search for "dotless" and check
the "Search in character details" box.

--Dale

-- 
Sign in coffee shop:  "Children left unattended will be given an
espresso and a Free Puppy."

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

RE: Entering Unicode characters

[Drew Adams]

> Together, these issues seem to me to suggest that it might be
> non-trivial to attempt to build a table of ASCII representations
> of various Unicode characters.

Next up: Unicode emoji.
http://unicode.org/emoji/charts/full-emoji-list.html

Re: Entering Unicode characters

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> CC: johnw@gnu.org, eller.helmut@gmail.com, emacs-devel@gnu.org
> Date: Tue, 09 Feb 2016 07:14:50 -0500
> 
> 			   (There's no white snowman, AFAIK, it's called
>   > "snowman without snow".)
> 
> Is that the space character?

:-) I wish.  No, it's the same shape as the other one, but just a
contour (the other one is filled with foreground color).

Re: Entering Unicode characters

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Date: Tue, 09 Feb 2016 07:14:02 -0500
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
>   > > char-displayable-p?
> 
>   > That is useless for a TTY frame.
> 
> It should be possible to make it smarter
> if it is told which font the tty is displaying.

When Emacs displays on a TTY frame, the font is out of its control.
The font is determined by the terminal emulator or the console config
files, and Emacs wouldn't know where to find the font even when told
its name.

In particular, if you login from another machine, the font used by the
terminal emulator is not accessible at all for Emacs that runs on the
remote.

So I think what you envision is impossible, or at least impractical.

Re: Entering Unicode characters

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Additionally, there are lots of opportunities for bikeshedding 
  > over the representations of certain characters. Take ☺, WHITE 
  > SMILING FACE. Off the top of my head, i can imagine that being 
  > represented by ':-)', but i frequently encounter people using ':)' 
  > and '(-:' to represent the 'smiling face' concept instead.

Just pick one.  Since the representation will be used only in this
one feature, the choice is not crucial.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Entering Unicode characters

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > 			   (There's no white snowman, AFAIK, it's called
  > >   > "snowman without snow".)

  > > Is that the space character?

  > No, it's not a space.

I was making a joke.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Entering Unicode characters

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Next up: Unicode emoji.
  > http://unicode.org/emoji/charts/full-emoji-list.html

We can probably find emoticons to stand for some of them.
And if we can't...so what?

I suggested that it would be good to come up with short ASCII visual
representations of SOME of these Unicode characters _for ubb-browse_.

You seem to be trying to prove we can't do this for ALL the Unicode
characters, but I never said we could do them all.  Doing it for just
some characters will be useful.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Entering Unicode characters

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > When Emacs displays on a TTY frame, the font is out of its control.
  > The font is determined by the terminal emulator or the console config
  > files,

That's right.

	   and Emacs wouldn't know where to find the font even when told
  > its name.

Why not?  Emacs could know the usual place where these font files are
stored, and some Lisp code could find a particular font file
just like the command that tells the terminal which font to use.

  > In particular, if you login from another machine, the font used by the
  > terminal emulator is not accessible at all for Emacs that runs on the
  > remote.

In theory, it might not be present on the machine Emacs is running on.
But in practice, it probably is present, if they are both GNU/Linux.

And if the font is NOT present there, you could copy it there and tell
Emacs the file name to find it in.

So I think what I proposed is doable, practically speaking.

It would take work, of course, but there is nothing difficult about it
in principle.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Entering Unicode characters

[Alexis]

Richard Stallman <rms@gnu.org> writes:

> I suggested that it would be good to come up with short ASCII 
> visual representations of SOME of these Unicode characters _for 
> ubb-browse_.
>
> You seem to be trying to prove we can't do this for ALL the 
> Unicode characters, but I never said we could do them all. 
> Doing it for just some characters will be useful.

What i was trying to say was that, setting aside the difficulty of 
create an ASCII representation for /all/ Unicode codepoints, i 
could foresee a lot of energy being sunk into debates over the 
'best' representation for even /some/ of the codepoints. You 
wrote:

> Just pick one.  Since the representation will be used only in 
> this one feature, the choice is not crucial. 

i tend to agree, but my experience with these sort of situations 
suggests that there will be a number of people feeling that the 
colour of the bikeshed /is/ a crucial matter. :-) But i guess 
we'll see what happens!


Alexis.

RE: Entering Unicode characters

[Drew Adams]

>   > Next up: Unicode emoji.
>   > http://unicode.org/emoji/charts/full-emoji-list.html
> 
> We can probably find emoticons to stand for some of them.
> And if we can't...so what?
> 
> I suggested that it would be good to come up with short ASCII visual
> representations of SOME of these Unicode characters _for ubb-browse_.
> 
> You seem to be trying to prove we can't do this for ALL the Unicode
> characters, but I never said we could do them all.  Doing it for just
> some characters will be useful.

I was kidding about emoji.

(You might be confusing me with someone else who spoke against
your idea of ASCII representations.)

Re: Entering Unicode characters

[Elias Mårtenson]

[-- Attachment #1: Type: text/plain, Size: 855 bytes --]

On 10 February 2016 at 10:07, Drew Adams <drew.adams@oracle.com> wrote:


> > You seem to be trying to prove we can't do this for ALL the Unicode
> > characters, but I never said we could do them all.  Doing it for just
> > some characters will be useful.
>
> I was kidding about emoji.
>
> (You might be confusing me with someone else who spoke against
> your idea of ASCII representations.)
>

I know this can't be used in a summary screen containing more than one
character, but how about drawing a bitmap version of the character? If you
use a, say, 32×32 grid you can represent pretty much anything. I can
envision this being a useful thing to include in the C-u C-x = output.

At least this would allow a user on an ASCII terminal (or even one on a
graphical terminal with incomplete fonts) to see what a character looks
like.

[-- Attachment #2: Type: text/html, Size: 1246 bytes --]

Re: Entering Unicode characters

[Jose E. Marchesi]

    
I propose to use the ASCII sequence 'vi' to represent U+1F4A9.

Re: Entering Unicode characters

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I know this can't be used in a summary screen containing more than one
  > character, but how about drawing a bitmap version of the character? If you
  > use a, say, 32×32 grid you can represent pretty much anything. I can
  > envision this being a useful thing to include in the C-u C-x = output.

  > At least this would allow a user on an ASCII terminal (or even one on a
  > graphical terminal with incomplete fonts) to see what a character looks
  > like.

It would be useful to be able to see a character that way, but it
would have to work via requesting to see one specific character.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Entering Unicode characters

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > i tend to agree, but my experience with these sort of situations 
  > suggests that there will be a number of people feeling that the 
  > colour of the bikeshed /is/ a crucial matter.

I think John Wiegley can end such arguments, if necessary.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Entering Unicode characters

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> CC: schwab@suse.de, monnier@iro.umontreal.ca, emacs-devel@gnu.org
> Date: Tue, 09 Feb 2016 19:12:34 -0500
> 
> 	   and Emacs wouldn't know where to find the font even when told
>   > its name.
> 
> Why not?  Emacs could know the usual place where these font files are
> stored, and some Lisp code could find a particular font file
> just like the command that tells the terminal which font to use.

How can it know?  Emacs compiled --without-x doesn't even have
font-related code compiled in.  And even if it had, the terminal
emulator might very well use a different font back-end, which keeps
font information in a different place.  Especially if the terminal
emulator runs on a different OS.

>   > In particular, if you login from another machine, the font used by the
>   > terminal emulator is not accessible at all for Emacs that runs on the
>   > remote.
> 
> In theory, it might not be present on the machine Emacs is running on.
> But in practice, it probably is present, if they are both GNU/Linux.

You can still have different versions of the same font even in this
special case.

> And if the font is NOT present there, you could copy it there and tell
> Emacs the file name to find it in.

So maybe a better solution would be a feature that would tell Emacs
explicitly, by user initiative, which characters are displayable?  The
user would run some utility to find that out, and then communicate the
results to Emacs via some customization.  That sounds more reliable to
me than have Emacs assume something about fonts in two different
places.

Re: Entering Unicode characters

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Why not?  Emacs could know the usual place where these font files are
  > > stored, and some Lisp code could find a particular font file
  > > just like the command that tells the terminal which font to use.

  > How can it know?  Emacs compiled --without-x doesn't even have
  > font-related code compiled in.  And even if it had, the terminal
  > emulator might very well use a different font back-end, which keeps
  > font information in a different place.  Especially if the terminal
  > emulator runs on a different OS.

So the feature won't work in that case.

We don't have to choose between perfection and nothing.

  > So maybe a better solution would be a feature that would tell Emacs
  > explicitly, by user initiative, which characters are displayable?  The
  > user would run some utility to find that out, and then communicate the
  > results to Emacs via some customization.  That sounds more reliable to
  > me than have Emacs assume something about fonts in two different
  > places.

That's a good idea.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Entering Unicode characters

[John Wiegley]

[-- Attachment #1: Type: text/plain, Size: 644 bytes --]

>>>>> Richard Stallman <rms@gnu.org> writes:

>> i tend to agree, but my experience with these sort of situations suggests
>> that there will be a number of people feeling that the colour of the
>> bikeshed /is/ a crucial matter.

> I think John Wiegley can end such arguments, if necessary.

Indeed. Now, Emacs-Tangents welcomes any and all bikeshedding discussions, so
please feel free if you believe that antique lace and desert bone should be
considered separate colors. :)

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 629 bytes --]

Re: Entering Unicode characters

[Helmut Eller]

[-- Attachment #1: Type: text/plain, Size: 937 bytes --]

On Sat, Feb 06 2016, Eli Zaretskii wrote:

> Maybe we should display with those +/- widgets that allow
> showing and hiding portions of the characters, so that each 256
> characters could be shown or hidden separately?

Here is an updated version that uses +/- buttons.

> Also, I think unassigned codepoints should not be shown, and the same
> for Surrogates and Private-Area codepoints.  They will never yield
> anything but those rectangles with a codepoint in hex inside them.

The categories Cn, Cs, and Co are now hidden by default.

For some reason, raw bytes also belong to the Cn category so are not
shown by default.

I discovered that Emacs displays 8-bit control codes the same as raw
bytes, e.g. the char-codes #x82 and #x3FFF82 are both displayed as \202.
I think this is confusing.

> Finally, I think a command to insert the character at point into
> another buffer would be a good addition.

I added that, too.

Helmut


[-- Attachment #2: ubb.el --]
[-- Type: application/emacs-lisp, Size: 36463 bytes --]

;;; ubb.el --- Unicode block browser      -*-coding:utf-8; lexical-binding:t-*-

;; Copyright (C) 2016 Free Software Foundation, Inc.

;; GNU Emacs is free software: you can redistribute it and/or modify
;; it under the terms of the GNU General Public License as published by
;; the Free Software Foundation, either version 3 of the License, or
;; (at your option) any later version.

;; GNU Emacs is distributed in the hope that it will be useful,
;; but WITHOUT ANY WARRANTY; without even the implied warranty of
;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;; GNU General Public License for more details.

;; You should have received a copy of the GNU General Public License
;; along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.

;;; Commentary:

;; This is a mode for browsing the characters in Unicode Blocks or
;; other collections of characters, like Emacs' "charsets".
;;
;; Start the browser with "M-x ubb-browse".  That displays the
;; characters of the "Basic Latin" block.  From there it's probably
;; the easiest to use the menu to explore the available commands.
;;
;; The browser displays the current "set" of the current "group".  A
;; set is a collection of codepoints, e.g. the Unicode Block "Box
;; Drawing" is such a set.  Groups are collections of sets, e.g. all
;; Unicode Blocks form a group.  The commands
;; `ubb-browse-group-by-name'/`ubb-select-set-by-name' can used to
;; select the current group/set.

;;; Code

(require 'cl-lib)

;; term/internal.el doesn't provide any feature.  Not sure what the proper
;; way is to load it.
(defvar IT-unicode-translations)
(unless (boundp 'IT-unicode-translations)
  (load "term/internal"))

\f
;;; Some type definitions

;; A ubb--set represents a set of codepoints.
(cl-defstruct (ubb--set (:constructor ubb--make-set% (name ranges%%))
			(:constructor nil)
			(:predicate nil) (:copier nil))
  ;; NAME is a string for display purposes.
  (name "" :type string :read-only t)
  ;; RANGES%% is either a list or a function.
  ;; If a list then it looks like ((START . END) ...)
  ;; START and END denote a range of codepoints.
  ;; START is inclusive; END is exclusive.
  ;; If a function, it returns a list of ranges of the above form.
  (ranges%% () :type (or list function) :read-only t))

(defun ubb--make-set (name ranges)
  (cl-check-type name string) (cl-check-type ranges (or list function))
  (ubb--make-set% name ranges))

(defun ubb--sort-ranges (ranges)
  (sort (copy-sequence ranges)
	(lambda (r1 r2)
	  (< (car r1) (car r2)))))

;; A "rangeset" is a list of ranges.  The ranges in a rangeset are
;; non-overlapping and ordered by start point.
(defun ubb--ranges-to-rangeset (ranges)
  (cond ((null ranges) '())
	((null (cdr ranges)) ranges)
	(t
	 (let ((sorted (ubb--sort-ranges ranges)))
	   (nreverse
	    (cl-reduce (lambda (left+m right)
			 (cl-destructuring-bind ((s1 &rest e1) &rest m) left+m
			   (cl-destructuring-bind (s2 &rest e2) right
			     (cl-assert (<= s1 s2))
			     (cond ((<= e1 s2) ; no overlap
				    (cons right left+m))
				   (t
				    (cons (cons (min s1 s2)
						(max e1 e2))
					  m))))))
		       (cdr sorted) :initial-value (list (car sorted))))))))

(defun ubb--set-ranges% (set)
  (let ((ranges%% (ubb--set-ranges%% set)))
    (cl-etypecase ranges%%
      (function (funcall ranges%%))
      (list ranges%%))))

(defun ubb--set-rangeset (set)
  (ubb--ranges-to-rangeset (ubb--set-ranges% set)))

(defun ubb--rangeset-member (rangeset codepoint)
  (cl-loop for (start . end) in rangeset
	   thereis (and (<= start codepoint) (< codepoint end))))

(defun ubb--set-member (set codepoint)
  (ubb--rangeset-member (ubb--set-rangeset set) codepoint))

(defun ubb--rangeset-foreach (rangeset fun)
  (cl-loop for (start . end) in rangeset
	   do (cl-loop for codepoint from start below end
		       do (funcall fun codepoint))))

(defun ubb--rangeset-size (rangeset)
  (cl-loop for (start . end) in rangeset
	   sum (- end start)))

(defun ubb--rangeset-difference (x y)
  (cond ((null y) x)
	(t
	 (let ((result '()))
	   (while (and x y)
	     (cl-destructuring-bind ((s1 &rest e1) &rest r1) x
	       (cl-destructuring-bind (s2 &rest e2) (car y)
		 (cond ((<= e1 s2) (push (pop x) result))
		       ((<= e2 s1) (pop y))
		       ((and (<= s2 s1) (<= e1 e2))
			(pop x))
		       ((< s1 s2)
			(push (cons s1 s2) result)
			(setq x (cons (cons s2 e1) r1)))
		       (t
			(cl-assert (<= s2 s1))
			(cl-assert (< e2 e1))
			(setq x (cons (cons e2 e1) r1)))))))
	   (cl-revappend result x)))))

;; Return a list of ranges corresponding to set of codepoints in the
;; string STRING.
(defun ubb--string-ranges (string)
  (let* ((sorted (cl-sort (copy-sequence string) #'<))
	 (ranges '())
	 (start nil)
	 (end nil))
    (cl-loop for c across sorted do
	     (cond ((not start)
		    (setq start c)
		    (setq end (1+ c)))
		   ((= c (1- end)))
		   ((= c end) (setq end (1+ c)))
		   (t (push (cons start end) ranges)
		      (setq start c)
		      (setq end (1+ c)))))
    (when start (push (cons start end) ranges))
    (reverse ranges)))

;; A ubb--group is used to represent a set of ubb--sets.
(cl-defstruct (ubb--group (:constructor ubb--make-group)
			  (:copier nil) (:predicate nil))
  ;; NAME is a string for display purposes
  (name "" :type string :read-only t)
  ;; SETS%% is either a function (with no arguments) that should return a
  ;; sequence of ubb--sets that belong to this group, or the cached result
  ;; of calling the function.
  (sets%% (error "Arg missing") :type (or function sequence))
  ;; HEADER is a function which receives a set as argument and
  ;; should the value for `header-line-format'.
  (header (error "Arg missing") :type function :read-only t))

(defun ubb--group-sets (group)
  (let ((sets%% (ubb--group-sets%% group)))
    (cl-etypecase sets%%
      (sequence sets%%)
      (function (setf (ubb--group-sets%% group) (funcall sets%%))))))

\f
;;; Unicode blocks

;; Local copy.
(defvar ubb--blocks-file-name
  (expand-file-name "../admin/unidata/Blocks.txt" data-directory))

;; Fallback if no local copy.
(defvar ubb--blocks-url "http://www.unicode.org/Public/UNIDATA/Blocks.txt")

;; Parse blocks in the format used by the Unicode Character Database.
(defun ubb--parse-blocks ()
  (goto-char (point-min))
  (let ((result '()))
    (while (re-search-forward
	    "^\\([0-9A-F]+\\)\\.\\.\\([0-9A-F]+\\);[ ]*\\([^ ].*\\)$"
	    nil t)
      (let* ((start (string-to-number (match-string 1) 16))
	     (end (1+ (string-to-number (match-string 2) 16)))
	     (name (match-string 3)))
	(push (ubb--make-set name (list (cons start end))) result)))
    (let ((r (cl-coerce (nreverse result) 'vector)))
      (cl-assert (let ((s (elt r 0)))
		   (and (equal (ubb--set-name s)  "Basic Latin")
			(equal (ubb--set-rangeset s) '((0 . #x80))))))
      r)))

;; Load block information from a file or if the file isn't present
;; download it from unicode.org.
(defun ubb--load-blocks ()
  (with-temp-buffer
    (cond ((file-exists-p ubb--blocks-file-name)
	   (let ((coding-system-for-read 'binary))
	     (insert-file-contents-literally ubb--blocks-file-name)))
	  (t
	   (insert
	    (with-current-buffer (url-retrieve-synchronously ubb--blocks-url)
	      (re-search-forward "\n\n")
	      (buffer-substring (point) (point-max))))))
    (ubb--parse-blocks)))

(defvar ubb--all-blocks-cache nil)

(defun ubb--all-blocks ()
  "Return a sequence of all blocks."
  (or ubb--all-blocks-cache
      (setq ubb--all-blocks-cache (ubb--load-blocks))))

(defun ubb--find-block-by-codepoint (codepoint)
  (cl-find-if (lambda (block)
		(ubb--set-member block codepoint))
	      (ubb--all-blocks)))

(defun ubb--unicode-block-header (set)
  (cl-destructuring-bind ((start &rest end)) (ubb--set-rangeset set)
    (format "Block: %s  %04X..%04X" (ubb--set-name set) start end)))

(defvar ubb--unicode-blocks-group
  (ubb--make-group :name "Unicode blocks"
		   :sets%% #'ubb--all-blocks
		   :header #'ubb--unicode-block-header))

\f
;;; Charsets

;; NOTE: `map-charset-chars' can call the function with overlapping
;; ranges.  Also the cons cell for the range argument is updated so
;; it's a good idea to copy the contents instead of using the cons
;; cell.
(defun ubb--charset-ranges (charset)
  (let ((ranges '()))
    (map-charset-chars (lambda (from+to _)
			 (cl-destructuring-bind (from &rest to) from+to
			   (cl-assert (characterp from))
			   (cl-assert (characterp to))
			   (push (cons from (1+ to)) ranges)))
		       charset)
    ranges))

(defun ubb--charset-to-set (charset)
  (ubb--make-set (or (get-charset-property charset :long-name)
		     (get-charset-property charset :short-name)
		     (format "%s" charset))
		 (lambda () (ubb--charset-ranges charset))))

(defun ubb--charsets-without-aliases ()
  (reverse ; ascii first, please
   (cl-remove-duplicates charset-list :key #'charset-plist)))

(defun ubb--all-charsets ()
  (mapcar #'ubb--charset-to-set (ubb--charsets-without-aliases)))

(defun ubb--charset-header (set)
  (format "Charset: %s" (ubb--set-name set)))

(defvar ubb--charsets-group
  (ubb--make-group :name "Charsets"
		   :sets%% #'ubb--all-charsets
		   :header #'ubb--charset-header))

\f
;;; Scripts

(defun ubb--all-scripts ()
  (let ((script2ranges (make-hash-table))
	(sets '()))
    (map-char-table (lambda (key script)
		      (setf (gethash script script2ranges)
			    (cons (cl-etypecase key
				    (character (cons key (1+ key)))
				    (cons (cons (car key) (1+ (cdr key)))))
				  (gethash script script2ranges))))
		    char-script-table)
    (maphash (lambda (script ranges)
	       (push (ubb--make-set (symbol-name script) ranges)
		     sets))
	     script2ranges)
    (cl-sort sets #'string< :key #'ubb--set-name)))

(defvar ubb--scripts-group
  (ubb--make-group :name "Scripts"
		   :sets%% #'ubb--all-scripts
		   :header (lambda (set)
			     (format "Script: %s" (ubb--set-name set)))))

\f
;;; Unicode categories

(defun ubb--all-unicode-categories ()
  (let ((cat2ranges (make-hash-table))
	(sets '()))
    (map-char-table (lambda (key cat)
		      (setf (gethash cat cat2ranges)
			    (cons (cl-etypecase key
				    (character (cons key (1+ key)))
				    (cons (cons (car key) (1+ (cdr key)))))
				  (gethash cat cat2ranges))))
		    unicode-category-table)
    (maphash (lambda (cat ranges)
	       (let* ((desc (char-code-property-description
			     'general-category cat))
		      (name (format "%s (%s)" cat desc)))
		 (push (ubb--make-set name ranges)
		       sets)))
	     cat2ranges)
    (cl-sort sets #'string< :key #'ubb--set-name)))

(defvar ubb--unicode-categories-group
  (ubb--make-group :name "Unicode categories"
		   :sets%% #'ubb--all-unicode-categories
		   :header (lambda (set)
			     (format "Unicode general category: %s"
				     (ubb--set-name set)))))

\f
;;; Languages

(defun ubb--all-languages ()
  (list (ubb--make-set "English" (ubb--string-ranges "“”‘’"))
	(ubb--make-set "French" (ubb--string-ranges "\
éàèùâêîôûëïüÿçœæ\
ÉÀÈÙÂÊÎÔÛËÏÜŸÇŒÆ\
«»‹›“”‘’€"))
	(ubb--make-set "German" (ubb--string-ranges "äöüßÄÖÜ„“‚‘’»«›‹€"))
	(ubb--make-set "Italian" (ubb--string-ranges "\
àèìòùéóî\
ÀÈÌÒÙÉÓÎ\
“”‘’«»‹›€"))
	(ubb--make-set "Spanish" (ubb--string-ranges "\
ñáéíóúü\
ÑÁÉÍÓÚÜ\
“”‘’«»¿¡"))
	(ubb--make-set "Turkish" (ubb--string-ranges "\
çşğıöü\
ÇŞĞİÖÜ\
“”‘’«»‹›"))
	(ubb--make-set "APL" (append (ubb--string-ranges "\
?⌈⌊⍴∼∣⍳⋆−+×÷,⌹○⍟⌽⊖⍋⍒⍎⍕⍉!−×÷⋆○?∈⌈⌊⍴↑↓⊥⊤∣,\/⍳⌹⌽⊖⍟⍕⍉!¨<≤=≥>≠∨∧⍱⍲/⌿\⍀.∘.")
				     `((,?⌶ . ,(1+ ?⍺)))))))

(defvar ubb--languages-group
  (ubb--make-group :name "Languages"
		   :sets%% #'ubb--all-languages
		   :header (lambda (set)
			     (format "Language: %s" (ubb--set-name set)))))


\f
;;; Games (important)

(defun ubb--all-game-sets ()
  (list (ubb--make-set "Chess" (ubb--string-ranges "♚♛♜♝♞♟♙♘♗♔♖♕"))
	(ubb--make-set "Domino" `((#x1f030 . #x1f0a0)))
	(ubb--make-set "Poker" `(,@(ubb--string-ranges "♠♤♥♡♦♢♣♧")
				 (#x1f0a0 . #x1f100)))))

(defun ubb--game-header (set)
  (format "Game: %s" (ubb--set-name set)))

(defvar ubb--games-group
  (ubb--make-group :name "Games"
		   :sets%% #'ubb--all-game-sets
		   :header #'ubb--game-header))

\f
;;; Groups

(defun ubb--all-groups ()
  (list ubb--unicode-blocks-group
	ubb--charsets-group
	ubb--scripts-group
	ubb--unicode-categories-group
	ubb--languages-group
	ubb--games-group))

(defvar ubb--read-group-name-history (list "Unicode blocks"))

(defun ubb--read-group-name (prompt)
  (let ((hist 'ubb--read-group-name-history)
	(completion-ignore-case t))
    (completing-read prompt
		     (mapcar #'ubb--group-name (ubb--all-groups))
		     nil t nil hist)))

(defun ubb--find-group-by-name (name)
  (cl-find name (ubb--all-groups) :key #'ubb--group-name :test #'equal))

(defvar ubb--read-set-name-history (list))

(defun ubb--read-set-name (group prompt)
  (let ((hist 'ubb--read-set-name-history)
	(completion-ignore-case t)
	(completion-styles (cl-adjoin 'substring completion-styles)))
    (completing-read prompt
		     (mapcar #'ubb--set-name (ubb--group-sets group))
		     nil t nil hist)))

(defun ubb--find-set-by-name (group name)
  (cl-find name (ubb--group-sets group)
	   :key #'ubb--set-name :test #'equal))

\f
;;; Codepoint filter

(eval-and-compile
  (defun ubb--unicode-category-symbols ()
    (let ((result '()))
      (map-char-table (lambda (_ cat) (cl-pushnew cat result))
		      unicode-category-table)
      result)))

(defcustom ubb-categories-to-hide '(Cn Cs Co)
  "List of Unicode category symbols.
Characters belonging to these categories will not be displayed."
  :group 'ubb
  :type (eval-when-compile
	  `(set ,@(cl-loop for c in (ubb--unicode-category-symbols)
			   collect `(const ,c)))))

(defmacro ubb--define-hide-category-toggle (name category)
  `(defun ,name ()
     (interactive)
     (cond ((memq ',category ubb-categories-to-hide)
	    (setq ubb-categories-to-hide
		  (remove ',category ubb-categories-to-hide)))
	   (t
	    (push ',category ubb-categories-to-hide)))))

(ubb--define-hide-category-toggle ubb-toggle-hide-not-assigned-codepoints Cn)
(ubb--define-hide-category-toggle ubb-toggle-hide-surrogates Cs)
(ubb--define-hide-category-toggle ubb-toggle-hide-private-use-codepoints Co)

;; Return the set of hidden codepoints.
(defun ubb--hidden-rangeset ()
  (cond ((null ubb-categories-to-hide) '())
	(t
	 (let ((ranges '()))
	   (map-char-table
	    (lambda (key cat)
	      (when (memq cat ubb-categories-to-hide)
		(cl-etypecase key
		  (character (push (cons key (1+ key)) ranges))
		  (cons (push (cons (car key) (1+ (cdr key))) ranges)))))
	    unicode-category-table)
	   (ubb--ranges-to-rangeset ranges)))))

\f
;;; Display

(defface ubb-invisible
  '((t :inherit tooltip))
  "Face used for code-points that would otherwise be invisible/transparent."
  :group 'ubb)

(defvar ubb--invisible-categories '(Cf Zs Zl Zp))

(defun ubb--propertize (start end codepoint)
  (add-text-properties start end (list 'codepoint codepoint
				       'help-echo #'ubb--help-echo))
  (when (memq (get-char-code-property codepoint 'general-category)
	      ubb--invisible-categories)
    (add-text-properties start end '(face ubb-invisible))))

(defun ubb--insert-codepoint (codepoint)
  (insert " ")
  (let* ((start (point))
	 (_ (insert (cl-case codepoint
		      (?\n "^J")
		      (t codepoint)))))
    (ubb--propertize start (point) codepoint)))

;; FIXME: only for compatibility with Emacs 24
(defun ubb--default-font-width ()
  (cond ((fboundp 'default-font-width)
	 (default-font-width))
	(t
	 (frame-char-width))))

(defvar ubb--right-margin 10)
(defvar ubb--left-margin 1)

;; Insert a set of codepoints, trying to create lines of equal width.
;;
;; Inserting large sets can be slow, so this calls redisplay for every
;; line to give some visual feedback to the user.  Also, the progress
;; is shown in percent in the echo area.
(defun ubb--insert-rangeset (rangeset)
  (let* ((win (get-buffer-window))
	 (_ (cl-assert win))
	 (font-width (ubb--default-font-width))
	 (right-limit (- (window-width win t)
			 (* ubb--right-margin font-width)))
	 (line-start (point))
	 (count (ubb--rangeset-size rangeset))
	 (i 0))
    (insert-char ?\s ubb--left-margin)
    (ubb--rangeset-foreach
     rangeset
     (lambda (codepoint)
       (ubb--insert-codepoint codepoint)
       (cl-incf i)
       (let ((w (car (window-text-pixel-size win line-start (point)))))
	 (when (<= right-limit w)
	   (insert "\n")
	   (setq line-start (point))
	   (insert-char ?\s ubb--left-margin)
	   (message "%.f%%" (* 100.0 (/ (float i) count)))
	   (redisplay)))))
    (message nil)))

\f
;;; Buttons

(defun ubb--insert-button-rangeset (rangeset)
  (ubb--insert-rangeset rangeset))

(defvar ubb--button-keymap (make-sparse-keymap))

(defun ubb--insert-button (rangeset collapsed?)
  (let ((start (point)))
    (insert (propertize (if collapsed? "[+]" "[-]")
			'keymap ubb--button-keymap))
    (unless collapsed?
      (insert "\n")
      (ubb--insert-button-rangeset rangeset))
    (add-text-properties start (point)
			 (list 'rangeset rangeset 'collapsed? collapsed?))))

(defun ubb-toggle-button ()
  (interactive)
  (when (not (get-text-property (point) 'rangeset))
    (user-error "Not at a button"))
  (let* ((collapsed? (get-text-property (point) 'collapsed?))
	 (rangeset (get-text-property (point) 'rangeset))
	 (end (next-single-char-property-change (point) 'rangeset))
	 (start (previous-single-char-property-change end 'rangeset))
	 (inhibit-read-only t))
    (delete-region start end)
    (ubb--insert-button rangeset (not collapsed?))
    (goto-char start))
  (message "toggle"))

(defun ubb--split-rangeset (rangeset n)
  (let ((left '())
	(count 0)
	(right rangeset))
    (while (and (< count n) right)
      (cl-destructuring-bind (start &rest end) (car right)
	(let ((len (- end start)))
	  (cond ((<= (+ count len) n)
		 (push (pop right) left)
		 (cl-incf count len))
		(t
		 (let ((k (- n count)))
		   (push (cons start (+ start k)) left)
		   (pop right)
		   (push (cons (+ start k) end) right)
		   (cl-incf count k)))))))
    (list (nreverse left) right)))

(defvar ubb--codepoints-per-button 256)

(defun ubb--insert-rangeset/buttons (rangeset)
  (let ((n ubb--codepoints-per-button))
    (cl-destructuring-bind (first rest) (ubb--split-rangeset rangeset n)
      (cond ((null rest)
	     (insert "\n")
	     (ubb--insert-rangeset first))
	    (t
	     (ubb--insert-button first nil)
	     (insert "\n")
	     (save-excursion
	       (set-window-point (get-buffer-window) (point-min))
	       (redisplay))
	     (while rest
	       (cl-destructuring-bind (r rs) (ubb--split-rangeset rest n)
		 (ubb--insert-button r t)
		 (insert "\n")
		 (setq rest rs))))))))

\f
;;; Mode

(defun ubb--clear-codepoint-info ()
  (message nil))

;; Return a short description for codepoint.  This basically the
;; Unicode name.
(defun ubb--short-description (codepoint)
  (let* ((name (get-char-code-property codepoint 'name))
	 (old (get-char-code-property codepoint 'old-name))
	 (cat (get-char-code-property codepoint 'general-category))
	 (catdesc (char-code-property-description 'general-category cat)))
    (format "\"%c\" %s (%s: %s) U+%04X" codepoint (or name old "[no name]")
	     cat catdesc codepoint)))

(defun ubb--show-codepoint-info (codepoint)
  (message "%s" (ubb--short-description codepoint)))

(defun ubb--current-codepoint (&optional noerror)
  (let ((codepoint (get-text-property (point) 'codepoint)))
    (cond (codepoint)
	  (noerror nil)
	  (t (user-error "No code-point selected")))))

;; This is called from post-command-hook.
(defun ubb--codepoint-sensor ()
  (unless (current-message)
    (let ((codepoint (ubb--current-codepoint t)))
      (cond (codepoint (ubb--show-codepoint-info codepoint))
	    (t (ubb--clear-codepoint-info))))))

;; This called if the mouse pointer hovers around.
(defun ubb--help-echo (_ __ pos)
  (let ((codepoint (get-text-property pos 'codepoint)))
    (if codepoint (ubb--short-description codepoint))))

(defvar ubb--buffer-set)   ; The currently displayed ubb--set
(defvar ubb--buffer-group) ; The group to which ubb--buffer-set belongs
(defvar ubb--buffer-hidden-codepoints-count)

(define-derived-mode ubb-mode fundamental-mode "ubb"
  "\\<ubb-mode-map>\
Mode for viewing the characters in Unicode blocks and other charsets.

The browser displays the current \"set\" of the current \"group\".  A
set is a collection of code-points, e.g. the Unicode Block \"Box
Drawing\" is such a set.  Groups are collections of sets, e.g. all
Unicode Blocks form a group.

Example: To view the \"Box Drawing\" block, first select the
\"Unicode blocks\" group with:
  \\[ubb-browse-group-by-name] \"Unicode blocks\" RET
then select the \"Box Drawing\" set with:
  \\[ubb-select-set-by-name] \"Box Drawing\" RET

Main commands:
\\[ubb-browse-group-by-name] - Select a group and display its first set.
\\[ubb-select-set-by-name] - Select a set from the current group.
\\[ubb-next-set] - Select the next set in the current group.
\\[ubb-prev-set] - Select the previous set in the current group.
\\[ubb-insert] - Insert the current code-point into a buffer.
\\[ubb-copy-to-kill-ring] - Copy current code-point to the kill-ring.
"
  (setq-local bidi-display-reordering nil)
  (setq-local truncate-lines t)
  (read-only-mode 1)
  (add-hook 'post-command-hook 'ubb--codepoint-sensor nil t))

(defun ubb--buffer-name () "*ubb*")

(defun ubb--get-buffer ()
  (or (get-buffer (ubb--buffer-name))
      (with-current-buffer (get-buffer-create (ubb--buffer-name))
	(ubb-mode)
	(current-buffer))))

(defcustom ubb-incremental-display 'buttons
  "Variable to customize the display algorithm.
This is primarily interesting for large sets which can be slow to
display.

If the value is 'buttons, only the first 256 characters are
visible initially.  The user must press the [+] \"buttons\" to
see the others.

If the value is 'nil then all characters are displayed as quickly
as possible.  This can be slow."
  :group 'ubb
  :type '(radio (const buttons) (const nil)))

(defun ubb--display-rangeset (rangeset)
  (let* ((inhibit-read-only t))
    (erase-buffer)
    (let* ((win (display-buffer (current-buffer)))
	   (_ (select-window win))
	   (pos (point)))
      (cl-ecase ubb-incremental-display
	((nil) (insert "\n") (ubb--insert-rangeset rangeset))
	((buttons) (ubb--insert-rangeset/buttons rangeset)))
      (set-window-point win pos)
      (current-buffer))))

;; This is the main entry point.
;;
;; Insert and display the ubb--set SET belonging to ubb--group GROUP.
;; Create a fresh buffer only if needed.  Display the buffer and
;; select its window.  Finally return the buffer.
(defun ubb--browse-set (group set)
  (with-current-buffer (ubb--get-buffer)
    (setq-local ubb--buffer-group group)
    (setq-local ubb--buffer-set set)
    (let* ((rs1 (ubb--set-rangeset set))
	   (rs (ubb--rangeset-difference rs1 (ubb--hidden-rangeset)))
	   (nhidden (- (ubb--rangeset-size rs1) (ubb--rangeset-size rs))))
      (setq header-line-format
	    (concat (funcall (ubb--group-header group) set)
		    (if (> nhidden 0) (format "  [%d hidden]" nhidden))))
      (ubb--display-rangeset rs))))

\f
;;; ASCII replacements

;; For fonts that don't have proper glyphs for some codepoints it may
;; be useful to display an ASCII string instead.  E.g. "i." could be
;; used instead of ı (dotless i).
;;
;; This uses buffer-display-table to do that.  The replacement strings
;; come from `IT-unicode-translations'.

(defun ubb--codepoint-displayable-p (codepoint)
  (char-displayable-p codepoint))

(defun ubb-toggle-ascii-replacements ()
  "Toggle whether to display some glyphs with ASCII replacements."
  (interactive)
  (cond (buffer-display-table (setq buffer-display-table nil))
	(t
	 (let ((dtab (make-display-table)))
	   (cl-loop for (from to vector) in IT-unicode-translations do
		    (cl-loop for c from from to to
			     for glyph across vector do
			     (when (and glyph
					(not (ubb--codepoint-displayable-p c)))
			       (let* ((string (cl-etypecase glyph
						(integer (string glyph))
						(string glyph)))
				      (vector (vconcat string)))
				 (aset dtab c vector)))))
	   (setq buffer-display-table dtab))))
  (recenter))

\f
;;; Commands

(defun ubb-describe-codepoint-briefly ()
  "Show name and category of the current code-point."
  (interactive)
  (ubb--show-codepoint-info (ubb--current-codepoint)))

(defun ubb-describe-codepoint ()
  "Describe the current code-point."
  (interactive)
  (when (ubb--current-codepoint)
    (describe-char (point))))

(defun ubb--next-set (next)
  (let* ((set ubb--buffer-set)
	 (group ubb--buffer-group)
	 (all (ubb--group-sets group))
	 (i (cl-position set all :test #'equal))
	 (j (funcall next i)))
    (cond ((and (<= 0 j) (< j (length all)))
	   (ubb--browse-set group (elt all j)))
	  (t
	   (user-error "No more sets (in group %S)"
		       (ubb--group-name group))))))

(defun ubb-next-set ()
  "Browse the next set of the group."
  (interactive)
  (ubb--next-set #'1+))

(defun ubb-prev-set ()
  "Browse the previous set of the group."
  (interactive)
  (ubb--next-set #'1-))

(defun ubb-select-set-by-name (name)
  "Select the set to browse by name."
  (interactive (list (ubb--read-set-name ubb--buffer-group "Set name: ")))
  (let ((set (or (ubb--find-set-by-name ubb--buffer-group name)
		 (user-error "No set with name: %S" name))))
    (ubb--browse-set ubb--buffer-group set)))

(defun ubb--search-property (prop &optional backward)
  "Search the next text range where PROP is non-nil.
Return the value of PROP.
If BACKWARD is non-nil, search backward."
  (let ((next (cond (backward #'previous-single-char-property-change)
		    (t #'next-single-char-property-change)))
        (start (point))
        (value nil))
    (while (progn
             (goto-char (funcall next (point) prop))
             (not (or (setq value (get-char-property (point) prop))
                      (eobp)
                      (bobp)))))
    (cond (value)
          (t (goto-char start) nil))))

(defun ubb-forward-codepoint ()
  "Move cursor to the next code-point."
  (interactive)
  (or (ubb--search-property 'codepoint nil)
      (user-error "No more code-points")))

(defun ubb-backward-codepoint ()
  "Move cursor to the previous code-point."
  (interactive)
  (or (ubb--search-property 'codepoint t)
      (user-error "No more code-points")))

(defun ubb-forward-button ()
  "Move cursor to the next button."
  (interactive)
  (or (ubb--search-property 'rangeset nil)
      (user-error "No more buttons")))

(defun ubb-backward-button ()
  "Move cursor to the previous button."
  (interactive)
  (or (ubb--search-property 'rangeset t)
      (user-error "No more buttons")))

(defun ubb--search-codepoint (codepoint)
  (or (let ((pos (text-property-any (point) (point-max) 'codepoint codepoint)))
	(cond (pos (goto-char pos) t)))
      (let ((start (point))
	    (found? nil))
	(while (and (not found?) (ubb--search-property 'rangeset))
	  (when (ubb--rangeset-member (ubb--search-property 'rangeset)
				      codepoint)
	    (setq found? t)))
	(cond (found? (when (get-text-property (point) 'collapsed?)
			(ubb-toggle-button)
			(ubb--search-codepoint codepoint))
		      t)
	      (t (goto-char start) nil)))
      (error "Not found: %c" codepoint)))

(defun ubb-browse-block (block &optional codepoint)
  "Browse the Unicode block BLOCK.
Interactively without prefix arg, prompt for the block name.
With negative prefix arg, use the character at point to find
the corresponding block.
With positive positive arg, prompt for the name or number of the code-point
\(see `read-char-by-name')."
  (interactive
   (cond ((not current-prefix-arg)
	  (let ((name (ubb--read-set-name ubb--unicode-blocks-group
					  "Block name: ")))
	    (list (or (ubb--find-set-by-name ubb--unicode-blocks-group name)
		      (user-error "No block with named: %S" name)))))
	 (t
	  (let* ((codepoint
		  (cond ((< (prefix-numeric-value current-prefix-arg) 0)
			 (char-after))
			(t
			 (read-char-by-name
			  "Code-point (Unicode name or hex): "))))
		 (block (or (ubb--find-block-by-codepoint codepoint)
			    (user-error "No block for code-point: %X"
					codepoint))))
	    (list block codepoint)))))
  (with-current-buffer (ubb--browse-set ubb--unicode-blocks-group block)
    (when codepoint
      (ubb--search-codepoint codepoint))))

(defun ubb-browse-block-by-codepoint ()
  "Very similar to `ubb-browse-block'.
The only difference is that when invoked without prefix arg,
prompt for the codepoint instead for the Unicode block."
  (interactive)
  (let ((current-prefix-arg (or current-prefix-arg '(4))))
    (call-interactively #'ubb-browse-block)))

(defun ubb--browse-group (group)
  (ubb--browse-set group (elt (ubb--group-sets group) 0)))

(defun ubb-browse-group-by-name (name)
  "Prompt for a group name and display the first set in the group."
  (interactive (list (ubb--read-group-name "Group name: ")))
  (let ((group (or (ubb--find-group-by-name name)
		   (user-error "No group with name: %S" name))))
    (ubb--browse-group group)))

(defun ubb-reset-text-scale ()
  (interactive)
  (text-scale-set 0))

(defun ubb-copy-to-kill-ring ()
  "Save the current code-point in the kill ring."
  (interactive)
  (kill-new (string (ubb--current-codepoint)))
  (message "Current kill is %S" (current-kill 0)))

(defun ubb-append-to-kill ()
  "Append the current code-point to the latest kill."
  (interactive)
  (kill-append (string (ubb--current-codepoint)) nil)
  (message "Current kill is %S" (current-kill 0)))

(defun ubb-insert (buffer)
  "Insert the current code-point into BUFFER and switch to that buffer.
Intractively, prompt for a buffer name (using `other-buffer' as default)."
  (interactive (list (progn
		       (ubb--current-codepoint)
		       (read-buffer "Insert into buffer: "
				    (list (other-buffer (current-buffer) t))
				    t))))
  (let ((codepoint (ubb--current-codepoint)))
    (pop-to-buffer buffer)
    (insert-char codepoint)))

(defun ubb-redraw ()
  "Redraw the current set."
  (interactive)
  (ubb--browse-set ubb--buffer-group ubb--buffer-set))

(defun ubb-browse ()
  "Start the character browser."
  (interactive)
  (let ((buffer (get-buffer (ubb--buffer-name))))
    (cond (buffer (with-current-buffer buffer
		    (ubb--browse-set ubb--buffer-group ubb--buffer-set)))
	  (t (ubb--browse-group ubb--unicode-blocks-group)))))

(defun ubb-quit ()
  "Close the UBB window."
  (interactive)
  (quit-restore-window nil 'bury))

\f
;;; Menu

;; Return an uninterned symbol with FUN set as it function.  This is a
;; trick to put closures into menus.
(defun ubb--fake-menu-symbol (fun)
  (let ((sym (make-symbol "fake-menu-filter-symbol")))
    (fset sym fun)
    sym))

(defun ubb--build-set-menu (group)
  (mapcar (lambda (set)
	    (vector (ubb--set-name set)
		    (ubb--fake-menu-symbol
		     (lambda ()
		       (interactive)
		       (ubb--browse-set group set)))))
	  (ubb--group-sets group)))

(defun ubb--set-menu-filter (_others)
  (ubb--build-set-menu ubb--buffer-group))

(defun ubb--build-group-menu ()
  (mapcar (lambda (group)
	    (list (ubb--group-name group)
		  :filter (ubb--fake-menu-symbol
			   (lambda (_) (ubb--build-set-menu group)))))
	  (ubb--all-groups)))

(easy-menu-define nil ubb-mode-map
  "Menu for UBB mode."
  `("Character-Browser"
    ,@(ubb--build-group-menu)
    "--"
    ["Select group by name" ubb-browse-group-by-name]
    ["Select set by name" ubb-select-set-by-name]
    ["Select Unicode block by code-point" ubb-browse-block-by-codepoint]
    ("Select set in current group"
     ("Sets in current group" :filter ubb--set-menu-filter)
     ["Next set in group" ubb-next-set :key-sequence ">"]
     ["Previous set in group" ubb-prev-set :key-sequence "<"])
    "--"
    ["Insert code-point into buffer" ubb-insert
     :active (ubb--current-codepoint t)]
    ["Copy code-point to kill-ring" ubb-copy-to-kill-ring
     :active (ubb--current-codepoint t)]
    ["Describe code-point briefly" ubb-describe-codepoint-briefly
     :active (ubb--current-codepoint t)]
    ["Show code-point details" ubb-describe-codepoint
     :active (ubb--current-codepoint t)]
    "--"
    ("Movement"
     ["Move to next code-point" ubb-forward-codepoint]
     ["Move to previous code-point" ubb-backward-codepoint]
     ["Move to next button" ubb-forward-button
      :active (text-property-not-all (point-min) (point-max) 'rangeset nil)]
     ["Move to previous button" ubb-backward-button
      :active (text-property-not-all (point-min) (point-max) 'rangeset nil)])
    ("Zoom"
     :visible window-system
     ["Increase scale factor" text-scale-increase]
     ["Decrease scale factor" text-scale-decrease]
     ["Reset scale factor" ubb-reset-text-scale])
    ("Options"
     ["Hide not assigned code-points" ubb-toggle-hide-not-assigned-codepoints
      :style toggle :selected (memq 'Cn ubb-categories-to-hide)]
     ["Hide code-points within private-use areas"
      ubb-toggle-hide-private-use-codepoints
      :style toggle :selected (memq 'Co ubb-categories-to-hide)]
     ["Hide surrogates" ubb-toggle-hide-surrogates
      :style toggle :selected (memq 'Cs ubb-categories-to-hide)]
     "--"
     ["Display buttons for large sets"
      (lambda () (interactive) (setq ubb-incremental-display 'buttons))
      :style radio :selected (eq ubb-incremental-display 'buttons)]
     ["Non-incremental display"
      (lambda () (interactive) (setq ubb-incremental-display nil))
      :style radio :selected (eq ubb-incremental-display 'nil)]
     "--"
     ["ASCII replacements" ubb-toggle-ascii-replacements
      :style toggle :selected buffer-display-table])
    "--"
    ["Redraw" ubb-redraw]
    ["Quit" ubb-quit]))

\f
;;; Key bindings

(define-key ubb-mode-map (kbd "SPC") #'ubb-describe-codepoint-briefly)
(define-key ubb-mode-map (kbd "D") #'ubb-describe-codepoint)
(define-key ubb-mode-map (kbd "f") #'ubb-forward-codepoint)
(define-key ubb-mode-map (kbd "b") #'ubb-backward-codepoint)
(define-key ubb-mode-map (kbd "n") #'ubb-next-set)
(define-key ubb-mode-map (kbd "p") #'ubb-prev-set)
(define-key ubb-mode-map (kbd ">") #'ubb-next-set)
(define-key ubb-mode-map (kbd "<") #'ubb-prev-set)
(define-key ubb-mode-map (kbd "TAB") #'ubb-forward-button)
(define-key ubb-mode-map (kbd "<backtab>") #'ubb-backward-button)
(define-key ubb-mode-map (kbd "N") #'ubb-select-set-by-name)
(define-key ubb-mode-map (kbd "G") #'ubb-browse-group-by-name)
(define-key ubb-mode-map (kbd "C") #'ubb-browse-block-by-codepoint)
(define-key ubb-mode-map (kbd "+") #'text-scale-increase)
(define-key ubb-mode-map (kbd "-") #'text-scale-decrease)
(define-key ubb-mode-map (kbd "*") #'ubb-reset-text-scale)
(define-key ubb-mode-map (kbd "c") #'ubb-copy-to-kill-ring)
(define-key ubb-mode-map (kbd "a") #'ubb-append-to-kill)
(define-key ubb-mode-map (kbd "RET") #'ubb-insert)
(define-key ubb-mode-map (kbd "~") #'ubb-toggle-ascii-replacements)
(define-key ubb-mode-map (kbd "g") #'ubb-redraw)
(define-key ubb-mode-map (kbd "q") #'ubb-quit)

(define-key ubb--button-keymap (kbd "RET") #'ubb-toggle-button)
(define-key ubb--button-keymap (kbd "<mouse-1>") #'ubb-toggle-button)

(provide 'ubb)

Re: Entering Unicode characters

[Helmut Eller]

On Sun, Feb 07 2016, Richard Stallman wrote:

> It needs to be a little more self-documenting; for instance, it should
> say how to switch to another block.

I tried to improve the docstring of ubb-mode.

> Many of the characters don't display on my console,
> Have we got a way to tell, in Emacs Lisp, whether a certain
> character code can actually display?  I think there is one.

char-displayable-p seems to do this and it has, or more precisely
terminal_glyph_code has, special code for the Linux console.  So I think
it works on the Linux console and X but not with terminal emulators like
xterm.

> Could this use that to show something meaningful,
> for characters that can't really display?  For instance,
> a sequence of 2 or 3 characters that somehow stands for
> the real character?
>
> For instance, the sequence << could stand for «, and
> the sequence i-. could stand for dotless i.

I added the command ubb-toggle-ascii-replacements (bound to ~) for this.
It uses the translations from term/internal.el.  E.g. dotless i becomes
"i.".

Helmut

Re: Entering Unicode characters

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

It sounds like this will soon be very usable.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.