Going Unicode all out in Emacs 26.1

Going Unicode all out in Emacs 26.1

[Sivaram Neelakantan]

As the Subject says, what settings are needed to make Emacs work in
Unicode for text files and buffers? for GNU Emacs 26.1 (build 1,
x86_64-w64-mingw32) of 2018-05-30.  For e.g. gnus, latex and plain
text stuff.

Presumably this encoding won't affect any programming scripts that I
write and Emacs will handle it automagically?

sivaram
-- 

Re: Going Unicode all out in Emacs 26.1

[Stefan Monnier]

> As the Subject says, what settings are needed to make Emacs work in
> Unicode for text files and buffers?  for GNU Emacs 26.1 (build 1,
> x86_64-w64-mingw32) of 2018-05-30.

There's no setting needed, it just works.


        Stefan

RE: Going Unicode all out in Emacs 26.1

[Drew Adams]

> > As the Subject says, what settings are needed to make Emacs work in
> > Unicode for text files and buffers?  for GNU Emacs 26.1 (build 1,
> > x86_64-w64-mingw32) of 2018-05-30.
> 
> There's no setting needed, it just works.

But keep in mind that you need to use a font that supports whatever Unicode chars you want to use.  If you don't see the chars displayed as you expect/hope, try another font.  (Eli typically recommends Symbola.)

Re: Going Unicode all out in Emacs 26.1

[Van L]

> Drew Adams writes:
> 
> But keep in mind that you need to use a font that supports whatever Unicode chars you want to use.

When the Unicode char has no representation they call that `tofu`.

`noto` is short for `no more tofu`.

  https://www.google.com/get/noto/

Re: Going Unicode all out in Emacs 26.1

[Eli Zaretskii]

> From: Sivaram Neelakantan <nsivaram.net@gmail.com>
> Date: Thu, 07 Jun 2018 01:27:30 +0530
> 
> As the Subject says, what settings are needed to make Emacs work in
> Unicode for text files and buffers?

Please explain what you mean by "work in Unicode".  I don't think I
understand the issues you are having or are envisioning to have.  And
"Unicode" has a very specific interpretation in the Windows world, in
some contexts.

Re: Going Unicode all out in Emacs 26.1

[Sivaram Neelakantan]

On Thu, Jun 07 2018,Eli Zaretskii wrote:

>> From: Sivaram Neelakantan <nsivaram.net@gmail.com>
>> Date: Thu, 07 Jun 2018 01:27:30 +0530
>> 
>> As the Subject says, what settings are needed to make Emacs work in
>> Unicode for text files and buffers?
>
> Please explain what you mean by "work in Unicode".  I don't think I
> understand the issues you are having or are envisioning to have.  And
> "Unicode" has a very specific interpretation in the Windows world, in
> some contexts.
>

I have no issues so far; all I wanted to know was whether there are
any .Emacs settings that needs to be fiddled with to make Unicode work
on Windows.  I just checked the Hello file, all languages seem to
render correctly in terms of the chars that I see.   I see that the
modeline has got a U with a tooltip 'utf-8-emacs-dos' which should
allow me to work in Emacs and nonemacs Windows application like
notepad(reading the file for e.g)?

sivaram
-- 

Re: Going Unicode all out in Emacs 26.1

[Eli Zaretskii]

> From: Sivaram Neelakantan <nsivaram.net@gmail.com>
> Date: Sat, 09 Jun 2018 15:50:08 +0530
> 
> > Please explain what you mean by "work in Unicode".  I don't think I
> > understand the issues you are having or are envisioning to have.  And
> > "Unicode" has a very specific interpretation in the Windows world, in
> > some contexts.
> 
> I have no issues so far; all I wanted to know was whether there are
> any .Emacs settings that needs to be fiddled with to make Unicode work
> on Windows.  I just checked the Hello file, all languages seem to
> render correctly in terms of the chars that I see.   I see that the
> modeline has got a U with a tooltip 'utf-8-emacs-dos' which should
> allow me to work in Emacs and nonemacs Windows application like
> notepad(reading the file for e.g)?

Well, that doesn't really answer my question, sop I'd try to answer
what I think you may be asking.

 . Visiting UTF-8 encoded files should "just work".
 . If you want to _create_ UTF-8 encoded files, you will need to
   request that explicitly using "C-x RET c" before saving a file.
 . Visiting UTF-16 encoded files may require "C-x RET c" before
   "C-x C-f".  (UTF-16 is what is called "Unicode" on Windows, which
   was one reason why I asked my question.)
 . File names are supported even if they include characters not
   supported by the current system codepage.
 . Receiving sub-process output encoded in UTF-8 should work with
   programs known to output UTF-8, such as Git; in other cases you may
   need to use "C-x RET c" or customize your process-coding-system-alist.
 . Beware of passing to programs command-line arguments encoded in
   UTF-8: that doesn't work.

HTH

Re: Going Unicode all out in Emacs 26.1

[Filipp Gunbin]

On 09/06/2018 15:50 +0530, Sivaram Neelakantan wrote:

> I have no issues so far; all I wanted to know was whether there are
> any .Emacs settings that needs to be fiddled with to make Unicode work
> on Windows.  I just checked the Hello file, all languages seem to
> render correctly in terms of the chars that I see.   I see that the
> modeline has got a U with a tooltip 'utf-8-emacs-dos' which should
> allow me to work in Emacs and nonemacs Windows application like
> notepad(reading the file for e.g)?

You may want to read (info "(emacs) Language Environments") to feel
yourself more confident about this.

If you need to customize something, you can use hook
`set-language-environment-hook'.

I have this in my .emacs:

(add-hook 'set-language-environment-hook
	  (lambda ()
	    (if (equal current-language-environment "UTF-8")
		(setq default-input-method 'russian-computer))))
(set-language-environment "UTF-8")

It says "just use utf-8 lang environment", ignoring env vars LANG,
LC_ALL, LC_CTYPE.  Then, on setting the lang env, it sets a suitable
input method.

But, as noted, things should work fine without customization.

Filipp

Re: Going Unicode all out in Emacs 26.1

[Eli Zaretskii]

> From: Filipp Gunbin <fgunbin@fastmail.fm>
> Date: Sat, 09 Jun 2018 18:00:40 +0300
> Cc: help-gnu-emacs@gnu.org
> 
> I have this in my .emacs:
> 
> (add-hook 'set-language-environment-hook
> 	  (lambda ()
> 	    (if (equal current-language-environment "UTF-8")
> 		(setq default-input-method 'russian-computer))))
> (set-language-environment "UTF-8")

Setting the language environment to UTF-8 is precisely something
that's NOT recommended to Windows users, because MS-Windows doesn't
support UTF-8 at all as a codeset.  Emacs on Windows does, but the
UTF-8 language environment will, among other things, cause Emacs to
pass command-line arguments to programs encoded in UTF-8, something
that will not work.

Emacs users on Windows have no reasons to change the language
environment from the default (which is determined by the system's
setup), because Emacs on Windows already uses Unicode-compliant APIs
and encodings where possible (e.g., when copying text to/from the
system clipboard, when accessing non-ASCII file names, etc.).  On the
contrary, changing the language environment can easily get you in
trouble on Windows.  Caveat emptor!

Re: Going Unicode all out in Emacs 26.1

[Filipp Gunbin]

On 09/06/2018 18:24 +0300, Eli Zaretskii wrote:

>> From: Filipp Gunbin <fgunbin@fastmail.fm>
>> Date: Sat, 09 Jun 2018 18:00:40 +0300
>> Cc: help-gnu-emacs@gnu.org
>>
>> I have this in my .emacs:
>>
>> (add-hook 'set-language-environment-hook
>> 	  (lambda ()
>> 	    (if (equal current-language-environment "UTF-8")
>> 		(setq default-input-method 'russian-computer))))
>> (set-language-environment "UTF-8")
>
> Setting the language environment to UTF-8 is precisely something
> that's NOT recommended to Windows users, because MS-Windows doesn't
> support UTF-8 at all as a codeset.  Emacs on Windows does, but the
> UTF-8 language environment will, among other things, cause Emacs to
> pass command-line arguments to programs encoded in UTF-8, something
> that will not work.
>
> Emacs users on Windows have no reasons to change the language
> environment from the default (which is determined by the system's
> setup), because Emacs on Windows already uses Unicode-compliant APIs
> and encodings where possible (e.g., when copying text to/from the
> system clipboard, when accessing non-ASCII file names, etc.).  On the
> contrary, changing the language environment can easily get you in
> trouble on Windows.  Caveat emptor!

Ah, thanks for clarification!  I'm not a Windows user, but that's nice
to know.

But what about Cygwin?  Can we assume that it will itself handle these
Windows-related complexities?  It seems that my language environment
setting dates back to the time when I used Cygwin.

Filipp

Re: Going Unicode all out in Emacs 26.1

[Xavier Maillard]

Hey Eli,

> > (set-language-environment "UTF-8")
> 
> Setting the language environment to UTF-8 is precisely something
> that's NOT recommended to Windows users, because MS-Windows doesn't
> support UTF-8 at all as a codeset.  Emacs on Windows does, but the
> UTF-8 language environment will, among other things, cause Emacs to
> pass command-line arguments to programs encoded in UTF-8, something
> that will not work.

What about other platforms ? Is it considered harmless to set it ?

Thank you.
-- 
Xavier Maillard                      
e/j:xavier@maillard.im               w:www.maillard.im
m: 06 52 18 63 43
GPG: 9983 DCA1 1FAC 8DA7 653A  F9AA BA49 09B7 8F04 DE1B

Re: Going Unicode all out in Emacs 26.1

[Eli Zaretskii]

> Date: Mon, 11 Jun 2018 06:59:10 +0200
> From: Xavier Maillard <xavier@maillard.im>
> CC: help-gnu-emacs@gnu.org
> 
> > Setting the language environment to UTF-8 is precisely something
> > that's NOT recommended to Windows users, because MS-Windows doesn't
> > support UTF-8 at all as a codeset.  Emacs on Windows does, but the
> > UTF-8 language environment will, among other things, cause Emacs to
> > pass command-line arguments to programs encoded in UTF-8, something
> > that will not work.
> 
> What about other platforms ? Is it considered harmless to set it ?

On Posix platforms, sure.  (But make sure your filesystem has file
names encoded in UTF-8, or else all file names will appear as
mojibake.)

Re: Going Unicode all out in Emacs 26.1

[Xavier Maillard]

> > What about other platforms ? Is it considered harmless to set it ?
> 
> On Posix platforms, sure.  (But make sure your filesystem has file
> names encoded in UTF-8, or else all file names will appear as
> mojibake.)

Humm, ok. I am not willing to mess up anything on this device then.
Thank you Eli.
-- 
Xavier Maillard                      
e/j:xavier@maillard.im               w:www.maillard.im
m: 06 52 18 63 43
GPG: 9983 DCA1 1FAC 8DA7 653A  F9AA BA49 09B7 8F04 DE1B

Re: Going Unicode all out in Emacs 26.1

[Sivaram Neelakantan]

On Sat, Jun 09 2018,Eli Zaretskii wrote:

>> From: Filipp Gunbin <fgunbin@fastmail.fm>
>> Date: Sat, 09 Jun 2018 18:00:40 +0300
>> Cc: help-gnu-emacs@gnu.org
>> 
>> I have this in my .emacs:
>> 
>> (add-hook 'set-language-environment-hook
>> 	  (lambda ()
>> 	    (if (equal current-language-environment "UTF-8")
>> 		(setq default-input-method 'russian-computer))))
>> (set-language-environment "UTF-8")
>
> Setting the language environment to UTF-8 is precisely something
> that's NOT recommended to Windows users, because MS-Windows doesn't
> support UTF-8 at all as a codeset.  Emacs on Windows does, but the
> UTF-8 language environment will, among other things, cause Emacs to
> pass command-line arguments to programs encoded in UTF-8, something
> that will not work.

Against all advice, I randomly copy Emacs snippets like the above
thinking that should be "good" for Windows too.  I think I have a mile
long cruft like this in .emacs.  I don't even go near my .gnus file
these days.


> Emacs users on Windows have no reasons to change the language
> environment from the default (which is determined by the system's
> setup), because Emacs on Windows already uses Unicode-compliant APIs
> and encodings where possible (e.g., when copying text to/from the
> system clipboard, when accessing non-ASCII file names, etc.).  On the
> contrary, changing the language environment can easily get you in
> trouble on Windows.  Caveat emptor!

I didn't do anything, so that seems to have saved the day for me.  For
now.


sivaram
-- 

Re: Going Unicode all out in Emacs 26.1

[Sivaram Neelakantan]

On Sat, Jun 09 2018,Eli Zaretskii wrote:


[snipped 16 lines]

> Well, that doesn't really answer my question, sop I'd try to answer
> what I think you may be asking.
>
>  . Visiting UTF-8 encoded files should "just work".


Thanks. That's reassuring.

>  . If you want to _create_ UTF-8 encoded files, you will need to
>    request that explicitly using "C-x RET c" before saving a file.
>  . Visiting UTF-16 encoded files may require "C-x RET c" before
>    "C-x C-f".  (UTF-16 is what is called "Unicode" on Windows, which
>    was one reason why I asked my question.)

Right, this was what I wanted to know.  What to fiddle with to
create/edit files in Windows.  Thanks, this helps.


>  . File names are supported even if they include characters not
>    supported by the current system codepage.

How will they render in the buffer then?

>  . Receiving sub-process output encoded in UTF-8 should work with
>    programs known to output UTF-8, such as Git; in other cases you may
>    need to use "C-x RET c" or customize your process-coding-system-alist.
>  . Beware of passing to programs command-line arguments encoded in
>    UTF-8: that doesn't work.
>
> HTH
>


sivaram
-- 

Re: Going Unicode all out in Emacs 26.1

[Eli Zaretskii]

> From: Sivaram Neelakantan <nsivaram.net@gmail.com>
> Date: Sun, 10 Jun 2018 01:14:21 +0530
> 
> >  . File names are supported even if they include characters not
> >    supported by the current system codepage.
> 
> How will they render in the buffer then?

Emacs can display any character in its buffers, it supports the entire
Unicode range of characters.  That is correct on Windows as well.  You
just need to have fonts installed to display them (and modern versions
of Windows have them out of the box for most of the characters).