bug#15402: 24.3; Emacs and Mac OS Dock bad launch behavior

bug#15402: 24.3; Emacs and Mac OS Dock bad launch behavior

[Constantine Vetoshev]

[-- Attachment #1: Type: text/plain, Size: 1135 bytes --]

When Emacs is launched from its icon in the Mac OS Dock, all modes
derived from term (such as M-x ansi-term) exhibit strange behavior:
printing a long line which contains a Unicode character causes line
truncation and generally puts the terminal in a bad state. Resetting the
terminal fixes the problem.

The problem does NOT occur when Emacs is launched from the command
line, i.e., the terminal modes all work correctly if Emacs is invoked by
running the following at a (non-Emacs) shell:
"/Applications/Emacs.app/Contents/MacOS/Emacs"

The problem does not seem to be related to any Emacs initialization code
in .emacs or init.el. It occurs consistently when all initialization
code is commented out.

Attached is a sample file which causes the problem; just save it, launch a
shell in ansi-term, and cat the file.

In addition, when launched from the Dock, ansi-term requires something
like the following to display Unicode characters correctly:

(defadvice ansi-term (after advice-ansi-term activate)
  (set-buffer-process-coding-system 'utf-8-unix 'utf-8-unix))

This is NOT required when launching Emacs from the command line!


[-- Attachment #2: test.txt --]
[-- Type: text/plain, Size: 1530 bytes --]

Started PUT "/projects/1" for 127.0.0.1 at 2013-09-15 14:35:50 -0700
Processing by ProjectsController#update as JS
  Parameters: {"utf8"=>"✓", "authenticity_token"=>"w016h+NdEQac52o0w1o5pq+dQbQkbCh7XtW3SriENXc=", "project"=>{"name"=>"Wilson lighting upgrade and solar install", "categories_attributes"=>{"0"=>{"id"=>"275"}, "1"=>{"id"=>"276"}, "2"=>{"name"=>"", "sub_type"=>""}}, "status"=>"Completed", "leader_id"=>"90", "proposed_cost"=>"155500", "construction_year"=>"1988", "approved"=>"November 10, 2010", "start_date"=>"November 29, 2010", "complete_date"=>"March  1, 2011", "lifespan"=>"10", "resources_attributes"=>{"1"=>{"name"=>"", "units"=>"", "cost_per_unit"=>"", "annual_savings"=>""}}, "description"=>"We plan on making this green, believe it!", "methodology"=>"Estimated", "methodology_description"=>"Here are some details, does this work for you guys?", "transactions_attributes"=>{"101"=>{"id"=>"101", "date"=>"November 19, 2010", "transaction_type"=>"Project Expenditure", "description"=>"", "amount"=>"-80000"}, "102"=>{"id"=>"102", "date"=>"March 01, 2011", "transaction_type"=>"Project Rebate", "description"=>"", "amount"=>"5000"}, "365"=>{"id"=>"365", "date"=>"December 30, 2011", "transaction_type"=>"Project Loan Repayment", "description"=>"", "amount"=>"17548"}, "366"=>{"id"=>"366", "date"=>"December 31, 2012", "transaction_type"=>"Project Loan Repayment", "description"=>"", "amount"=>"21000"}, "new"=>{"id"=>"", "date"=>"", "transaction_type"=>"", "description"=>"", "amount"=>""}}}, "id"=>"1"}

[-- Attachment #3: Type: text/plain, Size: 393 bytes --]



In GNU Emacs 24.3.1 (x86_64-apple-darwin12.2.0, NS apple-appkit-1187.34)
 of 2013-03-10 on athena.local
Windowing system distributor `Apple', version 10.3.1187
Configured using:
 `configure '--with-ns''

Important settings:
  value of $LC_COLLATE: C
  value of $LC_CTYPE: en_US.UTF-8
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix
  default enable-multibyte-characters: t

bug#15402: 24.3; Emacs and Mac OS Dock bad launch behavior

[Jan Djärv]

Hello.

17 sep 2013 kl. 10:25 skrev Constantine Vetoshev <vetoshev@gmail.com>:

> When Emacs is launched from its icon in the Mac OS Dock, all modes
> derived from term (such as M-x ansi-term) exhibit strange behavior:
> printing a long line which contains a Unicode character causes line
> truncation and generally puts the terminal in a bad state. Resetting the
> terminal fixes the problem.

By unicode character, I assume you mean non-ASCII?  All ASCII characters are Unicode characters.

> 
> The problem does NOT occur when Emacs is launched from the command
> line, i.e., the terminal modes all work correctly if Emacs is invoked by
> running the following at a (non-Emacs) shell:
> "/Applications/Emacs.app/Contents/MacOS/Emacs"
> 
> The problem does not seem to be related to any Emacs initialization code
> in .emacs or init.el. It occurs consistently when all initialization
> code is commented out.
> 
> Attached is a sample file which causes the problem; just save it, launch a
> shell in ansi-term, and cat the file.
> 
> In addition, when launched from the Dock, ansi-term requires something
> like the following to display Unicode characters correctly:
> 
> (defadvice ansi-term (after advice-ansi-term activate)
>  (set-buffer-process-coding-system 'utf-8-unix 'utf-8-unix))
> 
> This is NOT required when launching Emacs from the command line!

There is a problem here.  You don't need to cat your test file to see the problem, just input a non-ASCII character from the keyboard, for example å.
The dock-started Emacs will not show å, it will show \345.

	Jan D.

bug#15402: 24.3; Emacs and Mac OS Dock bad launch behavior

[Constantine Vetoshev]

Hi Jan.

On Sep 17, 2013, at 11:04, Jan Djärv <jan.h.d@swipnet.se> wrote:
> Hello.
> 
> 17 sep 2013 kl. 10:25 skrev Constantine Vetoshev <vetoshev@gmail.com>:
> When Emacs is launched from its icon in the Mac OS Dock, all modes
>> derived from term (such as M-x ansi-term) exhibit strange behavior:
>> printing a long line which contains a Unicode character causes line
>> truncation and generally puts the terminal in a bad state. Resetting the
>> terminal fixes the problem.
> 
> By unicode character, I assume you mean non-ASCII?  All ASCII characters are Unicode characters.

Yes, I mean non-ASCII. I have specifically experienced the problem with the check-mark character, ✓, and the em-dash, —.

> There is a problem here.  You don't need to cat your test file to see the problem, just input a non-ASCII character from the keyboard, for example å.
> The dock-started Emacs will not show å, it will show \345.

You are right, but I discovered the defadvice strangeness while investigating the problem I found with the test file. They are not quite the same, and I only guessed that the two are related, since both go away when I run Emacs from the command line.

Let me clarify what I meant by "puts the terminal in a bad state". When ansi-term prints the contents of the test file, it will happily print the non-ASCII character (assuming set-buffer-process-coding-system was called). It will then continue printing characters until it hits the edge of the Emacs buffer window, print a few more characters (usually two, they wrap around), not finish printing the entire line, then move the cursor to the next line, and begins printing the rest of the output on that next line. It completely stops moving the cursor down to the next line. If I try typing in new shell commands, their output is all collapsed into the same line.

I was very surprised that just launching Emacs from the command line makes the problem go away. I don't know if the incorrect encoding setup for Dock-launched Emacs is related to this term emulation problem, but they both seem related to encoding setup.

Thanks,
Constantine

bug#15402: 24.3; Emacs and Mac OS Dock bad launch behavior

[Stefan Monnier]

> This is NOT required when launching Emacs from the command line!

Typically, the difference between the two is that running Emacs from the
command line runs it with environment variables set by your shell's init
scripts, whereas running it from the GUI does not have those
env-vars set.

IOW please compare M-: process-environment RET in the two cases, it will
probably contain a clue.


        Stefan

bug#15402: 24.3; Emacs and Mac OS Dock bad launch behavior

[Jan Djärv]

Hello.

18 sep 2013 kl. 19:24 skrev Stefan Monnier <monnier@iro.umontreal.ca>:

>> This is NOT required when launching Emacs from the command line!
> 
> Typically, the difference between the two is that running Emacs from the
> command line runs it with environment variables set by your shell's init
> scripts, whereas running it from the GUI does not have those
> env-vars set.
> 
> IOW please compare M-: process-environment RET in the two cases, it will
> probably contain a clue.
> 

I already checked, LANG is needed to be set properly in the environment, and it isn't when laucnhng from dock.  M-x setenv doesn't help, as it really doesn't setenv anything (should be renamed IMHO).

So see launchd.conf and launchctl (setenv) on how to add to the environment of programs launched from dock.

	Jan D.

bug#15402: 24.3; Emacs and Mac OS Dock bad launch behavior

[Constantine Vetoshev]

On Sep 19, 2013, at 08:54, Jan Djärv <jan.h.d@swipnet.se> wrote:
> I already checked, LANG is needed to be set properly in the environment, and it isn't when laucnhng from dock.  M-x setenv doesn't help, as it really doesn't setenv anything (should be renamed IMHO).
> 
> So see launchd.conf and launchctl (setenv) on how to add to the environment of programs launched from dock.

Confirmed. That's a really sharp edge, and has symptoms which do not necessarily suggest this fix. Is a reasonable workaround possible? Perhaps the Mac port can infer a reasonable default value for LANG on startup, based on whatever locale information Cocoa provides?

Alternatively, Apple documentation says that using LSEnvironment in Info.plist to set an environment variable should help, but using it to set LANG does nothing for me.

Also, the real reason I submitted this bug report is bad behavior in term. This bad behavior happens to go away with a properly set LANG, but my init.el contains all sorts of invocations like:

(set-language-environment "UTF-8")
(prefer-coding-system 'utf-8)
(defadvice multi-term (after cv/multi-term activate)
  (set-buffer-process-coding-system 'utf-8-unix 'utf-8-unix))

This fixes the basic non-ASCII character display problems (I didn't even realize Emacs cared that much about LANG), but ansi-term still chokes on long lines containing non-ASCII characters — even though it displays them correctly. Shouldn't at least one of these settings get term working correctly, even without a properly-set LANG?

bug#15402: 24.3; Emacs and Mac OS Dock bad launch behavior

[Stefan Monnier]

>>> This is NOT required when launching Emacs from the command line!
>> Typically, the difference between the two is that running Emacs from the
>> command line runs it with environment variables set by your shell's init
>> scripts, whereas running it from the GUI does not have those
>> env-vars set.
>> IOW please compare M-: process-environment RET in the two cases, it will
>> probably contain a clue.
> I already checked, LANG is needed to be set properly in the environment, and
> it isn't when laucnhng from dock.  M-x setenv doesn't help, as it really
> doesn't setenv anything (should be renamed IMHO).

IIRC, under Darwin, everything along the lines of locale-coding-system,
keyboard-codind-system, terminal-coding-system, etc... should always be
utf-8.  So we probably don't need to look at LANG to make that decision.


        Stefan

bug#15402: 24.3; Emacs and Mac OS Dock bad launch behavior

[Jan Djärv]

Hello.

19 sep 2013 kl. 21:25 skrev Constantine Vetoshev <vetoshev@gmail.com>:

> Also, the real reason I submitted this bug report is bad behavior in term. This bad behavior happens to go away with a properly set LANG, but my init.el contains all sorts of invocations like:
> 
> (set-language-environment "UTF-8")
> (prefer-coding-system 'utf-8)
> (defadvice multi-term (after cv/multi-term activate)
>  (set-buffer-process-coding-system 'utf-8-unix 'utf-8-unix))
> 
> This fixes the basic non-ASCII character display problems (I didn't even realize Emacs cared that much about LANG), but ansi-term still chokes on long lines containing non-ASCII characters — even though it displays them correctly. Shouldn't at least one of these settings get term working correctly, even without a properly-set LANG?

I agree that some of these settings should be taken into account. Don't know which though.
Maybe we should set utf-8-unix as default on OSX if LANG is not set? 
FWIW M-x shell behaves correctly for me.

	Jan D.

bug#15402: 24.3; Emacs and Mac OS Dock bad launch behavior

[Constantine Vetoshev]

On Sep 20, 2013, at 00:05, Jan Djärv <jan.h.d@swipnet.se> wrote:
> I agree that some of these settings should be taken into account. Don't know which though.
> Maybe we should set utf-8-unix as default on OSX if LANG is not set? 
> FWIW M-x shell behaves correctly for me.


M-x shell does not use term.el, and it doesn't attempt to emulate a real terminal. It stands to reason that it would behave completely differently. I can't really avoid emulating the terminal for my use case.

Did you reproduce the ansi-term line handling bug I explained, with the sample file I attached? The main problem is that it does not go away with any of the Emacs-specific utf-8-unix settings. It only goes away with a correctly-set LANG.

We might be able to suggest a default LANG when it's not available (is that what you meant?), but is there a valid non-locale-specific value for it we can use? Mine is set to "en_US.UTF-8", and I thought LANG requires a locale prefix. We obviously shouldn't just default to the en_US locale.

CV

bug#15402: 24.3; Emacs and Mac OS Dock bad launch behavior

[Jan Djärv]

Hello.

20 sep 2013 kl. 09:57 skrev Constantine Vetoshev <vetoshev@gmail.com>:

> On Sep 20, 2013, at 00:05, Jan Djärv <jan.h.d@swipnet.se> wrote:
>> I agree that some of these settings should be taken into account. Don't know which though.
>> Maybe we should set utf-8-unix as default on OSX if LANG is not set? 
>> FWIW M-x shell behaves correctly for me.
> 
> 
> M-x shell does not use term.el, and it doesn't attempt to emulate a real terminal. It stands to reason that it would behave completely differently. I can't really avoid emulating the terminal for my use case.

I see.

> 
> Did you reproduce the ansi-term line handling bug I explained, with the sample file I attached?

Yes.

> The main problem is that it does not go away with any of the Emacs-specific utf-8-unix settings. It only goes away with a correctly-set LANG.
> 
> We might be able to suggest a default LANG when it's not available (is that what you meant?), but is there a valid non-locale-specific value for it we can use? Mine is set to "en_US.UTF-8", and I thought LANG requires a locale prefix. We obviously shouldn't just default to the en_US locale.

For this case, the language part is not really used, it is the UTF-8 part that matters.

	Jan D.

bug#15402: 24.3; Emacs and Mac OS Dock bad launch behavior

[Constantine Vetoshev]

On Sep 20, 2013, at 02:21, Jan Djärv <jan.h.d@swipnet.se> wrote:
> For this case, the language part is not really used, it is the UTF-8 part that matters.

You are absolutely right. I just confirmed that setting LANG to "UTF-8" works. So I agree that Mac startup code should just check LANG and, if it is not found, set it to "UTF-8".

CV

bug#15402: 24.3; Emacs and Mac OS Dock bad launch behavior

[Jan Djärv]

Hello.

Finally got round to fixing this.  Closing.

	Jan D.

19 sep 2013 kl. 23:30 skrev Stefan Monnier <monnier@iro.umontreal.ca>:

>>>> This is NOT required when launching Emacs from the command line!
>>> Typically, the difference between the two is that running Emacs from the
>>> command line runs it with environment variables set by your shell's init
>>> scripts, whereas running it from the GUI does not have those
>>> env-vars set.
>>> IOW please compare M-: process-environment RET in the two cases, it will
>>> probably contain a clue.
>> I already checked, LANG is needed to be set properly in the environment, and
>> it isn't when laucnhng from dock.  M-x setenv doesn't help, as it really
>> doesn't setenv anything (should be renamed IMHO).
> 
> IIRC, under Darwin, everything along the lines of locale-coding-system,
> keyboard-codind-system, terminal-coding-system, etc... should always be
> utf-8.  So we probably don't need to look at LANG to make that decision.
> 
> 
>        Stefan

bug#15402: 24.3; Emacs and Mac OS Dock bad launch behavior

[Constantine Vetoshev]

On Sep 29, 2013, at 10:11, Jan Djärv <jan.h.d@swipnet.se> wrote:
> Finally got round to fixing this.  Closing.

Great! Thank you very much.