Operating on yanked region

Operating on yanked region

[Karl Voit]

Hi!

When I paste text from a Windows application to my Emacs (running
mainly Org-mode), I get wrong charset.

For correcting such wrong character sets, I want to write a function
that corrects the charset. Additionally, I want to add some
search&replace stuff to convert Outlook-list items to Org-mode list
items.

So this is, what I want to achieve:

  1. clipboard-yank
  2. operate following things on the yanked region:
     1. recode-region (from mule.el)
        - from: windows-1252
        - to: latin-1
     2. replace list items
     3. replace enum items
     4. paste from clipboard

How can I work with the newly yanked region after clipboard-yank?

With a construct I named "with-region" the structure should follow following
concept:

(defun my-convert-windows-region ()
  (with-region (clipboard-yank)
     (recode-region ...)
     (string-replace ...)
     (string-replace ...)
  )
)

How can I accomplish this with Elisp in reality?

-- 
All in all, one of the most disturbing things today is the definitive
fact that the NSA, GCHQ, and many more government organizations are
massively terrorizing the freedom of us and the next generations.
                                                  http://Karl-Voit.at

Re: Operating on yanked region

[Yuri Khan]

On Wed, May 18, 2016 at 5:07 PM, Karl Voit <devnull@karl-voit.at> wrote:

> When I paste text from a Windows application to my Emacs (running
> mainly Org-mode), I get wrong charset.
>
> For correcting such wrong character sets, I want to write a function
> that corrects the charset.

I cannot help with your needs, but you should not have to jump through
hoops just to get text encoding right, as long as the originating
application puts Unicode text on clipboard. (And I’m pretty convinced
Outlook does.)

If Emacs on Windows takes encoded text from clipboard when Unicode is
available, it’s a bug in Emacs.

Emacs 24.5.1 has wrong charset on yanked text with Windows (was: Operating on yanked region)

[Karl Voit]

* Yuri Khan <yuri.v.khan@gmail.com> wrote:
> On Wed, May 18, 2016 at 5:07 PM, Karl Voit <devnull@karl-voit.at> wrote:
>
>> When I paste text from a Windows application to my Emacs (running
>> mainly Org-mode), I get wrong charset.
>>
>> For correcting such wrong character sets, I want to write a function
>> that corrects the charset.
>
> I cannot help with your needs, 

That's OK.

Besides the good points regarding charset below, I still need to
search&replace some things on the yanked region.

> but you should not have to jump through
> hoops just to get text encoding right, as long as the originating
> application puts Unicode text on clipboard. (And I’m pretty convinced
> Outlook does.)

How can I test this?

> If Emacs on Windows takes encoded text from clipboard when Unicode is
> available, it’s a bug in Emacs.

I am using a pre-compiled GNU Emacs 24.5.1 (i686-pc-mingw32) of
2015-04-11 on LEG570.

Yes, I'd prefer when Emacs yanks the text in the correct charset
which it doesn't at my side.

Can somebody confirm this wrong behaviour?

How can I help?

-- 
All in all, one of the most disturbing things today is the definitive
fact that the NSA, GCHQ, and many more government organizations are
massively terrorizing the freedom of us and the next generations.
                                                  http://Karl-Voit.at

Re: Emacs 24.5.1 has wrong charset on yanked text with Windows (was: Operating on yanked region)

[tomas]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Wed, May 18, 2016 at 02:13:52PM +0200, Karl Voit wrote:
> * Yuri Khan <yuri.v.khan@gmail.com> wrote:
> > On Wed, May 18, 2016 at 5:07 PM, Karl Voit <devnull@karl-voit.at> wrote:
> >
> >> When I paste text from a Windows application to my Emacs (running
> >> mainly Org-mode), I get wrong charset.
> >>
> >> For correcting such wrong character sets, I want to write a function
> >> that corrects the charset.
> >
> > I cannot help with your needs, 
> 
> That's OK.
> 
> Besides the good points regarding charset below, I still need to
> search&replace some things on the yanked region.
> 
> > but you should not have to jump through
> > hoops just to get text encoding right, as long as the originating
> > application puts Unicode text on clipboard. (And I’m pretty convinced
> > Outlook does.)
> 
> How can I test this?

I know very little about Windows these days. But the documentation
for the variables "selection-coding-system" and "x-select-request-type"
might contain relevant information. You can access this doc with

  C-h v <variable name>

Hope this gets you a step further

- -- t
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlc8YJUACgkQBcgs9XrR2kb3cwCeIzxVtUd+OcO/hID1vJRayBbN
ukkAn1Upbc1Rap6NCO22+KiyrbrenpHv
=EEwS
-----END PGP SIGNATURE-----

Re: Emacs 24.5.1 has wrong charset on yanked text with Windows (was: Operating on yanked region)

[Yuri Khan]

On Wed, May 18, 2016 at 6:13 PM, Karl Voit <devnull@karl-voit.at> wrote:

>> but you should not have to jump through
>> hoops just to get text encoding right, as long as the originating
>> application puts Unicode text on clipboard. (And I’m pretty convinced
>> Outlook does.)
>
> How can I test this?

Something like this (Note: Little Programs do little to no error checking):

#include <stdio.h>
#include <Windows.h>

int main()
{
    UINT format = 0;
    OpenClipboard(0);
    do
    {
        format = EnumClipboardFormats(format);
        printf("%d\n", format);
    } while (format);
    CloseClipboard();
}

Unicode text is format 13. ANSI text is format 1. OEM text is format
7. Formats are enumerated in decreasing order of preference, i.e. most
faithful representation first.

Re: Emacs 24.5.1 has wrong charset on yanked text with Windows (was: Operating on yanked region)

[Karl Voit]

* <tomas@tuxteam.de> <tomas@tuxteam.de> wrote:
>
> On Wed, May 18, 2016 at 02:13:52PM +0200, Karl Voit wrote:
>> * Yuri Khan <yuri.v.khan@gmail.com> wrote:
>> > On Wed, May 18, 2016 at 5:07 PM, Karl Voit <devnull@karl-voit.at> wrote:
>> >
>> >> When I paste text from a Windows application to my Emacs (running
>> >> mainly Org-mode), I get wrong charset.
>> >>
>> >> For correcting such wrong character sets, I want to write a function
>> >> that corrects the charset.
>> >
>> > I cannot help with your needs, 
>> 
>> That's OK.
>> 
>> Besides the good points regarding charset below, I still need to
>> search&replace some things on the yanked region.
>> 
>> > but you should not have to jump through
>> > hoops just to get text encoding right, as long as the originating
>> > application puts Unicode text on clipboard. (And I’m pretty convinced
>> > Outlook does.)
>> 
>> How can I test this?
>
> I know very little about Windows these days. But the documentation
> for the variables "selection-coding-system" and 

,----
| selection-coding-system is a variable defined in `select.el'.
| Its value is iso-latin-1-dos
| Original value was nil
`----

I once set this to 'utf-8 in my init.el. The Emacs help further
suggests setting it to 'utf-16le-dos instead when running on
Windows. With switching to this setting, the clipboard gets yanked
properly! :-)

> "x-select-request-type" might contain relevant information. 

(UTF8_STRING COMPOUND_TEXT TEXT STRING)
... seems legit.

Great, that solves the charset issue.

Thanks very much! :-)


Can you still show me how I yank and operate (string-replace) only
on the yanked text?

-- 
All in all, one of the most disturbing things today is the definitive
fact that the NSA, GCHQ, and many more government organizations are
massively terrorizing the freedom of us and the next generations.
                                                  http://Karl-Voit.at

Re: Operating on yanked region

[Eli Zaretskii]

> From: Karl Voit <devnull@Karl-Voit.at>
> Date: Wed, 18 May 2016 13:07:12 +0200
> 
> When I paste text from a Windows application to my Emacs (running
> mainly Org-mode), I get wrong charset.

That shouldn't happen, as Emacs uses Unicode to get text from the
clipboard.  There should be no charsets involved here.

Does this happen in "emacs -Q"?  If so, what version of Emacs is that,
and is it a native Windows build or a Cygwin build?

If this doesn't happen in "emacs -Q", look at your customizations to
find the culprit, especially those related to encoding/decoding
selections.

> For correcting such wrong character sets, I want to write a function
> that corrects the charset.

This shouldn't be needed (and if it is, then it's impossible in
general, because Windows attempts to translate characters, and does
that lossily).  Suggest instead to find out why Unicode retrieval
doesn't work.

Re: Operating on yanked region

[Eli Zaretskii]

> From: Yuri Khan <yuri.v.khan@gmail.com>
> Date: Wed, 18 May 2016 17:30:13 +0600
> Cc: "help-gnu-emacs@gnu.org" <help-gnu-emacs@gnu.org>
> 
> If Emacs on Windows takes encoded text from clipboard when Unicode is
> available, it’s a bug in Emacs.

What you call "Unicode" is actually UTF-16, so it's also an encoding.

Re: Emacs 24.5.1 has wrong charset on yanked text with Windows (was: Operating on yanked region)

[Eli Zaretskii]

> From: Karl Voit <devnull@Karl-Voit.at>
> Date: Wed, 18 May 2016 17:53:51 +0200
> 
> ,----
> | selection-coding-system is a variable defined in `select.el'.
> | Its value is iso-latin-1-dos
> | Original value was nil
> `----
> 
> I once set this to 'utf-8 in my init.el. The Emacs help further
> suggests setting it to 'utf-16le-dos instead when running on
> Windows. With switching to this setting, the clipboard gets yanked
> properly! :-)

You shouldn't set selection-coding-system to any value on MS-Windows,
just leave it alone.  It will work regardless.  Setting it to anything
is asking for trouble, as it forces Emacs to use that encoding instead
of using UTF-16 when available in the clipboard.

> Can you still show me how I yank and operate (string-replace) only
> on the yanked text?

Turn on transient-mark-mode, I think.

Re: Emacs 24.5.1 has wrong charset on yanked text with Windows (was: Operating on yanked region)

[Karl Voit]

* Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Karl Voit <devnull@Karl-Voit.at>
>> Date: Wed, 18 May 2016 17:53:51 +0200
>> 
>> ,----
>> | selection-coding-system is a variable defined in `select.el'.
>> | Its value is iso-latin-1-dos
>> | Original value was nil
>> `----
>> 
>> I once set this to 'utf-8 in my init.el. The Emacs help further
>> suggests setting it to 'utf-16le-dos instead when running on
>> Windows. With switching to this setting, the clipboard gets yanked
>> properly! :-)
>
> You shouldn't set selection-coding-system to any value on MS-Windows,
> just leave it alone.  It will work regardless.  Setting it to anything
> is asking for trouble, as it forces Emacs to use that encoding instead
> of using UTF-16 when available in the clipboard.

Before I set it to 'utf-16le-dos as suggested by the manual for
Windows, I tested with no value and it did not yank correctly.

-- 
All in all, one of the most disturbing things today is the definitive
fact that the NSA, GCHQ, and many more government organizations are
massively terrorizing the freedom of us and the next generations.
                                                  http://Karl-Voit.at

Re: Emacs 24.5.1 has wrong charset on yanked text with Windows (was: Operating on yanked region)

[Eli Zaretskii]

> From: Karl Voit <devnull@Karl-Voit.at>
> Date: Wed, 18 May 2016 22:30:47 +0200
> 
> > You shouldn't set selection-coding-system to any value on MS-Windows,
> > just leave it alone.  It will work regardless.  Setting it to anything
> > is asking for trouble, as it forces Emacs to use that encoding instead
> > of using UTF-16 when available in the clipboard.
> 
> Before I set it to 'utf-16le-dos as suggested by the manual for
> Windows, I tested with no value and it did not yank correctly.

Don't you see it being set to that value in "emacs -Q"?  That's what I
see here.

Re: Emacs 24.5.1 has wrong charset on yanked text with Windows (was: Operating on yanked region)

[tomas]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Wed, May 18, 2016 at 05:53:51PM +0200, Karl Voit wrote:
> * <tomas@tuxteam.de> <tomas@tuxteam.de> wrote:

[...]

> I once set this to 'utf-8 in my init.el. The Emacs help further
> suggests setting it to 'utf-16le-dos instead when running on
> Windows. With switching to this setting, the clipboard gets yanked
> properly! :-)

That's interesting, since Eli says it shouldn't be necessary on
Windows (and he sure knows a hell of a lot more about Emacs than
I do, and much more so specifically about Emacs on windows).

That'd mean that your Emacs is confused somehow, but why?

[...]

> Can you still show me how I yank and operate (string-replace) only
> on the yanked text?

I don't know exactly what you want to achieve (manual operation, or
ultimately some automatism?), but you might start here:

 - after a (normal) yank, the last mark is at the start of the
   yanked text and point at its end (but mark is not active).
   So if you activate it, e.g. by

     M-x eval-expression RET (activate-mark) RET

   you get the just yanked stuff "selected". You'll have to
   wrap some of that into commands to make it practical, though.
   Season to taste.

 - there isn't, AFAIK, a hook hanging off the yank event itself
   (a pity, IMHO), but if you somehow manage to attach the text
   property named 'yank-handler (having as value a function +
   arg provided by you), then this function gets the chance to
   do its thing just after yanking.

   Search for "yank-handler" in the Emacs Lisp manual. I'm a
   bit pressed now, but if you nudge me I'd be willing to whip
   up an example.

regards
- -- t
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlc9e4sACgkQBcgs9XrR2kYZOwCfQpAsCcNfDAh7YXGXMTAJIJMo
0c4AnRE/GyKY0fCri6uoQhQS+OcUNa/a
=6S6s
-----END PGP SIGNATURE-----

Re: Operating on yanked region

[Karl Voit]

* Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Karl Voit <devnull@Karl-Voit.at>
>> Date: Wed, 18 May 2016 13:07:12 +0200
>> 
>> When I paste text from a Windows application to my Emacs (running
>> mainly Org-mode), I get wrong charset.
>
> That shouldn't happen, as Emacs uses Unicode to get text from the
> clipboard.  There should be no charsets involved here.
>
> Does this happen in "emacs -Q"?  If so, what version of Emacs is that,
> and is it a native Windows build or a Cygwin build?

No, it did not happen in emacs -Q and I am using GNU Emacs 24.5.1
(i686-pc-mingw32) of 2015-04-11 on LEG570 (binary download).

> If this doesn't happen in "emacs -Q", look at your customizations to
> find the culprit, especially those related to encoding/decoding
> selections.

It *is* related to my customizations[1] and a snippet I got from
[2].

>> For correcting such wrong character sets, I want to write a function
>> that corrects the charset.
>
> This shouldn't be needed (and if it is, then it's impossible in
> general, because Windows attempts to translate characters, and does
> that lossily).  Suggest instead to find out why Unicode retrieval
> doesn't work.

With your help in this thread, I fixed my charset issue. However, I
was not able to search&replace within the yanked text or within the
text before being inserted in order to convert Outlook-list items to
Org-mode syntax.

[1] https://github.com/novoid/dot-emacs
[2] https://www.masteringemacs.org/article/working-coding-systems-unicode-emacs

-- 
All in all, one of the most disturbing things today is the definitive
fact that the NSA, GCHQ, and many more government organizations are
massively terrorizing the freedom of us and the next generations.
                                                  http://Karl-Voit.at

Re: Emacs 24.5.1 has wrong charset on yanked text with Windows (was: Operating on yanked region)

[Karl Voit]

* <tomas@tuxteam.de> <tomas@tuxteam.de> wrote:
>
> On Wed, May 18, 2016 at 05:53:51PM +0200, Karl Voit wrote:
>> * <tomas@tuxteam.de> <tomas@tuxteam.de> wrote:
>
> [...]
>
>> I once set this to 'utf-8 in my init.el. The Emacs help further
>> suggests setting it to 'utf-16le-dos instead when running on
>> Windows. With switching to this setting, the clipboard gets yanked
>> properly! :-)
>
> That's interesting, since Eli says it shouldn't be necessary on
> Windows (and he sure knows a hell of a lot more about Emacs than
> I do, and much more so specifically about Emacs on windows).
>
> That'd mean that your Emacs is confused somehow, but why?

I just answered to Elis posting which should also answer the
question of my (faulty) configuration.

> [...]
>
>> Can you still show me how I yank and operate (string-replace) only
>> on the yanked text?
>
> I don't know exactly what you want to achieve (manual operation, or
> ultimately some automatism?), but you might start here:

What I want to achieve (I should have started with this one in the
first place): yanked text from Outlook had wrong charset (fixed!)
and a different syntax for list items. I am trying to automate it so
that I can paste to Org-mode and get Org-mode syntax for list items.
Therefore I want to search&replace within the yanked text to look
for Outlook bullet point snippets and replace them accordingly.

>  - after a (normal) yank, the last mark is at the start of the
>    yanked text and point at its end (but mark is not active).
>    So if you activate it, e.g. by
>
>      M-x eval-expression RET (activate-mark) RET
>
>    you get the just yanked stuff "selected". You'll have to
>    wrap some of that into commands to make it practical, though.
>    Season to taste.

Wow, this is great news. Thanks!

>  - there isn't, AFAIK, a hook hanging off the yank event itself
>    (a pity, IMHO), but if you somehow manage to attach the text
>    property named 'yank-handler (having as value a function +
>    arg provided by you), then this function gets the chance to
>    do its thing just after yanking.
>
>    Search for "yank-handler" in the Emacs Lisp manual. I'm a
>    bit pressed now, but if you nudge me I'd be willing to whip
>    up an example.

Cool help!

I am not sure if I want to modify yanking in general. I was thinking
of defining my-outlook-yank or similar that does the additional
stuff.

-- 
All in all, one of the most disturbing things today is the definitive
fact that the NSA, GCHQ, and many more government organizations are
massively terrorizing the freedom of us and the next generations.
                                                  http://Karl-Voit.at

Re: Emacs 24.5.1 has wrong charset on yanked text with Windows (was: Operating on yanked region)

[tomas]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Thu, Jun 09, 2016 at 12:10:57PM +0200, Karl Voit wrote:
> * <tomas@tuxteam.de> <tomas@tuxteam.de> wrote:

[...]

> Cool help!

:-)

I've been helped so often here: glad to be of some help, then.

> I am not sure if I want to modify yanking in general. I was thinking
> of defining my-outlook-yank or similar that does the additional
> stuff.

Sounds reasonable

regards
- -- t
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAldZQm0ACgkQBcgs9XrR2kZudACggV18cVVwUWUFIYaDr/UYvUXB
o0kAn2dKIixwj2SErV8MTZYC7vFqxopF
=U6a9
-----END PGP SIGNATURE-----

Re: Operating on yanked region

[Eli Zaretskii]

> From: Karl Voit <devnull@Karl-Voit.at>
> Date: Thu, 9 Jun 2016 11:43:20 +0200
> 
> > If this doesn't happen in "emacs -Q", look at your customizations to
> > find the culprit, especially those related to encoding/decoding
> > selections.
> 
> It *is* related to my customizations[1] and a snippet I got from
> [2].

I don't recommend following the advice in [2] to change your defaults
to UTF-8, certainly not on Windows.  And even on Posix systems, that
advice might do more harm than help.