Removing duplicated lines

Removing duplicated lines

[Dani Moncayo]

Hi,

Does (vanilla) Emacs have some command for removing duplicated lines
in a given region?

This is a command I'm missing for some time.

I've done a quick search, but didn't found anything.

If there isn't such command, I think I'd file a feature request.

TIA.

-- 
Dani Moncayo

Re: Removing duplicated lines

[Sebastien Vauban]

Hi Dani,

Dani Moncayo wrote:
> Does (vanilla) Emacs have some command for removing duplicated lines
> in a given region?
>
> This is a command I'm missing for some time.
>
> I've done a quick search, but didn't found anything.
>
> If there isn't such command, I think I'd file a feature request.

Shell command on a region: `M-| uniq RET'

Best regards,
  Seb

-- 
Sebastien Vauban

Re: Removing duplicated lines

[Dani Moncayo]

> Shell command on a region: `M-| uniq RET'

Thanks, but this is not exactly what I'd like, because:
* It depends on a shell command, which might not be available
(specially under MS-Windows).
* It shows the compacted region in the echo area (and therefore also
in the *Messages* buffer), but I'd like the command to modify the
region in-place (and show some message in the echo area like "nn
duplicated lines removed").

Thanks for replying.

-- 
Dani Moncayo

Re: Removing duplicated lines

[Yuri Khan]

On Thu, Nov 29, 2012 at 6:06 PM, Sebastien Vauban
<wxhgmqzgwmuf@spammotel.com> wrote:
>> Does (vanilla) Emacs have some command for removing duplicated lines
>> in a given region?
> Shell command on a region: `M-| uniq RET'

+1 but I’d add that it will only detect consecutive duplicate lines.
sort|uniq or sort -u will remove all duplicates but will sort the
lines as a side effect, which may or may not be desired.

Re: Removing duplicated lines

[Yuri Khan]

On Thu, Nov 29, 2012 at 6:25 PM, Dani Moncayo <dmoncayo@gmail.com> wrote:
>> Shell command on a region: `M-| uniq RET'
>
> Thanks, but this is not exactly what I'd like, because:
> * It depends on a shell command, which might not be available
> (specially under MS-Windows).

True. But if you have Emacs on Windows, you can (should?) also have
UnxUtils or GnuWin32 or Cygwin or msys.

> * It shows the compacted region in the echo area (and therefore also
> in the *Messages* buffer), but I'd like the command to modify the
> region in-place (and show some message in the echo area like "nn
> duplicated lines removed").

To modify in-place, prefix with C-u.

Re: Removing duplicated lines

[Dani Moncayo]

>> Thanks, but this is not exactly what I'd like, because:
>> * It depends on a shell command, which might not be available
>> (specially under MS-Windows).
>
> True. But if you have Emacs on Windows, you can (should?) also have
> UnxUtils or GnuWin32 or Cygwin or msys.

Maybe.  IMO, the operation is simple and common enough to deserve and
implementation inside Emacs, so that no external utility is required.
And Emacs already have similar commands like `sort-lines' which could
have been delegated also to external tools.

>> * It shows the compacted region in the echo area (and therefore also
>> in the *Messages* buffer), but I'd like the command to modify the
>> region in-place (and show some message in the echo area like "nn
>> duplicated lines removed").
>
> To modify in-place, prefix with C-u.

Indeed, thanks!

-- 
Dani Moncayo

Re: Removing duplicated lines

[Jambunathan K]

Dani Moncayo <dmoncayo@gmail.com> writes:

> Hi,
>
> Does (vanilla) Emacs have some command for removing duplicated lines
> in a given region?
>
> This is a command I'm missing for some time.
>
> I've done a quick search, but didn't found anything.
>
> If there isn't such command, I think I'd file a feature request.

Something like this ...

    (defun delete-duplicate-lines (beg end)
      (interactive "r")
      (let ((lines (split-string (buffer-substring beg end) "\n")))
        (delete-region beg end)
        (insert
         (mapconcat #'identity (delete-dups lines) "\n"))))

> TIA.

-- 

Re: Removing duplicated lines

[Dani Moncayo]

> Something like this ...
>
>     (defun delete-duplicate-lines (beg end)
>       (interactive "r")
>       (let ((lines (split-string (buffer-substring beg end) "\n")))
>         (delete-region beg end)
>         (insert
>          (mapconcat #'identity (delete-dups lines) "\n"))))

Seem to do the job, indeed.  Thanks!

It could be improved by accepting a prefix argument to control an
optional reordering of the region (as side effect):
* If the argument is > 0 --> Region sorted in ascending order.
* If the argument is < 0 --> Region sorted in descending order.
* If the argument is 0 or nil --> Region is left unsorted.

-- 
Dani Moncayo

Re: Removing duplicated lines

[naisanza]

On Thursday, November 29, 2012 6:57:45 AM UTC-5, Yuri Khan wrote:
> On Thu, Nov 29, 2012 at 6:25 PM, Dani Moncayo <> wrote:
> 
> >> Shell command on a region: `M-| uniq RET'
> 
> >
> 
> > Thanks, but this is not exactly what I'd like, because:
> 
> > * It depends on a shell command, which might not be available
> 
> > (specially under MS-Windows).
> 
> 
> 
> True. But if you have Emacs on Windows, you can (should?) also have
> 
> UnxUtils or GnuWin32 or Cygwin or msys.
> 
> 
> 
> > * It shows the compacted region in the echo area (and therefore also
> 
> > in the *Messages* buffer), but I'd like the command to modify the
> 
> > region in-place (and show some message in the echo area like "nn
> 
> > duplicated lines removed").
> 
> 
> 
> To modify in-place, prefix with C-u.

awesome.

Re: Removing duplicated lines

[naisanza]

On Thursday, November 29, 2012 6:06:54 AM UTC-5, Sebastien Vauban wrote:
> Hi Dani,
> 
> 
> 
> Dani Moncayo wrote:
> 
> > Does (vanilla) Emacs have some command for removing duplicated lines
> 
> > in a given region?
> 
> >
> 
> > This is a command I'm missing for some time.
> 
> >
> 
> > I've done a quick search, but didn't found anything.
> 
> >
> 
> > If there isn't such command, I think I'd file a feature request.
> 
> 
> 
> Shell command on a region: `M-| uniq RET'
> 
> 
> 
> Best regards,
> 
>   Seb
> 
> 
> 
> -- 
> 
> Sebastien Vauban

This was just what I was looking for