uniq

uniq

[Tak Ota]

Do we have something equivalent to the next command?  I know the name
is bad as it is not same as UNIQ(1).  It works better because sorting
is not required.

-Tak

;;
;; uniq
;;
(defun uniq ()
  "Omit duplicated lines."
  (interactive)
  (save-excursion
    (goto-char (point-min))
    (while (not (= (point) (point-max)))
      (let* ((start (point))
	     (str (format "^%s"
			  (regexp-quote
			   (buffer-substring start
					     (progn (forward-line 1) (point)))))))
	(save-excursion
	  (while (re-search-forward str nil t)
	    (delete-region (match-beginning 0) (match-end 0))))))))

Re: uniq

[Stephen Berman]

On Fri, 3 Dec 2010 18:41:12 -0800 Tak Ota <Takaaki.Ota@am.sony.com> wrote:

> Do we have something equivalent to the next command?  I know the name
> is bad as it is not same as UNIQ(1).  It works better because sorting
> is not required.
>
> -Tak
>
> ;;
> ;; uniq
> ;;
> (defun uniq ()
>   "Omit duplicated lines."
>   (interactive)
>   (save-excursion
>     (goto-char (point-min))
>     (while (not (= (point) (point-max)))
>       (let* ((start (point))
> 	     (str (format "^%s"
> 			  (regexp-quote
> 			   (buffer-substring start
> 					     (progn (forward-line 1) (point)))))))
> 	(save-excursion
> 	  (while (re-search-forward str nil t)
> 	    (delete-region (match-beginning 0) (match-end 0))))))))

Would it be faster to avoid nested while-loops?

(defun uniq ()
  "Omit duplicated lines."
  (interactive)
  (let (lines)
    (save-excursion
      (goto-char (point-min))
      (while (not (eobp))
	(let* ((beg (line-beginning-position))
	       (end (line-end-position))
	       (line (buffer-substring beg end)))
	  (if (member line lines)
	      (delete-region beg (1+ end))
	    (push line lines)
	    (forward-line)))))))

Steve Berman

Re: uniq

[Stefan Monnier]

> Would it be faster to avoid nested while-loops?

Not sure if `member' is faster than `re-search-forward', but if you
replace re-search-forward with just search-forward (which requires
a bit more care since you need to manually check that matches are
anchored at bol and eol) I'm pretty sure that'll be even faster since it
will then use a more efficient search algorithm that gets faster the
longer the line of text you're looking for.


        Stefan

Re: uniq

[René Kyllingstad]

On Sat, Dec 4, 2010 at 3:09 PM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>> Would it be faster to avoid nested while-loops?
>
> Not sure if `member' is faster than `re-search-forward', but if you
> replace re-search-forward with just search-forward (which requires
> a bit more care since you need to manually check that matches are
> anchored at bol and eol) I'm pretty sure that'll be even faster since it
> will then use a more efficient search algorithm that gets faster the
> longer the line of text you're looking for.

Even better, use a hash table to keep the already seen lines. Maybe
call it delete-duplicate-lines, similar to delete-matching-lines.


-- René

Re: uniq

[Lennart Borgman]

2010/12/5 René Kyllingstad <listmailemacs@kyllingstad.com>:
> On Sat, Dec 4, 2010 at 3:09 PM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>>> Would it be faster to avoid nested while-loops?
>>
>> Not sure if `member' is faster than `re-search-forward', but if you
>> replace re-search-forward with just search-forward (which requires
>> a bit more care since you need to manually check that matches are
>> anchored at bol and eol) I'm pretty sure that'll be even faster since it
>> will then use a more efficient search algorithm that gets faster the
>> longer the line of text you're looking for.
>
> Even better, use a hash table to keep the already seen lines. Maybe
> call it delete-duplicate-lines, similar to delete-matching-lines.

Or maybe using a schwarzian transform to sort only line numbers and
then from this list just delete duplicates (moving backwards).