bug#23675: Feature request: Emacs 25.0.94: count-lines should offer a way to ignore invisible lines, e.g. outline mode

bug#23675: Feature request: Emacs 25.0.94: count-lines should offer a way to ignore invisible lines, e.g. outline mode

[Robert Weiner]

[-- Attachment #1: Type: text/plain, Size: 326 bytes --]

Often when particular lines are hidden, one  wants to count only visible
lines.  count-lines presently cannot do this.  XEmacs has for a long-time
had count-lines take an optional 3rd parameter flag which if non-nil,
ignores invisible lines in the line count.  This would be very helpful
whenever selective-display is in use.

[-- Attachment #2: Type: text/html, Size: 367 bytes --]

bug#23675: Feature request: Emacs 25.0.94: count-lines should offer a way to ignore invisible lines, e.g. outline mode

[Eli Zaretskii]

> From: Robert Weiner <rsw@gnu.org>
> Date: Wed, 1 Jun 2016 10:23:56 -0400
> 
> Often when particular lines are hidden, one wants to count only visible lines. count-lines presently cannot do
> this. XEmacs has for a long-time had count-lines take an optional 3rd parameter flag which if non-nil, ignores
> invisible lines in the line count. This would be very helpful whenever selective-display is in use.

Patches are welcome to add this.

Also, we have forward-visible-line, which might prove instrumental.

Thanks.

bug#23675: Feature request: Emacs 25.0.94: count-lines should offer a way to ignore invisible lines, e.g. outline mode

[Robert Weiner]

[-- Attachment #1.1: Type: text/plain, Size: 1993 bytes --]

Attached is a patch to add this feature based on 25.0.94.  Please consider
applying it to Emacs.  I have also included the full new version of the
function below.

---------------------
(defun count-lines (start end &optional ignore-invisible-lines-flag)
  "Return number of lines between START and END.
This is usually the number of newlines between them,
but can be one more if START is not equal to END
and the greater of them is not at the start of a line.

With optional IGNORE-INVISIBLE-LINES-FLAG non-nil,
lines collapsed with selective-display are excluded
from the line count."
  (save-excursion
    (save-restriction
      (narrow-to-region start end)
      (goto-char (point-min))
      (cond ((and (not ignore-invisible-lines-flag) (eq selective-display
t))
    (save-match-data
      (let ((done 0))
(while (re-search-forward "\n\\|\r[^\n]" nil t 40)
  (setq done (+ 40 done)))
(while (re-search-forward "\n\\|\r[^\n]" nil t 1)
  (setq done (+ 1 done)))
(goto-char (point-max))
(if (and (/= start end)
 (not (bolp)))
    (1+ done)
  done))))
   (ignore-invisible-lines-flag
    (- (buffer-size) (forward-line (buffer-size))
(let ((invisible-count 0)
     prop)
 (goto-char (point-min))
 (while (re-search-forward "\n\\|\r[^\n]" nil t)
   (setq prop (get-char-property (1- (point)) 'invisible))
   (if (if (eq buffer-invisibility-spec t)
   prop
 (or (memq prop buffer-invisibility-spec)
     (assq prop buffer-invisibility-spec)))
(setq invisible-count (1+ invisible-count))))
 invisible-count)))
   (t (- (buffer-size) (forward-line (buffer-size))))))))

---------------------


On Wed, Jun 1, 2016 at 10:23 AM, Robert Weiner <rsw@gnu.org> wrote:
>
> Often when particular lines are hidden, one  wants to count only visible
lines.  count-lines presently cannot do this.  XEmacs has for a long-time
had count-lines take an optional 3rd parameter flag which if non-nil,
ignores invisible lines in the line count.  This would be very helpful
whenever selective-display is in use.
>

[-- Attachment #1.2: Type: text/html, Size: 3843 bytes --]

[-- Attachment #2: count-lines-diff.txt --]
[-- Type: text/plain, Size: 2980 bytes --]

*** simple-orig.el	2016-06-06 19:24:50.000000000 -0400
--- simple.el	2016-06-06 19:24:50.000000000 -0400
***************
*** 1214,1241 ****
  	  (message "line %d (narrowed line %d)"
  		   (+ n (line-number-at-pos start) -1) n))))))
  
! (defun count-lines (start end)
    "Return number of lines between START and END.
  This is usually the number of newlines between them,
  but can be one more if START is not equal to END
! and the greater of them is not at the start of a line."
    (save-excursion
      (save-restriction
        (narrow-to-region start end)
        (goto-char (point-min))
!       (if (eq selective-display t)
! 	  (save-match-data
! 	    (let ((done 0))
!                      (while (re-search-forward "[\n\C-m]" nil t 40)
!                        (setq done (+ 40 done)))
!                      (while (re-search-forward "[\n\C-m]" nil t 1)
!                        (setq done (+ 1 done)))
!                      (goto-char (point-max))
!                      (if (and (/= start end)
! 		       (not (bolp)))
! 		  (1+ done)
! 		done)))
! 	(- (buffer-size) (forward-line (buffer-size)))))))
  
  (defun line-number-at-pos (&optional pos)
    "Return (narrowed) buffer line number at position POS.
--- 1214,1258 ----
  	  (message "line %d (narrowed line %d)"
  		   (+ n (line-number-at-pos start) -1) n))))))
  
! (defun count-lines (start end &optional ignore-invisible-lines-flag)
    "Return number of lines between START and END.
  This is usually the number of newlines between them,
  but can be one more if START is not equal to END
! and the greater of them is not at the start of a line.
! 
! With optional IGNORE-INVISIBLE-LINES-FLAG non-nil,
! lines collapsed with selective-display are excluded
! from the line count."
    (save-excursion
      (save-restriction
        (narrow-to-region start end)
        (goto-char (point-min))
!       (cond ((and (not ignore-invisible-lines-flag) (eq selective-display t))
! 	     (save-match-data
! 	       (let ((done 0))
! 		 (while (re-search-forward "\n\\|\r[^\n]" nil t 40)
! 		   (setq done (+ 40 done)))
! 		 (while (re-search-forward "\n\\|\r[^\n]" nil t 1)
! 		   (setq done (+ 1 done)))
! 		 (goto-char (point-max))
! 		 (if (and (/= start end)
! 			  (not (bolp)))
! 		     (1+ done)
! 		   done))))
! 	    (ignore-invisible-lines-flag
! 	     (- (buffer-size) (forward-line (buffer-size))
! 		(let ((invisible-count 0)
! 		      prop)
! 		  (goto-char (point-min))
! 		  (while (re-search-forward "\n\\|\r[^\n]" nil t)
! 		    (setq prop (get-char-property (1- (point)) 'invisible))
! 		    (if (if (eq buffer-invisibility-spec t)
! 			    prop
! 			  (or (memq prop buffer-invisibility-spec)
! 			      (assq prop buffer-invisibility-spec)))
! 			(setq invisible-count (1+ invisible-count))))
! 		  invisible-count)))
! 	    (t (- (buffer-size) (forward-line (buffer-size))))))))
  
  (defun line-number-at-pos (&optional pos)
    "Return (narrowed) buffer line number at position POS.

bug#23675: Feature request: Emacs 25.0.94: count-lines should offer a way to ignore invisible lines, e.g. outline mode

[Eli Zaretskii]

> From: Robert Weiner <rsw@gnu.org>
> Date: Mon, 6 Jun 2016 19:35:47 -0400
> Cc: emacs-devel <emacs-devel@gnu.org>
> 
> Attached is a patch to add this feature based on 25.0.94. Please consider applying it to Emacs. I have also
> included the full new version of the function below.

Thanks.

Any reason why you didn't use forward-visible-line, which would leave
the skipping of invisible text to that function, and make your code
simpler?

bug#23675: Feature request: Emacs 25.0.94: count-lines should offer a way to ignore invisible lines, e.g. outline mode

[Robert Weiner]

[-- Attachment #1: Type: text/plain, Size: 543 bytes --]

On Tue, Jun 7, 2016 at 11:58 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>
> Any reason why you didn't use forward-visible-line, which would leave
> the skipping of invisible text to that function, and make your code
> simpler?
>

Because unlike forward-line, forward-visible-line does not indicate how
many lines it failed to move when it cannot move the requested amount (of
course it should).  Therefore, I could not tell when it failed to move and
would have had to add other buffer location tests.  I hope that explains it
sufficiently.

Bob

[-- Attachment #2: Type: text/html, Size: 890 bytes --]

bug#23675: Feature request: Emacs 25.0.94: count-lines should offer a way to ignore invisible lines, e.g. outline mode

[Stefan Monnier]

>> Any reason why you didn't use forward-visible-line, which would leave
>> the skipping of invisible text to that function, and make your code
>> simpler?
> Because unlike forward-line, forward-visible-line does not indicate how
> many lines it failed to move when it cannot move the requested amount (of
> course it should).

Then let's fix that.


        Stefan

bug#23675: Feature request: Emacs 25.0.94: count-lines should offer a way to ignore invisible lines, e.g. outline mode

[Lars Ingebrigtsen]

Some comments on the patch:

Robert Weiner <rsw@gnu.org> writes:

> ! (defun count-lines (start end)
>     "Return number of lines between START and END.
>   This is usually the number of newlines between them,
>   but can be one more if START is not equal to END
> ! and the greater of them is not at the start of a line."
>     (save-excursion
>       (save-restriction
>         (narrow-to-region start end)
>         (goto-char (point-min))
> !       (if (eq selective-display t)
> ! 	  (save-match-data

Hm...  the current version of the function doesn't mention
selective-display at all, which it probably should, anyway...

> ! (defun count-lines (start end &optional ignore-invisible-lines-flag)

I think we tend to avoid -flag in arguments these days?

> ! With optional IGNORE-INVISIBLE-LINES-FLAG non-nil,
> ! lines collapsed with selective-display are excluded
> ! from the line count."

This is very confusing, because it only mentions selective-display when
this flag is non-nil, but not what the flag does otherwise.

> !       (cond ((and (not ignore-invisible-lines-flag) (eq selective-display t))
> ! 	     (save-match-data

[...]

> ! 		  (while (re-search-forward "\n\\|\r[^\n]" nil t)

And the selective-display bit saves match data and the
non-selective-display one doesn't.

But other than that, I think this sounds like a useful addition.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#23675: Feature request: Emacs 25.0.94: count-lines should offer a way to ignore invisible lines, e.g. outline mode

[Stefan Monnier]

>> ! (defun count-lines (start end)
>>     "Return number of lines between START and END.
>>   This is usually the number of newlines between them,
>>   but can be one more if START is not equal to END
>> ! and the greater of them is not at the start of a line."
>>     (save-excursion
>>       (save-restriction
>>         (narrow-to-region start end)
>>         (goto-char (point-min))
>> !       (if (eq selective-display t)
>> ! 	  (save-match-data
>
> Hm...  the current version of the function doesn't mention
> selective-display at all,

That's because the t value of selective-display has been described as
obsolete in the manual since 2013.


        Stefan

bug#23675: Feature request: Emacs 25.0.94: count-lines should offer a way to ignore invisible lines, e.g. outline mode

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Hm...  the current version of the function doesn't mention
>> selective-display at all,
>
> That's because the t value of selective-display has been described as
> obsolete in the manual since 2013.

Aha.  But then the new doc string (if we apply this patch) shouldn't
either, I guess...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#23675: Feature request: Emacs 25.0.94: count-lines should offer a way to ignore invisible lines, e.g. outline mode

[Lars Ingebrigtsen]

Robert Weiner <rsw@gnu.org> writes:

> Attached is a patch to add this feature based on 25.0.94.  Please
> consider applying it to Emacs.  I have also included the full new
> version of the function below.

There were some comments about implementing this based on other
functions, but that was never followed up, so I went ahead and applied
the patch to Emacs 28.  Further improvements are left as an exercise for
the reader.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#23675: 30.0.50: make count-lines optionally ignore invisible lines

[J.P.]

[-- Attachment #1: Type: text/plain, Size: 877 bytes --]

Closed in 2020 (Emacs 28).

Perhaps I'm not understanding how the `ignore-invisible-lines' parameter
for `count-lines' is supposed to work, but it doesn't seem to make the
function consider `invisible' text properties that have lists as values.
I bring this up because ERC will likely be needing a line-counting
function that's list-aware, at least in the manner shown in the attached
tests. The change to `count-lines' accompanying these tests was merely
lifted from `forward-visible-line' to make them pass, but it's quite
possibly flawed and/or incomplete. If a proper solution ever emerges to
address this, hopefully it'll come at the hands of someone better
informed than I in the ways of Emacs invisibility. In the meantime
(2023), ERC will likely be doing its own subpar rendition unless someone
takes up the challenge for 30.1 (and Compat agrees to adopt it). Thanks.


[-- Attachment #2: 0001-POC-Honor-invisible-text-prop-list-values-in-count-l.patch --]
[-- Type: text/x-patch, Size: 5105 bytes --]

From 68fb034a60f74ea3cab9acd8e35bdf52ba9d5024 Mon Sep 17 00:00:00 2001
From: "F. Jason Park" <jp@neverwas.me>
Date: Thu, 3 Aug 2023 22:24:57 -0700
Subject: [PATCH] [POC] Honor invisible text-prop list values in count-lines

* lisp/simple.el (count-lines): Use `invisible-p' when
`ignore-invisible-lines' parameter is non-nil.
* test/lisp/simple-tests.el (simple-tests-count-lines/invisible/interval,
simple-tests-count-lines/invisible/list/single,
simple-tests-count-lines/invisible/list/tail): Add tests.
(Bug#23675)
---
 lisp/simple.el            | 19 ++++--------
 test/lisp/simple-tests.el | 62 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 67 insertions(+), 14 deletions(-)

diff --git a/lisp/simple.el b/lisp/simple.el
index 6dc08ff0eb0..5c414fb364f 100644
--- a/lisp/simple.el
+++ b/lisp/simple.el
@@ -1782,20 +1782,11 @@ count-lines
 		   done))))
 	    (ignore-invisible-lines
              (goto-char (point-min))
-	     (save-match-data
-	       (- (buffer-size)
-                  (forward-line (buffer-size))
-		  (let ((invisible-count 0)
-		        prop)
-		    (goto-char (point-min))
-		    (while (re-search-forward "\n\\|\r[^\n]" nil t)
-		      (setq prop (get-char-property (1- (point)) 'invisible))
-		      (if (if (eq buffer-invisibility-spec t)
-			      prop
-			    (or (memq prop buffer-invisibility-spec)
-			        (assq prop buffer-invisibility-spec)))
-			  (setq invisible-count (1+ invisible-count))))
-		    invisible-count))))
+             (let ((count (if (eobp) 0 1)))
+               (while (and (zerop (forward-line)) (not (eobp)))
+                 (unless (invisible-p (1- (point)))
+                   (cl-incf count)))
+               count))
 	    (t
              (goto-char (point-max))
              (if (bolp)
diff --git a/test/lisp/simple-tests.el b/test/lisp/simple-tests.el
index 7dabb735522..e8e83c9a5cd 100644
--- a/test/lisp/simple-tests.el
+++ b/test/lisp/simple-tests.el
@@ -67,6 +67,68 @@ simple-test-count-lines/ignore-invisible-lines
     (insert (propertize "\nbar\nbaz\nzut" 'invisible t))
     (should (= (count-lines (point-min) (point-max) t) 2))))
 
+(ert-deftest simple-tests-count-lines/invisible/interval ()
+  (with-current-buffer (get-buffer-create "*count-lines/invisible/interval*")
+    (should (= 0 (count-lines (point-min) (point-max) t))) ; invisible
+    (insert "pre\n"
+            "before\n"
+            (propertize "0\n" 'invisible t)
+            (propertize "1\n" 'invisible t)
+            (propertize "2\n" 'invisible t)
+            (propertize "3\n" 'invisible t)
+            "after\n"
+            "post\n")
+    (should (equal buffer-invisibility-spec 't))
+    (with-restriction 5 (- (point-max) 5)
+      (goto-char (point-min))
+      (should (looking-at "before"))
+      (goto-char (point-max))
+      (should (looking-back "after\n")))
+    (should (= 6 (count-lines 5 (- (point-max) 5))))
+    (should (= 2 (count-lines 5 (- (point-max) 5) t))) ; invisible
+    (when noninteractive (kill-buffer))))
+
+(ert-deftest simple-tests-count-lines/invisible/list/single ()
+  (with-current-buffer (get-buffer-create "*count-lines/invisible/list/head*")
+    (insert "before\n"
+            (propertize "0\n" 'invisible '(0))
+            (propertize "1\n" 'invisible '(1))
+            (propertize "2\n" 'invisible '(2))
+            (propertize "3\n" 'invisible '(3))
+            "after\n")
+    (setq buffer-invisibility-spec '(0 1 (2 . t) 3 t))
+    (should (= 6 (count-lines (point-min) (point-max))))
+    (should (= 2 (count-lines (point-min) (point-max) t))) ; invisible
+    (when noninteractive (kill-buffer)))
+
+  (with-current-buffer (get-buffer-create "*count-lines/invisible/list/alt*")
+    (insert "before\n"
+            (propertize "0\n" 'invisible '(0))
+            "a\n"
+            (propertize "1\n" 'invisible '(1))
+            "b\n"
+            (propertize "2\n" 'invisible '(2))
+            "c\n"
+            (propertize "3\n" 'invisible '(3))
+            "after\n")
+    (setq buffer-invisibility-spec '(0 1 (2 . t) 3 t))
+    (should (= 9 (count-lines (point-min) (point-max))))
+    (should (= 5 (count-lines (point-min) (point-max) t))) ; invisible
+    (when noninteractive (kill-buffer))))
+
+(ert-deftest simple-tests-count-lines/invisible/list/tail ()
+  (with-current-buffer (get-buffer-create "*count-lines/invisible/list/tail*")
+    (insert "before\n"
+            (propertize "0\n" 'invisible '(x 0))
+            (propertize "1\n" 'invisible '(x 1))
+            (propertize "2\n" 'invisible '(x 2))
+            (propertize "3\n" 'invisible '(x 3))
+            "after\n")
+    (setq buffer-invisibility-spec '(0 (1 . t) 2 3 t))
+    (should (= 6 (count-lines (point-min) (point-max))))
+    (should (= 2 (count-lines (point-min) (point-max) t))) ; invisible
+    (when noninteractive (kill-buffer))))
+
 (ert-deftest simple-text-count-lines-non-ascii ()
   (with-temp-buffer
     (insert "あ\nい\nう\nえ\nお\n")
-- 
2.41.0

bug#23675: 30.0.50: make count-lines optionally ignore invisible lines

[Eli Zaretskii]

> Cc: emacs-erc@gnu.org, rsw@gnu.org
> From: "J.P." <jp@neverwas.me>
> Date: Fri, 04 Aug 2023 17:35:27 -0700
> 
> Perhaps I'm not understanding how the `ignore-invisible-lines' parameter
> for `count-lines' is supposed to work, but it doesn't seem to make the
> function consider `invisible' text properties that have lists as values.
> I bring this up because ERC will likely be needing a line-counting
> function that's list-aware, at least in the manner shown in the attached
> tests. The change to `count-lines' accompanying these tests was merely
> lifted from `forward-visible-line' to make them pass, but it's quite
> possibly flawed and/or incomplete. If a proper solution ever emerges to
> address this, hopefully it'll come at the hands of someone better
> informed than I in the ways of Emacs invisibility. In the meantime
> (2023), ERC will likely be doing its own subpar rendition unless someone
> takes up the challenge for 30.1 (and Compat agrees to adopt it). Thanks.

This is OK for master, but please don't use cl-incf in simple.el, as
there's no real need to do so there.

Stefan, any comments?

bug#23675: 30.0.50: make count-lines optionally ignore invisible lines

[J.P.]

Eli Zaretskii <eliz@gnu.org> writes:

>> Cc: emacs-erc@gnu.org, rsw@gnu.org
>> From: "J.P." <jp@neverwas.me>
>> Date: Fri, 04 Aug 2023 17:35:27 -0700
>> 
>> Perhaps I'm not understanding how the `ignore-invisible-lines' parameter
>> for `count-lines' is supposed to work, but it doesn't seem to make the
>> function consider `invisible' text properties that have lists as values.
>> I bring this up because ERC will likely be needing a line-counting
>> function that's list-aware, at least in the manner shown in the attached
>> tests. The change to `count-lines' accompanying these tests was merely
>> lifted from `forward-visible-line' to make them pass, but it's quite
>> possibly flawed and/or incomplete. If a proper solution ever emerges to
>> address this, hopefully it'll come at the hands of someone better
>> informed than I in the ways of Emacs invisibility. In the meantime
>> (2023), ERC will likely be doing its own subpar rendition unless someone
>> takes up the challenge for 30.1 (and Compat agrees to adopt it). Thanks.
>
> This is OK for master, but please don't use cl-incf in simple.el, as
> there's no real need to do so there.

Actually, it's looking like `count-screen-lines' will suit ERC's needs.
Thanks anyway for taking a peek, and apologies for the noise.