[PATCH] Fix `org-babel-detangle' handling of false positives

[PATCH] Fix `org-babel-detangle' handling of false positives

[Kevin Foley]

[-- Attachment #1: Type: text/plain, Size: 504 bytes --]

This patch fixes the way `org-babel-detangle' handles false positive
matches for links.  Without the patch it tries to use match data that
may not be present in a false positive.  I've also included a regression
test.

This is my first contribution to Org Mode or Emacs and my first patch
by mailing list so please let me know if I've overlooked anything.

Also note I have not assigned copyright to FSF at this time, however I
believe this change should be small enough to not require it.

Kevin Foley


[-- Attachment #2: patch --]
[-- Type: text/plain, Size: 4043 bytes --]

From 82e2d108536101c5a5ff9f8a0009051e5a308a3a Mon Sep 17 00:00:00 2001
From: "Kevin J. Foley" <kevin@kevinjfoley.me>
Date: Tue, 28 Jan 2020 17:51:29 -0500
Subject: [PATCH] Fix `org-babel-detangle' handling of false positives

* lisp/ob-tangle.el (org-babel-detangle): Handle false positive
matches of `org-link-bracket-re'

* testing/examples/babel.el: New file for babel detangle false
positive test

* testing/examples/babel.org (detangle): Add detangle/false positive
example

* testing/lisp/test-ob-tangle.el (ob-tangle/detangle-false-positive):
Add test for detangle false positive

TINYCHANGE
---
 lisp/ob-tangle.el              | 18 ++++++++++--------
 testing/examples/babel.el      |  5 +++++
 testing/examples/babel.org     | 13 +++++++++++++
 testing/lisp/test-ob-tangle.el | 11 +++++++++++
 4 files changed, 39 insertions(+), 8 deletions(-)
 create mode 100644 testing/examples/babel.el

diff --git a/lisp/ob-tangle.el b/lisp/ob-tangle.el
index 946039869..4dac0f786 100644
--- a/lisp/ob-tangle.el
+++ b/lisp/ob-tangle.el
@@ -513,14 +513,16 @@ (defun org-babel-detangle (&optional source-code-file)
     (goto-char (point-min))
     (let ((counter 0) new-body end)
       (while (re-search-forward org-link-bracket-re nil t)
-        (when (re-search-forward
-	       (concat " " (regexp-quote (match-string 2)) " ends here"))
-          (setq end (match-end 0))
-          (forward-line -1)
-          (save-excursion
-	    (when (setq new-body (org-babel-tangle-jump-to-org))
-	      (org-babel-update-block-body new-body)))
-          (setq counter (+ 1 counter)))
+        (if (and (match-string 2)
+		 (re-search-forward
+		  (concat " " (regexp-quote (match-string 2)) " ends here") nil t))
+	    (progn (setq end (match-end 0))
+		   (forward-line -1)
+		   (save-excursion
+		     (when (setq new-body (org-babel-tangle-jump-to-org))
+		       (org-babel-update-block-body new-body)))
+		   (setq counter (+ 1 counter)))
+	  (setq end (point)))
         (goto-char end))
       (prog1 counter (message "Detangled %d code blocks" counter)))))

diff --git a/testing/examples/babel.el b/testing/examples/babel.el
new file mode 100644
index 000000000..a7bb0ccf5
--- /dev/null
+++ b/testing/examples/babel.el
@@ -0,0 +1,5 @@
+(string-match-p "^#[[:digit:]]+$" "#123")
+
+;; [[id:73115FB0-6565-442B-BB95-50195A499EF4][detangle:1]]
+;; detangle changes
+;; linked content to detangle:1 ends here
diff --git a/testing/examples/babel.org b/testing/examples/babel.org
index c889d5d92..b0942800a 100644
--- a/testing/examples/babel.org
+++ b/testing/examples/babel.org
@@ -488,3 +488,16 @@ nil
 #+BEGIN_SRC emacs-lisp :output-dir xxx :file foo.bar
 nil
 #+END_SRC
+* detangle
+** false positive
+The =[[= causes a false positive which ~org-babel-detangle~ should handle properly
+#+begin_src emacs-lisp :tangle yes
+(string-match-p "^#[[:digit:]]+$" "#123")
+#+end_src
+** linked content to detangle
+:PROPERTIES:
+:ID:       73115FB0-6565-442B-BB95-50195A499EF4
+:END:
+#+begin_src emacs-lisp :tangle yes :comments link
+  ;; detangle
+#+end_src
diff --git a/testing/lisp/test-ob-tangle.el b/testing/lisp/test-ob-tangle.el
index 301f7aff7..ed75e6ca4 100644
--- a/testing/lisp/test-ob-tangle.el
+++ b/testing/lisp/test-ob-tangle.el
@@ -25,6 +25,8 @@
 \f
 ;;; Code:

+(require 'subr-x)
+
 ;; TODO
 ;; (ert-deftest ob-tangle/noweb-on-tangle ()
 ;;   "Noweb header arguments tangle correctly.
@@ -380,6 +382,15 @@ (ert-deftest ob-tangle/commented-src-blocks ()
 		    (org-split-string (buffer-string))))
 	      (delete-file file))))))

+(ert-deftest ob-tangle/detangle-false-positive ()
+  "Test handling of false positive link during detangle."
+  (org-test-in-example-file (expand-file-name "babel.el" org-test-example-dir)
+    (org-babel-detangle)
+    (org-test-at-id "73115FB0-6565-442B-BB95-50195A499EF4"
+    (org-babel-next-src-block)
+    (should (equal (string-trim (org-element-property :value (org-element-at-point)))
+		   ";; detangle changes")))))
+
 (provide 'test-ob-tangle)

 ;;; test-ob-tangle.el ends here
--
2.19.0

Re: [PATCH] Fix `org-babel-detangle' handling of false positives

[Bastien]

Hi Kevin,

this looks good.  The patch is significant enough that we need you to
sign the FSF copyright assignment papers.  Here is the form:

https://code.orgmode.org/bzg/org-mode/raw/master/request-assign-future.txt

Once you're done with this, we can push your commit.

Should you contribute more, we can give you push access.

Some comments below:

Foley <kevin@kevinjfoley.me> writes:

> This patch fixes the way `org-babel-detangle' handles false positive
> matches for links.  Without the patch it tries to use match data that
> may not be present in a false positive.  I've also included a regression
> test.
>
> This is my first contribution to Org Mode or Emacs and my first patch
> by mailing list so please let me know if I've overlooked anything.
>
> Also note I have not assigned copyright to FSF at this time, however I
> believe this change should be small enough to not require it.
>
> Kevin Foley
>
> From 82e2d108536101c5a5ff9f8a0009051e5a308a3a Mon Sep 17 00:00:00 2001
> From: "Kevin J. Foley" <kevin@kevinjfoley.me>
> Date: Tue, 28 Jan 2020 17:51:29 -0500
> Subject: [PATCH] Fix `org-babel-detangle' handling of false positives
>
> * lisp/ob-tangle.el (org-babel-detangle): Handle false positive
> matches of `org-link-bracket-re'
                                  ^
                                  There should be a "." at the end of
                                  sentences in changelog entries.

> * testing/examples/babel.el: New file for babel detangle false
> positive test
               ^
               Same here.

> * testing/examples/babel.org (detangle): Add detangle/false positive
> example
         ^
         And here.

> * testing/lisp/test-ob-tangle.el (ob-tangle/detangle-false-positive):
> Add test for detangle false positive
                                      ^ And here.
                                      
> TINYCHANGE

Well, there are more than 15 lines of changes.  Signing the FSF papers
will allow you to submit more changes later.

Thanks!

-- 
 Bastien

Re: [PATCH] Fix `org-babel-detangle' handling of false positives

[Kevin Foley]

Hi Bastien,

Thank you for the feedback!  I'm working with my employer regarding
copyright assignment and should have that sorted out shortly.  I'll
submit the proposed changes once that's taken care.

Kevin

Bastien <bzg@gnu.org> writes:

> Hi Kevin,
>
> this looks good.  The patch is significant enough that we need you to
> sign the FSF copyright assignment papers.  Here is the form:
>
> https://code.orgmode.org/bzg/org-mode/raw/master/request-assign-future.txt
>
> Once you're done with this, we can push your commit.
>
> Should you contribute more, we can give you push access.
>
> Some comments below:
>
> Foley <kevin@kevinjfoley.me> writes:
>
>> This patch fixes the way `org-babel-detangle' handles false positive
>> matches for links.  Without the patch it tries to use match data that
>> may not be present in a false positive.  I've also included a regression
>> test.
>>
>> This is my first contribution to Org Mode or Emacs and my first patch
>> by mailing list so please let me know if I've overlooked anything.
>>
>> Also note I have not assigned copyright to FSF at this time, however I
>> believe this change should be small enough to not require it.
>>
>> Kevin Foley
>>
>> From 82e2d108536101c5a5ff9f8a0009051e5a308a3a Mon Sep 17 00:00:00 2001
>> From: "Kevin J. Foley" <kevin@kevinjfoley.me>
>> Date: Tue, 28 Jan 2020 17:51:29 -0500
>> Subject: [PATCH] Fix `org-babel-detangle' handling of false positives
>>
>> * lisp/ob-tangle.el (org-babel-detangle): Handle false positive
>> matches of `org-link-bracket-re'
>                                   ^
>                                   There should be a "." at the end of
>                                   sentences in changelog entries.
>
>> * testing/examples/babel.el: New file for babel detangle false
>> positive test
>                ^
>                Same here.
>
>> * testing/examples/babel.org (detangle): Add detangle/false positive
>> example
>          ^
>          And here.
>
>> * testing/lisp/test-ob-tangle.el (ob-tangle/detangle-false-positive):
>> Add test for detangle false positive
>                                       ^ And here.
>                                       
>> TINYCHANGE
>
> Well, there are more than 15 lines of changes.  Signing the FSF papers
> will allow you to submit more changes later.
>
> Thanks!
>
> -- 
>  Bastien

Re: [PATCH] Fix `org-babel-detangle' handling of false positives

[Bastien]

Hi Kevin,

Kevin Foley <kevin@kevinjfoley.me> writes:

> Thank you for the feedback!  I'm working with my employer regarding
> copyright assignment and should have that sorted out shortly.  I'll
> submit the proposed changes once that's taken care.

great, thanks a lot for taking the time to go through this!

-- 
 Bastien

Re: [PATCH] Fix `org-babel-detangle' handling of false positives

[Kevin Foley]

Hi,

I've completed the copyright assignment so should be good to go with
this.

Haven't had a chance to see if any conflicts have come up in the
meantime but will try to take a look today.

Kevin

Re: [PATCH] Fix `org-babel-detangle' handling of false positives

[Bastien]

Hi Kevin,

Kevin Foley <kevin@kevinjfoley.me> writes:

> I've completed the copyright assignment so should be good to go with
> this.

great, thanks for letting us know!

> Haven't had a chance to see if any conflicts have come up in the
> meantime but will try to take a look today.

I tried to apply your patch but I was not able to apply it, perhaps
just a problem with extracting it from your email.

Can you send an (perhaps updated) version as an attachment?

Thanks,

-- 
 Bastien

Re: [PATCH] Fix `org-babel-detangle' handling of false positives

[Kevin Foley]

[-- Attachment #1: Type: text/plain, Size: 724 bytes --]

Hi Bastien,

Bastien <bzg@gnu.org> writes:

> I tried to apply your patch but I was not able to apply it, perhaps
> just a problem with extracting it from your email.
>
> Can you send an (perhaps updated) version as an attachment?

What issue did you have?

I was able to download and apply the original patch to the latest commit
on master (701c7bed) without any issues/conflicts using =git am --
<path/to/patch>=.  I'm new to patch based development so I may be
missing something.

I've tried recreating the patch and attaching to this message, hopefully
that resolves it.  I've also attached it as "text/x-patch" instead of
"text/plain" which I used last time, I'm not sure if that causes issues
or if one is preferred.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Fix-org-babel-detangle-handling-of-false-positives.patch --]
[-- Type: text/x-patch, Size: 4037 bytes --]

From eae83992e0c671d0162ba60f567629138ecc0074 Mon Sep 17 00:00:00 2001
From: "Kevin J. Foley" <kevin@kevinjfoley.me>
Date: Tue, 28 Jan 2020 17:51:29 -0500
Subject: [PATCH] Fix `org-babel-detangle' handling of false positives

* lisp/ob-tangle.el (org-babel-detangle): Handle false positive
matches of `org-link-bracket-re'

* testing/examples/babel.el: New file for babel detangle false
positive test

* testing/examples/babel.org (detangle): Add detangle/false positive
example

* testing/lisp/test-ob-tangle.el (ob-tangle/detangle-false-positive):
Add test for detangle false positive
---
 lisp/ob-tangle.el              | 18 ++++++++++--------
 testing/examples/babel.el      |  5 +++++
 testing/examples/babel.org     | 13 +++++++++++++
 testing/lisp/test-ob-tangle.el | 11 +++++++++++
 4 files changed, 39 insertions(+), 8 deletions(-)
 create mode 100644 testing/examples/babel.el

diff --git a/lisp/ob-tangle.el b/lisp/ob-tangle.el
index 8fd407478..4fe444532 100644
--- a/lisp/ob-tangle.el
+++ b/lisp/ob-tangle.el
@@ -516,14 +516,16 @@ (defun org-babel-detangle (&optional source-code-file)
     (goto-char (point-min))
     (let ((counter 0) new-body end)
       (while (re-search-forward org-link-bracket-re nil t)
-        (when (re-search-forward
-	       (concat " " (regexp-quote (match-string 2)) " ends here"))
-          (setq end (match-end 0))
-          (forward-line -1)
-          (save-excursion
-	    (when (setq new-body (org-babel-tangle-jump-to-org))
-	      (org-babel-update-block-body new-body)))
-          (setq counter (+ 1 counter)))
+        (if (and (match-string 2)
+		 (re-search-forward
+		  (concat " " (regexp-quote (match-string 2)) " ends here") nil t))
+	    (progn (setq end (match-end 0))
+		   (forward-line -1)
+		   (save-excursion
+		     (when (setq new-body (org-babel-tangle-jump-to-org))
+		       (org-babel-update-block-body new-body)))
+		   (setq counter (+ 1 counter)))
+	  (setq end (point)))
         (goto-char end))
       (prog1 counter (message "Detangled %d code blocks" counter)))))
 
diff --git a/testing/examples/babel.el b/testing/examples/babel.el
new file mode 100644
index 000000000..a7bb0ccf5
--- /dev/null
+++ b/testing/examples/babel.el
@@ -0,0 +1,5 @@
+(string-match-p "^#[[:digit:]]+$" "#123")
+
+;; [[id:73115FB0-6565-442B-BB95-50195A499EF4][detangle:1]]
+;; detangle changes
+;; linked content to detangle:1 ends here
diff --git a/testing/examples/babel.org b/testing/examples/babel.org
index c889d5d92..b0942800a 100644
--- a/testing/examples/babel.org
+++ b/testing/examples/babel.org
@@ -488,3 +488,16 @@ nil
 #+BEGIN_SRC emacs-lisp :output-dir xxx :file foo.bar
 nil
 #+END_SRC
+* detangle
+** false positive
+The =[[= causes a false positive which ~org-babel-detangle~ should handle properly
+#+begin_src emacs-lisp :tangle yes
+(string-match-p "^#[[:digit:]]+$" "#123")
+#+end_src
+** linked content to detangle
+:PROPERTIES:
+:ID:       73115FB0-6565-442B-BB95-50195A499EF4
+:END:
+#+begin_src emacs-lisp :tangle yes :comments link
+  ;; detangle
+#+end_src
diff --git a/testing/lisp/test-ob-tangle.el b/testing/lisp/test-ob-tangle.el
index 301f7aff7..ed75e6ca4 100644
--- a/testing/lisp/test-ob-tangle.el
+++ b/testing/lisp/test-ob-tangle.el
@@ -25,6 +25,8 @@
 \f
 ;;; Code:
 
+(require 'subr-x)
+
 ;; TODO
 ;; (ert-deftest ob-tangle/noweb-on-tangle ()
 ;;   "Noweb header arguments tangle correctly.
@@ -380,6 +382,15 @@ (ert-deftest ob-tangle/commented-src-blocks ()
 		    (org-split-string (buffer-string))))
 	      (delete-file file))))))
 
+(ert-deftest ob-tangle/detangle-false-positive ()
+  "Test handling of false positive link during detangle."
+  (org-test-in-example-file (expand-file-name "babel.el" org-test-example-dir)
+    (org-babel-detangle)
+    (org-test-at-id "73115FB0-6565-442B-BB95-50195A499EF4"
+    (org-babel-next-src-block)
+    (should (equal (string-trim (org-element-property :value (org-element-at-point)))
+		   ";; detangle changes")))))
+
 (provide 'test-ob-tangle)
 
 ;;; test-ob-tangle.el ends here
-- 
2.19.0


[-- Attachment #3: Type: text/plain, Size: 15 bytes --]


Thanks,
Kevin

Re: [PATCH] Fix `org-babel-detangle' handling of false positives

[Bastien]

Hi Kevin,

I was able to apply the patch, I just did so against master.

I also added you to the list of contributors here:
https://orgmode.org/worg/org-contribute.html

Thanks a lot!

-- 
 Bastien

Re: [PATCH] Fix `org-babel-detangle' handling of false positives

[Kyle Meyer]

Hi Kevin and Bastien,

This change (010d1e3b6) needs a follow-up patch.  Running the tests now
leaves the working tree dirty:

diff --git a/testing/examples/babel.org b/testing/examples/babel.org
index b0942800a..e95a65b85 100644
--- a/testing/examples/babel.org
+++ b/testing/examples/babel.org
@@ -499,5 +499,5 @@ ** linked content to detangle
 :ID:       73115FB0-6565-442B-BB95-50195A499EF4
 :END:
 #+begin_src emacs-lisp :tangle yes :comments link
-  ;; detangle
+  ;; detangle changes
 #+end_src

Kevin, could you look into updating the test to avoid changing the repo
state?

Thanks.

Re: [PATCH] Fix `org-babel-detangle' handling of false positives

[Kevin Foley]

Kyle Meyer <kyle@kyleam.com> writes:

> Kevin, could you look into updating the test to avoid changing the repo
> state?

Sorry about that.  Do you have any suggestions on how to avoid changing
the state or any examples of similar tests?

I tried the following to replace the file with it's original contents
but it doesn't seem to be working as I intend.  I also worry it's a
convoluted approach and there may be a simpler way.

diff --git a/testing/lisp/test-ob-tangle.el b/testing/lisp/test-ob-tangle.el
index ed75e6ca4..a91bd3446 100644
--- a/testing/lisp/test-ob-tangle.el
+++ b/testing/lisp/test-ob-tangle.el
@@ -384,12 +384,19 @@ (ert-deftest ob-tangle/commented-src-blocks ()

 (ert-deftest ob-tangle/detangle-false-positive ()
   "Test handling of false positive link during detangle."
-  (org-test-in-example-file (expand-file-name "babel.el" org-test-example-dir)
-    (org-babel-detangle)
-    (org-test-at-id "73115FB0-6565-442B-BB95-50195A499EF4"
-    (org-babel-next-src-block)
-    (should (equal (string-trim (org-element-property :value (org-element-at-point)))
-                  ";; detangle changes")))))
+  (let* ((babel-org-file (expand-file-name "babel.org" org-test-example-dir))
+        (orig-file-buffer
+         (get-buffer-create " bable-org")))
+    (with-current-buffer orig-file-buffer (insert-file-contents babel-org-file))
+    (org-test-in-example-file (expand-file-name "babel.el" org-test-example-dir)
+      (org-babel-detangle)
+      (org-test-at-id "73115FB0-6565-442B-BB95-50195A499EF4"
+       (org-babel-next-src-block)
+       (should (equal (string-trim (org-element-property :value (org-element-at-point)))
+                      ";; detangle changes"))))
+    (with-current-buffer orig-file-buffer
+      (kill-buffer (get-file-buffer babel-org-file))
+      (write-region (point-min) (point-max) babel-org-file))))

 (provide 'test-ob-tangle)


Thanks,
Kevin

Re: [PATCH] Fix `org-babel-detangle' handling of false positives

[Kyle Meyer]

Kevin Foley writes:

> Kyle Meyer <kyle@kyleam.com> writes:
>
>> Kevin, could you look into updating the test to avoid changing the repo
>> state?
>
> Sorry about that.  Do you have any suggestions on how to avoid changing
> the state or any examples of similar tests?
>
> I tried the following to replace the file with it's original contents
> but it doesn't seem to be working as I intend.  I also worry it's a
> convoluted approach and there may be a simpler way.

Thanks for taking a look.  The dirty working tree state happens when
running all the ob-tangle tests but not when running just the test you
added [*].  That makes me think the core issue is that the change from
your test stays around to be save by another test.  So I suspect you can
avoid the dirty state by just wrapping the test in unwind-protect,
making sure to kill the Org buffer before detangle-false-positive exits.

[*] Specifically, running

      make BTEST_RE='ob-tangle' test-dirty

    versus

      make BTEST_RE='detangle' test-dirty