Re: [david.hansen@gmx.net: Re: comint's directory tracking doesn't understand \( or \)]

[David Hansen <david.hansen@gmx.net>]

On Sun, 04 Mar 2007 21:55:17 -0500 Richard Stallman wrote:

> How is it that the problem in directory tracking is fixed by
> changing comint-arguments?

shell-mode watches your input for commands that change the directory
(mainly the shells `cd' command but also `popd', `pushd' and maybe
others).

shell-mode then just use the first argument - (comint-arguments 1 1) -
to this command to keep track of the working dir of the shell process.

> Can you add some comments to make your code clearer?

Hope that's enough comments...  I also forgot backticks in the first
version and simplified the code a bit.

> We can't install it unless we can prove to ourselves that
> it is correct.  But I think that might be clear once we have
> good comments to help show it.

I don't think the problem is to prove if it's correct or not.

First of all we can't be 100% correct here (e.g. if backticks or
environment variables are involved).  And even with my patch something
like $((41 + 1)) or other (very) special constructs will not be parsed
in a sensible way (wouldn't be such a big problem to add it but i don't
think it would do any good).

Also it's impossible to support every obscure shell out there.

`comint-arguments' job is to work with all widely used shells for simple
commands.

If you do a grep for `comint-arguments' on the emacs sources you'll see
what I mean.  e.g. telnet.el uses it to extract the hostname.

I think the reason why this patch isn't that popular is that there are
just so many comint based modes out there that it's impossible to say if
it breaks something or not.

I think the risk that it breaks something that ships with emacs is very
low (just do the grep).  Also i don't think there is any widely used
shell out there that treats backslashes very different.

> Also, would you please send a change log?

2007-03-05  David Hansen  <david.hansen@physik.fu-berlin.de>

	* comint.el (comint-delim-arg): Function removed.
        (comint-arguments): Backslash quotes arbitrary characters.
        Moved `comint-delim-arg' functionality here.

*** comint.el	04 Mar 2007 13:04:02 +0100	1.358
--- comint.el	05 Mar 2007 06:30:23 +0100	
***************
*** 105,110 ****
--- 105,112 ----
  ;;; Code:
  
  (require 'ring)
+ (eval-when-compile (require 'cl))
+ 
  \f
  ;; Buffer Local Variables:
  ;;============================================================================
***************
*** 1344,1371 ****
  			(t nth))))
  	(comint-arguments string nth mth)))))
  
- (defun comint-delim-arg (arg)
-   "Return a list of arguments from ARG.
- Break it up at the delimiters in `comint-delimiter-argument-list'.
- Returned list is backwards."
-   (if (null comint-delimiter-argument-list)
-       (list arg)
-     (let ((args nil)
- 	  (pos 0)
- 	  (len (length arg)))
-       (while (< pos len)
- 	(let ((char (aref arg pos))
- 	      (start pos))
- 	  (if (memq char comint-delimiter-argument-list)
- 	      (while (and (< pos len) (eq (aref arg pos) char))
- 		(setq pos (1+ pos)))
- 	    (while (and (< pos len)
- 			(not (memq (aref arg pos)
- 				   comint-delimiter-argument-list)))
- 	      (setq pos (1+ pos))))
- 	  (setq args (cons (substring arg start pos) args))))
-       args)))
- 
  (defun comint-arguments (string nth mth)
    "Return from STRING the NTH to MTH arguments.
  NTH and/or MTH can be nil, which means the last argument.
--- 1346,1351 ----
***************
*** 1373,1423 ****
  We assume whitespace separates arguments, except within quotes
  and except for a space or tab that immediately follows a backslash.
  Also, a run of one or more of a single character
! in `comint-delimiter-argument-list' is a separate argument.
! Argument 0 is the command name."
!   ;; The first line handles ordinary characters and backslash-sequences
!   ;; (except with w32 msdos-like shells, where backslashes are valid).
!   ;; The second matches "-quoted strings.
!   ;; The third matches '-quoted strings.
!   ;; The fourth matches `-quoted strings.
!   ;; This seems to fit the syntax of BASH 2.0.
!   (let* ((first (if (if (fboundp 'w32-shell-dos-semantics)
! 			(w32-shell-dos-semantics))
! 		    "[^ \n\t\"'`]+\\|"
! 		  "[^ \n\t\"'`\\]+\\|\\\\[\"'`\\ \t]+\\|"))
! 	 (argpart (concat first
! 			  "\\(\"\\([^\"\\]\\|\\\\.\\)*\"\\|\
! '[^']*'\\|\
! `[^`]*`\\)"))
! 	 (args ()) (pos 0)
! 	 (count 0)
! 	 beg str quotes)
!     ;; Build a list of all the args until we have as many as we want.
!     (while (and (or (null mth) (<= count mth))
! 		(string-match argpart string pos))
!       (if (and beg (= pos (match-beginning 0)))
! 	  ;; It's contiguous, part of the same arg.
! 	  (setq pos (match-end 0)
! 		quotes (or quotes (match-beginning 1)))
! 	;; It's a new separate arg.
! 	(if beg
! 	    ;; Put the previous arg, if there was one, onto ARGS.
! 	    (setq str (substring string beg pos)
! 		  args (if quotes (cons str args)
! 			 (nconc (comint-delim-arg str) args))))
! 	(setq count (length args))
! 	(setq quotes (match-beginning 1))
! 	(setq beg (match-beginning 0))
! 	(setq pos (match-end 0))))
!     (if beg
! 	(setq str (substring string beg pos)
! 	      args (if quotes (cons str args)
! 		     (nconc (comint-delim-arg str) args))))
!     (setq count (length args))
!     (let ((n (or nth (1- count)))
! 	  (m (if mth (1- (- count mth)) 0)))
!       (mapconcat
!        (function (lambda (a) a)) (nthcdr n (nreverse (nthcdr m args))) " "))))
  \f
  ;;
  ;; Input processing stuff
--- 1353,1459 ----
  We assume whitespace separates arguments, except within quotes
  and except for a space or tab that immediately follows a backslash.
  Also, a run of one or more of a single character
! in `comint-delimiter-argument-list' (unless quoted with a backslash)
! is a separate argument. Argument 0 is the command name."
!   (let ((len (length string))
!         (esc (unless (and (fboundp 'w32-shell-dos-semantics)
!                           (w32-shell-dos-semantics))
!                ?\\))          ; backslash escape character, none on MS-DOS
!         (ifs '(?\n ?\t ?\ ))  ; whitespace word delimiters (the bash default)
!         (quo '(?\" ?\' ?\`))  ; argument quoting characters
!         (i 0)                 ; character index of the string
!         (beg 0)               ; beginning of the currently parsed argument
!         state                 ; stack of parsing states (see below for details)
!         args)                 ; list of arguments parsed so far
!     (flet ((push-arg (new-beg)
!              ;; With the index `i' at the end of an argument push it to the
!              ;; list `args' and set the beginning of the next argument `beg'
!              ;; to NEW-BEG.  Also decrement MTH to keep track of the number of
!              ;; parsed arguments.
!              (push (substring string beg i) args)
!              (and mth (decf mth))
!              (setq beg new-beg)))
!       ;; Loop over the characters of STRING and maintain a stack of "parser
!       ;; states".  Each parser state is a character that describes the state.
!       ;;
!       ;; If it is a member of the list `quo' we are within a quoted string that
!       ;; is delimited by this character.
!       ;;
!       ;; If it is a member of `comint-delimiter-argument-list' it is the value
!       ;; of the prevously scanned character.  We need to keep track of it as a
!       ;; sequence of equal elements of `comint-delimiter-argument-list' are
!       ;; considered as one single argument (like '>>' or '&&').
!       ;;
!       ;; If it is `esc' (a backslash on most systems) the current characte is
!       ;; escaped by a backslash and treated like any ordinary non special
!       ;; character.
!       ;;
!       ;; We stop looping if we reached the end of the string or after the MTH
!       ;; argument is parsed.
!       (while (and (<= i len) (or (not mth) (>= mth 0)))
!         (let ((s (car state))                      ; current state
!               (c (and (< i len) (aref string i)))) ; current character
!           (cond
!             ;; If the current character is backslash escaped update `state'.
!             ((eq ?\\ s)
!              (pop state))
!             ;; If within a sequence of `comint-delimiter-argument-list'
!             ;; characters check for the end of it (some different character).
!             ((and (member s comint-delimiter-argument-list) (not (eq s c)))
!              (push-arg i)
!              (pop state)
!              ;; We need to parse the current character again as it may change
!              ;; the state.  Undo the following `incf' call.
!              (decf i))
!             ;; Check for the beginning or end of quote delimited strings.
!             ((member c quo)
!              (if (eq c s)
!                  ;; We are within a quote delimited string and the current
!                  ;; character is the same as the one that started the string.
!                  ;; We reached the end.  Update `state'.
!                  (pop state)
!                ;; The current character only starts a new quote delimited
!                ;; string if we aren't already in such a construct (which is
!                ;; equivalent to `s' being nil).  Keeping track of nested
!                ;; constructs doesn't make any sense when splitting arguments.
!                (or s (push c state))))
!             ;; If the current character is a backslash it quotes the next
!             ;; character unless we are within a `'' or ``' delimited string.
!             ((and (eq esc c) (not (or (eq ?\' s) (eq ?\` s))))
!              (push c state))
!             ;; Check for space delimiters.
!             ((and (not s) (member c ifs))
!              (if (= beg i)
!                  ;; Some other character befor this space already delimited an
!                  ;; argument.  We just need to adjust the beginning of the next
!                  ;; argument.
!                  (incf beg)
!                ;; We found the end of an argument.
!                (push-arg (1+ i))))
!             ;; Check for special argument delimiting characters.
!             ((and (not s) (member c comint-delimiter-argument-list))
!              (push c state)
!              (when (/= beg i)
!                ;; This character ends the previous argument (there are no
!                ;; spaces before it).
!                (push-arg i)))
!             ;; The end of the string marks the end of an argument but only if
!             ;; the string doesn't end with whitespaces.
!             ((not c)
!              (unless (= beg len)
!                (push-arg len))))
!           (incf i)))
!       (if (not nth)
!           ;; if MTH is non nil only return the last argument if there are no
!           ;; non parsed arguments left
!           (or (and (or (not mth)
!                        (>= i len)
!                        (string-match "^[\t\n ]*$" (substring string i)))
!                    (car args))
!               "")
!         (setf (nthcdr (- (length args) nth) args) nil)
!         (mapconcat #'identity (nreverse args) " ")))))
! 
  \f
  ;;
  ;; Input processing stuff