baby step toward thread patch

baby step toward thread patch

[Tom Tromey]

[-- Attachment #1: Type: text/plain, Size: 2735 bytes --]

The concurrency branch hasn't been getting much action lately, largely
because of a combination of lack of time and the difficulty of keeping
it up-to-date with trunk.  I'd like to solve the latter problem by
putting some needed infrastructure directly on trunk.  Once all the
infrastructure bits are merged, development can focus on just the
threading bits.

In general these infrastructure changes don't involve any runtime cost.
However, they may make the code uglier.

I'm posting this to get an ok on this change before checking it in.  In
the absence of comments (after a decent interval) I will just go ahead
with it.

In elisp, a let binding will always be thread-local.  For this to work
properly, variables defined in C must be thread-aware somehow.  This
patch is a step toward one implementation of that.  In particular, it
moves most such globals into a structure and redefines the "plain" name
of the variable as a macro that references this structure.  Then on the
concurrency branch, we'll change the definitions of these macros to
introduce thread locality.

E.g., consider 'Vgc_cons_percentage'.  Right now it is defined in
alloc.c:

    static Lisp_Object Vgc_cons_percentage;

After this patch it will appear in a structure in globals.h:

    struct emacs_globals
    {
    [...]
      Lisp_Object Vgc_cons_percentage;
    [...]
    };

    extern struct emacs_globals globals;

    #define Vgc_cons_percentage \
        globals.Vgc_cons_percentage


In case you're curious, for threading we'll introduce an indirection:
each thread will have its own copy of this structure, and one pointer
per field.  The pointers will be redirected to either the global or
thread-local copy of the object depending on whether it is let-bound.
Then the macros will just dereference these pointers.

If you read the concurrency branch, you'll see that this approach is
different from what is implemented there.  The concurrency branch does
not work for the DEFVAR_INT and DEFVAR_BOOL cases, but this approach
will.


This patch has a couple of other minor tweaks in it: I had to rename a
few local variables that had the same name as one of the new macros, I
had to remove some (redundant) extern declarations, and in one case
("scroll_step") I had to rename a global to avoid a clash with a Gtk
header file.

These little changes I did by hand.  The major part of the patch was
written by a script, which I'm attaching.

I can commit the lesser changes separately, if anybody cares.


One last thing is that this patch doesn't handle buffer_defaults
properly yet.  That is not hard; I just think it should be an
independent change.


I did not write a ChangeLog entry for this yet.  I will update the .el
script to do this.

Tom


[-- Attachment #2: rewrite-globals.el --]
[-- Type: text/plain, Size: 5762 bytes --]

;; Rewrite DEFVAR_LISP variables.
;; Compatibility defines are added to globals.h.
;; Invoke as:  emacs --script rewrite-globals.el

(defvar defvar-list '())

(defvar global-text-list '())

(defvar variables-written '())

(defun error-at (text)
  (save-buffer)
  (error "%s:%d: %s"
	 (buffer-file-name)
	 (line-number-at-pos (point))
	 text))

(defun message-at (text)
  (save-buffer)
  (message "%s:%d: %s"
	   (buffer-file-name)
	   (line-number-at-pos (point))
	   text))

(defun extract-defvars ()
  "Extract DEFVAR_{LISP,INT,BOOL} variable names for later processing.
Also, rewrite DEFVAR invocations to remove the `&' from the 2nd arg."
  (let ((case-fold-search nil))
    (while (re-search-forward "^[^#*]*\\(DEFVAR_[A-Z_]*\\)" nil 'move)
      (let ((kind (match-string 1)))
	(unless (member kind '("DEFVAR_KBOARD" "DEFVAR_PER_BUFFER"))
	  ;; Skip the paren and the first argument.
	  (skip-chars-forward " (")
	  (forward-sexp)
	  (skip-chars-forward ", \t\n")
	  ;; `...' and `no_cell' are special -- they mean that
	  ;; this is a phony DEFVAR.
	  ;; buffer_defaults will have to be handled separately.
	  (unless (looking-at "\\(no_cell\\|\\.\\.\\.\\|&buffer_defaults[^,]*\\),")
	    (if (looking-at "&\\(\\_<\\(\\sw\\|\\s_\\)+\\_>\\)")
		(let ((var-name (match-string 1)))
		  ;; This would improve safety but it interferes with
		  ;; buffer_defaults.
		  ;; (delete-char 1)
		  ;; (insert "f_")
		  (push var-name defvar-list)))))))))

(defun maybe-extract-comment ()
  (let ((here (point)))
    (skip-chars-backward " \t\n")
    (backward-char 2)
    (if (looking-at "[*]/")
	(progn
	 (search-backward "/*" nil 'move)
	 (if (bobp)
	     ""
	   (let ((result (buffer-substring (point) here)))
	     (delete-region (point) here)
	     result)))
      (goto-char here)
      "")))

(defun munge-V ()
  (interactive)
  (while (re-search-forward "^\\(extern \\|static \\)?\\(Lisp_Object\\|int\\|EMACS_INT\\) \\([^(;]\\|\n\\)+;" nil 'move)
    (goto-char (match-end 2))
    (forward-char)
    (let ((is-extern (equal (match-string 1) "extern "))
	  (type (match-string 2))
	  (start (match-beginning 0))
	  (this-text "")
	  (this-comment "")
	  (skipped-one nil)
	  (last-start (point))
	  (deleted-last nil))
      (while (not (looking-at ";"))
	(if (and (looking-at "[a-z0-9A-Z_]+")
		 (member (match-string 0) defvar-list))
	    (let ((var-name (match-string 0))
		  (var-start (match-beginning 0))
		  (var-end (match-end 0)))
	      ;; Output to globals.h.
	      (unless is-extern
		(cond
		 ((member var-name variables-written)
		  (message-at "Duplicate seen"))
		 ((string-match "buffer_defaults_" var-name)
		  ;; Ignore.
		  )
		 (t
		  (setq this-text (concat this-text
					  type " " ;; "f_"
					  var-name ";\n"))
		  (push var-name variables-written))))
	      ;; FIXME: must handle comment stuff here too.
	      ;; Remove it and trailing ",".
	      (goto-char var-end)
	      (skip-chars-forward " \t\n,")
	      (delete-region var-start (point))
	      (setq deleted-last t)
	      ;; Leave LAST-START pointing to the same spot.
	      )
	  ;; If the last was a delete, delete the "," before this
	  ;; one.
	  (if deleted-last
	      (delete-region last-start (point)))
	  (setq deleted-last nil)
	  ;; Skip sexps until we hit the next declaration.
	  (while (not (looking-at "[;,]"))
	    (forward-sexp)
	    (skip-chars-forward " \t\n"))
	  (setq skipped-one t)
	  (setq last-start (point)))
	;; Move to the start of the next declaration.
	(skip-chars-forward ", \t\n"))
      (cond
       ((not skipped-one)
	;; If we removed all the individual declarations, then remove
	;; the whole thing.
	(end-of-line)
	(forward-char)
	(delete-region start (point))
	(setq this-comment (maybe-extract-comment)))
       (deleted-last
	;; If the last was a delete, delete the "," now.
	(delete-region last-start (point))))
      ;; Only update the globals when we see the variable's
      ;; definition.
      (if is-extern
	  (if (not (equal this-comment ""))
	      (message "declaration has comment: %s" this-text))
	(push (cons this-comment this-text) global-text-list)))))

(defconst V-dir ".")

(defun no-newlines ()
  (let ((here (point)))
    (skip-chars-backward " \t\n")
    (delete-region (point) here)))

(defun just-one-newline ()
  (no-newlines)
  (insert "\n"))

(defun munge-V-directory ()
  ;; Compute files early to avoid problems with .# files.
  (let ((files (directory-files V-dir t "[ch]$")))

    ;; First extract all defvars.
    (dolist (file files)
      (save-excursion
	(message "Scanning %s" file)
	(find-file file)
	(extract-defvars)
	(save-buffer)))

    (setq defvar-list (delete-dups (sort defvar-list #'string<)))

    (dolist (file files)
      (save-excursion
	(message "Processing %s" file)
	(find-file file)
	(goto-char (point-min))
	(munge-V)
	(save-buffer)))

    (message "Creating globals.h")
    (find-file "globals.h")
    (erase-buffer)
    (insert "struct emacs_globals\n{")
    (dolist (item (nreverse global-text-list))
      (insert "\n")
      (unless (string= (car item) "")
	(insert (car item))
	(just-one-newline))
      (insert (cdr item))
      (just-one-newline))
    (insert "};\n\n")
    (insert "extern struct emacs_globals globals;\n\n")
    (indent-region (point-min) (point-max))
    (goto-char (point-max))
    (dolist (v defvar-list)
      (insert "#define " v " \\\n    "
	      (if (string-match "buffer_defaults_" v)
		  (concat "buffer_defaults." (substring v 16))
		(concat "globals." ;; "f_"
			v))
	      "\n"))
    (let ((version-control 'never))
      (save-buffer))

    (message "Updating lisp.h")
    (find-file "lisp.h")
    (goto-char (point-max))
    (search-backward "#endif")
    (insert "#include \"globals.h\"\n\n")
    (save-buffer)))

(munge-V-directory)

[-- Attachment #3: the patch --]
[-- Type: application/x-gzip, Size: 68143 bytes --]

Re: baby step toward thread patch

[Tom Tromey]

>>>>> "Tom" == Tom Tromey <tromey@redhat.com> writes:

Tom> I'd like to solve the latter problem by putting some needed
Tom> infrastructure directly on trunk.  Once all the infrastructure bits
Tom> are merged, development can focus on just the threading bits.

Tom> I'm posting this to get an ok on this change before checking it in.  In
Tom> the absence of comments (after a decent interval) I will just go ahead
Tom> with it.

It has been a couple of weeks without comment.

So, I have committed the latest version of all my changes.  This is a
bit different than what I sent two weeks ago, but not drastically so.

If it causes problems, please let me know, and I will see what I can do
to fix them.

I separated out each logical change.  However, the very last change
consists of a hand-generated patch combined with a quite large
automatically-generated patch.  I could not separate these without
breaking the build.

If anybody is interested, I can send the final version of the rewriting
script.

Tom> One last thing is that this patch doesn't handle buffer_defaults
Tom> properly yet.  That is not hard; I just think it should be an
Tom> independent change.

This remains true.  Updating buffer_defaults will be my next patch.  It
may be a while before I get to this, as my Emacs hacking time is
extremely limited.

After buffer defaults I will tackle per-buffer and per-keyboard
bindings.  For these, I will require some assistance, as the source
rewriter for this patch requires compiling Emacs, and I don't have
access to all platforms.

I think that will be all the needed infrastructure bits; then we can
look into rebasing the concurrency branch (or, more likely, making a new
branch).

Tom

Re: baby step toward thread patch

[Thien-Thi Nguyen]

() Tom Tromey <tromey@redhat.com>
() Tue, 18 Jan 2011 13:42:24 -0700

   If anybody is interested, I can send the final version
   of the rewriting script.

Perhaps you can add it under admin/ in a new directory
named edits/, with a README for the directory and comments
in the script referencing the particular commit(s).  Even
if the nature of the edit is extremely specific, it surely
uses techniques that can be used in the future.

Re: baby step toward thread patch

[Glenn Morris]

Tom Tromey wrote:

> So, I have committed the latest version of all my changes.

Something wrong in the following diff hunk?

   window.c:154:8: warning: extra tokens at end of #endif directive

*** src/window.c	2011-01-15 23:16:57 +0000
--- src/window.c	2011-01-18 20:45:37 +0000
***************
*** 194,210 ****
  #if 0 /* This isn't used anywhere.  */
  /* Nonzero means we can split a frame even if it is "unsplittable".  */
  static int inhibit_frame_unsplittable;
! #endif /* 0 */
! 
! extern EMACS_INT scroll_margin;
! 
! extern Lisp_Object Qwindow_scroll_functions, Vwindow_scroll_functions;
! 
! /* If non-nil, then the `recenter' command with a nil argument
!    the entire frame to be redrawn; the special value `tty' causes the
!    frame to be redrawn only if it is a tty frame.  */
! 
! static Lisp_Object Vrecenter_redisplay;
  
  \f
  DEFUN ("windowp", Fwindowp, Swindowp, 1, 1, 0,
--- 151,157 ----
  #if 0 /* This isn't used anywhere.  */
  /* Nonzero means we can split a frame even if it is "unsplittable".  */
  static int inhibit_frame_unsplittable;
! #endif extern Lisp_Object Qwindow_scroll_functions, Vwindow_scroll_functions;
  
  \f
  DEFUN ("windowp", Fwindowp, Swindowp, 1, 1, 0,

Re: baby step toward thread patch

[Tom Tromey]

>>>>> "GM" == Glenn Morris <rgm@gnu.org> writes:

GM> Something wrong in the following diff hunk?
GM>    window.c:154:8: warning: extra tokens at end of #endif directive

Thanks.  I am checking in a fix.
Just removing the stuff after the #endif is fine.

Tom

Re: baby step toward thread patch

[Tom Tromey]

Tom> Updating buffer_defaults will be my next patch.  It may be a while
Tom> before I get to this, as my Emacs hacking time is extremely
Tom> limited.

I looked at this a bit and then realized I was mistaken --
buffer_defaults will be handled fine by the same change that makes
ordinary buffers be handled correctly.  So, I am going to focus on that
patch instead.

Tom