>> Just let-bind text-quoting-style:
>>
>> (let ((text-quoting-style 'grave))
>>    (format-message "foo `bar' baz"))
>
> This won't work when I sometimes actually want the replacement to 
> happen:
>
> (format-message "This is variable `foo', and the code 'bar")
>

Indeed.  If you want both literal ` ' and interpreted ` ', until Stefan's 
suggestion is implemented, you can either use what Andreas suggested, or 
use a variant of format-message such as:

(defun format-message-alt (format &rest objects)
   "Format a string out of a format-string and arguments.

This acts like `format-message', which see, except that the grave
accent (\\=`) and apostrophe (\\=') can be escaped with `\\\\​=',
in which case they are not replaced by the left and right quote
replacement characters specified by `text-quoting-style'."
   (let* ((fq
 	  (replace-regexp-in-string
 	   "\\\\=`" "\uE001"
 	   (replace-regexp-in-string
 	    "\\\\='" "\uE000" format)))
 	 (fm (format-message fq objects))
 	 (fu
 	  (replace-regexp-in-string
 	   "\uE001" "`"
 	   (replace-regexp-in-string
 	    "\uE000" "'" fm))))
     fu))