From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED!not-for-mail From: Lars Ingebrigtsen Newsgroups: gmane.emacs.devel Subject: Re: libnettle/libhogweed WIP Date: Fri, 21 Apr 2017 20:45:58 +0200 Message-ID: References: <87lgr6yakj.fsf@lifelogs.com> <87wpamww9k.fsf@lifelogs.com> <8337daggnj.fsf@gnu.org> <87d1cdwxt6.fsf@lifelogs.com> <83tw5pg1q3.fsf@gnu.org> <87zifhulc2.fsf@lifelogs.com> <83h91og80k.fsf@gnu.org> <87pogbuhoe.fsf@lifelogs.com> <834lxndmd9.fsf@gnu.org> <87efwrug6z.fsf@lifelogs.com> <87r30qu5av.fsf@lifelogs.com> <874lxmtxyy.fsf@lifelogs.com> <87r30prvwt.fsf@lifelogs.com> <8337d4csez.fsf@gnu.org> <87r30nq9el.fsf@lifelogs.com> <83inlyc1k2.fsf@gnu.org> <87inlyrfni.fsf@lifelogs.com> <878tmurbhj.fsf@lifelogs.com> NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Trace: blaine.gmane.org 1492800382 12063 195.159.176.226 (21 Apr 2017 18:46:22 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Fri, 21 Apr 2017 18:46:22 +0000 (UTC) User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.0.50 (gnu/linux) To: emacs-devel@gnu.org Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Fri Apr 21 20:46:17 2017 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1d1dZR-0002xS-IZ for ged-emacs-devel@m.gmane.org; Fri, 21 Apr 2017 20:46:17 +0200 Original-Received: from localhost ([::1]:60994 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d1dZU-0006d6-0o for ged-emacs-devel@m.gmane.org; Fri, 21 Apr 2017 14:46:20 -0400 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:42671) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d1dZN-0006c7-Bh for emacs-devel@gnu.org; Fri, 21 Apr 2017 14:46:14 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1d1dZJ-0002Io-EL for emacs-devel@gnu.org; Fri, 21 Apr 2017 14:46:13 -0400 Original-Received: from hermes.netfonds.no ([80.91.224.195]:37606) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1d1dZJ-0002IL-49 for emacs-devel@gnu.org; Fri, 21 Apr 2017 14:46:09 -0400 Original-Received: from cm-84.213.17.174.getinternet.no ([84.213.17.174] helo=stories) by hermes.netfonds.no with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.84_2) (envelope-from ) id 1d1dZ8-0004kr-Jg for emacs-devel@gnu.org; Fri, 21 Apr 2017 20:46:01 +0200 Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAAGFBMVEXp3sxYUU1BNTMtGxwn DA+xqZwRAgWBeXIByN+GAAACVUlEQVQ4jXWUTW8aMRCG3cMq144R2ms9Rsk59hbOwIqzSWX5Hjbu FW9kzd/v2JhFqdoRCLSP5+sdzwon/mNOgYTFemp/vgt3weE4Go1Yvhq1HadT3HwI/64HYwFnoyD/ SKhfs3wuwE1W8zkFiNmtEeWO1BAHBnEcj8as3+yLEGciBfoUY2RwigxUH4jr8+TlvGogTuPRWsye S+y8OGsdv4Ce/M2S3N7BcLQGFVHi+lUPevvwQOTWikvXhYTbOFVw2jBQpW3ouXM5xHtVmwNKjlQ8 SjxrHyBlfnqTsyOLLUcc9uTJNZm7YPU9+Waf8vkq/gankT0ouG4B9h6KPfrsRMvh9Q14DnUAoHsK 7+0CxgMUj2JccfgCoHf1NNt5sIske8myZz5cPkk/wEFiuQcJuP18/gpQopIgWcrUOvc3dRkUiXta J3wA9pBzYTwO9w0Sg98V7AGkxRevVmZ2TxRifG+AkjTpmmmFs+syg8st1M5T2mXHGgfynQ9Ljp/c 3O57R+JzHTAQLR4FvIrOiV9cEwFXNS1AvLDsIXmY3/SwNLij7Hu6iiQDzp92rB4FvLIWQJm77nlX JE/w9CEyA8NPgZ/n0J+pT9zg9kNcGVi8GyjCMo8GDM6gEusxAyRWuYFhOPBRSNxdsweoe+kC1fVc wIX3pqyu6/i6S5hLH6cC4mBKXsUAFS9u6e+5gqkCvD6l8lPXI1bQyl2HG2hrkPm+1cEmHspcZqzx vXY+6SOaSnqos9cGL5pDTcdR8nIqXeOstLXjZRgZmCZGTcB6jLEl/+c7qXN/AB6uGTCfF6EJAAAA AElFTkSuQmCC In-Reply-To: <878tmurbhj.fsf@lifelogs.com> (Ted Zlatanov's message of "Thu, 20 Apr 2017 17:54:32 -0400") X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 80.91.224.195 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.org gmane.emacs.devel:214196 Archived-At: Ted Zlatanov writes: > The KEY is secret and ideally would come from a file and never be > seen at the Lisp level. But tests and other use cases may need it from a > buffer (more secure but still accessible to Lisp) or a string (visible > to all as a function parameter). Hm... Having a file that just has a passphrase in it sounds like an unusual use case. I think in Emacs these tokens would normally come from auth-source in most applications. At least that what I see when I salivate at use cases. :-) > Getting the INPUT from a file enables large files (not in the first > version probably) and other interesting use cases. Emacs buffers are surprisingly efficient at handling large files: They're basically just (sort of) contiguous areas of memory with some structs describing their contents. Here's how long it takes this machine to put a 4GB .iso file into a buffer (and then kill Emacs): [larsi@stories ~]$ time emacs -batch --eval "(with-temp-buffer (set-buffer-= multibyte nil) (let ((coding-system-for-read 'binary)) (insert-file-content= s \"~/Downloads/debian-8.6.0-amd64-DVD-1.iso\") (message \"%s\" (buffer-siz= e))))" 3994091520 real 0m1.008s user 0m0.012s sys 0m0.988s To compare, this is how long it takes this machine to just output it all to /dev/null: [larsi@stories ~]$ time cat ~/Downloads/debian-8.6.0-amd64-DVD-1.iso > /dev= /null =20 real 0m0.294s user 0m0.000s sys 0m0.292s So the Emacs primitives are definitely competitive in the "read a huge file" stakes. I think asking Emacs to encrypt a 4GB file will be a very common use case, but it's doable without creating special handling. If I understand the code correctly (and I may definitely not be doing that; I've just skimmed it very, very briefly), you may be able to point the encryption code at the Emacs buffer contents directly without copying it anywhere beforehand, and then (since the results are usually of very similar length) back to the same Emacs buffer afterwards. 4GB Emacs buffer -> encrypted to 4GB GnuTLS buffer -> 4GB Emacs buffer instead of 4GB Emacs buffer -> copy to 4GB gnutls.c buffer -> encrypted to 4GB GnuTLS buffer -> made into Emacs string or something so you save at least one 4GB buffer by just taking the data directly from the buffer and putting it back in the same place. (So 8GB total memory print instead of 12GB or even possibly 16GB in the current code.) > LI> In any case, the `file' case you're discussing here doesn't really fe= el > LI> that useful, but also makes things more complicated. If the user wan= ts > LI> to encrypt a file, then it's more flexible to just have the caller > LI> insert the file into a buffer and call the function as normal > > Aboslutely. It would be nice if the Emacs C core had "readers" like Java > or Go because then this discussion would be really simple: "did you use > a reader" - "yes" - "good" :) I guess what I'm saying is that Emacs has readers, and we call those "Emacs buffers". :-) The other problem with having a special file handler in the GnuTLS code is that users will expect to be able to encrypt all files that they see visible from Emacs, including the ones from Tramp, and application writers will also have differing opinions on whether encrypting a .gz file means encrypting the contents of the file or the file itself: That is, Emacs has a very rich file handler jungle that it would be nice if still works when you ask Emacs to encrypt something. You'd have to handle (file "~/foo) (file "c:/foo/bar") (file "H=C3=A9llo") ; in iso-8859-1 (file "/ssh:host:/tmp/foo") both as input and output specifiers if you never want the file contents to his Elisp Land... It all sounds a bit daunting. To me, at least. :-) Instead we have most of the primitives we need for safe handling of secrets in Emacs already; a few more should be added. But I think this pattern for handling secret files could be tweaked and macroised after some code review: (with-temp-buffer (set-buffer-multibyte nil) (let ((coding-system-for-read 'binary) (coding-system-for-write 'binary)) (unwind-protect (progn (insert-file-contents "My DVD.iso") (gnutls-encrypt ... ... (current-buffer)) (write-region ...)) (clear-buffer (current-buffer))))) ;; New function that runs memset ;; over the buffer area =20=20=20=20 Or something. We have to look at what buffers write-region creates and stuff, but in the 'binary case, I don't think it creates copies of the Emacs buffer anywhere. Of course, if these files read and written are via Tramp or a complex file handler, we can't guarantee that those don't leave a buffer anywhere, but... --=20 (domestic pets only, the antidote for overdose, milk.) bloggy blog: http://lars.ingebrigtsen.no