From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED!not-for-mail From: Ted Zlatanov Newsgroups: gmane.emacs.devel Subject: Re: libnettle/libhogweed WIP Date: Mon, 15 May 2017 17:55:34 -0400 Organization: =?utf-8?B?0KLQtdC+0LTQvtGAINCX0LvQsNGC0LDQvdC+0LI=?= @ Cienfuegos Message-ID: <87ziedpyy1.fsf@lifelogs.com> References: <83y3w5z1ez.fsf@gnu.org> <87lgr6yakj.fsf@lifelogs.com> <87wpamww9k.fsf@lifelogs.com> <8337daggnj.fsf@gnu.org> <87d1cdwxt6.fsf@lifelogs.com> <83tw5pg1q3.fsf@gnu.org> <87zifhulc2.fsf@lifelogs.com> <83h91og80k.fsf@gnu.org> <87pogbuhoe.fsf@lifelogs.com> <834lxndmd9.fsf@gnu.org> <87efwrug6z.fsf@lifelogs.com> <87r30qu5av.fsf@lifelogs.com> <874lxmtxyy.fsf@lifelogs.com> <87r30prvwt.fsf@lifelogs.com> <8337d4csez.fsf@gnu.org> <87r30nq9el.fsf@lifelogs.com> <83inlyc1k2.fsf@gnu.org> <87inlyrfni.fsf@lifelogs.com> <837f2eb845.fsf@gnu.org> NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Trace: blaine.gmane.org 1494885358 10511 195.159.176.226 (15 May 2017 21:55:58 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Mon, 15 May 2017 21:55:58 +0000 (UTC) User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.0.50 (gnu/linux) To: emacs-devel@gnu.org Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Mon May 15 23:55:53 2017 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dANy4-0002by-M2 for ged-emacs-devel@m.gmane.org; Mon, 15 May 2017 23:55:52 +0200 Original-Received: from localhost ([::1]:38997 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dANyA-0002sp-7H for ged-emacs-devel@m.gmane.org; Mon, 15 May 2017 17:55:58 -0400 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:57494) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dANy1-0002sT-6o for emacs-devel@gnu.org; Mon, 15 May 2017 17:55:52 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dANxx-0001VZ-3Q for emacs-devel@gnu.org; Mon, 15 May 2017 17:55:49 -0400 Original-Received: from [195.159.176.226] (port=47105 helo=blaine.gmane.org) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1dANxw-0001VN-PH for emacs-devel@gnu.org; Mon, 15 May 2017 17:55:45 -0400 Original-Received: from list by blaine.gmane.org with local (Exim 4.84_2) (envelope-from ) id 1dANxp-0002Ic-LO for emacs-devel@gnu.org; Mon, 15 May 2017 23:55:37 +0200 X-Injected-Via-Gmane: http://gmane.org/ Mail-Followup-To: emacs-devel@gnu.org Original-Lines: 156 Original-X-Complaints-To: usenet@blaine.gmane.org X-Face: bd.DQ~'29fIs`T_%O%C\g%6jW)yi[zuz6; d4V0`@y-~$#3P_Ng{@m+e4o<4P'#(_GJQ%TT= D}[Ep*b!\e,fBZ'j_+#"Ps?s2!4H2-Y"sx" Mail-Copies-To: never Cancel-Lock: sha1:oBRpLrfEo/137wtEe4GhzTUcAP4= X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 195.159.176.226 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.org gmane.emacs.devel:214865 Archived-At: On Fri, 21 Apr 2017 09:14:02 +0300 Eli Zaretskii wrote: EZ> Something like this at the beginning of the node: >> EZ> All of the functions described here accept argument lists of the EZ> form @code{(BUFFER-OR-STRING START END CODING-SYSTEM NOERROR)}, EZ> where BUFFER-OR-STRING ... >> >> OK, but I want to link to that from the C function docs too, since they >> also will reference that format. Would I link to the manual node? Is >> that OK for C functions, which are kind of standalone as far as docs go? EZ> If you mean primitives implemented in C that are exposed to Lisp, EZ> yes. For internal C functions inaccessible from Lisp, just describe EZ> that in a comment preceding the functions. OK, I'll add those links instead of the full text. EZ> I think both this and the other sub-thread arrived at a point where it EZ> is important to present a list of use cases we envision and would like EZ> to support, because these kinds of decisions are prone to errors if we EZ> don't hold the use cases in our minds. EZ> So could you please present such a list, describing the source of the EZ> text to be encrypted/decrypted/hashed, the purpose of the operation EZ> (i.e. some higher-level context), and the destination where the EZ> encrypted/decrypted/hashed text will go? The list doesn't have to be EZ> exhaustive, but it should include the most important use cases. Right now, I am mirroring the GnuTLS API. That API is in wide use. So I think the use cases are a large subset of the general GnuTLS use cases; we're enabling the same things. On Fri, 21 Apr 2017 09:21:14 +0300 Eli Zaretskii wrote: >> From: Ted Zlatanov >> Date: Thu, 20 Apr 2017 17:54:32 -0400 >> >> The KEY is secret and ideally would come from a file and never be >> seen at the Lisp level. But tests and other use cases may need it from a >> buffer (more secure but still accessible to Lisp) or a string (visible >> to all as a function parameter). EZ> For testing, we could always write the key to a file before using it. EZ> What other use cases would need the key from other sources? I think if the key is not on disk, we shouldn't force it to disk just to fit a always-a-file usage model. OTOH if it's already on disk, we shouldn't slurp it into memory to fit the always-in-memory usage model. Both of those situations expose the key data by making copies. >> Getting the INPUT from a file enables large files (not in the first >> version probably) and other interesting use cases. EZ> What other cases? Large files is only theoretically useful, since EZ> generally Emacs cannot do useful things on files larger than EZ> most-positive-fixnum, and on 64-bit machines that is far enough to not EZ> care. I think anything over 1 MB is pretty big. There also pipes (pretty easy to fit the file model) and network streams (probably a separate spec). EZ> I think we need to weigh flexibility against the complexity, and find EZ> the optimal balance. So making the interfaces too complicated for use EZ> cases that will happen only very rarely, if at all, should be avoided. I agree in general, BUT these are not end-user APIs. They are for application developers. They have to be flexible so we don't have to bolt-on these things later. Users will get something much simpler or they won't even see these interfaces (the application will hide them). On Fri, 21 Apr 2017 20:45:58 +0200 Lars Ingebrigtsen wrote: LI> Ted Zlatanov writes: LI> Hm... Having a file that just has a passphrase in it sounds like an LI> unusual use case. I think in Emacs these tokens would normally come LI> from auth-source in most applications. At least that what I see when I LI> salivate at use cases. :-) Private SSH keys are a good example; see https://github.com/jschauma/jass for instance. But generally as I said above, we shouldn't force a copy file->string or string->file if the private data is already in one form. LI> Emacs buffers are surprisingly efficient at handling large files: LI> They're basically just (sort of) contiguous areas of memory with some LI> structs describing their contents. OK, but buffers are a copy of the file data. I'd rather not make a copy. LI> If I understand the code correctly (and I may definitely not be doing LI> that; I've just skimmed it very, very briefly), you may be able to point LI> the encryption code at the Emacs buffer contents directly without LI> copying it anywhere beforehand, and then (since the results are usually LI> of very similar length) back to the same Emacs buffer afterwards. LI> 4GB Emacs buffer -> encrypted to 4GB GnuTLS buffer -> 4GB Emacs buffer LI> instead of LI> 4GB Emacs buffer -> copy to 4GB gnutls.c buffer -> encrypted to 4GB LI> GnuTLS buffer -> made into Emacs string or something Yes, definitely possible. But it's more secure, I think, to read chunks from the file and process them (possibly overwriting the data) in a small loop, narrowing the scope and risk of the data exposure. The GnuTLS APIs are designed for that usage. It's faster but less interruptible as well. So it's not ideal for every situation, but I would like to support it. On Fri, 21 Apr 2017 22:15:16 +0300 Eli Zaretskii wrote: EZ> The data will always leave traces, because doing the above involves EZ> reallocation of memory, so you are likely to leave traces in the page EZ> file and in memory. But I don't think you can avoid that, whatever EZ> you do: as long as data needs to be read into memory to process it, it EZ> will always leave traces. Would you agree the tight loop that overwrites the read block will leave fewer traces and offer fewer exposure opportunities? LI> The other problem with having a special file handler in the GnuTLS code LI> is that users will expect to be able to encrypt all files that they see LI> visible from Emacs, including the ones from Tramp, and application LI> writers will also have differing opinions on whether encrypting a .gz LI> file means encrypting the contents of the file or the file itself: That LI> is, Emacs has a very rich file handler jungle that it would be nice if LI> still works when you ask Emacs to encrypt something. LI> You'd have to handle LI> (file "~/foo) LI> (file "c:/foo/bar") LI> (file "Héllo") ; in iso-8859-1 LI> (file "/ssh:host:/tmp/foo") Right, I understand. I am OK with restricting the API to local files only but agree the name handling needs to be done carefully. Lars, I think your read-into-buffer macro would work nicely in a wrapper API (something like EPA's contexts). We can modify the macro if the `(file "foo")' spec becomes available, and end users won't know the difference. So how about a compromise for now: I can leave the `(file "foo")' capability out. I'll adjust the docs to remove mentions of it. Then, after the main patch is done, I can propose a followup patch to implement `(file "foo")' and we can decide if it's good or bad. That will get the GnuTLS API integration working, and we can have a separate discussion about `(file "foo")' later. Lars, Eli, would that be acceptable? Thanks! Ted