Upcoming GMime 3.0 changes

Upcoming GMime 3.0 changes

[Jeffrey Stedfast]

Hello notmuch devs,

I'm sending this email to inform you guys of some upcoming GMime 3.0 features and other changes that I've implemented.

1. Probably the most import change coming to GMime 3.0 is that each header field/value pair has its own raw value cache instead of using a cached stream of the entire head block, thus forcing the complete invalidation of the header cache when anything changed in the headers. In GMime 2.6, once the header cache was invalidated, each header would get re-folded when written back out to disk. This was obviously not ideal.

2. Next up is the replacement of the old custom GnuPG back-end with GpgMe. Also included with this change is full support for S/MIME via both multipart/signed and application/pkcs7-mime content-types using gpgsm (via GpgMe). During this change, I also took the liberty of simplifying the crypto API's a bit and so I was able to make it such that g_mime_multipart_signed_verify(), for example, no longer requires you to pass it a GMimeCryptoContext. Instead, GMime parses the Content-Type's protocol parameter and uses that to instantiate the correct crypto context (GMimeGpgContext for PGP and GMimePkcs7Context for S/MIME). Many of the various state properties have been replaced with bitflags that can be passed to encrypt() and decrypt(). The sign() method now also takes a detach argument (might make this into a bitflag instead?) in order to support encapsulated signing.

3. The GMimeMessage API's have been improved a bit. Gone are the old GMimeMessage API's that return strings for the From and Reply-To headers. Instead, *all* GMimeMessage API's return an InternetAddressList pointer which can be used to manipulate the parsed address header. Also improved is the g_mime_message_set_subject() method which now takes an optional charset parameter (or NULL to use the default).

4. New in GMime 3.0 is the GMimeParserOptions struct which can be passed to GMimeParser and other parser functions exposed in the lower-level API's. This structure helps define how strict/forgiving the various parsing routines should be with the input. This replaces the need for g_mime_init()'s flags so you can change these settings on the fly now.

5. Another new feature is GMimeTextPart which makes getting/setting text on a GMimePart simpler. GMimeTextPart is a subclass of GMimePart, so all of the GMimePart API's that you are used to still work but now you also have access to convenient functions like g_mime_text_part_get_text() and g_mime_text_part_set_text() which removes the need for creating streams, data wrappers, and setting them on the part.

6. Brand new rfc822 address parser which is more tolerant than the previous generation parser. What's not to love?

7. And finally we get to a nifty feature that I just hacked up while waiting for some other code to compile (hey, it takes an hour to compile... I needed something to do!) which is that GMimeParser now scans for -----BEGIN PGP MESSAGE-----/-----END PGP MESSAGE----- and -----BEGIN PGP SIGNED MESSAGE-----/-----END PGP SIGNED MESSAGE----- markers while looking for MIME boundaries and sets some state on the corresponding GMimePart that you can use to quickly decide if the part contains encapsulated OpenPGP data.


Note: for the full list of API changes made, see https://github.com/jstedfast/gmime/blob/master/PORTING

Jeff

Re: Upcoming GMime 3.0 changes

[David Bremner]

Jeffrey Stedfast <jestedfa@microsoft.com> writes:

> Hello notmuch devs,
>
> I'm sending this email to inform you guys of some upcoming GMime 3.0 features and other changes that I've implemented.
>

Thanks for the heads up. At least the GPGME stuff sounds like good news,
we may finally have working S/MIME encryption/decryption

d

RE: Upcoming GMime 3.0 changes

[Jeffrey Stedfast]

> -----Original Message-----
> From: David Bremner [mailto:david@tethera.net]
> Sent: Tuesday, March 14, 2017 9:18 AM
> To: Jeffrey Stedfast <jestedfa@microsoft.com>; notmuch@notmuchmail.org
> Subject: Re: Upcoming GMime 3.0 changes
> 
> Jeffrey Stedfast <jestedfa@microsoft.com> writes:
> 
> > Hello notmuch devs,
> >
> > I'm sending this email to inform you guys of some upcoming GMime 3.0
> features and other changes that I've implemented.
> >
> 
> Thanks for the heads up. At least the GPGME stuff sounds like good news,
> we may finally have working S/MIME encryption/decryption

No problem! Notmuch and Balsa are the 2 main projects I know of using GMime so I want to make sure you guys get a chance to comment on the changes I've been making for 3.0. I also want you guys to know that if there's anything that you wished GMime had that it doesn't, now is a great time to make the suggestion 😊

(oh, btw, I just added support for inline-pgp encrypt/decrypt/sign/verify this morning)

Daniel Kahn Gillmor has already been keeping in touch with me on improvements to the GPG support which is why I made the switch to GpgME, so once I did that, it was not much effort to add support for S/MIME (I believe someone from the notmuch group actually asked me what it would take to implement S/MIME support sometime this past year and I had started to look into it but never got a chance to finish until now).

Jeff

RE: Upcoming GMime 3.0 changes

[David Bremner]

Jeffrey Stedfast <jestedfa@microsoft.com> writes:

>> -----Original Message-----
>> From: David Bremner [mailto:david@tethera.net]
>> Sent: Tuesday, March 14, 2017 9:18 AM
>> To: Jeffrey Stedfast <jestedfa@microsoft.com>; notmuch@notmuchmail.org
>> Subject: Re: Upcoming GMime 3.0 changes
>> 
>> Jeffrey Stedfast <jestedfa@microsoft.com> writes:
>> 
>> > Hello notmuch devs,
>> >
>> > I'm sending this email to inform you guys of some upcoming GMime 3.0
>> features and other changes that I've implemented.
>> >

Just for the record, I have some patches in progress for porting to
gmime-3.0. The main issue is the multiplicity of memory management
models involved. I think the gmime 3.0 approach of using more stock glib
memory management makes sense, but it will require a bit of work to make
code that can compile against gmime-2.6 and gmime-3.0. I don't see us
being able to drop support for gmime-2.6 for a few years, unfortunately.

d

RE: Upcoming GMime 3.0 changes

[Daniel Kahn Gillmor]

On Tue 2017-05-09 10:37:30 -0300, David Bremner wrote:
> Just for the record, I have some patches in progress for porting to
> gmime-3.0. The main issue is the multiplicity of memory management
> models involved. I think the gmime 3.0 approach of using more stock glib
> memory management makes sense, but it will require a bit of work to make
> code that can compile against gmime-2.6 and gmime-3.0. I don't see us
> being able to drop support for gmime-2.6 for a few years, unfortunately.

out of curiosity, why do you think we won't be able to drop gmime-2.6
for a few years?  if it's due to the debian release cycle and wanting to
backport notmuch to stretch, i don't think i'd mind providing backports
of gmime 3.0 for that purpose.

   --dkg

RE: Upcoming GMime 3.0 changes

[David Bremner]

Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes:

>
> out of curiosity, why do you think we won't be able to drop gmime-2.6
> for a few years?  if it's due to the debian release cycle and wanting to
> backport notmuch to stretch, i don't think i'd mind providing backports
> of gmime 3.0 for that purpose.

That was what I was thinking about (and other distros with similar
release cycles). But even among our own developers I don't know we can
just throw a switch and say "install gmime 3.0 before you git pull".  I
guess the other point is even with backports, there is still the issue
of security support for older releases. We'll need to maintain the 2.6
stuff in a branch, even if it only gets security/severe bug updates.

Another question is how long Jeff plans to support 2.6 with bug fixes.

d

RE: Upcoming GMime 3.0 changes

[Tomi Ollila]

On Tue, May 09 2017, David Bremner wrote:

> Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes:
>
>>
>> out of curiosity, why do you think we won't be able to drop gmime-2.6
>> for a few years?  if it's due to the debian release cycle and wanting to
>> backport notmuch to stretch, i don't think i'd mind providing backports
>> of gmime 3.0 for that purpose.
>
> That was what I was thinking about (and other distros with similar
> release cycles). But even among our own developers I don't know we can
> just throw a switch and say "install gmime 3.0 before you git pull". 

I personally can compile gmime 3.0 for this (EOL'd) scientific linux 6.2
machine where I am writing this email, provided it does not need c/c++11
features(*). If gmime-3.0 could be statically linked to notmuch binary then
we could even provide an option to have it bundled into notmuch when
compiled (like I've included zlib...).

> I guess the other point is even with backports, there is still the issue
> of security support for older releases. We'll need to maintain the 2.6
> stuff in a branch, even if it only gets security/severe bug updates.

But these go to releases (how many old of those) which still have support
for gmime 2.6...

>
> Another question is how long Jeff plans to support 2.6 with bug fixes.
>
> d

Tomi

(*) I compiled Xapian 1.4 for Fedora 25; compiled fine and notmuch worked
with it. On SL6.2 compilation failed, and when running
/opt/rh/devtoolset-2/root/usr/bin/gcc compilation crashed at some point,