Status of Vim client

Status of Vim client

[J. Lewis Muir]

Hello, Notmuchers!

What is the status of the Vim client?  The last commit to
vim/notmuch.vim was over a year and a half ago (2015-01-23).

I'm also aware of two Vim clients on GitHub:

* https://github.com/felipec/notmuch-vim (last commit: 2014-05-22)

* https://github.com/imain/notmuch-vim (last commit: 2015-08-24)

So, are the clients really stable, or are they rotting because nobody
uses them?  My hope is the former, but I suspect the latter.  Are most
of the Vim people just using Mutt or NeoMutt?

Thanks!

Lewis

Re: Status of Vim client

[Lucas Hoffmann]

[-- Attachment #1: Type: text/plain, Size: 185 bytes --]

Quoting J. Lewis Muir (2016-09-29 17:21:16)
> Are most of the Vim people just using Mutt or NeoMutt?

At least one of them Vim people is using alot :)
https://github.com/pazz/alot

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

-----BEGIN PGP SIGNATURE-----

iQEcBAABCAAGBQJX7VTKAAoJEGC3H8cAGkGhOmAH/114almFvAwzrILo/RUembA9
J5+KKtdyhqImKn6tMYC6KvIs2JO85j69PnJnmOfgcpn+SnLjNThE+SYGO4dAZY5c
ntpYXynT1I2Q7ppKdGfbBKAp54WZs0J4swgNyODcJwdTG0ymFpQvoXpVIh/Jgleo
IAoKu53jLWiaz/wa9KFrPdQ9DrqK+HQ8utOQAYsPI6KgCmxKN0ZHUQaVQUDEPbVr
ZU1UfxXHFOSOY/F2cNP3f2JgN2DY/tVOZ256yom898e9qHXBh0RO3+gamILRPP03
hO6j1ftPAmN3d66IPNC/Zp4km/mRSNUTd8jhW/CD32ZfM139P8SoAhMR/VycPdE=
=IrqN
-----END PGP SIGNATURE-----

Re: Status of Vim client

[David Bremner]

"J. Lewis Muir" <jlmuir@imca-cat.org> writes:

> So, are the clients really stable, or are they rotting because nobody
> uses them?  My hope is the former, but I suspect the latter.  Are most
> of the Vim people just using Mutt or NeoMutt?

FWIW, I was recently considering moving the vim client to contrib, and
then dropping it completely from distribution after some time. So
basically was asking myself the same question.

d

Re: Status of Vim client

[Nick Howell]

Lewis wrote:
> I'm also aware of two Vim clients on GitHub:
> 
> * https://github.com/felipec/notmuch-vim (last commit: 2014-05-22)
> 
> * https://github.com/imain/notmuch-vim (last commit: 2015-08-24)
> 
> So, are the clients really stable, or are they rotting because nobody
> uses them?  My hope is the former, but I suspect the latter.

I have a few patchsets adding:
- attachment support in compose
- mailcap support for multipart messages
- pgp support (decorations for reading, controls for composing)
- block operations on messages when searching
as well as miscellaneous minor bugfixes/features.

These are in various states of cleanliness, but none are directly
upstreamable right now. The first three I tried to upstream to felipec
a year or two ago, but I guess we were both too busy to push it.
(Excuse the pun.) 

If you're interested, I'm happy to share. I could invest a bit of time
now; I just didn't think anyone but me cared.

Cheers,
Nick

Re: Status of Vim client

[Nick Howell]

Nick Howell wrote:
> I have a few patchsets adding:

> - attachment support in compose
> - mailcap support for multipart messages

Mildly clean (imo), independent versions of these have been sent to
the mailing list; they are meant to be applied on-top of felipec's
repository. (I was unaware of, and haven't looked at, the imain repo.)

> - pgp support (decorations for reading, controls for composing)

This is pretty messy right now, especially given that it requires
patching mail-gpg and ruby-gpgme, and also uses some features
command-line gpg exposes that ruby-gpgme doesn't. I don't think I can
get an RFC version out for awhile. For anything to be sane, we would
have to upstream to these two ruby libraries. I hope we wouldn't have
to upstream to gpgme (as this would involve new code, as opposed to
massaging what I already have).

> - block operations on messages when searching

I can probably get an RFC version of this out in a couple days.

> as well as miscellaneous minor bugfixes/features.

Hopefully many of these patches will disappear in the rebase/cleanup
process. (Though I know where policy for rebasing is probably defined,
I haven't looked at it.)

I can also send out a monster mess 75-patch series, which includes
everything; maybe this would be useful for folks who want to know what
the finished product will look like. But it's too big and unfocused to
ask RFC on it.

Cheers,
Nick

p.s. please let me know if I broke some rule of patch mailing
ettiquette, e.g. should I have included "vim:" or based the patches
off of notmuch.git?

Re: Status of Vim client

[David Bremner]

Nick Howell <nlhowell@gmail.com> writes:

> Nick Howell wrote:
>> I have a few patchsets adding:
>
>> - attachment support in compose
>> - mailcap support for multipart messages
>
> Mildly clean (imo), independent versions of these have been sent to
> the mailing list; they are meant to be applied on-top of felipec's
> repository. (I was unaware of, and haven't looked at, the imain repo.)

For whatever reason, I didn't see those messages on the (i.e. this) mailing list.

> p.s. please let me know if I broke some rule of patch mailing
> ettiquette, e.g. should I have included "vim:" or based the patches
> off of notmuch.git?

Yes, from our point of view notmuch.git is upstream, so all patches for
potential inclusion should be based off of that.

It would be handy to include vim: in the subject line, but it's not
absolutely crucial.

d

Re: Status of Vim client

[nlhowell]

David Bremner wrote:
> Nick Howell <nlhowell@gmail.com> writes:
> > Nick Howell wrote:
> >> I have a few patchsets adding:
> >
> >> - attachment support in compose - mailcap support for multipart
> >> messages
> >
> > Mildly clean (imo), independent versions of these have been sent
> > to the mailing list; they are meant to be applied on-top of
> > felipec's repository. (I was unaware of, and haven't looked at,
> > the imain repo.)
> 
> For whatever reason, I didn't see those messages on the (i.e. this)
> mailing list.

Strange. I will make ettiquette adjustments and re-send. Here's my
smtp log:
> Oct 10 17:32:12 host=smtp.gmail.com tls=on auth=on user=nlhowell from=nlhowell@gmail.com recipients=notmuch@notmuchmail.org,nlhowell@gmail.com mailsize=1023 smtpstatus=250 smtpmsg='250 2.0.0 OK 1476142322 m128sm382316ioa.41 - gsmtp' exitcode=EX_OK
> Oct 10 17:32:27 host=smtp.gmail.com tls=on auth=on user=nlhowell from=nlhowell@gmail.com recipients=notmuch@notmuchmail.org,nlhowell@gmail.com mailsize=4032 smtpstatus=250 smtpmsg='250 2.0.0 OK 1476142337 b133sm7677395iti.21 - gsmtp' exitcode=EX_OK
> Oct 10 17:32:43 host=smtp.gmail.com tls=on auth=on user=nlhowell from=nlhowell@gmail.com recipients=notmuch@notmuchmail.org,nlhowell@gmail.com mailsize=4085 smtpstatus=250 smtpmsg='250 2.0.0 OK 1476142353 n69sm7704802ita.0 - gsmtp' exitcode=EX_OK
> Oct 10 18:25:13 host=smtp.gmail.com tls=on auth=on user=nlhowell from=nlhowell@gmail.com recipients=notmuch@notmuchmail.org,nlhowell@gmail.com mailsize=1149 smtpstatus=250 smtpmsg='250 2.0.0 OK 1476145503 fi6sm530779pac.20 - gsmtp' exitcode=EX_OK
> Oct 10 18:25:18 host=smtp.gmail.com tls=on auth=on user=nlhowell from=nlhowell@gmail.com recipients=notmuch@notmuchmail.org,nlhowell@gmail.com mailsize=3608 smtpstatus=250 smtpmsg='250 2.0.0 OK 1476145508 n77sm507192pfi.82 - gsmtp' exitcode=EX_OK
> Oct 10 18:25:23 host=smtp.gmail.com tls=on auth=on user=nlhowell from=nlhowell@gmail.com recipients=notmuch@notmuchmail.org,nlhowell@gmail.com mailsize=1071 smtpstatus=250 smtpmsg='250 2.0.0 OK 1476145514 e1sm542594pap.11 - gsmtp' exitcode=EX_OK
> Oct 10 18:25:29 host=smtp.gmail.com tls=on auth=on user=nlhowell from=nlhowell@gmail.com recipients=notmuch@notmuchmail.org,nlhowell@gmail.com mailsize=1422 smtpstatus=250 smtpmsg='250 2.0.0 OK 1476145519 k67sm507280pfb.86 - gsmtp' exitcode=EX_OK
> Oct 10 18:25:34 host=smtp.gmail.com tls=on auth=on user=nlhowell from=nlhowell@gmail.com recipients=notmuch@notmuchmail.org,nlhowell@gmail.com mailsize=970 smtpstatus=250 smtpmsg='250 2.0.0 OK 1476145524 n2sm286444pfa.75 - gsmtp' exitcode=EX_OK
> Oct 10 18:25:39 host=smtp.gmail.com tls=on auth=on user=nlhowell from=nlhowell@gmail.com recipients=notmuch@notmuchmail.org,nlhowell@gmail.com mailsize=4136 smtpstatus=250 smtpmsg='250 2.0.0 OK 1476145530 j6sm497374paa.44 - gsmtp' exitcode=EX_OK
> Oct 10 18:25:44 host=smtp.gmail.com tls=on auth=on user=nlhowell from=nlhowell@gmail.com recipients=notmuch@notmuchmail.org,nlhowell@gmail.com mailsize=1653 smtpstatus=250 smtpmsg='250 2.0.0 OK 1476145535 g9sm503882paw.40 - gsmtp' exitcode=EX_OK
> Oct 10 18:40:12 host=smtp.gmail.com tls=on auth=on user=nlhowell from=nlhowell@gmail.com recipients=notmuch@notmuchmail.org,nlhowell@gmail.com,nlhowell@gmail.com mailsize=2058 smtpstatus=250 smtpmsg='250 2.0.0 OK 1476146402 n7sm558994pfn.62 - gsmtp' exitcode=EX_OK

I assumed they were getting moderated, but perhaps they got dropped?

> > p.s. please let me know if I broke some rule of patch mailing
> > ettiquette, e.g. should I have included "vim:" or based the
> > patches off of notmuch.git?
> 
> Yes, from our point of view notmuch.git is upstream, so all patches
> for potential inclusion should be based off of that.
> 
> It would be handy to include vim: in the subject line, but it's not
> absolutely crucial.

I'll make these two corrections.

Cheers,
Nick

[RFC PATCH 0/2] vim: add "attach" support to compose

[Nick Howell]

The following is a pair of patches implementing an "attach" action for
compose mode.

The first patch reorganizes the code for the magic "Notmuch-" headers
used in compose mode.

The second patch adds a magic Notmuch-Attach header, maps an input()
helper to add the header, and adds logic to process the header
on-send.

Nick Howell (2):
  vim: compose: support additional Notmuch- filters
  vim: compose: attachment support

 vim/notmuch.vim                | 67 ++++++++++++++++++++++++++++++++++++++----
 vim/syntax/notmuch-compose.vim |  5 ----
 2 files changed, 61 insertions(+), 11 deletions(-)

-- 
2.7.3

[RFC PATCH 1/2] vim: compose: support additional Notmuch- filters

[Nick Howell]

Add support for additional Notmuch-* headers: special headers read (and
then removed) by vnm upon send. The plan is to use these special headers
for all sorts of fancy features: attachments, PGP, whatever, so that the
inputs to these features are editable from vim.

For the moment, this just factors the code a bit: the Notmuch-Help lines
are expanded to list allowed headers, and the number of headers to
ignore is now decided at runtime, instead of being hardcoded.

The biggest change is the addition of dynamic highlighting for these: at
the end of the compose helper we add syntax highlighting for all the
special headers we interpret. Since these are now generated dynamically,
we remove the highlighting code from the static syntax files.
---
 vim/notmuch.vim                | 24 ++++++++++++++++++++----
 vim/syntax/notmuch-compose.vim |  5 -----
 2 files changed, 20 insertions(+), 9 deletions(-)

diff --git a/vim/notmuch.vim b/vim/notmuch.vim
index ad8b7c8..bea30f8 100644
--- a/vim/notmuch.vim
+++ b/vim/notmuch.vim
@@ -87,13 +87,19 @@ endfunction
 function! s:compose_send()
 	let b:compose_done = 1
 	let fname = expand('%')
-	let lines = getline(5, '$')
 
 ruby << EOF
 	# Generate proper mail to send
-	text = VIM::evaluate('lines').join("\n")
+	text = []
+	eval_nm_headers = true
+	VIM::evaluate('getline(0, \'$\')').each do |line|
+		unless eval_nm_headers and NotmuchHeaders.any? { |h| line.start_with?("Notmuch-#{h}: ") }
+			text << line
+			eval_nm_headers = false
+		end
+	end
 	fname = VIM::evaluate('fname')
-	transport = Mail.new(text)
+	transport = Mail.new(text.join("\n"))
 	transport.message_id = generate_message_id
 	transport.charset = 'utf-8'
 	File.write(fname, transport.to_s)
@@ -483,6 +489,8 @@ ruby << EOF
 	$messages = []
 	$mail_installed = defined?(Mail)
 
+	NotmuchHeaders = ["Help"]
+
 	def get_config_item(item)
 		result = ''
 		IO.popen(['notmuch', 'config', 'get', item]) { |out|
@@ -491,6 +499,7 @@ ruby << EOF
 		return result.rstrip
 	end
 
+
 	def get_config
 		$db_name = get_config_item('database.path')
 		$email_name = get_config_item('user.name')
@@ -547,8 +556,9 @@ ruby << EOF
 	def open_compose_helper(lines, cur)
 		help_lines = [
 			'Notmuch-Help: Type in your message here; to help you use these bindings:',
-			'Notmuch-Help:   ,s    - send the message (Notmuch-Help lines will be removed)',
+			'Notmuch-Help:   ,s    - send the message',
 			'Notmuch-Help:   ,q    - abort the message',
+			"Notmuch-Help: Magic Notmuch- headers (removed on send): #{NotmuchHeaders.join(", ")}.",
 			]
 
 		dir = File.expand_path('~/.notmuch/compose')
@@ -571,6 +581,12 @@ ruby << EOF
 			VIM::command("let s:reply_from='%s'" % $email_address)
 			VIM::command("call s:new_file_buffer('compose', '#{f.path}')")
 			VIM::command("call cursor(#{cur}, 0)")
+			VIM::command(<<-'SEOF' % NotmuchHeaders.collect { |h| "Notmuch-#{h}" }.join('\|'))
+				syntax region nmComposeNotmuchHeaders contains=nmComposeNotmuchHeaderLine start='^\(%1$s\):' end='^\(\(%1$s\):\)\@!'
+				syntax match nmComposeNotmuchHeaderLine /\(%1$s\):/ contained
+				highlight link nmComposeNotmuchHeaders Include
+				highlight link nmComposeNotmuchHeaderLine Error
+			SEOF
 		end
 	end
 
diff --git a/vim/syntax/notmuch-compose.vim b/vim/syntax/notmuch-compose.vim
index 19adb75..8d45636 100644
--- a/vim/syntax/notmuch-compose.vim
+++ b/vim/syntax/notmuch-compose.vim
@@ -1,7 +1,2 @@
 runtime! syntax/mail.vim
 
-syntax region nmComposeHelp          contains=nmComposeHelpLine start='^Notmuch-Help:\%1l' end='^\(Notmuch-Help:\)\@!'
-syntax match  nmComposeHelpLine      /Notmuch-Help:/ contained
-
-highlight link nmComposeHelp        Include
-highlight link nmComposeHelpLine    Error
-- 
2.7.3

[RFC PATCH 2/2] vim: compose: attachment support

[Nick Howell]

Adds attachment support through the mail gem. Notmuch-Attach headers
list files which are added to the message before sending.

Files are added by constructing a new post-processed transport Mail
object using the Notmuch-Attach header and the original Mail object. Any
attachments added (say by hand, or by some other method) to the
pre-transport message are preserved.

Adds a helper which prompts for an attachment (with filename completion)
and adds the header for you (bound to ,a).
---
 vim/notmuch.vim | 45 ++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 42 insertions(+), 3 deletions(-)

diff --git a/vim/notmuch.vim b/vim/notmuch.vim
index bea30f8..09685c3 100644
--- a/vim/notmuch.vim
+++ b/vim/notmuch.vim
@@ -45,6 +45,7 @@ let g:notmuch_show_maps = {
 
 let g:notmuch_compose_maps = {
 	\ ',s':		'compose_send()',
+	\ ',a':		'compose_attach()',
 	\ ',q':		'compose_quit()',
 	\ }
 
@@ -84,6 +85,20 @@ function! s:compose_quit()
 	call s:kill_this_buffer()
 endfunction
 
+function! s:compose_attach()
+ruby << EOF
+	raise "Attachment support unavailable." unless NotmuchHeaders.include?("Attach")
+EOF
+	let attachment = input('attach: ', '~/', 'file')
+ruby << EOF
+	r, c = $curwin.cursor
+	non_help_lines = (1..$curbuf.count).reject { |i| $curbuf[i].start_with?("Notmuch-Help: ") }
+	attachment = File.expand_path(VIM::evaluate('attachment'))
+	$curbuf.append(non_help_lines[0] - 1, "Notmuch-Attach: #{attachment}")
+	$curwin.cursor = [r + 1, c]
+EOF
+endfunction
+
 function! s:compose_send()
 	let b:compose_done = 1
 	let fname = expand('%')
@@ -91,18 +106,39 @@ function! s:compose_send()
 ruby << EOF
 	# Generate proper mail to send
 	text = []
+	attachments = []
 	eval_nm_headers = true
 	VIM::evaluate('getline(0, \'$\')').each do |line|
 		unless eval_nm_headers and NotmuchHeaders.any? { |h| line.start_with?("Notmuch-#{h}: ") }
 			text << line
 			eval_nm_headers = false
+		else
+			if line.start_with?("Notmuch-Attach: ")
+				attachments << line["Notmuch-Attach: ".length..-1]
+			end
 		end
 	end
 	fname = VIM::evaluate('fname')
 	transport = Mail.new(text.join("\n"))
 	transport.message_id = generate_message_id
 	transport.charset = 'utf-8'
-	File.write(fname, transport.to_s)
+
+	transport_fin = Mail.new do
+		transport.header_fields.reject do |f|
+			[
+				"Content-Type",
+				"Content-Transfer-Encoding",
+			].include?(f.name)
+		end.each do |f|
+			header[f.name] = f.value
+		end
+		transport.attachments.each { |a| add_file(:filename => a.filename, :mime_type => a.mime_type, :content => a.decoded) }
+		attachments.each { |a| add_file(a) }
+
+		body (transport.find_first_text || transport).body.to_s
+	end
+
+	File.write(fname, transport_fin.to_s)
 EOF
 
 	let cmdtxt = g:notmuch_sendmail . ' -t -f ' . s:reply_from . ' < ' . fname
@@ -489,7 +525,9 @@ ruby << EOF
 	$messages = []
 	$mail_installed = defined?(Mail)
 
-	NotmuchHeaders = ["Help"]
+	NotmuchHeaders = ["Help",
+		$mail_installed && "Attach",
+	].select { |s| s.is_a?(String) }
 
 	def get_config_item(item)
 		result = ''
@@ -557,9 +595,10 @@ ruby << EOF
 		help_lines = [
 			'Notmuch-Help: Type in your message here; to help you use these bindings:',
 			'Notmuch-Help:   ,s    - send the message',
+			NotmuchHeaders.include?("Attach") && 'Notmuch-Help:   ,a    - prompt for attachment',
 			'Notmuch-Help:   ,q    - abort the message',
 			"Notmuch-Help: Magic Notmuch- headers (removed on send): #{NotmuchHeaders.join(", ")}.",
-			]
+		].select { |s| s.is_a?(String) }
 
 		dir = File.expand_path('~/.notmuch/compose')
 		FileUtils.mkdir_p(dir)
-- 
2.7.3

[RFC PATCH 0/7] vim: add mailcap filtering to show

[Nick Howell]

This patch series adds the ability to display parts with non-plaintext
mime types. It uses the mailcap facility (see the commit message for
the first patch) to transform to plaintext. The latter patches add
special handling for multipart/alternative (instead of just treating
it as multipart/mixed).

Nick Howell (7):
  vim: show: add mailcap filtering
  vim: mailcap: forbid access to display server
  vim: mailcap: gracefully handle missing mimetype
  vim: mailcap: don't corrupt the view if filters write to stderr
  vim: mailcap: redraw after processing
  vim: mailcap: show all alternatives if none preferred
  vim: mailcap: handle mailcap failure gracefully

 vim/notmuch.vim | 63 +++++++++++++++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 59 insertions(+), 4 deletions(-)

-- 
2.7.3

[RFC PATCH 1/7] vim: show: add mailcap filtering

[Nick Howell]

Add mailcap filters, which allow us to display multiple parts.

See http://packages.debian.org/unstable/net/mime-support for details on
run-mailcap, the mailcap program we default to; the TLDR is that mailcap
is a program which accepts a mimetype argument, looks up a translation
program based on this, and hooks up stdin/stdout to this program.

This is useful for inline display of html, pdfs, docx, images (through
caca), etc, as well as external display (by launching an image viewer).
The implementation here linearizes the part tree of the message
(treating all multipart pieces as multipart/mixed) and then popens a
mailcap process for each. If mailcap fails for some reason (e.g. no
display rule), the contents are listed as "unavailable."

A sample $HOME/.mailcap configuration file:

application/pdf; pdftotext -layout '%s' - | fmt; copiousoutput;
text/html; vilistextum -u '%s' -w 72 -; copiousoutput;
text/plain; fmt -s

Some suggested converters:
- antiword, antixls, vilistextum, docx2txt
---
 vim/notmuch.vim | 34 ++++++++++++++++++++++++++++++----
 1 file changed, 30 insertions(+), 4 deletions(-)

diff --git a/vim/notmuch.vim b/vim/notmuch.vim
index ad8b7c8..c66c874 100644
--- a/vim/notmuch.vim
+++ b/vim/notmuch.vim
@@ -60,6 +60,7 @@ let s:notmuch_reader_default = 'mutt -f %s'
 let s:notmuch_sendmail_default = 'sendmail'
 let s:notmuch_folders_count_threads_default = 0
 let s:notmuch_compose_start_insert_default = 1
+let s:notmuch_mailcap_filter_default = "run-mailcap --action=view %s:-"
 
 function! s:new_file_buffer(type, fname)
 	exec printf('edit %s', a:fname)
@@ -320,9 +321,9 @@ ruby << EOF
 		msgs = q.search_messages
 		msgs.each do |msg|
 			m = Mail.read(msg.filename)
-			part = m.find_first_text
 			nm_m = Message.new(msg, m)
 			$messages << nm_m
+			mailcap = VIM::evaluate('g:notmuch_mailcap_filter')
 			date_fmt = VIM::evaluate('g:notmuch_datetime_format')
 			date = Time.at(msg.date).strftime(date_fmt)
 			nm_m.start = b.count
@@ -332,9 +333,31 @@ ruby << EOF
 			b << "Cc: %s" % msg['cc']
 			b << "Date: %s" % msg['date']
 			nm_m.body_start = b.count
-			b << "--- %s ---" % part.mime_type
-			part.convert.each_line do |l|
-				b << l.chomp
+			parts = [m]
+			while parts.any?(&:multipart?)
+				parts = parts.collect do |part|
+					if part.multipart?
+						part.parts
+					else
+						part
+					end
+				end.flatten
+			end
+
+			parts.each do |part|
+				b << "--- %s ---" % part.mime_type
+				begin
+					IO.popen(mailcap % part.mime_type, "r+b") do |io|
+						io.write(part.decoded)
+						io.close_write
+						io.each_line do |l|
+							b << l.chomp
+						end
+					end
+				rescue
+					b << "--- unavailable ---"
+				end
+				b << ""
 			end
 			b << ""
 			nm_m.end = b.count
@@ -393,6 +416,9 @@ endfunction
 "" root
 
 function! s:set_defaults()
+	if !exists('g:notmuch_mailcap_filter')
+		let g:notmuch_mailcap_filter = s:notmuch_mailcap_filter_default
+	endif
 	if !exists('g:notmuch_date_format')
 		if exists('g:notmuch_rb_date_format')
 			let g:notmuch_date_format = g:notmuch_rb_date_format
-- 
2.7.3

[RFC PATCH 2/7] vim: mailcap: forbid access to display server

[Nick Howell]

Default to prevent mailcap from accessing the display server. Potential
concerns: a malicious .mailcap file + attachment could read your
keystrokes, or if the .mailcap file is non-malicious, an attachment
exploiting a vulnerability in your mailcap viewer.

In principle we should probably try to run mailcap with super-low
privileges, but I haven't explored this.
---
 vim/notmuch.vim | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/vim/notmuch.vim b/vim/notmuch.vim
index c66c874..947fec0 100644
--- a/vim/notmuch.vim
+++ b/vim/notmuch.vim
@@ -60,7 +60,7 @@ let s:notmuch_reader_default = 'mutt -f %s'
 let s:notmuch_sendmail_default = 'sendmail'
 let s:notmuch_folders_count_threads_default = 0
 let s:notmuch_compose_start_insert_default = 1
-let s:notmuch_mailcap_filter_default = "run-mailcap --action=view %s:-"
+let s:notmuch_mailcap_filter_default = "DISPLAY= run-mailcap --action=view %s:-"
 
 function! s:new_file_buffer(type, fname)
 	exec printf('edit %s', a:fname)
-- 
2.7.3

[RFC PATCH 3/7] vim: mailcap: gracefully handle missing mimetype

[Nick Howell]

Some plaintext messages do not specify a mimetype. Instead of failing in
this situation, just assume the part is plaintext. (Maybe a better
strategy would be to check if the message contains non-unicode first?)
---
 vim/notmuch.vim | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/vim/notmuch.vim b/vim/notmuch.vim
index 947fec0..89fa2dc 100644
--- a/vim/notmuch.vim
+++ b/vim/notmuch.vim
@@ -345,9 +345,9 @@ ruby << EOF
 			end
 
 			parts.each do |part|
-				b << "--- %s ---" % part.mime_type
+				b << "--- %s ---" % ( [part.mime_type || "unknown", part.filename].find_all { |x| x }.join(" ") )
 				begin
-					IO.popen(mailcap % part.mime_type, "r+b") do |io|
+					IO.popen(mailcap % (part.mime_type || "text/plain"), "r+b") do |io|
 						io.write(part.decoded)
 						io.close_write
 						io.each_line do |l|
-- 
2.7.3

[RFC PATCH 4/7] vim: mailcap: don't corrupt the view if filters write to stderr

[Nick Howell]

---
 vim/notmuch.vim | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/vim/notmuch.vim b/vim/notmuch.vim
index 89fa2dc..a20dd86 100644
--- a/vim/notmuch.vim
+++ b/vim/notmuch.vim
@@ -347,7 +347,7 @@ ruby << EOF
 			parts.each do |part|
 				b << "--- %s ---" % ( [part.mime_type || "unknown", part.filename].find_all { |x| x }.join(" ") )
 				begin
-					IO.popen(mailcap % (part.mime_type || "text/plain"), "r+b") do |io|
+					IO.popen("%s 2>/dev/null" % (mailcap % (part.mime_type || "text/plain")), "r+b") do |io|
 						io.write(part.decoded)
 						io.close_write
 						io.each_line do |l|
-- 
2.7.3

[RFC PATCH 5/7] vim: mailcap: redraw after processing

[Nick Howell]

---
 vim/notmuch.vim | 1 +
 1 file changed, 1 insertion(+)

diff --git a/vim/notmuch.vim b/vim/notmuch.vim
index a20dd86..5cb8d20 100644
--- a/vim/notmuch.vim
+++ b/vim/notmuch.vim
@@ -370,6 +370,7 @@ ruby << EOF
 		VIM::command("syntax region nmShowMsg#{i}Body start='\\%%%il' end='\\%%%dl' contains=@nmShowMsgBody" % [msg.body_start, msg.end])
 	end
 EOF
+	normal \f
 	setlocal nomodifiable
 	call s:set_map(g:notmuch_show_maps)
 endfunction
-- 
2.7.3

[RFC PATCH 6/7] vim: mailcap: show all alternatives if none preferred

[Nick Howell]

Display all parts of a multipart/alternative message (or message-part)
if no alternative_preference matches any of the alternative mimetypes.

E.g. if a message is multipart/alternative with options text/plain and
text/html, and the alternative_preference is ['application/pdf'], no
preference option matches any of the parts. This patch would just
display both parts.

Better configuration is to have '*/*' as an option in your
alternative_preference, e.g. ['text/plain', 'text/*', '*/*'], so that
there is always a catch-all.
---
 vim/notmuch.vim | 19 ++++++++++++++++++-
 1 file changed, 18 insertions(+), 1 deletion(-)

diff --git a/vim/notmuch.vim b/vim/notmuch.vim
index 5cb8d20..da747eb 100644
--- a/vim/notmuch.vim
+++ b/vim/notmuch.vim
@@ -337,7 +337,24 @@ ruby << EOF
 			while parts.any?(&:multipart?)
 				parts = parts.collect do |part|
 					if part.multipart?
-						part.parts
+						if part.mime_type == "multipart/alternative"
+							case alternative_display
+							when "best-only"
+								alternative_preference.each.collect do |mime_type|
+									part.parts.find { |p| File.fnmatch(mime_type, p.mime_type) }
+								end.reject(&:nil?).first || part.parts
+							when "best-sort"
+								part.parts.sort_by do |p|
+									alternative_preference.collect.with_index do |mime_type, i|
+										(File.fnmatch(mime_type, p.mime_type) ? 0 : 1) << i
+									end.reduce(:+)
+								end
+							when "verbatim"
+								part.parts
+							end
+						else
+							part.parts
+						end
 					else
 						part
 					end
-- 
2.7.3

[RFC PATCH 7/7] vim: mailcap: handle mailcap failure gracefully

[Nick Howell]

---
 vim/notmuch.vim | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/vim/notmuch.vim b/vim/notmuch.vim
index da747eb..e9baeff 100644
--- a/vim/notmuch.vim
+++ b/vim/notmuch.vim
@@ -370,6 +370,14 @@ ruby << EOF
 						io.each_line do |l|
 							b << l.chomp
 						end
+						if $? != 0
+							b << "--- ERROR: mailcap failed ---"
+							part.decoded.each_line do |l|
+								b << l.chomp
+							end
+						end
+					rescue
+						b << "--- unavailable ---"
 					end
 				rescue
 					b << "--- unavailable ---"
-- 
2.7.3

[RFC PATCHv2 0/8] vim: add mailcap filtering to show

[Nick Howell]

This patch series adds the ability to display parts with non-plaintext
mime types. It uses the mailcap facility (see the commit message for
the first patch) to transform to plaintext. The latter patches add
special handling for multipart/alternative (instead of just treating
it as multipart/mixed).

Probably we should choose a more secure default, e.g. whitelisting
environment variables. It would be nice to recommend the use of a
sandbox.

Changes v1..v2:
- add missing patch "add multipart/alternative picking"
- correct poor conflict resolution in "handle mailcap failure
  gracefully"

Nick Howell (8):
  vim: show: add mailcap filtering
  vim: mailcap: forbid access to display server
  vim: mailcap: gracefully handle missing mimetype
  vim: mailcap: don't corrupt the view if filters write to stderr
  vim: mailcap: redraw after processing
  vim: show: add multipart/alternative picking
  vim: mailcap: show all alternatives if none preferred
  vim: mailcap: handle mailcap failure gracefully

 vim/notmuch.vim | 70 +++++++++++++++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 66 insertions(+), 4 deletions(-)

-- 
2.7.3

[RFC PATCHv2 1/8] vim: show: add mailcap filtering

[Nick Howell]

Add mailcap filters, which allow us to display multiple parts.

See http://packages.debian.org/unstable/net/mime-support for details on
run-mailcap, the mailcap program we default to; the TLDR is that mailcap
is a program which accepts a mimetype argument, looks up a translation
program based on this, and hooks up stdin/stdout to this program.

This is useful for inline display of html, pdfs, docx, images (through
caca), etc, as well as external display (by launching an image viewer).
The implementation here linearizes the part tree of the message
(treating all multipart pieces as multipart/mixed) and then popens a
mailcap process for each. If mailcap fails for some reason (e.g. no
display rule), the contents are listed as "unavailable."

A sample $HOME/.mailcap configuration file:

application/pdf; pdftotext -layout '%s' - | fmt; copiousoutput;
text/html; vilistextum -u '%s' -w 72 -; copiousoutput;
text/plain; fmt -s

Some suggested converters:
- antiword, antixls, vilistextum, docx2txt
---
 vim/notmuch.vim | 34 ++++++++++++++++++++++++++++++----
 1 file changed, 30 insertions(+), 4 deletions(-)

diff --git a/vim/notmuch.vim b/vim/notmuch.vim
index ad8b7c8..c66c874 100644
--- a/vim/notmuch.vim
+++ b/vim/notmuch.vim
@@ -60,6 +60,7 @@ let s:notmuch_reader_default = 'mutt -f %s'
 let s:notmuch_sendmail_default = 'sendmail'
 let s:notmuch_folders_count_threads_default = 0
 let s:notmuch_compose_start_insert_default = 1
+let s:notmuch_mailcap_filter_default = "run-mailcap --action=view %s:-"
 
 function! s:new_file_buffer(type, fname)
 	exec printf('edit %s', a:fname)
@@ -320,9 +321,9 @@ ruby << EOF
 		msgs = q.search_messages
 		msgs.each do |msg|
 			m = Mail.read(msg.filename)
-			part = m.find_first_text
 			nm_m = Message.new(msg, m)
 			$messages << nm_m
+			mailcap = VIM::evaluate('g:notmuch_mailcap_filter')
 			date_fmt = VIM::evaluate('g:notmuch_datetime_format')
 			date = Time.at(msg.date).strftime(date_fmt)
 			nm_m.start = b.count
@@ -332,9 +333,31 @@ ruby << EOF
 			b << "Cc: %s" % msg['cc']
 			b << "Date: %s" % msg['date']
 			nm_m.body_start = b.count
-			b << "--- %s ---" % part.mime_type
-			part.convert.each_line do |l|
-				b << l.chomp
+			parts = [m]
+			while parts.any?(&:multipart?)
+				parts = parts.collect do |part|
+					if part.multipart?
+						part.parts
+					else
+						part
+					end
+				end.flatten
+			end
+
+			parts.each do |part|
+				b << "--- %s ---" % part.mime_type
+				begin
+					IO.popen(mailcap % part.mime_type, "r+b") do |io|
+						io.write(part.decoded)
+						io.close_write
+						io.each_line do |l|
+							b << l.chomp
+						end
+					end
+				rescue
+					b << "--- unavailable ---"
+				end
+				b << ""
 			end
 			b << ""
 			nm_m.end = b.count
@@ -393,6 +416,9 @@ endfunction
 "" root
 
 function! s:set_defaults()
+	if !exists('g:notmuch_mailcap_filter')
+		let g:notmuch_mailcap_filter = s:notmuch_mailcap_filter_default
+	endif
 	if !exists('g:notmuch_date_format')
 		if exists('g:notmuch_rb_date_format')
 			let g:notmuch_date_format = g:notmuch_rb_date_format
-- 
2.7.3

[RFC PATCHv2 2/8] vim: mailcap: forbid access to display server

[Nick Howell]

Default to prevent mailcap from accessing the display server. Potential
concerns: a malicious .mailcap file + attachment could read your
keystrokes, or if the .mailcap file is non-malicious, an attachment
exploiting a vulnerability in your mailcap viewer.

In principle we should probably try to run mailcap with super-low
privileges, but I haven't explored this.
---
 vim/notmuch.vim | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/vim/notmuch.vim b/vim/notmuch.vim
index c66c874..947fec0 100644
--- a/vim/notmuch.vim
+++ b/vim/notmuch.vim
@@ -60,7 +60,7 @@ let s:notmuch_reader_default = 'mutt -f %s'
 let s:notmuch_sendmail_default = 'sendmail'
 let s:notmuch_folders_count_threads_default = 0
 let s:notmuch_compose_start_insert_default = 1
-let s:notmuch_mailcap_filter_default = "run-mailcap --action=view %s:-"
+let s:notmuch_mailcap_filter_default = "DISPLAY= run-mailcap --action=view %s:-"
 
 function! s:new_file_buffer(type, fname)
 	exec printf('edit %s', a:fname)
-- 
2.7.3

[RFC PATCHv2 3/8] vim: mailcap: gracefully handle missing mimetype

[Nick Howell]

Some plaintext messages do not specify a mimetype. Instead of failing in
this situation, just assume the part is plaintext. (Maybe a better
strategy would be to check if the message contains non-unicode first?)
---
 vim/notmuch.vim | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/vim/notmuch.vim b/vim/notmuch.vim
index 947fec0..89fa2dc 100644
--- a/vim/notmuch.vim
+++ b/vim/notmuch.vim
@@ -345,9 +345,9 @@ ruby << EOF
 			end
 
 			parts.each do |part|
-				b << "--- %s ---" % part.mime_type
+				b << "--- %s ---" % ( [part.mime_type || "unknown", part.filename].find_all { |x| x }.join(" ") )
 				begin
-					IO.popen(mailcap % part.mime_type, "r+b") do |io|
+					IO.popen(mailcap % (part.mime_type || "text/plain"), "r+b") do |io|
 						io.write(part.decoded)
 						io.close_write
 						io.each_line do |l|
-- 
2.7.3

[RFC PATCHv2 4/8] vim: mailcap: don't corrupt the view if filters write to stderr

[Nick Howell]

---
 vim/notmuch.vim | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/vim/notmuch.vim b/vim/notmuch.vim
index 89fa2dc..a20dd86 100644
--- a/vim/notmuch.vim
+++ b/vim/notmuch.vim
@@ -347,7 +347,7 @@ ruby << EOF
 			parts.each do |part|
 				b << "--- %s ---" % ( [part.mime_type || "unknown", part.filename].find_all { |x| x }.join(" ") )
 				begin
-					IO.popen(mailcap % (part.mime_type || "text/plain"), "r+b") do |io|
+					IO.popen("%s 2>/dev/null" % (mailcap % (part.mime_type || "text/plain")), "r+b") do |io|
 						io.write(part.decoded)
 						io.close_write
 						io.each_line do |l|
-- 
2.7.3

[RFC PATCHv2 5/8] vim: mailcap: redraw after processing

[Nick Howell]

---
 vim/notmuch.vim | 1 +
 1 file changed, 1 insertion(+)

diff --git a/vim/notmuch.vim b/vim/notmuch.vim
index a20dd86..5cb8d20 100644
--- a/vim/notmuch.vim
+++ b/vim/notmuch.vim
@@ -370,6 +370,7 @@ ruby << EOF
 		VIM::command("syntax region nmShowMsg#{i}Body start='\\%%%il' end='\\%%%dl' contains=@nmShowMsgBody" % [msg.body_start, msg.end])
 	end
 EOF
+	normal \f
 	setlocal nomodifiable
 	call s:set_map(g:notmuch_show_maps)
 endfunction
-- 
2.7.3

[RFC PATCHv2 6/8] vim: show: add multipart/alternative picking

[Nick Howell]

Instead of just blindly showing every part of multipart/*, add special
handling for multipart/alternative (which is designed for alternative
content-types containing the same message). Two new preferences are
added:

- g:notmuch_multipart_alternative_display
        = "best-only" (default)
        Display only the favorite content-type.

        = "best-sort"
        Display all content-types, sorted by preference.

        = "verbatim"
        Display all content-types in the order they appear.

- g:notmuch_multipart_alternative_preference
        Array which orders mimetypes based on preference; note that if a
        part has a mimetype which matches none of these, the part will
        not be displayed!

        e.g. ['text/plain', 'text/*', '*'] (default)

The RFC describing multipart/alternative has some subtleties regarding
ways the sender can influence which content-type is shown; we mostly
ignore these, and give the user the choice.

Note that the defaults break some mail programs (**cough apple**) which,
when attaching images, prefer to embed the image in text/html, but then
list a multipart/alternative of text/plain without attaching the image.
---
 vim/notmuch.vim | 31 ++++++++++++++++++++++++++++++-
 1 file changed, 30 insertions(+), 1 deletion(-)

diff --git a/vim/notmuch.vim b/vim/notmuch.vim
index 5cb8d20..795ab26 100644
--- a/vim/notmuch.vim
+++ b/vim/notmuch.vim
@@ -61,6 +61,8 @@ let s:notmuch_sendmail_default = 'sendmail'
 let s:notmuch_folders_count_threads_default = 0
 let s:notmuch_compose_start_insert_default = 1
 let s:notmuch_mailcap_filter_default = "DISPLAY= run-mailcap --action=view %s:-"
+let s:notmuch_multipart_alternative_display_default = "best-only"
+let s:notmuch_multipart_alternative_preference_default = ['text/plain', 'text/*', '*']
 
 function! s:new_file_buffer(type, fname)
 	exec printf('edit %s', a:fname)
@@ -324,6 +326,8 @@ ruby << EOF
 			nm_m = Message.new(msg, m)
 			$messages << nm_m
 			mailcap = VIM::evaluate('g:notmuch_mailcap_filter')
+			alternative_preference = VIM::evaluate('g:notmuch_multipart_alternative_preference')
+			alternative_display = VIM::evaluate('g:notmuch_multipart_alternative_display')
 			date_fmt = VIM::evaluate('g:notmuch_datetime_format')
 			date = Time.at(msg.date).strftime(date_fmt)
 			nm_m.start = b.count
@@ -337,7 +341,24 @@ ruby << EOF
 			while parts.any?(&:multipart?)
 				parts = parts.collect do |part|
 					if part.multipart?
-						part.parts
+						if part.mime_type == "multipart/alternative"
+							case alternative_display
+							when "best-only"
+								alternative_preference.each.collect do |mime_type|
+									part.parts.find { |p| File.fnmatch(mime_type, p.mime_type) }
+								end.first
+							when "best-sort"
+								part.parts.sort_by do |p|
+									alternative_preference.collect.with_index do |mime_type, i|
+										(File.fnmatch(mime_type, p.mime_type) ? 0 : 1) << i
+									end.reduce(:+)
+								end
+							when "verbatim"
+								part.parts
+							end
+						else
+							part.parts
+						end
 					else
 						part
 					end
@@ -417,6 +438,14 @@ endfunction
 "" root
 
 function! s:set_defaults()
+	if !exists('g:notmuch_multipart_alternative_display')
+		let g:notmuch_multipart_alternative_display = s:notmuch_multipart_alternative_display_default
+	endif
+
+	if !exists('g:notmuch_multipart_alternative_preference')
+		let g:notmuch_multipart_alternative_preference = s:notmuch_multipart_alternative_preference_default
+	endif
+
 	if !exists('g:notmuch_mailcap_filter')
 		let g:notmuch_mailcap_filter = s:notmuch_mailcap_filter_default
 	endif
-- 
2.7.3

[RFC PATCHv2 7/8] vim: mailcap: show all alternatives if none preferred

[Nick Howell]

Display all parts of a multipart/alternative message (or message-part)
if no alternative_preference matches any of the alternative mimetypes.

E.g. if a message is multipart/alternative with options text/plain and
text/html, and the alternative_preference is ['application/pdf'], no
preference option matches any of the parts. This patch would just
display both parts.

Better configuration is to have '*/*' as an option in your
alternative_preference, e.g. ['text/plain', 'text/*', '*/*'], so that
there is always a catch-all.
---
 vim/notmuch.vim | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/vim/notmuch.vim b/vim/notmuch.vim
index 795ab26..cdcaa5f 100644
--- a/vim/notmuch.vim
+++ b/vim/notmuch.vim
@@ -346,7 +346,7 @@ ruby << EOF
 							when "best-only"
 								alternative_preference.each.collect do |mime_type|
 									part.parts.find { |p| File.fnmatch(mime_type, p.mime_type) }
-								end.first
+								end.reject(&:nil?).first || part.parts
 							when "best-sort"
 								part.parts.sort_by do |p|
 									alternative_preference.collect.with_index do |mime_type, i|
-- 
2.7.3

[RFC PATCHv2 8/8] vim: mailcap: handle mailcap failure gracefully

[Nick Howell]

---
 vim/notmuch.vim | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/vim/notmuch.vim b/vim/notmuch.vim
index cdcaa5f..79a0e27 100644
--- a/vim/notmuch.vim
+++ b/vim/notmuch.vim
@@ -374,6 +374,12 @@ ruby << EOF
 						io.each_line do |l|
 							b << l.chomp
 						end
+						if $? != 0
+							b << "--- ERROR: mailcap failed ---"
+							part.decoded.each_line do |l|
+								b << l.chomp
+							end
+						end
 					end
 				rescue
 					b << "--- unavailable ---"
-- 
2.7.3

Re: Status of Vim client

[David Bremner]

nlhowell@gmail.com writes:

> I assumed they were getting moderated, but perhaps they got dropped?

Tomi can you check that possibility? It seems unlikely, given the other
messages are getting through, but who knows what evil lurks in the heart
of Mailman.

d