* clean up for nmweb/btsmail
@ 2018-02-05 4:19 Daniel Kahn Gillmor
2018-02-05 4:19 ` [PATCH 01/11] nmweb: (pylint) use __future__.absolute_import Daniel Kahn Gillmor
` (11 more replies)
0 siblings, 12 replies; 15+ messages in thread
From: Daniel Kahn Gillmor @ 2018-02-05 4:19 UTC (permalink / raw)
To: Notmuch Mail
The patches that follow are some proposed cleanups for
contrib/notmuch-web (aka nmweb, aka btsmail), which was originally
proposed by Brian Sniffen.
These are pretty mechanical changes -- nothing fancy added or removed,
just trying to make the code a little more maintainable and moving it
to python3.
You can also find this on the "nmweb" branch at
https://gitlab.com/dkg/notmuch
Regards,
--dkg
^ permalink raw reply [flat|nested] 15+ messages in thread* [PATCH 01/11] nmweb: (pylint) use __future__.absolute_import 2018-02-05 4:19 clean up for nmweb/btsmail Daniel Kahn Gillmor @ 2018-02-05 4:19 ` Daniel Kahn Gillmor 2018-02-05 4:19 ` [PATCH 02/11] nmweb: (pylint) use replace() function directly on string object Daniel Kahn Gillmor ` (10 subsequent siblings) 11 siblings, 0 replies; 15+ messages in thread From: Daniel Kahn Gillmor @ 2018-02-05 4:19 UTC (permalink / raw) To: Notmuch Mail pylint --py3k suggests always including this in code that might run in either python2 or python3. it selects the python3 default. (see "pylint --help-msg=no-absolute-import" for more details) --- contrib/notmuch-web/nmweb.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/contrib/notmuch-web/nmweb.py b/contrib/notmuch-web/nmweb.py index add0d631..c6d437c5 100755 --- a/contrib/notmuch-web/nmweb.py +++ b/contrib/notmuch-web/nmweb.py @@ -1,5 +1,7 @@ #!/usr/bin/env python +from __future__ import absolute_import + import web from notmuch import Database,Query,Message import urllib -- 2.15.1 ^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH 02/11] nmweb: (pylint) use replace() function directly on string object 2018-02-05 4:19 clean up for nmweb/btsmail Daniel Kahn Gillmor 2018-02-05 4:19 ` [PATCH 01/11] nmweb: (pylint) use __future__.absolute_import Daniel Kahn Gillmor @ 2018-02-05 4:19 ` Daniel Kahn Gillmor 2018-02-05 4:19 ` [PATCH 03/11] nmweb: (pylint) use spaces instead of tabs Daniel Kahn Gillmor ` (9 subsequent siblings) 11 siblings, 0 replies; 15+ messages in thread From: Daniel Kahn Gillmor @ 2018-02-05 4:19 UTC (permalink / raw) To: Notmuch Mail Use of string.replace() is deprecated, better to invoke replace() directly on the string object. --- contrib/notmuch-web/nmweb.py | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/contrib/notmuch-web/nmweb.py b/contrib/notmuch-web/nmweb.py index c6d437c5..1f6e0145 100755 --- a/contrib/notmuch-web/nmweb.py +++ b/contrib/notmuch-web/nmweb.py @@ -10,7 +10,6 @@ from mailbox import MaildirMessage import os import mimetypes import email -import string import re from jinja2 import Environment, FileSystemLoader # FIXME to PackageLoader from jinja2 import Markup @@ -166,8 +165,7 @@ def format_message_walk(msg,mid): yield '<ul>' for subpart in part.get_payload(): yield ('<li><a href="#%s">%s</a></li>' % - (string.replace(subpart.get_content_type(), - '/', '-'), + (subpart.get_content_type().replace('/', '-'), subpart.get_content_type())) yield '</ul>' elif part.get_content_type() == 'message/rfc822': @@ -191,7 +189,7 @@ def format_message_walk(msg,mid): yield '<iframe class="embedded-html" src="%s">' % os.path.join(prefix,cachedir,mid,filename) yield '</div>' else: - yield '<div id="%s">' % string.replace(part.get_content_type(),'/','-') + yield '<div id="%s">' % (part.get_content_type().replace('/','-'),) (filename,cid) = link_to_cached_file(part,mid,counter) counter += 1 yield '<a href="%s">%s (%s)</a>' % (os.path.join(prefix, @@ -222,7 +220,7 @@ def format_message_walk(msg,mid): env.globals['format_message'] = format_message def replace_cids(body,mid): - return string.replace(body,'cid:',os.path.join(prefix,cachedir,mid)+'/') + return body.replace('cid:',os.path.join(prefix,cachedir,mid)+'/') def find_cids(body): return re.findall(r'cid:([^ "\'>]*)', body) -- 2.15.1 ^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH 03/11] nmweb: (pylint) use spaces instead of tabs 2018-02-05 4:19 clean up for nmweb/btsmail Daniel Kahn Gillmor 2018-02-05 4:19 ` [PATCH 01/11] nmweb: (pylint) use __future__.absolute_import Daniel Kahn Gillmor 2018-02-05 4:19 ` [PATCH 02/11] nmweb: (pylint) use replace() function directly on string object Daniel Kahn Gillmor @ 2018-02-05 4:19 ` Daniel Kahn Gillmor 2018-02-05 4:19 ` [PATCH 04/11] nmweb: (pylint) normalize whitespace Daniel Kahn Gillmor ` (8 subsequent siblings) 11 siblings, 0 replies; 15+ messages in thread From: Daniel Kahn Gillmor @ 2018-02-05 4:19 UTC (permalink / raw) To: Notmuch Mail --- contrib/notmuch-web/nmweb.py | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/contrib/notmuch-web/nmweb.py b/contrib/notmuch-web/nmweb.py index 1f6e0145..b56c6b12 100755 --- a/contrib/notmuch-web/nmweb.py +++ b/contrib/notmuch-web/nmweb.py @@ -47,7 +47,7 @@ class index: tags = db.get_all_tags() return template.render(tags=tags, title="Notmuch webmail", - prefix=prefix, + prefix=prefix, sprefix=webprefix) class search: @@ -80,7 +80,7 @@ class search: ts=ts, title=terms, prefix=prefix, - sprefix=webprefix) + sprefix=webprefix) def format_time_range(start,end): if end-start < (60*60*24): @@ -137,7 +137,7 @@ class show: mid=mid, title=m.get_header('Subject'), prefix=prefix, - sprefix=webprefix) + sprefix=webprefix) def format_message(fn,mid): msg = MaildirMessage(open(fn)) @@ -184,9 +184,9 @@ def format_message_walk(msg,mid): decoded = decodeAnyway(unb64,part.get_content_charset('ascii')) cid_refd += find_cids(decoded) part.set_payload(replace_cids(decoded,mid).encode(part.get_content_charset('ascii'))) - (filename,cid) = link_to_cached_file(part,mid,counter) - counter +=1 - yield '<iframe class="embedded-html" src="%s">' % os.path.join(prefix,cachedir,mid,filename) + (filename,cid) = link_to_cached_file(part,mid,counter) + counter +=1 + yield '<iframe class="embedded-html" src="%s">' % os.path.join(prefix,cachedir,mid,filename) yield '</div>' else: yield '<div id="%s">' % (part.get_content_type().replace('/','-'),) -- 2.15.1 ^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH 04/11] nmweb: (pylint) normalize whitespace 2018-02-05 4:19 clean up for nmweb/btsmail Daniel Kahn Gillmor ` (2 preceding siblings ...) 2018-02-05 4:19 ` [PATCH 03/11] nmweb: (pylint) use spaces instead of tabs Daniel Kahn Gillmor @ 2018-02-05 4:19 ` Daniel Kahn Gillmor 2018-03-17 12:37 ` David Bremner 2018-02-05 4:19 ` [PATCH 05/11] nmweb: (pylint) import built-in modules before third-party modules Daniel Kahn Gillmor ` (7 subsequent siblings) 11 siblings, 1 reply; 15+ messages in thread From: Daniel Kahn Gillmor @ 2018-02-05 4:19 UTC (permalink / raw) To: Notmuch Mail I've settled on: * 2-space indent * one space after comma * one space on either side of assignment and comparison operators * no trailing whitespace This now satisfies the indentation/whitespace checks of: pylint --indent-string=' ' --indent-after-paren=2 nmweb.py --- contrib/notmuch-web/nmweb.py | 248 ++++++++++++++++++++++--------------------- 1 file changed, 125 insertions(+), 123 deletions(-) diff --git a/contrib/notmuch-web/nmweb.py b/contrib/notmuch-web/nmweb.py index b56c6b12..bee4896e 100755 --- a/contrib/notmuch-web/nmweb.py +++ b/contrib/notmuch-web/nmweb.py @@ -3,7 +3,7 @@ from __future__ import absolute_import import web -from notmuch import Database,Query,Message +from notmuch import Database, Query, Message import urllib from datetime import datetime from mailbox import MaildirMessage @@ -27,14 +27,14 @@ urls = ( '/', 'index', '/search/(.*)', 'search', '/show/(.*)', 'show', - ) +) def urlencode_filter(s): - if type(s) == 'Markup': - s = s.unescape() - s = s.encode('utf8') - s = urllib.quote_plus(s) - return Markup(s) + if type(s) == 'Markup': + s = s.unescape() + s = s.encode('utf8') + s = urllib.quote_plus(s) + return Markup(s) env.filters['url'] = urlencode_filter class index: @@ -51,7 +51,7 @@ class index: sprefix=webprefix) class search: - def GET(self,terms): + def GET(self, terms): redir = False if web.input(terms=None).terms: redir = True @@ -59,20 +59,20 @@ class search: if web.input(afters=None).afters: afters = web.input(afters=None).afters[:-3] else: - afters='0' + afters = '0' if web.input(befores=None).befores: befores = web.input(befores=None).befores else: befores = '4294967296' # 2^32 if int(afters) > 0 or int(befores) < 4294967296: redir = True - terms += ' %s..%s' % (afters,befores) + terms += ' %s..%s' % (afters, befores) if redir: raise web.seeother('/search/%s' % urllib.quote_plus(terms)) web.header('Content-type', 'text/html') web.header('Transfer-Encoding', 'chunked') db = Database() - q = Query(db,terms) + q = Query(db, terms) q.set_sort(Query.SORT.NEWEST_FIRST) ts = q.search_threads() template = env.get_template('search.html') @@ -82,19 +82,19 @@ class search: prefix=prefix, sprefix=webprefix) -def format_time_range(start,end): +def format_time_range(start, end): if end-start < (60*60*24): time = datetime.fromtimestamp(start).strftime('%Y %b %d %H:%M') else: start = datetime.fromtimestamp(start).strftime("%Y %b %d") end = datetime.fromtimestamp(end).strftime("%Y %b %d") - time = "%s through %s" % (start,end) + time = "%s through %s" % (start, end) return time env.globals['format_time_range'] = format_time_range def mailto_addrs(frm): - frm = email.utils.getaddresses([frm]) - return ','.join(['<a href="mailto:%s">%s</a> ' % ((l,p) if p else (l,l)) for (p,l) in frm]) + frm = email.utils.getaddresses([frm]) + return ','.join(['<a href="mailto:%s">%s</a> ' % ((l, p) if p else (l, l)) for (p, l) in frm]) env.globals['mailto_addrs'] = mailto_addrs def show_msgs(msgs): @@ -102,32 +102,33 @@ def show_msgs(msgs): for msg in msgs: red = 'black' flag = msg.get_flag(Message.FLAG.MATCH) - if flag: red='red' + if flag: red = 'red' frm = msg.get_header('From') frm = mailto_addrs(frm) subj = msg.get_header('Subject') lnk = urllib.quote_plus(msg.get_message_id()) rs = show_msgs(msg.get_replies()) - r += '<li><font color=%s>%s—<a href="%s/show/%s">%s</a></font> %s</li>' % (red,frm,prefix,lnk,subj,rs) + r += '<li><font color=%s>%s—<a href="%s/show/%s">%s</a></font> %s</li>' % ( + red, frm, prefix, lnk, subj, rs) r += '</ul>' return r env.globals['show_msgs'] = show_msgs # As email.message.walk, but showing close tags as well def mywalk(self): - yield self - if self.is_multipart(): - for subpart in self.get_payload(): - for subsubpart in mywalk(subpart): - yield subsubpart - yield 'close-div' + yield self + if self.is_multipart(): + for subpart in self.get_payload(): + for subsubpart in mywalk(subpart): + yield subsubpart + yield 'close-div' class show: - def GET(self,mid): + def GET(self, mid): web.header('Content-type', 'text/html') web.header('Transfer-Encoding', 'chunked') db = Database() - q = Query(db,'id:'+mid) + q = Query(db, 'id:'+mid) m = list(q.search_messages())[0] template = env.get_template('show.html') # FIXME add reply-all link with email.urils.getaddresses @@ -139,11 +140,11 @@ class show: prefix=prefix, sprefix=webprefix) -def format_message(fn,mid): - msg = MaildirMessage(open(fn)) - return format_message_walk(msg,mid) +def format_message(fn, mid): + msg = MaildirMessage(open(fn)) + return format_message_walk(msg, mid) -def decodeAnyway(txt,charset='ascii'): +def decodeAnyway(txt, charset='ascii'): try: out = txt.decode(charset) except UnicodeDecodeError: @@ -153,63 +154,45 @@ def decodeAnyway(txt,charset='ascii'): out = txt.decode('latin1') return out -def format_message_walk(msg,mid): - counter = 0 - cid_refd = [] - for part in mywalk(msg): - if part=='close-div': - yield '</div>' - elif part.get_content_maintype() == 'multipart': - yield '<div class="multipart-%s">' % part.get_content_subtype() - if part.get_content_subtype() == 'alternative': - yield '<ul>' - for subpart in part.get_payload(): - yield ('<li><a href="#%s">%s</a></li>' % - (subpart.get_content_type().replace('/', '-'), - subpart.get_content_type())) - yield '</ul>' - elif part.get_content_type() == 'message/rfc822': - # FIXME extract subject, date, to/cc/from into a separate template and use it here - yield '<div class="message-rfc822">' - elif part.get_content_maintype() == 'text': - if part.get_content_subtype() == 'plain': - yield '<div id="text-plain"><pre>' - out = part.get_payload(decode=True) - out = decodeAnyway(out,part.get_content_charset('ascii')) - yield out - yield '</pre></div>' - elif part.get_content_subtype() == 'html': - yield '<div id="text-html">' - unb64 = part.get_payload(decode=True) - decoded = decodeAnyway(unb64,part.get_content_charset('ascii')) - cid_refd += find_cids(decoded) - part.set_payload(replace_cids(decoded,mid).encode(part.get_content_charset('ascii'))) - (filename,cid) = link_to_cached_file(part,mid,counter) - counter +=1 - yield '<iframe class="embedded-html" src="%s">' % os.path.join(prefix,cachedir,mid,filename) - yield '</div>' - else: - yield '<div id="%s">' % (part.get_content_type().replace('/','-'),) - (filename,cid) = link_to_cached_file(part,mid,counter) - counter += 1 - yield '<a href="%s">%s (%s)</a>' % (os.path.join(prefix, - cachedir, - mid, - filename), - filename, - part.get_content_type()) - yield '</div>' - elif part.get_content_maintype() == 'image': - (filename,cid) = link_to_cached_file(part,mid,counter) - if cid not in cid_refd: - counter += 1 - yield '<img src="%s" alt="%s">' % (os.path.join(prefix, - cachedir, - mid, - filename), - filename) +def format_message_walk(msg, mid): + counter = 0 + cid_refd = [] + for part in mywalk(msg): + if part == 'close-div': + yield '</div>' + elif part.get_content_maintype() == 'multipart': + yield '<div class="multipart-%s">' % part.get_content_subtype() + if part.get_content_subtype() == 'alternative': + yield '<ul>' + for subpart in part.get_payload(): + yield ('<li><a href="#%s">%s</a></li>' % + (subpart.get_content_type().replace('/', '-'), + subpart.get_content_type())) + yield '</ul>' + elif part.get_content_type() == 'message/rfc822': + # FIXME extract subject, date, to/cc/from into a separate template and use it here + yield '<div class="message-rfc822">' + elif part.get_content_maintype() == 'text': + if part.get_content_subtype() == 'plain': + yield '<div id="text-plain"><pre>' + out = part.get_payload(decode=True) + out = decodeAnyway(out, part.get_content_charset('ascii')) + yield out + yield '</pre></div>' + elif part.get_content_subtype() == 'html': + yield '<div id="text-html">' + unb64 = part.get_payload(decode=True) + decoded = decodeAnyway(unb64, part.get_content_charset('ascii')) + cid_refd += find_cids(decoded) + part.set_payload(replace_cids(decoded, mid).encode(part.get_content_charset('ascii'))) + (filename, cid) = link_to_cached_file(part, mid, counter) + counter += 1 + yield '<iframe class="embedded-html" src="%s">' % ( + os.path.join(prefix, cachedir, mid, filename),) + yield '</div>' else: - (filename,cid) = link_to_cached_file(part,mid,counter) + yield '<div id="%s">' % (part.get_content_type().replace('/', '-'),) + (filename, cid) = link_to_cached_file(part, mid, counter) counter += 1 yield '<a href="%s">%s (%s)</a>' % (os.path.join(prefix, cachedir, @@ -217,50 +200,69 @@ def format_message_walk(msg,mid): filename), filename, part.get_content_type()) + yield '</div>' + elif part.get_content_maintype() == 'image': + (filename, cid) = link_to_cached_file(part, mid, counter) + if cid not in cid_refd: + counter += 1 + yield '<img src="%s" alt="%s">' % (os.path.join(prefix, + cachedir, + mid, + filename), + filename) + else: + (filename, cid) = link_to_cached_file(part, mid, counter) + counter += 1 + yield '<a href="%s">%s (%s)</a>' % (os.path.join(prefix, + cachedir, + mid, + filename), + filename, + part.get_content_type()) env.globals['format_message'] = format_message -def replace_cids(body,mid): - return body.replace('cid:',os.path.join(prefix,cachedir,mid)+'/') +def replace_cids(body, mid): + return body.replace('cid:', os.path.join(prefix, cachedir, mid)+'/') def find_cids(body): - return re.findall(r'cid:([^ "\'>]*)', body) + return re.findall(r'cid:([^ "\'>]*)', body) -def link_to_cached_file(part,mid,counter): - filename = part.get_filename() - if not filename: - ext = mimetypes.guess_extension(part.get_content_type()) - if not ext: - ext = '.bin' - filename = 'part-%03d%s' % (counter, ext) +def link_to_cached_file(part, mid, counter): + filename = part.get_filename() + if not filename: + ext = mimetypes.guess_extension(part.get_content_type()) + if not ext: + ext = '.bin' + filename = 'part-%03d%s' % (counter, ext) + try: + os.makedirs(os.path.join(cachedir, mid)) + except OSError: + pass + fn = os.path.join(cachedir, mid, filename) # FIXME escape mid,filename + fp = open(fn, 'wb') + if part.get_content_maintype() == 'text': + data = part.get_payload(decode=True) + data = decodeAnyway(data, part.get_content_charset('ascii')).encode('utf-8') + else: try: - os.makedirs(os.path.join(cachedir,mid)) + data = part.get_payload(decode=True) + except: + data = part.get_payload(decode=False) + if data: fp.write(data) + fp.close() + if 'Content-ID' in part: + cid = part['Content-ID'] + if cid[0] == '<' and cid[-1] == '>': cid = cid[1:-1] + cid_fn = os.path.join(cachedir, mid, cid) # FIXME escape mid,cid + try: + os.unlink(cid_fn) except OSError: pass - fn = os.path.join(cachedir, mid, filename) # FIXME escape mid,filename - fp = open(fn, 'wb') - if part.get_content_maintype()=='text': - data = part.get_payload(decode=True) - data = decodeAnyway(data,part.get_content_charset('ascii')).encode('utf-8') - else: - try: - data = part.get_payload(decode=True) - except: - data = part.get_payload(decode=False) - if data: fp.write(data) - fp.close() - if 'Content-ID' in part: - cid = part['Content-ID'] - if cid[0]=='<' and cid[-1]=='>': cid = cid[1:-1] - cid_fn = os.path.join(cachedir, mid, cid) # FIXME escape mid,cid - try: - os.unlink(cid_fn) - except OSError: - pass - os.link(fn,cid_fn) - return (filename,cid) - else: - return (filename,None) + os.link(fn, cid_fn) + return (filename, cid) + else: + return (filename, None) -if __name__ == '__main__': - app = web.application(urls, globals()) - app.run() +if __name__ == '__main__': + app = web.application(urls, globals()) + app.run() -- 2.15.1 ^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: [PATCH 04/11] nmweb: (pylint) normalize whitespace 2018-02-05 4:19 ` [PATCH 04/11] nmweb: (pylint) normalize whitespace Daniel Kahn Gillmor @ 2018-03-17 12:37 ` David Bremner 0 siblings, 0 replies; 15+ messages in thread From: David Bremner @ 2018-03-17 12:37 UTC (permalink / raw) To: Daniel Kahn Gillmor, Notmuch Mail Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes: > I've settled on: > > * 2-space indent > * one space after comma > * one space on either side of assignment and comparison operators > * no trailing whitespace > > This now satisfies the indentation/whitespace checks of: > > pylint --indent-string=' ' --indent-after-paren=2 nmweb.py No objection to the patch, but would it be possible to also update .dir-locals.el (or a directory local version) for emacs? d ^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH 05/11] nmweb: (pylint) import built-in modules before third-party modules 2018-02-05 4:19 clean up for nmweb/btsmail Daniel Kahn Gillmor ` (3 preceding siblings ...) 2018-02-05 4:19 ` [PATCH 04/11] nmweb: (pylint) normalize whitespace Daniel Kahn Gillmor @ 2018-02-05 4:19 ` Daniel Kahn Gillmor 2018-02-05 4:19 ` [PATCH 06/11] nmweb: (pylint) put if clause on separate line from predicate Daniel Kahn Gillmor ` (6 subsequent siblings) 11 siblings, 0 replies; 15+ messages in thread From: Daniel Kahn Gillmor @ 2018-02-05 4:19 UTC (permalink / raw) To: Notmuch Mail --- contrib/notmuch-web/nmweb.py | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/contrib/notmuch-web/nmweb.py b/contrib/notmuch-web/nmweb.py index bee4896e..6c687554 100755 --- a/contrib/notmuch-web/nmweb.py +++ b/contrib/notmuch-web/nmweb.py @@ -2,15 +2,16 @@ from __future__ import absolute_import -import web -from notmuch import Database, Query, Message import urllib from datetime import datetime from mailbox import MaildirMessage -import os import mimetypes import email import re +import os + +import web +from notmuch import Database, Query, Message from jinja2 import Environment, FileSystemLoader # FIXME to PackageLoader from jinja2 import Markup -- 2.15.1 ^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH 06/11] nmweb: (pylint) put if clause on separate line from predicate 2018-02-05 4:19 clean up for nmweb/btsmail Daniel Kahn Gillmor ` (4 preceding siblings ...) 2018-02-05 4:19 ` [PATCH 05/11] nmweb: (pylint) import built-in modules before third-party modules Daniel Kahn Gillmor @ 2018-02-05 4:19 ` Daniel Kahn Gillmor 2018-02-05 4:19 ` [PATCH 07/11] nmweb: shim for launching from gunicorn Daniel Kahn Gillmor ` (5 subsequent siblings) 11 siblings, 0 replies; 15+ messages in thread From: Daniel Kahn Gillmor @ 2018-02-05 4:19 UTC (permalink / raw) To: Notmuch Mail --- contrib/notmuch-web/nmweb.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/contrib/notmuch-web/nmweb.py b/contrib/notmuch-web/nmweb.py index 6c687554..0217fce0 100755 --- a/contrib/notmuch-web/nmweb.py +++ b/contrib/notmuch-web/nmweb.py @@ -249,7 +249,8 @@ def link_to_cached_file(part, mid, counter): data = part.get_payload(decode=True) except: data = part.get_payload(decode=False) - if data: fp.write(data) + if data: + fp.write(data) fp.close() if 'Content-ID' in part: cid = part['Content-ID'] -- 2.15.1 ^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH 07/11] nmweb: shim for launching from gunicorn 2018-02-05 4:19 clean up for nmweb/btsmail Daniel Kahn Gillmor ` (5 preceding siblings ...) 2018-02-05 4:19 ` [PATCH 06/11] nmweb: (pylint) put if clause on separate line from predicate Daniel Kahn Gillmor @ 2018-02-05 4:19 ` Daniel Kahn Gillmor 2018-02-05 4:19 ` [PATCH 08/11] nmweb: use urllib.quote_plus() as plain quote_plus() Daniel Kahn Gillmor ` (4 subsequent siblings) 11 siblings, 0 replies; 15+ messages in thread From: Daniel Kahn Gillmor @ 2018-02-05 4:19 UTC (permalink / raw) To: Notmuch Mail file:///usr/share/doc/python-webpy-doc/html/deploying.html suggests that gunicorn is an efficient way to deploy a webpy-based WSGI daemon. This script is glue between nmweb.py and gunicorn. --- contrib/notmuch-web/nmgunicorn.py | 11 +++++++++++ 1 file changed, 11 insertions(+) create mode 100644 contrib/notmuch-web/nmgunicorn.py diff --git a/contrib/notmuch-web/nmgunicorn.py b/contrib/notmuch-web/nmgunicorn.py new file mode 100644 index 00000000..e71ba12a --- /dev/null +++ b/contrib/notmuch-web/nmgunicorn.py @@ -0,0 +1,11 @@ +#!/usr/bin/env python3 + +# to launch nmweb from gunicorn. + +from nmweb import urls, index, search, show +import web + +app = web.application(urls, globals()) + +# get the wsgi app from web.py application object +wsgiapp = app.wsgifunc() -- 2.15.1 ^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH 08/11] nmweb: use urllib.quote_plus() as plain quote_plus() 2018-02-05 4:19 clean up for nmweb/btsmail Daniel Kahn Gillmor ` (6 preceding siblings ...) 2018-02-05 4:19 ` [PATCH 07/11] nmweb: shim for launching from gunicorn Daniel Kahn Gillmor @ 2018-02-05 4:19 ` Daniel Kahn Gillmor 2018-02-05 4:19 ` [PATCH 09/11] nmweb: explicitly move to python3 Daniel Kahn Gillmor ` (3 subsequent siblings) 11 siblings, 0 replies; 15+ messages in thread From: Daniel Kahn Gillmor @ 2018-02-05 4:19 UTC (permalink / raw) To: Notmuch Mail This will make it easier to import quote_plus from urllib.parse, which is where it is in python3. --- contrib/notmuch-web/nmweb.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/contrib/notmuch-web/nmweb.py b/contrib/notmuch-web/nmweb.py index 0217fce0..31351399 100755 --- a/contrib/notmuch-web/nmweb.py +++ b/contrib/notmuch-web/nmweb.py @@ -2,7 +2,7 @@ from __future__ import absolute_import -import urllib +from urllib import quote_plus from datetime import datetime from mailbox import MaildirMessage import mimetypes @@ -34,7 +34,7 @@ def urlencode_filter(s): if type(s) == 'Markup': s = s.unescape() s = s.encode('utf8') - s = urllib.quote_plus(s) + s = quote_plus(s) return Markup(s) env.filters['url'] = urlencode_filter @@ -69,7 +69,7 @@ class search: redir = True terms += ' %s..%s' % (afters, befores) if redir: - raise web.seeother('/search/%s' % urllib.quote_plus(terms)) + raise web.seeother('/search/%s' % quote_plus(terms)) web.header('Content-type', 'text/html') web.header('Transfer-Encoding', 'chunked') db = Database() @@ -107,7 +107,7 @@ def show_msgs(msgs): frm = msg.get_header('From') frm = mailto_addrs(frm) subj = msg.get_header('Subject') - lnk = urllib.quote_plus(msg.get_message_id()) + lnk = quote_plus(msg.get_message_id()) rs = show_msgs(msg.get_replies()) r += '<li><font color=%s>%s—<a href="%s/show/%s">%s</a></font> %s</li>' % ( red, frm, prefix, lnk, subj, rs) -- 2.15.1 ^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH 09/11] nmweb: explicitly move to python3 2018-02-05 4:19 clean up for nmweb/btsmail Daniel Kahn Gillmor ` (7 preceding siblings ...) 2018-02-05 4:19 ` [PATCH 08/11] nmweb: use urllib.quote_plus() as plain quote_plus() Daniel Kahn Gillmor @ 2018-02-05 4:19 ` Daniel Kahn Gillmor 2018-02-05 4:19 ` [PATCH 10/11] nmweb: hide numeric date fields' Daniel Kahn Gillmor ` (2 subsequent siblings) 11 siblings, 0 replies; 15+ messages in thread From: Daniel Kahn Gillmor @ 2018-02-05 4:19 UTC (permalink / raw) To: Notmuch Mail --- contrib/notmuch-web/nmweb.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/contrib/notmuch-web/nmweb.py b/contrib/notmuch-web/nmweb.py index 31351399..eaeeb507 100755 --- a/contrib/notmuch-web/nmweb.py +++ b/contrib/notmuch-web/nmweb.py @@ -1,8 +1,8 @@ -#!/usr/bin/env python +#!/usr/bin/env python3 from __future__ import absolute_import -from urllib import quote_plus +from urllib.parse import quote_plus from datetime import datetime from mailbox import MaildirMessage import mimetypes -- 2.15.1 ^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH 10/11] nmweb: hide numeric date fields' 2018-02-05 4:19 clean up for nmweb/btsmail Daniel Kahn Gillmor ` (8 preceding siblings ...) 2018-02-05 4:19 ` [PATCH 09/11] nmweb: explicitly move to python3 Daniel Kahn Gillmor @ 2018-02-05 4:19 ` Daniel Kahn Gillmor 2018-02-05 4:19 ` [PATCH 11/11] nmweb: handle non-numeric timestamp inputs Daniel Kahn Gillmor 2018-03-24 23:05 ` clean up for nmweb/btsmail David Bremner 11 siblings, 0 replies; 15+ messages in thread From: Daniel Kahn Gillmor @ 2018-02-05 4:19 UTC (permalink / raw) To: Notmuch Mail These are populated by javascript, but the user shouldn't see them. --- contrib/notmuch-web/templates/base.html | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/contrib/notmuch-web/templates/base.html b/contrib/notmuch-web/templates/base.html index 6d204562..e3743c93 100644 --- a/contrib/notmuch-web/templates/base.html +++ b/contrib/notmuch-web/templates/base.html @@ -20,9 +20,9 @@ src="{{sprefix}}/js/jquery-ui-1.8.21.custom.min.js"></script> <form action="{{prefix}}/search/" method="GET" data-ajax="false"> <label for="terms">Terms</label><input id="terms" name="terms"> <label for="after">After</label><input id="after" -name="after"><input id="afters" name="afters"> +name="after"><input type="hidden" id="afters" name="afters"> <label for="before">Before</label><input id="before" -name="before"><input id="befores" name="befores"> +name="before"><input id="befores" type="hidden" name="befores"> <input type="submit" name="submit" id="submit" value="Search"> </form> {% endblock searchform %} -- 2.15.1 ^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH 11/11] nmweb: handle non-numeric timestamp inputs 2018-02-05 4:19 clean up for nmweb/btsmail Daniel Kahn Gillmor ` (9 preceding siblings ...) 2018-02-05 4:19 ` [PATCH 10/11] nmweb: hide numeric date fields' Daniel Kahn Gillmor @ 2018-02-05 4:19 ` Daniel Kahn Gillmor 2018-03-17 12:57 ` David Bremner 2018-03-24 23:05 ` clean up for nmweb/btsmail David Bremner 11 siblings, 1 reply; 15+ messages in thread From: Daniel Kahn Gillmor @ 2018-02-05 4:19 UTC (permalink / raw) To: Notmuch Mail Without this check, it's trivial to crash the nmweb daemon with a ValueError by putting a non-numeric value in befores or afters. --- contrib/notmuch-web/nmweb.py | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/contrib/notmuch-web/nmweb.py b/contrib/notmuch-web/nmweb.py index eaeeb507..21276b66 100755 --- a/contrib/notmuch-web/nmweb.py +++ b/contrib/notmuch-web/nmweb.py @@ -65,9 +65,12 @@ class search: befores = web.input(befores=None).befores else: befores = '4294967296' # 2^32 - if int(afters) > 0 or int(befores) < 4294967296: - redir = True - terms += ' %s..%s' % (afters, befores) + try: + if int(afters) > 0 or int(befores) < 4294967296: + redir = True + terms += ' %s..%s' % (afters, befores) + except ValueError: + pass if redir: raise web.seeother('/search/%s' % quote_plus(terms)) web.header('Content-type', 'text/html') -- 2.15.1 ^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: [PATCH 11/11] nmweb: handle non-numeric timestamp inputs 2018-02-05 4:19 ` [PATCH 11/11] nmweb: handle non-numeric timestamp inputs Daniel Kahn Gillmor @ 2018-03-17 12:57 ` David Bremner 0 siblings, 0 replies; 15+ messages in thread From: David Bremner @ 2018-03-17 12:57 UTC (permalink / raw) To: Daniel Kahn Gillmor, Notmuch Mail [-- Attachment #1: Type: text/plain, Size: 1619 bytes --] Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes: > Without this check, it's trivial to crash the nmweb daemon with a > ValueError by putting a non-numeric value in befores or afters. I don't really understand what's going on here enough to comment on this patch. Perhaps Brian can confirm. I notice currently the web page is generating URL's like https://nmbug.notmuchmail.org/btsmail/search/crash+date%3A%401516161600..%401520910000000 and those in turn are causing internal server errors (at least on the instance on nmbug. > --- > contrib/notmuch-web/nmweb.py | 9 ++++++--- > 1 file changed, 6 insertions(+), 3 deletions(-) > > diff --git a/contrib/notmuch-web/nmweb.py b/contrib/notmuch-web/nmweb.py > index eaeeb507..21276b66 100755 > --- a/contrib/notmuch-web/nmweb.py > +++ b/contrib/notmuch-web/nmweb.py > @@ -65,9 +65,12 @@ class search: > befores = web.input(befores=None).befores > else: > befores = '4294967296' # 2^32 > - if int(afters) > 0 or int(befores) < 4294967296: > - redir = True > - terms += ' %s..%s' % (afters, befores) > + try: > + if int(afters) > 0 or int(befores) < 4294967296: > + redir = True > + terms += ' %s..%s' % (afters, befores) > + except ValueError: > + pass > if redir: > raise web.seeother('/search/%s' % quote_plus(terms)) > web.header('Content-type', 'text/html') > -- > 2.15.1 > > _______________________________________________ > notmuch mailing list > notmuch@notmuchmail.org > https://notmuchmail.org/mailman/listinfo/notmuch [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 658 bytes --] ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: clean up for nmweb/btsmail 2018-02-05 4:19 clean up for nmweb/btsmail Daniel Kahn Gillmor ` (10 preceding siblings ...) 2018-02-05 4:19 ` [PATCH 11/11] nmweb: handle non-numeric timestamp inputs Daniel Kahn Gillmor @ 2018-03-24 23:05 ` David Bremner 11 siblings, 0 replies; 15+ messages in thread From: David Bremner @ 2018-03-24 23:05 UTC (permalink / raw) To: Daniel Kahn Gillmor, Notmuch Mail Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes: > The patches that follow are some proposed cleanups for > contrib/notmuch-web (aka nmweb, aka btsmail), which was originally > proposed by Brian Sniffen. So, the main problem with this series is that contrib/notmuch-web doesn't exist on master. Assuming we still want to include it, we need one or more patches to do so. d ^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2018-03-24 23:05 UTC | newest] Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-02-05 4:19 clean up for nmweb/btsmail Daniel Kahn Gillmor 2018-02-05 4:19 ` [PATCH 01/11] nmweb: (pylint) use __future__.absolute_import Daniel Kahn Gillmor 2018-02-05 4:19 ` [PATCH 02/11] nmweb: (pylint) use replace() function directly on string object Daniel Kahn Gillmor 2018-02-05 4:19 ` [PATCH 03/11] nmweb: (pylint) use spaces instead of tabs Daniel Kahn Gillmor 2018-02-05 4:19 ` [PATCH 04/11] nmweb: (pylint) normalize whitespace Daniel Kahn Gillmor 2018-03-17 12:37 ` David Bremner 2018-02-05 4:19 ` [PATCH 05/11] nmweb: (pylint) import built-in modules before third-party modules Daniel Kahn Gillmor 2018-02-05 4:19 ` [PATCH 06/11] nmweb: (pylint) put if clause on separate line from predicate Daniel Kahn Gillmor 2018-02-05 4:19 ` [PATCH 07/11] nmweb: shim for launching from gunicorn Daniel Kahn Gillmor 2018-02-05 4:19 ` [PATCH 08/11] nmweb: use urllib.quote_plus() as plain quote_plus() Daniel Kahn Gillmor 2018-02-05 4:19 ` [PATCH 09/11] nmweb: explicitly move to python3 Daniel Kahn Gillmor 2018-02-05 4:19 ` [PATCH 10/11] nmweb: hide numeric date fields' Daniel Kahn Gillmor 2018-02-05 4:19 ` [PATCH 11/11] nmweb: handle non-numeric timestamp inputs Daniel Kahn Gillmor 2018-03-17 12:57 ` David Bremner 2018-03-24 23:05 ` clean up for nmweb/btsmail David Bremner
Code repositories for project(s) associated with this public inbox https://yhetil.org/notmuch.git/ This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).