[PATCH 00/17] nmbug-status: Python-3-compabitility and general refactoring

[PATCH 00/17] nmbug-status: Python-3-compabitility and general refactoring

[W. Trevor King]

I noticed that nmbug-status was written in Python :), but not
compatible with Python 3 :(.  I started cleaning up a few print
statements, but this quickly turned into a more general refactoring.
Let me know if this is too much to bite off at once.  I tried to keep
each patch fairly contained, but the Page / HtmlPage addition is still
pretty big.  Despite increasing the size of this module by almost 50%,
I think my final version is more readable.  However, it's always
easier to read your own code, so feel free to tell me that this is a
step in the completely wrong direction ;).

Also anyone with asthetic sensibilities is free to pick nicer colors
in the final patch.  I'm too partial to the EFF permutation symmetry
to be able to pick other colors myself ;).

W. Trevor King (17):
  nmbug-status: Convert to Python-3-compatible print functions
  nmbug-status: Use email.utils instead of rfc822
  nmbug-status: Decode Popen output using the user's locale
  nmbug-status: Factor config-loading out into read_config
  nmbug-status: Add metavars for --config and --get-query
  nmbug-status: Consolidate functions and main code
  nmbug-status: Don't require write access
  nmbug-status: Consolidate HTML header printing
  nmbug-status: Add a Python-3-compatible urllib.parse.quote import
  nmbug-status: Add Page and HtmlPage for modular rendering
  nmbug-status: Normalize table HTML indentation
  nmbug-status: Convert from XHTML 1.0 to HTML 5
  nmbug-status: Encode output using the user's locale
  nmbug-status: Anchor with h3 ids instead of a names
  nmbug-status: Quote the title when using it as an id
  nmbug-status: Use <code> and <p> markup where appropriate
  nmbug-status: Color threads in HTML output

 devel/nmbug/nmbug-status | 412 ++++++++++++++++++++++++++++++-----------------
 1 file changed, 261 insertions(+), 151 deletions(-)

-- 
1.8.5.2.8.g0f6c0d1

[PATCH 01/17] nmbug-status: Convert to Python-3-compatible print functions

[W. Trevor King]

We shouldn't require folks to install Python 2 to run nmbug-status.
---
 devel/nmbug/nmbug-status | 44 +++++++++++++++++++++++---------------------
 1 file changed, 23 insertions(+), 21 deletions(-)

diff --git a/devel/nmbug/nmbug-status b/devel/nmbug/nmbug-status
index 934c895..b1feee9 100755
--- a/devel/nmbug/nmbug-status
+++ b/devel/nmbug/nmbug-status
@@ -6,6 +6,8 @@
 #       - python 2.6 for json
 #       - argparse; either python 2.7, or install separately
 
+from __future__ import print_function
+
 import datetime
 import rfc822
 import urllib
@@ -49,12 +51,12 @@ config = json.load(fp)
 
 if args.list_views:
     for view in config['views']:
-        print view['title']
+        print(view['title'])
     sys.exit(0)
 elif args.get_query != None:
     for view in config['views']:
         if args.get_query == view['title']:
-            print ' and '.join(view['query'])
+            print(' and '.join(view['query']))
     sys.exit(0)
 else:
     # only import notmuch if needed
@@ -75,7 +77,7 @@ class Thread:
 
 def output_with_separator(threadlist, sep):
     outputs = (thread.join_utf8_with_newlines() for thread in threadlist)
-    print sep.join(outputs)
+    print(sep.join(outputs))
 
 headers = ['date', 'from', 'subject']
 
@@ -93,13 +95,13 @@ def print_view(title, query, comment):
     lines = None
 
     if output_format == 'html':
-        print '<h3><a name="%s" />%s</h3>' % (title, title)
-        print comment
-        print 'The view is generated from the following query:'
-        print '<blockquote>'
-        print query_string
-        print '</blockquote>'
-        print '<table>\n'
+        print('<h3><a name="%s" />%s</h3>' % (title, title))
+        print(comment)
+        print('The view is generated from the following query:')
+        print('<blockquote>')
+        print(query_string)
+        print('</blockquote>')
+        print('<table>\n')
 
     for m in q_new.search_messages():
 
@@ -156,7 +158,7 @@ def print_view(title, query, comment):
     if output_format == 'html':
         output_with_separator(threadlist,
                               '\n<tr><td colspan="2"><br /></td></tr>\n')
-        print '</table>'
+        print('</table>')
     else:
         output_with_separator(threadlist, '\n\n')
 
@@ -165,26 +167,26 @@ def print_view(title, query, comment):
 db = notmuch.Database(mode=notmuch.Database.MODE.READ_WRITE)
 
 if output_format == 'html':
-    print '''<?xml version="1.0" encoding="utf-8" ?>
+    print('''<?xml version="1.0" encoding="utf-8" ?>
 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
 <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
 <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
 <title>Notmuch Patches</title>
 </head>
-<body>'''
-    print '<h2>Notmuch Patches</h2>'
-    print 'Generated: %s<br />' % datetime.datetime.utcnow().date()
-    print 'For more infomation see <a href="http://notmuchmail.org/nmbug">nmbug</a>'
+<body>''')
+    print('<h2>Notmuch Patches</h2>')
+    print('Generated: %s<br />' % datetime.datetime.utcnow().date())
+    print('For more infomation see <a href="http://notmuchmail.org/nmbug">nmbug</a>')
 
-    print '<h3>Views</h3>'
-    print '<ul>'
+    print('<h3>Views</h3>')
+    print('<ul>')
     for view in config['views']:
-        print '<li><a href="#%(title)s">%(title)s</a></li>' % view
-    print '</ul>'
+        print('<li><a href="#%(title)s">%(title)s</a></li>' % view)
+    print('</ul>')
 
 for view in config['views']:
     print_view(**view)
 
 if output_format == 'html':
-    print '</body>\n</html>'
+    print('</body>\n</html>')
-- 
1.8.5.2.8.g0f6c0d1

[PATCH 02/17] nmbug-status: Use email.utils instead of rfc822

[W. Trevor King]

rfc822 has been deprecated since Python 2.3, and it's gone in
Python 3 [1].

[1]: http://docs.python.org/2/library/rfc822.html
---
 devel/nmbug/nmbug-status | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/devel/nmbug/nmbug-status b/devel/nmbug/nmbug-status
index b1feee9..6525176 100755
--- a/devel/nmbug/nmbug-status
+++ b/devel/nmbug/nmbug-status
@@ -9,7 +9,7 @@
 from __future__ import print_function
 
 import datetime
-import rfc822
+import email.utils
 import urllib
 import json
 import argparse
@@ -128,7 +128,7 @@ def print_view(title, query, comment):
                 val = str.join(' ', val.split(None)[1:4])
                 val = str(datetime.datetime.strptime(val, '%d %b %Y').date())
             elif header == 'from':
-                (val, addr) = rfc822.parseaddr(val)
+                (val, addr) = email.utils.parseaddr(val)
                 if val == '':
                     val = addr.split('@')[0]
 
-- 
1.8.5.2.8.g0f6c0d1

[PATCH 03/17] nmbug-status: Decode Popen output using the user's locale

[W. Trevor King]

Avoid:

  $ ./nmbug-status --list-views
  Traceback (most recent call last):
    File "./nmbug-status", line 47, in <module>
      'cat-file', 'blob', sha1+':status-config.json'],
  TypeError: can't concat bytes to str

by explicitly converting the byte-stream read from Popen into a
Unicode string.  On Python 2, this conversion is str -> unicode; on
Python 3 it is bytes -> str.

_ENCODING is derived from the user's locale (or system default) in an
attempt to match Git's output encoding.  It may be more robust to skip
the encoding/decoding by using a Python wrapper like pygit2 [1] for
Git access.  That's a fairly heavy dependency though, and using the
locale will probably work.

[1]: http://www.pygit2.org/
---
 devel/nmbug/nmbug-status | 25 ++++++++++++++++---------
 1 file changed, 16 insertions(+), 9 deletions(-)

diff --git a/devel/nmbug/nmbug-status b/devel/nmbug/nmbug-status
index 6525176..a7a391d 100755
--- a/devel/nmbug/nmbug-status
+++ b/devel/nmbug/nmbug-status
@@ -8,8 +8,10 @@
 
 from __future__ import print_function
 
+import codecs
 import datetime
 import email.utils
+import locale
 import urllib
 import json
 import argparse
@@ -17,6 +19,10 @@ import os
 import sys
 import subprocess
 
+
+_ENCODING = locale.getpreferredencoding() or sys.getdefaultencoding()
+
+
 # parse command line arguments
 
 parser = argparse.ArgumentParser()
@@ -37,15 +43,16 @@ else:
     nmbhome = os.getenv('NMBGIT', os.path.expanduser('~/.nmbug'))
 
     # read only the first line from the pipe
-    sha1 = subprocess.Popen(['git', '--git-dir', nmbhome,
-                             'show-ref', '-s', 'config'],
-                            stdout=subprocess.PIPE).stdout.readline()
-
-    sha1 = sha1.rstrip()
-
-    fp = subprocess.Popen(['git', '--git-dir', nmbhome,
-                           'cat-file', 'blob', sha1+':status-config.json'],
-                          stdout=subprocess.PIPE).stdout
+    sha1_bytes = subprocess.Popen(
+        ['git', '--git-dir', nmbhome, 'show-ref', '-s', 'config'],
+        stdout=subprocess.PIPE).stdout.readline()
+    sha1 = sha1_bytes.decode(_ENCODING).rstrip()
+
+    fp_byte_stream = subprocess.Popen(
+        ['git', '--git-dir', nmbhome, 'cat-file', 'blob',
+         sha1+':status-config.json'],
+        stdout=subprocess.PIPE).stdout
+    fp = codecs.getreader(encoding=_ENCODING)(stream=fp_byte_stream)
 
 config = json.load(fp)
 
-- 
1.8.5.2.8.g0f6c0d1

[PATCH 04/17] nmbug-status: Factor config-loading out into read_config

[W. Trevor King]

By isolating this peripheral handling, we make the core logic of
nmbug-status easier to read.
---
 devel/nmbug/nmbug-status | 45 +++++++++++++++++++++++++--------------------
 1 file changed, 25 insertions(+), 20 deletions(-)

diff --git a/devel/nmbug/nmbug-status b/devel/nmbug/nmbug-status
index a7a391d..6dfbe4d 100755
--- a/devel/nmbug/nmbug-status
+++ b/devel/nmbug/nmbug-status
@@ -23,6 +23,30 @@ import subprocess
 _ENCODING = locale.getpreferredencoding() or sys.getdefaultencoding()
 
 
+def read_config(path=None, encoding=None):
+    "Read config from json file"
+    if not encoding:
+        encoding = _ENCODING
+    if path:
+        fp = open(path)
+    else:
+        nmbhome = os.getenv('NMBGIT', os.path.expanduser('~/.nmbug'))
+
+        # read only the first line from the pipe
+        sha1_bytes = subprocess.Popen(
+            ['git', '--git-dir', nmbhome, 'show-ref', '-s', 'config'],
+            stdout=subprocess.PIPE).stdout.readline()
+        sha1 = sha1_bytes.decode(encoding).rstrip()
+
+        fp_byte_stream = subprocess.Popen(
+            ['git', '--git-dir', nmbhome, 'cat-file', 'blob',
+             sha1+':status-config.json'],
+            stdout=subprocess.PIPE).stdout
+        fp = codecs.getreader(encoding=encoding)(stream=fp_byte_stream)
+
+    return json.load(fp)
+
+
 # parse command line arguments
 
 parser = argparse.ArgumentParser()
@@ -35,26 +59,7 @@ parser.add_argument('--get-query', help='get query for view')
 
 args = parser.parse_args()
 
-# read config from json file
-
-if args.config != None:
-    fp = open(args.config)
-else:
-    nmbhome = os.getenv('NMBGIT', os.path.expanduser('~/.nmbug'))
-
-    # read only the first line from the pipe
-    sha1_bytes = subprocess.Popen(
-        ['git', '--git-dir', nmbhome, 'show-ref', '-s', 'config'],
-        stdout=subprocess.PIPE).stdout.readline()
-    sha1 = sha1_bytes.decode(_ENCODING).rstrip()
-
-    fp_byte_stream = subprocess.Popen(
-        ['git', '--git-dir', nmbhome, 'cat-file', 'blob',
-         sha1+':status-config.json'],
-        stdout=subprocess.PIPE).stdout
-    fp = codecs.getreader(encoding=_ENCODING)(stream=fp_byte_stream)
-
-config = json.load(fp)
+config = read_config(path=args.config)
 
 if args.list_views:
     for view in config['views']:
-- 
1.8.5.2.8.g0f6c0d1

[PATCH 05/17] nmbug-status: Add metavars for --config and --get-query

[W. Trevor King]

Now the suggested usage (listed by 'nmbug-status --help') is:

  usage: nmbug-status [-h] [--text] [--config PATH] [--list-views]
                      [--get-query VIEW]

instead of the less obvious:

  usage: nmbug-status [-h] [--text] [--config CONFIG] [--list-views]
                      [--get-query GET_QUERY]
---
 devel/nmbug/nmbug-status | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/devel/nmbug/nmbug-status b/devel/nmbug/nmbug-status
index 6dfbe4d..22eeb5c 100755
--- a/devel/nmbug/nmbug-status
+++ b/devel/nmbug/nmbug-status
@@ -52,10 +52,12 @@ def read_config(path=None, encoding=None):
 parser = argparse.ArgumentParser()
 parser.add_argument('--text', help='output plain text format',
                     action='store_true')
-parser.add_argument('--config', help='load config from given file')
+parser.add_argument('--config', help='load config from given file',
+                    metavar='PATH')
 parser.add_argument('--list-views', help='list views',
                     action='store_true')
-parser.add_argument('--get-query', help='get query for view')
+parser.add_argument('--get-query', help='get query for view',
+                    metavar='VIEW')
 
 args = parser.parse_args()
 
-- 
1.8.5.2.8.g0f6c0d1

[PATCH 06/17] nmbug-status: Consolidate functions and main code

[W. Trevor King]

The definitions of Thread, output_with_separator, and print_view were
between the main argparse and view-printing code.  Group them together
with our existing read_config at the top of the module, which makes
for easier reading in the main section.

I also:

* Made 'headers' a print_view argument instead of a module-level
  global.  The list -> tuple conversion avoids having a mutable
  default argument, which makes some people jumpy ;).

* Made 'db' a print_view argument instead of relying on the global
  namespace to access it from print_view.
---
 devel/nmbug/nmbug-status | 78 +++++++++++++++++++++++++-----------------------
 1 file changed, 40 insertions(+), 38 deletions(-)

diff --git a/devel/nmbug/nmbug-status b/devel/nmbug/nmbug-status
index 22eeb5c..199892f 100755
--- a/devel/nmbug/nmbug-status
+++ b/devel/nmbug/nmbug-status
@@ -47,40 +47,6 @@ def read_config(path=None, encoding=None):
     return json.load(fp)
 
 
-# parse command line arguments
-
-parser = argparse.ArgumentParser()
-parser.add_argument('--text', help='output plain text format',
-                    action='store_true')
-parser.add_argument('--config', help='load config from given file',
-                    metavar='PATH')
-parser.add_argument('--list-views', help='list views',
-                    action='store_true')
-parser.add_argument('--get-query', help='get query for view',
-                    metavar='VIEW')
-
-args = parser.parse_args()
-
-config = read_config(path=args.config)
-
-if args.list_views:
-    for view in config['views']:
-        print(view['title'])
-    sys.exit(0)
-elif args.get_query != None:
-    for view in config['views']:
-        if args.get_query == view['title']:
-            print(' and '.join(view['query']))
-    sys.exit(0)
-else:
-    # only import notmuch if needed
-    import notmuch
-
-if args.text:
-    output_format = 'text'
-else:
-    output_format = 'html'
-
 class Thread:
     def __init__(self, last, lines):
         self.last = last
@@ -89,16 +55,17 @@ class Thread:
     def join_utf8_with_newlines(self):
         return '\n'.join( (line.encode('utf-8') for line in self.lines) )
 
+
 def output_with_separator(threadlist, sep):
     outputs = (thread.join_utf8_with_newlines() for thread in threadlist)
     print(sep.join(outputs))
 
-headers = ['date', 'from', 'subject']
 
-def print_view(title, query, comment):
+def print_view(database, title, query, comment,
+               headers=('date', 'from', 'subject')):
 
     query_string = ' and '.join(query)
-    q_new = notmuch.Query(db, query_string)
+    q_new = notmuch.Query(database, query_string)
     q_new.set_sort(notmuch.Query.SORT.OLDEST_FIRST)
 
     last_thread_id = ''
@@ -176,6 +143,41 @@ def print_view(title, query, comment):
     else:
         output_with_separator(threadlist, '\n\n')
 
+
+# parse command line arguments
+
+parser = argparse.ArgumentParser()
+parser.add_argument('--text', help='output plain text format',
+                    action='store_true')
+parser.add_argument('--config', help='load config from given file',
+                    metavar='PATH')
+parser.add_argument('--list-views', help='list views',
+                    action='store_true')
+parser.add_argument('--get-query', help='get query for view',
+                    metavar='VIEW')
+
+args = parser.parse_args()
+
+config = read_config(path=args.config)
+
+if args.list_views:
+    for view in config['views']:
+        print(view['title'])
+    sys.exit(0)
+elif args.get_query != None:
+    for view in config['views']:
+        if args.get_query == view['title']:
+            print(' and '.join(view['query']))
+    sys.exit(0)
+else:
+    # only import notmuch if needed
+    import notmuch
+
+if args.text:
+    output_format = 'text'
+else:
+    output_format = 'html'
+
 # main program
 
 db = notmuch.Database(mode=notmuch.Database.MODE.READ_WRITE)
@@ -200,7 +202,7 @@ if output_format == 'html':
     print('</ul>')
 
 for view in config['views']:
-    print_view(**view)
+    print_view(database=db, **view)
 
 if output_format == 'html':
     print('</body>\n</html>')
-- 
1.8.5.2.8.g0f6c0d1

[PATCH 07/17] nmbug-status: Don't require write access

[W. Trevor King]

The database in only used for notmuch.Query, so there's no need for
write access.  This allows nmbug-status to run while the database is
being updated, without raising:

  A Xapian exception occurred opening database: Unable to get write lock on …: already locked
  Traceback (most recent call last):
    File "./nmbug-status", line 182, in <module>
      db = notmuch.Database(mode=notmuch.Database.MODE.READ_WRITE)
    File "/…/notmuch/database.py", line 154, in __init__
      self.open(path, mode)
    File "/…/notmuch/database.py", line 214, in open
      raise NotmuchError(status)
  notmuch.errors.XapianError
---
 devel/nmbug/nmbug-status | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/devel/nmbug/nmbug-status b/devel/nmbug/nmbug-status
index 199892f..be3e28e 100755
--- a/devel/nmbug/nmbug-status
+++ b/devel/nmbug/nmbug-status
@@ -180,7 +180,7 @@ else:
 
 # main program
 
-db = notmuch.Database(mode=notmuch.Database.MODE.READ_WRITE)
+db = notmuch.Database(mode=notmuch.Database.MODE.READ_ONLY)
 
 if output_format == 'html':
     print('''<?xml version="1.0" encoding="utf-8" ?>
-- 
1.8.5.2.8.g0f6c0d1

[PATCH 08/17] nmbug-status: Consolidate HTML header printing

[W. Trevor King]

Make this all one big string, using '...{date}...'.format(date=...) to
inject the date [1].  This syntax was added in Python 2.6, and is
preferred to %-formatting in Python 3 [1].

[1]: http://docs.python.org/2/library/stdtypes.html#str.format
---
 devel/nmbug/nmbug-status | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)

diff --git a/devel/nmbug/nmbug-status b/devel/nmbug/nmbug-status
index be3e28e..3aa83b6 100755
--- a/devel/nmbug/nmbug-status
+++ b/devel/nmbug/nmbug-status
@@ -190,13 +190,12 @@ if output_format == 'html':
 <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
 <title>Notmuch Patches</title>
 </head>
-<body>''')
-    print('<h2>Notmuch Patches</h2>')
-    print('Generated: %s<br />' % datetime.datetime.utcnow().date())
-    print('For more infomation see <a href="http://notmuchmail.org/nmbug">nmbug</a>')
-
-    print('<h3>Views</h3>')
-    print('<ul>')
+<body>
+<h2>Notmuch Patches</h2>
+Generated: {date}<br />
+For more infomation see <a href="http://notmuchmail.org/nmbug">nmbug</a>
+<h3>Views</h3>
+<ul>'''.format(date=datetime.datetime.utcnow().date()))
     for view in config['views']:
         print('<li><a href="#%(title)s">%(title)s</a></li>' % view)
     print('</ul>')
-- 
1.8.5.2.8.g0f6c0d1

[PATCH 09/17] nmbug-status: Add a Python-3-compatible urllib.parse.quote import

[W. Trevor King]

Python 2's urllib.quote [1] has moved to urllib.parse.quote in Python
3 [2].

[1]: http://docs.python.org/2/library/urllib.html#urllib.quote
[2]: http://docs.python.org/3/library/urllib.parse.html#urllib.parse.quote
---
 devel/nmbug/nmbug-status | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/devel/nmbug/nmbug-status b/devel/nmbug/nmbug-status
index 3aa83b6..22b6b10 100755
--- a/devel/nmbug/nmbug-status
+++ b/devel/nmbug/nmbug-status
@@ -12,7 +12,10 @@ import codecs
 import datetime
 import email.utils
 import locale
-import urllib
+try:  # Python 3
+    from urllib.parse import quote
+except ImportError:  # Python 2
+    from urllib import quote
 import json
 import argparse
 import os
@@ -124,8 +127,8 @@ def print_view(database, title, query, comment,
 
         if output_format == 'html':
 
-            out['subject'] = '<a href="http://mid.gmane.org/%s">%s</a>' \
-                % (urllib.quote(mid), out['subject'])
+            out['subject'] = '<a href="http://mid.gmane.org/%s">%s</a>' % (
+                quote(mid), out['subject'])
 
             lines.append(' <tr><td>%s' % out['date'])
             lines.append('</td><td>%s' % out['id'])
-- 
1.8.5.2.8.g0f6c0d1

[PATCH 10/17] nmbug-status: Add Page and HtmlPage for modular rendering

[W. Trevor King]

I was having trouble understanding the logic of the longish print_view
function, so I refactored the output generation into modular bits.
The basic text rendering is handled by Page, which has enough hooks
that HtmlPage can borrow the logic and slot-in HTML generators.

By modularizing the logic it should also be easier to build other
renderers if folks want to customize the layout for other projects.

Timezones
=========

This commit has not effect on the output, except that some dates have
been converted from the sender's timezone to UTC due to:

  -            val = m.get_header(header)
  -            ...
  -            if header == 'date':
  -                val = str.join(' ', val.split(None)[1:4])
  -                val = str(datetime.datetime.strptime(val, '%d %b %Y').date())
  ...
  +                value = str(datetime.datetime.utcfromtimestamp(
  +                    message.get_date()).date())

I also tweaked the HTML header date to be utcnow instead of the local
now() to make all times independent of the generator's local time.
This matches Gmane, which converts all Date headers to UTC (although
they use a 'GMT' suffix).  Notmuch uses
g_mime_utils_header_decode_date to calculate the UTC timestamps, but
uses a NULL tz_offset which drops the information we'd need to get
back to the sender's local time [1].  With the generator's local time
arbitrarily different from the sender's and viewer's local time,
sticking with UTC seems the best bet.

[1]: https://developer.gnome.org/gmime/stable/gmime-gmime-utils.html#g-mime-utils-header-decode-date
---
 devel/nmbug/nmbug-status | 292 +++++++++++++++++++++++++++--------------------
 1 file changed, 171 insertions(+), 121 deletions(-)

diff --git a/devel/nmbug/nmbug-status b/devel/nmbug/nmbug-status
index 22b6b10..7778029 100755
--- a/devel/nmbug/nmbug-status
+++ b/devel/nmbug/nmbug-status
@@ -5,10 +5,13 @@
 # dependencies
 #       - python 2.6 for json
 #       - argparse; either python 2.7, or install separately
+#       - collections; python 2.7
 
 from __future__ import print_function
+from __future__ import unicode_literals
 
 import codecs
+import collections
 import datetime
 import email.utils
 import locale
@@ -24,6 +27,7 @@ import subprocess
 
 
 _ENCODING = locale.getpreferredencoding() or sys.getdefaultencoding()
+_PAGES = {}
 
 
 def read_config(path=None, encoding=None):
@@ -50,104 +54,175 @@ def read_config(path=None, encoding=None):
     return json.load(fp)
 
 
-class Thread:
-    def __init__(self, last, lines):
-        self.last = last
-        self.lines = lines
-
-    def join_utf8_with_newlines(self):
-        return '\n'.join( (line.encode('utf-8') for line in self.lines) )
-
-
-def output_with_separator(threadlist, sep):
-    outputs = (thread.join_utf8_with_newlines() for thread in threadlist)
-    print(sep.join(outputs))
-
-
-def print_view(database, title, query, comment,
-               headers=('date', 'from', 'subject')):
-
-    query_string = ' and '.join(query)
-    q_new = notmuch.Query(database, query_string)
-    q_new.set_sort(notmuch.Query.SORT.OLDEST_FIRST)
-
-    last_thread_id = ''
-    threads = {}
-    threadlist = []
-    out = {}
-    last = None
-    lines = None
-
-    if output_format == 'html':
-        print('<h3><a name="%s" />%s</h3>' % (title, title))
-        print(comment)
-        print('The view is generated from the following query:')
-        print('<blockquote>')
-        print(query_string)
-        print('</blockquote>')
-        print('<table>\n')
-
-    for m in q_new.search_messages():
-
-        thread_id = m.get_thread_id()
-
-        if thread_id != last_thread_id:
-            if threads.has_key(thread_id):
-                last = threads[thread_id].last
-                lines = threads[thread_id].lines
+class Thread (list):
+    def __init__(self):
+        self.running_data = {}
+
+
+class Page (object):
+    def __init__(self, header=None, footer=None):
+        self.header = header
+        self.footer = footer
+
+    def write(self, database, views, stream=None):
+        if not stream:
+            try:  # Python 3
+                byte_stream = sys.stdout.buffer
+            except AttributeError:  # Python 2
+                byte_stream = sys.stdout
+            stream = codecs.getwriter(encoding='UTF-8')(stream=byte_stream)
+        self._write_header(views=views, stream=stream)
+        for view in views:
+            self._write_view(database=database, view=view, stream=stream)
+        self._write_footer(views=views, stream=stream)
+
+    def _write_header(self, views, stream):
+        if self.header:
+            stream.write(self.header)
+
+    def _write_footer(self, views, stream):
+        if self.footer:
+            stream.write(self.footer)
+
+    def _write_view(self, database, view, stream):
+        if 'query-string' not in view:
+            query = view['query']
+            view['query-string'] = ' and '.join(query)
+        q = notmuch.Query(database, view['query-string'])
+        q.set_sort(notmuch.Query.SORT.OLDEST_FIRST)
+        threads = self._get_threads(messages=q.search_messages())
+        self._write_view_header(view=view, stream=stream)
+        self._write_threads(threads=threads, stream=stream)
+
+    def _get_threads(self, messages):
+        threads = collections.OrderedDict()
+        for message in messages:
+            thread_id = message.get_thread_id()
+            if thread_id in threads:
+                thread = threads[thread_id]
             else:
-                last = {}
-                lines = []
-                thread = Thread(last, lines)
+                thread = Thread()
                 threads[thread_id] = thread
-                for h in headers:
-                    last[h] = ''
-                threadlist.append(thread)
-            last_thread_id = thread_id
-
+            thread.running_data, display_data = self._message_display_data(
+                running_data=thread.running_data, message=message)
+            thread.append(display_data)
+        return list(threads.values())
+
+    def _write_view_header(self, view, stream):
+        pass
+
+    def _write_threads(self, threads, stream):
+        for thread in threads:
+            for message_display_data in thread:
+                stream.write(
+                    ('{date:10.10s} {from:20.20s} {subject:40.40s}\n'
+                     '{message-id-term:>72}\n'
+                     ).format(**message_display_data))
+            if thread != threads[-1]:
+                stream.write('\n')
+
+    def _message_display_data(self, running_data, message):
+        headers = ('thread-id', 'message-id', 'date', 'from', 'subject')
+        data = {}
         for header in headers:
-            val = m.get_header(header)
-
-            if header == 'date':
-                val = str.join(' ', val.split(None)[1:4])
-                val = str(datetime.datetime.strptime(val, '%d %b %Y').date())
-            elif header == 'from':
-                (val, addr) = email.utils.parseaddr(val)
-                if val == '':
-                    val = addr.split('@')[0]
-
-            if header != 'subject' and last[header] == val:
-                out[header] = ''
+            if header == 'thread-id':
+                value = message.get_thread_id()
+            elif header == 'message-id':
+                value = message.get_message_id()
+                data['message-id-term'] = 'id:"{}"'.format(value)
+            elif header == 'date':
+                value = str(datetime.datetime.utcfromtimestamp(
+                    message.get_date()).date())
             else:
-                out[header] = val
-                last[header] = val
-
-        mid = m.get_message_id()
-        out['id'] = 'id:"%s"' % mid
-
-        if output_format == 'html':
-
-            out['subject'] = '<a href="http://mid.gmane.org/%s">%s</a>' % (
-                quote(mid), out['subject'])
-
-            lines.append(' <tr><td>%s' % out['date'])
-            lines.append('</td><td>%s' % out['id'])
-            lines.append('</td></tr>')
-            lines.append(' <tr><td>%s' % out['from'])
-            lines.append('</td><td>%s' % out['subject'])
-            lines.append('</td></tr>')
-        else:
-            lines.append('%(date)-10.10s %(from)-20.20s %(subject)-40.40s\n%(id)72s' % out)
-
-    if output_format == 'html':
-        output_with_separator(threadlist,
-                              '\n<tr><td colspan="2"><br /></td></tr>\n')
-        print('</table>')
-    else:
-        output_with_separator(threadlist, '\n\n')
-
+                value = message.get_header(header)
+            if header == 'from':
+                (value, addr) = email.utils.parseaddr(value)
+                if not value:
+                    value = addr.split('@')[0]
+            data[header] = value
+        next_running_data = data.copy()
+        for header, value in data.items():
+            if header in ['message-id', 'subject']:
+                continue
+            if value == running_data.get(header, None):
+                data[header] = ''
+        return (next_running_data, data)
+
+
+class HtmlPage (Page):
+    def _write_header(self, views, stream):
+        super(HtmlPage, self)._write_header(views=views, stream=stream)
+        stream.write('<ul>\n')
+        for view in views:
+            stream.write(
+                '<li><a href="#{title}">{title}</a></li>\n'.format(**view))
+        stream.write('</ul>\n')
+
+    def _write_view_header(self, view, stream):
+        stream.write('<h3><a name="{title}" />{title}</h3>\n'.format(**view))
+        if 'comment' in view:
+            stream.write(view['comment'])
+            stream.write('\n')
+        for line in [
+                'The view is generated from the following query:',
+                '<blockquote>',
+                view['query-string'],
+                '</blockquote>',
+                ]:
+            stream.write(line)
+            stream.write('\n')
+
+    def _write_threads(self, threads, stream):
+        if not threads:
+            return
+        stream.write('<table>\n')
+        for thread in threads:
+            for message_display_data in thread:
+                stream.write((
+                    '<tr><td>{date}\n'
+                    '</td><td>{message-id-term}\n'
+                    '</td></tr>\n'
+                    '<tr><td>{from}\n'
+                    '</td><td>{subject}\n'
+                    '</td></tr>\n'
+                    ).format(**message_display_data))
+            if thread != threads[-1]:
+                stream.write('<tr><td colspan="2"><br /></td></tr>\n')
+        stream.write('</table>\n')
+
+    def _message_display_data(self, *args, **kwargs):
+        running_data, display_data = super(
+            HtmlPage, self)._message_display_data(
+                *args, **kwargs)
+        if 'subject' in display_data and 'message-id' in display_data:
+            d = {
+                'message-id': quote(display_data['message-id']),
+                'subject': display_data['subject'],
+                }
+            display_data['subject'] = (
+                '<a href="http://mid.gmane.org/{message-id}">{subject}</a>'
+                ).format(**d)
+        return (running_data, display_data)
+
+
+_PAGES['text'] = Page()
+_PAGES['html'] = HtmlPage(
+    header='''<?xml version="1.0" encoding="utf-8" ?>
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+<head>
+<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
+<title>Notmuch Patches</title>
+</head>
+<body>
+<h2>Notmuch Patches</h2>
+Generated: {date}<br />
+For more infomation see <a href="http://notmuchmail.org/nmbug">nmbug</a>
+<h3>Views</h3>
+'''.format(date=datetime.datetime.utcnow().date()),
+    footer='</body>\n</html>\n',
+    )
 
-# parse command line arguments
 
 parser = argparse.ArgumentParser()
 parser.add_argument('--text', help='output plain text format',
@@ -177,34 +252,9 @@ else:
     import notmuch
 
 if args.text:
-    output_format = 'text'
+    page = _PAGES['text']
 else:
-    output_format = 'html'
-
-# main program
+    page = _PAGES['html']
 
 db = notmuch.Database(mode=notmuch.Database.MODE.READ_ONLY)
-
-if output_format == 'html':
-    print('''<?xml version="1.0" encoding="utf-8" ?>
-<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
-<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
-<head>
-<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
-<title>Notmuch Patches</title>
-</head>
-<body>
-<h2>Notmuch Patches</h2>
-Generated: {date}<br />
-For more infomation see <a href="http://notmuchmail.org/nmbug">nmbug</a>
-<h3>Views</h3>
-<ul>'''.format(date=datetime.datetime.utcnow().date()))
-    for view in config['views']:
-        print('<li><a href="#%(title)s">%(title)s</a></li>' % view)
-    print('</ul>')
-
-for view in config['views']:
-    print_view(database=db, **view)
-
-if output_format == 'html':
-    print('</body>\n</html>')
+page.write(database=db, views=config['views'])
-- 
1.8.5.2.8.g0f6c0d1

[PATCH 11/17] nmbug-status: Normalize table HTML indentation

[W. Trevor King]

I don't think I've ever seen '</td><td>{value}\n' before :p.  The new
formatting avoids mixing tag levels and content across lines.
---
 devel/nmbug/nmbug-status | 14 ++++++++------
 1 file changed, 8 insertions(+), 6 deletions(-)

diff --git a/devel/nmbug/nmbug-status b/devel/nmbug/nmbug-status
index 7778029..fdafda0 100755
--- a/devel/nmbug/nmbug-status
+++ b/devel/nmbug/nmbug-status
@@ -179,12 +179,14 @@ class HtmlPage (Page):
         for thread in threads:
             for message_display_data in thread:
                 stream.write((
-                    '<tr><td>{date}\n'
-                    '</td><td>{message-id-term}\n'
-                    '</td></tr>\n'
-                    '<tr><td>{from}\n'
-                    '</td><td>{subject}\n'
-                    '</td></tr>\n'
+                    '<tr>\n'
+                    '  <td>{date}</td>\n'
+                    '  <td>{message-id-term}</td>\n'
+                    '</tr>\n'
+                    '<tr>\n'
+                    '  <td>{from}</td>\n'
+                    '  <td>{subject}</td>\n'
+                    '</tr>\n'
                     ).format(**message_display_data))
             if thread != threads[-1]:
                 stream.write('<tr><td colspan="2"><br /></td></tr>\n')
-- 
1.8.5.2.8.g0f6c0d1

[PATCH 12/17] nmbug-status: Convert from XHTML 1.0 to HTML 5

[W. Trevor King]

HTML 5 for the win :).  I also de-namespaced the language; the HTML 5
spec allows a vestigial xml:lang attribute, but it's a no-op [1], so I
stripped it.

This shouldn't break anything at tethera, which already serves the
status as text/html:

  $ wget -S http://nmbug.tethera.net/status/
  --2014-02-02 21:20:39--  http://nmbug.tethera.net/status/
  Resolving nmbug.tethera.net... 87.98.215.224
  Connecting to nmbug.tethera.net|87.98.215.224|:80... connected.
  HTTP request sent, awaiting response...
    HTTP/1.1 200 OK
    Vary: Accept-Encoding
    Content-Type: text/html
  ...

This also matches the Content-Type in the generated HTML's http-equiv
meta.

[1]: http://www.w3.org/TR/html5/dom.html#the-lang-and-xml:lang-attributes
---
 devel/nmbug/nmbug-status | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/devel/nmbug/nmbug-status b/devel/nmbug/nmbug-status
index fdafda0..b67b86b 100755
--- a/devel/nmbug/nmbug-status
+++ b/devel/nmbug/nmbug-status
@@ -209,9 +209,8 @@ class HtmlPage (Page):
 
 _PAGES['text'] = Page()
 _PAGES['html'] = HtmlPage(
-    header='''<?xml version="1.0" encoding="utf-8" ?>
-<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
-<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+    header='''<!DOCTYPE html>
+<html lang="en">
 <head>
 <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
 <title>Notmuch Patches</title>
-- 
1.8.5.2.8.g0f6c0d1

[PATCH 13/17] nmbug-status: Encode output using the user's locale

[W. Trevor King]

Instead of always writing UTF-8, allow the user to configure the
output encoding using their locale.  This is useful for previewing
output in the terminal, for poor souls that don't use UTF-8 locales
;).
---
 devel/nmbug/nmbug-status | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/devel/nmbug/nmbug-status b/devel/nmbug/nmbug-status
index b67b86b..e86ea2a 100755
--- a/devel/nmbug/nmbug-status
+++ b/devel/nmbug/nmbug-status
@@ -70,7 +70,7 @@ class Page (object):
                 byte_stream = sys.stdout.buffer
             except AttributeError:  # Python 2
                 byte_stream = sys.stdout
-            stream = codecs.getwriter(encoding='UTF-8')(stream=byte_stream)
+            stream = codecs.getwriter(encoding=_ENCODING)(stream=byte_stream)
         self._write_header(views=views, stream=stream)
         for view in views:
             self._write_view(database=database, view=view, stream=stream)
@@ -212,7 +212,7 @@ _PAGES['html'] = HtmlPage(
     header='''<!DOCTYPE html>
 <html lang="en">
 <head>
-<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
+<meta http-equiv="Content-Type" content="text/html; charset={encoding}" />
 <title>Notmuch Patches</title>
 </head>
 <body>
@@ -220,7 +220,8 @@ _PAGES['html'] = HtmlPage(
 Generated: {date}<br />
 For more infomation see <a href="http://notmuchmail.org/nmbug">nmbug</a>
 <h3>Views</h3>
-'''.format(date=datetime.datetime.utcnow().date()),
+'''.format(date=datetime.datetime.utcnow().date(),
+           encoding=_ENCODING),
     footer='</body>\n</html>\n',
     )
 
-- 
1.8.5.2.8.g0f6c0d1

[PATCH 14/17] nmbug-status: Anchor with h3 ids instead of a names

[W. Trevor King]

HTML 5 allows id attributes on all HTML elements [1], but restricts
names to particular cases [2].  Attaching the id attribute to the h3
element allows us to drop the anchor a element altogether.

[1]: http://www.w3.org/TR/html5/dom.html#the-id-attribute
[2]: http://www.w3.org/TR/html5/index.html#attributes-1
---
 devel/nmbug/nmbug-status | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/devel/nmbug/nmbug-status b/devel/nmbug/nmbug-status
index e86ea2a..3597389 100755
--- a/devel/nmbug/nmbug-status
+++ b/devel/nmbug/nmbug-status
@@ -159,7 +159,7 @@ class HtmlPage (Page):
         stream.write('</ul>\n')
 
     def _write_view_header(self, view, stream):
-        stream.write('<h3><a name="{title}" />{title}</h3>\n'.format(**view))
+        stream.write('<h3 id="{title}">{title}</h3>\n'.format(**view))
         if 'comment' in view:
             stream.write(view['comment'])
             stream.write('\n')
-- 
1.8.5.2.8.g0f6c0d1

[PATCH 15/17] nmbug-status: Quote the title when using it as an id

[W. Trevor King]

Also allow manual id overrides from the JSON config.  Quoting avoids
errors like:

   Bad value '#Possible bugs' for attribute href on element a:
   Whitespace in fragment component. Use %20 in place of spaces.

from http://validator.w3.org.
---
 devel/nmbug/nmbug-status | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/devel/nmbug/nmbug-status b/devel/nmbug/nmbug-status
index 3597389..9bdb34f 100755
--- a/devel/nmbug/nmbug-status
+++ b/devel/nmbug/nmbug-status
@@ -154,12 +154,14 @@ class HtmlPage (Page):
         super(HtmlPage, self)._write_header(views=views, stream=stream)
         stream.write('<ul>\n')
         for view in views:
+            if 'id' not in view:
+                view['id'] = quote(view['title'])
             stream.write(
-                '<li><a href="#{title}">{title}</a></li>\n'.format(**view))
+                '<li><a href="#{id}">{title}</a></li>\n'.format(**view))
         stream.write('</ul>\n')
 
     def _write_view_header(self, view, stream):
-        stream.write('<h3 id="{title}">{title}</h3>\n'.format(**view))
+        stream.write('<h3 id="{id}">{title}</h3>\n'.format(**view))
         if 'comment' in view:
             stream.write(view['comment'])
             stream.write('\n')
-- 
1.8.5.2.8.g0f6c0d1

[PATCH 16/17] nmbug-status: Use <code> and <p> markup where appropriate

[W. Trevor King]

* Wrap free text in <p> tags.
* Convert <blockquote> to <p><code> for query strings.
* Wrap message-id-term (id:"...") in <code>.

The <code> tags get nicer default markup (smaller monospace font) for
notmuch query terms [1].  The <p> tags don't have much effect without
attached CSS, but bare text (phrasing content [2]) in <body> (which
expects flow content [3,4]) feels wrong.

[1]: http://www.w3.org/TR/html5/text-level-semantics.html#the-code-element
[2]: http://www.w3.org/TR/html5/dom.html#phrasing-content-1
[3]: http://www.w3.org/TR/html5/dom.html#flow-content-1
[4]: http://www.w3.org/TR/html5/sections.html#the-body-element
---
 devel/nmbug/nmbug-status | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/devel/nmbug/nmbug-status b/devel/nmbug/nmbug-status
index 9bdb34f..21fb481 100755
--- a/devel/nmbug/nmbug-status
+++ b/devel/nmbug/nmbug-status
@@ -162,14 +162,18 @@ class HtmlPage (Page):
 
     def _write_view_header(self, view, stream):
         stream.write('<h3 id="{id}">{title}</h3>\n'.format(**view))
+        stream.write('<p>\n')
         if 'comment' in view:
             stream.write(view['comment'])
             stream.write('\n')
         for line in [
                 'The view is generated from the following query:',
-                '<blockquote>',
+                '</p>',
+                '<p>',
+                '  <code>',
                 view['query-string'],
-                '</blockquote>',
+                '  </code>',
+                '</p>',
                 ]:
             stream.write(line)
             stream.write('\n')
@@ -183,7 +187,7 @@ class HtmlPage (Page):
                 stream.write((
                     '<tr>\n'
                     '  <td>{date}</td>\n'
-                    '  <td>{message-id-term}</td>\n'
+                    '  <td><code>{message-id-term}</code></td>\n'
                     '</tr>\n'
                     '<tr>\n'
                     '  <td>{from}</td>\n'
@@ -219,8 +223,10 @@ _PAGES['html'] = HtmlPage(
 </head>
 <body>
 <h2>Notmuch Patches</h2>
+<p>
 Generated: {date}<br />
 For more infomation see <a href="http://notmuchmail.org/nmbug">nmbug</a>
+</p>
 <h3>Views</h3>
 '''.format(date=datetime.datetime.utcnow().date(),
            encoding=_ENCODING),
-- 
1.8.5.2.8.g0f6c0d1

[PATCH 17/17] nmbug-status: Color threads in HTML output

[W. Trevor King]

Add tbody sections so we don't have to color every row.  Multiple
tbody sections are allowed [1].  Use CSS 3's nth-child to handle
even/odd coloring (skipping the spacer rows) [2], which is supported
on the major browsers [3].

border-spacing is from CCS 2.1 [4,5].  I'm using it to avoid
whitespace between td cells.

border-radius is from CCS 3 [6,7].  I'm using it to make the colored
sections a bit less harsh.  I tried adding rounded borders to the
tbody itself doesn't work, but I couldn't get that to work without
setting the tbody's display to 'block'.  That rounded the corners, but
collapsed the cell spacing (e.g. columns were no longer aligned).
This commit's by-corner-td approach is not particularly elegant, but
it works.  The td padding entries just ensure that the cell body is
suitably far from the edges that it doesn't fall outside of the
rounded corners.

The doubled-braces are escapes from Python's str.format.

[1]: http://www.w3.org/TR/html5/tabular-data.html#the-table-element
[2]: http://www.w3.org/TR/css3-selectors/#nth-child-pseudo
[3]: https://developer.mozilla.org/en-US/docs/Web/CSS/:nth-child#Browser_compatibility
[4]: http://www.w3.org/TR/CSS2/tables.html#propdef-border-spacing
[5]: https://developer.mozilla.org/en-US/docs/Web/CSS/border-spacing#Browser_compatibility
[6]: http://www.w3.org/TR/css3-background/#the-border-radius
[7]: https://developer.mozilla.org/en-US/docs/Web/CSS/border-radius#Browser_compatibility
---
 devel/nmbug/nmbug-status | 52 ++++++++++++++++++++++++++++++++++++++----------
 1 file changed, 41 insertions(+), 11 deletions(-)

diff --git a/devel/nmbug/nmbug-status b/devel/nmbug/nmbug-status
index 21fb481..f1adf65 100755
--- a/devel/nmbug/nmbug-status
+++ b/devel/nmbug/nmbug-status
@@ -183,19 +183,22 @@ class HtmlPage (Page):
             return
         stream.write('<table>\n')
         for thread in threads:
+            stream.write('  <tbody>\n')
             for message_display_data in thread:
                 stream.write((
-                    '<tr>\n'
-                    '  <td>{date}</td>\n'
-                    '  <td><code>{message-id-term}</code></td>\n'
-                    '</tr>\n'
-                    '<tr>\n'
-                    '  <td>{from}</td>\n'
-                    '  <td>{subject}</td>\n'
-                    '</tr>\n'
+                    '    <tr>\n'
+                    '      <td>{date}</td>\n'
+                    '      <td><code>{message-id-term}</code></td>\n'
+                    '    </tr>\n'
+                    '    <tr>\n'
+                    '      <td>{from}</td>\n'
+                    '      <td>{subject}</td>\n'
+                    '    </tr>\n'
                     ).format(**message_display_data))
+            stream.write('  </tbody>\n')
             if thread != threads[-1]:
-                stream.write('<tr><td colspan="2"><br /></td></tr>\n')
+                stream.write(
+                    '  <tbody><tr><td colspan="2"><br /></td></tr></tbody>\n')
         stream.write('</table>\n')
 
     def _message_display_data(self, *args, **kwargs):
@@ -218,8 +221,35 @@ _PAGES['html'] = HtmlPage(
     header='''<!DOCTYPE html>
 <html lang="en">
 <head>
-<meta http-equiv="Content-Type" content="text/html; charset={encoding}" />
-<title>Notmuch Patches</title>
+  <meta http-equiv="Content-Type" content="text/html; charset={encoding}" />
+  <title>Notmuch Patches</title>
+  <style media="screen" type="text/css">
+    table {{
+      border-spacing: 0;
+    }}
+    td {{
+      padding-left: 1em;
+      padding-right: 1em;
+    }}
+    tr:first-child td:first-child {{
+      border-top-left-radius: 1em;
+    }}
+    tr:first-child td:last-child {{
+      border-top-right-radius: 1em;
+    }}
+    tr:last-child td:first-child {{
+      border-bottom-left-radius: 1em;
+    }}
+    tr:last-child td:last-child {{
+      border-bottom-right-radius: 1em;
+    }}
+    tbody:nth-child(4n+1) tr td {{
+      background-color: #EFF;
+    }}
+    tbody:nth-child(4n+3) tr td {{
+      background-color: #FEF;
+    }}
+  </style>
 </head>
 <body>
 <h2>Notmuch Patches</h2>
-- 
1.8.5.2.8.g0f6c0d1

Re: [PATCH 00/17] nmbug-status: Python-3-compabitility and general refactoring

[Tomi Ollila]

On Mon, Feb 03 2014, "W. Trevor King" <wking@tremily.us> wrote:

> I noticed that nmbug-status was written in Python :), but not
> compatible with Python 3 :(.  I started cleaning up a few print
> statements, but this quickly turned into a more general refactoring.
> Let me know if this is too much to bite off at once.  I tried to keep
> each patch fairly contained, but the Page / HtmlPage addition is still
> pretty big.  Despite increasing the size of this module by almost 50%,
> I think my final version is more readable.  However, it's always
> easier to read your own code, so feel free to tell me that this is a
> step in the completely wrong direction ;).
>
> Also anyone with asthetic sensibilities is free to pick nicer colors
> in the final patch.  I'm too partial to the EFF permutation symmetry
> to be able to pick other colors myself ;).

Nice series. One problem with python 2.6 though:

Traceback (most recent call last):
  File "devel/nmbug/nmbug-status", line 300, in <module>
    page.write(database=db, views=config['views'])
  File "devel/nmbug/nmbug-status", line 76, in write
    self._write_view(database=database, view=view, stream=stream)
  File "devel/nmbug/nmbug-status", line 93, in _write_view
    threads = self._get_threads(messages=q.search_messages())
  File "devel/nmbug/nmbug-status", line 107, in _get_threads
    running_data=thread.running_data, message=message)
  File "devel/nmbug/nmbug-status", line 207, in _message_display_data
    *args, **kwargs)
  File "devel/nmbug/nmbug-status", line 132, in _message_display_data
    data['message-id-term'] = 'id:"{}"'.format(value)
ValueError: zero length field name in format

Using python 2.6.6

$ python -c 'print "{}".format("foo")'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ValueError: zero length field name in format
zsh: exit 1     python -c '"{}".format("foo")'

python -c 'print "{0}".format("foo")'
foo

With python 2.7.x python -c 'print "{}".format("foo")' works OK.

Argh, when I changed that one, next:

Traceback (most recent call last):
  File "devel/nmbug/nmbug-status", line 300, in <module>
    page.write(database=db, views=config['views'])
  File "devel/nmbug/nmbug-status", line 76, in write
    self._write_view(database=database, view=view, stream=stream)
  File "devel/nmbug/nmbug-status", line 95, in _write_view
    self._write_threads(threads=threads, stream=stream)
  File "devel/nmbug/nmbug-status", line 197, in _write_threads
    ).format(**message_display_data))
  File "/usr/lib64/python2.6/codecs.py", line 351, in write
    data, consumed = self.encode(object, self.errors)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u017b' in
  position 176: ordinal not in range(128)

This one is harder...

Tomi


>
> W. Trevor King (17):
>   nmbug-status: Convert to Python-3-compatible print functions
>   nmbug-status: Use email.utils instead of rfc822
>   nmbug-status: Decode Popen output using the user's locale
>   nmbug-status: Factor config-loading out into read_config
>   nmbug-status: Add metavars for --config and --get-query
>   nmbug-status: Consolidate functions and main code
>   nmbug-status: Don't require write access
>   nmbug-status: Consolidate HTML header printing
>   nmbug-status: Add a Python-3-compatible urllib.parse.quote import
>   nmbug-status: Add Page and HtmlPage for modular rendering
>   nmbug-status: Normalize table HTML indentation
>   nmbug-status: Convert from XHTML 1.0 to HTML 5
>   nmbug-status: Encode output using the user's locale
>   nmbug-status: Anchor with h3 ids instead of a names
>   nmbug-status: Quote the title when using it as an id
>   nmbug-status: Use <code> and <p> markup where appropriate
>   nmbug-status: Color threads in HTML output
>
>  devel/nmbug/nmbug-status | 412 ++++++++++++++++++++++++++++++-----------------
>  1 file changed, 261 insertions(+), 151 deletions(-)
>
> -- 
> 1.8.5.2.8.g0f6c0d1
>
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH 00/17] nmbug-status: Python-3-compabitility and general refactoring

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 1613 bytes --]

On Mon, Feb 03, 2014 at 11:10:23PM +0200, Tomi Ollila wrote:
>     data['message-id-term'] = 'id:"{}"'.format(value)
> ValueError: zero length field name in format

Oops, Python 2.6 still needs explicit indexes ('{0}', not '{}').  It's
an easy fix, so I'll queue it for v2.  You're still going to need
Python 2.7 or greater for collections.OrderedDict().  We could word
around that too, but do we really care about 2.6?  I don't expect that
the installed nmbug-status userbase is so large and backward that
upgrading to 2.7 will be that hard ;).  2.6 isn't even getting
security fixes anymore [1], so I think it's time to migrate :p.

>   File "devel/nmbug/nmbug-status", line 197, in _write_threads
>     ).format(**message_display_data))
>   File "/usr/lib64/python2.6/codecs.py", line 351, in write
>     data, consumed = self.encode(object, self.errors)
> UnicodeEncodeError: 'ascii' codec can't encode character u'\u017b' in
>   position 176: ordinal not in range(128)

Hmm.  __future__'s unicode_literals should be giving us a Unicode
target, so I'm not sure why we'd have trouble injecting Unicode.  This
works fine for me on Python 2.7 and 3.3.  Maybe you just have a funky
encoding?  What is your:

  $ locale
  LANG=en_US.UTF-8
  …
  $ python -c 'import locale, sys; print(locale.getpreferredencoding() or sys.getdefaultencoding())'
  UTF-8

Cheers,
Trevor

[1]: http://www.python.org/download/releases/2.6.9/

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 00/17] nmbug-status: Python-3-compabitility and general refactoring

[Tomi Ollila]

On Tue, Feb 04 2014, "W. Trevor King" <wking@tremily.us> wrote:

> On Mon, Feb 03, 2014 at 11:10:23PM +0200, Tomi Ollila wrote:
>>     data['message-id-term'] = 'id:"{}"'.format(value)
>> ValueError: zero length field name in format
>
> Oops, Python 2.6 still needs explicit indexes ('{0}', not '{}').  It's
> an easy fix, so I'll queue it for v2.  You're still going to need
> Python 2.7 or greater for collections.OrderedDict().  We could word
> around that too, but do we really care about 2.6?  I don't expect that
> the installed nmbug-status userbase is so large and backward that
> upgrading to 2.7 will be that hard ;).  2.6 isn't even getting
> security fixes anymore [1], so I think it's time to migrate :p.

Probably not many cares about 2.6; I already use argparse and I can add
that OrderedDict() too. Still {0} is easy enough to do :D. I am running
this notmuch & nmbug in Scientific Linux 6.2 machine which has python 2.6
-- and this is the only machine where I can review your nmbug-status
changes ;)

>>   File "devel/nmbug/nmbug-status", line 197, in _write_threads
>>     ).format(**message_display_data))
>>   File "/usr/lib64/python2.6/codecs.py", line 351, in write
>>     data, consumed = self.encode(object, self.errors)
>> UnicodeEncodeError: 'ascii' codec can't encode character u'\u017b' in
>>   position 176: ordinal not in range(128)
>
> Hmm.  __future__'s unicode_literals should be giving us a Unicode
> target, so I'm not sure why we'd have trouble injecting Unicode.  This
> works fine for me on Python 2.7 and 3.3.  Maybe you just have a funky
> encoding?  What is your:
>
>   $ locale
>   LANG=en_US.UTF-8
>   …
>   $ python -c 'import locale, sys; print(locale.getpreferredencoding() or sys.getdefaultencoding())'
>   UTF-8

LANG=en_US.UTF-8
all other LC_* variables en_US.UTF-8 except
LC_TIME=en_GB.utf8
LC_ALL empty (naturally)

python -c 'import locale; print(locale.getpreferredencoding())'
UTF-8
python -c 'import sys; print(sys.getdefaultencoding())'
ascii


anyway, if this doesn't resolve out and there is no resistance to require
python 2.7 (that means from David) I can hack around this to get this
reviewed.

>
> Cheers,
> Trevor

Tomi

>
> [1]: http://www.python.org/download/releases/2.6.9/

Re: [PATCH 00/17] nmbug-status: Python-3-compabitility and general refactoring

[David Bremner]

Tomi Ollila <tomi.ollila@iki.fi> writes:

> anyway, if this doesn't resolve out and there is no resistance to require
> python 2.7 (that means from David) I can hack around this to get this
> reviewed.

Currently the production copy of nmbug-status is running with python2.6
on Debian squeeze.  I _should_ upgrade that machine to wheezy, but I'm
not sure if there are any complications preventing that.

d

Re: [PATCH 00/17] nmbug-status: Python-3-compabitility and general refactoring

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 653 bytes --]

On Tue, Feb 04, 2014 at 09:07:45AM -0400, David Bremner wrote:
> Tomi Ollila writes:
> > anyway, if this doesn't resolve out and there is no resistance to require
> > python 2.7 (that means from David) I can hack around this to get this
> > reviewed.
>
> Currently the production copy of nmbug-status is running with python2.6
> on Debian squeeze.

That's two votes for Python 2.6 and none for later versions, so I'll
just try a bit harder for 2.6 compatibility ;).

Cheers,
Trevor

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 00/17] nmbug-status: Python-3-compabitility and general refactoring

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 2450 bytes --]

On Tue, Feb 04, 2014 at 12:30:30PM +0200, Tomi Ollila wrote:
> On Tue, Feb 04 2014, "W. Trevor King" wrote:
> > On Mon, Feb 03, 2014 at 11:10:23PM +0200, Tomi Ollila wrote:
> >>   File "devel/nmbug/nmbug-status", line 197, in _write_threads
> >>     ).format(**message_display_data))
> >>   File "/usr/lib64/python2.6/codecs.py", line 351, in write
> >>     data, consumed = self.encode(object, self.errors)
> >> UnicodeEncodeError: 'ascii' codec can't encode character u'\u017b' in
> >>   position 176: ordinal not in range(128)
> >
> > Hmm.  __future__'s unicode_literals should be giving us a Unicode
> > target, so I'm not sure why we'd have trouble injecting Unicode.  This
> > works fine for me on Python 2.7 and 3.3.  Maybe you just have a funky
> > encoding?  …
> 
> LANG=en_US.UTF-8
> all other LC_* variables en_US.UTF-8 except
> LC_TIME=en_GB.utf8
> LC_ALL empty (naturally)
> 
> python -c 'import locale; print(locale.getpreferredencoding())'
> UTF-8
> python -c 'import sys; print(sys.getdefaultencoding())'
> ascii

That's very strange.  On Python 2.6.9, with the same encodings:

  >>> from __future__ import unicode_literals
  >>> import codecs
  >>> import locale
  >>> import sys
  >>> print(locale.getpreferredencoding())  # same as yours
  UTF-8
  >>> print(sys.getdefaultencoding())  # same as yours
  ascii
  >>> _ENCODING = locale.getpreferredencoding() or sys.getdefaultencoding()
  >>> print(_ENCODING)  # double-check default encodings
  UTF-8
  >>> byte_stream = sys.stdout  # copied from Page.write
  >>> stream = codecs.getwriter(encoding=_ENCODING)(stream=byte_stream)
  >>> data = {'from': '\u017b'}  # fake the troublesome data
  >>> print(type(data['from']))  # double-check unicode_literals
  <type 'unicode'>
  >>> string = '  <td>{from}</td>\n'.format(**data)
  >>> stream.write(string)
    <td>Ż</td>

It looks like you'll have the same _ENCODING as I do (UTF-8).  That
means your stream should be wrapped in a UTF-8 StreamWriter, so I
don't understand why it's converting to ASCII.  Can you run through
the above on your troublesome machine and confirm that stream.write()
is still raising the exception?  If it doesn't work, can you just
paste that whole run in your next email?

Thanks,
Trevor

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 00/17] nmbug-status: Python-3-compabitility and general refactoring

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 913 bytes --]

On Tue, Feb 04, 2014 at 07:50:32AM -0800, W. Trevor King wrote:
> That's two votes for Python 2.6 and none for later versions, so I'll
> just try a bit harder for 2.6 compatibility ;).

As a v2 preview, and to help with further review, I've posted my
current v1+ version of this branch at:

  git://tremily.us/notmuch.git nmbug-status-python3

Changes since v1:

* '{}' → '{0}' for Python 2's str.format().
* Added an OrderedDict stub for Python 2.6 (and earlier, but they'll
  be missing json).

This v1+ works on my local box with Python 2.6.9.  I can push it as v2
now, but it's a long series and I don't want to spam the list just for
Python 2.6 fixups.  If folks want a v2 pushed to the list, just let me
know.

Cheers,
Trevor

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 00/17] nmbug-status: Python-3-compabitility and general refactoring

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 1203 bytes --]

On Tue, Feb 04, 2014 at 12:30:30PM +0200, Tomi Ollila wrote:
> On Tue, Feb 04 2014, W. Trevor King wrote:
> > On Mon, Feb 03, 2014 at 11:10:23PM +0200, Tomi Ollila wrote:
> >>   File "devel/nmbug/nmbug-status", line 197, in _write_threads
> >>     ).format(**message_display_data))
> >>   File "/usr/lib64/python2.6/codecs.py", line 351, in write
> >>     data, consumed = self.encode(object, self.errors)
> >> UnicodeEncodeError: 'ascii' codec can't encode character u'\u017b' in
> >>   position 176: ordinal not in range(128)
> > …
> 
> LANG=en_US.UTF-8

My current guess is that LANG is set in your terminal, but that you
ran nmbug-status from a cron job where it wasn't set.  If that's the
case, try adding:

  export LANG=en_US.UTF-8

to your cron job (or replace with whichever encoding you like).

We don't care about the preferred language yet, but we could [1].
There's actually not all that much that needs translating for
nmbug-status.

Cheers,
Trevor

[1]: http://docs.python.org/3/library/gettext.html

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 00/17] nmbug-status: Python-3-compabitility and general refactoring

[Tomi Ollila]

On Tue, Feb 04 2014, "W. Trevor King" <wking@tremily.us> wrote:

> On Tue, Feb 04, 2014 at 12:30:30PM +0200, Tomi Ollila wrote:
>> On Tue, Feb 04 2014, W. Trevor King wrote:
>> > On Mon, Feb 03, 2014 at 11:10:23PM +0200, Tomi Ollila wrote:
>> >>   File "devel/nmbug/nmbug-status", line 197, in _write_threads
>> >>     ).format(**message_display_data))
>> >>   File "/usr/lib64/python2.6/codecs.py", line 351, in write
>> >>     data, consumed = self.encode(object, self.errors)
>> >> UnicodeEncodeError: 'ascii' codec can't encode character u'\u017b' in
>> >>   position 176: ordinal not in range(128)
>> > …
>> 
>> LANG=en_US.UTF-8
>
> My current guess is that LANG is set in your terminal, but that you
> ran nmbug-status from a cron job where it wasn't set.  If that's the
> case, try adding:
>
>   export LANG=en_US.UTF-8
>
> to your cron job (or replace with whichever encoding you like).

Yes guess is good, but incorrect I run it on the command line.

Now I did the following:

$ git remote add tremily git://tremily.us/notmuch.git
$ git fetch --all
$ git checkout nmbug-status-python3

commit d64ef215b62fa74a96fdb4c93e1f6f7abf7c80c6

and then rerun nmbug-status -- and I am still having the same problem.
I will investigate this further...

> We don't care about the preferred language yet, but we could [1].
> There's actually not all that much that needs translating for
> nmbug-status.

Yes, that is unfortunate that David (Taavetti in Finnish) did an uneducated
choice by choosing english as the preferred language. It should always
have been written in Finnish so we would not have this conversation >;)

>
> Cheers,
> Trevor
>
> [1]: http://docs.python.org/3/library/gettext.html


Tomi

Re: [PATCH 00/17] nmbug-status: Python-3-compabitility and general refactoring

[Tomi Ollila]

On Tue, Feb 04 2014, "W. Trevor King" <wking@tremily.us> wrote:

>
>   >>> from __future__ import unicode_literals
>   >>> import codecs
>   >>> import locale
>   >>> import sys
>   >>> print(locale.getpreferredencoding())  # same as yours
>   UTF-8
>   >>> print(sys.getdefaultencoding())  # same as yours
>   ascii
>   >>> _ENCODING = locale.getpreferredencoding() or sys.getdefaultencoding()
>   >>> print(_ENCODING)  # double-check default encodings
>   UTF-8
>   >>> byte_stream = sys.stdout  # copied from Page.write
>   >>> stream = codecs.getwriter(encoding=_ENCODING)(stream=byte_stream)
>   >>> data = {'from': '\u017b'}  # fake the troublesome data
>   >>> print(type(data['from']))  # double-check unicode_literals
>   <type 'unicode'>
>   >>> string = '  <td>{from}</td>\n'.format(**data)
>   >>> stream.write(string)
>     <td>Ż</td>
>
> It looks like you'll have the same _ENCODING as I do (UTF-8).  That
> means your stream should be wrapped in a UTF-8 StreamWriter, so I
> don't understand why it's converting to ASCII.  Can you run through
> the above on your troublesome machine and confirm that stream.write()
> is still raising the exception?  If it doesn't work, can you just
> paste that whole run in your next email?

I don't know what to paste, so i paste this:

$ python
Python 2.6.6 (r266:84292, Nov 21 2013, 12:39:37) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> data = {'from': '\u017b'}
>>> print(type(data['from'])) 
<type 'str'>
>>> string = '  <td>{from}</td>\n'.format(**data)
>>> print string
  <td>\u017b</td>

and then:

>>> data = {'from': u'\u017b'}
>>> print(type(data['from'])) 
<type 'unicode'>
>>> string = '  <td>{from}</td>\n'.format(**data)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\u017b' in
>>> position 0: ordinal not in range(128)

... and ...

>>> import os
>>> print os.environ['LANG']
en_US.UTF-8


> Thanks,
> Trevor


Tomi

Re: [PATCH 00/17] nmbug-status: Python-3-compabitility and general refactoring

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 2845 bytes --]

On Tue, Feb 04, 2014 at 08:40:18PM +0200, Tomi Ollila wrote:
> On Tue, Feb 04 2014, W. Trevor King wrote:
> >
> >   >>> from __future__ import unicode_literals
> >   >>> import codecs
> >   >>> import locale
> >   >>> import sys
> >   >>> print(locale.getpreferredencoding())  # same as yours
> >   UTF-8
> >   >>> print(sys.getdefaultencoding())  # same as yours
> >   ascii
> >   >>> _ENCODING = locale.getpreferredencoding() or sys.getdefaultencoding()
> >   >>> print(_ENCODING)  # double-check default encodings
> >   UTF-8
> >   >>> byte_stream = sys.stdout  # copied from Page.write
> >   >>> stream = codecs.getwriter(encoding=_ENCODING)(stream=byte_stream)
> >   >>> data = {'from': '\u017b'}  # fake the troublesome data
> >   >>> print(type(data['from']))  # double-check unicode_literals
> >   <type 'unicode'>
> >   >>> string = '  <td>{from}</td>\n'.format(**data)
> >   >>> stream.write(string)
> >     <td>Ż</td>
> >
> > It looks like you'll have the same _ENCODING as I do (UTF-8).  That
> > means your stream should be wrapped in a UTF-8 StreamWriter, so I
> > don't understand why it's converting to ASCII.  Can you run through
> > the above on your troublesome machine and confirm that stream.write()
> > is still raising the exception?  If it doesn't work, can you just
> > paste that whole run in your next email?
> 
> I don't know what to paste, so i paste this:
> 
> $ python
> Python 2.6.6 (r266:84292, Nov 21 2013, 12:39:37) 
> [GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.

It looks like you left out:

  from __future__ import unicode_literals

Can you try again with that line as the first command?

> >>> data = {'from': '\u017b'}
> >>> print(type(data['from'])) 
> <type 'str'>

which is why your data is a 'str' and not a 'unicode' instance.

> >>> string = '  <td>{from}</td>\n'.format(**data)
> >>> print string
>   <td>\u017b</td>
> 
> and then:
> 
> >>> data = {'from': u'\u017b'}

This works around the lack of unicode_literals with an explicit u''.

> >>> print(type(data['from'])) 
> <type 'unicode'>
> >>> string = '  <td>{from}</td>\n'.format(**data)
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> UnicodeEncodeError: 'ascii' codec can't encode character u'\u017b' in

However, without unicode_literals or an explicit u'', you're format
string '…{from}' is a str (it should be a 'unicode' instance with
unicode_literals).

> >>> import os
> >>> print os.environ['LANG']
> en_US.UTF-8

That's good anyway ;).  Thanks for digging into this :).

Cheers,
Trevor

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 00/17] nmbug-status: Python-3-compabitility and general refactoring

[Tomi Ollila]

On Tue, Feb 04 2014, "W. Trevor King" <wking@tremily.us> wrote:

>> 
>> I don't know what to paste, so i paste this:
>> 
>> $ python
>> Python 2.6.6 (r266:84292, Nov 21 2013, 12:39:37) 
>> [GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] on linux2
>> Type "help", "copyright", "credits" or "license" for more information.
>
> It looks like you left out:
>
>   from __future__ import unicode_literals
>
> Can you try again with that line as the first command?
>
>> >>> data = {'from': '\u017b'}
>> >>> print(type(data['from'])) 
>> <type 'str'>
>
> which is why your data is a 'str' and not a 'unicode' instance.
>
>> >>> string = '  <td>{from}</td>\n'.format(**data)
>> >>> print string
>>   <td>\u017b</td>

Quick update before getting to sleep:

$ python 
Python 2.6.6 (r266:84292, Nov 21 2013, 12:39:37) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from __future__ import print_function
>>> from __future__ import unicode_literals
>>> data = {'from': '\u017b'}
>>> print(type(data['from']))
<type 'unicode'>
>>> string = '  <td>{from}</td>\n'.format(**data)
>>> print(string)
  <td>Ż</td>

so getting success there

$ PYTHONPATH=$PWD/bindings/python/ ./devel/nmbug/nmbug-status
...
Traceback (most recent call last):
  File "devel/nmbug/nmbug-status", line 318, in <module>
    page.write(database=db, views=config['views'])
  File "devel/nmbug/nmbug-status", line 94, in write
    self._write_view(database=database, view=view, stream=stream)
  File "devel/nmbug/nmbug-status", line 113, in _write_view
    self._write_threads(threads=threads, stream=stream)
  File "devel/nmbug/nmbug-status", line 215, in _write_threads
    ).format(**message_display_data))
  File "/usr/lib64/python2.6/codecs.py", line 351, in write
    data, consumed = self.encode(object, self.errors)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u017b' in
position 176: ordinal not in range(128)


I'll dig deeper tomorrow.


Tomi

Re: [PATCH 00/17] nmbug-status: Python-3-compabitility and general refactoring

[Tomi Ollila]

On Tue, Feb 04 2014, Tomi Ollila <tomi.ollila@iki.fi> wrote:

> On Tue, Feb 04 2014, "W. Trevor King" <wking@tremily.us> wrote:
>
>
> $ PYTHONPATH=$PWD/bindings/python/ ./devel/nmbug/nmbug-status
> ...
> Traceback (most recent call last):
>   File "devel/nmbug/nmbug-status", line 318, in <module>
>     page.write(database=db, views=config['views'])
>   File "devel/nmbug/nmbug-status", line 94, in write
>     self._write_view(database=database, view=view, stream=stream)
>   File "devel/nmbug/nmbug-status", line 113, in _write_view
>     self._write_threads(threads=threads, stream=stream)
>   File "devel/nmbug/nmbug-status", line 215, in _write_threads
>     ).format(**message_display_data))
>   File "/usr/lib64/python2.6/codecs.py", line 351, in write
>     data, consumed = self.encode(object, self.errors)
> UnicodeEncodeError: 'ascii' codec can't encode character u'\u017b' in
> position 176: ordinal not in range(128)
>
>
> I'll dig deeper tomorrow.

PEBKAC. I (actually) used wrapper to run nmbug-status -- to run right
notmuch binary. While it mostly sets environment to be right it internally
uses LC_ALL=C for consistency -- and the wrapper forgot to restore
LC_ALL before running nmbug-status.

(that also confirms that the suggestion you made I used cron was somewhat
accurate ;)

After I fixed the wrapper the program works fine on python 2.6.6.

I have some comments on the new look -- which is pretty nice btw.

now every other thread is background-colored (girly) pink (fef?)
and every other light blue (eff). for me the light blue is not
visible (there are no body background color set?)

... hmm, i added 'body { background-color: #FFF; }' to the page and
still light blue is not visible on my screen (checked w/ #f0f that
the above line actually has some effect)... #DFF for the light-blue
works better for me.

I think rounded corners are nice but IMHO the round radius is too big
-- using 0.5mm for the radius would IMHO be better.

>
> Tomi

Tomi

Re: [PATCH 00/17] nmbug-status: Python-3-compabitility and general refactoring

[Tomi Ollila]

On Wed, Feb 05 2014, Tomi Ollila <tomi.ollila@iki.fi> wrote:

> I have some comments on the new look -- which is pretty nice btw.
>
> now every other thread is background-colored (girly) pink (fef?)
> and every other light blue (eff). for me the light blue is not
> visible (there are no body background color set?)
>
> ... hmm, i added 'body { background-color: #FFF; }' to the page and
> still light blue is not visible on my screen (checked w/ #f0f that
> the above line actually has some effect)... #DFF for the light-blue
> works better for me.
>
> I think rounded corners are nice but IMHO the round radius is too big
> -- using 0.5mm for the radius would IMHO be better.

Forgot to mention that the <, >, &  etc. characters could also be encoded
in this patch series (to &lt;, &gt;, &amp; etc.)

>
> Tomi

Tomi

Re: [PATCH 00/17] nmbug-status: Python-3-compabitility and general refactoring

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 1051 bytes --]

On Wed, Feb 05, 2014 at 05:00:45PM +0200, Tomi Ollila wrote:
> After I fixed the wrapper the program works fine on python 2.6.6.

:)

> now every other thread is background-colored (girly) pink (fef?)
> and every other light blue (eff). for me the light blue is not
> visible (there are no body background color set?)

#EFF is visible to me, but it is light.

> #DFF for the light-blue works better for me.

I'm fine with that.  As I said in my cover letter, I'm happy to have
alternative color schemes suggested.  Does anyone else want to chime
in on better defaults?  This should probably also be configurable via
status-config.json.  Maybe:

   { "style": {"html": {"thread-colors": ["#DFF", "#FEF"]}}}

?

> I think rounded corners are nice but IMHO the round radius is too
> big -- using 0.5mm for the radius would IMHO be better.

No problem.

Cheers,
Trevor

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 00/17] nmbug-status: Python-3-compabitility and general refactoring

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 542 bytes --]

On Wed, Feb 05, 2014 at 05:24:56PM +0200, Tomi Ollila wrote:
> Forgot to mention that the <, >, & etc. characters could also be
> encoded in this patch series (to &lt;, &gt;, &amp; etc.)

Good point.  I'll xml.sax.saxutils.escape the message-id, from, and
subject at the end of HtmlPage._message_display_data, and squash that
into the Page / HtmlPage commit.

Thanks,
Trevor

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 00/17] nmbug-status: Python-3-compabitility and general refactoring

[Tomi Ollila]

On Wed, Feb 05 2014, "W. Trevor King" <wking@tremily.us> wrote:

> On Wed, Feb 05, 2014 at 05:00:45PM +0200, Tomi Ollila wrote:
>> After I fixed the wrapper the program works fine on python 2.6.6.
>
> :)
>
>> now every other thread is background-colored (girly) pink (fef?)
>> and every other light blue (eff). for me the light blue is not
>> visible (there are no body background color set?)
>
> #EFF is visible to me, but it is light.
>
>> #DFF for the light-blue works better for me.
>
> I'm fine with that.  As I said in my cover letter, I'm happy to have
> alternative color schemes suggested.  Does anyone else want to chime
> in on better defaults?  This should probably also be configurable via
> status-config.json.  Maybe:
>
>    { "style": {"html": {"thread-colors": ["#DFF", "#FEF"]}}}

Maybe we can persuade David to test-run new notmuch-status in
his production server before it is pushed to notmuch master, and
bikeshed the style in http://nmbug.tethera.net/status/

> ?

probably no-one is interested to configure those later (if we even
get bikeshed comments... ;)

Tomi

>> I think rounded corners are nice but IMHO the round radius is too
>> big -- using 0.5mm for the radius would IMHO be better.
>
> No problem.
>
> Cheers,
> Trevor

Re: [PATCH 00/17] nmbug-status: Python-3-compabitility and general refactoring

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 1385 bytes --]

On Thu, Feb 06, 2014 at 12:54:56AM +0200, Tomi Ollila wrote:
> On Wed, Feb 05 2014, "W. Trevor King" wrote:
> > On Wed, Feb 05, 2014 at 05:00:45PM +0200, Tomi Ollila wrote:
> >> #DFF for the light-blue works better for me.
> >
> > I'm fine with that.  As I said in my cover letter, I'm happy to have
> > alternative color schemes suggested.  Does anyone else want to chime
> > in on better defaults?  This should probably also be configurable via
> > status-config.json.  Maybe:
> >
> >    { "style": {"html": {"thread-colors": ["#DFF", "#FEF"]}}}
> 
> Maybe we can persuade David to test-run new notmuch-status in
> his production server before it is pushed to notmuch master, and
> bikeshed the style in http://nmbug.tethera.net/status/

I'll remove this once the series lands, but I've currently got a
preview up at http://tremily.us/status.html

> probably no-one is interested to configure those later (if we even
> get bikeshed comments... ;)

I suppose we can revisit this when someone with an existing color
scheme decides to use nmbug and nmbug-status as their bugtracker (red
a gray for Red Hat?  Blue and yellow for Python? ;).  Until then, I'm
fine with hardcoding this.

Cheers,
Trevor

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 00/17] nmbug-status: Python-3-compabitility and general refactoring

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 1005 bytes --]

On Wed, Feb 05, 2014 at 07:31:27AM -0800, W. Trevor King wrote:
> On Wed, Feb 05, 2014 at 05:24:56PM +0200, Tomi Ollila wrote:
> > Forgot to mention that the <, >, & etc. characters could also be
> > encoded in this patch series (to &lt;, &gt;, &amp; etc.)
> 
> Good point.  I'll xml.sax.saxutils.escape the message-id, from, and
> subject at the end of HtmlPage._message_display_data, and squash that
> into the Page / HtmlPage commit.

I added this as a new commit instead of squashing (since the escaping
wasn't present in the original nmbug-status), and also pushed the
color and radius reductions [1] to my nmbug-status-python3 branch.

Is this at the point where I should push v2 to the list, or do we want
to wait for more feedback first?

Cheers,
Trevor

[1]: http://mid.gmane.org/20140205152738.GJ14197@odin.tremily.us

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 00/17] nmbug-status: Python-3-compabitility and general refactoring

[David Bremner]

[-- Attachment #1: Type: text/plain, Size: 483 bytes --]

"W. Trevor King" <wking@tremily.us> writes:

 > I'll remove this once the series lands, but I've currently got a
 > preview up at http://tremily.us/status.html

 yeah, the colour scheme is not my favourite. For the sake of being
 semi-constructive, I attach an alternative suggestion using colours
 #ffd96e and #bce.

 More importantly some of the threads are run together: e.g. 

 id:"4eddf2b1.4288980a.0b74.5557@mx.google.com" and
 id:"E1RYMYd-0003wu-Ea@thinkbox.jade-hamburg.de"


[-- Attachment #2: colour.png --]
[-- Type: image/png, Size: 89089 bytes --]

Re: [PATCH 00/17] nmbug-status: Python-3-compabitility and general refactoring

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 1003 bytes --]

On Sat, Feb 08, 2014 at 09:54:28AM -0400, David Bremner wrote:
> W. Trevor King writes:
> > I'll remove this once the series lands, but I've currently got a
> > preview up at http://tremily.us/status.html
> 
> yeah, the colour scheme is not my favourite. For the sake of being
> semi-constructive, I attach an alternative suggestion #ffd96e and
> #bce.

Works for me.

> More importantly some of the threads are run together: e.g. 
> 
> id:"4eddf2b1.4288980a.0b74.5557@mx.google.com" and
> id:"E1RYMYd-0003wu-Ea@thinkbox.jade-hamburg.de"

Both of those messages are part of the same thread
(thread:000000000000eaab on my box, but I doubt thread IDs are
portable), so I don't add a thread-separating space between them.
Would you like more message-separating space even between messages in
the same thread?

Cheers,
Trevor

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 00/17] nmbug-status: Python-3-compabitility and general refactoring

[Tomi Ollila]

On Sat, Feb 08 2014, "W. Trevor King" <wking@tremily.us> wrote:

> On Sat, Feb 08, 2014 at 09:54:28AM -0400, David Bremner wrote:
>> W. Trevor King writes:
>> > I'll remove this once the series lands, but I've currently got a
>> > preview up at http://tremily.us/status.html
>> 
>> yeah, the colour scheme is not my favourite. For the sake of being
>> semi-constructive, I attach an alternative suggestion #ffd96e and
>> #bce.
>
> Works for me.

I'm fine with that too.

>
>> More importantly some of the threads are run together: e.g. 
>> 
>> id:"4eddf2b1.4288980a.0b74.5557@mx.google.com" and
>> id:"E1RYMYd-0003wu-Ea@thinkbox.jade-hamburg.de"
>
> Both of those messages are part of the same thread
> (thread:000000000000eaab on my box, but I doubt thread IDs are
> portable), so I don't add a thread-separating space between them.
> Would you like more message-separating space even between messages in
> the same thread?

They're in the same (notmuch) thread. Easily testable with

notmuch search id:"4eddf2b1.4288980a.0b74.5557@mx.google.com" or id:"E1RYMYd-0003wu-Ea@thinkbox.jade-hamburg.de"

and therefore shown together. The commit 
02cafc84b4540cd0fb878121dcb3551b4ecd9fd1 made this happen always
(but without this this could have happened if there were no
other patches in new threads coming along in between.

I think the more space does not fix anything but it might help regognizing
message boundaries a bit (and thus would be a nice feature).

> Cheers,
> Trevor

Tomi

Re: [PATCH 00/17] nmbug-status: Python-3-compabitility and general refactoring

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 1735 bytes --]

On Sat, Feb 08, 2014 at 08:29:41PM +0200, Tomi Ollila wrote:
> On Sat, Feb 08 2014, W. Trevor King wrote:
> > On Sat, Feb 08, 2014 at 09:54:28AM -0400, David Bremner wrote:
> >> W. Trevor King writes:
> >> > I'll remove this once the series lands, but I've currently got a
> >> > preview up at http://tremily.us/status.html
> >> 
> >> yeah, the colour scheme is not my favourite. For the sake of being
> >> semi-constructive, I attach an alternative suggestion #ffd96e and
> >> #bce.
> >
> > Works for me.
> 
> I'm fine with that too.

Updated in my example and nmbug-status-python3 branch.

> >> More importantly some of the threads are run together: e.g. 
> >> 
> >> id:"4eddf2b1.4288980a.0b74.5557@mx.google.com" and
> >> id:"E1RYMYd-0003wu-Ea@thinkbox.jade-hamburg.de"
> >
> > Both of those messages are part of the same thread
> > (thread:000000000000eaab on my box, but I doubt thread IDs are
> > portable), so I don't add a thread-separating space between them.
> > Would you like more message-separating space even between messages
> > in the same thread?
> 
> …
> 
> I think the more space does not fix anything but it might help
> regognizing message boundaries a bit (and thus would be a nice
> feature).

Added to my example and nmbug-status-python3 branch.  I'm using CSS
for this new spacing, so non-CSS browsers (e.g. w3m) won't render the
inter-message spacing.  I think the colored-link (second message row)
vs. default id (first message row) makes inter-message separation
clear enough in that case.

Cheers,
Trevor

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 00/17] nmbug-status: Python-3-compabitility and general refactoring

[Tomi Ollila]

On Sat, Feb 08 2014, "W. Trevor King" <wking@tremily.us> wrote:

> On Sat, Feb 08, 2014 at 08:29:41PM +0200, Tomi Ollila wrote:
>> On Sat, Feb 08 2014, W. Trevor King wrote:
>> > On Sat, Feb 08, 2014 at 09:54:28AM -0400, David Bremner wrote:
>> >> 
>> >> yeah, the colour scheme is not my favourite. For the sake of being
>> >> semi-constructive, I attach an alternative suggestion #ffd96e and
>> >> #bce.
>> >
>> > Works for me.
>> 
>> I'm fine with that too.
>
> Updated in my example and nmbug-status-python3 branch.
>
>> >> More importantly some of the threads are run together: e.g. 
>> >> 
>> >> id:"4eddf2b1.4288980a.0b74.5557@mx.google.com" and
>> >> id:"E1RYMYd-0003wu-Ea@thinkbox.jade-hamburg.de"
>> >
>> > Both of those messages are part of the same thread
>> > (thread:000000000000eaab on my box, but I doubt thread IDs are
>> > portable), so I don't add a thread-separating space between them.
>> > Would you like more message-separating space even between messages
>> > in the same thread?
>> 
>> …
>> 
>> I think the more space does not fix anything but it might help
>> regognizing message boundaries a bit (and thus would be a nice
>> feature).
>
> Added to my example and nmbug-status-python3 branch.  I'm using CSS
> for this new spacing, so non-CSS browsers (e.g. w3m) won't render the
> inter-message spacing.  I think the colored-link (second message row)
> vs. default id (first message row) makes inter-message separation
> clear enough in that case.

The 2 0.5em:s adds up to 1em between messages and that IMHO makes the
messages be too far apart from each other. I changed the padding-top
and padding-bottom values to 0.25em which IMHO is better...

> Cheers,
> Trevor

Tomi

Re: [PATCH 00/17] nmbug-status: Python-3-compabitility and general refactoring

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 1756 bytes --]

On Sat, Feb 08, 2014 at 09:37:43PM +0200, Tomi Ollila wrote:
> On Sat, Feb 08 2014, W. Trevor King wrote:
> > On Sat, Feb 08, 2014 at 08:29:41PM +0200, Tomi Ollila wrote:
> >> On Sat, Feb 08 2014, W. Trevor King wrote:
> >> > On Sat, Feb 08, 2014 at 09:54:28AM -0400, David Bremner wrote:
> >> >> More importantly some of the threads are run together: e.g. 
> >> >> 
> >> >> id:"4eddf2b1.4288980a.0b74.5557@mx.google.com" and
> >> >> id:"E1RYMYd-0003wu-Ea@thinkbox.jade-hamburg.de"
> >> >
> >> > Both of those messages are part of the same thread
> >> > (thread:000000000000eaab on my box, but I doubt thread IDs are
> >> > portable), so I don't add a thread-separating space between them.
> >> > Would you like more message-separating space even between messages
> >> > in the same thread?
> >> 
> >> …
> >> 
> >> I think the more space does not fix anything but it might help
> >> regognizing message boundaries a bit (and thus would be a nice
> >> feature).
> >
> > Added to my example and nmbug-status-python3 branch.  I'm using CSS
> > for this new spacing, so non-CSS browsers (e.g. w3m) won't render the
> > inter-message spacing.  I think the colored-link (second message row)
> > vs. default id (first message row) makes inter-message separation
> > clear enough in that case.
> 
> The 2 0.5em:s adds up to 1em between messages and that IMHO makes the
> messages be too far apart from each other. I changed the padding-top
> and padding-bottom values to 0.25em which IMHO is better...

0.25em works for me.  Branch and example updated.

Cheers,
Trevor

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 15/17] nmbug-status: Quote the title when using it as an id

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 582 bytes --]

On Mon, Feb 03, 2014 at 02:59:33AM -0800, W. Trevor King wrote:
> +            if 'id' not in view:
> +                view['id'] = quote(view['title'])

Stuff like href="#Maybe%20Ready%20%28Core%20and%20Emacs%29" wasn't
working for me on Firefox 24.2.0, so I've tweaked this commit to slug
titles instead of quoting them.  Pushed to my nmbug-status-python3
branch and queued for my v2 submission.

Cheers,
Trevor

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 15/17] nmbug-status: Quote the title when using it as an id

[Tomi Ollila]

On Sun, Feb 09 2014, "W. Trevor King" <wking@tremily.us> wrote:

> On Mon, Feb 03, 2014 at 02:59:33AM -0800, W. Trevor King wrote:
>> +            if 'id' not in view:
>> +                view['id'] = quote(view['title'])
>
> Stuff like href="#Maybe%20Ready%20%28Core%20and%20Emacs%29" wasn't
> working for me on Firefox 24.2.0, so I've tweaked this commit to slug
> titles instead of quoting them.  Pushed to my nmbug-status-python3
> branch and queued for my v2 submission.

$ git checkout nmbug-status-python3
$ git pull --rebase
$ ./devel/nmbug/nmbug-status

Looks pretty good to me. I am not sure whether the commit messages should
be *that* verbose, though...

>
> Cheers,
> Trevor

Tomi