Xapers

Xapers

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 1154 bytes --]

Hey, folks.  I wanted to let people know about a program that I've been
working on that may be of interest to people on this list.  It's also
very heavily inspired by notmuch.

It's a personal paper management/indexing system called Xapers, geared
somewhat specifically towards academic journal articles.  For those
familiar with Papers [0], it's like that but for the command line set.

Give it a pdf and a source identifier (like doi url), and it downloads
bibtex from the online library, stores the pdf and bibtex in a local
document store, and indexes them both into a Xapian database.  Documents
can be arbitrarily tagged.  Documents and bibtex can be exported from
arbitrary searches.  It's a python library underneath, but there a cli
and a simple curses ui that can display search results, tag documents,
and retrieve files.

README and a screenshot of the curses ui here:

http://finestructure.net/xapers/README
http://finestructure.net/xapers/screenshot.png

It's certainly not perfect, but it's been working fairly well for me for
a while.  Feedback/bugs/etc (off list) are very much appreciated.

jamie.

[0] http://www.papersapp.com/papers/

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: Xapers

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 398 bytes --]

On Mon, Apr 15 2013, Jameson Graef Rollins <jrollins@finestructure.net> wrote:
> README and a screenshot of the curses ui here:
>
> http://finestructure.net/xapers/README
> http://finestructure.net/xapers/screenshot.png

I maybe should have been a little clearer that all necessary
instructions should be in the README, including the source repo:

  git://finestructure.net/xapers

Thanks.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: Xapers

[Patrick Totzke]

[-- Attachment #1: Type: text/plain, Size: 4543 bytes --]

Quoting Jameson Graef Rollins (2013-04-15 18:18:00)
> On Mon, Apr 15 2013, Jameson Graef Rollins <jrollins@finestructure.net> wrote:
> > README and a screenshot of the curses ui here:
> >
> > http://finestructure.net/xapers/README
> > http://finestructure.net/xapers/screenshot.png
> 
> I maybe should have been a little clearer that all necessary
> instructions should be in the README, including the source repo:
> 
>   git://finestructure.net/xapers


Brilliant idea!

here's some early feedback:
* is there a way to batch import a bunch of pdfs? or better yet, a bibtex file with entries pointing
  to files like the ones produced by say jabref?
  There, entries have some line like this:
  file = {Boz2012.pdf:papers/Boz2012.pdf:PDF}

* got a bunch of traces:
  -----------------------------
  $xapers import --source=ABK2011.pdf                                                                                     1 ↵ master 
  Reading bibtex... done.
  Adding bibtex... 
  
  Traceback (most recent call last):
    File "/home/pazz/.local/bin/xapers", line 145, in <module>
      docid = cli.add(query_string, infile=infile, source=source, tags=tags, prompt=prompt)
    File "/home/pazz/.local/lib/python2.7/site-packages/xapers/cli/ui.py", line 228, in add
      doc.add_bibtex(bibtex)
    File "/home/pazz/.local/lib/python2.7/site-packages/xapers/documents.py", line 373, in add_bibtex
      self.bibentry = xapers.bibtex.Bibentry(bibtex)
    File "/home/pazz/.local/lib/python2.7/site-packages/xapers/bibtex.py", line 25, in __init__
      if os.path.exists(bibtex):
    File "/usr/lib/python2.7/genericpath.py", line 18, in exists
      os.stat(path)
  TypeError: must be encoded string without NULL bytes, not str
  ----------------------------
  
  $ xapers show, then i moved around a bit

  ----------------------------
  Traceback (most recent call last):
    File "/home/pazz/.local/bin/xapers", line 228, in <module>
      xapers.nci.UI(xroot, cmd=args)
    File "/home/pazz/.local/lib/python2.7/site-packages/xapers/nci/ui.py", line 73, in __init__
      self.mainloop.run()
    File "/home/pazz/.local/lib/python2.7/site-packages/urwid-1.2.0_dev-py2.7-linux-x86_64.egg/urwid/main_loop.py", line 272, in run
      self.screen.run_wrapper(self._run)
    File "/home/pazz/.local/lib/python2.7/site-packages/urwid-1.2.0_dev-py2.7-linux-x86_64.egg/urwid/raw_display.py", line 242, in run_wrapper
      return fn()
    File "/home/pazz/.local/lib/python2.7/site-packages/urwid-1.2.0_dev-py2.7-linux-x86_64.egg/urwid/main_loop.py", line 337, in _run
      self.event_loop.run()
    File "/home/pazz/.local/lib/python2.7/site-packages/urwid-1.2.0_dev-py2.7-linux-x86_64.egg/urwid/main_loop.py", line 708, in run
      self._loop()
    File "/home/pazz/.local/lib/python2.7/site-packages/urwid-1.2.0_dev-py2.7-linux-x86_64.egg/urwid/main_loop.py", line 787, in _loop
      self._watch_files[fd]()
    File "/home/pazz/.local/lib/python2.7/site-packages/urwid-1.2.0_dev-py2.7-linux-x86_64.egg/urwid/main_loop.py", line 388, in _update
      self.process_input(keys)
    File "/home/pazz/.local/lib/python2.7/site-packages/urwid-1.2.0_dev-py2.7-linux-x86_64.egg/urwid/main_loop.py", line 488, in process_input
      k = self._topmost_widget.keypress(self.screen_size, k)
    File "/home/pazz/.local/lib/python2.7/site-packages/urwid-1.2.0_dev-py2.7-linux-x86_64.egg/urwid/container.py", line 1120, in keypress
      return self.body.keypress( (maxcol, remaining), key )
    File "/home/pazz/.local/lib/python2.7/site-packages/xapers/nci/search.py", line 304, in keypress
      eval(cmd)
    File "<string>", line 1, in <module>
    File "/home/pazz/.local/lib/python2.7/site-packages/xapers/nci/search.py", line 155, in nextEntry
      self.listbox.set_focus(pos + 1)
    File "/home/pazz/.local/lib/python2.7/site-packages/urwid-1.2.0_dev-py2.7-linux-x86_64.egg/urwid/listbox.py", line 567, in set_focus
      self.body.set_focus(position)
    File "/home/pazz/.local/lib/python2.7/site-packages/urwid-1.2.0_dev-py2.7-linux-x86_64.egg/urwid/listbox.py", line 178, in set_focus
      raise IndexError, "No widget at position %s" % (position,)
  IndexError: No widget at position 1
  ----------------------------


* did you already set something up to report bugs? github issues, a maling list?

Can't wait until someone forks alot to use this. I've been waiting for someone (else)
to start a project like this for some time now :)

rock on,
/p


[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iEYEABECAAYFAlFsSxUACgkQlDQDZ9fWxaoRsgCbB6Xj1P05hJsch71+5jC1BBdr
Ko8AniviyvGPECY3FACqnAfiW4xc5PQT
=eXpE
-----END PGP SIGNATURE-----

Re: Xapers

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 5216 bytes --]

On Mon, Apr 15 2013, Patrick Totzke <patricktotzke@gmail.com> wrote:
> Brilliant idea!

Thanks!  I give Carl all the credit, though, since it's really all just
based on the notmuch idea.

> here's some early feedback:
> * is there a way to batch import a bunch of pdfs? or better yet, a bibtex file with entries pointing
>   to files like the ones produced by say jabref?
>   There, entries have some line like this:
>   file = {Boz2012.pdf:papers/Boz2012.pdf:PDF}

There's not currently a prescribed way to batch import, but if you can
programmatically get all pdf/bibtex pairs then it should be easy to
script:

  xapers import --file=PDF --source=BIBTEX

> * got a bunch of traces:
>   -----------------------------
>   $xapers import --source=ABK2011.pdf                                                                                     1 ↵ master 
>   Reading bibtex... done.
>   Adding bibtex... 
>   
>   Traceback (most recent call last):
>     File "/home/pazz/.local/bin/xapers", line 145, in <module>
>       docid = cli.add(query_string, infile=infile, source=source, tags=tags, prompt=prompt)
>     File "/home/pazz/.local/lib/python2.7/site-packages/xapers/cli/ui.py", line 228, in add
>       doc.add_bibtex(bibtex)
>     File "/home/pazz/.local/lib/python2.7/site-packages/xapers/documents.py", line 373, in add_bibtex
>       self.bibentry = xapers.bibtex.Bibentry(bibtex)
>     File "/home/pazz/.local/lib/python2.7/site-packages/xapers/bibtex.py", line 25, in __init__
>       if os.path.exists(bibtex):
>     File "/usr/lib/python2.7/genericpath.py", line 18, in exists
>       os.stat(path)
>   TypeError: must be encoded string without NULL bytes, not str
>   ----------------------------

Yikes.  Sorry about that.  Clearly more error checking is needed.

The problem here is that PDFs should passed in through the --file=
option, while source library URLs or bibtex files should be passed in
through --source=.

>   $ xapers show, then i moved around a bit
>
>   ----------------------------
>   Traceback (most recent call last):
>     File "/home/pazz/.local/bin/xapers", line 228, in <module>
>       xapers.nci.UI(xroot, cmd=args)
>     File "/home/pazz/.local/lib/python2.7/site-packages/xapers/nci/ui.py", line 73, in __init__
>       self.mainloop.run()
>     File "/home/pazz/.local/lib/python2.7/site-packages/urwid-1.2.0_dev-py2.7-linux-x86_64.egg/urwid/main_loop.py", line 272, in run
>       self.screen.run_wrapper(self._run)
>     File "/home/pazz/.local/lib/python2.7/site-packages/urwid-1.2.0_dev-py2.7-linux-x86_64.egg/urwid/raw_display.py", line 242, in run_wrapper
>       return fn()
>     File "/home/pazz/.local/lib/python2.7/site-packages/urwid-1.2.0_dev-py2.7-linux-x86_64.egg/urwid/main_loop.py", line 337, in _run
>       self.event_loop.run()
>     File "/home/pazz/.local/lib/python2.7/site-packages/urwid-1.2.0_dev-py2.7-linux-x86_64.egg/urwid/main_loop.py", line 708, in run
>       self._loop()
>     File "/home/pazz/.local/lib/python2.7/site-packages/urwid-1.2.0_dev-py2.7-linux-x86_64.egg/urwid/main_loop.py", line 787, in _loop
>       self._watch_files[fd]()
>     File "/home/pazz/.local/lib/python2.7/site-packages/urwid-1.2.0_dev-py2.7-linux-x86_64.egg/urwid/main_loop.py", line 388, in _update
>       self.process_input(keys)
>     File "/home/pazz/.local/lib/python2.7/site-packages/urwid-1.2.0_dev-py2.7-linux-x86_64.egg/urwid/main_loop.py", line 488, in process_input
>       k = self._topmost_widget.keypress(self.screen_size, k)
>     File "/home/pazz/.local/lib/python2.7/site-packages/urwid-1.2.0_dev-py2.7-linux-x86_64.egg/urwid/container.py", line 1120, in keypress
>       return self.body.keypress( (maxcol, remaining), key )
>     File "/home/pazz/.local/lib/python2.7/site-packages/xapers/nci/search.py", line 304, in keypress
>       eval(cmd)
>     File "<string>", line 1, in <module>
>     File "/home/pazz/.local/lib/python2.7/site-packages/xapers/nci/search.py", line 155, in nextEntry
>       self.listbox.set_focus(pos + 1)
>     File "/home/pazz/.local/lib/python2.7/site-packages/urwid-1.2.0_dev-py2.7-linux-x86_64.egg/urwid/listbox.py", line 567, in set_focus
>       self.body.set_focus(position)
>     File "/home/pazz/.local/lib/python2.7/site-packages/urwid-1.2.0_dev-py2.7-linux-x86_64.egg/urwid/listbox.py", line 178, in set_focus
>       raise IndexError, "No widget at position %s" % (position,)
>   IndexError: No widget at position 1
>   ----------------------------

Hrm.  Don't know what this is about.  I'll look into it.  Sorry again!

> * did you already set something up to report bugs? github issues, a maling list?

No, not yet.  I wanted to gauge interest first.  Maybe I'll go ahead and
set something up on github.  In the meantime please just report bugs to
be directly and I'll note them in the TODO.

> Can't wait until someone forks alot to use this. I've been waiting for
> someone (else) to start a project like this for some time now :)

Well actually the curses UI is ripped almost entirely from alot already!
It clearly needs some more work, though.

Thanks for the feedback and interest!

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: Xapers

[Rainer M. Krug]

Jameson Graef Rollins <jrollins@finestructure.net> writes:

> On Mon, Apr 15 2013, Patrick Totzke <patricktotzke@gmail.com> wrote:
>> Brilliant idea!
>
> Thanks!  I give Carl all the credit, though, since it's really all just
> based on the notmuch idea.
>
>> here's some early feedback:
>> * is there a way to batch import a bunch of pdfs? or better yet, a bibtex file with entries pointing
>>   to files like the ones produced by say jabref?
>>   There, entries have some line like this:
>>   file = {Boz2012.pdf:papers/Boz2012.pdf:PDF}
>
> There's not currently a prescribed way to batch import, but if you can
> programmatically get all pdf/bibtex pairs then it should be easy to
> script:
>

It would be quite important to be able to do something along the lines
of

  xapers import --bib=TheBibtexFile.bib

and xapers extracts the info it needs. If this info of the source of the
data is put into a config file, one could, as with notmuch, just update
xapers by doing:

  xapers new

and all new papers in the config file source (bibtex, possibly directory
with pdf, text file, ...) would be added to xapers.

Most people already have there papers in bibtex files, so a feature like
this wold make xapers a *very* interesting tool - I would most
definitely use it,bu not if I have to re-enter each file I add to the
pdf.

Is there a mailing list or place where you anounce new versions?

Looking forward to the bibtex interface,

Rainer
  

>   xapers import --file=PDF --source=BIBTEX
>
>> * got a bunch of traces:
>>   -----------------------------
>>   $xapers import --source=ABK2011.pdf                                                                                     1 ↵ master 
>>   Reading bibtex... done.
>>   Adding bibtex... 
>>   
>>   Traceback (most recent call last):
>>     File "/home/pazz/.local/bin/xapers", line 145, in <module>
>>       docid = cli.add(query_string, infile=infile, source=source, tags=tags, prompt=prompt)
>>     File "/home/pazz/.local/lib/python2.7/site-packages/xapers/cli/ui.py", line 228, in add
>>       doc.add_bibtex(bibtex)
>>     File "/home/pazz/.local/lib/python2.7/site-packages/xapers/documents.py", line 373, in add_bibtex
>>       self.bibentry = xapers.bibtex.Bibentry(bibtex)
>>     File "/home/pazz/.local/lib/python2.7/site-packages/xapers/bibtex.py", line 25, in __init__
>>       if os.path.exists(bibtex):
>>     File "/usr/lib/python2.7/genericpath.py", line 18, in exists
>>       os.stat(path)
>>   TypeError: must be encoded string without NULL bytes, not str
>>   ----------------------------
>
> Yikes.  Sorry about that.  Clearly more error checking is needed.
>
> The problem here is that PDFs should passed in through the --file=
> option, while source library URLs or bibtex files should be passed in
> through --source=.
>
>>   $ xapers show, then i moved around a bit
>>
>>   ----------------------------
>>   Traceback (most recent call last):
>>     File "/home/pazz/.local/bin/xapers", line 228, in <module>
>>       xapers.nci.UI(xroot, cmd=args)
>>     File "/home/pazz/.local/lib/python2.7/site-packages/xapers/nci/ui.py", line 73, in __init__
>>       self.mainloop.run()
>>     File "/home/pazz/.local/lib/python2.7/site-packages/urwid-1.2.0_dev-py2.7-linux-x86_64.egg/urwid/main_loop.py", line 272, in run
>>       self.screen.run_wrapper(self._run)
>>     File "/home/pazz/.local/lib/python2.7/site-packages/urwid-1.2.0_dev-py2.7-linux-x86_64.egg/urwid/raw_display.py", line 242, in run_wrapper
>>       return fn()
>>     File "/home/pazz/.local/lib/python2.7/site-packages/urwid-1.2.0_dev-py2.7-linux-x86_64.egg/urwid/main_loop.py", line 337, in _run
>>       self.event_loop.run()
>>     File "/home/pazz/.local/lib/python2.7/site-packages/urwid-1.2.0_dev-py2.7-linux-x86_64.egg/urwid/main_loop.py", line 708, in run
>>       self._loop()
>>     File "/home/pazz/.local/lib/python2.7/site-packages/urwid-1.2.0_dev-py2.7-linux-x86_64.egg/urwid/main_loop.py", line 787, in _loop
>>       self._watch_files[fd]()
>>     File "/home/pazz/.local/lib/python2.7/site-packages/urwid-1.2.0_dev-py2.7-linux-x86_64.egg/urwid/main_loop.py", line 388, in _update
>>       self.process_input(keys)
>>     File "/home/pazz/.local/lib/python2.7/site-packages/urwid-1.2.0_dev-py2.7-linux-x86_64.egg/urwid/main_loop.py", line 488, in process_input
>>       k = self._topmost_widget.keypress(self.screen_size, k)
>>     File "/home/pazz/.local/lib/python2.7/site-packages/urwid-1.2.0_dev-py2.7-linux-x86_64.egg/urwid/container.py", line 1120, in keypress
>>       return self.body.keypress( (maxcol, remaining), key )
>>     File "/home/pazz/.local/lib/python2.7/site-packages/xapers/nci/search.py", line 304, in keypress
>>       eval(cmd)
>>     File "<string>", line 1, in <module>
>>     File "/home/pazz/.local/lib/python2.7/site-packages/xapers/nci/search.py", line 155, in nextEntry
>>       self.listbox.set_focus(pos + 1)
>>     File "/home/pazz/.local/lib/python2.7/site-packages/urwid-1.2.0_dev-py2.7-linux-x86_64.egg/urwid/listbox.py", line 567, in set_focus
>>       self.body.set_focus(position)
>>     File "/home/pazz/.local/lib/python2.7/site-packages/urwid-1.2.0_dev-py2.7-linux-x86_64.egg/urwid/listbox.py", line 178, in set_focus
>>       raise IndexError, "No widget at position %s" % (position,)
>>   IndexError: No widget at position 1
>>   ----------------------------
>
> Hrm.  Don't know what this is about.  I'll look into it.  Sorry again!
>
>> * did you already set something up to report bugs? github issues, a maling list?
>
> No, not yet.  I wanted to gauge interest first.  Maybe I'll go ahead and
> set something up on github.  In the meantime please just report bugs to
> be directly and I'll note them in the TODO.
>
>> Can't wait until someone forks alot to use this. I've been waiting for
>> someone (else) to start a project like this for some time now :)
>
> Well actually the curses UI is ripped almost entirely from alot already!
> It clearly needs some more work, though.
>
> Thanks for the feedback and interest!
>
> jamie.
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch
<#secure method=pgpmime mode=sign>

-- 
Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation Biology, UCT), Dipl. Phys. (Germany)

Centre of Excellence for Invasion Biology
Stellenbosch University
South Africa

Tel :       +33 - (0)9 53 10 27 44
Cell:       +33 - (0)6 85 62 59 98
Fax :       +33 - (0)9 58 10 27 44

Fax (D):    +49 - (0)3 21 21 25 22 44

email:      Rainer@krugs.de

Skype:      RMkrug

Re: Xapers

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 961 bytes --]

On Tue, Apr 16 2013, "Rainer M. Krug" <Rainer@krugs.de> wrote:
> It would be quite important to be able to do something along the lines
> of
>
>   xapers import --bib=TheBibtexFile.bib

Hi, Rainer:

Xapers does currently do something similar to this.  The following takes
the first bibtex entry from bibtex.bib and creates a new Xapers
document:

  xapers add --source=bibtex.bib

However, I suspect you're looking for a way to import a full bibtex
database with multiple entries, and create a new Xapers document for
each.  This shouldn't be difficult to implement.  Particularly if there
is a standard bibtex field for specifying associated file paths.
Patrick mentioned that jabref uses something, so maybe I can start with
that.

> Is there a mailing list or place where you anounce new versions?

It does look like there might be a need.  I'll look into putting
something together.

In the mean time I'm hanging out on #xapers on irc.freenode.net.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: Xapers

[Rainer M. Krug]

Jameson Graef Rollins <jrollins@finestructure.net> writes:

> On Tue, Apr 16 2013, "Rainer M. Krug" <Rainer@krugs.de> wrote:
>> It would be quite important to be able to do something along the lines
>> of
>>
>>   xapers import --bib=TheBibtexFile.bib
>
> Hi, Rainer:
>
> Xapers does currently do something similar to this.  The following takes
> the first bibtex entry from bibtex.bib and creates a new Xapers
> document:
>
>   xapers add --source=bibtex.bib
>
> However, I suspect you're looking for a way to import a full bibtex
> database with multiple entries, and create a new Xapers document for

Exactly - batch importing my several thousand papers from the bibtex file.

> each.  This shouldn't be difficult to implement.  Particularly if
> there

That's good to hear.
'
> is a standard bibtex field for specifying associated file paths.
> Patrick mentioned that jabref uses something, so maybe I can start with
> that.

Mendeley for example uses the file field:

file = {:home/rkrug/Documents/Mendeley Desktop/Knight et al/Knight et
al._2006.pdf:pdf},

I would actually make the pdf an optional entry. Most papers have URLs
attached, which make it possible to get the paper if one is interested
in. And xapers would be really useful in assessing potential papers
before downloading the pdfs.

And another thought: jabref stores info about the paper in the pdf's
meta data - maybe xapers could use that data as well, so just pointing
to a directory full of pdf could then also work.

>
>> Is there a mailing list or place where you anounce new versions?
>
> It does look like there might be a need.  I'll look into putting
> something together.

Would be good - xapers looks really useful.

>
> In the mean time I'm hanging out on #xapers on irc.freenode.net.

Ok - and please send updates about the mailing list here as well.

Cheers,

Rainer

>
> jamie.
<#secure method=pgpmime mode=sign>

-- 
Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation Biology, UCT), Dipl. Phys. (Germany)

Centre of Excellence for Invasion Biology
Stellenbosch University
South Africa

Tel :       +33 - (0)9 53 10 27 44
Cell:       +33 - (0)6 85 62 59 98
Fax :       +33 - (0)9 58 10 27 44

Fax (D):    +49 - (0)3 21 21 25 22 44

email:      Rainer@krugs.de

Skype:      RMkrug

Re: Xapers

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 393 bytes --]

Hi, folks.  One last bit of spam about Xapers (which I announced
previously [0]).  We now have an improved web presence [1] and a mailing
list [2].  Version 0.5.2 has also been uploaded to Debian, and will
hopefully be available in unstable soon.

jamie.

[0] id:87zjx0tdwt.fsf@servo.finestructure.net
[1] http://finestructure.net/xapers
[2] https://lists.mayfirst.org/mailman/listinfo/xapers

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]