contributing to work on citations

contributing to work on citations

[Matt Price]

[-- Attachment #1: Type: text/plain, Size: 1878 bytes --]

So, from the "some projects" thread I have the sense there is a group of
users with some interest in improving citation support. What is the best
way to kickstart that process? it seems to me that, at a minimum, the
following is needed:

1. Finalization of the citation syntax.

IIUC, This involves refining and merging the work in the wip-cite git
branch.

2. Choice of a citation creation tool

IIUC, the two main candidates are pandoc-citeproc and citeproc-js/node.
Pandoc-citeproc works well but requires users to install the whole pandoc
infrastructure, and needs to be wrapped in a thin tool for use with org.
Citeproc-node has fewer dependencies (I guess), but needs to be updated to
support additional output formats (e.g. latex, odt, and org) and is a
little more elaborate to run from the command line (runs as a service, can
only be communicated with over http).

3. Handling of citation links in the various export engines
Functions must be written to handle citations in, at minimum, latex, html,
and org

4. Interface with backends
Org should be able to talk to bibtex and zotero databases, at least (and
maybe more).

A lot of this work has already been done. Richard has written org-citeproc
as a wrapper for pandoc-citeproc https://github.com/wyleyr/org-citeproc .
Aaron has written some functions for org-citeproc in the wip-cite-awe
branch. Nicolas has developed the syntax in wip-cite. I even have a fork of
citeproc-node that could be used to develop the missing features we need.
And of course John has org-ref (
http://kitchingroup.cheme.cmu.edu/blog/2014/05/13/Using-org-ref-for-citations-and-references/)
while Erik H has zotxt-emacs for Zotero integration (
https://github.com/egh/zotxt-emacs).

What would be the best next move? It seems to me there are enough moving
parts that the process will be quite a bit easier if we co-ordinate.

thanks,
Matt

[-- Attachment #2: Type: text/html, Size: 2356 bytes --]

Re: contributing to work on citations

[Aaron Ecay]

Hi Matt,

Thanks for your willingness to help with the project.

2015ko urriak 30an, Matt Price-ek idatzi zuen:
> 
> So, from the "some projects" thread I have the sense there is a group of
> users with some interest in improving citation support. What is the best
> way to kickstart that process? it seems to me that, at a minimum, the
> following is needed:
> 
> 1. Finalization of the citation syntax.
> 
> IIUC, This involves refining and merging the work in the wip-cite git
> branch.

There was a very voluminous discussion on this last time.  Lots of
people proposed various additions.  I’d be inclined to go with whatever
syntax is implemented currently, and let any needed extensions sort
themselves out through real-world testing.

> 
> 2. Choice of a citation creation tool
> 
> IIUC, the two main candidates are pandoc-citeproc and citeproc-js/node.
> Pandoc-citeproc works well but requires users to install the whole pandoc
> infrastructure, and needs to be wrapped in a thin tool for use with org.
> Citeproc-node has fewer dependencies (I guess), but needs to be updated to
> support additional output formats (e.g. latex, odt, and org) and is a
> little more elaborate to run from the command line (runs as a service, can
> only be communicated with over http).

IIUC, citeproc-node just wraps citeproc-js, which has a “normal” API.
So it should be possible to write a CLI wrapper for citeproc-js.  (But I
have not studied citeproc-js at all, so I could be wrong).

My plan is to create preliminary citation support based on Richard’s
haskell tool, with the intention of moving to a JS-based tool when
possible.  So implementing a work-alike of Richard’s tool in JS would be
a good development.

> 
> 3. Handling of citation links in the various export engines
> Functions must be written to handle citations in, at minimum, latex, html,
> and org
> 
> 4. Interface with backends
> Org should be able to talk to bibtex and zotero databases, at least (and
> maybe more).

The code I wrote should have enough hooks to enable this.  But I haven’t
looked at zotero at all.  If zotero support is important to you, you
could investigate the “lookup types” in the citation code I wrote, see
if they are adequate, and try to use them to implement zotero lookup.
From the docstring:

"Types of citation lookup backends.

Alist from type to list of:

- Function called at the beginning of export, with the rest of
  the keyword line after #+BIBDB: type, and the info plist.
  Should cache whatever it needs in the info plist.

- Function to lookup a citation.  Called with the key and the
  info plist.  Will be memoized by
  `org-export-cite--lookup' (TODO).  Should return an alist of
  keys and values about the citation (author, year, title, etc.)

- A boolean; non-nil = this lookup type is remote.  All local
  lookups will be tried before any remote one is.
  TODO: not yet implemented"
  
> 
> A lot of this work has already been done. Richard has written org-citeproc
> as a wrapper for pandoc-citeproc https://github.com/wyleyr/org-citeproc .
> Aaron has written some functions for org-citeproc in the wip-cite-awe
> branch. Nicolas has developed the syntax in wip-cite. I even have a fork of
> citeproc-node that could be used to develop the missing features we need.
> And of course John has org-ref (
> http://kitchingroup.cheme.cmu.edu/blog/2014/05/13/Using-org-ref-for-citations-and-references/)

The org-ref features provide a good aspirational horizon for us, I think.
I.e. org’s core should aim to support almost all of them eventually.  But
the support for citation syntax is not so far along that creating such
support is an immediate prospect.

> while Erik H has zotxt-emacs for Zotero integration (
> https://github.com/egh/zotxt-emacs).
> 
> What would be the best next move? It seems to me there are enough moving
> parts that the process will be quite a bit easier if we co-ordinate.

Based on the interests you’ve expressed, I think working on the JS
citation processor and/or zotero integration would be good contributions
you can make.  But you can work on other things as well.  I think the
best way to coordinate will be to keep the list apprised of your
progress.  If you have push access, you can push WIP code as branches in
the org-mode repository.  (Otherwise, you can use github or some other
git host, or ask for push access.)

Thanks again,

-- 
Aaron Ecay

Re: contributing to work on citations

[Will Monroe]

Aaron, Matt:

> Based on the interests you’ve expressed, I think working on the JS
> citation processor and/or zotero integration would be good contributions
> you can make.  But you can work on other things as well.  I think the
> best way to coordinate will be to keep the list apprised of your
> progress.

As someone who uses Zotero and org-mode quite a bit, I'd be glad to help
with testing, if that would be useful.  I'm learning Elisp but I'm still
a beginner.  Nevertheless, I have a keen interest in bringing together
these tools.

All the best,

Will

Re: contributing to work on citations

[William Denton]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 398 bytes --]

On 4 November 2015, Will Monroe wrote:

> As someone who uses Zotero and org-mode quite a bit, I'd be glad to help
> with testing, if that would be useful.  I'm learning Elisp but I'm still
> a beginner.  Nevertheless, I have a keen interest in bringing together
> these tools.

Same here---I'm glad to test.

Bill
-- 
William Denton ↔  Toronto, Canada ↔  https://www.miskatonic.org/

Re: contributing to work on citations

[John Wiegley]

>>>>> Will Monroe <wtmonroe.ls@gmail.com> writes:

> As someone who uses Zotero and org-mode quite a bit, I'd be glad to help
> with testing, if that would be useful. I'm learning Elisp but I'm still a
> beginner. Nevertheless, I have a keen interest in bringing together these
> tools.

Me too, I'm a fan and active user of both tools as well.

John

Re: contributing to work on citations

[Matt Price]

[-- Attachment #1: Type: text/plain, Size: 5665 bytes --]

On Wed, Nov 4, 2015 at 10:18 AM, Aaron Ecay <aaronecay@gmail.com> wrote:

> Hi Matt,
>
> Thanks for your willingness to help with the project.
>
> 2015ko urriak 30an, Matt Price-ek idatzi zuen:
> >
> > So, from the "some projects" thread I have the sense there is a group of
> > users with some interest in improving citation support. What is the best
> > way to kickstart that process? it seems to me that, at a minimum, the
> > following is needed:
> >
> > 1. Finalization of the citation syntax.
> >
> > IIUC, This involves refining and merging the work in the wip-cite git
> > branch.
>
> There was a very voluminous discussion on this last time.  Lots of
> people proposed various additions.  I’d be inclined to go with whatever
> syntax is implemented currently, and let any needed extensions sort
> themselves out through real-world testing.
>

OK, great.  Does this mean just checking out the org-wip-cite branch? Or
has someone perhaps merged more recent changes into wip-cite somewhere
else?

>
> >
> > 2. Choice of a citation creation tool
> >
> > IIUC, the two main candidates are pandoc-citeproc and citeproc-js/node.
> > Pandoc-citeproc works well but requires users to install the whole pandoc
> > infrastructure, and needs to be wrapped in a thin tool for use with org.
> > Citeproc-node has fewer dependencies (I guess), but needs to be updated
> to
> > support additional output formats (e.g. latex, odt, and org) and is a
> > little more elaborate to run from the command line (runs as a service,
> can
> > only be communicated with over http).
>
> IIUC, citeproc-node just wraps citeproc-js, which has a “normal” API.
> So it should be possible to write a CLI wrapper for citeproc-js.  (But I
> have not studied citeproc-js at all, so I could be wrong).
>

I am not sure but I don't think it's trivial; and alas i am slow at eveyr
kind of coding, but I will certainly look into this.

>
> My plan is to create preliminary citation support based on Richard’s
> haskell tool, with the intention of moving to a JS-based tool when
> possible.  So implementing a work-alike of Richard’s tool in JS would be
> a good development.
>
>
OK, I'll pay more attention to Richard's tool and how it works, with the
goal of making something like a drop-in replacement.

> >
> > 3. Handling of citation links in the various export engines
> > Functions must be written to handle citations in, at minimum, latex,
> html,
> > and org
> >
> > 4. Interface with backends
> > Org should be able to talk to bibtex and zotero databases, at least (and
> > maybe more).
>
> The code I wrote should have enough hooks to enable this.  But I haven’t
> looked at zotero at all.  If zotero support is important to you, you
> could investigate the “lookup types” in the citation code I wrote, see
> if they are adequate, and try to use them to implement zotero lookup.
> From the docstring:
>
> "Types of citation lookup backends.
>
> Alist from type to list of:
>
> - Function called at the beginning of export, with the rest of
>   the keyword line after #+BIBDB: type, and the info plist.
>   Should cache whatever it needs in the info plist.
>
> - Function to lookup a citation.  Called with the key and the
>   info plist.  Will be memoized by
>   `org-export-cite--lookup' (TODO).  Should return an alist of
>   keys and values about the citation (author, year, title, etc.)
>
> - A boolean; non-nil = this lookup type is remote.  All local
>   lookups will be tried before any remote one is.
>   TODO: not yet implemented"
>
>
I'll start here.  Thanks for the pointer.


> >
> > A lot of this work has already been done. Richard has written
> org-citeproc
> > as a wrapper for pandoc-citeproc https://github.com/wyleyr/org-citeproc
> .
> > Aaron has written some functions for org-citeproc in the wip-cite-awe
> > branch. Nicolas has developed the syntax in wip-cite. I even have a fork
> of
> > citeproc-node that could be used to develop the missing features we need.
> > And of course John has org-ref (
> >
> http://kitchingroup.cheme.cmu.edu/blog/2014/05/13/Using-org-ref-for-citations-and-references/
> )
>
> The org-ref features provide a good aspirational horizon for us, I think.
> I.e. org’s core should aim to support almost all of them eventually.  But
> the support for citation syntax is not so far along that creating such
> support is an immediate prospect.
>
> > while Erik H has zotxt-emacs for Zotero integration (
> > https://github.com/egh/zotxt-emacs).
> >
> > What would be the best next move? It seems to me there are enough moving
> > parts that the process will be quite a bit easier if we co-ordinate.
>
> Based on the interests you’ve expressed, I think working on the JS
> citation processor and/or zotero integration would be good contributions
> you can make.  But you can work on other things as well.  I think the
> best way to coordinate will be to keep the list apprised of your
> progress.  If you have push access, you can push WIP code as branches in
> the org-mode repository.  (Otherwise, you can use github or some other
> git host, or ask for push access.)
>
>
OK, will do. I think it would be great if you, Nicolas, Richard, and Erik H
if he's still interested, could also keep the rest of us up to date with
your efforts -- in all likelihood you will be moving faster than me (though
I can speak for Will, Bill and John).  I'll try to stay current with what
you're doing and make what contributtions I can.  thanks Aaron!


> Thanks again,
>
> --
> Aaron Ecay
>

[-- Attachment #2: Type: text/html, Size: 7595 bytes --]

Re: contributing to work on citations

[Matt Price]

[-- Attachment #1: Type: text/plain, Size: 1901 bytes --]

On Wed, Nov 4, 2015 at 10:18 AM, Aaron Ecay <aaronecay@gmail.com> wrote:

> >
> > 3. Handling of citation links in the various export engines
> > Functions must be written to handle citations in, at minimum, latex,
> html,
> > and org
> >
> > 4. Interface with backends
> > Org should be able to talk to bibtex and zotero databases, at least (and
> > maybe more).
>
> The code I wrote should have enough hooks to enable this.  But I haven’t
> looked at zotero at all.  If zotero support is important to you, you
> could investigate the “lookup types” in the citation code I wrote, see
> if they are adequate, and try to use them to implement zotero lookup.
> From the docstring:
>
> "Types of citation lookup backends.
>
> Alist from type to list of:
>
> - Function called at the beginning of export, with the rest of
>   the keyword line after #+BIBDB: type, and the info plist.
>   Should cache whatever it needs in the info plist.
>
> - Function to lookup a citation.  Called with the key and the
>   info plist.  Will be memoized by
>   `org-export-cite--lookup' (TODO).  Should return an alist of
>   keys and values about the citation (author, year, title, etc.)
>
> - A boolean; non-nil = this lookup type is remote.  All local
>   lookups will be tried before any remote one is.
>   TODO: not yet implemented"
>

Hey, Aaron,
Am I right that I need to check out your wip-cite-awe branch from ~8 months
ago? Would it make sense to try to merge recent changes on master into that
branch first?

Anyway, I figure I will try to modify zotxt-emacs so it provides these
functions and generates conformant cite: links.  That seems the easiest
solution, and maybe Erik will give me some help once I've made some initial
progress.  Meanwhile I will try to learn enough about javascript to get
started on the citeproc-js ocmmand-line tool.  thx,
matt

[-- Attachment #2: Type: text/html, Size: 2424 bytes --]

Re: contributing to work on citations

[Aaron Ecay]

Hi Matt,

2015ko azaroak 6an, Matt Price-ek idatzi zuen:

> Hey, Aaron,
> Am I right that I need to check out your wip-cite-awe branch from ~8 months
> ago? 

Richard Lawrence forked that branch, and made some changes on top of it,
so you should probably start with his branch.  His changes aren’t in the
org-mode repo but rather github: <https://github.com/wyleyr/org-mode.git>
(That should be a url suitable for git clone or git remote add; the branch
with his work in it is called wip-cite-org-citeproc and not master).

> Would it make sense to try to merge recent changes on master into that
> branch first?

That’s a good first step.  Hopefully any conflicts will be minor, but
don’t hesitate to shout for help if there are problems!

> 
> Anyway, I figure I will try to modify zotxt-emacs so it provides these
> functions and generates conformant cite: links.

NB the new citation support doesn’t use links à la org-ref, but rather a
dedicated syntax object (maybe that’s what you meant, but calling the
new citations “links” is not technically correct).

> That seems the easiest solution, and maybe Erik will give me some help
> once I've made some initial progress.  Meanwhile I will try to learn
> enough about javascript to get started on the citeproc-js ocmmand-line
> tool.  thx, matt

Thanks,

-- 
Aaron Ecay

Re: contributing to work on citations

[Matt Price]

[-- Attachment #1: Type: text/plain, Size: 1473 bytes --]

On Fri, Nov 6, 2015 at 8:10 AM, Aaron Ecay <aaronecay@gmail.com> wrote:

> Hi Matt,
>
> 2015ko azaroak 6an, Matt Price-ek idatzi zuen:
>
> > Hey, Aaron,
> > Am I right that I need to check out your wip-cite-awe branch from ~8
> months
> > ago?
>
> Richard Lawrence forked that branch, and made some changes on top of it,
> so you should probably start with his branch.  His changes aren’t in the
> org-mode repo but rather github: <https://github.com/wyleyr/org-mode.git>
> (That should be a url suitable for git clone or git remote add; the branch
> with his work in it is called wip-cite-org-citeproc and not master).
>
> > Would it make sense to try to merge recent changes on master into that
> > branch first?
>
> That’s a good first step.  Hopefully any conflicts will be minor, but
> don’t hesitate to shout for help if there are problems!
>
> OK, I tried rebasing on current master, is online here & pull request has
been sent ot richard:

https://github.com/titaniumbones/org-mode



> >
> > Anyway, I figure I will try to modify zotxt-emacs so it provides these
> > functions and generates conformant cite: links.
>
> NB the new citation support doesn’t use links à la org-ref, but rather a
> dedicated syntax object (maybe that’s what you meant, but calling the
> new citations “links” is not technically correct).
>
> Right. Of course. Thank you! I'll try to get started on this in the next
few days.

m

[-- Attachment #2: Type: text/html, Size: 2326 bytes --]

Re: contributing to work on citations

[Richard Lawrence]

Hi Matt and all,

Matt Price <moptop99@gmail.com> writes:
>> OK, I tried rebasing on current master, is online here & pull request has
>> been sent ot richard:
>
> https://github.com/titaniumbones/org-mode

Sorry it took me so long to get around to this.  Because there have been
commits to master in the meantime, and because I wanted to clean up the
(unnecessarily complicated) history of my branch, I have performed
another rebase of my branch onto master and put the result in a new
branch, here:

https://github.com/wyleyr/org-mode/tree/wip-cite-rebase

This branch linearizes the history of the various changes that went into
org-element (parsing citations) and org-cite (handling citations), and
cleans up my commit messages, which were produced by an old magit and
pretty ugly to deal with.

Hopefully that will make it easier for someone to continue this work!

Best,
Richard

Re: contributing to work on citations

[Rasmus]

Hi,

Thanks for looking at this, Matt and Aaron.

Matt Price <moptop99@gmail.com> writes:

> Anyway, I figure I will try to modify zotxt-emacs so it provides these
> functions and generates conformant cite: links.  That seems the easiest
> solution, and maybe Erik will give me some help once I've made some initial
> progress.  Meanwhile I will try to learn enough about javascript to get
> started on the citeproc-js ocmmand-line tool.  thx,

I use JS sometimes, and FWIW I thinking your plan of altering citeproc-js
upstream is the best approach proposed thus far.

ATM I don’t have much time, and I having a hard time forming a overview of
where to aid wrt. adding citation support.

Rasmus

-- 
And when I’m finished thinking, I have to die a lot