Citations, continued

Citations, continued

[Richard Lawrence]

Hi all,

I wanted to continue the discussion that began in this thread about
adding citation support to Org:
http://thread.gmane.org/gmane.emacs.orgmode/94352/focus=94412

Here are some thoughts I have after reviewing that discussion:

1) Lots of people seem to need/want better support for citations in Org
documents.  Other projects (Pandoc among them) already have support for
citations which is good enough that at least some people are using them
to fill this perceived gap in Org's support.

2) There are at least several different backend reference database
formats (BibTeX, Zotero, etc.) used by Orgsters.  Not all such databases
use human-readable keys.  Org also has a nice internal format for
storing reference information: org-bibtex.

3) There are also several different frontend solutions for making
citations in Org documents (org-ref, various `home-brewed' solutions,
raw LaTeX \cite commands, etc.).  The variety here is at least in part
due to our different requirements for the format of exported documents:
e.g., some people only care about exporting to LaTeX, others need
something that will work for HTML or ODT; some people need to specify
pre- and post-text for citations; some people need to specify citation
types.

4) Because individual Orgsters have widely varying needs, there is some
disagreement about what `proper' citation support should look like.  (Do
we need new syntax, or can existing syntax be used?  Which features need
to be supported by Org, and which can be provided by external tools?
etc.)

It seems there are three distinct but related problems:
  - representing citations in Org documents
  - exporting citations in an Org document to a backend document format
    like LaTeX, ODT, or HTML
  - searching for and manipulating keys in a reference database from
    within Org, and otherwise fostering good communication between
    such a database and citations in Org documents

Here's my personal opinion about how we might solve them.

As for the first problem, I think a good case can be made for adding new
syntax to Org to represent citations, instead of repurposing/extending
existing syntax (most notably, the link syntax).  

Here's why.  Citations are complicated, and it's clear that some sort of
new syntax is needed to represent them.  Even link-based solutions
introduce new syntax `inside' the link syntax, such as using `::' to
separate pre-text and post-text in a link description, as org-ref does.
Thus, the issue is not *whether* there should be additional syntax, but
just *how constrained* it should be.  In particular, the question is
whether we want to make citation syntax a subset of link syntax, or
whether citations and links should be distinct types of syntactic
elements.

It seems to me that the needs of citation users are wide enough and
complicated enough that it is worth shedding the constraints imposed by
the link syntax.  Eventually, packing all the representations we need
for a general solution (citation type, pre- and post-text, suppressing
author name, etc. etc.) into the link syntax will tend to make citations
unreadable.

Moreover, citations are not really links, even though it is often useful
to treat them (or parts of them) as links.  For example, a link can only
point to one target; yet a single citation often points to multiple
sources.  And what if you need different pre- or post-text for different
sources within the same citation, like ``See @Doe1999 for an overview; a
more extensive discussion is in @Foobar2000''?

Finally, it seems to me that Org strives to be a document format which
is usable independently of any particular export format.  This counts in
favor of having citation syntax as a first-class citizen in Org; at
present, Org documents can only contain citations through a combination
of extensions and external tools.

For these reasons, I would support a separate citation syntax, but one
that can behave like a link when useful.  For example, suppose we
borrowed the Pandoc [ ... @key1 ...; ... @key2 ...] syntax.  When point
is on `@key1', C-c C-o could be bound to find the key in the reference
database, or another useful action, depending on the reference database
format.

As I mentioned in the earlier thread, I think the Pandoc syntax is a
good place to start, and I think it would be valuable to have the two
syntaxes be compatible.  But even Pandoc's citation syntax might not be
general enough to satisfy everyone's needs, so the first step for
introducing citation syntax to Org should be compiling a list of all the
things such a syntax should represent.

The other problems, I think, must wait until a stable citation syntax
emerges -- export support in particular.  (Using an existing syntax from
another project could help ease the transition here: if people can
export citations using an existing tool, they'll be able to switch to
that syntax immediately, and use the external tool in the meantime while
Org-internal support for it catches up.)

I hope this is a useful starting point for further discussion!

Best,
Richard

Re: Citations, continued

[Nicolas Goaziou]

Hello,

Richard Lawrence <richard.lawrence@berkeley.edu> writes:

> As I mentioned in the earlier thread, I think the Pandoc syntax is a
> good place to start, and I think it would be valuable to have the two
> syntaxes be compatible.  But even Pandoc's citation syntax might not be
> general enough to satisfy everyone's needs, so the first step for
> introducing citation syntax to Org should be compiling a list of all the
> things such a syntax should represent.

See also 

  <http://permalink.gmane.org/gmane.emacs.orgmode/72446>

BTW, I agree that a dedicated syntax is needed, but we failed to find
one so far.


Regards,

-- 
Nicolas Goaziou

Re: Citations, continued

[Richard Lawrence]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Richard Lawrence <richard.lawrence@berkeley.edu> writes:
>
>> ...so the first step for introducing citation syntax to Org should be
>> compiling a list of all the things such a syntax should represent.
>
> See also 
>
>   <http://permalink.gmane.org/gmane.emacs.orgmode/72446>

I reviewed that discussion, and also took a (relatively quick) look at
the citation features provided by BibLaTeX and Citation Style Language
processors.  Here's what I've come up with for an initial list of
requirements for citation syntax:

A citation is a textual reference to one or more individual works,
together with other information about those works, grouped together in
a single place.

Within a citation, each reference to an individual work needs to be
capable of containing:
  1) a database key that references the cited work
  2) prefix / pre-text
  3) suffix / post-text
  4) references to page/chapter/section/whatever numbers and ranges.
     This is likely part of the prefix or suffix, but might be worth
     parsing separately for localization or link-following behavior.
  5) a way of indicating backend-agnostic formatting properties.
     Examples of some properties users might want to specify are:
     - displaying only some fields (or suppressing some fields) from a
       reference record (e.g., journal, date, author)
     - indicating that the referenced works should *only* appear in
       the bibliography of the exported document (equivalent of LaTeX
       \nocite)
       
Citations as a whole also need:
  6) [@6] a way of indicating formatting properties for specific export
     backends.  Examples of some properties that users might want to
     specify are:
     - a citation command to use for each individual reference (LaTeX;
       others?)
     - a multi-cite command to apply to all references together
       (LaTeX)
     - CSS or other styling class (HTML and derived backends; also
       ODT?)
     - properties describing how to treat emphasis and other
       formatting that cannot appear in plain text (ASCII and other
       plain text backends)

In addition to the syntax of citations themselves, the Org document
would also need to represent the following metadata to support
citations:
  7) [@7] a pointer to one or more backend reference databases,
     including in-document databases in org-bibtex format
  8) a reference to a citation style or style file
  9) a reference to a locale file 
  10) an indication of where the bibliography should be found in the
      exported document (equivalent to \printbibliography, etc. in
      LaTeX)

I would like to know if others can think of anything else that should go
on this list.  I am particularly interested in hearing from people who
use (or want to use) citations with non-LaTeX export backends, since I
am least familiar with how citations work in those types of documents.

I have also been working on a proposal for citation syntax that I think
will meet these requirements, which I will post separately.

Best,
Richard

Re: Citations, continued

[Rasmus]

Richard Lawrence <richard.lawrence@berkeley.edu> writes:

> Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:
>
>> Richard Lawrence <richard.lawrence@berkeley.edu> writes:
>>
>>> ...so the first step for introducing citation syntax to Org should be
>>> compiling a list of all the things such a syntax should represent.
>>
>> See also 
>>
>>   <http://permalink.gmane.org/gmane.emacs.orgmode/72446>
>
> Within a citation, each reference to an individual work needs to be
> capable of containing:
>   1) a database key that references the cited work
>   2) prefix / pre-text
>   3) suffix / post-text
>   4) references to page/chapter/section/whatever numbers and ranges.
>      This is likely part of the prefix or suffix, but might be worth
>      parsing separately for localization or link-following behavior.
>   5) a way of indicating backend-agnostic formatting properties.
>      Examples of some properties users might want to specify are:

>      - displaying only some fields (or suppressing some fields) from a
>        reference record (e.g., journal, date, author)

Would this not be properties of the bibliography and not the citation?


> Citations as a whole also need:
>   6) [@6] a way of indicating formatting properties for specific export
>      backends.

I think the idea would be /not/ to have to consider specific backends.  If
you want special properties (say bold) for HTML could it not be solved by
a macro or a filter?  Probably I'm misunderstanding.

>      ...
>      - CSS or other styling class (HTML and derived backends; also
>        ODT?)

The user solves this by writing CSS.  Of course citations would be wrapped
in a span or whatever.

>      - properties describing how to treat emphasis and other
>        formatting that cannot appear in plain text (ASCII and other
>        plain text backends)

IMO this is solved by ox-ascii.el already.

> In addition to the syntax of citations themselves, the Org document
> would also need to represent the following metadata to support
> citations:
>   7) [@7] a pointer to one or more backend reference databases,
>      including in-document databases in org-bibtex format

This would be a huge win.

>   8) a reference to a citation style or style file

How does this work outside of LaTeX?

>   9) a reference to a locale file

There's already a #+BIBLIOGRAPHY or #+REFERENCES in ox-bibtex.el.  But
it's quite limited.

>   10) an indication of where the bibliography should be found in the
>       exported document (equivalent to \printbibliography, etc. in
>       LaTeX)


> I would like to know if others can think of anything else that should go
> on this list.  I am particularly interested in hearing from people who
> use (or want to use) citations with non-LaTeX export backends, since I
> am least familiar with how citations work in those types of documents.

I would use citations in html and even odt.  Put it's a hard problem
'cause there's nothing quite like bib(la)tex (to the best of my
knowledge).

> I have also been working on a proposal for citation syntax that I think
> will meet these requirements, which I will post separately.

Cool!  Let me know if I can help.  

I have mainly worked on regexps for the syntax I proposed in another email.

—Rasmus

-- 
The second rule of Fight Club is: You do not talk about Fight Club

Re: Citations, continued

[Richard Lawrence]

Hi Rasmus and all,

Rasmus <rasmus@gmx.us> writes:

> Richard Lawrence <richard.lawrence@berkeley.edu> writes:

>> Within a citation, each reference to an individual work needs to be
>> capable of containing:
>>   1) a database key that references the cited work
>>   2) prefix / pre-text
>>   3) suffix / post-text
>>   4) references to page/chapter/section/whatever numbers and ranges.
>>      This is likely part of the prefix or suffix, but might be worth
>>      parsing separately for localization or link-following behavior.
>>   5) a way of indicating backend-agnostic formatting properties.
>>      Examples of some properties users might want to specify are:
>
>>      - displaying only some fields (or suppressing some fields) from a
>>        reference record (e.g., journal, date, author)
>
> Would this not be properties of the bibliography and not the citation?

No, I mean things that can vary from one citation to the next -- like
what you'd write in LaTeX as

\citet{Doe99} once thought foo, but in his \citeyear{Doe2014}, he
revises his position to bar.

The second citation should only display the year, since the author's
name has already been mentioned in the sentence.  This type of
formatting information would be nice to represent regardless of export
backend.

>> Citations as a whole also need:
>>   6) [@6] a way of indicating formatting properties for specific export
>>      backends.
>
> I think the idea would be /not/ to have to consider specific backends.  If
> you want special properties (say bold) for HTML could it not be solved by
> a macro or a filter?  Probably I'm misunderstanding.

I agree that we should probably want to minimize and segregate
backend-specific formatting information, but it is inevitable that
someone will need this ability; and I'm a `provide reasonable defaults,
but also an escape hatch when you need it' kind of guy.

The things I have in mind are things like telling the LaTeX backend to
use a particular citation command for this citation, or the HTML backend
to use/add a particular CSS class.  Maybe this could be done with macros
or filters, but I think that would prove complicated for all but the
simplest cases, since citations have argument structure that filters
might not necessarily `see'.

>> In addition to the syntax of citations themselves, the Org document
>> would also need to represent the following metadata to support
>> citations:
>>   7) [@7] a pointer to one or more backend reference databases,
>>      including in-document databases in org-bibtex format
>
> This would be a huge win.
>
>>   8) a reference to a citation style or style file
>
> How does this work outside of LaTeX?

Well, Pandoc for example processes citations using the citeproc-hs
implementation of the Citation Style Language, which is an XML format
that allows describing how citations and bibliographies should be
formatted.  Thus, for example, you could tell Pandoc to process your
citations in APA style, or any of the other styles in this repo:

https://www.zotero.org/styles

CSL is an XML format, and I shudder to think about implementing it in
Elisp, but that's how its done.  In fact, Pandoc uses this even for
LaTeX output, rather than trying to map citations to the various \cite
commands.
 
>>   9) a reference to a locale file
>
> There's already a #+BIBLIOGRAPHY or #+REFERENCES in ox-bibtex.el.  But
> it's quite limited.

Yes, I think we should leverage this but modify its syntax a bit.

Best,
Richard

Re: Citations, continued

[Rasmus]

Hi,

Richard Lawrence <richard.lawrence@berkeley.edu> writes:

> Hi Rasmus and all,
>
> Rasmus <rasmus@gmx.us> writes:
>
>> Richard Lawrence <richard.lawrence@berkeley.edu> writes:
>
>>> Within a citation, each reference to an individual work needs to be
>>> capable of containing:
>>>   1) a database key that references the cited work
>>>   2) prefix / pre-text
>>>   3) suffix / post-text
>>>   4) references to page/chapter/section/whatever numbers and ranges.
>>>      This is likely part of the prefix or suffix, but might be worth
>>>      parsing separately for localization or link-following behavior.
>>>   5) a way of indicating backend-agnostic formatting properties.
>>>      Examples of some properties users might want to specify are:
>>
>>>      - displaying only some fields (or suppressing some fields) from a
>>>        reference record (e.g., journal, date, author)
>>
>> Would this not be properties of the bibliography and not the citation?
>
> No, I mean things that can vary from one citation to the next -- like
> what you'd write in LaTeX as
>
> \citet{Doe99} once thought foo, but in his \citeyear{Doe2014}, he
> revises his position to bar.

Okay, I misunderstood you then.q I though you wanted something like
\AtEveryBibitem (of biblatex) which literally alters fields, e.g.:
\AtEveryBibitem{\clearfield{month}}.

>>> Citations as a whole also need:
>>>   6) [@6] a way of indicating formatting properties for specific export
>>>      backends.
>> I think the idea would be /not/ to have to consider specific backends.  If
>> you want special properties (say bold) for HTML could it not be solved by
>> a macro or a filter?  Probably I'm misunderstanding.
> [...]
> use a particular citation command for this citation, or the HTML backend
> to use/add a particular CSS class.  Maybe this could be done with macros
> or filters, but I think that would prove complicated for all but the
> simplest cases, since citations have argument structure that filters
> might not necessarily `see'.

I see.  It's possible via macros.  I don't have strong opinions on this.

>>>   8) a reference to a citation style or style file
>>
>> How does this work outside of LaTeX?
>
> Well, Pandoc for example processes citations using the citeproc-hs

It seems to use pandoc-citeproc which is based on citeproc-hs.

> implementation of the Citation Style Language, which is an XML format
> that allows describing how citations and bibliographies should be
> formatted.  Thus, for example, you could tell Pandoc to process your
> citations in APA style, or any of the other styles in this repo:
>
> https://www.zotero.org/styles
>
> CSL is an XML format, and I shudder to think about implementing it in
> Elisp, but that's how its done.  In fact, Pandoc uses this even for
> LaTeX output, rather than trying to map citations to the various \cite
> commands.

I wonder if Zotero can be used to format such citations.  It can do
something for rtf at least:

    https://www.zotero.org/support/rtf_scan

—Rasmus

-- 
A page of history is worth a volume of logic

Re: Citations, continued

[Erik Hetzner]

On Sat, 31 Jan 2015 at 10:26:05 PST,
Richard Lawrence <richard.lawrence@berkeley.edu> wrote:
> 
> Hi all,
>
> […]
>
> As I mentioned in the earlier thread, I think the Pandoc syntax is a
> good place to start, and I think it would be valuable to have the two
> syntaxes be compatible.  But even Pandoc's citation syntax might not be
> general enough to satisfy everyone's needs, so the first step for
> introducing citation syntax to Org should be compiling a list of all the
> things such a syntax should represent.

Hi Richard,

It would probably be worth reviewing the discussions that led to the
pandoc citation syntax:

  https://groups.google.com/d/msg/pandoc-discuss/v54VxMXtoWM/7ezu4xDSN8gJ

Of the requirements you later mentioned:

>   1) a database key that references the cited work
>   2) prefix / pre-text
>   3) suffix / post-text
>   4) references to page/chapter/section/whatever numbers and ranges.
>      This is likely part of the prefix or suffix, but might be worth
>      parsing separately for localization or link-following behavior.
>   5) a way of indicating backend-agnostic formatting properties.
>      Examples of some properties users might want to specify are:
>      - displaying only some fields (or suppressing some fields) from a
>        reference record (e.g., journal, date, author)
>      - indicating that the referenced works should *only* appear in
>        the bibliography of the exported document (equivalent of LaTeX
>        \nocite)

pandoc syntax handles 1, 2, 3, 4 (locator, which is separate from
suffix) and some of 5 (author suppression; I’m not sure the use cases
for journal/date/etc. suppression). It does not handle \nocite, but
this is something that they have discussed in the past and could
easily be supported in org-mode using, e.g. a #+NO_CITE block.
Personally I feel this covers the 99% of uses while being human
readable and writable.

I agree that citekey management is something that should be handle
separately, as this will depend on the backend (bibtex file, zotero,
etc.) Whether these citekeys are also links doesn’t really matter that
much either and could depend on the backend.

best, Erik
--
Sent from my free software system <http://fsf.org/>.

Re: Citations, continued

[John Kitchin]

[-- Attachment #1: Type: text/plain, Size: 9293 bytes --]

>
>
>
Here's my personal opinion about how we might solve them.
>
> As for the first problem, I think a good case can be made for adding new
> syntax to Org to represent citations, instead of repurposing/extending
> existing syntax (most notably, the link syntax).
>

I think links are remarkable flexible, and solve most of the pre/post text
issue. You can use literal links if you don't want to see them collapsed.
Links already have a type, so they are easily differentiable from other
links and elements. The only limitation I can see is your example below
where I couldn't put two pre/post text citations in one link. I would write
that as:

See cite:Doe1999 for an overview; a more extensive discussion is in
cite:Foobar2000

if the pre/post text is really critical somehow, you can do this.

[[cite:Doe1999][See::for an overview]]; a more extensive discussion is in
cite:Foobar2000

I guess this would also be ok in orgref:
[[cite:Doe1999][See::for an overview]][[cite:Foobar2000][; a more extensive
discussion is in]]

Depending on your backend and citation style, it might be necessary to use
Ref. citenum:Foobar2000 for it to look right on export.

Either way, none of these are more readable than the first one I think, and
they would probably all render the same in any backend.



> Here's why.  Citations are complicated, and it's clear that some sort of
> new syntax is needed to represent them.  Even link-based solutions
> introduce new syntax `inside' the link syntax, such as using `::' to
> separate pre-text and post-text in a link description, as org-ref does.
> Thus, the issue is not *whether* there should be additional syntax, but
> just *how constrained* it should be.  In particular, the question is
> whether we want to make citation syntax a subset of link syntax, or
> whether citations and links should be distinct types of syntactic
> elements.
>
> It seems to me that the needs of citation users are wide enough and
> complicated enough that it is worth shedding the constraints imposed by
> the link syntax.  Eventually, packing all the representations we need
> for a general solution (citation type, pre- and post-text, suppressing
> author name, etc. etc.) into the link syntax will tend to make citations
> unreadable.
>

Making anything unreadable should be avoided at all cost!


> Moreover, citations are not really links, even though it is often useful
> to treat them (or parts of them) as links.  For example, a link can only
> point to one target; yet a single citation often points to multiple
> sources.  And what if you need different pre- or post-text for different
> sources within the same citation, like ``See @Doe1999 for an overview; a
> more extensive discussion is in @Foobar2000''?
>

An org link can point to many targets. An org-mode link executes code,
which can give you many options to consider. In your example, I would
consider that two citations. Even a single cite link in org-ref considers
each comma-separated key a separate clickable entity; the code figures out
which key you clicked on, and gives you several options of what to do with
it, like open the entry, the pdf, etc... which it figures out how to do
based on the key.


>
> Finally, it seems to me that Org strives to be a document format which
> is usable independently of any particular export format.  This counts in
> favor of having citation syntax as a first-class citizen in Org; at
> present, Org documents can only contain citations through a combination
> of extensions and external tools.
>
> For these reasons, I would support a separate citation syntax, but one
> that can behave like a link when useful.  For example, suppose we
> borrowed the Pandoc [ ... @key1 ...; ... @key2 ...] syntax.  When point
> is on `@key1', C-c C-o could be bound to find the key in the reference
> database, or another useful action, depending on the reference database
> format.
>

How does the pandoc syntax handle different link types. e.g. the 40ish
different types supported by biblatex? Links manage this easily, and with
the export framework, you can convert them to all kinds of other formats.
It looks to me like the pandoc syntax is too simple for that.

  '("cite" "nocite" ;; the default latex cite commands
    ;; natbib cite commands,
http://ctan.unixbrain.com/macros/latex/contrib/natbib/natnotes.pdf
    "citet" "citet*" "citep" "citep*"
    "citealt" "citealt*" "citealp" "citealp*"
    "citenum" "citetext"
    "citeauthor" "citeauthor*"
    "citeyear" "citeyear*"
    "Citet" "Citep" "Citealt" "Citealp" "Citeauthor"
    ;; biblatex commands
    ;; http://ctan.mirrorcatalogs.com/macros/latex/contrib/biblatex/doc/biblatex.pdf
    "Cite"
    "parencite" "Parencite"
    "footcite" "footcitetext"
    "textcite" "Textcite"
    "smartcite" "Smartcite"
    "cite*" "parencite*" "supercite"
    "autocite" "Autocite" "autocite*" "Autocite*"
    "Citeauthor*"
    "citetitle" "citetitle*"
    "citedate" "citedate*"
    "citeurl"
    "fullcite" "footfullcite"
    ;; "volcite" "Volcite" cannot support the syntax
    "notecite" "Notecite"
    "pnotecite" "Pnotecite"
    "fnotecite"
    ;; multicites. Very limited support for these.
    "cites" "Cites" "parencites" "Parencites"
    "footcites" "footcitetexts"
    "smartcites" "Smartcites" "textcites" "Textcites"
    "supercites" "autocites" "Autocites"
    ;; for the bibentry package
    "bibentry"
    )



I think you need a citation type for the backend, because you often need
more than one type in a document, e.g. the regular citations, sometimes a
citation number, author or year. For a pandoc like link you could consider
(adapted from the pandoc examples):

cite@key1,key2,key3

[optional pre text citenum@key optional post text]

or a group like this

[see citetext@item1 p. 34-35; also citet*@item3 chap. 3].


where optional text might include additional citations

cite@key[citeauthor@key2, p. 30; see also Ref. citenum@key3)]

I can't say I look forward to figuring out how to convert that to latex (or
what the latex equivalent even is!)). There would have to be some mechanism
to define new citation types, analogous to adding new link types. I suppose
if org-element-map parsed these into data structures it might not be too
bad.

These look alot like replacing : with @ in a link to me, but maybe the
added flexibility is worth it. The simplest cases would act like links.
Looking back at this, I am not seeing a critical advantage over the links,
other than eliminating the :: separator in the description. Except for the
last example. I would never dream of  writing a citation like that though!
We never use pre/post text in citations in our work, and they don't even
make sense with all citation formats, e.g. superscripted numbers. Maybe
someone could provide some real life citation examples that links can't
handle?

I suspect a lot of pre/post text issues can be solved manually as:

(see cite:key1, pg33-4; also cite:key2, chapter 3)

and you will get what you want in the output.

I have no idea how hard it would be to implement something like that
syntax.


As I mentioned in the earlier thread, I think the Pandoc syntax is a
> good place to start, and I think it would be valuable to have the two
> syntaxes be compatible.  But even Pandoc's citation syntax might not be
> general enough to satisfy everyone's needs, so the first step for
> introducing citation syntax to Org should be compiling a list of all the
> things such a syntax should represent.
>

The backend responsible for rendering the citation needs:
1. how to render the citation in the text (some hint about a type, pre and
post text)
2. how to render the citation in the bibliography (it will get all the
information from somewhere, based on the key of the citation, and prepare
the citation).

In Latex, you specify a style somewhere that says how to format a cite,
cite*, citenum, etc... and what the bibliography entries look like (unless
you use thebibliography, in which case you do that work yourself).

in org-ref, it gets the type from a link, the key from the link, and
pre/post text from the link, and converts that to the latex form needed by
the latex bibliography backend. Something similar happens in the html
export, except I build the bibliography from the bibtex file, and then go
back and put html links in place for citations. nothing fancy.

In word, I suppose there are little fields in the main document, and you
run some function that fires up zotero/endnote/mendeley, etc... that does
the formatting.


>
> The other problems, I think, must wait until a stable citation syntax
> emerges -- export support in particular.  (Using an existing syntax from
> another project could help ease the transition here: if people can
> export citations using an existing tool, they'll be able to switch to
> that syntax immediately, and use the external tool in the meantime while
> Org-internal support for it catches up.)
>
> I hope this is a useful starting point for further discussion!
>

So, after working through all that, I still think links are good enough for
a large portion of citations. org should not try to solve all of these
problems. That said, I would be interested again to see real life citation
needs that links are too awkward to handle.



>
> Best,
> Richard
>
>
>

[-- Attachment #2: Type: text/html, Size: 19589 bytes --]

Re: Citations, continued

[Richard Lawrence]

Hi John and all,

John Kitchin <jkitchin@andrew.cmu.edu> writes:

>> As for the first problem, I think a good case can be made for adding new
>> syntax to Org to represent citations, instead of repurposing/extending
>> existing syntax (most notably, the link syntax).
>>
>
> I think links are remarkable flexible, and solve most of the pre/post text
> issue. You can use literal links if you don't want to see them collapsed.

I agree; links *are* remarkably flexible.  But I think that's actually
sort of the problem in this case.  It's easy to use the link syntax to
implement a solution for citations that is `good enough' for 80+% of the
use cases.  That becomes a barrier to adopting something which better
advances the goals of Org and its users.

If I was implementing a citations solution just for my own Org
documents, I would do it exactly the way org-ref does: use the link
syntax, and put the pre- and post-text in the description, with a
separator that's easy to remember, like

[[cite:Doe1999][See::for an overview]] 

or more realistically: 

[[cite:Doe1999][Cf. a recent discussion in::, on sluicing.]]

Does this work?  Yes.  Is it easy to implement on top of the existing
features?  Yes.  Again, for my own personal use, this is exactly what I
would do.

But I would not want to see this syntax canonized as the official Org
citation syntax.  It feels like a hack, not a proper solution, and it
definitely strikes me as less readable than

[Cf. a recent discussion in @Doe1999, on sluicing.] 

especially if you have the default setup, where the URLS of links that
have a description part are hidden.

My point is not that the link syntax *can't* do enough.  Rather, my
point is that citations are conceptually distinct from links, and if we
are going to adopt an official syntax for them, that syntax should
reflect this conceptual distinction.  This is better for document
authors, because it is less work for us.  It gives us the right tool for
this particular job, instead of re-purposing a tool that, despite its
power, is designed for a different job.  It is thus better for the Org
community as a whole.

>> Moreover, citations are not really links, even though it is often useful
>> to treat them (or parts of them) as links.  For example, a link can only
>> point to one target; yet a single citation often points to multiple
>> sources.  And what if you need different pre- or post-text for different
>> sources within the same citation, like ``See @Doe1999 for an overview; a
>> more extensive discussion is in @Foobar2000''?
>>
>
> An org link can point to many targets. An org-mode link executes code,
> which can give you many options to consider. In your example, I would
> consider that two citations. Even a single cite link in org-ref considers
> each comma-separated key a separate clickable entity; the code figures out
> which key you clicked on, and gives you several options of what to do with
> it, like open the entry, the pdf, etc... which it figures out how to do
> based on the key.

This is a good example of how extending the link syntax to citations
will end up obscuring important conceptual distinctions, which makes
more work for the user.  

Normally, when you click on a link (or follow it with C-c C-o), you
don't have to pay attention to /where/ on the link you click.  By making
citation keys significant sub-parts of a link, you force the user to
always pay attention to this for every link, at least until he or she
determines whether the link is a citation link or a `real' link.  (If
that doesn't seem like a big deal, consider how it would affect someone
who edits Org documents using a screen reader.)

A link has two significant parts: a URL, and a description.  A citation
potentially has at least three significant parts for each work it
references (key, pre-text and post-text).  

I am certainly not opposed to having parts of citations exhibit
link-like behavior, though.  I think it would be great if one could
`follow' each reference key within a citation to do something useful
with it, as org-ref does.  I just think the link syntax is not the right
underlying syntax to express this.  It does not contain enough
significant parts -- and giving it more significant parts makes links
into something else.

> How does the pandoc syntax handle different link types. e.g. the 40ish
> different types supported by biblatex? Links manage this easily, and with
> the export framework, you can convert them to all kinds of other formats.
> It looks to me like the pandoc syntax is too simple for that.
> ...
> I think you need a citation type for the backend, because you often need
> more than one type in a document, e.g. the regular citations, sometimes a
> citation number, author or year.

Yes, I agree here.  As far as I can tell, the Pandoc syntax does not
handle different types of citations within the same document, except
in a limited way, via the distinctions between

Smith says blah [@Smith99, p. 3].
@Smith99 [p. 3] say blah.
Smith once thought foo [-@Smith99], but now thinks bar [-@Smith2014].

which basically correspond in the LaTeX world to natbib's \citep,
\citet, and \citeyear in the default (Chicago) citation styling.  Pandoc
does not map these different citation types to LaTeX commands directly,
though.  Instead, it formats citations using citeproc-hs' implmentation
of the Citation Style Language, then inserts the resulting text into the
document where the citation is supposed to appear.

I agree that some sort of fallback may be needed so that you can
instruct the LaTeX exporter, in particular, which command to use for a
given citation.  

But Org is a document format, not a nicer-looking frontend for LaTeX (as
other people on this list have had to remind me in the past!).  The
official syntax should distinguish between citations, on the one hand,
and instructions to particular backends for formatting them, on the
other.  Users should not have to learn the vagaries of LaTeX citation
commands in order to represent citations in an Org document. 

There is existing syntax for giving instructions to particular export
backends about how to format Org elements: the #+ATTR_BACKEND syntax.
Something like this is the right way to pass citation formatting
information to LaTeX when needed.  Unfortunately, this can't be applied
to inline citations directly, so something else is required.  I have an
idea about how to do this, which I will post when I finish writing it
up.

Best,
Richard

Re: Citations, continued

[Thomas S. Dye]

Aloha Richard,

Richard Lawrence <richard.lawrence@berkeley.edu> writes:

> My point is not that the link syntax *can't* do enough.  Rather, my
> point is that citations are conceptually distinct from links, and if we
> are going to adopt an official syntax for them, that syntax should
> reflect this conceptual distinction.  This is better for document
> authors, because it is less work for us.  It gives us the right tool for
> this particular job, instead of re-purposing a tool that, despite its
> power, is designed for a different job.  It is thus better for the Org
> community as a whole.

I agree that citations are conceptually distinct from links, but at the
same time they share many features.  Both can refer to something outside
the Org mode document.  Both can be replaced in the Org mode export with
something from outside the Org mode document.  The fact that links can
be made to handle most users' citation needs is practical proof that the
similarities are more than superficial.

Now, I agree with you that Org mode links are not ideal for citations.
Parsing the description is humbug and error-prone, and the descriptions
look ungainly in the Org mode document.  I never remember to click
citation links in the "right" place!  There is much room for improvement
here.

You and others are advocating a separate syntax for links and citations,
which might indeed be the way to go.  I can see it being much nicer than
the current state of affairs with Org mode links.  The downside is that
it will mean learning another set of rules, in addition to the existing
rules for links.

Several years ago, Samuel Wales suggested an extensible syntax example
using link features
(https://lists.gnu.org/archive/html/emacs-orgmode/2010-08/msg00404.html).
At the time it seemed to me that this was a Lisp-y approach because it
solved particular problems by generalizing or abstracting a language
feature to include particulars that had previously fallen outside its
ken.  I wanted something like this while I was working on implementing
citation links for export to LaTeX.

Would it be feasible to generalize Org mode's link syntax, or make it
extensible, so the overlap of link with citation is complete?  

All the best,
Tom

-- 
Thomas S. Dye
http://www.tsdye.com

Re: Citations, continued

[John Kitchin]

Thomas S. Dye writes:

> Aloha Richard,
>
> Richard Lawrence <richard.lawrence@berkeley.edu> writes:
>
>> My point is not that the link syntax *can't* do enough.  Rather, my
>> point is that citations are conceptually distinct from links, and if we
>> are going to adopt an official syntax for them, that syntax should
>> reflect this conceptual distinction.  This is better for document
>> authors, because it is less work for us.  It gives us the right tool for
>> this particular job, instead of re-purposing a tool that, despite its
>> power, is designed for a different job.  It is thus better for the Org
>> community as a whole.
>
> I agree that citations are conceptually distinct from links, but at the
> same time they share many features.  Both can refer to something outside
> the Org mode document.  Both can be replaced in the Org mode export with
> something from outside the Org mode document.  The fact that links can
> be made to handle most users' citation needs is practical proof that the
> similarities are more than superficial.
>
> Now, I agree with you that Org mode links are not ideal for citations.
> Parsing the description is humbug and error-prone, and the descriptions
> look ungainly in the Org mode document.  I never remember to click
> citation links in the "right" place!  There is much room for improvement
> here.

I am not sure how much better it could get. What did you have in mind? I
could imagine for a cite link with several citations the click action
could give you an ido-completing/helm selection buffer and you choose
what you want to do from there. In org-ref the click action is user
definable, so you can get what you want.

>
> You and others are advocating a separate syntax for links and citations,
> which might indeed be the way to go.  I can see it being much nicer than
> the current state of affairs with Org mode links.  The downside is that
> it will mean learning another set of rules, in addition to the existing
> rules for links.
>
> Several years ago, Samuel Wales suggested an extensible syntax example
> using link features
> (https://lists.gnu.org/archive/html/emacs-orgmode/2010-08/msg00404.html).
> At the time it seemed to me that this was a Lisp-y approach because it
> solved particular problems by generalizing or abstracting a language
> feature to include particulars that had previously fallen outside its
> ken.  I wanted something like this while I was working on implementing
> citation links for export to LaTeX.
>

I am totally for this idea! I have been pondering how to make a link
element with extra data in it for a while.


> Would it be feasible to generalize Org mode's link syntax, or make it
> extensible, so the overlap of link with citation is complete?
>
> All the best,
> Tom

--
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu

Re: Citations, continued

[Thomas S. Dye]

Hi John,

John Kitchin <jkitchin@andrew.cmu.edu> writes:

>> Now, I agree with you that Org mode links are not ideal for citations.
>> Parsing the description is humbug and error-prone, and the descriptions
>> look ungainly in the Org mode document.  I never remember to click
>> citation links in the "right" place!  There is much room for improvement
>> here.
>
> I am not sure how much better it could get. What did you have in mind? I
> could imagine for a cite link with several citations the click action
> could give you an ido-completing/helm selection buffer and you choose
> what you want to do from there. In org-ref the click action is user
> definable, so you can get what you want.

I didn't mean to imply any criticism of org-ref, which I haven't used.
I've been using my own home-grown solution for several years now, have
grown used to its limitations, and of course now have all those legacy
documents to support ...

It would be nice to have the link syntax extended or generalized to
indicate pre- and post-note text, so my home-grown system would use
links compatible with org-ref, say.  I really like the direction Richard
is heading for this reason.  If the distinctions needed for citations
were recognized in the Org mode core, then citation links used by
different systems might be compatible with one another.

More control over how links are displayed would be nice (perhaps let the
user pass in a function?).  When I first worked on setting up citations,
I thought it would be great to color citation links differently from
other links, and I still kind of like that idea.  Also, in my setup, I
don't want to see the full link because the bibtex keys are long and the
full links really break up the flow of the text.  Among other things,
this means I can't be sure just looking at the buffer in its typical
state what kind of link I'm using (footcite, parencite, textcite, etc.).
A little indicator of some kind would be really nice here, but I haven't
found an easy way to display one.

I'm glad you like Samuel's idea about extensible syntax for links.  I
don't know if it is practical or not, but it was one of those ideas that
seemed right on when I first read it.

All the best,
Tom

-- 
T.S. Dye & Colleagues, Archaeologists
735 Bishop St, Suite 315, Honolulu, HI 96813
Tel: 808-529-0866, Fax: 808-529-0884
http://www.tsdye.com

Re: Citations, continued

[John Kitchin]

You can color your own links like this:

 (highlight-regexp "cite:\\([a-zA-Z0-9]*[-_:]*\\)*" 'dired-warning)

 (highlight-regexp "citenum:\\([a-zA-Z0-9]*[-_:]*\\)*" 'dired-marked)

The regexps are not as robust as what org uses, the plain link on my
system looks like:

"\\<\\(Autocite[*s]?\\|Cite\\(?:a\\(?:l[pt]\\|uthor\\*?\\)\\|[pst]\\)?\\|Gls\\(?:pl\\)?\\|Notecite\\|P\\(?:arencites?\\|notecite\\)\\|ResearcherID\\|Smartcites?\\|Textcites?\\|a\\(?:dd\\(?:bibresource\\|ressbook\\)\\|ns\\|ssignment\\|ttachfile\\|utocite[*s]?\\)\\|b\\(?:bdb\\|ib\\(?:entry\\|liography\\(?:style\\)?\\|tex\\)\\)\\|cite\\(?:a\\(?:l\\(?:[pt]\\*\\|[pt]\\)\\|uthor\\*?\\)\\|date\\*?\\|num\\|p\\*\\|t\\(?:\\*\\|ext\\|itle\\*?\\)\\|url\\|year\\*?\\|[*pst]\\)?\\|do\\(?:cview\\|i\\)\\|e\\(?:l\\(?:feed\\|isp\\)\\|qref\\|xercise\\)\\|f\\(?:ile\\(?:\\+\\(?:\\(?:emac\\|sy\\)s\\)\\)?\\|notecite\\|oot\\(?:cite\\(?:s\\|texts?\\)?\\|fullcite\\)\\|tp\\|u\\(?:llcite\\|nc\\)\\)\\|g\\(?:ls\\(?:pl\\)?\\|nus\\|oogle\\)\\|https?\\|i\\(?:d\\|n\\(?:dex\\|fo\\)\\|rc\\)\\|l\\(?:abel\\|ist-of-\\(?:\\(?:figur\\|tabl\\)es\\)\\)\\|m\\(?:a\\(?:c-outlook\\|ilto\\)\\|c\\|essage\\|he\\|od\\|sx\\|u4e\\)\\|n\\(?:ameref\\|ew\\(?:glossaryentry\\|s\\)\\|ihmsid\\|o\\(?:bibliography\\|\\(?:te\\)?cite\\)\\)\\|orcid\\|p\\(?:a\\(?:geref\\|rencite[*s]?\\)\\|m\\(?:c?id\\)\\|notecite\\|rint\\(?:bibliography\\|index\\)\\|ydoc\\)\\|r\\(?:ef\\|mail\\)\\|s\\(?:hell\\|kim\\|link\\|martcites?\\|olution\\|upercites?\\)\\|t\\(?:extcites?\\|q-index\\)\\|x-together-item\\):\\([^
\n()<>]+\\(?:([[:word:]0-9_]+)\\|\\([^[:punct:]
\n]\\|/\\)\\)\\)"


But, if you put that in init files, then your links will be colored the
way you want them.

Thomas S. Dye writes:

> Hi John,
>
> John Kitchin <jkitchin@andrew.cmu.edu> writes:
>
>>> Now, I agree with you that Org mode links are not ideal for citations.
>>> Parsing the description is humbug and error-prone, and the descriptions
>>> look ungainly in the Org mode document.  I never remember to click
>>> citation links in the "right" place!  There is much room for improvement
>>> here.
>>
>> I am not sure how much better it could get. What did you have in mind? I
>> could imagine for a cite link with several citations the click action
>> could give you an ido-completing/helm selection buffer and you choose
>> what you want to do from there. In org-ref the click action is user
>> definable, so you can get what you want.
>
> I didn't mean to imply any criticism of org-ref, which I haven't used.
> I've been using my own home-grown solution for several years now, have
> grown used to its limitations, and of course now have all those legacy
> documents to support ...
>
> It would be nice to have the link syntax extended or generalized to
> indicate pre- and post-note text, so my home-grown system would use
> links compatible with org-ref, say.  I really like the direction Richard
> is heading for this reason.  If the distinctions needed for citations
> were recognized in the Org mode core, then citation links used by
> different systems might be compatible with one another.
>
> More control over how links are displayed would be nice (perhaps let the
> user pass in a function?).  When I first worked on setting up citations,
> I thought it would be great to color citation links differently from
> other links, and I still kind of like that idea.  Also, in my setup, I
> don't want to see the full link because the bibtex keys are long and the
> full links really break up the flow of the text.  Among other things,
> this means I can't be sure just looking at the buffer in its typical
> state what kind of link I'm using (footcite, parencite, textcite, etc.).
> A little indicator of some kind would be really nice here, but I haven't
> found an easy way to display one.
>
> I'm glad you like Samuel's idea about extensible syntax for links.  I
> don't know if it is practical or not, but it was one of those ideas that
> seemed right on when I first read it.
>
> All the best,
> Tom

--
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu

Re: Citations, continued

[John Kitchin]

> I'm glad you like Samuel's idea about extensible syntax for links.  I
> don't know if it is practical or not, but it was one of those ideas that
> seemed right on when I first read it.

I am glad you mentioned, it was an inspiration! Although this is sure to
move away from a standard new syntax, it is straightforward to subvert a
link like the following example to get more readable pre/post text
example. The quotes are necessary to get the list read
correctly. Whether this is useful remains to be seen, but it was fun to
work it out.

#+BEGIN_SRC emacs-lisp :results silent
(org-add-link-type
 "slink"
 ;;  follow function
 (lambda (path)
   (let* ((data (read (concat "(" path ")")))
          (head (car data))
          (plist (cadr data)))
     (message-box "%s\n%s\n%s" head plist  (plist-get plist :type))))
 ;; format function
 (lambda (path description backend)
   (let* ((data (read (concat "(" path ")")))
          (head (car data))
          (plist (cadr data)))
     (format "\\%s[%s][%s]{%s}"
             (plist-get plist :type)
             (plist-get plist :pre)
             (plist-get plist :post)
             head))))
#+END_SRC

[[slink:kitchin-2010 (:pre "See for example" :post "page 47" :type cite)]]

Exports to:

#+BEGIN_EXAMPLE
\cite[See for example][page 47]{kitchin-2010}
#+END_EXAMPLE




>
> All the best,
> Tom

--
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu

Re: Citations, continued

[Rasmus]

John Kitchin <jkitchin@andrew.cmu.edu> writes:

> #+BEGIN_SRC emacs-lisp :results silent
> (org-add-link-type
>  "slink" ...)
> #+END_SRC

Thanks John, this is great!

I managed to chop down my citation setup to the following:

[[cite:key :pre pre :post post :type type]]

with reasonable support for other backends.  I only use bibtex and I used
to separate pre/post with ";" so there's some legacy code in there.
Everything is hard-coded to my system/taste, written quickly and (very)
lightly tested, but maybe it will still be useful to somebody. . .

—Rasmus


(with-eval-after-load 'org
  (require 'reftex-cite)
  
  (defmacro rasmus/org-bib-add-type (type)
    ;; TODO: maybe this can be made more effective?
    ;; Seems to work OK...
    `(org-add-link-type
      ,type
      'rasmus/org-bib-follow
      ,(lambda (path description backend)
	 (funcall 'rasmus/org-bib-format path description backend
                  ;; cite defaults to textcite
		  (if (equal type "cite") "textcite" type)))))
  
  (mapc (lambda (type) (funcall 'rasmus/org-bib-add-type type))
        '("cite" "textcite" "parentcite" "citeyear" "citeauthor"))
  
  (defun rasmus/org-bib-follow (path)
    "Find the pdf version of citation."
    (let* ((stream (read (format "(%s)" path))))
      (rasmus/find-lit (car head))))

  (defun rasmus/find-lit (key)
    "Open pdf file associated with KEY from `reftex-default-bibliography'."
    (let* ((bib (file-name-directory (car reftex-default-bibliography)))
           (file (concat bib path (concat "/" key ".pdf"))))
      (when (file-exists-p file) (find-file file))))

  (defun rasmus/org-bib-format (path description backend &optional type*)
    "Format a org-link citation.

Support links of the type

[[type*:key :pre PRE :post POST :type TYPE**]]

Or

[[Key*:key][POST;PRE]]

Based on John K's great post here:
    http://permalink.gmane.org/gmane.emacs.orgmode/94575"
    (let* ((key
            ;; key is a single symbol by assumption
            (and (string-match "\\` *\\([^ ]+\\) *" path)
                 (prog1 (match-string 1 path)
                   (setq path (replace-match "" nil nil path)))))
           ;; generate plist
           (data (read (format "(%s)"
                               (replace-regexp-in-string
                                "\\(:\\w+\\) \\([^:]+\\) ?" "\\1 \"\\2\" "
                                path))))
           (type (or (plist-get data :type) type* "textcite"))
           (pre  (org-trim (or (plist-get data :pre)
			       ;; support my "old" syntax
			       (and description
				    (cadr (split-string description ";"))) "")))
	   (post (org-trim
		  (or (funcall (lambda (txt)
				 (and txt
				      (let ((res (string-to-number txt)))
					(if (zerop res) txt
					  (concat (if (> (length txt) 1)  "pp." "p.") " " txt)))))
			       (plist-get data :post))
		      (and description
			   (car (split-string description ";")))
		      "")))
	   (entry (or (save-window-excursion
                        (bibtex-search-entry key t 0)
                        (bibtex-parse-entry))
                      (error (format "unknown key: %s" key))))
           (author (or (reftex-format-citation entry "%2a") ""))
           (year (or  (reftex-format-citation entry "%y") "")))
      (if (org-export-derived-backend-p backend 'latex)
          (format "\\%s[%s][%s]{%s}" type pre post key)
        ;; TODO: This should probably be wrapped in <cite>.</cite> with html...
        (cl-case (intern type)
          (parencite
           (format "(%s %s %s %s)"
                   pre author
                   (or (and (org-string-nw-p year) (concat year ", ")) "")
                   post))
          (citeyear
           (format "%s %s %s" pre year post))
          (citeauthor
           (format "%s %s %s" pre author post))
          (fullcite
           (reftex-format-citation entry reftex-cite-view-format))
          (t ;; textcite
           (format "%s (%s%s%s)"
                   author
                   (and (org-string-nw-p pre) (concat pre " "))
                   year
                   (and (org-string-nw-p post) (concat ", " post)))))))))


-- 
Dung makes an excellent fertilizer

Re: Citations, continued

[Thomas S. Dye]

Hi John,

Wow.  It's inspiring to see Samuel's idea in action.

Presumably, something like this would also be possible?

;;  follow function
(lambda (path)
  (let* ((data (read (concat "(" path ")")))
         (head (car data))
         (plist (cadr data)))
    (funcall (plist-get plist :follow) head)))

[[slink:kitchin-2010 (:pre "See for example" :post "page 47" :type
cite :follow #'ebib-open-org-link)]]

All the best,
Tom

John Kitchin <jkitchin@andrew.cmu.edu> writes:

>> I'm glad you like Samuel's idea about extensible syntax for links.  I
>> don't know if it is practical or not, but it was one of those ideas that
>> seemed right on when I first read it.
>
> I am glad you mentioned, it was an inspiration! Although this is sure to
> move away from a standard new syntax, it is straightforward to subvert a
> link like the following example to get more readable pre/post text
> example. The quotes are necessary to get the list read
> correctly. Whether this is useful remains to be seen, but it was fun to
> work it out.
>
> #+BEGIN_SRC emacs-lisp :results silent
> (org-add-link-type
>  "slink"
>  ;;  follow function
>  (lambda (path)
>    (let* ((data (read (concat "(" path ")")))
>           (head (car data))
>           (plist (cadr data)))
>      (message-box "%s\n%s\n%s" head plist  (plist-get plist :type))))
>  ;; format function
>  (lambda (path description backend)
>    (let* ((data (read (concat "(" path ")")))
>           (head (car data))
>           (plist (cadr data)))
>      (format "\\%s[%s][%s]{%s}"
>              (plist-get plist :type)
>              (plist-get plist :pre)
>              (plist-get plist :post)
>              head))))
> #+END_SRC
>
> 
>
> Exports to:
>
> #+BEGIN_EXAMPLE
> \cite[See for example][page 47]{kitchin-2010}
> #+END_EXAMPLE
>
>
>
>
>>
>> All the best,
>> Tom
>
> --
> Professor John Kitchin
> Doherty Hall A207F
> Department of Chemical Engineering
> Carnegie Mellon University
> Pittsburgh, PA 15213
> 412-268-7803
> @johnkitchin
> http://kitchingroup.cheme.cmu.edu
>
>

-- 
Thomas S. Dye
http://www.tsdye.com

Re: Citations, continued

[John Kitchin]

Thanks! I had been thinking about how to do that for a while, and seeing
Samuel's idea crytallized it for me. Thanks Samuel, and Tom for
remembering it from long ago.

yes that should also be possible. once you open the rabbit hole of
embedded lisp, many things are possible ;) even beyond citations.

One note might be related to security that is worth thinking about. org
is conservative in the beginning in making you choose to turn off the
questions asking do you want to run this for babel, shell and elisp. In
your example, you are opening a hole I think. Suppose this is buried
deep in the document ;)

[[slink:kitchin-2010 (:pre "See for example" :post "page 47" :type cite
:follow (lambda (x) (message-box "I could be a shell command. please
please please click me or export me!")))]]

I tried it, and indeed it runs that function.

... Anyway, that is totally possible by many other means right now with
links, so it is not totally new, but it is probably worth thinking about
a little since code execution like this could make it out to the export.

Anyway, I look forward to seeing where it goes!


Thomas S. Dye writes:

> Hi John,
>
> Wow.  It's inspiring to see Samuel's idea in action.
>
> Presumably, something like this would also be possible?
>
> ;;  follow function
> (lambda (path)
>   (let* ((data (read (concat "(" path ")")))
>          (head (car data))
>          (plist (cadr data)))
>     (funcall (plist-get plist :follow) head)))
>
> [[slink:kitchin-2010 (:pre "See for example" :post "page 47" :type
> cite :follow #'ebib-open-org-link)]]
>
> All the best,
> Tom
>
> John Kitchin <jkitchin@andrew.cmu.edu> writes:
>
>>> I'm glad you like Samuel's idea about extensible syntax for links.  I
>>> don't know if it is practical or not, but it was one of those ideas that
>>> seemed right on when I first read it.
>>
>> I am glad you mentioned, it was an inspiration! Although this is sure to
>> move away from a standard new syntax, it is straightforward to subvert a
>> link like the following example to get more readable pre/post text
>> example. The quotes are necessary to get the list read
>> correctly. Whether this is useful remains to be seen, but it was fun to
>> work it out.
>>
>> #+BEGIN_SRC emacs-lisp :results silent
>> (org-add-link-type
>>  "slink"
>>  ;;  follow function
>>  (lambda (path)
>>    (let* ((data (read (concat "(" path ")")))
>>           (head (car data))
>>           (plist (cadr data)))
>>      (message-box "%s\n%s\n%s" head plist  (plist-get plist :type))))
>>  ;; format function
>>  (lambda (path description backend)
>>    (let* ((data (read (concat "(" path ")")))
>>           (head (car data))
>>           (plist (cadr data)))
>>      (format "\\%s[%s][%s]{%s}"
>>              (plist-get plist :type)
>>              (plist-get plist :pre)
>>              (plist-get plist :post)
>>              head))))
>> #+END_SRC
>>
>>
>>
>> Exports to:
>>
>> #+BEGIN_EXAMPLE
>> \cite[See for example][page 47]{kitchin-2010}
>> #+END_EXAMPLE
>>
>>
>>
>>
>>>
>>> All the best,
>>> Tom
>>
>> --
>> Professor John Kitchin
>> Doherty Hall A207F
>> Department of Chemical Engineering
>> Carnegie Mellon University
>> Pittsburgh, PA 15213
>> 412-268-7803
>> @johnkitchin
>> http://kitchingroup.cheme.cmu.edu
>>
>>

--
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu

Re: Citations, continued

[Rasmus]

tsd@tsdye.com (Thomas S. Dye) writes:

> You and others are advocating a separate syntax for links and citations,
> which might indeed be the way to go.  I can see it being much nicer than
> the current state of affairs with Org mode links.  The downside is that
> it will mean learning another set of rules, in addition to the existing
> rules for links.

So we bump the version number?  The [[cite:key]] would not stop working.

> Several years ago, Samuel Wales suggested an extensible syntax example
> using link features
> (https://lists.gnu.org/archive/html/emacs-orgmode/2010-08/msg00404.html).
> At the time it seemed to me that this was a Lisp-y approach because it
> solved particular problems by generalizing or abstracting a language
> feature to include particulars that had previously fallen outside its
> ken.  I wanted something like this while I was working on implementing
> citation links for export to LaTeX.
>
> Would it be feasible to generalize Org mode's link syntax, or make it
> extensible, so the overlap of link with citation is complete?  

I like :key entry-links and it would be powerful.  Certain like types
would also need to support custom display functions.  You could then
implement a cite-operator '@' as a handler for a generalized link.

—Rasmus

-- 
Governments should be afraid of their people

Re: Citations, continued

[Richard Lawrence]

tsd@tsdye.com (Thomas S. Dye) writes:

> You and others are advocating a separate syntax for links and citations,
> which might indeed be the way to go.  I can see it being much nicer than
> the current state of affairs with Org mode links.  The downside is that
> it will mean learning another set of rules, in addition to the existing
> rules for links.

Yes, I agree, and I see the import of this concern.  Links and citations
have enough similarities that some more generalized syntax could
probably cover them both, as you suggest below.

The main upside, as I see it, of adopting a totally different syntax for
citations is that we have the option to adopt a syntax that is already
known to be good enough (for some set of uses) and to work with other
tools (like Pandoc).  But this is a `merely practical' concern and maybe
does not outweight the considerations above.

> Several years ago, Samuel Wales suggested an extensible syntax example
> using link features
> (https://lists.gnu.org/archive/html/emacs-orgmode/2010-08/msg00404.html).
> At the time it seemed to me that this was a Lisp-y approach because it
> solved particular problems by generalizing or abstracting a language
> feature to include particulars that had previously fallen outside its
> ken.  I wanted something like this while I was working on implementing
> citation links for export to LaTeX.
>
> Would it be feasible to generalize Org mode's link syntax, or make it
> extensible, so the overlap of link with citation is complete?  

This is an interesting idea!

I think the minimal change that would be necessary would be to allow the
new extended links (call them `elinks' for short) to be defined in such
a way that they can have varying numbers of significant parts, depending
on the type.  An elink definition for a given type would specify a
number and order of parts.  All elinks would use the same syntax to
delimit the parts.  To keep things simple and more or less consistent
with the existing syntax, we could just surround each part with square
brackets, and assume that only the first part is required.

Thus, the relevant definitions could look like:

href: URL DESCRIPTION
cite: KEY PRE-TEXT POST-TEXT 

with examples like:

[href: [http://www.google.com]]
[href: [http://www.google.com][Evil search engine]]

[cite: [Smith99]]
[cite: [Smith99][Cf.]]
[cite: [Smith99][Cf.][for a discussion]]

though this doesn't quite solve the readability problem with having
optional pre and post text appear after the key.

In theory, this syntax could even allow complicated nestings:

multi-cite: CITE-ELINK ...

[multi-cite: [cite: [Smith99]]
             [cite: [Doe2000]]
             [cite: [Foobar2014][a summary appears in]]
             [cite: [Baz2014][][which is available at [href: [http://baz.org]]]]

though that might quickly get unwieldy (especially for non-Lispers). :)

Link handlers would become functions that accept one argument plus a
&rest list, and the choice of link handler would dispatch on the type.

An alternative to the brackets would be to use something like plist
syntax for the optional arguments (as Samuel originally suggested and
Rasmus has echoed).  Links would really just start to look like Lisp
function calls. 

I'm just spitballing here; not sure I'd ultimately endorse something
like this, but I think it's worthwhile to explore the question of how
link and citation syntax could usefully be generalized.

Best,
Richard

Re: Citations, continued

[Rasmus]

John Kitchin <jkitchin@andrew.cmu.edu> writes:

> See cite:Doe1999 for an overview; a more extensive discussion is in
> cite:Foobar2000

This is ok and supported by ox-bibtex.el.

> if the pre/post text is really critical somehow, you can do this.
>
> [[cite:Doe1999][See::for an overview]]; a more extensive discussion is in
> cite:Foobar2000

This is displayed as "See::for an overview" with an underline, which is
not really good enough.

> I guess this would also be ok in orgref:
> [[cite:Doe1999][See::for an overview]][[cite:Foobar2000][; a more extensive
> discussion is in]]

Which is displayed as 

See::for an overview; a more extensive discussion is in

Which is formidably unreadable!


> How does the pandoc syntax handle different link types.

We can make it more powerful as necessary.

> We never use pre/post text in citations in our work, and they don't even
> make sense with all citation formats, e.g. superscripted numbers. Maybe
> someone could provide some real life citation examples that links can't
> handle?

[[cite:key][prenote postnote]]

Per your examples above. 

> I suspect a lot of pre/post text issues can be solved manually as:
>
> (see cite:key1, pg33-4; also cite:key2, chapter 3)
> and you will get what you want in the output.

Note that \parencite[pre][post]{key} becomes (see author, year post).  I
don't know how to construct this in a simple way as (see author (year) post) 
is not good enough.

> In word, I suppose there are little fields in the main document, and you
> run some function that fires up zotero/endnote/mendeley, etc... that does
> the formatting.

I guess...

>>
>> The other problems, I think, must wait until a stable citation syntax
>> emerges -- export support in particular.  (Using an existing syntax from
>> another project could help ease the transition here: if people can
>> export citations using an existing tool, they'll be able to switch to
>> that syntax immediately, and use the external tool in the meantime while
>> Org-internal support for it catches up.)
>>
>> I hope this is a useful starting point for further discussion!
>>
>
> So, after working through all that, I still think links are good enough for
> a large portion of citations.

Not surprisingly I disagree.

It "breaks" whenever you have pre and post notes.  Author-Year is pretty
common and pre and post is pretty damn important.  Readability is
important and links fail at this step for any citation but the most simple
ones.

Links are "hard" to type outside of Emacs and logically/syntax-wise
unpleasant and are displayed poorly within Emacs.

—Rasmus

-- 
The second rule of Fight Club is: You do not talk about Fight Club

Re: Citations, continued

[Rasmus]

Richard Lawrence <richard.lawrence@berkeley.edu> writes:

> 2) There are at least several different backend reference database
> formats (BibTeX, Zotero, etc.) used by Orgsters.  Not all such databases
> use human-readable keys.  Org also has a nice internal format for
> storing reference information: org-bibtex.

Human readable keys are not a deal-breaker IMO.  Important is bibtex and
an entity-like display (e.g. Org displays \alpha as α etc.).

> 4) Because individual Orgsters have widely varying needs, there is some
> disagreement about what `proper' citation support should look like.  (Do
> we need new syntax, or can existing syntax be used?  Which features need
> to be supported by Org, and which can be provided by external tools?
> etc.)
>
> It seems there are three distinct but related problems:

>   1. representing citations in Org documents

>   2. exporting citations in an Org document to a backend document format
>      like LaTeX, ODT, or HTML

>   3. searching for and manipulating keys in a reference database from
>      within Org, and otherwise fostering good communication between
>      such a database and citations in Org documents

I would add:

    4a. Support backends  ...
    4b. ... and represent them internally.

    5. Formatting of citations.

    6. Nice representation in org-mode similar to entities.

As John K said, it is much preferable to have external tools handle 5.  If
there exist an external tool that can format handle formatting based on a
given backend, problem 2. and 5. disappear more or less.

If 5. is solved in elisp (e.g. using bibtex.el), problem 2. is "easy"
across all backends, but results might only be easily clickable/dynamic in
html.

1  → is an org-element.el problem.
3  → is easy for bibtex at least via Reftex.  My understanding is that
     Zotero also has well-supported selection functionality.
4b → allows for easy sharing of complete documents.

> I think a good case can be made for adding new syntax to Org to
> represent citations, instead of repurposing/extending existing syntax
> (most notably, the link syntax).

+1.

> For these reasons, I would support a separate citation syntax, but one
> that can behave like a link when useful.  For example, suppose we
> borrowed the Pandoc [ ... @key1 ...; ... @key2 ...] syntax.  When point
> is on `@key1', C-c C-o could be bound to find the key in the reference
> database, or another useful action, depending on the reference database
> format.

I think we should use "almost-pandoc" cf.

  http://permalink.gmane.org/gmane.emacs.orgmode/94412

Namely [@bibtex-key :key value] with handy shortcuts.
E.g. for latex constructs

      @key                       → \textcite{key}
      (@key)                     → \parencite{key}
      [@key :key value]          → ?
      [@key :type mytype]        → \mytype{key}
      [-@key]                    → \nocite{key}
      (pre @key post)            → \parencite[pre][post]{key}
      (pre @key post :key value) → ? 
      
Something like (pre1 @key1 post1 pre1 @key2 post2) is hard to represent
though ('cause it disallows).  Perhaps (pre1 @key1 post1) (pre2 @key2
post2) could be merged like how subscripts are collected?

> As I mentioned in the earlier thread, I think the Pandoc syntax is a
> good place to start, and I think it would be valuable to have the two
> syntaxes be compatible.  But even Pandoc's citation syntax might not be
> general enough to satisfy everyone's needs, so the first step for
> introducing citation syntax to Org should be compiling a list of all the
> things such a syntax should represent.

I think allowing for arbitrary keys is abstract enough to solve all
issues.  It would also be easy to add user-written support.

> The other problems, I think, must wait until a stable citation syntax
> emerges -- export support in particular.

+1.

—Rasmus

-- 
It was you, Jezebel, it was you

Re: Citations, continued

[Matt Price]

[-- Attachment #1: Type: text/plain, Size: 6903 bytes --]

I have very little of substance to say, but many thanks to Richard for
raising the level of discourse to a much more sophisticated one than I was
able to achieve in my initial post.

I don't feel qualified to comment on whether links or a new citation syntax
is appropriate.  But I do think that Richard and the rest of you have
identified the key features that any future syntax should follow.  I guess
I would emphasize that Richard's insistence on the experience of a document
writer is very important -- it should be straightforward to add citations
from whatever sources we use, to read and work with those citations in org
itself, and to export them painlessly to the other major formats in which
our work is shared (I think the main ones right now and for the foreseeable
future are LaTex, ODT/Docx, and HTML). And that should be true even for
people who are not so good at fiddling with presentation details.

From my perspective, citation support is really the only writerly feature
"missing" from org; if it can be put in place, I will be thrilled and
thankful.

Finally, I did want to describe one other syntax I've recently learned
about, which is used by Zotero's odf-scan plugin (
http://zotero-odf-scan.github.io/zotero-odf-scan/); these are text
citations of this type:

{pre | -Author (Date) | p. XX | post | ZOTEROKEY }

The scanner will convert these to ODT citations, and multiple citations
strung together will be formatted as a multi-source citation.

I find these pretty easy to read, though there is certainly room for
improvement.

On Sat, Jan 31, 2015 at 1:26 PM, Richard Lawrence <
richard.lawrence@berkeley.edu> wrote:

> Hi all,
>
> I wanted to continue the discussion that began in this thread about
> adding citation support to Org:
> http://thread.gmane.org/gmane.emacs.orgmode/94352/focus=94412
>
> Here are some thoughts I have after reviewing that discussion:
>
> 1) Lots of people seem to need/want better support for citations in Org
> documents.  Other projects (Pandoc among them) already have support for
> citations which is good enough that at least some people are using them
> to fill this perceived gap in Org's support.
>
> 2) There are at least several different backend reference database
> formats (BibTeX, Zotero, etc.) used by Orgsters.  Not all such databases
> use human-readable keys.  Org also has a nice internal format for
> storing reference information: org-bibtex.
>
> 3) There are also several different frontend solutions for making
> citations in Org documents (org-ref, various `home-brewed' solutions,
> raw LaTeX \cite commands, etc.).  The variety here is at least in part
> due to our different requirements for the format of exported documents:
> e.g., some people only care about exporting to LaTeX, others need
> something that will work for HTML or ODT; some people need to specify
> pre- and post-text for citations; some people need to specify citation
> types.
>
> 4) Because individual Orgsters have widely varying needs, there is some
> disagreement about what `proper' citation support should look like.  (Do
> we need new syntax, or can existing syntax be used?  Which features need
> to be supported by Org, and which can be provided by external tools?
> etc.)
>
> It seems there are three distinct but related problems:
>   - representing citations in Org documents
>   - exporting citations in an Org document to a backend document format
>     like LaTeX, ODT, or HTML
>   - searching for and manipulating keys in a reference database from
>     within Org, and otherwise fostering good communication between
>     such a database and citations in Org documents
>
> Here's my personal opinion about how we might solve them.
>
> As for the first problem, I think a good case can be made for adding new
> syntax to Org to represent citations, instead of repurposing/extending
> existing syntax (most notably, the link syntax).
>
> Here's why.  Citations are complicated, and it's clear that some sort of
> new syntax is needed to represent them.  Even link-based solutions
> introduce new syntax `inside' the link syntax, such as using `::' to
> separate pre-text and post-text in a link description, as org-ref does.
> Thus, the issue is not *whether* there should be additional syntax, but
> just *how constrained* it should be.  In particular, the question is
> whether we want to make citation syntax a subset of link syntax, or
> whether citations and links should be distinct types of syntactic
> elements.
>
> It seems to me that the needs of citation users are wide enough and
> complicated enough that it is worth shedding the constraints imposed by
> the link syntax.  Eventually, packing all the representations we need
> for a general solution (citation type, pre- and post-text, suppressing
> author name, etc. etc.) into the link syntax will tend to make citations
> unreadable.
>
> Moreover, citations are not really links, even though it is often useful
> to treat them (or parts of them) as links.  For example, a link can only
> point to one target; yet a single citation often points to multiple
> sources.  And what if you need different pre- or post-text for different
> sources within the same citation, like ``See @Doe1999 for an overview; a
> more extensive discussion is in @Foobar2000''?
>
> Finally, it seems to me that Org strives to be a document format which
> is usable independently of any particular export format.  This counts in
> favor of having citation syntax as a first-class citizen in Org; at
> present, Org documents can only contain citations through a combination
> of extensions and external tools.
>
> For these reasons, I would support a separate citation syntax, but one
> that can behave like a link when useful.  For example, suppose we
> borrowed the Pandoc [ ... @key1 ...; ... @key2 ...] syntax.  When point
> is on `@key1', C-c C-o could be bound to find the key in the reference
> database, or another useful action, depending on the reference database
> format.
>
> As I mentioned in the earlier thread, I think the Pandoc syntax is a
> good place to start, and I think it would be valuable to have the two
> syntaxes be compatible.  But even Pandoc's citation syntax might not be
> general enough to satisfy everyone's needs, so the first step for
> introducing citation syntax to Org should be compiling a list of all the
> things such a syntax should represent.
>
> The other problems, I think, must wait until a stable citation syntax
> emerges -- export support in particular.  (Using an existing syntax from
> another project could help ease the transition here: if people can
> export citations using an existing tool, they'll be able to switch to
> that syntax immediately, and use the external tool in the meantime while
> Org-internal support for it catches up.)
>
> I hope this is a useful starting point for further discussion!
>
> Best,
> Richard
>
>
>

[-- Attachment #2: Type: text/html, Size: 7917 bytes --]

Re: Citations, continued

[Richard Lawrence]

[-- Attachment #1: Type: text/plain, Size: 13112 bytes --]

Hi all,

Here is the citation syntax proposal I have mentioned in a couple of
posts now.  I have attached it as an Org document for better
readability, and also reproduced the text below.  Let me know what you
think!

Best,
Richard


#+TITLE: A Proposal for Org citation syntax
#+AUTHOR: Richard Lawrence

* Introduction
In brief, the proposal is:

1. Use the Pandoc syntax for basic, inline citations.
2. Extend the Pandoc syntax modestly to accommodate backend-agnostic
   formatting of inline citations.
3. Also allow non-inline citation definitions, with a syntax
   comparable to non-inline footnotes, to accommodate
   backend-specific formatting.

Basing this proposal on the Pandoc syntax is a `merely practical'
choice.  It might not be the most Org-like, and it might produce too
much conceptual divergence between citations and links.  But it is a
syntax that is already well-tested and known to work elsewhere, and
which has easily-scriptable tools for processing it (namely, Pandoc's
own), which Org users could rely on in the meantime, while Org's own
implementation of this syntax catches up.

Beyond the features provided by the basic Pandoc syntax, I have tried
to ensure that the other features are as Org-like as possible, are
already in use in Org documents, and (I hope) could be implemented
with minimal work.

* Citation syntax
(I repeat the list of requirements I posted earlier here, for easy
reference; so far, I don't think anyone has suggested we need any
others.)

A citation is a textual reference to one or more individual works,
together with other information about those works, grouped together in
a single place.

Within a citation, each reference to an individual work needs to be
capable of containing:
1. a database key that references the cited work
2. prefix / pre-text
3. suffix / post-text
4. references to page/chapter/section/whatever numbers and ranges.
   This is likely part of the prefix or suffix, but might be worth
   parsing separately for localization or link-following behavior.
5. a way of indicating backend-agnostic formatting properties.
   Examples of some properties users might want to specify are:
   - displaying only some fields (or suppressing some fields) from a
     reference record (e.g., journal, date, author)
   - indicating that the referenced works should *only* appear in
     the bibliography of the exported document (equivalent of LaTeX
     \nocite)

Citations as a whole also need:
6. [@6] a way of indicating formatting properties for specific export
   backends.  Examples of some properties that users might want to
   specify are:
   - a citation command to use for each individual reference (LaTeX;
     others?)
   - a multi-cite command to apply to all references together
     (LaTeX)
   - CSS or other styling class (HTML and derived backends; also
     ODT?)
   - properties describing how to treat emphasis and other
     formatting that cannot appear in plain text (ASCII and other
     plain text backends)

** Starting point
I assume, to start, the basic Pandoc [ ... @key1 ...; ... @key2 ...]
syntax for inline citations, documented [[http://pandoc.org/README.html#citations][here]].  This defines a syntax
for inline citations that allows grouping multiple individual
references together between brackets, with semicolons as separators.

Previous discussions have suggested beginning citation definitions
with a tag, like [cite: ...] or [citation: ...], by analogy with
footnotes and links.  As far as I can see, the tag doesn't really
provide any advantages for inline citations, and is just unnecessary
markup.  This is because the syntax of citations is (or should be)
more constrained than footnotes or links; a citation is already
recognizable, and parseable as such, by the required presence of a
reference key.  The tag would also immediately break compatibility
with the basic Pandoc syntax if it were required for inline citation
definitions, a result which I am trying to avoid in this proposal.

A syntax for /non-inline/ citation definitions, however, comparable to
the syntax for footnotes, would make good use of such a tag.  This is
what I propose below.

** Backend-agnostic formatting properties
*** Selecting specific fields
Selecting specific fields to display could be done by appending field
names to cite keys after colons, much like Org tags:
#+BEGIN_QUOTE
[See @Doe99, pp. 34--45; also @Doe00:year, section 6] 

[See their article in @Doe99:journal:year.] 
#+END_QUOTE
Note that this would make for an extension of Pandoc syntax.  This
extension is not a strict superset, since Pandoc allows internal `:'
characters in cite keys, and thus would treat `@Doe99:journal:year' as
a single key, rather than treating the key as ending at the first
colon, with other data afterward.  (More compatible but uglier
alternatives for the field selector include `!', `{', `}', and `^'.
If an alternative is desired, I suggest `@Doe99{journal,year}'.)

When specific fields are requested, ONLY data from those fields should
appear in the exported document.  Backends would choose how to export
these citations based on the selected fields.

I would think the default behavior during export should be something
like: get the reference record from the database, then pass it and a
list of the requested fields to a user-customizable function which is
expected to return a string to insert in the output.  (The default
function could, say, intersperse the requested field data with
whitespace and add parentheses.  More sophisticated functions could
rely on external tools to format the citation using the Citation Style
Language.)  Of course, this assumes that the exporter has a way of
querying the reference database, which would be fine for bibtex and
org-bibtex databases, but may not be a good assumption in general.

Specific backends could also do something different with field
selectors when it makes sense to do so.  For example, the LaTeX
backend could choose \citeyear as the command to place in the exported
document when just `:year' is requested in the citation.

*** Non-cited works that should appear in the bibliography
A special field selector `:nocite' would be one way to achieve
citations that, for whatever reason, should appear in the Org source
and in the exported bibliography, but should not appear in the
exported text where they are placed.  This would allow referencing
them at relevant places in the document, like:
#+BEGIN_QUOTE
Smith said a lot of things, but no one can remember what they
were. [@Smith79:nocite]
#+END_QUOTE

One drawback of this syntax is that it does not provide an easy way to
list all the nocite references, since the user would have to add
`:nocite' to each one individually.  This is not a huge problem for
small numbers of refernces, but it would also be nice to have some
equivalent of LaTeX's \nocite{*}.  On this point, see the proposal for
non-inline citation definitions below.

** Non-inline citation definitions and backend-specific formatting
The syntax proposed above assumes citations are defined inline.  A
complementary alternative would be to treat citations like
(non-inline) footnotes, with an inline reference and a definition
elsewhere in the document.  This could be convenient for citations
that have lots of pre- or post-text.

In that case, a citation could look like:
#+BEGIN_QUOTE
    Doe provides an interesting analysis. [cite:1]

    ...

    * Citations

    [cite:1] See @Doe99, pp. 34--45; also @Doe2000:year, ch. 1.
#+END_QUOTE
That is, a citation /pointer/ would occur inline in the document text,
which refers (via a number or a label) to a citation /definition/ in a
specially-named subtree.  The definition begins by repeating the
pointer, and has the same syntax as proposed above, minus the
enclosing square brackets.

This approach could peacefully coexist with the above proposal for
inline citations, in the same way that inline and non-inline footnote
definitions now peacefully coexist.  

*** Backend-specific formatting
In general, it would be nice to avoid formatting properties which are
specific to a particular export backend when a backend-agnostic
solution is available, but some backend-specific formatting needs are
probably inevitable, so we need a syntax for specifying them.

Another advantage of the non-inline citation syntax is that it would
allow using the existing #+ATTR_BACKEND syntax to specify
backend-specific formatting properties, since the citation definitions
would be block-level elements:
#+BEGIN_QUOTE
    * Citations

    #+ATTR_LATEX: :command citet
    #+ATTR_HTML: :class my-citation
    [cite:1] See @Doe99, pp. 34--45; @Foobar2000, ch.1.
#+END_QUOTE
This automatically makes the syntax readily extensible as new needs
come up and target formats evolve.

(Originally, I had thought about how to extend the inline citation
definition syntax above to include backend-specific formatting
information.  But everything I came up with seemed pretty ugly, and
not worth the extra syntax it would require.  When I realized that
non-inline definitions could leverage the existing syntax for
backend-specific properties, I tossed that part of the proposal,
though I'm happy to share it if anyone wants to see it.)

Thus, I propose that, for authors who /need/ backend-specific
formatting, this should be the way to do it.  The above inline
citation syntax should remain limited to uses where no
backend-specific behavior is required.

Note however that there is a tension here with the proposal above for
backend-agnostic field selectors.  I am not sure what should happen
if, say, the user selects individual fields in the citation but also
requests an incompatible citation command for a particular backend.

*** Bibliography-only entries
Non-inline definitions would also provide a convenient place to list
non-cited references that should appear in the bibliography.  For
example:
#+BEGIN_QUOTE
    * Citations
    ...
    [nocite:] @Doe99; @Foobar2000; @Baz98.
#+END_QUOTE
As a special case,
#+BEGIN_QUOTE
    * Citations

    [nocite:*] 
#+END_QUOTE
could introduce bibliography entries for everything in the reference
database.

* Document metadata
In addition to the syntax of citations themselves, the Org document
would also need to represent the following metadata to support
citations:
7. [@7] a pointer to one or more backend reference databases,
   including in-document databases in org-bibtex format
8. a reference to a citation style or style file
9. a reference to a locale file
10. an indication of where the bibliography should be found in the
    exported document (equivalent to \printbibliography, etc. in
    LaTeX)

** #+BIBLIOGRAPHY: reference database, style, locale
The #+BIBLIOGRAPHY keyword already exists, in ox-bibtex.el (in
contrib), though its current syntax does not quite meet all the above
needs.  I suggest changing the syntax to support in-file databases and
a locale file.

The point of specifying the style and locale as part of
the #+BIBLIOGRAPHY definition is for compatibility with both LaTeX and
Citation Style Language bibliography and citation formatting.

In keeping with other metadata keyword lines (like #+OPTIONS), I
suggest a key:value syntax for the arguments to #+BIBLIOGRAPHY, like
so:
#+BEGIN_QUOTE
#+BIBLIOGRAPHY: db:/path/to/some/file.bib style:chicago

#+BIBLIOGRAPHY: db:/path/to/some/file.bib style:plain locale:en_GB

#+BIBLIOGRAPHY: db:"*Reference DB"
#+END_QUOTE
In the last example, the leading "*" is meant to indicate that the
reference database is a subtree with headline "Reference DB", whose
branches are in org-bibtex format.

By specifying where the reference data is (and implicitly what format
it is in, e.g., via the file extension), link-following and export
behavior for citations can differ depending on the format of this
database.  For example, if the database is a .bib file, `following' a
citation key could mean finding the corresponding entry in this file.
If the database is an in-document tree in org-bibtex format, following
a key could mean jumping to the headline whose :CUSTOM_ID: property
agrees with that key.

Likewise, if the database is in a format that the exporter knows how
to read, then export backends could potentially look up information
from it to create bibliography entries and citations in the exported
document, possibly relying on an external tool (like citeproc-*) to
transform them into the requested style.  This would be particularly
useful for non-LaTeX backends (which is what ox-bibtex.el focuses on
at the moment).

** Bibliography placement
The other issue is that Org documents must say where the bibliography
should appear in exported documents.

A reasonable default would be placing the bibliography at the end of
the document.  But some documents, in particular long ones, may need
more flexibility in specifying where to place the bibliography.

The simplest solution seems to be just allowing the #+BIBLIOGRAPHY
keyword to appear anywhere in the document, to be replaced on export
with the formatted bibliography.  I think this is what ox-bibtex now
does.  


[-- Attachment #2: proposal.org --]
[-- Type: text/plain, Size: 12909 bytes --]

# Created 2015-02-02 Mon 09:49
#+TITLE: A Proposal for Org citation syntax
#+AUTHOR: Richard Lawrence

* Introduction
In brief, the proposal is:

1. Use the Pandoc syntax for basic, inline citations.
2. Extend the Pandoc syntax modestly to accommodate backend-agnostic
   formatting of inline citations.
3. Also allow non-inline citation definitions, with a syntax
   comparable to non-inline footnotes, to accommodate
   backend-specific formatting.

Basing this proposal on the Pandoc syntax is a `merely practical'
choice.  It might not be the most Org-like, and it might produce too
much conceptual divergence between citations and links.  But it is a
syntax that is already well-tested and known to work elsewhere, and
which has easily-scriptable tools for processing it (namely, Pandoc's
own), which Org users could rely on in the meantime, while Org's own
implementation of this syntax catches up.

Beyond the features provided by the basic Pandoc syntax, I have tried
to ensure that the other features are as Org-like as possible, are
already in use in Org documents, and (I hope) could be implemented
with minimal work.

* Citation syntax
(I repeat the list of requirements I posted earlier here, for easy
reference; so far, I don't think anyone has suggested we need any
others.)

A citation is a textual reference to one or more individual works,
together with other information about those works, grouped together in
a single place.

Within a citation, each reference to an individual work needs to be
capable of containing:
1. a database key that references the cited work
2. prefix / pre-text
3. suffix / post-text
4. references to page/chapter/section/whatever numbers and ranges.
   This is likely part of the prefix or suffix, but might be worth
   parsing separately for localization or link-following behavior.
5. a way of indicating backend-agnostic formatting properties.
   Examples of some properties users might want to specify are:
   - displaying only some fields (or suppressing some fields) from a
     reference record (e.g., journal, date, author)
   - indicating that the referenced works should *only* appear in
     the bibliography of the exported document (equivalent of LaTeX
     \nocite)

Citations as a whole also need:
6. [@6] a way of indicating formatting properties for specific export
   backends.  Examples of some properties that users might want to
   specify are:
   - a citation command to use for each individual reference (LaTeX;
     others?)
   - a multi-cite command to apply to all references together
     (LaTeX)
   - CSS or other styling class (HTML and derived backends; also
     ODT?)
   - properties describing how to treat emphasis and other
     formatting that cannot appear in plain text (ASCII and other
     plain text backends)

** Starting point
I assume, to start, the basic Pandoc [ ... @key1 ...; ... @key2 ...]
syntax for inline citations, documented [[http://pandoc.org/README.html#citations][here]].  This defines a syntax
for inline citations that allows grouping multiple individual
references together between brackets, with semicolons as separators.

Previous discussions have suggested beginning citation definitions
with a tag, like [cite: ...] or [citation: ...], by analogy with
footnotes and links.  As far as I can see, the tag doesn't really
provide any advantages for inline citations, and is just unnecessary
markup.  This is because the syntax of citations is (or should be)
more constrained than footnotes or links; a citation is already
recognizable, and parseable as such, by the required presence of a
reference key.  The tag would also immediately break compatibility
with the basic Pandoc syntax if it were required for inline citation
definitions, a result which I am trying to avoid in this proposal.

A syntax for /non-inline/ citation definitions, however, comparable to
the syntax for footnotes, would make good use of such a tag.  This is
what I propose below.

** Backend-agnostic formatting properties
*** Selecting specific fields
Selecting specific fields to display could be done by appending field
names to cite keys after colons, much like Org tags:
#+BEGIN_QUOTE
[See @Doe99, pp. 34--45; also @Doe00:year, section 6] 

[See their article in @Doe99:journal:year.] 
#+END_QUOTE
Note that this would make for an extension of Pandoc syntax.  This
extension is not a strict superset, since Pandoc allows internal `:'
characters in cite keys, and thus would treat `@Doe99:journal:year' as
a single key, rather than treating the key as ending at the first
colon, with other data afterward.  (More compatible but uglier
alternatives for the field selector include `!', `{', `}', and `^'.
If an alternative is desired, I suggest `@Doe99{journal,year}'.)

When specific fields are requested, ONLY data from those fields should
appear in the exported document.  Backends would choose how to export
these citations based on the selected fields.

I would think the default behavior during export should be something
like: get the reference record from the database, then pass it and a
list of the requested fields to a user-customizable function which is
expected to return a string to insert in the output.  (The default
function could, say, intersperse the requested field data with
whitespace and add parentheses.  More sophisticated functions could
rely on external tools to format the citation using the Citation Style
Language.)  Of course, this assumes that the exporter has a way of
querying the reference database, which would be fine for bibtex and
org-bibtex databases, but may not be a good assumption in general.

Specific backends could also do something different with field
selectors when it makes sense to do so.  For example, the LaTeX
backend could choose \citeyear as the command to place in the exported
document when just `:year' is requested in the citation.

*** Non-cited works that should appear in the bibliography
A special field selector `:nocite' would be one way to achieve
citations that, for whatever reason, should appear in the Org source
and in the exported bibliography, but should not appear in the
exported text where they are placed.  This would allow referencing
them at relevant places in the document, like:
#+BEGIN_QUOTE
Smith said a lot of things, but no one can remember what they
were. [@Smith79:nocite]
#+END_QUOTE

One drawback of this syntax is that it does not provide an easy way to
list all the nocite references, since the user would have to add
`:nocite' to each one individually.  This is not a huge problem for
small numbers of refernces, but it would also be nice to have some
equivalent of LaTeX's \nocite{*}.  On this point, see the proposal for
non-inline citation definitions below.

** Non-inline citation definitions and backend-specific formatting
The syntax proposed above assumes citations are defined inline.  A
complementary alternative would be to treat citations like
(non-inline) footnotes, with an inline reference and a definition
elsewhere in the document.  This could be convenient for citations
that have lots of pre- or post-text.

In that case, a citation could look like:
#+BEGIN_QUOTE
    Doe provides an interesting analysis. [cite:1]

    ...

    * Citations

    [cite:1] See @Doe99, pp. 34--45; also @Doe2000:year, ch. 1.
#+END_QUOTE
That is, a citation /pointer/ would occur inline in the document text,
which refers (via a number or a label) to a citation /definition/ in a
specially-named subtree.  The definition begins by repeating the
pointer, and has the same syntax as proposed above, minus the
enclosing square brackets.

This approach could peacefully coexist with the above proposal for
inline citations, in the same way that inline and non-inline footnote
definitions now peacefully coexist.  

*** Backend-specific formatting
In general, it would be nice to avoid formatting properties which are
specific to a particular export backend when a backend-agnostic
solution is available, but some backend-specific formatting needs are
probably inevitable, so we need a syntax for specifying them.

Another advantage of the non-inline citation syntax is that it would
allow using the existing #+ATTR_BACKEND syntax to specify
backend-specific formatting properties, since the citation definitions
would be block-level elements:
#+BEGIN_QUOTE
    * Citations

    #+ATTR_LATEX: :command citet
    #+ATTR_HTML: :class my-citation
    [cite:1] See @Doe99, pp. 34--45; @Foobar2000, ch.1.
#+END_QUOTE
This automatically makes the syntax readily extensible as new needs
come up and target formats evolve.

(Originally, I had thought about how to extend the inline citation
definition syntax above to include backend-specific formatting
information.  But everything I came up with seemed pretty ugly, and
not worth the extra syntax it would require.  When I realized that
non-inline definitions could leverage the existing syntax for
backend-specific properties, I tossed that part of the proposal,
though I'm happy to share it if anyone wants to see it.)

Thus, I propose that, for authors who /need/ backend-specific
formatting, this should be the way to do it.  The above inline
citation syntax should remain limited to uses where no
backend-specific behavior is required.

Note however that there is a tension here with the proposal above for
backend-agnostic field selectors.  I am not sure what should happen
if, say, the user selects individual fields in the citation but also
requests an incompatible citation command for a particular backend.

*** Bibliography-only entries
Non-inline definitions would also provide a convenient place to list
non-cited references that should appear in the bibliography.  For
example:
#+BEGIN_QUOTE
    * Citations
    ...
    [nocite:] @Doe99; @Foobar2000; @Baz98.
#+END_QUOTE
As a special case,
#+BEGIN_QUOTE
    * Citations

    [nocite:*] 
#+END_QUOTE
could introduce bibliography entries for everything in the reference
database.

* Document metadata
In addition to the syntax of citations themselves, the Org document
would also need to represent the following metadata to support
citations:
7. [@7] a pointer to one or more backend reference databases,
   including in-document databases in org-bibtex format
8. a reference to a citation style or style file
9. a reference to a locale file
10. an indication of where the bibliography should be found in the
    exported document (equivalent to \printbibliography, etc. in
    LaTeX)

** #+BIBLIOGRAPHY: reference database, style, locale
The #+BIBLIOGRAPHY keyword already exists, in ox-bibtex.el (in
contrib), though its current syntax does not quite meet all the above
needs.  I suggest changing the syntax to support in-file databases and
a locale file.

The point of specifying the style and locale as part of
the #+BIBLIOGRAPHY definition is for compatibility with both LaTeX and
Citation Style Language bibliography and citation formatting.

In keeping with other metadata keyword lines (like #+OPTIONS), I
suggest a key:value syntax for the arguments to #+BIBLIOGRAPHY, like
so:
#+BEGIN_QUOTE
#+BIBLIOGRAPHY: db:/path/to/some/file.bib style:chicago

#+BIBLIOGRAPHY: db:/path/to/some/file.bib style:plain locale:en_GB

#+BIBLIOGRAPHY: db:"*Reference DB"
#+END_QUOTE
In the last example, the leading "*" is meant to indicate that the
reference database is a subtree with headline "Reference DB", whose
branches are in org-bibtex format.

By specifying where the reference data is (and implicitly what format
it is in, e.g., via the file extension), link-following and export
behavior for citations can differ depending on the format of this
database.  For example, if the database is a .bib file, `following' a
citation key could mean finding the corresponding entry in this file.
If the database is an in-document tree in org-bibtex format, following
a key could mean jumping to the headline whose :CUSTOM_ID: property
agrees with that key.

Likewise, if the database is in a format that the exporter knows how
to read, then export backends could potentially look up information
from it to create bibliography entries and citations in the exported
document, possibly relying on an external tool (like citeproc-*) to
transform them into the requested style.  This would be particularly
useful for non-LaTeX backends (which is what ox-bibtex.el focuses on
at the moment).

** Bibliography placement
The other issue is that Org documents must say where the bibliography
should appear in exported documents.

A reasonable default would be placing the bibliography at the end of
the document.  But some documents, in particular long ones, may need
more flexibility in specifying where to place the bibliography.

The simplest solution seems to be just allowing the #+BIBLIOGRAPHY
keyword to appear anywhere in the document, to be replaced on export
with the formatted bibliography.  I think this is what ox-bibtex now
does.  

Re: Citations, continued

[Rasmus]

Hi,

Richard Lawrence <richard.lawrence@berkeley.edu> writes:

> Here is the citation syntax proposal I have mentioned in a couple of
> posts now.

Thanks.  I think it's a good start, but I find it too far away from Org in
some respects.  Hence comments follow.

> ** Backend-agnostic formatting properties
> *** Selecting specific fields
> Selecting specific fields to display could be done by appending field
> names to cite keys after colons, much like Org tags:
> #+BEGIN_QUOTE
> [See @Doe99, pp. 34--45; also @Doe00:year, section 6] 
>
> [See their article in @Doe99:journal:year.] 
> #+END_QUOTE

First, I think we should use @key for inline and (@key) for parenthesis
expressions.  This give us two short keys.  [@Key ⋯] can be reserved for
complicated references.

I don't like "@Doe99:journal:year".  It's too unlike existing syntax.

Rather, I'd just introduce types as hinted previously, [@Doe99 :type
my-journal-year-type].  Org can provide as many predefined :type as we
care for, and let the user define the rest as necessary.

> Note that this would make for an extension of Pandoc syntax.  This
> extension is not a strict superset, since Pandoc allows internal `:'
> characters in cite keys, and thus would treat `@Doe99:journal:year' as
> a single key, rather than treating the key as ending at the first
> colon, with other data afterward.

Compatibility with pandoc is a non-issue IMO.  If that's necessary we can
use ox.

> When specific fields are requested, ONLY data from those fields should
> appear in the exported document.  Backends would choose how to export
> these citations based on the selected fields.

What happens when a field is undefined?

> *** Non-cited works that should appear in the bibliography
> A special field selector `:nocite' would be one way to achieve
> citations that, for whatever reason, should appear in the Org source
> and in the exported bibliography, but should not appear in the
> exported text where they are placed.  This would allow referencing
> them at relevant places in the document, like:
> #+BEGIN_QUOTE
> Smith said a lot of things, but no one can remember what they
> were. [@Smith79:nocite]
> #+END_QUOTE

I think R-markdown uses something like [-@Smith79].  Again, I don't like
the [@key:nocite].


> One drawback of this syntax is that it does not provide an easy way to
> list all the nocite references, since the user would have to add
> `:nocite' to each one individually.  This is not a huge problem for
> small numbers of refernces, but it would also be nice to have some
> equivalent of LaTeX's \nocite{*}.  On this point, see the proposal for
> non-inline citation definitions below.

#+NOCITE: key1 ⋯ keyN

> ** Non-inline citation definitions and backend-specific formatting
> [...]
> #+BEGIN_QUOTE
>     Doe provides an interesting analysis. [cite:1]
>     ...
>     * Citations
>     [cite:1] See @Doe99, pp. 34--45; also @Doe2000:year, ch. 1.
> #+END_QUOTE

This is cool.  I'd use it in some cases.

> That is, a citation /pointer/ would occur inline in the document text,
> which refers (via a number or a label) to a citation /definition/ in a
> specially-named subtree.

It would refer to it anywhere.  No reason to make it different from
footnotes.

> *** Backend-specific formatting
> In general, it would be nice to avoid formatting properties which are
> specific to a particular export backend when a backend-agnostic
> solution is available, but some backend-specific formatting needs are
> probably inevitable, so we need a syntax for specifying them.
>
> Another advantage of the non-inline citation syntax is that it would
> allow using the existing #+ATTR_BACKEND syntax to specify
> backend-specific formatting properties, since the citation definitions
> would be block-level elements:
> #+BEGIN_QUOTE
>     * Citations
>
>     #+ATTR_LATEX: :command citet
>     #+ATTR_HTML: :class my-citation
>     [cite:1] See @Doe99, pp. 34--45; @Foobar2000, ch.1.
> #+END_QUOTE

Why not.  Since footnote-definition is a greater element it /does/ take
affiliated keywords, but I have never seen this used.

For inline maybe something like this:
    [@Key :type_html my-citation :type_latex citet] 


> *** Bibliography-only entries
> Non-inline definitions would also provide a convenient place to list
> non-cited references that should appear in the bibliography.  For
> example:
> #+BEGIN_QUOTE
>     * Citations
>     ...
>     [nocite:] @Doe99; @Foobar2000; @Baz98.
> #+END_QUOTE

As stated above, I find #+NOCITE more in line with current syntax.

> * Document metadata
> In addition to the syntax of citations themselves, the Org document
> would also need to represent the following metadata to support
> citations:
> 7. [@7] a pointer to one or more backend reference databases,
>    including in-document databases in org-bibtex format

If org-bibtex entries are present they should just take precedence.  That
way you can easily insert new entries and customize them for your project
and have a portable document.

> 8. a reference to a citation style or style file

> 9. a reference to a locale file

#+BIBLIOGRAPHY_FILES: file1 file2

> 10. an indication of where the bibliography should be found in the
>     exported document (equivalent to \printbibliography, etc. in
>     LaTeX)

From experience, the biblatex model of separating the loading of files,
styles and printing into different commands is a great advantage.

> ** #+BIBLIOGRAPHY: reference database, style, locale

I think more keywords are better.

> The point of specifying the style and locale as part of
> the #+BIBLIOGRAPHY definition is for compatibility with both LaTeX and
> Citation Style Language bibliography and citation formatting.

Local is defined by #+LANGUAGE.  AFAIK, Org doesn't support many
languages.  E.g. here's the definition of LANGUAGE in ox.el:

(:language "LANGUAGE" nil org-export-default-language t)

> In keeping with other metadata keyword lines (like #+OPTIONS), I
> suggest a key:value syntax for the arguments to #+BIBLIOGRAPHY, like
> so:
> #+BEGIN_QUOTE
> #+BIBLIOGRAPHY: db:/path/to/some/file.bib style:chicago

The db is redundant.  E.g. #+INCLUDE automatically knows that the first
word is the file in question.

There may be many bib files, so there should be a #+BIBLIOGRAPHY_FILES.

I prefer :style chicago to be consistent with #+INCLUDE.

> #+BIBLIOGRAPHY: db:/path/to/some/file.bib style:plain locale:en_GB

Locale is "dangerous" as it can get out of sync with LANGUAGE.

> #+BIBLIOGRAPHY: db:"*Reference DB"

If the citation-key is defined in document use it.  Otherwise check
BIBLIOGRAPHY_FILES.

> In the last example, the leading "*" is meant to indicate that the
> reference database is a subtree with headline "Reference DB", whose
> branches are in org-bibtex format.

Too much micromanagement.  I prefer how footnote-definition works. 


> ** Bibliography placement
> The simplest solution seems to be just allowing the #+BIBLIOGRAPHY
> keyword to appear anywhere in the document, to be replaced on export
> with the formatted bibliography.  I think this is what ox-bibtex now
> does.  

#+TOC works the same way.

—Rasmus

-- 
C is for Cookie

Re: Citations, continued

[Richard Lawrence]

Hi Rasmus and all,

Thanks for your comments!

Rasmus <rasmus@gmx.us> writes:

>> ** Backend-agnostic formatting properties
>> *** Selecting specific fields
>> Selecting specific fields to display could be done by appending field
>> names to cite keys after colons, much like Org tags:
>> #+BEGIN_QUOTE
>> [See @Doe99, pp. 34--45; also @Doe00:year, section 6] 
>>
>> [See their article in @Doe99:journal:year.] 
>> #+END_QUOTE
>
> First, I think we should use @key for inline and (@key) for parenthesis
> expressions.  This give us two short keys.  [@Key ⋯] can be reserved for
> complicated references.

That sounds fine to me.  I think you may be using `inline' differently
than me, though: do you mean `author's name appears in the text (not in
parentheses)'?  (I was using it to talk about where the citation
definition appears in the document, not where the author's name appears
relative to parentheses.)

> I don't like "@Doe99:journal:year".  It's too unlike existing syntax.

I agree it's a little clunky, but I think most of the time there would
just be one selector.  I was thinking of this on analogy with heading
properties and tags...is there a better existing syntax to refer to a
property value?

> Rather, I'd just introduce types as hinted previously, [@Doe99 :type
> my-journal-year-type].  Org can provide as many predefined :type as we
> care for, and let the user define the rest as necessary.

I don't like this, because it seems like a lot more work for me as a
user to achieve something that should be simple, and it trades 
specifying /what/ data I want for describing it more indirectly.

Suppose I'm writing a document, and I know I just want to reference the
journal and year of a particular publication, in that order.  Being a
studious keeper of my org-bibtex database, I already know that these
fields are called "journal" and "year".  But if, in addition to
reference database field names, I have to remember names for /types/ of
/combinations/ of field names, that's too much.  I'll end up taking
endless trips to the manual to figure out which type I need in this
case.  Do I need :type journal-before-year? :type journal-and-year?
etc.  This feels a bit too much like having to remember (or look up) all
the different LaTeX citation commands.

I expect that many of the common cases would have the same name, but
then I still have to remember which of my uses are `common'.  And
[@Doe99:year] is less verbose than [@Doe99 :type year].

What about just separating the field names off, as keys?  Like:

[See Doe's review @Doe99 :journal :year]

>> When specific fields are requested, ONLY data from those fields should
>> appear in the exported document.  Backends would choose how to export
>> these citations based on the selected fields.
>
> What happens when a field is undefined?

I guess I would suggest the same thing as happens in LaTeX: you get a
nice, bold "??" in the output where the missing data should be. 

>> *** Non-cited works that should appear in the bibliography
>> A special field selector `:nocite' would be one way to achieve
>> citations that, for whatever reason, should appear in the Org source
>> and in the exported bibliography, but should not appear in the
>> exported text where they are placed.  This would allow referencing
>> them at relevant places in the document, like:
>> #+BEGIN_QUOTE
>> Smith said a lot of things, but no one can remember what they
>> were. [@Smith79:nocite]
>> #+END_QUOTE
>
> I think R-markdown uses something like [-@Smith79].  Again, I don't like
> the [@key:nocite].

Doesn't [-@Smith79] mean something different, namely, "cite @Smith79
without the author name"?  It produces output like: "(1979)".

The idea is that the :nocite selector produces no output, like LaTeX's
\nocite.  I don't know how important it is to have this, though.

>> *** Backend-specific formatting
>> In general, it would be nice to avoid formatting properties which are
>> specific to a particular export backend when a backend-agnostic
>> solution is available, but some backend-specific formatting needs are
>> probably inevitable, so we need a syntax for specifying them.
>>
>> Another advantage of the non-inline citation syntax is that it would
>> allow using the existing #+ATTR_BACKEND syntax to specify
>> backend-specific formatting properties, since the citation definitions
>> would be block-level elements:
>> #+BEGIN_QUOTE
>>     * Citations
>>
>>     #+ATTR_LATEX: :command citet
>>     #+ATTR_HTML: :class my-citation
>>     [cite:1] See @Doe99, pp. 34--45; @Foobar2000, ch.1.
>> #+END_QUOTE
>
> Why not.  Since footnote-definition is a greater element it /does/ take
> affiliated keywords, but I have never seen this used.

Right, that's the point here...(were you disagreeing?)

> For inline maybe something like this:
>     [@Key :type_html my-citation :type_latex citet] 

Actually, this is a lot like the syntax I was thinking about for the
inline case, but in the end I thought it was too complicated and new to
be worth it, when the #+ATTR_BACKEND syntax will already work for the
out-of-line case.  I'm not opposed to something like this in principle,
but I really think we should try to keep the inline case very simple and
obvious to use, even if that means restricting its expressiveness a bit.

>> *** Bibliography-only entries
>> Non-inline definitions would also provide a convenient place to list
>> non-cited references that should appear in the bibliography.  For
>> example:
>> #+BEGIN_QUOTE
>>     * Citations
>>     ...
>>     [nocite:] @Doe99; @Foobar2000; @Baz98.
>> #+END_QUOTE
>
> As stated above, I find #+NOCITE more in line with current syntax.

That's fine with me.

>> * Document metadata
>> In addition to the syntax of citations themselves, the Org document
>> would also need to represent the following metadata to support
>> citations:
>> 7. [@7] a pointer to one or more backend reference databases,
>>    including in-document databases in org-bibtex format
>
> If org-bibtex entries are present they should just take precedence.  That
> way you can easily insert new entries and customize them for your project
> and have a portable document.

I can't think of any reason why that wouldn't work.

>> 8. a reference to a citation style or style file
>
>> 9. a reference to a locale file
>
> #+BIBLIOGRAPHY_FILES: file1 file2
>
>> 10. an indication of where the bibliography should be found in the
>>     exported document (equivalent to \printbibliography, etc. in
>>     LaTeX)
>
> From experience, the biblatex model of separating the loading of files,
> styles and printing into different commands is a great advantage.

OK.  I'd even be happy with

#+BIBLIOGRAPHY:
#+BIBLIGRAPHY_STYLE:
#+PRINT_BIBLIOGRAPHY:

where the first could be specified as many times as desired, to indicate
external reference databases.

>> The point of specifying the style and locale as part of
>> the #+BIBLIOGRAPHY definition is for compatibility with both LaTeX and
>> Citation Style Language bibliography and citation formatting.
>
> Local is defined by #+LANGUAGE.  AFAIK, Org doesn't support many
> languages.  E.g. here's the definition of LANGUAGE in ox.el:
>
> (:language "LANGUAGE" nil org-export-default-language t)

Ah, OK, I didn't know about #+LANGUAGE.  Is there any reason why the
locale of the bibliography might be different than the locale of the
document?  (I'm monolingual, I'm afraid, so I doubt I could think of one
if there were...)

Best,
Richard

Re: Citations, continued

[Vikas Rawal]

Org-ref is very functional and has so far been able to deal with much of my needs. So, I just hope we are not trying to fix something that is not broken.

The real need in the context of citations is to somehow extend the bibtex/biblatex integration to other export formats (odt/html, most importantly). Will all the new stuff that is being proposed take us in that direction? 

Vikas



> On 03-Feb-2015, at 7:26 am, Richard Lawrence <richard.lawrence@berkeley.edu> wrote:
> 
> Hi Rasmus and all,
> 
> Thanks for your comments!
> 
> Rasmus <rasmus@gmx.us> writes:
> 
>>> ** Backend-agnostic formatting properties
>>> *** Selecting specific fields
>>> Selecting specific fields to display could be done by appending field
>>> names to cite keys after colons, much like Org tags:
>>> #+BEGIN_QUOTE
>>> [See @Doe99, pp. 34--45; also @Doe00:year, section 6] 
>>> 
>>> [See their article in @Doe99:journal:year.] 
>>> #+END_QUOTE

Re: Citations, continued

[Rasmus]

Vikas Rawal <vikaslists@agrarianresearch.org> writes:

> Org-ref is very functional and has so far been able to deal with much
> of my needs. So, I just hope we are not trying to fix something that
> is not broken.

IMO for the same reason it is attractive to limit *bold* to imply bold.
[At one point it was possible to alter the (export) interpretation of
emphases].

Home-brewed solution are not stable compared to syntax.  Getting ad-hoc
citations in Org is very and there might be as many implementations as
there are users.

E.g. org-ruby would not (I hope) support [MY-CITE:MY-DOI]...

> The real need in the context of citations is to somehow extend the
> bibtex/biblatex integration to other export formats (odt/html, most
> importantly). Will all the new stuff that is being proposed take us in
> that direction?

That is /a/ problem but not /the/ problem.  E.g. some people prefer
Zotero.  Some might prefer DOI.

If you can live with \cite-only there's already ox-bibtex.el that will
support bibtex citations in html.

—Rasmus

-- 
May contains speling mistake

Re: Citations, continued

[Julian M. Burgos]

Thanks everyone for thinking about citations.  I wish I knew enough lisp
to make a contribution to this work... for now I can only sit on the
side and clap.

I am also a big fan of org-ref.  Although my needs are not complex
(basically citing from a BibLatex file and exporting to LaTex), I found
that having "actionable" citations is tremendously useful.  In
particular, clicking on the citation allows to open the pdf file with
the reference, the .bib file, or an org mode file with notes.  I hope
this feature is kept in whatever new org-mode reference system you guys
develop.

Keep up the good work!
All the best,

Julian

Vikas Rawal writes:

> Org-ref is very functional and has so far been able to deal with much of my needs. So, I just hope we are not trying to fix something that is not broken.
>
> The real need in the context of citations is to somehow extend the bibtex/biblatex integration to other export formats (odt/html, most importantly). Will all the new stuff that is being proposed take us in that direction? 
>
> Vikas
>
>
>
>> On 03-Feb-2015, at 7:26 am, Richard Lawrence <richard.lawrence@berkeley.edu> wrote:
>> 
>> Hi Rasmus and all,
>> 
>> Thanks for your comments!
>> 
>> Rasmus <rasmus@gmx.us> writes:
>> 
>>>> ** Backend-agnostic formatting properties
>>>> *** Selecting specific fields
>>>> Selecting specific fields to display could be done by appending field
>>>> names to cite keys after colons, much like Org tags:
>>>> #+BEGIN_QUOTE
>>>> [See @Doe99, pp. 34--45; also @Doe00:year, section 6] 
>>>> 
>>>> [See their article in @Doe99:journal:year.] 
>>>> #+END_QUOTE

-- 
Julian Mariano Burgos, PhD
Hafrannsóknastofnun/Marine Research Institute
Skúlagata 4, 121 Reykjavík, Iceland
Sími/Telephone : +354-5752037
Bréfsími/Telefax:  +354-5752001
Netfang/Email: julian@hafro.is

Re: Citations, continued

[John Kitchin]

None of those features need to change, don't worry!

Julian M. Burgos writes:

> Thanks everyone for thinking about citations.  I wish I knew enough lisp
> to make a contribution to this work... for now I can only sit on the
> side and clap.
>
> I am also a big fan of org-ref.  Although my needs are not complex
> (basically citing from a BibLatex file and exporting to LaTex), I found
> that having "actionable" citations is tremendously useful.  In
> particular, clicking on the citation allows to open the pdf file with
> the reference, the .bib file, or an org mode file with notes.  I hope
> this feature is kept in whatever new org-mode reference system you guys
> develop.
>
> Keep up the good work!
> All the best,
>
> Julian
>
> Vikas Rawal writes:
>
>> Org-ref is very functional and has so far been able to deal with much of my needs. So, I just hope we are not trying to fix something that is not broken.
>>
>> The real need in the context of citations is to somehow extend the bibtex/biblatex integration to other export formats (odt/html, most importantly). Will all the new stuff that is being proposed take us in that direction?
>>
>> Vikas
>>
>>
>>
>>> On 03-Feb-2015, at 7:26 am, Richard Lawrence <richard.lawrence@berkeley.edu> wrote:
>>>
>>> Hi Rasmus and all,
>>>
>>> Thanks for your comments!
>>>
>>> Rasmus <rasmus@gmx.us> writes:
>>>
>>>>> ** Backend-agnostic formatting properties
>>>>> *** Selecting specific fields
>>>>> Selecting specific fields to display could be done by appending field
>>>>> names to cite keys after colons, much like Org tags:
>>>>> #+BEGIN_QUOTE
>>>>> [See @Doe99, pp. 34--45; also @Doe00:year, section 6]
>>>>>
>>>>> [See their article in @Doe99:journal:year.]
>>>>> #+END_QUOTE

--
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu

Re: Citations, continued

[Rasmus]

Richard Lawrence <richard.lawrence@berkeley.edu> writes:

> Hi Rasmus and all,
>
> Thanks for your comments!
>
> Rasmus <rasmus@gmx.us> writes:
>>> #+BEGIN_QUOTE
>>> [See @Doe99, pp. 34--45; also @Doe00:year, section 6] 
>>>
>>> [See their article in @Doe99:journal:year.] 
>>> #+END_QUOTE
>>
>> First, I think we should use @key for inline and (@key) for parenthesis
>> expressions.  This give us two short keys.  [@Key ⋯] can be reserved for
>> complicated references.
>
> That sounds fine to me.  I think you may be using `inline' differently
> than me, though: do you mean `author's name appears in the text (not in
> parentheses)'?  (I was using it to talk about where the citation
> definition appears in the document, not where the author's name appears
> relative to parentheses.)

I applied my usecase which is \textcite and \parentcite.  The point it
that you have got two types of citations at hand: could be \textcite and
\footcite if you care more for that.

>> I don't like "@Doe99:journal:year".  It's too unlike existing syntax.
>
> I agree it's a little clunky, but I think most of the time there would
> just be one selector.  I was thinking of this on analogy with heading
> properties and tags...is there a better existing syntax to refer to a
> property value?

Perhaps it's similar to properties and tags.  I have key values in mind
which are either key:value or :key value as in OPTIONS lines and
MY_KEYWORD lines...  Perhaps it is not the correct reference.

>> Rather, I'd just introduce types as hinted previously, [@Doe99 :type
>> my-journal-year-type].  Org can provide as many predefined :type as we
>> care for, and let the user define the rest as necessary.
>
> I don't like this, because it seems like a lot more work for me as a
> user to achieve something that should be simple, and it trades 
> specifying /what/ data I want for describing it more indirectly.
>
> Suppose I'm writing a document, and I know I just want to reference the
> journal and year of a particular publication, in that order.  Being a
> studious keeper of my org-bibtex database, I already know that these
> fields are called "journal" and "year".  But if, in addition to
> reference database field names, I have to remember names for /types/ of
> /combinations/ of field names, that's too much.  

Reftex will do this for you.

> I'll end up taking endless trips to the manual to figure out which type
> I need in this case.  Do I need :type journal-before-year? :type
> journal-and-year?  etc.  This feels a bit too much like having to
> remember (or look up) all the different LaTeX citation commands.

Might be true.  I don't expect that problem much.

> What about just separating the field names off, as keys?  Like:
>
> [See Doe's review @Doe99 :journal :year]

That looks much better ("Org-ish"), though it implies

     [See Doe's review @Doe99 :journal nil :year nil]

Which is kind of the opposite of the desired...  Or perhaps I'm just
misreading it.

>>> When specific fields are requested, ONLY data from those fields should
>>> appear in the exported document.  Backends would choose how to export
>>> these citations based on the selected fields.
>>
>> What happens when a field is undefined?
>
> I guess I would suggest the same thing as happens in LaTeX: you get a
> nice, bold "??" in the output where the missing data should be. 

Or better, throw an error.

>> I think R-markdown uses something like [-@Smith79].  Again, I don't like
>> the [@key:nocite].
>
> Doesn't [-@Smith79] mean something different, namely, "cite @Smith79
> without the author name"?  It produces output like: "(1979)".

Thanks for the clarification.


>>> #+BEGIN_QUOTE
>>>     * Citations
>>>
>>>     #+ATTR_LATEX: :command citet
>>>     #+ATTR_HTML: :class my-citation
>>>     [cite:1] See @Doe99, pp. 34--45; @Foobar2000, ch.1.
>>> #+END_QUOTE
>>
>> Why not.  Since footnote-definition is a greater element it /does/ take
>> affiliated keywords, but I have never seen this used.
>
> Right, that's the point here...(were you disagreeing?)

No.

>> For inline maybe something like this:
>>     [@Key :type_html my-citation :type_latex citet] 
>
> Actually, this is a lot like the syntax I was thinking about for the
> inline case, but in the end I thought it was too complicated and new to
> be worth it, when the #+ATTR_BACKEND syntax will already work for the
> out-of-line case.  I'm not opposed to something like this in principle,
> but I really think we should try to keep the inline case very simple and
> obvious to use, even if that means restricting its expressiveness a bit.

The thing is it makes it very readable and obvious.  You can fix display
issues separately if you want.

>> From experience, the biblatex model of separating the loading of files,
>> styles and printing into different commands is a great advantage.
>
> OK.  I'd even be happy with
>
> #+BIBLIOGRAPHY:
> #+BIBLIGRAPHY_STYLE:
> #+PRINT_BIBLIOGRAPHY:
>
> where the first could be specified as many times as desired, to indicate
> external reference databases.

Yeah, I had those in an earlier draft.  My only issue is
that #+PRINT_BIBLIOGRAPHY: is awkward if nothing comes after the ':'.

>>> The point of specifying the style and locale as part of
>>> the #+BIBLIOGRAPHY definition is for compatibility with both LaTeX and
>>> Citation Style Language bibliography and citation formatting.
>>
>> Local is defined by #+LANGUAGE.  AFAIK, Org doesn't support many
>> languages.  E.g. here's the definition of LANGUAGE in ox.el:
>>
>> (:language "LANGUAGE" nil org-export-default-language t)
>
> Ah, OK, I didn't know about #+LANGUAGE.  Is there any reason why the
> locale of the bibliography might be different than the locale of the
> document?  (I'm monolingual, I'm afraid, so I doubt I could think of one
> if there were...)

AFAIK, #+LANGUAGE is a single element, e.g. 'en' or 'de'.  With ox-latex
you can load several languages via babel and org-latex-packages-alist.
Loading locals via other mechanism seems like a bug.

—Rasmus

-- 
With monopolies the cake is a lie!

Re: Citations, continued

[Eric S Fraga]

On Tuesday,  3 Feb 2015 at 11:35, Rasmus wrote:

[...]

I'm enjoying following this thread.  I look forward to the community
converging on some solution.

For me, any solution will likely do just fine as my use of citations is
quite straightforward.  I seldom, if ever, have pre or post text but I
do use a couple of alternative citation types (author, year; year only).

I have only one suggestion to keep in mind:

>>> What happens when a field is undefined?
>>
>> I guess I would suggest the same thing as happens in LaTeX: you get a
>> nice, bold "??" in the output where the missing data should be. 
>
> Or better, throw an error.

A *warning* would be better than an error, i.e. something that does
indicate a problem but that doesn't stop the export completing.  LaTeX
does this (as noted above).

When writing long articles, I often have dangling references which I
don't resolve until later.  I don't want to interrupt the writing part
(i.e. the creative process) by getting caught up in bookkeeping.  It's
sometimes hard enough to just get started... ;-)

Interestingly, I have just had a paper accepted for publication which
was written *entirely* in org.  I used the [[cite:fraga-etal-2014]]
approach for handling citations.  The paper made significant use of
babel to have everything in one place (data, code, results).  Very
pleasing and painless experience.  I did have to resort to LaTeX
specific commands a few times but mostly for the preamble (title,
authors, etc.).  I would share the org file except that it has
proprietary data.

thanks,
eric
-- 
: Eric S Fraga (0xFFFCF67D), Emacs 24.4.1, Org release_8.3beta-727-ga1cdc6.dirty

Re: Citations, continued

[Richard Lawrence]

Hi Eric and all,

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> On Tuesday,  3 Feb 2015 at 11:35, Rasmus wrote:

> I'm enjoying following this thread.  I look forward to the community
> converging on some solution.

Me too!

> For me, any solution will likely do just fine as my use of citations is
> quite straightforward.  I seldom, if ever, have pre or post text but I
> do use a couple of alternative citation types (author, year; year only).

Just to clarify: these are only `alternative' citation types if you're
not using a citation style where they are the default types, like
Chicago, right?  I assume you are using a numeric style, like ACM?

(This raises an interesting question, actually: what does the Pandoc
syntax do with author suppression for numeric citation styles?  Does
[-@Smith99] still output the year, or does it produce the same numeric
reference as [@Smith99]?)

> I have only one suggestion to keep in mind:
>
>>>> What happens when a field is undefined?
>>>
>>> I guess I would suggest the same thing as happens in LaTeX: you get a
>>> nice, bold "??" in the output where the missing data should be. 
>>
>> Or better, throw an error.
>
> A *warning* would be better than an error, i.e. something that does
> indicate a problem but that doesn't stop the export completing.  LaTeX
> does this (as noted above).

Agreed.  Something easily greppable, but not process-stopping.

> Interestingly, I have just had a paper accepted for publication which
> was written *entirely* in org.  I used the [[cite:fraga-etal-2014]]
> approach for handling citations.  The paper made significant use of
> babel to have everything in one place (data, code, results).  Very
> pleasing and painless experience.  I did have to resort to LaTeX
> specific commands a few times but mostly for the preamble (title,
> authors, etc.).

Cool!
 
Best,
Richard

Re: Citations, continued

[Eric S Fraga]

On Tuesday,  3 Feb 2015 at 08:27, Richard Lawrence wrote:

[...]

>> For me, any solution will likely do just fine as my use of citations is
>> quite straightforward.  I seldom, if ever, have pre or post text but I
>> do use a couple of alternative citation types (author, year; year only).
>
> Just to clarify: these are only `alternative' citation types if you're
> not using a citation style where they are the default types, like
> Chicago, right?  I assume you are using a numeric style, like ACM?

It depends on the journal I am writing for.  For those with numeric
styles (Chicago), i.e. "blah blah [1]", there are no alternatives so I
just use [[cite:blah]]; for journals expecting a Harvard (natbib) style
(blah, 1999), I will use both citet "Blah et al. (1999)" and citep
"(Blah et al, 1999)" as appropriate.

I prefer numeric styles but it's not typically up to me.  Harvard styles
are a throwback to typewriters (and people using Word etc. without any
automated reference software ;-).  IMO, of course!

So there needs to be some way to distinguish between styles, as already
noted I think (although much of the discussion has been about how to
include extra information...).
-- 
: Eric S Fraga (0xFFFCF67D), Emacs 25.0.50.1, Org release_8.3beta-750-gb6fce5.dirty

Re: Citations, continued

[Erik Hetzner]

[-- Attachment #1: Type: text/plain, Size: 3072 bytes --]

On Mon,  2 Feb 2015 at 10:02:41 PST,
Richard Lawrence <richard.lawrence@berkeley.edu> wrote:
> 
> Hi all,
> 
> Here is the citation syntax proposal I have mentioned in a couple of
> posts now.  I have attached it as an Org document for better
> readability, and also reproduced the text below.  Let me know what you
> think!

Hi Richard,

I am really, really glad to see people discussing citations in
org-mode. But I have some concerns about this proposal.

Before extensions are proposed to the pandoc format, I think it is
important to understand how flexible the combination of pandoc, and
what citeproc provides. I believe that pandoc can cover most of what
you want.

I also believe it would be a mistake to start from the idea of a
pandoc-style citation syntax that deviates from pandoc. Better instead
to start from what pandoc does now and find out what isn’t working for
org-mode users before extending pandoc, especially in ways that are
not compatible with pandoc.

And if extensions are proposed, it would be best to propose them on
the pandoc-discuss mailing list. It would be wonderful for users if
the syntax in pandoc-markdown and org-mode could stay aligned.

For more info on the flexibility of pandoc+citeproc, see
http://johnmacfarlane.net/pandoc/README.html#citations and
http://johnmacfarlane.net/pandoc/demos.html. It is also important to
distinguish what are features of the citeproc style (e.g. inline v.
footnote citations) and what are determined by the author and thus
should be present in the syntax (e.g. use or do not use a suffix or
locator).

From the example document:

1.   @item1 says blah.
2.   @item1 [p. 30] says blah.
3.   @item1 [p. 30, with suffix] says blah.
4.   @item1 [-@item2 p. 30; see also @item3] says blah.
5.   A citation group [see @item1 p. 34-35; also @item3 chap. 3].
6.   Another one [see @item1 p. 34-35].
7.   Citation with a suffix and locator [@item1 pp. 33, 35-37, and nowhere else].
8.   A citation without locators [@item3].
9.   Citation with suffix only [@item1 and nowhere else].
10.  Like a citation without author: [-@item1], and now Doe with a locator [-@item2 p. 44].

How this is rendered depends on the note style. In chicago author date it will have:

1.   Doe (2005) says blah.
2.   Doe (2005, 30) says blah.
3.   Doe (2005, 30, with suffix) says blah.
4.   Doe (2005; 2006, 30; see also Doe and Roe 2007) says blah.
5.   A citation group (see Doe 2005, 34–35; also Doe and Roe 2007, chap. 3).
6.   Another one (see Doe 2005, 34–35).
7.   Citation with a suffix and locator (Doe 2005, 33, 35–37, and nowhere else).
8.   A citation without locators (Doe and Roe 2007).
9.   Citation with suffix only (Doe 2005 and nowhere else).
10.  Like a citation without author: (2005), and now Doe with a locator (2006, 44).

with a bibliography, while in chicago fullnote bibliography everything
will be in footnotes (this is easier to see in HTML:
http://johnmacfarlane.net/pandoc/demo/example24b.html) or attached.

best, Erik


[-- Attachment #2: example24b.html --]
[-- Type: text/html, Size: 4458 bytes --]

[-- Attachment #3: Type: text/plain, Size: 56 bytes --]

--
Sent from my free software system <http://fsf.org/>.

Re: Citations, continued

[Richard Lawrence]

Hi Erik and all,

Erik Hetzner <egh@e6h.org> writes:

> I am really, really glad to see people discussing citations in
> org-mode. But I have some concerns about this proposal.
>
> Before extensions are proposed to the pandoc format, I think it is
> important to understand how flexible the combination of pandoc, and
> what citeproc provides. I believe that pandoc can cover most of what
> you want.

> I also believe it would be a mistake to start from the idea of a
> pandoc-style citation syntax that deviates from pandoc. Better instead
> to start from what pandoc does now and find out what isn’t working for
> org-mode users before extending pandoc, especially in ways that are
> not compatible with pandoc.

Actually, I totally agree.  For my own use, I would be completely happy
with just using the Pandoc syntax for citations in Org, without any
modifications.

The only reason I proposed anything else was that it seemed like other
people already know that they need more than the Pandoc syntax provides.
I think the main realistic cases are those where, in LaTeX, you'd use
commands like \citetitle, \citedate, or \citejournal -- citation
commands that pull in just a particular field from the reference,
because that is what the context around the citation requires.  I don't
see a way to do that in the Pandoc syntax.  (But am I missing
something?)  Hence my proposed field-selectors extension.

Personally, I need commands like these so little that I am happy to do
without them.  So maybe my proposal was a bit hasty.  Could we hear from
other people about how badly they need what such commands provide?

> And if extensions are proposed, it would be best to propose them on
> the pandoc-discuss mailing list. It would be wonderful for users if
> the syntax in pandoc-markdown and org-mode could stay aligned.

Yes, I again totally agree.  If people here settle on a syntax that is
close, but not quite the same as, Pandoc's, I will certainly do that.

Best,
Richard

Re: Citations, continued

[Erik Hetzner]

Hi Richard,

On Mon,  2 Feb 2015 at 20:41:06 PST,
Richard Lawrence <richard.lawrence@berkeley.edu> wrote:
> 
> Hi Erik and all,
> 
> Actually, I totally agree.  For my own use, I would be completely happy
> with just using the Pandoc syntax for citations in Org, without any
> modifications.

Great!

> The only reason I proposed anything else was that it seemed like other
> people already know that they need more than the Pandoc syntax provides.
> I think the main realistic cases are those where, in LaTeX, you'd use
> commands like \citetitle, \citedate, or \citejournal -- citation
> commands that pull in just a particular field from the reference,
> because that is what the context around the citation requires.  I don't
> see a way to do that in the Pandoc syntax.  (But am I missing
> something?)  Hence my proposed field-selectors extension.

If this is needed (and I still have a hard time seeing the use cases,
but I am not an academic) perhaps it could mimic the -@doe (suppress
author) syntax already used in pandoc (e.g. +title@doe). But
citeproc-js/hs only support suppress author or author only, so these
would not work in a pandoc export, nor any other that might depend on
citeproc-js.

> Personally, I need commands like these so little that I am happy to do
> without them.  So maybe my proposal was a bit hasty.  Could we hear from
> other people about how badly they need what such commands provide?
> 
> > And if extensions are proposed, it would be best to propose them on
> > the pandoc-discuss mailing list. It would be wonderful for users if
> > the syntax in pandoc-markdown and org-mode could stay aligned.
> 
> Yes, I again totally agree.  If people here settle on a syntax that is
> close, but not quite the same as, Pandoc's, I will certainly do that.

Again, this is great. I really do appreciate your getting this
proposal out there. I hope that I can finish porting my pandoc parser
to elisp within a week or so, so we can have an implementation to
start with.

best, Erik
--
Sent from my free software system <http://fsf.org/>.

Re: Citations, continued

[Richard Lawrence]

Hi Erik and all,

Erik Hetzner <egh@e6h.org> writes:

> On Mon,  2 Feb 2015 at 20:41:06 PST,
> Richard Lawrence <richard.lawrence@berkeley.edu> wrote:

>> The only reason I proposed anything else was that it seemed like other
>> people already know that they need more than the Pandoc syntax provides.
>> I think the main realistic cases are those where, in LaTeX, you'd use
>> commands like \citetitle, \citedate, or \citejournal -- citation
>> commands that pull in just a particular field from the reference,
>> because that is what the context around the citation requires.  I don't
>> see a way to do that in the Pandoc syntax.  (But am I missing
>> something?)  Hence my proposed field-selectors extension.
>
> If this is needed (and I still have a hard time seeing the use cases,
> but I am not an academic) perhaps it could mimic the -@doe (suppress
> author) syntax already used in pandoc (e.g. +title@doe). But
> citeproc-js/hs only support suppress author or author only, so these
> would not work in a pandoc export, nor any other that might depend on
> citeproc-js.

Yes.  I'll have to take a look at the Pandoc citeproc code and see how
easy it would be to add support for something like this.  If there are
Org people that need citation types that select specific fields, I
imagine there are Pandoc people who do as well.  The ideal would be if
citeproc would take care of proper formatting of all such citation
types, given just an ordered list of the fields that should appear.  I
don't know if CSL supports this, though; do you?

The +title@doe syntax would work, but I don't personally think that's
any easier to read (or parse) than @doe:title, or @doe+title (which
actually is another option I had thought of, given the +/- syntax for
tag matching in Org).  

> Again, this is great. I really do appreciate your getting this
> proposal out there. I hope that I can finish porting my pandoc parser
> to elisp within a week or so, so we can have an implementation to
> start with.

Thanks for taking the first step on this!  I had a brief look at your
parser code when you posted it the other day.

I don't know if you are familiar with Org's parser, in org-element.el.
The important thing will be to have a parser for citations return a data
structure in the format used by org-element.  In the language of
org-element, I think a citation is an object (as opposed to an element).
I am somewhat (but not super-) familiar with org-element and Elisp, so
if you want some help with this, let me know.

Best,
Richard

Re: Citations, continued

[Erik Hetzner]

Hi Richard,

On Tue,  3 Feb 2015 at 08:11:19 PST,
Richard Lawrence <richard.lawrence@berkeley.edu> wrote:
> 

[…]

> Yes.  I'll have to take a look at the Pandoc citeproc code and see how
> easy it would be to add support for something like this.  If there are
> Org people that need citation types that select specific fields, I
> imagine there are Pandoc people who do as well.  The ideal would be if
> citeproc would take care of proper formatting of all such citation
> types, given just an ordered list of the fields that should appear.  I
> don't know if CSL supports this, though; do you?

I’m not entirely sure what you mean. The authors of citeproc have come
up with a huge number of styles which seem to satisfy people’s needs.
What appears in the in-text citation is configurable, see:

  http://citationstyles.org/downloads/specification-csl101-20120903.html#citation

> The +title@doe syntax would work, but I don't personally think that's
> any easier to read (or parse) than @doe:title, or @doe+title (which
> actually is another option I had thought of, given the +/- syntax for
> tag matching in Org).  

The only advantage of it is that it doesn’t break existing cite keys,
which can contain :, and it is closer to existing syntax. But for the
time being it’s something I wouldn’t worry about.

> Thanks for taking the first step on this!  I had a brief look at your
> parser code when you posted it the other day.
> 
> I don't know if you are familiar with Org's parser, in org-element.el.
> The important thing will be to have a parser for citations return a data
> structure in the format used by org-element.  In the language of
> org-element, I think a citation is an object (as opposed to an element).
> I am somewhat (but not super-) familiar with org-element and Elisp, so
> if you want some help with this, let me know.

I concentrated on getting the parser to recognize valid citations
first. I have now finished this part (excepting any bugs, of course :)
and will need to add code to generate a proper parse tree. Then it can
be integrated into org-element.el.

best, Erik
--
Sent from my free software system <http://fsf.org/>.

Re: Citations, continued

[Nicolas Goaziou]

Hello,

Erik Hetzner <egh@e6h.org> writes:

> I concentrated on getting the parser to recognize valid citations
> first. I have now finished this part (excepting any bugs, of course :)
> and will need to add code to generate a proper parse tree. Then it can
> be integrated into org-element.el.

The point of "org-element.el" is to generate a parse tree. If you
generate it yourself, you do not need the library. Therefore, I'm not
sure to understand the "integration" part.

Also, AFAIU, the syntax for valid citations is not defined explicitly so
far. For example, I don't think it was discussed if any subset of Org
objects (e.g., macros or bold text) is allowed in a citation. Is there
any document we could merge into Org syntax document?


Regards,

-- 
Nicolas Goaziou

Re: Citations, continued

[Richard Lawrence]

Hi Nicolas,

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Also, AFAIU, the syntax for valid citations is not defined explicitly so
> far. For example, I don't think it was discussed if any subset of Org
> objects (e.g., macros or bold text) is allowed in a citation.

This is a good question that, as you say, deserves more discussion.

I am not sure, but I am *thinking* that inline citations are objects
(not elements) that can contain some other objects.

Specifically I think we need the following categories, all of which
would be objects:
  - key
  - prefix / pre-text
  - suffix / post-text
  - locator
  - individual citation
  - bracketed citation
  - unbracketed citation

These should have a grammar like the following, based on my
(reverse-engineered) understanding of the Pandoc syntax for citations:

  - A bracketed citation is a list of one or more individual citations, 
    separated by ';' if there are two or more, and surrounded by '[' ']'
  - An individual citation is formatted like: PREFIX KEY LOCATOR SUFFIX
    The key is obligatory, and the prefix, locator and suffix
    are optional.
  - A key optionally begins with '-', and obligatorily contains '@'
    followed by a string of charcters which begins with a letter or '_',
    and may contain alphanumeric characters and the following internal
    punctuation characters:
       :.#$%&-+?<>~/
  - A prefix or suffix is a text object (that may contain markup like
    emphasis or macros)
  - An unbracketed citation consists of a key, optionally followed by a
    locator which is enclosed in '[' ']'
    
I am not sure about the syntax of locators.  In particular, I do not
know if they should allow internal markup, I do not know if they have an
internal syntax, and I do not know if a comma is required to separate
them from a key in a bracketed citation.

Best,
Richard

Re: Citations, continued

[Nicolas Goaziou]

Richard Lawrence <richard.lawrence@berkeley.edu> writes:

Thanks for this reverse engineering.

> Specifically I think we need the following categories, all of which
> would be objects:
>   - key
>   - prefix / pre-text
>   - suffix / post-text
>   - locator
>   - individual citation
>   - bracketed citation
>   - unbracketed citation
>
> These should have a grammar like the following, based on my
> (reverse-engineered) understanding of the Pandoc syntax for citations:
>
>   - A bracketed citation is a list of one or more individual citations, 
>     separated by ';' if there are two or more, and surrounded by '[' ']'
>   - An individual citation is formatted like: PREFIX KEY LOCATOR SUFFIX
>     The key is obligatory, and the prefix, locator and suffix
>     are optional.
>   - A key optionally begins with '-', and obligatorily contains '@'
>     followed by a string of charcters which begins with a letter or '_',
>     and may contain alphanumeric characters and the following internal
>     punctuation characters:
>        :.#$%&-+?<>~/
>   - A prefix or suffix is a text object (that may contain markup like
>     emphasis or macros)
>   - An unbracketed citation consists of a key, optionally followed by a
>     locator which is enclosed in '[' ']'

I don't think all should be objects. For example, prefix and suffix can
be properties in a `full-citation' object (like :tag in items).

IIUC, we need three objects (I'm not wedded to the names):

  - short-citation (aka unbracketed citation), with :cite-key
    and :locator properties, both being strings and :suppress-author as
    a boolean ;
    
  - full-citation (aka individual citation), with, in addition to the
    properties above, :prefix and :suffix, both being parsed string.

Since full citations can only exist in a bracketed citation, there is no
reason to create a third object type for the latter. It acts as a mere
container only useful for lexer.

> I am not sure about the syntax of locators.  In particular, I do not
> know if they should allow internal markup, I do not know if they have an
> internal syntax, and I do not know if a comma is required to separate
> them from a key in a bracketed citation.

This needs to be decided indeed. Is there any reason to allow markup
there?

My only concern is speed. A bracketed citation can induce a lot of
backtracking since it can be triggered each time a square bracket is
opened, which is not too uncommon, I think. Basically, at each "[", we
need to find corresponding "]", and if there is, any key between the
two. That's some overhead.

Also, syntax is ambiguous. For example, in

  [[http://orgmode.org][some @key]]

it is not clear if @key should be treated as a short-citation in a link
description, or included in a full citation with
"[http://orgmode.org][some " as its prefix. I mean, the answer is clear
for you and me, but not necessarily at lexer's level. For example,
Eric's parser chose the former, which is good, but also disallows square
brackets in prefix, which rules out some objects from this location
(mainly links and footnotes).

That's why I suggested the [cite: ...] part in the first place, which
you dismissed quickly. It reduces backtracking a lot and can solve
easily some confusing situations.

Of course I understand the need for compatibility with existing Pandoc
syntax, but I wouldn't want us to shoot ourselves in the foot. Even if
we don't use "cite:" markup, I think we should carefully specify current
syntax to avoid loopholes.


Regards,

-- 
Nicolas Goaziou

Re: Citations, continued

[Richard Lawrence]

Hi Nicolas and all,

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Richard Lawrence <richard.lawrence@berkeley.edu> writes:
>
> Thanks for this reverse engineering.
>
>> Specifically I think we need the following categories, all of which
>> would be objects:
>>   - key
>>   - prefix / pre-text
>>   - suffix / post-text
>>   - locator
>>   - individual citation
>>   - bracketed citation
>>   - unbracketed citation
>>
>> These should have a grammar like the following, based on my
>> (reverse-engineered) understanding of the Pandoc syntax for citations:
>>
>>   - A bracketed citation is a list of one or more individual citations, 
>>     separated by ';' if there are two or more, and surrounded by '[' ']'
>>   - An individual citation is formatted like: PREFIX KEY LOCATOR SUFFIX
>>     The key is obligatory, and the prefix, locator and suffix
>>     are optional.
>>   - A key optionally begins with '-', and obligatorily contains '@'
>>     followed by a string of charcters which begins with a letter or '_',
>>     and may contain alphanumeric characters and the following internal
>>     punctuation characters:
>>        :.#$%&-+?<>~/
>>   - A prefix or suffix is a text object (that may contain markup like
>>     emphasis or macros)
>>   - An unbracketed citation consists of a key, optionally followed by a
>>     locator which is enclosed in '[' ']'
>
> I don't think all should be objects. For example, prefix and suffix can
> be properties in a `full-citation' object (like :tag in items).

Yes, sorry, this was dumb of me...for some reason, I was thinking
"everything in Org syntax has to be an object or an element, and these
aren't elements, so they're objects".  But obviously, some of these
categories are merely internal or merely represent properties of
objects.

> IIUC, we need three objects (I'm not wedded to the names):
>
>   - short-citation (aka unbracketed citation), with :cite-key
>     and :locator properties, both being strings and :suppress-author as
>     a boolean ;
>     
>   - full-citation (aka individual citation), with, in addition to the
>     properties above, :prefix and :suffix, both being parsed string.

> Since full citations can only exist in a bracketed citation, there is no
> reason to create a third object type for the latter. It acts as a mere
> container only useful for lexer.

I think this is not quite right: in my original terminology, `individual
citation' is just an intermediate category.  A bracketed/full citation
contains at least one, but may contain many, `individual' citations,
like:

[See @Doe99, p. 3; also @Doe2000, p. 989.]

This is a bracketed/full citation containing two individual citations,
each with their own prefix and suffix.   

>> I am not sure about the syntax of locators.  In particular, I do not
>> know if they should allow internal markup, I do not know if they have an
>> internal syntax, and I do not know if a comma is required to separate
>> them from a key in a bracketed citation.
>
> This needs to be decided indeed. Is there any reason to allow markup
> there?

I had a look at the Pandoc parser; see:
https://github.com/jgm/pandoc/blob/master/src/Text/Pandoc/Readers/Markdown.hs
(Citation stuff starts at line 1843.)

My Haskell is about as good as my German (which is to say: not very),
but I think I learned a few interesting things.

First, from what I can tell, there is actually no separate category of
`locators', despite the documentation.  There's just the suffix, which
is anything between the key and `;' or `]'.  (The bareloc function seems
to just look for a regular suffix.  But I could be missing something.)
Thus, maybe we can drop that.

Also, it appears that you can write things like

@Smith99 [p. 33; see also @Doe2014] says something interesting.

That is, an in-text citation with a suffix may also contain further
citations in the brackets, after the suffix for the in-text citations.
  
> My only concern is speed. A bracketed citation can induce a lot of
> backtracking since it can be triggered each time a square bracket is
> opened, which is not too uncommon, I think. Basically, at each "[", we
> need to find corresponding "]", and if there is, any key between the
> two. That's some overhead.

Good point.  I hadn't thought about this at all.

> Also, syntax is ambiguous. For example, in
>
>   [[http://orgmode.org][some @key]]
>
> it is not clear if @key should be treated as a short-citation in a link
> description, or included in a full citation with
> "[http://orgmode.org][some " as its prefix. I mean, the answer is clear
> for you and me, but not necessarily at lexer's level. For example,
> Eric's parser chose the former, which is good, but also disallows square
> brackets in prefix, which rules out some objects from this location
> (mainly links and footnotes).

Yes, good point.  Also, inline export snippets (@@latex: ...@@)
could prove problematic.

I do think it's important to allow some markup in the prefix and suffix,
because there are obvious uses where you might want emphasis, etc.

My initial thought is that a prefix or suffix should only allow:
  - Entities and LaTeX fragments
  - Line breaks?
  - Macros
  - Text markup

I'd also be happy without macros and line breaks, personally.

> That's why I suggested the [cite: ...] part in the first place, which
> you dismissed quickly. It reduces backtracking a lot and can solve
> easily some confusing situations.
>
> Of course I understand the need for compatibility with existing Pandoc
> syntax, but I wouldn't want us to shoot ourselves in the foot. Even if
> we don't use "cite:" markup, I think we should carefully specify current
> syntax to avoid loopholes.

Another interesting thing I learned from the Pandoc source is that,
should we want to adopt "[cite: ...]" syntax, I think it would be pretty
trivial for Pandoc to support it.  (Worst case, they can copy-and-paste
the Markdown citation parser and then add "cite:" in a couple of
places.)  So if this is necessary on the Org side for performance or
ambiguity reasons, I am not against it.

One question, though, is how this should work with in-text citations.
Should I have to write:

@Smith99 [cite:p. 33]

or

@Smith99 [cite:p. 33; see also @Doe2014]

?

Best,
Richard

Re: Citations, continued

[Nicolas Goaziou]

Richard Lawrence <richard.lawrence@berkeley.edu> writes:

>> Since full citations can only exist in a bracketed citation, there is no
>> reason to create a third object type for the latter. It acts as a mere
>> container only useful for lexer.
>
> I think this is not quite right: in my original terminology, `individual
> citation' is just an intermediate category.  A bracketed/full citation
> contains at least one, but may contain many, `individual' citations,
> like:
>
> [See @Doe99, p. 3; also @Doe2000, p. 989.]
>
> This is a bracketed/full citation containing two individual citations,
> each with their own prefix and suffix.

You're right. I was confused about citations.

> I do think it's important to allow some markup in the prefix and suffix,
> because there are obvious uses where you might want emphasis, etc.
>
> My initial thought is that a prefix or suffix should only allow:
>   - Entities and LaTeX fragments
>   - Line breaks?
>   - Macros
>   - Text markup
>
> I'd also be happy without macros and line breaks, personally.

Fine.

What about the following set?

 bold code entity italic latex-fragment line-break strike-through 
 subscript superscript underline superscript

>> That's why I suggested the [cite: ...] part in the first place, which
>> you dismissed quickly. It reduces backtracking a lot and can solve
>> easily some confusing situations.
>>
>> Of course I understand the need for compatibility with existing Pandoc
>> syntax, but I wouldn't want us to shoot ourselves in the foot. Even if
>> we don't use "cite:" markup, I think we should carefully specify current
>> syntax to avoid loopholes.
>
> Another interesting thing I learned from the Pandoc source is that,
> should we want to adopt "[cite: ...]" syntax, I think it would be pretty
> trivial for Pandoc to support it.  (Worst case, they can copy-and-paste
> the Markdown citation parser and then add "cite:" in a couple of
> places.)  So if this is necessary on the Org side for performance or
> ambiguity reasons, I am not against it.
>
> One question, though, is how this should work with in-text citations.
> Should I have to write:
>
> @Smith99 [cite:p. 33]
>
> or
>
> @Smith99 [cite:p. 33; see also @Doe2014]
>
> ?

To be clear, much like Rasmus, I don't like much in-text citations
syntax above. Actually, I would suggest to mimic footnotes, and handle
in-text citations with the same syntax as named footnotes.

Using the example from Erik Hetzner in the same thread, what about:

  1. [cite:@item1] says blah.
  2. [cite:@item1: p. 30] says blah.
  3. [cite:@item1: p. 30, with suffix] says blah.
  4. [cite:@item1: -@item2 p. 30; see also @item3] says blah.
  5. A citation group [cite:: see @item1 p. 34-35; also @item3 chap. 3].
  6. Another one [cite::see @item1 p. 34-35].
  7. Citation with a suffix and locator [cite:: @item1 pp. 33, 35-37, and nowhere else].
  8. A citation without locators [cite:: @item3].
  9. Citation with suffix only [cite:: @item1 and nowhere else].
  10. Like a citation without author: [cite:: -@item1], and now Doe
      with a locator [cite:: -@item2 p. 44].

As a reminder, here is a possible output from the text above

  1.   Doe (2005) says blah.
  2.   Doe (2005, 30) says blah.
  3.   Doe (2005, 30, with suffix) says blah.
  4.   Doe (2005; 2006, 30; see also Doe and Roe 2007) says blah.
  5.   A citation group (see Doe 2005, 34–35; also Doe and Roe 2007, chap. 3).
  6.   Another one (see Doe 2005, 34–35).
  7.   Citation with a suffix and locator (Doe 2005, 33, 35–37, and nowhere else).
  8.   A citation without locators (Doe and Roe 2007).
  9.   Citation with suffix only (Doe 2005 and nowhere else).
  10.  Like a citation without author: (2005), and now Doe with a locator (2006, 44).

Note that space after the second colon is not mandatory. More
explicitly, syntax would be either

  [cite:IN-TEXT-KEY]

or

  [cite:IN-TEXT-KEY?:SPACE* CITATIONS]

where CITATIONS is any number of

  PREFIX? KEY SUFFIX? 

separated with semi-colons.

It is slightly more verbose, but also more regular and faster to parse.


Regards,

Re: Citations, continued

[Richard Lawrence]

Hi Nicolas,

I just want to say thanks for continuing the conversation, by the way:
I know this thread has gotten long, but I'm glad people are still
paying attention, and Nicolas, your opinion counts for a lot.

On Sat, Feb 7, 2015 at 2:43 PM, Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:

> What about the following set?
>
>  bold code entity italic latex-fragment line-break strike-through
>  subscript superscript underline superscript

That would work fine for me in prefixes and suffixes.

> To be clear, much like Rasmus, I don't like much in-text citations
> syntax above. Actually, I would suggest to mimic footnotes, and handle
> in-text citations with the same syntax as named footnotes.
>
> Using the example from Erik Hetzner in the same thread, what about:
>
>   1. [cite:@item1] says blah.
>   2. [cite:@item1: p. 30] says blah.
>   3. [cite:@item1: p. 30, with suffix] says blah.
>   4. [cite:@item1: -@item2 p. 30; see also @item3] says blah.
>   5. A citation group [cite:: see @item1 p. 34-35; also @item3 chap. 3].
>   6. Another one [cite::see @item1 p. 34-35].
>   7. Citation with a suffix and locator [cite:: @item1 pp. 33, 35-37, and nowhere else].
>   8. A citation without locators [cite:: @item3].
>   9. Citation with suffix only [cite:: @item1 and nowhere else].
>   10. Like a citation without author: [cite:: -@item1], and now Doe
>       with a locator [cite:: -@item2 p. 44].
>
> explicitly, syntax would be either
>
>   [cite:IN-TEXT-KEY]
>
> or
>
>   [cite:IN-TEXT-KEY?:SPACE* CITATIONS]
>
> where CITATIONS is any number of
>
>   PREFIX? KEY SUFFIX?
>
> separated with semi-colons.
>
> It is slightly more verbose, but also more regular and faster to parse.

I guess I could live with this, but to be honest, I much prefer the Pandoc way.

The Pandoc syntax has a nice congruence between the source file and
the output: if a cite key is inside the brackets in the source, the
reference is inside the brackets in the output, and if it's outside in
the source, it's outside in the output.  This convention seems
natural, easy to remember, and very readable -- at least if, like me
(and I would guess many others), you use author names in cite keys.

By contrast, I'm used to thinking of footnote labels as having no
direct correlation with the output, so it seems incongruent to use the
analogous position in citation syntax to represent in-text citations.

So as an author, I prefer the Pandoc way, but I understand there are
other considerations.  If we must have the tag for performance
reasons, I would prefer using two different tags to represent the two
cases; I suggest borrowing (from LaTeX's natbib package) "citet" for
in-text and "citep" for bracketed citations , but I don't really care
as long as they're easy to type, and it's easy to change one to the
other.

What do others think?

Best,
Richard

Re: Citations, continued

[John Kitchin]

My only concern is that it remains possible to support this relatively
full set of citation options on export:

(defcustom org-ref-cite-types
  '("cite" "nocite" ;; the default latex cite commands
    ;; natbib cite commands, http://ctan.unixbrain.com/macros/latex/contrib/natbib/natnotes.pdf
    "citet" "citet*" "citep" "citep*"
    "citealt" "citealt*" "citealp" "citealp*"
    "citenum" "citetext"
    "citeauthor" "citeauthor*"
    "citeyear" "citeyear*"
    "Citet" "Citep" "Citealt" "Citealp" "Citeauthor"
    ;; biblatex commands
    ;; http://ctan.mirrorcatalogs.com/macros/latex/contrib/biblatex/doc/biblatex.pdf
    "Cite"
    "parencite" "Parencite"
    "footcite" "footcitetext"
    "textcite" "Textcite"
    "smartcite" "Smartcite"
    "cite*" "parencite*" "supercite"
    "autocite" "Autocite" "autocite*" "Autocite*"
    "Citeauthor*"
    "citetitle" "citetitle*"
    "citedate" "citedate*"
    "citeurl"
    "fullcite" "footfullcite"
    ;; "volcite" "Volcite" cannot support the syntax
    "notecite" "Notecite"
    "pnotecite" "Pnotecite"
    "fnotecite"
    ;; multicites. Very limited support for these.
    "cites" "Cites" "parencites" "Parencites"
    "footcites" "footcitetexts"
    "smartcites" "Smartcites" "textcites" "Textcites"
    "supercites" "autocites" "Autocites"
    ;; for the bibentry package
    "bibentry"
    )
  "List of citation types known in org-ref"
  :type '(repeat :tag "List of citation types" string)
  :group 'org-ref)

which we are currently able to do. I never type any of those in, org-ref
does it automatically from a key selection tool.

From a user point of view, these to me look almost identical to link
syntax, which could be confusing.

 [cite:IN-TEXT-KEY]
 [cite:IN-TEXT-KEY?:SPACE* CITATIONS]

I don't see the payoff in adding it. you can already have those as
links, and export to pandoc syntac trivial in the first case. The
payoff only comes from item 4 in the list below, with a nested citation,
which I have never seen in the wild in scientific publications. Perhaps
it is more common in other fields.

I don't personally find the pandoc notation easier to understand, eg
[@item] and [-@item]

vs

\cite{item} and \citeauthor{item}

The congruence you mention is not relevant for superscripted references,
there are no brackets or parentheses in that case.



Richard Lawrence writes:

> Hi Nicolas,
>
> I just want to say thanks for continuing the conversation, by the way:
> I know this thread has gotten long, but I'm glad people are still
> paying attention, and Nicolas, your opinion counts for a lot.
>
> On Sat, Feb 7, 2015 at 2:43 PM, Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:
>
>> What about the following set?
>>
>>  bold code entity italic latex-fragment line-break strike-through
>>  subscript superscript underline superscript
>
> That would work fine for me in prefixes and suffixes.
>
>> To be clear, much like Rasmus, I don't like much in-text citations
>> syntax above. Actually, I would suggest to mimic footnotes, and handle
>> in-text citations with the same syntax as named footnotes.
>>
>> Using the example from Erik Hetzner in the same thread, what about:
>>
>>   1. [cite:@item1] says blah.
>>   2. [cite:@item1: p. 30] says blah.
>>   3. [cite:@item1: p. 30, with suffix] says blah.
>>   4. [cite:@item1: -@item2 p. 30; see also @item3] says blah.
>>   5. A citation group [cite:: see @item1 p. 34-35; also @item3 chap. 3].
>>   6. Another one [cite::see @item1 p. 34-35].
>>   7. Citation with a suffix and locator [cite:: @item1 pp. 33, 35-37, and nowhere else].
>>   8. A citation without locators [cite:: @item3].
>>   9. Citation with suffix only [cite:: @item1 and nowhere else].
>>   10. Like a citation without author: [cite:: -@item1], and now Doe
>>       with a locator [cite:: -@item2 p. 44].
>>
>> explicitly, syntax would be either
>>
>>   [cite:IN-TEXT-KEY]
>>
>> or
>>
>>   [cite:IN-TEXT-KEY?:SPACE* CITATIONS]
>>
>> where CITATIONS is any number of
>>
>>   PREFIX? KEY SUFFIX?
>>
>> separated with semi-colons.
>>
>> It is slightly more verbose, but also more regular and faster to parse.
>
> I guess I could live with this, but to be honest, I much prefer the Pandoc way.
>
> The Pandoc syntax has a nice congruence between the source file and
> the output: if a cite key is inside the brackets in the source, the
> reference is inside the brackets in the output, and if it's outside in
> the source, it's outside in the output.  This convention seems
> natural, easy to remember, and very readable -- at least if, like me
> (and I would guess many others), you use author names in cite keys.
>
> By contrast, I'm used to thinking of footnote labels as having no
> direct correlation with the output, so it seems incongruent to use the
> analogous position in citation syntax to represent in-text citations.
>
> So as an author, I prefer the Pandoc way, but I understand there are
> other considerations.  If we must have the tag for performance
> reasons, I would prefer using two different tags to represent the two
> cases; I suggest borrowing (from LaTeX's natbib package) "citet" for
> in-text and "citep" for bracketed citations , but I don't really care
> as long as they're easy to type, and it's easy to change one to the
> other.
>
> What do others think?
>
> Best,
> Richard

--
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu

Re: Citations, continued

[Richard Lawrence]

Hi John and all,

On Sun, Feb 8, 2015 at 1:46 AM, John Kitchin <jkitchin@andrew.cmu.edu> wrote:
> My only concern is that it remains possible to support this relatively
> full set of citation options on export:
> ...
> which we are currently able to do. I never type any of those in, org-ref
> does it automatically from a key selection tool.

My original proposal was that we achieve this by allowing splitting
citations into an in-line pointer and out-of-line definition.  Since
the out-of-line definition would be a greater element, it could take
#+ATTR_BACKEND properties, which could be used for this, like so:

The literature is divided on this point. [cite:1]

#+ATTR_LATEX: :command autocites
[cite:1] For Position A see @Doe99; @Smith99; for Position B see @Foobar87.

What I like about this is that it separates the citation part from the
LaTeX-specific part, and it leverages existing syntax for the
LaTeX-specific part.  Something like this seems right to me.

Would that work for you, most of the time?

I am not sure we could do this under Nicolas' current proposal, but
maybe we could.

Also, I realize now that this out-of-line definition style doesn't
mesh super well with any way of expressing the distinction between
bracketed and in-text citations that has been suggested, so it still
needs a bit more thought, at least if this style can be used for
in-text citations.

> From a user point of view, these to me look almost identical to link
> syntax, which could be confusing.

I'm not overly worried about this, because it doesn't seem worse than
distinguishing links and footnotes.  (Highlighting can help, too.)

> I don't personally find the pandoc notation easier to understand, eg
> [@item] and [-@item]
>
> vs
>
> \cite{item} and \citeauthor{item}

Well, I think it's a *little* better than straight LaTeX syntax.  As
far as readability goes, it's nice that you can put the prefix and
suffix in the order they will read ("see @Doe99 for more" vs.
"\cite[see][for more]{Doe99}").

But the main advantage is not that it is easier to understand (it
isn't, at least not if you already know LaTeX), but that it is
abstracted from LaTeX syntax, or any other target format's syntax.
We're looking to make citations first-class syntactic citizens in Org.
  (We *could* even do that just by adopting LaTeX syntax as our
own...but I don't know if anyone would plump for that. :)

> The congruence you mention is not relevant for superscripted references,
> there are no brackets or parentheses in that case.

True.  Do you use superscripted references a lot?  I never use them, personally.

Best,
Richard

Re: Citations, continued

[Thomas S. Dye]

Aloha all,

I'm having a hard time relating the various syntax proposals to a
common, shared goal for Org mode users.  This might just be me--I often
struggle to understand conversations on the Org mode list.  However, I'm
also convinced that lack of clearly shared goals really gets in the way
of project success.  So, because I'd like to see Org mode settle on a
citation syntax, I'll risk exposing my limitations by describing what I
think a reasonable goal might be and then offer some comments on a
recent post.

IIUC, Org mode citation syntax needs to capture four pieces of
information for an *individual* citation: a =key= into one or more
stores of bibliographic information; a =citation-command= that is
understood by the =citation-style= specified for the document; a
=pre-note= of arbitrary text in any language; and a =post-note= of
arbitrary text in any language.  At least, this is how the LaTeX world
accommodates the almost unconstrained and ever-growing variability in
bibliographic styles in the wild.

Note that =key= and =citation-command= are limited for any one document
by the keys used in the store of bibliographic information and the
commands defined for the citation style.  In practice, though, both
=key= and =citation-command= are arbitrarily complex, because there are
no universal constraints on either the content of keys used by current
and future bibliographic information stores, or the commands defined by
current and future citation styles.  Also, there is no guarantee that
the Org mode author created any of the keys and it is likely that the
Org mode author did not name any of the citation commands.

When I have targeted a specific citation style (typically specified by a
publisher) and a particular group of objects to cite, I want Org mode to
1) treat me kindly when I am creating a citation for my document, 2)
give me just enough information in the Org mode buffer so I can easily
identify the four pieces of information I've entered *without having to
do anything except read*, and 3) help me change from one citation
command to another when I'm editing.

During citation creation I want access to all the keys in the
bibliographic store(s), enough information so I can unambiguously
identify the object associated with each key, easy access to all the
commands defined by the citation style, and no constraints at all on
what I can write for the pre-note and post-note.  I want this part to be
as verbose as possible, but limited to the information that I really need.

When I look at the citation in the Org mode buffer I want to 1) see the
pre-note and post-note, preferably in the correct relationship to the
citation, 2) know what citation command I selected, and 3) know which
object I've cited.  *I don't want to see the key*; the keys I create
myself are long and ugly and those created by others are typically long,
ugly, and (for me) often indecipherable.  I also don't want to see the
citation command, just an indicator that distinguishes among the handful
or so possibilities allowed by the citation style and actually used in
the document.

When I'm rewriting or editing, I often need to change the citation
command, usually to switch between, e.g., (Lawrence 2014) and Lawrence
(2014).  In my current setup this is one of the actions that has its own
hot key (I have only a few hot keys because I have a hard time
remembering them).

Now that I've specified goals, there are a few comments interspersed
below.

Richard Lawrence <richard.lawrence@berkeley.edu> writes:

> Hi John and all,
>
> On Sun, Feb 8, 2015 at 1:46 AM, John Kitchin <jkitchin@andrew.cmu.edu> wrote:
>> My only concern is that it remains possible to support this relatively
>> full set of citation options on export:
>> ...
>> which we are currently able to do. I never type any of those in, org-ref
>> does it automatically from a key selection tool.

I'd rather just see the citation commands associated with a particular
citation style.  I've been switching from natbib to biblatex over the
last few years (natbib at work with my colleagues there, and biblatex
for my research documents not associated with work) and I want Org mode
to remember which one I'm using, if possible, so I don't have to think
about it.

>
> My original proposal was that we achieve this by allowing splitting
> citations into an in-line pointer and out-of-line definition.  Since
> the out-of-line definition would be a greater element, it could take
> #+ATTR_BACKEND properties, which could be used for this, like so:
>
> The literature is divided on this point. [cite:1]
>
> #+ATTR_LATEX: :command autocites
> [cite:1] For Position A see @Doe99; @Smith99; for Position B see @Foobar87.
>

This is extremely problematic for me.  IIUC, I'd often need to click on
[cite:1] to learn what I'd cited and then I'd need to click again to
figure out which object was associated with each key.  I want to be able
to jump directly to the entry in the bibliographic store from the
in-text citation, and I want to jump into the bibliographic store only
when I want to pick out some specific information from the entry.

> What I like about this is that it separates the citation part from the
> LaTeX-specific part, and it leverages existing syntax for the
> LaTeX-specific part.  Something like this seems right to me.

I don't think the citation command should be thought of as LaTeX
specific, but rather as one of the four pieces of information required
to create arbitrarily complex citations in the output.  IIUC, it should
be possible to generate the required information for any document
preparation system from the information potentially supplied by
=pre-note=, =post-note=, =citation-command=, and =key=.

All the best,
Tom

-- 
Thomas S. Dye
http://www.tsdye.com

Re: Citations, continued

[e.fraga]

Tom,

thanks for this contribution.  From my point of view, you have
summarised the requirements very well.  I agree with them completely,
especially the need to expose and to hide the right information when
perusing the org file.

-- 
Eric S Fraga (GnuPG: 0xC89193D8FFFCF67D)

Re: Citations, continued

[Rasmus]

Aloha Tom,

Were obviously have similar requirements.  However, there's one point
where I'm entirely on the same page as you.  I want to figure out if
that's 'cause I haven't thought carefully enough about it, or because I
have simpler requirements.

tsd@tsdye.com (Thomas S. Dye) writes:

> I don't think the citation command should be thought of as LaTeX
> specific, but rather as one of the four pieces of information required
> to create arbitrarily complex citations in the output.  IIUC, it should
> be possible to generate the required information for any document
> preparation system from the information potentially supplied by
> =pre-note=, =post-note=, =citation-command=, and =key=.

The question: 

     In any given document, do you typically need more than two types of
     citations, i.e. {citet, citep} OR {textcite, parentcite}?

I do use other citation types, in particular a genitive version of
textcite, but not very often.  That was why I initially wanted something
like this:

simple  inline:         @KEY 
complex inline:         [PRE @KEY POST :key VAL]
        parent:         (PRE @KEY POST :key VAL)

Where :type was the only key I was clever enough to think about (heavily
biased by LaTeX).  Then you would be able to put in the top of your
document what "inline" and that "parent" means.  Note, as Nicolas rightly
pointed out that (·) shouldn't be used for syntax, so the above is to
understand needs.  In any case, if you, or Eric, or anybody else for that
matter, often rely on much more than two types of citations in any given
document perhaps this is better:

      [TYPE: PRE @KEY POST :key VAL]

:key VAL may not be needed at all (but e.g. the new cool
\textcites()()[][]{} commands have even more arguments).  That's
essentially the "generalized link" you were talking about earlier.

Display in the buffer is indeed something that should be tackled, but this
is an issue of overlays, like entities, and not the immediate issue.  But
I have it in mind as well.  It's tough though, when you have other
citations types than author-year.

Cheers,
Rasmus

-- 
Lasciate ogni speranza, voi che leggete questo.

Re: Citations, continued

[Nicolas Goaziou]

Rasmus <rasmus@gmx.us> writes:

> The question: 
>
>      In any given document, do you typically need more than two types of
>      citations, i.e. {citet, citep} OR {textcite, parentcite}?
>
> I do use other citation types, in particular a genitive version of
> textcite, but not very often.  That was why I initially wanted something
> like this:
>
> simple  inline:         @KEY 
> complex inline:         [PRE @KEY POST :key VAL]
>         parent:         (PRE @KEY POST :key VAL)
>
> Where :type was the only key I was clever enough to think about (heavily
> biased by LaTeX).  Then you would be able to put in the top of your
> document what "inline" and that "parent" means.  Note, as Nicolas rightly
> pointed out that (·) shouldn't be used for syntax, so the above is to
> understand needs.  In any case, if you, or Eric, or anybody else for that
> matter, often rely on much more than two types of citations in any given
> document perhaps this is better:
>
>       [TYPE: PRE @KEY POST :key VAL]
>
> :key VAL may not be needed at all (but e.g. the new cool
> \textcites()()[][]{} commands have even more arguments).  That's
> essentially the "generalized link" you were talking about earlier.

This is an important issue indeed.

It seems to me that :type is a LaTeX-only feature and, as such, should
be handled in "ox-latex". In the general case, I think that Org should
only support inline and parenthesized citations.

If more than two different keys are needed in a single document, use of
custom links or raw LaTeX would then be unavoidable. OTOH, this gives us
very readable citations within the buffer in most cases.

Re: Citations, continued

[Rasmus]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Rasmus <rasmus@gmx.us> writes:
>
>> The question: 
>>
>>      In any given document, do you typically need more than two types of
>>      citations, i.e. {citet, citep} OR {textcite, parentcite}?
>>
>> I do use other citation types, in particular a genitive version of
>> textcite, but not very often.  That was why I initially wanted something
>> like this:
>>
>> simple  inline:         @KEY 
>> complex inline:         [PRE @KEY POST :key VAL]
>>         parent:         (PRE @KEY POST :key VAL)
>>
>> Where :type was the only key I was clever enough to think about (heavily
>> biased by LaTeX).  Then you would be able to put in the top of your
>> document what "inline" and that "parent" means.  Note, as Nicolas rightly
>> pointed out that (·) shouldn't be used for syntax, so the above is to
>> understand needs.  In any case, if you, or Eric, or anybody else for that
>> matter, often rely on much more than two types of citations in any given
>> document perhaps this is better:
>>
>>       [TYPE: PRE @KEY POST :key VAL]
>>
>> :key VAL may not be needed at all (but e.g. the new cool
>> \textcites()()[][]{} commands have even more arguments).  That's
>> essentially the "generalized link" you were talking about earlier.
>
> This is an important issue indeed.
>
> It seems to me that :type is a LaTeX-only feature and, as such, should
> be handled in "ox-latex". In the general case, I think that Org should
> only support inline and parenthesized citations.

Not necessarily.  I could do:

   (defun rasmus/gentive-citation (citation-element backend)
          (case backend ...) ...)

   (add-to-list 'org-cite-types 'rasmus/gentive-citation )

E.g. for genitive citations such as "Smith's (1984) model", which can be
mostly emulated using primitives (:author and :year) if biblatex is not
available.

> If more than two different keys are needed in a single document, use of
> custom links or raw LaTeX would then be unavoidable. OTOH, this gives us
> very readable citations within the buffer in most cases.

Perhaps.  But then we are back at not having proper support across
backends—which might be unavoidable in any case.  Yet, aim high!

—Rasmus

-- 
Tack, ni svenska vakttorn. Med plutonium tvingar vi dansken på knä!

Re: Citations, continued

[Nicolas Goaziou]

Rasmus <rasmus@gmx.us> writes:

> Not necessarily.  I could do:
>
>    (defun rasmus/gentive-citation (citation-element backend)
>           (case backend ...) ...)
>
>    (add-to-list 'org-cite-types 'rasmus/gentive-citation )
>
> E.g. for genitive citations such as "Smith's (1984) model", which can be
> mostly emulated using primitives (:author and :year) if biblatex is not
> available.

As explained in another message, I'd like to avoid custom citations
(portability issues).

>> If more than two different keys are needed in a single document, use of
>> custom links or raw LaTeX would then be unavoidable. OTOH, this gives us
>> very readable citations within the buffer in most cases.
>
> Perhaps.  But then we are back at not having proper support across
> backends—which might be unavoidable in any case.  Yet, aim high!

Obtaining reasonable support for citations in every major back-end is
already aiming high in my book.

Anyway, I'm fine with having more powerful citation syntax for LaTeX,
but let's keep the syntax as simple as possible. As I once was told, Org
is not a programming language.


Regards,

Re: Citations, continued

[Thomas S. Dye]

Aloha Rasmus,

Rasmus <rasmus@gmx.us> writes:

>      In any given document, do you typically need more than two types of
>      citations, i.e. {citet, citep} OR {textcite, parentcite}?

Yes, I typically use what I call a multicite to get multiple citations
with biblatex.  It just inserts {key}.  I precede two or more of these
with a placeholder--π for parencites, † for textcites, or ƒ for
footcites--and then use a filter to insert \parencites, etc.  This is
crude, and wouldn't work in a more general setting, but it serves for
the work I do.

BTW, parencites et al. were a major step forward in biblatex, IMO.  Note
that CSL doesn't yet support multiple citations.

hth,
Tom

-- 
Thomas S. Dye
http://www.tsdye.com

Re: Citations, continued

[Nicolas Goaziou]

tsd@tsdye.com (Thomas S. Dye) writes:

> Yes, I typically use what I call a multicite to get multiple citations
> with biblatex.  It just inserts {key}.  I precede two or more of these
> with a placeholder--π for parencites, † for textcites, or ƒ for
> footcites--and then use a filter to insert \parencites, etc.  This is
> crude, and wouldn't work in a more general setting, but it serves for
> the work I do.

If you write something like

  [cite: pre1 @k1 post1; pre2 @k2 post2]

wouldn't it possible to guess you want to use multicite? IOW, does using
"multicite" really implies a change in the syntax?


Regards,

Re: Citations, continued

[Rasmus]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> If you write something like
>
>   [cite: pre1 @k1 post1; pre2 @k2 post2]
>
> wouldn't it possible to guess you want to use multicite? IOW, does using
> "multicite" really implies a change in the syntax?

To fully support multicite you need keyless citations in the beginning and
the end of the list, e.g.

    [cite: common pre; pre1 @k1 post1; pre2 @k2 post2; common post]

Of course, for textcite it's easy to replicate wihtout.  For parencites
and footcites, less so.

—Rasmus

-- 
May contains speling mistake

Re: Citations, continued

[Nicolas Goaziou]

Rasmus <rasmus@gmx.us> writes:

> Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:
>
>> If you write something like
>>
>>   [cite: pre1 @k1 post1; pre2 @k2 post2]
>>
>> wouldn't it possible to guess you want to use multicite? IOW, does using
>> "multicite" really implies a change in the syntax?
>
> To fully support multicite you need keyless citations in the beginning and
> the end of the list, e.g.
>
>     [cite: common pre; pre1 @k1 post1; pre2 @k2 post2; common post]
>
> Of course, for textcite it's easy to replicate wihtout.  For parencites
> and footcites, less so.

I think

  [cite: common pre; pre1 @k1 post1; pre2 @k2 post2; common post]

is regular and readable enough. So, there's no need to add special
support for multicite.


Regards,

Re: Citations, continued

[Rasmus]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> I think
>
>   [cite: common pre; pre1 @k1 post1; pre2 @k2 post2; common post]
>
> is regular and readable enough. So, there's no need to add special
> support for multicite.

Definitely.

In latex you'd just use multicite whenever more than two keys within one
cite-group.

     [cite: common pre; pre1 @k1 post1; pre2 @k2 post2; common post]
   → \textcites(common pre)(common post)[pre1][post1]{k1}[pre2][post2]{k2}
             ^^^ 
And:

   [cite: @k1; k2] → \textcites()()[][]{k1}[][]{k2}
                  or \textcites{k1}{k2}

—Rasmus

-- 
Lasciate ogni speranza o voi che entrate: siete nella mani di'machellaio

Re: Citations, continued

[Thomas S. Dye]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> tsd@tsdye.com (Thomas S. Dye) writes:
>
>> Yes, I typically use what I call a multicite to get multiple citations
>> with biblatex.  It just inserts {key}.  I precede two or more of these
>> with a placeholder--π for parencites, † for textcites, or ƒ for
>> footcites--and then use a filter to insert \parencites, etc.  This is
>> crude, and wouldn't work in a more general setting, but it serves for
>> the work I do.
>
> If you write something like
>
>   [cite: pre1 @k1 post1; pre2 @k2 post2]
>
> wouldn't it possible to guess you want to use multicite? IOW, does using
> "multicite" really implies a change in the syntax?
>
>
> Regards,

Yes, but it doesn't capture the citation command, AFAICT.

Something like

[cite: pre1 @k1 post1; pre2 @k2 post2 :command paren|text|foot]

would do.

All the best,
Tom

-- 
T.S. Dye & Colleagues, Archaeologists
735 Bishop St, Suite 315, Honolulu, HI 96813
Tel: 808-529-0866, Fax: 808-529-0884
http://www.tsdye.com

Re: Citations, continued

[John Kitchin]

Rasmus writes:
> The question:
>
>      In any given document, do you typically need more than two types of
>      citations, i.e. {citet, citep} OR {textcite, parentcite}?
>
> I do use other citation types, in particular a genitive version of
> textcite, but not very often.  That was why I initially wanted something
> like this:

We often use cite, citenum in one document. I don't use citeauthor, but
have manually written something like "Doe et al said... cite:Doe99",
which could be written with a citeauthor like command I think. There are
similar use cases for a citeyear command. It is not too far fetched to
see these four types in a document, in addition to a citep command.

>
> simple  inline:         @KEY
> complex inline:         [PRE @KEY POST :key VAL]
>         parent:         (PRE @KEY POST :key VAL)
>
> Where :type was the only key I was clever enough to think about (heavily
> biased by LaTeX).  Then you would be able to put in the top of your
> document what "inline" and that "parent" means.  Note, as Nicolas rightly
> pointed out that (·) shouldn't be used for syntax, so the above is to
> understand needs.  In any case, if you, or Eric, or anybody else for that
> matter, often rely on much more than two types of citations in any given
> document perhaps this is better:
>
>       [TYPE: PRE @KEY POST :key VAL]
>
> :key VAL may not be needed at all (but e.g. the new cool
> \textcites()()[][]{} commands have even more arguments).  That's
> essentially the "generalized link" you were talking about earlier.
>
> Display in the buffer is indeed something that should be tackled, but this
> is an issue of overlays, like entities, and not the immediate issue.  But
> I have it in mind as well.  It's tough though, when you have other
> citations types than author-year.
>
> Cheers,
> Rasmus

--
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu

Re: Citations, continued

[Richard Lawrence]

Hi Tom and all,

tsd@tsdye.com (Thomas S. Dye) writes:

> IIUC, Org mode citation syntax needs to capture four pieces of
> information for an *individual* citation: a =key= into one or more
> stores of bibliographic information; a =citation-command= that is
> understood by the =citation-style= specified for the document; a
> =pre-note= of arbitrary text in any language; and a =post-note= of
> arbitrary text in any language.  At least, this is how the LaTeX world
> accommodates the almost unconstrained and ever-growing variability in
> bibliographic styles in the wild.

I think the key, pre-note and post-note are common ground, and everyone
agrees that they need to be represented in a citation syntax.

I am a bit more hesitant about the citation command, though.  My reasons
for hestitation are the same as other people have expressed here, but I
think it might be useful to summarize.

As Tom notes, there is basically unlimited complexity in the LaTeX world
for citation commands.  There's a different subset of commands for every
distinction you might want to express in a given style.

The trouble is that there is close to zero support for this complexity
in the other export formats that Org supports.  As far as I am aware
(though I could be wrong), HTML, ODT, and other document formats barely
even have a way of indicating that a piece of text is a citation.
(HTML5 does have a "cite" element, but it seems to be basically an
unconstrained semantic element.)

Many of us (including me) are exporting to LaTeX, and if that is what
you are doing, it is natural to want to have access to all the
distinctions that LaTeX provides.  But if we try to capture those
distinctions in a high-level Org citation syntax, then we have two
problems: (a) finding the right syntax to express those distinctions,
and (b) exporting citations that make use of those distinctions to
non-LaTeX formats, which provide little support for them.  The more
distinctions we try to capture, the more difficult these problems will
be to solve.

Thus, I think the right question to ask is: which distinctions are both
*simple enough* and *important enough* that they are worth encoding in
Org syntax and supporting in non-LaTeX backends?  I think that is the
right place to draw the line between features of citations that are
encoded in `citation syntax proper' vs. `escape hatches' that give more
information about how to format a citation in a particular backend.

My sense is that a lot of the complexity in LaTeX citations should fall
in the latter category, but we need to think more about what falls in
the first category.

One clear candidate for the first category that I think everyone agrees
on is the distinction between in-text vs. parenthetical citations
(\citet vs. \citep and similar in LaTeX).  For my own case, this is the
only distinction I need to be directly expressed in syntax.

Nicolas expressed a similar opinion:

> It seems to me that :type is a LaTeX-only feature and, as such, should
> be handled in "ox-latex". In the general case, I think that Org should
> only support inline and parenthesized citations.

But in response to a question from Rasmus, Tom also suggested that
multi-cites are a candidate, in addition to the in-text/parenthetical
distinction:

> Yes, I typically use what I call a multicite to get multiple citations
> with biblatex.

I myself never use these, so I don't understand exactly what would be
required to support them properly in non-LaTeX backends, and I can't
speak to which category they belong to. 

Are there other candidates we should discuss?  What other distinctions
are so important that they are worth figuring out how to express in
syntax that can be exported by all backends?

As for supporting the `escape hatch' category, it seems that more
thought is needed here, too.  Right now, the #+ATTR_BACKEND syntax is
the only way I know of to specify arbitrary export properties for a
piece of Org syntax.  But it only applies to greater elements, and Tom
at least does not think it is sufficient to only allow specifying the
citation command, in particular, in an out-of-line (because greater
element) citation definition.

So maybe we need some kind of inline syntax for backend-specific
properties of citations, perhaps along the lines that Rasmus has
suggested.  I think this could be a good idea; my only concern is that
we make a clear separation between this syntax and the main syntax for
citations.  There are two reasons for this: if we don't clearly make
this separation, then (1) it becomes much harder to figure out and agree
on which distinctions should be expressed in `the' citation syntax; and
(2) there is a danger that the complexity of backend-specific properties
will bleed over into the main citation syntax, which all backends have
to support.

Best,
Richard

Re: Citations, continued

[Rasmus]

Richard Lawrence <richard.lawrence@berkeley.edu> writes:

> tsd@tsdye.com (Thomas S. Dye) writes:
>
>> IIUC, Org mode citation syntax needs to capture four pieces of
>> information for an *individual* citation: a =key= into one or more
>> stores of bibliographic information; a =citation-command= that is
>> understood by the =citation-style= specified for the document; a
>> =pre-note= of arbitrary text in any language; and a =post-note= of
>> arbitrary text in any language.  At least, this is how the LaTeX world
>> accommodates the almost unconstrained and ever-growing variability in
>> bibliographic styles in the wild.
>
> I think the key, pre-note and post-note are common ground, and everyone
> agrees that they need to be represented in a citation syntax.

False.  There has been discussions as to whether prenote should be
included for inline citations.

> Thus, I think the right question to ask is: which distinctions are both
> *simple enough* and *important enough* that they are worth encoding in
> Org syntax and supporting in non-LaTeX backends?  I think that is the
> right place to draw the line between features of citations that are
> encoded in `citation syntax proper' vs. `escape hatches' that give more
> information about how to format a citation in a particular backend.

Ideally we find TOOL that can handle this.  Worse case: bibtex.el, but
hopefully something less bare-bone, that knows about styles would be
great.  E.g. "Zotero for html" or similar.

> My sense is that a lot of the complexity in LaTeX citations should fall
> in the latter category, but we need to think more about what falls in
> the first category.

I don't know.  If you think of a type as a receipt it makes sense to allow
it to some extend, I guess.  Most LaTeX "receipts" very easy to use 'cause
somebody took care of the details.

> But in response to a question from Rasmus, Tom also suggested that
> multi-cites are a candidate, in addition to the in-text/parenthetical
> distinction:

Multicite is pretty easy.  A couple of days ago I showed that you can even
do it with the current link syntax.

Examples:

\textcites(pre-both)(post-both)[pre1][post1]{bohringer14}[pre2][post2]{davis14}
→ pre-both Böhringer et al. (pre1 2014, post1) and Davis and Schiraldi (pre2
2014, post2), post-both

\parencites(pre-both)(post-both)[pre1][post1]{bohringer14}[pre2][post2]{davis14} 
→ (pre-both pre1 Böhringer et al. 2014, post1; pre2 Davis and Schiraldi 2014, post2, post-both)

But multicite is surely a ox-latex feature, but it's just a convenience
wrapper around normal cite commands which can be constructed using
primitives, namely :author, :year :pre and :post.  You could imagine
something like

  [cite: pre-both; pre1 @k1 post1; pre2 @k2 post2; post-both]

Which could be simplified to

  [cite: pre-both pre1 @k1 post1; pre2 @k2 post2 post-both]


> As for supporting the `escape hatch' category, it seems that more
> thought is needed here, too.  Right now, the #+ATTR_BACKEND syntax is
> the only way I know of to specify arbitrary export properties for a
> piece of Org syntax.

From a user perspective, links take a backend argument.

> So maybe we need some kind of inline syntax for backend-specific
> properties of citations, perhaps along the lines that Rasmus has
> suggested.  I think this could be a good idea; my only concern is that
> we make a clear separation between this syntax and the main syntax for
> citations.  There are two reasons for this: if we don't clearly make
> this separation, then (1) it becomes much harder to figure out and agree
> on which distinctions should be expressed in `the' citation syntax; and
> (2) there is a danger that the complexity of backend-specific properties
> will bleed over into the main citation syntax, which all backends have
> to support.

Those are indeed valid concerns.  One reason why I like :type is that all
complexity is hidden away somewhere else.  Much like links.  This may not
be a good thing.  I'm heating up on (my interpretation) of Nicolas
idea:

[cite: pre-both; pre1 @k1 post1; pre2 @k2 post2; post-both]

It's still an improvement (though small) over links, and you might still
get one "free" type which can be expressed as [@k].

—Rasmus

-- 
What will be next?

Re: Citations, continued

[John Kitchin]

Richard Lawrence writes:

> Hi Tom and all,
>
> tsd@tsdye.com (Thomas S. Dye) writes:
>
>> IIUC, Org mode citation syntax needs to capture four pieces of
>> information for an *individual* citation: a =key= into one or more
>> stores of bibliographic information; a =citation-command= that is
>> understood by the =citation-style= specified for the document; a
>> =pre-note= of arbitrary text in any language; and a =post-note= of
>> arbitrary text in any language.  At least, this is how the LaTeX world
>> accommodates the almost unconstrained and ever-growing variability in
>> bibliographic styles in the wild.
>
> I think the key, pre-note and post-note are common ground, and everyone
> agrees that they need to be represented in a citation syntax.
>
> I am a bit more hesitant about the citation command, though.  My reasons
> for hestitation are the same as other people have expressed here, but I
> think it might be useful to summarize.

I think the critical point is that the syntax must be user
extendable. It should be possible to add these different types, even if
most people do not use them. Otherwise, links will continue to be used
anyway.

>
> As Tom notes, there is basically unlimited complexity in the LaTeX world
> for citation commands.  There's a different subset of commands for every
> distinction you might want to express in a given style.
>
> The trouble is that there is close to zero support for this complexity
> in the other export formats that Org supports.  As far as I am aware
> (though I could be wrong), HTML, ODT, and other document formats barely
> even have a way of indicating that a piece of text is a citation.
> (HTML5 does have a "cite" element, but it seems to be basically an
> unconstrained semantic element.)

This should not prevent us from making a syntax that can be simple for
these backends, and functional for Latex.

>
> Many of us (including me) are exporting to LaTeX, and if that is what
> you are doing, it is natural to want to have access to all the
> distinctions that LaTeX provides.  But if we try to capture those
> distinctions in a high-level Org citation syntax, then we have two
> problems: (a) finding the right syntax to express those distinctions,
> and (b) exporting citations that make use of those distinctions to
> non-LaTeX formats, which provide little support for them.  The more
> distinctions we try to capture, the more difficult these problems will
> be to solve.

Again, user extendability is the key to this. You would not use these
for export to other formats, and you do not want to have to use one
notation for some backends, and another for latex. One also does not use
the sophisticated machinery of latex by accident. So I don't think we
lose anything by making it possible.

>
> Thus, I think the right question to ask is: which distinctions are both
> *simple enough* and *important enough* that they are worth encoding in
> Org syntax and supporting in non-LaTeX backends?  I think that is the
> right place to draw the line between features of citations that are
> encoded in `citation syntax proper' vs. `escape hatches' that give more
> information about how to format a citation in a particular backend.

I think the new citations should support an export mechanism like links
do. Each backend whould be responsible to take the information it gets,
and use it to create what it needs. Syntax like [see cite@Doe99 for
example] contains all that information, and is pretty readable.

>
> My sense is that a lot of the complexity in LaTeX citations should fall
> in the latter category, but we need to think more about what falls in
> the first category.
>
> One clear candidate for the first category that I think everyone agrees
> on is the distinction between in-text vs. parenthetical citations
> (\citet vs. \citep and similar in LaTeX).  For my own case, this is the
> only distinction I need to be directly expressed in syntax.

If these are the default types, I think it is fine. It should be
possible to define new types though. It should be possible to define
this: [citeauthor@Doe99] showed in [citeyear@Doe99] that this was
possible. An alternative approach is illustrated in
Ref. [citenum@Doe100].

>
> Nicolas expressed a similar opinion:
>
>> It seems to me that :type is a LaTeX-only feature and, as such, should
>> be handled in "ox-latex". In the general case, I think that Org should
>> only support inline and parenthesized citations.

I don't think it is a latex only feature. Word could have this feature
too where you differentiate between types, and html, and ascii,
etc... This is handled by the backend.

>
> But in response to a question from Rasmus, Tom also suggested that
> multi-cites are a candidate, in addition to the in-text/parenthetical
> distinction:
>
>> Yes, I typically use what I call a multicite to get multiple citations
>> with biblatex.

If a multicite is something that might render as [1-3] or [1,6,9] in a
backend then yes, multicite is a necessary capability for most
scientific publications.

These could look like (I am mixing :/@ below just to see what it looks like):

[cite:key1,key2] or [cite:key1; cite:key2]

[cite@key1,key2] or [cite@key1; cite@key2]

[cite@key1; see also cite@key2]

>
> I myself never use these, so I don't understand exactly what would be
> required to support them properly in non-LaTeX backends, and I can't
> speak to which category they belong to.
>
> Are there other candidates we should discuss?  What other distinctions
> are so important that they are worth figuring out how to express in
> syntax that can be exported by all backends?

I think the examples above cover a lot of the use cases we have
considered. Maybe there is some subtle issues of a citation in a
citation, or of other markup in a citation, e.g. bold/italics, that I
overlooked.

I am pretty sure all those formats can be integrated pretty easily into
something like org-ref for inserting citations. As long as some part of
them is clickable, with a user-defined action, e.g. the cite@key part,
then functionally this would be the same as what we have with links in
org-ref.

converting multicites this way to latex might be a little tricky, but I
think it is doable.

>
> As for supporting the `escape hatch' category, it seems that more
> thought is needed here, too.  Right now, the #+ATTR_BACKEND syntax is
> the only way I know of to specify arbitrary export properties for a
> piece of Org syntax.  But it only applies to greater elements, and Tom
> at least does not think it is sufficient to only allow specifying the
> citation command, in particular, in an out-of-line (because greater
> element) citation definition.
>
> So maybe we need some kind of inline syntax for backend-specific
> properties of citations, perhaps along the lines that Rasmus has
> suggested.  I think this could be a good idea; my only concern is that
> we make a clear separation between this syntax and the main syntax for
> citations.  There are two reasons for this: if we don't clearly make
> this separation, then (1) it becomes much harder to figure out and agree
> on which distinctions should be expressed in `the' citation syntax; and
> (2) there is a danger that the complexity of backend-specific properties
> will bleed over into the main citation syntax, which all backends have
> to support.
>
> Best,
> Richard

--
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu

Re: Citations, continued

[Richard Lawrence]

Hi John and all,

John Kitchin <jkitchin@andrew.cmu.edu> writes:

> I think the critical point is that the syntax must be user
> extendable. It should be possible to add these different types, even if
> most people do not use them. Otherwise, links will continue to be used
> anyway.

I completely agree.  Some form of user extensibility is needed, not only
because our existing needs are complex and diverse, but because it's
hard for each of us to know how those needs will change in the future. 

My concern is just that we clearly distinguish the `main' or `proper'
citation syntax from the user-extensible part, as I said here:

>> So maybe we need some kind of inline syntax for backend-specific
>> properties of citations, perhaps along the lines that Rasmus has
>> suggested.  I think this could be a good idea; my only concern is that
>> we make a clear separation between this syntax and the main syntax for
>> citations.  There are two reasons for this: if we don't clearly make
>> this separation, then (1) it becomes much harder to figure out and agree
>> on which distinctions should be expressed in `the' citation syntax; and
>> (2) there is a danger that the complexity of backend-specific properties
>> will bleed over into the main citation syntax, which all backends have
>> to support.

To put it a different way, I think the main citation syntax should
express all and only those features that are important enough that they
should be supported `out of the box' -- i.e., without any user extension
or customization -- for all backends.

Anything else should be possible to express via whatever syntax we
decide on for user extensions, but I think there should be a really
clear line between the two.     

> I think the new citations should support an export mechanism like links
> do. Each backend whould be responsible to take the information it gets,
> and use it to create what it needs. Syntax like [see cite@Doe99 for
> example] contains all that information, and is pretty readable.

> It should be possible to define new types though. It should be
> possible to define this: [citeauthor@Doe99] showed in [citeyear@Doe99]
> that this was possible. An alternative approach is illustrated in
> Ref. [citenum@Doe100].

Rasmus has also expressed support for something like this, and I can see
that it is important for a user to be able to define types which are
backend-independent (and can thus be exported differently for different
backends by some user-supplied function), much like links.    

On the other hand, before we adopt such a facility, it's important to
think about what the interface would look like.  What can reasonably be
expected of the user function?  What information needs to be given to it?
(Just the citation and its properties? a reference to the bibligraphy
file? a data structure representing the referenced work?)  What happens
when the user doesn't define behavior for a particular backend?

I don't quite like the examples you have illustrated here because they
don't make the distinction I was urging above, between the `main'
citation syntax and the user-extensible part.  As a result it's really
unclear to an author/editor, when reading the document, which backends
can be expected to support which citations, and which citations can be
expected to break if all you have is a default setup.  (Imagine you
didn't write the document or the custom citation type!) From the
exporter's perspective, it's also really unclear what should happen when
there's no behavior defined for, say, the "citeauthor" type for the HTML
backend.  If it defaults to just a "normal" citation, the output is very
likely to be unreadable in the general case, but what other option is
there?

It's easy (and correct) to say: "These are the user's problems."  But
we're all users, so let's think some more about how to make things easy
for ourselves. :)

For these reasons, I'd prefer something like Rasmus' suggestion; maybe
syntax like

[@Doe99 :custom-type author-only] showed in [@Doe99 :custom-type year-only]
that this was possible. An alternative approach is illustrated in
[@Doe :custom-type refnum].

where, basically, ":custom-type" is a nice big flag that says: "Here be
dragons: you are responsible for exporting this citation yourself."

> If a multicite is something that might render as [1-3] or [1,6,9] in a
> backend then yes, multicite is a necessary capability for most
> scientific publications.

Can you (or Tom, or someone else) make the case that it is important
enough to have multicites that non-LaTeX backends should support them
out of the box?  (I'm not doubting it, I just don't have any idea why
since I don't use them myself.)

Best,
Richard

Re: Citations, continued

[John Kitchin]

Richard Lawrence writes:

>
> My concern is just that we clearly distinguish the `main' or `proper'
> citation syntax from the user-extensible part, as I said here:
>

> Rasmus has also expressed support for something like this, and I can see
> that it is important for a user to be able to define types which are
> backend-independent (and can thus be exported differently for different
> backends by some user-supplied function), much like links.
>
> On the other hand, before we adopt such a facility, it's important to
> think about what the interface would look like.  What can reasonably be
> expected of the user function?  What information needs to be given to it?
> (Just the citation and its properties? a reference to the bibligraphy
> file? a data structure representing the referenced work?)  What happens
> when the user doesn't define behavior for a particular backend?

The pieces of information we have converged on (I think) are key, pre
text, post text, command, and that could be packaged in some data
structure. You also need the backend, and there is a final piece that is
important and that is the bibliography database(s). In org-ref we use a
bibliography link or rely on a default variable setting if that is not
found. Another option is #+BIBLIOGRAPHY: information.

The user function should take that information, and provide the
formatted string to replace the citation. As with latex, the function
may have to run a few times, e.g. to get the citations, construct the
bibliography, and then go back through the text to put in the formatted
numbers, author-year, etc... in latex we have the convenience to let
latex/bibtex handle that, and other backends would need to duplicate at
least some of that capability.

This can also be used to specify where a bibliography should be placed
in the document.

For undefined behavior, the only thing that can happen is the the same
thing that happens with links. Nothing.


> I don't quite like the examples you have illustrated here because they
> don't make the distinction I was urging above, between the `main'
> citation syntax and the user-extensible part.  As a result it's really
> unclear to an author/editor, when reading the document, which backends
> can be expected to support which citations, and which citations can be
> expected to break if all you have is a default setup.  (Imagine you
> didn't write the document or the custom citation type!) From the
> exporter's perspective, it's also really unclear what should happen when
> there's no behavior defined for, say, the "citeauthor" type for the HTML
> backend.  If it defaults to just a "normal" citation, the output is very
> likely to be unreadable in the general case, but what other option is
> there?

That is true enough. That is already the case for every custom link
anyone makes in org-mode. the default behavior there is that the link is
dead and exports as plain text. This is also the case in Latex, if you
do not use the right package, sometimes you cannot get the right format
out.

>
> It's easy (and correct) to say: "These are the user's problems."  But
> we're all users, so let's think some more about how to make things easy
> for ourselves. :)
>
> For these reasons, I'd prefer something like Rasmus' suggestion; maybe
> syntax like
>
> [@Doe99 :custom-type author-only] showed in [@Doe99 :custom-type year-only]
> that this was possible. An alternative approach is illustrated in
> [@Doe :custom-type refnum].
>
> where, basically, ":custom-type" is a nice big flag that says: "Here be
> dragons: you are responsible for exporting this citation yourself."
>
>> If a multicite is something that might render as [1-3] or [1,6,9] in a
>> backend then yes, multicite is a necessary capability for most
>> scientific publications.
>
> Can you (or Tom, or someone else) make the case that it is important
> enough to have multicites that non-LaTeX backends should support them
> out of the box?  (I'm not doubting it, I just don't have any idea why
> since I don't use them myself.)

My case is that if this is not supported out of the box, we could not
use the citation syntax for any of our scientific publishing needs.  In
a recently published paper we had four citations with 4 references in
it, four with three references, 10 with two references and 15 single
citations. That is probably typical in our work, and in review work it
is common to have even more than four references in a citation. This is
a feature of modern publications because of tools like Endnote, zotero,
etc... which make it trivial to add citations, and independent of the
ultimate format (which is usually pdf, but sometimes html, and sometimes
Word).

If the new syntax allows us to handle citations the way links do today,
I can see a transition pathway to the new syntax. If not, I doubt it can
be adopted for scientific writing if you need two syntaxes for
citations. There are other, important types of writing this may support,
and maybe it makes sense to have separate approaches.


>
> Best,
> Richard

--
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu

Re: Citations, continued

[Thomas S. Dye]

John Kitchin <jkitchin@andrew.cmu.edu> writes:

>> Can you (or Tom, or someone else) make the case that it is important
>> enough to have multicites that non-LaTeX backends should support them
>> out of the box?  (I'm not doubting it, I just don't have any idea why
>> since I don't use them myself.)
>
> My case is that if this is not supported out of the box, we could not
> use the citation syntax for any of our scientific publishing needs.  In
> a recently published paper we had four citations with 4 references in
> it, four with three references, 10 with two references and 15 single
> citations. That is probably typical in our work, and in review work it
> is common to have even more than four references in a citation. This is
> a feature of modern publications because of tools like Endnote, zotero,
> etc... which make it trivial to add citations, and independent of the
> ultimate format (which is usually pdf, but sometimes html, and sometimes
> Word).
>
> If the new syntax allows us to handle citations the way links do today,
> I can see a transition pathway to the new syntax. If not, I doubt it can
> be adopted for scientific writing if you need two syntaxes for
> citations. There are other, important types of writing this may support,
> and maybe it makes sense to have separate approaches.

My experience echoes John's--many citations in my archaeology work refer
to multiple objects.  However, this was possible in bibtex.  The problem
solved by multicites in biblatex is that they make it possible to
associate a pre-note and a post-note to each object in a citation with
multiple objects.  This is often useful in author-date styles, but it is
absolutely essential in the footnote styles used in the humanities.  In
history, for example, citation practice differs substantially from the
sciences.  Scientists tend to use fine-grained citations to support
specific statements.  Historians tend to make a paragraph-long argument
which ends in a footnote that might look something like this "1. For the
popularity of pork in eighteenth century England see Foo 1923, Bar 1946,
and references therein; animal husbandry in Oceania is discussed at
length by Baz 1963."

I don't know what word-processing (document preparation) software is
prevalent in history or the humanities generally, though my guess is
Word, not LaTeX.

All the best,
Tom

-- 
Thomas S. Dye
http://www.tsdye.com

Re: Citations, continued

[Nicolas Goaziou]

Richard Lawrence <richard.lawrence@berkeley.edu> writes:

> On Sat, Feb 7, 2015 at 2:43 PM, Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:
>
>> What about the following set?
>>
>>  bold code entity italic latex-fragment line-break strike-through
>>  subscript superscript underline superscript
>
> That would work fine for me in prefixes and suffixes.

Fair enough.

> I guess I could live with this, but to be honest, I much prefer the
> Pandoc way.

Actually, there is another, shorter possibility:

  - in-text citation 

      [KEY] or [KEY suffix]

    [@item1] or [@item1 p. 30] or [@item1 p. 30, with suffix]

  - out-text citation 

      [cite: prefix? key suffix?; prefix2? key2 suffix2? ...]

    [cite: see @item1 p. 34-35] or [cite: @item1 pp. 33, 35-37, and
    nowhere else] or [cite: -@item1 p. 44] or even [cite: see @item1 p.
    34-35; also @item3 chap. 3]

IMO it is quite readable.

> The Pandoc syntax has a nice congruence between the source file and
> the output: if a cite key is inside the brackets in the source, the
> reference is inside the brackets in the output, and if it's outside in
> the source, it's outside in the output.  This convention seems
> natural, easy to remember, and very readable -- at least if, like me
> (and I would guess many others), you use author names in cite keys.

This can be partly (i.e. visually) solved with overlays, e.g., you write
[@item1: p. 30] and you see @item1 (p. 30) in the buffer. IMO, this is
overkill, though.

> So as an author, I prefer the Pandoc way, but I understand there are
> other considerations.  If we must have the tag for performance
> reasons, I would prefer using two different tags to represent the two
> cases; I suggest borrowing (from LaTeX's natbib package) "citet" for
> in-text and "citep" for bracketed citations , but I don't really care
> as long as they're easy to type, and it's easy to change one to the
> other.

IMO, my current proposal is clearer.

Also, you can switch to in-text to out-text (or the other way) just by
adding (or removing) cite: at the beginning of the citation.


Regards,

Re: Citations, continued

[Richard Lawrence]

Hi Nicolas and all,

On Sun, Feb 8, 2015 at 1:58 AM, Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:
>
>   - in-text citation
>
>       [KEY] or [KEY suffix]
>
>     [@item1] or [@item1 p. 30] or [@item1 p. 30, with suffix]
>
>   - out-text citation
>
>       [cite: prefix? key suffix?; prefix2? key2 suffix2? ...]
>
>     [cite: see @item1 p. 34-35] or [cite: @item1 pp. 33, 35-37, and
>     nowhere else] or [cite: -@item1 p. 44] or even [cite: see @item1 p.
>     34-35; also @item3 chap. 3]
>
> IMO it is quite readable.

Yes, I like this better than your first proposal, or the citet/citep approach.

But doesn't the first case pose the same performance problems as the
Pandoc syntax, since it does away with the tag?  Or are you thinking
it's easier to parse because the key occurs immediately after the
bracket, without a prefix...?

Best,
Richard

Re: Citations, continued

[Nicolas Goaziou]

Richard Lawrence <richard.lawrence@berkeley.edu> writes:

> But doesn't the first case pose the same performance problems as the
> Pandoc syntax, since it does away with the tag?  Or are you thinking
> it's easier to parse because the key occurs immediately after the
> bracket, without a prefix...?

Correct. It is easier to dismiss false positives.

Regards,

Re: Citations, continued

[Rasmus]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Using the example from Erik Hetzner in the same thread, what about:
>
>   1. [cite:@item1] says blah.
>   2. [cite:@item1: p. 30] says blah.

Why is "p." stripped here?

>   3. [cite:@item1: p. 30, with suffix] says blah.
>   4. [cite:@item1: -@item2 p. 30; see also @item3] says blah.

If item{1,2} have the same author biblatex[-chicago?] is smart enough to
compress it to "author (year1, year2)". So this example seems like a
downgrade if "-" is required to get the suggested output.

>   5. A citation group [cite:: see @item1 p. 34-35; also @item3 chap. 3].

Why is chap. *not* stripped here?

>   6. Another one [cite::see @item1 p. 34-35].
>   7. Citation with a suffix and locator [cite:: @item1 pp. 33, 35-37, and nowhere else].

Where does suffix and locator end here.  E.g. what is the output of

     [cite:: @item1 33, pp. 35-37, and nowhere else].

>   9. Citation with suffix only [cite:: @item1 and nowhere else].

How do I know this is a suffix?  Is locator a regexp like 
    \`[p\.0-9 ]+? 

What is [cite:@K s. 12] or [cite:@K side.? 12]?

>   10. Like a citation without author: [cite:: -@item1], and now Doe
>       with a locator [cite:: -@item2 p. 44].

>   2.   Doe (2005, 30) says blah.
>   5.   A citation group (see Doe 2005, 34–35; also Doe and Roe 2007, chap. 3).
>   7.   Citation with a suffix and locator (Doe 2005, 33, 35–37, and nowhere else).


> Note that space after the second colon is not mandatory. More
> explicitly, syntax would be either
>
>   [cite:IN-TEXT-KEY]
>
> or
>
>   [cite:IN-TEXT-KEY?:SPACE* CITATIONS]
>
> where CITATIONS is any number of
>
>   PREFIX? KEY SUFFIX? 
>
> separated with semi-colons.

What if I need several text cite keys.  Say @K{1,2} is the same author A,
and @K3 is B.  Then  [cite:@K1,@K2,@K3] should/could be something like 
A (Y1, Y2), and B (Y3).  How do I express this?

Some comments.

  1. Am I supposed to distinguish between a text citations and parenthesis
     citation based on a single ":"?  That's hard.  Why not distinguish
     based on the initial label?  E.g. {textcite, parentcite} or {citet,
     citep}.

  2. The idea of locator /and/ suffix is confusing.  The fact that your
     examples suggest seemingly random dropping of data from locator makes
     me want to avoid it even more.  It's a 'can of worms' to use a
     frequently emerging expression from this list.

  3. This is almost full circle.  The proposal above seems no better (and
     IMO worse) than e.g. the generalized links that Tom suggested, e.g.
     [TYPE: KEY :pre PRE :post SUF] or [TYPE: PRE @KEY POST]. 
     Or [[TYPE: KEY :pre PRE :post SUF]] or [[TYPE: PRE @KEY POST]].

  4. The reason for suggested syntax seems to be support some benchmark
     like A1 (Y1, Y2 also A2, Y3).  I have never come across a nested
     citation like this.  I have seen either (A1 Y1, Y2 and A2, Y3) or 
     A1 (Y1, Y2) and A2 (Y3). . .

  5. . . . Yet I still don't know how to get A1 (PRE Y2) with the above.
     Is the benchmark correct?

If parsing speed is key here I think that
[citet: pre1 @k1 post1; pre2 @k2 post2] and [citep: pre1 @k1 post1; pre2 @k2 post2]
are clearer solutions.  But this is clearly closer to a LaTeX than pandoc.

—Rasmus

-- 
Dung makes an excellent fertilizer

Re: Citations, continued

[Nicolas Goaziou]

Rasmus <rasmus@gmx.us> writes:

> Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:
>
>> Using the example from Erik Hetzner in the same thread, what about:
>>
>>   1. [cite:@item1] says blah.
>>   2. [cite:@item1: p. 30] says blah.
>
> Why is "p." stripped here?

I don't understand. Anyway, I now suggest

  [@item1] and [@item1 p. 30]

>>   3. [cite:@item1: p. 30, with suffix] says blah.
>>   4. [cite:@item1: -@item2 p. 30; see also @item3] says blah.
>
> If item{1,2} have the same author biblatex[-chicago?] is smart enough to
> compress it to "author (year1, year2)". So this example seems like a
> downgrade if "-" is required to get the suggested output.

  [@item1 -@item2 p. 30]

Downgrade is a bit strong.

>>   5. A citation group [cite:: see @item1 p. 34-35; also @item3 chap. 3].
>
> Why is chap. *not* stripped here?

I do not understand either.

> Where does suffix and locator end here.  E.g. what is the output of
>
>      [cite:: @item1 33, pp. 35-37, and nowhere else].

  [cite: @item1 pp. 33, 35-37, and nowhere else]

suffix and locator are merged (AFAIU, in practice, there is no
distinction between locator and suffix): "pp. 33, 35-37, and nowehere
else".

>>   9. Citation with suffix only [cite:: @item1 and nowhere else].
>
> How do I know this is a suffix?  Is locator a regexp like 
>     \`[p\.0-9 ]+?

See above.

> What is [cite:@K s. 12] or [cite:@K side.? 12]?

See above.

> What if I need several text cite keys.  Say @K{1,2} is the same author A,
> and @K3 is B.  Then  [cite:@K1,@K2,@K3] should/could be something like 
> A (Y1, Y2), and B (Y3).  How do I express this?

Since A and B do not appear in the same parenthesis, two citations are
needed:

  [@K1 -@K2], and [@K3]

> Some comments.
> 
>   1. Am I supposed to distinguish between a text citations and parenthesis
>      citation based on a single ":"?  That's hard.  Why not distinguish
>      based on the initial label?  E.g. {textcite, parentcite} or {citet,
>      citep}.

In fact, you're right, we don't need the colon, hence my other proposal.

>   2. The idea of locator /and/ suffix is confusing.  The fact that your
>      examples suggest seemingly random dropping of data from locator makes
>      me want to avoid it even more.  It's a 'can of worms' to use a
>      frequently emerging expression from this list.

Again, there's no real need to extract a locator. At least, not at the
parser level.

>   3. This is almost full circle.  The proposal above seems no better (and
>      IMO worse) than e.g. the generalized links that Tom suggested, e.g.
>      [TYPE: KEY :pre PRE :post SUF] or [TYPE: PRE @KEY POST]. 
>      Or [[TYPE: KEY :pre PRE :post SUF]] or [[TYPE: PRE @KEY POST]].

In [type: KEY :pre PRE :post SUF], PRE comes after KEY, which impedes
readability, IMO.

Double brackets are link syntax: there will be conflict if TYPE belongs
to `org-link-types'.

>   5. . . . Yet I still don't know how to get A1 (PRE Y2) with the above.
>      Is the benchmark correct?

You can't. Is this needed?

> If parsing speed is key here I think that
> [citet: pre1 @k1 post1; pre2 @k2 post2] and [citep: pre1 @k1 post1; pre2 @k2 post2]
> are clearer solutions.  But this is clearly closer to a LaTeX than
> pandoc.

If "A1 (PRE Y2)" is really needed, then yes, I think that's good enough.
Otherwise I think [@k1] is terse and nice.


Regards,

Re: Citations, continued

[Rasmus]

Hi,

Thanks for the quick reply.  A very colorful (in Gnus at least) reply
follows.

>>>   1. [cite:@item1] says blah.
>>>   2. [cite:@item1: p. 30] says blah.
>>
>> Why is "p." stripped here?
>
> I don't understand. Anyway, I now suggest

This is what I'm talking about:

>>>  2. [cite:@item1: p. 30] says blah.
>>> ...              ^^^^
>>>  2.   Doe (2005, 30) says blah.
>>>                ^^^


>>>   3. [cite:@item1: p. 30, with suffix] says blah.
>>>   4. [cite:@item1: -@item2 p. 30; see also @item3] says blah.
>>
>> If item{1,2} have the same author biblatex[-chicago?] is smart enough to
>> compress it to "author (year1, year2)". So this example seems like a
>> downgrade if "-" is required to get the suggested output.
>
>   [@item1 -@item2 p. 30]
>
> Downgrade is a bit strong.

If I have to think about the /formatting/ out output rather than the
/contents/ downgrade is not too strong (IMO).  In the example above, why
not [@item1 @item2 p. 30]?


Another issue is that it's not transpose-words safe.  E.g. this output
seems bad: [-@k1 @k2 30] => Y1 A2 (Y2, 30).

>>>   5. A citation group [cite:: see @item1 p. 34-35; also @item3 chap. 3].
>>
>> Why is chap. *not* stripped here?
>
> I do not understand either.

Compare to example 1 where p. is stripped.  Here chap. is /not/ stripped.
>>>  5.   A citation group (see Doe 2005, 34–35; also Doe and Roe 2007, chap. 3).


>> Where does suffix and locator end here.  E.g. what is the output of
>>
>>      [cite:: @item1 33, pp. 35-37, and nowhere else].
>
>   [cite: @item1 pp. 33, 35-37, and nowhere else]
>
> suffix and locator are merged (AFAIU, in practice, there is no
> distinction between locator and suffix): "pp. 33, 35-37, and nowehere
> else".

But in your previous examples data is stripped from the locator.  If
there's no difference I think it would be better to not talk about this
locator 'cause it's extremely confusing.

>>>   9. Citation with suffix only [cite:: @item1 and nowhere else].
>>
>> How do I know this is a suffix?  Is locator a regexp like 
>>     \`[p\.0-9 ]+?
>
> See above.
>> What is [cite:@K s. 12] or [cite:@K side.? 12]?
>
> See above.

See also above.  From you explanation I would guess that at least these
two examples are wrong.  Is that correct?

>>>  2. [cite:@item1: p. 30] says blah.
>>>  2.   Doe (2005, 30) says blah.
>>>  3. [cite:@item1: p. 30, with suffix] says blah.
>>>  3.   Doe (2005, 30, with suffix) says blah.


>> What if I need several text cite keys.  Say @K{1,2} is the same author A,
>> and @K3 is B.  Then  [cite:@K1,@K2,@K3] should/could be something like 
>> A (Y1, Y2), and B (Y3).  How do I express this?
>
> Since A and B do not appear in the same parenthesis, two citations are
> needed:
>
>   [@K1 -@K2], and [@K3]

This is a minor downgrade from biblatex.  The year thing is worse.

>> Some comments.
>> 
>>   1. Am I supposed to distinguish between a text citations and parenthesis
>>      citation based on a single ":"?  That's hard.  Why not distinguish
>>      based on the initial label?  E.g. {textcite, parentcite} or {citet,
>>      citep}.
>
> In fact, you're right, we don't need the colon, hence my other proposal.

This is much better.

>>   2. The idea of locator /and/ suffix is confusing.  The fact that your
>>      examples suggest seemingly random dropping of data from locator makes
>>      me want to avoid it even more.  It's a 'can of worms' to use a
>>      frequently emerging expression from this list.
>
> Again, there's no real need to extract a locator. At least, not at the
> parser level.

Let's stop talking about this locator then.  It appears nowhere else in
LaTeX or Org documentation.

>>   5. . . . Yet I still don't know how to get A1 (PRE Y2) with the above.
>>      Is the benchmark correct?
>
> You can't. Is this needed?

It's not unheard of.  I have used it in the past.  In LaTeX it's something like: 

    \citet[C]{k}    → A    (Y, C)
    \citet[B][]{k}  → A (B, Y)
    \citet[B][C]{k} → A (B, Y, C)

>> If parsing speed is key here I think that
>> [citet: pre1 @k1 post1; pre2 @k2 post2] and [citep: pre1 @k1 post1; pre2 @k2 post2]
>> are clearer solutions.  But this is clearly closer to a LaTeX than
>> pandoc.
>
> If "A1 (PRE Y2)" is really needed, then yes, I think that's good enough.
> Otherwise I think [@k1] is terse and nice.

It's nice.  @k1 / [pre @k1 post] for text and (pre @k1 post) for
parentheses expressions is nicer, but that's details.  I trust your
judgment on the technical merit of one idea versus the next.

Thanks,
Rasmus

-- 
El Rey ha muerto. ¡Larga vida al Rey!

Re: Citations, continued

[Nicolas Goaziou]

Rasmus <rasmus@gmx.us> writes:

>>>>  2. [cite:@item1: p. 30] says blah.
>>>> ...              ^^^^
>>>>  2.   Doe (2005, 30) says blah.
>>>>                ^^^

According to Erik, this is just a possible output. It belongs to export
configuration.

>>>>   3. [cite:@item1: p. 30, with suffix] says blah.
>>>>   4. [cite:@item1: -@item2 p. 30; see also @item3] says blah.
>>>
>>> If item{1,2} have the same author biblatex[-chicago?] is smart enough to
>>> compress it to "author (year1, year2)". So this example seems like a
>>> downgrade if "-" is required to get the suggested output.
>>
>>   [@item1 -@item2 p. 30]
>>
>> Downgrade is a bit strong.
>
> If I have to think about the /formatting/ out output rather than the
> /contents/ downgrade is not too strong (IMO).  In the example above, why
> not [@item1 @item2 p. 30]?

All back-end may not be smart enough to compress years. However, it is
probably equivalent for those that can.

> Another issue is that it's not transpose-words safe.  E.g. this output
> seems bad: [-@k1 @k2 30] => Y1 A2 (Y2, 30).

This citation is invalid. The point of [@k1] is that the author is
mandatory, since it is in-text, so [-@k1] doesn't make sense. 

Also, it is not possible (as in Pandoc) to extract only year from
a bibliography entry, so there's no way to have Y1 so far.

> But in your previous examples data is stripped from the locator.  If
> there's no difference I think it would be better to not talk about this
> locator 'cause it's extremely confusing.

Again, I was re-using examples. Let's forget about the "locator".

> See also above.  From you explanation I would guess that at least these
> two examples are wrong.  Is that correct?
>
>>>>  2. [cite:@item1: p. 30] says blah.
>>>>  2.   Doe (2005, 30) says blah.
>>>>  3. [cite:@item1: p. 30, with suffix] says blah.
>>>>  3.   Doe (2005, 30, with suffix) says blah.

They are not wrong. The output may decide to strip "p. ". This is not
our problem at this stage of the discussion.

>>> What if I need several text cite keys.  Say @K{1,2} is the same author A,
>>> and @K3 is B.  Then  [cite:@K1,@K2,@K3] should/could be something like 
>>> A (Y1, Y2), and B (Y3).  How do I express this?
>>
>> Since A and B do not appear in the same parenthesis, two citations are
>> needed:
>>
>>   [@K1 -@K2], and [@K3]
>
> This is a minor downgrade from biblatex.  The year thing is worse.

Org doesn't pretend to replace either LaTeX or biblatex. If extracting
data from an entry is required, then I suggest to extend key syntax,
e.g.:

  @K1:year

But, again, if this is a LaTeX-only feature, org-ref and even raw LaTeX
code is good enough. We shouldn't try to implement every possible
feature, or we will fail.

So, it is only an option if it can be implemented in most major export
back-ends.

> It's not unheard of.  I have used it in the past.  In LaTeX it's something like: 
>
>     \citet[C]{k}    → A    (Y, C)
>     \citet[B][]{k}  → A (B, Y)
>     \citet[B][C]{k} → A (B, Y, C)

I understand, but would it be needed to have both A (Y, C) and A (B, Y)
in the same document? Otherwise, [@k text] can be treated either as
A (text, Y) or A (Y, text) consistently throughout the document,
according to bibliographic style.

In any case, if we allow @key:tag syntax, then it is possible to do

  [@k:author] [cite: B -@k C]

and get

  A (B, Y, C)

> It's nice. @k1 / [pre @k1 post] for text and (pre @k1 post) for
> parentheses expressions is nicer, but that's details. I trust your
> judgment on the technical merit of one idea versus the next.

[pre @k1 post] is slower to parse. (pre @k1 post) is worse because "("
is probably more common than "[". Really, round brackets should not be
used for Org syntax.

I haven't much against @k1, but it introduces more false positives than
[@k1].

Regards,

Re: Citations, continued

[Rasmus]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

>> Another issue is that it's not transpose-words safe.  E.g. this output
>> seems bad: [-@k1 @k2 30] => Y1 A2 (Y2, 30).
>
> This citation is invalid. The point of [@k1] is that the author is
> mandatory, since it is in-text, so [-@k1] doesn't make sense. 

In that case I prefer the explicit extraction of keys below, since I don't
understand why [-@k1] is invalid ("in her seminal -@k @k:journal article,
@k:author demonstrated that ⋯").  Probably I don't understand pandoc well
enough...

>> See also above.  From you explanation I would guess that at least these
>> two examples are wrong.  Is that correct?
>>
>>>>>  2. [cite:@item1: p. 30] says blah.
>>>>>  2.   Doe (2005, 30) says blah.
>>>>>  3. [cite:@item1: p. 30, with suffix] says blah.
>>>>>  3.   Doe (2005, 30, with suffix) says blah.
>
> They are not wrong. The output may decide to strip "p. ". This is not
> our problem at this stage of the discussion.

It seemed to be the main characteristics of this "locator" that you all
were talking about...

>>>> What if I need several text cite keys.  Say @K{1,2} is the same author A,
>>>> and @K3 is B.  Then  [cite:@K1,@K2,@K3] should/could be something like 
>>>> A (Y1, Y2), and B (Y3).  How do I express this?
>>>
>>> Since A and B do not appear in the same parenthesis, two citations are
>>> needed:
>>>
>>>   [@K1 -@K2], and [@K3]
>>
>> This is a minor downgrade from biblatex.  The year thing is worse.
>
> Org doesn't pretend to replace either LaTeX or biblatex.

Of course not.  That would be idiotic.  But I would rather try to imitate
the state-of-the-art, which in my book is biblatex, not pandocs.

> If extracting data from an entry is required, then I suggest to extend
> key syntax, e.g.:
>
>   @K1:year

Extracting from a plist-like thingy is easy.  What is hard is proper
formatting of things like author, e.g. how many author to include, adding
"and" etc.  It would be amazing to let TOOL handle this!  Unfortunately,
good TOOL is scare outside of the latex-world (maybe Zotero?  I don't know
how commandline friend it is).

>> it's not unheard of.  I have used it in the past.  In LaTeX it's something like: 
>>
>>     \citet[C]{k}    → A    (Y, C)
>>     \citet[B][]{k}  → A (B, Y)
>>     \citet[B][C]{k} → A (B, Y, C)


> I understand, but would it be needed to have both A (Y, C) and A (B, Y)
> in the same document?

Sure, why not?  The citation is often an active part of the sentence in
author-year style ("⋯ as AUTHOR-TEAM (see e.g. 1993, chapter 3) ⋯").  In
my experience postnotes are much more common, though.

> In any case, if we allow @key:tag syntax, then it is possible to do
>
>   [@k:author] [cite: B -@k C]
>
> and get
>
>   A (B, Y, C)

OK.  I don't particularly like it.  E.g. when I find out that the abstract
of @k was misleading and that @KK is the right citation I have to
carefully change @k to @KK two places.  This is probably minor, though.

>> It's nice. @k1 / [pre @k1 post] for text and (pre @k1 post) for
>> parentheses expressions is nicer, but that's details. I trust your
>> judgment on the technical merit of one idea versus the next.
>
> [pre @k1 post] is slower to parse. 

Is that because you'd have to check all occurrences or [⋯]?

> (pre @k1 post) is worse because "(" is probably more common than
> "[". Really, round brackets should not be used for Org syntax.

Probably true.

> I haven't much against @k1, but it introduces more false positives than
> [@k1].

It could check if k1 is a known key and interpret "@k1" accordingly.  In
my limited experience with bibtex.el it's only a bit slow to look up a key
the first time.  But who knows how this scale on very citation-intensive
documents.

On the interwebs some people use @USERNAME a lot (maybe it's a google
thing?).  That might speak against this.

—Rasmus

PS: Here's a quick link "proof-of-concept" (not really) with biblatex
only, and cite it textcitation.  Documents with this type of syntax are
indeed pleasant to the eye.

[[cite: pre1 @bohringer14 post1; pre2 @davis14 post2]]
→ \textcites[pre1][post1]{bohringer14}[pre2][post2]{davis14}

(org-add-link-type
 "cite"
 'identity
 (lambda (path description backend)
   (funcall 'rasmus/org-bib-format path description backend "textcite")))

(defun rasmus/interpret-cite-string (string)
  (mapcar
   (lambda (data)
     (string-match "\\([^@]*\\)@\\([^ \t]*\\)\\([^@]*\\)" data)
     (list :key (match-string 2 data)
           :pre (org-trim (or (match-string 1 data) ""))
           :post (org-trim (or (match-string 3 data) "")))) 
   (org-split-string string ";")))

(defun rasmus/org-bib-format (path description backend type)
  (let ((citations (rasmus/interpret-cite-string path)))
    (if (org-export-derived-backend-p backend 'latex)
        (format "\\%s%s%s"
                type (if (> (length citations) 1) "s" "")
                (mapconcat (lambda (cite)
                             (format "[%s][%s]{%s}" (plist-get cite :pre)
                                     (plist-get cite :post)
                                     (plist-get cite :key)))
                           citations "")))))


-- 
Summon the Mothership!

Re: Citations, continued

[Nicolas Goaziou]

Rasmus <rasmus@gmx.us> writes:

> In that case I prefer the explicit extraction of keys below, since I don't
> understand why [-@k1] is invalid ("in her seminal -@k @k:journal article,
> @k:author demonstrated that ⋯").  Probably I don't understand pandoc well
> enough...

In the initial suggestion @k:journal or @k:author was not possible. In
pandoc, AFAIU, in-text means author is outside parenthesis. Nothing
fancier. So [-@k1] would mean: "author" is outside parenthesis, but
should not be displayed anyway.

> It seemed to be the main characteristics of this "locator" that you all
> were talking about...

See, you are talking again about locator...

> Of course not.  That would be idiotic.  But I would rather try to imitate
> the state-of-the-art, which in my book is biblatex, not pandocs.

I'm sure biblatex is powerful, but it's clearly not the state-of-the-art
in readability, and I guess only `latex' would support its features.

Really my concerns are about parsing speed and readability within the
document. Heavy biblatex users will eventually have to fall-back to
LaTeX-specific syntax at some point anyway.

>> If extracting data from an entry is required, then I suggest to extend
>> key syntax, e.g.:
>>
>>   @K1:year
>
> Extracting from a plist-like thingy is easy.

I don't understand.

> What is hard is proper formatting of things like author, e.g. how many
> author to include, adding "and" etc. It would be amazing to let TOOL
> handle this! Unfortunately, good TOOL is scare outside of the
> latex-world (maybe Zotero? I don't know how commandline friend it is).

In any case, this is not the job of Org. At the moment, the goal is to
find a readable syntax that allows to present citation information to
external processors (biblatex, zotero...).

>> I understand, but would it be needed to have both A (Y, C) and A (B, Y)
>> in the same document?
>
> Sure, why not?

I don't know. Pandoc doesn't allow it, and, apparently, nobody
complained enough to add this feature to Pandoc citations. So, either it
is not that useful, or Pandoc citations are hardly used.

>> [pre @k1 post] is slower to parse. 
>
> Is that because you'd have to check all occurrences or [⋯]?

Correct. 

Also, it is ambiguous with link syntax (e.g., if pre begin with "[") and
footnotes syntax.

>> I haven't much against @k1, but it introduces more false positives than
>> [@k1].
>
> It could check if k1 is a known key and interpret "@k1" accordingly.

No it couldn't. Org doesn't know about keys. Or, more precisely, syntax
mustn't depend on known keys. I don't want to make the same mistake as
export blocks (i.e., #+begin_html doesn't mean the same thing if
"ox-html" is loaded or not).

@k1, for some subset of chars within k1, would always be parsed a key,
valid or not. Again, this is not particularly wrong, but it may
introduce false positive from time to time. However, in these cases, we
could use entities to escape "@" from the key.


Regards,

Re: Citations, continued

[Rasmus]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> In the initial suggestion @k:journal or @k:author was not possible. In
> pandoc, AFAIU, in-text means author is outside parenthesis. Nothing
> fancier. So [-@k1] would mean: "author" is outside parenthesis, but
> should not be displayed anyway.

If [-@k1] has non-obvious, interchangeable meanings depending on where it
happens to be typed I like it even less.  "-@" seems flaky and
over-complicated IMO.

> Really my concerns are about parsing speed and readability within the
> document. 

Agreed.

> Heavy biblatex users will eventually have to fall-back to
> LaTeX-specific syntax at some point anyway.

That's unfortunate.


>>> I understand, but would it be needed to have both A (Y, C) and A (B, Y)
>>> in the same document?
>>
>> Sure, why not?
>
> I don't know. Pandoc doesn't allow it, and, apparently, nobody
> complained enough to add this feature to Pandoc citations. So, either it
> is not that useful, or Pandoc citations are hardly used.

Perhaps Org-Mode users write more sophisticated documents

See Tom's post for a more careful analysis of what data a citation
contains.

> Also, it is ambiguous with link syntax (e.g., if pre begin with "[") and
> footnotes syntax.

So don't allow footnotes and links within citations.  Emphasis is enough.
This is a less severe loss than PRENOTE.  Also, nested citations can be
dropped: e.g. A1 (Y1, POST1, PRE1 A2 Y2 POST2) if that makes parsing
easier.

>>> I haven't much against @k1, but it introduces more false positives than
>>> [@k1].
>>
>> It could check if k1 is a known key and interpret "@k1" accordingly.
>
> No it couldn't. Org doesn't know about keys. Or, more precisely, syntax
> mustn't depend on known keys. I don't want to make the same mistake as
> export blocks (i.e., #+begin_html doesn't mean the same thing if
> "ox-html" is loaded or not).

That's valid.  I remember having run into this (and maybe even report it
as a bug).

—Rasmus

-- 
With monopolies the cake is a lie!

Re: Citations, continued

[Nicolas Goaziou]

Rasmus <rasmus@gmx.us> writes:

> PS: Here's a quick link "proof-of-concept" (not really) with biblatex
> only, and cite it textcitation.  Documents with this type of syntax are
> indeed pleasant to the eye.
>
> [[cite: pre1 @bohringer14 post1; pre2 @davis14 post2]]

As pointed out already, the double square bracket is not needed, and can
introduce ambiguity with links.

I'm fine with

  [cite: pre1 @bohringer14 post1; pre2 @davis14 post2]

but how do you distinguish between in-text and in-parenthesis?

Also, how do you extract year or journal from the key (since you
suggested these features in the first place)?


Regards,

Re: Citations, continued

[Rasmus]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Rasmus <rasmus@gmx.us> writes:
>
>> PS: Here's a quick link "proof-of-concept" (not really) with biblatex
>> only, and cite it textcitation.  Documents with this type of syntax are
>> indeed pleasant to the eye.
>>
>> [[cite: pre1 @bohringer14 post1; pre2 @davis14 post2]]
>
> As pointed out already, the double square bracket is not needed, and can
> introduce ambiguity with links.

Hence the double warning:

>>"proof-of-concept" (not really)

> I'm fine with
>
>   [cite: pre1 @bohringer14 post1; pre2 @davis14 post2]
>
> but how do you distinguish between in-text and in-parenthesis?

In the above with "citet" and "citep" or similar.

The above was only meant to try out the [PRE @KEY POST; PRE @KEY POST]
format and show that it indeed is nice.  That's all.
   
> Also, how do you extract year or journal from the key (since you
> suggested these features in the first place)?

Actually, I first suggested only having /types/ of citations.  Richard
suggested extraction of fields.

—Rasmus

-- 
The right to be left alone is a human right

Re: Citations, continued

[John Kitchin]

Nicolas Goaziou writes:

> Rasmus <rasmus@gmx.us> writes:
>
>> PS: Here's a quick link "proof-of-concept" (not really) with biblatex
>> only, and cite it textcitation.  Documents with this type of syntax are
>> indeed pleasant to the eye.
>>
>> [[cite: pre1 @bohringer14 post1; pre2 @davis14 post2]]
>
> As pointed out already, the double square bracket is not needed, and can
> introduce ambiguity with links.
>
> I'm fine with
>
>   [cite: pre1 @bohringer14 post1; pre2 @davis14 post2]

Why cannot there be a list of acceptable keywords eg

[citenum:
[citeyear:
[citeauthor:

which a backend would be responsible for handling, including a default
handler for unknown keywords?

>
> but how do you distinguish between in-text and in-parenthesis?
>
> Also, how do you extract year or journal from the key (since you
> suggested these features in the first place)?
>
>
> Regards,

--
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu

Re: Citations, continued

[Nicolas Goaziou]

John Kitchin <jkitchin@andrew.cmu.edu> writes:

> Why cannot there be a list of acceptable keywords eg
>
> [citenum:
> [citeyear:
> [citeauthor:
>
> which a backend would be responsible for handling, including a default
> handler for unknown keywords?

It has the same problem as [pre @key post] syntax: it is slower to
parser. This is because you cannot know beforehand the keyword, so you
have to check almost every [...] in the document.

Another option is to mimic custom links, if that's what you're thinking
of, which means to store every user-defined keyword in a variable and
build a regexp out of it. I dislike it even more because the document is
not portable anymore, as it requires you to share your custom keywords.

If year, or author, are needed, I suggested to append some optional
parameter to the key, e.g.,

  [cite: pre @key:year post]

Note that it rules out colons from KEY syntax (but we can use another
less common character, e.g. "|").


Regards,

Re: Citations, continued

[Rasmus]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Another option is to mimic custom links, if that's what you're thinking
> of, which means to store every user-defined keyword in a variable and
> build a regexp out of it. I dislike it even more because the document is
> not portable anymore, as it requires you to share your custom keywords.

So, the (opinionated) useful defaults in biblatex are:
    cite(s), parencite(s), footcite(s), texcite(s), fullcite,
    footfullcite, nocite

Citation types for extracting parts:
     citeauthor, citetitle, citeyear, citedate, citeurl,

From natbib:
    citet (== textcite), citep (== parencite).

Keys I don't care about, since they are quite biblatex specific:
    smartcite, autocide, parentcite*, uppercase variants. *volcites(s) (any objections?)

In natbib:
   citealt, citalp, starred variants

So that's 17 support keys and two aliases.  I guess this would deter most
authors from needing custom styles.

> Note that it rules out colons from KEY syntax (but we can use another
> less common character, e.g. "|").

The default bibtex.el style generates keys like "%A%y:%t", so I think ":"
is no good, appealing as it is.

—Rasmus


Footnotes: 
¹   which is just
    [cite: common pre; pre1 @k1 post1; ⋯; preN @kN postN;  common post]

-- 
Er du tosset for noge' lårt!

Re: Citations, continued

[Nicolas Goaziou]

Rasmus <rasmus@gmx.us> writes:

> So, the (opinionated) useful defaults in biblatex are:
>     cite(s), parencite(s), footcite(s), texcite(s), fullcite,
>     footfullcite, nocite

Isn't footcite/footfullcite a choice made at the document's level
instead of per citation? If that's the case, it could go in a keyword,
e.g.,

  #+LATEX_CITATION: :style footcite

> Citation types for extracting parts:
>      citeauthor, citetitle, citeyear, citedate, citeurl,

Can't this be attached to the key, as a filter?

> From natbib:
>     citet (== textcite), citep (== parencite).
>
> Keys I don't care about, since they are quite biblatex specific:
>     smartcite, autocide, parentcite*, uppercase variants. *volcites(s) (any objections?)
>
> In natbib:
>    citealt, citalp, starred variants
>
> So that's 17 support keys and two aliases.  I guess this would deter most
> authors from needing custom styles.

Then what about

  [cite:command: common pre; pre1 @key1 post1; ... ; common post]

where command is anything matching is constituted of alphanumeric
characters only (this is just a guess, a proper regexp is yet to be
determined).

LaTeX back-end will see "command" and less advanced back-ends "cite", so
that the same document can be exported through multiple back-ends.

Also

  [cite: common pre; pre1 @key1 post1; ... ; common post]

would be equivalent to

  [cite:default_command: common pre; pre1 @key1 post1; ... ; common post]

where "default_command" would be set in a defcustom within
"ox-latex.el".

However, this syntax doesn't handle in-text citation for other back-ends
than LaTeX. Hence the [@key post] proposal, or even @key [post], which
I find more elegant than

  [citet: ...] / [citep: ...]

> The default bibtex.el style generates keys like "%A%y:%t", so I think ":"
> is no good, appealing as it is.

Then "/" (filter) or "|" (pipe).


Regards,

Re: Citations, continued

[Rasmus]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

>> So, the (opinionated) useful defaults in biblatex are:
>>     cite(s), parencite(s), footcite(s), texcite(s), fullcite,
>>     footfullcite, nocite
>
> Isn't footcite/footfullcite a choice made at the document's level
> instead of per citation? If that's the case, it could go in a keyword,
> e.g.,
>
>   #+LATEX_CITATION: :style footcite

I guess you'd distinguish between fullcite and footfullcite then?  I have
only ever used fullcite for illustrative purposes, e.g. demonstrating the
citation style.  And I guess footcite is an alternative to
{textcite, parencite}.

>> Citation types for extracting parts:
>>      citeauthor, citetitle, citeyear, citedate, citeurl,
>
> Can't this be attached to the key, as a filter?

Do you mean an ox-filter here or the slash "/"?  It's more complex and but
probably also prettier.  "[@K/author]" looks nice.  I haven't seen "/" in
bibtex keys.

In any case, an ox-filter is no good.  You sometimes need it for
constructing sentences, e.g. I like to keep out the year when it's obvious
to ease reading::

    A (Y) showed foo.  Note that A assumed that ...

> Then what about
>
>   [cite:command: common pre; pre1 @key1 post1; ... ; common post]

Could work.

> where command is anything matching is constituted of alphanumeric
> characters only (this is just a guess, a proper regexp is yet to be
> determined).
>
> LaTeX back-end will see "command" and less advanced back-ends "cite", so
> that the same document can be exported through multiple back-ends.

OK.  But what if I want to use, say, my "genitive" citation, "A's (Y)", in
html?  This is perhaps a question of whether we'll manage to find a tool
to handle this for us, or we'll have to do it lisp.

> However, this syntax doesn't handle in-text citation for other back-ends
> than LaTeX. Hence the [@key post] proposal, or even @key [post], which
> I find more elegant than
>
>   [citet: ...] / [citep: ...]

So [@key post] is equivalent to [cite:default_command: @key post].  Does
it scale to an arbitrary length of keys, e.g. [@k1 post1; ⋯; @kN postN]?
Could [@: pre1 @k1 post1; ⋯; preN @kN postN] be used if you need prenotes?
Or only [cite:⋯].  

Would you "expand" all short citations in the early ox parsing?

I don't care for "@key [post]"

>> The default bibtex.el style generates keys like "%A%y:%t", so I think ":"
>> is no good, appealing as it is.
>
> Then "/" (filter) or "|" (pipe).

Why do you write "filter" after the slash?  Am I supposed to think about
ox-filters?
 
—Rasmus

-- 
Governments should be afraid of their people

Re: Citations, continued

[Andreas Leha]

Hi,

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:
> Rasmus <rasmus@gmx.us> writes:
>
>> So, the (opinionated) useful defaults in biblatex are:
>>     cite(s), parencite(s), footcite(s), texcite(s), fullcite,
>>     footfullcite, nocite
>
> Isn't footcite/footfullcite a choice made at the document's level
> instead of per citation? If that's the case, it could go in a keyword,
> e.g.,
>
>   #+LATEX_CITATION: :style footcite
>

I have only used footcite in beamer presentations so far.  It is quite
common to show a fullframe image and have a footcitation.

But I also mixed them with parencites, when a specicic item on the page
is attributed to a given reference.

Maybe one could debate about using something other that footcite for the
former use case.  So, I don't have too strong an opinion here.

I just wanted to throw in a use-case where this decision was not made on
the per-document level.

Regards,
Andreas

Re: Citations, continued

[John Kitchin]

Rasmus writes:

> Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:
>
>> Another option is to mimic custom links, if that's what you're thinking
>> of, which means to store every user-defined keyword in a variable and
>> build a regexp out of it. I dislike it even more because the document is
>> not portable anymore, as it requires you to share your custom keywords.
>
> So, the (opinionated) useful defaults in biblatex are:
>     cite(s), parencite(s), footcite(s), texcite(s), fullcite,
>     footfullcite, nocite
>
> Citation types for extracting parts:
>      citeauthor, citetitle, citeyear, citedate, citeurl,

If citenum was also in that list, then I agree. It is not that likely
there is little need for custom style.

>
> From natbib:
>     citet (== textcite), citep (== parencite).
>
> Keys I don't care about, since they are quite biblatex specific:
>     smartcite, autocide, parentcite*, uppercase variants. *volcites(s)
>     (any objections?)
None from me.

>
> In natbib:
>    citealt, citalp, starred variants
>
> So that's 17 support keys and two aliases.  I guess this would deter most
> authors from needing custom styles.
>
>> Note that it rules out colons from KEY syntax (but we can use another
>> less common character, e.g. "|").
>
> The default bibtex.el style generates keys like "%A%y:%t", so I think ":"
> is no good, appealing as it is.
>
> —Rasmus
>
>
> Footnotes:
> ¹   which is just
>     [cite: common pre; pre1 @k1 post1; ⋯; preN @kN postN;  common post]

--
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu

Re: Citations, continued

[Rasmus]

John Kitchin <jkitchin@andrew.cmu.edu> writes:

> Rasmus writes:
>
>> Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:
>>
>>> Another option is to mimic custom links, if that's what you're thinking
>>> of, which means to store every user-defined keyword in a variable and
>>> build a regexp out of it. I dislike it even more because the document is
>>> not portable anymore, as it requires you to share your custom keywords.
>>
>> So, the (opinionated) useful defaults in biblatex are:
>>     cite(s), parencite(s), footcite(s), texcite(s), fullcite,
>>     footfullcite, nocite
>>
>> Citation types for extracting parts:
>>      citeauthor, citetitle, citeyear, citedate, citeurl,
>
> If citenum was also in that list, then I agree. It is not that likely
> there is little need for custom style.

Ok, sorry I didn't check the natbib manual carefully.  AFAIK you get
numbers with biblatex without any author-year options so:

 \cite{k}, \parencite{k} → [Num]
 \textcite{k} → A [Num]

Is this similar to \numcite?  From natbib is seems to be intended for
people who use author-year, but still wants numbers.  Is that correct?

—Rasmus

-- 
Vote for proprietary math!

Re: Citations, continued

[John Kitchin]

>
> Ok, sorry I didn't check the natbib manual carefully.  AFAIK you get
> numbers with biblatex without any author-year options so:
>
>  \cite{k}, \parencite{k} → [Num]
>  \textcite{k} → A [Num]
>
> Is this similar to \numcite?  From natbib is seems to be intended for
> people who use author-year, but still wants numbers.  Is that correct?

I don't think so. It is just the number that the citation has in the
bibliography, not in brackets or parens, and usually not
superscripted. We use it to get something like this in the exported
document:

"This is seen in Ref. 9"

With the other commands you get something like:
"This is seen in Ref. [9]"

or if you have a superscripted style:
"This is seen in Ref. ^9"


>
> —Rasmus

--
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu

Re: Citations, continued

[Richard Lawrence]

Rasmus <rasmus@gmx.us> writes:

> So, the (opinionated) useful defaults in biblatex are:
>     cite(s), parencite(s), footcite(s), texcite(s), fullcite,
>     footfullcite, nocite

So that is to say we need to be able to express the following
distinctions (did I miss anything?):
  - in-text vs. parenthetical (parencite vs. textcite)
  - normal citation vs. multi-cite citation with common pre and
    post notes (-s variants)
  - producing a full bibliography entry, or not (-full- variants)
  - footnoted, or not (foot- variants)
  - producing output, or not (nocite)

I am not sure about that the last two need to be represented in citation
syntax itself.  Do we need a separate way of indicating that a citation
should appear in a footnote?  Org already has footnote syntax...can't
authors just put citations in an Org footnote? 

As for nocites, I liked your earlier suggestion that we have a #+NOCITE
keyword (which could be specified multiple times).  So I am not
sure this needs to be in citation syntax proper, either.

> Citation types for extracting parts:
>      citeauthor, citetitle, citeyear, citedate, citeurl,

As I've said in other posts, I think maybe we should not think of these
as `citation' commands and thus don't need to represent them in citation
syntax.  Instead I suggest we give authors tools to insert this
information into documents directly.

Best,
Richard

Re: Citations, continued

[Stefan Nobis]

Richard Lawrence <richard.lawrence@berkeley.edu> writes:

> Rasmus <rasmus@gmx.us> writes:

>> Citation types for extracting parts:
>>      citeauthor, citetitle, citeyear, citedate, citeurl,

> As I've said in other posts, I think maybe we should not think of
> these as `citation' commands and thus don't need to represent them
> in citation syntax. Instead I suggest we give authors tools to
> insert this information into documents directly.

This would render changes quite hard. Maybe I misspelled something in
the database or I choosed the wrong reference: With above part
extractors all I have to do at most is to replace the @key. But if
data is copied verbatim, I have to search for all years, author names,
titles, urls etc. Very error prone.

I think, even citeauthor or citeurl are citations. A normal "\cite"
command is nothing else than a short reference to all the detail data
in the bibliography. And sometimes context allows to make these
references even shorter, by only using the author name or a year etc.
I don't really see the distinction between citation and indirection -
each citation is as much an indirection as e.g. citeyear is.

-- 
Until the next mail...,
Stefan.

Re: Citations, continued

[Richard Lawrence]

Hi Stefan,

Stefan Nobis <stefan-ml@snobis.de> writes:

> Richard Lawrence <richard.lawrence@berkeley.edu> writes:
>
>> Rasmus <rasmus@gmx.us> writes:
>
>>> Citation types for extracting parts:
>>>      citeauthor, citetitle, citeyear, citedate, citeurl,
>
>> As I've said in other posts, I think maybe we should not think of
>> these as `citation' commands and thus don't need to represent them
>> in citation syntax. Instead I suggest we give authors tools to
>> insert this information into documents directly.
>
> This would render changes quite hard. Maybe I misspelled something in
> the database or I choosed the wrong reference: With above part
> extractors all I have to do at most is to replace the @key. But if
> data is copied verbatim, I have to search for all years, author names,
> titles, urls etc. Very error prone.

That's true, and I realize that's a disadvantage.  (Though I think these
are errors that, for the most part, normal proof-reading will correct
anyway.)

I know these commands are convenient, and that not having them would
introduce this class of errors, but the question is whether they are so
important that it's worth providing an equivalent for them in non-LaTeX
backends.  For my part, it seems like the convenience is not worth the
effort that would be required to make the exporter handle these
correctly in general.  (For example, it seems the exporter would then
have to worry about things like quoting and emphasizing document titles
-- which means worrying about context, document type, locale and
language, quotation styles, etc.)  But maybe I'm wrong; can you make the
case that it is indeed worth it?

Best,
Richard

Re: Citations, continued

[Stefan Nobis]

Richard Lawrence <richard.lawrence@berkeley.edu> writes:

> I know these commands are convenient, and that not having them would
> introduce this class of errors, but the question is whether they are
> so important that it's worth providing an equivalent for them in
> non-LaTeX backends.

Hmmm... I don't see this as a big problem. Either the exporter or some
tool has to be able to read from the bibliography database and has to
understand at least parts of the available fields (e.g. author and
year to enable author-year style citations). Based on this it should
be easy to only output some of the fields (e.g. only author).

> For my part, it seems like the convenience is not worth the effort
> that would be required to make the exporter handle these correctly
> in general. (For example, it seems the exporter would then have to
> worry about things like quoting and emphasizing document titles --
> which means worrying about context, document type, locale and
> language, quotation styles, etc.)

Does the exporter really habe to worry about these things? But anyway:
Some tool is needed to generate the bibliography with all its data -
this tool has to handle all these details and therefore it should be
not too hard to get partial data from it. BTW: I don't think any
special formatting should be required - ASCII or even HTML would never
look the same as a LaTeX generated PDF. So minor drawbacks are IMHO
not as important as to be able to express important details in the
source.

I think, the syntax should be quite flexible (at least easy to extend,
with compact, nice looking extension-syntax). If some backend lacks
support for some feature, maybe someone finds the time to fix it (and
then org-mode would rule the world :)). Otherwise a simple fallback
(default citation style, output citation string unchanged,...) will be
used.

-- 
Until the next mail...,
Stefan.

Re: Citations, continued

[Richard Lawrence]

Stefan Nobis <stefan-ml@snobis.de> writes:

> Richard Lawrence <richard.lawrence@berkeley.edu> writes:
>
>> I know these commands are convenient, and that not having them would
>> introduce this class of errors, but the question is whether they are
>> so important that it's worth providing an equivalent for them in
>> non-LaTeX backends.
>
> Hmmm... I don't see this as a big problem. Either the exporter or some
> tool has to be able to read from the bibliography database and has to
> understand at least parts of the available fields (e.g. author and
> year to enable author-year style citations). Based on this it should
> be easy to only output some of the fields (e.g. only author).

Well, a lot of the time, it probably is.  But what if the author's name
is, say, `John von Neumann' (to use a recently-mentioned example)?  Then
the exporter needs to figure out (1) whether to output the full name or
just the last name; if the latter, (2) whether the last name is
`Neumann' or `von Neumann'; and if the latter, (3) how to capitalize the
last name in the context where it appears.

> But anyway: Some tool is needed to generate the bibliography with all
> its data - this tool has to handle all these details and therefore it
> should be not too hard to get partial data from it.

That is true.  Some tool does have to do this, and there are tools that
are designed for it outside of LaTeX (like CSL processors) that Org
could rely on.  But I really don't have any idea how easy it would be to
make the exporter interact with them in a fine-grained way like this, or
what constraints they would place on citation support in Org.  Does
anyone have a sense of this?

> BTW: I don't think any special formatting should be required - ASCII
> or even HTML would never look the same as a LaTeX generated PDF. So
> minor drawbacks are IMHO not as important as to be able to express
> important details in the source.

That is true.  The question we are trying to reach a consensus on is
just what constitutes `important'. :)

> I think, the syntax should be quite flexible (at least easy to extend,
> with compact, nice looking extension-syntax).

Yes, I agree.  Still, as I said elsewhere, I think it's really important
to draw a clear line between the `main' citation syntax, which encodes
features that are important enough to support fully on all backends, and
the syntax for extensions, which encode features that might only work on
some backends or in one's personal setup.  I think this is important for
document portability and so authors and editors know what they can rely
on.

I also think it's important because I suspect the only way we're going
to get a working Org citation syntax is if we try very hard to limit
what goes into the first category to something (much) less than
`everything we can now do with LaTeX'.  We can always move more stuff
into the first category later on, if it becomes clear that it's needed.

I certainly don't mean to be telling people that the LaTeX features they
are now relying on are not useful, or not important.  (I apologize if
I'm coming off that way!) I just think some of them might not be *so*
useful that they are worth the effort to fully support on other
backends, at least initially.

> If some backend lacks support for some feature, maybe someone finds
> the time to fix it (and then org-mode would rule the world
> :)). Otherwise a simple fallback (default citation style, output
> citation string unchanged,...) will be used.

IMHO, this is really only acceptable for features that fall in the
second category.  If there are `core' features of Org citations that
don't work on all backends, then there isn't really anything gained by
using Org citations instead of just sticking with LaTeX (or another
backend-specific citation format).

I am personally totally fine with a syntax that allows you to say things
like:

[cite: see @Doe99 for more. :latex-command SomeEsotericCitationCommand]

or

[cite: see @Doe99 for more. :custom-type my-custom-type]

as long as it is obvious to an author that these citations might not
export correctly on all backends, in contrast to a citation makes use of
only the `main' Org citation syntax.  Individual authors are in a better
position to decide when that tradeoff is acceptable to them.

Best,
Richard

Re: Citations, continued

[Matt Price]

[-- Attachment #1: Type: text/plain, Size: 1010 bytes --]

On Wed, Feb 11, 2015 at 11:51 AM, Richard Lawrence <
richard.lawrence@berkeley.edu> wrote:

> Stefan Nobis <stefan-ml@snobis.de> writes:
>
> > Richard Lawrence <richard.lawrence@berkeley.edu> writes:
> >
> > But anyway: Some tool is needed to generate the bibliography with all
> > its data - this tool has to handle all these details and therefore it
> > should be not too hard to get partial data from it.
>
> That is true.  Some tool does have to do this, and there are tools that
> are designed for it outside of LaTeX (like CSL processors) that Org
> could rely on.  But I really don't have any idea how easy it would be to
> make the exporter interact with them in a fine-grained way like this, or
> what constraints they would place on citation support in Org.  Does
> anyone have a sense of this?
>
>
I think Erik Hetzner knows a lot about this -- he has done all that work on
zotxt, which involves talking
to Zotero's internal citeproc.js, which is, I think, the most widely-used
CSL processor.

Matt

[-- Attachment #2: Type: text/html, Size: 1598 bytes --]

Re: Citations, continued

[Aaron Ecay]

Hi Rasmus,

2015ko otsailak 10an, Rasmus-ek idatzi zuen:

[...]


> Keys I don't care about, since they are quite biblatex specific:
>     smartcite, autocide, parentcite*, uppercase variants. *volcites(s) (any objections?)

Smartcite is like footcite in regular text and parencite in footnotes.
I think we probably want functionality like that, but it should be in
lisp so that it works for all backends.

The uppercase variants are important for correctness with names like
“von Neumann”.  When we write a sentence which begins “Von Neumann
(19xx) claimed that...” the “von” must be capitalized, though it
should be lowercase in non-sentence-initial position.  Of course, we
don’t need to copy the biblatex convention of capitalized command
names to represent this in org.  Perhaps a :capitalize t property
could be added, assuming one of the proposals is adopted that allows
the inclusion of a property list in the citation.

-- 
Aaron Ecay

Re: Citations, continued

[Rasmus]

Aaron Ecay <aaronecay@gmail.com> writes:

> The uppercase variants are important for correctness with names like
> “von Neumann”.  When we write a sentence which begins “Von Neumann
> (19xx) claimed that...” the “von” must be capitalized, though it
> should be lowercase in non-sentence-initial position.  Of course, we
> don’t need to copy the biblatex convention of capitalized command
> names to represent this in org.  Perhaps a :capitalize t property
> could be added, assuming one of the proposals is adopted that allows
> the inclusion of a property list in the citation.

So I guess Org cannot be used for game theory or CS papers. . .

I guess, in ox-latex it should use magically use the uppercase variants
depending on sentence-end or similar voodoo.  Like textcites would
automatically be used if there's more than one key.

Anyway, you are right.  But they should still not be exposed in the case a
limited set of biblatex keys are adopted.

—Rasmus

-- 
In theory, practice and theory are the same. In practice they are not

Re: Citations, continued

[Richard Lawrence]

Hi Nicolas and all,

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> If year, or author, are needed, I suggested to append some optional
> parameter to the key, e.g.,
>
>   [cite: pre @key:year post]

I proposed exactly this earlier in the thread, but then I came to the
conclusion that we shouldn't do it.

Conceptually, something like `@key:year' isn't a citation, but merely
indirection, because it doesn't actually provide the reader of the
rendered document enough information to look up the reference.  I think
we can cut down on the number of `citation' types that the syntax should
support if we distinguish citations from indirection like this.

Practically speaking, I think this would also be too hard to implement
correctly.  Once you can say things like "@key:title", you have to worry
about how this will be formatted in the exported document (in quotes?
italics?  etc.), and it's just not clear how to do that in general.

Instead, I suggest that Org supports this kind of indirection via other
means, perhaps by providing functions to look up the required data and
insert it directly into the text, where the author can format it as
desired.

So I would suggest that we *don't* try to capture what LaTeX does with
\citeyear, \citeauthor, etc. in the citation syntax.

Best,
Richard

Re: Citations, continued

[Thomas S. Dye]

Richard Lawrence <richard.lawrence@berkeley.edu> writes:

> Conceptually, something like `@key:year' isn't a citation, but merely
> indirection, because it doesn't actually provide the reader of the
> rendered document enough information to look up the reference.  I think
> we can cut down on the number of `citation' types that the syntax should
> support if we distinguish citations from indirection like this.

I don't think this concept holds in the LaTeX world.  I'm fairly certain
that citation commands like \citeauthor and \citeyear create an entry in
the bibliography.

All the best,
Tom

-- 
Thomas S. Dye
http://www.tsdye.com

Re: Citations, continued

[Richard Lawrence]

Hi Tom and all,

tsd@tsdye.com (Thomas S. Dye) writes:

> Richard Lawrence <richard.lawrence@berkeley.edu> writes:
>
>> Conceptually, something like `@key:year' isn't a citation, but merely
>> indirection, because it doesn't actually provide the reader of the
>> rendered document enough information to look up the reference.  I think
>> we can cut down on the number of `citation' types that the syntax should
>> support if we distinguish citations from indirection like this.
>
> I don't think this concept holds in the LaTeX world.  I'm fairly certain
> that citation commands like \citeauthor and \citeyear create an entry in
> the bibliography.

Fair enough.  I just meant that something like

"As the reader may verify, \citeyear{Doe99} fails to make any progress on this issue."

doesn't render into a form that allows the reader to know which work is
being talked about, even if that work appears in the bibliography; the
author has to supply more context for it to make sense.  Thus, \citeyear
and friends are more of a convenience for the author than commands to
produce a (complete) citation that the reader can use.

But I don't really care so much about the right definition of "citation"
as about the fact that supporting an equivalent for these commands in
non-LaTeX backends strikes me as really hard, which makes me wonder if
the effort required to support them is worth the convenience gained by
representing them in the main citation syntax.

It seems like it would be hard because providing equivalents for things
like \citeauthor or \citetitle in, say, HTML would require the exporter
to know a *lot* about how to format names and titles in the context
where those citations appear.  This is a very non-trivial problem.

But perhaps the exporter could rely on an external CSL processor for
things like this?  I don't really know if CSL can handle this kind of
`partial' citation -- if it can, and if CSL is part of the plan for
exporting citations in non-LaTeX backends, then I have no real objection
to representing them in the citation syntax, because they are indeed
convenient.

Best,
Richard

Re: Citations, continued

[Thomas S. Dye]

Richard Lawrence <richard.lawrence@berkeley.edu> writes:

> Hi Tom and all,
>
> tsd@tsdye.com (Thomas S. Dye) writes:
>
>> Richard Lawrence <richard.lawrence@berkeley.edu> writes:
>>
>>> Conceptually, something like `@key:year' isn't a citation, but merely
>>> indirection, because it doesn't actually provide the reader of the
>>> rendered document enough information to look up the reference.  I think
>>> we can cut down on the number of `citation' types that the syntax should
>>> support if we distinguish citations from indirection like this.
>>
>> I don't think this concept holds in the LaTeX world.  I'm fairly certain
>> that citation commands like \citeauthor and \citeyear create an entry in
>> the bibliography.
>
> Fair enough.  I just meant that something like
>
> "As the reader may verify, \citeyear{Doe99} fails to make any progress
> on this issue."
>
> doesn't render into a form that allows the reader to know which work is
> being talked about, even if that work appears in the bibliography; the
> author has to supply more context for it to make sense.  Thus, \citeyear
> and friends are more of a convenience for the author than commands to
> produce a (complete) citation that the reader can use.
>
> But I don't really care so much about the right definition of "citation"
> as about the fact that supporting an equivalent for these commands in
> non-LaTeX backends strikes me as really hard, which makes me wonder if
> the effort required to support them is worth the convenience gained by
> representing them in the main citation syntax.
>
> It seems like it would be hard because providing equivalents for things
> like \citeauthor or \citetitle in, say, HTML would require the exporter
> to know a *lot* about how to format names and titles in the context
> where those citations appear.  This is a very non-trivial problem.
>
> But perhaps the exporter could rely on an external CSL processor for
> things like this?  I don't really know if CSL can handle this kind of
> `partial' citation -- if it can, and if CSL is part of the plan for
> exporting citations in non-LaTeX backends, then I have no real objection
> to representing them in the citation syntax, because they are indeed
> convenient.
>
> Best,
> Richard

Couldn't the syntax be extensible so it accommodates any citation
command?

Perhaps each backend could support an appropriate set of commands and
fallback to a default if the user tries an unsupported command?

If a LaTeX user wants all the citation commands for an html document,
there are converters such as tex4ht.  Org mode doesn't need to reinvent
that wheel.

I expect that ox-latex will often need to support new citation commands
as the humanities continue to develop styles based on biblatex.  Perhaps
we should view extensible syntax as "backend extensible" rather than
"user extensible"?

All the best,
Tom

-- 
Thomas S. Dye
http://www.tsdye.com

Re: Citations, continued

[Rasmus]

Hi,

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

>   - full-citation (aka individual citation), with, in addition to the
>     properties above, :prefix and :suffix, both being parsed string.

Full-citation is confusing.  A full citation is (to me) what you have in
the bibliography.

> Since full citations can only exist in a bracketed citation, there is no
> reason to create a third object type for the latter. It acts as a mere
> container only useful for lexer.

Maybe I'm misunderstanding, but if you by bracketed citation mean a
parentheses expression your claim in false.  This is a valid and common
enough citation:

       Smith (see e.g. 1991, pp. 31)

\textcite[see in particular][pp. 31]{smith}

On the other hand if "bracketed" refers to how it's written in the source,
how do I get both AUTHOR (PRE YEAR SUF) and (PRE AUTHOR, YEAR SUF)
citations in the same document?

> . For example,
> Eric's parser chose the former, which is good, but also disallows square
> brackets in prefix, which rules out some objects from this location
> (mainly links and footnotes).

\textcite[test\footnote{test}][]{key} won't compile either, though you
could probably work around if you really wanted to.

> Of course I understand the need for compatibility with existing Pandoc
> syntax, but I wouldn't want us to shoot ourselves in the foot. Even if
> we don't use "cite:" markup, I think we should carefully specify current
> syntax to avoid loopholes.

I /don't/ understand why comparability with pandoc is needed.  For better
or worse Org is a different markup.

How would cite: work with prefix, posfix and citationtype?

—Rasmus

-- 
There are known knowns; there are things we know that we know

Re: Citations, continued

[Rasmus]

Hi,

I realize you list Pandoc features, but I will still point out some issues
with this syntax.

Richard Lawrence <richard.lawrence@berkeley.edu> writes:

> Specifically I think we need the following categories, all of which
> would be objects:
>   - key
>   - prefix / pre-text
>   - suffix / post-text
>   - locator

What is the point of an locator?  Why not just suffix?  Presumably, if I
want to put pp. in front of my page numbers or whatever I can do it with
a filter or manually.


> These should have a grammar like the following, based on my
> (reverse-engineered) understanding of the Pandoc syntax for citations:
> ...
>   - A key optionally begins with '-', and obligatorily contains '@'
>     followed by a string of charcters which begins with a letter or '_',
>     and may contain alphanumeric characters and the following internal
>     punctuation characters:
>        :.#$%&-+?<>~/

I fail to understand this feature.  The Pandoc manual uses something like
this an example: A said X in @-key; which I think is bad practice.  In
latex you'd write \citeauthor{key} said X in \citeyear{key}.  Unless we
can access other keys, why adopt a special operator for year?  Why not
title or author which are e.g. useful when using number citations?q

>   - An unbracketed citation consists of a key, optionally followed by a
>     locator which is enclosed in '[' ']'

This is another, to me, illogical structure.

     [A @key B]
     @key [B]

It is not obvious that [B] relates to @key in the second example.

–Rasmus

-- 
This message is brought to you by the department of redundant departments

Re: Citations, continued

[Erik Hetzner]

On Wed,  4 Feb 2015 at 04:06:36 PST,
Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:
> 
> Hello,
> 
> Erik Hetzner <egh@e6h.org> writes:
> 
> > I concentrated on getting the parser to recognize valid citations
> > first. I have now finished this part (excepting any bugs, of course :)
> > and will need to add code to generate a proper parse tree. Then it can
> > be integrated into org-element.el.
> 
> The point of "org-element.el" is to generate a parse tree. If you
> generate it yourself, you do not need the library. Therefore, I'm not
> sure to understand the "integration" part.
> 
> Also, AFAIU, the syntax for valid citations is not defined explicitly so
> far. For example, I don't think it was discussed if any subset of Org
> objects (e.g., macros or bold text) is allowed in a citation. Is there
> any document we could merge into Org syntax document?

Hi Nicolas,

I didn’t mean to push too far ahead. I’m hoping that when we have
working code, we can work out some of these tricky issues and get the
syntax properly documented. Embedding italic text and links in
citations is, for instance, something that I think is important to
support.

Regarding org-element.el, I meant that this code would extend
org-element to generate org-element.el compatible objects for the
citations. Does that make sense? I admit that I have not read
org-element.el carefully, so I might be wrong about how to proceed.

The pandoc grammar is, I believe, more complicated than any structure
currently present in org-mode, and so the parser was a little
trickier.

best, Erik
--
Sent from my free software system <http://fsf.org/>.

Re: Citations, continued

[Richard Lawrence]

Erik Hetzner <egh@e6h.org> writes:

>> The ideal would be if citeproc would take care of proper formatting
>> of all such citation types, given just an ordered list of the fields
>> that should appear.  I don't know if CSL supports this, though; do
>> you?
>
> I’m not entirely sure what you mean. The authors of citeproc have come
> up with a huge number of styles which seem to satisfy people’s needs.
> What appears in the in-text citation is configurable, see:
>
>   http://citationstyles.org/downloads/specification-csl101-20120903.html#citation

Sorry, I wasn't clear.  What I mean is, is there a way to tell an
implementation of CSL "hey, this particular citation right here should
only contain the author (or year, or journal...) of the referenced work,
even though the citation style for this document is (e.g.) numeric?"

The link you referenced makes it seem like the <citation> element
describes how citations should be formatted for a whole document, but
maybe I don't understand it.  (Can there be multiple citation formatting
styles specified by a CSL stylesheet? or multiple stylesheets used to
format the citations in a document?)

The idea is, a citation like "As Doe says in @Doe99:title, ..." should
render like "As Doe says in /The Title/, ...", not like "As Doe says in
Doe (1999), ...", even if "@Doe99" citations in the document generally
render like the latter.  I suspect this must be possible with
citeproc/CSL, but I don't actually know, since Pandoc doesn't provide
syntax for this kind of case.
 
Best,
Richard

Re: Citations, continued

[Erik Hetzner]

On Wed,  4 Feb 2015 at 07:59:46 PST,
Richard Lawrence <richard.lawrence@berkeley.edu> wrote:
> 
> Erik Hetzner <egh@e6h.org> writes:
> 
> >> The ideal would be if citeproc would take care of proper formatting
> >> of all such citation types, given just an ordered list of the fields
> >> that should appear.  I don't know if CSL supports this, though; do
> >> you?
> >
> > I’m not entirely sure what you mean. The authors of citeproc have come
> > up with a huge number of styles which seem to satisfy people’s needs.
> > What appears in the in-text citation is configurable, see:
> >
> >   http://citationstyles.org/downloads/specification-csl101-20120903.html#citation
> 
> Sorry, I wasn't clear.  What I mean is, is there a way to tell an
> implementation of CSL "hey, this particular citation right here should
> only contain the author (or year, or journal...) of the referenced work,
> even though the citation style for this document is (e.g.) numeric?"
> 
> The link you referenced makes it seem like the <citation> element
> describes how citations should be formatted for a whole document, but
> maybe I don't understand it.  (Can there be multiple citation formatting
> styles specified by a CSL stylesheet? or multiple stylesheets used to
> format the citations in a document?)
> 
> The idea is, a citation like "As Doe says in @Doe99:title, ..." should
> render like "As Doe says in /The Title/, ...", not like "As Doe says in
> Doe (1999), ...", even if "@Doe99" citations in the document generally
> render like the latter.  I suspect this must be possible with
> citeproc/CSL, but I don't actually know, since Pandoc doesn't provide
> syntax for this kind of case.

Hi Richard,

Thanks for the reply. I believe there is some discussion of this here:

  https://groups.google.com/d/msg/pandoc-discuss/QcAnk7hsZD8/e9MYHu0BA5IJ

and probably elsewhere on the pandoc-discuss list if you search the
archives.

It’s not supported in existing citeproc implementations, but I think
it would be possible to support something like this. On the other
hand, this is also something that is easily done by hand, so I don’t
know if it’s worth the trouble.

best, Erik
--
Sent from my free software system <http://fsf.org/>.

Re: Citations, continued

[Richard Lawrence]

Erik Hetzner <egh@e6h.org> writes:

> On Wed,  4 Feb 2015 at 07:59:46 PST,
> Richard Lawrence <richard.lawrence@berkeley.edu> wrote:
 
>> The idea is, a citation like "As Doe says in @Doe99:title, ..." should
>> render like "As Doe says in /The Title/, ...", not like "As Doe says in
>> Doe (1999), ...", even if "@Doe99" citations in the document generally
>> render like the latter.  I suspect this must be possible with
>> citeproc/CSL, but I don't actually know, since Pandoc doesn't provide
>> syntax for this kind of case.

> It’s not supported in existing citeproc implementations, but I think
> it would be possible to support something like this. On the other
> hand, this is also something that is easily done by hand, so I don’t
> know if it’s worth the trouble.

Actually, I've changed my mind; it occurs to me that there is a good
argument for /not/ adopting this syntax.  First of all, the cases it is
meant to cover are not really cases of /citations/ so much as cases of
/indirections/.  Writing "@Doe99:title" instead of the title of the work
is just to aid the author; it does not produce enough information in the
output to let the reader look up the referenced work, so it isn't really
a citation.

Second, it would be difficult to get the formatting of these selected
fields right in general.  Should titles be emphasized, or put in
quotes?  When do we insert commas between fields? etc.

Instead, I think Org should provide functions that provide the same
aid to document authors without the indirection, functions like:
  - org-get-fields-from-citation
  - org-insert-fields-from-citation-at-point
  - org-replace-citation-at-point-with-fields
These would take care of looking up the desired data in the reference
database and inserting it into the Org document, where the author
could format it as desired.

Furthermore, if someone *really* needs the indirection, I think it
would be relatively straightforward to implement as an export filter,
especially if the above functions are provided.  This case should be
uncommon enough that users can be expected to handle it themselves.

So, unless someone thinks it's really important to find a workable
solution, I hereby drop this (the `field selectors') part of my
proposal.  That means citations in the inline case can remain fully
compatible with Pandoc.

Best,
Richard