wip-cite status question and feedback

wip-cite status question and feedback

[Bruce D'Arcus]

[-- Attachment #1: Type: text/plain, Size: 4215 bytes --]

Hi everyone; first post.

From what I can tell, there was a lot of activity here a few years back to
define org citation link support, which was reflected in what seems to be a
fairly developed wip-cite branch of the git repo.

But that stalled somewhere, and we still have diverse tools, using diverse
and incompatible methods to define citation links.

I see from the archive there was an encouraging thread from April of 2018
<https://lists.gnu.org/archive/html/emacs-orgmode/2018-04/msg00336.html>
(so, two years ago) that seemed to suggest merging to master was close,
with perhaps some uncertainty around syntax being the primary hold up?

My main question: how do we get this done?


I'm happy to help resolve any syntax questions, if that's the primary hold
up, though my coding skills in general are minimal.

While I’m here, some feedback and thoughts below. I think the proposed
syntax is basically sound, but maybe could be simplified.

For background, I created the Citation Style Language (CSL), and the
initial XSLT-based proof-of-concept formatter, which I used to format the
citations and bibliography in my first book.

I work in a field that has pretty diverse citation practices, and myself
work at the borders of the social sciences and humanities, which influences
my thinking on design priorities around citation technology.

I also had a hand in the pandoc citation syntax, which is consistent with
the basic design goals I had, including:


   1.

   simple as possible plain-text format, consistent with markdown
   2.

   prefix and suffix free text, page number locators, etc.
   3.

   ability to radically change output style (for example, from author-date
   to footnote-based), without changing source.


I believe the pandoc syntax achieves all of these goals.

Which brings me to the org proposal.

The only documentation I can find on the proposal is at the citeproc org
project.

https://github.com/andras-simonyi/citeproc-org#citations

If the above is correct, then:


   1.

   There are two forms of citations: short and long.
   2.

   Short is of this syntax, and can only have a single citekey as content:
   [@doe2019].
   3.

   Anything else (multiple citations, locators, prefixes and/or suffixes)
   requires the long form.
   4.

   In addition, there is a standard long form, and what the doc above calls
   “parenthetical” form of the long form.


So do I have all that right?

First question:

Why #4 above; a distinction between two different long forms?

Moving on, assuming the above is correct, let’s borrow these examples from
the pandoc citation syntax for markdown
<https://pandoc.org/MANUAL.html#citations> (which it also supports in org):

Blah blah [see @doe99, pp. 33-35; also @smith04, chap. 1].

Blah blah [@doe99, pp. 33-35, 38-39 and *passim*].

Blah blah [@smith04; @doe99].

So from a model perspective, a citation in pandoc is a list of one-or-more
cites (or whatever we want to call them), each of which has optional:

   -

   prefix
   -

   Locators (page numbers and such)
   -

   suffix


I believe this is similar to the proposed org model, with one difference:
the org proposal adds prefix and suffix for the list (the citation) as a
whole.

So the pandoc version is a flat list, and the org version has two levels.

Am I still understanding everything correctly?

So ... am I correct that in none of these cases can one use the “short”
form, and that the above would look like this converted to the proposed org
syntax?

Blah blah [cite:see @doe99 pp. 33-35; also @smith04 chap. 1].

Blah blah [cite:@doe99 pp. 33-35, 38-39 and *passim*].

Blah blah [cite:@smith04; @doe99].

Or the first one might treat the “see” as a prefix for the list, though I’m
not sure what the practical benefit of that more hierarchical modelling is.

While I do wonder if the syntax could be simplified, my main hope is that
it actually gets merged to master and deployed, to harmonize the variety of
different ways emacs tools (org-ref, ebib, pandoc, etc.) are dealing with
citation links in org currently.

[-- Attachment #2: Type: text/html, Size: 20288 bytes --]

Re: wip-cite status question and feedback

[Nicolas Goaziou]

Hello,

"Bruce D'Arcus" <bdarcus@gmail.com> writes:

> Hi everyone; first post.

Welcome!

> I see from the archive there was an encouraging thread from April of 2018
> <https://lists.gnu.org/archive/html/emacs-orgmode/2018-04/msg00336.html>
> (so, two years ago) that seemed to suggest merging to master was close,
> with perhaps some uncertainty around syntax being the primary hold up?
>
> My main question: how do we get this done?

Good question. I don't think syntax is the primary hold up. It seems to
me that the glue part between the parser and the export back-ends is
missing. If we merge wip-cite branch in master, where do we plug it? How
do Org folks use it, out of the box? It also need tooling. For example,
citations could be a link to a database.

Of course, that doesn't mean the syntax is perfect, but as it is,
I assume it theoretically goes a long way towards solving most citation
needs.

> I'm happy to help resolve any syntax questions, if that's the primary hold
> up, though my coding skills in general are minimal.

Actually, the coding part is the easiest one. I can help on that part.
Unfortunately, I do not use citations, so I cannot speak for the people
who would make use of them in Org.

> While I’m here, some feedback and thoughts below. I think the proposed
> syntax is basically sound, but maybe could be simplified.

Great!

> The only documentation I can find on the proposal is at the citeproc org
> project.
>
> https://github.com/andras-simonyi/citeproc-org#citations
>
> If the above is correct, then:
>
>    1. There are two forms of citations: short and long.
>    2. Short is of this syntax, and can only have a single citekey as content:
>       [@doe2019].
>    3. Anything else (multiple citations, locators, prefixes and/or suffixes)
>       requires the long form.
>    4. In addition, there is a standard long form, and what the doc above calls
>       “parenthetical” form of the long form.
>
> So do I have all that right?

You do.

> First question:
>
> Why #4 above; a distinction between two different long forms?

I think this is because you may want to mix two styles in the same
document: inline and parenthesised. I think the idea comes from BibTeX.

> Moving on, assuming the above is correct, let’s borrow these examples from
> the pandoc citation syntax for markdown
> <https://pandoc.org/MANUAL.html#citations> (which it also supports in org):
>
> Blah blah [see @doe99, pp. 33-35; also @smith04, chap. 1].
>
> Blah blah [@doe99, pp. 33-35, 38-39 and *passim*].
>
> Blah blah [@smith04; @doe99].
>
> So from a model perspective, a citation in pandoc is a list of one-or-more
> cites (or whatever we want to call them), each of which has optional:
>
>    - prefix
>    - Locators (page numbers and such)
>    - suffix
>
> I believe this is similar to the proposed org model, with one difference:
> the org proposal adds prefix and suffix for the list (the citation) as a
> whole.
>
> So the pandoc version is a flat list, and the org version has two levels.
>
> Am I still understanding everything correctly?

Org syntax has no notion of locators. Is that an issue for later
interpretation?

It just uses optional prefix and suffix around keys. The "(prefix) key
(suffix)" is called a citation reference. A full citation may contain
one ore more citation references, separated by semicolons.

You are right about the global prefix and suffix. I'm not sure why they
are here, but were probably suggested at some point.

If there is no practical use for them, we can also drop them.

> So ... am I correct that in none of these cases can one use the “short”
> form, and that the above would look like this converted to the proposed org
> syntax?

True. I think the short form is just a quick reference that should be
expanded in a predefined style, e.g., "(author, year)".

> Blah blah [cite:see @doe99 pp. 33-35; also @smith04 chap. 1].
>
> Blah blah [cite:@doe99 pp. 33-35, 38-39 and *passim*].
>
> Blah blah [cite:@smith04; @doe99].

Actually, barring the "cite:" prefix, you can use the exact same syntax
as Pandoc, i.e.:

  [cite:see @doe99, pp. 33-35; also @smith04, chap. 1]
  [cite:@doe99, pp. 33-35, 38-39 and *passim*]
  [cite:@smith04; @doe99]

Only the semicolon is meaningful in there, not the commas. Also, for
readability, spacing after the "cite:" prefix is ignored, so one can
write, e.g.,

  [cite: see @doe99, pp. 33-35; also @smith04, chap. 1]

> Or the first one might treat the “see” as a prefix for the list, though I’m
> not sure what the practical benefit of that more hierarchical modelling is.

No, IIRC, the common prefix is the part before the first semicolon, but
only if that part contains no citation key. IOW, in the example above,
"see " is the prefix of the first citation reference, not a global
prefix.

> While I do wonder if the syntax could be simplified, my main hope is that
> it actually gets merged to master and deployed, to harmonize the variety of
> different ways emacs tools (org-ref, ebib, pandoc, etc.) are dealing with
> citation links in org currently.

If the syntax is practical, we could merge it, yes. Maybe that's the way
forward, after all. But then? What library is going to use it?

Org Ref is well established and is unlikely to switch to the new syntax.
It would be nice if both syntax could converge at some point, though.
The point of the new syntax is to make it easier for this kind of
extension.

"citeproc-el" is not included in Emacs, so integrating "citeproc-org" in
Org may not be helpful at this point. Since Org is bundled with Emacs,
it cannot use external libraries.

WDYT?

Regards,

-- 
Nicolas Goaziou

Re: wip-cite status question and feedback

[Bruce D'Arcus]

Thank you for this thorough reply, Nicholas!

See below.

On Tue, Apr 7, 2020 at 1:51 PM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:
>
> Hello,
>
> "Bruce D'Arcus" <bdarcus@gmail.com> writes:
>
> > Hi everyone; first post.
>
> Welcome!
>
> > I see from the archive there was an encouraging thread from April of 2018
> > <https://lists.gnu.org/archive/html/emacs-orgmode/2018-04/msg00336.html>
> > (so, two years ago) that seemed to suggest merging to master was close,
> > with perhaps some uncertainty around syntax being the primary hold up?
> >
> > My main question: how do we get this done?
>
> Good question. I don't think syntax is the primary hold up. It seems to
> me that the glue part between the parser and the export back-ends is
> missing. If we merge wip-cite branch in master, where do we plug it? How
> do Org folks use it, out of the box? It also need tooling. For example,
> citations could be a link to a database.

OK, gotcha. That does explain the "hold up."

I'll come back to this below, but for now simply say that ideal would
probably be a default formatting library and database, with room to
replace with other options.

For the database, it would make to do as existing tools do; allow
users to define default bib(la)tex files.

[snip]

> Actually, barring the "cite:" prefix, you can use the exact same syntax
> as Pandoc, i.e.:
>
>   [cite:see @doe99, pp. 33-35; also @smith04, chap. 1]
>   [cite:@doe99, pp. 33-35, 38-39 and *passim*]
>   [cite:@smith04; @doe99]
>
> Only the semicolon is meaningful in there, not the commas. Also, for
> readability, spacing after the "cite:" prefix is ignored, so one can
> write, e.g.,
>
>   [cite: see @doe99, pp. 33-35; also @smith04, chap. 1]

OK, that's good.

Note that in CSL processors, the locators are meaningful key-values,
basically; not plain text strings.

This is because different citation styles have different requirements
for rendering them.

Description from the spec:

a cite-specific pinpointer within the item (e.g. a page number within
a book, or a volume in a multi-volume work). Must be accompanied in
the input data by a label indicating the locator type (see the
Locators term list), which determines which term is rendered by
cs:label when the “locator” variable is selected.

And this is the list of CSL locators keys:

http://docs.citationstyles.org/en/stable/specification.html#locators

> > Or the first one might treat the “see” as a prefix for the list, though I’m
> > not sure what the practical benefit of that more hierarchical modelling is.
>
> No, IIRC, the common prefix is the part before the first semicolon, but
> only if that part contains no citation key. IOW, in the example above,
> "see " is the prefix of the first citation reference, not a global
> prefix.
>
> > While I do wonder if the syntax could be simplified, my main hope is that
> > it actually gets merged to master and deployed, to harmonize the variety of
> > different ways emacs tools (org-ref, ebib, pandoc, etc.) are dealing with
> > citation links in org currently.
>
> If the syntax is practical, we could merge it, yes. Maybe that's the way
> forward, after all. But then? What library is going to use it?
>
> Org Ref is well established and is unlikely to switch to the new syntax.
> It would be nice if both syntax could converge at some point, though.
> The point of the new syntax is to make it easier for this kind of
> extension.

While I of course can't speak for John, I would hope and expect org
ref to support the new syntax.

I would hope and expect the same of other packages (like ebib) that
use their own custom syntax.

> "citeproc-el" is not included in Emacs, so integrating "citeproc-org" in
> Org may not be helpful at this point. Since Org is bundled with Emacs,
> it cannot use external libraries.

I hadn't considered that.

I'm thinking two options:

1. just have a super simple citation export formatter, with (per
above) room to configure an external one

2. while I don't know its status, include citeproc-org in org and emacs

I'd say if citeproc-org is far along and free of substantial bugs, 2
might make sense. I am, of course, biased, but CSL's been widely
deployed in the wild for more than a decade, and there are thousands
of available styles.

Otherwise, and more likely, 1 would be the best path; users get basic
output formatting for citations, but then can plug in more elaborate
tools (citeproc-org, citeproc-hs*, etc.) if they need them.

Obviously, if a user is exporting to LaTeX, the formatting is
straightforward, because you're just exporting to the latex cite code,
as pandoc does (though it also has an option to use citeproc to
produce "cooked" formatted output in latex).

For HTML, the output doesn't need to be anything fancy.

So the #1 path is the more iterative approach to implementing the functionality.

WDY about that?

Bruce

* citeproc-hs is a newer effort, funded by zotero, to create a very
high performance, fully compliant, CSL implementation, that is easy to
embed and use pretty much anywhere; I haven't checked on the status
recently, but I'm optimistic this will be the best option long-term

Bruce

Re: wip-cite status question and feedback

[Bruce D'Arcus]

Sorry for the typos, and also a more important mistake: a didn't mean
citeproc-hs, but citeproc-rs.

https://github.com/cormacrelf/citeproc-rs

On Tue, Apr 7, 2020 at 2:27 PM Bruce D'Arcus <bdarcus@gmail.com> wrote:
>
> Thank you for this thorough reply, Nicholas!
>
> See below.
>
> On Tue, Apr 7, 2020 at 1:51 PM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:
> >
> > Hello,
> >
> > "Bruce D'Arcus" <bdarcus@gmail.com> writes:
> >
> > > Hi everyone; first post.
> >
> > Welcome!
> >
> > > I see from the archive there was an encouraging thread from April of 2018
> > > <https://lists.gnu.org/archive/html/emacs-orgmode/2018-04/msg00336.html>
> > > (so, two years ago) that seemed to suggest merging to master was close,
> > > with perhaps some uncertainty around syntax being the primary hold up?
> > >
> > > My main question: how do we get this done?
> >
> > Good question. I don't think syntax is the primary hold up. It seems to
> > me that the glue part between the parser and the export back-ends is
> > missing. If we merge wip-cite branch in master, where do we plug it? How
> > do Org folks use it, out of the box? It also need tooling. For example,
> > citations could be a link to a database.
>
> OK, gotcha. That does explain the "hold up."
>
> I'll come back to this below, but for now simply say that ideal would
> probably be a default formatting library and database, with room to
> replace with other options.
>
> For the database, it would make to do as existing tools do; allow
> users to define default bib(la)tex files.
>
> [snip]
>
> > Actually, barring the "cite:" prefix, you can use the exact same syntax
> > as Pandoc, i.e.:
> >
> >   [cite:see @doe99, pp. 33-35; also @smith04, chap. 1]
> >   [cite:@doe99, pp. 33-35, 38-39 and *passim*]
> >   [cite:@smith04; @doe99]
> >
> > Only the semicolon is meaningful in there, not the commas. Also, for
> > readability, spacing after the "cite:" prefix is ignored, so one can
> > write, e.g.,
> >
> >   [cite: see @doe99, pp. 33-35; also @smith04, chap. 1]
>
> OK, that's good.
>
> Note that in CSL processors, the locators are meaningful key-values,
> basically; not plain text strings.
>
> This is because different citation styles have different requirements
> for rendering them.
>
> Description from the spec:
>
> a cite-specific pinpointer within the item (e.g. a page number within
> a book, or a volume in a multi-volume work). Must be accompanied in
> the input data by a label indicating the locator type (see the
> Locators term list), which determines which term is rendered by
> cs:label when the “locator” variable is selected.
>
> And this is the list of CSL locators keys:
>
> http://docs.citationstyles.org/en/stable/specification.html#locators
>
> > > Or the first one might treat the “see” as a prefix for the list, though I’m
> > > not sure what the practical benefit of that more hierarchical modelling is.
> >
> > No, IIRC, the common prefix is the part before the first semicolon, but
> > only if that part contains no citation key. IOW, in the example above,
> > "see " is the prefix of the first citation reference, not a global
> > prefix.
> >
> > > While I do wonder if the syntax could be simplified, my main hope is that
> > > it actually gets merged to master and deployed, to harmonize the variety of
> > > different ways emacs tools (org-ref, ebib, pandoc, etc.) are dealing with
> > > citation links in org currently.
> >
> > If the syntax is practical, we could merge it, yes. Maybe that's the way
> > forward, after all. But then? What library is going to use it?
> >
> > Org Ref is well established and is unlikely to switch to the new syntax.
> > It would be nice if both syntax could converge at some point, though.
> > The point of the new syntax is to make it easier for this kind of
> > extension.
>
> While I of course can't speak for John, I would hope and expect org
> ref to support the new syntax.
>
> I would hope and expect the same of other packages (like ebib) that
> use their own custom syntax.
>
> > "citeproc-el" is not included in Emacs, so integrating "citeproc-org" in
> > Org may not be helpful at this point. Since Org is bundled with Emacs,
> > it cannot use external libraries.
>
> I hadn't considered that.
>
> I'm thinking two options:
>
> 1. just have a super simple citation export formatter, with (per
> above) room to configure an external one
>
> 2. while I don't know its status, include citeproc-org in org and emacs
>
> I'd say if citeproc-org is far along and free of substantial bugs, 2
> might make sense. I am, of course, biased, but CSL's been widely
> deployed in the wild for more than a decade, and there are thousands
> of available styles.
>
> Otherwise, and more likely, 1 would be the best path; users get basic
> output formatting for citations, but then can plug in more elaborate
> tools (citeproc-org, citeproc-hs*, etc.) if they need them.
>
> Obviously, if a user is exporting to LaTeX, the formatting is
> straightforward, because you're just exporting to the latex cite code,
> as pandoc does (though it also has an option to use citeproc to
> produce "cooked" formatted output in latex).
>
> For HTML, the output doesn't need to be anything fancy.
>
> So the #1 path is the more iterative approach to implementing the functionality.
>
> WDY about that?
>
> Bruce
>
> * citeproc-hs is a newer effort, funded by zotero, to create a very
> high performance, fully compliant, CSL implementation, that is easy to
> embed and use pretty much anywhere; I haven't checked on the status
> recently, but I'm optimistic this will be the best option long-term
>
> Bruce

Re: wip-cite status question and feedback

[Joost Kremers]

On Tue, Apr 07 2020, Bruce D'Arcus wrote:
> While I of course can't speak for John, I would hope and expect 
> org
> ref to support the new syntax.
>
> I would hope and expect the same of other packages (like ebib) 
> that
> use their own custom syntax.

Ebib maintainer here. I would certainly include support in Ebib 
for any citation syntax that Org would adopt. 

What would help, BTW, is if there's an easy way to find out what 
the bibliography file or files are that are associated with the 
current Org buffer.

-- 
Joost Kremers
Life has its moments

Re: wip-cite status question and feedback

[Bruce D'Arcus]

On Tue, Apr 7, 2020 at 5:13 PM Joost Kremers <joostkremers@fastmail.fm> wrote:

> On Tue, Apr 07 2020, Bruce D'Arcus wrote:
> > While I of course can't speak for John, I would hope and expect
> > org
> > ref to support the new syntax.
> >
> > I would hope and expect the same of other packages (like ebib)
> > that
> > use their own custom syntax.
>
> Ebib maintainer here. I would certainly include support in Ebib
> for any citation syntax that Org would adopt.
>
> What would help, BTW, is if there's an easy way to find out what
> the bibliography file or files are that are associated with the
> current Org buffer.

I guess the simplest and most flexible option would be to specify the
bib file in the header?

So if present use that; otherwise use the global bib file(s) specified
in the init?

Bruce

Re: wip-cite status question and feedback

[Joost Kremers]

On Wed, Apr 08 2020, Bruce D'Arcus wrote:
> On Tue, Apr 7, 2020 at 5:13 PM Joost Kremers 
> <joostkremers@fastmail.fm> wrote:
>> What would help, BTW, is if there's an easy way to find out 
>> what
>> the bibliography file or files are that are associated with the
>> current Org buffer.
>
> I guess the simplest and most flexible option would be to 
> specify the
> bib file in the header?
>
> So if present use that; otherwise use the global bib file(s) 
> specified
> in the init?

That sounds like a good approach, yes. I don't know if Org would 
want to support having different bib files for different headers 
(i.e., be able to specify bib files in a :PROPERTIES: block). I 
believe that would be a bad idea, but who knows?

For my purpose, the best thing would be to have a function that 
returns a list of bibliography files for the current buffer (or 
header, if you want to go that way). That way, I wouldn't have to 
come up with my own (bug-prone) way of figuring out what bib files 
a user expects a particular Org file to use. Since (presumably) 
such a function is going to be needed anyway for exporting, it 
would be a simple matter of making that function public (i.e., 
name it with one dash, not two).

Thanks,

-- 
Joost Kremers
Life has its moments

Re: wip-cite status question and feedback

[Nicolas Goaziou]

Hello,

"Bruce D'Arcus" <bdarcus@gmail.com> writes:

> Note that in CSL processors, the locators are meaningful key-values,
> basically; not plain text strings.

OK, but it is enough for Org to feed a CSL processor with, e.g.,

  key    -> "@doe99"
  prefix -> "see "
  suffix -> ", pp. 33-35"

Then CSL processor does its job to extract whatever information it
needs. Am I right?

> While I of course can't speak for John, I would hope and expect org
> ref to support the new syntax.
>
> I would hope and expect the same of other packages (like ebib) that
> use their own custom syntax.

I hope so, too. But it would help if developers could chime in and tell
what API Org should provide. This is particularly important since Org
will only provide the API, without any specialized implementation. More
on this at the end of the message.

> 1. just have a super simple citation export formatter, with (per
>    above) room to configure an external one

Sure. Org should provide default export for citations in LaTeX, ASCII,
HTML, Texinfo and ODT. Suggestions are more than welcome.

> 2. while I don't know its status, include citeproc-org in org and
> emacs

I think citeproc-el should ship with Emacs, but I cannot speak for
Emacs; it would be nice to ask Emacs developers about it.

> I'd say if citeproc-org is far along and free of substantial bugs, 2
> might make sense. I am, of course, biased, but CSL's been widely
> deployed in the wild for more than a decade, and there are thousands
> of available styles.
>
> Otherwise, and more likely, 1 would be the best path; users get basic
> output formatting for citations, but then can plug in more elaborate
> tools (citeproc-org, citeproc-hs*, etc.) if they need them.

Option 2 makes a lot of sense if citeproc-el is integrated with Emacs.
Until them, I agree that option 1 is the way to go at the moment.

> WDY about that?

Sounds like a plan. Let me summarize it a bit :

1. [ ] Finalize Org citation syntax. It is mostly good, but we need to
   decide about the following points:

   - [ ] Should it include both "(cite):" and "cite:", i.e., does it
     make sense to provide a (very limited) style specification piece
     wise?

   - [ ] Should it include /global/ prefix and suffix?

   - [ ] Should we keep the short specification, i.e., "[@key]"?

   - [ ] What kind of markup do we allow in a citation? At the moment
     the exhaustive list is: bold, code, entity, italic, latex-fragment,
     strike-through, subscript, superscript, underline, verbatim and
     line breaks.

2. [ ] Decide about API Org should provide for it to be useful. Here are
   some low hanging fruits:

   - [ ] List all ".bib" files associated to the document,

   - [ ] List all citations,

   - [ ] Return citation key at point, if any.

   - Anything else?

   Moreover, we need to decide about how external processors could plug
   into the export framework.

   - [ ] For example, it could be a simple variable bound to a list of
     functions. Each function accepts three arguments: the export
     back-end, as a symbol, the full citation, as a list of triplets
     (key, prefix, suffix) along with global prefix/suffix, and the
     usual INFO communication channel. Does it need more?

   - [ ] Also, the prefix/suffix may contain some Org markup, so this
     needs to be also processed. Should it happen before, or after the
     external processor does its job? I.e., should the function
     translate into Org or target format?

3. [ ] Specify the kind of basic translation that be done out of the
   box? Ideally, every non-derived back-end should do something, even
   basic. Therefore, we need to propose some translation pattern for
   each of the following:

   - [ ] ASCII

   - [ ] HTML

   - [ ] LaTeX

   - [ ] ODT

   - [ ] Texinfo

4. Anything else?


We need help, or at least opinion, from actual implementors to go
further. I'm CC'ing some of them.

I can take care of the implementation (with some time, my plate is full
at the moment), but I'm mostly incompetent about design issues.

So, what about ticking off some items?

Regards,

-- 
Nicolas Goaziou

Re: wip-cite status question and feedback

[Bruce D'Arcus]

On Wed, Apr 8, 2020 at 5:32 AM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:
>
> Hello,
>
> "Bruce D'Arcus" <bdarcus@gmail.com> writes:
>
> > Note that in CSL processors, the locators are meaningful key-values,
> > basically; not plain text strings.
>
> OK, but it is enough for Org to feed a CSL processor with, e.g.,
>
>   key    -> "@doe99"
>   prefix -> "see "
>   suffix -> ", pp. 33-35"
>
> Then CSL processor does its job to extract whatever information it
> needs. Am I right?

On this, I would defer to András and Albert (who maintains the pandoc
org code, I believe).

Bruce



Bruce

Re: wip-cite status question and feedback

[John Kitchin]

[-- Attachment #1: Type: text/plain, Size: 3432 bytes --]

org-ref relies very heavily on the link functionality to provide actions on
a cite, e.g. to open the pdf, or url, allow sorting, to change the cite
color when it is a bad key, etc If the new syntax also has that capability,
e.g. through font-lock, then I would consider integrating it into org-ref,
but if not I think it would be a big regression in org-ref functionality.

If I were to dream, each cite would have text-properties that include the
key (so it is easy to get the key at point and do something with info in
the corresponding database), and a help-echo function that could be
user-defined, a face function that could be user-defined, a user-defined
keymap, and some properties that define the bounds of the cite. While at
it, maybe it is a good idea to allow a custom display, so one could toggle
between a short cite (e.g. number or author year) and the full cites. These
do not need to be part of the implementation, but if they were possible
from the implementation it would be a lot more useful for something like
org-ref.

It would be a gain in quality of export, especially for non-LaTeX documents
though, if there was an integrated citeproc.

For the bibliography you need to support a few variants, IMO. One is
bibtex-like, where you specify the source of the bibliography(ies) in the
place where you want it to appear. The other is biblatex like, where the
bibliography(ies) can be specified in a header or as properties, and you
have another way to specify where in the document you want the bibliography
to be.

It should also be possible to have no bibliography, but the correct
citations. And it should be possible to have the bibliography go to another
file.

Finally, the most common thing I do is use a default bibliography that is
defined in a variable in my init file. This lets me put citations in
org-files conveniently, but I almost never export these as they are usually
just notes.

If that all seemed possible, most likely it would make sense to start a new
generation of org-ref that largely eliminated the links. I would probably
still have to keep label and the ref links. There is not currently a way to
reference equations otherwise. Tables and Figures seem ok with native org
links.

A new org-ref wouldn't happen fast, I guess it would be a year long
project. But a clean slate would have some advantages to clean up and
consolidate some things.

John

-----------------------------------
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu



On Wed, Apr 8, 2020 at 8:19 AM Bruce D'Arcus <bdarcus@gmail.com> wrote:

> On Wed, Apr 8, 2020 at 5:32 AM Nicolas Goaziou <mail@nicolasgoaziou.fr>
> wrote:
> >
> > Hello,
> >
> > "Bruce D'Arcus" <bdarcus@gmail.com> writes:
> >
> > > Note that in CSL processors, the locators are meaningful key-values,
> > > basically; not plain text strings.
> >
> > OK, but it is enough for Org to feed a CSL processor with, e.g.,
> >
> >   key    -> "@doe99"
> >   prefix -> "see "
> >   suffix -> ", pp. 33-35"
> >
> > Then CSL processor does its job to extract whatever information it
> > needs. Am I right?
>
> On this, I would defer to András and Albert (who maintains the pandoc
> org code, I believe).
>
> Bruce
>
>
>
> Bruce
>

[-- Attachment #2: Type: text/html, Size: 4421 bytes --]

Re: wip-cite status question and feedback

[Bruce D'Arcus]

On Wed, Apr 8, 2020 at 9:39 AM John Kitchin <jkitchin@andrew.cmu.edu> wrote:

> If I were to dream, each cite would have text-properties that include the key (so it is easy to get the key at point and do something with info in the corresponding database), and a help-echo function that could be user-defined, a face function that could be user-defined, a user-defined keymap, and some properties that define the bounds of the cite. While at it, maybe it is a good idea to allow a custom display, so one could toggle between a short cite (e.g. number or author year) and the full cites.

+1!!!

Bruce

Re: wip-cite status question and feedback

[Albert Krewinkel]

Hello,

Bruce D'Arcus writes:

> On Wed, Apr 8, 2020 at 5:32 AM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:
>> "Bruce D'Arcus" <bdarcus@gmail.com> writes:
>>
>> > Note that in CSL processors, the locators are meaningful key-values,
>> > basically; not plain text strings.
>>
>> OK, but it is enough for Org to feed a CSL processor with, e.g.,
>>
>>   key    -> "@doe99"
>>   prefix -> "see "
>>   suffix -> ", pp. 33-35"
>>
>> Then CSL processor does its job to extract whatever information it
>> needs. Am I right?
>
> On this, I would defer to András and Albert (who maintains the pandoc
> org code, I believe).

Yes, that is correct. Pandoc parses `prefix` and `suffix` as markup, so
the pp in ", /pp/. 33-35." would be italicized.

--
Albert Krewinkel
GPG: 8eed e3e2 e8c5 6f18 81fe  e836 388d c0b2 1f63 1124

Re: wip-cite status question and feedback

[Bruce D'Arcus]

On Thu, Apr 9, 2020 at 3:38 AM Albert Krewinkel <albert@zeitkraut.de> wrote:
>
> Hello,
>
> Bruce D'Arcus writes:
>
> > On Wed, Apr 8, 2020 at 5:32 AM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:
> >> "Bruce D'Arcus" <bdarcus@gmail.com> writes:
> >>
> >> > Note that in CSL processors, the locators are meaningful key-values,
> >> > basically; not plain text strings.
> >>
> >> OK, but it is enough for Org to feed a CSL processor with, e.g.,
> >>
> >>   key    -> "@doe99"
> >>   prefix -> "see "
> >>   suffix -> ", pp. 33-35"
> >>
> >> Then CSL processor does its job to extract whatever information it
> >> needs. Am I right?
> >
> > On this, I would defer to András and Albert (who maintains the pandoc
> > org code, I believe).
>
> Yes, that is correct. Pandoc parses `prefix` and `suffix` as markup, so
> the pp in ", /pp/. 33-35." would be italicized.

And Pandoc parses that "suffix" into two parts: the locators, and the
suffix proper.

In this example, from the CSL perspective, there is no suffix; just a locator.

So to answer Nicholes' "CSL processor does its job" question, the
answer would be "yes'; correct?

The only thing I would add on that is if the org citation syntax has
no notion of locators, and simply folds them into the suffix, that
could cause problems for users. This may or may not be a problem to
worry about now; I don't know.

Maybe if org and emacs were to include citeproc-org, you support it,
and otherwise keep simpler?

FWIW, I asked about this on the CSL developers subforum, and got a
very helpful reply from Denis Maier, first discussing the pandoc
citation model and parsing, and then the org citation syntax and
global affixes it supports.

https://discourse.citationstyles.org/t/ideal-api-to-expose-for-csl-processor/1622/3
https://discourse.citationstyles.org/t/ideal-api-to-expose-for-csl-processor/1622/4

He suggests considering adding support for the global prefix to CSL 1.2.

Bruce

Re: wip-cite status question and feedback

[Bruce D'Arcus]

On Thu, Apr 9, 2020 at 5:30 AM Bruce D'Arcus <bdarcus@gmail.com> wrote:

...

> FWIW, I asked about this on the CSL developers subforum, and got a
> very helpful reply from Denis Maier, first discussing the pandoc
> citation model and parsing, and then the org citation syntax and
> global affixes it supports.
>
> https://discourse.citationstyles.org/t/ideal-api-to-expose-for-csl-processor/1622/3
> https://discourse.citationstyles.org/t/ideal-api-to-expose-for-csl-processor/1622/4
>
> He suggests considering adding support for the global prefix to CSL 1.2.

PS - he also suggests here that it looks like the global affixes comes
from biblatex.

I just confirmed that. From the manual (section 3.8.3):

\cites(multiprenote)(multipostnote)[prenote][postnote]{key}...[prenote][postnote]{key}

I have a lot of respect for biblatex, and know it's designed for
handling more complex real world styles.


Bruce

Re: wip-cite status question and feedback

[Bruce D'Arcus]

Another question on the syntax.

Does it support something like this in the pandoc syntax?

Doe [-@doe19] argues ...

The minus sign signals to suppress the author rendering, so that you
end up with:

Doe (2019)

... instead of:

Doe (Doe, 2019)

I know latex works differently, but this is simple, and very flexible.
--
Bruce

Re: wip-cite status question and feedback

[Nicolas Goaziou]

Hello,

"Bruce D'Arcus" <bdarcus@gmail.com> writes:

> Another question on the syntax.
>
> Does it support something like this in the pandoc syntax?
>
> Doe [-@doe19] argues ...

No it doesn't. That seems reasonable to include. I add it to my TODO
list.

> The minus sign signals to suppress the author rendering, so that you
> end up with:
>
> Doe (2019)
>
> ... instead of:
>
> Doe (Doe, 2019)
>
> I know latex works differently, but this is simple, and very flexible.

Speaking of which, I read about the "citation modes", which may be
preferred over "suppress author". What are these citation modes? How do
you combine them if "suppress author" is one of the "citation modes". Do
citation modes apply on a reference, or a complete citation, which
possibly contains multiple references?

Also, on a related subject, now I remember [@key] is not a shorthand for
[cite:@key]; the former is meant as "author in text" style.

Regards,

-- 
Nicolas Goaziou

Re: wip-cite status question and feedback

[Bruce D'Arcus]

On Thu, Apr 9, 2020 at 7:17 PM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:
>
> Hello,
>
> "Bruce D'Arcus" <bdarcus@gmail.com> writes:
>
> > Another question on the syntax.
> >
> > Does it support something like this in the pandoc syntax?
> >
> > Doe [-@doe19] argues ...
>
> No it doesn't. That seems reasonable to include. I add it to my TODO
> list.
>
> > The minus sign signals to suppress the author rendering, so that you
> > end up with:
> >
> > Doe (2019)
> >
> > ... instead of:
> >
> > Doe (Doe, 2019)
> >
> > I know latex works differently, but this is simple, and very flexible.
>
> Speaking of which, I read about the "citation modes", which may be
> preferred over "suppress author". What are these citation modes?

Pandoc's code for its citation modes is here

https://github.com/jgm/pandoc/blob/f2b337768e95c8903f65e597f7d8cd81938dacc3/src/Text/Pandoc/Readers/Org/Inlines.hs#L336

And the part that reflects its citation syntax looks to be this:

https://github.com/jgm/pandoc/blob/f2b337768e95c8903f65e597f7d8cd81938dacc3/src/Text/Pandoc/Readers/Org/Inlines.hs#L362

> How do you combine them if "suppress author" is one of the "citation modes". Do
> citation modes apply on a reference, or a complete citation, which
> possibly contains multiple references?

Here's some pandoc markdown examples:

#+BEGIN_SRC
Blah [see @doeA, pp. 33-35; also @doeB, chap. 1].

Blah [@doeA, pp. 33-35, 38-39 and *passim*].

Blah [@doeA; @doeB].

Blah [-@doeA].

Blah [-@doeA; -@doeB].

Blah [-@doeA; @doeB].

@doeA and -@doeB
#+END_SRC

When exported to HTML, we get:

#+BEGIN_EXAMPLE
Blah (see Doe 2018, 33–35, also 2017, chap. 1).

Blah (Doe 2018, 33–35, 38–39 and passim).

Blah (Doe 2018, 2017).

Blah (2018).

Blah (2018, 2017).

Blah (2018, 2017).

Doe (2018) and (2017)
#+END_EXAMPLE

The second to last one is interesting, because my understanding was
the suppress_author flag applies to the individual cite references,
rather than the complete citation.

But in the rendering, it's clear pandoc (I think appropriately)
applies it to both cites, and so the whole citation.

> Also, on a related subject, now I remember [@key] is not a shorthand for
> [cite:@key]; the former is meant as "author in text" style.

OK, sounds good.

Bruce

Re: wip-cite status question and feedback

[Albert Krewinkel]

Bruce D'Arcus writes:

> On Thu, Apr 9, 2020 at 3:38 AM Albert Krewinkel <albert@zeitkraut.de> wrote:
>> Bruce D'Arcus writes:
>>
>> > On Wed, Apr 8, 2020 at 5:32 AM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:
>> >> "Bruce D'Arcus" <bdarcus@gmail.com> writes:
>> >>
>> >> > Note that in CSL processors, the locators are meaningful key-values,
>> >> > basically; not plain text strings.
>> >>
>> >> OK, but it is enough for Org to feed a CSL processor with, e.g.,
>> >>
>> >>   key    -> "@doe99"
>> >>   prefix -> "see "
>> >>   suffix -> ", pp. 33-35"
>> >>
>> >> Then CSL processor does its job to extract whatever information it
>> >> needs. Am I right?
>> >
>> > On this, I would defer to András and Albert (who maintains the pandoc
>> > org code, I believe).
>>
>> Yes, that is correct. Pandoc parses `prefix` and `suffix` as markup, so
>> the pp in ", /pp/. 33-35." would be italicized.
>
> And Pandoc parses that "suffix" into two parts: the locators, and the
> suffix proper.

"Pandoc", as used above, is probably meant to include pandoc-citeproc,
the CSL processor shipped with pandoc. Pandoc itself does not process
the prefix/suffix values further, but leaves that to pandoc-citeproc:

https://github.com/jgm/pandoc-citeproc/blob/d7eb5fd15980ba40d7bffd6d2f031a229881668f/src/Text/CSL/Pandoc.hs#L471

> In this example, from the CSL perspective, there is no suffix; just a locator.

From pandoc's point of view, there is no difference. Org-mode could
probably take a similar stance.

> So to answer Nicholes' "CSL processor does its job" question, the
> answer would be "yes'; correct?

Agree.


--
Albert Krewinkel
GPG: 8eed e3e2 e8c5 6f18 81fe  e836 388d c0b2 1f63 1124

Re: wip-cite status question and feedback

[denis.maier.lists]

[-- Attachment #1: Type: text/plain, Size: 2354 bytes --]

Hi,

very good to see these things are getting discussed again. (Back then, I have been following the citation syntax discussion from a distance rather then participating actively, so that's my first post here.)

It would be great if Org had a closer integration with a Citeproc. This would be a huge improvement for those who use Emacs and Org for academic writing.

Anyway, concerning this:

> Speaking of which, I read about the "citation modes", which may be preferred over "suppress author". What are these citation modes? How do you combine them if "suppress author" is one of the "citation modes".

Bruce has already sent a link to Pandoc's org-mode Reader where you can find the citation modes. The standard (markdown) implementation can be found here: https://github.com/jgm/pandoc/blob/f2b337768e95c8903f65e597f7d8cd81938dacc3/src/Text/Pandoc/Readers/Markdown.hs

Anyway, currently there are basically three modes:
- NormalCitation
- SuppressAuthor
- AuthorInText

At the moment, there is no way to combine modes.

A simple markdown sample:

#+BEGIN_SRC
[@doe]

[-@doe]

@doe argues ...

-@doe [23]
#+END_SRC

Gives:

#+BEGIN_EXAMPLE
[Para [Cite [Citation {citationId = "doe", citationPrefix = [], citationSuffix = [], citationMode = NormalCitation, citationNoteNum = 0, citationHash = 0}] [Str "[@doe]"]]
,Para [Cite [Citation {citationId = "doe", citationPrefix = [], citationSuffix = [], citationMode = SuppressAuthor, citationNoteNum = 0, citationHash = 0}] [Str "[-@doe]"]]
,Para [Cite [Citation {citationId = "doe", citationPrefix = [], citationSuffix = [], citationMode = AuthorInText, citationNoteNum = 0, citationHash = 0}] [Str "@doe"],Space,Str "argues",Space,Str "\8230"]
,Para [Cite [Citation {citationId = "doe", citationPrefix = [], citationSuffix = [Str "23"], citationMode = SuppressAuthor, citationNoteNum = 0, citationHash = 0}] [Str "@doe",Space,Str "[23]"]]
#+END_EXAMPLE

So:
[@doe] => NormalCitation
[-@doe] => SuppressAuthor
@doe argues ... => AuthorInText
-@doe [23] => SuppressAuthor

I do admit that this last example is a bit pointless, but it shows a point. => SuppressAuthor takes precedence over the other modes in both cases.

Depending on CSL's future development there might well be other citations modes. I personally think there's much to learn from biblatex, but that's a different debate...

Best,
Denis



[-- Attachment #2: Type: text/html, Size: 4904 bytes --]

Re: wip-cite status question and feedback

[Bruce D'Arcus]

On Fri, Apr 10, 2020 at 5:30 AM <denis.maier.lists@mailbox.org> wrote:

...

> [Para [Cite [Citation {citationId = "doe", citationPrefix = [], citationSuffix = [], citationMode = NormalCitation, citationNoteNum = 0, citationHash = 0}] [Str "[@doe]"]]

I just wanted to add a quick note on this, in case anyone is wondering:

That `citationNoteNum` key there would be used internally where the
style "class" was "note-based"; as in, where the processor was
generating citations as footnotes.

In the CSL implementations I'm aware of (pandoc-citeproc with markdown
and org docs, or Zotero with LibreWrtier or MS Word), one can go
back-and-forth between "note-based" and "in-text" without modifying
the markdown source. The code is smart enough to know the different
between a citation in the text (that needs to be "footnoted"), and a
citation already in a footnote (which simply renders to be rendered,
without adding a new footnote), and how to get the correct footnote
numbering across the output document.

Bruce

Re: wip-cite status question and feedback

[Nicolas Goaziou]

Hello,

denis.maier.lists@mailbox.org writes:

> Bruce has already sent a link to Pandoc's org-mode Reader where you
> can find the citation modes. The standard (markdown) implementation
> can be found here:
> https://github.com/jgm/pandoc/blob/f2b337768e95c8903f65e597f7d8cd81938dacc3/src/Text/Pandoc/Readers/Markdown.hs
>

Thank you.

> Anyway, currently there are basically three modes:
> - NormalCitation
> - SuppressAuthor
> - AuthorInText

[...]

> So:
> [@doe] => NormalCitation
> [-@doe] => SuppressAuthor
> @doe argues ... => AuthorInText
> -@doe [23] => SuppressAuthor

Let's assume Org implements SuppressAuthor as "-@doe", so far it has:

| Syntax                    | Mode                            | Sample output |
|---------------------------+---------------------------------+---------------|
| @doe or [cite:@doe]       | AuthorInText                    | Doe (2020)    |
| -@doe or [cite:-@doe]     | AuthorInText + SuppressAuthor   | (2020)        |
| [@doe] or [(cite):@doe]   | NormalCitation                  | (Doe, 2020)   |
| [-@doe] or [(cite):-@doe] | NormalCitation + SuppressAuthor | (2020)        |

So, we could mix suppress author with some other style, although this
seems useless for the time being.

Current syntax is not great because it is ad-hoc, and not very
future-proof, if new modes appear. Also, the "(cite)" key is a bit
mouthful, "citep" might have been prettier. Anyway, is it complete
enough?

Also, there is some ambiguity between modes applying reference-wise,
i.e., "suppress author", and modes applying to the whole reference,
i.e., NormalCitation and AuthorInText.

Therefore, I'm wondering: has the following any meaning?

  [cite: -@doe; @foo]
  [(cite): -@doe; @foo]

> Depending on CSL's future development there might well be other
> citations modes. I personally think there's much to learn from
> biblatex, but that's a different debate...

As a related node, from Biblatex, we may also need, e.g., [(Cite):...],
but this was rejected for some reason a long time ago.

Regards,

-- 
Nicolas Goaziou

Re: wip-cite status question and feedback

[Bruce D'Arcus]

On Wed, Apr 8, 2020, 5:32 AM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:

[snip]

> Sounds like a plan. Let me summarize it a bit :
>
> 1. [ ] Finalize Org citation syntax. It is mostly good, but we need to
>    decide about the following points:
>
>    - [ ] Should it include both "(cite):" and "cite:", i.e., does it
>      make sense to provide a (very limited) style specification piece
>      wise?

There are two questions here; right?

So (cite:) is the standard citation form (in the pandoc syntax
[@doe19]), and cite: is the bare variant @doe19?

On the latter, probably yes, depending on how ready citeproc.el/org is
to take over processing for more advanced needs.

>    - [ ] Should it include /global/ prefix and suffix?

On further reflection, and discussion on the csl forum, I think yes.

The flat pandoc syntax works well, but does break down in one situation:

Where you have a multicite, and where the output processor resorts them.

Global affixes would fix that.

>    - [ ] Should we keep the short specification, i.e., "[@key]"?

If you kept it, would it be possible to allow more than one key?

So something like [@doe17; @doe18]?

>    - [ ] What kind of markup do we allow in a citation? At the moment
>      the exhaustive list is: bold, code, entity, italic, latex-fragment,
>      strike-through, subscript, superscript, underline, verbatim and
>      line breaks.

I don't see any downside to being liberal here. Do you?

Though most common by far, I think, would be just bold and italic.

> 2. [ ] Decide about API Org should provide for it to be useful. Here are
>    some low hanging fruits:
>
>    - [ ] List all ".bib" files associated to the document,
>
>    - [ ] List all citations,
>
>    - [ ] Return citation key at point, if any.
>
>    - Anything else?
>
>    Moreover, we need to decide about how external processors could plug
>    into the export framework.
>
>    - [ ] For example, it could be a simple variable bound to a list of
>      functions. Each function accepts three arguments: the export
>      back-end, as a symbol, the full citation, as a list of triplets
>      (key, prefix, suffix) along with global prefix/suffix, and the
>      usual INFO communication channel. Does it need more?
>
>    - [ ] Also, the prefix/suffix may contain some Org markup, so this
>      needs to be also processed. Should it happen before, or after the
>      external processor does its job? I.e., should the function
>      translate into Org or target format?

Obviously on the above group of questions, would be really good to
hear from Andras, but the citeproc-el docs should be helpful?

https://github.com/andras-simonyi/citeproc-el#usage

> 3. [ ] Specify the kind of basic translation that be done out of the
>    box? Ideally, every non-derived back-end should do something, even
>    basic. Therefore, we need to propose some translation pattern for
>    each of the following:
>
>    - [ ] ASCII
>
>    - [ ] HTML
>
>    - [ ] LaTeX
>
>    - [ ] ODT
>
>    - [ ] Texinfo

So this would the default output for each format. This would be
assuming, to go back to earlier posts here, basic formatting built it,
and not integrating citeproc-el?

Bruce

Re: wip-cite status question and feedback

[denis.maier.lists]

Hi,

> Let's assume Org implements SuppressAuthor as "-@doe", so far it has:
> 
> | Syntax                    | Mode                            | Sample output |
> |---------------------------+---------------------------------+---------------|
> | @doe or [cite:@doe]       | AuthorInText                    | Doe (2020)    |
> | -@doe or [cite:-@doe]     | AuthorInText + SuppressAuthor   | (2020)        |
> | [@doe] or [(cite):@doe]   | NormalCitation                  | (Doe, 2020)   |
> | [-@doe] or [(cite):-@doe] | NormalCitation + SuppressAuthor | (2020)        |
> 
> So, we could mix suppress author with some other style, although this
> seems useless for the time being.
> 
> Current syntax is not great because it is ad-hoc, and not very
> future-proof, if new modes appear. Also, the "(cite)" key is a bit
> mouthful, "citep" might have been prettier. Anyway, is it complete
> enough?

Well, that depends on your target. If you aim for CSL than that's already fairly complete. (Even it's more than complete since the current CSL specification only knows normal citations. Suppressing authors is done by calling applications, and AuthorInText is a pandoc-specific addition.)
If you aim for biblatex than there's still a long way to go.
Concerning "cite", "(cite)", and "citep"

1. In CSL the basic citation depends on the class of the citation style. Citations appear either as parenthetical citations or in notes. 
It's not a huge deal, but nevertheless...

2. "citep" is much better than "(cite)", but it is also inaccurate as long as CSL is concerned.

So, I'd rather suggest something along these lines:
 
cite: => note citation or parenthetical citation => (Doe 2020)
citet => narrative cite => Doe (2020)

Of course, you might argue that this is too closely tied to CSL. You could also adopt natbib's cite, citet, citep commands, or biblatex's many cite commands...
 
> Also, there is some ambiguity between modes applying reference-wise,
> i.e., "suppress author", and modes applying to the whole reference,
> i.e., NormalCitation and AuthorInText.
> 
> Therefore, I'm wondering: has the following any meaning?
> 
>   [cite: -@doe; @foo]
>   [(cite): -@doe; @foo]

Good question: I think "[(cite): -@doe; @foo]" is absolutely possible. That would be "(2019; Foo 2020)". But what about "[cite: -@doe; @foo]"? That is certainly more complicated. How do you render multiple narrative citations?

Best,
Denis

Re: wip-cite status question and feedback

[Bruce D'Arcus]

On Sat, Apr 11, 2020 at 5:43 PM <denis.maier.lists@mailbox.org> wrote:
...

> Well, that depends on your target. If you aim for CSL than that's already fairly complete. (Even it's more than complete since the current CSL specification only knows normal citations. Suppressing authors is done by calling applications, and AuthorInText is a pandoc-specific addition.)

Suppress author is not specific to pandoc; it's used in other CSL
applications like Zotero and Mendeley.

> If you aim for biblatex than there's still a long way to go.
> Concerning "cite", "(cite)", and "citep"
>
> 1. In CSL the basic citation depends on the class of the citation style. Citations appear either as parenthetical citations or in notes.
> It's not a huge deal, but nevertheless...
>
> 2. "citep" is much better than "(cite)", but it is also inaccurate as long as CSL is concerned.
>
> So, I'd rather suggest something along these lines:
>
> cite: => note citation or parenthetical citation => (Doe 2020)
> citet => narrative cite => Doe (2020)
>
> Of course, you might argue that this is too closely tied to CSL. You could also adopt natbib's cite, citet, citep commands, or biblatex's many cite commands...

My only suggestion is to retain the goal of being able to change
back-and-forth from in-text and note-based styles without needing to
modifying the document source.

The pandoc syntax meets that goal.

Biblatex (and likely bibtex) does not, where you have dedicated
citation commands for footcites and such, and so have to modify
document source to make those changes in output.

The downside to the suppress author approach is you have to explicitly
write the author name in the text when doing things like the citet
example above.

So the above would be "Doe [-doe2020]" in pandoc.

I've never found that a problem, but reasonable people disagree.

I'm not 100% sure, but I think citet meets that goal also, so Denis'
suggestion might work.

Bruce

Re: wip-cite status question and feedback

[Nicolas Goaziou]

Hello,

"Bruce D'Arcus" <bdarcus@gmail.com> writes:

> I'm not 100% sure, but I think citet meets that goal also, so Denis'
> suggestion might work.

I hadn't realised that AuthorInText was a Citeproc(-hs) interpretation
of CSL.

Anyway, Org cite syntax should:
- fully support CSL,
- allow changing globally style of cites,
- be extensible enough to support « advanced » citation markup (NatBib,
  Biblatex…),
- degrade gracefully when using less advanced markup,
- be short enough

Here's a proposal for a definitive citation syntax.

This syntax implements one default cite command, and specialized (or
typed) ones.

The default citation command is, as usual:

  [cite: ... @key1...]

with allowed global pre/post strings, and a minus marker to specify
SuppressAuthor per cite. 

I assume that [cite:@key] is common enough, so bare @key is a shorthand
for it. Likewise, -@key is a shorthand for [cite:-@key].

Default citations uses the default citation style, which could be
defined globally (by a defcustom), locally (keyword, or property), or
per ".bib" file.

The syntax also provides typed citations:

  [citeX: ... @key1...]

where X stands for any alphanumeric character.

A typed citation is meant to locally override default style. Each
citation back-ends may interpret "X" type, but if they don't, "citeX"
should be treated as "cite".

For example, assuming Citeproc treats

  [cite:@doe]

as (Doe, 2020), then

  [citet:@doe]

could be interpreted as AuthorInText by Citeproc, i.e.,

  Doe (2020)

but

  [citep:@doe]

could be ignored, and therefore become

  (Doe, 2020)

Of course Biblatex may interpret it differently.

Note that the following proposal drops support for [@key1], which
I found awkward anyway.


WDYT?

-- 
Nicolas Goaziou

Re: wip-cite status question and feedback

[Bruce D'Arcus]

Hi,

On Sun, Apr 12, 2020 at 6:38 AM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:

> Anyway, Org cite syntax should:
> - fully support CSL,
> - allow changing globally style of cites,
> - be extensible enough to support « advanced » citation markup (NatBib,
>   Biblatex…),
> - degrade gracefully when using less advanced markup,
> - be short enough

Perfect.

> Here's a proposal for a definitive citation syntax.
>
> This syntax implements one default cite command, and specialized (or
> typed) ones.
>
> The default citation command is, as usual:
>
>   [cite: ... @key1...]
>
> with allowed global pre/post strings, and a minus marker to specify
> SuppressAuthor per cite.
>
> I assume that [cite:@key] is common enough, so bare @key is a shorthand
> for it. Likewise, -@key is a shorthand for [cite:-@key].

All good.

> Default citations uses the default citation style, which could be
> defined globally (by a defcustom), locally (keyword, or property), or
> per ".bib" file.

I'm just a little confused here, particularly on the last item. Why
would one set a style per bib file?

> The syntax also provides typed citations:
>
>   [citeX: ... @key1...]
>
> where X stands for any alphanumeric character.
>
> A typed citation is meant to locally override default style. Each
> citation back-ends may interpret "X" type, but if they don't, "citeX"
> should be treated as "cite".
>
> For example, assuming Citeproc treats
>
>   [cite:@doe]
>
> as (Doe, 2020), then
>
>   [citet:@doe]
>
> could be interpreted as AuthorInText by Citeproc, i.e.,
>
>   Doe (2020)
>
> but
>
>   [citep:@doe]
>
> could be ignored, and therefore become
>
>   (Doe, 2020)
>
> Of course Biblatex may interpret it differently.

This looks to be an elegant solution to the goals you articulated at the top.

On the "could be ignored" part, you are referring to the optional type
character, so that citey: becomes cite:; correct?

Finally, what does the above example look like when, say, there are
two cites (say @doe2020 and @doe2019), and a global prefix?

Is it this?

[cite:see ;@doe2020;@doe2019]

And a SuppressAuthor variant would be this?

[cite:see ;-@doe2020;-@doe2019]

Bruce

Re: wip-cite status question and feedback

[Nicolas Goaziou]

> I'm just a little confused here, particularly on the last item. Why
> would one set a style per bib file?

No idea. The need exists though:
<https://tex.stackexchange.com/questions/10104/two-bibliographies-with-two-different-styles-in-the-same-document>

This is a natural following step. Does Org need to standardize styles?
Or is it up to each citation back-end to handle this? 

My naive thinking was to allow something like:

   #+bibliography: "something.bib" :style author-year

but maybe the "style" part is too citation back-end dependant, and
should not be standardized.

I would be nice, however, to standardize two keywords: one to define
a bibliography, and another one to insert it in a document, upon export.

Suggestions welcome!

> On the "could be ignored" part, you are referring to the optional type
> character, so that citey: becomes cite:; correct?

Yes, basically, the parser returns, e.g., '(:style "y"). It is up to the
citation back-end to interpret, or not, that :style attribute. If it
ignores it, the citation effectively becomes equivalent to '(:style
nil), i.e., "cite:". Is that clearer?

> Finally, what does the above example look like when, say, there are
> two cites (say @doe2020 and @doe2019), and a global prefix?
>
> Is it this?
>
> [cite:see ;@doe2020;@doe2019]

Yes, and a "t-styled" citation would be:

  [citet:see;@doe2020;@doe2019]

Barring the prefix, the syntax of the citation does not change wrt to
"wip-cite" branch. However, this is enough to be slightly incompatible,
hence the "wip".

> And a SuppressAuthor variant would be this?
>
> [cite:see ;-@doe2020;-@doe2019]

Indeed.

How does that sound?

Re: wip-cite status question and feedback

[Bruce D'Arcus]

I'm going to split off the syntax part of your email, Nicholas, for
quick comment.

I need to think more about the other questions.

On Sun, Apr 12, 2020 at 10:02 AM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:

> > Finally, what does the above example look like when, say, there are
> > two cites (say @doe2020 and @doe2019), and a global prefix?
> >
> > Is it this?
> >
> > [cite:see ;@doe2020;@doe2019]
>
> Yes, and a "t-styled" citation would be:
>
>   [citet:see;@doe2020;@doe2019]
>
> Barring the prefix, the syntax of the citation does not change wrt to
> "wip-cite" branch. However, this is enough to be slightly incompatible,
> hence the "wip".
>
> > And a SuppressAuthor variant would be this?
> >
> > [cite:see ;-@doe2020;-@doe2019]
>
> Indeed.
>
> How does that sound?

Good; no issues that I see with this at all.

Only question is I see you removed whitespace after the prefix on your
citet: example.

Is the expectation (which is reasonable; am just asking) that prefixes
would add the whitespace after it on output, so users don't have worry
about this?

So in other words, the value of an affix would be a trimmed string?

Bruce

Re: wip-cite status question and feedback

[Nicolas Goaziou]

"Bruce D'Arcus" <bdarcus@gmail.com> writes:

> On Sun, Apr 12, 2020 at 10:02 AM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:

>> Yes, and a "t-styled" citation would be:
>>
>>   [citet:see;@doe2020;@doe2019]
>>
>> Barring the prefix, the syntax of the citation does not change wrt to
>> "wip-cite" branch. However, this is enough to be slightly incompatible,
>> hence the "wip".
>
> Good; no issues that I see with this at all.

Great!

I'll wait a bit for others to comment. If there is no objection, I'll
implement this in "wip-cite" and rebase that branch on top of "master"
for easier testing and feedback.

> Only question is I see you removed whitespace after the prefix on your
> citet: example.
>
> Is the expectation (which is reasonable; am just asking) that prefixes
> would add the whitespace after it on output, so users don't have worry
> about this?
>
> So in other words, the value of an affix would be a trimmed string?

That was a typo. But that's a good question anyway.

Generally speaking, I'd rather avoid any magic, so the parser should not
add any space whatsoever. However, should it remove some?

AFAICT, Biblatex would probably ignore spacing since it provides its own
mechanism to separate multiple cites. I don't know about Citeproc. Maybe
trimming prefixes and postfixes is the way to go. What would you suggest
here?

In any case, consecutive spaces ought to be packed into a single one.
This allows auto-filling a paragraph at a citation, e.g.,

  Some very long explanation [cite:see @doe2020 
  pp. 12-15].

is equivalent to

  Some very long explanation [cite:see @doe2020 pp. 12-15]

Re: wip-cite status question and feedback

[Bruce D'Arcus]

On Sun, Apr 12, 2020 at 11:32 AM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:
>
> "Bruce D'Arcus" <bdarcus@gmail.com> writes:

...

> > So in other words, the value of an affix would be a trimmed string?
>
> That was a typo. But that's a good question anyway.
>
> Generally speaking, I'd rather avoid any magic, so the parser should not
> add any space whatsoever. However, should it remove some?
>
> AFAICT, Biblatex would probably ignore spacing since it provides its own
> mechanism to separate multiple cites. I don't know about Citeproc. Maybe
> trimming prefixes and postfixes is the way to go. What would you suggest
> here?

In that case, since it was a typo, I would do as pandoc does: treat it
as a string.

The below syntax examples are more clear.

> In any case, consecutive spaces ought to be packed into a single one.
> This allows auto-filling a paragraph at a citation, e.g.,
>
>   Some very long explanation [cite:see @doe2020
>   pp. 12-15].
>
> is equivalent to
>
>   Some very long explanation [cite:see @doe2020 pp. 12-15]

That makes sense.

Bruce

Re: wip-cite status question and feedback

[Nicolas Goaziou]

"Bruce D'Arcus" <bdarcus@gmail.com> writes:

> In that case, since it was a typo, I would do as pandoc does: treat it
> as a string.

Do you mean: keep all spaces wherever they may be?

Re: wip-cite status question and feedback

[Bruce D'Arcus]

[-- Attachment #1: Type: text/plain, Size: 339 bytes --]

On Sun, Apr 12, 2020 at 11:58 AM Nicolas Goaziou <mail@nicolasgoaziou.fr>
wrote:
>
> "Bruce D'Arcus" <bdarcus@gmail.com> writes:
>
> > In that case, since it was a typo, I would do as pandoc does: treat it
> > as a string.
>
> Do you mean: keep all spaces wherever they may be?

No, I thought your point about auto-filling was a good one.

[-- Attachment #2: Type: text/html, Size: 649 bytes --]

Re: wip-cite status question and feedback

[denis.maier.lists]

Nicolas Goaziou <mail@nicolasgoaziou.fr> hat am 12. April 2020 17:32 geschrieben:
> 
>  
> "Bruce D'Arcus" <bdarcus@gmail.com> writes:
> 
> > On Sun, Apr 12, 2020 at 10:02 AM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:
> 
> >> Yes, and a "t-styled" citation would be:
> >>
> >>   [citet:see;@doe2020;@doe2019]
> >>
> >> Barring the prefix, the syntax of the citation does not change wrt to
> >> "wip-cite" branch. However, this is enough to be slightly incompatible,
> >> hence the "wip".
> >
> > Good; no issues that I see with this at all.
> 
> Great!
> 
> I'll wait a bit for others to comment. If there is no objection, I'll
> implement this in "wip-cite" and rebase that branch on top of "master"
> for easier testing and feedback.

Your proposal looks good to me. Just one question concerning typed citations. citeX is good and concise, but why limit this to only one character? What about allowing something more verbose? Perhaps "cite-intext:" or "cite:intext:"? 
The simple syntax is great for most cases, but if you want to support some of those not so common biblatex commands, this might be better.
What do you think? (I think there's been a discussion about that, but I don't remember the exact details.) Anyway, I think your proposal is a good start, and it can be extended afterwards.

Concerning some other open questions, I suggest sticking to what citeproc-org uses:

1. For the bibliography:
#+bibliography: something.bib
(Could this be a list containing multiple files?)

2. Placing the bibliography with:
#+bibliography: here
(Ideally, it would be possible to have this multiple times, perhaps with some filters, like printing only the works of a certain author, or with certain keywords, or so. But that's, of course something for later...)

3. Setting the style:
#+CSL_STYLE: "some-style.csl"

Of course, if you're using biblatex or natbib you'll need another option for that.

Best,
Denis

Re: wip-cite status question and feedback

[Nicolas Goaziou]

Hello,

denis.maier.lists@mailbox.org writes:

> Just one question concerning typed citations. citeX is good and
> concise, but why limit this to only one character?

Because… it is good and concise? ;)

> What about allowing something more verbose? Perhaps
> "cite-intext:" or "cite:intext:"?

Note the latter introduces an ambiguity: [cite:see: @doe was right!].
Fixing it requires two colons in default cite prefix: [cite::@doe].
I don't think we want this.

The former doesn't have this bias.

> The simple syntax is great for most cases, but if you want to support
> some of those not so common biblatex commands, this might be better.

Alphanumeric suffix provides 62 combinations, which should hopefully be
enough for any citation back-end out there (I'm looking at you
biblatex). It's not terribly readable, tho, as you point out.

> What do you think?

This is a conciseness versus readability problem, not a technical one,
as long as we do not allow too much, from a parser point of view.

I have no strong opinion on the topic. It would be more valuable to hear
from actual citations users. What would they prefer?

> Concerning some other open questions, I suggest sticking to what
> citeproc-org uses:
>
> 1. For the bibliography:
>
> #+bibliography: something.bib
> (Could this be a list containing multiple files?)

Multiple keywords may be more appropriate, particularly if you need to
spell out absolute file names.

Org can provide a function listing all of them anyway.

> 2. Placing the bibliography with:
>
> #+bibliography: here
> (Ideally, it would be possible to have this multiple times, perhaps
> with some filters, like printing only the works of a certain author,
> or with certain keywords, or so. But that's, of course something for
> later...)

It is smart, but I'm not sure I like using the same keyword for two
different things. OTOH, I don't have a better idea.

> 3. Setting the style:
> #+CSL_STYLE: "some-style.csl"
>
> Of course, if you're using biblatex or natbib you'll need another
> option for that.

I think this part is out of Org's scope. Since values between various
citation back-ends are probably not compatible, e.g., some may require
a file, others a style name, normalization is not useful here. They can
use whatever keyword they fancy.

Regards,

-- 
Nicolas Goaziou

Re: wip-cite status question and feedback

[Stefan Nobis]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Alphanumeric suffix provides 62 combinations, which should hopefully
> be enough for any citation back-end out there (I'm looking at you
> biblatex). It's not terribly readable, tho, as you point out.

I second that. Some of the many BibLaTeX commands are due to
compatibility with other (older) packages and to ease transitions.

Another aspect: Maybe it would be a good idea to reserve some chars
(e.g. the numeric ones) for user defined citation commands (a
minimalistic reserved namespace).


[Placing bibliography with "#+bibliography: here"]
> It is smart, but I'm not sure I like using the same keyword for two
> different things. OTOH, I don't have a better idea.

I personally also dislike one keyword for completely different
purposes. Therefore I suggest to take the idea from BibLaTeX and use a
keyword like "printbibliography" the mark the place where the
bibliography should be output.

This command may also have parameters like filtering options (maybe
depending on the backend processor; I only know BibLaTeX so I can't
say if it would be easy to abstract away differences between different
engines). In the case of BiBLaTeX the printbibliography command
optionally accepts multiple key-value parameters. Some examples for
the keys are "heading" for the chapter/section heading, "type" to
output/print only entries of a certain type (like "book"; or type
"online" and with the additional key "nottype" separate non-online and
online sources), "keyword" to filter entries with the associated
keyword etc.

Another nice feature of BibLaTeX is the possibility to generate
bibliography per chapter/section (e.g. setting conference
proceedings). In the simplest case each chapter/section is marked, in
the case of LaTeX/BibLaTeX it is wrapped with "\begin{refsection}" and
"\end{refsection}" and the printbibliography command occurs inside
this refsection block. In this case BibLaTeX defaults to output only
references used inside the marked block.

[Style selection]
> I think this part is out of Org's scope. Since values between
> various citation back-ends are probably not compatible, e.g., some
> may require a file, others a style name, normalization is not useful
> here. They can use whatever keyword they fancy.

Yes, I second that. But it may be worth thinking a second about using
one Org document to generate output with different backends (e.g. PDF
via LaTeX and BibLaTeX, HTML, and ASCII). It would be nice, to have
some way to configure each citation engine form the same document.
Either use different keywords like "#+CSL_STYLE" and
"#+BIBLATEX_STYLE" or we use your original suggestion of a ":style"
parameter to the "#+BIBLIOGRAPHY" keyword and provide some means to
translate its value individually for each engine - e.g. an alist or
some function provided by the user. And if no translation is provided,
just give the value verbatim to the engine, thus if a user only
targets a single backend, he does not need to provide any
extra configuration.

Just my 2¢.

-- 
Until the next mail...,
Stefan.

Re: wip-cite status question and feedback

[Bruce D'Arcus]

On Sun, Apr 12, 2020 at 6:19 PM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:

> denis.maier.lists@mailbox.org writes:
>
> > Just one question concerning typed citations. citeX is good and
> > concise, but why limit this to only one character?
>
> Because… it is good and concise? ;)
>
> > What about allowing something more verbose? Perhaps
> > "cite-intext:" or "cite:intext:"?
>
> Note the latter introduces an ambiguity: [cite:see: @doe was right!].
> Fixing it requires two colons in default cite prefix: [cite::@doe].
> I don't think we want this.
>
> The former doesn't have this bias.

So cite-intext vs citet?

> > The simple syntax is great for most cases, but if you want to support
> > some of those not so common biblatex commands, this might be better.
>
> Alphanumeric suffix provides 62 combinations, which should hopefully be
> enough for any citation back-end out there (I'm looking at you
> biblatex). It's not terribly readable, tho, as you point out.
>
> > What do you think?
>
> This is a conciseness versus readability problem, not a technical one,
> as long as we do not allow too much, from a parser point of view.
>
> I have no strong opinion on the topic. It would be more valuable to hear
> from actual citations users. What would they prefer?

For me, I'm less concerned with the syntax question than I am with the
flexibility and clarity of knowing how to get the output I want.

If I have to look at some table of commands, or pause to remember
which command I need, that's providing friction to my work.

I will add that the biggest request of Zotero's citation support
(which only has the two forms: parenthetical and suppress author) is
textual citations, like natbib's citet.

Perhaps it would make sense to reserve that style in org?


Bruce

Re: wip-cite status question and feedback

[Denis Maier]

> Stefan Nobis <stefan-ml@snobis.de> hat am 13. April 2020 10:33 geschrieben:
> 
>  
> Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:
> 
> > Alphanumeric suffix provides 62 combinations, which should hopefully
> > be enough for any citation back-end out there (I'm looking at you
> > biblatex). It's not terribly readable, tho, as you point out.
> 
> I second that. Some of the many BibLaTeX commands are due to
> compatibility with other (older) packages and to ease transitions.
> 
> Another aspect: Maybe it would be a good idea to reserve some chars
> (e.g. the numeric ones) for user defined citation commands (a
> minimalistic reserved namespace).

My main concern is not so much the sheer number of available commands. I am just not sure if something like [cite6: @doe] will increase readability. (Will you remember what that is supposed to mean?) Also: With biblatex you have \nocite and \notecite, which one will be [citen: @doe]? I guess here I'd use citen for the more common option (nocite), but there still could be the need for a notecite version. Perhaps [cite-note: @doe] or [cite/note: @doe]. Again, a set of simple commands cite, citet, and perhaps a few others might a good starting point, but I am not sure if that is enough in the long run. (I currently use mainly CSL so I don't have much need for them myself, but I think extensability might become an issue at some point.) Also, some might prefer to have more verbose commands.

By the way, I think you should add a nocite version to your proposal. (citen, citeno or something similar.)

Concerning the fallback idea (citep being equal to cite if citep is not defined by a backend.) That's really good, but perhaps there should be a way to customize the fallback rules? Like a certain citex should be treated as cite, while citey is equal to citet... 
WDYT?


> [Placing bibliography with "#+bibliography: here"]
> > It is smart, but I'm not sure I like using the same keyword for two
> > different things. OTOH, I don't have a better idea.
> 
> I personally also dislike one keyword for completely different
> purposes. Therefore I suggest to take the idea from BibLaTeX and use a
> keyword like "printbibliography" the mark the place where the
> bibliography should be output.

Ok, fair enough.
So:
#+BIBLIOGRAPHY: file1.bib
#+BIBLIOGRAPHY: file2.bib
[Rest of file]

#+PRINTBIBLIOGRAPHY

Or perhaps:

#+BIBLIOGRAPHYFILE: file1.bib
#+BIBLIOGRAPHYFILE: file2.bib
[Rest of file]

#+BIBLIOGRAPHY

But ultimately, each will be fine. I don't think that matters much...

> This command may also have parameters like filtering options (maybe
> depending on the backend processor; I only know BibLaTeX so I can't
> say if it would be easy to abstract away differences between different
> engines). In the case of BiBLaTeX the printbibliography command
> optionally accepts multiple key-value parameters. Some examples for
> the keys are "heading" for the chapter/section heading, "type" to
> output/print only entries of a certain type (like "book"; or type
> "online" and with the additional key "nottype" separate non-online and
> online sources), "keyword" to filter entries with the associated
> keyword etc.

Yes, biblatex is very powerful here, and much ahead of other solutions. Pandoc has some support for multiple bibliographies, but it's nowhere as advanced. So offering something here would be great, but the question is how this can be done in a output agnostic way.

> [Style selection]
> > I think this part is out of Org's scope. Since values between
> > various citation back-ends are probably not compatible, e.g., some
> > may require a file, others a style name, normalization is not useful
> > here. They can use whatever keyword they fancy.
> 
> Yes, I second that. But it may be worth thinking a second about using
> one Org document to generate output with different backends (e.g. PDF
> via LaTeX and BibLaTeX, HTML, and ASCII). It would be nice, to have
> some way to configure each citation engine form the same document.
> Either use different keywords like "#+CSL_STYLE" and
> "#+BIBLATEX_STYLE" or we use your original suggestion of a ":style"
> parameter to the "#+BIBLIOGRAPHY" keyword and provide some means to
> translate its value individually for each engine - e.g. an alist or
> some function provided by the user. And if no translation is provided,
> just give the value verbatim to the engine, thus if a user only
> targets a single backend, he does not need to provide any
> extra configuration.

These are very good questions. Looking again at pandoc: There you have two options:
a) use pandoc-citeproc to produce the citations for each target format
b) use a native bibliographic tool (e.g. biblatex or natbib in latex output).

Option b) is clearly more powerful as you can use 
But option a) can be used if you need the same output in DOCX, HTML and PDF. (Say you have an article written for a journal, and the manuscript is uploaded to your institution's repository.) That should be possible with Org as well.

Best,
Denis

Re: wip-cite status question and feedback

[Bruce D'Arcus]

On Mon, Apr 13, 2020 at 4:34 AM Stefan Nobis <stefan-ml@snobis.de> wrote:

> [Placing bibliography with "#+bibliography: here"]
> > It is smart, but I'm not sure I like using the same keyword for two
> > different things. OTOH, I don't have a better idea.
>
> I personally also dislike one keyword for completely different
> purposes. Therefore I suggest to take the idea from BibLaTeX and use a
> keyword like "printbibliography" the mark the place where the
> bibliography should be output.

So we need one keyword to designate the bibliographic data file, and
another where to place the formatted output.

> This command may also have parameters like filtering options (maybe
> depending on the backend processor; I only know BibLaTeX so I can't
> say if it would be easy to abstract away differences between different
> engines). In the case of BiBLaTeX the printbibliography command
> optionally accepts multiple key-value parameters. Some examples for
> the keys are "heading" for the chapter/section heading, "type" to
> output/print only entries of a certain type (like "book"; or type
> "online" and with the additional key "nottype" separate non-online and
> online sources), "keyword" to filter entries with the associated
> keyword etc.
>
> Another nice feature of BibLaTeX is the possibility to generate
> bibliography per chapter/section (e.g. setting conference
> proceedings). In the simplest case each chapter/section is marked, in
> the case of LaTeX/BibLaTeX it is wrapped with "\begin{refsection}" and
> "\end{refsection}" and the printbibliography command occurs inside
> this refsection block. In this case BibLaTeX defaults to output only
> references used inside the marked block.

Indeed, this is a good summary, and also a feature often requested on
the Zotero forums.

I think the way biblatex handles this is general, based on reading the
docs, is good.

It'd be nice to have, but maybe not essential for a first step?

Bruce

Re: wip-cite status question and feedback

[denis.maier.lists]

> Nicolas Goaziou <mail@nicolasgoaziou.fr> hat am 13. April 2020 00:19 geschrieben:
> 
>  
> Hello,
> 
> denis.maier.lists@mailbox.org writes:
> 
> > Just one question concerning typed citations. citeX is good and
> > concise, but why limit this to only one character?
> 
> Because… it is good and concise? ;)
> 
> > What about allowing something more verbose? Perhaps
> > "cite-intext:" or "cite:intext:"?
> 
> Note the latter introduces an ambiguity: [cite:see: @doe was right!].
> Fixing it requires two colons in default cite prefix: [cite::@doe].
> I don't think we want this.
> 
> The former doesn't have this bias.

Ok, ambiguity is not good. So we need something else. (As in the other message: cite/note or cite-intext?)

> 
> > The simple syntax is great for most cases, but if you want to support
> > some of those not so common biblatex commands, this might be better.
> 
> Alphanumeric suffix provides 62 combinations, which should hopefully be
> enough for any citation back-end out there (I'm looking at you
> biblatex). It's not terribly readable, tho, as you point out.
> 
> > What do you think?
> 
> This is a conciseness versus readability problem, not a technical one,
> as long as we do not allow too much, from a parser point of view.
> 
> I have no strong opinion on the topic. It would be more valuable to hear
> from actual citations users. What would they prefer?

What about allowing both, just like most command line tools have short and long options (e.g., -o and --open)?

Best,
Denis

Re: wip-cite status question and feedback

[Joost Kremers]

On Mon, Apr 13 2020, Nicolas Goaziou wrote:
> denis.maier.lists@mailbox.org writes:
>> What about allowing something more verbose? Perhaps
>> "cite-intext:" or "cite:intext:"?
[...]
>> The simple syntax is great for most cases, but if you want to 
>> support
>> some of those not so common biblatex commands, this might be 
>> better.
>
> Alphanumeric suffix provides 62 combinations, which should 
> hopefully be
> enough for any citation back-end out there (I'm looking at you
> biblatex). It's not terribly readable, tho, as you point out.

62 combinations might sound like a lot, but if you want your cite 
commands to be mnemonic, you'll run out of options much more 
quickly.

> This is a conciseness versus readability problem, not a 
> technical one,
> as long as we do not allow too much, from a parser point of 
> view.
>
> I have no strong opinion on the topic. It would be more valuable 
> to hear
> from actual citations users. What would they prefer?

Not sure if my opinion counts, given that I mainly use LaTeX + 
biblatex to write my texts, but I would definitely allow more than 
one character. The more common commands (=citep=, =citet=) can 
still use a single character (and thus remain concise), but for 
less common commands, the ability to have more descriptive names 
is to be preferred. Imagine looking at a document you wrote a few 
years back and having to figure out what =citeQ= or =cite7= was 
meant for again, or finding that =citeF= was changed from 
=\fullcite= to =\footfullcite= because at some point the 
developers figured the latter would be used more often.

I don't think it's necessary to use a dash (or any other 
character) in longer cite commands, though. =citeintext= isn't 
that much more difficult to read than =cite-intext=. (Biblatex 
does just fine without dashes, and there's always camelCase if 
you're so inclined.)

>> 1. For the bibliography:
>>
>> #+bibliography: something.bib
>> (Could this be a list containing multiple files?)
>
> Multiple keywords may be more appropriate, particularly if you 
> need to
> spell out absolute file names.
>
> Org can provide a function listing all of them anyway.

Yes, and please make it a public (one-dash) function. :-)

>> 2. Placing the bibliography with:
>>
>> #+bibliography: here
>> (Ideally, it would be possible to have this multiple times, 
>> perhaps
>> with some filters, like printing only the works of a certain 
>> author,
>> or with certain keywords, or so. But that's, of course 
>> something for
>> later...)
>
> It is smart, but I'm not sure I like using the same keyword for 
> two
> different things. OTOH, I don't have a better idea.

As someone already suggested, using something like 
=#+printbibliography:= would work. And if that is too 
biblatex-like, you could instead opt for e.g. 
=#+list-of-references:=. (Output formats such as HTML or epub 
don't involve any printing anyway, so... ;-)


-- 
Joost Kremers
Life has its moments

Re: wip-cite status question and feedback

[denis.maier.lists]

> > Stefan Nobis <stefan-ml@snobis.de> hat am 13. April 2020 10:33 geschrieben:
> > 
> >  
> > Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:
> > 
> > > Alphanumeric suffix provides 62 combinations, which should hopefully
> > > be enough for any citation back-end out there (I'm looking at you
> > > biblatex). It's not terribly readable, tho, as you point out.
> > 
> > I second that. Some of the many BibLaTeX commands are due to
> > compatibility with other (older) packages and to ease transitions.
> > 
> > Another aspect: Maybe it would be a good idea to reserve some chars
> > (e.g. the numeric ones) for user defined citation commands (a
> > minimalistic reserved namespace).
> 
> My main concern is not so much the sheer number of available commands. I am just not sure if something like [cite6: @doe] will increase readability. (Will you remember what that is supposed to mean?) Also: With biblatex you have \nocite and \notecite, which one will be [citen: @doe]? I guess here I'd use citen for the more common option (nocite), but there still could be the need for a notecite version. Perhaps [cite-note: @doe] or [cite/note: @doe]. Again, a set of simple commands cite, citet, and perhaps a few others might a good starting point, but I am not sure if that is enough in the long run. (I currently use mainly CSL so I don't have much need for them myself, but I think extensability might become an issue at some point.) Also, some might prefer to have more verbose commands.
> 
> By the way, I think you should add a nocite version to your proposal. (citen, citeno or something similar.)
> 
> Concerning the fallback idea (citep being equal to cite if citep is not defined by a backend.) That's really good, but perhaps there should be a way to customize the fallback rules? Like a certain citex should be treated as cite, while citey is equal to citet... 
> WDYT?
> 
> 
> > [Placing bibliography with "#+bibliography: here"]
> > > It is smart, but I'm not sure I like using the same keyword for two
> > > different things. OTOH, I don't have a better idea.
> > 
> > I personally also dislike one keyword for completely different
> > purposes. Therefore I suggest to take the idea from BibLaTeX and use a
> > keyword like "printbibliography" the mark the place where the
> > bibliography should be output.
> 
> Ok, fair enough.
> So:
> #+BIBLIOGRAPHY: file1.bib
> #+BIBLIOGRAPHY: file2.bib
> [Rest of file]
> 
> #+PRINTBIBLIOGRAPHY
> 
> Or perhaps:
> 
> #+BIBLIOGRAPHYFILE: file1.bib
> #+BIBLIOGRAPHYFILE: file2.bib
> [Rest of file]
> 
> #+BIBLIOGRAPHY
> 
> But ultimately, each will be fine. I don't think that matters much...
> 
> > This command may also have parameters like filtering options (maybe
> > depending on the backend processor; I only know BibLaTeX so I can't
> > say if it would be easy to abstract away differences between different
> > engines). In the case of BiBLaTeX the printbibliography command
> > optionally accepts multiple key-value parameters. Some examples for
> > the keys are "heading" for the chapter/section heading, "type" to
> > output/print only entries of a certain type (like "book"; or type
> > "online" and with the additional key "nottype" separate non-online and
> > online sources), "keyword" to filter entries with the associated
> > keyword etc.
> 
> Yes, biblatex is very powerful here, and much ahead of other solutions. Pandoc has some support for multiple bibliographies, but it's nowhere as advanced. So offering something here would be great, but the question is how this can be done in a output agnostic way.
> 
> > [Style selection]
> > > I think this part is out of Org's scope. Since values between
> > > various citation back-ends are probably not compatible, e.g., some
> > > may require a file, others a style name, normalization is not useful
> > > here. They can use whatever keyword they fancy.
> > 
> > Yes, I second that. But it may be worth thinking a second about using
> > one Org document to generate output with different backends (e.g. PDF
> > via LaTeX and BibLaTeX, HTML, and ASCII). It would be nice, to have
> > some way to configure each citation engine form the same document.
> > Either use different keywords like "#+CSL_STYLE" and
> > "#+BIBLATEX_STYLE" or we use your original suggestion of a ":style"
> > parameter to the "#+BIBLIOGRAPHY" keyword and provide some means to
> > translate its value individually for each engine - e.g. an alist or
> > some function provided by the user. And if no translation is provided,
> > just give the value verbatim to the engine, thus if a user only
> > targets a single backend, he does not need to provide any
> > extra configuration.
> 
> These are very good questions. Looking again at pandoc: There you have two options:
> a) use pandoc-citeproc to produce the citations for each target format
> b) use a native bibliographic tool (e.g. biblatex or natbib in latex output).
> 
> Option b) is clearly more powerful as you can use 
> But option a) can be used if you need the same output in DOCX, HTML and PDF. (Say you have an article written for a journal, and the manuscript is uploaded to your institution's repository.) That should be possible with Org as well.
> 
> Best,
> Denis

Re: wip-cite status question and feedback

[denis.maier.lists]

Sorry, my last message was unreadable. (and possibly sent twice, once from a wrong account... don't know if this will come through)
 
> Stefan Nobis <stefan-ml@snobis.de> hat am 13. April 2020 10:33 geschrieben:
> 
>  
> Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:
> 
> > Alphanumeric suffix provides 62 combinations, which should hopefully
> > be enough for any citation back-end out there (I'm looking at you
> > biblatex). It's not terribly readable, tho, as you point out.
> 
> I second that. Some of the many BibLaTeX commands are due to
> compatibility with other (older) packages and to ease transitions.
> 
> Another aspect: Maybe it would be a good idea to reserve some chars
> (e.g. the numeric ones) for user defined citation commands (a
> minimalistic reserved namespace).

My main concern is not so much the sheer number of available commands. I am just not sure if something like [cite6: @doe] will increase readability. (Will you remember what that is supposed to mean?) Also: With biblatex you have \nocite and \notecite, which one will be [citen: @doe]? I guess here I'd use citen for the more common option (nocite), but there still could be the need for a notecite version. Perhaps [cite-note: @doe] or [cite/note: @doe]. Again, a set of simple commands cite, citet, and perhaps a few others might a good starting point, but I am not sure if that is enough in the long run. (I currently use mainly CSL so I don't have much need for them myself, but I think extensability might become an issue at some point.) Also, some might prefer to have more verbose commands.

By the way, I think you should add a nocite version to your proposal. (citen, citeno or something similar.)

Concerning the fallback idea (citep being equal to cite if citep is not defined by a backend.) That's really good, but perhaps there should be a way to customize the fallback rules? Like a certain citex should be treated as cite, while citey is equal to citet... 
WDYT?


> [Placing bibliography with "#+bibliography: here"]
> > It is smart, but I'm not sure I like using the same keyword for two
> > different things. OTOH, I don't have a better idea.
> 
> I personally also dislike one keyword for completely different
> purposes. Therefore I suggest to take the idea from BibLaTeX and use a
> keyword like "printbibliography" the mark the place where the
> bibliography should be output.

Ok, fair enough.
So:
#+BIBLIOGRAPHY: file1.bib
#+BIBLIOGRAPHY: file2.bib
[Rest of file]

#+PRINTBIBLIOGRAPHY

Or perhaps:

#+BIBLIOGRAPHYFILE: file1.bib
#+BIBLIOGRAPHYFILE: file2.bib
[Rest of file]

#+BIBLIOGRAPHY

But ultimately, each will be fine. I don't think that matters much...

> This command may also have parameters like filtering options (maybe
> depending on the backend processor; I only know BibLaTeX so I can't
> say if it would be easy to abstract away differences between different
> engines). In the case of BiBLaTeX the printbibliography command
> optionally accepts multiple key-value parameters. Some examples for
> the keys are "heading" for the chapter/section heading, "type" to
> output/print only entries of a certain type (like "book"; or type
> "online" and with the additional key "nottype" separate non-online and
> online sources), "keyword" to filter entries with the associated
> keyword etc.

Yes, biblatex is very powerful here, and much ahead of other solutions. Pandoc has some support for multiple bibliographies, but it's nowhere as advanced. So offering something here would be great, but the question is how this can be done in a output agnostic way.

> [Style selection]
> > I think this part is out of Org's scope. Since values between
> > various citation back-ends are probably not compatible, e.g., some
> > may require a file, others a style name, normalization is not useful
> > here. They can use whatever keyword they fancy.
> 
> Yes, I second that. But it may be worth thinking a second about using
> one Org document to generate output with different backends (e.g. PDF
> via LaTeX and BibLaTeX, HTML, and ASCII). It would be nice, to have
> some way to configure each citation engine form the same document.
> Either use different keywords like "#+CSL_STYLE" and
> "#+BIBLATEX_STYLE" or we use your original suggestion of a ":style"
> parameter to the "#+BIBLIOGRAPHY" keyword and provide some means to
> translate its value individually for each engine - e.g. an alist or
> some function provided by the user. And if no translation is provided,
> just give the value verbatim to the engine, thus if a user only
> targets a single backend, he does not need to provide any
> extra configuration.

These are very good questions. Looking again at pandoc: There you have two options:
a) use pandoc-citeproc to produce the citations for each target format
b) use a native bibliographic tool (e.g. biblatex or natbib in latex output).

Option b) is clearly more powerful as you can use 
But option a) can be used if you need the same output in DOCX, HTML and PDF. (Say you have an article written for a journal, and the manuscript is uploaded to your institution's repository.) That should be possible with Org as well.

Best,
Denis

Re: wip-cite status question and feedback

[Stefan Nobis]

Joost Kremers <joostkremers@fastmail.fm> writes:

> I don't think it's necessary to use a dash (or any other character)
> in longer cite commands, though. =citeintext= isn't that much more
> difficult to read than =cite-intext=. (Biblatex does just fine
> without dashes, and there's always camelCase if you're so inclined.)

It is not necessary, but it may be an opportunity to introduce
namespaces. For example cite and citeX (with a single letter) is
strictly reserved for Org core. Package may introduce new commands
like "cite-subcommand" with a single dash and something like
"cite--subcommand" or "cite/subcommand" is reserved for local use by
users and should not be used by packages.

I think citation needs are quite manifold and thus there might be many
upcoming packages each supporting some special needs. Therefore it
would be nice to a some simple kind of namespaces for commands to
avoid name clashes when using multiple specialised support packages at
the same time.

-- 
Until the next mail...,
Stefan.

RE: wip-cite status question and feedback

[Gustav Wikström]

Hi,

I'm curious. So take this for what it is; I.e. curiosity. What /exactly/ is meant with a citation here? Is it a new general concept in Org mode, or is it something more narrow, as an extension for some specific third party software? Would I be able to use it without that third party software? What would the content of a citation be? Is it a link to some source plus annotations and formatting? Is it only the link? Is it also the formatting? Is it something else entirely? I'm wondering since Org mode has existing facilities for much of this already. But maybe not packaged together for citations just yet? Is there any purpose of thinking of citations as a wrapper for already existing functionality? Or could the links-syntax be extended with more properties and auxiliary functionality to fulfill the need for citations?

Maybe these questions have been discussed already. Sorry in that case, it was a long thread and I admit to not having read everything. I'm simply wondering since there may be reason for reusing and extending existing functionality before creating something new. Hence the question of what exactly a citation consists of above. Because Org mode can already provide links to external content. And, to me at least, a citation is simply that: a link. A link which is the key to the information that is to be marked as the source of the citation. Auxiliary functionality on top of that could provide facilities to collect all such links into bibliographies based on link-backend and properties of that link. Configurations (emacs config or org mode properties, [info:org#Property Syntax]) could instruct on formatting.

But as I said in the beginning. Take this as curiosity. And take it as questions for someone not very much into the academic world at this time, not having to bother with typesetting, formatting and latex related issues.

Kind regards
Gustav

> -----Original Message-----
> From: Emacs-orgmode <emacs-orgmode-bounces+gustav=whil.se@gnu.org> On
> Behalf Of Nicolas Goaziou
> Sent: den 8 april 2020 11:32
> To: Bruce D'Arcus <bdarcus@gmail.com>
> Cc: Joost Kremers <joostkremers@fastmail.fm>; emacs-orgmode@gnu.org;
> András Simonyi <andras.simonyi@gmail.com>; John Kitchin
> <jkitchin@andrew.cmu.edu>
> Subject: Re: wip-cite status question and feedback
> 
> Hello,
> 
> "Bruce D'Arcus" <bdarcus@gmail.com> writes:
> 
> > Note that in CSL processors, the locators are meaningful key-values,
> > basically; not plain text strings.
> 
> OK, but it is enough for Org to feed a CSL processor with, e.g.,
> 
>   key    -> "@doe99"
>   prefix -> "see "
>   suffix -> ", pp. 33-35"
> 
> Then CSL processor does its job to extract whatever information it
> needs. Am I right?
> 
> > While I of course can't speak for John, I would hope and expect org
> > ref to support the new syntax.
> >
> > I would hope and expect the same of other packages (like ebib) that
> > use their own custom syntax.
> 
> I hope so, too. But it would help if developers could chime in and tell
> what API Org should provide. This is particularly important since Org
> will only provide the API, without any specialized implementation. More
> on this at the end of the message.
> 
> > 1. just have a super simple citation export formatter, with (per
> >    above) room to configure an external one
> 
> Sure. Org should provide default export for citations in LaTeX, ASCII,
> HTML, Texinfo and ODT. Suggestions are more than welcome.
> 
> > 2. while I don't know its status, include citeproc-org in org and
> > emacs
> 
> I think citeproc-el should ship with Emacs, but I cannot speak for
> Emacs; it would be nice to ask Emacs developers about it.
> 
> > I'd say if citeproc-org is far along and free of substantial bugs, 2
> > might make sense. I am, of course, biased, but CSL's been widely
> > deployed in the wild for more than a decade, and there are thousands
> > of available styles.
> >
> > Otherwise, and more likely, 1 would be the best path; users get basic
> > output formatting for citations, but then can plug in more elaborate
> > tools (citeproc-org, citeproc-hs*, etc.) if they need them.
> 
> Option 2 makes a lot of sense if citeproc-el is integrated with Emacs.
> Until them, I agree that option 1 is the way to go at the moment.
> 
> > WDY about that?
> 
> Sounds like a plan. Let me summarize it a bit :
> 
> 1. [ ] Finalize Org citation syntax. It is mostly good, but we need to
>    decide about the following points:
> 
>    - [ ] Should it include both "(cite):" and "cite:", i.e., does it
>      make sense to provide a (very limited) style specification piece
>      wise?
> 
>    - [ ] Should it include /global/ prefix and suffix?
> 
>    - [ ] Should we keep the short specification, i.e., "[@key]"?
> 
>    - [ ] What kind of markup do we allow in a citation? At the moment
>      the exhaustive list is: bold, code, entity, italic, latex-
> fragment,
>      strike-through, subscript, superscript, underline, verbatim and
>      line breaks.
> 
> 2. [ ] Decide about API Org should provide for it to be useful. Here
> are
>    some low hanging fruits:
> 
>    - [ ] List all ".bib" files associated to the document,
> 
>    - [ ] List all citations,
> 
>    - [ ] Return citation key at point, if any.
> 
>    - Anything else?
> 
>    Moreover, we need to decide about how external processors could plug
>    into the export framework.
> 
>    - [ ] For example, it could be a simple variable bound to a list of
>      functions. Each function accepts three arguments: the export
>      back-end, as a symbol, the full citation, as a list of triplets
>      (key, prefix, suffix) along with global prefix/suffix, and the
>      usual INFO communication channel. Does it need more?
> 
>    - [ ] Also, the prefix/suffix may contain some Org markup, so this
>      needs to be also processed. Should it happen before, or after the
>      external processor does its job? I.e., should the function
>      translate into Org or target format?
> 
> 3. [ ] Specify the kind of basic translation that be done out of the
>    box? Ideally, every non-derived back-end should do something, even
>    basic. Therefore, we need to propose some translation pattern for
>    each of the following:
> 
>    - [ ] ASCII
> 
>    - [ ] HTML
> 
>    - [ ] LaTeX
> 
>    - [ ] ODT
> 
>    - [ ] Texinfo
> 
> 4. Anything else?
> 
> 
> We need help, or at least opinion, from actual implementors to go
> further. I'm CC'ing some of them.
> 
> I can take care of the implementation (with some time, my plate is full
> at the moment), but I'm mostly incompetent about design issues.
> 
> So, what about ticking off some items?
> 
> Regards,
> 
> --
> Nicolas Goaziou

Re: wip-cite status question and feedback

[Bruce D'Arcus]

On Mon, Apr 13, 2020 at 8:05 AM Gustav Wikström <gustav@whil.se> wrote:

> I'm curious. So take this for what it is; I.e. curiosity. What /exactly/ is meant with a citation here? Is it a new general concept in Org mode, or is it something more narrow, as an extension for some specific third party software? Would I be able to use it without that third party software? What would the content of a citation be? Is it a link to some source plus annotations and formatting? Is it only the link? Is it also the formatting? Is it something else entirely? I'm wondering since Org mode has existing facilities for much of this already. But maybe not packaged together for citations just yet? Is there any purpose of thinking of citations as a wrapper for already existing functionality? Or could the links-syntax be extended with more properties and auxiliary functionality to fulfill the need for citations?

I'm not sure of the value of this sort of question thrown in the
middle of a long-running, many year, conversation. You seem to assume
nobody considered this.

But to answer anyway ...

Citations are references to bibliographic sources.

They are a convention for citing references that developed over
decades, and even centuries, before the internet even existed.

So, they are links of sorts, but lot more.

They're complicated because of the addition of diverse, prescriptive,
and often obscure output formatting requirements, and also because we
have to include additional information beyond just the source.

For example, if I cite some thing, I may need to include page number
references; like (Doe, 2018:18). This says, X info from the Doc, 2018
source, page 18.

So practically, if I submit a manuscript to one journal, it has one
set of formatting requirements; perhaps the above citation style, and
then a separate bibliography style.

If it's rejected and I resubmit it to another journal, it will have a
different set of formatting requirements; perhaps citations look like
this instead (Doe, 2018 p. 18), or are rendered as footnotes.

So we use these systems to decouple authoring from those output requirements.

One can say that academic citation practice is inflexible and archaic,
and one would be right. But it's the world we live in, and very slow
to change. 100 years from now it likely won't look too different!

Effectively, I see this long thread, and similar ones before it, as
how best to reconcile well-known requirements as reflected in existing
systems, but to do so in a way that fits with org.

I think we're close!

HTH.

Bruce

Re: wip-cite status question and feedback

[Richard Lawrence]

Hi all,

Nice to see this issue being discussed again!

I don't have a lot to add and at the moment I don't have a lot of time
to contribute, but I wanted to make one point about this issue:

Joost Kremers <joostkremers@fastmail.fm> writes:

> On Mon, Apr 13 2020, Nicolas Goaziou wrote:
>> denis.maier.lists@mailbox.org writes:
>>> What about allowing something more verbose? Perhaps
>>> "cite-intext:" or "cite:intext:"?
> [...]
>>> The simple syntax is great for most cases, but if you want to
>>> support some of those not so common biblatex commands, this might be
>>> better.
>>
>> Alphanumeric suffix provides 62 combinations, which should hopefully
>> be enough for any citation back-end out there (I'm looking at you
>> biblatex). It's not terribly readable, tho, as you point out.
>
> 62 combinations might sound like a lot, but if you want your cite
> commands to be mnemonic, you'll run out of options much more quickly.

This came up in the discussion in 2015, too. So maybe this can help
avoid repeating a long discussion about this:

I think it is worth pointing out to Bib(La)TeX users that it is useful
to avoid a proliferation of citation commands in Org syntax. The syntax
discussed so far achieves this by "factoring out" formatting information
that BibLaTeX puts into the command into other parts of the syntax and
into the choice of citation stylesheet. For example, instead of having
\footnotecite and \parencite as separate commands, you can just have a
single cite command, and the choice of stylesheet determines whether
citations get formatted as footnotes or as in-text parenthetical
citations or as something else.

This obviously has the drawback that if you only have single citation
command, you only get to make the choice about formatting once for the
whole document (via the stylesheet). So, I think the relevant question
is: how many different basic citation types are needed *within a single
document*, keeping in mind that these basic types will be formatted in
different ways, depending on the choice of stylesheet?

My experience is that it's typically just two (e.g. parenthetical and
author-in-text), and my memory of the earlier conversation was that most
people agreed. This is also borne out in the Pandoc syntax. As long as
we have two basic types of citations, the finer points of formatting
them can be achieved via other syntax, including the choice of
stylesheet.

-- 
Best,
Richard

Re: wip-cite status question and feedback

[Joost Kremers]

On Wed, Apr 15 2020, Richard Lawrence wrote:
>> 62 combinations might sound like a lot, but if you want your 
>> cite
>> commands to be mnemonic, you'll run out of options much more 
>> quickly.
[...]
> So, I think the relevant question
> is: how many different basic citation types are needed *within a 
> single
> document*, keeping in mind that these basic types will be 
> formatted in
> different ways, depending on the choice of stylesheet?
>
> My experience is that it's typically just two (e.g. 
> parenthetical and
> author-in-text), and my memory of the earlier conversation was 
> that most
> people agreed. This is also borne out in the Pandoc syntax. As 
> long as
> we have two basic types of citations, the finer points of 
> formatting
> them can be achieved via other syntax, including the choice of
> stylesheet.

Good points. I guess what this boils down to is whether Org wants 
to be like LaTeX, where simple things are doable and complicated 
things possible, or Pandoc, where simple things are simple indeed 
and complicated things essentially impossible.

To clarify: in LaTeX (biblatex) you can mix footnote and in-text 
citations in a single document, Pandoc doesn't allow that. 
Pandoc's functionality is sufficient for a great majority of 
cases, but if you want or need to go beyond it, things get very 
difficult.

My suggestion would still be not to hard-code a limit on possible 
citation commands. Org itself should probably just provide the 
basics, but users and add-on packages should be allowed to define 
more specific commands with readable names and there should be a 
well-defined interface for doing so (just like users and packages 
can add new link types, for example).

Just my €0.02, of course.

-- 
Joost Kremers
Life has its moments

Re: wip-cite status question and feedback

[Richard Lawrence]

Joost Kremers <joostkremers@fastmail.fm> writes:

> Good points. I guess what this boils down to is whether Org wants 
> to be like LaTeX, where simple things are doable and complicated 
> things possible, or Pandoc, where simple things are simple indeed 
> and complicated things essentially impossible.
>
> To clarify: in LaTeX (biblatex) you can mix footnote and in-text 
> citations in a single document, Pandoc doesn't allow that. 
> Pandoc's functionality is sufficient for a great majority of 
> cases, but if you want or need to go beyond it, things get very 
> difficult.

Right. The Pandoc syntax trades some of the flexibility of (Bib)LaTeX
for the ability to render the citations it *does* support in a whole
bunch of non-LaTeX formats.

I personally think this is a good tradeoff, and one I would like to see
Org adopt. In both Org and Pandoc, you can use embedded LaTeX if you
need it. If you need the full power of BibLaTeX citations, then you are
confined to LaTeX export anyway, so you might as well just use BibLaTeX
commands in your document. But if you fall into the great majority of
use cases, you can use specialized citation syntax, and thereby get
reasonable behavior in other export formats too.

> My suggestion would still be not to hard-code a limit on possible 
> citation commands. Org itself should probably just provide the 
> basics, but users and add-on packages should be allowed to define 
> more specific commands with readable names and there should be a 
> well-defined interface for doing so (just like users and packages 
> can add new link types, for example).

I agree with this. I see no problem with having an analogue of
org-add-link-type for citations, and I think it's reasonable to have the
syntax allow for such extensions, so that e.g.

[cite/my-custom-cite-type: ...]

can still be recognized by the parser as a citation, which extensions
can then give a semantics to. But I think there needs to be a clear
syntactic delimitation between citations that are expected to work "out
of the box" (which to me primarily means: exported correctly in the
built-in backends) vs. those that need some additional extension to
export correctly or support additional behavior (which doesn't need to
be available on all backends, and could e.g. support BibLaTeX-only
users).  Otherwise the problem of getting citations working is too big a
project.

Best,
Richard

Re: wip-cite status question and feedback

[Bruce D'Arcus]

Just one question, Richard ...

On Sat, Apr 18, 2020 at 5:50 AM Richard Lawrence
<richard.lawrence@uni-tuebingen.de> wrote:

[...]

> I think it is worth pointing out to Bib(La)TeX users that it is useful
> to avoid a proliferation of citation commands in Org syntax. The syntax
> discussed so far achieves this by "factoring out" formatting information
> that BibLaTeX puts into the command into other parts of the syntax and
> into the choice of citation stylesheet. For example, instead of having
> \footnotecite and \parencite as separate commands, you can just have a
> single cite command, and the choice of stylesheet determines whether
> citations get formatted as footnotes or as in-text parenthetical
> citations or as something else.

Before the question, I did just want to add that this is an excellent point.

[...]

> My experience is that it's typically just two (e.g. parenthetical and
> author-in-text), and my memory of the earlier conversation was that most
> people agreed.

Just to align what you're saying and what I'm saying:

I see three commands in the pandoc syntax: standard/parenthetical,
author-in-text, and suppress-author; that look like so:

[@doe17]
@doe17
-@doe17

Implicit in what you wrote is the last one is not needed.

The question, then: Is that what you're saying; we don't need suppress-author?

I think I actually agree, though will add a topic that came up in the
CSL implementation discussion for the author-in-text styles in the
past few days.

Here's a common way a citation might be integrated in a narrative text:

Doe, by contrast, found negative results (2017).

So we have the author name in-text, than some text, then the year-only citation.

The traditional way to do that in pandoc is to use the suppress-author
command at the end.

Doe, by contrast, found negative results [-@doe17].

So the piece of information I refer to above is that one of the CSL
implementers (Frank Bennett) figured out  how to make the above
example an author-in-text variant, so that you don't need
suppress-author, and the entire sentence is the citation.

He did this by adding an optional "infix" variable to the citation.

So in that example, you would have:

- command: "author-in-text"
- citekey: "doe17"
- infix: "by contrast, found negative results"

This is arguably an edge case, but it does relate to the question of
whether we need two (standard and author-in-text) or three commands
(adding the suppress-author).

One could make the reasonable argument (I think, though not everyone
would agree) that the workaround for the above example is to use
author-in-text command but restructure the sentence:

@doe17, by contrast, found negative results.

From that perspective, I guess we indeed need only two commands:
standard (parenthetical) and author-in-text.

Bruce

Re: wip-cite status question and feedback

[Richard Lawrence]

Hi Bruce and all,

"Bruce D'Arcus" <bdarcus@gmail.com> writes:

> Just to align what you're saying and what I'm saying:
>
> I see three commands in the pandoc syntax: standard/parenthetical,
> author-in-text, and suppress-author; that look like so:
>
> [@doe17]
> @doe17
> -@doe17
>
> Implicit in what you wrote is the last one is not needed.
>
> The question, then: Is that what you're saying; we don't need suppress-author?

Ah, no, I didn't intend it like that. I am not very familiar with the
implementation details of pandoc-citeproc and wasn't aware that
suppress-author was a different type of citation command. I was
(vaguely) thinking of the third case as a "variant" of an in-text
citation type, rather than a separate type.

Actually, the Pandoc example you give seems to support this way of
thinking about it:  

> Doe, by contrast, found negative results [-@doe17].

That is a fourth case, right? "[-@doe17]" is not equivalent to "-@doe17"?

In other words, what we have here are two orthogonal distinctions:
parenthetical vs. in-text, and normal vs. author-suppressed. So, at
least on my funny way of counting ;), that's two citation "types", with
two "variants" within each of those types.

> one of the CSL implementers (Frank Bennett) figured out how to make
> the above example an author-in-text variant, so that you don't need
> suppress-author, and the entire sentence is the citation.
>
> He did this by adding an optional "infix" variable to the citation.
>
> So in that example, you would have:
>
> - command: "author-in-text"
> - citekey: "doe17"
> - infix: "by contrast, found negative results"
>
> This is arguably an edge case, but it does relate to the question of
> whether we need two (standard and author-in-text) or three commands
> (adding the suppress-author).
>
> One could make the reasonable argument (I think, though not everyone
> would agree) that the workaround for the above example is to use
> author-in-text command but restructure the sentence:
>
> @doe17, by contrast, found negative results.
>
> From that perspective, I guess we indeed need only two commands:
> standard (parenthetical) and author-in-text.

This way of writing the sentence seems less obvious to me than the
pandoc syntax. It also has the potential disadvantage that the choice
between rendering

"Doe (2017), by contrast, found negative results"

and

"Doe, by contrast, found negative results (2017)"

now has to be made at the level of the stylesheet instead of at the
level of the sentence where that citation appears. My instinct is that
this choice is informed by individual writing style and better made at
the level of the sentence. But you probably have a better sense than I
do of whether this is something that should at least sometimes be
controlled by the stylesheet. (Are there e.g. journals that always want
in-text citations to look like the latter case? I have no idea.)

In any case, if I'm right that this choice is usually better made at the
sentence level, then I think the syntax needs to support all four cases.
.The pandoc syntax does this, and I think the suppress-author variation
is probably needed often enough that we should have something similar.

-- 
Best,
Richard

Re: wip-cite status question and feedback

[Bruce D'Arcus]

On Sat, Apr 18, 2020 at 8:48 AM Richard Lawrence
<richard.lawrence@uni-tuebingen.de> wrote:
>
> Hi Bruce and all,
>
> "Bruce D'Arcus" <bdarcus@gmail.com> writes:
>
> > Just to align what you're saying and what I'm saying:
> >
> > I see three commands in the pandoc syntax: standard/parenthetical,
> > author-in-text, and suppress-author; that look like so:
> >
> > [@doe17]
> > @doe17
> > -@doe17
> >
> > Implicit in what you wrote is the last one is not needed.
> >
> > The question, then: Is that what you're saying; we don't need suppress-author?
>
> Ah, no, I didn't intend it like that.

Glad I asked then!

> I am not very familiar with the
> implementation details of pandoc-citeproc and wasn't aware that
> suppress-author was a different type of citation command. I was
> (vaguely) thinking of the third case as a "variant" of an in-text
> citation type, rather than a separate type.
>
> Actually, the Pandoc example you give seems to support this way of
> thinking about it:
>
> > Doe, by contrast, found negative results [-@doe17].
>
> That is a fourth case, right? "[-@doe17]" is not equivalent to "-@doe17"?

I think, notwithstanding a mistake I think I made in my previous
message, the "-" wouldn't be relevant to a bare author-in-text
citation command; the latter case.

So I think that's still three commands. Hopefully the below explains
why, but please let me know.

> In other words, what we have here are two orthogonal distinctions:
> parenthetical vs. in-text, and normal vs. author-suppressed. So, at
> least on my funny way of counting ;), that's two citation "types", with
> two "variants" within each of those types.

Just for clarity, for the record, "parenthetical" is the language of
author-date citation styles.

But what we're talking about with citet-like citations is broader than this.

In a numeric style, for example, you could have "Doe [3]"; so this
really applies to any style type (including end/footnote-based).

What we're doing is putting the author in the sentence; and outside
the citation.

This is why I'm using the more general language of  "author-in-text."

So three output cases, in author-date/numeric, where I've placed
content output by the citation processor in braces to distinguish it
from content entered by the user:

1. "Blah blah {(Doe, 2017)}"/"Blah blah {[3]}" -> default cite command
2. "{Doe (2017)}"/"{Doe [3]}" -> author-in-text command
3. "Doe blah blah {(2017)}"/"Doe blah blah {[3]}" -> suppress-author command

I can't see that it's necessary to have a fourth, because I think the
result of that would be this, which doesn't make any sense.

4.  "Doe blah blah {2017}"/"Doe blah blah {[3]}" ->
author-in-text+suppress-author command

Let us know what you think?

Bruce

Re: wip-cite status question and feedback

[Bruce D'Arcus]

But ...

On Sat, Apr 18, 2020 at 9:17 AM Bruce D'Arcus <bdarcus@gmail.com> wrote:

...

> I can't see that it's necessary to have a fourth, because I think the
> result of that would be this, which doesn't make any sense.
>
> 4.  "Doe blah blah {2017}"/"Doe blah blah {[3]}" ->
> author-in-text+suppress-author command

... notwithstanding that, I think Nicolas' latest proposed syntax
would support this anyway.

[citet:-@doe17]

Re: wip-cite status question and feedback

[Denis Maier]

> Bruce D'Arcus <bdarcus@gmail.com> hat am 18. April 2020 15:22 geschrieben:
> 
>  
> But ...
> 
> On Sat, Apr 18, 2020 at 9:17 AM Bruce D'Arcus <bdarcus@gmail.com> wrote:
> 
> ...
> 
> > I can't see that it's necessary to have a fourth, because I think the
> > result of that would be this, which doesn't make any sense.
> >
> > 4.  "Doe blah blah {2017}"/"Doe blah blah {[3]}" ->
> > author-in-text+suppress-author command
> 
> ... notwithstanding that, I think Nicolas' latest proposed syntax
> would support this anyway.
> 
> [citet:-@doe17]

Perhaps that can be used as author-only?

That would be:
[cite: @doe17] => (Doe 2017) [or a footnote, of course.)
[cite: -@doe17] => (2017)
[citet: @doe17] => Doe (2017)
[citet: -@doe17] => Doe

So we have like two basic citation types: One that is part of the narrative, one where the citation is set off from the main text. Both citations can be modified with a minus prefix. In one case this leads to "suppress author", in the other case this means "author only". (Using the same prefix for two different things can be considered problematic.)

Best,
Denis

Re: wip-cite status question and feedback

[denis.maier.lists]

> Bruce D'Arcus <bdarcus@gmail.com> hat am 18. April 2020 15:22 geschrieben:
> 
>  
> But ...
> 
> On Sat, Apr 18, 2020 at 9:17 AM Bruce D'Arcus <bdarcus@gmail.com> wrote:
> 
> ...
> 
> > I can't see that it's necessary to have a fourth, because I think the
> > result of that would be this, which doesn't make any sense.
> >
> > 4.  "Doe blah blah {2017}"/"Doe blah blah {[3]}" ->
> > author-in-text+suppress-author command
> 
> ... notwithstanding that, I think Nicolas' latest proposed syntax
> would support this anyway.
> 
> [citet:-@doe17]

Perhaps that can be used as author-only?

That would be:
[cite: @doe17] =>  (Doe 2017) [or a footnote, of course.)
[cite: -@doe17] =>  (2017)
[citet: @doe17] =>  Doe (2017)
[citet: -@doe17] =>  Doe

So we have like two basic citation types: One that is part of the narrative, one where the citation is set off from the main text. Both citations can be modified with a minus prefix. In one case this leads to "suppress author", in the other case this means "author only". (Using the same prefix for two different things can be considered problematic.)

Best,
Denis

Re: wip-cite status question and feedback

[Joost Kremers]

On Sat, Apr 18 2020, Bruce D'Arcus wrote:
> The question, then: Is that what you're saying; we don't need 
> suppress-author?
>
> I think I actually agree, though will add a topic that came up 
> in the
> CSL implementation discussion for the author-in-text styles in 
> the
> past few days.
>
> Here's a common way a citation might be integrated in a 
> narrative text:
>
> Doe, by contrast, found negative results (2017).

There are other constructions in with suppress-author is useful. 
For example, possessives:

Doe's (2017) results... 

or 

Doe's (2017) ground braking paper...

Such use is not uncommon in my field.

Also, consider languages that require a case ending on the author 
name. And who knows what quirks other languages may have?

> So we have the author name in-text, than some text, then the 
> year-only citation.
>
> The traditional way to do that in pandoc is to use the 
> suppress-author
> command at the end.
>
> Doe, by contrast, found negative results [-@doe17].
>
> So the piece of information I refer to above is that one of the 
> CSL
> implementers (Frank Bennett) figured out  how to make the above
> example an author-in-text variant, so that you don't need
> suppress-author, and the entire sentence is the citation.
>
> He did this by adding an optional "infix" variable to the 
> citation.

I doubt that would work in every case. Consider e.g.,
;
In her original (2017) study, Doe argues that... 

[...]
> This is arguably an edge case, but it does relate to the 
> question of
> whether we need two (standard and author-in-text) or three 
> commands
> (adding the suppress-author).

I would definitely say that a way to suppress the author is 
necessary. In fact, I would argue that there should also be a way 
to cite just the author name, if only to make sure it's not 
misspelled or written inconsistently. (I'm a regular user of 
\citeauthor in BibLaTeX... ;-) But I admit that's less of a 
necessity.


-- 
Joost Kremers
Life has its moments

Re: wip-cite status question and feedback

[Richard Lawrence]

"Bruce D'Arcus" <bdarcus@gmail.com> writes:

> I can't see that it's necessary to have a fourth, because I think the
> result of that would be this, which doesn't make any sense.
>
> 4.  "Doe blah blah {2017}"/"Doe blah blah {[3]}" ->
> author-in-text+suppress-author command
>
> Let us know what you think?

I think this could sometimes make sense. Granted, it wouldn't be very
often, but if e.g. you are citing something inside a wider parenthetical
remark, like:

  (Blah blah. However, Doe showed that not-blah; see her -@doe17.)

I can imagine that some style guides might forbid putting nested
parentheses in that position, so having a way to render "2017" instead
of "(2017)" would be useful.

Another case: I can imagine citation styles that use e.g. a work's title
(instead of its year) as the non-author identifier, in which case it
would often make sense to say things like

  Doe depicts blah in her -@doe17

as a way to output things like

  Doe depicts blah in her /Wondrous Novel/

Again, I don't know how important this is, or how widely used it would
be, but those are at least a couple of possibilities.

On the other hand, I notice that pandoc does not distinguish these
cases, at least with the default citation style; pandoc renders both
-@doe17 and [-@doe17] like "(2017)", so maybe it's not that important.

> ... notwithstanding that, I think Nicolas' latest proposed syntax
> would support this anyway.
>
> [citet:-@doe17]

Great. No objections from this corner, then!

-- 
Best,
Richard

Re: wip-cite status question and feedback

[Nicolas Goaziou]

Hello,

I cannot answer all open questions as the thread would spread too thin.
So, I'll try to subsume where Org is at the moment, and what need to be
decided.

First things first, I pushed a new branch, "wip-cite-new" in the
repository, with an modified implementation of citation syntax,
hopefully taking into consideration remarks made so far. I didn't squash
"wip-cite" because citeproc-org library suggests to use it, and I didn't
want to break it.

Instead of formally describing the syntax, here are a few boring
examples:

  [cite:@key]
  [cite:-@key]
  [cite:pre @key post]
  [cite:pre @key post; pre -@key2 post]
  [cite: common prefix; pre @key post; pre @key2 post; common suffix]

There is a limitation for prefixes and suffixes: they cannot contain
semicolons or closing square brackets. I'm not sure it is worth
implementing some character escape mechanism, tho.

Now with styles:

  [cite/style: ...]
  [cite/style/substyle: ...]
  [cite/mycitationprocessor/fullcite: ...]
  [cite/foot/text: ...]
  [cite/style/substyle/subsubstyle/OMG: ...]

The forward slash separator gives us local citation style and a name
space. There's no limit on the depth of sub-styles. However style
strings are limited to alphanumeric characters only.

I also removed short citations: 

  Lorem ipsum @doe09 dolor sit amet

They look nice, but I realized this was the same mistake as allowing [1]
to be a footnote reference. False positives are way too common. This
could generate frustration upon export.

I assume [cite:...] is the default citation style, defined at the
citation processor's level. Styled citations override locally the
default style. Again, a processor not handling a given style is expected
to fallback to default style.

As a consequence, there is no special syntax for "author-in-text" style.
But we can suggest one for back-end processors. We might want to stick
to the most complete one, BibLaTeX, IIUC, and /require/ processors to
support, at least:

  [cite/text: ...]
  [cite/paren: ...]

With this bare minimum, we ensure documents are somehow portable between
processors, and, therefore, export back-ends.

Again, this is only a proposal, feedback is welcome. Hopefully we can
move onto the next step: how should we interface citation processors and
Org?

First, I think we agreed on the BIBLIOGRAPHY keyword, with the following
syntax:

  #+BIBLIOGRAPHY: file
  #+BIBLIOGRAPHY: "file"

There can be multiple BIBLIOGRAPHY keywords in a document (and
equivalent node properties). Also, I think Org should support a global
variable, e.g., `org-citation-default-bibliography'.

I also think we defined a keyword to insert a bibliography:

  #+PRINT_BIBLIOGRAPHY: ... options ... (may be specific to citation processors)

I expect the processors to provide Org with a maximum two different
actions: manage and export. For example, AFAIK, Org Ref manages and
exports, citeproc-org/citeproc-el only exports, and a default processor
might realistically limit export to LaTeX and derived and management to
a default fontification of citations.

- Management :: 

  As suggested by John Kitchin, we want something like
  `org-link-parameters'. However, I don't think it makes much sense to
  let different citation processors handle different citations in the
  same buffer.

  Instead we can provide one global function for each of these features:
  - completion
  - fontification (possibly special keymaps, help-echo, etc)
  - following
  - am I missing something?

  Citation processors may operate on the bibliography file, but that's
  out of the scope of Org.
  
  For example, we could introduce the variable
  `org-citation-follow-function', which contains a function called with
  two arguments: the key of the citation to follow, and the list of
  bibliography files active in the buffer. Each citation processor could
  set this function. By default, it would probably be

    (defun org-citation-follow-default (&rest _)
      (message "Please activate a citation processor to follow citations")
      nil)
  
- Export :: 

  In this case, we may want to allow multiple processors for various
  export back-ends. I thought about declaring active processors in
  a document with a keyword, e.g.,

    #+CITATION_PROCESSOR: org-ref :default-style foo :back-ends (latex)
    #+CITATION_PROCESSOR: citeproc :default-style bar 

  and with a global variable, e.g., `org-citation-export-default' which
  could be, e.g.,

    '((default :defaut-style "authoryear" :back-ends (latex)))

  but could become, with appropriate libraries

    '((org-ref :defaut-style "authoryear" :back-ends (latex))
      (citeproc-org :default-style "style-file.csl" :back-ends nil))

  where more specialized back-ends are used first. Note that `latex'
  would mean `latex' and derived back-ends, e.g., `beamer'.

Well, that's all for now. Again, I am not a citation specialist, so
I need feedback. Let's keep the ball rolling!


Regards,

-- 
Nicolas Goaziou

Re: wip-cite status question and feedback

[Bruce D'Arcus]

First, thanks for your work on this Nicolas; really awesome to see the progress!

I'm just going to address your syntax/cite command question.

I don't have concerns about the other details, and I think others are
better positioned to comment on those ...

On Sat, Apr 25, 2020 at 12:20 PM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:

...

> I assume [cite:...] is the default citation style, defined at the
> citation processor's level. Styled citations override locally the
> default style. Again, a processor not handling a given style is expected
> to fallback to default style.
>
> As a consequence, there is no special syntax for "author-in-text" style.
> But we can suggest one for back-end processors. We might want to stick
> to the most complete one, BibLaTeX, IIUC, and /require/ processors to
> support, at least:
>
>   [cite/text: ...]
>   [cite/paren: ...]
>
> With this bare minimum, we ensure documents are somehow portable between
> processors, and, therefore, export back-ends.

...

So in this approach, we have a single core "cite" command, and
everything else is a namespaced extension?

My understanding, though, is that org "cite" would default to your
last example I quote above (in natibib, citep); that there's no need
for a dedicated "cite/paren" command, either reserved or not.

So by default, the "cite" command might yield something like this on
output (of course, depending on processor)?

- to natbib/latex = "\citep{doe18}"

For final HTML output (say using citeproc-el/org), something like:

- author-date = "(Doe, 2018)"
- number = "[3]"
- note = "2" (represented as a footnote or endnote, of course)

... etc.

And then we need a mechanism to do the textual variant (natbib citet);
"cite/text" makes sense to me.

Given how common that is (In natbib, it and citep are the two core
commands), is there any downside to reserving that?

And then I guess the "suppress-author" variant would be something like
"cite/year" or "cite/suppress-author"?

Bruce

Re: wip-cite status question and feedback

[Nicolas Goaziou]

Hello,

"Bruce D'Arcus" <bdarcus@gmail.com> writes:

> On Sat, Apr 25, 2020 at 12:20 PM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:
>

[...]

>>
>>   [cite/text: ...]
>>   [cite/paren: ...]
>>

> So in this approach, we have a single core "cite" command, and
> everything else is a namespaced extension?

Indeed.

> My understanding, though, is that org "cite" would default to your
> last example I quote above (in natibib, citep); that there's no need
> for a dedicated "cite/paren" command, either reserved or not.

Not necessarily. "cite" means default value, whatever that is. It could,
for example, mean: "cite/text" for every citation, if that is what you
use the most. In that case, "cite/paren" is necessary, to override it
locally. It could also be, e.g., "cite/footnote", then both "cite/text"
and "cite/paren" could be of some use. That was suggested by Richard
Lawrence in this thread, if my memory serves me right.

Does that make sense?

> So by default, the "cite" command might yield something like this on
> output (of course, depending on processor)?
>
> - to natbib/latex = "\citep{doe18}"
>
> For final HTML output (say using citeproc-el/org), something like:
>
> - author-date = "(Doe, 2018)"
> - number = "[3]"
> - note = "2" (represented as a footnote or endnote, of course)
>
> ... etc.
>
> And then we need a mechanism to do the textual variant (natbib citet);
> "cite/text" makes sense to me.

I assume this would be the more common configuration, indeed.

> Given how common that is (In natbib, it and citep are the two core
> commands), is there any downside to reserving that?

As I wrote, we can reserve "cite/text" already.

Could we find something shorter for such a common need? Well, I didn't
find any syntax compelling enough—I don't like special casing. For
example, having both "citeX" and "cite/XXX", as suggested by Denis
Maier, is a bit convoluted, IMO. E.g., having both "citet" and
"cite/text" would just add confusion to the system, IMO.

Besides, "cite/text" is not that difficult to type. Moreover, you would
probably use a tool to insert the citation anyway.

This is not an irrevocable decision, of course. I merely suggested and
implemented one syntax, but I'm still open to suggestions.

> And then I guess the "suppress-author" variant would be something like
> "cite/year" or "cite/suppress-author"?

The syntax still includes the "suppress-author" mechanism:
[cite:-@doe20].

It could be redundant with "cite/suppress-author", indeed. We can keep
it nonetheless. We can also decide to remove the "-@key" special syntax.
Or, we could also consider this idea to be an interesting one, and
extend it, with, e.g., [cite:!@doe20], which could be a shortcut for
[cite/text:@doe20].

Special cases… Everything's possible. You tell me.

Regards,

-- 
Nicolas Goaziou

Re: wip-cite status question and feedback

[Bruce D'Arcus]

On Sat, Apr 25, 2020 at 4:03 PM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:

...

> > My understanding, though, is that org "cite" would default to your
> > last example I quote above (in natibib, citep); that there's no need
> > for a dedicated "cite/paren" command, either reserved or not.
>
> Not necessarily. "cite" means default value, whatever that is. It could,
> for example, mean: "cite/text" for every citation, if that is what you
> use the most. In that case, "cite/paren" is necessary, to override it
> locally. It could also be, e.g., "cite/footnote", then both "cite/text"
> and "cite/paren" could be of some use. That was suggested by Richard
> Lawrence in this thread, if my memory serves me right.
>
> Does that make sense?

I think so. I'll defer to Richard on this, since he was making this point.

...

> > Given how common that is (In natbib, it and citep are the two core
> > commands), is there any downside to reserving that?
>
> As I wrote, we can reserve "cite/text" already.
>
> Could we find something shorter for such a common need? Well, I didn't
> find any syntax compelling enough—I don't like special casing. For
> example, having both "citeX" and "cite/XXX", as suggested by Denis
> Maier, is a bit convoluted, IMO. E.g., having both "citet" and
> "cite/text" would just add confusion to the system, IMO.
>
> Besides, "cite/text" is not that difficult to type. Moreover, you would
> probably use a tool to insert the citation anyway.
>
> This is not an irrevocable decision, of course. I merely suggested and
> implemented one syntax, but I'm still open to suggestions.

I support this decision, for all the reasons you mention; no problem at all.

> > And then I guess the "suppress-author" variant would be something like
> > "cite/year" or "cite/suppress-author"?
>
> The syntax still includes the "suppress-author" mechanism:
> [cite:-@doe20].

Oh right; good.

> It could be redundant with "cite/suppress-author", indeed. We can keep
> it nonetheless. We can also decide to remove the "-@key" special syntax.
> Or, we could also consider this idea to be an interesting one, and
> extend it, with, e.g., [cite:!@doe20], which could be a shortcut for
> [cite/text:@doe20].
>
> Special cases… Everything's possible. You tell me.

I vote to keep the "-".

On your last suggestion (the "!" citet shortcut), how valuable it would be would
really depend on the UX; how commonly, per your point above, one would be using
a tool to insert this citation command.

Bruce

Re: wip-cite status question and feedback

[Denis Maier]

Hello,

Am 25.04.2020 um 18:19 schrieb Nicolas Goaziou:
> Hello,
>
> I cannot answer all open questions as the thread would spread too thin.
> So, I'll try to subsume where Org is at the moment, and what need to be
> decided.

Thanks for the suggestion. Looks like a pretty solid approach.

> There is a limitation for prefixes and suffixes: they cannot contain
> semicolons or closing square brackets. I'm not sure it is worth
> implementing some character escape mechanism, tho.

What about using quotes if someone needs this, like so [cite: "common 
prefix; still common prefix"; pre @key post; pre @key2 post; common 
suffix] ?


>
> Now with styles:
>
>    [cite/style: ...]
>    [cite/style/substyle: ...]
>    [cite/mycitationprocessor/fullcite: ...]
>    [cite/foot/text: ...]
>    [cite/style/substyle/subsubstyle/OMG: ...]
>
> The forward slash separator gives us local citation style and a name
> space. There's no limit on the depth of sub-styles. However style
> strings are limited to alphanumeric characters only.

Good solution.

> I assume [cite:...] is the default citation style, defined at the
> citation processor's level. Styled citations override locally the
> default style. Again, a processor not handling a given style is expected
> to fallback to default style.
>
> As a consequence, there is no special syntax for "author-in-text" style.
> But we can suggest one for back-end processors. We might want to stick
> to the most complete one, BibLaTeX, IIUC, and /require/ processors to
> support, at least:
>
>    [cite/text: ...]
>    [cite/paren: ...]
>
> With this bare minimum, we ensure documents are somehow portable between
> processors, and, therefore, export back-ends.
>

> Hopefully we can
> move onto the next step: how should we interface citation processors and
> Org?
>
> First, I think we agreed on the BIBLIOGRAPHY keyword, with the following
> syntax:
>
>    #+BIBLIOGRAPHY: file
>    #+BIBLIOGRAPHY: "file"
>
> There can be multiple BIBLIOGRAPHY keywords in a document (and
> equivalent node properties). Also, I think Org should support a global
> variable, e.g., `org-citation-default-bibliography'.
>
> I also think we defined a keyword to insert a bibliography:
>
>    #+PRINT_BIBLIOGRAPHY: ... options ... (may be specific to citation processors)

Looks good.

>
> - Export ::
>
>    In this case, we may want to allow multiple processors for various
>    export back-ends. I thought about declaring active processors in
>    a document with a keyword, e.g.,
>
>      #+CITATION_PROCESSOR: org-ref :default-style foo :back-ends (latex)
>      #+CITATION_PROCESSOR: citeproc :default-style bar
>
>    and with a global variable, e.g., `org-citation-export-default' which
>    could be, e.g.,
>
>      '((default :defaut-style "authoryear" :back-ends (latex)))
>
>    but could become, with appropriate libraries
>
>      '((org-ref :defaut-style "authoryear" :back-ends (latex))
>        (citeproc-org :default-style "style-file.csl" :back-ends nil))
>
>    where more specialized back-ends are used first. Note that `latex'
>    would mean `latex' and derived back-ends, e.g., `beamer'.
>
> Well, that's all for now. Again, I am not a citation specialist, so
> I need feedback. Let's keep the ball rolling!

Does that mean you'll be able to have the same or different processors 
for different backends? (Like biblatex for latex and citeproc-el for 
ODT/HTML/etc.; or when you need identical output you can use citeproc-el 
even for latex?)

Again, thanks for all your work here.

All the best,

Denis

Re: wip-cite status question and feedback

[Richard Lawrence]

"Bruce D'Arcus" <bdarcus@gmail.com> writes:

>> > My understanding, though, is that org "cite" would default to your
>> > last example I quote above (in natibib, citep); that there's no need
>> > for a dedicated "cite/paren" command, either reserved or not.
>>
>> Not necessarily. "cite" means default value, whatever that is. It could,
>> for example, mean: "cite/text" for every citation, if that is what you
>> use the most. In that case, "cite/paren" is necessary, to override it
>> locally. It could also be, e.g., "cite/footnote", then both "cite/text"
>> and "cite/paren" could be of some use. That was suggested by Richard
>> Lawrence in this thread, if my memory serves me right.
>>
>> Does that make sense?
>
> I think so. I'll defer to Richard on this, since he was making this point.

Sorry to take so long to reply. The point I made earlier was that, as
far as I understand, the choice of CSL stylesheet is the main factor
determining how a given citation gets rendered into the output (assuming
you process citations with CSL). So yes, it makes sense to have "cite"
mean default value as determined by the choice of stylesheet.

I've been skimming the CSL documentation, and I'm realizing that I
actually don't have a very good understanding of how these different
types of commands would be represented at the level of a CSL processor.

Bruce, is it possible to have a CSL stylesheet that would be able to
accommodate both e.g. "cite/paren" and "cite/footnote" in the same
document? Can a stylesheet support an arbitrary numbers of different
citation types like this, and can a CSL processor choose among those
types based on their *names*?

If so, then I think Nicolas' proposal to have "cite" mean default and
make non-default citations available as "cite/xxx" makes sense
(especially with the other syntax supporting suppress-author, etc.).

If not, then the "cite/xxx" syntax makes less sense to me; it just sort
of looks like a different way of writing BibLaTeX commands, and will be
hard to support when LaTeX is not the output format. I would be hesitant
in that case to make "cite/xxx" the standard way to express "this
citation should be rendered in manner xxx, instead of the default".

Best,
Richard

Re: wip-cite status question and feedback

[Bruce D'Arcus]

On Fri, May 1, 2020 at 1:38 PM Richard Lawrence
<richard.lawrence@uni-tuebingen.de> wrote:
>
> "Bruce D'Arcus" <bdarcus@gmail.com> writes:
>
> >> > My understanding, though, is that org "cite" would default to your
> >> > last example I quote above (in natibib, citep); that there's no need
> >> > for a dedicated "cite/paren" command, either reserved or not.
> >>
> >> Not necessarily. "cite" means default value, whatever that is. It could,
> >> for example, mean: "cite/text" for every citation, if that is what you
> >> use the most. In that case, "cite/paren" is necessary, to override it
> >> locally. It could also be, e.g., "cite/footnote", then both "cite/text"
> >> and "cite/paren" could be of some use. That was suggested by Richard
> >> Lawrence in this thread, if my memory serves me right.
> >>
> >> Does that make sense?
> >
> > I think so. I'll defer to Richard on this, since he was making this point.
>
> Sorry to take so long to reply. The point I made earlier was that, as
> far as I understand, the choice of CSL stylesheet is the main factor
> determining how a given citation gets rendered into the output (assuming
> you process citations with CSL). So yes, it makes sense to have "cite"
> mean default value as determined by the choice of stylesheet.

I was trying to not assume CSL in this discussion, but certainly
that's how I was thinking.

> I've been skimming the CSL documentation, and I'm realizing that I
> actually don't have a very good understanding of how these different
> types of commands would be represented at the level of a CSL processor.

Yes, because it's currently silent on this issue. In part because of
this conversation, we've decided to add a section that deals with this
in the context of a generic API.

> Bruce, is it possible to have a CSL stylesheet that would be able to
> accommodate both e.g. "cite/paren" and "cite/footnote" in the same
> document?

No. We could extend CSL to do that, but I don't actually recall ever
getting that request.

> Can a stylesheet support an arbitrary numbers of different
> citation types like this, and can a CSL processor choose among those
> types based on their *names*?

CSL currently has a single citation template, so effectively one
system, defined in that style file.

CSL is silent on the rest, since we wanted to see how the ecosystem
evolved to respond to user needs. We left it up to the implementers on
how to address integrating.

But per above, we have enough information at this point to make the
silent explicit.

Based on that, what we have seen is what I've described here: two core
commands (cite and suppress-author), and now a third iteration of that
which we will add explicitly to the spec and schema in the next
release: the equivalent of citet.

This turns out relatively simple to implement by building on existing
CSL concepts, where output becomes, for example, "[author-only] " +
"([suppress-author])".

This is already supported in CSL, but we will add a new "intext"
element to allow one to define the author-only rendering for styles
(like APA) where the output is different than it is in the citation.

So to sum up, I expect we will explicitly define three commands:
default (the one defined in the citation template of the style),
suppress-author (which need not be explicitly defined in the style,
since the processor knows how to achieve this), and cite-text.

> If so, then I think Nicolas' proposal to have "cite" mean default and
> make non-default citations available as "cite/xxx" makes sense
> (especially with the other syntax supporting suppress-author, etc.).
>
> If not, then the "cite/xxx" syntax makes less sense to me; it just sort
> of looks like a different way of writing BibLaTeX commands, and will be
> hard to support when LaTeX is not the output format. I would be hesitant
> in that case to make "cite/xxx" the standard way to express "this
> citation should be rendered in manner xxx, instead of the default".
>
> Best,
> Richard

Re: wip-cite status question and feedback

[Nicolas Goaziou]

Hello,

Thank you for the feedback.

Denis Maier <denis.maier.lists@mailbox.org> writes:

> What about using quotes if someone needs this, like so [cite: "common
> prefix; still common prefix"; pre @key post; pre @key2 post; common
> suffix] ?

Then we would have to find a way to escape the quote…

> Does that mean you'll be able to have the same or different processors
> for different backends? (Like biblatex for latex and citeproc-el for
> ODT/HTML/etc.; or when you need identical output you can use
> citeproc-el even for latex?)

With the suggested set-up, you associate a single processor to each
export back-end. The same processor can be associated to multiple export
back-ends.

Moreover, you can change this association globally, or per document.

Is that sufficient?

Regards,

-- 
Nicolas Goaziou

Re: wip-cite status question and feedback

[Bruce D'Arcus]

On Sat, May 2, 2020 at 5:51 AM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:

...

> > Does that mean you'll be able to have the same or different processors
> > for different backends? (Like biblatex for latex and citeproc-el for
> > ODT/HTML/etc.; or when you need identical output you can use
> > citeproc-el even for latex?)
>
> With the suggested set-up, you associate a single processor to each
> export back-end. The same processor can be associated to multiple export
> back-ends.
>
> Moreover, you can change this association globally, or per document.
>
> Is that sufficient?

To my mind, it's perfect.

Bruce

Re: wip-cite status question and feedback

[Nicolas Goaziou]

Hello,

Richard Lawrence <richard.lawrence@uni-tuebingen.de> writes:

> If so, then I think Nicolas' proposal to have "cite" mean default and
> make non-default citations available as "cite/xxx" makes sense
> (especially with the other syntax supporting suppress-author, etc.).
>
> If not, then the "cite/xxx" syntax makes less sense to me; it just sort
> of looks like a different way of writing BibLaTeX commands, and will be
> hard to support when LaTeX is not the output format. I would be hesitant
> in that case to make "cite/xxx" the standard way to express "this
> citation should be rendered in manner xxx, instead of the default".

Note that I only followed as many requests from participants to this
thread as possible.

Anyway, I'm a bit lost here. The point to the syntax is to support
Citeproc as well as Bib(La)TeX and Org Ref, so it has to deal with both
processors using a limited set of cite commands, and processors with an
awful lot of cite commands. On top of that, some users requested name
spaces for custom commands, which only makes sense in Bib(La)TeX
context, IIUC. So, there we are.

I suggested to support at least "cite", "cite/text" and "cite/paren",
but it sounds like "cite/paren" is not possible with Citeproc. This
doesn't matter much, we can limit the supported set to "cite" and
"cite/text" in Citeproc.

I think there are really two paths here: either we only support the
common denominator between all processors, like, e.g., Pandoc, or we
handle every possible command, knowing that most of them will not be
portable anyways.

WDYT?

Regards,

-- 
Nicolas Goaziou

Re: wip-cite status question and feedback

[Bruce D'Arcus]

On Sat, May 2, 2020 at 9:13 AM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:

> I suggested to support at least "cite", "cite/text" and "cite/paren",
> but it sounds like "cite/paren" is not possible with Citeproc. This
> doesn't matter much, we can limit the supported set to "cite" and
> "cite/text" in Citeproc.

Just to clarify here (hopefully I do; I know this is complicated), in
author-date CSL styles, the default is effectively cite/paren.

Here's the citation template layout for APA, for example:

https://github.com/citation-style-language/styles/blob/master/apa.csl#L1672

Which means "(Doe, 2018)".

Suppress-author and intext variants modify that default.

Bruce

Re: wip-cite status question and feedback

[Nicolas Goaziou]

Hello,

"Bruce D'Arcus" <bdarcus@gmail.com> writes:

> So to sum up, I expect we will explicitly define three commands:
> default (the one defined in the citation template of the style),
> suppress-author (which need not be explicitly defined in the style,
> since the processor knows how to achieve this), and cite-text.

So, is there anything wrong with [cite:@key], [cite:-@key] and
[cite/text:@key] per above?

In particular, cite-text sounds like another non-default style to me,
rather than a derivative of the default style, and if it does, this
warrants introducing a "cite/text" syntax. E.g., what happens if default
style is footnote-like and cite-text is used?

Also, I've had a cursory look at "citeproc-el" implementation, and there
is apparent support for capitalized citations. You don't seem to talk
about this. If such a thing exists, we need to introduce another marker
at the cite key level (like suppress-author).

As a last, more technical point, I'm thinking about rendering citations
in a pre-export phase, where the processor is handled a list of all
citations as Org objects (so you can extract context about them, e.g.,
footnote label around it if applicable) where all prefixes and suffixes
are already in the target format.

More specifically, as an inaccurate but enough for the point example, in
the document

  Go ahead, make my day [cite:@harry83 at *0:23:18*].

assuming target is LaTeX code, the processor would see something like.

  ((citation ... (citation-reference :key "harry83" :suffix " at \bold{0:23:18}")))

In particular, does Citeproc handle raw LaTeX, or more generally, any
code in (pre|suf)fix, as long as the locator is accessible? I assume so,
but I'd rather ask.

Regards,  

-- 
Nicolas Goaziou

Re: wip-cite status question and feedback

[Nicolas Goaziou]

It seems you didn't copy the list. I add it again.

> No, I think that should be fine. (Perhaps also a fourth one for
> author-only. And what about nocite?)

Sorry. I wasn't clear.

There is still full support for styles behind the suggested syntax,
e.g., [cite/author: ...], [cite/nocite: ...] (this one is odd). I was
pointing out that we cover Citeproc needs, and more.

> Author in text, the rest in a footnote.

So it is not really a new style; you can have cite-text on top of any
style. This might be a problem. Either we invent an alternate syntax,
with duplicated styles, e.g.

  [cite: ...]     [cite/style: ...] 
  [cite*: ...]    [cite*/style: ...]

this was already suggested in this thread (with "citet").

Or we make use of sub-styles, e.g.

  [cite: ...]       [cite/foot: ...]
  [cite/text: ...]  [cite/foot/text: ...]

This is ambiguous, tho: is it "cite/foot/text" or "cite/text/foot"?

Of course, this is an issue for BibLaTeX only. AFAIU, [cite/text: ...]
is totally unambiguous for Citeproc.

What do Bib(La)TeX users think about it?

> That doesn't exist in CSL. It could be useful though.

It is odd that citeproc-el offers this, then.

> citeproc-js handles pseudo-html, with pandoc-citeproc it's possible to
> use markdown, but I think also raw HTML should be supported...

It sounds good enough, then. Besides, i assume markup in prefix/suffix
is not common.

Thank you.

Re: wip-cite status question and feedback

[Denis Maier]

Am 02.05.2020 um 18:34 schrieb Nicolas Goaziou:

> It seems you didn't copy the list. I add it again.
>
>> No, I think that should be fine. (Perhaps also a fourth one for
>> author-only. And what about nocite?)
> Sorry. I wasn't clear.
>
> There is still full support for styles behind the suggested syntax,
> e.g., [cite/author: ...], [cite/nocite: ...] (this one is odd). I was
> pointing out that we cover Citeproc needs, and more.
Yeah, and I was pointing out that these might be necessary from a 
CSL/citeproc perspective.
>
>> Author in text, the rest in a footnote.
> So it is not really a new style; you can have cite-text on top of any
> style. This might be a problem.
Why? I can't follow you here...
> Either we invent an alternate syntax,
> with duplicated styles, e.g.
>
>    [cite: ...]     [cite/style: ...]
>    [cite*: ...]    [cite*/style: ...]
>
> this was already suggested in this thread (with "citet").
>
> Or we make use of sub-styles, e.g.
>
>    [cite: ...]       [cite/foot: ...]
>    [cite/text: ...]  [cite/foot/text: ...]
>
> This is ambiguous, tho: is it "cite/foot/text" or "cite/text/foot"?
>
> Of course, this is an issue for BibLaTeX only. AFAIU, [cite/text: ...]
> is totally unambiguous for Citeproc.
>
> What do Bib(La)TeX users think about it?

I don't think it's a real problem. In CSL it's really clear. The CSL 
style defines the overall style, i.e.:

cite => "Doe 2020" in parentheses or in a note
cite/test => "Doe" in text, "2020" in parentheses or in a note.

And I doubt it's a problem for biblatex:

cite => autocite (or just cite, but I think autocite is a better choice)
cite/text => textcite
cite/foot => footcite

I don't think duplicate styles or sub-styles are needed.

Best,

Denis

Re: wip-cite status question and feedback

[Richard Lawrence]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> I think there are really two paths here: either we only support the
> common denominator between all processors, like, e.g., Pandoc, or we
> handle every possible command, knowing that most of them will not be
> portable anyways.

Yes, I think that is the core issue: which way to do we want to go here?

My opinion has always been that it makes more sense to just support the
common denominator at the level of Org citation syntax, for two reasons:
(1) it makes implementing a good solution that will work for a lot of
cases much more feasible, and (2) anyone who really needs more than the
common denominator -- that is to say, anyone who needs BibLaTeX -- can
already write arbitrary LaTeX snippets directly in an Org document. Thus
the latter group doesn't really lose anything if the syntax only
supports the common denominator, while everyone else has a lot to gain
from an implementation of citation syntax that can be exported on other
backends.

On the other hand, this opinion is narrowly focused on the issue of how
citation syntax can be rendered into citations when exporting a
document. When I think about the other uses for the syntax that e.g.
John Kitchin has talked about in this thread, and that something like a
future org-ref could support, then I see that people who need to export
citations as BibLaTeX *would* still be missing out if they couldn't use
the citation syntax.

So, I think it is good if the syntax can support advanced BibLaTeX users
too, and it looks to me like the "cite/xxx" syntax can do that. I have
no objections to the syntax you've proposed, Nicolas.

I *do* think it's worth marking a clear distinction between citation
syntax that can vs. cannot be expected to export correctly on non-LaTeX
backends. It looks to me at the moment like that distinction will be
expressed as the difference between "cite" and "cite/xxx". For me,
that's a reason to make "cite/text" a special case and give it its own
syntax, since this is such an important and widespread use case, and CSL
supports it. But I don't feel that strongly about this. For me, it would
be fine if it's mentioned as a special case in the manual.

-- 
Best,
Richard

Re: wip-cite status question and feedback

[Bastien]

Hi Bruce,

"Bruce D'Arcus" <bdarcus@gmail.com> writes:

> I'm not sure of the value of this sort of question thrown in the
> middle of a long-running, many year, conversation. You seem to assume
> nobody considered this.

Well, this sounded a bit harsh, problably more than what was intended.

> But to answer anyway ...

And your answer was precisely what I was (also) looking for, so thanks
for it.  I haven't followed nor helped developments in this area but I
hope this can settle down and be widely available.

Best,

-- 
 Bastien

Re: wip-cite status question and feedback

[Bruce D'Arcus]

Hi Bastian,


On Sun, May 24, 2020 at 8:12 AM Bastien <bzg@gnu.org> wrote:
>
> Hi Bruce,
>
> "Bruce D'Arcus" <bdarcus@gmail.com> writes:
>
> > I'm not sure of the value of this sort of question thrown in the
> > middle of a long-running, many year, conversation. You seem to assume
> > nobody considered this.
>
> Well, this sounded a bit harsh, problably more than what was intended.

Yes, probably true.

I was trying to be direct, to avoid a potentially long-winded diversion.

But it was a bit harsh. Sorry Gustav.

> > But to answer anyway ...
>
> And your answer was precisely what I was (also) looking for, so thanks
> for it.  I haven't followed nor helped developments in this area but I
> hope this can settle down and be widely available.

Indeed; thanks!

Bruce

Re: wip-cite status question and feedback

[András Simonyi]

Hi,

apologies for reacting that late (it seems that I messed up my email filtering
royally) -- it is very nice to see progress in this area. Thanks to all of you
for trying to bring this forward, and, of course, to Bruce for initiating the
thread.

I think that the syntax that has crystallized is a good starting point
for having
dedicated citation support in Org, so in the following I'll concentrate on some
of the infrastructure issues raised.

(i) Default ("built in") citation processor in Org

I think citeproc-el could be a reasonable solution for this. The core is
well-tested (thanks to the CSL test suite), implements a fairly large subset of
the CSL 1.01 standard, and can output citations in a number of formats including
in Org syntax. If I understood correctly, this would require citeproc-el to be
part of Emacs, and not even an ELPA package would suffice. In principle I'm open
to this, but I'm not familiar with the process and there are potential pitfalls,
e.g. citeproc-el currently depends on some "modern" libraries, e.g., "s" or
"dash" which AFAIK are not in Emacs.

(ii) Citation API

NG == Nicolas Goaziou wrote on 8 Apr 2020

  NG> 2. [ ] Decide about API Org should provide for it to be useful. Here are
  NG> some low hanging fruits:
  NG>    - [ ] List all ".bib" files associated to the document,
  NG>    - [ ] List all citations,
  NG>    - [ ] Return citation key at point, if any.
  NG>    - Anything else?

From the citeproc-el (and CSL) perspective, the list of bibliography database
files, the place where the bibliography should be placed (if it's specified) and
the list of citations would be enough, I think. For the proper handling of
footnote-based styles it would be important to provide footnote information
about each citation: is a citation in a footnote and what is its footnote number
(index in the list of footnotes)? (citeproc-org contains a function to collect
this info.)

  NG> Moreover, we need to decide about how external processors could plug into
  NG> the export framework.
  NG>    - [ ] For example, it could be a simple variable bound to a list of
  NG> functions. Each function accepts three arguments: the export back-end, as
  NG> a symbol, the full citation, as a list of triplets (key, prefix, suffix)
  NG> along with global prefix/suffix, and the usual INFO communication channel.
  NG> Does it need more?

Unfortunately I'm unfamiliar with the Org exporter, so this might not be a
problem at all, but there is a seemingly tricky point during export, related,
again, to footnotes in footnote-based citation styles: If a footnote style is
used and a citation is not already in a footnote then a footnote has to be
dynamically generated around the citation during export. The simplest way of
doing this is to wrap the rendered citation (which can contain backend-specific
font-properties, e.g. smallcaps) into an inline Org footnote block. E.g.,
citeproc-org, which basically acts as an Org->Org preprocessor, generates
dynamical footnote citations for html export along the lines of

[fn:: @@html:<span style="font-variant:small-caps">Gödel</span> 1931@@]

If this scenario can be handled by the proposed mechanism then I don't expect
any other problems.

  NG>    - [ ] Also, the prefix/suffix may contain some Org markup, so this
  NG> needs to be also processed. Should it happen before, or after the external
  NG> processor does its job? I.e., should the function translate into Org or
  NG> target format?

This is a very good (and a bit tricky...) question! Since the proposed Org
citation syntax does not contain locator-related slots, the citation processor
has to locate this information in the affixes (in citeproc-org, in the suffix),
parse and remove it. This part should definitely be done in Org format, so I'd
say the Org markup rendering should be after the citation rendering. (With the
proviso that part of the citation processor's output can already be rendered in
the target format as in the previous example.)

  NG> 3. [ ] Specify the kind of basic translation that be done out of the box?
  NG> Ideally, every non-derived back-end should do something, even basic.
  NG> Therefore, we need to propose some translation pattern for each of the
  NG> following:
  NG>    - [ ] ASCII
  NG>    - [ ] HTML
  NG>    - [ ] LaTeX
  NG>    - [ ] ODT
  NG>    - [ ] Texinfo

As an Org->Org preprocessor, citeproc-org supports all these back-ends in the
sense that for HTML and LaTeX it outputs target specific rendering using the
@@latex and @@html syntax and generic Org for the rest.

(iii) Citation visualization

JK == John Kitchin wrote on 8 Apr 2020

  JK> org-ref relies very heavily on the link functionality to provide actions
  JK> on a cite, e.g. to open the pdf, or url, allow sorting, to change the cite
  JK> color when it is a bad key, etc If the new syntax also has that
  JK> capability

I'd add to this that it would be very useful if citations -- similarly to links
-- had a "descriptive" (as opposed to "literal") rendering mode in the buffer as
well, which would show the rendered citation, as produced by a citation
processor (if available). Similarly to links one could toggle between the two
display modes.

(iv) Bibliography format

This might go without saying but perhaps it's worth specifying which
bibliography format(s) should be supported by the "out of the box"
functionality.
My impression is that despite the clear superiority of biblatex and CSL, BibTeX
is still the most widely used format, and should be the first/main target. Later
on, of course, it would be nice to support others, e.g., using a CSL processor
one basically gets CSL-JSON support for free, and there could be even an Org-CSL
format, by analogy with Org-BibTex. What do you think?

Sorry again for replying only now & best regards,

András

On Sun, 24 May 2020 at 15:18, Bruce D'Arcus <bdarcus@gmail.com> wrote:
>
> Hi Bastian,
>
>
> On Sun, May 24, 2020 at 8:12 AM Bastien <bzg@gnu.org> wrote:
> >
> > Hi Bruce,
> >
> > "Bruce D'Arcus" <bdarcus@gmail.com> writes:
> >
> > > I'm not sure of the value of this sort of question thrown in the
> > > middle of a long-running, many year, conversation. You seem to assume
> > > nobody considered this.
> >
> > Well, this sounded a bit harsh, problably more than what was intended.
>
> Yes, probably true.
>
> I was trying to be direct, to avoid a potentially long-winded diversion.
>
> But it was a bit harsh. Sorry Gustav.
>
> > > But to answer anyway ...
> >
> > And your answer was precisely what I was (also) looking for, so thanks
> > for it.  I haven't followed nor helped developments in this area but I
> > hope this can settle down and be widely available.
>
> Indeed; thanks!
>
> Bruce

Re: wip-cite status question and feedback

[Bruce D'Arcus]

On Fri, May 29, 2020 at 6:00 PM András Simonyi <andras.simonyi@gmail.com> wrote:
>
> Hi,
>
> apologies for reacting that late (it seems that I messed up my email filtering
> royally) ...

I wondered what happened to you; glad you sorted it out though!

...

> (i) Default ("built in") citation processor in Org
>
> I think citeproc-el could be a reasonable solution for this. The core is
> well-tested (thanks to the CSL test suite), implements a fairly large subset of
> the CSL 1.01 standard, and can output citations in a number of formats including
> in Org syntax. If I understood correctly, this would require citeproc-el to be
> part of Emacs, and not even an ELPA package would suffice. In principle I'm open
> to this, but I'm not familiar with the process and there are potential pitfalls,
> e.g. citeproc-el currently depends on some "modern" libraries, e.g., "s" or
> "dash" which AFAIK are not in Emacs.

It would very cool if you all could figure out how to make this happen.

It would ensure pretty awesome out-of-box functionality.

Bruce

Re: wip-cite status question and feedback

[Bruce D'Arcus]

One thing on a detail:

On Fri, May 29, 2020 at 6:00 PM András Simonyi <andras.simonyi@gmail.com> wrote:

...

> From the citeproc-el (and CSL) perspective, the list of bibliography database
> files, the place where the bibliography should be placed (if it's specified) ...

Particularly if citeproc.el gets incorporated into emacs, keep in mind
the possibility
(at least at some point) of more than one bibliography in a document.

I'm not super familiar with biblatex, but I do know it supports this
functionality.

Here's one example I found; LaTeX of course:

\printbibliography[title=Primary Sources, keyword=primary]
\printbibliography[title=Secondary Sources, keyword=secondary]

And there's the use case of per/chapter endnote bibliographies and such.

Bruce

Re: wip-cite status question and feedback

[Bastien]

Hi all,

just a quick word in this thread to say that we are in feature freeze
for Org core features (ie. everything in org*.el files, + ob.el/ol.el).

So let's take the time to discuss this in details for 9.5 (or later,
when it's ready.)

Thanks!

-- 
 Bastien

Re: wip-cite status question and feedback

[Bruce D'Arcus]

On Wed, Jun 3, 2020 at 10:40 AM Bastien <bzg@gnu.org> wrote:

> Hi all,
>
> just a quick word in this thread to say that we are in feature freeze
> for Org core features (ie. everything in org*.el files, + ob.el/ol.el).
>
> So let's take the time to discuss this in details for 9.5 (or later,
> when it's ready.)

What about the question of including citeproc.el and citeproc.org in org proper?

That seems the most immediate practical question, as it would impact what code
would be (further) developed, and therefore what would need to be tested, etc.

Andras has expressed openness to that, but also questions.

Bruce

Re: wip-cite status question and feedback

[Bruce D'Arcus]

Could Nicholas or Bastien please update on where this stands at this point?

On Wed, Jun 3, 2020 at 10:53 AM Bruce D'Arcus <bdarcus@gmail.com> wrote:
>
> On Wed, Jun 3, 2020 at 10:40 AM Bastien <bzg@gnu.org> wrote:
>
> > Hi all,
> >
> > just a quick word in this thread to say that we are in feature freeze
> > for Org core features (ie. everything in org*.el files, + ob.el/ol.el).
> >
> > So let's take the time to discuss this in details for 9.5 (or later,
> > when it's ready.)
>
> What about the question of including citeproc.el and citeproc.org in org proper?
>
> That seems the most immediate practical question, as it would impact what code
> would be (further) developed, and therefore what would need to be tested, etc.
>
> Andras has expressed openness to that, but also questions.
>
> Bruce

Re: wip-cite status question and feedback

[Bruce D'Arcus]

[-- Attachment #1: Type: text/plain, Size: 1332 bytes --]

I know Bastien and Nicolas are less active on the list these days, but does
anyone know what the plan is with the wip-cite branch?

My understanding is the syntax work that Nicolas did was done enough that
it was ready for testing, and that the remaining questions were really what
more needed to be done before merging.

So lingering questions included possibility of incorporating citeproc-org
directly, so rich formatting was available out of box, and if not that,
what was minimum functionality needed for release.

At least that's what I recall; it's been awhile.

Anyone know anything else?

On Wed, Jun 3, 2020 at 10:53 AM Bruce D'Arcus <bdarcus@gmail.com> wrote:

> On Wed, Jun 3, 2020 at 10:40 AM Bastien <bzg@gnu.org> wrote:
>
> > Hi all,
> >
> > just a quick word in this thread to say that we are in feature freeze
> > for Org core features (ie. everything in org*.el files, + ob.el/ol.el).
> >
> > So let's take the time to discuss this in details for 9.5 (or later,
> > when it's ready.)
>
> What about the question of including citeproc.el and citeproc.org in org
> proper?
>
> That seems the most immediate practical question, as it would impact what
> code
> would be (further) developed, and therefore what would need to be tested,
> etc.
>
> Andras has expressed openness to that, but also questions.
>
> Bruce
>

[-- Attachment #2: Type: text/html, Size: 1921 bytes --]

Re: wip-cite status question and feedback

[M. ‘quintus’ Gülker]

Am 24. März 2021 um 09:22 Uhr -0400 schrieb Bruce D'Arcus:
> Anyone know anything else?

Not really. I would just like to say that I am eagerly watching this
thread and am earnestly curious about what will happen to wip-cite. A
proper cite system for org would be such a useful feature.

  -quintus

-- 
Dipl.-Jur. M. Gülker | https://mg.guelker.eu |    For security:
Passau, Germany      | kontakt@guelker.eu    | () Avoid HTML e-mail
European Union       | PGP: see homepage     | /\ http://asciiribbon.org

Re: wip-cite status question and feedback

[Bruce D'Arcus]

Maybe since Nicolas has been around lately, he can weigh in?

On Wed, Mar 24, 2021 at 2:28 PM M. ‘quintus’ Gülker
<post+orgmodeml@guelker.eu> wrote:
>
> Am 24. März 2021 um 09:22 Uhr -0400 schrieb Bruce D'Arcus:
> > Anyone know anything else?
>
> Not really. I would just like to say that I am eagerly watching this
> thread and am earnestly curious about what will happen to wip-cite. A
> proper cite system for org would be such a useful feature.
>
>   -quintus
>
> --
> Dipl.-Jur. M. Gülker | https://mg.guelker.eu |    For security:
> Passau, Germany      | kontakt@guelker.eu    | () Avoid HTML e-mail
> European Union       | PGP: see homepage     | /\ http://asciiribbon.org

Re: wip-cite status question and feedback

[Nicolas Goaziou]

Hello,

"Bruce D'Arcus" writes:

> Maybe since Nicolas has been around lately, he can weigh in?

I guess I can make a summary about the current state of the citations
branch, i.e., what is done, what is missing.

There are three major steps to complete in order to add citations in
Org: defining the syntax, designing the Org API for citation processors,
and writing a default processor.

The syntax is complete in "wip-cite-new" branch. For the record, in its
full glory, it can look like this:

  [cite/style: global prefix; prefix -@key suffix ; ... ; global suffix]

"/style", "global prefix", "prefix", "-" marker, "suffix" and "global
suffix" are all optional. So, in its minimal form, it can be as simple
as:

  [cite:@Doe:1995a]

The syntax also includes a new #+bibliography keyword, which, when
paired with a new `org-cite-global-bibliography', defines global or
local bibliographies.

For exporting needs, I also introduced #+print_bibliography,
#+citation_style and #+bibliography_style keywords.

Now about the API, which is partly implemented on a local branch.

Citations processors, in addition to any tools they may provide, can
integrate into Org in three distinct areas: opening (with
`org-open-at-point'), fontification, and export.

  - "opening" action is straightforward. All is needed for the processor
    is to provide a function accepting two arguments: the citation key,
    as a string, and possibly a universal argument, which it may ignore,
    or not.

    All this is already implemented locally.

  - "exporting" action is trickier, because there are multiple ways to
    do the integration, and, since I'm not an implementor for citation
    processors, I don't have an accurate view about what is the best
    design. Anyway, here is the

    First, export happens as pre-process, before export back-ends are
    introduced. IOW, export back-ends are never going to see a citation
    object, which means no support whatsoever is needed on their end.

    Support export requires two functions. The first function is
    responsible for rendering a bibliography. Its arguments are the list
    of citations in the document, the list of bibliography files, the
    style, if any, and the export back-end. It should return a string.

    The second mandatory function is obviously responsible for rendering
    citations. It is called with a citation object, the desired style,
    if any, and the export back-end, the full list of citations objets
    in the document, and the list of bibliography files. It should also
    return a string. Org provides a helper function to determine the
    footnote containing a citation (and its label, or number) from
    a citation object.

    In the functions described above, I don't know if the arguments are
    sufficient. I would love to hear about citeproc-org and org-ref
    developers about this.

    Also, note that style is an indication. Export is requested to
    handle regular [cite:...] syntax. Unknown styles should fall-back to
    this.

  - "fontification" is meant to give full access to face selection, what
    is really displayed, additional keymaps, all using a single
    function.

    At the moment, I have no idea about what arguments would be useful.
    I think John Kitchin gave ideas about this already on this ML.
    I have to re-read his posts on the subject. In any case, feedback
    welcome.

    This not implemented yet.

A citation processor does not need to provide integration in all these
areas. Users may be able to mix and match processors. This is another
(minor) point which is yet to be designed. How is a user supposed to
select a processor for each integration area? It could be done through
three variables, e.g.,

  (setq org-cite-display-processor 'org-ref)
  (setq org-cite-export-processor 'citeproc)
  (setq org-cite-follow-processor 'default)

I think it is unlikely for a user to locally select "display" and
"follow" processors. However, we need a way to use a local export
processor for a given document. I may need to introduce
a #+citation_processor keyword during export. Any other idea?

The last step is implementing a default processor. The point is to
provide a self-contained, very basic processor handling all three areas
described above.

I started implementing one. It relies on built-in bibtex.el library, so
it assumes bibliography is written as a BibTex file. At the moment it
properly "follows" citations. It also exports citations as (Name, date).
However, it doesn't export bibliographies yet. It does not fontify
either.

As a conclusion, besides the syntax, the branch is not ready for
inclusion yet. There are a few design questions about the API to answer.
Once done, and as long as no one has high expectations about the default
processor, this last part should not be too hard to complete.


Regards,
-- 
Nicolas Goaziou

Re: wip-cite status question and feedback

[András Simonyi]

Dear All,

thank you very much for bringing this forward and thanks to Nicholas
for his work and detailed write-up on the API! Unfortunately, I'm
extremely busy right now, but will try to comment in detail on the
coming days, most probably on Thursday. I'm very excited by the new
developments!

best regards,
András

On Mon, 12 Apr 2021 at 15:19, Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:
>
> Hello,
>
> "Bruce D'Arcus" writes:
>
> > Maybe since Nicolas has been around lately, he can weigh in?
>
> I guess I can make a summary about the current state of the citations
> branch, i.e., what is done, what is missing.
>
> There are three major steps to complete in order to add citations in
> Org: defining the syntax, designing the Org API for citation processors,
> and writing a default processor.
>
> The syntax is complete in "wip-cite-new" branch. For the record, in its
> full glory, it can look like this:
>
>   [cite/style: global prefix; prefix -@key suffix ; ... ; global suffix]
>
> "/style", "global prefix", "prefix", "-" marker, "suffix" and "global
> suffix" are all optional. So, in its minimal form, it can be as simple
> as:
>
>   [cite:@Doe:1995a]
>
> The syntax also includes a new #+bibliography keyword, which, when
> paired with a new `org-cite-global-bibliography', defines global or
> local bibliographies.
>
> For exporting needs, I also introduced #+print_bibliography,
> #+citation_style and #+bibliography_style keywords.
>
> Now about the API, which is partly implemented on a local branch.
>
> Citations processors, in addition to any tools they may provide, can
> integrate into Org in three distinct areas: opening (with
> `org-open-at-point'), fontification, and export.
>
>   - "opening" action is straightforward. All is needed for the processor
>     is to provide a function accepting two arguments: the citation key,
>     as a string, and possibly a universal argument, which it may ignore,
>     or not.
>
>     All this is already implemented locally.
>
>   - "exporting" action is trickier, because there are multiple ways to
>     do the integration, and, since I'm not an implementor for citation
>     processors, I don't have an accurate view about what is the best
>     design. Anyway, here is the
>
>     First, export happens as pre-process, before export back-ends are
>     introduced. IOW, export back-ends are never going to see a citation
>     object, which means no support whatsoever is needed on their end.
>
>     Support export requires two functions. The first function is
>     responsible for rendering a bibliography. Its arguments are the list
>     of citations in the document, the list of bibliography files, the
>     style, if any, and the export back-end. It should return a string.
>
>     The second mandatory function is obviously responsible for rendering
>     citations. It is called with a citation object, the desired style,
>     if any, and the export back-end, the full list of citations objets
>     in the document, and the list of bibliography files. It should also
>     return a string. Org provides a helper function to determine the
>     footnote containing a citation (and its label, or number) from
>     a citation object.
>
>     In the functions described above, I don't know if the arguments are
>     sufficient. I would love to hear about citeproc-org and org-ref
>     developers about this.
>
>     Also, note that style is an indication. Export is requested to
>     handle regular [cite:...] syntax. Unknown styles should fall-back to
>     this.
>
>   - "fontification" is meant to give full access to face selection, what
>     is really displayed, additional keymaps, all using a single
>     function.
>
>     At the moment, I have no idea about what arguments would be useful.
>     I think John Kitchin gave ideas about this already on this ML.
>     I have to re-read his posts on the subject. In any case, feedback
>     welcome.
>
>     This not implemented yet.
>
> A citation processor does not need to provide integration in all these
> areas. Users may be able to mix and match processors. This is another
> (minor) point which is yet to be designed. How is a user supposed to
> select a processor for each integration area? It could be done through
> three variables, e.g.,
>
>   (setq org-cite-display-processor 'org-ref)
>   (setq org-cite-export-processor 'citeproc)
>   (setq org-cite-follow-processor 'default)
>
> I think it is unlikely for a user to locally select "display" and
> "follow" processors. However, we need a way to use a local export
> processor for a given document. I may need to introduce
> a #+citation_processor keyword during export. Any other idea?
>
> The last step is implementing a default processor. The point is to
> provide a self-contained, very basic processor handling all three areas
> described above.
>
> I started implementing one. It relies on built-in bibtex.el library, so
> it assumes bibliography is written as a BibTex file. At the moment it
> properly "follows" citations. It also exports citations as (Name, date).
> However, it doesn't export bibliographies yet. It does not fontify
> either.
>
> As a conclusion, besides the syntax, the branch is not ready for
> inclusion yet. There are a few design questions about the API to answer.
> Once done, and as long as no one has high expectations about the default
> processor, this last part should not be too hard to complete.
>
>
> Regards,
> --
> Nicolas Goaziou

Re: wip-cite status question and feedback

[András Simonyi]

Dear All,

apologies for being this late with my reaction --  here  are some
comments/questions on Nicolas's summary:

On Mon, 12 Apr 2021 at 15:19, Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:

> suffix" are all optional. So, in its minimal form, it can be as simple
> as:
>
>   [cite:@Doe:1995a]

are short-form citations like @doe2018 or  [@doe2018] also supported?

> Now about the API, which is partly implemented on a local branch.

>   - "opening" action is straightforward. All is needed for the processor
>     is to provide a function accepting two arguments: the citation key,
>     as a string, and possibly a universal argument, which it may ignore,
>     or not.
>
>     All this is already implemented locally.

I can imagine situations in which the location (e.g. page range) of
the citation is also important, e.g., when the
processor opens the cited document, so it'd be nice to pass this
information (when available) as well.

>     First, export happens as pre-process, before export back-ends are
>     introduced. IOW, export back-ends are never going to see a citation
>     object, which means no support whatsoever is needed on their end.

This is pretty close to how citeproc-org operates right now.

>     Support export requires two functions. The first function is
>     responsible for rendering a bibliography. Its arguments are the list
>     of citations in the document, the list of bibliography files, the
>     style, if any, and the export back-end. It should return a string.

Am I correct in assuming that the returned bibliography should be
rendered as a legitimate Org fragment regardless of the back-end?

>     The second mandatory function is obviously responsible for rendering
>     citations. It is called with a citation object, the desired style,
>     if any, and the export back-end, the full list of citations objets
>     in the document, and the list of bibliography files. It should also
>     return a string. Org provides a helper function to determine the
>     footnote containing a citation (and its label, or number) from
>     a citation object.

This might not be an important point, but citeproc-el is rather
citation list oriented in the sense that it processes the list of all
citations and returns a list of their formatted counterparts, so it
might be useful to provide this "bulk" API as well, at least as an
optional addition.

What is very important and was rather tricky to implement as I recall
is that the list of citations should be in the order they actually
appear in the exported document even for citations in footnotes,
because certain styles format citations differently if another
reference to the same work occurred in a nearby, earlier note. In
addition to the precise order of the citations and whether they are in
a footnote or not, the concrete identity  of the notes is also
important, because formatting can differ depending on whether two
citations are in the same note or in different ones.

>   - "fontification" is meant to give full access to face selection, what
>     is really displayed, additional keymaps, all using a single
>     function.

>     At the moment, I have no idea about what arguments would be useful.
>     I think John Kitchin gave ideas about this already on this ML.
>     I have to re-read his posts on the subject. In any case, feedback
>     welcome.

I'm thinking about implementing a "fontification" solution which would
use citeproc-el with a standard style to produce nice preview-like
representations of the citations in the buffer. This would require
basically the same pieces of information as citation export I think,
although it might be made strictly  local, working only with the
single citation object plus the bibliography information. Providing
really precise previews is probably out of scope here,because some
details depend on global information about other citations which can
frequently change and would be difficult to keep up-to-date.

> A citation processor does not need to provide integration in all these
> areas. Users may be able to mix and match processors. This is another
> (minor) point which is yet to be designed. How is a user supposed to
> select a processor for each integration area? It could be done through
> three variables, e.g.,
>
>   (setq org-cite-display-processor 'org-ref)
>   (setq org-cite-export-processor 'citeproc)
>   (setq org-cite-follow-processor 'default)
>
> I think it is unlikely for a user to locally select "display" and
> "follow" processors. However, we need a way to use a local export
> processor for a given document. I may need to introduce
> a #+citation_processor keyword during export. Any other idea?

All of these solutions seem to be good starting points.  As for
setting the "display" and "follow" processors locally, this leads to a
question which probably has to be addressed at a certain point: that
of bibliography formats. The Emacs world is currently rather BibTeX
centered, but biblatex is an important (and rather different)
alternative, and there is CSL as well which I expect to become more
and more relevant (it's citeproc-el's native format). Moreover, these
formats have some variants, e.g., for BibTeX there is also org-bibtex,
for CSL there is CSL-JSON and a CSL-YAML etc. If different "display"
and "follow" processors will be able to handle different formats then
the users might want to change those settings locally as well, based
on the bibliography format, but I'm not sure what kind of
infrastructure would be the best way of supporting this. (E.g.,
registering format support information about the processors and
choosing on this basis?)

> The last step is implementing a default processor. The point is to
> provide a self-contained, very basic processor handling all three areas
> described above.
>
> I started implementing one. It relies on built-in bibtex.el library, so
> it assumes bibliography is written as a BibTex file. At the moment it
> properly "follows" citations. It also exports citations as (Name, date).
> However, it doesn't export bibliographies yet. It does not fontify
> either.
>
> As a conclusion, besides the syntax, the branch is not ready for
> inclusion yet. There are a few design questions about the API to answer.
> Once done, and as long as no one has high expectations about the default
> processor, this last part should not be too hard to complete.
>
>
> Regards,
> --
> Nicolas Goaziou

Thanks again for working on this and best regards,
András

Re: wip-cite status question and feedback

[Bruce D'Arcus]

On Fri, Apr 16, 2021 at 1:06 PM András Simonyi <andras.simonyi@gmail.com> wrote:

> The Emacs world is currently rather BibTeX
> centered, but biblatex is an important (and rather different)
> alternative, and there is CSL as well which I expect to become more
> and more relevant (it's citeproc-el's native format). Moreover, these
> formats have some variants, e.g., for BibTeX there is also org-bibtex,
> for CSL there is CSL-JSON and a CSL-YAML etc.

I might just jump in here and mention two recent, more general, CSL
processor projects, both designed with similar goals of being usable
as libraries, and so in theory should be able to hook up to this
system at some point with some elisp glue.

1. the haskell based https://github.com/jgm/citeproc, now used in pandoc
2. the rust based https://github.com/zotero/citeproc-rs (I expect this
to be incorporated into Zotero in the future)

Here's the most clear natural language description of the API in the
first, which should be similar in the second.

https://github.com/jgm/citeproc#how-to-use-it

Bruce

Re: wip-cite status question and feedback

[M. ‘quintus’ Gülker]

Hi,

Am 12. April 2021 um 15:19 Uhr +0200 schrieb Nicolas Goaziou:
> The syntax is complete in "wip-cite-new" branch. For the record, in its
> full glory, it can look like this:
> 
>   [cite/style: global prefix; prefix -@key suffix ; ... ; global suffix]
> 
> [...]
>   - "exporting" action is trickier, because there are multiple ways to
>     do the integration, and, since I'm not an implementor for citation
>     processors, I don't have an accurate view about what is the best
>     design. [...]
>     The second mandatory function is obviously responsible for rendering
>     citations. It is called with a citation object, the desired style,
>     if any, and the export back-end, the full list of citations objets
>     in the document, and the list of bibliography files. It should also
>     return a string. Org provides a helper function to determine the
>     footnote containing a citation (and its label, or number) from
>     a citation object.
> 
>     In the functions described above, I don't know if the arguments are
>     sufficient.

The citation object will provide access to all elements of the new
cite syntax I assume, including things like key, prefix and suffix?

Several styles I am normally confronted with require crossreferencing
in citation footnotes (example: “Doe (see above Fn. 24), pp. 35-37”).
Formatting this requires access to the place where an @key first
occured in a footnote. The full list of citation objects probably
suffices for that information; on a first thought I would either use
the first citation object from that list with the @key at hand unequal
to the active citation object (if the list is in the order in which
the citations appear in the exported document, which might not match
the order in the org source) or use the citation object whose footnote
label has the lowest number and is unequal to the active citation
object (if the list is not guaranteed to be in said order). I would
prefer the former approach, because sometimes I deal with footnotes
with numbers like “4a” (a footnote inserted at a late stage in the
authoring process between footnotes 4 and 5), which defeats the
lowest-number approach.

For non-footnote-based citations, the “helper function to determine
the footnote containing a citation” should probably return nil.

  -quintus

-- 
Dipl.-Jur. M. Gülker | https://mg.guelker.eu |    For security:
Passau, Germany      | kontakt@guelker.eu    | () Avoid HTML e-mail
European Union       | PGP: see homepage     | /\ http://asciiribbon.org

Re: wip-cite status question and feedback

[Nicolas Goaziou]

Hello,

András Simonyi <andras.simonyi@gmail.com> writes:

> are short-form citations like @doe2018 or  [@doe2018] also supported?

No, I removed them in last year's iteration, because they are prone to
false positives. I didn't want to repeat the footnote syntax mistake,
when "[1]" was equivalent to "[fn:1]".

I am also leaning towards removing the possibility to include Org syntax
(e.g., bold) in prefixes/suffixes. Indeed, this doesn't sound terribly
useful (you can move the bold part outside of the citation object), and
yet, adds a layer of complexity on top of the citation object.

Can we assume (global)prefix/suffix are just plain text?

>> Now about the API, which is partly implemented on a local branch.
>
>>   - "opening" action is straightforward. All is needed for the processor
>>     is to provide a function accepting two arguments: the citation key,
>>     as a string, and possibly a universal argument, which it may ignore,
>>     or not.
>>
>>     All this is already implemented locally.
>
> I can imagine situations in which the location (e.g. page range) of
> the citation is also important, e.g., when the
> processor opens the cited document, so it'd be nice to pass this
> information (when available) as well.

In my mind, "opening" leads to the bibliography reference, not to the
original document. IIUC, in this situation, the location does not matter
much, does it?

In any case, Org has no clue about the "location" of the citation. It
can provide the suffix associated to the citation key, but it is up to
the citation processor to extract page information out of it. If this is
desirable, we need to provide the full citation object instead of the
key. I don't know if it is worth the trouble.

>>     First, export happens as pre-process, before export back-ends are
>>     introduced. IOW, export back-ends are never going to see a citation
>>     object, which means no support whatsoever is needed on their end.
>
> This is pretty close to how citeproc-org operates right now.

I had a quick glance at citeproc-org for inspiration. I agree this is
a sane way to proceed.

>>     Support export requires two functions. The first function is
>>     responsible for rendering a bibliography. Its arguments are the list
>>     of citations in the document, the list of bibliography files, the
>>     style, if any, and the export back-end. It should return a string.
>
> Am I correct in assuming that the returned bibliography should be
> rendered as a legitimate Org fragment regardless of the back-end?

I think it should be rendered as target format, to be on par with
citation handling.

You may, however, use Org format as an intermediate step, and then, call
`org-export-data-with-backend' (from "ox.el") on it. If this is
desirable, I need to also provide a fifth `info' argument to the list
above.

>>     The second mandatory function is obviously responsible for rendering
>>     citations. It is called with a citation object, the desired style,
>>     if any, and the export back-end, the full list of citations objets
>>     in the document, and the list of bibliography files. It should also
>>     return a string. Org provides a helper function to determine the
>>     footnote containing a citation (and its label, or number) from
>>     a citation object.
>
> This might not be an important point, but citeproc-el is rather
> citation list oriented in the sense that it processes the list of all
> citations and returns a list of their formatted counterparts, so it
> might be useful to provide this "bulk" API as well, at least as an
> optional addition.

I don't think any addition to the suggested API is required. If you
cannot split citation processing, you may fill a temporary cache holding
all exports on first citation (the full list of citations objects is
provided right from the start), then read from that cache on subsequent
citations.

WDYT?

> What is very important and was rather tricky to implement as I recall
> is that the list of citations should be in the order they actually
> appear in the exported document even for citations in footnotes,
> because certain styles format citations differently if another
> reference to the same work occurred in a nearby, earlier note.

Noted.

IIRC, `org-export--footnote-reference-map' deals with a similar issue,
i.e., how to apply a function on footnote references _in order_.

Situation in nested footnotes can be ambiguous, tho. In the following
example, is the expected order @a then @b then @c?

--8<---------------cut here---------------start------------->8---
Text[fn:1] [@cite:@c]

[fn:1] This[fn:2] is [cite:@b].

[fn:2] [cite:@a]
--8<---------------cut here---------------end--------------->8---

> In addition to the precise order of the citations and whether they are
> in a footnote or not, the concrete identity of the notes is also
> important, because formatting can differ depending on whether two
> citations are in the same note or in different ones.

I don't understand what you call the "identity of the notes". Could you
provide an example?

>>   - "fontification" is meant to give full access to face selection, what
>>     is really displayed, additional keymaps, all using a single
>>     function.
>
>>     At the moment, I have no idea about what arguments would be useful.
>>     I think John Kitchin gave ideas about this already on this ML.
>>     I have to re-read his posts on the subject. In any case, feedback
>>     welcome.
>
> I'm thinking about implementing a "fontification" solution which would
> use citeproc-el with a standard style to produce nice preview-like
> representations of the citations in the buffer. This would require
> basically the same pieces of information as citation export I think,
> although it might be made strictly  local, working only with the
> single citation object plus the bibliography information.

OK. Citation object and list of bibliography files as arguments are
a starting point.

>> A citation processor does not need to provide integration in all these
>> areas. Users may be able to mix and match processors. This is another
>> (minor) point which is yet to be designed. How is a user supposed to
>> select a processor for each integration area? It could be done through
>> three variables, e.g.,
>>
>>   (setq org-cite-display-processor 'org-ref)
>>   (setq org-cite-export-processor 'citeproc)
>>   (setq org-cite-follow-processor 'default)
>>
>> I think it is unlikely for a user to locally select "display" and
>> "follow" processors. However, we need a way to use a local export
>> processor for a given document. I may need to introduce
>> a #+citation_processor keyword during export. Any other idea?
>
> All of these solutions seem to be good starting points.  As for
> setting the "display" and "follow" processors locally, this leads to a
> question which probably has to be addressed at a certain point: that
> of bibliography formats. The Emacs world is currently rather BibTeX
> centered, but biblatex is an important (and rather different)
> alternative, and there is CSL as well which I expect to become more
> and more relevant (it's citeproc-el's native format). Moreover, these
> formats have some variants, e.g., for BibTeX there is also org-bibtex,
> for CSL there is CSL-JSON and a CSL-YAML etc. If different "display"
> and "follow" processors will be able to handle different formats then
> the users might want to change those settings locally as well, based
> on the bibliography format, but I'm not sure what kind of
> infrastructure would be the best way of supporting this. (E.g.,
> registering format support information about the processors and
> choosing on this basis?)

I don't have an idea about it either. This is not a blocker, tho. We can
revisit it later, once we have "something" working.

Thanks for your feedback.

Regards,
-- 
Nicolas Goaziou

Re: wip-cite status question and feedback

[Ihor Radchenko]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> In my mind, "opening" leads to the bibliography reference, not to the
> original document. IIUC, in this situation, the location does not matter
> much, does it?
>
> In any case, Org has no clue about the "location" of the citation. It
> can provide the suffix associated to the citation key, but it is up to
> the citation processor to extract page information out of it. If this is
> desirable, we need to provide the full citation object instead of the
> key. I don't know if it is worth the trouble.

From my experience using org-ref for citations, going to the
bibliography reference is rarely useful. Most of time, I want to jump to
the document I am citing (or even page in the document).

I think that full citation object should be worth passing to the
citation backend.

Best,
Ihor

Re: wip-cite status question and feedback

[Bruce D'Arcus]

On Sun, Apr 18, 2021 at 9:31 AM Ihor Radchenko <yantar92@gmail.com> wrote:
>
> Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:
>
> > In my mind, "opening" leads to the bibliography reference, not to the
> > original document. IIUC, in this situation, the location does not matter
> > much, does it?
> >
> > In any case, Org has no clue about the "location" of the citation. It
> > can provide the suffix associated to the citation key, but it is up to
> > the citation processor to extract page information out of it. If this is
> > desirable, we need to provide the full citation object instead of the
> > key. I don't know if it is worth the trouble.
>
> From my experience using org-ref for citations, going to the
> bibliography reference is rarely useful. Most of time, I want to jump to
> the document I am citing (or even page in the document).
>
> I think that full citation object should be worth passing to the
> citation backend.

I agree.

FWIW, this is how the pandoc API describes this point, which I believe
is consistent with Nicolas' meaning of "citation objects"?

"a list of Citations (each of which may have multiple CitationItems)."

Bruce

Re: wip-cite status question and feedback

[Nicolas Goaziou]

Hello,

M. ‘quintus’ Gülker <post+orgmodeml@guelker.eu> writes:

> The citation object will provide access to all elements of the new
> cite syntax I assume, including things like key, prefix and suffix?

Indeed. Also global prefix and suffix.

> Several styles I am normally confronted with require crossreferencing
> in citation footnotes (example: “Doe (see above Fn. 24), pp. 35-37”).
> Formatting this requires access to the place where an @key first
> occured in a footnote. The full list of citation objects probably
> suffices for that information; on a first thought I would either use
> the first citation object from that list with the @key at hand unequal
> to the active citation object 

This would work. If it is a common need, Org could also provide such
a helper function.

> or use the citation object whose footnote label has the lowest number
> and is unequal to the active citation object (if the list is not
> guaranteed to be in said order). I would prefer the former approach,
> because sometimes I deal with footnotes with numbers like “4a” (a
> footnote inserted at a late stage in the authoring process between
> footnotes 4 and 5), which defeats the lowest-number approach.

Note that export process provides its own footnote numbering, which does
not rely on the label used. See `org-export-get-footnote-number'.

So you can also use the second method.

> For non-footnote-based citations, the “helper function to determine
> the footnote containing a citation” should probably return nil.

Indeed. If there is no footnote definition containing the citation, it
returns nil.

Regards,
-- 
Nicolas Goaziou

Re: wip-cite status question and feedback

[Matt Price]

[-- Attachment #1: Type: text/plain, Size: 235 bytes --]

As a user, is there any way I can participate at this point? I'm not in a
position to contribute code tight now but really do want to have this
feature as soon as possible. It's going to improve my students' lives quite
a bit.


>
>
>

[-- Attachment #2: Type: text/html, Size: 445 bytes --]

Re: wip-cite status question and feedback

[Nicolas Goaziou]

Hello,

Matt Price <moptop99@gmail.com> writes:

> As a user, is there any way I can participate at this point? I'm not in a
> position to contribute code tight now but really do want to have this
> feature as soon as possible. It's going to improve my students' lives quite
> a bit.

Thank you for your interest.

I have completed step 2, barring some documentation to write. At this
point, there is a defined citation syntax (step 1), and a complete API
to operate on citations (step 2). I will rebase "wip-cite-new" branch
and post a wrap-up soon.

Once done, the next step is to write a limited default back-end, and,
more importantly, to convert existing citation systems to use the API,
and extend it if required.

I will take care about the default back-end (hopefully before the end of
the week but don't hold your breath), but can only provide guidance for
the conversion part. I hope there is enough motivated people to give it
a try.

So, while participation is mostly code-related at this point, you can
certainly comment about the usability of the API from a user POV. Of
course, at this point, we need something that works well enough, not
necessarily something feature complete.

Also, if all goes well, there will be a need for a complete
documentation in the manual… Hear, hear.

Regards,
-- 
Nicolas Goaziou

Re: wip-cite status question and feedback

[Bruce D'Arcus]

On Wed, Apr 21, 2021 at 1:08 PM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:

> I will take care about the default back-end (hopefully before the end of
> the week but don't hold your breath), but can only provide guidance for
> the conversion part. I hope there is enough motivated people to give it
> a try.

My lisp skills are rather weak, but I recently started a project
called bibtex-actions, which is similar to ivy/helm-bibtex but for
completing-read.

If someone wants to help implement support for inserting citations
using the new cite syntax, I've created an issue with some ideas here:

https://github.com/bdarcus/bibtex-actions/issues/108

Bruce

Re: wip-cite status question and feedback

[John Kitchin]

I guess that the actions I use most often when "opening" a citation are,
opening the pdf, going to the webpage for it, and then opening the
bibtex entry (usually to fix capitalization or something). In org-ref
though, there are a whole bunch of other potential actions, like
searching for related citations, copying the key or formatted citation
to the clipboard, etc. I guess my point is there are a lot of things
that opening might mean to different people.

in org-ref, with any of those, the first step is finding where the key
sits in your bib-files, and then getting that entry. It is in a way a
primitive citation processor before the one that is used at export.


Ihor Radchenko <yantar92@gmail.com> writes:

> Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:
>
>> In my mind, "opening" leads to the bibliography reference, not to the
>> original document. IIUC, in this situation, the location does not matter
>> much, does it?
>>
>> In any case, Org has no clue about the "location" of the citation. It
>> can provide the suffix associated to the citation key, but it is up to
>> the citation processor to extract page information out of it. If this is
>> desirable, we need to provide the full citation object instead of the
>> key. I don't know if it is worth the trouble.
>
> From my experience using org-ref for citations, going to the
> bibliography reference is rarely useful. Most of time, I want to jump to
> the document I am citing (or even page in the document).
>
> I think that full citation object should be worth passing to the
> citation backend.
>
> Best,
> Ihor


--
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu
Pronouns: he/him/his

Re: wip-cite status question and feedback

[Bruce D'Arcus]

On Wed, Apr 21, 2021 at 3:57 PM John Kitchin <jkitchin@andrew.cmu.edu> wrote:

> I guess that the actions I use most often when "opening" a citation are,
> opening the pdf, going to the webpage for it, and then opening the
> bibtex entry (usually to fix capitalization or something). In org-ref
> though, there are a whole bunch of other potential actions, like
> searching for related citations, copying the key or formatted citation
> to the clipboard, etc. I guess my point is there are a lot of things
> that opening might mean to different people.

Good point, which I missed.

In bibtex-actions, which uses bibtex-completion as a backend, I have
the following "open" commands:

- open-pdf
- open-link (doi or url)
- open (pdf, or link if not present)
- open-entry (bibtex, to edit)
- open-notes (to review, edit)

All of those backend functions take KEYS as input.

Bruce

Re: wip-cite status question and feedback

[John Kitchin]

>>>   - "fontification" is meant to give full access to face selection, what
>>>     is really displayed, additional keymaps, all using a single
>>>     function.
>>
>>>     At the moment, I have no idea about what arguments would be useful.
>>>     I think John Kitchin gave ideas about this already on this ML.
>>>     I have to re-read his posts on the subject. In any case, feedback
>>>     welcome.

For fontification in an org-file, what I currently find helpful in org-ref:

1. the citations have a face that makes them stand out (it is usually a
green color, unless the key is not found in the bibfiles, and then it is
red). I have a way to turn that off though, because it can cause
performance issues in large documents with large bibliographies.

2. There is a tooltip that shows a lightly formatted reference. The
tooltip is fairly crude, and not related to any particular bibliography
style. It would be great if it did, but so far the effort to achieve
that has been too high. With an integrated citation processor it might
be better, but I am skeptical they will ever look quite like they would
in a latex export, especially if you use latex code in your bibtex
files. This tooltip only needs to be good enough in my opinion to tell
what the key is representing at that point.

3. I like having keymaps on the citations. The most common use for me is to
press S-up to sort a sequence of citations by year, followed by S-left
and S-right to transpose citations in a sequence, followed by
M-left/right to navigate across citations.

4. I tend to have my follow function launch a hydra menu, which provides
many action choices. I think this is easier than trying to remember a lot of
different commands that also work at the citation at point. I started
this with helm, then ivy and now hydra. I can't tell if a new generation
of approaches like selectrum, or the package bibtex-actions will
eventually replace these. This isn't quite fontification, but since you
can put keymaps on with fontification, it isn't fully indpendent either.

5. I mostly think the citations should be displayed as plain text, i.e.
not replaced by a numbered overlay, or equivalent. I could see hiding
the [], but also guess that would be more confusing than beneficial.

>>
>> I'm thinking about implementing a "fontification" solution which would
>> use citeproc-el with a standard style to produce nice preview-like
>> representations of the citations in the buffer. This would require
>> basically the same pieces of information as citation export I think,
>> although it might be made strictly  local, working only with the
>> single citation object plus the bibliography information.
>
> OK. Citation object and list of bibliography files as arguments are
> a starting point.
>
>>> A citation processor does not need to provide integration in all these
>>> areas. Users may be able to mix and match processors. This is another
>>> (minor) point which is yet to be designed. How is a user supposed to
>>> select a processor for each integration area? It could be done through
>>> three variables, e.g.,
>>>
>>>   (setq org-cite-display-processor 'org-ref)
>>>   (setq org-cite-export-processor 'citeproc)
>>>   (setq org-cite-follow-processor 'default)
>>>
>>> I think it is unlikely for a user to locally select "display" and
>>> "follow" processors. However, we need a way to use a local export
>>> processor for a given document. I may need to introduce
>>> a #+citation_processor keyword during export. Any other idea?
>>
>> All of these solutions seem to be good starting points.  As for
>> setting the "display" and "follow" processors locally, this leads to a
>> question which probably has to be addressed at a certain point: that
>> of bibliography formats. The Emacs world is currently rather BibTeX
>> centered, but biblatex is an important (and rather different)
>> alternative, and there is CSL as well which I expect to become more
>> and more relevant (it's citeproc-el's native format). Moreover, these
>> formats have some variants, e.g., for BibTeX there is also org-bibtex,
>> for CSL there is CSL-JSON and a CSL-YAML etc. If different "display"
>> and "follow" processors will be able to handle different formats then
>> the users might want to change those settings locally as well, based
>> on the bibliography format, but I'm not sure what kind of
>> infrastructure would be the best way of supporting this. (E.g.,
>> registering format support information about the processors and
>> choosing on this basis?)
>
> I don't have an idea about it either. This is not a blocker, tho. We can
> revisit it later, once we have "something" working.

This suggests that there is an API of required functions that have a
defined interface for these. That is sort of what org-ref does to
support helm, and a few ivy backends for completion. It is mostly done
by defining some variables that point to functions in those backend
libraries, and the variables are funcalled in a core library.

for example in the follow processor there would be a function that takes
a key, and some bibliography source information, and it would be
responsible for doing what you mean by follow.

export is trickier, but same idea.


>
> Thanks for your feedback.
>
> Regards,


--
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu
Pronouns: he/him/his

Re: wip-cite status question and feedback

[Bruce D'Arcus]

A big +1 to everything John said. On this:

On Wed, Apr 21, 2021 at 4:26 PM John Kitchin <jkitchin@andrew.cmu.edu> wrote:

> 4. I tend to have my follow function launch a hydra menu, which provides
> many action choices. I think this is easier than trying to remember a lot of
> different commands that also work at the citation at point. I started
> this with helm, then ivy and now hydra. I can't tell if a new generation
> of approaches like selectrum, or the package bibtex-actions will
> eventually replace these.

The approach I take to menus of contextual commands in bibtex-actions
is to rely on the Embark package, which will associate categorized
completion candidates with a keymap of commands, and so make the
commands available in context, display the menus depending on how you
configure it (default embark menu vs which-key).

https://github.com/oantolin/embark

So I have no code specific to this, except that a) I give the
candidates a "bibtex" metadata category in completing-read, and b) I
provide a keymap. The user has to specify the association between the
two, or to their own custom keymap.

Conceptually, I'd guess we want something similar; for it to be easy
to use hydra or embark or whatever to associate commands/functions
with citation items (keys).

Bruce

Re: wip-cite status question and feedback

[John Kitchin]

and for completeness from the org-ref point of view, probably all of
them call something like (org-ref-find-bibliography) inside those
functions to get a list of bib sources from a hierarchy of local
definitions in the buffer to env vars, to a default source variable
defined in elisp. I think something similar is done in the
bibtex-completion commands.


Bruce D'Arcus <bdarcus@gmail.com> writes:

> On Wed, Apr 21, 2021 at 3:57 PM John Kitchin <jkitchin@andrew.cmu.edu> wrote:
>
>> I guess that the actions I use most often when "opening" a citation are,
>> opening the pdf, going to the webpage for it, and then opening the
>> bibtex entry (usually to fix capitalization or something). In org-ref
>> though, there are a whole bunch of other potential actions, like
>> searching for related citations, copying the key or formatted citation
>> to the clipboard, etc. I guess my point is there are a lot of things
>> that opening might mean to different people.
>
> Good point, which I missed.
>
> In bibtex-actions, which uses bibtex-completion as a backend, I have
> the following "open" commands:
>
> - open-pdf
> - open-link (doi or url)
> - open (pdf, or link if not present)
> - open-entry (bibtex, to edit)
> - open-notes (to review, edit)
>
> All of those backend functions take KEYS as input.
>
> Bruce


--
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu
Pronouns: he/him/his

Re: wip-cite status question and feedback

[András Simonyi]

Dear All,

On Sun, 18 Apr 2021 at 15:11, Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:

> I am also leaning towards removing the possibility to include Org syntax
> (e.g., bold) in prefixes/suffixes. Indeed, this doesn't sound terribly
> useful (you can move the bold part outside of the citation object), and
> yet, adds a layer of complexity on top of the citation object.
>
> Can we assume (global)prefix/suffix are just plain text?
>

well, I think there might be some legitimate use cases, e.g.,
(see Smith 1997, p. 2, see also p. 33, pp. 45-47, but /cf./ p. 103)
here one might want cf. to be set in italics, and I can imagine
situations in which emphasizing a phrase in a prefix/suffix can be
useful,
but IMHO we can certainly start with assuming just plain text.

> In any case, Org has no clue about the "location" of the citation. It
> can provide the suffix associated to the citation key, but it is up to
> the citation processor to extract page information out of it. If this is
> desirable, we need to provide the full citation object instead of the
> key. I don't know if it is worth the trouble.

It would definitely be useful for making possible more sophisticated
opening actions.

> > Am I correct in assuming that the returned bibliography should be
> > rendered as a legitimate Org fragment regardless of the back-end?
>
> I think it should be rendered as target format, to be on par with
> citation handling.
>
> You may, however, use Org format as an intermediate step, and then, call
> `org-export-data-with-backend' (from "ox.el") on it. If this is
> desirable, I need to also provide a fifth `info' argument to the list
> above.

This is a crucial point: when a note CSL style is used, the export has
to generate footnotes "around"  those citations  which are not already
in a footnote.
Since citeproc-org generates only Org fragments, this is very simple
to do (with anonymous footnotes), but for other formats it can be
rather complicated. Would calling `org-export-data-with-backend'
handle this? If yes then adding an argument  to request this call
would be very useful.

> >
> > This might not be an important point, but citeproc-el is rather
> > citation list oriented in the sense that it processes the list of all
> > citations and returns a list of their formatted counterparts, so it
> > might be useful to provide this "bulk" API as well, at least as an
> > optional addition.
>
> I don't think any addition to the suggested API is required. If you
> cannot split citation processing, you may fill a temporary cache holding
> all exports on first citation (the full list of citations objects is
> provided right from the start), then read from that cache on subsequent
> citations.
>
> WDYT?

Basically I agree,  it would probably be premature optimization to
provide anything else at this point.

> I don't understand what you call the "identity of the notes". Could you
> provide an example?

I meant only that knowing whether a citation is in a footnote or not
is not enough -- if a numerical note index is available which
specifies
the precise position of the note among the footnotes then this is
exactly the information what is needed.

> Situation in nested footnotes can be ambiguous, tho. In the following
> example, is the expected order @a then @b then @c?

--8<---------------cut here---------------start------------->8---
Text[fn:1] [@cite:@c]

[fn:1] This[fn:2] is [cite:@b].

[fn:2] [cite:@a]
--8<---------------cut here---------------end--------------->8---

Yes, I think the order here is @a  @b and @c.

>> there is CSL-JSON and a CSL-YAML etc. If different "display"
>> and "follow" processors will be able to handle different formats then
>> the users might want to change those settings locally as well, based
>> on the bibliography format, but I'm not sure what kind of
>> infrastructure would be the best way of supporting this. (E.g.,
>> registering format support information about the processors and
>> choosing on this basis?)

>I don't have an idea about it either. This is not a blocker, tho. We can
>revisit it later, once we have "something" working.

Absolutely, in the long run probably a way of explicitly specifying
the bibliography format/type will be needed, but just a (file)name  is
a good starting point.

thanks & best regards,
András


> > infrastructure would be the best way of supporting this. (E.g.,
> > registering format support information about the processors and
> > choosing on this basis?)
>
> I don't have an idea about it either. This is not a blocker, tho. We can
> revisit it later, once we have "something" working.
>
> Thanks for your feedback.
>
> Regards,
> --
> Nicolas Goaziou

Re: wip-cite status question and feedback

[Nicolas Goaziou]

Hello,

András Simonyi <andras.simonyi@gmail.com> writes:

> This is a crucial point: when a note CSL style is used, the export has
> to generate footnotes "around"  those citations  which are not already
> in a footnote.
> Since citeproc-org generates only Org fragments, this is very simple
> to do (with anonymous footnotes), but for other formats it can be
> rather complicated. Would calling `org-export-data-with-backend'
> handle this? If yes then adding an argument  to request this call
> would be very useful.

No, I don't think `org-export-data' would handle this. It may require
a more specific tool.

Do you have a small example (org + desired output) so I can experiment
with it?

Regards,
-- 
Nicolas Goaziou

Re: wip-cite status question and feedback

[Bruce D'Arcus]

On Wed, Apr 21, 2021 at 7:51 PM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:
>
> Hello,
>
> András Simonyi <andras.simonyi@gmail.com> writes:
>
> > This is a crucial point: when a note CSL style is used, the export has
> > to generate footnotes "around"  those citations  which are not already
> > in a footnote.
> > Since citeproc-org generates only Org fragments, this is very simple
> > to do (with anonymous footnotes), but for other formats it can be
> > rather complicated. Would calling `org-export-data-with-backend'
> > handle this? If yes then adding an argument  to request this call
> > would be very useful.
>
> No, I don't think `org-export-data' would handle this. It may require
> a more specific tool.
>
> Do you have a small example (org + desired output) so I can experiment
> with it?

In general (conceptually), if you have one footnote in text, and one
in a footnote, with a note style, both will be rendered as footnotes.

So in an adapted version from earlier:

----
Text 1 [@cite:@a].
Text 2[fn:1].

[fn:1] This is [cite:@b].
----

Output would put both citations in respective footnotes.

I included this originally in CSL to make it possible for a user to
switch between note-based and other styles without having to modify
their documents.

Bruce

Re: wip-cite status question and feedback

[Nicolas Goaziou]

Hello,

"Bruce D'Arcus" <bdarcus@gmail.com> writes:

> In general (conceptually), if you have one footnote in text, and one
> in a footnote, with a note style, both will be rendered as footnotes.
>
> So in an adapted version from earlier:
>
> ----
> Text 1 [@cite:@a].
> Text 2[fn:1].
>
> [fn:1] This is [cite:@b].
> ----
>
> Output would put both citations in respective footnotes.

So it would produce the same output as

---
Text 1 [fn::[@cite:@a]].
Text 2[fn:1].

[fn:1] This is [cite:@b].
---

?

If that's correct, this requires a specific tool in "oc.el", for example 

  (defun org-cite-wrap-footnote (citation)
   "Wrap CITATION object within an anonymous footnote."
   ...)

which can be used upon "exporting" the footnote, after having done
appropriate checks.

Regards,
-- 
Nicolas Goaziou

Re: wip-cite status question and feedback

[Timothy]

John Kitchin <jkitchin@andrew.cmu.edu> writes:

> 5. I mostly think the citations should be displayed as plain text, i.e.
> not replaced by a numbered overlay, or equivalent. I could see hiding
> the [], but also guess that would be more confusing than beneficial.

As someone who works author-year inline citations into the text,
APA-style, e.g.
 "as seen in Goaziou et al. (2021) citations in Org shall soon be possible"
Seeing that as
 "as seen in [1] citations in Org shall soon be possible"
is worse than
 "as seen in [cite:@goaziou2021] et al. citations in Org shall soon be possible"

I think what would be ideal, would be if common citation styles could
define a method which produces a display string, like "Goaziou et al.
(2021)". If nothing is defined, then no overlay should be produced.

--
Timothy

Re: wip-cite status question and feedback

[Bruce D'Arcus]

On Wed, Apr 21, 2021 at 10:47 PM Timothy <tecosaur@gmail.com> wrote:

> I think what would be ideal, would be if common citation styles could
> define a method which produces a display string, like "Goaziou et al.
> (2021)". If nothing is defined, then no overlay should be produced.

Make sense, but with a caveat: not all styles work well for preview.

Might be worth looking at how the Zettle markdown editor implements
citations in general, as it's well-done, well-documented, and gone
through a few iterations.

It uses pandoc for the citations syntax and processing.

https://docs.zettlr.com/en/academic/citations/

If point is within the citation brackets, you see the raw syntax, so
you can edit it.

Once you are outside the brackets, having already inserted the
citation, you see a preview of the output.

Examples; the top raw, bottom previewed version of the same.

[@doe19]
(Doe, 2019)

Aside: this is somewhat similar to Nicolas' demo.

That preview is always the same author-date style, regardless of the
output style selected.

So if you choose a footnote-based style for export, you still see the
author-date preview.

This makes sense, because the purpose is only to confirm you have the
right citation.

You also have a "references" sidebar that shows you a previewed list
of all cited references.

Bruce

Re: wip-cite status question and feedback

[Timothy]

Bruce D'Arcus <bdarcus@gmail.com> writes:

> If point is within the citation brackets, you see the raw syntax, so
> you can edit it.
>
> Once you are outside the brackets, having already inserted the
> citation, you see a preview of the output.

This sounds quite sensible to me. I'd be happy with some sensible
author-year format (a customisation variable/fn would be nice though),
or the current style so long as you see the raw entry when the cursor is
inside.

--
Timothy

Re: wip-cite status question and feedback

[Nicolas Goaziou]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> "Bruce D'Arcus" <bdarcus@gmail.com> writes:
>
>> In general (conceptually), if you have one footnote in text, and one
>> in a footnote, with a note style, both will be rendered as footnotes.
>>
>> So in an adapted version from earlier:
>>
>> ----
>> Text 1 [@cite:@a].
>> Text 2[fn:1].
>>
>> [fn:1] This is [cite:@b].
>> ----
>>
>> Output would put both citations in respective footnotes.
>
> So it would produce the same output as
>
> ---
> Text 1 [fn::[@cite:@a]].
> Text 2[fn:1].
>
> [fn:1] This is [cite:@b].
> ---
>
> ?
>
> If that's correct, this requires a specific tool in "oc.el", for example 
>
>   (defun org-cite-wrap-footnote (citation)
>    "Wrap CITATION object within an anonymous footnote."
>    ...)
>
> which can be used upon "exporting" the footnote, after having done
> appropriate checks.

I went ahead and implemented `org-cite-wrap-citation' function in
"oc.el" (from wip-cite-new branch). Here's a quick demo.

--8<---------------cut here---------------start------------->8---
(defun ngz-cite-demo-export-citation (citation &rest _)
  (unless (org-cite-inside-footnote-p citation)
    (org-cite-wrap-citation citation))
  "(Doe, 1999)")

(org-cite-register-processor 'demo
  :export-citation #'ngz-cite-demo-export-citation)
--8<---------------cut here---------------end--------------->8---

When I export the following document to ASCII

--8<---------------cut here---------------start------------->8---
#+cite_export: demo

Regular footnote[fn:1].

Reference outside footnote: [cite:@key]

Footnote with a reference[fn:2].

[fn:1] Definition 1

[fn:2] Reference [cite:@key].
--8<---------------cut here---------------end--------------->8---

I get

--8<---------------cut here---------------start------------->8---
Regular footnote[1].

Reference outside footnote: [2]

Footnote with a reference[3].



Footnotes
─────────

[1] Definition 1

[2] (Doe, 1999)

[3] Reference (Doe, 1999).
--8<---------------cut here---------------end--------------->8---

Regards,
-- 
Nicolas Goaziou

Re: wip-cite status question and feedback

[Nicolas Goaziou]

Hello,

András Simonyi <andras.simonyi@gmail.com> writes:

> well, I think there might be some legitimate use cases, e.g.,
> (see Smith 1997, p. 2, see also p. 33, pp. 45-47, but /cf./ p. 103)

This use-case is not convincing, because it ought to be the task of the
citation processor to italicize "cf." (perhaps through an option), for
consistency across citations.

> here one might want cf. to be set in italics, and I can imagine
> situations in which emphasizing a phrase in a prefix/suffix can be
> useful,

Hmmm.

A drawback with allowing emphasis there is that prefix and suffix become
parsed data and not plain string anymore. As a consequence, searching
through them, e.g., when looking for locator names, requires an
additional level of indirection, since you need to first transform
parsed data back into plain text.

Of course, this is not a deal breaker, just something to know in order
to eventually make an informed choice.


Regards,
-- 
Nicolas Goaziou

Re: wip-cite status question and feedback

[András Simonyi]

Dear All,

On Fri, 23 Apr 2021 at 13:49, Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:

> I went ahead and implemented `org-cite-wrap-citation' function in
> "oc.el" (from wip-cite-new branch). Here's a quick demo.

> --8<---------------cut here---------------start------------->8---
> (defun ngz-cite-demo-export-citation (citation &rest _)
>   (unless (org-cite-inside-footnote-p citation)
>     (org-cite-wrap-citation citation))
>   "(Doe, 1999)")
>
> (org-cite-register-processor 'demo
>   :export-citation #'ngz-cite-demo-export-citation)
> --8<---------------cut here---------------end--------------->8---
>
> When I export the following document to ASCII
>
> --8<---------------cut here---------------start------------->8---
> #+cite_export: demo
>
> Regular footnote[fn:1].
>
> Reference outside footnote: [cite:@key]
>
> Footnote with a reference[fn:2].
>
> [fn:1] Definition 1
>
> [fn:2] Reference [cite:@key].
> --8<---------------cut here---------------end--------------->8---
>
> I get
>
> --8<---------------cut here---------------start------------->8---
> Regular footnote[1].
>
> Reference outside footnote: [2]
>
> Footnote with a reference[3].

> Footnotes
> ─────────
>
> [1] Definition 1
>
> [2] (Doe, 1999)
>
> [3] Reference (Doe, 1999).
> --8<---------------cut here---------------end--------------->8---

Thank you, this is very promising!  I've checked the behaviour of
citeproc-org with and without a note style now and there is only one
additional minor difference which I forgot to mention, I don't know
how difficult would it be to implement it: When a citation is wrapped
into an anonymous footnote because a note style is used then the space
between the citation and the previous token (if there was one) is
removed. So for the fragment (using cite links for now)
--8<---------------cut here---------------start------------->8---
text1 cite:Smullyan-1995-Logic text2[fn::text3 cite:Sierpinski-1965-Card text4]
--8<---------------cut here---------------end--------------->8---
exporting to text with he Chicago author-date CSL style produces the output
--8<---------------cut here---------------start------------->8---
text1 (Smullyan, 1995) text2[1]

Footnotes
─────────

[1] text3 (Sierpiński, 1965) text4
--8<---------------cut here---------------end--------------->8---
while with the Chicago fullnote style
--8<---------------cut here---------------start------------->8---
text1[1] text2[2]

Footnotes
─────────

[1] Raymond M. Smullyan, /First-Order Logic/, 2nd, corrected ed. (Dover,
1995).

[2] text3 Wacław Sierpiński, /Cardinal and Ordinal Numbers/, 2nd,
revised ed. (PWN–Polish Scientific Publishers, 1965). text4
--8<---------------cut here---------------end--------------->8---

I think this is important for making it possible to switch between
note and non-note export styles without changing the Org document, and
would be very interested to hear  your and other list members' views
on this.

thanks & best regards,
András

Re: wip-cite status question and feedback

[Bruce D'Arcus]

On Fri, Apr 23, 2021 at 8:56 AM András Simonyi <andras.simonyi@gmail.com> wrote:

> Thank you, this is very promising!  I've checked the behaviour of
> citeproc-org with and without a note style now and there is only one
> additional minor difference which I forgot to mention, I don't know
> how difficult would it be to implement it: When a citation is wrapped
> into an anonymous footnote because a note style is used then the space
> between the citation and the previous token (if there was one) is
> removed.

...

> I think this is important for making it possible to switch between
> note and non-note export styles without changing the Org document, and
> would be very interested to hear  your and other list members' views
> on this.

I also forgot to mention this detail, and agree it's important to be
able to do: to modify punctuation around such "footnoted-citations".

Bruce

Re: wip-cite status question and feedback

[Bruce D'Arcus]

On Fri, Apr 23, 2021 at 9:10 AM Bruce D'Arcus <bdarcus@gmail.com> wrote:

> I also forgot to mention this detail, and agree it's important to be
> able to do: to modify punctuation around such "footnoted-citations".

While I'm on it, one other related detail:

It can be that not only does the space get removed, but the note mark
is moved outside the period.

So if you have ...

Some sentence with a concluding citation [cite:@key].

... that should end up like this:

Some sentence with a concluding citation.[1]

Aside: looking through the CSL spec, it doesn't seem this is
documented. It obviously should be.

And I don't remember if that convention is locale-specific; e.g. if
while that's the standard in English, it could be different in France.

In any case, this sort of punctuation modification should be possible.

Bruce

Re: wip-cite status question and feedback

[András Simonyi]

Dear All,

On Fri, 23 Apr 2021 at 14:03, Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:

>> well, I think there might be some legitimate use cases, e.g.,
>> (see Smith 1997, p. 2, see also p. 33, pp. 45-47, but /cf./ p. 103)

> This use-case is not convincing, because it ought to be the task of the
> citation processor to italicize "cf." (perhaps through an option), for
> consistency across citations.

This could and maybe should be the case, but I think the currently
available citation processors do not add markup to the suffixes and
prefixes.

> A drawback with allowing emphasis there is that prefix and suffix become
> parsed data and not plain string anymore. As a consequence, searching
> through them, e.g., when looking for locator names, requires an
> additional level of indirection, since you need to first transform
> parsed data back into plain text.

> Of course, this is not a deal breaker, just something to know in order
> to eventually make an informed choice.

I agree that it'd add a considerable amount of complexity, so this
must depend on how important this feature would be. If there are no
objections (I certainly won't object) then simply disallowing markup
in citations, at least in this iteration, seems absolutely acceptable
to me.

thanks & best regards,
András

>
> Regards,
> --
> Nicolas Goaziou

Re: wip-cite status question and feedback

[András Simonyi]

On Fri, 23 Apr 2021 at 15:24, Bruce D'Arcus <bdarcus@gmail.com> wrote:

> It can be that not only does the space get removed, but the note mark
> is moved outside the period.
>
> So if you have ...
>
> Some sentence with a concluding citation [cite:@key].
>
> ... that should end up like this:
>
> Some sentence with a concluding citation.[1]
>
> Aside: looking through the CSL spec, it doesn't seem this is
> documented. It obviously should be.
>
> And I don't remember if that convention is locale-specific; e.g. if
> while that's the standard in English, it could be different in France.

Very good point, and it has also reminded me that the information
available to the citation processor should include in one form or
another the document's locale -- citeproc-org currently uses the
#+LANG option of the Org document (if set) for this purpose.

best regards,
András

Re: wip-cite status question and feedback

[Bruce D'Arcus]

On Fri, Apr 23, 2021 at 9:24 AM Bruce D'Arcus <bdarcus@gmail.com> wrote:

> Aside: looking through the CSL spec, it doesn't seem this is
> documented. It obviously should be.
>
> And I don't remember if that convention is locale-specific; e.g. if
> while that's the standard in English, it could be different in France.
>
> In any case, this sort of punctuation modification should be possible.

I added an issue on this to the CSL docs repo, and got this very
helpful comment from Denis Meier.

https://github.com/citation-style-language/documentation/issues/139#issuecomment-825934813

Upshot: yes, these punctuation conventions do vary by locale.

Bruce

Re: wip-cite status question and feedback

[Nicolas Goaziou]

Hello,

András Simonyi <andras.simonyi@gmail.com> writes:

> Thank you, this is very promising!  I've checked the behaviour of
> citeproc-org with and without a note style now and there is only one
> additional minor difference which I forgot to mention, I don't know
> how difficult would it be to implement it: When a citation is wrapped
> into an anonymous footnote because a note style is used then the space
> between the citation and the previous token (if there was one) is
> removed.

Done.

Note, however, that this introduces some discrepancy with regards to
footnotes. In the following document

--8<---------------cut here---------------start------------->8---
1. Text [fn::[cite:@key]].

2. Text [cite:@key].

3. Text [fn:1].
--8<---------------cut here---------------end--------------->8---

only the space on the second line will be removed (assuming the cite has
note style) because Org does not move space around footnotes.

It does not sound to bad, tho.

Regards,
-- 
Nicolas Goaziou

Re: wip-cite status question and feedback

[M. ‘quintus’ Gülker]

Am 23. April 2021 um 09:24 Uhr -0400 schrieb Bruce D'Arcus:
> It can be that not only does the space get removed, but the note mark
> is moved outside the period.
> 
> So if you have ...
> 
> Some sentence with a concluding citation [cite:@key].
> 
> ... that should end up like this:
> 
> Some sentence with a concluding citation.[1]
> 
> Aside: looking through the CSL spec, it doesn't seem this is
> documented. It obviously should be.
> 
> And I don't remember if that convention is locale-specific; e.g. if
> while that's the standard in English, it could be different in
> France.

As for German, the semantics are different. In Law discipline, in some
journals (not all) both styles can be used within the very same
document and have different meanings. That is,

> This is an example sentence, with a half-sentence following.¹

means that the citation ¹ references the entire sentence, whereas

> This is an example sentence, with a half-sentence following¹.

means that it references only the part following the comma ("with a
half-sentence following"). Normalising this into one uniform style
would be a semantic error. Not all journals handle it like that,
though. Some do prefer uniform look and glance over the semantic
difference.

  -quintus

-- 
Dipl.-Jur. M. Gülker | https://mg.guelker.eu |    For security:
Passau, Germany      | kontakt@guelker.eu    | () Avoid HTML e-mail
European Union       | PGP: see homepage     | /\ http://asciiribbon.org

Re: wip-cite status question and feedback

[Nicolas Goaziou]

Hello,

"Bruce D'Arcus" <bdarcus@gmail.com> writes:

> Some sentence with a concluding citation [cite:@key].
>
> ... that should end up like this:
>
> Some sentence with a concluding citation.[1]
>
> Aside: looking through the CSL spec, it doesn't seem this is
> documented. It obviously should be.
>
> And I don't remember if that convention is locale-specific; e.g. if
> while that's the standard in English, it could be different in France.
>
> In any case, this sort of punctuation modification should be possible.

Could you show more example of this, possibly including quotes the
citation, or better, a precise description of the punctuation
modification you have in mind?

What may be possible to do is to have `org-cite-wrap-footnote' doing
such a modification according to an optional argument. So any caller
(i.e., a processor) could choose to activate it or not, perhaps
according to language.


Regards,
-- 
Nicolas Goaziou

Re: wip-cite status question and feedback

[Bruce D'Arcus]

On Sat, Apr 24, 2021 at 1:47 PM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:
>
> Hello,
>
> "Bruce D'Arcus" <bdarcus@gmail.com> writes:
>
> > Some sentence with a concluding citation [cite:@key].
> >
> > ... that should end up like this:
> >
> > Some sentence with a concluding citation.[1]
> >
> > Aside: looking through the CSL spec, it doesn't seem this is
> > documented. It obviously should be.
> >
> > And I don't remember if that convention is locale-specific; e.g. if
> > while that's the standard in English, it could be different in France.
> >
> > In any case, this sort of punctuation modification should be possible.
>
> Could you show more example of this, possibly including quotes the
> citation, or better, a precise description of the punctuation
> modification you have in mind?

Yes.

Denis lays it out in this comment:

https://github.com/citation-style-language/documentation/issues/139#issuecomment-825934813

What he's arguing is that the rules vary by locale, with German, for
example (he's employed at a Swiss-German institution, I believe),
having different conventions than English, and American English
different than British English.

But an example from American English for illustration, derived from
Denis' examples.

"A simple quote" [cite:@doe].

When rendered, that should be this in an author-date style:

"A simple quote" (Doe 2021).

... and this in a note style:

"A simple quote."[^1]

So that rule would suggest something like:

- if a citation concludes a sentence, move the note mark and whatever
trailing quotation mark, outside the period.

But, Denis continues, "While this is perfectly acceptable in American
English, it is not in German, or even in British English. Here we have
to know whether the final period is part of the original quotation. If
yes, it will be put inside the quotes, otherwise outside." I'll paste
the rest of his examples at the end.

It's possible his rule here is more general, and would still be
acceptable in American English.

Does that help explain?

> What may be possible to do is to have `org-cite-wrap-footnote' doing
> such a modification according to an optional argument. So any caller
> (i.e., a processor) could choose to activate it or not, perhaps
> according to language.

This sounds like what would be needed, maybe with a sensible default.

Bruce

>>> Denis' other examples:

Quotation ending with a period:

"A simple quote."[^1]

Quotation not ending with a period:

Variant a)

"A simple quote".[^1]

Variant b)

"A simple quote"[^1].

Re: wip-cite status question and feedback

[Bruce D'Arcus]

I had an idea on this, though it may not be a good one ...

On Sat, Apr 24, 2021 at 2:39 PM Bruce D'Arcus <bdarcus@gmail.com> wrote:
>
> On Sat, Apr 24, 2021 at 1:47 PM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:
> >
> > Hello,
> >
> > "Bruce D'Arcus" <bdarcus@gmail.com> writes:
> >
> > > Some sentence with a concluding citation [cite:@key].
> > >
> > > ... that should end up like this:
> > >
> > > Some sentence with a concluding citation.[1]
> > >
> > > Aside: looking through the CSL spec, it doesn't seem this is
> > > documented. It obviously should be.
> > >
> > > And I don't remember if that convention is locale-specific; e.g. if
> > > while that's the standard in English, it could be different in France.
> > >
> > > In any case, this sort of punctuation modification should be possible.
> >
> > Could you show more example of this, possibly including quotes the
> > citation, or better, a precise description of the punctuation
> > modification you have in mind?
>
> Yes.
>
> Denis lays it out in this comment:
>
> https://github.com/citation-style-language/documentation/issues/139#issuecomment-825934813
>
> What he's arguing is that the rules vary by locale, with German, for
> example (he's employed at a Swiss-German institution, I believe),
> having different conventions than English, and American English
> different than British English.
>
> But an example from American English for illustration, derived from
> Denis' examples.
>
> "A simple quote" [cite:@doe].
>
> When rendered, that should be this in an author-date style:
>
> "A simple quote" (Doe 2021).
>
> ... and this in a note style:
>
> "A simple quote."[^1]
>
> So that rule would suggest something like:
>
> - if a citation concludes a sentence, move the note mark and whatever
> trailing quotation mark, outside the period.
>
> But, Denis continues, "While this is perfectly acceptable in American
> English, it is not in German, or even in British English. Here we have
> to know whether the final period is part of the original quotation. If
> yes, it will be put inside the quotes, otherwise outside." I'll paste
> the rest of his examples at the end.
>
> It's possible his rule here is more general, and would still be
> acceptable in American English.

The idea is this: make use of a "quote" style and abuse the item
prefix for the quote content?

So using his example:

[cite/quote:;A simple quote. @doe20]

A processor could then know the citation is associated with a quote
that ends a sentence, vs (note missing period):

[cite/quote:;A simple quote @doe20]

... and then more easily adjust accordingly, without needing to know
anything about the surrounding punctuation.

Does that make sense?

Bruce

Re: wip-cite status question and feedback

[Denis Maier]

Am 26.04.2021 um 16:54 schrieb Bruce D'Arcus:
> I had an idea on this, though it may not be a good one ...
> 
> On Sat, Apr 24, 2021 at 2:39 PM Bruce D'Arcus <bdarcus@gmail.com> wrote:
>>
>> On Sat, Apr 24, 2021 at 1:47 PM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:
>>>
>>> Hello,
>>>
>>> "Bruce D'Arcus" <bdarcus@gmail.com> writes:
>>>
>>>> Some sentence with a concluding citation [cite:@key].
>>>>
>>>> ... that should end up like this:
>>>>
>>>> Some sentence with a concluding citation.[1]
>>>>
>>>> Aside: looking through the CSL spec, it doesn't seem this is
>>>> documented. It obviously should be.
>>>>
>>>> And I don't remember if that convention is locale-specific; e.g. if
>>>> while that's the standard in English, it could be different in France.
>>>>
>>>> In any case, this sort of punctuation modification should be possible.
>>>
>>> Could you show more example of this, possibly including quotes the
>>> citation, or better, a precise description of the punctuation
>>> modification you have in mind?
>>
>> Yes.
>>
>> Denis lays it out in this comment:
>>
>> https://github.com/citation-style-language/documentation/issues/139#issuecomment-825934813
>>
>> What he's arguing is that the rules vary by locale, with German, for
>> example (he's employed at a Swiss-German institution, I believe),
>> having different conventions than English, and American English
>> different than British English.
>>
>> But an example from American English for illustration, derived from
>> Denis' examples.
>>
>> "A simple quote" [cite:@doe].
>>
>> When rendered, that should be this in an author-date style:
>>
>> "A simple quote" (Doe 2021).
>>
>> ... and this in a note style:
>>
>> "A simple quote."[^1]
>>
>> So that rule would suggest something like:
>>
>> - if a citation concludes a sentence, move the note mark and whatever
>> trailing quotation mark, outside the period.
>>
>> But, Denis continues, "While this is perfectly acceptable in American
>> English, it is not in German, or even in British English. Here we have
>> to know whether the final period is part of the original quotation. If
>> yes, it will be put inside the quotes, otherwise outside." I'll paste
>> the rest of his examples at the end.
>>
>> It's possible his rule here is more general, and would still be
>> acceptable in American English.
> 
> The idea is this: make use of a "quote" style and abuse the item
> prefix for the quote content?
> 
> So using his example:
> 
> [cite/quote:;A simple quote. @doe20]
> 
> A processor could then know the citation is associated with a quote
> that ends a sentence, vs (note missing period):
> 
> [cite/quote:;A simple quote @doe20]
> 
> ... and then more easily adjust accordingly, without needing to know
> anything about the surrounding punctuation.
> 
> Does that make sense?

Reminds me a bit of how the latex csquotes package works, which is often 
used together with biblatex, it even was developped by the original 
biblatex author before he mysteriously disappeared about a decade ago.

In addition to simple commands like \enquote{something} that simply 
render the argument in quotes this package also has more complex 
commands such as \textcquote or blockcquote. With these commands you can 
combine a quotation with a citation:
\textcquote[〈prenote〉][〈postnote〉]{〈key〉}[〈punct〉]{〈text〉} 
〈tpunct〉

\textcquote[45]{doe}[.]{This is the quotation}
Note the punctuation of the original quote is included in a special 
argument in brackets.
If the period weren't part of the quotation, you'd simply do:
\textcquote[45]{doe}{This is the quotation}.

Regarding the proposal: I think that could go in the right direction, 
but in the current form it has the downside that you can't use the 
prefix anymore, right? What about keeping this separate, like so:

[quote: A simple quote.][cite: @doe p. 45]

But maybe even that is not necessary, and there might be simpler ways.

WDYT?

Denis

Re: wip-cite status question and feedback

[Bruce D'Arcus]

On Mon, Apr 26, 2021 at 4:35 PM Denis Maier <denismaier@mailbox.org> wrote:

> Regarding the proposal: I think that could go in the right direction,
> but in the current form it has the downside that you can't use the
> prefix anymore, right?

Not sure why you would need a prefix for this case, but org-cite has
two levels of affixes. So maybe:

[cite/quote:Prefix ;quote @doe19]

> What about keeping this separate, like so:
>
> [quote: A simple quote.][cite: @doe p. 45]
>
> But maybe even that is not necessary, and there might be simpler ways.

Hmm .. org doesn't have an inline quote, I guess?

Bruce

Re: wip-cite status question and feedback

[Timothy]

Bruce D'Arcus <bdarcus@gmail.com> writes:

> Hmm .. org doesn't have an inline quote, I guess?

I don't think so, unless @@quote:@@ is recognised.

--
Timothy

Re: wip-cite status question and feedback

[Denis Maier]

Am 27.04.2021 um 12:12 schrieb Bruce D'Arcus:
> On Mon, Apr 26, 2021 at 4:35 PM Denis Maier <denismaier@mailbox.org> wrote:
> 
>> Regarding the proposal: I think that could go in the right direction,
>> but in the current form it has the downside that you can't use the
>> prefix anymore, right?
> 
> Not sure why you would need a prefix for this case, but org-cite has
> two levels of affixes. So maybe:
> 
> [cite/quote:Prefix ;quote @doe19]
> 
>> What about keeping this separate, like so:
>>
>> [quote: A simple quote.][cite: @doe p. 45]
>>
>> But maybe even that is not necessary, and there might be simpler ways.
> 
> Hmm .. org doesn't have an inline quote, I guess?

Don't know. That was just a suggestion.

Regarding simpler ways, why not just:

"A simple quote." [cite: @doe p. 45]

That way no information is lost, and you could just move around 
punctuation as appropriate. A distinct syntax might be more powerful, 
but also less readable.

Denis

Re: wip-cite status question and feedback

[Bruce D'Arcus]

On Tue, Apr 27, 2021 at 7:44 AM Denis Maier <denismaier@mailbox.org> wrote:

> Regarding simpler ways, why not just:
>
> "A simple quote." [cite: @doe p. 45]

No, that's worse ;-)

Let's review basic requirements:

1. a plain text document really should be readable and its semantics clear
2. a user should be able to write their documents, and use different
output styles, without modifying that source document

Your example fails on 1 because it suggests the citation attaches to
(starts) the following sentence.

2 is hard given the details we've laid out, but if whatever solution
works for the majority of cases, and at worst requires the user either
a) to be aware of how the processing works and adjust accordingly+,
and/or b) to edit the final document a bit, that seems a reasonable
balance.

Bruce

+ a lot of the corner cases in citations come from a mismatch between
the need for a certain regularity and rigor required for
machine--processing, and users' desire for complete flexibility.

Re: wip-cite status question and feedback

[Denis Maier]

Am 27.04.2021 um 14:32 schrieb Bruce D'Arcus:
> On Tue, Apr 27, 2021 at 7:44 AM Denis Maier <denismaier@mailbox.org> wrote:
> 
>> Regarding simpler ways, why not just:
>>
>> "A simple quote." [cite: @doe p. 45]
> 
> No, that's worse ;-)
> 
> Let's review basic requirements:
> 
> 1. a plain text document really should be readable and its semantics clear
> 2. a user should be able to write their documents, and use different
> output styles, without modifying that source document
> 
> Your example fails on 1 because it suggests the citation attaches to
> (starts) the following sentence.
Only if you mentally parse it as a parenthetical author-date citation. 
If you read it as note citation then your conclusion doesn't follow 
naturally.

> 
> 2 is hard given the details we've laid out, but if whatever solution
> works for the majority of cases, and at worst requires the user either
> a) to be aware of how the processing works and adjust accordingly+,
> and/or b) to edit the final document a bit, that seems a reasonable
> balance.
> 
> Bruce
> 
> + a lot of the corner cases in citations come from a mismatch between
> the need for a certain regularity and rigor required for
> machine--processing, and users' desire for complete flexibility

Yes, that's indeed the question.

I'm after all not totally sure we need a one-size-fits-all solution. By 
way of analogy, biblatex has an \autocite command that scans ahead for 
following punctuation and moves that around.

Bla bla \autocite{doe20}.
can be
Bla bla (Doe 2020).
Or
Bla bla.\footnote{Doe 2020.)

"Bla bla" \autocite{doe20}.
can be
"Bla bla" (Doe 2020).
Or
"Bla bla".\footnote{Doe 2020.)

Then, for more complex cases you'll need special markup provided by the 
csquotes package. Maybe such a modular solution could be useful?

Denis

Re: wip-cite status question and feedback

[Bruce D'Arcus]

On Tue, Apr 27, 2021 at 9:58 AM Denis Maier <denismaier@mailbox.org> wrote:

> > Your example fails on 1 because it suggests the citation attaches to
> > (starts) the following sentence.
>
> Only if you mentally parse it as a parenthetical author-date citation.

I don't think so. A period is a way we delimit sentences; is it not?

...

> I'm after all not totally sure we need a one-size-fits-all solution.

This particular sub-thread started when Andras noted he needed a way
to handle this case, and Nicolas coded up an initial example.

So I think we just need to establish what a reasonable default is.

Bruce

Re: wip-cite status question and feedback

[Denis Maier]

Am 27.04.2021 um 16:07 schrieb Bruce D'Arcus:
> On Tue, Apr 27, 2021 at 9:58 AM Denis Maier <denismaier@mailbox.org> wrote:
>
>>> Your example fails on 1 because it suggests the citation attaches to
>>> (starts) the following sentence.
>> Only if you mentally parse it as a parenthetical author-date citation.
> I don't think so. A period is a way we delimit sentences; is it not?

Sure, but what about this:
This is a sentence with a footnote attached to it.[fn:: This footnote 
belongs to the preceding sentence, doesn't it?]

>
> ...
>
>> I'm after all not totally sure we need a one-size-fits-all solution.
> This particular sub-thread started when Andras noted he needed a way
> to handle this case, and Nicolas coded up an initial example.
>
> So I think we just need to establish what a reasonable default is.

Absolutely, I'm just asking what can be changed afterwards, maybe by 
providing an extended inline quotes syntax or so.

Denis

Re: wip-cite status question and feedback

[Nicolas Goaziou]

Hello,

"Bruce D'Arcus" <bdarcus@gmail.com> writes:

> But an example from American English for illustration, derived from
> Denis' examples.
>
> "A simple quote" [cite:@doe].
>
> When rendered, that should be this in an author-date style:
>
> "A simple quote" (Doe 2021).
>
> ... and this in a note style:
>
> "A simple quote."[^1]
>
> So that rule would suggest something like:
>
> - if a citation concludes a sentence, move the note mark and whatever
> trailing quotation mark, outside the period.
>
> But, Denis continues, "While this is perfectly acceptable in American
> English, it is not in German, or even in British English. Here we have
> to know whether the final period is part of the original quotation. If
> yes, it will be put inside the quotes, otherwise outside." I'll paste
> the rest of his examples at the end.
>
> It's possible his rule here is more general, and would still be
> acceptable in American English.
>
> Does that help explain?

OK. I wrote a POC, and I would appreciate some feedback about it.

In order to test it, one need to evaluate the following, from an
up-to-date "wip-cite-new" branch:

--8<---------------cut here---------------start------------->8---
(defun org-cite--swap-punctuation (object info)
  "Move punctuation following OBJECT before it, if applicable.

When OBJECT is followed by some terminal punctuation character, such as
a period or a question mark, and preceded by some text, move the punctuation
at the end of the previous text.  If that text ends with a double quote,
move the punctuation before that quote, too.

INFO is the export state, as a property list."
  (let ((next (org-export-get-next-element object info))
        (previous
         (org-last
          (org-element-map (org-export-get-previous-element object info)
              'plain-text
            #'identity info)))
        (punct-re
         (rx string-start
             (group (or "." "!" "?" "..." "…"))
             (or eol (any " " "\t")))))
    (when (and previous
               (stringp next)
               (string-match punct-re next))
      (let* ((punct (match-string 1 next))
             (new-next (substring next (match-end 1)))
             (quote-re
              (rx (opt (and (group (or string-start (not (any "." "!" "?" "…"))))
                            (group "\"")))
                  (group (0+ (any " " "\t")))
                  string-end))
             (new-previous
              (replace-regexp-in-string
               quote-re (concat "\\1" punct "\\2\\3") previous)))
        (org-element-set-element next new-next)
        (org-element-set-element previous new-previous)))))

(defun org-cite-wrap-citation (citation info &optional move-punctuation)
  "Wrap an anonymous inline footnote around CITATION object in the parse tree.

INFO is the export state, as a property list.  When optional argument
MOVE-PUNCTUATION is non-nil, move punctuation character following citation before
it, when applicable.  If a quotation mark precedes the citation, move
punctuation before it, too.

The parse tree is modified by side-effect."
  (let ((footnote
         (list 'footnote-reference
               (list :label nil
                     :type 'inline
                     :contents-begin (org-element-property :begin citation)
                     :contents-end (org-element-property :end citation)
                     :post-blank (org-element-property :post-blank citation)))))
    ;; Remove any white space before citation.
    (org-cite--set-previous-post-blank citation 0 info)
    ;; Possibly swap punctuation around citation.
    (when move-punctuation (org-cite--swap-punctuation citation info))
    ;; Footnote swallows citation.
    (org-element-insert-before footnote citation)
    (org-element-adopt-elements footnote
    (org-element-extract-element citation))))

;; test citation processor

(defun test-export-citation (citation nil nil info)
  (org-cite-wrap-citation citation info t)
  "...")

(org-cite-register-processor 'test
  :export-citation #'test-export-citation)
--8<---------------cut here---------------end--------------->8---

For example, the following document:

--8<---------------cut here---------------start------------->8---
#+cite_export: test

This is a test [cite:@a].

This is a test [cite:@a]?

This is a test [cite:@a]...

This is a "test" [cite:@a].

This is a "test." [cite:@a].

This is a *"test"* [cite:@a].

This is a *some /covoluted/ "test"* [cite:@a].

# Do nothing in the following cases.

This is a test [cite:@a]

This is a test. [cite:@a]

[cite:@a].

This is a "test" [cite:@a]
--8<---------------cut here---------------end--------------->8---

would become in ASCII export (without the uninteresting footnotes part):

--8<---------------cut here---------------start------------->8---
This is a test.[1]

This is a test?[2]

This is a test…[3]

This is a « test. »[4]

This is a « test. ».[5]

This is a *« test. »*[6]

This is a *some /covoluted/ « test. »*[7]

This is a test[8]

This is a test.[9]

[10].

This is a « test »[11]
--8<---------------cut here---------------end--------------->8---

Is it what you had in mind?

Also, I have questions about punctuation. 

Currently, these changes only apply to terminal punctuation I could
think of: . ! ? ... …

Should the above apply to any punctuation, i.e., [:punct:]? Should it be
limited to a different set of elements instead? If so which one? Should
it be configurable?

Regards,
-- 
Nicolas Goaziou

Re: wip-cite status question and feedback

[Denis Maier]

Hi Nicolas,

thanks for all you work on this one. I don't have a setup where I can 
test this, but from what I can tell this looks quite good already.

Am 30.04.2021 um 15:28 schrieb Nicolas Goaziou:
[...]

> 
> OK. I wrote a POC, and I would appreciate some feedback about it.
> 
> In order to test it, one need to evaluate the following, from an
> up-to-date "wip-cite-new" branch:
> 
> --8<---------------cut here---------------start------------->8---
> (defun org-cite--swap-punctuation (object info)
>    "Move punctuation following OBJECT before it, if applicable.
> 
> When OBJECT is followed by some terminal punctuation character, such as
> a period or a question mark, and preceded by some text, move the punctuation
> at the end of the previous text.  If that text ends with a double quote,
> move the punctuation before that quote, too.
> 
> INFO is the export state, as a property list."
>    (let ((next (org-export-get-next-element object info))
>          (previous
>           (org-last
>            (org-element-map (org-export-get-previous-element object info)
>                'plain-text
>              #'identity info)))
>          (punct-re
>           (rx string-start
>               (group (or "." "!" "?" "..." "…"))
>               (or eol (any " " "\t")))))
>      (when (and previous
>                 (stringp next)
>                 (string-match punct-re next))
>        (let* ((punct (match-string 1 next))
>               (new-next (substring next (match-end 1)))
>               (quote-re
>                (rx (opt (and (group (or string-start (not (any "." "!" "?" "…"))))
>                              (group "\"")))
>                    (group (0+ (any " " "\t")))
>                    string-end))
>               (new-previous
>                (replace-regexp-in-string
>                 quote-re (concat "\\1" punct "\\2\\3") previous)))
>          (org-element-set-element next new-next)
>          (org-element-set-element previous new-previous)))))
> 
> (defun org-cite-wrap-citation (citation info &optional move-punctuation)
>    "Wrap an anonymous inline footnote around CITATION object in the parse tree.
> 
> INFO is the export state, as a property list.  When optional argument
> MOVE-PUNCTUATION is non-nil, move punctuation character following citation before
> it, when applicable.  If a quotation mark precedes the citation, move
> punctuation before it, too.
> 
> The parse tree is modified by side-effect."
>    (let ((footnote
>           (list 'footnote-reference
>                 (list :label nil
>                       :type 'inline
>                       :contents-begin (org-element-property :begin citation)
>                       :contents-end (org-element-property :end citation)
>                       :post-blank (org-element-property :post-blank citation)))))
>      ;; Remove any white space before citation.
>      (org-cite--set-previous-post-blank citation 0 info)
>      ;; Possibly swap punctuation around citation.
>      (when move-punctuation (org-cite--swap-punctuation citation info))
>      ;; Footnote swallows citation.
>      (org-element-insert-before footnote citation)
>      (org-element-adopt-elements footnote
>      (org-element-extract-element citation))))
> 
> ;; test citation processor
> 
> (defun test-export-citation (citation nil nil info)
>    (org-cite-wrap-citation citation info t)
>    "...")
> 
> (org-cite-register-processor 'test
>    :export-citation #'test-export-citation)
> --8<---------------cut here---------------end--------------->8---
> 
> For example, the following document:
> 
> --8<---------------cut here---------------start------------->8---
> #+cite_export: test
> 
> This is a test [cite:@a].
> 
> This is a test [cite:@a]?
> 
> This is a test [cite:@a]...
> 
> This is a "test" [cite:@a].
> 
> This is a "test." [cite:@a].
> 
> This is a *"test"* [cite:@a].
> 
> This is a *some /covoluted/ "test"* [cite:@a].
> 
> # Do nothing in the following cases.
> 
> This is a test [cite:@a]
> 
> This is a test. [cite:@a]
> 
> [cite:@a].
> 
> This is a "test" [cite:@a]
> --8<---------------cut here---------------end--------------->8---
> 
> would become in ASCII export (without the uninteresting footnotes part):
> 
> --8<---------------cut here---------------start------------->8---
> This is a test.[1]
> 
> This is a test?[2]
> 
> This is a test…[3]
> 
> This is a « test. »[4]
> 
> This is a « test. ».[5]
> 
> This is a *« test. »*[6]
> 
> This is a *some /covoluted/ « test. »*[7]
> 
> This is a test[8]
> 
> This is a test.[9]
> 
> [10].
> 
> This is a « test »[11]
> --8<---------------cut here---------------end--------------->8---
> 
> Is it what you had in mind?

Yes, this should be equivalent to the behaviour in pandoc.

However, as I've said before, this behaviour is only correct in American 
English. TO quuote the Chicago Manual of Style 6.9: "In an alternative 
system, sometimes called British Style (as described in the /New Oxford 
Style Manual ...) ... only those punctuation points that appeared in the 
original material are included within the quotation marks." The same 
would be correct for German. Do you have an idea if/how this could be 
implemented?

> 
> Also, I have questions about punctuation.
> 
> Currently, these changes only apply to terminal punctuation I could
> think of: . ! ? ... …
> 
> Should the above apply to any punctuation, i.e., [:punct:]? Should it be
> limited to a different set of elements instead? If so which one? Should
> it be configurable?

Again the Chicago Manual, 6.10: "Colons and semicolons---unlike periods 
and commas---follow closing quotation marks; question marks and 
exclamation points follow closing quoation marks unless they belong 
within the quoted matter."

I don't know if there is a general rule and how much of this should be 
configurable. How costly would that be?

> 
> Regards,
> 

Re: wip-cite status question and feedback

[Nicolas Goaziou]

Hello,

Denis Maier <denismaier@mailbox.org> writes:

> However, as I've said before, this behaviour is only correct in
> American English. TO quuote the Chicago Manual of Style 6.9: "In an
> alternative system, sometimes called British Style (as described in
> the /New Oxford Style Manual ...) ... only those punctuation points
> that appeared in the original material are included within the
> quotation marks." The same would be correct for German. Do you have an
> idea if/how this could be implemented?

This punctuation dance is done through an optional argument in
`org-cite-wrap-citation' (off by default). It is up to the citation
processors to activate it, according to their own rules, e.g., after
checking "language" keyword with

  (plist-get info :language)

or simply by letting users decide what to do through a defcustom.

I don't think we can do more in that area.

> Again the Chicago Manual, 6.10: "Colons and semicolons---unlike
> periods and commas---follow closing quotation marks; question marks
> and exclamation points follow closing quoation marks unless they
> belong within the quoted matter."

I understand colons and semicolons should not move within the quote. But
should they be moved before the note? I.e., should the following happen

               ... foo" [cite:@a];     =>    ... foo";[1]

or should it be?

               ... foo" [cite:@a];     =>    ... foo"[1];

IIUC, commas should also be moved within the quotes:

               ... foo" [cite:@a],     =>    ... foo,"[1]

Am I correct?

Concerning question marks and exclamation points, I assume we can safely
consider that if they belong to the quoted matter, they are already
included in the quotes. So the only case left to consider is the
following:

               ... foo" [cite:@a]?     =>    ... foo"?[1]

Is that right?

> I don't know if there is a general rule and how much of this should be
> configurable. How costly would that be?

I'd like to avoid any configuration variable in "oc.el", which is only
meant to be as the toolbox for processor developers. 

However, if required, configuration would happen through an optional
argument, possibly the same as the one activating the punctuation dance.
I.e., when non-nil, it can only provide a list of punctuation characters
to consider moving around, with rules explaining if they should be
included in a preceding quote.

Since this would be backward compatible, we don't need to implement it
for now if the use-case is, at this point, purely hypothetical.

Thank you for the feedback!

Regards,
-- 
Nicolas Goaziou

Re: wip-cite status question and feedback

[Bruce D'Arcus]

On Fri, Apr 30, 2021 at 5:48 PM Denis Maier <denismaier@mailbox.org> wrote:

> Yes, this should be equivalent to the behaviour in pandoc.
>
> However, as I've said before, this behaviour is only correct in American
> English.

Denis and I are working on sorting out the details of how to address
this off-list ATM.

But I'm tentatively thinking this could be addressed by splitting the
MOVE-PUNCTUATION parameter in two, so that we have:

MOVE-PUNCTUATION: Move punctuation character following citation before
it, when applicable (for example [TODO]).

PUNCTUATION-INSIDE-QUOTES: If a quotation mark precedes the citation,
move punctuation before it, too, unless [TODO]

So for the examples Nicolas posted, this:

(org-cite-wrap-citation citation info t)

… would change to this:

(org-cite-wrap-citation citation info t t)

... and to get the British/German output it would be:

(org-cite-wrap-citation citation info t nil)

More when we figure out if this is feasible, with example input/output, etc.

Bruce

Re: wip-cite status question and feedback

[Denis Maier]

While evaluating different aspects of punctuation moving I had another 
look at the csquotes package. p. 21 f. and p. 27 ff. in the manual 
(http://mirrors.ctan.org/macros/latex/contrib/csquotes/csquotes.pdf) are 
quite instructive.[1] This package a structured representation of a 
quotation, final punctuation in the quotation, the citation, and 
trailing punctuation, which all can then be reassembled according to 
different rules.
Now, I was wondering if something similar could be used for org as well. 
I'm thinking of something like this:

1. At first parse quotations and citations into lists 
'(quotation-content final-punct-in-quotation citation trailing-punctuation)
2. Fed that into a second function that reassembles the elements.

In that case, the function org-cite-wrap-punctuation would take a 
object, info, a boolean and a function as arguments, the boolean would 
enable/disable the function, the last argument could specify how the 
elements should be reassembled, so:
(defun org-cite-wrap-citation (citation info &optional move-punctuation 
move-punctuation-function)
=>
(org-cite-wrap-citation citation info t American)

Does that make sense? Would that be possible?

I'm thinking whether this could make the system more flexible and 
adaptable. And it would remedy the need to come up with all possible 
patterns as it should be easy to add those later. WDYT?

Denis

[1]:  As it seems, french typography seems to place note symbols just 
before the closing quotation mark.

Am 01.05.2021 um 15:26 schrieb Bruce D'Arcus:
> On Fri, Apr 30, 2021 at 5:48 PM Denis Maier <denismaier@mailbox.org> wrote:
>
>> Yes, this should be equivalent to the behaviour in pandoc.
>>
>> However, as I've said before, this behaviour is only correct in American
>> English.
> Denis and I are working on sorting out the details of how to address
> this off-list ATM.
>
> But I'm tentatively thinking this could be addressed by splitting the
> MOVE-PUNCTUATION parameter in two, so that we have:
>
> MOVE-PUNCTUATION: Move punctuation character following citation before
> it, when applicable (for example [TODO]).
>
> PUNCTUATION-INSIDE-QUOTES: If a quotation mark precedes the citation,
> move punctuation before it, too, unless [TODO]
>
> So for the examples Nicolas posted, this:
>
> (org-cite-wrap-citation citation info t)
>
> … would change to this:
>
> (org-cite-wrap-citation citation info t t)
>
> ... and to get the British/German output it would be:
>
> (org-cite-wrap-citation citation info t nil)
>
> More when we figure out if this is feasible, with example input/output, etc.
>
> Bruce

Re: wip-cite status question and feedback

[Bruce D'Arcus]

On Sun, May 2, 2021 at 5:59 PM Denis Maier <denismaier@mailbox.org> wrote:

> I'm thinking whether this could make the system more flexible and
> adaptable.

We'd still need to discuss details of course (like including sensible
defaults, etc.) if this were possible, but Denis and I agree that
having a second optional parameter as a function would be ideal.

> And it would remedy the need to come up with all possible
> patterns as it should be easy to add those later. WDYT?

And even more, presumably allow third party developers to come up with them?

Bruce

Re: wip-cite status question and feedback

[Bruce D'Arcus]

On the substance of these rules, my conclusion (and Denis knows this
are better than I, so can amend this) is the primary difference
between what Chicago calls "American" punctuation rules and "British"
is that the former puts trailing punctuation within the closing quote,
and the latter does not, except in rare circumstances.

This is independent of citations, BTW. It's just that the citations
expose them when we have to switch between different styles.

Given this, we could imagine a default function that itself took a
parameter to address that difference, with possible values like
"inside-punctuation" and "outside-punctuation," which of course would
be more general than calling them "American" and "British".

But those are among those "details" I noted in my previous message.

The function would be pretty close to the logic of what Nicolas
already wrote, but with some additional logic to deal with the
"British" case.

That would give pretty robust and flexible support out-of-box.

But allowing one to plug in a different function would give still more
flexibility, of course. It could well be my characterization of the
fundamental difference between those two punctuation rules breaks down
in other cases.

Bruce

Re: wip-cite status question and feedback

[Bruce D'Arcus]

On Sun, May 2, 2021 at 6:18 PM Bruce D'Arcus <bdarcus@gmail.com> wrote:

> On Sun, May 2, 2021 at 5:59 PM Denis Maier <denismaier@mailbox.org> wrote:
>
> > I'm thinking whether this could make the system more flexible and
> > adaptable.
>
> We'd still need to discuss details of course (like including sensible
> defaults, etc.) if this were possible, but Denis and I agree that
> having a second optional parameter as a function would be ideal.

If this is NOT practical, just to summarize what we've learned/concluded:

Assuming this basic example:

A sentence ending in a "cited quote" [cite:doe].

We found three rules:

1. what Chicago calls "American"
2. what it calls "British"
3. French (though Denis is still confirming how these work in actual books)

The output in each, when formatting as a note:

1. A sentence ending in a "cited quote."[1]
2. A sentence ending in a "cited quote".[1]
3. A sentence ending in a "cited quote[1]."

Note: Nicolas' POC correctly handles case 1.

Can be hard to see the nuances, but note that all three move the
punctuation, and remove the space between quote and note mark.

The difference is how they reassemble these components; in particular
where they put that trailing punctuation (period, etc.), and where
they put the note mark.

1. move punctuation inside quote
2. move punctuation outside quote
3. move note mark AND then punctuation inside quote

It could be they each need a boolean parameter, similar to what I
suggested earlier.

Then "American" (current) could be:

- punctuation-inside-quote t
- mark-inside-quote nil

"British" could be:

- punctuation-inside-quote nil
- mark-inside-quote nil

And "french":

- punctuation-inside-quote t
- mark-inside-quote t

But a potential (?) problem with this is one couldn't specify the
relative order of punctuation and note mark.

So Denis had an idea to simply allow the order of those components to
be configured, something like this (the lisp is probably wrong, but it
demonstrates the idea):

`(final-punct trailing-punct closing-quotation citation)

I think that covers current status.

So what to do?

Bruce

PS - we know, this seems a lot of effort for what seems a corner case.
But there will be some users for which this matters a lot.

Re: wip-cite status question and feedback

[M. ‘quintus’ Gülker]

Am 05. Mai 2021 um 09:46 Uhr -0400 schrieb Bruce D'Arcus:
> We found three rules:
> 
> 1. what Chicago calls "American"
> 2. what it calls "British"
> 3. French (though Denis is still confirming how these work in actual books)
> 
> The output in each, when formatting as a note:
> 
> 1. A sentence ending in a "cited quote."[1]
> 2. A sentence ending in a "cited quote".[1]
> 3. A sentence ending in a "cited quote[1]."

While I have never seen it stated authoritatively somewhere, in German
it appears to be common to use 1) if the terminal period is in the
cited source, and 2) if it is not, that is, just being exact in
quoting. As a result, both variants can occur in the same document,
because it depends on the cited source.

3) is doubtful in German. It would mean that there is a footnote 1 in
the cited source, but there is not reference for the cited source.
Correct it would have to be

    4) A sentence ending in a "cited quote¹"³.

or

    5) A sentence ending in a "cited quote¹".³

If the cited quote referenced by footnote 3 indeed does have a
footnote 1 in that position. That being said, I never saw such a
construction.

At least, that are the rules I recall from my school time and which I
used ever since with nobody complaining.

I wonder, can the placement of the footnote not just be left to the
author...? I have the impression that something is being
over-engineered here with the attempt to automate this, but maybe this
is just me.

  -quintus

-- 
Dipl.-Jur. M. Gülker | https://mg.guelker.eu |    For security:
Passau, Germany      | kontakt@guelker.eu    | () Avoid HTML e-mail
European Union       | PGP: see homepage     | /\ http://asciiribbon.org

Re: wip-cite status question and feedback

[Bruce D'Arcus]

On Wed, May 5, 2021 at 2:15 PM M. ‘quintus’ Gülker
<post+orgmodeml@guelker.eu> wrote:

> I wonder, can the placement of the footnote not just be left to the
> author...? I have the impression that something is being
> over-engineered here with the attempt to automate this, but maybe this
> is just me.

The use case is for users who may author with the expectation of using
one style for output (for example, author-date), but then find they
have to use another (say note-based).

It's not super common, since most scholars operate in fields where
practices are more-or-less consistent.

Scientists, for example, do not use note-based styles, and humanities
people do not use numeric ones.

But people (like me) who, for example, work at the borders of the
humanities and social sciences, can be required to switch among very
different styles. For example, they may submit a manuscript to one
journal that uses an author-date style, and it gets rejected there,
and then submit to another journal which is note-based.

So it is somewhat rare, but is not a purely hypothetical requirement.

Without this, the user would have to modify their entire document to
adjust the formatting, which is a hassle.

It's also not a hypothetical solution, as it's already implemented in
solutions like pandoc, and works well there.

But that currently works correctly for the "American" rules only,
which is why Denis put in an issue there for that, and why he has
weighed in here.

Hope that explains.

Bruce

Re: wip-cite status question and feedback

[Denis Maier]

Am 05.05.2021 um 20:14 schrieb M. ‘quintus’ Gülker:
> Am 05. Mai 2021 um 09:46 Uhr -0400 schrieb Bruce D'Arcus:
>> We found three rules:
>>
>> 1. what Chicago calls "American"
>> 2. what it calls "British"
>> 3. French (though Denis is still confirming how these work in actual books)
>>
>> The output in each, when formatting as a note:
>>
>> 1. A sentence ending in a "cited quote."[1]
>> 2. A sentence ending in a "cited quote".[1]
>> 3. A sentence ending in a "cited quote[1]."
> 
> While I have never seen it stated authoritatively somewhere, in German
> it appears to be common to use 1) if the terminal period is in the
> cited source, and 2) if it is not, that is, just being exact in
> quoting. As a result, both variants can occur in the same document,
> because it depends on the cited source.
> 
> 3) is doubtful in German. It would mean that there is a footnote 1 in
> the cited source, but there is not reference for the cited source.
> 

Just to be clear, these are meant as examples for the three language 
specific rules outlined above.
1. is an example for the "American" style, which consistently puts 
punctuation (commas and periods) inside quotation marks even if the 
punctuation mark does not appear in the original quotation.
2. is an example for the "British" style, which seems to conform to what 
seems to be correct in German.
3. is an example for what the latex csquotes describes as "French" style 
where the footnote mark seems to be included in the quotation just 
before the punctuation mark. Yesterday, I've tried to find examples for 
this rule in French book, and I couldn't. I found this:
	- Punctuation is placed inside or outside the quotation marks depending 
on whether you're quoting a complete sentence.
	- If punctuation is placed inside the quotation marks the order is: ."[1]
	- If punctuation is placed outside the quotation marks the order is: "[1].
	- If there is no preceding quotation the order is: [1].
Maybe a native French speaker can shed some light on this?

Denis

Re: wip-cite status question and feedback

[M. ‘quintus’ Gülker]

Am 05. Mai 2021 um 14:27 Uhr -0400 schrieb Bruce D'Arcus:
> Hope that explains.

Sure, thank you! I just wanted to provide some possibly useful input.
I am not to critise these exciting efforts.

  -quintus

-- 
Dipl.-Jur. M. Gülker | https://mg.guelker.eu |    For security:
Passau, Germany      | kontakt@guelker.eu    | () Avoid HTML e-mail
European Union       | PGP: see homepage     | /\ http://asciiribbon.org