are super-hidden technical blocks required?

are super-hidden technical blocks required?

[Torsten Wagner]

Hi,

I notice that with all this upcoming syncing and interaction with
third party programs as well as with programming languages org-mode
starts to require more and more information which are actually not
intend to be read in plain text. E.g. a UUID-number for each task to
enable two-way syncs with calendars.
Following the mailing list, I notice that it gets more and more
difficult for the devs to keep track of how to handle the text parsing
and each fix in the parsers need to be carefully checked against a
steadily increasing amount of cases to avoid the introduction of new
problems.

First I was wondering if it would be time to switch org-mode from text
to some sort of XML. The advantages would be a much much easier way to
parse data and it would make many many things for the devs much
easier. However, switching org-mode to some sort of XML would be
against one of its key-features... being plain text.

There are property-blocks and drawers to "hide" stuff, but at the
moment they contain a mixed set of data relevant and non-relevant for
a user.
Would it help to introduce a technical-property block which only
contains information intend to be used by other programs and parsers?
This blocks could be hidden under all normal means unlike really
someone want to see them and hit a special key-combo. That might be a
way to keep the format clean and plain text but get some of the
features known e.g. from XML. I know this might be tricky esp. copy
and paste operations need to take care that those blocks get correctly
moved.

What do you think?

Torsten

Re: are super-hidden technical blocks required?

[Bastien]

Hi Torsten,

I don't think this is about plain text vs à la XML (because XML and
friends are plain text too) but about whether we should allow a new
type of block to keep non-human-readable stuff out of the way.

One thing I don't get is how these blocks would be more hidden than
normal blocks or drawers.

Another thing I'm not sure about is what kind of technical stuff
you have in mind.  I see what you mean for UUID, but note that this
is not a problem of the :ID: property itself (which can hold a human
readable value), but of the value of this property when using UUIDs.

Speculating about Org possibilities is one of the fun here, so I
would not mind reading a more precise example.

But in general, as one can more or less guess, I'm more of a less
person.  :)

-- 
 Bastien

Re: are super-hidden technical blocks required?

[Rasmus]

Bastien <bzg@gnu.org> writes:

> I don't think this is about plain text vs à la XML (because XML and
> friends are plain text too) but about whether we should allow a new
> type of block to keep non-human-readable stuff out of the way.

How about just allowing for a list of properties being hidden.  For
instance all uuid properties could be hidden.

–Rasmus

-- 
A clever person solves a problem. A wise person avoids it

Re: are super-hidden technical blocks required?

[Russell Adams]

On Mon, Jul 30, 2012 at 11:27:28AM +0100, Rasmus wrote:
> Bastien <bzg@gnu.org> writes:
>
> > I don't think this is about plain text vs ?? la XML (because XML and
> > friends are plain text too) but about whether we should allow a new
> > type of block to keep non-human-readable stuff out of the way.
>
> How about just allowing for a list of properties being hidden.  For
> instance all uuid properties could be hidden.

Aren't property drawers already hidden by default?

------------------------------------------------------------------
Russell Adams                            RLAdams@AdamsInfoServ.com

PGP Key ID:     0x1160DCB3           http://www.adamsinfoserv.com/

Fingerprint:    1723 D8CA 4280 1EC9 557F  66E8 1154 E018 1160 DCB3

Fwd: are super-hidden technical blocks required?

[Torsten Wagner]

Sorry I forgot to CC the list.


---------- Forwarded message ----------
From: Torsten Wagner <torsten.wagner@gmail.com>
Date: 30 July 2012 17:42
Subject: Re: are super-hidden technical blocks required?
To: Bastien <bzg@gnu.org>


Hi Bastien,

I just think normal blocks or drawers are used by many org-user for
very different reasons and many store personal data into it.
After all, we love the flexibility in org-mode right :)

However, something which is saved in a org-mode file by another
application might not be intended to be read (nor modified) by users
directly. Hence having a mix of user data and non-user data in the
same drawer or property seems to me kind of annoying for the user and
maybe even dangerous, since you might brake something accidentely.

Thus, I was thinking of a block specially reserved for non-user
interaction. Completely hidden in all kind of views and maybe only
visible for debugging or expert-usage.

Examples would be the ID properties used by MobileOrg or UUID recently
used by org-caldav.el. Basically, all and everything which comes from
a automatic import/export/two-way sync and which is not intend to be
changed by the user directly. Guess there might be useful for stuff
from odf-import/export too. Other programs might have properties (e.g.
title color, notify sound, etc.) which have no equivalent in org-mode
but it would be nice to save them for a two-way sync.
As far as I understood two-way syncs with other programs are still
tricky because there is no standard way how to save information
necessary for the other side of the sync-chain.

My idea was something like

* Test
:DATA-PROPERTIES:
:UUID:
:CALENDAR-NAME:
:SERVER_ADDRESS:
:TIMESTAMP_SERVER:
:HASH-TAG:
:PROGRAM-FOO-PROPERTY:
:END:

which should be simply

* Test

in all normal org-mode views (or use a very minimalistic out of the
way indicator ?!)

not sure how well this would work out and maybe some people do not
like it since it hiding stuff. Others might like it because it removes
certain cluttering of (not as useful) information from the normal
views.

Just an idea


Torsten

On 30 July 2012 16:26, Bastien <bzg@gnu.org> wrote:
> Hi Torsten,
>
> I don't think this is about plain text vs ą la XML (because XML and
> friends are plain text too) but about whether we should allow a new
> type of block to keep non-human-readable stuff out of the way.
>
> One thing I don't get is how these blocks would be more hidden than
> normal blocks or drawers.
>
> Another thing I'm not sure about is what kind of technical stuff
> you have in mind.  I see what you mean for UUID, but note that this
> is not a problem of the :ID: property itself (which can hold a human
> readable value), but of the value of this property when using UUIDs.
>
> Speculating about Org possibilities is one of the fun here, so I
> would not mind reading a more precise example.
>
> But in general, as one can more or less guess, I'm more of a less
> person.  :)
>
> --
>  Bastien

Re: are super-hidden technical blocks required?

[Ivy Foster]

On 30 Jul 2012, at 11:26 am +0900, Torsten Wagner wrote:
> Hi,

Hi,

> [Because of the problems of syncing and interaction with
> third-party programs] I was wondering if it would be time
> to switch org-mode from text to some sort of XML.

I mostly lurk on this list, but reading the preceding
proposal I figured I should note that, as a user, one of the
key features of org-mode is its lovely simplicity of syntax
and interface. If I really wanted to keep my files in
hand-hacked or generated XML, I could, but I'd much rather
keep 'em in, well, org (-: .

> Would it help [alleviate the problem of property-blocks
> containing mixed user & technical data] to introduce a
> technical-property block which only contains information
> intend to be used by other programs and parsers? 

Sounds like an interesting idea.

> This blocks could be hidden under all normal means unlike
> really someone want to see them and hit a special
> key-combo.

Hmm, personally I'd rather have it visible but clearly
labeled. Transparency is nearly always a good thing.

It's great that you're thinking about this stuff, and I'll
look forward to seeing where these ideas go.

Cheers,
iff

Re: are super-hidden technical blocks required?

[Jonathan Leech-Pepin]

Hi,

On Mon, Jul 30, 2012 at 10:42 AM, Ivy Foster <joyfulgirl@archlinux.us> wrote:
> On 30 Jul 2012, at 11:26 am +0900, Torsten Wagner wrote:
>> Hi,
>
> Hi,
>
>> [Because of the problems of syncing and interaction with
>> third-party programs] I was wondering if it would be time
>> to switch org-mode from text to some sort of XML.
>
> I mostly lurk on this list, but reading the preceding
> proposal I figured I should note that, as a user, one of the
> key features of org-mode is its lovely simplicity of syntax
> and interface. If I really wanted to keep my files in
> hand-hacked or generated XML, I could, but I'd much rather
> keep 'em in, well, org (-: .
>
>> Would it help [alleviate the problem of property-blocks
>> containing mixed user & technical data] to introduce a
>> technical-property block which only contains information
>> intend to be used by other programs and parsers?
>
> Sounds like an interesting idea.

It sounds interesting however my first instinct is that it will not be
easy to make the distinctions.  Is :ID: meant as technical-data or
user-data?  Columns and Archive properties are more 'technical', yet
they are for use by Org.  With the new exporter/org-element you
retrieve the properties using =org-element-property= so the unneeded
properties don't need to be parsed by the exporters.

>> This blocks could be hidden under all normal means unlike
>> really someone want to see them and hit a special
>> key-combo.
>
> Hmm, personally I'd rather have it visible but clearly
> labeled. Transparency is nearly always a good thing.
>

Agreed.  If it's there I'd want to know it was there.  the :ARCHIVED:
tag does well enough at keeping content hidden for that purpose, but
you still see that it is present.  (So just don't open the drawer
unless you need it.)

> It's great that you're thinking about this stuff, and I'll
> look forward to seeing where these ideas go.
>
> Cheers,
> iff
>

Regards,
Jon

Re: are super-hidden technical blocks required?

[Robert Horn]

Jonathan Leech-Pepin <jonathan.leechpepin@gmail.com> writes:

> Hi,
>>> This blocks could be hidden under all normal means unlike
>>> really someone want to see them and hit a special
>>> key-combo.
>>
>> Hmm, personally I'd rather have it visible but clearly
>> labeled. Transparency is nearly always a good thing.
>>
>
I agree.  The real use needs more clarification.  Things like ID are
already well hidden as :PROPERTIES: until the user explicitly opens the
drawer for viewing.  I don't understand the need to hide those further, so
a better explanation of why is needed.

I could see a use for a "blob" property.  Suppose I have a JPEG of
someone's portrait that I want to incorporate into a contact's
properties.  Some mechanism to indicate:

:PROPERTIES:
:Portrait: blob-flag application/jpeg alternate-text="Joe's picture" <<hidden-content>>
:END:

with a third level of lisp-function that will open the blob contents as
ugly encoded text or with a decoding/viewing function instead of the
text "<<hidden-content>>".  I could see this being useful for both
textual items like S-exprs, XML, JSON, etc. and for binary things like
JPEG, PDF, etc.  It could be similar to the current embedding done for
literate programming with a way to separate blob content from org
content.

Right now the link capabilities are available for blob-like uses, but
links don't work if I need to move the org-mode files between machines,
etc.

R Horn
rjhorn@alum.mit.edu

Re: are super-hidden technical blocks required?

[Torsten Wagner]

Hi Robert,

Please see my follow up post with a more detailed description and a
(as I find) already better solution.
In summary, it is about providing a way to store data in org-mode
which is not intended to be read/modified by humans.
Your idea would be one part of it and I was thinking of that too.
> I could see a use for a "blob" property.  Suppose I have a JPEG of
> someone's portrait that I want to incorporate into a contact's
> properties.  Some mechanism to indicate:
Unfortunately, I notice that Emacs and org-mode can easily slow down
for large documents (I noticed, e.g. by using  thousand lines of
(collapsed) code blocks) . I frighten that an JPEG or PNG which would
easily fill hundreds or thousands of (hidden) lines might slow down
org-mode. That is, if there would be such a function, one would have
to make sure it does not influence the performance.

Torsten

Re: are super-hidden technical blocks required?

[Ilya Shlyakhter]

On 7/31/2012 9:23 AM, Robert Horn wrote:
> I agree.  The real use needs more clarification.  Things like ID are
> already well hidden as :PROPERTIES: until the user explicitly opens the
> drawer for viewing.  I don't understand the need to hide those further, so
> a better explanation of why is needed.

One common use would be to store the creation & last-modification dates 
of each entry.  I've tried various ways of doing it and they all were 
too obtrusive to use on _every_ entry.   Time-stamping of all entries
would be extremely useful, just as time-stamping of files is.
But I don't want to see the timestamps during normal Org usage.

Re: are super-hidden technical blocks required?

[Bastien]

Hi Ilya,

Ilya Shlyakhter <ilya_shl@alum.mit.edu> writes:

> But I don't want to see the timestamps during normal Org usage.

What do you think of "hiding" them by having a new face for properties
matching a custom regexp?  This has the advantage of letting the user
decide what to do with such properties: either hiding or highlighting
them.  This is also easier to implement.

-- 
 Bastien

Re: are super-hidden technical blocks required?

[Ilya Shlyakhter]

On 8/5/2012 5:16 AM, Bastien wrote:
> Hi Ilya,
>
> Ilya Shlyakhter <ilya_shl@alum.mit.edu> writes:
>
>> But I don't want to see the timestamps during normal Org usage.
>
> What do you think of "hiding" them by having a new face for properties
> matching a custom regexp?  This has the advantage of letting the user
> decide what to do with such properties: either hiding or highlighting
> them.  This is also easier to implement.

Then _every_ entry will still have a PROPERTIES drawer.
The problem wasn't what's in the drawer (which you hardly ever open), 
but having to see (and step around) that PROPERTIES line on every entry.

What about a HIDDEN_PROPERTIES drawer that, when folded, folds
completely (so that its title line is hidden too), and have a key to 
reveal such
drawers (the way M-tab opens archived entries)?

Re: are super-hidden technical blocks required?

[Nicolas Goaziou]

Hello,

> What about a HIDDEN_PROPERTIES drawer that, when folded, folds
> completely (so that its title line is hidden too), and have a key to
> reveal such drawers (the way M-tab opens archived entries)?

This is begging for problems.  At some point, an user will start to
notice weird behaviour he cannot explain from, let's say, the exporter,
because he will have completely forgotten about one totally hidden
drawer setting export options.

I don't think anything should be made totally invisible. There should
always be visual clues for such things.

PROPERTIES drawer is as hidden as it can decently be. Since its contents
are mostly for Org internals, the average user is, usually, not supposed
to expand it. If you often need to have a look at some properties, you
may use column view instead.


Regards,

-- 
Nicolas Goaziou

Re: are super-hidden technical blocks required?

[Ilya Shlyakhter]

On Sun, Aug 5, 2012 at 6:20 PM, Nicolas Goaziou <n.goaziou@gmail.com> wrote:
>> What about a HIDDEN_PROPERTIES drawer that, when folded, folds
>> completely (so that its title line is hidden too), and have a key to
>> reveal such drawers (the way M-tab opens archived entries)?
>
> This is begging for problems.  At some point, an user will start to
> notice weird behaviour he cannot explain from, let's say, the exporter,
> because he will have completely forgotten about one totally hidden
> drawer setting export options.
>
> I don't think anything should be made totally invisible. There should
> always be visual clues for such things.

But if _every_ entry has a visible PROPERTIES drawer (because a
creation and last-modification
dates of the entry are stored there), then entries for which custom
export options were set no longer stand out.
Just as, if you BOLD every word in a document, BOLD no longer acts as
a highlight.

So, I agree that unique customizations should lead to a visible indicator.
But for properties that aren't really customizations ( creation date,
modification date, unique ID ),
a visible indicator only clutters things without acting as a useful highlight.
So, what about making a UNIVERSAL_PROPERTIES drawer that is normally
completely hidden,
unless you explicitly reveal it?

Re: are super-hidden technical blocks required?

[Torsten Wagner]

Hey,

during this discussions people already claimed that they would prefer
to know what is stored and I can understand this.
That was the reason for the proposal of a HIDDEN_PROP: line to mark
certain properties hidden.
The benefit of this approach, people are actively aware of what they
hide and they can hide whatever they want, even stuff which they might
under other working schemes. E.g., for a certain workflow they might
prefer to hide each and all properties.

I can see the point that the property drawer header can be annoying
too. Actually, when I used orgmobile for the first time I was not too
happy to see all this property drawers suddenly appearing in my files.

Even proposing it first, I would not suggest to introduce a second
kind of drawer anymore. Chances are great that people start mixing
informations between two different drawers and then stuff could get
ugly and messy. It also introduce much new syntax and bug-fixing,
problem-solving, and further improvements have to be done for two
almost identically cases.

Alternatively to a new kind of drawer, I would think of the
HIDDEN_PROP: line and an additional method which hides a drawer even
more if it only contains hidden property elements. That could be done,
for example, by the already mentioned custom face.

That is, a drawer is clearly visible if it contains properties intend
to be read/changed by the user (not marked invisible).
A drawer is less visible, if it contains only properties marked as hidden.

Hidden properties do not reveal by the normal way of opening a drawer.
However, there presence might be indicate by a "..." line at the end.
A special key-combo reveals them (e.g. for debugging, or as some kind
of expert-mode).

That would really give the largest amount of flexibility, is backward
compatible, does not introduce to much new syntax ("just" a new header
parameter) and people can easily change what should be visible and
invisible by changing one single line.

BTW. Timestamps of the last changes are really a good example for a
hidden property. I was thinking already of a hashtag created by the
title and body of a node. That could be compared by synctools with the
present hash and thus, local changes could be identified.

If I just would now more about elisp ;)

Greetings

Torsten

Re: are super-hidden technical blocks required?

[Ilya Shlyakhter]

On Sun, Aug 5, 2012 at 10:46 PM, Torsten Wagner
<torsten.wagner@gmail.com> wrote:
> I can see the point that the property drawer header can be annoying
> too. Actually, when I used orgmobile for the first time I was not too
> happy to see all this property drawers suddenly appearing in my files.

> Alternatively to a new kind of drawer, I would think of the
> HIDDEN_PROP: line and an additional method which hides a drawer even
> more if it only contains hidden property elements. That could be done,
> for example, by the already mentioned custom face.
>
> That is, a drawer is clearly visible if it contains properties intend
> to be read/changed by the user (not marked invisible).
> A drawer is less visible, if it contains only properties marked as hidden.

That would be great.  Except, if a PROPERTIES drawer holds only hidden
properties,
it would be best to completely hide the :PROPERTIES: line (using
outline-flag-region), rather than display it in
a dimmed font.  So that, unless you explicitly ask to reveal these
lines, you edit the file as if they weren't there.

Re: are super-hidden technical blocks required?

[Jonathan Leech-Pepin]

Hello,

On Sun, Aug 5, 2012 at 11:01 PM, Ilya Shlyakhter <ilya_shl@alum.mit.edu> wrote:
> On Sun, Aug 5, 2012 at 10:46 PM, Torsten Wagner
> <torsten.wagner@gmail.com> wrote:
>> I can see the point that the property drawer header can be annoying
>> too. Actually, when I used orgmobile for the first time I was not too
>> happy to see all this property drawers suddenly appearing in my files.
>
>> Alternatively to a new kind of drawer, I would think of the
>> HIDDEN_PROP: line and an additional method which hides a drawer even
>> more if it only contains hidden property elements. That could be done,
>> for example, by the already mentioned custom face.
>>
>> That is, a drawer is clearly visible if it contains properties intend
>> to be read/changed by the user (not marked invisible).
>> A drawer is less visible, if it contains only properties marked as hidden.
>
> That would be great.  Except, if a PROPERTIES drawer holds only hidden
> properties,
> it would be best to completely hide the :PROPERTIES: line (using
> outline-flag-region), rather than display it in
> a dimmed font.  So that, unless you explicitly ask to reveal these
> lines, you edit the file as if they weren't there.
>

The issue I can see with completely hiding :PROPERTIES: line is
that you would then run the risk of adding text at the wrong
location (between the headline and the drawer for example).  At
the moment when the drawer is folded you know if the point is
before, within or after the drawer (even though you still can
remove parts of :end: by accident with backspaces), if it isn't
visible at all you don't have that ability.

Re: are super-hidden technical blocks required?

[Ilya Shlyakhter]

On Mon, Aug 6, 2012 at 8:12 AM, Jonathan Leech-Pepin
<jonathan.leechpepin@gmail.com> wrote:
> The issue I can see with completely hiding :PROPERTIES: line is
> that you would then run the risk of adding text at the wrong
> location (between the headline and the drawer for example).  At
> the moment when the drawer is folded you know if the point is
> before, within or after the drawer (even though you still can
> remove parts of :end: by accident with backspaces), if it isn't
> visible at all you don't have that ability.

Maybe, Org should allow text between the headline and the drawer?
Is there a reason not to?

Alternately, if a UNIVERSAL_PROPERTIES drawer was always on the line immediately
following the headline, there would be no way to place text between it
and the headilne.

Also, maybe a hidden drawer could be protected by setting character
properties on it
to make it unmodifiable?

Re: are super-hidden technical blocks required?

[Allen S. Rout]

On 08/04/2012 02:10 PM, Ilya Shlyakhter wrote:
>
> One common use would be to store the creation & last-modification dates
> of each entry.  I've tried various ways of doing it and they all were
> too obtrusive to use on _every_ entry.   Time-stamping of all entries
> would be extremely useful, just as time-stamping of files is.
> But I don't want to see the timestamps during normal Org usage.


As a user, if your code is decorating my tree, I want to know it.  If 
you hide it, I'd be mad.   Org is my life in plain text, not WordPerfect 
with reveal-codes.


- Allen S. Rout

Re: are super-hidden technical blocks required?

[Michael Brand]

On Mon, Aug 6, 2012 at 8:16 PM, Allen S. Rout <asr@ufl.edu> wrote:
> Org is my life in plain text, not WordPerfect with reveal-codes.

I always wondered what Ford Prefect is doing in the Org Manual and why
he is related with Org. :-))

Michael

Re: are super-hidden technical blocks required?

[Ilya Shlyakhter]

On 8/6/2012 2:16 PM, Allen S. Rout wrote:
>> One common use would be to store the creation & last-modification dates
>> of each entry.  I've tried various ways of doing it and they all were
>> too obtrusive to use on _every_ entry.   Time-stamping of all entries
>> would be extremely useful, just as time-stamping of files is.
>> But I don't want to see the timestamps during normal Org usage.
>
> As a user, if your code is decorating my tree, I want to know it.  If
> you hide it, I'd be mad.   Org is my life in plain text, not WordPerfect
> with reveal-codes.

For decorations that change behavior, e.g. export options or inherited 
properties, sure.
But meta-information such as creation/modification times or unique node 
ids, which do not change behavior and are known to be associated with 
every node, displaying them is a distraction.  If you already know that
every node has a creation time, what is added by seeing a :PROPERTIES: 
line for that node?  If anything, it obscures nodes that do have unique
properties you want to know about.

Re: are super-hidden technical blocks required?

[Torsten Wagner]

Hi,

I was aware that some people would prefer a more transparent way.
I think a technical property block accessible for third party sync
tools would make it much easier for the developers of those tools.
On the other hand I understand that people want to know what is stored
along within there org-files.

One Idea I had and which was mentioned by Rasmus already too, would be
to use the same property block but being able to hide certain
properties

#+ HIDE_PROP: ID, UUID, ODF_PROP, MOBILE_ORG_PROP

I guess many people use only one or the other third party tool and
hence once set-up correctly in there files, its rather easy to mask
them invisible or visible depending on personal preferences and needs.
This approach makes people also actively aware of those properties and
does not try to hide them.

Maybe for the upcoming future if programs would like to introduce
rather many properties the list could be sensitive to wildcards?!
so #+ HIDE_PROP: ODF_*
would hide all properties introduced for ODF in- and export.

even a pure technical block could be realized by this if there is a
rule to start properties of sole technical purpose by a certain
key-character e.g. a underline (if this is legal as property name, not
sure about). Then _* would hide all technical properties. _ODF_* hides
all technical properties for ODF _OM_* does the same for orgmobile,
etc.  Developers could decide whether a property is more of technical
nature or if user should be more aware of it (and possibly change it).
Using not only wildcards but real regular expression (which might not
be to difficult in elisp) would make this even more flexible.

To indicate that a property block contains hidden technical properties
I guess, as usual in org-mode a  line of "..." could indicate this.
E.g.,
#+ HIDE_PROP: _*
* Test
:PROPERTIES:
:name: Max Muster
:address: Muster-Street 1,
:tel: 11223344
...
:END:

#+ HIDE_PROP:
* Test
:PROPERTIES:
:name: Max Muster
:address: Muster-Street 1,
:tel: 11223344
:_UUID: 123j2k2j32kj332k1j2h1ghj1g321
:_OM_COLOUR: red
:_OM_image: ./max_muster.jpg
:_SYNC_TS_UNIX: 1342483647
:_OR_CODE: ./max_muster_QR.png
:END:

I guess this solution would fit to everyone, people who like to know
what is stored within there org-files and people who want a clutter
free org-file for easy reading (and sure people can easily switch
between both approaches). Finally, developers would have a rather easy
and out-of-the-way possibility to save as much information within the
org-file as needed to e.g. enable 2-way syncs with other programs.
After all, everything still keeps plain text.

Any thoughts?

Torsten

Re: are super-hidden technical blocks required?

[Bastien]

Hi Thorsten,

thanks for the detailed example.  As I said, I tend to be conversative
about such topics.  Not because I'm already too old, but because this is
often not worth the time-to-implement/complexity-in-code.  So I'm still
open to read a very compelling case where "tech" properties need to be
hidden...  

Of course, "need" is subjective -- let's say if you manage to have at
least 3 friends complaining about tech properties being visible when
unfolding a drawer, I'm all ears :)

-- 
 Bastien

Re: are super-hidden technical blocks required?

[Torsten Wagner]

Hey Bhastien,

thanks for keeping the topic up. Well, I guess people who are dealing
with import/export from third-party programs might have an idea how to
use this functionality (and can tell us how useful this would be). I
can try to contact the authors of mobileorg for iphone and android as
well as some other authors of sync-tools (if they are not already
contributing to this discussion). Lets see what is there opinion.

All the best

Torsten


On 1 August 2012 22:29, Bastien <bzg@gnu.org> wrote:
> Hi Thorsten,
>
> thanks for the detailed example.  As I said, I tend to be conversative
> about such topics.  Not because I'm already too old, but because this is
> often not worth the time-to-implement/complexity-in-code.  So I'm still
> open to read a very compelling case where "tech" properties need to be
> hidden...
>
> Of course, "need" is subjective -- let's say if you manage to have at
> least 3 friends complaining about tech properties being visible when
> unfolding a drawer, I'm all ears :)
>
> --
>  Bastien

Re: are super-hidden technical blocks required?

[Christopher J. White]

Hi Folks,

I thought I'd throw in my 2c on the topic.  I work on org-toodledo which 
syncs TODO items with Toodledo.com.  On first sync, it creates adds a 
"ToodledID" property to track the ID assigned by the server.

In my use case, that majority of TODO items have *no* other properties. 
  As such, many items have a PROPERTIES drawer with just the one entry.

What I see is visual clutter.  Many of my TODO items are also very small 
-- often no body at all.  So the only thing beneath the item is the 
property drawer plus other "properties" like DEADLINE/SCHEDULED/CLOSED. 
  When trying to browse my todo list, it gets a little painful when 
every other line is ":PROPERTIES:...", or DEADLINE, etc.

I rarely (never?) edit any of these properties directly manually.  I 
either modify them via agenda mode, keys (C-c C-s), or via column view 
that pulls out interesting properties that I like to edit.

So for me, I want the entire *drawer* to disappear, as well as 
SCHEDULED/DEADLINE and CLOSED lines.

I've personally thought there should be an extra step in the visibility 
cycling:

<TAB>
   -> FOLDED -> CHILDREN -> SUBTREE -> PROPERTIES

S-<TAB>

   -> OVERVIEW -> CONTENTS -> SHOW ALL (minus PROPS) -> PROPERTIES

...cj


On 8/1/12 9:19 PM, Torsten Wagner wrote:
> Hey Bastien,
>
> thanks for keeping the topic up. Well, I guess people who are dealing
> with import/export from third-party programs might have an idea how to
> use this functionality (and can tell us how useful this would be). I
> can try to contact the authors of mobileorg for iphone and android as
> well as some other authors of sync-tools (if they are not already
> contributing to this discussion). Lets see what is there opinion.
>
> All the best
>
> Torsten
>
>
> On 1 August 2012 22:29, Bastien <bzg@gnu.org> wrote:
>> Hi Thorsten,
>>
>> thanks for the detailed example.  As I said, I tend to be conversative
>> about such topics.  Not because I'm already too old, but because this is
>> often not worth the time-to-implement/complexity-in-code.  So I'm still
>> open to read a very compelling case where "tech" properties need to be
>> hidden...
>>
>> Of course, "need" is subjective -- let's say if you manage to have at
>> least 3 friends complaining about tech properties being visible when
>> unfolding a drawer, I'm all ears :)
>>
>> --
>>   Bastien
>
>

Re: are super-hidden technical blocks required?

[Achim Gratz]

Torsten Wagner writes:
> One Idea I had and which was mentioned by Rasmus already too, would be
> to use the same property block but being able to hide certain
> properties
>
> #+ HIDE_PROP: ID, UUID, ODF_PROP, MOBILE_ORG_PROP

I don't think that moves us into the right direction...

Let me again suggest that drawers might be extended to have "stops"
which would enable them to unfold from stop to stop instead of just
being fully opened or closed.  As long as the "uninteresting" properties
are kept after the last stop, they don't get into the way of normal
operation.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Waldorf MIDI Implementation & additional documentation:
http://Synth.Stromeko.net/Downloads.html#WaldorfDocs

Re: are super-hidden technical blocks required?

[Bernt Hansen]

Achim Gratz <Stromeko@nexgo.de> writes:

> Torsten Wagner writes:
>> One Idea I had and which was mentioned by Rasmus already too, would be
>> to use the same property block but being able to hide certain
>> properties
>>
>> #+ HIDE_PROP: ID, UUID, ODF_PROP, MOBILE_ORG_PROP
>
> I don't think that moves us into the right direction...
>
> Let me again suggest that drawers might be extended to have "stops"
> which would enable them to unfold from stop to stop instead of just
> being fully opened or closed.  As long as the "uninteresting" properties
> are kept after the last stop, they don't get into the way of normal
> operation.

Or just use two drawers... PROPERTIES and SYNCDATA (or some other
appropriate name) so you unfold the on you care about and leave the
other folded.  That seems a lot simpler than stops in drawers...

Regards,
Bernt

Re: are super-hidden technical blocks required?

[Achim Gratz]

Bernt Hansen writes:
> Or just use two drawers... PROPERTIES and SYNCDATA (or some other
> appropriate name) so you unfold the on you care about and leave the
> other folded.  That seems a lot simpler than stops in drawers...

It might seem to, but I'm actually using this right now (LOGBOOK and
CLOSED) and it really sucks having to move the overrun in LOGBOOk to
CLOSED once a week or so.  Just like a real drawer of any size, there
are always some things you expect to find quickly and some other things
you know are there, butt need some rummaging around to find them.  That
is different from two drawers...


Regards.
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for KORG EX-800 and Poly-800MkII V0.9:
http://Synth.Stromeko.net/Downloads.html#KorgSDada

Re: are super-hidden technical blocks required?

[Torsten Wagner]

Hi Achim,

well the HIDE_PROP approach would have the benefit that one can easily
and quickly change what he wants to see and what he wants to hide at a
single place within his org-file. E.g., I might like to hide almost
all and everything in a drawer normally, but need to work on special
properties from time to time.
Other than this, I agree that some stop function in a drawer might be
better compared to an extra/individual drawer.

Torsten


On 2 August 2012 02:11, Achim Gratz <Stromeko@nexgo.de> wrote:
> Torsten Wagner writes:
>> One Idea I had and which was mentioned by Rasmus already too, would be
>> to use the same property block but being able to hide certain
>> properties
>>
>> #+ HIDE_PROP: ID, UUID, ODF_PROP, MOBILE_ORG_PROP
>
> I don't think that moves us into the right direction...
>
> Let me again suggest that drawers might be extended to have "stops"
> which would enable them to unfold from stop to stop instead of just
> being fully opened or closed.  As long as the "uninteresting" properties
> are kept after the last stop, they don't get into the way of normal
> operation.
>
>
> Regards,
> Achim.
> --
> +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+
>
> Waldorf MIDI Implementation & additional documentation:
> http://Synth.Stromeko.net/Downloads.html#WaldorfDocs
>
>

Re: are super-hidden technical blocks required?

[Bastien]

If the whole point is to make some properties less visible,
why not a solution based on fontification?

We could have a user-defined regexp to highlight (or "dim")
certain properties.

I don't believe in a solution that would change the current
flow of cycling through drawers.  I feel that's too much.

Patch welcome,

-- 
 Bastien

Re: are super-hidden technical blocks required?

[Ilya Shlyakhter]

On 8/2/2012 11:10 AM, Bastien wrote:
> If the whole point is to make some properties less visible,
> why not a solution based on fontification?
>
> We could have a user-defined regexp to highlight (or "dim")
> certain properties.

That would still leave the :PROPERTIES: line visible, which is problem
for "universal" properties assigned to every entry.

> I don't believe in a solution that would change the current
> flow of cycling through drawers.  I feel that's too much.

I agree.  Showing hidden drawers is a rarely-used operation and should
be a separate command, not part of the commonly-used cycling.

babel awk with table input: Code block produced no output.

[Greg Minshall]

hi.  it appears that a left or right paren in an entry in a table makes
awk not execute.  here's an example (change ":stdin fails" to ":stdin
works" to see it work).  cheers, Greg
----
#+tblname: fails
| proto          | no c code                   |            |
| pscl           | c code, just fine           |            |
| quadprog       | (minimal) c code, just fine |            |

#+tblname: works
| proto          | no c code                   |            |
| pscl           | c code, just fine           |            |
| quadprog       | minimal c code, just fine   |            |

#+begin_src awk :stdin fails
  BEGIN {
      print "starting"
  }
  {
      print $0
  }
#+end_src

Re: babel awk with table input: Code block produced no output.

[Sebastien Vauban]

Hi Greg,

Greg Minshall wrote:
> hi.  it appears that a left or right paren in an entry in a table makes
> awk not execute.  here's an example (change ":stdin fails" to ":stdin
> works" to see it work).  cheers, Greg
> ----
> #+tblname: fails
> | proto          | no c code                   |            |
> | pscl           | c code, just fine           |            |
> | quadprog       | (minimal) c code, just fine |            |
>
> #+tblname: works
> | proto          | no c code                   |            |
> | pscl           | c code, just fine           |            |
> | quadprog       | minimal c code, just fine   |            |
>
> #+begin_src awk :stdin fails
>   BEGIN {
>       print "starting"
>   }
>   {
>       print $0
>   }
> #+end_src

Confirmed.

Though, no idea why.

Best regards,
  Seb

-- 
Sebastien Vauban

[BUG] Traceback on Org-Export

[Luis Anaya]

Guys:

I pulled the latest and greatest from  Git after I uploaded by changes
on the exporter and I got the following trace during import of org-export. 

??

Luis

Debugger entered--Lisp error: (void-variable org-outline-regexp-bol)
  (concat "^[   ]*$" "\\|" org-outline-regexp-bol "\\|" "^[     ]*#\\+" $
  (defconst org-element-paragraph-separate (concat "^[  ]*$" "\\|" org-o$
  eval-buffer(#<buffer  *load*<4>> nil "/opt/home/papo/.emacs.d/org/lisp$
  load-with-code-conversion("/opt/home/papo/.emacs.d/org/lisp/org-elemen$
  require(org-element)
  eval-buffer(#<buffer  *load*<3>> nil "/opt/home/papo/.emacs.d/org/cont$
  load-with-code-conversion("/opt/home/papo/.emacs.d/org/contrib/lisp/or$
  require(org-export)
  eval-buffer(#<buffer  *load*<2>> nil "/opt/home/papo/.emacs.d/org/cont$
  load-with-code-conversion("/opt/home/papo/.emacs.d/org/contrib/lisp/or$
  require(org-e-groff)
  eval-buffer(#<buffer  *load*> nil "/opt/home/papo/.emacs" nil t)  ; Re$
  load-with-code-conversion("/opt/home/papo/.emacs" "/opt/home/papo/.ema$
  load("~/.emacs" t t)
  #[nil "^H\205\264^@   \306=\203^Q^@\307^H\310Q\2027^@ \311=\2033^@\312$
  command-line()
  normal-top-level()

-- 
Luis R. Anaya
papo anaya aroba hot mail punto com
"Do not use 100 words if you can say it in 10" - Yamamoto Tsunetomo

Re: [BUG] Traceback on Org-Export

[Bastien]

Hi Luis,

Luis Anaya <papoanaya@hotmail.com> writes:

> I pulled the latest and greatest from  Git after I uploaded by changes
> on the exporter and I got the following trace during import of
> org-export. 

I can't reproduce this.  Perhaps ~$ make cleanall and ~$ make again?

-- 
 Bastien

Re: [BUG] Traceback on Org-Export

[Luis Anaya]

Bastien <bzg@gnu.org> writes:

> Hi Luis,
>
> Luis Anaya <papoanaya@hotmail.com> writes:
>
> I can't reproduce this.  Perhaps ~$ make cleanall and ~$ make again?

Tried it... same error. I'm getting the following compilation error
which is too much of a coincidence. 

I'll keep digging.

Luis


In org-toggle-heading:
org.el:19364:26:Warning: attempt to let-bind constant `org-outline-regexp-bol'

In org-mark-subtree:
org.el:20638:28:Warning: attempt to let-bind constant `org-outline-regexp-bol'

In org-yank-generic:
org.el:21355:37:Warning: attempt to let-bind constant `org-outline-regexp-bol'

In org-yank-folding-would-swallow-text:
org.el:21384:37:Warning: attempt to let-bind constant `org-outline-regexp-bol'




-- 
Luis Anaya
papo anaya aroba hot mail punto com
"Do not use 100 words if you can say it in 10" - Yamamoto Tsunetomo

Re: [BUG] Traceback on Org-Export

[Achim Gratz]

Luis Anaya writes:
> In org-toggle-heading:
> org.el:19364:26:Warning: attempt to let-bind constant `org-outline-regexp-bol'

Are you using Emacs23? Because I don't get these on Emacs24 and they
look positively wierd...

Plus, it's one of those instances where it seems to make a difference in
which order the files get compiled.  If I compile _only_ org.el, I only
get:

In toplevel form:
org.el:4994:1:Error: Symbol's value as variable is void: org-outline-regexp-bol

at the line of (require 'org-element).  Org-element can't require org
since that would become recursive, so it should actually declare those
variables, but that doesn't work since it evaluates some constants at
compile-time and without a (require 'org) these aren't defined yet.
Trying to workaround this issue with eval-and-compile seems possible,
but doesn't really appeal to me.

This seems to be further fallout from moving the require forms in org.el
past the constant and variable definitions (which does make a difference
for Emacs23 vs Emacs24 since the scope rules are now different).
Bastien, what was the problem you were trying to solve there?  In any
case, the interaction between org and org-element needs a serious
overhaul and while we're at it this work should be extended to other
subsystems of org.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Terratec KOMPLEXER:
http://Synth.Stromeko.net/Downloads.html#KomplexerWaves

Re: [BUG] Traceback on Org-Export

[Achim Gratz]

Achim Gratz writes:
> In any case, the interaction between org and org-element needs a
> serious overhaul and while we're at it this work should be extended to
> other subsystems of org.

I've just pushed two new make targets to master that will compile org
with a single Emacs process per file, compile-single and compile-source.

The second of these targets actually removes each compiled file as soon
as it is created so that Emacs will always work from _only_ sources for
each compilation.  I might add a target that does the opposite, namely
compiling all files and then removing just one single compiled file and
compile it again, just to have another point of reference.

These should show unclean interdependencies between sources more
cleanly and it does.  Besides the org-outline-regexp-bol stuff Luis has
been stumbling on, there are quite a few more — mostly in Babel and
org-element, if only becausae these have seen the most development
activity recently.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for KORG EX-800 and Poly-800MkII V0.9:
http://Synth.Stromeko.net/Downloads.html#KorgSDada

Re: [BUG] Traceback on Org-Export

[Bastien]

Achim Gratz <Stromeko@nexgo.de> writes:

> This seems to be further fallout from moving the require forms in org.el
> past the constant and variable definitions (which does make a difference
> for Emacs23 vs Emacs24 since the scope rules are now different).
> Bastien, what was the problem you were trying to solve there?  

I was trying to get rid of the many compiler warnings that you get from
requiring libraries that relies on such variables and constants.

> In any case, the interaction between org and org-element needs a
> serious overhaul and while we're at it this work should be extended to
> other subsystems of org.

Yes.  The whole maze needs a treat.  Please submit a patch if you feel
like undertaking this task.  I managed to get zero warnings with Emacs
24.1 and did not take the time to check with Emacs 23.

Thanks,

-- 
 Bastien

Re: [BUG] Traceback on Org-Export

[Achim Gratz]

Bastien writes:
> Yes.  The whole maze needs a treat.  Please submit a patch if you feel
> like undertaking this task.  I managed to get zero warnings with Emacs
> 24.1 and did not take the time to check with Emacs 23.

That is going to be one big hairy patch that touches almost everything.
Maybe it is is possible to chop that down into a more manageable series
of smaller changes, starting with those that will rescue org-element.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Waldorf MIDI Implementation & additional documentation:
http://Synth.Stromeko.net/Downloads.html#WaldorfDocs

Re: [BUG] Traceback on Org-Export

[Bastien]

Achim Gratz <Stromeko@nexgo.de> writes:

> Bastien writes:
>> Yes.  The whole maze needs a treat.  Please submit a patch if you feel
>> like undertaking this task.  I managed to get zero warnings with Emacs
>> 24.1 and did not take the time to check with Emacs 23.
>
> That is going to be one big hairy patch that touches almost everything.
> Maybe it is is possible to chop that down into a more manageable series
> of smaller changes, starting with those that will rescue org-element.

All right.  If useful, please push a public branch for this on
orgmode.org.  I will try downloading Emacs 23 and see what warnings
will have here.

-- 
 Bastien

Re: [BUG] Traceback on Org-Export

[Achim Gratz]

Bastien writes:
> All right.  If useful, please push a public branch for this on
> orgmode.org.  I will try downloading Emacs 23 and see what warnings
> will have here.

Done, branch name is "Globals".  Merge it fast or be prepared for tons
of merge conflicts.  I already resolved the ones created by your recent
changes, but not the warnings introduced by them.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf rackAttack:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: [BUG] Traceback on Org-Export

[Luis Anaya]

Bastien <bzg@gnu.org> writes:

> I can't reproduce this.  Perhaps ~$ make cleanall and ~$ make again?

Ok... my bad... pilot error... missing (require 'org). I would've
thought (org-install) would've loaded it. Anyway, this solved the
problem by adding the require on my .emacs file.

Now I need some tea...



Luis


-- 
Luis Anaya
papo anaya aroba hot mail punto com
"Do not use 100 words if you can say it in 10" - Yamamoto Tsunetomo

Re: [BUG] Traceback on Org-Export

[Giovanni Ridolfi]

Da: Luis Anaya <papoanaya@hotmail.com>
Inviato: Venerdì 3 Agosto 2012 12:43

> Ok... my bad... pilot error... missing (require 'org). 

> [...]  Anyway, this solved the
> problem by adding the require on my .emacs file.


?-)

why is this needed? 
The documentation doesn't mention it and I commented it out in my .emacs
few weeks ago.


Perhaps you do run the export in batch mode?


Giovanni /a bit puzzled


Org-mode version 7.8.11 (4732f153f

Re: [BUG] Traceback on Org-Export

[Luis Anaya]

Giovanni Ridolfi <giovanni.ridolfi@yahoo.it> writes:

>> Ok... my bad... pilot error... missing (require 'org). 
>
> ?-)
>
> why is this needed? 

Good question :) Like I said, I thought that org-install would have
loaded it. But being that I'm more interested in testing the Groff
exporter to the latest pull from git, it became a secondary task. 

(and I'm now fixing the exporter to meet parameter changes in the 
org-export.el API :( )


> Perhaps you do run the export in batch mode?

No. Not while I'm debugging code. I should try for testing though, in
this way I can automate the results. But for now, this is done on the
console. 

> Giovanni /a bit puzzled

Luis

-- 
Luis Anaya
papo anaya aroba hot mail punto com
"Do not use 100 words if you can say it in 10" - Yamamoto Tsunetomo

Re: [BUG] Traceback on Org-Export

[Nick Dokos]

Luis Anaya <papoanaya@hotmail.com> wrote:

> Giovanni Ridolfi <giovanni.ridolfi@yahoo.it> writes:
> 
> >> Ok... my bad... pilot error... missing (require 'org). 
> >
> > ?-)
> >
> > why is this needed? 
> 
> Good question :) Like I said, I thought that org-install would have
> loaded it. But being that I'm more interested in testing the Groff
> exporter to the latest pull from git, it became a secondary task. 
> 
> (and I'm now fixing the exporter to meet parameter changes in the 
> org-export.el API :( )
> 
> 
> > Perhaps you do run the export in batch mode?
> 
> No. Not while I'm debugging code. I should try for testing though, in
> this way I can automate the results. But for now, this is done on the
> console. 
> 
> > Giovanni /a bit puzzled
> 

It's probably a missing autoload, perhaps, maybe...

org-install does not require anything: it just provides autoloads
so that when you use a function, its autoload takes care of loading
the appropriate file(s).

Nick

PS And if I've got it wrong, Achim will surely correct me :-)

> Luis
> 
> -- 
> Luis Anaya
> papo anaya aroba hot mail punto com
> "Do not use 100 words if you can say it in 10" - Yamamoto Tsunetomo
> 

Re: [BUG] Traceback on Org-Export

[Achim Gratz]

Luis Anaya writes:
> Bastien <bzg@gnu.org> writes:
> Ok... my bad... pilot error... missing (require 'org). I would've
> thought (org-install) would've loaded it. Anyway, this solved the
> problem by adding the require on my .emacs file.

You've just hidden the real problem, which seems to be a missing
(require 'org) or declaration for that constant along that load chain
(most likely org-element from a very cursory look around the sources).

But there's plenty of work to do in that department for you as well.
:-)

--8<---------------cut here---------------start------------->8---
Compiling /home/gratz/lisp/org-mode/lisp/org-e-groff.el...

In toplevel form:
org-e-groff.el:555:7:Warning: assignment to free variable
    `registered-references'

In org-e-groff-template:
org-e-groff.el:731:13:Warning: assignment to free variable `hyphenate'
org-e-groff.el:725:13:Warning: assignment to free variable `justify-right'
org-e-groff.el:726:19:Warning: reference to free variable `justify-right'
org-e-groff.el:732:19:Warning: reference to free variable `hyphenate'

In org-e-groff-footnote-reference:
org-e-groff.el:924:32:Warning: assignment to free variable `ref-id'
org-e-groff.el:925:22:Warning: reference to free variable `ref-id'
org-e-groff.el:928:27:Warning: reference to free variable
    `registered-references'
org-e-groff.el:929:55:Warning: assignment to free variable
    `registered-references'

In org-e-groff-item:
org-e-groff.el:1142:40:Warning: value returned from (plist-get (car (cdr
    element)) property) is unused

In org-e-groff-link--inline-image:
org-e-groff.el:1301:18:Warning: assignment to free variable `placement'
org-e-groff.el:1301:37:Warning: assignment to free variable `width'
org-e-groff.el:1301:43:Warning: assignment to free variable `height'
org-e-groff.el:1302:14:Warning: assignment to free variable `disable-caption'
org-e-groff.el:1302:53:Warning: assignment to free variable `caption'
org-e-groff.el:1299:42:Warning: reference to free variable `raw-path'
org-e-groff.el:1302:31:Warning: reference to free variable `placement'
org-e-groff.el:1302:31:Warning: reference to free variable `width'
org-e-groff.el:1302:31:Warning: reference to free variable `height'
org-e-groff.el:1302:31:Warning: reference to free variable `disable-caption'
org-e-groff.el:1302:31:Warning: reference to free variable `caption'

In org-e-groff-paragraph:
org-e-groff.el:1405:9:Warning: assignment to free variable `parent'
org-e-groff.el:1406:30:Warning: reference to free variable `parent'

In org-e-groff-src-block:
org-e-groff.el:1622:19:Warning: assignment to free variable `disable-caption'
org-e-groff.el:1657:19:Warning: reference to free variable `disable-caption'

In org-e-groff-table--align-string:
org-e-groff.el:1771:22:Warning: assignment to free variable `align'
org-e-groff.el:1776:23:Warning: reference to free variable `align'

In org-e-groff-table--org-table:
org-e-groff.el:1864:11:Warning: assignment to free variable `lines'
org-e-groff.el:1854:42:Warning: assignment to free variable `attr-list'
org-e-groff.el:1873:37:Warning: assignment to free variable `title-line'
org-e-groff.el:1911:30:Warning: assignment to free variable `disable-caption'
org-e-groff.el:1891:37:Warning: assignment to free variable `long-cells'
org-e-groff.el:1858:54:Warning: reference to free variable `attr-list'
org-e-groff.el:1869:30:Warning: assignment to free variable `table-format'
org-e-groff.el:1869:7:Warning: reference to free variable `lines'
org-e-groff.el:1874:53:Warning: assignment to free variable `first-line'
org-e-groff.el:1874:53:Warning: reference to free variable `table-format'
org-e-groff.el:1875:41:Warning: reference to free variable `title-line'
org-e-groff.el:1881:53:Warning: reference to free variable `first-line'
org-e-groff.el:1893:47:Warning: assignment to free variable `long-line'
org-e-groff.el:1893:47:Warning: reference to free variable `long-cells'
org-e-groff.el:1901:57:Warning: reference to free variable `long-line'
org-e-groff.el:1909:36:Warning: reference to free variable `disable-caption'

In org-e-groff-table-cell:
org-e-groff.el:1925:36:Warning: reference to free variable `orgtbl-exp-regexp'

In org-e-groff-export-to-groff:
org-e-groff.el:2050:4:Warning: misplaced interactive spec: `(interactive)'
org-e-groff.el:2048:9:Warning: assignment to free variable
    `registered-references'
--8<---------------cut here---------------end--------------->8---


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Q+, Q and microQ:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: babel awk with table input: Code block produced no output.

[Greg Minshall]

hi.  i did a bit of poking around.  here are my findings, but i'm not
conversant enough with the semantics and interconnectivity of org-mode
to know what should be done.

to summarize: if a Babel awk script returns something that starts out
with a left paren ("("), i get "Code block returned no value".

basically, org-babel-execute:awk calls org-babel-import-elisp-from-file:
which calls org-babel-string-read [1] with each cell of the result.

org-babel-string-read calls org-babel-read.  org-babel-read
looks to see if the first character of the cell is one of {[,(,',`} and,
if it is, tries to evaluate the cell as e-lisp.  in my case (below),
what is in the cell is "(minimal)" and since there is no minimal
function in e-lisp, an error is thrown.

so, the first question is, are the semantics of parsing results such
that random e-lisp-looking code should be executed?  (this seems
dangerous, but may nevertheless be the intended semantics.)  if one did
*not* want that behavior, one can call org-babel-read with the
inhibit-lisp-eval parameter 't, which causes it to *not* try to execute
any embedded lisp-looking code [2].

the error that eval throws is caught by
org-babel-import-elisp-from-file, which then just silently returns a
nil.  the second question is, should it report an error to the user?

cheers, Greg Minshall

----

> hi.  it appears that a left or right paren in an entry in a table makes
> awk not execute.  here's an example (change ":stdin fails" to ":stdin
> works" to see it work).  cheers, Greg
> ----
> #+tblname: fails
> | proto          | no c code                   |            |
> | pscl           | c code, just fine           |            |
> | quadprog       | (minimal) c code, just fine |            |
> 
> #+tblname: works
> | proto          | no c code                   |            |
> | pscl           | c code, just fine           |            |
> | quadprog       | minimal c code, just fine   |            |
> 
> #+begin_src awk :stdin fails
>   BEGIN {
>       print "starting"
>   }
>   {
>       print $0
>   }
> #+end_src
----

[1] in spite of its documentation, i'm not sure what
org-babel-string-read does.  i thought it was removing quotation marks
from strings (but wasn't sure why).  but, running this in *scratch*
gives:
----
(org-babel-string-read "this is \"a\" test")
"a"
----
rather than "this is a test", as i had assumed.  maybe that was a bogus
test?

[2] changing org-babel-string-read to call org-babel-read with
inhibit-lisp-eval 't causes *my code* to work.  my code *also* works if
i say ":results output" or ":results scalar"; i will defensively use one
of these for my code.

Re: babel awk with table input: Code block produced no output.

[Thomas S. Dye]

Hi Greg,

:results output sounds right for awk as a report formatter.  I'm
wondering if there is a need for :results value with awk?

All the best,
Tom

Greg Minshall <minshall@umich.edu> writes:

> hi.  i did a bit of poking around.  here are my findings, but i'm not
> conversant enough with the semantics and interconnectivity of org-mode
> to know what should be done.
>
> to summarize: if a Babel awk script returns something that starts out
> with a left paren ("("), i get "Code block returned no value".
>
> basically, org-babel-execute:awk calls org-babel-import-elisp-from-file:
> which calls org-babel-string-read [1] with each cell of the result.
>
> org-babel-string-read calls org-babel-read.  org-babel-read
> looks to see if the first character of the cell is one of {[,(,',`} and,
> if it is, tries to evaluate the cell as e-lisp.  in my case (below),
> what is in the cell is "(minimal)" and since there is no minimal
> function in e-lisp, an error is thrown.
>
> so, the first question is, are the semantics of parsing results such
> that random e-lisp-looking code should be executed?  (this seems
> dangerous, but may nevertheless be the intended semantics.)  if one did
> *not* want that behavior, one can call org-babel-read with the
> inhibit-lisp-eval parameter 't, which causes it to *not* try to execute
> any embedded lisp-looking code [2].
>
> the error that eval throws is caught by
> org-babel-import-elisp-from-file, which then just silently returns a
> nil.  the second question is, should it report an error to the user?
>
> cheers, Greg Minshall
>
> ----
>
>> hi.  it appears that a left or right paren in an entry in a table makes
>> awk not execute.  here's an example (change ":stdin fails" to ":stdin
>> works" to see it work).  cheers, Greg
>> ----
>> #+tblname: fails
>> | proto          | no c code                   |            |
>> | pscl           | c code, just fine           |            |
>> | quadprog       | (minimal) c code, just fine |            |
>> 
>> #+tblname: works
>> | proto          | no c code                   |            |
>> | pscl           | c code, just fine           |            |
>> | quadprog       | minimal c code, just fine   |            |
>> 
>> #+begin_src awk :stdin fails
>>   BEGIN {
>>       print "starting"
>>   }
>>   {
>>       print $0
>>   }
>> #+end_src
> ----
>
> [1] in spite of its documentation, i'm not sure what
> org-babel-string-read does.  i thought it was removing quotation marks
> from strings (but wasn't sure why).  but, running this in *scratch*
> gives:
> ----
> (org-babel-string-read "this is \"a\" test")
> "a"
> ----
> rather than "this is a test", as i had assumed.  maybe that was a bogus
> test?
>
> [2] changing org-babel-string-read to call org-babel-read with
> inhibit-lisp-eval 't causes *my code* to work.  my code *also* works if
> i say ":results output" or ":results scalar"; i will defensively use one
> of these for my code.
>
>

-- 
Thomas S. Dye
http://www.tsdye.com

Re: babel awk with table input: Code block produced no output.

[Greg Minshall]

Tom,

> :results output sounds right for awk as a report formatter.  I'm
> wondering if there is a need for :results value with awk?

yes, i agree.  i didn't get far enough into the code to see where the
"value" result was coming from.

cheers, Greg

Change: no longer automatically evaluate embedded elisp in code block output Was: babel awk with table input: Code block produced no output.

[Eric Schulte]

> so, the first question is, are the semantics of parsing results such
> that random e-lisp-looking code should be executed?  (this seems
> dangerous, but may nevertheless be the intended semantics.)  if one did
> *not* want that behavior, one can call org-babel-read with the
> inhibit-lisp-eval parameter 't, which causes it to *not* try to execute
> any embedded lisp-looking code [2].
>

The reading of values that "look like lisp" as lisp is intended behavior
of the `org-babel-read' function at large (e.g., this is how we are able
to pass in the current file name through a header argument), however I
agree that this should probably not be the default behavior when reading
results from a code block.  It is much more likely that a person will
intentionally embed elisp in a header argument, but much less likely
that someone would intentionally write an awk script which returns elisp
(and if that is their goal they can simple pass the awk output to an
emacs-lisp code block).

For this reason I've just pushed up a change which does as you suggest
and inhibits lisp evaluation in the `org-babel-import-elisp-from-file'
function which is only used to import emacs-lisp results from code
blocks.

If anyone is relying on the current behavior please complain about this
change and we can look at other options.

>
> the error that eval throws is caught by
> org-babel-import-elisp-from-file, which then just silently returns a
> nil.  the second question is, should it report an error to the user?
>

This will now report the error with a message, but will still return nil
for the table cell's value.

Thanks,

>
> cheers, Greg Minshall
>
> ----
>
>> hi.  it appears that a left or right paren in an entry in a table makes
>> awk not execute.  here's an example (change ":stdin fails" to ":stdin
>> works" to see it work).  cheers, Greg
>> ----
>> #+tblname: fails
>> | proto          | no c code                   |            |
>> | pscl           | c code, just fine           |            |
>> | quadprog       | (minimal) c code, just fine |            |
>> 
>> #+tblname: works
>> | proto          | no c code                   |            |
>> | pscl           | c code, just fine           |            |
>> | quadprog       | minimal c code, just fine   |            |
>> 
>> #+begin_src awk :stdin fails
>>   BEGIN {
>>       print "starting"
>>   }
>>   {
>>       print $0
>>   }
>> #+end_src
> ----
>
> [1] in spite of its documentation, i'm not sure what
> org-babel-string-read does.  i thought it was removing quotation marks
> from strings (but wasn't sure why).  but, running this in *scratch*
> gives:
> ----
> (org-babel-string-read "this is \"a\" test")
> "a"
> ----
> rather than "this is a test", as i had assumed.  maybe that was a bogus
> test?
>
> [2] changing org-babel-string-read to call org-babel-read with
> inhibit-lisp-eval 't causes *my code* to work.  my code *also* works if
> i say ":results output" or ":results scalar"; i will defensively use one
> of these for my code.
>

-- 
Eric Schulte
http://cs.unm.edu/~eschulte

Re: Change: no longer automatically evaluate embedded elisp in code block output Was: babel awk with table input: Code block produced no output.

[Greg Minshall]

...
> For this reason I've just pushed up a change which does as you suggest
> and inhibits lisp evaluation in the `org-babel-import-elisp-from-file'
> function which is only used to import emacs-lisp results from code
> blocks.
...
> This will now report the error with a message, but will still return nil
> for the table cell's value.

thanks, and thanks!

Re: are super-hidden technical blocks required?

[Torsten Wagner]

Hi,

I would say this discussion is just showing how difficult it becomes
to save all  extra information provided by more and more 3rd party
tools in a smart way in  plain-text.

I can understand both arguments
* hide stuff which is not useful or needed for the user vs.
* its my data and my file, I want to know what is stored in it.

The extra property line is really verbose if the only element inside
is a ID, a timestamp, or some other "technical" stuff.
Imagine a single list of task each only a view words on a single line.
The property drawer adds three lines to this just to add an ID. Even
collapsed it still doubles the amount of lines.

However, hiding even the collapsed property line has the danger that
people might forget about it and that devs need to make sure that by
any possible way of copy and pasting (and emacs knows many ways) those
hidden lines are not left behind, that text get not added in between
and the syntax remain valid for all kind of operation.

I am unsure how to deal with this. Any kind of font face, special
character, etc. on the next line below the title does not really give
any win. There will be still a second line per tree-node.

Adding something to the title line itself is tricky too. The title
line is rather full already with all the possible features.
I still prefer the proposed selective masking of properties to hide
them. In addition maybe a flag can be set to
+ hide a property drawer line,
+ set a different font face,
+ or leave it as it is
for the special case that the property drawer only contains hidden properties.

But I frighten that this, even it might be the most flexible solution,
might also be the most complex one to implement.

So how to satisfy both views, a clutter free view and the awareness of
what is saved in your file?


Torsten

Re: are super-hidden technical blocks required?

[Bastien]

Torsten Wagner <torsten.wagner@gmail.com> writes:

> So how to satisfy both views, a clutter free view and the awareness of
> what is saved in your file?

I think we must untangle two issues here: one is about the visibility by
itself (what should be visible, invisible, how visible, how invisible?)
and the other one is about cycling (changing visibility interactively.)

For example, if a user wants to consider that a drawer doesn't make an
entry non-empty while cycling, this can be done, avoiding cluttering the
with stuff you don't want when cycling.

But that's a different problem than the one of visibility /by itself/.

I know I'm making the question even more complex, but that's hopefully
just a temporary step in this discussion.

-- 
 Bastien

Re: are super-hidden technical blocks required?

[Torsten Wagner]

[-- Attachment #1: Type: text/plain, Size: 1670 bytes --]

Hey Bastien,

On 7 August 2012 19:23, Bastien <bzg@gnu.org> wrote:
> that a drawer doesn't make an
> entry non-empty while cycling,

ohhh you challenge us... "does not ... non-empty".... is in fact the
same like "if there is only a drawer, the entry is still empty"....
right ?!

Yes, I agree that should be separated.

Maybe an idea would be a rule like....
if all properties in a drawer are marked as hidden and there is
nothing else for the particular entry (no body), do not open the entry
for the next cycling rounds.
I just tested a bit and org-mode is clever enough already to avoid any
text-insertion before the property drawer if text get added to a
collapsed entry.
Thus, this rule just might work and hide "technical" properties
completely during cycling.
Combined with a #+HIDDEN_PROP: line each and everyone can adjust
individually how much and what he likes to hide.

#+HIDDEN_PROP: * <----- all properties are hidden
would be the extreme and all property drawers will be hidden in case
they are the only element of a entry. In case other elements are
included, they collapsed drawer line will be dimmed by a different
face to indicate that only "hidden" properties are included

#+HIDDEN_PROP:    <---- this means no properties are hidden
would be the other extreme and nothing would be hidden (that
essentially would represent the present state).

I created two mock-ups. One shows the present solution and the other
shows how certain properties can be marked "hidden" and which effect
does this have on different level and combinations. Hope that helps
within this discussion. I choose a arbitrary colour scheme to make it
rather good visible.

Torsten

[-- Attachment #2: org_hidden_drawer_mockup_proposed_method.png --]
[-- Type: image/png, Size: 11419 bytes --]

[-- Attachment #3: org_hidden_drawer_mockup_orignal.png --]
[-- Type: image/png, Size: 9231 bytes --]

Re: are super-hidden technical blocks required?

[Christopher J. White]

Nice!  I like this approach.  The only slight change I would make is to 
the "All entries are unfolded one level".  If there are only "hidden" 
properties but there is other content, show the other content but not 
the PROPERTIES drawer:

    * All entries are unfolded one level
    ** Only "hidden" properties with other content
       This is more content

The ":PROPERTIES:" is not shown.

Question -- are you proposing a new step in cycling that opens all 
property drawers, or is this already available via some command or 
setting?  I've never seen a way to open everything including PROPERTIES 
via Tab or S-Tab cycling.

...cj


On 8/7/12 9:20 AM, Torsten Wagner wrote:
> Hey Bastien,
>
> On 7 August 2012 19:23, Bastien <bzg@gnu.org> wrote:
>> that a drawer doesn't make an
>> entry non-empty while cycling,
>
> ohhh you challenge us... "does not ... non-empty".... is in fact the
> same like "if there is only a drawer, the entry is still empty"....
> right ?!
>
> Yes, I agree that should be separated.
>
> Maybe an idea would be a rule like....
> if all properties in a drawer are marked as hidden and there is
> nothing else for the particular entry (no body), do not open the entry
> for the next cycling rounds.
> I just tested a bit and org-mode is clever enough already to avoid any
> text-insertion before the property drawer if text get added to a
> collapsed entry.
> Thus, this rule just might work and hide "technical" properties
> completely during cycling.
> Combined with a #+HIDDEN_PROP: line each and everyone can adjust
> individually how much and what he likes to hide.
>
> #+HIDDEN_PROP: * <----- all properties are hidden
> would be the extreme and all property drawers will be hidden in case
> they are the only element of a entry. In case other elements are
> included, they collapsed drawer line will be dimmed by a different
> face to indicate that only "hidden" properties are included
>
> #+HIDDEN_PROP:    <---- this means no properties are hidden
> would be the other extreme and nothing would be hidden (that
> essentially would represent the present state).
>
> I created two mock-ups. One shows the present solution and the other
> shows how certain properties can be marked "hidden" and which effect
> does this have on different level and combinations. Hope that helps
> within this discussion. I choose a arbitrary colour scheme to make it
> rather good visible.
>
> Torsten
>

Re: are super-hidden technical blocks required?

[Torsten Wagner]

Hey Christopher,

>    * All entries are unfolded one level
>    ** Only "hidden" properties with other content
>       This is more content
>
> The ":PROPERTIES:" is not shown.

I left it there, because some people claimed the dislike to hide
property drawers to much. A different face colour might be a good
compromise.


> Question -- are you proposing a new step in cycling that opens all property
> drawers, or is this already available via some command or setting?  I've
> never seen a way to open everything including PROPERTIES via Tab or S-Tab
> cycling.

No, I do not know how to open drawers by tab-cycling and would not
propose it. This view is just to show the most verbose way by open
each property drawer in addition manually.

The nice part on this solution, change the properties in the
HIDDEN_PROP line and you can get a complete different view only focus
on what is important for a certain task. It would rather easy to adapt
it quickly to the different requirements. I believe this goes even
further rather then only hide "technical" properties. E.g. entries
with many properties could be limited to show only what you are
working on.
See for example the following entry which I stole from Thomas S. Dyes
post about org-bibtex.el

** {Cultural Resources of Naval Air Station, Barbers
    :PROPERTIES:
    :TYPE:     book
    :CUSTOM_ID: tuggle94:_cultur_resour_naval_air_station_barber_point
    :MONTH:    December}
    :ADDRESS:  Honolulu
    :SERIES:   Prepared for Belt Collins Hawaii
    :YEAR:     1994
    :PUBLISHER: iarii
    :AUTHOR:   {H. David Tuggle and M. J. Tomonari-Tuggle}
    :END:

This has many properties. If you have hundred of this entries you
might be interested to see only AUTHOR and YEAR line for some task. I
am aware of the column view but with the proposed method people could
get a plain text view only showing what they just need. If the
HIDDEN_PROP line could include regular expressions one could even
negate a list to get a "hide all but" behavior. To get

** {Cultural Resources of Naval Air Station, Barbers
    :PROPERTIES:
    :YEAR:     1994
    :AUTHOR:   {H. David Tuggle and M. J. Tomonari-Tuggle
    ...
    :END:

You simple would have to add #+HIDDEN_PROP: ^[YEAR AUHOR]

Torsten

Re: are super-hidden technical blocks required?

[Robert Horn]

Separating out the issue of how to hide and expose the content, why not
use s-expressions for the hidden content?  Org is built on a lisp engine
and these will fit nicely into automation.  It avoids a lot of parsing
and other headaches, and an s-expression can hold any of the discussed information.

It could be an a-list that is designed for the purpose, or it could be a
list structured like the custom configuration list used in emacs config.
I think of the latter when considering the potential to steal code for
using the contents, updating the contents, and providing a user
interface when that is needed.  It also provides a model for sharing the
list among independent groups of developers with overlapping interests.
It permits automatically generated comments, advice, and warnings for
those who look at it, much like you find when looking over the tail end
of a .emacs file.

The property construction could then be a simple 
:HIDDEN-ALIST-PROPERTIES: (( stuff ........))

It would be very unreadable and unfriendly to the novice, but that is
already a characteristic of the data being discussed.

R Horn
rjhorn@alum.mit.edu

Re: are super-hidden technical blocks required?

[Bastien]

Hi Torsten,

thanks for the mock-ups -- very useful to get a quick overview.

I'm still reluctant to implement what you propose, because the 
two issues (cycling and hiding) are still too intertwined for me.

Here is how I would like the problems:

1. The fact that property drawers with only "technical" properties
   are shown when cycling.

2. The fact that property drawers with only "technical" properties
   are shown altogether.

3. The fact that "technical" properties are shown when unfolding 
   a drawer.

4. The fact that "technical" properties are shown altogether.


We started from (4), moved to (3), then stumbled on (1) and (2).

It seems to me that (4) is fundamental -- other issues depend on it.

It also occurred to me you only want this kind of feature when you 
are actually editing your org file -- not when it is processed by a 
third-part tool or by the Agenda.  Chances are that you don't want
the properties to be always invisible... and editing HIDE_PROPS just
to hide/unhide some properties is not handy.

So I implemented this simple solution, now available in the git repo:

  M-: (setq org-custom-properties '("ID")) RET
  M-x org-toggle-custom-properties-visibility RET

will hide all :ID: properties in the buffer.  The same command again
will show them.  If `org-catch-invisible-edits' is non-nil, trying to 
edit an invisible property will throw a question/error.

I hope this suits most people needs in this thread.

Thanks again for triggering this dicussion!

-- 
 Bastien