Just sharing another orgmode use: usability tests

Just sharing another orgmode use: usability tests

[Eduardo Mercovich]

Dear all.

We all know that emacs+orgmode are not tools, but meta-tools. This 
is why it can be so daunting at first(*1) and totally awesome in 
the long run. ;)

This mail is just to share with you another great application of 
orgmode: usability tests.

For those of you who never experienced it yet, a usability test is 
basically a critical observation of certain people (Users) doing 
certain things (tasks) in a particular setting (context). And 
critical means that there are some defined things that you look 
specifically to, like task success of fail (what means "success" 
is previously defined), times, User observations, navigation, etc.

During a test there is
+ a User (usually one, but in certain specific cases can be more),
+ a Facilitator (who cares for the User and ensures that the 
experimental protocol is followed as it should) and
+ Observers (who takes notes of these defined criteria and many 
other observations). 

Test notes are usually comprised of task start and stop time, 
results (succes, failure or so-so), User quotes, navigation steps, 
intermediate steps results, Observer's hypothesis to be reviewed, 
etc. That is, the prescribed issues, plus any other notable fact. 

In some specific tools, like the industry standard setting Morae 
(https://www.techsmith.com/morae.html, US$ 1,995) you can take 
those notes with relative easy, because it automates the time 
counting and you can set codes (usually one letter) to specify the 
type of observation. For example, q for 'user quote' or v for 
'video' (something interesting to review after the test) or n for 
navigation, and so on. 

Handwrited notes are great because of their fluidity, except that 
is hard to count times (you can look at an external chronometer, 
of course, but it takes your attention away from the User) and in 
my case, my handwriting is fast but so bad that I can hardly read 
it after. ;)

And here comes orgmode. 

In my last test I finally tried to use orgmode for this (why not, 
I use it for almost any other important task) and the results 
where impressive, even in the 1st try, even without any 
customization or heavy data metabolism after. 

What do we have out of the box: 

+ integration of script and notes (I was the Facilitator, but took 
notes also) so I can read the user script and take notes 
integrated with each task and context. Each task is a heading, and 
notes are directly inside it with...

+ ... task results easily annotated, using the task state (TODO in 
red for failure, ENCURSO in brown for so-so and DONE in green for 
success). 

+ time stamping, not only start and end, but anything in between 
too. 

+ abbreviations allowed me to enter notes faster (chording may 
help to do it faster even, but I still don't use it). 

+ regular expression highlighting makes incredibly sweet to review 
the notes. For example, I use !! to mark something important and 
!!! for critical, so those go for line highlighting with yellow 
and red background respectively, and user quotes are between ", so 
those get separated from the normal text too. 


What I believe could be easely done with a bit of work:

+ automatic task clocking, including automated time stamping of 
every observation. 

+ automatic recollection of User quotes, like "let's see everyhing 
that Users said in task X". 

+ observation tagging (there are repeating patterns of use or 
interaction problems or sources that you could cath on the fly, so 
any observation could be related to those common issues). 

+ automatic filling of User/task-results-and-time matrix (the most 
fundamental metric of usability tests). 


In brief: orgmode is great for usability studies. :D

While I don't know if any of you will find these observations 
useful, I'm sure you all enjoy knowing that this incredible 
meta-tool many of you created and still make it grow has found 
another great use. 

Have a great orgmoding time... :)



*1: Yes, I know, not for us on this list but yes for many people 
 that's used to more specific, maket-centered tools. ;)
-- 
eduardo mercovich

 Donde se cruzan tus talentos 
 con las necesidades del mundo, 
 ahí está tu vocación. 
 (Anónimo)

Re: Just sharing another orgmode use: usability tests

[Eric S Fraga]

[-- Attachment #1: Type: text/plain, Size: 735 bytes --]

On Thursday, 15 Jun 2017 at 20:08, Eduardo Mercovich wrote:
> This mail is just to share with you another great application of 
> orgmode: usability tests.

Thanks.  Very interesting.


[...]

> What I believe could be easely done with a bit of work:
>
> + automatic task clocking, including automated time stamping of 
> every observation. 

This should be trivial.  Use a capture template for observations and you
can clock these automatically (I think) and/or time stamp as
required.  In fact, many of your other suggested needs could be met with
capture templates.

But, being emacs, I am sure there will be plenty of alternatives!

Thanks again,
eric

-- 
: Eric S Fraga (0xFFFCF67D), Emacs 26.0.50, Org release_9.0.7-531-g530113

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 194 bytes --]

Re: Just sharing another orgmode use: usability tests

[Eduardo Mercovich]

Hi Eric.

>> This mail is just to share with you another great application 
>> of orgmode: usability tests.

[...]

>> What I believe could be easely done with a bit of work:

>> + automatic task clocking, including automated time stamping of 
>> every observation.

> This should be trivial.  Use a capture template for observations 
> and you
> can clock these automatically (I think) and/or time stamp as
> required.  In fact, many of your other suggested needs could be 
> met with
> capture templates.

The standard clocking task mechanism will do it fine.
However, since there are many multiple small observations along 
one task, maybe a capture template is a bit overdo... 

> But, being emacs, I am sure there will be plenty of 
> alternatives!

Yes, I do totally agree... :)

I will continue slowly to experiment with this. Clocking task 
in-out is one of the 1st issues to be tried along with timers, and 
also adding the seconds (which in a task could be important).

Best... 

-- 
eduardo mercovich

 Donde se cruzan tus talentos 
 con las necesidades del mundo, 
 ahí está tu vocación. 
 (Anónimo)

Re: Just sharing another orgmode use: usability tests

[Eric S Fraga]

[-- Attachment #1: Type: text/plain, Size: 660 bytes --]

On Friday, 16 Jun 2017 at 14:19, Eduardo Mercovich wrote:

[...]

> The standard clocking task mechanism will do it fine.

Great.

> However, since there are many multiple small observations along 
> one task, maybe a capture template is a bit overdo... 

I find that capturing is a lightweight activity.  It all depends on what
you expect to have to do.  For instance, I log some activities by simply
typing "C-c c l COUPLE OF KEYWORDS C-c C-c" and I'm done!  The entry is
time stamped automatically and it simply allows me to note something
without really intruding into a workflow.

But, YMMV of course :-)

-- 
Eric S Fraga, ericsfraga@<most social media>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 194 bytes --]

Re: Just sharing another orgmode use: usability tests

[Eduardo Mercovich]

Hi Eric.

>> The standard clocking task mechanism will do it fine.
>
> Great.
>
>> However, since there are many multiple small observations along 
>> one task, maybe a capture template is a bit overdo... 

> I find that capturing is a lightweight activity.  It all depends 
> on what
> you expect to have to do.  For instance, I log some activities 
> by simply
> typing "C-c c l COUPLE OF KEYWORDS C-c C-c" and I'm done!  The 
> entry is
> time stamped automatically and it simply allows me to note 
> something
> without really intruding into a workflow.

It could be... maybe I feel it like too much it's because some obs 
are really quite short (like 1 name/place) and I took them as 
lines, so headings or list items are more than enough.
I'll have to try it. :)

OTOH, I just found the perfect timestamping method: org-timer. 
:)))

Best...


-- 
eduardo mercovich

 Donde se cruzan tus talentos 
 con las necesidades del mundo, 
 ahí está tu vocación. 
 (Anónimo)

Re: Just sharing another orgmode use: usability tests

[Karl Voit]

* Eduardo Mercovich <eduardo@mercovich.net> wrote:
>
> + automatic task clocking, including automated time stamping of 
> every observation. 

You seem to use headings for any events.

If list items does the trick as well, you can take a look at timers
for doing the clocking thing: http://orgmode.org/manual/Timers.html

> + observation tagging (there are repeating patterns of use or 
> interaction problems or sources that you could cath on the fly, so 
> any observation could be related to those common issues). 

You already mentioned templates. I'd go with yasnippets for that as
well.

> + automatic filling of User/task-results-and-time matrix (the most 
> fundamental metric of usability tests). 

Wow, this would be great.

-- 
get mail|git|SVN|photos|postings|SMS|phonecalls|RSS|CSV|XML into Org-mode:
       > get Memacs from https://github.com/novoid/Memacs <
Personal Information Management > http://Karl-Voit.at/tags/pim/
Emacs-related > http://Karl-Voit.at/tags/emacs/

Re: Just sharing another orgmode use: usability tests

[Eduardo Mercovich]

Hi Karl.

>> + automatic task clocking, including automated time stamping of 
>> every observation.

> You seem to use headings for any events.
> If list items does the trick as well, you can take a look at 
> timers
> for doing the clocking thing: 
> http://orgmode.org/manual/Timers.html

Totally agree, that's exactly what I use now. :)
Lists are used inside each task, at 1 item per (short) 
observation. 

>> + observation tagging (there are repeating patterns of use or 
>> interaction problems or sources that you could catch on the 
>> fly, so any observation could be related to those common 
>> issues).

> You already mentioned templates. I'd go with yasnippets for that 
> as well.

Sorry I wasn't clear, I meant more the use of tags related to 
common observed patterns mark and select after (with sparse tree) 
those in which that pattern or issue is relevant. Great to group 
things (tasks, observations, etc.) related to the same issue. 

>> + automatic filling of User/task-results-and-time matrix (the 
>> most fundamental metric of usability tests).

> Wow, this would be great.

Yes, but for that I have to learn Lisp. ;)
I'm starting with "An Introduction To Programming in Emacs Lisp" 
by Robert J. Chassell. Just minutes a week sadly, but this is what 
I can do now. :)

In the document structure we have: 
* Users,
** Pre-task interview
** tasks (with TODO states as indicators of success|failure|so-so; 
   maybe we can use properties).
 - each one with one or more timer entry
 - so total task time is the last item timer value,
** Post-task interview

To make the User/task-results-and-time matrix we'll have to walk 
the hierarchy and get for each User the sequence of tasks, and of 
each, the success (or not), time and tag (issue) related and put 
that in the results matrix. 

Since I couldn't find other references about orgmode used for 
usability, I am organizing an informal workshop to share this 
experience next week with 10 people or so from the usability 
community in Buenos Aires.
If someone is interested, we can arrange another in English after 
this 1st one. ;)

Best regards... 


-- 
eduardo mercovich

 Donde se cruzan tus talentos 
 con las necesidades del mundo, 
 ahí está tu vocación. 
 (Anónimo)