Gathering data on user preferences

Gathering data on user preferences

[Tim Cross]

Recent threads on proposed changes to default settings, provision of
configuration profiles, surveying Emacs users etc make me wonder if we
could use ELPA more effectively to gather valuable data on settings of
interest.

My thinking is that we could create an ELPA package which works
in a similar way to report emacs bug (in fact, it could probably
leverage off some of that functionality). Essentially, it would generate a
buffer containing details of variables of interest and their current
setting in a set format which could then be emailed to a data gathering
address. The set format would make it possible to process these messages
via scripts to collate the data.

The idea would be to make it easy for users to submit details about
their current settings which could be used to help inform decisions
regarding default settings.

This would not be meant to replace other data gathering approaches, such
as user surveys which can provide users with an avenue to express their
desires, preferences, issues etc. It would be just another information
source which would be easy for users to provide and easy to process in
an automated manner to provide a basic snapshot of current settings
being used.

It would even be possible to tailor the package for specific areas of
interest by releasing new/updated versions and provide concrete data on
which options people are changing from default values.

Unlike data gathering processes of other systems, as the data is first
written to a buffer, the user has full control over what information is
reported. If they don't want to reveal their setting for some reason,
they can just remove it from the buffer before sending it, plus the data
only gets gathered if and when the user is willing to submit it.

We could then setup scripts to process messages sent to the gathering
email address which maintains a simple database that collates the
responses and provides basic reporting functionality. In time, this
database can also show us how things are evolving over time and possibly
indicate default settings which may need review. If this proved
valuable, we could even consider adding functionality to politely ask
users to submit their data X weeks/months after installing or upgrading
their Emacs.  

Re: Gathering data on user preferences

[Howard Melman]

Tim Cross <theophilusx@gmail.com> writes:

> My thinking is that we could create an ELPA package which works
> in a similar way to report emacs bug (in fact, it could probably
> leverage off some of that functionality). Essentially, it would generate a
> buffer containing details of variables of interest and their current
> setting in a set format which could then be emailed to a data gathering
> address. The set format would make it possible to process these messages
> via scripts to collate the data.

I proposed this almost exactly a year ago...
https://lists.gnu.org/archive/html/emacs-devel/2020-09/msg00451.html
I got no responses.

I still think it's a good idea and I hope people respond to you.

-- 

Howard

Re: Gathering data on user preferences

[Eli Zaretskii]

> From: Howard Melman <hmelman@gmail.com>
> Date: Tue, 07 Sep 2021 00:25:29 -0400
> 
> I proposed this almost exactly a year ago...
> https://lists.gnu.org/archive/html/emacs-devel/2020-09/msg00451.html
> I got no responses.
> 
> I still think it's a good idea and I hope people respond to you.

If by "respond" you mean you expect someone to write such a command
and submit it to Emacs, then the chances of that seem to be low, given
the time that passed with no one volunteering.

How about if you do this job yourself?

(We will still need to ask people to use this command and hope they
will be willing to tell us about their habits through this mechanism.
Personally, I'm not sure this will work, given how people feel about
telemetry features.  But it won't hurt us to try.)

Thanks.

Re: Gathering data on user preferences

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1413 bytes --]

On Tue, Sep 07, 2021 at 01:22:46PM +1000, Tim Cross wrote:
> 
> Recent threads on proposed changes to default settings, provision of
> configuration profiles, surveying Emacs users etc make me wonder if we
> could use ELPA more effectively to gather valuable data on settings of
> interest.
> 
> My thinking is that we could create an ELPA package [...]
>                            [...] for users to submit details about
> their current settings which could be used to help inform decisions
> regarding default settings.

Basically a good idea. There's well-established precedent with
Debian's popcon [1]. At install you are asked whether you want
to take part in it, the default being "no". So it is active
"opt in".

The original idea was to have some basis for deciding which packages
go into the first CD (remember those?).

Some preliminary work has to be done by Someone [TM]. For example,
you wouldn't like your IMAP account credentials published in some
statistics :-)

I.e. some categorising of variables into publishable and private
(perhaps with more than just two levels? Decisions, decisions)
seems in order (reminds one of that "safe variable" thing, doesn't
it?).

So I think some discussion and some design work are needed; but it
seems a good idea. And popcon (impressively) shows that it actually
/can/ work.

Cheers

[1] https://popcon.debian.org/

 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Gathering data on user preferences

[Tim Cross]

<tomas@tuxteam.de> writes:

> [[PGP Signed Part:Undecided]]
> On Tue, Sep 07, 2021 at 01:22:46PM +1000, Tim Cross wrote:
>> 
>> Recent threads on proposed changes to default settings, provision of
>> configuration profiles, surveying Emacs users etc make me wonder if we
>> could use ELPA more effectively to gather valuable data on settings of
>> interest.
>> 
>> My thinking is that we could create an ELPA package [...]
>>                            [...] for users to submit details about
>> their current settings which could be used to help inform decisions
>> regarding default settings.
>
> Basically a good idea. There's well-established precedent with
> Debian's popcon [1]. At install you are asked whether you want
> to take part in it, the default being "no". So it is active
> "opt in".
>
> The original idea was to have some basis for deciding which packages
> go into the first CD (remember those?).
>
> Some preliminary work has to be done by Someone [TM]. For example,
> you wouldn't like your IMAP account credentials published in some
> statistics :-)
>
> I.e. some categorising of variables into publishable and private
> (perhaps with more than just two levels? Decisions, decisions)
> seems in order (reminds one of that "safe variable" thing, doesn't
> it?).
>
> So I think some discussion and some design work are needed; but it
> seems a good idea. And popcon (impressively) shows that it actually
> /can/ work.
>

I was thinking of only getting data on 'non-sensitive' variables and
only aggregate data would be available (i.e. make sure it is not
possible to say person x submitted values a, b and c). It could possibly
even be setup as a very simple web service which just accepts a JSON
object in the body of a POST request. That would avoid anyone having to
provide an email address. The possible downside is that this would be
open to 'gaming' the system, but I'm not sure we need to worry about
that too much - there is little reward in doing so.

The system would definitely be 'opt in' and the user would be shown
exactly what is going to be sent (and provided with the ability to
remove anything they are not comfortable with). 

Re: Gathering data on user preferences

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1372 bytes --]

On Tue, Sep 07, 2021 at 05:54:09PM +1000, Tim Cross wrote:

[...]

> I was thinking of only getting data on 'non-sensitive' variables

Of course, I didn't assume otherwise. What I was aiming at is that
classifying /which/ variables can be considered 'non-sensitive' is
already some non-negligible work, that's all (and leaving the
decision to each single user doesn't seem fair).

>                                                                and
> only aggregate data would be available (i.e. make sure it is not
> possible to say person x submitted values a, b and c).

Of course. Correlations between values might be of interest.

> [...] The possible downside is that this would be
> open to 'gaming' the system, but I'm not sure we need to worry about
> that too much - there is little reward in doing so.

This is why I pointed to Debian's popcon: they just live with that,
and that doesn't seem to be a problem.

> The system would definitely be 'opt in' and the user would be shown
> exactly what is going to be sent (and provided with the ability to
> remove anything they are not comfortable with). 

Also, see popcon.

As a small nit: why JSON and not S-expressions? We're lispies, after
all ;-)

But the latter are just details. I guess who gets to implement gets
to choose. That might be an incentive ;-P

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Gathering data on user preferences

[Tim Cross]

tomas@tuxteam.de writes:

> As a small nit: why JSON and not S-expressions? We're lispies, after
> all ;-)
>

Only because from a web perspective, middleware to process body contents
consisting of JSON are plentiful, while libraries for s-expressions are
not.  

Re: Gathering data on user preferences

[Arthur Miller]

Tim Cross <theophilusx@gmail.com> writes:

> tomas@tuxteam.de writes:
>
>> As a small nit: why JSON and not S-expressions? We're lispies, after
>> all ;-)
>>
>
> Only because from a web perspective, middleware to process body contents
> consisting of JSON are plentiful, while libraries for s-expressions are
> not.  

But you would be processing them in Emacs which is very good at processing both
text and s-expressions? :)

Re: Gathering data on user preferences

[Howard Melman]

> On Sep 7, 2021, at 2:28 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Howard Melman <hmelman@gmail.com>
>> Date: Tue, 07 Sep 2021 00:25:29 -0400
>> 
>> I proposed this almost exactly a year ago...
>> https://lists.gnu.org/archive/html/emacs-devel/2020-09/msg00451.html
>> I got no responses.
>> 
>> I still think it's a good idea and I hope people respond to you.
> 
> If by "respond" you mean you expect someone to write such a command
> and submit it to Emacs, then the chances of that seem to be low, given
> the time that passed with no one volunteering.
> 
> How about if you do this job yourself?
> 
> (We will still need to ask people to use this command and hope they
> will be willing to tell us about their habits through this mechanism.
> Personally, I'm not sure this will work, given how people feel about
> telemetry features.  But it won't hurt us to try.)

By "respond" I meant no one sent an email in support of or opposed to the idea.

Howard

Re: Gathering data on user preferences

[Tim Cross]

Arthur Miller <arthur.miller@live.com> writes:

> Tim Cross <theophilusx@gmail.com> writes:
>
>> tomas@tuxteam.de writes:
>>
>>> As a small nit: why JSON and not S-expressions? We're lispies, after
>>> all ;-)
>>>
>>
>> Only because from a web perspective, middleware to process body contents
>> consisting of JSON are plentiful, while libraries for s-expressions are
>> not.  
>
> But you would be processing them in Emacs which is very good at processing both
> text and s-expressions? :)

Why would you be processing them in Emacs? If you had a web service, the
last thing you would have is Emacs sitting behind the web server. You
would use something better suited which would process the data from the
POST request which then stores the data in some format. You might have
Emacs be able to query that data to get reports. 

Re: Gathering data on user preferences

[Arthur Miller]

Tim Cross <theophilusx@gmail.com> writes:

> Arthur Miller <arthur.miller@live.com> writes:
>
>> Tim Cross <theophilusx@gmail.com> writes:
>>
>>> tomas@tuxteam.de writes:
>>>
>>>> As a small nit: why JSON and not S-expressions? We're lispies, after
>>>> all ;-)
>>>>
>>>
>>> Only because from a web perspective, middleware to process body contents
>>> consisting of JSON are plentiful, while libraries for s-expressions are
>>> not.  
>>
>> But you would be processing them in Emacs which is very good at processing both
>> text and s-expressions? :)
>
> Why would you be processing them in Emacs? If you had a web service, the
> last thing you would have is Emacs sitting behind the web server. You
> would use something better suited which would process the data from the
> POST request which then stores the data in some format. You might have
> Emacs be able to query that data to get reports. 

Just save a byte stream, and then process in Emacs?

Re: Gathering data on user preferences

[Tim Cross]

Arthur Miller <arthur.miller@live.com> writes:

> Tim Cross <theophilusx@gmail.com> writes:
>
>> Arthur Miller <arthur.miller@live.com> writes:
>>
>>> Tim Cross <theophilusx@gmail.com> writes:
>>>
>>>> tomas@tuxteam.de writes:
>>>>
>>>>> As a small nit: why JSON and not S-expressions? We're lispies, after
>>>>> all ;-)
>>>>>
>>>>
>>>> Only because from a web perspective, middleware to process body contents
>>>> consisting of JSON are plentiful, while libraries for s-expressions are
>>>> not.  
>>>
>>> But you would be processing them in Emacs which is very good at processing both
>>> text and s-expressions? :)
>>
>> Why would you be processing them in Emacs? If you had a web service, the
>> last thing you would have is Emacs sitting behind the web server. You
>> would use something better suited which would process the data from the
>> POST request which then stores the data in some format. You might have
>> Emacs be able to query that data to get reports. 
>
> Just save a byte stream, and then process in Emacs?

But why? There are loads of libraries which can do all the necessary
processing and storage of results that would be almost trivial to
assemble and the data would be processed as soon as it arrives - all
done without need for human intervention. Putting emacs into the mix
would only complicate the processes and make it a more manual process. A
boring task best left to machines I think. All we are really interested
in is the aggregate data, not each submission. 

Re: Gathering data on user preferences

[Arthur Miller]

Tim Cross <theophilusx@gmail.com> writes:

> Arthur Miller <arthur.miller@live.com> writes:
>
>> Tim Cross <theophilusx@gmail.com> writes:
>>
>>> Arthur Miller <arthur.miller@live.com> writes:
>>>
>>>> Tim Cross <theophilusx@gmail.com> writes:
>>>>
>>>>> tomas@tuxteam.de writes:
>>>>>
>>>>>> As a small nit: why JSON and not S-expressions? We're lispies, after
>>>>>> all ;-)
>>>>>>
>>>>>
>>>>> Only because from a web perspective, middleware to process body contents
>>>>> consisting of JSON are plentiful, while libraries for s-expressions are
>>>>> not.  
>>>>
>>>> But you would be processing them in Emacs which is very good at processing both
>>>> text and s-expressions? :)
>>>
>>> Why would you be processing them in Emacs? If you had a web service, the
>>> last thing you would have is Emacs sitting behind the web server. You
>>> would use something better suited which would process the data from the
>>> POST request which then stores the data in some format. You might have
>>> Emacs be able to query that data to get reports. 
>>
>> Just save a byte stream, and then process in Emacs?
>
> But why? There are loads of libraries which can do all the necessary
> processing and storage of results that would be almost trivial to
> assemble and the data would be processed as soon as it arrives - all
> done without need for human intervention.

Who said human intervention? :-)

I see emacs as a good tool for shell scripting and text processing. In general I
am surprised that lisp(s) are second grade citizens in the land of web scraping,
text processing etc, python being No 1 atm and js no2.

Thomas asked why not s-exps? And I also wonder. If we sent s-exps over the wire,
maybe it would be less processing involved? Write incomming s-exp to emacs
server socket? Is it too wild to think of? :)

> boring task best left to machines I think.

Of course, my thoughts as well, anytime!

Re: Gathering data on user preferences

[Tim Cross]

Arthur Miller <arthur.miller@live.com> writes:

>
> Who said human intervention? :-)
>
> I see emacs as a good tool for shell scripting and text processing. In general I
> am surprised that lisp(s) are second grade citizens in the land of web scraping,
> text processing etc, python being No 1 atm and js no2.
>
> Thomas asked why not s-exps? And I also wonder. If we sent s-exps over the wire,
> maybe it would be less processing involved? Write incomming s-exp to emacs
> server socket? Is it too wild to think of? :)
>

Yes it is. Your creating lots of work to create a solution which would
perform badly and be overly fragile and resource expensive. Emacs is
simply the wrong tool for this task.

We could use s-exp, but I don't see any benefit as I would not be using
Emacs to process the data. Its like asking "Why don't we use Emacs as a
bug tracker?" - the answer is you could, but there are better bug
trackers out there.

I also think your not considering the concurrent nature of web servers.
Requests are not serial and multiple requests can come in at the same
time. If you were going to write the data to Emacs via a socket, you
need to handle multiple socket connections at the same time and respond
to the web server in a timely manner to prevent timeout errors etc. 

We would have to disagree here. I don't think Emacs is a good general
purpose tool for shell scripting and text processing in this scenario at
all. It is completely the wrong tool for the task.

- It is huge compared to other tools. Why would you fire up Emacs on a
  server just to process a small chunk of data from a POST request?

- Emacs is not multi-threaded, so you would either have to fire up an
  Emacs instance for every connection (terribly slow) or you wold have
  to write some handler code to write the data to intermediate files and
  then have some Emacs process fire up at some point to process the
  received data. You would also need to implement some form of
  locking mechanism to ensure Emacs doesn't try to read data which
  hasn't completed writing yet and you still have to write some form of
  handler in something just to generate the intermediate data files. The
  solution would end up more complex and prone to bugs than necessary.

- Compared to other scripting solutions, Emacs is extremely slow. This
  is partially because it has an internal structure oriented towards
  being an editor, not a general purpose scripting environment. 

Just because you can do almost anything with Emacs doesn't mean you
should. If you really want to use a lisp language, there are far better
choices for this scenario (like one of the many lisp based shells like
lsh, schsh, rep, guile etc). 

Re: Gathering data on user preferences

[Philip Kaludercic]

<tomas@tuxteam.de> writes:

> On Tue, Sep 07, 2021 at 01:22:46PM +1000, Tim Cross wrote:
>> 
>> Recent threads on proposed changes to default settings, provision of
>> configuration profiles, surveying Emacs users etc make me wonder if we
>> could use ELPA more effectively to gather valuable data on settings of
>> interest.
>> 
>> My thinking is that we could create an ELPA package [...]
>>                            [...] for users to submit details about
>> their current settings which could be used to help inform decisions
>> regarding default settings.
>
> Basically a good idea. There's well-established precedent with
> Debian's popcon [1]. At install you are asked whether you want
> to take part in it, the default being "no". So it is active
> "opt in".

[...]

> I.e. some categorising of variables into publishable and private
> (perhaps with more than just two levels? Decisions, decisions)
> seems in order (reminds one of that "safe variable" thing, doesn't
> it?).

At this point, couldn't the information attached to bug reports be
used. I remember someone making the suggestion in last years
discussion. The advantage is that there is already a lot of data
available right now, and that this data contains the evolution of trends
over a number of years.

I might also imagine that if people knew bug reports contribute to a
better representation of the entire user-base.

-- 
	Philip Kaludercic

Re: Gathering data on user preferences

[Tim Cross]

Philip Kaludercic <philipk@posteo.net> writes:

> <tomas@tuxteam.de> writes:
>
>> On Tue, Sep 07, 2021 at 01:22:46PM +1000, Tim Cross wrote:
>>> 
>>> Recent threads on proposed changes to default settings, provision of
>>> configuration profiles, surveying Emacs users etc make me wonder if we
>>> could use ELPA more effectively to gather valuable data on settings of
>>> interest.
>>> 
>>> My thinking is that we could create an ELPA package [...]
>>>                            [...] for users to submit details about
>>> their current settings which could be used to help inform decisions
>>> regarding default settings.
>>
>> Basically a good idea. There's well-established precedent with
>> Debian's popcon [1]. At install you are asked whether you want
>> to take part in it, the default being "no". So it is active
>> "opt in".
>
> [...]
>
>> I.e. some categorising of variables into publishable and private
>> (perhaps with more than just two levels? Decisions, decisions)
>> seems in order (reminds one of that "safe variable" thing, doesn't
>> it?).
>
> At this point, couldn't the information attached to bug reports be
> used. I remember someone making the suggestion in last years
> discussion. The advantage is that there is already a lot of data
> available right now, and that this data contains the evolution of trends
> over a number of years.
>
> I might also imagine that if people knew bug reports contribute to a
> better representation of the entire user-base.

I would imagine the package I'm proposing would likely leverage off some
of the code which generates the data attached to bug reports. I also
think it would be good to be able to mine the data which exists in bug
reports, but have no idea how easily that could be done.

The reasons for considering a new package are

- Specific to purpose. Bug reports also include data we are not
  interested in for this specific exercise

- Formatted data which would be structured to make collation and
  processing easy

- Not yet sure if the data included in bug reports is the same as the
  data we want for analysis of user preferences and settings. I expect
  there is overlap, but there is likely data in bug reports we are not
  interested in and some which is missing that we do want.

- People tend to only report bugs when they have a problem. We would
  want people to report this information (possibly after a 'call' to do
  so) even when everything is working fine. We wouldn't want people just
  logging bug reports solely to report configurations as I suspect this
  would just clutter the bug tracker with non bug data. 

- I think it wold be good if the 'package' allowed easy configuration of
  what is requested. We could then update the package with specific
  'profiles' to report on specific areas etc.

I do think there is probably some really valuable data currently sitting
in the Emacs bug tracker and it would be an interesting project to try
and mine that repository to see what we could get out of it. However, I
have no idea what API the system provides or how easy this would be to
do (without significant human intervention). 

Re: Gathering data on user preferences

[Daniel Fleischer]

Tim Cross [2021-09-08 Wed 18:14] wrote:

> I would imagine the package I'm proposing would likely leverage off some
> of the code which generates the data attached to bug reports. I also
> think it would be good to be able to mine the data which exists in bug
> reports, but have no idea how easily that could be done.
>
> I do think there is probably some really valuable data currently sitting
> in the Emacs bug tracker and it would be an interesting project to try
> and mine that repository to see what we could get out of it. However, I
> have no idea what API the system provides or how easy this would be to
> do (without significant human intervention). 

I can download the mbox files for each month, going back a few years and
then aggregate them according to month or even week. I'll do it in
python and can share the data and code if I get something interesting.

-- 

Daniel Fleischer

Re: Gathering data on user preferences

[Philip Kaludercic]

Tim Cross <theophilusx@gmail.com> writes:

> I would imagine the package I'm proposing would likely leverage off some
> of the code which generates the data attached to bug reports. I also
> think it would be good to be able to mine the data which exists in bug
> reports, but have no idea how easily that could be done.

You are probably right that a separate package would provide better and
more accurate data, I just wonder what the incentive is for someone who
uses Emacs, has opinions but isn't part of any "community". Unless this
is automated, which become annoying, it seems like this is a quality vs
quantity problem -- and if that is the case, it might be more helpful to
improve the quality of the quantity (more detail in bug reports) than
improve the quantity of the quality.

-- 
	Philip Kaludercic

Re: Gathering data on user preferences

[Tim Cross]

Philip Kaludercic <philipk@posteo.net> writes:

> Tim Cross <theophilusx@gmail.com> writes:
>
>> I would imagine the package I'm proposing would likely leverage off some
>> of the code which generates the data attached to bug reports. I also
>> think it would be good to be able to mine the data which exists in bug
>> reports, but have no idea how easily that could be done.
>
> You are probably right that a separate package would provide better and
> more accurate data, I just wonder what the incentive is for someone who
> uses Emacs, has opinions but isn't part of any "community". Unless this
> is automated, which become annoying, it seems like this is a quality vs
> quantity problem -- and if that is the case, it might be more helpful to
> improve the quality of the quantity (more detail in bug reports) than
> improve the quantity of the quality.

I suspect multiple channels will be required as it is unlikely one
engagement model will work for all. Bug reports, user survey and a
telemetry package could all be part of the picture. 

Re: Gathering data on user preferences

[Tim Cross]

Daniel Fleischer <danflscr@gmail.com> writes:

> Tim Cross [2021-09-08 Wed 18:14] wrote:
>
>> I would imagine the package I'm proposing would likely leverage off some
>> of the code which generates the data attached to bug reports. I also
>> think it would be good to be able to mine the data which exists in bug
>> reports, but have no idea how easily that could be done.
>>
>> I do think there is probably some really valuable data currently sitting
>> in the Emacs bug tracker and it would be an interesting project to try
>> and mine that repository to see what we could get out of it. However, I
>> have no idea what API the system provides or how easy this would be to
>> do (without significant human intervention). 
>
> I can download the mbox files for each month, going back a few years and
> then aggregate them according to month or even week. I'll do it in
> python and can share the data and code if I get something interesting.

It will be interesting to see what you get and how hard it was to
extract. I suspect the bug reports could be  a gold mine of valuable
data, but it may be difficult to process.

Re: Gathering data on user preferences

[Philip Kaludercic]

Howard Melman <hmelman@gmail.com> writes:

>> On Sep 7, 2021, at 2:28 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>> 
>>> From: Howard Melman <hmelman@gmail.com>
>>>
>>> I proposed this almost exactly a year ago...
>>> https://lists.gnu.org/archive/html/emacs-devel/2020-09/msg00451.html
>>> I got no responses.
>>
>> How about if you do this job yourself?
>
> By "respond" I meant no one sent an email in support of or opposed to the idea.

In my experience, the best thing you can do in that case is prepare a
prototype, so that something substantial can be discussed. No response
usually just means nobody has anything worthwhile to add.

-- 
	Philip Kaludercic