[ANN] org-ql 0.4 released

[ANN] org-ql 0.4 released

[Adam Porter]

Hi friends,

FYI, I've released org-ql 0.4.  It includes many improvements since 0.3.

https://github.com/alphapapa/org-ql

Thanks,
Adam

Re: [ANN] org-ql 0.4 released

[Michael Alan Dorman]

Adam Porter <adam@alphapapa.net> writes:

> Hi friends,
>
> FYI, I've released org-ql 0.4.  It includes many improvements since 0.3.
>
> https://github.com/alphapapa/org-ql

It would be nice if you could do a stable release of org-super-agenda so
that it could be installed from melpa-stable...

Mike.

Re: [ANN] org-ql 0.4 released

[Adam Porter]

Michael Alan Dorman <mdorman@jaunder.io> writes:

>> Hi friends,
>>
>> FYI, I've released org-ql 0.4.  It includes many improvements since 0.3.
>>
>> https://github.com/alphapapa/org-ql
>
> It would be nice if you could do a stable release of org-super-agenda so
> that it could be installed from melpa-stable...

Comments like yours lead me to the conclusion that MELPA Stable needs to
be abolished.  I have been a proponent of the idea of MELPA Stable, so I
don't say that lightly.

I'll assume that you don't know what the technical issues are and offer
an explanation.  Briefly:

+ MELPA Stable is nothing like what one might assume it's intended to be
  like, e.g. Debian Stable or Debian Testing.  It provides none of the
  benefits that Debian Stable and Testing provide.

+ MELPA Stable is, just like "regular" MELPA, a "rolling" collection of
  packages developed without any coordination between maintainers.

+ The only difference is that MELPA Stable contains whatever versions of
  packages that their maintainers have decided to tag with a version
  number, rather than the latest commit to the master branch.  These
  versions are not necessarily better, more stable, more reliable, or
  more trustworthy than the untagged versions which appear in "regular"
  MELPA.

+ Due to the lack of coordination between dependencies and their APIs,
  version conflicts and breakage are a regular occurrence.  For example,
  if package A depends on package B, and package B makes an API change
  and tags a new MELPA Stable release, users of package A's MELPA Stable
  version will see package A cease to work properly until package A, not
  only commits a fix, but tags a new MELPA Stable version containing the
  fix.  Since packages A and B do not share the same development
  schedule, it is likely that their tagged-version release schedules
  will not synchronize well.

  If you are familiar with Debian, imagine if any upstream changes were
  automatically pushed to Testing despite any freeze that might be in
  place.  It would be virtually impossible to complete a freeze and make
  a new stable release, and Testing and Stable would cease to be useful,
  leaving only Unstable as a usable target.  This is the situation
  between "regular" MELPA and MELPA Stable.

For my packages, I tag stable versions for a few reasons:

+ To help users track changes in the changelog.

+ To help me separate new, possibly bug-introducing changes from
  working, debugged code.

+ To help packagers in systems like Debian and Guix, who package stable
  versions of some Elisp packages--and who, in so doing, take
  responsibility for breakage.

Now, I sympathize with not wanting to be vulnerable to potential
breakage caused by the uncoordinated release of changes to packages on
"regular" MELPA.  That is a real problem.  But the solution is not to
use MELPA Stable.  The solution is to take charge of what packages you
upgrade and when, rather than being at the mercy of whatever commits
happen to be in MELPA at the moment.

For myself, I commit the ~/.emacs.d/elpa directory to Git with the rest
of my config, and I upgrade packages intentionally.  If breakage
happens, I can easily revert and deal with it later.  Other users use
alternative package managers, like Borg or Straight or Quelpa, which
pull changes directly from Git repos and allow pinning to commits, tags,
etc.

So, for yourself, I can only recommend that you abandon MELPA Stable and
install packages by other means.  If you don't have the time or
inclination to redo your whole config like that, then just use something
like Quelpa to install the current version of org-super-agenda directly.
It's a couple lines of use-package in your config, and you can upgrade
it manually from then on, e.g. with
<https://github.com/alphapapa/unpackaged.el#upgrade-a-quelpa-use-package-forms-package>.
As always, your Emacs config is up to you.

Now, I'm off to to the discussions on MELPA's tracker to add my vote to
abolish MELPA Stable, or to at least allow packages to opt-out of it.

Re: [ANN] org-ql 0.4 released

[Tim Cross]

I don't disagree with most of what you write. I do think a big part of
the problem is inconsistency and poor practices by some maintainers
which is at the hart of the issue. Sometimes it seems that package
maintainers put too much emphasis on getting the latest package out and
forget about the stability and dependencies that users have.

All of this leaves me wondering why the ELPA system doens't adopt a
semantic versioning scheme and configure the repositories to
keep/maintain previous versions. Emacs was very late to the whole
'package' concept and it is a little disappointing that more time wasn't
invested in understanding the challenges and solutions implemented by
other systems like deb, rpm, CPAN, Java, ruby, python etc. There is
little unique to the Emacs environment that other environments haven't
had to resolve one way or the other.  

What would be good is an approach based on sematic versioning similar to
npmjs.com, maven, clojars and other packaging systems. 

Semantic versioning won't solve all the issues. The lack of namespaces
or ability of libraries to depend on specific versions of a library are
both significant challenges. However, a semantic version does provide
more information than something like version 20200123, which just tells
me the release date. At least with complaint semantic versions I can
have a better idea as to whether the changes are just a bug fix, an
enhancement or major API changes that may break existing dependencies. 

Tim

Adam Porter <adam@alphapapa.net> writes:

> Michael Alan Dorman <mdorman@jaunder.io> writes:
>
>>> Hi friends,
>>>
>>> FYI, I've released org-ql 0.4.  It includes many improvements since 0.3.
>>>
>>> https://github.com/alphapapa/org-ql
>>
>> It would be nice if you could do a stable release of org-super-agenda so
>> that it could be installed from melpa-stable...
>
> Comments like yours lead me to the conclusion that MELPA Stable needs to
> be abolished.  I have been a proponent of the idea of MELPA Stable, so I
> don't say that lightly.
>
> I'll assume that you don't know what the technical issues are and offer
> an explanation.  Briefly:
>
> + MELPA Stable is nothing like what one might assume it's intended to be
>   like, e.g. Debian Stable or Debian Testing.  It provides none of the
>   benefits that Debian Stable and Testing provide.
>
> + MELPA Stable is, just like "regular" MELPA, a "rolling" collection of
>   packages developed without any coordination between maintainers.
>
> + The only difference is that MELPA Stable contains whatever versions of
>   packages that their maintainers have decided to tag with a version
>   number, rather than the latest commit to the master branch.  These
>   versions are not necessarily better, more stable, more reliable, or
>   more trustworthy than the untagged versions which appear in "regular"
>   MELPA.
>
> + Due to the lack of coordination between dependencies and their APIs,
>   version conflicts and breakage are a regular occurrence.  For example,
>   if package A depends on package B, and package B makes an API change
>   and tags a new MELPA Stable release, users of package A's MELPA Stable
>   version will see package A cease to work properly until package A, not
>   only commits a fix, but tags a new MELPA Stable version containing the
>   fix.  Since packages A and B do not share the same development
>   schedule, it is likely that their tagged-version release schedules
>   will not synchronize well.
>
>   If you are familiar with Debian, imagine if any upstream changes were
>   automatically pushed to Testing despite any freeze that might be in
>   place.  It would be virtually impossible to complete a freeze and make
>   a new stable release, and Testing and Stable would cease to be useful,
>   leaving only Unstable as a usable target.  This is the situation
>   between "regular" MELPA and MELPA Stable.
>
> For my packages, I tag stable versions for a few reasons:
>
> + To help users track changes in the changelog.
>
> + To help me separate new, possibly bug-introducing changes from
>   working, debugged code.
>
> + To help packagers in systems like Debian and Guix, who package stable
>   versions of some Elisp packages--and who, in so doing, take
>   responsibility for breakage.
>
> Now, I sympathize with not wanting to be vulnerable to potential
> breakage caused by the uncoordinated release of changes to packages on
> "regular" MELPA.  That is a real problem.  But the solution is not to
> use MELPA Stable.  The solution is to take charge of what packages you
> upgrade and when, rather than being at the mercy of whatever commits
> happen to be in MELPA at the moment.
>
> For myself, I commit the ~/.emacs.d/elpa directory to Git with the rest
> of my config, and I upgrade packages intentionally.  If breakage
> happens, I can easily revert and deal with it later.  Other users use
> alternative package managers, like Borg or Straight or Quelpa, which
> pull changes directly from Git repos and allow pinning to commits, tags,
> etc.
>
> So, for yourself, I can only recommend that you abandon MELPA Stable and
> install packages by other means.  If you don't have the time or
> inclination to redo your whole config like that, then just use something
> like Quelpa to install the current version of org-super-agenda directly.
> It's a couple lines of use-package in your config, and you can upgrade
> it manually from then on, e.g. with
> <https://github.com/alphapapa/unpackaged.el#upgrade-a-quelpa-use-package-forms-package>.
> As always, your Emacs config is up to you.
>
> Now, I'm off to to the discussions on MELPA's tracker to add my vote to
> abolish MELPA Stable, or to at least allow packages to opt-out of it.


-- 
Tim Cross

Re: [ANN] org-ql 0.4 released

[Adam Porter]

Tim Cross <theophilusx@gmail.com> writes:

> I don't disagree with most of what you write. I do think a big part of
> the problem is inconsistency and poor practices by some maintainers
> which is at the hart of the issue. Sometimes it seems that package
> maintainers put too much emphasis on getting the latest package out and
> forget about the stability and dependencies that users have.
>
> All of this leaves me wondering why the ELPA system doens't adopt a
> semantic versioning scheme and configure the repositories to
> keep/maintain previous versions. Emacs was very late to the whole
> 'package' concept and it is a little disappointing that more time wasn't
> invested in understanding the challenges and solutions implemented by
> other systems like deb, rpm, CPAN, Java, ruby, python etc. There is
> little unique to the Emacs environment that other environments haven't
> had to resolve one way or the other.  

The problems you identified in the first paragraph are why the solution
identified in the second paragraph doesn't work (for this case; I like
SemVer in general).

> However, a semantic version does provide more information than
> something like version 20200123, which just tells me the release
> date. At least with complaint semantic versions I can have a better
> idea as to whether the changes are just a bug fix, an enhancement or
> major API changes that may break existing dependencies.

Yes, and that's why I like it, too.  But we can't make other developers
use it.  Even if MELPA required SemVer-like numbering, nothing would
stop package maintainers from incrementing major-only version numbers.
It's their code.

Developing with proper use of semantic versioning is a discipline that
requires extra diligence.  It's rarely more fun, and for most Emacs
packages, it's not worth the trouble.  Most packages are dependents
rather than dependencies, leaves at the ends of the branches, and they
can be freely upgraded without breaking anything else.  Most packages'
interoperability with other packages is minor.

If SemVer were required, MELPA would probably have 5-10% of the packages
it does today.  The lax, welcoming approach used by MELPA has drawbacks,
but it also has tremendous benefits, and I think it's to credit for much
of the vibrancy of the Emacs package "ecosystem" and community today.

The problem I'm writing about here is that users' perception of MELPA
Stable doesn't match the reality.  Discussions about how to change MELPA
Stable and MELPA versioning for the better have been going on for years.
Ultimately that doesn't matter, because package authors can do whatever
they want.  We should help users understand and accept the technical
reality--and MELPA Stable misleads them, so it should be removed.  As I
posted in my proposal on the MELPA tracker, we can pursue better
technical solutions then, ones that may be more in line with the
limitations of the platform.

If you're interested in the topic, please join the discussions on the
MELPA tracker.

Re: [ANN] org-ql 0.4 released

[Michael Alan Dorman]

Adam,

I do recognize that melpa-stable is not in any practical way curated,
guaranteed, cross-tested, etc.---and that emacs' packaging doesn't even
necessarily provide what might be required for anyone to try to do any
of that.

On the other hand, I *also* don't assume that maintainers are incapable
of making a reasonable assessment of the stability of their packages, or
of making a personal choice to try to maintain API compatibility in some
sensible way, and so forth.

And you know what: my personal experience over the last five years
hasn't been subject to the problems you identify; perhaps I'm just
lucky.

Nevertheless, my experience leads me to be of the opinion that
abolishing melpa-stable would reek of making the perfect the enemy of
the good---breakage can happen, and some things on it are no better than
whatever happens to be up on melpa, and it's unlikely to ever improve to
be a perfectly curated set of package versions, and got knows some stuff
lags behind enormously...but I think that it is still an improvement to
give maintainers *some* strategy for trying to manage their packages and
their dependencies and communicate all this to their users, rather than
consigning every emacs user to doing individual curation for every
single package they ever use.

Given your position, though, could I suggest that you at least remove
dependencies from your packgaes that feature versions that can only make
sense with melpa-stable?  That's what ultimately started this: the fact
that your new release of org-ql depends on a version of org-super-agenda
that *looks* like you care about melpa stable.

Mike.

Adam Porter <adam@alphapapa.net> writes:
> Michael Alan Dorman <mdorman@jaunder.io> writes:
>
>>> Hi friends,
>>>
>>> FYI, I've released org-ql 0.4.  It includes many improvements since 0.3.
>>>
>>> https://github.com/alphapapa/org-ql
>>
>> It would be nice if you could do a stable release of org-super-agenda so
>> that it could be installed from melpa-stable...
>
> Comments like yours lead me to the conclusion that MELPA Stable needs to
> be abolished.  I have been a proponent of the idea of MELPA Stable, so I
> don't say that lightly.
>
> I'll assume that you don't know what the technical issues are and offer
> an explanation.  Briefly:
>
> + MELPA Stable is nothing like what one might assume it's intended to be
>   like, e.g. Debian Stable or Debian Testing.  It provides none of the
>   benefits that Debian Stable and Testing provide.
>
> + MELPA Stable is, just like "regular" MELPA, a "rolling" collection of
>   packages developed without any coordination between maintainers.
>
> + The only difference is that MELPA Stable contains whatever versions of
>   packages that their maintainers have decided to tag with a version
>   number, rather than the latest commit to the master branch.  These
>   versions are not necessarily better, more stable, more reliable, or
>   more trustworthy than the untagged versions which appear in "regular"
>   MELPA.
>
> + Due to the lack of coordination between dependencies and their APIs,
>   version conflicts and breakage are a regular occurrence.  For example,
>   if package A depends on package B, and package B makes an API change
>   and tags a new MELPA Stable release, users of package A's MELPA Stable
>   version will see package A cease to work properly until package A, not
>   only commits a fix, but tags a new MELPA Stable version containing the
>   fix.  Since packages A and B do not share the same development
>   schedule, it is likely that their tagged-version release schedules
>   will not synchronize well.
>
>   If you are familiar with Debian, imagine if any upstream changes were
>   automatically pushed to Testing despite any freeze that might be in
>   place.  It would be virtually impossible to complete a freeze and make
>   a new stable release, and Testing and Stable would cease to be useful,
>   leaving only Unstable as a usable target.  This is the situation
>   between "regular" MELPA and MELPA Stable.
>
> For my packages, I tag stable versions for a few reasons:
>
> + To help users track changes in the changelog.
>
> + To help me separate new, possibly bug-introducing changes from
>   working, debugged code.
>
> + To help packagers in systems like Debian and Guix, who package stable
>   versions of some Elisp packages--and who, in so doing, take
>   responsibility for breakage.
>
> Now, I sympathize with not wanting to be vulnerable to potential
> breakage caused by the uncoordinated release of changes to packages on
> "regular" MELPA.  That is a real problem.  But the solution is not to
> use MELPA Stable.  The solution is to take charge of what packages you
> upgrade and when, rather than being at the mercy of whatever commits
> happen to be in MELPA at the moment.
>
> For myself, I commit the ~/.emacs.d/elpa directory to Git with the rest
> of my config, and I upgrade packages intentionally.  If breakage
> happens, I can easily revert and deal with it later.  Other users use
> alternative package managers, like Borg or Straight or Quelpa, which
> pull changes directly from Git repos and allow pinning to commits, tags,
> etc.
>
> So, for yourself, I can only recommend that you abandon MELPA Stable and
> install packages by other means.  If you don't have the time or
> inclination to redo your whole config like that, then just use something
> like Quelpa to install the current version of org-super-agenda directly.
> It's a couple lines of use-package in your config, and you can upgrade
> it manually from then on, e.g. with
> <https://github.com/alphapapa/unpackaged.el#upgrade-a-quelpa-use-package-forms-package>.
> As always, your Emacs config is up to you.
>
> Now, I'm off to to the discussions on MELPA's tracker to add my vote to
> abolish MELPA Stable, or to at least allow packages to opt-out of it.

Re: [ANN] org-ql 0.4 released

[Adam Porter]

Michael Alan Dorman <mdorman@jaunder.io> writes:

> On the other hand, I *also* don't assume that maintainers are incapable
> of making a reasonable assessment of the stability of their packages, or
> of making a personal choice to try to maintain API compatibility in some
> sensible way, and so forth.

If it were only a matter of maintainers assessing the stability of
individual packages, there would be no problem.  The problem is that
some packages depend on other packages, and their maintainers don't
coordinate the tagging of stable versions.  Everyone does their own
thing in their own time.  MELPA Stable can't solve that.

> And you know what: my personal experience over the last five years
> hasn't been subject to the problems you identify; perhaps I'm just
> lucky.

It works pretty well, except when it doesn't.

> Nevertheless, my experience leads me to be of the opinion that
> abolishing melpa-stable would reek of making the perfect the enemy of
> the good

AFAICT no good comes from using MELPA Stable.

> but I think that it is still an improvement to give maintainers *some*
> strategy for trying to manage their packages and their dependencies
> and communicate all this to their users

Even the most widely used, most impressive packages, like Magit and
Helm, don't consistently tag stable versions and don't use actual
semantic versioning.  There are no rules, so even if a few developers
were disciplined in their development and tagging, it would not justify
a user using MELPA Stable, because the other packages on it would not
live up to the same standards.  It doesn't offer what you seem to think
it does.

> rather than consigning every emacs user to doing individual curation
> for every single package they ever use.

That's not necessary.  Just put ~/.emacs.d/elpa in version control, and
when your config and package versions seem to be working, commit
everything.  Next time you upgrade packages, if something breaks and you
don't have time to fix it, rollback to what works and try again later.
If everything seems to work, commit the upgraded packages.  It's very
simple, and unlike MELPA Stable, it will give you actual stability.

> Given your position, though, could I suggest that you at least remove
> dependencies from your packgaes that feature versions that can only make
> sense with melpa-stable?  That's what ultimately started this: the fact
> that your new release of org-ql depends on a version of org-super-agenda
> that *looks* like you care about melpa stable.

I care about stability, not MELPA Stable.  It's your choice to use MELPA
Stable, and you're free to upgrade or downgrade individual packages to
work around such occasional, temporary breakage caused by it--the pieces
are yours to keep.  I'm sorry for any inconvenience, but your config is
up to you.

Re: [ANN] org-ql 0.4 released

[David R]

[-- Attachment #1: Type: text/plain, Size: 1592 bytes --]

On Saturday, January 25, 2020, Adam Porter <adam@alphapapa.net> wrote:

> I care about stability, not MELPA Stable.  It's your choice to use MELPA
> Stable, and you're free to upgrade or downgrade individual packages to
> work around such occasional, temporary breakage caused by it--the pieces
> are yours to keep.  I'm sorry for any inconvenience, but your config is
> up to you.


I'm making an extremely late reply, sorry.

It seems to me that this last statement ("Your config is up to you"), or
perhaps the point of view that produces it, is not self-evident when
applied to package versions. I think that in some way it's near the heart
of the controversy.

Maybe for me personally, my config being up to me (regarding package
versions) is a disadvantage. I gratefully make use of a number of packages
that I don't fully understand, and if I was required to study all of them
until I was confident that I *did* fully understand them before installing,
I'd just give up using Emacs at all.

I don't think there's likely to be any person who uses all of the packages
offered in the list. It appears to me that there are sort of "interest
groups" of Emacs users, where members of each group tend to all install,
use, and collectively debug a similar subset of what's available. I've
discovered that if I install the packages that are most popular among the
org-mode group, Emacs does what I want and everything seems to work. A sort
of de facto "group curation system", that has created a sort of unofficial
ad-hoc "org-mode group consensus distro of Emacs".

-- 
David

-- 
David Rogers

[-- Attachment #2: Type: text/html, Size: 1819 bytes --]

Re: [ANN] org-ql 0.4 released

[Adam Porter]

David R <davidandrewrogers@gmail.com> writes:

> On Saturday, January 25, 2020, Adam Porter <adam@alphapapa.net> wrote:
>
>> I care about stability, not MELPA Stable.  It's your choice to use MELPA
>> Stable, and you're free to upgrade or downgrade individual packages to
>> work around such occasional, temporary breakage caused by it--the pieces
>> are yours to keep.  I'm sorry for any inconvenience, but your config is
>> up to you.
>
> It seems to me that this last statement ("Your config is up to you"),
> or perhaps the point of view that produces it, is not self-evident
> when applied to package versions. I think that in some way it's near
> the heart of the controversy.
>
> Maybe for me personally, my config being up to me (regarding package
> versions) is a disadvantage. I gratefully make use of a number of
> packages that I don't fully understand, and if I was required to study
> all of them until I was confident that I *did* fully understand them
> before installing, I'd just give up using Emacs at all.

That is not what I am suggesting.

I suggest that you:

1.  Keep your Emacs config backed up, preferably with version control,
including all installed packges.

2.  Upgrade packages deliberately, not "upgrade all upgradable
packages."

3.  When you discover a problem caused by an upgrade, roll-back to a
known-good configuration until you have time to debug the problem.

This is what I recommend to all Emacs users.  It does not require
understanding any packages' source code.

Elisp has no inter-package API.  It's just a Lisp machine.  Elisp
packages are just symbols that you load into your image.  There is no
actual separation between packages' symbols.  There are no versioned
APIs, no synchronized transitions.

Each user's configuation, image, set of installed packages, etc. is that
user's responsibility.  In that sense, it's no different from a computer
system as a whole: you choose what software to install on it.  If you
like or need a certain version of certain software, it's up to you to
ensure that you have a copy of it available.  If you upgrade some
software and it doesn't work anymore with some other software version,
it's up to you to deal with it.

One of Emacs's chief strengths is user empowerment.  That doesn't mean
that users need to know how everything works--not even the core Emacs
developers do.  It means that you should know enough to maintain the
operability of the system, like you generally do for the rest of your
computer system.

The Emacs package "ecosystem" is not a "vendored" system.  It's not like
Debian, with thousands of relatively curated packages maintained as a
middleman, expected to work together in a stable release.  In Emacsland,
Each package (outside of Emacs and ELPA, and somewhat within ELPA as
well) is developed independently.

Nor should Emacs be treated like other software systems that are
live-updated whenever the developers hit the "push" button, with users
expecting the latest everything to work together all the time because
one party (ostensibly) takes responsibility for ensuring that.

Emacs gives the user the power.  And with great power...well, you know.

Having said all that, the problems with MELPA Stable are, in a sense,
artificial, and they're orthogonal to these general issues.  I can only
recommend, again, to not use it.  If you're lucky, it will work fine.
When it doesn't, the solution will involve not using it.  So you might
as well just skip it.  If you want less-frequent package upgrades, just
don't upgrade your packages so frequently.  Or use something like Quelpa
or Straight or Borg, where you can easily install the package versions
you want.