Dmitry Gutov writes: > I don't immediately see that new generics are needed. But if someone > finds a workable approach (with hard numbers), good. See benchmarks. >> But of course we will be outlawing every extension like the :m6-sparse >> example I gave. >> And also, IMHO, we end up with a much poorer protocol in terms >> of versatility (no find-by-key :key, no :from-end, no :start/:end). >> But this part is not about performance, rather interface, and that's >> not the topic of this subthread. > > cl-lib is more flexible in one aspect (its additional keywords > vocabulary which basically multiplies the provided interface by 100x), > but it's more rigid in the data it works with. Exactly. It's the CL tradeoff, and it give you superior performance. CL was never meant for custom sequences. One should use seq.el for that, and IMO, _only_ for that. The seq.el tradeoff gives you inferior performance and poorer interface. But it's not only a matter of performance. Take the 'seq-remove-at-position' generic. Presumably someone thought that operationq was common enough to merit a separate entry point. In CL, this very operation can be done (probably faster) with cl-remove using keyword arguments. But say you want to remove two elements at two consecutive positions? Two calls with two slowdowns? What about n positions? cl-remove can do all of that in the same call with :COUNT and :START for example. If you want to cut that time down in seq.el you either have to introduce 'seq-remove-at-consecutive-positions' generic or pay what is probably a tremedous price by using `seq-drop-while` which calls the seq-elt generic in a hot loop. And so, in general, the seq.el interface either bloats up, or is slow, or is inherently inflexible. I think seq.el can obviate any one of these problems at best, but not two or three simultaneously. cl-lib obviates all three of them at the cost of one drawback: custom sequence support. So in general, different tools, different jobs. >>> If that is the only drawback we can find, it is an unavoidable one. > Could you try to explain what I should find in the second example? > What do you mean by "outlaw"? What I meant is that the only way to get the same performance out of seq.el is to have early #'sequencep checks that bypass the generics completely, and this makes custom sequences based on sequences impossible. > Does causing worse performance for a short time constitute "outlawing" > something? But you should also find in that "m6sparse" example that the logic is broken -- not only in terms of performance -- until its author implements seq-contains-pred. So this is pure "Incompatible Lisp changes" material (which I also think tanked performance should be btw.) But in fact is is already broken by all the "list optimzations" in seq.el. Optimizations, before yours, that caused whatever the contract was to be violated. To be able to use `seq-drop-while` for my m6sparse sequence, I have to add implementations to all those generic entry points, which is just akward. >> And even outlaw more stuff. For example, these generics even have >> specializers on all mandatory arguments? >> For example, why does seq-do have FUNCTION as a required argument??? > > Because FUNCTION is applied to SEQUENCE? > >> It should be &optional or a &key, with a default value. Which >> cl-degeneric obviously supports. That way specializations on FUNCTION >> become impossible, or at least become much harder and there's less >> risk of tanking user code. Design mistake IMO. > > I'm reasonably sure nobody expects function to be anything but a > straight function (symbol or a lambda), because that's how 'seq-do' is > used throughout the code. Yes, but putting as a required argument in the arglist means users can specialize for it, and that isn't needed. Not sure anyone does it or even if that makes the generic even slower. >>> But that is the price one has to pay for correcting a design mistake. >>> We're not going to do this every Tuesday. >> Sure, but that price increases manyfold if we start suggesting >> seq.el as a replacement for all your sequence processing needs. > > We can first fix the mistake and then go on to continue "suggesting it > as a replacement". Or not. > > I don't exactly see it that way, though. And you give an impression of > arguing for the opposite: toward never using it at all. Not at all. Maybe you missed some of my previous messages. I think seq.el's support for polymorphic sequences, though a little bit flawed in some respects, is very useful. For example, I've been pondering using it in eglot.el to try to speed up JSON parsing. If some kind of seq-plist-get and seq-plist-destructuring-bind can be designed, I might be able to skip consing much of the useless elements of a gigantic JSON blob and parse just the parts I need. Of course, not easy, but I think seq.el is the tool for that. >> Why working on the :m6-sparse extension, I noticed Emacs becomes >> noticeably slower, and I suspect that's because while I was editing, >> all the seq functions I was writing where being probed for >> applicability, while core things like completion, buffer-switching >> etc are calling seq-foo generics. > > It could be helpful to do some profiling and see where the slowdown > came from. Could it come exactly from the set operations? Not sure. It might have come from tracing seq.el functions, for example. You might say that it's my fault I was tracing them, but should I be punished in Emacs usability just for trying to use Emacs to iteratively develop a seq.el extension? Anyway tracing basic staples such as seq-do and seq-let gives some insight as to where they are used and what shape of arguments they are called with in your normal programming activities. Small lists seem to appear a lot more often. But expect a massive slowdown while tracing: even with modest seq.el usage in current core, these generics are called a lot already. >> I find this performance aspect very bad. Maybe it can be obviated, >> but only if you drop the '(if (sequence-p seq)' bomb into seq.el >> somehow. I don't see how we can avoid that one. > > I don't quite see the need. And it's unlikely to be of reliable help: > my observations say that method dispatch simply becomes slower as soon > as a generic function gets a second implementation. And that > implementation might arrive from any third-party code. Exactly. The entry point generics probably can never be avoided. I think we agreed that noone -- user or library -- should add implementations to them. That's why I think not making them defuns was another design mistake. But other intermediary generics _can_ be skipped and would bring a performance boost to seq.el. Alright. That all said, here's the latest results, which I gathered using the attached sequence-benchmarks.el are also attached in results.txt. I gathered each set of timings by running these two things src/emacs -Q -nw sequence-benchmarks.el -f emacs-lisp-byte-compile-and-load and then src/emacs -Q -nw -l m6sparse.el sequence-benchmarks.el -f emacs-lisp-byte-compile-and-load And I also attach m6sparse.el. The branch I used was feature/cl-lib-improvements where I also pushed your seq-difference-3 patch. This email is long enough, so take your conclusions. I don't think any results are exactly flattering to seq.el, especially -- but not only -- when you compare to the destructive versions of some utils that cl-lib offers. But this one stands out. I hope you can read this macro more or less. (joaot/with-benchmark-group "destructuring" ((list1 . (make-list 3 0))) 100000 (joaot/bench (pcase-let ((`(,a ,b ,c) list1)) (+ a b c))) (joaot/bench (cl-destructuring-bind (a b c) list1 (+ a b c))) (joaot/bench (seq-let (a b c) list1 (+ a b c)))) Running in Emacs -Q: ("destructuring" (cl-destructuring-bind "FASTEST" 0.010702393 0 0.0) (pcase-let "1.3x SLOWER" 0.014360937999999998 0 0.0) (seq-let "3.5x (rel 2.6x) SLOWER" 0.03706726 0 0.0)) Where after loading m6sparse.el we go to this: (("destructuring" (cl-destructuring-bind "FASTEST" 0.010157632 0 0.0) (pcase-let "1.3x SLOWER" 0.013152518000000002 0 0.0) (seq-let "14.8x (rel 11.4x) SLOWER" 0.15057331499999999 6 0.04785139399999849)) You may try to optimize this, but you'll probably have to introduce yet another generic. And this seq-let thing should also remind us that this should never only be about "fast enough". Eli was suggesting seq-let as an alternative to pcase-let the other day. Let-like forms do appear in tight loops (and tight loops aplenty do exist), I expect a let-like form on a list to expand to little more than some car, cadr, etc calls, not some immensely slow generic call by comparison. So, in summary. YES to seq.el for custom sequences, we need more of those (probably in core even) and NO to seq.el as a drop-in general-purpose sequence processing library. João