Re: [elpa] master 88578a4: Increase the default number of hits

Re: [elpa] master 88578a4: Increase the default number of hits

[Michael Albinus]

Lars Ingebrigtsen <larsi@gnus.org> writes:

Hi Lars,

>     Increase the default number of hits
>     
>     * debbugs-gnu.el (debbugs-gnu-default-hits-per-page): Increase the
>     default number of hits, because it doesn't make much sense to look
>     at partial bug lists when doing triage.
> ---
>  packages/debbugs/debbugs-gnu.el |    2 +-
>  1 files changed, 1 insertions(+), 1 deletions(-)
>
> diff --git a/packages/debbugs/debbugs-gnu.el b/packages/debbugs/debbugs-gnu.el
> index bada55b..9ae0bc8 100644
> --- a/packages/debbugs/debbugs-gnu.el
> +++ b/packages/debbugs/debbugs-gnu.el
> @@ -220,7 +220,7 @@
>    (mapcar 'cadr (cdr (get 'debbugs-gnu-default-packages 'custom-type)))
>    "*List of all possible package names.")
>  
> -(defcustom debbugs-gnu-default-hits-per-page 500
> +(defcustom debbugs-gnu-default-hits-per-page 3000
>    "*The number of bugs shown per page."
>    :group 'debbugs-gnu
>    :type 'integer

IIRC, Glenn has asked to use 500 hits per default only. Otherwise, he
feared that there might be performance problems.

Another approach might be to show the newest hits first. Would this help
you?

Best regards, Michael.

Re: [elpa] master 88578a4: Increase the default number of hits

[Lars Ingebrigtsen]

Michael Albinus <michael.albinus@gmx.de> writes:

> IIRC, Glenn has asked to use 500 hits per default only. Otherwise, he
> feared that there might be performance problems.

Back in the olden days, the debbugs server was slow.  This doesn't seem
to be a problem any more (judging by how long debbugs-gnu uses to fetch
the data).

And I doubt that Emacs developers are hitting the interface very often.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: [elpa] master 88578a4: Increase the default number of hits

[Glenn Morris]

Lars Ingebrigtsen wrote:

> -(defcustom debbugs-gnu-default-hits-per-page 500
> +(defcustom debbugs-gnu-default-hits-per-page 3000

Please set this back to 500. We already went over why, and nothing's
changed since then (except the system has more users). 1000 might be ok,
but 3000 is just too much.

http://lists.gnu.org/archive/html/emacs-devel/2011-07/msg00155.html

This is written in the file's commentary.

Re: [elpa] master 88578a4: Increase the default number of hits

[Lars Ingebrigtsen]

Glenn Morris <rgm@gnu.org> writes:

> Lars Ingebrigtsen wrote:
>
>> -(defcustom debbugs-gnu-default-hits-per-page 500
>> +(defcustom debbugs-gnu-default-hits-per-page 3000
>
> Please set this back to 500. We already went over why, and nothing's
> changed since then (except the system has more users). 1000 might be ok,
> but 3000 is just too much.

Ok, I'll change it back, but I don't really understand why this is a
problem.  Fetching all 4400 bugs seems to be pretty quick, and doing bug
triage if you don't have the full list of bugs is awkward, because it's
difficult to use `C-s' to search for bugs and `/' to narrow to all the
bugs in the area you're interested in (in a quick and convenient manner).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: [elpa] master 88578a4: Increase the default number of hits

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Sat, 26 Dec 2015 19:52:57 +0100
> Cc: emacs-devel@gnu.org
> 
> Glenn Morris <rgm@gnu.org> writes:
> 
> > Lars Ingebrigtsen wrote:
> >
> >> -(defcustom debbugs-gnu-default-hits-per-page 500
> >> +(defcustom debbugs-gnu-default-hits-per-page 3000
> >
> > Please set this back to 500. We already went over why, and nothing's
> > changed since then (except the system has more users). 1000 might be ok,
> > but 3000 is just too much.
> 
> Ok, I'll change it back, but I don't really understand why this is a
> problem.  Fetching all 4400 bugs seems to be pretty quick

It took 45 sec here.

> and doing bug triage if you don't have the full list of bugs is
> awkward, because it's difficult to use `C-s' to search for bugs and
> `/' to narrow to all the bugs in the area you're interested in (in a
> quick and convenient manner).

Well, debbugs-gnu does ask you how many to fetch, so you can very
easily override this, right?  IOW, why change the default if you can
easily customize it or override it?

Re: [elpa] master 88578a4: Increase the default number of hits

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> Ok, I'll change it back, but I don't really understand why this is a
>> problem.  Fetching all 4400 bugs seems to be pretty quick
>
> It took 45 sec here.

26 seconds here, of which the SOAP call itself took about 18 seconds...

> Well, debbugs-gnu does ask you how many to fetch, so you can very
> easily override this, right?  IOW, why change the default if you can
> easily customize it or override it?

Of course I override it.  But I want the defaults to be sensible for
other users, too, or they'll get the impression that debbugs-gnu is
slightly useless.  Which it is if you accept the default, in my opinion.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: [elpa] master 88578a4: Increase the default number of hits

[Michael Albinus]

Lars Ingebrigtsen <larsi@gnus.org> writes:

>> Well, debbugs-gnu does ask you how many to fetch, so you can very
>> easily override this, right?  IOW, why change the default if you can
>> easily customize it or override it?
>
> Of course I override it.  But I want the defaults to be sensible for
> other users, too, or they'll get the impression that debbugs-gnu is
> slightly useless.  Which it is if you accept the default, in my opinion.

You have a similar situation in `vc-print-log', which is limited by
`vc-log-show-limit'. Do you have the impression, that this function is
"slightly useless"?

And I have also in mind to mirror `debbugs-gnu-*' functions by
`debbugs-debian-*' functions, which would retrieve bugs from
bugs.debian.org instead of debbugs.gnu.org. Here we don't speak about
5.000 bugs to be retrieved.

Best regards, Michael.

Re: [elpa] master 88578a4: Increase the default number of hits

[Lars Ingebrigtsen]

Michael Albinus <michael.albinus@gmx.de> writes:

> You have a similar situation in `vc-print-log', which is limited by
> `vc-log-show-limit'. Do you have the impression, that this function is
> "slightly useless"?

No, why should it?  I don't limit and rearrange stuff in that buffer...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: [elpa] master 88578a4: Increase the default number of hits

[Glenn Morris]

Lars Ingebrigtsen wrote:

> Of course I override it.  But I want the defaults to be sensible for
> other users, too, or they'll get the impression that debbugs-gnu is
> slightly useless.  Which it is if you accept the default, in my opinion.

I'm baffled as to how "only" getting 500 search results per page makes
the results "slightly useless".

Per page search limits:

mailman: 100
gmane: 10
Google: ~ 10

Re: [elpa] master 88578a4: Increase the default number of hits

[Michael Albinus]

Glenn Morris <rgm@gnu.org> writes:

> I'm baffled as to how "only" getting 500 search results per page makes
> the results "slightly useless".

`debbugs-gnu' has a nifty narrowing feature. This works the better the
more bugs are present per page. That's why I've proposed to fill the
page request by request, instead of creating a new page per request.

Best regards, Michael.

Re: [elpa] master 88578a4: Increase the default number of hits

[Lars Ingebrigtsen]

Glenn Morris <rgm@gnu.org> writes:

> Lars Ingebrigtsen wrote:
>
>> Of course I override it.  But I want the defaults to be sensible for
>> other users, too, or they'll get the impression that debbugs-gnu is
>> slightly useless.  Which it is if you accept the default, in my opinion.
>
> I'm baffled as to how "only" getting 500 search results per page makes
> the results "slightly useless".

Try it yourself.  Imagine that you're switching between limiting the bug
lists from "moreinfo" to "ssl" (as I did just now to get back to dealing
with real bugs again).  

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: [elpa] master 88578a4: Increase the default number of hits

[Glenn Morris]

Lars Ingebrigtsen wrote:

> Try it yourself.  Imagine that you're switching between limiting the bug
> lists from "moreinfo" to "ssl" (as I did just now to get back to dealing
> with real bugs again).  

I just don't work that way.
I just do a separate search for "ssl" (or whatever). 6 whole results!
There are only ~ 250 "moreinfo" bugs.
So I don't see why you need to list 3000.

If you are saying "I want to have a complete listing of all the bugs and
then do my own local searches on the results", then I'd ask you to
find some other way to implement that that doesn't put such a load on
the server.

Re: [elpa] master 88578a4: Increase the default number of hits

[Michael Albinus]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Ok, I'll change it back, but I don't really understand why this is a
> problem.  Fetching all 4400 bugs seems to be pretty quick, and doing bug
> triage if you don't have the full list of bugs is awkward, because it's
> difficult to use `C-s' to search for bugs and `/' to narrow to all the
> bugs in the area you're interested in (in a quick and convenient manner).

If the number of bugs to be displayed exceeds
`debbugs-gnu-default-hits-per-page', several pages in different buffers
are shown successively. What if we show them all in the same buffer?
That is, you get the first 1000 bugs, then you ask for more and you get
the next 1000 bugs in the same buffer, you ask for more and you get the
missing 141 bugs in the same buffer. By this, you could still use the
narrowing commands during your triage, but the default hits per page is
still accesptable.

Best regards, Michael.

Re: [elpa] master 88578a4: Increase the default number of hits

[Lars Ingebrigtsen]

Michael Albinus <michael.albinus@gmx.de> writes:

> If the number of bugs to be displayed exceeds
> `debbugs-gnu-default-hits-per-page', several pages in different buffers
> are shown successively. What if we show them all in the same buffer?
> That is, you get the first 1000 bugs, then you ask for more and you get
> the next 1000 bugs in the same buffer, you ask for more and you get the
> missing 141 bugs in the same buffer. By this, you could still use the
> narrowing commands during your triage, but the default hits per page is
> still accesptable.

Well, I want all the bugs.  :-)  So that I can switch between different
subsets of the bug reports like *snap*.  

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: [elpa] master 88578a4: Increase the default number of hits

[Michael Albinus]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Michael Albinus <michael.albinus@gmx.de> writes:
>
>> If the number of bugs to be displayed exceeds
>> `debbugs-gnu-default-hits-per-page', several pages in different buffers
>> are shown successively. What if we show them all in the same buffer?
>> That is, you get the first 1000 bugs, then you ask for more and you get
>> the next 1000 bugs in the same buffer, you ask for more and you get the
>> missing 141 bugs in the same buffer. By this, you could still use the
>> narrowing commands during your triage, but the default hits per page is
>> still accesptable.
>
> Well, I want all the bugs.  :-)  So that I can switch between different
> subsets of the bug reports like *snap*.  

You'll have all bugs at the end, in the same buffer. You just need to
say three times "download the next debbugs-gnu-default-hits-per-page bugs",
instead of saying it once. Not too uncomfortable, when you start a
triage over all bugs.

Best regards, Michael.

Re: [elpa] master 88578a4: Increase the default number of hits

[Lars Ingebrigtsen]

Michael Albinus <michael.albinus@gmx.de> writes:

> You'll have all bugs at the end, in the same buffer. You just need to
> say three times "download the next debbugs-gnu-default-hits-per-page bugs",
> instead of saying it once. Not too uncomfortable, when you start a
> triage over all bugs.

As a user interface, having to wait for four times with commands
inbetween is not a very good user interface.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: [elpa] master 88578a4: Increase the default number of hits

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Michael Albinus <michael.albinus@gmx.de> writes:
>
>> You'll have all bugs at the end, in the same buffer. You just need to
>> say three times "download the next debbugs-gnu-default-hits-per-page bugs",
>> instead of saying it once. Not too uncomfortable, when you start a
>> triage over all bugs.
>
> As a user interface, having to wait for four times with commands
> inbetween is not a very good user interface.

(But the debbugs-get-bugs function could just split the request up into
chunks of 1000, if you think that's a better idea.  It'll be more pings
back and forth with the server, so it might be slower (unless the server
starts thrashing on account of 5000 requests being too much), but
probably not that much slower...)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: [elpa] master 88578a4: Increase the default number of hits

[Michael Albinus]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> As a user interface, having to wait for four times with commands
> inbetween is not a very good user interface.

It is the compromise with Glenn's request for a proper
`debbugs-gnu-default-hits-per-page'. Of course you could always set this
custom option to 3000 on your own, while running your triage. Glenn
might tolerate it just-for-you (don't know, just a guess).

Best regards, Michael.

Re: [elpa] master 88578a4: Increase the default number of hits

[David Kastrup]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Michael Albinus <michael.albinus@gmx.de> writes:
>
>> You'll have all bugs at the end, in the same buffer. You just need to
>> say three times "download the next debbugs-gnu-default-hits-per-page
>> bugs", instead of saying it once. Not too uncomfortable, when you
>> start a triage over all bugs.
>
> As a user interface, having to wait for four times with commands
> inbetween is not a very good user interface.

Better than if have to say three times "download the next
debbugs-gnu-default-hits-per-page bugs", click the heels of your golden
slippers together, close your eyes and turn around.

Maybe this should take a universal-argument via (interactive "P")?  Just
C-u means everything?

-- 
David Kastrup

Re: [elpa] master 88578a4: Increase the default number of hits

[Michael Albinus]

David Kastrup <dak@gnu.org> writes:

> Maybe this should take a universal-argument via (interactive "P")?  Just
> C-u means everything?

`debbugs-gnu' uses the universal-argument already for further questions.

Best regards, Michael.

Re: [elpa] master 88578a4: Increase the default number of hits

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > You'll have all bugs at the end, in the same buffer. You just need to
  > > say three times "download the next debbugs-gnu-default-hits-per-page bugs",
  > > instead of saying it once. Not too uncomfortable, when you start a
  > > triage over all bugs.

  > As a user interface, having to wait for four times with commands
  > inbetween is not a very good user interface.

This is Lisp code; we can easily give users a way to ask for whatever
they want.

  > Because the more results you fetch at once, the more memory it uses on
  > the server, thus increasing the likelihood of a bunch of simultaneous
  > requests causing an out-of-memory condition. I did already explain this,
  > not sure why it's hard to get across. Most (all?) search engines limit
  > the number of results you get per page.

If it causes problems to request all the bug reports
in a single command, the Lisp code could send multiple commands
to fetch them 500 at a time until it has got them all.


-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: [elpa] master 88578a4: Increase the default number of hits

[Glenn Morris]

Lars Ingebrigtsen wrote:

> Ok, I'll change it back, but I don't really understand why this is a
> problem. 

Because the more results you fetch at once, the more memory it uses on
the server, thus increasing the likelihood of a bunch of simultaneous
requests causing an out-of-memory condition. I did already explain this,
not sure why it's hard to get across. Most (all?) search engines limit
the number of results you get per page.

In fact, there really should be a hard limit that the server imposes.
(Michael, could you look at implementing that?)

Re: [elpa] master 88578a4: Increase the default number of hits

[Michael Albinus]

Glenn Morris <rgm@gnu.org> writes:

> In fact, there really should be a hard limit that the server imposes.
> (Michael, could you look at implementing that?)

Will check. What would be an acceptable hard limit? 5.000?

I will also see, whether bugs.debian.org has implemented such a server
side limit meanwhile. In their web interface on
<http://bugs-search.debian.org/cgi-bin/search.cgi>, they have the limit
of 200 hits per search request.

Best regards, Michael.

Re: [elpa] master 88578a4: Increase the default number of hits

[Glenn Morris]

Michael Albinus wrote:

> Glenn Morris <rgm@gnu.org> writes:
>
>> In fact, there really should be a hard limit that the server imposes.
>> (Michael, could you look at implementing that?)
>
> Will check. What would be an acceptable hard limit? 5.000?

Personally I'd want to see the hard limit set at 1000.
5000 is just too much. It may seem to work ok for one person, but then
if two of you happen to do that at the same time, bang.
(I just run debbugs-gnu and watch memory usage in top.)

If you want to implement clever ways to run separate queries one after
the other then stitch the results together via local caching, I guess
that's ok.

Re: [elpa] master 88578a4: Increase the default number of hits

[Michael Albinus]

Glenn Morris <rgm@gnu.org> writes:

> If you want to implement clever ways to run separate queries one after
> the other then stitch the results together via local caching, I guess
> that's ok.

Will do. It will take some days; our children and grand-children are
visiting us over Xmas.

Re: [elpa] master 88578a4: Increase the default number of hits

[Michael Albinus]

Glenn Morris <rgm@gnu.org> writes:

> If you want to implement clever ways to run separate queries one after
> the other then stitch the results together via local caching, I guess
> that's ok.

I gave it a first shot. Lars, could you check whether it works for you?

It is not complete yet. I want to start printing of the tabulated list
after retrieving the first chunk of data, in parallel to the subsequent
queries. This should result in better performance.

And the documentation must be adapted, of course.

Best regards, Michael.

Re: [elpa] master 88578a4: Increase the default number of hits

[Lars Ingebrigtsen]

Michael Albinus <michael.albinus@gmx.de> writes:

> I gave it a first shot. Lars, could you check whether it works for you?

It works, and it's just 20% slower than the previous version (which is
about what I would have expected).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: [elpa] master 88578a4: Increase the default number of hits

[Michael Albinus]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> It works, and it's just 20% slower than the previous version (which is
> about what I would have expected).

I've committed a patch, which allows to retrieve the bugs
asynchronously. You must install GNU ELPA's "async" package.

Best regards, Michael.

Re: [elpa] master 88578a4: Increase the default number of hits

[Dmitry Gutov]

On 12/29/2015 01:00 AM, Michael Albinus wrote:

> I've committed a patch, which allows to retrieve the bugs
> asynchronously. You must install GNU ELPA's "async" package.

I don't think it's in GNU ELPA.

Re: [elpa] master 88578a4: Increase the default number of hits

[Michael Albinus]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 12/29/2015 01:00 AM, Michael Albinus wrote:
>
>> I've committed a patch, which allows to retrieve the bugs
>> asynchronously. You must install GNU ELPA's "async" package.
>
> I don't think it's in GNU ELPA.

Oops, yes. It's in MELPA.

John, would it be OK to add the async package to GNU ELPA?

Best regards, Michael.

Re: [elpa] master 88578a4: Increase the default number of hits

[John Wiegley]

[-- Attachment #1: Type: text/plain, Size: 324 bytes --]

>>>>> Michael Albinus <michael.albinus@gmx.de> writes:

> John, would it be OK to add the async package to GNU ELPA?

I would be very happy to see it in core ELPA.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 629 bytes --]

Adding async package to GNU ELPA (was: [elpa] master 88578a4: Increase the default number of hits)

[Michael Albinus]

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> Michael Albinus <michael.albinus@gmx.de> writes:
>
>> John, would it be OK to add the async package to GNU ELPA?
>
> I would be very happy to see it in core ELPA.

Thanks John!

I must confess I have no idea how to add the package to ELPA. I suppose
it shall be an external package.

Could somebody who knows do it? Or tell me where I could get the needed
information how-to?

TIA, and best regards, Michael.

Re: Adding async package to GNU ELPA (was: [elpa] master 88578a4: Increase the default number of hits)

[Artur Malabarba]

> I must confess I have no idea how to add the package to ELPA. I suppose
> it shall be an external package.
>
> Could somebody who knows do it? Or tell me where I could get the needed
> information how-to?

See the README file in elpa.git. I've pasted the file below for your
convenience.
The "External branches" section explains how to use subtrees or a
separate branch.


-----
Copyright (C) 2010-2011, 2014, 2015 Free Software Foundation, Inc.
See the end of the file for license conditions.


This branch contains the sources, deployment scripts, and auxiliary
files for the Emacs Lisp package archive (elpa.gnu.org).

This file explains the branch layout, how to add and edit packages,
and how to deploy the archive (either on elpa.gnu.org, or a local copy
for testing purposes).


* DIRECTORY LAYOUT

** admin/    -- scripts for administering the package archive.
** html/     -- HTML for the elpa.gnu.org website.
** packages/ -- source code for the packages.


* PACKAGES

** Contents of the packages/ directory:
This directory holds the package sources, with one subdirectory for
each package.

Each directory in here corresponds to a package, which can be
either a single-file package or a multifile package.

A nightly cron job refreshes the GNU ELPA archive from this repository.

This cron job only creates a new package when the "version" (as specified in
the "Version:" header) of a package is modified.  This means that you can
safely work on the next version here without worrying about the unstable
code making it to GNU ELPA, and simply update the "version" when you want to
release the new code.

** To add a package: (submission, submit)

Adding a basic package is very simple. There are thorough
instructional, but the gist is that you:

1. Notify emacs-devel@gnu.org.
2. Place all files inside `packages/<pkg-name>/'.
3. `git add', `git commit' and `git push'.

If you don't have push access to the repository, someone will do steps
2 and 3 for you.

*** Notify emacs-devel@gnu.org

There is no approval process for GNU Elpa packages.  Still,
you must send an email to emacs-devel for several reasons:

- Notifying other developers;
- Making sure the package doesn't break FSF rules;
- Checking if the package is not reinventing the wheel;
- Ensuring that first-time developers are doing it right.

Before doing anything, please ensure your package follows the
conventions described in the `** Format' section.  Then, send an email
to the list with the subject:
    [ELPA] New package: <pkg-name>

Start your message with an explanation about the package.  A
copy-paste of the package's Summary and Commentary is perfectly fine
here, but you can write more or less than that if you'd like.

At the bottom of the message contents include the changes you're going
to make (the patch).  For a single-file package this can be the
package file itself instead of the patch.  If you prefer (and if you
have push access), you can push your changes to a branch called
`scratch/<pkg-name>', and mention the branch in your message.

After 48h, or once any issues have been addressed, someone will push
your changes for you.  You should probably also subscribe to
emacs-devel@gnu.org, since that's where we discuss about GNU Elpa, and
to bug-gnu-emacs@gnu.org, since that's where people will report bugs
about your package.

*** Add a simple (1-file) package as packages/<pkg-name>/<pkg-name>.el.

The file needs to follow the usual coding conventions (most
importantly start with ";;; <file> --- <description>") and have a
"Version:" and "Maintainer:" pseudo-header (see the "Format"
subsection below).

For some examples, see
    (info "(elisp) Simple Packages")

*** Add a multi-file package as a directory, packages/<pkg-name>.

It needs to have a file named packages/<pkg-name>/<pkg-name>.el which
follows the
same rules as above.

It additionally follows the same guidelines described in
    (info "(elisp) Multi-file Packages")
with the exception that it is not a tar package (it's a plain
directory) and it must not contain a "<pkg-name>-pkg.el" file (this
will be created for you).

*** Commit your changes the usual way ("git add", "git commit", etc).

Changes in the Git repository do not immediately propagate to the
user-facing archive (what users see when they do `M-x list-packages').
That is done by deploying the archive, which happens automatically
once a day, and the changes are only reflected when the "Version:"
header changes.

** Format

Each package should follow the ELPA packaging conventions, but there are
some differences due to the way the deployment script creates the packages
and the web-pages from this source code:
- Multi-file packages put the package metadata in the main <pkg-name>.el file
  in the format used for single-file packages: the <pkg-name>-pkg.el file is
  auto-generated from it.
- Every package should have both a "Version:" *and* a "Maintainer:".
- the "URL:" header can be used to specify the home page
  of the package, if it's maintained externally.
- A "News:" section (or "NEWS" file) can/should be used to list the
  user-visible changes of each version.
- The "Package-Type:" header can be used to force the type of package
  created (can be either `simple' for single-file packages or `multi' for
  tarballs).  By default the type is decided based on whether there are
  several Elisp files in the source.
- If you want some files to not be included in the tarball, you can
  put a `.elpaignore' file in the root of your package directory, where you
  can list patterns of files to ignore (this file is passed to tar's -X).

** External branches

The above instructions are enough to add regular packages, those that
are maintained primarily here in the repository.  The instructions
below are for those maintainers who prefer to use a dedicated
repository or branch for the package.

There are two ways to do that: subtrees and externals.

Either way, such packages should always be listed in the
`externals-list' file.

In both cases, a copy of the code is kept in the `elpa' repository
(not necessarily in the master branch) and should be sync'd with the
upstream every once in a while.  This copy may include local changes,
although these should be kept to a minimum.

If know you don't want a local package, but don't know which of these
two options you prefer, then use a subtree.

*** Subtrees

In the `subtree' case, the copy of the code is kept here in the master
branch, inside its corresponding `packages/<pkg-name>' directory just
as if it were a local package.

In fact, a subtree package is essentially indistinguishable from a
local package.  The only difference is that, instead of developing it
here, you do it in some remote repository and pull in the changes.

Instead of manually creating the directory, you should be able to use:

    git subtree add --prefix=packages/<pkg-name> <remote-repo> <remote-branch>

Later, when you make some changes to the remote and want to publish
them here, simply do:

    git subtree pull --prefix=packages/<pkg-name> <remote-repo> <remote-branch>

On older git versions "git subtree" might not be available.  You can
try "git merge -s subtree", or just update git.

- <remote-repo> is the remote's URL.  If you've previously used "git
  remote add", then this can be the remote's name.
- <remote-branch> is the branch you want to pull (probably "master").

If you want the local code to be slightly different from the remote,
simply commit further changes to it here.  Of course, this may trigger
merge conflicts when you do a "subtree pull" in the future, so it's
best to avoid these local changes.

If someone makes changes to your package here on elpa.git and you want
to push them to your remote, it's easiest to just copy these changes
over to the remote repo.  Trying to push a subtree with git is likely
to induce headache.

**** When you're adding and pulling, DO NOT --SQUASH!!

Don't worry about flooding elpa.git's commit log with your package's
commit messages.  Your package is part of elpa.git.  Squashing doesn't
help and only gets in the way.

*** Externals

In the `external' case, the copy of the code is not kept here but in the
`externals/<pkg-name>' branch in the `elpa' repository.

You can check out all the external packages into the `packages' directory
with the command:

   make externals

You can check out a specific external PACKAGE into the `packages'
directory with these commands:

   cd packages
   git clone --reference .. --single-branch --branch externals/PACKAGE
$(git config remote.origin.url) PACKAGE

If you already have a packages/PACKAGE directory with a previous
checkout, you can update it like this:

   cd packages/PACKAGE
   git pull

** Public incubation

If you want to develop a package publicly prior to its first release (to
benefit from others' feedback, primarily), but not in an external repo,
you have 2 choices:
- you can simply put "Version: 0" to indicate that this should not be
  released yet.
- or you can push to an "ephemeral" branch -- subject to rebase and eventual
  removal upon finishing merge -- for the duration of the incubation.

* DEPLOYMENT

** To install all the packages "in place":

   make externals
   make

This compiles and generates autoloads for all the packages in the
packages/ directory.  You can then add that directory, e.g. with:

(eval-after-load 'package
  '(add-to-list 'package-directory-list ".../elpa/packages"))

** To deploy the package repository as a remotely-accessible archive:

   git clone .../elpa
   (cd elpa; git clone .../emacs)    #If you want to generate :core packages.
   mkdir build
   cd build
   (cd ../elpa; git log --format=%H | tail -n 1) >.changelog-witness
   ln -s ../elpa/admin
   ln -s ../elpa/GNUmakefile
   admin/update-archive.sh

This deploys the packages to the staging/ directory (sibling of "build").
Unlike "make", this makes a full copy of the packages, tars up
multi-file packages, and doesn't byte-compile any files.

** To access a deployed archive

To access the archive via HTTP, have a symlink (say) /var/www/packages
pointing to DEST/packages, and set up Emacs with

  (setq package-archives '(("new-elpa" . "http://foo.com/packages")))

You can also access the archive via normal file access.  Such "local
archives" are useful for debugging:

  (setq package-archives '(("local-elpa" . ".../elpa/packages")))

** Notes specific to elpa.gnu.org

The way things are set up on this machine, we refresh the archive by
a cron job.  You can do it by hand by logging in (access set up by FSF
admins), and

   su elpa
   cd ~elpa/build
   admin/update-archive.sh

Which makes a full archive deployment, as discussed above.  The symlink
/var/www/packages points to the staging package directory under
/home/elpa/.

The Org mode dailies are also fetched and added by the script
admin/org-synch.sh, run as a cron job.

This file is part of GNU Emacs.

GNU Emacs is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

GNU Emacs is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.

Local variables:
mode: outline
paragraph-separate: "[ ]*$"
end:

Re: Adding async package to GNU ELPA

[Michael Albinus]

Artur Malabarba <bruce.connor.am@gmail.com> writes:

> See the README file in elpa.git. I've pasted the file below for your
> convenience.
> The "External branches" section explains how to use subtrees or a
> separate branch.

Thanks. I've added async as subtree package, as explained. When
compiling, I get the following error:

--8<---------------cut here---------------start------------->8---
~/src/elpa> make
Generating autoloads for packages/async/async-autoloads.el
Missing ‘version’ header
GNUmakefile:132: recipe for target 'packages/async/async-autoloads.el' failed
make: *** [packages/async/async-autoloads.el] Error 255
--8<---------------cut here---------------end--------------->8---

The async package has an own async-pkg.el. But this doesn't seem to be
taken into account; instead a ;;; Version: header in async.el seems to
be needed. I could fix it, but I don't want to touch the inherited
files. What else could I do?

Best regards, Michael.

Re: Adding async package to GNU ELPA

[Artur Malabarba]

> --8<---------------cut here---------------start------------->8---
> ~/src/elpa> make
> Generating autoloads for packages/async/async-autoloads.el
> Missing ‘version’ header
> GNUmakefile:132: recipe for target 'packages/async/async-autoloads.el' failed
> make: *** [packages/async/async-autoloads.el] Error 255
> --8<---------------cut here---------------end--------------->8---
>
> The async package has an own async-pkg.el. But this doesn't seem to be
> taken into account; instead a ;;; Version: header in async.el seems to
> be needed. I could fix it, but I don't want to touch the inherited
> files. What else could I do?

The package archives take care of creating the -pkg.el file for you,
any reason why you want to provide one yourself?

Re: Adding async package to GNU ELPA

[Michael Albinus]

Artur Malabarba <bruce.connor.am@gmail.com> writes:

>> The async package has an own async-pkg.el. But this doesn't seem to be
>> taken into account; instead a ;;; Version: header in async.el seems to
>> be needed. I could fix it, but I don't want to touch the inherited
>> files. What else could I do?
>
> The package archives take care of creating the -pkg.el file for you,
> any reason why you want to provide one yourself?

It's not about me. I have added John's async package from its git
repository as subtree. There are async.el w/o Version: header, and
async-pkg.el. If possible, I would like to avoid modifying them locally.

Best regards, Michael.

Re: Adding async package to GNU ELPA

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 461 bytes --]

On 30 Dec 2015 7:31 pm, "Michael Albinus" <michael.albinus@gmx.de> wrote:
> >
> > The package archives take care of creating the -pkg.el file for you,
> > any reason why you want to provide one yourself?
>
> It's not about me. I have added John's async package from its git
> repository as subtree.

Ah then let's pass this question on to him. :-)

John?

The package archives take care of creating the -pkg.el file, any reason why
async provides one manually?

[-- Attachment #2: Type: text/html, Size: 659 bytes --]

Re: Adding async package to GNU ELPA

[John Wiegley]

>>>>> Artur Malabarba <bruce.connor.am@gmail.com> writes:

> The package archives take care of creating the -pkg.el file, any reason why
> async provides one manually?

I have no idea; I think Thierry may have added that file. We should drop it,
it sounds like.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Adding async package to GNU ELPA

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 443 bytes --]

On 31 Dec 2015 2:03 pm, "John Wiegley" <jwiegley@gmail.com> wrote:
> > The package archives take care of creating the -pkg.el file, any reason
why
> > async provides one manually?
>
> I have no idea; I think Thierry may have added that file. We should drop
it,
> it sounds like.

Then I guess the solution is to delete this file from the source repo. John
or Thierry can do that, or Michael can open a PR for that, linking to this
discussion.

[-- Attachment #2: Type: text/html, Size: 573 bytes --]

Re: Adding async package to GNU ELPA

[Michael Albinus]

Artur Malabarba <bruce.connor.am@gmail.com> writes:

> On 31 Dec 2015 2:03 pm, "John Wiegley" <jwiegley@gmail.com> wrote:
>> > The package archives take care of creating the -pkg.el file, any
> reason why
>> > async provides one manually?
>>
>> I have no idea; I think Thierry may have added that file. We should
> drop it,
>> it sounds like.
>
> Then I guess the solution is to delete this file from the source repo.
> John or Thierry can do that, or Michael can open a PR for that,
> linking to this discussion.

And please add a ;;; Version: header to async.el. I don't believe we
need a bug report ...

Best regards, Michael.

Re: Adding async package to GNU ELPA

[Thierry Volpiatto]

BTW, my contribution to async will ends here if it is now on ELPA, as I
have not write access there...

-- 
Thierry
https://emacs-helm.github.io/helm/

Re: Adding async package to GNU ELPA

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 286 bytes --]

On 31 Dec 2015 5:53 pm, "Thierry Volpiatto" <thierry.volpiatto@gmail.com>
wrote:
>
>
> BTW, my contribution to async will ends here if it is now on ELPA, as I
> have not write access there...

Michael offered to push to Elpa, so that development of this package would
remain on github.

[-- Attachment #2: Type: text/html, Size: 413 bytes --]

Re: Adding async package to GNU ELPA

[Thierry Volpiatto]

Artur Malabarba <bruce.connor.am@gmail.com> writes:

> On 31 Dec 2015 5:48 pm, "Thierry Volpiatto" <thierry.volpiatto@gmail.com> wrote:
>> >
>> > Then I guess the solution is to delete this file from the source
>> > repo. John or Thierry can do that, or Michael can open a PR for that,
>> > linking to this discussion.
>>
>> AFAIK the *-pkg.el file is needed for multi files packages, at least on
>> MELPA.
>
> It's not. :-) I have several multi file packages on Melpa and I've never written a -pkg file.
>
>> Anyway as I already said, it is silly to write informations in source file,
>> then parse this file
>> and then write a *-pkg.el file, we
>> (the author(s) of package) have better time providing directly the *-pkg.el file
>> with all info inside.
>
> I never had any issue with it, and I've never heard other package authors complain.

Of course, but I said it is bad to have to insert informations in source
files when we can add a *-pkg.el file especially for this.
I guess you are doing that because some code on ELPA expect informations
in source file headers and then create the *-pkg.el file from this, and
IMO this should be changed for the benefit of ELPA.

>> (which one ?)
>
> The file with the same name as the package (in this case, async.el).

Not necessarily, the contrary is also possible:

You can have a package named foo-core with 3 source files:

foo
bar
baz
and one file named foo-core-pkg.el

Like helm.

And it is a nice feature, I don't know if ELPA support this, probably not.

-- 
Thierry
https://emacs-helm.github.io/helm/

Re: Adding async package to GNU ELPA

[Michael Albinus]

Thierry Volpiatto <thierry.volpiatto@gmail.com> writes:

> Of course, but I said it is bad to have to insert informations in source
> files when we can add a *-pkg.el file especially for this.
> I guess you are doing that because some code on ELPA expect informations
> in source file headers and then create the *-pkg.el file from this, and
> IMO this should be changed for the benefit of ELPA.
>
>>> (which one ?)
>>
>> The file with the same name as the package (in this case, async.el).
>
> Not necessarily, the contrary is also possible:
>
> You can have a package named foo-core with 3 source files:
>
> foo
> bar
> baz
> and one file named foo-core-pkg.el
>
> Like helm.
>
> And it is a nice feature, I don't know if ELPA support this, probably not.

I would scond this proposal (providing an own *-pkg.el file). I'm
thinking about adding Tramp to ELPA, and tramp.el does not carry version
related information. A file tramp-pkg.el would solve this issue for
Tramp.

Best regards, Michael.

Re: Adding async package to GNU ELPA

[Artur Malabarba]

> I would scond this proposal (providing an own *-pkg.el file). I'm
> thinking about adding Tramp to ELPA, and tramp.el does not carry version
> related information. A file tramp-pkg.el would solve this issue for
> Tramp.
>
> Best regards, Michael.

Adding a version header to tramp.el would also solve this "issue", and
wouldn't involve extra work on anyone. :-)

Re: Adding async package to GNU ELPA

[Michael Albinus]

Artur Malabarba <bruce.connor.am@gmail.com> writes:

> Adding a version header to tramp.el would also solve this "issue", and
> wouldn't involve extra work on anyone. :-)

Tramp needs its version string at different places. Therefore, the
version is kept only in configure.ac. During the run of ./configure, the
files trampver.el and trampver.texi are generated. An additional file
tramp-pkg.el could be generated as well.

Alternatively, the package manager could retrieve the Version: header
from another file but tramp.el. trampver.el carries it already.

Best regards, Michael.

Re: Adding async package to GNU ELPA

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 474 bytes --]

On Jan 4, 2016 5:35 AM, "Michael Albinus" <michael.albinus@gmx.de> wrote:
>
> Tramp needs its version string at different places. Therefore, the
> version is kept only in configure.ac.

Would it be possible to keep it in tramp.el instead, and retrieve it using
`lm-version'?

> Alternatively, the package manager could retrieve the Version: header
> from another file but tramp.el. trampver.el carries it already.

Indeed. This might even be possible already, I'm not sure.

[-- Attachment #2: Type: text/html, Size: 678 bytes --]

Re: Adding async package to GNU ELPA

[Michael Albinus]

Artur Malabarba <bruce.connor.am@gmail.com> writes:

>> Tramp needs its version string at different places. Therefore, the
>> version is kept only in configure.ac.
>
> Would it be possible to keep it in tramp.el instead, and retrieve it
> using `lm-version'?

No. The whole idea of trampver.el is to keep the version string away
from tramp.el. If the version string would be generated into
tramp.el, the development must be applied in tramp.el.in. Not applicable.

>> Alternatively, the package manager could retrieve the Version: header
>> from another file but tramp.el. trampver.el carries it already.
>
> Indeed. This might even be possible already, I'm not sure.

Some days ago, I've patched finder.el to support this. Since then, Tramp
is visible as built-in package. Would be great, if ELPA could support
this as well, if the approach with an own <foo>-pkg.el does not work.
However, I believe the approach with <foo>-pkg.el is superior.

Best regards, Michael.

Re: Adding async package to GNU ELPA

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 1016 bytes --]

On Jan 7, 2016 11:20 AM, "Michael Albinus" <michael.albinus@gmx.de> wrote:
>
> Artur Malabarba <bruce.connor.am@gmail.com> writes:
>
> >> Tramp needs its version string at different places. Therefore, the
> >> version is kept only in configure.ac.
> >
> > Would it be possible to keep it in tramp.el instead, and retrieve it
> > using `lm-version'?
>
> No. The whole idea of trampver.el is to keep the version string away
> from tramp.el. If the version string would be generated into
> tramp.el, the development must be applied in tramp.el.in. Not applicable.

I'm not talking about generating the version string into tramp.el.

IIUC, you manually write the version string only in configure.ac. (did I
understand this correctly?)

What I'm saying is: could you write the version string only in tramp.el,
instead of in configure.ac?
You could easily retrieve the version during ./configure with a simple
(grep | sed) or with lm-version (in elisp).

This way tramp.el would still be static (no need for tramp.el.in).

[-- Attachment #2: Type: text/html, Size: 1502 bytes --]

Re: Adding async package to GNU ELPA

[Michael Albinus]

Artur Malabarba <bruce.connor.am@gmail.com> writes:

>> No. The whole idea of trampver.el is to keep the version string away
>> from tramp.el. If the version string would be generated into
>> tramp.el, the development must be applied in tramp.el.in. Not
> applicable.
>
> I'm not talking about generating the version string into tramp.el. 
>
> IIUC, you manually write the version string only in configure.ac. (did
> I understand this correctly?) 

Yes.

> What I'm saying is: could you write the version string only in
> tramp.el, instead of in configure.ac?
> You could easily retrieve the version during ./configure with a simple
> (grep | sed) or with lm-version (in elisp). 

As I said, the version string is used at several places in different
files of Tramp. This must be consistent, it is essential for Tramp.

Your propasal would mean to write an own expansion mechanism, instead of
using AC_SUBST.

Too much trouble. Since there is no urgent need, I'll wait how it
evolves with the request from Thierry (do not generate <foo>-pkg.el
mandatory). I believe this would be the best, consistent solution.

Best regards, Michael.

Re: Adding async package to GNU ELPA

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 255 bytes --]

On Jan 7, 2016 6:00 PM, "Michael Albinus" <michael.albinus@gmx.de> wrote:
>
> Your propasal would mean to write an own expansion mechanism, instead of
> using AC_SUBST.

I see. That would be quite a bit of trouble indeed.

I'll ponder on this a bit more.

[-- Attachment #2: Type: text/html, Size: 390 bytes --]

Re: Adding async package to GNU ELPA

[Andreas Schwab]

Michael Albinus <michael.albinus@gmx.de> writes:

> Your propasal would mean to write an own expansion mechanism, instead of
> using AC_SUBST.

You can still use AC_SUBST.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Adding async package to GNU ELPA

[Artur Malabarba]

Andreas Schwab <schwab@linux-m68k.org> writes:

> You can still use AC_SUBST.

Thanks Andreas. I really know nothing about configure.ac and the likes,
so it would be great if you could elaborate on this for the benefit of
Michael and the others.

Re: Adding async package to GNU ELPA

[Andreas Schwab]

Artur Malabarba <bruce.connor.am@gmail.com> writes:

> Andreas Schwab <schwab@linux-m68k.org> writes:
>
>> You can still use AC_SUBST.
>
> Thanks Andreas. I really know nothing about configure.ac and the likes,
> so it would be great if you could elaborate on this for the benefit of
> Michael and the others.

Just continue to use it.  If you have any problem with it, please
elaborate.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Adding async package to GNU ELPA

[Thierry Volpiatto]

Artur Malabarba <bruce.connor.am@gmail.com> writes:

>> I would scond this proposal (providing an own *-pkg.el file). I'm
>> thinking about adding Tramp to ELPA, and tramp.el does not carry version
>> related information. A file tramp-pkg.el would solve this issue for
>> Tramp.
>>
>> Best regards, Michael.
>
> Adding a version header to tramp.el would also solve this "issue", and
> wouldn't involve extra work on anyone. :-)

I don't see why ELPA shouldn't have the same features as MELPA which is
supporting very well *-pkg.el files.

-- 
Thierry

Re: Adding async package to GNU ELPA

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 382 bytes --]

On Jan 4, 2016 5:50 AM, "Thierry Volpiatto" <thierry.volpiatto@gmail.com>
wrote:
> I don't see why ELPA shouldn't have the same features as MELPA which is
> supporting very well *-pkg.el files.

I never said it shouldn't. :-)
It just doesn't support this ATM.

Although I am hesitant to add support for a second syntax, as this kind of
thing tends to add to the maintenance burden.

[-- Attachment #2: Type: text/html, Size: 536 bytes --]

Re: Adding async package to GNU ELPA

[Artur Malabarba]

Michael Albinus <michael.albinus@gmx.de> writes:

> Thierry Volpiatto <thierry.volpiatto@gmail.com> writes:
>
>> Of course, but I said it is bad to have to insert informations in source
>> files when we can add a *-pkg.el file especially for this.
>> I guess you are doing that because some code on ELPA expect informations
>> in source file headers and then create the *-pkg.el file from this, and
>> IMO this should be changed for the benefit of ELPA.
>
> I would scond this proposal (providing an own *-pkg.el file). I'm
> thinking about adding Tramp to ELPA, and tramp.el does not carry version
> related information. A file tramp-pkg.el would solve this issue for
> Tramp.

Ok, I've had a look at Gelpa's admin/archive-contents.el, and I think
this can be done without introducing complexity.

Right now, it is structured more or less like this:
1. Read information from package headers;
2. Do some stuff with that information, one of which is to generate the
-pkg.el file.

Ideally, it would be reworked like this:
1. If absent, generate the -pkg.el file from the package headers (this
is the only moment where package headers would be read);
2. Read information from the -pkg.el file;
3. Do some stuff with that information.


That said, I can't be the one to do it for the foreseeable future.
Therefore, unless someone else steps up, the quickest way to get any of
the aforementioned packages on Gelpa is for them to comply with the
current framework.

Cheers,
Artur

Re: [elpa] master 88578a4: Increase the default number of hits

[Lars Magne Ingebrigtsen]

Michael Albinus <michael.albinus@gmx.de> writes:

> I've committed a patch, which allows to retrieve the bugs
> asynchronously.

Nice!

But I think something went wrong in the charset dept somewhere.  I'm
getting listings like this:

22262 normal		   [Tadeusz J Liszka	   ] 24.5; case-fold-search breaks fortran comment regexp
22263 normal,done	   [Clément Pit--Claudel]	Failed assertion (no good repro yet)
22264 normal,moreinfo	   [Clément Pit--Claudel]	Another assertion failed (without a clear repro)
22265 normal		   [Federico Beffa	   ] 24.5; xscheme minibuffer

Which looks like doubly-encoded utf-8, perhaps?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: [elpa] master 88578a4: Increase the default number of hits

[Michael Albinus]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> But I think something went wrong in the charset dept somewhere.  I'm
> getting listings like this:
>
> 22262 normal [Tadeusz J Liszka ] 24.5; case-fold-search breaks fortran
> comment regexp
> 22263 normal,done [Clément Pit--Claudel] Failed assertion (no good
> repro yet)
> 22264 normal,moreinfo [Clément Pit--Claudel] Another assertion
> failed (without a clear repro)
> 22265 normal		   [Federico Beffa	   ] 24.5; xscheme minibuffer
>
> Which looks like doubly-encoded utf-8, perhaps?

Same here. I'm not aware having changed something in this area. I
believe I have seen this a while ago. Then this was fixed (special
characters were displayed properly), and now this happened again.

Hmm. I'll try to debug when time permits.

Best regards, Michael.

Re: [elpa] master 88578a4: Increase the default number of hits

[Michael Albinus]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> But I think something went wrong in the charset dept somewhere.  I'm
> getting listings like this:
>
> 22263 normal,done [Clément Pit--Claudel] Failed assertion (no good
> repro yet)
>
> Which looks like doubly-encoded utf-8, perhaps?

Well, this is a side-effect of using the async package (which spawns a
new Emacs process doing its job).

I've just realized that soap-client 3.0 offers a soap-invoke-async
function. Maybe I'll give it a try.

Best regards, Michael.

Re: [elpa] master 88578a4: Increase the default number of hits

[Lars Magne Ingebrigtsen]

Michael Albinus <michael.albinus@gmx.de> writes:

> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>
>> But I think something went wrong in the charset dept somewhere.  I'm
>> getting listings like this:
>>
>> 22263 normal,done [Clément Pit--Claudel] Failed assertion (no good
>> repro yet)
>>
>> Which looks like doubly-encoded utf-8, perhaps?
>
> Well, this is a side-effect of using the async package (which spawns a
> new Emacs process doing its job).

This is a bug in async, then?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: [elpa] master 88578a4: Increase the default number of hits

[Michael Albinus]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

>>> Which looks like doubly-encoded utf-8, perhaps?
>>
>> Well, this is a side-effect of using the async package (which spawns a
>> new Emacs process doing its job).
>
> This is a bug in async, then?

Don't know. I'll write a bug report when I have the feeling it is ...
(say: I have a small recipe which shows this).

Best regards, Michael.

Re: [elpa] master 88578a4: Increase the default number of hits

[Michael Albinus]

Glenn Morris <rgm@gnu.org> writes:

>>> In fact, there really should be a hard limit that the server imposes.
>>> (Michael, could you look at implementing that?)
>>
>> Will check. What would be an acceptable hard limit? 5.000?
>
> Personally I'd want to see the hard limit set at 1000.

I've added  a hard limit of 1000 in Debbugs::SOAP::get_status. I'm
asking myself whether we shall make this configurable, and whether we
shall apply it in other functions as well. But this would mean, that our
Debbugs package diverges more and more from the original one; it would
be better to merge our changes back to the Debian source, or to handle
it as a real fork.

Best regards, Michael.

Re: [elpa] master 88578a4: Increase the default number of hits

[Glenn Morris]

Michael Albinus wrote:

> But this would mean, that our Debbugs package diverges more and more
> from the original one; it would be better to merge our changes back to
> the Debian source, or to handle it as a real fork.

That ship sailed a long, long time ago.
Our debbugs differs all over the place.
A change in soap is the least of our worries.

Re: [elpa] master 88578a4: Increase the default number of hits

[Lars Ingebrigtsen]

Glenn Morris <rgm@gnu.org> writes:

> Because the more results you fetch at once, the more memory it uses on
> the server, thus increasing the likelihood of a bunch of simultaneous
> requests causing an out-of-memory condition. I did already explain this,
> not sure why it's hard to get across.

It hard to get across because it's just so odd.  :-)  Since this is an
obviously useful functionality that would make Emacs bug triage easier,
why can't we have the VM where this is running provisioned with more
RAM?

I can send money to the FSF for 32GB extra RAM if that helps.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: [elpa] master 88578a4: Increase the default number of hits

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> I can send money to the FSF for 32GB extra RAM if that helps.

(Sorry, I meant 128GB RAM.  This is 2015, after all.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: [elpa] master 88578a4: Increase the default number of hits

[Glenn Morris]

Lars Ingebrigtsen wrote:

> It hard to get across because it's just so odd.  :-)  Since this is an
> obviously useful functionality that would make Emacs bug triage easier,
> why can't we have the VM where this is running provisioned with more
> RAM?
>
> I can send money to the FSF for 32GB extra RAM if that helps.

That's a kind offer.

If enough people tell me that this issue is affecting them, I will ask
FSF if we can have more memory. At the moment it seems to me that you
have an atypical way of working (no offence meant!). Given that all
resources are shared, I don't like to ask for more than we actually
need.

Re: [elpa] master 88578a4: Increase the default number of hits

[Lars Ingebrigtsen]

Glenn Morris <rgm@gnu.org> writes:

> If enough people tell me that this issue is affecting them, I will ask
> FSF if we can have more memory. At the moment it seems to me that you
> have an atypical way of working (no offence meant!).

Well, not a lot of people are using debbugs-gnu at all.  And I suspect
that's because the defaults are less than ideal.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: [elpa] master 88578a4: Increase the default number of hits

[Glenn Morris]

Lars Ingebrigtsen wrote:

> Glenn Morris <rgm@gnu.org> writes:
>
>> If enough people tell me that this issue is affecting them, I will ask
>> FSF if we can have more memory. At the moment it seems to me that you
>> have an atypical way of working (no offence meant!).
>
> Well, not a lot of people are using debbugs-gnu at all.  And I suspect
> that's because the defaults are less than ideal.  :-)

I've got to admit I don't use it, I use the web interface to search.

I don't want to stand in the way of people getting work done.
If it was my own server, I would add more memory.
But with it being an FSF resource, I'm reluctant to ask for a bigger
share of the pie. Maybe this is a silly attitude? For all I know,
they have memory sitting around unused...

So I guess I'll defer to whatever the group wisdom decides.

But it sounds like maybe Michael solved this in software in the meantime?

BTW, if there's a way to limit the number of simultaneous SOAP
connections, that should probably be implemented on the server too.

Another random thought - is it possible to implement some local cache,
so only changed bugs are fetched? This could be a silly or impractical
idea, feel free to ignore it - I'm totally unfamiliar with the implementation.

In summary: shout at me if I need to do a thing.

Re: [elpa] master 88578a4: Increase the default number of hits

[Michael Albinus]

Glenn Morris <rgm@gnu.org> writes:

> BTW, if there's a way to limit the number of simultaneous SOAP
> connections, that should probably be implemented on the server too.

Will check.

> Another random thought - is it possible to implement some local cache,
> so only changed bugs are fetched? This could be a silly or impractical
> idea, feel free to ignore it - I'm totally unfamiliar with the implementation.

Every bug carries a "last_modified" attribute, so it shall be
possible. I put it on my TODO list.

(Unfortunately, TODO is rapidly growing these days. So I cannot promise
a due date)

Best regards, Michael.

Re: [elpa] master 88578a4: Increase the default number of hits

[Glenn Morris]

I noticed today many out-of-memory errors on the machine, apparently due
to soap.cgi. This is a PITA, since oomkiller kills off random things,
which can include mail handling.

I rebooted the machine to recover from this, but at time of writing it
has not come back up. All I can do at this point is mail the FSF
sysadmin, and due to the season I would not be surprised if it takes a
while to sort out.

Bah, humbug.

Re: [elpa] master 88578a4: Increase the default number of hits

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Date: Thu, 31 Dec 2015 21:35:29 -0500
> Cc: Lars Ingebrigtsen <larsi@gnus.org>, emacs-devel@gnu.org
> 
> I rebooted the machine to recover from this, but at time of writing it
> has not come back up. All I can do at this point is mail the FSF
> sysadmin, and due to the season I would not be surprised if it takes a
> while to sort out.

That's too bad (it's not back up yet as of this writing, and the FSF
status page doesn't even mention the problem).  And it's a PITA when
that happens, because all the bug-related discussions get lost without
being archived with the bugs.  Given these problems, I think we should
ask the admins to look into what happens on that machine when it's
rebooted, so that these incidents wouldn't be so painful.

Re: [elpa] master 88578a4: Increase the default number of hits

[Michael Albinus]

Glenn Morris <rgm@gnu.org> writes:

> I noticed today many out-of-memory errors on the machine, apparently due
> to soap.cgi. This is a PITA, since oomkiller kills off random things,
> which can include mail handling.

I fear I am guilty. During my tests yesterday, one test ran out of
control (a loop counter didn't change properly), and many soap requests
were sent. I'm very sorry for this!

Hmm, is there a chance to get a reduced version of the debbugs server I
could run as VM? This could avoid such clashes in the future.

> Bah, humbug.

Best regards, Michael.

Re: [elpa] master 88578a4: Increase the default number of hits

[Glenn Morris]

Michael Albinus wrote:

>> I noticed today many out-of-memory errors on the machine, apparently due
>> to soap.cgi. This is a PITA, since oomkiller kills off random things,
>> which can include mail handling.
>
> I fear I am guilty. During my tests yesterday, one test ran out of
> control (a loop counter didn't change properly), and many soap requests
> were sent. I'm very sorry for this!

OK, it's good to have a scapegoat, I mean, explanation, for what happened. :)
It seems to be back as of 10 mins ago.

> Hmm, is there a chance to get a reduced version of the debbugs server I
> could run as VM? This could avoid such clashes in the future.

Good idea (something I should have done myself ages ago).
Just set up a minimal Debian stable VM, then copy the list of installed
packages from d.g.org. Don't install the standard debbugs package,
instead just rsync our version (files live in various locations).
IIRC the only things that are customized are exim, mailman, and apache -
look in /etc for settings.

Re: [elpa] master 88578a4: Increase the default number of hits

[Lars Magne Ingebrigtsen]

I've been trying to debug the charset issue, but I'm failing because I
can't get it to debug properly.

I think the issue may be with encoding as utf-8 (in a multibyte buffer),
and then base64-ing the result, which will most likely make base64
encode each character as a series of bytes, and it's already been
encoded as utf-8, so...  double.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: [elpa] master 88578a4: Increase the default number of hits

[Lars Magne Ingebrigtsen]

Oh, if anybody wants a quick and easy test case that displays the error
in async.el:

(async-get (async-start
            `(lambda ()
               (load ,(locate-library "debbugs"))
               (debbugs-get-status 22285))))
=> (((source . "unknown") (originator . "Clément Pit--Claudel <clement.pitclaudel@live.com>") (tags) (package "emacs") (id . 22285) (severity . "normal") (archived) (summary) (msgid . "<5686A38C.1010002@live.com>") (location . "db-h") (blocks) (subject . "isearch and repeat do no play well together (overlays are not cleared after repeating isearch-forward-exit-minibuffer)") (unarchived) (affects) (log_modified . 1451699523) (last_modified . 1451699523) (mergedwith) (fixed_date) (fixed) (date . 1451699522) (found) (forwarded) (owner) (done) (keywords) (pending . "pending") (found_versions) (found_date) (bug_num . 22285) (blockedby) (fixed_versions)))               

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: [elpa] master 88578a4: Increase the default number of hits

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Date: Sat, 02 Jan 2016 07:47:13 +0100
> 
> Oh, if anybody wants a quick and easy test case that displays the error
> in async.el:
> 
> (async-get (async-start
>             `(lambda ()
>                (load ,(locate-library "debbugs"))
>                (debbugs-get-status 22285))))
> => (((source . "unknown") (originator . "Clément Pit--Claudel <clement.pitclaudel@live.com>") (tags) (package "emacs") (id . 22285) (severity . "normal") (archived) (summary) (msgid . "<5686A38C.1010002@live.com>") (location . "db-h") (blocks) (subject . "isearch and repeat do no play well together (overlays are not cleared after repeating isearch-forward-exit-minibuffer)") (unarchived) (affects) (log_modified . 1451699523) (last_modified . 1451699523) (mergedwith) (fixed_date) (fixed) (date . 1451699522) (found) (forwarded) (owner) (done) (keywords) (pending . "pending") (found_versions) (found_date) (bug_num . 22285) (blockedby) (fixed_versions)))               

Any idea why this is different with async than without it?

Re: [elpa] master 88578a4: Increase the default number of hits

[Lars Magne Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Any idea why this is different with async than without it?

It's the inter-process communication and the base64 that's affecting
it.  I think.  Possibly.  (async.el works by starting a separate Emacs,
and then talking to it over a socket...)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: [elpa] master 88578a4: Increase the default number of hits

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Sat, 02 Jan 2016 10:01:10 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Any idea why this is different with async than without it?
> 
> It's the inter-process communication and the base64 that's affecting
> it.  I think.  Possibly.  (async.el works by starting a separate Emacs,
> and then talking to it over a socket...)

How is process-coding-system set up for communicating with the
subsidiary Emacs?  it should be no-conversion, I think, at least in
the input direction (i.e. from the subsidiary Emacs to the parent
one).

Re: [elpa] master 88578a4: Increase the default number of hits

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

> How is process-coding-system set up for communicating with the
> subsidiary Emacs?  it should be no-conversion, I think, at least in
> the input direction (i.e. from the subsidiary Emacs to the parent
> one).

Since all inter-process communication is base64 encoded (thus
pure-ASCII) any coding system will do.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: [elpa] master 88578a4: Increase the default number of hits

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: Lars Magne Ingebrigtsen <larsi@gnus.org>,  emacs-devel@gnu.org
> Date: Sat, 02 Jan 2016 11:05:28 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > How is process-coding-system set up for communicating with the
> > subsidiary Emacs?  it should be no-conversion, I think, at least in
> > the input direction (i.e. from the subsidiary Emacs to the parent
> > one).
> 
> Since all inter-process communication is base64 encoded

Thanks, I didn't know that.

> (thus pure-ASCII) any coding system will do.

Right.

So the problem is likely in the receiving part: it should make the
buffer with the base64 encoded stuff unibyte before decoding it.

Re: [elpa] master 88578a4: Increase the default number of hits

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

> So the problem is likely in the receiving part: it should make the
> buffer with the base64 encoded stuff unibyte before decoding it.

It's not that easy, a simple test works as expected.

(async-start (lambda () (format "%c" 233))
             (lambda (result) (message "%s" result)))

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: [elpa] master 88578a4: Increase the default number of hits

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: larsi@gnus.org,  emacs-devel@gnu.org
> Date: Sat, 02 Jan 2016 12:08:37 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > So the problem is likely in the receiving part: it should make the
> > buffer with the base64 encoded stuff unibyte before decoding it.
> 
> It's not that easy, a simple test works as expected.
> 
> (async-start (lambda () (format "%c" 233))
>              (lambda (result) (message "%s" result)))

Does the first lambda cause a UTF-8 encoding be produced somewhere,
before it's base64 encoded?  If not, this is not a faithful emulation
of what's going on.

Re: [elpa] master 88578a4: Increase the default number of hits

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

> Does the first lambda cause a UTF-8 encoding be produced somewhere,
> before it's base64 encoded?

The async communication is based on printed representation of lisp
expressions, so this question doesn't make sense.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: [elpa] master 88578a4: Increase the default number of hits

[Andreas Schwab]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Oh, if anybody wants a quick and easy test case that displays the error
> in async.el:
>
> (async-get (async-start
>             `(lambda ()
>                (load ,(locate-library "debbugs"))
>                (debbugs-get-status 22285))))
> => (((source . "unknown") (originator . "Clément Pit--Claudel <clement.pitclaudel@live.com>") (tags) (package "emacs") (id . 22285) (severity . "normal") (archived) (summary) (msgid . "<5686A38C.1010002@live.com>") (location . "db-h") (blocks) (subject . "isearch and repeat do no play well together (overlays are not cleared after repeating isearch-forward-exit-minibuffer)") (unarchived) (affects) (log_modified . 1451699523) (last_modified . 1451699523) (mergedwith) (fixed_date) (fixed) (date . 1451699522) (found) (forwarded) (owner) (done) (keywords) (pending . "pending") (found_versions) (found_date) (bug_num . 22285) (blockedby) (fixed_versions)))               

Try this patch:

diff --git a/packages/async/async.el b/packages/async/async.el
index 7286c48..c3b5dc3 100644
--- a/packages/async/async.el
+++ b/packages/async/async.el
@@ -129,7 +129,9 @@ as follows:
     (eval sexp)))
 
 (defun async--insert-sexp (sexp)
-  (let (print-level print-length)
+  (let (print-level
+	print-length
+	(print-escape-multibyte t))
     (prin1 sexp (current-buffer))
     ;; Just in case the string we're sending might contain EOF
     (encode-coding-region (point-min) (point-max) 'utf-8-unix)

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: [elpa] master 88578a4: Increase the default number of hits

[Michael Albinus]

Andreas Schwab <schwab@linux-m68k.org> writes:

> Try this patch:
>
> diff --git a/packages/async/async.el b/packages/async/async.el
> index 7286c48..c3b5dc3 100644
> --- a/packages/async/async.el
> +++ b/packages/async/async.el
> @@ -129,7 +129,9 @@ as follows:
>      (eval sexp)))
>  
>  (defun async--insert-sexp (sexp)
> -  (let (print-level print-length)
> +  (let (print-level
> +	print-length
> +	(print-escape-multibyte t))
>      (prin1 sexp (current-buffer))
>      ;; Just in case the string we're sending might contain EOF
>      (encode-coding-region (point-min) (point-max) 'utf-8-unix)

Unfortunately, it didn't work.

> Andreas.

Best regards, Michael.

Re: [elpa] master 88578a4: Increase the default number of hits

[Michael Albinus]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> I've been trying to debug the charset issue, but I'm failing because I
> can't get it to debug properly.
>
> I think the issue may be with encoding as utf-8 (in a multibyte buffer),
> and then base64-ing the result, which will most likely make base64
> encode each character as a series of bytes, and it's already been
> encoded as utf-8, so...  double.

I'll check later. Maybe you create a ticket, in order not to forget?

Thanks, and best regards, Michael.

Re: [elpa] master 88578a4: Increase the default number of hits

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Date: Sat, 02 Jan 2016 07:44:46 +0100
> Cc: Glenn Morris <rgm@gnu.org>, emacs-devel@gnu.org
> 
> I've been trying to debug the charset issue, but I'm failing because I
> can't get it to debug properly.
> 
> I think the issue may be with encoding as utf-8 (in a multibyte buffer),
> and then base64-ing the result, which will most likely make base64
> encode each character as a series of bytes, and it's already been
> encoded as utf-8, so...  double.

Did you try making that buffer unibyte, or maybe encoding into a
separate unibyte buffer?  Then base64 would run in a unibyte buffer,
as it should.

Re: [elpa] master 88578a4: Increase the default number of hits

[Lars Magne Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Did you try making that buffer unibyte, or maybe encoding into a
> separate unibyte buffer?  Then base64 would run in a unibyte buffer,
> as it should.

I just couldn't quite make sense of what async.el was doing.  I fiddled
with the stuff for half an hour, making stuff unibyte and not, and I
just couldn't get it to do what I wanted to.  Probably because I don't
understand quite what's going on in the "remote" Emacs, and how it's
loaded and started...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: [elpa] master 88578a4: Increase the default number of hits

[John Wiegley]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>> Did you try making that buffer unibyte, or maybe encoding into a separate
>> unibyte buffer? Then base64 would run in a unibyte buffer, as it should.

> I just couldn't quite make sense of what async.el was doing. I fiddled with
> the stuff for half an hour, making stuff unibyte and not, and I just
> couldn't get it to do what I wanted to. Probably because I don't understand
> quite what's going on in the "remote" Emacs, and how it's loaded and
> started...

It's quite possible the charset encoding has been wrong, since I didn't have
specific tests for this case. The creation of such tests would be a nice
outcome to Michael's debugging.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: [elpa] master 88578a4: Increase the default number of hits

[Lars Magne Ingebrigtsen]

John Wiegley <jwiegley@gmail.com> writes:

> It's quite possible the charset encoding has been wrong, since I didn't have
> specific tests for this case. The creation of such tests would be a nice
> outcome to Michael's debugging.

Here's one test:

(async-get (async-start `(lambda () "báh")))
=> "báh"

But I'm not sure whether it's testing wrong encoding during transmission
to the other Emacs or back again.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: [elpa] master 88578a4: Increase the default number of hits

[Andreas Schwab]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Here's one test:
>
> (async-get (async-start `(lambda () "báh")))
> => "báh"

This works for me.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: [elpa] master 88578a4: Increase the default number of hits

[Lars Magne Ingebrigtsen]

Andreas Schwab <schwab@linux-m68k.org> writes:

> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>
>> Here's one test:
>>
>> (async-get (async-start `(lambda () "báh")))
>> => "báh"
>
> This works for me.

This is with the latest trunk, with -Q, with async.el from ELPA git.
Hm...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: [elpa] master 88578a4: Increase the default number of hits

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  rgm@gnu.org,  michael.albinus@gmx.de,  emacs-devel@gnu.org
> Date: Sat, 02 Jan 2016 18:36:36 +0100
> 
> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
> 
> > Here's one test:
> >
> > (async-get (async-start `(lambda () "báh")))
> > => "báh"
> 
> This works for me.

Lars confessed at some point that he is overriding his locale's
settings, so maybe that is the reason.  Lars, is this in an Emacs
where you call set-locale-environment (or maybe it was
set-language-environment?) by hand?

Re: [elpa] master 88578a4: Increase the default number of hits

[Lars Magne Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Lars confessed at some point that he is overriding his locale's
> settings, so maybe that is the reason.  Lars, is this in an Emacs
> where you call set-locale-environment (or maybe it was
> set-language-environment?) by hand?

No, it's "emacs -Q"...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: [elpa] master 88578a4: Increase the default number of hits

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Cc: Andreas Schwab <schwab@linux-m68k.org>,  rgm@gnu.org,  michael.albinus@gmx.de,  emacs-devel@gnu.org
> Date: Sat, 02 Jan 2016 18:51:36 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Lars confessed at some point that he is overriding his locale's
> > settings, so maybe that is the reason.  Lars, is this in an Emacs
> > where you call set-locale-environment (or maybe it was
> > set-language-environment?) by hand?
> 
> No, it's "emacs -Q"...

Aha.  And your locale is...?

Re: [elpa] master 88578a4: Increase the default number of hits

[Lars Magne Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Aha.  And your locale is...?

Uhm...  is it this one?

---
locale-coding-system is a variable defined in ‘C source code’.
Its value is ‘iso-latin-1-unix’

Documentation:
Coding system to use with system messages.
Also used for decoding keyboard input on X Window system, and for
encoding standard output and error streams.
---

It should, in any case, be kinda irrelevant.  Only ASCII goes over the
socket...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: [elpa] master 88578a4: Increase the default number of hits

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Cc: rgm@gnu.org,  schwab@linux-m68k.org,  michael.albinus@gmx.de,  emacs-devel@gnu.org
> Date: Sun, 03 Jan 2016 10:07:33 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Aha.  And your locale is...?
> 
> Uhm...  is it this one?
> 
> ---
> locale-coding-system is a variable defined in ‘C source code’.
> Its value is ‘iso-latin-1-unix’

Close enough, thanks.

So my crystal ball says that Andreas has his value set to utf-8, which
is why you observe a different behavior than he does.

Re: [elpa] master 88578a4: Increase the default number of hits

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Date: Sat, 02 Jan 2016 18:28:11 +0100
> Cc: rgm@gnu.org, michael.albinus@gmx.de, emacs-devel@gnu.org
> 
> John Wiegley <jwiegley@gmail.com> writes:
> 
> > It's quite possible the charset encoding has been wrong, since I didn't have
> > specific tests for this case. The creation of such tests would be a nice
> > outcome to Michael's debugging.
> 
> Here's one test:
> 
> (async-get (async-start `(lambda () "báh")))
> => "báh"
> 
> But I'm not sure whether it's testing wrong encoding during transmission
> to the other Emacs or back again.  :-)

According to my testing, the problem is in the other direction, from
the subsidiary Emacs to the main one.  I see a few gotchas in the code
that can at least explain part of the problem.

John, is it okay to commit changes to 'async' directly to ELPA, or
would you like to see them in advance?

Thanks.

Re: [elpa] master 88578a4: Increase the default number of hits

[John Wiegley]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

> John, is it okay to commit changes to 'async' directly to ELPA, or would you
> like to see them in advance?

Feel free to commit fixes directly to ELPA.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: [elpa] master 88578a4: Increase the default number of hits

[Eli Zaretskii]

> From: John Wiegley <jwiegley@gmail.com>
> Cc: Lars Magne Ingebrigtsen <larsi@gnus.org>,  rgm@gnu.org,  michael.albinus@gmx.de,  emacs-devel@gnu.org
> Date: Wed, 06 Jan 2016 09:23:35 -0800
> 
> >>>>> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > John, is it okay to commit changes to 'async' directly to ELPA, or would you
> > like to see them in advance?
> 
> Feel free to commit fixes directly to ELPA.

OK, done.

Lars, please update both the emacs-25 branch and elpa, and try again
accessing debbugs bugs with non-ASCII characters (using emacs-25.0.50,
not 25.1.50 from master).  At least your simplified example:

  (async-get (async-start `(lambda () "báh")))

now works for me.  I don't think debbugs will, but maybe now the patch
proposed by Andreas will help.  If not, I will continue debugging this
later.

Re: [elpa] master 88578a4: Increase the default number of hits

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

> Lars, please update both the emacs-25 branch and elpa, and try again
> accessing debbugs bugs with non-ASCII characters (using emacs-25.0.50,
> not 25.1.50 from master).  At least your simplified example:
>
>   (async-get (async-start `(lambda () "báh")))
>
> now works for me.  I don't think debbugs will, but maybe now the patch
> proposed by Andreas will help.  If not, I will continue debugging this
> later.

The problem is that unibyte strings are received as multibyte strings.

ELISP> (async-get (async-start (lambda () "\303")))
"Ã"

Binding print-escape-nonascii fixes that.  I have also enabled
print-circle to avoid problems with circular data structures.

(There was also a bug in soap-client in the way it handles a
base64Binary data element, which was the reason the unibyte string was
returned in the debbugs query in the first place.)

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: [elpa] master 88578a4: Increase the default number of hits

[Lars Magne Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Lars, please update both the emacs-25 branch and elpa, and try again
> accessing debbugs bugs with non-ASCII characters (using emacs-25.0.50,
> not 25.1.50 from master).  At least your simplified example:
>
>   (async-get (async-start `(lambda () "báh")))
>
> now works for me.

Yup; this now works for me.  Thanks.

> I don't think debbugs will, but maybe now the patch proposed by
> Andreas will help.  If not, I will continue debugging this later.

After Andreas' fix, debbugs works fine, too.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: [elpa] master 88578a4: Increase the default number of hits

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Cc: John Wiegley <jwiegley@gmail.com>,  rgm@gnu.org,  michael.albinus@gmx.de,  emacs-devel@gnu.org
> Date: Thu, 07 Jan 2016 05:16:47 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Lars, please update both the emacs-25 branch and elpa, and try again
> > accessing debbugs bugs with non-ASCII characters (using emacs-25.0.50,
> > not 25.1.50 from master).  At least your simplified example:
> >
> >   (async-get (async-start `(lambda () "báh")))
> >
> > now works for me.
> 
> Yup; this now works for me.  Thanks.
> 
> > I don't think debbugs will, but maybe now the patch proposed by
> > Andreas will help.  If not, I will continue debugging this later.
> 
> After Andreas' fix, debbugs works fine, too.

Great, thanks.

Re: [elpa] master 88578a4: Increase the default number of hits

[Michael Albinus]

Michael Albinus <michael.albinus@gmx.de> writes:

> Glenn Morris <rgm@gnu.org> writes:
>
>> Another random thought - is it possible to implement some local cache,
>> so only changed bugs are fetched? This could be a silly or impractical
>> idea, feel free to ignore it - I'm totally unfamiliar with the implementation.
>
> Every bug carries a "last_modified" attribute, so it shall be
> possible. I put it on my TODO list.

It's implemented now. Cached bugs expire after an hour, or when a
control message is sent from debbugs-gnu.el.

Let's see how it works. The expiration time could be adapted in
`debbugs-cache-expiry'; maybe we need to tune it further.

Best regards, Michael.

Re: [elpa] master 88578a4: Increase the default number of hits

[Lars Magne Ingebrigtsen]

Michael Albinus <michael.albinus@gmx.de> writes:

> It's implemented now. Cached bugs expire after an hour, or when a
> control message is sent from debbugs-gnu.el.

If somebody else changes the status of the bugs, will it show a stale
status?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: [elpa] master 88578a4: Increase the default number of hits

[Michael Albinus]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

>> It's implemented now. Cached bugs expire after an hour, or when a
>> control message is sent from debbugs-gnu.el.
>
> If somebody else changes the status of the bugs, will it show a stale
> status?

Yes. Don't know how often this happens.

What I'm thinking about is a command which disables the cache
temporarily, while retrieving bugs. For example "C-u g", but this would
retrieve *all* bugs in the buffer (could be some thousands). This is too
coarse-grained.

Best regards, Michael.

Re: [elpa] master 88578a4: Increase the default number of hits

[Lars Magne Ingebrigtsen]

Michael Albinus <michael.albinus@gmx.de> writes:

> What I'm thinking about is a command which disables the cache
> temporarily, while retrieving bugs. For example "C-u g", but this would
> retrieve *all* bugs in the buffer (could be some thousands). This is too
> coarse-grained.

There's no way to ask debbugs "what bug reports have changed since
<time>"?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: [elpa] master 88578a4: Increase the default number of hits

[Michael Albinus]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Michael Albinus <michael.albinus@gmx.de> writes:
>
>> What I'm thinking about is a command which disables the cache
>> temporarily, while retrieving bugs. For example "C-u g", but this would
>> retrieve *all* bugs in the buffer (could be some thousands). This is too
>> coarse-grained.
>
> There's no way to ask debbugs "what bug reports have changed since
> <time>"?

Unfortunately, not. The SOAP function get_bugs allows only to query for
package, submitter, maint, src, severity, status, tag, owner, bugs,
affects, archive. See <https://wiki.debian.org/DebbugsSoapInterface>.

Bugs carry a last_modified attribute, but we cannot use it for queries.
This would require to extend the SOAP interface.

Best regards, Michael.

Re: [elpa] master 88578a4: Increase the default number of hits

[Michael Albinus]

Michael Albinus <michael.albinus@gmx.de> writes:

> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>
>> Michael Albinus <michael.albinus@gmx.de> writes:
>>
>>> What I'm thinking about is a command which disables the cache
>>> temporarily, while retrieving bugs. For example "C-u g", but this would
>>> retrieve *all* bugs in the buffer (could be some thousands). This is too
>>> coarse-grained.
>>
>> There's no way to ask debbugs "what bug reports have changed since
>> <time>"?
>
> Unfortunately, not. The SOAP function get_bugs allows only to query for
> package, submitter, maint, src, severity, status, tag, owner, bugs,
> affects, archive. See <https://wiki.debian.org/DebbugsSoapInterface>.
>
> Bugs carry a last_modified attribute, but we cannot use it for queries.
> This would require to extend the SOAP interface.

I have made debbugs-cache-expiry a defcustom, so you could decrease the
time to cache. And "C-u g" ignores the cache now.

Best regards, Michael.