bug#71893: 27.1; Gnus pop3 download progress goes over 100%

bug#71893: 27.1; Gnus pop3 download progress goes over 100%

[Daniel Bastos]

After seeing Gnus download mail from a POP3 server various times and the
progress report going over 100%, I decided to look into it and edited
the source so I'd get better logging of the fact.  It doesn't happen at
all times.  It's less than 50% of the times, say.  But it's frequent
enough to make it easy to see it at work, though I haven't spotted what
exactly makes the problem happen.

Here's one instance.

Checking new news...
Reading active file via nnml...
Reading incoming mail from pop...
size 0, total-size 80988
pop3 retrieved 0KB (0%)
size 237, total-size 80988
pop3 retrieved 0KB (0%)
size 751, total-size 80988
pop3 retrieved 0KB (0%)
size 1777, total-size 80988
[...]
pop3 retrieved 80KB (99%)
size 81659, total-size 80988
pop3 retrieved 81KB (100%)
size 81865, total-size 80988
pop3 retrieved 81KB (101%)
size 83255, total-size 80988
pop3 retrieved 83KB (102%)
size 84645, total-size 80988
[...]
size 511138, total-size 80988
pop3 retrieved 511KB (631%)
Wrote c:/sys/emacs/usr/Mail/mail/misc/1799
Wrote c:/sys/emacs/usr/Mail/mail/misc/1800
Wrote c:/sys/emacs/usr/Mail/mail/misc/1801
Wrote c:/sys/emacs/usr/Mail/mail/misc/1802
Wrote c:/sys/emacs/usr/Mail/mail/misc/1803
Wrote c:/sys/emacs/usr/Mail/mail/misc/1804
Wrote c:/sys/emacs/usr/Mail/mail/misc/1805
Wrote c:/sys/emacs/usr/Mail/mail/misc/1806
nnml: Reading incoming mail (8 new)...done
--8<-------------------------------------------------------->8---

Here's the size on disk of each message downaloded:

--8<-------------------------------------------------------->8---
%ls -l | awk '$8 >= 1799 && $8 <= 1806 { total += $5; print; } \
  END { print(total) }'
-rw-rw-rw-   1 x              None       446971 2024-05-21 17:43 1799
-rw-rw-rw-   1 x              None         8441 2024-05-21 17:43 1800
-rw-rw-rw-   1 x              None        16925 2024-05-21 17:43 1801
-rw-rw-rw-   1 x              None        20494 2024-05-21 17:43 1802
-rw-rw-rw-   1 x              None         6513 2024-05-21 17:43 1803
-rw-rw-rw-   1 x              None         5688 2024-05-21 17:43 1804
-rw-rw-rw-   1 x              None         2925 2024-05-21 17:43 1805
-rw-rw-rw-   1 x              None         4781 2024-05-21 17:43 1806
512738
--8<-------------------------------------------------------->8---

The total-size calculated by Gnus was 80988.  Where did that number come
from?  It seems to come from straight from the POP3 server, which in
this case it's the Gmail POP3 server.  I managed to get a POP3 list
before downloading, here's what I find.

--8<-------------------------------------------------------->8---
+OK Gpop ready for requests from 34.197.192.71 o12mb164492058vst
+OK send PASS
+OK Welcome.
+OK 8 messages (80988 bytes)
1 20295
2 4583
3 20295
4 20295
5 2725
6 5487
7 2725
8 4583
.
+OK Farewell.
--8<-------------------------------------------------------->8---

(You don't see my input in the POP3 session above.  You only see the
output from the POP3 server.  My input is authentication plus a LIST
command.)

I collected other instances of the problem as well, but they all seem to
have the same in common---the POP3 server seems to announce a total size
that doesn't match the fact (after downloading each message).

In other words, this doesn't look like a Gnus problem at all.  I'm
reporting to this, however, because I'm a little bit curious as to is
the problem exactly.  (You might easily know what explains this.)

It's very odd to me in the session above, we find 3 messages with
exactly the same size of 20295.  There are two with 4583 bytes and there
are two with 2725 bytes.  I'm not getting any repeated messages at all.

I would appreciate any analysis you might provide on this.  Thank you.


In GNU Emacs 27.1 (build 1, x86_64-w64-mingw32)
 of 2020-08-21 built on CIRROCUMULUS
Repository revision: 86d8d76aa36037184db0b2897c434cdaab1a9ae8
Repository branch: HEAD
Windowing system distributor 'Microsoft Corp.', version 10.0.19045
System Description: Microsoft Windows 10 Pro N (v10.0.2009.19045.4291)

Configured using:
 'configure --without-dbus --host=x86_64-w64-mingw32
 --without-compress-install 'CFLAGS=-O2 -static''

Configured features:
XPM JPEG TIFF GIF PNG RSVG SOUND NOTIFY W32NOTIFY ACL GNUTLS LIBXML2
HARFBUZZ ZLIB TOOLKIT_SCROLL_BARS MODULES THREADS JSON PDUMPER LCMS2 GMP

Important settings:
  value of $LC_ALL: C
  value of $LANG: ENU
  locale-coding-system: cp1252

Major mode: Article

Minor modes in effect:
  electric-pair-mode: t
  show-paren-mode: t
  recentf-mode: t
  save-place-mode: t
  winner-mode: t
  display-time-mode: t
  shell-dirtrack-mode: t
  global-eldoc-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
c:/sys/emacs/usr/.emacs.d/elpa/slime-20231228.1755/slime hides c:/sys/emacs/usr/quicklisp/dists/quicklisp/software/slime-v2.28/slime
c:/sys/emacs/usr/.emacs.d/elpa/slime-20231228.1755/slime-tests hides c:/sys/emacs/usr/quicklisp/dists/quicklisp/software/slime-v2.28/slime-tests
c:/sys/emacs/usr/.emacs.d/elpa/slime-20231228.1755/slime-autoloads hides c:/sys/emacs/usr/quicklisp/dists/quicklisp/software/slime-v2.28/slime-autoloads
c:/sys/emacs/usr/.emacs.d/elpa/pos-tip-20191227.1356/pos-tip hides c:/sys/emacs/share/emacs/site-lisp/my-pos-tip/pos-tip
c:/sys/emacs/usr/.emacs.d/elpa/pos-tip-20191227.1356/pos-tip-pkg hides c:/sys/emacs/share/emacs/site-lisp/my-pos-tip/pos-tip-pkg
c:/sys/emacs/usr/.emacs.d/elpa/pos-tip-20191227.1356/pos-tip-autoloads hides c:/sys/emacs/share/emacs/site-lisp/my-pos-tip/pos-tip-autoloads
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-xp hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-xp
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-xp-complete hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-xp-complete
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-wsl hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-wsl
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-visit hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-visit
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-util hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-util
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-unicode-input-method hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-unicode-input-method
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-stepper hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-stepper
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-smart-open hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-smart-open
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-show hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-show
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-repl hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-repl
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-repl-buffer-name hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-repl-buffer-name
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-profile hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-profile
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-ppss hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-ppss
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-parens hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-parens
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-mode hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-mode
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-logger hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-logger
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-keywords-and-builtins hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-keywords-and-builtins
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-indent hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-indent
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-imenu hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-imenu
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-font-lock hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-font-lock
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-eldoc hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-eldoc
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-edit hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-edit
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-doc hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-doc
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-describe hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-describe
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-debug hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-debug
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-custom hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-custom
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-complete hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-complete
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-common hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-common
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-collection hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-collection
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-cmd hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-cmd
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-bug-report hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-bug-report
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-browse-url hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-browse-url
c:/sys/emacs/usr/.emacs.d/elpa/seq-2.23/seq hides c:/sys/emacs/share/emacs/27.1/lisp/emacs-lisp/seq

Features:
(shadow emacsbug org-element avl-tree ol-eww ol-rmail ol-mhe ol-irc
ol-info ol-gnus ol-docview doc-view image-mode exif ol-bibtex bibtex
ol-bbdb ol-w3m org ob ob-tangle ob-ref ob-lob ob-table ob-exp org-macro
org-footnote org-src ob-comint org-pcomplete org-list org-faces
org-entities org-version ob-emacs-lisp ob-core ob-eval org-table ol
org-keys org-compat org-macs org-loaddefs cal-menu calendar cal-loaddefs
skeleton dabbrev macros shr-color color rect grep eieio-opt speedbar
sb-image ezimage dframe ibuf-ext ibuffer ibuffer-loaddefs macrostep-c
cmacexp cc-mode cc-fonts cc-guess cc-menus cc-cmds cc-styles cc-align
cc-engine asm-mode pcmpl-unix canlock gnus-draft jka-compr em-unix
em-term term ehelp em-script em-prompt em-ls em-hist em-pred em-glob
em-dirs esh-var em-cmpl em-basic em-banner em-alias esh-mode eshell
esh-cmd esh-ext esh-opt esh-proc esh-io esh-arg esh-module esh-groups
esh-util tex-mode latexenc python tramp-sh misearch multi-isearch
gnus-html url-queue help-fns radix-tree url-cache mm-url mule-util
flow-fill mailalias smtpmail sendmail nnir sort gnus-cite mm-archive
mail-extr gnus-async gnus-bcklg qp gnus-ml gnus-topic pop3 nndraft nnmh
nnfolder gnutls nnml gnus-agent gnus-srvr gnus-score score-mode
nnvirtual gnus-msg gnus-art mm-uu mml2015 mm-view mml-smime smime dig
gnus-cache gnus-sum url url-proxy url-privacy url-expand url-methods
url-history mailcap shr url-cookie url-domsuf url-util svg xml dom
network-stream nsm nntp gnus-group gnus-undo gnus-start gnus-cloud
nnimap nnmail mail-source utf7 netrc nnoo gnus-spec gnus-int gnus-range
message rmc puny dired dired-loaddefs rfc822 mml mml-sec epa epg
epg-config mm-decode mm-bodies mm-encode mail-parse rfc2231 mailabbrev
gmm-utils mailheader gnus-win gnus nnheader gnus-util rmail
rmail-loaddefs rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils
rx text-property-search paredit finder-inf info package url-handlers
url-parse url-vars aggressive-indent tramp tramp-loaddefs trampver
tramp-integration files-x tramp-compat parse-time iso8601 time-date
auth-source password-cache json subr-x map elec-pair paren slime-fancy
slime-indentation slime-cl-indent cl-indent slime-trace-dialog
slime-fontifying-fu slime-package-fu slime-references
slime-compiler-notes-tree slime-scratch slime-presentations bridge
slime-macrostep macrostep slime-mdot-fu slime-enclosing-context
slime-fuzzy slime-fancy-trace slime-fancy-inspector slime-c-p-c
slime-editing-commands slime-autodoc slime-repl elp slime-parse slime
warnings derived cl-extra edmacro kmacro lisp-mnt gud apropos etags
fileloop generator xref cl-seq project eieio eieio-core eieio-loaddefs
arc-mode archive-mode noutline outline easy-mmode hyperspec
slime-autoloads cl-macs recentf tree-widget wid-edit saveplace winner
windmove time ido ess-toolbar ess-mouse mouseme thingatpt browse-url
ess-menu ess-swv ess-noweb ess-noweb-font-lock-mode ess-bugs-l essd-els
ess-sas-d ess-sas-l ess-sas-a shell pcomplete ess-sta-d ess-sta-l
cc-vars cc-defs make-regexp ess-sp6w-d ess-sp3-d ess-julia julia-mode
ert pp ewoc debug backtrace help-mode find-func ess-r-d ess-r-syntax
ess-r-completion ess-tracebug format-spec ess-roxy advice hideshow
ess-help ess-developer ess-s-l ess ess-inf compile comint ansi-color
ring ess-mode ess-noweb-mode ess-utils cl cl-loaddefs cl-lib ess-custom
executable easymenu ess-compat ess-site my seq byte-opt gv bytecomp
byte-compile cconv tooltip eldoc electric uniquify ediff-hook vc-hooks
lisp-float-type mwheel dos-w32 ls-lisp disp-table term/w32-win w32-win
w32-vars term/common-win tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode elisp-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch timer
select scroll-bar mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame minibuffer cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite charscript charprop case-table epa-hook
jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice loaddefs
button faces cus-face macroexp files text-properties overlay sha1 md5
base64 format env code-pages mule custom widget hashtable-print-readable
backquote threads w32notify w32 lcms2 multi-tty make-network-process
emacs)

Memory information:
((conses 16 993168 152026)
 (symbols 48 46017 1)
 (strings 32 250528 17715)
 (string-bytes 1 6645953)
 (vectors 16 96624)
 (vector-slots 8 4132847 166420)
 (floats 8 938 892)
 (intervals 56 23645 900)
 (buffers 1000 246))

bug#71893: 27.1; Gnus pop3 download progress goes over 100%

[Jeremy Bryant via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Daniel Bastos <dbastos@ic.ufrj.br> writes:

> In GNU Emacs 27.1 (build 1, x86_64-w64-mingw32)
>  of 2020-08-21 built on CIRROCUMULUS
> Repository revision: 86d8d76aa36037184db0b2897c434cdaab1a9ae8
> Repository branch: HEAD
> Windowing system distributor 'Microsoft Corp.', version 10.0.19045
> System Description: Microsoft Windows 10 Pro N (v10.0.2009.19045.4291)

Are you constrained to use 27.1?

Have you tried to reproduce this bug in a more current version of Emacs?
(the latest release is 29.4)

bug#71893: 27.1; Gnus pop3 download progress goes over 100%

[Eli Zaretskii]

Ping!

> Cc: 71893@debbugs.gnu.org
> Date: Tue, 02 Jul 2024 23:50:21 +0100
> From:  Jeremy Bryant via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> 
> Daniel Bastos <dbastos@ic.ufrj.br> writes:
> 
> > In GNU Emacs 27.1 (build 1, x86_64-w64-mingw32)
> >  of 2020-08-21 built on CIRROCUMULUS
> > Repository revision: 86d8d76aa36037184db0b2897c434cdaab1a9ae8
> > Repository branch: HEAD
> > Windowing system distributor 'Microsoft Corp.', version 10.0.19045
> > System Description: Microsoft Windows 10 Pro N (v10.0.2009.19045.4291)
> 
> Are you constrained to use 27.1?
> 
> Have you tried to reproduce this bug in a more current version of Emacs?
> (the latest release is 29.4)
> 
> 
> 
> 

bug#71893: 27.1; Gnus pop3 download progress goes over 100%

[Eli Zaretskii]

Ping! Ping!  Can we please make some progress here?

> Cc: 71893@debbugs.gnu.org
> Date: Sat, 20 Jul 2024 13:07:38 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> Ping!
> 
> > Cc: 71893@debbugs.gnu.org
> > Date: Tue, 02 Jul 2024 23:50:21 +0100
> > From:  Jeremy Bryant via "Bug reports for GNU Emacs,
> >  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> > 
> > Daniel Bastos <dbastos@ic.ufrj.br> writes:
> > 
> > > In GNU Emacs 27.1 (build 1, x86_64-w64-mingw32)
> > >  of 2020-08-21 built on CIRROCUMULUS
> > > Repository revision: 86d8d76aa36037184db0b2897c434cdaab1a9ae8
> > > Repository branch: HEAD
> > > Windowing system distributor 'Microsoft Corp.', version 10.0.19045
> > > System Description: Microsoft Windows 10 Pro N (v10.0.2009.19045.4291)
> > 
> > Are you constrained to use 27.1?
> > 
> > Have you tried to reproduce this bug in a more current version of Emacs?
> > (the latest release is 29.4)
> > 
> > 
> > 
> > 
> 
> 
> 
> 

bug#71893: 27.1; Gnus pop3 download progress goes over 100%

[Daniel Bastos]

I'll get to it as soon as possible.

On Sun, Aug 4, 2024 at 4:54 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
> Ping! Ping!  Can we please make some progress here?
>
> > Cc: 71893@debbugs.gnu.org
> > Date: Sat, 20 Jul 2024 13:07:38 +0300
> > From: Eli Zaretskii <eliz@gnu.org>
> >
> > Ping!
> >
> > > Cc: 71893@debbugs.gnu.org
> > > Date: Tue, 02 Jul 2024 23:50:21 +0100
> > > From:  Jeremy Bryant via "Bug reports for GNU Emacs,
> > >  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> > >
> > > Daniel Bastos <dbastos@ic.ufrj.br> writes:
> > >
> > > > In GNU Emacs 27.1 (build 1, x86_64-w64-mingw32)
> > > >  of 2020-08-21 built on CIRROCUMULUS
> > > > Repository revision: 86d8d76aa36037184db0b2897c434cdaab1a9ae8
> > > > Repository branch: HEAD
> > > > Windowing system distributor 'Microsoft Corp.', version 10.0.19045
> > > > System Description: Microsoft Windows 10 Pro N (v10.0.2009.19045.4291)
> > >
> > > Are you constrained to use 27.1?
> > >
> > > Have you tried to reproduce this bug in a more current version of Emacs?
> > > (the latest release is 29.4)
> > >
> > >
> > >
> > >
> >
> >
> >
> >

bug#71893: 27.1; Gnus pop3 download progress goes over 100%

[Eli Zaretskii]

Any progress there?

> From: Daniel Bastos <dbastos@ic.ufrj.br>
> Date: Mon, 12 Aug 2024 09:54:59 -0300
> Cc: jb@jeremybryant.net, 71893@debbugs.gnu.org
> 
> I'll get to it as soon as possible.
> 
> On Sun, Aug 4, 2024 at 4:54 AM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > Ping! Ping!  Can we please make some progress here?
> >
> > > Cc: 71893@debbugs.gnu.org
> > > Date: Sat, 20 Jul 2024 13:07:38 +0300
> > > From: Eli Zaretskii <eliz@gnu.org>
> > >
> > > Ping!
> > >
> > > > Cc: 71893@debbugs.gnu.org
> > > > Date: Tue, 02 Jul 2024 23:50:21 +0100
> > > > From:  Jeremy Bryant via "Bug reports for GNU Emacs,
> > > >  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> > > >
> > > > Daniel Bastos <dbastos@ic.ufrj.br> writes:
> > > >
> > > > > In GNU Emacs 27.1 (build 1, x86_64-w64-mingw32)
> > > > >  of 2020-08-21 built on CIRROCUMULUS
> > > > > Repository revision: 86d8d76aa36037184db0b2897c434cdaab1a9ae8
> > > > > Repository branch: HEAD
> > > > > Windowing system distributor 'Microsoft Corp.', version 10.0.19045
> > > > > System Description: Microsoft Windows 10 Pro N (v10.0.2009.19045.4291)
> > > >
> > > > Are you constrained to use 27.1?
> > > >
> > > > Have you tried to reproduce this bug in a more current version of Emacs?
> > > > (the latest release is 29.4)
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> 

bug#71893: 27.1; Gnus pop3 download progress goes over 100%

[Daniel Bastos]

On Sat, Aug 31, 2024 at 4:53 AM Eli Zaretskii <eliz@gnu.org> wrote:
> Any progress there?

I've the 29.3 set up and I have not yet managed to reproduce.  I've
tested it three times already with a set of messages to be downloaded
that I would believe it would hit the problem.  I'm going to try to
change my POP3 settings so that I can download the set of messages
from both versions 27.1 e 29.3 to see if I can compare.  What I'm very
puzzled with is that I would not have thought that this is a Gnus
problem.  It seemed clear to me that the POP3 server was reporting
incorrect results.  Since I know very little about POP3 servers,
perhaps there's more to the story than I can see.  If 29.3 solves the
problem, I'm going to upgrade.  Sorry about the delay and thanks for
your assistance!  I'll write again soon.

bug#71893: 27.1; Gnus pop3 download progress goes over 100%

[Daniel Bastos]

On Mon, Sep 9, 2024 at 8:29 PM Daniel Bastos <dbastos@ic.ufrj.br> wrote:
>
> On Sat, Aug 31, 2024 at 4:53 AM Eli Zaretskii <eliz@gnu.org> wrote:
> > Any progress there?
>
> I've the 29.3 set up and I have not yet managed to reproduce.

Now I did.  The problem happens in

  GNU Emacs 29.3 (build 2, x86_64-w64-mingw32) of 2024-03-24

too.  Here's what I found in the POP3 before fetching my mail.

+OK 69 messages (2041560 bytes)
1 1243
2 56203
3 5694
4 1858
5 8704
6 2264
7 1626
8 1626
9 1626
10 8704
11 15079
12 15079
13 15079
14 1858
15 6303
16 1303
17 1303
18 1858
19 8704
20 8704
21 8581
22 8704
23 8581
24 13969
25 13969
26 61517
27 9590
28 47193
29 61517
30 47193
31 13969
32 16591
33 1880
34 56203
35 16591
36 16591
37 61517
38 12830
39 56203
40 56203
41 13969
42 16591
43 16591
44 8581
45 1880
46 1880
47 56203
48 12830
49 8704
50 12830
51 61517
52 1880
53 6303
54 6303
55 61517
56 61517
57 61517
58 61517
59 61517
60 61517
61 9590
62 9590
63 237515
64 236784
65 56203
66 56203
67 56203
68 12830
69 17268
.
+OK Farewell.

If I sum the bytes of each message, I get

%awk '{ sum += $2; } END { print(sum) }' pop3-list.txt
2041560

which is exactly the advertised total at the top.  The Gmail POP3
makes minimum sense.  And here's the log after having Gnus fetch all
the messages.  (I couldn't keep the beginning lines because the
*messages* buffer erased them, but the log shows that Gnus is going
well beyond 100%.)

pop3 retrieved 6945KB (340%)
[...]
pop3 retrieved 9992KB (489%)
pop3 retrieved 10386KB (508%)
Saving file ~/.pop3-uidl...
Wrote c:/Users/x/AppData/Roaming/.pop3-uidl
Wrote c:/Users/x/AppData/Roaming/Mail/mail/misc/118
Wrote c:/Users/x/AppData/Roaming/Mail/mail/misc/119
[...]
Wrote c:/Users/x/AppData/Roaming/Mail/mail/misc/186
nnml: Reading incoming mail (69 new)...done
Reading active file via nnml...done
Checking new news...done

Here's a check of how much data I get after having written each
message to the file system.

%ls -l | awk '$8 >= 118 && $8 <= 186 { total += $5; print; } END {
print(total) }'
-rw-rw-rw-   1 x              None         1472 2024-09-19 16:00 118
-rw-rw-rw-   1 x              None       876549 2024-09-19 16:00 119
-rw-rw-rw-   1 x              None         5925 2024-09-19 16:00 120
-rw-rw-rw-   1 x              None          963 2024-09-19 16:00 121
-rw-rw-rw-   1 x              None         7307 2024-09-19 16:00 122
-rw-rw-rw-   1 x              None         2494 2024-09-19 16:00 123
-rw-rw-rw-   1 x              None         1459 2024-09-19 16:00 124
-rw-rw-rw-   1 x              None         7773 2024-09-19 16:00 125
-rw-rw-rw-   1 x              None         1855 2024-09-19 16:00 126
-rw-rw-rw-   1 x              None         2380 2024-09-19 16:00 127
-rw-rw-rw-   1 x              None       573608 2024-09-19 16:00 128
-rw-rw-rw-   1 x              None        13685 2024-09-19 16:00 129
-rw-rw-rw-   1 x              None        15324 2024-09-19 16:00 130
-rw-rw-rw-   1 x              None         7416 2024-09-19 16:00 131
-rw-rw-rw-   1 x              None      6056691 2024-09-19 16:00 132
-rw-rw-rw-   1 x              None         6596 2024-09-19 16:00 133
-rw-rw-rw-   1 x              None         1532 2024-09-19 16:00 134
-rw-rw-rw-   1 x              None         2088 2024-09-19 16:00 135
-rw-rw-rw-   1 x              None         9087 2024-09-19 16:00 136
-rw-rw-rw-   1 x              None         1398 2024-09-19 16:00 137
-rw-rw-rw-   1 x              None        10892 2024-09-19 16:00 138
-rw-rw-rw-   1 x              None         7740 2024-09-19 16:00 139
-rw-rw-rw-   1 x              None         3172 2024-09-19 16:00 140
-rw-rw-rw-   1 x              None        14922 2024-09-19 16:00 141
-rw-rw-rw-   1 x              None        15983 2024-09-19 16:00 142
-rw-rw-rw-   1 x              None         9209 2024-09-19 16:00 143
-rw-rw-rw-   1 x              None        12690 2024-09-19 16:00 144
-rw-rw-rw-   1 x              None        49851 2024-09-19 16:00 145
-rw-rw-rw-   1 x              None         3123 2024-09-19 16:00 146
-rw-rw-rw-   1 x              None        47433 2024-09-19 16:00 147
-rw-rw-rw-   1 x              None        13070 2024-09-19 16:00 148
-rw-rw-rw-   1 x              None       186296 2024-09-19 16:00 149
-rw-rw-rw-   1 x              None         1108 2024-09-19 16:00 150
-rw-rw-rw-   1 x              None       202280 2024-09-19 16:00 151
-rw-rw-rw-   1 x              None        16064 2024-09-19 16:00 152
-rw-rw-rw-   1 x              None        17431 2024-09-19 16:00 153
-rw-rw-rw-   1 x              None        39122 2024-09-19 16:00 154
-rw-rw-rw-   1 x              None        10617 2024-09-19 16:00 155
-rw-rw-rw-   1 x              None        32890 2024-09-19 16:00 156
-rw-rw-rw-   1 x              None        34388 2024-09-19 16:00 157
-rw-rw-rw-   1 x              None        14204 2024-09-19 16:00 158
-rw-rw-rw-   1 x              None        15540 2024-09-19 16:00 159
-rw-rw-rw-   1 x              None        16822 2024-09-19 16:00 160
-rw-rw-rw-   1 x              None         8825 2024-09-19 16:00 161
-rw-rw-rw-   1 x              None         6727 2024-09-19 16:00 162
-rw-rw-rw-   1 x              None         6521 2024-09-19 16:00 163
-rw-rw-rw-   1 x              None       934797 2024-09-19 16:00 164
-rw-rw-rw-   1 x              None        11879 2024-09-19 16:00 165
-rw-rw-rw-   1 x              None         8935 2024-09-19 16:00 166
-rw-rw-rw-   1 x              None         3363 2024-09-19 16:00 167
-rw-rw-rw-   1 x              None        36427 2024-09-19 16:00 168
-rw-rw-rw-   1 x              None         2109 2024-09-19 16:00 169
-rw-rw-rw-   1 x              None         1886 2024-09-19 16:00 170
-rw-rw-rw-   1 x              None         6534 2024-09-19 16:00 171
-rw-rw-rw-   1 x              None        46503 2024-09-19 16:00 172
-rw-rw-rw-   1 x              None        44079 2024-09-19 16:00 173
-rw-rw-rw-   1 x              None        54344 2024-09-19 16:00 174
-rw-rw-rw-   1 x              None        52465 2024-09-19 16:00 175
-rw-rw-rw-   1 x              None        62311 2024-09-19 16:00 176
-rw-rw-rw-   1 x              None        61752 2024-09-19 16:00 177
-rw-rw-rw-   1 x              None         8929 2024-09-19 16:00 178
-rw-rw-rw-   1 x              None         9823 2024-09-19 16:00 179
-rw-rw-rw-   1 x              None       237754 2024-09-19 16:00 180
-rw-rw-rw-   1 x              None       237022 2024-09-19 16:00 181
-rw-rw-rw-   1 x              None        49595 2024-09-19 16:00 182
-rw-rw-rw-   1 x              None        53055 2024-09-19 16:00 183
-rw-rw-rw-   1 x              None        56433 2024-09-19 16:00 184
-rw-rw-rw-   1 x              None        13062 2024-09-19 16:00 185
-rw-rw-rw-   1 x              None        17499 2024-09-19 16:00 186
10403078

I get a lot more data---10,403,078 bytes.  So it makes me feel that
Gnus is not at fault here; that it's the POP3 server that sometimes
advertises a total amount, but ends up providing a lot more than the
advertised value.  I wouldn't assume this is an obvious bug of any
POP3 server, but it's what the evidence seems to suggest---that the
Gmail POP3 server says one thing and does another.

(*) How I generated the log in *messages*

I added a (message "...") call to the procedure pop3-wait-for-messages.

(defun pop3-wait-for-messages (process count total-size start-point)
  (while (> count 0)
    (goto-char start-point)
    (while (or (and (re-search-forward "^\\+OK" nil t)
    (or (not total-size)
(re-search-forward "^\\.\r?\n" nil t)))
       (re-search-forward "^-ERR " nil t))
      (cl-decf count)
      (setq start-point (point)))
    (unless (memq (process-status process) '(open run))
      (error "pop3 process died"))
    (when total-size
      (let ((size 0))
(goto-char (point-min))
(while (re-search-forward "^\\+OK.*\n" nil t)
  (setq size (+ size (- (point))
(if (re-search-forward "^\\.\r?\n" nil 'move)
    (match-beginning 0)
  (point)))))
(message "pop3 retrieved %dKB (%d%%)"
(truncate (/ size 1000))
(truncate (* (/ (* size 1.0) total-size) 100)))))
    (pop3-accept-process-output process))
  start-point)