bug#71893: 27.1; Gnus pop3 download progress goes over 100%

[Daniel Bastos <dbastos@ic.ufrj.br>]

After seeing Gnus download mail from a POP3 server various times and the
progress report going over 100%, I decided to look into it and edited
the source so I'd get better logging of the fact.  It doesn't happen at
all times.  It's less than 50% of the times, say.  But it's frequent
enough to make it easy to see it at work, though I haven't spotted what
exactly makes the problem happen.

Here's one instance.

Checking new news...
Reading active file via nnml...
Reading incoming mail from pop...
size 0, total-size 80988
pop3 retrieved 0KB (0%)
size 237, total-size 80988
pop3 retrieved 0KB (0%)
size 751, total-size 80988
pop3 retrieved 0KB (0%)
size 1777, total-size 80988
[...]
pop3 retrieved 80KB (99%)
size 81659, total-size 80988
pop3 retrieved 81KB (100%)
size 81865, total-size 80988
pop3 retrieved 81KB (101%)
size 83255, total-size 80988
pop3 retrieved 83KB (102%)
size 84645, total-size 80988
[...]
size 511138, total-size 80988
pop3 retrieved 511KB (631%)
Wrote c:/sys/emacs/usr/Mail/mail/misc/1799
Wrote c:/sys/emacs/usr/Mail/mail/misc/1800
Wrote c:/sys/emacs/usr/Mail/mail/misc/1801
Wrote c:/sys/emacs/usr/Mail/mail/misc/1802
Wrote c:/sys/emacs/usr/Mail/mail/misc/1803
Wrote c:/sys/emacs/usr/Mail/mail/misc/1804
Wrote c:/sys/emacs/usr/Mail/mail/misc/1805
Wrote c:/sys/emacs/usr/Mail/mail/misc/1806
nnml: Reading incoming mail (8 new)...done
--8<-------------------------------------------------------->8---

Here's the size on disk of each message downaloded:

--8<-------------------------------------------------------->8---
%ls -l | awk '$8 >= 1799 && $8 <= 1806 { total += $5; print; } \
  END { print(total) }'
-rw-rw-rw-   1 x              None       446971 2024-05-21 17:43 1799
-rw-rw-rw-   1 x              None         8441 2024-05-21 17:43 1800
-rw-rw-rw-   1 x              None        16925 2024-05-21 17:43 1801
-rw-rw-rw-   1 x              None        20494 2024-05-21 17:43 1802
-rw-rw-rw-   1 x              None         6513 2024-05-21 17:43 1803
-rw-rw-rw-   1 x              None         5688 2024-05-21 17:43 1804
-rw-rw-rw-   1 x              None         2925 2024-05-21 17:43 1805
-rw-rw-rw-   1 x              None         4781 2024-05-21 17:43 1806
512738
--8<-------------------------------------------------------->8---

The total-size calculated by Gnus was 80988.  Where did that number come
from?  It seems to come from straight from the POP3 server, which in
this case it's the Gmail POP3 server.  I managed to get a POP3 list
before downloading, here's what I find.

--8<-------------------------------------------------------->8---
+OK Gpop ready for requests from 34.197.192.71 o12mb164492058vst
+OK send PASS
+OK Welcome.
+OK 8 messages (80988 bytes)
1 20295
2 4583
3 20295
4 20295
5 2725
6 5487
7 2725
8 4583
.
+OK Farewell.
--8<-------------------------------------------------------->8---

(You don't see my input in the POP3 session above.  You only see the
output from the POP3 server.  My input is authentication plus a LIST
command.)

I collected other instances of the problem as well, but they all seem to
have the same in common---the POP3 server seems to announce a total size
that doesn't match the fact (after downloading each message).

In other words, this doesn't look like a Gnus problem at all.  I'm
reporting to this, however, because I'm a little bit curious as to is
the problem exactly.  (You might easily know what explains this.)

It's very odd to me in the session above, we find 3 messages with
exactly the same size of 20295.  There are two with 4583 bytes and there
are two with 2725 bytes.  I'm not getting any repeated messages at all.

I would appreciate any analysis you might provide on this.  Thank you.


In GNU Emacs 27.1 (build 1, x86_64-w64-mingw32)
 of 2020-08-21 built on CIRROCUMULUS
Repository revision: 86d8d76aa36037184db0b2897c434cdaab1a9ae8
Repository branch: HEAD
Windowing system distributor 'Microsoft Corp.', version 10.0.19045
System Description: Microsoft Windows 10 Pro N (v10.0.2009.19045.4291)

Configured using:
 'configure --without-dbus --host=x86_64-w64-mingw32
 --without-compress-install 'CFLAGS=-O2 -static''

Configured features:
XPM JPEG TIFF GIF PNG RSVG SOUND NOTIFY W32NOTIFY ACL GNUTLS LIBXML2
HARFBUZZ ZLIB TOOLKIT_SCROLL_BARS MODULES THREADS JSON PDUMPER LCMS2 GMP

Important settings:
  value of $LC_ALL: C
  value of $LANG: ENU
  locale-coding-system: cp1252

Major mode: Article

Minor modes in effect:
  electric-pair-mode: t
  show-paren-mode: t
  recentf-mode: t
  save-place-mode: t
  winner-mode: t
  display-time-mode: t
  shell-dirtrack-mode: t
  global-eldoc-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
c:/sys/emacs/usr/.emacs.d/elpa/slime-20231228.1755/slime hides c:/sys/emacs/usr/quicklisp/dists/quicklisp/software/slime-v2.28/slime
c:/sys/emacs/usr/.emacs.d/elpa/slime-20231228.1755/slime-tests hides c:/sys/emacs/usr/quicklisp/dists/quicklisp/software/slime-v2.28/slime-tests
c:/sys/emacs/usr/.emacs.d/elpa/slime-20231228.1755/slime-autoloads hides c:/sys/emacs/usr/quicklisp/dists/quicklisp/software/slime-v2.28/slime-autoloads
c:/sys/emacs/usr/.emacs.d/elpa/pos-tip-20191227.1356/pos-tip hides c:/sys/emacs/share/emacs/site-lisp/my-pos-tip/pos-tip
c:/sys/emacs/usr/.emacs.d/elpa/pos-tip-20191227.1356/pos-tip-pkg hides c:/sys/emacs/share/emacs/site-lisp/my-pos-tip/pos-tip-pkg
c:/sys/emacs/usr/.emacs.d/elpa/pos-tip-20191227.1356/pos-tip-autoloads hides c:/sys/emacs/share/emacs/site-lisp/my-pos-tip/pos-tip-autoloads
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-xp hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-xp
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-xp-complete hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-xp-complete
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-wsl hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-wsl
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-visit hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-visit
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-util hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-util
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-unicode-input-method hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-unicode-input-method
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-stepper hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-stepper
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-smart-open hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-smart-open
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-show hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-show
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-repl hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-repl
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-repl-buffer-name hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-repl-buffer-name
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-profile hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-profile
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-ppss hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-ppss
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-parens hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-parens
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-mode hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-mode
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-logger hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-logger
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-keywords-and-builtins hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-keywords-and-builtins
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-indent hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-indent
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-imenu hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-imenu
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-font-lock hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-font-lock
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-eldoc hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-eldoc
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-edit hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-edit
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-doc hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-doc
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-describe hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-describe
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-debug hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-debug
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-custom hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-custom
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-complete hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-complete
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-common hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-common
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-collection hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-collection
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-cmd hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-cmd
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-bug-report hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-bug-report
c:/sys/emacs/usr/.emacs.d/elpa/racket-mode-20231224.2126/racket-browse-url hides c:/sys/emacs/share/emacs/site-lisp/my-racket-mode/racket-browse-url
c:/sys/emacs/usr/.emacs.d/elpa/seq-2.23/seq hides c:/sys/emacs/share/emacs/27.1/lisp/emacs-lisp/seq

Features:
(shadow emacsbug org-element avl-tree ol-eww ol-rmail ol-mhe ol-irc
ol-info ol-gnus ol-docview doc-view image-mode exif ol-bibtex bibtex
ol-bbdb ol-w3m org ob ob-tangle ob-ref ob-lob ob-table ob-exp org-macro
org-footnote org-src ob-comint org-pcomplete org-list org-faces
org-entities org-version ob-emacs-lisp ob-core ob-eval org-table ol
org-keys org-compat org-macs org-loaddefs cal-menu calendar cal-loaddefs
skeleton dabbrev macros shr-color color rect grep eieio-opt speedbar
sb-image ezimage dframe ibuf-ext ibuffer ibuffer-loaddefs macrostep-c
cmacexp cc-mode cc-fonts cc-guess cc-menus cc-cmds cc-styles cc-align
cc-engine asm-mode pcmpl-unix canlock gnus-draft jka-compr em-unix
em-term term ehelp em-script em-prompt em-ls em-hist em-pred em-glob
em-dirs esh-var em-cmpl em-basic em-banner em-alias esh-mode eshell
esh-cmd esh-ext esh-opt esh-proc esh-io esh-arg esh-module esh-groups
esh-util tex-mode latexenc python tramp-sh misearch multi-isearch
gnus-html url-queue help-fns radix-tree url-cache mm-url mule-util
flow-fill mailalias smtpmail sendmail nnir sort gnus-cite mm-archive
mail-extr gnus-async gnus-bcklg qp gnus-ml gnus-topic pop3 nndraft nnmh
nnfolder gnutls nnml gnus-agent gnus-srvr gnus-score score-mode
nnvirtual gnus-msg gnus-art mm-uu mml2015 mm-view mml-smime smime dig
gnus-cache gnus-sum url url-proxy url-privacy url-expand url-methods
url-history mailcap shr url-cookie url-domsuf url-util svg xml dom
network-stream nsm nntp gnus-group gnus-undo gnus-start gnus-cloud
nnimap nnmail mail-source utf7 netrc nnoo gnus-spec gnus-int gnus-range
message rmc puny dired dired-loaddefs rfc822 mml mml-sec epa epg
epg-config mm-decode mm-bodies mm-encode mail-parse rfc2231 mailabbrev
gmm-utils mailheader gnus-win gnus nnheader gnus-util rmail
rmail-loaddefs rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils
rx text-property-search paredit finder-inf info package url-handlers
url-parse url-vars aggressive-indent tramp tramp-loaddefs trampver
tramp-integration files-x tramp-compat parse-time iso8601 time-date
auth-source password-cache json subr-x map elec-pair paren slime-fancy
slime-indentation slime-cl-indent cl-indent slime-trace-dialog
slime-fontifying-fu slime-package-fu slime-references
slime-compiler-notes-tree slime-scratch slime-presentations bridge
slime-macrostep macrostep slime-mdot-fu slime-enclosing-context
slime-fuzzy slime-fancy-trace slime-fancy-inspector slime-c-p-c
slime-editing-commands slime-autodoc slime-repl elp slime-parse slime
warnings derived cl-extra edmacro kmacro lisp-mnt gud apropos etags
fileloop generator xref cl-seq project eieio eieio-core eieio-loaddefs
arc-mode archive-mode noutline outline easy-mmode hyperspec
slime-autoloads cl-macs recentf tree-widget wid-edit saveplace winner
windmove time ido ess-toolbar ess-mouse mouseme thingatpt browse-url
ess-menu ess-swv ess-noweb ess-noweb-font-lock-mode ess-bugs-l essd-els
ess-sas-d ess-sas-l ess-sas-a shell pcomplete ess-sta-d ess-sta-l
cc-vars cc-defs make-regexp ess-sp6w-d ess-sp3-d ess-julia julia-mode
ert pp ewoc debug backtrace help-mode find-func ess-r-d ess-r-syntax
ess-r-completion ess-tracebug format-spec ess-roxy advice hideshow
ess-help ess-developer ess-s-l ess ess-inf compile comint ansi-color
ring ess-mode ess-noweb-mode ess-utils cl cl-loaddefs cl-lib ess-custom
executable easymenu ess-compat ess-site my seq byte-opt gv bytecomp
byte-compile cconv tooltip eldoc electric uniquify ediff-hook vc-hooks
lisp-float-type mwheel dos-w32 ls-lisp disp-table term/w32-win w32-win
w32-vars term/common-win tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode elisp-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch timer
select scroll-bar mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame minibuffer cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite charscript charprop case-table epa-hook
jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice loaddefs
button faces cus-face macroexp files text-properties overlay sha1 md5
base64 format env code-pages mule custom widget hashtable-print-readable
backquote threads w32notify w32 lcms2 multi-tty make-network-process
emacs)

Memory information:
((conses 16 993168 152026)
 (symbols 48 46017 1)
 (strings 32 250528 17715)
 (string-bytes 1 6645953)
 (vectors 16 96624)
 (vector-slots 8 4132847 166420)
 (floats 8 938 892)
 (intervals 56 23645 900)
 (buffers 1000 246))