bug#50988: 26.3; gnus-cloud gets wrong chunk byte count writing sync from windows

bug#50988: 26.3; gnus-cloud gets wrong chunk byte count writing sync from windows

[sb]

This happens for me on emacs 26.3, but has happened ever since I first
tried to use gnus-cloud in 2016, so I've never been able to use
gnus-cloud (I've continued to use its predecessor gnus-sync, but with
emacs 27, gnus-sync no longer worked).

When gnus-cloud-upload-all-data is run in gnus on emacs on windows, the
byte count in the sync data chunks is wrong, so that the data is broken
when attemting to sync on a debian machine.

The first chunk has the following header:
(:type :file :file-name "~/.gnus.el" :timestamp "2021-09-19T15:04:33+0200" :length 16241)

The byte count here is 16241, but it should have been 15754.

This results in the contents of the score files that follows .gnus.el to
be copied into .gnus.el and the .gnus.el file becomes unparsable.

The difference in length corresponds to the number of lines in .gnus.el,
so I'm guessing it's because the CR in CRLF line endings is stripped
away.

The place where the length is set, is here
  https://git.savannah.gnu.org/cgit/emacs.git/tree/lisp/gnus/gnus-cloud.el#n98

I've tried taking the length of the string instead of the size of the
buffer, but the count for .gnus.el still came out as 16241 (so the CR
stripping doesn't take place there).

I've also, as an experiment, replaced insert-file-contents-literally,
with insert-file-contents, and then windows gnus wrote data that
gnus-cloud-download-all-data on debian gnus could read.

But that's probably not a fix, since insert-file-contents-literally is
used intentionally?

This is with
(gnus-cloud-storage-method nil)
which is what I've been using for debugging because attempts to using
base64 with gzip caused error messages of the type "not base64"
(presumably also because the byte count was wrong?).

bug#50988: 26.3; gnus-cloud gets wrong chunk byte count writing sync from windows

[Eli Zaretskii]

> From: sb@dod.no
> Date: Sun, 03 Oct 2021 10:34:06 +0200
> 
> When gnus-cloud-upload-all-data is run in gnus on emacs on windows, the
> byte count in the sync data chunks is wrong, so that the data is broken
> when attemting to sync on a debian machine.
> 
> The first chunk has the following header:
> (:type :file :file-name "~/.gnus.el" :timestamp "2021-09-19T15:04:33+0200" :length 16241)
> 
> The byte count here is 16241, but it should have been 15754.
> 
> This results in the contents of the score files that follows .gnus.el to
> be copied into .gnus.el and the .gnus.el file becomes unparsable.
> 
> The difference in length corresponds to the number of lines in .gnus.el,
> so I'm guessing it's because the CR in CRLF line endings is stripped
> away.
> 
> The place where the length is set, is here
>   https://git.savannah.gnu.org/cgit/emacs.git/tree/lisp/gnus/gnus-cloud.el#n98
> 
> I've tried taking the length of the string instead of the size of the
> buffer, but the count for .gnus.el still came out as 16241 (so the CR
> stripping doesn't take place there).
> 
> I've also, as an experiment, replaced insert-file-contents-literally,
> with insert-file-contents, and then windows gnus wrote data that
> gnus-cloud-download-all-data on debian gnus could read.

Does the file read by that code have DOS-style CRLF end-of-line
format?  And what does that code have to do with your .gnus.el?  (I
don't use Gnus, so apologies for asking questions whose answers are
trivial.)

In general, if that function is supposed to read user-written files
(as opposed to files Gnus itself writes), then it should not assume
the EOL format is Unix.  But I'm not sure insert-file-contents is the
right solution here, because I don't know what will Gnus do with the
file.  Note that the function reads the file into a unibyte buffer,
while insert-file-contents performs decoding, and these two don't mix
well, usually.

Do you understand why the byte count "should have been 15754"?  If
gnus-cloud synchronizes your files with a remote repository, then the
files on the remote should have the same EOL format as on your
original disk, so where does Emacs strip or ignore the CR characters
to come up with a smaller number of bytes in the actual transfer?

HTH

bug#50988: 26.3; gnus-cloud gets wrong chunk byte count writing sync from windows

[Steinar Bang]

>>>>> Eli Zaretskii <eliz@gnu.org>:

> Does the file read by that code have DOS-style CRLF end-of-line
> format?

Yes.

> And what does that code have to do with your .gnus.el?  (I don't use
> Gnus, so apologies for asking questions whose answers are trivial.)

The purpose of gnus-cloud is to create IMAP like functionality for NNTP
groups.  Ie. what's read and followed up on on one computer, is marked
as read and followed up on a different computer (I use gnus on 3
different computers and sync state between them).

gnus-cloud also syncs setup in the .gnus.el and the current state of the
score files. The .gnus.el contents is the first chunk in the sync file.

The mechanism of the sync is an IMAP folder with new syncs written as
messages, with the body of the message consisting chunks that have a
lisp-ish header, like this:
(:type :file :file-name "~/.gnus.el" :timestamp "2021-09-19T15:04:33+0200" :length 16241)
and then the contents of the chunk, until the next header

An example:
 https://gist.github.com/steinarb/155c43fd102c1a929ea441b0703492a3

When the byte count in "length" is wrong, the contents of bytes past the
end, are added to ~/.gnus.el in the computer synced to.

> In general, if that function is supposed to read user-written files
> (as opposed to files Gnus itself writes), then it should not assume
> the EOL format is Unix.  But I'm not sure insert-file-contents is the
> right solution here, because I don't know what will Gnus do with the
> file.

Yes, I'm pretty sure insert-file-contents isn't the right fix either,
but now it sort of works, as opposed to not working at all.  So I'll
limp along on that as a workaround until a real fix is.

> Note that the function reads the file into a unibyte buffer, while
> insert-file-contents performs decoding, and these two don't mix well,
> usually.

> Do you understand why the byte count "should have been 15754"?

Yes.

The .gnus.el file became unparsable on the synched to machine after
sync, because of stuff that was added at the end (which sort of looked
like gnus score files).

So I looked into the contents of the file and used head to return the
first 16241 bytes of the file following the header
(:type :file :file-name "~/.gnus.el" :timestamp "2021-09-19T15:04:33+0200" :length 16241)
and saw that this included the stuff that broke the parsing of
.gnus.el:
 https://gist.github.com/steinarb/4ed3a4b716aab22f855df9af387cce78
(search for :type and you'll find the point where .gnus.el parsing
breaks after sync).

At this point I had concluded that the byte count was wrong, and the
only thing I could think about as the cause, was CRLF vs LF.

My .gnus.el has 487 lines and (- 16241 487) is 15754, so I tried using
head to extract the first 15754 bytes of the chunk and ended up with the
correct results:
 https://gist.github.com/steinarb/650839cec8b9961bb614b95be9c026b2

So the evidence is circumstantial but compelling (compelling to me, at
least). 

> If gnus-cloud synchronizes your files with a remote repository, then
> the files on the remote should have the same EOL format as on your
> original disk, so where does Emacs strip or ignore the CR characters
> to come up with a smaller number of bytes in the actual transfer?

I'm not sure where the conversion happens and have been unable to figure
it out.

But I'm guessing the buffer inserted into here, has eol lf-only and that
this is where/when the conversion happens:
 https://git.savannah.gnu.org/cgit/emacs.git/tree/lisp/gnus/gnus-cloud.el#n104

FWIW I didn't really want gnus-cloud to sync .gnus.el, since that file
is handled as a config file checked into git.  But excluding it is not a
part of gnus-cloud config (at least not documented).

bug#50988: 26.3; gnus-cloud gets wrong chunk byte count writing sync from windows

[Eli Zaretskii]

> From: Steinar Bang <sb@dod.no>
> Cc: 50988@debbugs.gnu.org
> Date: Sun, 03 Oct 2021 12:32:51 +0200
> 
> > If gnus-cloud synchronizes your files with a remote repository, then
> > the files on the remote should have the same EOL format as on your
> > original disk, so where does Emacs strip or ignore the CR characters
> > to come up with a smaller number of bytes in the actual transfer?
> 
> I'm not sure where the conversion happens and have been unable to figure
> it out.
> 
> But I'm guessing the buffer inserted into here, has eol lf-only and that
> this is where/when the conversion happens:
>  https://git.savannah.gnu.org/cgit/emacs.git/tree/lisp/gnus/gnus-cloud.el#n104

Sounds unlikely: inserting a string into a buffer doesn't change the
EOL format, or at least isn't supposed to.

bug#50988: 26.3; gnus-cloud gets wrong chunk byte count writing sync from windows

[Lars Ingebrigtsen]

sb@dod.no writes:

> The first chunk has the following header:
> (:type :file :file-name "~/.gnus.el" :timestamp
> "2021-09-19T15:04:33+0200" :length 16241)
>
> The byte count here is 16241, but it should have been 15754.

I'm not able to reproduce this exactly, but I think I've identified a
possible problem in this area now.  Do you still see these issues on the
current "master" branch?

bug#50988: 26.3; gnus-cloud gets wrong chunk byte count writing sync from windows

[Steinar Bang]

>>>>> Lars Ingebrigtsen <larsi@gnus.org>:
> sb@dod.no writes:

>> The first chunk has the following header:
>> (:type :file :file-name "~/.gnus.el" :timestamp
>> "2021-09-19T15:04:33+0200" :length 16241)
>> 
>> The byte count here is 16241, but it should have been 15754.

> I'm not able to reproduce this exactly, but I think I've identified a
> possible problem in this area now.  Do you still see these issues on
> the current "master" branch?

No, sorry!

I removed gnus-cloud sync of .gnus.el and .authinfo.gpg on "Wed Oct 6 20:50:09 2021 +0200"

modified   .emacs
@@ -783,6 +783,7 @@ to be something different.")
  '(gnus-cloud-interactive nil)
  '(gnus-cloud-method "nnimap:privat")
  '(gnus-cloud-storage-method nil)
+ '(gnus-cloud-synced-files '((:directory "~/News" :match ".*.SCORE\\'")))
  '(gnus-picon-style 'right)
  '(gnus-treat-fill-long-lines nil)
  '(gnus-treat-newsgroups-picon nil)

(also, I'm now using a debian GNU/linux laptop, so I don't see any
CRLF/LF related issues (which I still think this was, even if Eli
doesn't). I.e. not easy to reproduce)

bug#50988: 26.3; gnus-cloud gets wrong chunk byte count writing sync from windows

[Lars Ingebrigtsen]

Steinar Bang <sb@dod.no> writes:

> (also, I'm now using a debian GNU/linux laptop, so I don't see any
> CRLF/LF related issues (which I still think this was, even if Eli
> doesn't). I.e. not easy to reproduce)

OK, then I guess there won't be any further progress in this bug report,
and I'm closing it.  If you do see this problem again, please respond to
the debbugs address and we'll reopen.

bug#50988: 26.3; gnus-cloud gets wrong chunk byte count writing sync from windows

[Steinar Bang]

>>>>> Lars Ingebrigtsen <larsi@gnus.org>:

> Steinar Bang <sb@dod.no> writes:
>> (also, I'm now using a debian GNU/linux laptop, so I don't see any
>> CRLF/LF related issues (which I still think this was, even if Eli
>> doesn't).

(Because if I remember correctly, the diff in byte count matched the
line number...)

>> I.e. not easy to reproduce)

> OK, then I guess there won't be any further progress in this bug report,
> and I'm closing it.  If you do see this problem again, please respond to
> the debbugs address and we'll reopen.

+1