bug#56499: 28.1; Unable to open large file

bug#56499: 28.1; Unable to open large file

[Juan José García Ripoll]

I am trying to load a file with
(find-file-literally "~/OneDrive/Mail/archive/Archives-2017.mbox")
either as lisp code or using key commands. Any alternative leads to a
lisp error with the following backtrace
Debugger entered--Lisp error: (wrong-type-argument arrayp nil)
  file-truename(nil)
  find-file-noselect-1(#<buffer Archives-2017.mbox<juanj/OneDrive/Mail/archive>> "~/OneDrive/Mail/archive/Archives-2017.mbox" nil t "~/OneDrive/Mail/archive/Archives-2017.mbox" (281474976939729 250058804))
  find-file-noselect("~/OneDrive/Mail/archive/Archives-2017.mbox" nil t)
  find-file-literally("~/OneDrive/Mail/archive/Archives-2017.mbox")
  eval-expression((find-file-literally "~/OneDrive/Mail/archive/Archives-2017.mbox") nil nil 127)
  funcall-interactively(eval-expression (find-file-literally "~/OneDrive/Mail/archive/Archives-2017.mbox") nil nil 127)
  command-execute(eval-expression)
All this was done from a bare emacs (using -Q)


In GNU Emacs 28.1 (build 2, x86_64-w64-mingw32)
 of 2022-04-21 built on AVALON
Windowing system distributor 'Microsoft Corp.', version 10.0.22000
System Description: Microsoft Windows 10 Pro for Workstations (v10.0.2009.22000.778)

Configured using:
 'configure --with-modules --without-dbus --with-native-compilation
 --without-compress-install CFLAGS=-O2'

Configured features:
ACL GIF GMP GNUTLS HARFBUZZ JPEG JSON LCMS2 LIBXML2 MODULES NATIVE_COMP
NOTIFY W32NOTIFY PDUMPER PNG RSVG SOUND THREADS TIFF TOOLKIT_SCROLL_BARS
XPM ZLIB

(NATIVE_COMP present but libgccjit not available)

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: cp1252

Major mode: Summary

Minor modes in effect:
  shell-dirtrack-mode: t
  savehist-mode: t
  save-place-mode: t
  which-key-mode: t
  marginalia-mode: t
  vertico-mode: t
  gcmh-mode: t
  override-global-mode: t
  global-eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  temp-buffer-resize-mode: t
  buffer-read-only: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t

Load-path shadows:
~/OneDrive/Library/Emacs/src/biblio.el/biblio hides c:/Users/juanj/OneDrive/Library/Emacs/elpa-28/biblio-0.2/biblio
~/OneDrive/Library/Emacs/src/biblio.el/biblio-pkg hides c:/Users/juanj/OneDrive/Library/Emacs/elpa-28/biblio-0.2/biblio-pkg
~/OneDrive/Library/Emacs/src/biblio.el/biblio-hal hides c:/Users/juanj/OneDrive/Library/Emacs/elpa-28/biblio-0.2/biblio-hal
~/OneDrive/Library/Emacs/src/biblio.el/biblio-download hides c:/Users/juanj/OneDrive/Library/Emacs/elpa-28/biblio-0.2/biblio-download
~/OneDrive/Library/Emacs/src/biblio.el/biblio-doi hides c:/Users/juanj/OneDrive/Library/Emacs/elpa-28/biblio-0.2/biblio-doi
~/OneDrive/Library/Emacs/src/biblio.el/biblio-dissemin hides c:/Users/juanj/OneDrive/Library/Emacs/elpa-28/biblio-0.2/biblio-dissemin
~/OneDrive/Library/Emacs/src/biblio.el/biblio-dblp hides c:/Users/juanj/OneDrive/Library/Emacs/elpa-28/biblio-0.2/biblio-dblp
~/OneDrive/Library/Emacs/src/biblio.el/biblio-crossref hides c:/Users/juanj/OneDrive/Library/Emacs/elpa-28/biblio-0.2/biblio-crossref
~/OneDrive/Library/Emacs/src/biblio.el/biblio-arxiv hides c:/Users/juanj/OneDrive/Library/Emacs/elpa-28/biblio-0.2/biblio-arxiv
~/OneDrive/Library/Emacs/src/biblio.el/biblio-core hides c:/Users/juanj/OneDrive/Library/Emacs/elpa-28/biblio-core-0.2/biblio-core
~/OneDrive/Library/Emacs/src/ebib/org-ebib hides c:/Users/juanj/OneDrive/Library/Emacs/elpa-28/ebib-2.34/org-ebib
~/OneDrive/Library/Emacs/src/ebib/ebib hides c:/Users/juanj/OneDrive/Library/Emacs/elpa-28/ebib-2.34/ebib
~/OneDrive/Library/Emacs/src/ebib/ebib-utils hides c:/Users/juanj/OneDrive/Library/Emacs/elpa-28/ebib-2.34/ebib-utils
~/OneDrive/Library/Emacs/src/ebib/ebib-reading-list hides c:/Users/juanj/OneDrive/Library/Emacs/elpa-28/ebib-2.34/ebib-reading-list
~/OneDrive/Library/Emacs/src/ebib/ebib-notes hides c:/Users/juanj/OneDrive/Library/Emacs/elpa-28/ebib-2.34/ebib-notes
~/OneDrive/Library/Emacs/src/ebib/ebib-keywords hides c:/Users/juanj/OneDrive/Library/Emacs/elpa-28/ebib-2.34/ebib-keywords
~/OneDrive/Library/Emacs/src/ebib/ebib-filters hides c:/Users/juanj/OneDrive/Library/Emacs/elpa-28/ebib-2.34/ebib-filters
~/OneDrive/Library/Emacs/src/ebib/ebib-db hides c:/Users/juanj/OneDrive/Library/Emacs/elpa-28/ebib-2.34/ebib-db
~/OneDrive/Library/Emacs/src/ebib/ebib-biblio hides c:/Users/juanj/OneDrive/Library/Emacs/elpa-28/ebib-2.34/ebib-biblio

Features:
(shadow emacsbug misearch multi-isearch pcmpl-unix shell pcomplete
comint ring help-fns radix-tree cl-print nnfolder jka-compr smtpmail
flyspell ispell vc-git diff-mode vc-dispatcher mailalias sendmail
bbdb-mua bbdb-com crm bbdb bbdb-site timezone face-remap flow-fill sort
smiley ansi-color gnus-cite mm-archive mail-extr gnus-bcklg gnus-async
qp gnus-ml gnus-topic gnus-demon utf-7 nndraft nnmh pp url-cache
gnus-oauth2 google-oauth oauth2 url-http url-auth url-gw plstore gnutls
network-stream nsm nnml nnnil gnus-agent gnus-srvr gnus-score score-mode
nnvirtual gnus-msg gnus-art mm-uu mml2015 mm-view mml-smime smime dig
nntp gnus-cache gnus-sum shr kinsoku svg dom gnus-group gnus-undo
gnus-start gnus-dbus dbus xml gnus-cloud nnimap nnmail mail-source utf7
netrc nnoo parse-time iso8601 gnus-spec gnus-int gnus-range message rmc
puny dired dired-loaddefs rfc822 mml mml-sec mm-decode mm-bodies
mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader gnus-win
gnus nnheader gnus-util rmail rmail-loaddefs rfc2047 rfc2045 ietf-drums
text-property-search mail-utils mm-util mail-prsvr wid-edit mule-util
time-date savehist saveplace debug backtrace find-func which-key
marginalia vertico epa-file epa derived epg rfc6068 epg-config advice
doom-themes-org doom-spacegrey-theme doom-themes doom-themes-common
edmacro kmacro gcmh diminish cl-extra help-mode use-package
use-package-ensure use-package-delight use-package-diminish
use-package-bind-key bind-key easy-mmode use-package-core finder-inf
tex-site info package browse-url url url-proxy url-privacy url-expand
url-methods url-history url-cookie url-domsuf url-util mailcap
url-handlers url-parse auth-source cl-seq eieio eieio-core cl-macs
eieio-loaddefs password-cache json subr-x map url-vars seq byte-opt gv
bytecomp byte-compile cconv cl-loaddefs cl-lib iso-transl tooltip eldoc
paren electric uniquify ediff-hook vc-hooks lisp-float-type elisp-mode
mwheel dos-w32 ls-lisp disp-table term/w32-win w32-win w32-vars
term/common-win tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode lisp-mode prog-mode register
page tab-bar menu-bar rfn-eshadow isearch easymenu timer select
scroll-bar mouse jit-lock font-lock syntax font-core term/tty-colors
frame minibuffer cl-generic cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932
hebrew greek romanian slovak czech european ethiopic indian cyrillic
chinese composite emoji-zwj charscript charprop case-table epa-hook
jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice button
loaddefs faces cus-face macroexp files window text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote threads w32notify w32 lcms2 multi-tty
make-network-process native-compile emacs)

Memory information:
((conses 16 628432 254574)
 (symbols 48 27192 58)
 (strings 32 152697 41177)
 (string-bytes 1 4616721)
 (vectors 16 66699)
 (vector-slots 8 1855300 504688)
 (floats 8 443 2122)
 (intervals 56 4078 2388)
 (buffers 992 50))

bug#56499: 28.1; Unable to open large file

[Eli Zaretskii]

> From: Juan José García Ripoll
>  <juanjose.garcia.ripoll@csic.es>
> Date: Mon, 11 Jul 2022 19:09:49 +0200
> 
> 
> I am trying to load a file with
> (find-file-literally "~/OneDrive/Mail/archive/Archives-2017.mbox")
> either as lisp code or using key commands. Any alternative leads to a
> lisp error with the following backtrace
> Debugger entered--Lisp error: (wrong-type-argument arrayp nil)
>   file-truename(nil)
>   find-file-noselect-1(#<buffer Archives-2017.mbox<juanj/OneDrive/Mail/archive>> "~/OneDrive/Mail/archive/Archives-2017.mbox" nil t "~/OneDrive/Mail/archive/Archives-2017.mbox" (281474976939729 250058804))
>   find-file-noselect("~/OneDrive/Mail/archive/Archives-2017.mbox" nil t)
>   find-file-literally("~/OneDrive/Mail/archive/Archives-2017.mbox")
>   eval-expression((find-file-literally "~/OneDrive/Mail/archive/Archives-2017.mbox") nil nil 127)
>   funcall-interactively(eval-expression (find-file-literally "~/OneDrive/Mail/archive/Archives-2017.mbox") nil nil 127)
>   command-execute(eval-expression)
> All this was done from a bare emacs (using -Q)

Does this happen with any file?  If not, what is special about this
file?

It sounds like buffer-file-name is nil for the buffer where the file
is visited, and I don't think I understand how could that happen with
a regular file.

In any case, I cannot reproduce this with random files I tried here.

bug#56499: 28.1; Unable to open large file

[Juan José García-Ripoll]

The file is living in OneDrive in Windows 10/11. I will try with other files in the same file system, but it has hit me hard this week, which I am in remote mode.

Juanjo

Eli Zaretskii <eliz@gnu.org> writes:
>> From: Juan José García Ripoll
>>  <juanjose.garcia.ripoll@csic.es>
>> Date: Mon, 11 Jul 2022 19:09:49 +0200
>> 
>> 
>> I am trying to load a file with
>> (find-file-literally "~/OneDrive/Mail/archive/Archives-2017.mbox")
>> either as lisp code or using key commands. Any alternative leads to a
>> lisp error with the following backtrace
>> Debugger entered--Lisp error: (wrong-type-argument arrayp nil)
>>   file-truename(nil)
>>   find-file-noselect-1(#<buffer Archives-2017.mbox<juanj/OneDrive/Mail/archive>> "~/OneDrive/Mail/archive/Archives-2017.mbox" nil t "~/OneDrive/Mail/archive/Archives-2017.mbox" (281474976939729 250058804))
>>   find-file-noselect("~/OneDrive/Mail/archive/Archives-2017.mbox" nil t)
>>   find-file-literally("~/OneDrive/Mail/archive/Archives-2017.mbox")
>>   eval-expression((find-file-literally "~/OneDrive/Mail/archive/Archives-2017.mbox") nil nil 127)
>>   funcall-interactively(eval-expression (find-file-literally "~/OneDrive/Mail/archive/Archives-2017.mbox") nil nil 127)
>>   command-execute(eval-expression)
>> All this was done from a bare emacs (using -Q)
>
> Does this happen with any file?  If not, what is special about this
> file?
>
> It sounds like buffer-file-name is nil for the buffer where the file
> is visited, and I don't think I understand how could that happen with
> a regular file.
>
> In any case, I cannot reproduce this with random files I tried here.

-- 
Juan José García Ripoll

Quantum Information and Foundations Group
Institute of Fundamental Physics IFF-CSIC
Calle Serrano 113b, Madrid 28006 Spain
http://quinfog.hbar.es - http://juanjose.garciaripoll.com

bug#56499: 28.1; Unable to open large file

[Lars Ingebrigtsen]

Juan José García-Ripoll <juanjose.garcia.ripoll@csic.es> writes:

> The file is living in OneDrive in Windows 10/11. I will try with other
> files in the same file system, but it has hit me hard this week, which
> I am in remote mode.

As Eli says, buffer-file-name apparently being nil in this case, and
that's very odd.

Could there be a special file name handler for OneDrive that's doing
something odd?

What does

M-: (find-file-name-handler "~/OneDrive/Mail/archive/Archives-2017.mbox" 'insert-file-contents) RET

say?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#56499: 28.1; Unable to open large file

[Juan Jose Garcia Ripoll]

[-- Attachment #1: Sólo texto --]
[-- Type: text/plain, Size: 1104 bytes --]

  I tried evaluating find-file-name-handler as suggested and it  
returns nil. I suspect it is due to OneDrive (and now also  
Nextcloud's) virtual files, which live online until requested. Somehow  
opening the file does not trigger the required download.

Juanjo

Lars Ingebrigtsen <larsi@gnus.org> escribió:

> Juan José García-Ripoll <juanjose.garcia.ripoll@csic.es> writes:
>
>> The file is living in OneDrive in Windows 10/11. I will try with other
>> files in the same file system, but it has hit me hard this week, which
>> I am in remote mode.
>
> As Eli says, buffer-file-name apparently being nil in this case, and
> that's very odd.
>
> Could there be a special file name handler for OneDrive that's doing
> something odd?
>
> What does
>
> M-: (find-file-name-handler  
> "~/OneDrive/Mail/archive/Archives-2017.mbox" 'insert-file-contents)  
> RET
> say?
   Juan José García Ripoll

Instituto de Física Fundamental, CSIC
Calle Serrano 113b, Madrid 28006, Spain
Phone: (+34) 915616800 (dial 943107 when hearing the voice)
http://quinfog.iff.csic.es / http://juanjose.garciaripoll.com

[-- Attachment #2: Mensaje HTML --]
[-- Type: text/html, Size: 2053 bytes --]

bug#56499: 28.1; Unable to open large file

[Eli Zaretskii]

> Date: Tue, 12 Jul 2022 15:46:40 +0200
> From: Juan Jose Garcia Ripoll <juanjose.garcia.ripoll@csic.es>
> Cc: Eli Zaretskii <eliz@gnu.org>, 56499@debbugs.gnu.org
> 
> I tried evaluating find-file-name-handler as suggested and it returns nil. I suspect it is due to OneDrive (and
> now also Nextcloud's) virtual files, which live online until requested. Somehow opening the file does not
> trigger the required download.

Does the problem only happen with files that are not downloaded from
the cloud?  What if you tell OneDrive to download that file, then try
again -- does the problem go away then?

bug#56499: 28.1; Unable to open large file

[Lars Ingebrigtsen]

Juan Jose Garcia Ripoll <juanjose.garcia.ripoll@csic.es> writes:

> I tried evaluating find-file-name-handler as suggested and it returns
> nil. I suspect it is due to OneDrive (and now also Nextcloud's)
> virtual files, which live online until requested. Somehow opening the
> file does not trigger the required download.

It should be transparent to Emacs, but perhaps the OS/OneDrive is
returning an odd value to Emacs which confuses Emacs?  I don't use
Windows and OneDrive, so I can't test myself.  I guess the relevant code
is in `insert-file-contents', which is implemened in C, so it's not
trivial to debug.

Eli, do you have any ideas?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#56499: 28.1; Unable to open large file

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  56499@debbugs.gnu.org
> Date: Tue, 12 Jul 2022 15:55:39 +0200
> 
> Juan Jose Garcia Ripoll <juanjose.garcia.ripoll@csic.es> writes:
> 
> > I tried evaluating find-file-name-handler as suggested and it returns
> > nil. I suspect it is due to OneDrive (and now also Nextcloud's)
> > virtual files, which live online until requested. Somehow opening the
> > file does not trigger the required download.
> 
> It should be transparent to Emacs, but perhaps the OS/OneDrive is
> returning an odd value to Emacs which confuses Emacs?  I don't use
> Windows and OneDrive, so I can't test myself.  I guess the relevant code
> is in `insert-file-contents', which is implemened in C, so it's not
> trivial to debug.
> 
> Eli, do you have any ideas?

Not at this time.

I asked Juan Jose a question, and I will try to find a machine with
OneDrive files to try to reproduce this.

bug#56499: 28.1; Unable to open large file

[Juan Jose Garcia Ripoll]

[-- Attachment #1: Sólo texto --]
[-- Type: text/plain, Size: 1332 bytes --]

  Apologies for the noise. I have found out that this problem is  
related to "metered connections". On those ones, Emacs cannot trigger  
opening of the file and is not capable of returning the right error  
message. Once I disable the option "metered connection" from the  
network, Emacs operates as expected. I suspect this cannot really be  
attributed to Emacs, except for the lack of error message in the first  
situation.

Eli Zaretskii <eliz@gnu.org> escribió:

>> Date: Tue, 12 Jul 2022 15:46:40 +0200
>> From: Juan Jose Garcia Ripoll <juanjose.garcia.ripoll@csic.es>
>> Cc: Eli Zaretskii <eliz@gnu.org>, 56499@debbugs.gnu.org
>>
>> I tried evaluating find-file-name-handler as suggested and it  
>> returns nil. I suspect it is due to OneDrive (and
>> now also Nextcloud's) virtual files, which live online until  
>> requested. Somehow opening the file does not
>> trigger the required download.
>
> Does the problem only happen with files that are not downloaded from
> the cloud?  What if you tell OneDrive to download that file, then  
> tryagain -- does the problem go away then?
   Juan José García Ripoll

Instituto de Física Fundamental, CSIC
Calle Serrano 113b, Madrid 28006, Spain
Phone: (+34) 915616800 (dial 943107 when hearing the voice)
http://quinfog.iff.csic.es / http://juanjose.garciaripoll.com

[-- Attachment #2: Mensaje HTML --]
[-- Type: text/html, Size: 2313 bytes --]

bug#56499: 28.1; Unable to open large file

[Eli Zaretskii]

> Date: Wed, 13 Jul 2022 10:22:09 +0200
> From: Juan Jose Garcia Ripoll <juanjose.garcia.ripoll@csic.es>
> Cc: larsi@gnus.org, 56499@debbugs.gnu.org
> 
> Apologies for the noise. I have found out that this problem is related to "metered connections".

Thanks.  I was about to tell that I couldn't reproduce the problem on
a machine where I have OneDrive set up.

> On those
> ones, Emacs cannot trigger opening of the file and is not capable of returning the right error message. Once
> I disable the option "metered connection" from the network, Emacs operates as expected. I suspect this
> cannot really be attributed to Emacs, except for the lack of error message in the first situation.

Would you mind please telling more about the situation where it
happens and about the symptoms?  Does this happen only with
find-file-literally? does it happen only with large files? and what
exactly are "metered connections" and how does one disable that
option?

The reason for the questions is that I'd like to describe the problem
and the solution in etc/PROBLEMS.

Thanks.

bug#56499: 28.1; Unable to open large file

[Juan Jose Garcia Ripoll]

[-- Attachment #1: Sólo texto --]
[-- Type: text/plain, Size: 2456 bytes --]

  Metered connections are network connections that charge by volume of  
data transferred. That happens, for instance, with 4G modems, which I  
am using this week.

When one tethers a 4G mobile phone to a laptop, it is possible to  
select that the connection is metered, to avoid that Windows uses that  
link to download big but not so important data (e.g. updates).

I was unaware that OneDrive deactivates itself in metered connections  
even if I ask to synchronize data. When I opened a big file (the ones  
that OneDrive typically removes from the computer to save space and  
leaves in the cloud) Emacs got a system error that led to the behavior  
described before. It also happens with smaller files, I now realize,  
provided they were not cached in the computer.

It is possible to disable the "metered connection" status from the  
wifi network properties (Windows 10 and Windows 11 settings interface,  
not control panel). Then OneDrive spins up again and Emacs is left  
waiting until the file is downloaded, with no error message.

Juanjo

Eli Zaretskii <eliz@gnu.org> escribió:

>> Date: Wed, 13 Jul 2022 10:22:09 +0200
>> From: Juan Jose Garcia Ripoll <juanjose.garcia.ripoll@csic.es>
>> Cc: larsi@gnus.org, 56499@debbugs.gnu.org
>>
>> Apologies for the noise. I have found out that this problem is  
>> related to "metered connections".
>
> Thanks.  I was about to tell that I couldn't reproduce the problem on
> a machine where I have OneDrive set up.
>
>> On those
>> ones, Emacs cannot trigger opening of the file and is not capable  
>> of returning the right error message. Once
>> I disable the option "metered connection" from the network, Emacs  
>> operates as expected. I suspect this
>> cannot really be attributed to Emacs, except for the lack of error  
>> message in the first situation.
>
> Would you mind please telling more about the situation where it
> happens and about the symptoms?  Does this happen only with
> find-file-literally? does it happen only with large files? and what
> exactly are "metered connections" and how does one disable that
> option?
>
> The reason for the questions is that I'd like to describe the problem
> and the solution in etc/PROBLEMS.
> Thanks.
   Juan José García Ripoll

Instituto de Física Fundamental, CSIC
Calle Serrano 113b, Madrid 28006, Spain
Phone: (+34) 915616800 (dial 943107 when hearing the voice)
http://quinfog.iff.csic.es / http://juanjose.garciaripoll.com

[-- Attachment #2: Mensaje HTML --]
[-- Type: text/html, Size: 3584 bytes --]

bug#56499: 28.1; Unable to open large file

[Eli Zaretskii]

> Date: Wed, 13 Jul 2022 18:11:04 +0200
> From: Juan Jose Garcia Ripoll <juanjose.garcia.ripoll@csic.es>
> Cc: larsi@gnus.org, 56499@debbugs.gnu.org
> 
> Metered connections are network connections that charge by volume of data transferred. That happens, for
> instance, with 4G modems, which I am using this week.
> 
> When one tethers a 4G mobile phone to a laptop, it is possible to select that the connection is metered, to
> avoid that Windows uses that link to download big but not so important data (e.g. updates).
> 
> I was unaware that OneDrive deactivates itself in metered connections even if I ask to synchronize data.
> When I opened a big file (the ones that OneDrive typically removes from the computer to save space and
> leaves in the cloud) Emacs got a system error that led to the behavior described before. It also happens with
> smaller files, I now realize, provided they were not cached in the computer.
> 
> It is possible to disable the "metered connection" status from the wifi network properties (Windows 10 and
> Windows 11 settings interface, not control panel). Then OneDrive spins up again and Emacs is left waiting
> until the file is downloaded, with no error message.

Thanks, I've now added an entry in PROBLEMS about this, and I'm
closing the bug.