tramp: Maximum buffer size exceeded

tramp: Maximum buffer size exceeded

[Kin Cho]

Hi,

tramp (2.0.35) seems to have trouble with large file size.

I was trying to copy /large-file (210 Meg) from foobar to a local
directory.

Perhaps there is another tramp method which doesn't require first
fetching a remote file to a local temp file?

Thanks.

-kin

tramp: Fetching /scp:root@foobar:/large-file to tmp file /tmp/tramp.61036yh...done
tramp: Inserting local temp file `/tmp/tramp.61036yh'...
setq: Maximum buffer size exceeded

Re: tramp: Maximum buffer size exceeded

[Phillip Lord]

>>>>> "Kin" == Kin Cho <kin@neoscale.com> writes:

  Kin> Hi,

  Kin> tramp (2.0.35) seems to have trouble with large file size.

  Kin> I was trying to copy /large-file (210 Meg) from foobar to a
  Kin> local directory.

  Kin> Perhaps there is another tramp method which doesn't require
  Kin> first fetching a remote file to a local temp file?

  Kin> Thanks.

  Kin> -kin

  Kin> tramp: Fetching /scp:root@foobar:/large-file to tmp file
  Kin> /tmp/tramp.61036yh...done tramp: Inserting local temp file
  Kin> `/tmp/tramp.61036yh'...  setq: Maximum buffer size exceeded

 
This is not a problem with tramp per se, but with emacs. It has a
maximum buffer size (which is I think something like 2^26
characters). Your 210 meg file is too large for it. 

Try it locally. You would still have the same problem. 

There is essentially no way around this, other than bad hacks, like
splitting the file up into two. 

Phil

Re: tramp: Maximum buffer size exceeded

[Kin Cho]

Phillip Lord <p.lord@russet.org.uk> writes:

> >>>>> "Kin" == Kin Cho <kin@neoscale.com> writes:
> 
>   Kin> Hi,
> 
>   Kin> tramp (2.0.35) seems to have trouble with large file size.
> 
>   Kin> I was trying to copy /large-file (210 Meg) from foobar to a
>   Kin> local directory.
> 
>   Kin> Perhaps there is another tramp method which doesn't require
>   Kin> first fetching a remote file to a local temp file?
> 
>   Kin> Thanks.
> 
>   Kin> -kin
> 
>   Kin> tramp: Fetching /scp:root@foobar:/large-file to tmp file
>   Kin> /tmp/tramp.61036yh...done tramp: Inserting local temp file
>   Kin> `/tmp/tramp.61036yh'...  setq: Maximum buffer size exceeded
> 
>  
> This is not a problem with tramp per se, but with emacs. It has a
> maximum buffer size (which is I think something like 2^26
> characters). Your 210 meg file is too large for it. 

But scp is an out-of-band method, why does tramp has to process
the file content in emacs?

> Try it locally. You would still have the same problem. 

Manually using scp works fine.

-kin

> There is essentially no way around this, other than bad hacks, like
> splitting the file up into two. 
> 
> Phil

Re: tramp: Maximum buffer size exceeded

[Kevin Rodgers]

Kin Cho wrote:

> Phillip Lord <p.lord@russet.org.uk> writes:
>>>>>>>"Kin" == Kin Cho <kin@neoscale.com> writes:
>>  Kin> tramp (2.0.35) seems to have trouble with large file size.
>>
>>  Kin> I was trying to copy /large-file (210 Meg) from foobar to a
>>  Kin> local directory.
>>
>>  Kin> Perhaps there is another tramp method which doesn't require
>>  Kin> first fetching a remote file to a local temp file?
>>
>>  Kin> Thanks.
>>
>>  Kin> -kin
>>
>>  Kin> tramp: Fetching /scp:root@foobar:/large-file to tmp file
>>  Kin> /tmp/tramp.61036yh...done tramp: Inserting local temp file
>>  Kin> `/tmp/tramp.61036yh'...  setq: Maximum buffer size exceeded
>> 
>>This is not a problem with tramp per se, but with emacs. It has a
>>maximum buffer size (which is I think something like 2^26
>>characters). Your 210 meg file is too large for it. 
> 
> But scp is an out-of-band method, why does tramp has to process
> the file content in emacs?


It has to insert the file contents into a buffer so you can edit it.


>>Try it locally. You would still have the same problem. 
> 
> Manually using scp works fine.

The copy succeeded.  The failure occurred trying to insert the local

copy of the file into an Emacs buffer.

-- 
Kevin Rodgers

Re: tramp: Maximum buffer size exceeded

[Kin Cho]

Kevin Rodgers <ihs_4664@yahoo.com> writes:

> Kin Cho wrote:
> 
> > Phillip Lord <p.lord@russet.org.uk> writes:
> >>>>>>>"Kin" == Kin Cho <kin@neoscale.com> writes:
> >>  Kin> tramp (2.0.35) seems to have trouble with large file size.
> >>
> >>  Kin> I was trying to copy /large-file (210 Meg) from foobar to a
> >>  Kin> local directory.
> >>
> >>  Kin> Perhaps there is another tramp method which doesn't require
> >>  Kin> first fetching a remote file to a local temp file?
> >>
> >>  Kin> Thanks.
> >>
> >>  Kin> -kin
> >>
> >>  Kin> tramp: Fetching /scp:root@foobar:/large-file to tmp file
> >>  Kin> /tmp/tramp.61036yh...done tramp: Inserting local temp file
> >>  Kin> `/tmp/tramp.61036yh'...  setq: Maximum buffer size exceeded
> >> This is not a problem with tramp per se, but with emacs. It
> >> has a
> >>maximum buffer size (which is I think something like 2^26
> >> characters). Your 210 meg file is too large for it.
> > But scp is an out-of-band method, why does tramp has to process
> > the file content in emacs?
> 
> 
> It has to insert the file contents into a buffer so you can edit it.

I wasn't trying to edit the file.

I was copying /large-file (dwim style) from a (tramp) dired
buffer to a (non-tramp) dired buffer.

-kin

> 
> >> Try it locally. You would still have the same problem.
> > Manually using scp works fine.
> 
> The copy succeeded.  The failure occurred trying to insert the local
> 
> copy of the file into an Emacs buffer.
> 
> -- 
> Kevin Rodgers

Re: tramp: Maximum buffer size exceeded

[Kai Großjohann]

Kin Cho <kin@neoscale.com> writes:

> tramp (2.0.35) seems to have trouble with large file size.
>
> I was trying to copy /large-file (210 Meg) from foobar to a local
> directory.

It's a pity.  I didn't think of this optimization.  If you look at
the function tramp-do-copy-or-rename-file, then you will see that it
deals with the following special cases:

* Both are Tramp files.  In that case, I look whether they are from
  the same user/host/method combo.  If so, I invoke "mv"/"cp" on the
  remote system directly.  If not, see the last case.

* Neither are Tramp files.  Error.

* This is the default case: fetch file into a buffer, then save the
  buffer.

I should add another case which looks whether exactly one of them is
remote and the method is an out of band method.  In that case, I can
invoke rcp/scp/... directly.

Want to code this?
-- 
Two cafe au lait please, but without milk.

Re: tramp: Maximum buffer size exceeded

[Kin Cho]

kai.grossjohann@gmx.net (Kai Großjohann) writes:

> Kin Cho <kin@neoscale.com> writes:
> 
> > tramp (2.0.35) seems to have trouble with large file size.
> >
> > I was trying to copy /large-file (210 Meg) from foobar to a local
> > directory.
> 
> It's a pity.  I didn't think of this optimization.  If you look at
> the function tramp-do-copy-or-rename-file, then you will see that it
> deals with the following special cases:
> 
> * Both are Tramp files.  In that case, I look whether they are from
>   the same user/host/method combo.  If so, I invoke "mv"/"cp" on the
>   remote system directly.  If not, see the last case.
> 
> * Neither are Tramp files.  Error.
> 
> * This is the default case: fetch file into a buffer, then save the
>   buffer.
> 
> I should add another case which looks whether exactly one of them is
> remote and the method is an out of band method.  In that case, I can
> invoke rcp/scp/... directly.
> 
> Want to code this?

I'll take a shot at it.

However, since I don't copy these large files often (may be once
a month), and even then it's easy to manually type the scp
command anyway.  My sense of urgency to work on this will be low.

Speaking of urgency, one thing that comes to mind as being more
urgently needed is to optimize tramp startup connection time.

An idea I had was to cache the little things that tramp sends to
the remote host.  That way, tramp would startup much quicker the
next time it connects to the same host.

-kin