[prev in list] [next in list] [prev in thread] [next in thread] 

List:       r-devel
Subject:    Re: [Rd] memory footprint of readRDS()
From:       Joris Meys <jorismeys () gmail ! com>
Date:       2018-09-18 16:23:34
Message-ID: CAO1zAVYKOwG36Sqj9F3as3pk_6x3xcmCTDdCr1WSwZxyqEin-A () mail ! gmail ! com
[Download RAW message or body]

Thx William and Brian for your swift responses, very insightful. I'll have
to hunt for more memory.

Cheers
Joris

On Tue, Sep 18, 2018 at 6:16 PM William Dunlap <wdunlap@tibco.com> wrote:

> The ratio of object size to rds file size depends on the object.  Some
> variation is due to how header information is stored in memory and in the
> file but I suspect most is due to how compression works (e.g., a vector of
> repeated values can be compressed into a smaller file than a bunch of
> random bytes).
>
> f <- function (data, ...)  {
>     force(data)
>     tf <- tempfile()
>     on.exit(unlink(tf))
>     save(data, file = tf)
>     c(`obj/file size` = as.numeric(object.size(data)/file.size(tf)))
> }
>
> > f(rep(0,1e6))
> obj/file size
>      1021.456
> > f(rep(0,1e6), compress=FALSE)
> obj/file size
>     0.9999986
> > f(rep(89.7,1e6))
> obj/file size
>      682.6555
> > f(log(1:1e6))
> obj/file size
>      1.309126
> > f(vector("list",1e6))
> obj/file size
>      2021.744
> > f(as.list(log(1:1e6)))
> obj/file size
>      8.907579
> > f(sample(as.raw(0:255),size=8e6,replace=TRUE))
> obj/file size
>     0.9998433
> > f(rep(as.raw(0:255),length=8e6))
> obj/file size
>      254.5595
> > f(as.character(1:1e6))
> obj/file size
>       23.5567
>
>
>
> Bill Dunlap
> TIBCO Software
> wdunlap tibco.com
>
> On Tue, Sep 18, 2018 at 8:28 AM, Joris Meys <jorismeys@gmail.com> wrote:
>
>> Dear all,
>>
>> I tried to read in a 3.8Gb RDS file on a computer with 16Gb available
>> memory. To my astonishment, the memory footprint of R rises quickly to
>> over
>> 13Gb and the attempt ends with an error that says "cannot allocate vector
>> of size 5.8Gb".
>>
>> I would expect that 3 times the memory would be enough to read in that
>> file, but apparently I was wrong. I checked the memory.limit() and that
>> one
>> gave me a value of more than 13Gb. So I wondered if this was to be
>> expected, or if there could be an underlying reason why this file doesn't
>> want to open.
>>
>> Thank you in advance
>> Joris
>>
>> --
>> Joris Meys
>> Statistical consultant
>>
>> Department of Data Analysis and Mathematical Modelling
>> Ghent University
>> Coupure Links 653, B-9000 Gent (Belgium)
>> <
>> https://maps.google.com/?q=Coupure+links+653,%C2%A0B-9000+Gent,%C2%A0Belgium&entry=gmail&source=g
>> >
>>
>> -----------
>> Biowiskundedagen 2017-2018
>> http://www.biowiskundedagen.ugent.be/
>>
>> -------------------------------
>> Disclaimer : http://helpdesk.ugent.be/e-maildisclaimer.php
>>
>>         [[alternative HTML version deleted]]
>>
>> ______________________________________________
>> R-devel@r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-devel
>>
>
>

-- 
Joris Meys
Statistical consultant

Department of Data Analysis and Mathematical Modelling
Ghent University
Coupure Links 653, B-9000 Gent (Belgium)
<https://maps.google.com/?q=Coupure+links+653,%C2%A0B-9000+Gent,%C2%A0Belgium&entry=gmail&source=g>

-----------
Biowiskundedagen 2017-2018
http://www.biowiskundedagen.ugent.be/

-------------------------------
Disclaimer : http://helpdesk.ugent.be/e-maildisclaimer.php

	[[alternative HTML version deleted]]

______________________________________________
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
[prev in list] [next in list] [prev in thread] [next in thread]