[prev in list] [next in list] [prev in thread] [next in thread]
List: sas-l
Subject: Re: Can PC/SAS really take advantage of powerful new hardware?
From: "William W. Viergever" <william () VIERGEVER ! NET>
Date: 2012-07-31 20:52:54
Message-ID: 7FF7CE3AE7397343A8C4A39769A7C6AAC22A0F3A25 () EXVMBX018-11 ! exch018 ! msoutlookonline ! net
[Download RAW message or body]
more disk/spindles do help: e.g., OS on C:, swap (pagefile) on D:, and temp on J:, \
input data on another (G:, H:, I:, L:, or K:), output data to a disk you did not read \
from ... all this is better than having (e.g., as in a laptop) just one drive
fyi - i've been running a 2 processor (dual-core - Opteron 2220SE's) box since '07 \
... all hard drives are 15K rpm Seagate Cheetah SAS drives which has served me well \
but have begun to look for a new box and may go back to Intel
also, re. SSD's ... managed to score an OEM Samsung 256gb mSata card (this goes where \
a 3G wireless card would go) and spent the weekend tearing down my Thinkpad W520 \
(2.20Ghz quad code i7 cpu w/ 32gb ram) and installed the mSata to use as my boot \
drive (and programs) and kept the 500gb 7200 rpm drive for Data ... for now ... may \
get a 500gb Samsung 830 SSD when i can get some spare change
so far this puppy SCREAMS ... don't have apps/programs re-installed yet, but when i \
do will run some test jobs to compare against current desktop
p.s. the W520 also has 2 USB 3.0 ports, so have a couple of 1TB USB 3.0 external \
drives i can use to get back to my multi-spindle setup
--------------------------------------------------------------
William W. Viergever
Viergever & Associates
Health Data Analysis / Systems Design & Development
2920 Arden Way Suite N
Sacramento, CA 95825
william@viergever.net
www.viergever.net
(916) 483-8398
--------------------------------------------------------------
-----Original Message-----
From: SAS(r) Discussion [mailto:SAS-L@LISTSERV.UGA.EDU] On Behalf Of bbser 2009
Sent: Tuesday, July 31, 2012 11:11 AM
To: SAS-L@LISTSERV.UGA.EDU
Subject: Re: Can PC/SAS really take advantage of powerful new hardware?
Thanks Mark and Alan.
I understand fast IO will improve performance. But why does installing multiple hard \
disks improve performance too?
-----Original Message-----
From: SAS(r) Discussion [mailto:SAS-L@LISTSERV.UGA.EDU] On Behalf Of Mark Miller
Sent: July-31-12 1:54 PM
To: SAS-L@LISTSERV.UGA.EDU
Subject: Re: [SAS-L] Can PC/SAS really take advantage of powerful new hardware?
For the last two years I've been using SAS on big multi-core servers attached to EMC \
storage array. Servers are Windows Server 2008R 64-bit with 32-cores (actually 4 \
CPU Chips) with 256GB or 512GB ram
Big memory is truly wonderful - imagine
loading 100GB file into text editor for examination (Notebook++)
SAS in-core sort for 200GB file
running side-by-side comparison of two 100GB files
Multicore is also cool --
consider running multiple SAS jobs in parallel -- e.g. like 12-14
or running two simulataneous instances of Stata on 100GB data
But I would second Alan Churchill and focus on I/O, I/O, i/O.....
Successful use of big memory and multicore depends on moving data between disk and \
core.
If you're on a desktop unit you probably cannot get an EMC storage array, but you \
could get an SSD. On each of our servers, we have ~100-200 GB of SSD assigned as \
temporary storage (e.g. SAS WORK) It is very effective for most work, but since we \
occasionally have to process 800GB/1.5TB files, it doesn't help much.
When I was doing a lot of SAS processing on an XP system I found that performance was \
really improved by using multiple drives and even installing multiple disk \
controllers like an Adaptec for IDE and another Firewire controller. ( I commonly had \
2 internal drives and 5 external drives)
... Mark
On 2012-07-31 9:49 AM, Alan Churchill wrote:
> Not sure where the information on memory allocation came from but that
might
> have been true in 32 bit days. However, you can certainly use every
> ounce
of
> memory now in a hash object.
>
> Thanks,
>
> Alan
>
> Alan Churchill
> O: 719-687-5954
> C: 719-310-4870
>
> -----Original Message-----
> From: bbser 2009 [mailto:bbser2009@gmail.com]
> Sent: Tuesday, July 31, 2012 10:37 AM
> To: 'Alan Churchill'; SAS-L@LISTSERV.UGA.EDU
> Subject: RE: [SAS-L] Can PC/SAS really take advantage of powerful new
> hardware?
>
> Speaking of CPU and Memory, I read somewhere, for a four cores CPU PC,
> if the memory is 16GB say, then each CPU only has 4 GM memory. If this
> is
true,
> can I define a hash object to hold data set more than 4 GB in the
> physical memory?
>
> Regards, Max
> (Maaxx)
>
>
>
> -----Original Message-----
> From: SAS(r) Discussion [mailto:SAS-L@LISTSERV.UGA.EDU] On Behalf Of
> Alan Churchill
> Sent: July-31-12 11:44 AM
> To: SAS-L@LISTSERV.UGA.EDU
> Subject: Re: [SAS-L] Can PC/SAS really take advantage of powerful new
> hardware?
>
> PC SAS can handle large data volumes. 100GB is really not that large
> and
can
> be dealt with no issues. I just did a project in late 2011 that is
> much larger than your volumes with no issues.
>
> Your current specs are way underpowered. Consumer-level hardware can
> be
had
> now that is more powerful than many servers. 96GB of RAM, for example,
> is doable in a consumer PC.
>
> I am going to repeat some posters but 100% focus on I/O. I would
> recommend solid state drives. Look at the OCZ series. If you can swing
> a PCI/e
drive,
> go for it. Plan on around 5-10x as much space as your largest dataset
> and have a less expensive drive for archiving with loads of storage.
> More RAM means more hashing so get as much as you can afford.
>
> On your spec below, your RAM is way low. You can get 32GB for around
> $250
on
> Newegg. If you really want to beef the system up though go with a dual
> CPU MB and maximize the RAM.
>
> Thanks,
> Alan
>
> Alan Churchill
> O: 719-687-5954
> C: 719-310-4870
>
> -----Original Message-----
> From: Joe Matise [mailto:snoopy369@GMAIL.COM]
> Sent: Tuesday, July 31, 2012 5:51 AM
> Subject: Re: Can PC/SAS really take advantage of powerful new hardware?
>
> PC-SAS licensing isn't based on number of cores, that's only server SAS.
>
> -Joe
>
> On Tue, Jul 31, 2012 at 1:24 AM, .. <ajayohri@yahoo.com> wrote:
>
> > Hi Ya
> >
> > PC/SAS is not really designed for 100 Gb + data but some of the
> > newer HPC procs may help. You need to help wiith some more details on
> > the
> > data- is it numeric or text, structured enough to be in RDBMS or you
> > need to store in Hadoop/NoSQL
> >
> > Changing hardware- I am not sure but I think your SAS licensing was
> > based on number of cores- correct me if wrong?
> >
> > I have put queries on 150 Gb data (in 2004) using PC/SAS but that
> > involved the data in Oracle and using remote submit (additional
> > components licensing). Also Unix /SAS (is it there now?) can help
> >
> > The roughest and cheapest thing you can do is add more hard disk, put
> > a MySQL database and dump the data there, and aggregate/query using
> > SQL , and statistically whatever using PC/SAS
> >
> > What exactly are you doing?
> >
> > Rgds
> >
> > A
> >
> >
> >
> > ----- Original Message -----
> > > From: Joe Matise <snoopy369@GMAIL.COM>
> > > To: SAS-L@LISTSERV.UGA.EDU
> > > Cc:
> > > Sent: Tuesday, July 31, 2012 3:06 AM
> > > Subject: Re: Can PC/SAS really take advantage of powerful new hardware?
> > >
> > > What are you doing with your data? Just running through it? CPU
> > intensive
> > > PROCs? What's your main bottleneck?
> > >
> > > Odds are with 100GB+ of data, you're definitely hitting a disk
> > bottleneck.
> > > SSD instead of hard disk is a good start. More RAM than 8GB I'd say
> > > -
> > 8GB
> > > is probably starter level nowadays - 16 or more is a good idea, if
> > > you're doing PROCs that like to keep a lot of stuff in memory
> > > (though a SSD will help with things like PROC SORT, which you aren't
> > > going to have the 400GB needed to sort a 100GB dataset with). Work
> > > out what memory size your
> > PROCs
> > > need to do things in one pass in memory and that will help. For
> > > 4GB+
> > RAM
> > > you'll need 64 bit windows (XP or 7, doesn't really matter).
> > >
> > > -Joe
> > >
> > > On Mon, Jul 30, 2012 at 4:31 PM, Ya Huang <ya.huang@amylin.com> wrote:
> > >
> > > > Hi there,
> > > >
> > > > I've never really needed to process really huge data before, a few
> > > > GB
> > > at
> > > > most. So I was happy with the performance of my PC/SAS (XP/v9.2).
> > > > Sometimes, I have to wait for a few minutes to an hour, I can live
> > > > with it, since it doesn't happen too often. I recently started to
> > > process
> > > > really big data (100+GB, millions of records), as expected, I can't
> > > endure
> > > > the slowness anymore.
> > > >
> > > > The two options for me now: 1) maybe I can ask IT to somehow load
> > > > the
> > data
> > > > to our Oracle database, and let Oracle handle the basic query and
> > > > aggregation, then download to PC for some statistics. 2) I can ask
> > > > for a more powerful PC, since SAS server is not an option for now.
> > > >
> > > > I wonder how much difference a new more powerful hardware can
> > > > really
> > make?
> > > > Let's say, from the current environment of Core duo/XP/2GB RAM/V9.2
> > > > to Quad core/W7/8GB RAM/v9.2?
> > > >
> > > > Anyone has a rough idea?
> > > >
> > > > Thanks
> > > >
> > > > Ya
> > > >
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic