[prev in list] [next in list] [prev in thread] [next in thread] 

List:       beowulf
Subject:    RE: [Beowulf] NASTRAN on cluster
From:       "Lombard, David N" <david.n.lombard () intel ! com>
Date:       2005-04-12 13:57:37
Message-ID: 187D3A7CAB42A54DB61F1D05F012572205D6AD2F () orsmsx402 ! amr ! corp ! intel ! com
[Download RAW message or body]

From: Mark Hahn on Monday, April 11, 2005 9:40 PM
> 
> > > We just installed a small cluster and are running NASTRAN 2005 on
> > it...
> 
> (on a cluster of intel duals, and it doesn't seem to scale well
> when both cpus on a node are busy.)
> 
> > Nastran doesn't really want to run more than one job (MPI rank) per
> > node.
> 
> I bet that isn't true on dual-opterons.

Pretty much true.  Think disk I/O.

> > The distro can/will have a significant impact on allocatable memory.
> > Nastran uses brk(2) to allocate memory, so the TASK_UNMAPPED_BASE is
> > significant.
[...]
> 
> on ia32, TASK_UNMAPPED_BASE, by default, is at 1GB rather than ~1.3.
> easy to change, though, at least to give 2-2.5 GB on ia32.

It may *not* be easy to change, depending on the distro and glibc.  But,
if you do whatever work is needed, you can push the memory allocation up
to about 2.1-2.3 GiB.

> > I can't comment on SATA, but PATA disks are a really bad choice, as
they
> > require too much effort from the CPU to drive them--SCSI is MUCH
> > preferred in that case.
> 
> this is one of the longest-lived fallacies I've ever personally
> experienced.
> it was true 10+ years ago when PIO was the norm for ATA disks.
> busmastering
> has been the norm for PATA for a long while.

Benchmarks prove my point...  PATA disks are *horrible* for Nastran.
And, as I stated above, I don't have info on SATA.

> > As for CPU v. I/O.  The factors are (in no order):
> > 
> > fp performance
> > memory b/w
> > disk b/w
> > memory size
> > 
> > Which of the above dominates the analysis depends on the analysis.
> 
> for the reported symptoms (poor scaling when using the second
processor),
> the first doesn't fit.  memory bw certainly does, and is Intel's main
> weak spot right now.  then again, disk bw and memory size could also
fit
> the symptoms (since they're also resources shared on a dual), but would
be
> diagnosable by other means (namely, both would result in low %CPU
> utilization;
> the latter (thrashing) would be pretty obvious from swap
> traffic/utilization.)

Correct on possible causes, also noted above, but I don't recall any
information on other symptoms to identify a specific causality.

> like I said, I bet the problem is memory bandwidth.  mainly because I
just
> don't
> see programs waiting on disk that much anymore

Nastran can be one of those.  Think terabytes of I/O to tens of gigabyte
files.

> it does happen, but large
> writes these days will stream at 50+ MB/s, and reads are often cached.

50 MB/s?  That's *very* slow.

For example, if typically running large modal frequency analyses, you
should be providing >300 MB/s uncached on ia32 and >600 MB/s uncached on
IPF.

> I should mention that if HT is enabled on these duals, the problem
could be
> poor HT support in your kernels.  (HT creates two virtual processors
for
> each physical one.  if the scheduler treats HT-virtual processors as
real,
> you will get very poor speedup.  this would also be diagnosable by
simply
> running 'top' during a test.)

I'm fairly certain this was fixed in 2.6, but could be wrong.

To repeat, "which of [several factors] dominates the analysis depends on
the analysis".

-- 
David N. Lombard
 
My comments represent my opinions, not those of Intel Corporation.

_______________________________________________
Beowulf mailing list, Beowulf@beowulf.org
To change your subscription (digest mode or unsubscribe) visit \
http://www.beowulf.org/mailman/listinfo/beowulf


[prev in list] [next in list] [prev in thread] [next in thread]