[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
One for Linux ... TERRA EARTH MODEL USES PGI'S PGF90 ON BEOWOLFCLUSTERS (fwd)
Subject: One for Linux ... TERRA EARTH MODEL USES PGI'S PGF90 ON BEOWOLF
CLUSTERS
TERRA EARTH MODEL USES PGI'S PGF90 ON BEOWOLF CLUSTERS
By Hans-Peter Bunge, Dept of Geosciences Princeton Univ.
==================================================================
Princeton, NJ -- The TERRA Earth model was originally developed by John
Baumgardner in 1985, and parallelized by Hans-Peter Bunge for RISC
workstation clusters in 1993. Since that time the model has been continuously
upgraded and enhanced, and is now widely used by geodynamicists.
A version of this model is a designated high performance target code for
the NASA HPCC Earth and Space Sciences Grand Challenge Project, due to its
excellent efficiency on parallel machines. Recently the model demonstrated
sustained performance of 100 GFLOPS on the 1024 processor NASA Goddard CRAY
T3E. Over the past few months, TERRA has been ported to dedicated high-speed
Beowulf clusters using the PGF90 Fortran 90 compiler from the Portland Group,
Inc. A Beowulf cluster is a parallel high-performance computing system
constructed using off-the-shelf PC hardware in a cluster configuration,
typically late-generation Intel Pentium II systems interconnected using fast
ethernet.
Terra is a finite element code defined on a nested geodesic grid, which
is derived from the regular icosahedron. The mesh allows an almost uniform
discretization of the sphere and avoids the pole-problem' of traditional
latitude-longitude grids (in such grids the narrow polar cells impose
unnecessarily small time-steps). An efficient multigrid solver computes the
velocity and pressure fields from equations of momentum and mass conservation
at every time step. TERRA's ability to handle strong lateral variations of
viscosity is especially important to model the deformation history of
silicate rocks, which can vary in strength by many orders of magnitude.
In the TERRA model, the primitive equations that describe the dynamics and
thermodynamics of the slow internal deformation of the earth are solved.
These equations are essentially the Navier-Stokes equations for a highly
viscous fluid. Because rocks deform very slowly (over many thousands of
years), it is appropriate to neglect the effects from inertia and rotation.
The slight variations in density, due to variations in pressure, temperature
and chemistry of rocks, give rise to buoyancy forces, which allow the Earth
interior to overturn slowly with time. Such overturns may have occurred as
often as 20 times during Earth's history, taking about 200 million years each
for their completion. They are the most effective mechanism to cool the
planet, and also provide the driving force to move tectonic plates and rift
the continents apart.
The parallel version of TERRA described here uses message-passing in
conjunction with Fortran 90. The computational domain is decomposed and
distributed across processors. Because communication is mostly local to
connect sub-domain boundaries, the parallel overhead shows a simple scaling
that goes with the surface-to-volume ratio of each sub-domain. Off-the-shelf
Intel processor-based PC hardware now has sufficient speed and memory to
handle relatively large sub-domains, such that the penalty from communication
is inconsequential. A parallel efficiency of more than 90 percent on
dedicated Beowulf clusters can be obtained at modest cost.
Porting TERRA to a Beowulf cluster turned out to be quite simple. PGF90
for Intel processor-based workstations proved to be the compiler of choice,
resulting in very fast machine code that significantly surpassed our
expectations. For example, we verified a sustained per-processor speed of
more than 150 MFLOPS on a single 450 Mhz Intel Pentium II. This per-processor
performance is comparable to some of the fastest RISC processors available
[table 1]. At 90 percent parallel efficiency, sustained performance of 10
GFLOPS is expected on the new 72 processor Pentium II Beowulf cluster under
construction at Princeton's Geosciences Department. Such high speed, but at
reasonable cost, is essential when attempting to perform large 3-D
simulations at the level of university departments.
<pre>
Hardware Processors MFLOPS/Processor Total MFLOPS
IBM SP2 32 310 9900
DEC ALPHA 32 210 6700
SGI Origin 32 160 5100
Pentium II 4 150 600
</pre>
Table 1: TERRA Performance on various microprocessor-based parallel systems
Equally important in our choice of PGF90 for Beowulf clusters was the
wide availability of Portland Group High Performance Fortran (PGHPF) on
high-end systems. With installations on 39 of the Top 100 systems worldwide,
and nearly 20% of the Top 500, PGHPF is effectively a de facto standard.
Using the PGI Fortran compilers has enabled our source-code to be ported
across platforms at minimal cost. In the case of TERRA, we have run identical
source-code on a dedicated 4 processor Pentium II cluster at the Department
of Numerical Modelisation at the Institute de Physique du Globe in Paris
using PGF90, as well as the 512 processor Edinburgh CRAY T3D using PGHPF
operating in F90 mode.
Presented here are simulations of the slow overturn of the Earth's mantle
(the 3000 km deep region between the surface and the molten iron core of the
Earth) at a snapshot in time. More than 10 million finite elements are needed
to represent this simulation. We show the temperature distribution using a
simple color scheme: Blue is cold and heavy, while red is hot and buoyant.
The upper 200 km of the Earth are removed, to gain an un-obscured view of the
Earth's interior. A simple reference model does not reproduce the geologic
observation of large oceanic and continental plates occasionally disconnected
by relatively few and elongated subduction zones (recall that the "Ring of
Fire" surrounding the Pacific ocean is powered by a prominent subduction
system, along which old oceanic plates sink back into the Earth's interior).
Instead, the surface sinks back into the interior along relatively dense and
point-like structures, quite unlike the plate tectonic style observed on
Earth.
A 'one-parameter' change from the reference model, making rocks of the
deep-Earth 30 times stronger than rocks of the shallower regions for all
depth levels below 670 km, results in a much more realistic simulation. Such
increase in the strength of deep-Earth rocks is strongly suggested by many
different geophysical observations. The one-parameter change results in a
convection structure that is radically different from the simple reference
model. The cold downwelling regions are now organized along a widely spaced
and interconnected system of downwelling lines, remarkably similar to the
pattern observed in subduction zones on Earth. The result demonstrates that
the deep Earth plays an important role in structuring the scale and pattern
of geologic motion at the surface. It also shows how geodynamicists can use
computer simulations to infer the secrets of the deep Earth interior.
-----
Hans-Peter Bunge is a researcher and faculty member in the Department of
Geosciences at Princeton University. He can be reached by email at
bunge@geo.princeton.edu.
Acknowledgement: This work was generously supported by the Institute of
Geophysics and Planetary Physics (IGPP) and the Advanced Computing Laboratory
(ACL) both at Los Alamos National Laboratory (LANL). All computer simulations
were performed at the ACL.
Note: an HTML version of this article, including graphics, can be found at
http://www.pgroup.com/p6_success/Terra.htm
--
To unsubscribe, send email to majordomo@silug.org with
"unsubscribe silug-discuss" in the body.