The Component Retrieval Language

SIMPLIFYING COMPLEX
SOFTWARE ASSEMBLY
THE COMPONENT RETRIEVAL LANGUAGE AND
IMPLEMENTATION

Presenter:
Eric Seidel
Dept. of Computer Science Co-authors:
Gabrielle Allen, Steven Brandt, Frank Löfﬂer, and Erik Schnetter
City College of New York Center for Computation & Technology
eric@eseidel.org Louisiana State University

Wednesday, August 4, 2010

COMPONENT FRAMEWORKS
• Set of individual software modules coordinated by glue
framework
• Each component (module) performs a speciﬁc task and
encapsulates a set of related functions data
• Frameworks can range from having a few components to many
• Components communicate via interfaces
• Used for various purposes, HPC examples include
• Cactus Framework
• CCA Frameworks (e.g. Caffeine)
• Domain speciﬁc frameworks (e.g. Earth System Modeling
Framework)

CACTUS
• Component Framework
• Over 500 unique components
• Distributed around the world
• Flesh
• Core application
• Thorns
• Independent modules
• Perform actual computation
• High Performance Computing www.cactuscode.org
• Massively parallel
• Runs on high end supercomputer clusters
• Supports many applications
• Numerical Relativity
• Quantum Gravity
• Computational Fluid Dynamics


CACTUS WORKFLOW

!REPOSITORY_TYPE pserver
• Managed using “Thornlists” !REPOSITORY_LOCATION cvs.cactuscode.org
!REPOSITORY_NAME /cactusdevcvs
• Plaintext list of thorns !REPOSITORY_USER eric9

required for a speciﬁc CactusBase/Boundary
conﬁguration CactusBase/CartGrid3D
CactusBase/CoordBase
• Used to checkout, update, CactusBase/IOASCII
CactusBase/IOBasic
build, and test the source CactusBase/IOUtil
CactusBase/InitBase
code CactusBase/LocalInterp


EINSTEIN TOOLKIT

• Toolkit for relativistic
astrophysical simulations
• Developed using Cactus
• Comprised of 135 thorns
• Initial Data, Evolution/
Analysis methods, Utilities
• First ofﬁcial release 2 months
ago www.einsteintoolkit.org


MOTIVATION

• Distributed Software Frameworks are hard to assemble and
manage
• Einstein Toolkit comprised of 135 individual components
• Very tedious to manually checkout or update
• Large barrier to entry for new users


VERSION CONTROL SYSTEMS
• Used to track revisions in source code
• Concurrent Versions System (cvs)
• Released in 1990
• Uses client-server model
• Server stores full history of repository
• Clients retrieve speciﬁc revision
• Subversion (svn)
• Successor to cvs
• Also uses client-server model
• Git
• Uses distributed model
• Everyone has copy of full history
http://en.wikipedia.org/wiki/
File:Revision_controlled_project_
visualization-2010-24-02.svg

GETCACTUS

• Designed to checkout and !REPOSITORY_TYPE pserver
!REPOSITORY_LOCATION cvs.cactuscode.org
update Cactus thorns and flesh !REPOSITORY_NAME /cactusdevcvs
!REPOSITORY_USER eric9
• Specific to Cactus Framework
CactusBase/Boundary
• Originally designed for CVS CactusBase/CartGrid3D
CactusBase/CoordBase
CactusBase/IOASCII
• SVN and git added later CactusBase/IOBasic
CactusBase/IOUtil
• Still difficult to distribute the CactusBase/InitBase
framework CactusBase/LocalInterp

• Users must edit the
thornlist


COMPONENT RETRIEVAL
LANGUAGE
# NAME is an alphanumeric or ’.’ character and

• Designed to fix problems
If
DOCUMENT : DIRECTIVES ; nent
error
with original GetCactus script
DIRECTIVE : DEFINE NAME ’=’ PATH EOL
| CHECKOUT ’=’ COMPONENTLIST EOL rors
| CHECKOUT ’=’ EOL COMPONENTLIST EOL inclu
| REPO_LOC ’=’ LOC EOL error

• Provides unified, tool agnostic
| AUTH_LOC ’=’ LOC EOL nent
| PATH_DIRECTIVE ’=’ PATH EOL and
# !REPO_PATH, !CHECKOUT, !TARGET,
M
syntax # !ANON_PASS, !NAME
| NAME_DIRECTIVE ’=’ NAME EOL
# !CRL_VERSION, !AUTH_USER,
whic
proc
# !ANON_USER, !TYPE spec
• Abstracts authentication ; load
the c
DIRECTIVES : DIRECTIVE

procedures | DIRECTIVES DIRECTIVE load
; tion,
perfo
LOC : PSERVER PATH # CVS repository a dis
• General-Purpose | NAME ’:’ ’/’ ’/’ PATH
| NAME ’@’ NAME ’:’ PATH
# Git/SVN repository
# Git repository
tory
with
;
histo

• No longer specific to PATH : NAME of di
| ’/’ NAME only
| PATH ’/’ NAME (som
Cactus ;

COMPONENTLIST : PATH
for e
115M
| COMPONENTLIST EOL PATH ; Ge
can e
All o
are p
Figure 2: Grammar for the CRL in Bison format comp

SAMPLE CRL FILE
!DEFINE ROOT = Cactus
!DEFINE ARR = $ROOT/arrangements

!TARGET = $ROOT !TARGET = $ARR
!TYPE = svn !TYPE = git
!AUTH_URL = https://svn.cactuscode.org/flesh/trunk !URL = git://github.com/ianhinder/Kranc.git
!URL = http://svn.cactuscode.org/flesh/trunk !AUTH_URL = git@github.com:ianhinder/Kranc.git
!CHECKOUT = Cactus !REPO_PATH= Auxiliary/Cactus
!NAME = . !CHECKOUT =
KrancNumericalTools/GenericFD
!TARGET = $ROOT
!TYPE = svn # McLachlan, the spacetime code
!URL = https://svn.cct.lsu.edu/repos/numrel/$1/trunk !TARGET = $ARR
!CHECKOUT = simfactory !TYPE = git
!URL = git://carpetcode.dyndns.org/McLachlan
!TARGET = $ARR !AUTH_URL = carpetgit@carpetcode.dyndns.org:McLachlan
!TYPE = svn !REPO_PATH= $2
!AUTH_URL = https://svn.cactuscode.org/arrangements/$1/$2/trunk !CHECKOUT = McLachlan/doc McLachlan/m McLachlan/par
!URL = http://svn.cactuscode.org/arrangements/$1/$2/trunk McLachlan/ML_BSSN
!CHECKOUT = McLachlan/ML_BSSN_Helper
CactusArchive/ADM McLachlan/ML_BSSN_O2
CactusBase/Boundary McLachlan/ML_BSSN_O2_Helper
CactusBase/CartGrid3D McLachlan/ML_BSSN_Test
CactusBase/CoordBase McLachlan/ML_ADMConstraints


GETCOMPONENTS

• Designed to be very modular Einstein Toolkit

Einstein Toolkit Core Einstein Toolkit

• Currently supports 5
svn.partnersite.org svn.einsteintoolkit.org
Cactus Flesh
and CCTK

version control systems and
svn.cactuscode.org

Tools, Parameter Files,
Carpet AMR
& Data
git.carpetcode.org

http/ftp downloads
svn.einsteintoolkit.org

• Very easy to add more Assemble
GetComponents
Simulation

• Can take input as local ﬁle or
URL Research Groups

Group Modules Individual Modules

• Manages all authentication
svn.groupthorns.org git.mythorns.org

Group Modules

issues ftp.groupthorns.org

./GetComponents http://tinyurl.com/einsteintoolkit-2010-06

AUTHENTICATION
Anonymous
mode
selected?

• Authentication handled entirely by yes no

VCS tools
Are components Is username

• GetComponents stores list of available
anonymously?
for URL
known?

authenticated repositories and no

users
yes
no
Prompt for
username yes

• Also tracks repositories with Use
anonymous
Print error
and ignore

speciﬁed anonymous access checkout component

Verify Use known

• Very secure
access username

• GetComponents never sees any Checkout
components

passwords!


CHECKOUT VS. UPDATE SPEED
1300

Serial Checkout Parallel Checkout

975
Time (seconds)

650

325

0
Abe Frost Kraken Lincoln LoneStar Longhorn Queen Bee Ranger Spur Steele
TeraGrid Resource


GETCOMPONENTS

• Generating component lists is still time-consuming and tedious
• Barrier/impossible for new users
• Don’t need all Einstein Toolkit modules to run a simulation
• How to determine which components are needed for a
particular simulation?
• e.g. what is needed to model two black holes, or a coastal
surge?


COMPONENT DEPENDENCIES

• Dependency tracking could
allow custom built simulations
• Specify one component IDScalarWaveC

containing data about the
simulation WaveToyC

• Initial values, type of Boundary CartGrid3D

simulation, etc
SymBase PUGH CoordBase
• Then recursively check
component dependencies


-- WAVETOY EXAMPLE
Interface Inheritance
Function Requirement
IDScalarWaveC WaveBinarySource LocalInterp LocalReduce Time Direct Thorn Dependency
Shared Variable Dependency
Capability Requirement

WaveToyC IsoSurfacer HTTPDEextra IOBasic IOAscii IOJpeg

Boundary CartGrid3D HTTPD IOUtil PUGHReduce PUGHSlab jpeg6b

SymBase CoordBase Socket PUGH


-- QUANTUM GRAVITY
Nerve PUGH Direct Thorn Dependency

AntichainEvol RandomAntichain CFlatSprinkle

BinaryCauset MonteCarlo Distributions

IOUtil RNGs


-- EINSTEIN TOOLKIT
ellbase idbrilldata
admmacros

idfileadm
httpd
time
idaxioddbrillbh

terminationtrigger
httpdextra

socket

localreduce ml_bssn_test
exact

aeilocalinterp
coordgauge
fortran Direct Thorn Dependency
idaxibrillbh
tatelliptic
idconstraintviolate

idlinearwaves
adm
formaline
iobasic ioutil
ml_bssn_helper
ellsor

iojpeg mol
localinterp
distortedbhivp idanalyticbh carpetevolutionmask newrad
tmunubase
admanalysis

calck
ahfinder ehfinder
iohdf5util
extract
carpetiobasic
ml_bssn_o2_helper
rotatingdbhivp
staticconformal

timerreport
admconstraints

tgrtensor
legoexcision weylscal4 quasilocalmeasures
ahfinderdirect
cartgrid3d admbase
ml_bssn
carpetioscalar

carpetioascii
boundary
carpetslab dissipation
multipole carpetmask
noexcision
slabtest
ml_bssn_o2
carpet genericfd

ml_admconstraints

twopunctures rotatingsymmetry90
iohdf5 noise spacemask
ioascii grhydro_initdata
carpetlib ml_admquantities
periodic
pugh grhydro

coordbase
sphericalsurface
pughslab meudon_bin_bh tovsolver

rotatingsymmetry180
carpetinterp
reflectionsymmetry
carpetregrid
carpetiohdf5 cartoon2d

hydro_initexcision symbase
meudon_mag_ns
hydro_analysis meudon_bin_ns
slab
jpeg6b
nice norms eosg_idealfluid

lapack

pughreduce admcoupling
eos_polytrope
carpetinterp2 carpetregrid2 eos_base
initbase gsl
carpetreduce loopcontrol
hydrobase summationbyparts

eos_idealfluid
hdf5
blas
nanchecker pughinterp carpettracker eosg_polytrope eos_hybrid
eosg_base constants
setmask_sphericalsurface lorene
eosg_hybrid


-- EINSTEIN TOOLKIT

ellbase idbrilldata
admmacros

idfileadm
httpd
idaxioddbrillbh time

terminationtrigger
httpdextra

socket

localreduce ml_bssn_test
exact

aeilocalinterp
coordgauge
fortran Direct Thorn Dependency
idaxibrillbh
tatelliptic
idconstraintviolate

idlinearwaves
adm
formaline
iobasic ioutil
ml_bssn_helper
ellsor

iojpeg mol
localinterp
distortedbhivp idanalyticbh carpetevolutionmask newrad
tmunubase
admanalysis

calck
ahfinder ehfinder
iohdf5util
extract
carpetiobasic
ml_bssn_o2_helper
rotatingdbhivp
staticconformal

timerreport
admconstraints

tgrtensor
legoexcision weylscal4 quasilocalmeasures
ahfinderdirect
cartgrid3d admbase
ml_bssn
carpetioscalar

carpetioascii
boundary
carpetslab dissipation
multipole carpetmask
noexcision
slabtest
ml_bssn_o2
carpet genericfd

ml_admconstraints

twopunctures rotatingsymmetry90
iohdf5 noise spacemask
ioascii grhydro_initdata
carpetlib ml_admquantities
periodic
pugh grhydro

coordbase
sphericalsurface
pughslab meudon_bin_bh tovsolver

rotatingsymmetry180
carpetinterp
reflectionsymmetry
carpetregrid
carpetiohdf5 cartoon2d

hydro_initexcision symbase
meudon_mag_ns
hydro_analysis meudon_bin_ns
slab
jpeg6b
nice norms eosg_idealfluid

lapack

pughreduce admcoupling
eos_polytrope
carpetinterp2 carpetregrid2 eos_base
initbase gsl
carpetreduce loopcontrol
hydrobase summationbyparts

eos_idealfluid
hdf5
pughinterp eosg_polytrope blas
nanchecker carpettracker eos_hybrid
eosg_base constants
setmask_sphericalsurface lorene
eosg_hybrid


DISTRIBUTION

• GetComponents is freely available with an open-source license
• www.eseidel.org/download/GetComponents
• Full documentation available
• ./GetComponents --man


ACKNOWLEDGEMENTS

• Many thanks to Gabrielle Allen, Steve Brandt, Frank Löfﬂer, and
Erik Schnetter


The Component Retrieval Language

Recommandé

Recommandé

Contenu connexe

Similaire à The Component Retrieval Language

Similaire à The Component Retrieval Language (20)

Dernier

Dernier (20)

The Component Retrieval Language