Meandre Architecture Ws Apr 2009

SEASR:

Meandre: !
Semantic-Driven Data-Intensive !
Flows in the Clouds
Xavier Llora, Bernie Acs, Loretta Auvil, Boris Capitanu, Michael Welge, David Goldberg

National Center for Supercomputing Applications!
University of Illinois at Urbana-Champaign

{xllora, acs1, lauvil, capitanu, mwelge, deg}@illinois.edu
The SEASR project and its Meandre infrastructure!
are sponsored by The Andrew W. Mellon Foundation

SEASR: Design Goals
•  Transparency
–  From a single laptop to a HPC cluster
–  Not bound to a particular computation fabric

–  Allow heterogeneous development

•  Intuitive programming paradigm
–  Modular Components, Flows, and Reusable

–  Foster Collaboration and Sharing

•  Open Source
•  Service Orientated Architecture (SOA)

Meandre: Infrastructure
•  SEASR/Meandre Infrastructure:
–  Dataﬂow execution paradigm
–  Semantic-web driven
–  Web Oriented
–  Supports publishing services
–  Modular components
–  Encapsulation and execution mechanism
–  Promotes reuse, sharing, and collaboration


Meandre: Data Driven Execution
•  Execution Paradigms
–  Conventional programs perform computational tasks by
executing a sequence of instructions.
–  Data driven execution revolves around the idea of
applying transformation operations to a ﬂow or stream
of data when it is available.

•  Dataﬂow Approach
–  May have zero to many inputs
–  May have zero to many outputs
–  Performs a logical operation when data is available

Meandre: Dataﬂow Example

Value1
Sum
Value2


Meandre: Dataﬂow Example
•  Dataﬂow Addition Example
–  Logical Operation ‘+’
Value1
–  Requires two inputs
Sum
–  Produces one output
Value2

•  When two inputs are available
–  Logical operation can be preformed

–  Sum is output

•  When output is produced
–  Reset internal values

–  Wait for two new input values to become available

Meandre: The Dataﬂow Component
•  Data dictates component execution semantics

Inputs Outputs

Component

P

Descriptor in RDF! The component !
of its behavior
implementation

Meandre: Component Metadata
•  Describes a component
•  Separates:
–  Components semantics (black box)
–  Components implementation

•  Provides a uniﬁed framework:
–  Basic building blocks or units (components)
–  Complex tasks (ﬂows)
–  Standardized metadata


Meandre: Semantic Web Concepts
•  Relies on the usage of the resource description framework
(RDF) which uses simple notation to express graph relations
written usually as XML to provide a set of conventions and
common means to exchange information
•  Provides a common framework to share and reuse data
across application, enterprise, and community boundaries
•  Focuses on common formats for integration and combination
of data drawn from diverse sources
•  Pays special attention to the language used for recording how
the data relates to real world objects
•  Allows navigation to sets of data resources that are
semantically connected.

Meandre: Metadata Ontologies
•  Meandre's metadata relies on three ontologies:
–  The RDF ontology serves as a base for deﬁning
Meandre descriptors
–  The Dublin Core Elements ontology provides basic
publishing and descriptive capabilities in the description
of Meandre descriptors
–  The Meandre ontology describes a set of relationships
that model valid components, as understood by the
Meandre execution engine architecture


Meandre: Components in RDF
@prefix meandre: <http://www.meandre.org/ontology/> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
Existing!
@prefix dc: <http://purl.org/dc/elements/1.1/> .
Standards
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix : <#> .

<http://dita.ncsa.uiuc.edu/meandre/e2k/components/limited-iterations>
meandre:name quot;Limited iterationsquot;^^xsd:string ;
rdf:type meandre:executable_component ;
dc:creator quot;Xavier Lloraquot;^^xsd:string ;
dc:date quot;2007-11-17T00:32:35quot;^^xsd:date ;
dc:description quot;Allows only a limited number of
iterationsquot;^^xsd:string ;
dc:format quot;java/classquot;^^xsd:string ;
dc:rights quot;University of Illinois/NCSA Open Source
Licensequot;^^xsd:string ;
meandre:execution_context

<http://norma.ncsa.uiuc.edu/public-dav/Meandre/demos/E2K/V1/resources/
colt.jar> ,

<http://norma.ncsa.uiuc.edu/public-dav/Meandre/demos/E2K/V1/resources/
gacore.jar> ,

<http://dita.ncsa.uiuc.edu/meandre/e2k/components/limited-
iterations/implementation/> ,

<http://norma.ncsa.uiuc.edu/public-dav/Meandre/demos/E2K/V1/
resources/gacore-meandre.jar> ,
<http://norma.ncsa.uiuc.edu/public-dav/Meandre/demos/E2K/V1/

Meandre: Components Types
•  Components are the basic building block of any
computational task.

•  There are two kinds of Meandre components:
–  Executable components

•  Perform computational tasks that require no human
interactions during runtime

•  Processes are initialized during flow startup and are fired when
in accordance to the policies defined for it.

–  Control components

•  Used to pause dataflow during user interaction cycles

•  WebUI may be a HTML Form, Applet, or Other user interface

Meandre: Component Assemblies
•  Deﬁned by connecting outputs from one component to the
inputs of another.
–  Cyclical connections are supported

–  Components may have
•  Zero to many inputs

•  Zero to many output

•  Properties that control runtime behavior

•  Described using RDF
–  Enables storage, reuse, and sharing like components

–  Allows discovery and dynamic execution


Meandre: Flow (Complex Tasks)
•  A ﬂow is a collection of connected components

Read
Merge
P

P

Show
Get
P
P

Do
P

Dataflow execution

Meandre: Create, Publish, & Share
•  “Components” and “Flows” have RDF descriptors
–  Easily shared, fosters sharing, & reuse

–  Allow machines to read and interpret
–  Independent of the implementations

–  Combine different implementation & platforms

–  Components: Java, Python, Lisp, Web Services

–  Execution: On a Laptop or a High Performance Cluster

•  A “Location” is RDF descriptor of one to many
components, one to many ﬂows, and their
implementations


Meandre: Repository & Locations
•  Each location represents a set components/flows
•  Users can
–  Combine different locations together

–  Create components

–  Assemble flows

–  Share components and flows

•  Repositories Help
–  Administrate complex environments

–  Organize components and flows


Meandre: Metadata Properties
•  Components and Flows share properties such as
component name, creator, creation date, description, tags,
and rights.
•  Components specific metadata to describe the
components' behavior, it’s location, type of
implementation, firing policy, runnable, format, resource
location, and execution context
•  Flow specific metadata describes the directed graph of
components, components instances, connectors,
connector instance data port source, connector, instance
data port target, connector instance source, connector
instance target, instance name


A Little more on the Nuts & Bolts!

Programming Paradigm
• What does Meandre Execution Engine Do?

• What are the possible Component Scenarios

• Relationship between Engine and Components

• Data Driven Flow Creation (Workbench/Zigzag)

Meandre Server Prepares a Flow
•  The Meandre Server prepares Data Intensive Flow by reading the
RDF Descriptors component plumping
–  Executable Components and the connection the between them are
prepared by constructing a Queue mechanism which is use to aggregate
data available events.
•  Meandre provides each component an executing thread for processing.
•  Meandre manages logic queue represent by component connections in a ﬂow
•  Meandre activates component for initialization, data events, and termination
•  Meandre provides components with access to runtime resources.

Context A Queue Context B

Meandre Sever Relationship to Component

Meandre Server
Infrastructure Input & Ouput Data 
Deﬁnes Firing.Policy  Ports that require a 
logical queue to be 
ALL or ANY 
Component RDF Descriptor defines: managed by server 

Meandre  Server  Component  Meandre  Server 

Pull Inputs

Component 
Push Outputs

Meandre Server Flows & Connectors
Flows are made up of “One or More” components
Flows may contain connectors that
with “None to Many” connectors that are described
are cyclical over one or more
to the Mendre Server for management
components

Flows must contain at minimum one
component with NO Inputs to cause
an Execute call to be made.
*Outputs are Always Optional.

Flow components may have
multiple connectors assigned
to any input data port
Flows can have any number of components with
 “None to Many” Inputs data port s
 “None to Many” Output data ports

Meandre: Programming Paradigm

•  The programming paradigm creates complex
tasks by linking together a bunch of specialized
components. Meandre's publishing mechanism
allows components developed by third parties to
be assembled in a new ﬂow.
•  There are two ways to develop ﬂows :
–  Meandre’s Workbench visual programming tool
–  Meandre’s ZigZag scripting language


Meandre: Workbench Existing Flow

Components

Flows

Locations


Meandre: ZigZag Script Language
•  ZigZag is a simple language for describing data-
intensive flows
–  Modeled on Python for simplicity.
–  ZigZag is declarative language for expressing the
directed graphs that describe flows.

•  Command-line tools allow ZigZag files to compile
and execute.
–  A compiler is provided to transform a ZigZag program
(.zz) into Meandre archive unit (.mau).
–  Mau(s) can then be executed by a Meandre engine.

•  As an example the Flow Diagram
–  The ﬂow below pushes two strings that get concatenated and
printed to the console

– 


•  ZigZag code that represents example ﬂow:
#
# Imports the three required components and creates the component aliases
#
import <http://localhost:1714/public/services/demo_repository.rdf>
alias <http://test.org/component/push_string> as PUSH
alias <http://test.org/component/concatenate-strings> as CONCAT
alias <http://test.org/component/print-object> as PRINT
#
# Creates four instances for the flow
#
push_hello, push_world, concat, print = PUSH(), PUSH(), CONCAT(), PRINT()
#
# Sets up the properties of the instances
#
push_hello.message, push_world.message = quot;Hello quot;, quot;world!quot;
#
# Describes the data-intensive flow
#
@phres, @pwres = push_hello(), push_world()
@cres = concat( string_one: phres.string; string_two: pwres.string )
print( object: cres.concatenated_string )
#


•  Automatic Parallelization
–  Multiple instances of a component could be run in parallel to boost
throughput.

–  Specialized operator available in ZigZag Scripting to cause multiple
instances of a given component to used
•  Consider a simple ﬂow example show in the diagram

•  The dataﬂow declaration would look like
#
#
@pu = push()
@pt = pass( string:pu.string )
print( object:pt.string )

–  Adding the operator [+AUTO] to middle component
#
@pu = push()
@pt = pass( string:pu.string ) [+AUTO]
print( object:pt.string )

–  [+AUTO] tells the ZigZag compiler to parallelize the “pass
component instance” by the number of cores available on
system.
–  [+AUTO] may also be written [+N] where N is an numeric
value to use for example [+10].


–  Adding the operator [+4] would result in a directed grap

# Describes the data-intensive flow # Describes the data-intensive flow
# #
@pu = push() @pu = push()
@pt = pass( string:pu.string ) [+4] @pt = pass( string:pu.string ) [+4!]
print( object:pt.string ) print( object:pt.string )


Meandre: Flows to MAU
•  Flows can be executed using their RDF
descriptors
•  Flows can be compiled into MAU
•  MAU is:
–  Self-contained representation
–  Ready for execution
–  Portable
–  The base of ﬂow execution in grid environments


Meandre: The Architecture
•  The design of the Meandre architecture follows
three directives:
–  provide a robust and transparent scalable solution from
a laptop to large-scale clusters
–  create an uniﬁed solution for batch and interactive tasks
–  encourage reusing and sharing components

•  To ensure such goals, the designed architecture
relies on four stacked layers and builds on top of
service-oriented architectures (SOA)


Meandre: Basic Single Server


Meandre MDX: Cloud Computing
•  Servers can be
–  instantiated on demand
–  disposed when done or on demand

•  A cluster is formed by at least one server
•  The Meandre Distributed Exchange (MDX)
–  Orchestrates operational integrity by managing cluster
conﬁguration and membership using a shared database
resource.


Meandre MDX: The Picture
MDX Backbone 


Meandre MDX: The Architecture
•  Virtualization infrastructure
–  Provide a uniform access to the underlying execution environment.
It relies on virtualization of machines and the usage of Java for
hardware abstraction.

•  IO standardization
–  A uniﬁed layer provides access to shared data stores, distributed
ﬁle-system, specialized metadata stores, and access to other
service-oriented architecture gateways.


Meandre MDX: The Architecture
•  Data-intensive ﬂow infrastructure
–  Provide the basic Meandre execution engine for data-intensive
ﬂows, component repositories and discovery mechanisms,
extensible plugins and web user interfaces (webUIs).

•  Interaction layer
–  Can provide self-contained applications via webUIs, create plugins
for third-party services, interact with the embedding application
that relies on the Meandre engine, or provide services to the cloud.


Meandre Architecture Ws Apr 2009

Recommandé

Recommandé

Contenu connexe

En vedette

En vedette (6)

Similaire à Meandre Architecture Ws Apr 2009

Similaire à Meandre Architecture Ws Apr 2009 (20)

Plus de Loretta Auvil

Plus de Loretta Auvil (8)

Dernier

Dernier (20)

Meandre Architecture Ws Apr 2009