SlideShare une entreprise Scribd logo
1  sur  4
Télécharger pour lire hors ligne
Something about DataStage, DataStage Administration, Job Designing,Developing, DataStage troubleshooting, DataStage Installation & Configuration, ETL, DataWareHousing, DB2,
Teradata, Oracle and Scripting.
Nuts & Bolts of DataStage
Home Interview Questions DataStage Scenarios Series Posts E­Books About Me !!
Friday, May 16, 2014
DataStage Scenario Problem ­­>  DataStage Scenario ­ Problem6
  
Solution Design :
a) Job Design :
Below is the design which can achieve the output as we needed. Here, we are reading seq file as a input, then data is passing through a
Transformer stage to achieve the output.
b) Transformer Stage Properties
Here, Create a new column in output which contains the Occurrence of characters
and their derivations are ­
Count :  Count(In_xfm.Name,'A')+Count(In_xfm.Name,'E')+Count(In_xfm.Name,'I')
 +Count(In_xfm.Name,'O')+Count(In_xfm.Name,'U')+Count(In_xfm.Name,'a')
+Count(In_xfm.Name,'e')+Count(In_xfm.Name,'i')+Count(In_xfm.Name,'o')
+Count(In_xfm.Name,'u')
DataStage Scenario ­ Design6 ­ job1
Total Pageviews
1 4 5 4 6 1 7
Search
Try Me
DataSet in DataStage
Issuing commands to a Queue Manager (runmqsc)
Hash Files in DataStage
XMeta DB : Datastage Repository
InfoSphere DataStage Jobstatus returned Codes from
dsjob
Conductor Node in Datastage
Schema File in Datastage
Sort stage to remove duplicate
14 Good design tips in Datastage
Datastage Coding Checklist
Must Reads
1   More    Next Blog» Create Blog   Sign In
DataStage
Scenario ­ Design8
­ job1
DataStage
Scenario ­
Design7­ job1
DataStage
Scenario ­ Design
2 ­ job1
DataStage
Scenario ­
Problem6
DataStage
Scenario ­ Design3
­ job1
Create a new column in output which contains the Occurrence of characters and assigned the StageVar.
c) OutPut File 
Now, Output file have the output
Name    Count
Priya Awasthi      5
Diya Singh        3
Sunil Verma        4
Rashmi Arya       4
Neha Tomar    4
Akash Aggrawal    5
Anil chahal        4
Kashish Patel     4
Rashid Patel        4
Gopal Joshi         4
For More ­­­> VISIT THIS LINK
Suggested Reading......
Linkwithin
By ETL DataStage at 08:46  0 Comments
Get daily dose of Tech Food
Email address... Submit
DataStage4You
111 have us in circles View all
Follow
tech foodies
▼  2014 (103)
►  October (7)
►  September (9)
►  August (5)
►  July (12)
►  June (10)
▼  May (13)
Surrogate Key Generator ­ Create/Update
State File...
Surrogate Key Generator ­ Create State
File
Some Oracle SQL should known by
Developer ­ 1
Interview Questions : DataWareHouse ­
Part6
DataStage Scenario ­ Design8 ­ job1
Data Warehouse Testing Checklist
DataStage Scenario ­ Design7­ job1
DataStage Scenario ­ Design6 ­ job1
What ETL is not?
Framework ( usually followed ) in ETL
Testing
Blog Archive
Newer Post Older PostHome
Subscribe to: Post Comments (Atom)
Labels: Code, DataSet, DataStage, design, develop, function, input, Job, output, problem, scenario, Seq File, transformer
0 Comments DataStage4You  Login
Sort by Best Share ⤤
Start the discussion…
Be the first to comment.
Subscribe✉ Add Disqus to your sited Privacy
Favorite ★
ETL Testing : Trends & Challenges
ETL Testing : Approach
DataStage Scenario ­ Problem19
►  April (10)
►  March (9)
►  February (16)
►  January (12)
►  2013 (167)
►  2012 (175)
►  2011 (8)
Administration 
application  authorities
client  Code  column
commands  Concept
Configuration 
create  Data  database  DataSet
DataStage  DataWareHouse  DB2  DBMS
debug  delete  design  develop
difference  director
Documentation  dsenv  dsjob  DSRPC 
environment Errors  ETL 
file 
function 
Information input  install  Interview 
Job  keys  Link Linux list 
Logging Logical  logs lookup 
managers  message  queue
Metadata Model  MQ  names
Optimizing  Oracle 
output  Parallel  parameter  partition
performance  Physical 
port  problem  process 
Project  Putty Questions 
remove 
Tags Cloud
&PH& 421  advantage Agents aggregator
Answers  architecture ASB attribute 
backup basic binary block books Buffer certification  change
channel  checkpoint  cleanup  clear 
Column  Generator  compiler 
Conceptual  conductor  container  copy
counter  Crontab 
deadlock  deploy 
dimension  Dimensional 
DSparam  dump
duplicate encrypt engine  exception
execution export fact factless  FAQ  FileSet filter free ftp
fun  fundamentals granularity  Guest hadoop handling
hash  head hide horizontal  Host huge  hyperlink  import  increase
index  issue  istool Java
jdbc  join  leaders  listener load
local locks  Login  macro mail
maintenance  memory  merge 
modify Monitor  MQSC multiple 
NLS  node  notes  notification  odbc  odbc.ini  operator
orchadmin  ORLogging  orphan  OS  osh
package  Parallelism 
password peek  Perl phantom  pivot
player  Practices  profile
programming  purge  read registry
reject  release  report  Resource  Restart  Roles
routine  rows 
scenario  Schema  Script 
Seq  File  sequence  Server  Service  Setting
Shell  shell  scripting 
sort source  SQL  stages
Start  Stop 
surrogate  table  target  teradata
tips  tool  transformer 
Troubleshoot  Tutorial  Unix User
Utility  UV  variables 
warnings  WAS  websphere
windows  XMETA 
row  generator  RTLogging  run  sample  SCD
scheduler  score  Scratch  section
session 
Share  shortcuts  show  slowly
snowflake solution  space  SSH 
Standards  Star  statistics  status  storage
switch system  tail  temporary 
time  trace  transformation  trigger
tuning  type unique 
uvodbc.config  version  videos  view
Vincent  McBurney  Virtual 
write Write Range Map  xml z/OS
The postings on this site are my own and don't necessarily represent IBM's or other companies positions, strategies or opinions. All content provided on this blog is for informational purposes only. The owner of this
blog makes no representations as to the accuracy or completeness of any information on this site or found by following any link on this site. The owner will not be liable for any errors or omissions in this information
nor for the availability of this information. The owner will not be liable for any losses, injuries, or damages from the display or use of his information. //­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­
Disclaimer
Did you find this Blog helpful ?? Let me know @ www.facebook.com/datastage4you. Ethereal template. Powered by Blogger.

Contenu connexe

Similaire à Data stage scenario design6 - job1

Data stage scenario design2 - job3
Data stage scenario   design2 - job3Data stage scenario   design2 - job3
Data stage scenario design2 - job3Naresh Bala
 
Data stage scenario design 2 - job1
Data stage scenario   design 2 - job1Data stage scenario   design 2 - job1
Data stage scenario design 2 - job1Naresh Bala
 
Best Practices for Building and Deploying Data Pipelines in Apache Spark
Best Practices for Building and Deploying Data Pipelines in Apache SparkBest Practices for Building and Deploying Data Pipelines in Apache Spark
Best Practices for Building and Deploying Data Pipelines in Apache SparkDatabricks
 
ETL and pivoting in spark
ETL and pivoting in sparkETL and pivoting in spark
ETL and pivoting in sparkSubhasish Guha
 
ETL and pivoting in spark
ETL and pivoting in sparkETL and pivoting in spark
ETL and pivoting in sparkSubhasish Guha
 
Rocky Nevin's presentation at eComm 2008
Rocky Nevin's presentation at eComm 2008Rocky Nevin's presentation at eComm 2008
Rocky Nevin's presentation at eComm 2008eComm2008
 
Spark Based Distributed Deep Learning Framework For Big Data Applications
Spark Based Distributed Deep Learning Framework For Big Data Applications Spark Based Distributed Deep Learning Framework For Big Data Applications
Spark Based Distributed Deep Learning Framework For Big Data Applications Humoyun Ahmedov
 
Avoiding Bad Database Surprises: Simulation and Scalability - Steven Lott
Avoiding Bad Database Surprises: Simulation and Scalability - Steven LottAvoiding Bad Database Surprises: Simulation and Scalability - Steven Lott
Avoiding Bad Database Surprises: Simulation and Scalability - Steven LottPyData
 
SQL Server 2000 Research Series - Transact SQL
SQL Server 2000 Research Series - Transact SQLSQL Server 2000 Research Series - Transact SQL
SQL Server 2000 Research Series - Transact SQLJerry Yang
 
Ml pipelines with Apache spark and Apache beam - Ottawa Reactive meetup Augus...
Ml pipelines with Apache spark and Apache beam - Ottawa Reactive meetup Augus...Ml pipelines with Apache spark and Apache beam - Ottawa Reactive meetup Augus...
Ml pipelines with Apache spark and Apache beam - Ottawa Reactive meetup Augus...Holden Karau
 
New Features of SQL Server 2016
New Features of SQL Server 2016New Features of SQL Server 2016
New Features of SQL Server 2016Mir Mahmood
 
Datastage Online Training in Hyderabad
Datastage Online Training in HyderabadDatastage Online Training in Hyderabad
Datastage Online Training in HyderabadUgs8008
 
Beyond Shuffling, Tips and Tricks for Scaling Apache Spark updated for Spark ...
Beyond Shuffling, Tips and Tricks for Scaling Apache Spark updated for Spark ...Beyond Shuffling, Tips and Tricks for Scaling Apache Spark updated for Spark ...
Beyond Shuffling, Tips and Tricks for Scaling Apache Spark updated for Spark ...Holden Karau
 
Quantitative Methods for Lawyers - Class #14 - R Boot Camp - Part 1 - Profess...
Quantitative Methods for Lawyers - Class #14 - R Boot Camp - Part 1 - Profess...Quantitative Methods for Lawyers - Class #14 - R Boot Camp - Part 1 - Profess...
Quantitative Methods for Lawyers - Class #14 - R Boot Camp - Part 1 - Profess...Daniel Katz
 
When to NoSQL and when to know SQL
When to NoSQL and when to know SQLWhen to NoSQL and when to know SQL
When to NoSQL and when to know SQLSimon Elliston Ball
 
Yufeng Guo | Coding the 7 steps of machine learning | Codemotion Madrid 2018
Yufeng Guo |  Coding the 7 steps of machine learning | Codemotion Madrid 2018 Yufeng Guo |  Coding the 7 steps of machine learning | Codemotion Madrid 2018
Yufeng Guo | Coding the 7 steps of machine learning | Codemotion Madrid 2018 Codemotion
 
Jump Start into Apache® Spark™ and Databricks
Jump Start into Apache® Spark™ and DatabricksJump Start into Apache® Spark™ and Databricks
Jump Start into Apache® Spark™ and DatabricksDatabricks
 
Graph Database Query Languages
Graph Database Query LanguagesGraph Database Query Languages
Graph Database Query LanguagesJay Coskey
 

Similaire à Data stage scenario design6 - job1 (20)

Data stage scenario design2 - job3
Data stage scenario   design2 - job3Data stage scenario   design2 - job3
Data stage scenario design2 - job3
 
Data stage scenario design 2 - job1
Data stage scenario   design 2 - job1Data stage scenario   design 2 - job1
Data stage scenario design 2 - job1
 
Best Practices for Building and Deploying Data Pipelines in Apache Spark
Best Practices for Building and Deploying Data Pipelines in Apache SparkBest Practices for Building and Deploying Data Pipelines in Apache Spark
Best Practices for Building and Deploying Data Pipelines in Apache Spark
 
ETL and pivoting in spark
ETL and pivoting in sparkETL and pivoting in spark
ETL and pivoting in spark
 
ETL and pivoting in spark
ETL and pivoting in sparkETL and pivoting in spark
ETL and pivoting in spark
 
Mssql to oracle
Mssql to oracleMssql to oracle
Mssql to oracle
 
Rocky Nevin's presentation at eComm 2008
Rocky Nevin's presentation at eComm 2008Rocky Nevin's presentation at eComm 2008
Rocky Nevin's presentation at eComm 2008
 
Spark Based Distributed Deep Learning Framework For Big Data Applications
Spark Based Distributed Deep Learning Framework For Big Data Applications Spark Based Distributed Deep Learning Framework For Big Data Applications
Spark Based Distributed Deep Learning Framework For Big Data Applications
 
Avoiding Bad Database Surprises: Simulation and Scalability - Steven Lott
Avoiding Bad Database Surprises: Simulation and Scalability - Steven LottAvoiding Bad Database Surprises: Simulation and Scalability - Steven Lott
Avoiding Bad Database Surprises: Simulation and Scalability - Steven Lott
 
SQL Server 2000 Research Series - Transact SQL
SQL Server 2000 Research Series - Transact SQLSQL Server 2000 Research Series - Transact SQL
SQL Server 2000 Research Series - Transact SQL
 
Ml pipelines with Apache spark and Apache beam - Ottawa Reactive meetup Augus...
Ml pipelines with Apache spark and Apache beam - Ottawa Reactive meetup Augus...Ml pipelines with Apache spark and Apache beam - Ottawa Reactive meetup Augus...
Ml pipelines with Apache spark and Apache beam - Ottawa Reactive meetup Augus...
 
New Features of SQL Server 2016
New Features of SQL Server 2016New Features of SQL Server 2016
New Features of SQL Server 2016
 
Datastage Online Training in Hyderabad
Datastage Online Training in HyderabadDatastage Online Training in Hyderabad
Datastage Online Training in Hyderabad
 
Beyond Shuffling, Tips and Tricks for Scaling Apache Spark updated for Spark ...
Beyond Shuffling, Tips and Tricks for Scaling Apache Spark updated for Spark ...Beyond Shuffling, Tips and Tricks for Scaling Apache Spark updated for Spark ...
Beyond Shuffling, Tips and Tricks for Scaling Apache Spark updated for Spark ...
 
Quantitative Methods for Lawyers - Class #14 - R Boot Camp - Part 1 - Profess...
Quantitative Methods for Lawyers - Class #14 - R Boot Camp - Part 1 - Profess...Quantitative Methods for Lawyers - Class #14 - R Boot Camp - Part 1 - Profess...
Quantitative Methods for Lawyers - Class #14 - R Boot Camp - Part 1 - Profess...
 
When to NoSQL and when to know SQL
When to NoSQL and when to know SQLWhen to NoSQL and when to know SQL
When to NoSQL and when to know SQL
 
My Master's Thesis
My Master's ThesisMy Master's Thesis
My Master's Thesis
 
Yufeng Guo | Coding the 7 steps of machine learning | Codemotion Madrid 2018
Yufeng Guo |  Coding the 7 steps of machine learning | Codemotion Madrid 2018 Yufeng Guo |  Coding the 7 steps of machine learning | Codemotion Madrid 2018
Yufeng Guo | Coding the 7 steps of machine learning | Codemotion Madrid 2018
 
Jump Start into Apache® Spark™ and Databricks
Jump Start into Apache® Spark™ and DatabricksJump Start into Apache® Spark™ and Databricks
Jump Start into Apache® Spark™ and Databricks
 
Graph Database Query Languages
Graph Database Query LanguagesGraph Database Query Languages
Graph Database Query Languages
 

Data stage scenario design6 - job1

  • 1. Something about DataStage, DataStage Administration, Job Designing,Developing, DataStage troubleshooting, DataStage Installation & Configuration, ETL, DataWareHousing, DB2, Teradata, Oracle and Scripting. Nuts & Bolts of DataStage Home Interview Questions DataStage Scenarios Series Posts E­Books About Me !! Friday, May 16, 2014 DataStage Scenario Problem ­­>  DataStage Scenario ­ Problem6    Solution Design : a) Job Design : Below is the design which can achieve the output as we needed. Here, we are reading seq file as a input, then data is passing through a Transformer stage to achieve the output. b) Transformer Stage Properties Here, Create a new column in output which contains the Occurrence of characters and their derivations are ­ Count :  Count(In_xfm.Name,'A')+Count(In_xfm.Name,'E')+Count(In_xfm.Name,'I')  +Count(In_xfm.Name,'O')+Count(In_xfm.Name,'U')+Count(In_xfm.Name,'a') +Count(In_xfm.Name,'e')+Count(In_xfm.Name,'i')+Count(In_xfm.Name,'o') +Count(In_xfm.Name,'u') DataStage Scenario ­ Design6 ­ job1 Total Pageviews 1 4 5 4 6 1 7 Search Try Me DataSet in DataStage Issuing commands to a Queue Manager (runmqsc) Hash Files in DataStage XMeta DB : Datastage Repository InfoSphere DataStage Jobstatus returned Codes from dsjob Conductor Node in Datastage Schema File in Datastage Sort stage to remove duplicate 14 Good design tips in Datastage Datastage Coding Checklist Must Reads 1   More    Next Blog» Create Blog   Sign In
  • 2. DataStage Scenario ­ Design8 ­ job1 DataStage Scenario ­ Design7­ job1 DataStage Scenario ­ Design 2 ­ job1 DataStage Scenario ­ Problem6 DataStage Scenario ­ Design3 ­ job1 Create a new column in output which contains the Occurrence of characters and assigned the StageVar. c) OutPut File  Now, Output file have the output Name    Count Priya Awasthi      5 Diya Singh        3 Sunil Verma        4 Rashmi Arya       4 Neha Tomar    4 Akash Aggrawal    5 Anil chahal        4 Kashish Patel     4 Rashid Patel        4 Gopal Joshi         4 For More ­­­> VISIT THIS LINK Suggested Reading...... Linkwithin By ETL DataStage at 08:46  0 Comments Get daily dose of Tech Food Email address... Submit DataStage4You 111 have us in circles View all Follow tech foodies ▼  2014 (103) ►  October (7) ►  September (9) ►  August (5) ►  July (12) ►  June (10) ▼  May (13) Surrogate Key Generator ­ Create/Update State File... Surrogate Key Generator ­ Create State File Some Oracle SQL should known by Developer ­ 1 Interview Questions : DataWareHouse ­ Part6 DataStage Scenario ­ Design8 ­ job1 Data Warehouse Testing Checklist DataStage Scenario ­ Design7­ job1 DataStage Scenario ­ Design6 ­ job1 What ETL is not? Framework ( usually followed ) in ETL Testing Blog Archive
  • 3. Newer Post Older PostHome Subscribe to: Post Comments (Atom) Labels: Code, DataSet, DataStage, design, develop, function, input, Job, output, problem, scenario, Seq File, transformer 0 Comments DataStage4You  Login Sort by Best Share ⤤ Start the discussion… Be the first to comment. Subscribe✉ Add Disqus to your sited Privacy Favorite ★ ETL Testing : Trends & Challenges ETL Testing : Approach DataStage Scenario ­ Problem19 ►  April (10) ►  March (9) ►  February (16) ►  January (12) ►  2013 (167) ►  2012 (175) ►  2011 (8) Administration  application  authorities client  Code  column commands  Concept Configuration  create  Data  database  DataSet DataStage  DataWareHouse  DB2  DBMS debug  delete  design  develop difference  director Documentation  dsenv  dsjob  DSRPC  environment Errors  ETL  file  function  Information input  install  Interview  Job  keys  Link Linux list  Logging Logical  logs lookup  managers  message  queue Metadata Model  MQ  names Optimizing  Oracle  output  Parallel  parameter  partition performance  Physical  port  problem  process  Project  Putty Questions  remove  Tags Cloud &PH& 421  advantage Agents aggregator Answers  architecture ASB attribute  backup basic binary block books Buffer certification  change channel  checkpoint  cleanup  clear  Column  Generator  compiler  Conceptual  conductor  container  copy counter  Crontab  deadlock  deploy  dimension  Dimensional  DSparam  dump duplicate encrypt engine  exception execution export fact factless  FAQ  FileSet filter free ftp fun  fundamentals granularity  Guest hadoop handling hash  head hide horizontal  Host huge  hyperlink  import  increase index  issue  istool Java jdbc  join  leaders  listener load local locks  Login  macro mail maintenance  memory  merge  modify Monitor  MQSC multiple  NLS  node  notes  notification  odbc  odbc.ini  operator orchadmin  ORLogging  orphan  OS  osh package  Parallelism  password peek  Perl phantom  pivot player  Practices  profile programming  purge  read registry reject  release  report  Resource  Restart  Roles
  • 4. routine  rows  scenario  Schema  Script  Seq  File  sequence  Server  Service  Setting Shell  shell  scripting  sort source  SQL  stages Start  Stop  surrogate  table  target  teradata tips  tool  transformer  Troubleshoot  Tutorial  Unix User Utility  UV  variables  warnings  WAS  websphere windows  XMETA  row  generator  RTLogging  run  sample  SCD scheduler  score  Scratch  section session  Share  shortcuts  show  slowly snowflake solution  space  SSH  Standards  Star  statistics  status  storage switch system  tail  temporary  time  trace  transformation  trigger tuning  type unique  uvodbc.config  version  videos  view Vincent  McBurney  Virtual  write Write Range Map  xml z/OS The postings on this site are my own and don't necessarily represent IBM's or other companies positions, strategies or opinions. All content provided on this blog is for informational purposes only. The owner of this blog makes no representations as to the accuracy or completeness of any information on this site or found by following any link on this site. The owner will not be liable for any errors or omissions in this information nor for the availability of this information. The owner will not be liable for any losses, injuries, or damages from the display or use of his information. //­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­ Disclaimer Did you find this Blog helpful ?? Let me know @ www.facebook.com/datastage4you. Ethereal template. Powered by Blogger.