SlideShare une entreprise Scribd logo
1  sur  23
Télécharger pour lire hors ligne
Process Automation for
Hydrological Data Mapping
over GIS Software


By Rohan Jain (08AG1016)
Introduction
● Weather data is available from various
  organisations like IMD, CWC through their
  stations spanning all over the country,
  periodically.
● The data available from these places can be
  used for further processing.
● Processing is done via various GIS Software
  available.
● ArcGIS is one such popular software. It is
  used for this project
Introduction: Problem
● Data is not available in format ArcGIS support
● So it cannot be directly imported
● Manually importing 10s of thousands of data
  is not possible.
● Hence data needs to be automatically
  converted into an ArcGIS format.
● But again data from all the sources is not in a
  standardised format.
● So each data source needs special attention
Objectives
● Automatic conversion of existing
  hydrological data of Mahanadi river basin
  into a universal time-series format
● Mapping of the data into ArcHydro model of
  the ArcGIS software
Study Area: Description
● Mahandi river basin, located between
  longitudes 800 30' and 870 E, and latitudes
  190 21' and 230 35' N
● 4.3% of the total geographical area of India
● Mahanadi was notorious for its devastating
  floods.
● Hirakud Dam, one of the longest dams
  improved the situation greatly.
Mahanadi river
basin
Study Area: Data Available
● Data from India Meteorological Department
  and Central Water Commission (CWC)
● Rainfall data
● Escape Discharge data
● Water Level Data
● Data from remote sensing
Methodology: Requirements
● ArcGIS (Version 9.3)
● ArcHydro tools (Version 1.4) and ArcHydro
  data model
● Python Programming Language (Version >
  2.6)
● External Python Libraries
  ○ xlrd (for reading spreadsheets)
  ○ dbfpy (for writing dBase files)
Methodology: Study Material
● Book: ArcHydro - GIS for Water Resources
  by David R. Maidment[7]
● Book: Arc Hydro Tools - Tutorials
● GIS Course Content - University of Texas
● Web Resources, Lectures made available by
  ESRI[8] (ArcGIS Developer organisation)
Methodology
● For interfacing with ArcGIS dBase (*.dbf)
  database file format used
● dBase is a popular database and ArcGIS
  relies on it itself for storing data, so a good
  choice for using it for our task
● Python libraries available (dbfpy)
● For data model to store the time series, used
  the TimeSeries model from ArcHydro data
  models.
Methodology: Data Model
● FeatureID: ID of the feature for which this
  time series data exists. IMD Stations, CWC
  Gauges etc.
● TSTypeID: ID of the time series type. We
  have Precipitaion, Discharge, Water Level
  etc defined
● TSDateTime: The date and time of individual
  data
● TSValue: Individual data value
Methodology: Automation
1. The data obtained from various organisations
   is converted into a format which follows
   python data structures.
2. Separate (dBase) files contain information
   about HydroIDs (which will help find
   FeatureID). The information is extracted and
   used to find FeatureIDs for station names
3. Time Series is generated and then further
   published as dBase files for use with ArcGIS
   software.
The Data Conversion
Process
Methodology: Code Written
● Modules
  ○ These are for generic tasks which are applicable to
      all data sources
   ○ timeseries.py
     ■ Takes care of timeseries related internal tasks
      ■ Also generates the dBase files
   ○ stations.py:
     ■ Process the HydroIDs (FeatureIDs in Time
         Series database)
      ■ Fetches ID - Name info about the stations
Methodology: Code Written
● Individual Data Source Scripts
  ○ Since each data source provides information in a
    different format, they all need a separate script.
  ○ These scripts process the raw data to pythonic
    format and then generate time series database
● Written in Python Programming Language
● Total roughly 450 lines of python code
● A C/Java equivalent will easily measure 2-3
  times
Results
● Set up an initial project with correct directory
  hierarchy and install python + the required
  libraries
● Then, on execution of the scripts the time
  series files are generated automatically
● The time series files can then be imported
  into ArcGIS table
Results: Loading Data




Loading data
into a Time
Series table in
ArcCatalog
Result: Loading Data




ArcCatalog data loading dialogs
Result: Loading Data
                       Displaying data
                       after being
                       imported.
Result: Processing Data
                   Processing the
                   data in
                   ArcMap using
                   ArcHydro tools
Result: Processing Data




ArcMap Processing the Discharge Time Series
Future Work
● Rewrite the modules using Object Oriented
  Approach to improve the code quality and
  future additions of code easier
● Apart from this Rainfall, Discharge, Water
  Level series more data can be obtained and
  added
Thank You

Contenu connexe

Tendances

Introduction to Data Engineer and Data Pipeline at Credit OK
Introduction to Data Engineer and Data Pipeline at Credit OKIntroduction to Data Engineer and Data Pipeline at Credit OK
Introduction to Data Engineer and Data Pipeline at Credit OKKriangkrai Chaonithi
 
DataOps - The Foundation for Your Agile Data Architecture
DataOps - The Foundation for Your Agile Data ArchitectureDataOps - The Foundation for Your Agile Data Architecture
DataOps - The Foundation for Your Agile Data ArchitectureDATAVERSITY
 
The Future of Data Warehousing and Data Integration
The Future of Data Warehousing and Data IntegrationThe Future of Data Warehousing and Data Integration
The Future of Data Warehousing and Data IntegrationEric Kavanagh
 
The Apache Spark File Format Ecosystem
The Apache Spark File Format EcosystemThe Apache Spark File Format Ecosystem
The Apache Spark File Format EcosystemDatabricks
 
SAP-APO-SNP Training
SAP-APO-SNP TrainingSAP-APO-SNP Training
SAP-APO-SNP TrainingAJAY
 
Snowflake for Data Engineering
Snowflake for Data EngineeringSnowflake for Data Engineering
Snowflake for Data EngineeringHarald Erb
 
Washington DC DataOps Meetup -- Nov 2019
Washington DC DataOps Meetup   -- Nov 2019Washington DC DataOps Meetup   -- Nov 2019
Washington DC DataOps Meetup -- Nov 2019DataKitchen
 
Cross language information retrieval (clir)slide
Cross language information retrieval (clir)slideCross language information retrieval (clir)slide
Cross language information retrieval (clir)slideMohd Iqbal Al-farabi
 
Fire in the Sky: An Introduction to Monitoring Apache Spark in the Cloud with...
Fire in the Sky: An Introduction to Monitoring Apache Spark in the Cloud with...Fire in the Sky: An Introduction to Monitoring Apache Spark in the Cloud with...
Fire in the Sky: An Introduction to Monitoring Apache Spark in the Cloud with...Spark Summit
 
Webinar future dataintegration-datamesh-and-goldengatekafka
Webinar future dataintegration-datamesh-and-goldengatekafkaWebinar future dataintegration-datamesh-and-goldengatekafka
Webinar future dataintegration-datamesh-and-goldengatekafkaJeffrey T. Pollock
 
Data Lake: A simple introduction
Data Lake: A simple introductionData Lake: A simple introduction
Data Lake: A simple introductionIBM Analytics
 
Demystifying data engineering
Demystifying data engineeringDemystifying data engineering
Demystifying data engineeringThang Bui (Bob)
 
Practical introduction to hadoop
Practical introduction to hadoopPractical introduction to hadoop
Practical introduction to hadoopinside-BigData.com
 
SAP Extended ECM by OpenText 10.5 - What's New?
SAP Extended ECM by OpenText 10.5 - What's New?SAP Extended ECM by OpenText 10.5 - What's New?
SAP Extended ECM by OpenText 10.5 - What's New?Thomas Demmler
 
Data Lake Architecture
Data Lake ArchitectureData Lake Architecture
Data Lake ArchitectureDATAVERSITY
 
Dataiku Data Science Studio (datasheet)
Dataiku Data Science Studio (datasheet)Dataiku Data Science Studio (datasheet)
Dataiku Data Science Studio (datasheet)John Cann
 
Försäkringskassan: Neo4j as an Information Hub (GraphSummit Stockholm 2023)
Försäkringskassan: Neo4j as an Information Hub (GraphSummit Stockholm 2023)Försäkringskassan: Neo4j as an Information Hub (GraphSummit Stockholm 2023)
Försäkringskassan: Neo4j as an Information Hub (GraphSummit Stockholm 2023)Neo4j
 
Integrating Big Data Technologies
Integrating Big Data TechnologiesIntegrating Big Data Technologies
Integrating Big Data TechnologiesDATAVERSITY
 

Tendances (20)

Introduction to Data Engineer and Data Pipeline at Credit OK
Introduction to Data Engineer and Data Pipeline at Credit OKIntroduction to Data Engineer and Data Pipeline at Credit OK
Introduction to Data Engineer and Data Pipeline at Credit OK
 
DataOps - The Foundation for Your Agile Data Architecture
DataOps - The Foundation for Your Agile Data ArchitectureDataOps - The Foundation for Your Agile Data Architecture
DataOps - The Foundation for Your Agile Data Architecture
 
The Future of Data Warehousing and Data Integration
The Future of Data Warehousing and Data IntegrationThe Future of Data Warehousing and Data Integration
The Future of Data Warehousing and Data Integration
 
The Apache Spark File Format Ecosystem
The Apache Spark File Format EcosystemThe Apache Spark File Format Ecosystem
The Apache Spark File Format Ecosystem
 
SAP-APO-SNP Training
SAP-APO-SNP TrainingSAP-APO-SNP Training
SAP-APO-SNP Training
 
Snowflake for Data Engineering
Snowflake for Data EngineeringSnowflake for Data Engineering
Snowflake for Data Engineering
 
Washington DC DataOps Meetup -- Nov 2019
Washington DC DataOps Meetup   -- Nov 2019Washington DC DataOps Meetup   -- Nov 2019
Washington DC DataOps Meetup -- Nov 2019
 
Cross language information retrieval (clir)slide
Cross language information retrieval (clir)slideCross language information retrieval (clir)slide
Cross language information retrieval (clir)slide
 
Fire in the Sky: An Introduction to Monitoring Apache Spark in the Cloud with...
Fire in the Sky: An Introduction to Monitoring Apache Spark in the Cloud with...Fire in the Sky: An Introduction to Monitoring Apache Spark in the Cloud with...
Fire in the Sky: An Introduction to Monitoring Apache Spark in the Cloud with...
 
Webinar future dataintegration-datamesh-and-goldengatekafka
Webinar future dataintegration-datamesh-and-goldengatekafkaWebinar future dataintegration-datamesh-and-goldengatekafka
Webinar future dataintegration-datamesh-and-goldengatekafka
 
Data Lake: A simple introduction
Data Lake: A simple introductionData Lake: A simple introduction
Data Lake: A simple introduction
 
Demystifying data engineering
Demystifying data engineeringDemystifying data engineering
Demystifying data engineering
 
Practical introduction to hadoop
Practical introduction to hadoopPractical introduction to hadoop
Practical introduction to hadoop
 
Data engineering
Data engineeringData engineering
Data engineering
 
SAP Extended ECM by OpenText 10.5 - What's New?
SAP Extended ECM by OpenText 10.5 - What's New?SAP Extended ECM by OpenText 10.5 - What's New?
SAP Extended ECM by OpenText 10.5 - What's New?
 
Data Sharing with Snowflake
Data Sharing with SnowflakeData Sharing with Snowflake
Data Sharing with Snowflake
 
Data Lake Architecture
Data Lake ArchitectureData Lake Architecture
Data Lake Architecture
 
Dataiku Data Science Studio (datasheet)
Dataiku Data Science Studio (datasheet)Dataiku Data Science Studio (datasheet)
Dataiku Data Science Studio (datasheet)
 
Försäkringskassan: Neo4j as an Information Hub (GraphSummit Stockholm 2023)
Försäkringskassan: Neo4j as an Information Hub (GraphSummit Stockholm 2023)Försäkringskassan: Neo4j as an Information Hub (GraphSummit Stockholm 2023)
Försäkringskassan: Neo4j as an Information Hub (GraphSummit Stockholm 2023)
 
Integrating Big Data Technologies
Integrating Big Data TechnologiesIntegrating Big Data Technologies
Integrating Big Data Technologies
 

En vedette

JHydro - an implementation of the digital watershed
JHydro - an implementation of the digital watershedJHydro - an implementation of the digital watershed
JHydro - an implementation of the digital watershedsilli
 
Impacts of landuse change on sediment transport in the yali reservoir catchment
Impacts of landuse change on sediment transport in the yali reservoir catchmentImpacts of landuse change on sediment transport in the yali reservoir catchment
Impacts of landuse change on sediment transport in the yali reservoir catchmentCPWF Mekong
 
Climate Smart Landscape-Based Integrated Watershed Management: Experiences fr...
Climate Smart Landscape-Based Integrated Watershed Management: Experiences fr...Climate Smart Landscape-Based Integrated Watershed Management: Experiences fr...
Climate Smart Landscape-Based Integrated Watershed Management: Experiences fr...CIFOR-ICRAF
 
Thesis Presentation
Thesis PresentationThesis Presentation
Thesis Presentationgawump
 
Application of RS and GIS in Groundwater Prospects Zonation
Application of RS and GIS in Groundwater Prospects ZonationApplication of RS and GIS in Groundwater Prospects Zonation
Application of RS and GIS in Groundwater Prospects ZonationVishwanath Awati
 
Presentation on Aerosols, cloud properties
Presentation on Aerosols, cloud properties Presentation on Aerosols, cloud properties
Presentation on Aerosols, cloud properties Esayas Meresa
 
Analysis of runoff for vishwamitri river watershed using scs cn method and ge...
Analysis of runoff for vishwamitri river watershed using scs cn method and ge...Analysis of runoff for vishwamitri river watershed using scs cn method and ge...
Analysis of runoff for vishwamitri river watershed using scs cn method and ge...vishvam Pancholi
 
Soil Erosion for Vishwamitri River watershed using RS and GIS
Soil Erosion for Vishwamitri River watershed using RS and GISSoil Erosion for Vishwamitri River watershed using RS and GIS
Soil Erosion for Vishwamitri River watershed using RS and GISvishvam Pancholi
 
Presentation on remote sensing & gis and watershed copy
Presentation on remote sensing & gis and watershed   copyPresentation on remote sensing & gis and watershed   copy
Presentation on remote sensing & gis and watershed copydivya sahgal
 
REMOTE SENSING & GIS APPLICATIONS IN WATERSHED MANAGEMENT
REMOTE SENSING & GIS APPLICATIONS IN WATERSHED MANAGEMENT REMOTE SENSING & GIS APPLICATIONS IN WATERSHED MANAGEMENT
REMOTE SENSING & GIS APPLICATIONS IN WATERSHED MANAGEMENT Sumant Diwakar
 
Iirs overview -Remote sensing and GIS application in Water Resources Management
Iirs overview -Remote sensing and GIS application in Water Resources ManagementIirs overview -Remote sensing and GIS application in Water Resources Management
Iirs overview -Remote sensing and GIS application in Water Resources ManagementTushar Dholakia
 
APPLICATIONS OF REMOTE SENSING AND GIS IN WATERSHED MANAGEMENT
APPLICATIONS OF REMOTE SENSING AND GIS IN WATERSHED MANAGEMENTAPPLICATIONS OF REMOTE SENSING AND GIS IN WATERSHED MANAGEMENT
APPLICATIONS OF REMOTE SENSING AND GIS IN WATERSHED MANAGEMENTSriram Chakravarthy
 
Application of gis and remote sensing in agriculture
Application of gis and remote sensing in agricultureApplication of gis and remote sensing in agriculture
Application of gis and remote sensing in agricultureRehana Qureshi
 
Application of Remote Sensing in Agriculture
Application of Remote Sensing in AgricultureApplication of Remote Sensing in Agriculture
Application of Remote Sensing in AgricultureUTTAM KUMAR
 
Introduction to remote sensing and gis
Introduction to remote sensing and gisIntroduction to remote sensing and gis
Introduction to remote sensing and gisMohsin Siddique
 
Integration of the MODFLOW Lak7 package in the FREEWAT GIS modelling environment
Integration of the MODFLOW Lak7 package in the FREEWAT GIS modelling environmentIntegration of the MODFLOW Lak7 package in the FREEWAT GIS modelling environment
Integration of the MODFLOW Lak7 package in the FREEWAT GIS modelling environmentMassimiliano Cannata
 

En vedette (20)

JHydro - an implementation of the digital watershed
JHydro - an implementation of the digital watershedJHydro - an implementation of the digital watershed
JHydro - an implementation of the digital watershed
 
Jaysukh C Songara
Jaysukh C SongaraJaysukh C Songara
Jaysukh C Songara
 
Spatial Earth Profile2
Spatial Earth Profile2Spatial Earth Profile2
Spatial Earth Profile2
 
Impacts of landuse change on sediment transport in the yali reservoir catchment
Impacts of landuse change on sediment transport in the yali reservoir catchmentImpacts of landuse change on sediment transport in the yali reservoir catchment
Impacts of landuse change on sediment transport in the yali reservoir catchment
 
Climate Smart Landscape-Based Integrated Watershed Management: Experiences fr...
Climate Smart Landscape-Based Integrated Watershed Management: Experiences fr...Climate Smart Landscape-Based Integrated Watershed Management: Experiences fr...
Climate Smart Landscape-Based Integrated Watershed Management: Experiences fr...
 
Thesis Presentation
Thesis PresentationThesis Presentation
Thesis Presentation
 
Gnd water
Gnd waterGnd water
Gnd water
 
Application of RS and GIS in Groundwater Prospects Zonation
Application of RS and GIS in Groundwater Prospects ZonationApplication of RS and GIS in Groundwater Prospects Zonation
Application of RS and GIS in Groundwater Prospects Zonation
 
Intergrated Water Resourcses system
Intergrated Water Resourcses systemIntergrated Water Resourcses system
Intergrated Water Resourcses system
 
Presentation on Aerosols, cloud properties
Presentation on Aerosols, cloud properties Presentation on Aerosols, cloud properties
Presentation on Aerosols, cloud properties
 
Analysis of runoff for vishwamitri river watershed using scs cn method and ge...
Analysis of runoff for vishwamitri river watershed using scs cn method and ge...Analysis of runoff for vishwamitri river watershed using scs cn method and ge...
Analysis of runoff for vishwamitri river watershed using scs cn method and ge...
 
Soil Erosion for Vishwamitri River watershed using RS and GIS
Soil Erosion for Vishwamitri River watershed using RS and GISSoil Erosion for Vishwamitri River watershed using RS and GIS
Soil Erosion for Vishwamitri River watershed using RS and GIS
 
Presentation on remote sensing & gis and watershed copy
Presentation on remote sensing & gis and watershed   copyPresentation on remote sensing & gis and watershed   copy
Presentation on remote sensing & gis and watershed copy
 
REMOTE SENSING & GIS APPLICATIONS IN WATERSHED MANAGEMENT
REMOTE SENSING & GIS APPLICATIONS IN WATERSHED MANAGEMENT REMOTE SENSING & GIS APPLICATIONS IN WATERSHED MANAGEMENT
REMOTE SENSING & GIS APPLICATIONS IN WATERSHED MANAGEMENT
 
Iirs overview -Remote sensing and GIS application in Water Resources Management
Iirs overview -Remote sensing and GIS application in Water Resources ManagementIirs overview -Remote sensing and GIS application in Water Resources Management
Iirs overview -Remote sensing and GIS application in Water Resources Management
 
APPLICATIONS OF REMOTE SENSING AND GIS IN WATERSHED MANAGEMENT
APPLICATIONS OF REMOTE SENSING AND GIS IN WATERSHED MANAGEMENTAPPLICATIONS OF REMOTE SENSING AND GIS IN WATERSHED MANAGEMENT
APPLICATIONS OF REMOTE SENSING AND GIS IN WATERSHED MANAGEMENT
 
Application of gis and remote sensing in agriculture
Application of gis and remote sensing in agricultureApplication of gis and remote sensing in agriculture
Application of gis and remote sensing in agriculture
 
Application of Remote Sensing in Agriculture
Application of Remote Sensing in AgricultureApplication of Remote Sensing in Agriculture
Application of Remote Sensing in Agriculture
 
Introduction to remote sensing and gis
Introduction to remote sensing and gisIntroduction to remote sensing and gis
Introduction to remote sensing and gis
 
Integration of the MODFLOW Lak7 package in the FREEWAT GIS modelling environment
Integration of the MODFLOW Lak7 package in the FREEWAT GIS modelling environmentIntegration of the MODFLOW Lak7 package in the FREEWAT GIS modelling environment
Integration of the MODFLOW Lak7 package in the FREEWAT GIS modelling environment
 

Similaire à Btp presentation

Understanding Hadoop
Understanding HadoopUnderstanding Hadoop
Understanding HadoopAhmed Ossama
 
SF Big Analytics 20191112: How to performance-tune Spark applications in larg...
SF Big Analytics 20191112: How to performance-tune Spark applications in larg...SF Big Analytics 20191112: How to performance-tune Spark applications in larg...
SF Big Analytics 20191112: How to performance-tune Spark applications in larg...Chester Chen
 
11. From Hadoop to Spark 1:2
11. From Hadoop to Spark 1:211. From Hadoop to Spark 1:2
11. From Hadoop to Spark 1:2Fabio Fumarola
 
Analyzing Data at Scale with Apache Spark
Analyzing Data at Scale with Apache SparkAnalyzing Data at Scale with Apache Spark
Analyzing Data at Scale with Apache SparkNicola Ferraro
 
Hoodie: How (And Why) We built an analytical datastore on Spark
Hoodie: How (And Why) We built an analytical datastore on SparkHoodie: How (And Why) We built an analytical datastore on Spark
Hoodie: How (And Why) We built an analytical datastore on SparkVinoth Chandar
 
Incremental Processing on Large Analytical Datasets with Prasanna Rajaperumal...
Incremental Processing on Large Analytical Datasets with Prasanna Rajaperumal...Incremental Processing on Large Analytical Datasets with Prasanna Rajaperumal...
Incremental Processing on Large Analytical Datasets with Prasanna Rajaperumal...Databricks
 
Spark Meetup at Uber
Spark Meetup at UberSpark Meetup at Uber
Spark Meetup at UberDatabricks
 
Data Analytics and Machine Learning: From Node to Cluster on ARM64
Data Analytics and Machine Learning: From Node to Cluster on ARM64Data Analytics and Machine Learning: From Node to Cluster on ARM64
Data Analytics and Machine Learning: From Node to Cluster on ARM64Ganesh Raju
 
BKK16-404B Data Analytics and Machine Learning- from Node to Cluster
BKK16-404B Data Analytics and Machine Learning- from Node to ClusterBKK16-404B Data Analytics and Machine Learning- from Node to Cluster
BKK16-404B Data Analytics and Machine Learning- from Node to ClusterLinaro
 
BKK16-408B Data Analytics and Machine Learning From Node to Cluster
BKK16-408B Data Analytics and Machine Learning From Node to ClusterBKK16-408B Data Analytics and Machine Learning From Node to Cluster
BKK16-408B Data Analytics and Machine Learning From Node to ClusterLinaro
 
Streamsets and spark in Retail
Streamsets and spark in RetailStreamsets and spark in Retail
Streamsets and spark in RetailHari Shreedharan
 
Analytic Insights in Retail Using Apache Spark with Hari Shreedharan
Analytic Insights in Retail Using Apache Spark with Hari ShreedharanAnalytic Insights in Retail Using Apache Spark with Hari Shreedharan
Analytic Insights in Retail Using Apache Spark with Hari ShreedharanDatabricks
 
Spark Driven Big Data Analytics
Spark Driven Big Data AnalyticsSpark Driven Big Data Analytics
Spark Driven Big Data Analyticsinoshg
 
Dache: A Data Aware Caching for Big-Data using Map Reduce framework
Dache: A Data Aware Caching for Big-Data using Map Reduce frameworkDache: A Data Aware Caching for Big-Data using Map Reduce framework
Dache: A Data Aware Caching for Big-Data using Map Reduce frameworkSafir Shah
 

Similaire à Btp presentation (20)

Understanding Hadoop
Understanding HadoopUnderstanding Hadoop
Understanding Hadoop
 
Geospatial Data Abstraction Library (GDAL) Enhancement for ESDIS (GEE)
Geospatial Data Abstraction Library (GDAL) Enhancement for ESDIS (GEE)Geospatial Data Abstraction Library (GDAL) Enhancement for ESDIS (GEE)
Geospatial Data Abstraction Library (GDAL) Enhancement for ESDIS (GEE)
 
HDF and netCDF Data Support in ArcGIS
HDF and netCDF Data Support in ArcGISHDF and netCDF Data Support in ArcGIS
HDF and netCDF Data Support in ArcGIS
 
SF Big Analytics 20191112: How to performance-tune Spark applications in larg...
SF Big Analytics 20191112: How to performance-tune Spark applications in larg...SF Big Analytics 20191112: How to performance-tune Spark applications in larg...
SF Big Analytics 20191112: How to performance-tune Spark applications in larg...
 
HDF-EOS Data Product Developer's Guide
HDF-EOS Data Product Developer's GuideHDF-EOS Data Product Developer's Guide
HDF-EOS Data Product Developer's Guide
 
11. From Hadoop to Spark 1:2
11. From Hadoop to Spark 1:211. From Hadoop to Spark 1:2
11. From Hadoop to Spark 1:2
 
Analyzing Data at Scale with Apache Spark
Analyzing Data at Scale with Apache SparkAnalyzing Data at Scale with Apache Spark
Analyzing Data at Scale with Apache Spark
 
Hoodie: How (And Why) We built an analytical datastore on Spark
Hoodie: How (And Why) We built an analytical datastore on SparkHoodie: How (And Why) We built an analytical datastore on Spark
Hoodie: How (And Why) We built an analytical datastore on Spark
 
Incremental Processing on Large Analytical Datasets with Prasanna Rajaperumal...
Incremental Processing on Large Analytical Datasets with Prasanna Rajaperumal...Incremental Processing on Large Analytical Datasets with Prasanna Rajaperumal...
Incremental Processing on Large Analytical Datasets with Prasanna Rajaperumal...
 
Spark Meetup at Uber
Spark Meetup at UberSpark Meetup at Uber
Spark Meetup at Uber
 
design_doc
design_docdesign_doc
design_doc
 
Data Analytics and Machine Learning: From Node to Cluster on ARM64
Data Analytics and Machine Learning: From Node to Cluster on ARM64Data Analytics and Machine Learning: From Node to Cluster on ARM64
Data Analytics and Machine Learning: From Node to Cluster on ARM64
 
BKK16-404B Data Analytics and Machine Learning- from Node to Cluster
BKK16-404B Data Analytics and Machine Learning- from Node to ClusterBKK16-404B Data Analytics and Machine Learning- from Node to Cluster
BKK16-404B Data Analytics and Machine Learning- from Node to Cluster
 
BKK16-408B Data Analytics and Machine Learning From Node to Cluster
BKK16-408B Data Analytics and Machine Learning From Node to ClusterBKK16-408B Data Analytics and Machine Learning From Node to Cluster
BKK16-408B Data Analytics and Machine Learning From Node to Cluster
 
Streamsets and spark in Retail
Streamsets and spark in RetailStreamsets and spark in Retail
Streamsets and spark in Retail
 
Analytic Insights in Retail Using Apache Spark with Hari Shreedharan
Analytic Insights in Retail Using Apache Spark with Hari ShreedharanAnalytic Insights in Retail Using Apache Spark with Hari Shreedharan
Analytic Insights in Retail Using Apache Spark with Hari Shreedharan
 
Big Data Processing
Big Data ProcessingBig Data Processing
Big Data Processing
 
Spark Driven Big Data Analytics
Spark Driven Big Data AnalyticsSpark Driven Big Data Analytics
Spark Driven Big Data Analytics
 
Working with HDF and netCDF Data in ArcGIS: Tools and Case Studies
Working with HDF and netCDF Data in ArcGIS: Tools and Case StudiesWorking with HDF and netCDF Data in ArcGIS: Tools and Case Studies
Working with HDF and netCDF Data in ArcGIS: Tools and Case Studies
 
Dache: A Data Aware Caching for Big-Data using Map Reduce framework
Dache: A Data Aware Caching for Big-Data using Map Reduce frameworkDache: A Data Aware Caching for Big-Data using Map Reduce framework
Dache: A Data Aware Caching for Big-Data using Map Reduce framework
 

Dernier

Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps ScriptAutomating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Scriptwesley chun
 
presentation ICT roal in 21st century education
presentation ICT roal in 21st century educationpresentation ICT roal in 21st century education
presentation ICT roal in 21st century educationjfdjdjcjdnsjd
 
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsIAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsEnterprise Knowledge
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...Martijn de Jong
 
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024The Digital Insurer
 
Presentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreterPresentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreternaman860154
 
Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...Enterprise Knowledge
 
How to convert PDF to text with Nanonets
How to convert PDF to text with NanonetsHow to convert PDF to text with Nanonets
How to convert PDF to text with Nanonetsnaman860154
 
Evaluating the top large language models.pdf
Evaluating the top large language models.pdfEvaluating the top large language models.pdf
Evaluating the top large language models.pdfChristopherTHyatt
 
08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking MenDelhi Call girls
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationSafe Software
 
What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?Antenna Manufacturer Coco
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerThousandEyes
 
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEarley Information Science
 
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationScaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationRadu Cotescu
 
Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024The Digital Insurer
 
Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024The Digital Insurer
 
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024The Digital Insurer
 
CNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of ServiceCNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of Servicegiselly40
 
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...Igalia
 

Dernier (20)

Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps ScriptAutomating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Script
 
presentation ICT roal in 21st century education
presentation ICT roal in 21st century educationpresentation ICT roal in 21st century education
presentation ICT roal in 21st century education
 
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsIAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI Solutions
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...
 
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
 
Presentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreterPresentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreter
 
Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...
 
How to convert PDF to text with Nanonets
How to convert PDF to text with NanonetsHow to convert PDF to text with Nanonets
How to convert PDF to text with Nanonets
 
Evaluating the top large language models.pdf
Evaluating the top large language models.pdfEvaluating the top large language models.pdf
Evaluating the top large language models.pdf
 
08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
 
What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
 
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationScaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organization
 
Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024
 
Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024
 
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
 
CNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of ServiceCNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of Service
 
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
 

Btp presentation

  • 1. Process Automation for Hydrological Data Mapping over GIS Software By Rohan Jain (08AG1016)
  • 2. Introduction ● Weather data is available from various organisations like IMD, CWC through their stations spanning all over the country, periodically. ● The data available from these places can be used for further processing. ● Processing is done via various GIS Software available. ● ArcGIS is one such popular software. It is used for this project
  • 3. Introduction: Problem ● Data is not available in format ArcGIS support ● So it cannot be directly imported ● Manually importing 10s of thousands of data is not possible. ● Hence data needs to be automatically converted into an ArcGIS format. ● But again data from all the sources is not in a standardised format. ● So each data source needs special attention
  • 4. Objectives ● Automatic conversion of existing hydrological data of Mahanadi river basin into a universal time-series format ● Mapping of the data into ArcHydro model of the ArcGIS software
  • 5. Study Area: Description ● Mahandi river basin, located between longitudes 800 30' and 870 E, and latitudes 190 21' and 230 35' N ● 4.3% of the total geographical area of India ● Mahanadi was notorious for its devastating floods. ● Hirakud Dam, one of the longest dams improved the situation greatly.
  • 7. Study Area: Data Available ● Data from India Meteorological Department and Central Water Commission (CWC) ● Rainfall data ● Escape Discharge data ● Water Level Data ● Data from remote sensing
  • 8. Methodology: Requirements ● ArcGIS (Version 9.3) ● ArcHydro tools (Version 1.4) and ArcHydro data model ● Python Programming Language (Version > 2.6) ● External Python Libraries ○ xlrd (for reading spreadsheets) ○ dbfpy (for writing dBase files)
  • 9. Methodology: Study Material ● Book: ArcHydro - GIS for Water Resources by David R. Maidment[7] ● Book: Arc Hydro Tools - Tutorials ● GIS Course Content - University of Texas ● Web Resources, Lectures made available by ESRI[8] (ArcGIS Developer organisation)
  • 10. Methodology ● For interfacing with ArcGIS dBase (*.dbf) database file format used ● dBase is a popular database and ArcGIS relies on it itself for storing data, so a good choice for using it for our task ● Python libraries available (dbfpy) ● For data model to store the time series, used the TimeSeries model from ArcHydro data models.
  • 11. Methodology: Data Model ● FeatureID: ID of the feature for which this time series data exists. IMD Stations, CWC Gauges etc. ● TSTypeID: ID of the time series type. We have Precipitaion, Discharge, Water Level etc defined ● TSDateTime: The date and time of individual data ● TSValue: Individual data value
  • 12. Methodology: Automation 1. The data obtained from various organisations is converted into a format which follows python data structures. 2. Separate (dBase) files contain information about HydroIDs (which will help find FeatureID). The information is extracted and used to find FeatureIDs for station names 3. Time Series is generated and then further published as dBase files for use with ArcGIS software.
  • 14. Methodology: Code Written ● Modules ○ These are for generic tasks which are applicable to all data sources ○ timeseries.py ■ Takes care of timeseries related internal tasks ■ Also generates the dBase files ○ stations.py: ■ Process the HydroIDs (FeatureIDs in Time Series database) ■ Fetches ID - Name info about the stations
  • 15. Methodology: Code Written ● Individual Data Source Scripts ○ Since each data source provides information in a different format, they all need a separate script. ○ These scripts process the raw data to pythonic format and then generate time series database ● Written in Python Programming Language ● Total roughly 450 lines of python code ● A C/Java equivalent will easily measure 2-3 times
  • 16. Results ● Set up an initial project with correct directory hierarchy and install python + the required libraries ● Then, on execution of the scripts the time series files are generated automatically ● The time series files can then be imported into ArcGIS table
  • 17. Results: Loading Data Loading data into a Time Series table in ArcCatalog
  • 18. Result: Loading Data ArcCatalog data loading dialogs
  • 19. Result: Loading Data Displaying data after being imported.
  • 20. Result: Processing Data Processing the data in ArcMap using ArcHydro tools
  • 21. Result: Processing Data ArcMap Processing the Discharge Time Series
  • 22. Future Work ● Rewrite the modules using Object Oriented Approach to improve the code quality and future additions of code easier ● Apart from this Rainfall, Discharge, Water Level series more data can be obtained and added